[yaml2obj][obj2yaml] - Teach tools to work with regular archives.

[grimar]

Summary

This teaches obj2yaml to dump valid regular (not thin) archives.
This also teaches yaml2obj to recognize archives YAML descriptions,
what allows to craft all different kinds of archives (valid and broken ones).

[grimar]

Now I am going to use this to drop precompiled binaries from https://github.com/llvm/llvm-project/blob/master/llvm/test/tools/llvm-objdump/malformed-archives.test
and to see how well this approach works. Hope to post a patch soon.

[jhenderson]

llvm/include/llvm/ObjectYAML/ArchiveYAML.h

26

I feel like Child or Member are more common terms for the parts of an archive. Probably best to pick one and use that? Same goes for the strings used in the YAML.

llvm/include/llvm/ObjectYAML/yaml2obj.h

53

Maybe rename to yaml2archive, since "arch" could be ambiguous (e.g. meaning "architecture").

llvm/lib/ObjectYAML/ArchiveYAML.cpp

24

I think the magic should be optional, and default to a regular archive's magic.

28–30

Could you explain why it doesn't get invoked? Presumably the equivalent functions for other formats are?

49

I'd like these to be optional too, with sensible defaults for the various fields. Having to specify them in most cases just generates noise.

50

This should be optional, with a default of no content.

llvm/test/tools/obj2yaml/Archives/regular.yaml

59–67

Can we omit the Entries here?

llvm/test/tools/yaml2obj/Archives/regular.yaml

4

7

llvm-ar options traditionally don't have -.

32

(fun fact: the ability to change the magic bytes arbitrarily allows someone to use yaml2obj as a kind of "echo" tool, by just specifying the arbitrary bytes to anything they want and omitting content and entries)

47

Maybe we don't need the "same size" case?

60–74

Rather than using od, I wonder if you could just use FileCheck directly on the archive, since the archive is a textual format?

# RUN: FileCheck --input-file=%t.two.a --check-prefix=TWO-DATA --strict-whitespace --match-full-lines --implicit-check-not={{.}}

#      TWO-DATA:!<arch>
# TWO-DATA-NEXT:bbbbbbbbbbbbbbb/1234567890abqwertyasdfgh876543217`
# TWO-DATA-NEXT:<some ASCII data>
...

I think the only change you'd need to make to the YAML is to use an ASCII character for the padding byte (e.g. 0x0a).

90

llvm/tools/obj2yaml/archive2yaml.cpp

18

Personal opinion: I don't see any benefit to final here, or even in general, since it prevents users extending things. I don't think that's useful.

29

50–56

If we make the header fields optional in the YAML, it's probably worth omitting them from obj2yaml's output.

61

This message maybe should include the Size field contents.

64

This should maybe list the expected size and the buffer size.

67

Is there testing for the Buffer.size() > Size aspect of this? In other words, showing that if the member ends at the end of the file, no padding.

[grimar]

• Addressed review comments.

llvm/lib/ObjectYAML/ArchiveYAML.cpp

28–30

Presumably the equivalent functions for other formats are?

No. Below is the piece of the code that calls mapping for the top level for each format:

void MappingTraits<YamlObjectFile>::mapping(IO &IO,
                                            YamlObjectFile &ObjectFile) {
  if (IO.outputting()) {
...
  } else {
    Input &In = (Input &)IO;
    if (IO.mapTag("!Arch")) {
      ObjectFile.Arch.reset(new ArchYAML::Archive());
      MappingTraits<ArchYAML::Archive>::mapping(IO, *ObjectFile.Arch);
    } else if (IO.mapTag("!ELF")) {
      ObjectFile.Elf.reset(new ELFYAML::Object());
      MappingTraits<ELFYAML::Object>::mapping(IO, *ObjectFile.Elf);
    } else if (IO.mapTag("!COFF")) {
      ObjectFile.Coff.reset(new COFFYAML::Object());
      MappingTraits<COFFYAML::Object>::mapping(IO, *ObjectFile.Coff);
    } else if (IO.mapTag("!mach-o")) {
      ObjectFile.MachO.reset(new MachOYAML::Object());
      MappingTraits<MachOYAML::Object>::mapping(IO, *ObjectFile.MachO);
    } else if (IO.mapTag("!fat-mach-o")) {
      ObjectFile.FatMachO.reset(new MachOYAML::UniversalBinary());
      MappingTraits<MachOYAML::UniversalBinary>::mapping(IO,
                                                         *ObjectFile.FatMachO);
    } else if (IO.mapTag("!minidump")) {
      ObjectFile.Minidump.reset(new MinidumpYAML::Object());
      MappingTraits<MinidumpYAML::Object>::mapping(IO, *ObjectFile.Minidump);
    } else if (IO.mapTag("!WASM")) {
      ObjectFile.Wasm.reset(new WasmYAML::Object());
      MappingTraits<WasmYAML::Object>::mapping(IO, *ObjectFile.Wasm);
    } else if (const Node *N = In.getCurrentNode()) {
      if (N->getRawTag().empty())
        IO.setError("YAML Object File missing document type tag!");
      else
        IO.setError("YAML Object File unsupported document type tag '" +
                    N->getRawTag() + "'!");
    }
  }
}

The code above just doesn't call validate. I think it just because it doesn't need it atm. E.g. when
we describe an ELF file, we can't specify both "Sections" and "Content" as ELF parser doesn't allow
us to set a content.

I've moved this piece of code to MappingTraits<YamlObjectFile>::mapping

llvm/test/tools/obj2yaml/Archives/regular.yaml

59–67

Now, when all fields are optional - yes.

llvm/test/tools/yaml2obj/Archives/regular.yaml

32

Not exactly. They still need to provide a file type first, e.g. "--- !Arch".

47

OK. I've removed the "same size" case.

60–74

Done.

llvm/tools/obj2yaml/archive2yaml.cpp

67

No. I've added the test.

[jhenderson]

llvm/test/tools/obj2yaml/Archives/regular.yaml

2

Possible additional test case: dumping the output when the header fields are empty, where such are "legal" (i.e. I think all but the name).

60

I think this comment better explains what's going on?

95

105

Looks like you could use '1' as the default here?

107

llvm/tools/obj2yaml/archive2yaml.cpp

38

I think it might be useful to include the offset of the bad header here. What do you think?

Same comment probably applies for most other messages below.

[grimar]

• Addressed review comments.

llvm/test/tools/obj2yaml/Archives/regular.yaml

2

Done.

i.e. I think all but the name

The Size value should be a valid integer to read the content. The name can be empty, but it`s the default value for it, so it is not dumped.

60

I think so.

llvm/tools/obj2yaml/archive2yaml.cpp

38

Done.

[jhenderson]

llvm/tools/obj2yaml/archive2yaml.cpp

39

I think the "at offset" bit is necessary, as otherwise it's not immediately clear what the number represents (it might be an index for example). Same applies below.

[grimar]

• Addressed review comment.

[jhenderson]

LGTM!

llvm/include/llvm/ObjectYAML/ArchiveYAML.h

+//===- ArchiveYAML.h - Archive YAMLIO implementation ------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file declares classes for handling the YAML representation of archives.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_OBJECTYAML_ARCHIVEYAML_H
+#define LLVM_OBJECTYAML_ARCHIVEYAML_H
+
+#include "llvm/Support/YAMLTraits.h"
+#include "llvm/ObjectYAML/YAML.h"
+#include "llvm/ADT/MapVector.h"
+
+namespace llvm {
+namespace ArchYAML {
+
+struct Archive {
+  struct Child {
+    struct Field {
+      Field() = default;
+      Field(StringRef Default, unsigned Length)
+          : DefaultValue(Default), MaxLength(Length) {}
+      StringRef Value;
+      StringRef DefaultValue;
+      unsigned MaxLength;
+    };
+
+    Child() {
+      Fields["Name"] = {"", 16};
+      Fields["LastModified"] = {"0", 12};
+      Fields["UID"] = {"0", 6};
+      Fields["GID"] = {"0", 6};
+      Fields["AccessMode"] = {"0", 8};
+      Fields["Size"] = {"0", 10};
+      Fields["Terminator"] = {"`\n", 2};
+    }
+
+    MapVector<StringRef, Field> Fields;
+
+    Optional<yaml::BinaryRef> Content;
+    Optional<llvm::yaml::Hex8> PaddingByte;
+  };
+
+  StringRef Magic;
+  Optional<std::vector<Child>> Members;
+  Optional<yaml::BinaryRef> Content;
+};
+
+} // end namespace ArchYAML
+} // end namespace llvm
+
+LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::ArchYAML::Archive::Child)
+
+namespace llvm {
+namespace yaml {
+
+template <> struct MappingTraits<ArchYAML::Archive> {
+  static void mapping(IO &IO, ArchYAML::Archive &A);
+  static std::string validate(IO &, ArchYAML::Archive &A);
+};
+
+template <> struct MappingTraits<ArchYAML::Archive::Child> {
+  static void mapping(IO &IO, ArchYAML::Archive::Child &C);
+  static std::string validate(IO &, ArchYAML::Archive::Child &C);
+};
+
+} // end namespace yaml
+} // end namespace llvm
+
+#endif // LLVM_OBJECTYAML_ARCHIVEYAML_H

llvm/include/llvm/ObjectYAML/ObjectYAML.h

 //===- ObjectYAML.h ---------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_OBJECTYAML_OBJECTYAML_H
 #define LLVM_OBJECTYAML_OBJECTYAML_H
 
+#include "llvm/ObjectYAML/ArchiveYAML.h"
 #include "llvm/ObjectYAML/COFFYAML.h"
 #include "llvm/ObjectYAML/ELFYAML.h"
 #include "llvm/ObjectYAML/MachOYAML.h"
 #include "llvm/ObjectYAML/MinidumpYAML.h"
 #include "llvm/ObjectYAML/WasmYAML.h"
 #include "llvm/Support/YAMLTraits.h"
 #include <memory>
 
 namespace llvm {
 namespace yaml {
 
 class IO;
 
 struct YamlObjectFile {
+  std::unique_ptr<ArchYAML::Archive> Arch;
   std::unique_ptr<ELFYAML::Object> Elf;
   std::unique_ptr<COFFYAML::Object> Coff;
   std::unique_ptr<MachOYAML::Object> MachO;
   std::unique_ptr<MachOYAML::UniversalBinary> FatMachO;
   std::unique_ptr<MinidumpYAML::Object> Minidump;
   std::unique_ptr<WasmYAML::Object> Wasm;
 };
 
 template <> struct MappingTraits<YamlObjectFile> {
   static void mapping(IO &IO, YamlObjectFile &ObjectFile);
 };
 
 } // end namespace yaml
 } // end namespace llvm
 
 #endif // LLVM_OBJECTYAML_OBJECTYAML_H

llvm/include/llvm/ObjectYAML/yaml2obj.h

 namespace MinidumpYAML {
 struct Object;
 }
 
 namespace WasmYAML {
 struct Object;
 }
 
+namespace ArchYAML {
+struct Archive;
+}
+
 namespace yaml {
 class Input;
 struct YamlObjectFile;
 
 using ErrorHandler = llvm::function_ref<void(const Twine &Msg)>;
 
+bool yaml2archive(ArchYAML::Archive &Doc, raw_ostream &Out, ErrorHandler EH);
 bool yaml2coff(COFFYAML::Object &Doc, raw_ostream &Out, ErrorHandler EH);
 bool yaml2elf(ELFYAML::Object &Doc, raw_ostream &Out, ErrorHandler EH,
               uint64_t MaxSize);
 bool yaml2macho(YamlObjectFile &Doc, raw_ostream &Out, ErrorHandler EH);
 bool yaml2minidump(MinidumpYAML::Object &Doc, raw_ostream &Out,
                    ErrorHandler EH);
 bool yaml2wasm(WasmYAML::Object &Doc, raw_ostream &Out, ErrorHandler EH);
 

llvm/lib/ObjectYAML/ArchiveEmitter.cpp

+//===- ArchiveEmitter.cpp ---------------------------- --------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ObjectYAML/ArchiveYAML.h"
+#include "llvm/ObjectYAML/yaml2obj.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace llvm;
+using namespace ArchYAML;
+
+namespace llvm {
+namespace yaml {
+
+bool yaml2archive(ArchYAML::Archive &Doc, raw_ostream &Out, ErrorHandler EH) {
+  Out.write(Doc.Magic.data(), Doc.Magic.size());
+
+  if (Doc.Content) {
+    Doc.Content->writeAsBinary(Out);
+    return true;
+  }
+
+  if (!Doc.Members)
+    return true;
+
+  auto WriteField = [&](StringRef Field, uint8_t Size) {
+    Out.write(Field.data(), Field.size());
+    for (size_t I = Field.size(); I != Size; ++I)
+      Out.write(' ');
+  };
+
+  for (const Archive::Child &C : *Doc.Members) {
+    for (auto &P : C.Fields)
+      WriteField(P.second.Value, P.second.MaxLength);
+
+    if (C.Content)
+      C.Content->writeAsBinary(Out);
+    if (C.PaddingByte)
+      Out.write(*C.PaddingByte);
+  }
+
+  return true;
+}
+
+} // namespace yaml
+} // namespace llvm

llvm/lib/ObjectYAML/ArchiveYAML.cpp

+//===- ArchiveYAML.cpp - ELF YAMLIO implementation -------------------- ----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines classes for handling the YAML representation of archives.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ObjectYAML/ArchiveYAML.h"
+
+namespace llvm {
+
+namespace yaml {
+
+void MappingTraits<ArchYAML::Archive>::mapping(IO &IO, ArchYAML::Archive &A) {
+  assert(!IO.getContext() && "The IO context is initialized already");
+  IO.setContext(&A);
+  IO.mapTag("!Arch", true);
+  IO.mapOptional("Magic", A.Magic, "!<arch>\n");
+  IO.mapOptional("Members", A.Members);
+  IO.mapOptional("Content", A.Content);
+  IO.setContext(nullptr);
+}
+
+std::string MappingTraits<ArchYAML::Archive>::validate(IO &,
+                                                       ArchYAML::Archive &A) {
+  if (A.Members && A.Content)
+    return "\"Content\" and \"Members\" cannot be used together";
+  return "";
+}
+
+void MappingTraits<ArchYAML::Archive::Child>::mapping(
+    IO &IO, ArchYAML::Archive::Child &E) {
+  assert(IO.getContext() && "The IO context is not initialized");
+  for (auto &P : E.Fields)
+    IO.mapOptional(P.first.data(), P.second.Value, P.second.DefaultValue);
+  IO.mapOptional("Content", E.Content);
+  IO.mapOptional("PaddingByte", E.PaddingByte);
+}
+
+std::string
+MappingTraits<ArchYAML::Archive::Child>::validate(IO &,
+                                                  ArchYAML::Archive::Child &C) {
+  for (auto &P : C.Fields)
+    if (P.second.Value.size() > P.second.MaxLength)
+      return ("the maximum length of \"" + P.first + "\" field is " +
+              Twine(P.second.MaxLength))
+          .str();
+  return "";
+}
+
+} // end namespace yaml
+
+} // end namespace llvm

llvm/lib/ObjectYAML/CMakeLists.txt

 add_llvm_component_library(LLVMObjectYAML
+  ArchiveEmitter.cpp
+  ArchiveYAML.cpp
   CodeViewYAMLDebugSections.cpp
   CodeViewYAMLSymbols.cpp
   CodeViewYAMLTypeHashing.cpp
   CodeViewYAMLTypes.cpp
   COFFEmitter.cpp
   COFFYAML.cpp
   DWARFEmitter.cpp
   DWARFYAML.cpp

llvm/lib/ObjectYAML/ObjectYAML.cpp

if (ObjectFile.Coff)
       MappingTraits<COFFYAML::Object>::mapping(IO, *ObjectFile.Coff);
     if (ObjectFile.MachO)
       MappingTraits<MachOYAML::Object>::mapping(IO, *ObjectFile.MachO);
     if (ObjectFile.FatMachO)
       MappingTraits<MachOYAML::UniversalBinary>::mapping(IO,
                                                          *ObjectFile.FatMachO);
   } else {
     Input &In = (Input &)IO;
-    if (IO.mapTag("!ELF")) {
+    if (IO.mapTag("!Arch")) {
+      ObjectFile.Arch.reset(new ArchYAML::Archive());
+      MappingTraits<ArchYAML::Archive>::mapping(IO, *ObjectFile.Arch);
+      std::string Err =
+          MappingTraits<ArchYAML::Archive>::validate(IO, *ObjectFile.Arch);
+      if (!Err.empty())
+        IO.setError(Err);
+    } else if (IO.mapTag("!ELF")) {
       ObjectFile.Elf.reset(new ELFYAML::Object());
       MappingTraits<ELFYAML::Object>::mapping(IO, *ObjectFile.Elf);
     } else if (IO.mapTag("!COFF")) {
       ObjectFile.Coff.reset(new COFFYAML::Object());
       MappingTraits<COFFYAML::Object>::mapping(IO, *ObjectFile.Coff);
     } else if (IO.mapTag("!mach-o")) {
       ObjectFile.MachO.reset(new MachOYAML::Object());
       MappingTraits<MachOYAML::Object>::mapping(IO, *ObjectFile.MachO);

llvm/lib/ObjectYAML/yaml2obj.cpp

do {
 
     yaml::YamlObjectFile Doc;
     YIn >> Doc;
     if (std::error_code EC = YIn.error()) {
       ErrHandler("failed to parse YAML input: " + EC.message());
       return false;
     }
 
+    if (Doc.Arch)
+      return yaml2archive(*Doc.Arch, Out, ErrHandler);
     if (Doc.Elf)
       return yaml2elf(*Doc.Elf, Out, ErrHandler, MaxSize);
     if (Doc.Coff)
       return yaml2coff(*Doc.Coff, Out, ErrHandler);
     if (Doc.MachO || Doc.FatMachO)
       return yaml2macho(Doc, Out, ErrHandler);
     if (Doc.Minidump)
       return yaml2minidump(*Doc.Minidump, Out, ErrHandler);

llvm/test/tools/obj2yaml/Archives/regular.yaml

+## Check how obj2yaml dumps regular archives.
+
+## Check how we dump an empty archive.
+
+# RUN: rm -f %t.empty.a
+# RUN: llvm-ar rc %t.empty.a
+# RUN: obj2yaml %t.empty.a | FileCheck %s --check-prefix=EMPTY
+
+# EMPTY:      --- !Arch
+# EMPTY-NEXT: Members: []
+# EMPTY-NEXT: ...
+
+## Check how we dump archives with multiple members.
+## Check we don't dump excessive spaces when dumping fields.
+## Check we don't dump fields with values that are equal to default values.
+## Check how we dump empty field values.
+
+# RUN: yaml2obj %s --docnum=1 -o %t.multiple.a
+# RUN: obj2yaml %t.multiple.a | FileCheck %s --check-prefix=MULTIPLE1
+
+# MULTIPLE:      --- !Arch
+# MULTIPLE-NEXT: Members:
+# MULTIPLE-NEXT:   - Name:         'bbb/'
+# MULTIPLE-NEXT:     LastModified: '1'
+# MULTIPLE-NEXT:     UID:          '2'
+# MULTIPLE-NEXT:     GID:          '3'
+# MULTIPLE-NEXT:     AccessMode:   '644'
+# MULTIPLE-NEXT:     Size:         '6'
+# MULTIPLE-NEXT:     Content:      20616161200A
+# MULTIPLE-NEXT:   - Name:         'dddd/'
+# MULTIPLE-NEXT:     LastModified: '4'
+# MULTIPLE-NEXT:     UID:          '5'
+# MULTIPLE-NEXT:     GID:          '6'
+# MULTIPLE-NEXT:     AccessMode:   '987'
+# MULTIPLE-NEXT:     Size:         '7'
+# MULTIPLE-NEXT:     Content:      2063636363200A
+# MULTIPLE-NEXT:     PaddingByte:  0x0A
+# MULTIPLE-NEXT:   - LastModified: ''
+# MULTIPLE-NEXT:     UID:          ''
+# MULTIPLE-NEXT:     GID:          ''
+# MULTIPLE-NEXT:     AccessMode:   ''
+# MULTIPLE-NEXT:     Terminator:   ''
+# MULTIPLE-NEXT:     Content:      ''
+# MULTIPLE-NEXT:   - {}
+# MULTIPLE-NEXT: ...
+
+--- !Arch
+Members:
+  - Name:         'bbb/'
+    LastModified: '1'
+    UID:          '2'
+    GID:          '3'
+    AccessMode:   '644'
+    Size:         '6'
+    Terminator:   "`\n"
+    Content:      20616161200A ## " aaa \n"
+  - Name:         'dddd/'
+    LastModified: '4'
+    UID:          '5'
+    GID:          '6'
+    AccessMode:   '987'
+    Size:         '7'
+    Terminator:   "`\n"
+    Content:      2063636363200A ## " cccc \n"
+    PaddingByte:  0x0A
+## All fields are empty (where possible).
+  - Name:         ''
+    LastModified: ''
+    UID:          ''
+    GID:          ''
+    AccessMode:   ''
+    Size:         '0'
+    Terminator:   ''
+    Content:      ''
+## All fields are explicitly set to the default values.
+  - Name:         ''
+    LastModified: '0'
+    UID:          '0'
+    GID:          '0'
+    AccessMode:   '0'
+    Size:         '0'
+    Terminator:   "`\n"
+    Content:      ""
+...
+
+## Check we report an error for non-regular archives.
+
+# RUN: yaml2obj %s --docnum=2 -o %t.not.regular.a
+# RUN: not obj2yaml %t.not.regular.a 2>&1 | \
+# RUN:   FileCheck %s -DFILE=%t.not.regular.a --check-prefix=NOT-REGULAR-ERR
+
+# NOT-REGULAR-ERR: Error reading file: [[FILE]]: only regular archives are supported
+
+--- !Arch
+Magic: "!<thin>\n"
+Members:
+  - {}
+
+## Check we report an error when unable to read the header of an archive member.
+
+# RUN: yaml2obj %s --docnum=3 -o %t.truncated.a
+# RUN: not obj2yaml %t.truncated.a 2>&1 | \
+# RUN:   FileCheck %s -DFILE=%t.truncated.a --check-prefix=TRUNCATED-ERR
+
+# TRUNCATED-ERR: Error reading file: [[FILE]]: unable to read the header of a child at 0x8
+
+--- !Arch
+Content: "00"
+
+## Check we report an error when unable to read the data of an archive member.
+
+# RUN: yaml2obj %s --docnum=4 -o %t.entdata.a
+# RUN: not obj2yaml %t.entdata.a 2>&1 | \
+# RUN:   FileCheck %s -DFILE=%t.entdata.a --check-prefix=ENTDATA-ERR
+
+# ENTDATA-ERR: Error reading file: [[FILE]]: unable to read the data of a child at 0x8 of size 1: the remaining archive size is 0
+
+--- !Arch
+Members:
+  - Size: [[SIZE='1']]
+
+## Check we report an error when unable to read the size of an archive member.
+
+# RUN: yaml2obj %s --docnum=4 -DSIZE='x' -o %t.entsize.a
+# RUN: not obj2yaml %t.entsize.a 2>&1 | \
+# RUN:   FileCheck %s -DFILE=%t.entsize.a --check-prefix=ENTSIZE-ERR
+
+# ENTSIZE-ERR: Error reading file: [[FILE]]: unable to read the size of a child at 0x8 as integer: "x"
+
+## Check we don't try to dump the padding byte when the size of the content is odd and
+## the content ends at the end of a file.
+
+# RUN: yaml2obj %s --docnum=5 -DCONTENT="61" -o %t.no.padding.byte.a
+# RUN: obj2yaml %t.no.padding.byte.a | FileCheck %s --check-prefix=NO-PADDING-BYTE
+
+#      NO-PADDING-BYTE: --- !Arch
+# NO-PADDING-BYTE-NEXT: Members:
+# NO-PADDING-BYTE-NEXT:   - Size:    '1'
+# NO-PADDING-BYTE-NEXT:     Content: '61'
+# NO-PADDING-BYTE-NEXT: ...
+
+--- !Arch
+Members:
+  - Size:    '1'
+    Content: [[CONTENT]]
+
+## Check we dump the padding byte when the size of the content is odd and the content ends
+## before the end of a file.
+
+# RUN: yaml2obj %s --docnum=5 -DCONTENT="610A" -o %t.padding.byte.a
+# RUN: obj2yaml %t.padding.byte.a | FileCheck %s --check-prefix=PADDING-BYTE
+
+#      PADDING-BYTE: --- !Arch
+# PADDING-BYTE-NEXT: Members:
+# PADDING-BYTE-NEXT:   - Size:        '1'
+# PADDING-BYTE-NEXT:     Content:     '61'
+# PADDING-BYTE-NEXT:     PaddingByte: 0x0A
+# PADDING-BYTE-NEXT: ...

llvm/test/tools/yaml2obj/Archives/regular.yaml

+## Check how yaml2obj creates archives.
+
+## Check we create an empty archive when neither "Members" nor "Content" are specified.
+
+# RUN: yaml2obj --docnum=1 %s -o %t.empty.a
+# RUN: llvm-ar t %t.empty.a | FileCheck %s --allow-empty --implicit-check-not={{.}}
+# RUN: wc -c < %t.empty.a | FileCheck %s --check-prefix=EMPTY-SIZE
+# RUN: od -t x1 -v %t.empty.a | FileCheck %s --ignore-case --check-prefix=EMPTY-DATA
+
+# EMPTY-SIZE: 8{{$}}
+# EMPTY-DATA: 21 3c 61 72 63 68 3e 0a
+
+--- !Arch
+Magic:   "[[MAGIC=!<arch>\n]]"
+Content: [[CONTENT=<none>]]
+Members: [[MEMBERS=<none>]]
+
+## Check we report an error when both "Content" and "Members" keys are used together.
+
+# RUN: not yaml2obj --docnum=1 -DMEMBERS="[]" -DCONTENT="00" %s 2>&1 | FileCheck %s --check-prefix=BOTH
+
+## BOTH: error: "Content" and "Members" cannot be used together
+
+# RUN: yaml2obj --docnum=1 -DCONTENT="12" %s -o %t.content.a
+# RUN: od -t x1 -v %t.content.a | FileCheck %s --ignore-case --check-prefix=CONTENT
+
+# CONTENT: 21 3c 61 72 63 68 3e 0a 12{{$}}
+
+## Check we can specify magic bytes of size greater than the normal size (size of "!<arch>\n").
+
+# RUN: yaml2obj --docnum=1 -DMAGIC="123456789" %s -o %t.magic2.a
+# RUN: wc -c < %t.magic2.a | FileCheck %s --check-prefix=MAGIC-SIZE-GR
+# RUN: od -t x1 -v %t.magic2.a | FileCheck %s --ignore-case --check-prefix=MAGIC-DATA-GR
+
+# MAGIC-SIZE-GR: 9{{$}}
+# MAGIC-DATA-GR: 31 32 33 34 35 36 37 38 39
+
+## Check we can specify magic bytes of size less than the normal size (size of "!<arch>\n").
+
+# RUN: yaml2obj --docnum=1 -DMAGIC="1234567" %s -o %t.magic3.a
+# RUN: wc -c < %t.magic3.a | FileCheck %s --check-prefix=MAGIC-SIZE-LESS
+# RUN: od -t x1 -v %t.magic3.a | FileCheck %s --ignore-case --check-prefix=MAGIC-DATA-LESS
+
+# MAGIC-SIZE-LESS: 7{{$}}
+# MAGIC-DATA-LESS: 31 32 33 34 35 36 37
+
+## Check we can produce a valid archive with multiple members.
+## Check we are able to omit the "Magic" key and this defaults to "!<arch>\n".
+
+# RUN: yaml2obj --docnum=2 %s -o %t.two.a
+# RUN: llvm-ar -t %t.two.a | FileCheck %s --check-prefix=TWO
+# RUN: FileCheck --input-file=%t.two.a %s \
+# RUN:   --match-full-lines --strict-whitespace --check-prefix=TWO-DATA
+
+# TWO:      {{^}}bbbbbbbbbbbbbbb{{$}}
+# TWO-NEXT: {{^}}a{{$}}
+# TWO-NOT:  {{.}}
+
+#      TWO-DATA:!<arch>
+# TWO-DATA-NEXT:bbbbbbbbbbbbbbb/1234567890abqwertyasdfgh876543217         `
+# TWO-DATA-NEXT: cccc {{$}}
+# TWO-DATA-NEXT:za/              1           2     3     456     6         `
+# TWO-DATA-NEXT: aaa {{$}}
+#  TWO-DATA-NOT:{{.}}
+
+--- !Arch
+Members:
+## An arbitrary entry where each of fields has maximum allowed length.
+  - Name:         'bbbbbbbbbbbbbbb/'
+    LastModified: '1234567890ab'
+    UID:          'qwerty'
+    GID:          'asdfgh'
+    AccessMode:   '87654321'
+    Size:         '7'
+    Terminator:   "`\n"
+    Content:      "2063636363200A"
+    PaddingByte:  0x7a ## 'z'
+## An arbitrary entry to demonstrate that we use the 0x20 byte (space character)
+## to fill gaps between field values.
+  - Name:         'a/'
+    LastModified: '1'
+    UID:          '2'
+    GID:          '3'
+    AccessMode:   '456'
+    Size:         '6'
+    Terminator:   "`\n"
+    Content:      "20616161200A"
+
+## Check how we validate maximum sizes of fields.
+
+# RUN: not yaml2obj --docnum=3 -DNAME="123456789ABCDEF01" %s 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERROR -DFIELD="Name" -DVAL=16
+# RUN: not yaml2obj --docnum=3 -DLAST="123456789ABCD" %s 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERROR -DFIELD="LastModified" -DVAL=12
+# RUN: not yaml2obj --docnum=3 -DUID="1234567" %s 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERROR -DFIELD="UID" -DVAL=6
+# RUN: not yaml2obj --docnum=3 -DGID="1234567" %s 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERROR -DFIELD="GID" -DVAL=6
+# RUN: not yaml2obj --docnum=3 -DACCESSMODE="123456789" %s 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERROR -DFIELD="AccessMode" -DVAL=8
+# RUN: not yaml2obj --docnum=3 -DSIZE="123456789AB" %s 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERROR -DFIELD="Size" -DVAL=10
+# RUN: not yaml2obj --docnum=3 -DTERMINATOR="123" %s 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERROR -DFIELD="Terminator" -DVAL=2
+
+# ERROR: error: the maximum length of "[[FIELD]]" field is [[VAL]]
+
+--- !Arch
+Members:
+  - Name:         '[[NAME=""]]'
+    LastModified: '[[LAST=""]]'
+    UID:          '[[UID=""]]'
+    GID:          '[[GID=""]]'
+    AccessMode:   '[[ACCESSMODE=""]]'
+    Size:         '[[SIZE=""]]'
+    Terminator:   '[[TERMINATOR=""]]'
+
+## Check that all keys are optional for members.
+
+# RUN: yaml2obj --docnum=4 %s -o %t.all.defaults.a
+# RUN: FileCheck --input-file=%t.all.defaults.a %s \
+# RUN:   --match-full-lines --strict-whitespace --check-prefix=DEFAULTS
+
+#      DEFAULTS:!<arch>
+# DEFAULTS-NEXT:                0           0     0     0       0         `
+# DEFAULTS-NEXT:                0           0     0     0       0         `
+#  DEFAULTS-NOT:{{.}}
+
+--- !Arch
+Members:
+  - {}
+  - {}

llvm/tools/obj2yaml/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   BinaryFormat
   DebugInfoCodeView
   DebugInfoDWARF
   Object
   ObjectYAML
   Support
   )
 
 add_llvm_utility(obj2yaml
+  archive2yaml.cpp
   obj2yaml.cpp
   coff2yaml.cpp
   dwarf2yaml.cpp
   elf2yaml.cpp
   macho2yaml.cpp
   minidump2yaml.cpp
   xcoff2yaml.cpp
   wasm2yaml.cpp
   )

llvm/tools/obj2yaml/archive2yaml.cpp

+//===------ utils/archive2yaml.cpp - obj2yaml conversion tool ---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "obj2yaml.h"
+#include "llvm/BinaryFormat/Magic.h"
+#include "llvm/ObjectYAML/ArchiveYAML.h"
+
+using namespace llvm;
+
+namespace {
+
+class ArchiveDumper {
+public:
+  Expected<ArchYAML::Archive *> dump(MemoryBufferRef Source) {
+    StringRef Buffer = Source.getBuffer();
+    assert(file_magic::archive == identify_magic(Buffer));
+
+    std::unique_ptr<ArchYAML::Archive> Obj =
+        std::make_unique<ArchYAML::Archive>();
+
+    StringRef Magic = "!<arch>\n";
+    if (!Buffer.startswith(Magic))
+      return createStringError(std::errc::not_supported,
+                               "only regular archives are supported");
+    Obj->Magic = Magic;
+    Buffer = Buffer.drop_front(Magic.size());
+
+    Obj->Members.emplace();
+    while (!Buffer.empty()) {
+      uint64_t Offset = Buffer.data() - Source.getBuffer().data();
+      if (Buffer.size() < sizeof(ArchiveHeader))
+        return createStringError(
+            std::errc::illegal_byte_sequence,
+            "unable to read the header of a child at 0x%" PRIx64, Offset);
+
+      const ArchiveHeader &Hdr =
+          *reinterpret_cast<const ArchiveHeader *>(Buffer.data());
+      Buffer = Buffer.drop_front(sizeof(ArchiveHeader));
+
+      auto ToString = [](ArrayRef<char> V) {
+        // We don't want to dump excessive spaces.
+        return StringRef(V.data(), V.size()).rtrim(' ');
+      };
+
+      ArchYAML::Archive::Child C;
+      C.Fields["Name"].Value = ToString(Hdr.Name);
+      C.Fields["LastModified"].Value = ToString(Hdr.LastModified);
+      C.Fields["UID"].Value = ToString(Hdr.UID);
+      C.Fields["GID"].Value = ToString(Hdr.GID);
+      C.Fields["AccessMode"].Value = ToString(Hdr.AccessMode);
+      StringRef SizeStr = ToString(Hdr.Size);
+      C.Fields["Size"].Value = SizeStr;
+      C.Fields["Terminator"].Value = ToString(Hdr.Terminator);
+
+      uint64_t Size;
+      if (SizeStr.getAsInteger(10, Size))
+        return createStringError(
+            std::errc::illegal_byte_sequence,
+            "unable to read the size of a child at 0x%" PRIx64
+            " as integer: \"%s\"",
+            Offset, SizeStr.str().c_str());
+      if (Buffer.size() < Size)
+        return createStringError(
+            std::errc::illegal_byte_sequence,
+            "unable to read the data of a child at 0x%" PRIx64
+            " of size %" PRId64 ": the remaining archive size is %zu",
+            Offset, Size, Buffer.size());
+      if (!Buffer.empty())
+        C.Content = arrayRefFromStringRef(Buffer.take_front(Size));
+
+      const bool HasPaddingByte = (Size & 1) && Buffer.size() > Size;
+      if (HasPaddingByte)
+        C.PaddingByte = Buffer[Size];
+
+      Obj->Members->push_back(C);
+      // If the size is odd, consume a padding byte.
+      Buffer = Buffer.drop_front(HasPaddingByte ? Size + 1 : Size);
+    }
+
+    return Obj.release();
+  }
+
+private:
+  struct ArchiveHeader {
+    char Name[16];
+    char LastModified[12];
+    char UID[6];
+    char GID[6];
+    char AccessMode[8];
+    char Size[10];
+    char Terminator[2];
+  };
+};
+
+} // namespace
+
+Error archive2yaml(raw_ostream &Out, MemoryBufferRef Source) {
+  ArchiveDumper Dumper;
+  Expected<ArchYAML::Archive *> YAMLOrErr = Dumper.dump(Source);
+  if (!YAMLOrErr)
+    return YAMLOrErr.takeError();
+
+  std::unique_ptr<ArchYAML::Archive> YAML(YAMLOrErr.get());
+  yaml::Output Yout(Out);
+  Yout << *YAML;
+
+  return Error::success();
+}

llvm/tools/obj2yaml/obj2yaml.h

 #ifndef LLVM_TOOLS_OBJ2YAML_OBJ2YAML_H
 #define LLVM_TOOLS_OBJ2YAML_OBJ2YAML_H
 
 #include "llvm/Object/COFF.h"
 #include "llvm/Object/Minidump.h"
 #include "llvm/Object/Wasm.h"
 #include "llvm/Object/XCOFFObjectFile.h"
 #include "llvm/Support/raw_ostream.h"
+#include "llvm/Support/MemoryBufferRef.h"
 #include <system_error>
 
 std::error_code coff2yaml(llvm::raw_ostream &Out,
                           const llvm::object::COFFObjectFile &Obj);
 llvm::Error elf2yaml(llvm::raw_ostream &Out,
                          const llvm::object::ObjectFile &Obj);
 llvm::Error macho2yaml(llvm::raw_ostream &Out,
                            const llvm::object::Binary &Obj);
 llvm::Error minidump2yaml(llvm::raw_ostream &Out,
                           const llvm::object::MinidumpFile &Obj);
 std::error_code xcoff2yaml(llvm::raw_ostream &Out,
                            const llvm::object::XCOFFObjectFile &Obj);
 std::error_code wasm2yaml(llvm::raw_ostream &Out,
                           const llvm::object::WasmObjectFile &Obj);
+llvm::Error archive2yaml(llvm::raw_ostream &Out, llvm::MemoryBufferRef Source);
 
 // Forward decls for dwarf2yaml
 namespace llvm {
 class DWARFContext;
 namespace DWARFYAML {
 struct Data;
 }
 }

llvm/tools/obj2yaml/obj2yaml.cpp

 //===------ utils/obj2yaml.cpp - obj2yaml conversion tool -------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "obj2yaml.h"
+#include "llvm/BinaryFormat/Magic.h"
 #include "llvm/Object/Archive.h"
 #include "llvm/Object/COFF.h"
 #include "llvm/Object/Minidump.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/InitLLVM.h"
 
 using namespace llvm;
static Error dumpObject(const ObjectFile &Obj) {
 
   if (Obj.isWasm())
     return errorCodeToError(wasm2yaml(outs(), cast<WasmObjectFile>(Obj)));
 
   llvm_unreachable("unexpected object file format");
 }
 
 static Error dumpInput(StringRef File) {
-  Expected<OwningBinary<Binary>> BinaryOrErr = createBinary(File);
-  if (!BinaryOrErr)
-    return BinaryOrErr.takeError();
+  ErrorOr<std::unique_ptr<MemoryBuffer>> FileOrErr =
+      MemoryBuffer::getFileOrSTDIN(File, /*FileSize=*/-1,
+                                   /*RequiresNullTerminator=*/false);
+  if (std::error_code EC = FileOrErr.getError())
+    return errorCodeToError(EC);
+  std::unique_ptr<MemoryBuffer> &Buffer = FileOrErr.get();
+  MemoryBufferRef MemBuf = Buffer->getMemBufferRef();
+  if (file_magic::archive == identify_magic(MemBuf.getBuffer()))
+    return archive2yaml(outs(), MemBuf);
+
+  Expected<std::unique_ptr<Binary>> BinOrErr =
+      createBinary(MemBuf, /*Context=*/nullptr);
+  if (!BinOrErr)
+    return BinOrErr.takeError();
 
-  Binary &Binary = *BinaryOrErr.get().getBinary();
+  Binary &Binary = *BinOrErr->get();
   // Universal MachO is not a subclass of ObjectFile, so it needs to be handled
   // here with the other binary types.
   if (Binary.isMachO() || Binary.isMachOUniversalBinary())
     return macho2yaml(outs(), Binary);
-  // TODO: If this is an archive, then burst it and dump each entry
   if (ObjectFile *Obj = dyn_cast<ObjectFile>(&Binary))
     return dumpObject(*Obj);
   if (MinidumpFile *Minidump = dyn_cast<MinidumpFile>(&Binary))
     return minidump2yaml(outs(), *Minidump);
 
   return Error::success();
 }