[yaml2obj] - Support reading a content as an array of bytes using the new 'ContentArray' key.

[grimar]

Summary

It implements the way to describe a section content using a multi line description. E.g:

- Name:         .foo
  Type:         SHT_PROGBITS
  ContentArray: [ 0x11, 0x22, 0x33, 0x44,                                ## .long 11223344
                  0x55, 0x66,                                            ## .short 5566.
                  0x77,                                                  ## .byte 0x77
                  0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF, 0x00 ] ## .quad 0x8899aabbccddeeff

It was briefly discussed in D75123 thread previously.

[MaskRay]

This implements .byte. Shall we support counterparts of .asciz, .short, and .long? For strings, specifying a string instead of indivial codepoints will be useful. Sometimes the content represents an integer wider than a byte, being able to specify the integer instead of individual bytes may be useful.

[grimar]

In D82366#2109569, @MaskRay wrote:

This implements .byte. Shall we support counterparts of .asciz, .short, and .long?

At first I thought about implementing a ContentAsm key and wanted to support .byte, .short, .long and .quad for start.
It could look like the following:

ContentAsm:
 - .byte:  0x11
 - .short: 0x2233
 - .long: 0x11223344

The problem is that it could require much more code and I wasn't sure how much it is useful given that such sections
are probably supposed to be temporary, for the cases when user wants to create a YAML test, but a particular SHT_* type
is not yet fully supported by yaml2obj. Also I can't say that such syntax looks nice to me.

I see no other way to implement things like ".asciz, .short, and .long". Particularly, even if we had an array of strings to support
.byte, .short, .long and .quad, like:

ContentArray: [ "0x11", "0x2233", "0x44556677" ...

Then it wouldn't allow to implement .asciz, because it is not clear how to distinguish between a string and a hex integer.

For strings, specifying a string instead of indivial codepoints will be useful.

It is also not clear how to implement null bytes for string values. It would need supporting excape value '\0' probably.

So given that such sections are probably temporary ones, and looking on the concerns above, I'd probably not do something
much more complex here than this patch does.

[jhenderson]

I have no particular opinion on using some specific sizing value or not as the case may be, so I'm happy with whichever approach is agreed upon.

llvm/include/llvm/ObjectYAML/ELFYAML.h

256–257

It probably makes sense to place this and Content next to each other, since they are logically related.

llvm/test/tools/yaml2obj/ELF/content-array.yaml

5

sections -> section

[grimar]

llvm/include/llvm/ObjectYAML/ELFYAML.h

256–257

I've put it here, because ContentBuf is not a key, while Content, Size and Info are.

[grimar]

• Addressed review comments.

[grimar]

@MaskRay, are you OK with this approach?

[jhenderson]

Thinking about it further, I think the ContentArray approach is fine by me - I don't see a massive benefit to explicitly specifying the width of fields. I think the ability to provide a string to write could prove useful for writing string tables (see the various examples where the string table contents are hand-written), but I think that might want to be a separate change, perhaps with a ContentString field instead? LGTM, but please wait for @MaskRay. I don't have any real objections to his approach either.

[MaskRay]

In D82366#2116205, @jhenderson wrote:

Thinking about it further, I think the ContentArray approach is fine by me - I don't see a massive benefit to explicitly specifying the width of fields. I think the ability to provide a string to write could prove useful for writing string tables (see the various examples where the string table contents are hand-written), but I think that might want to be a separate change, perhaps with a ContentString field instead? LGTM, but please wait for @MaskRay. I don't have any real objections to his approach either.

A separate ContentString assumes the content is a string array. To represent malformed section contents, e.g. broken .eh_frame or other .debug_* , it may be handy to mix strings with int8/int16/int32 literals.

llvm/include/llvm/ObjectYAML/ELFYAML.h

struct RawContentSection : Section {
   Optional<llvm::yaml::Hex64> Size;
   Optional<llvm::yaml::Hex64> Info;
 
   RawContentSection() : Section(ChunkKind::RawContent) {}
 
   static bool classof(const Chunk *S) {
     return S->Kind == ChunkKind::RawContent;
   }
+
+  // Is used when a content is read as an array of bytes.
+  Optional<std::vector<uint8_t>> ContentBuf;
 };
 
 struct NoBitsSection : Section {
   llvm::yaml::Hex64 Size;
 
   NoBitsSection() : Section(ChunkKind::NoBits) {}
 
   static bool classof(const Chunk *S) { return S->Kind == ChunkKind::NoBits; }

llvm/lib/ObjectYAML/ELFYAML.cpp

static void sectionMapping(IO &IO, ELFYAML::DynamicSection &Section) {
   commonSectionMapping(IO, Section);
   IO.mapOptional("Entries", Section.Entries);
   IO.mapOptional("Content", Section.Content);
 }
 
 static void sectionMapping(IO &IO, ELFYAML::RawContentSection &Section) {
   commonSectionMapping(IO, Section);
   IO.mapOptional("Content", Section.Content);
+
+  // We also support reading a content as array of bytes using the ContentArray
+  // key. obj2yaml never prints this field.
+  assert(!IO.outputting() || !Section.ContentBuf.hasValue());
+  IO.mapOptional("ContentArray", Section.ContentBuf);
+  if (Section.ContentBuf) {
+    if (Section.Content)
+      IO.setError("Content and ContentArray can't be used together");
+    Section.Content = yaml::BinaryRef(*Section.ContentBuf);
+  }
+
   IO.mapOptional("Size", Section.Size);
   IO.mapOptional("Info", Section.Info);
 }
 
 static void sectionMapping(IO &IO, ELFYAML::StackSizesSection &Section) {
   commonSectionMapping(IO, Section);
   IO.mapOptional("Content", Section.Content);
   IO.mapOptional("Size", Section.Size);

llvm/test/tools/yaml2obj/ELF/content-array.yaml

+## Check we are able to describe the content of a section
+## using the ContentArray key.
+
+## Check we are able to use ContentArray to create multi-line descriptions
+## of sections contents with comments on the same line.
+# RUN: yaml2obj --docnum=1 %s -o %t1
+# RUN: llvm-readobj --sections --section-data %t1 | FileCheck %s
+
+# CHECK:      Section {
+# CHECK:        Index: 1
+# CHECK-NEXT:   Name: .foo
+# CHECK-NEXT:   Type: SHT_PROGBITS
+# CHECK-NEXT:   Flags [
+# CHECK-NEXT:   ]
+# CHECK-NEXT:   Address: 0x0
+# CHECK-NEXT:   Offset: 0x40
+# CHECK-NEXT:   Size: 16
+# CHECK-NEXT:   Link: 0
+# CHECK-NEXT:   Info: 0
+# CHECK-NEXT:   AddressAlignment: 0
+# CHECK-NEXT:   EntrySize: 0
+# CHECK-NEXT:   SectionData (
+# CHECK-NEXT:     0000: 11223344 55667788 99AABBCC DDEEFF00
+# CHECK-NEXT:   )
+# CHECK-NEXT: }
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_DYN
+  Machine: EM_X86_64
+Sections:
+  - Name:         .foo
+    Type:         SHT_PROGBITS
+    ContentArray: [ 0x11, 0x22, 0x33, 0x44,                                ## .long 11223344
+                    0x55, 0x66,                                            ## .short 5566.
+                    0x77,                                                  ## .byte 0x77
+                    0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF, 0x00 ] ## .quad 0x8899aabbccddeeff00
+
+## Check we do not allow using 'Content' and 'ContentArray' at the same time.
+# RUN: not yaml2obj --docnum=2 %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=BOTH
+# BOTH: error: Content and ContentArray can't be used together
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_DYN
+  Machine: EM_X86_64
+Sections:
+  - Name:         .foo
+    Type:         SHT_PROGBITS
+    Content:      [ 0x0 ]
+    ContentArray: [ 0x1 ]
+
+## Check how the "Size" and the "ContentArray" keys can be used together.
+
+## Case A: check that we report an error when the the value of "Size" is less than the content size.
+# RUN: not yaml2obj --docnum=3 -DSIZE=1 %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=SIZE-LESS
+# SIZE-LESS: error: Section size must be greater than or equal to the content size
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_DYN
+  Machine: EM_X86_64
+Sections:
+  - Name:         .foo
+    Type:         SHT_PROGBITS
+    ContentArray: [ 0x11, 0x22 ]
+    Size:         [[SIZE]]
+
+## Case B: check we are able to produce an output when the value of "Size" is equal
+##         to the content size. In this case the "Size" key has no effect.
+# RUN: yaml2obj --docnum=3 -DSIZE=2 %s -o %t3.eq
+# RUN: llvm-readobj --sections --section-data %t3.eq | FileCheck %s --check-prefix=SIZE-EQ
+
+# SIZE-EQ:      Name: .foo
+# SIZE-EQ:      SectionData (
+# SIZE-EQ-NEXT:   0000: 1122 |
+# SIZE-EQ-NEXT: )
+
+## Case C: check we are able to produce an output when the value of "Size" is greater
+##         than the content size. In this case zeroes are added as padding after the
+##         specified content.
+# RUN: yaml2obj --docnum=3 -DSIZE=3 %s -o %t4.gr
+# RUN: llvm-readobj --sections --section-data %t4.gr | FileCheck %s --check-prefix=SIZE-GR
+
+# SIZE-GR:      Name: .foo
+# SIZE-GR:      SectionData (
+# SIZE-GR-NEXT:   0000: 112200 |
+# SIZE-GR-NEXT: )