[yaml2obj] - Introduce a way to set default values when using ShOffset, ShName and ShSize fields.

[grimar]

Summary

For some fields it might be convenient to specify the DEFAULT as a value.
E.g. instead of:

## 0x18 is the standard entsize value.
EntSize: [[ENTSIZE=0x18]]

we could write just:

EntSize: [[ENTSIZE=DEFAULT]]

and just ignore the EntSize key, like it doesn'y exist at all.

This patch adds a support for the DEFAULT value for ShOffset,
ShName and ShSize fields. It also updates a test for llvm-readobj,
which can use the new feature now.

If the whole idea is fine, I'll post patches for another
fields we have that might benefit from this approach.

[jhenderson]

To be clear, is the intent here is to allow yaml2obj to produce two different outputs form the same input where the only difference is that one provides a custom value and the other doesn't? How about a slightly more general concept: have a "None" special value (or "NONE" etc), parallel to llvm::Optional, which is equivalent to not specifying the field at all. This would be applicable to any optional field then, and would allow testing cases where a field may or may not exist.

For example, it could be used to allow testing Size/Content interactions:

...
Sections:
  - Name: .aaa
    Size: [[SIZE=None]]
    Content: [[CONTENT=None]]

# RUN: yaml2obj -o %t-with-size -DSIZE=42
# RUN: yaml2obj -o %t-with-content -DCONTENT=aabbccddeeff
# RUN: yaml2obj -o %t-with-both -DSIZE=42 -DCONTENT=aabbccddeeff

For ShName etc, it works the same as the proposed DEFAULT, because their values would be the default if the field wasn't specified at all.

[MaskRay]

The None proposal looks good to me. My only minor concern is how we differentiate it from a value called None. If we can't find other suitable syntactic form, then =None seems fine.

Other syntactic forms include ShOffset?: [[TEST]]. I don't particularly like the idea that ? is not in [[ ]], but maybe there are some transformations making this appealing.
At the worst, we can use [[TEST=ShOffset: 0]]

[grimar]

In D84398#2169466, @jhenderson wrote:

To be clear, is the intent here is to allow yaml2obj to produce two different outputs form the same input where the only difference is that one provides a custom value and the other doesn't?

Right.

How about a slightly more general concept: have a "None" special value (or "NONE" etc), parallel to llvm::Optional, which is equivalent to not specifying the field at all.

I like the idea too. I've prototyped a patch, it looks fine, I think, now I am working on tests, hope to post it soon.
(I've used <none> as a special value, btw. I think it has less chance to conflict with None/none. I've avoided using NONE because it looks like a macro, though I'd slightly
prefer to use NONE rather than None if we deside that <none> doesn't work for whatever reason).

I am abandoning this revision.

llvm/lib/ObjectYAML/ELFYAML.cpp

 
 StringRef MappingTraits<ELFYAML::Symbol>::validate(IO &IO,
                                                    ELFYAML::Symbol &Symbol) {
   if (Symbol.Index && Symbol.Section.data())
     return "Index and Section cannot both be specified for Symbol";
   return StringRef();
 }
 
+// For some fields we support a special "DEFAULT" value, what means no-op. It is
+// useful in YAML descriptions that use macros sometimes.
+static void maybeMapHex64(IO &IO, const char *Tag,
+                          Optional<llvm::yaml::Hex64> &Val) {
+  StringRef Str;
+  if (!IO.outputting())
+    IO.mapOptional(Tag, Str);
+  if (Str != "DEFAULT")
+    IO.mapOptional(Tag, Val);
+}
+
 static void commonSectionMapping(IO &IO, ELFYAML::Section &Section) {
   IO.mapOptional("Name", Section.Name, StringRef());
   IO.mapRequired("Type", Section.Type);
   IO.mapOptional("Flags", Section.Flags);
   IO.mapOptional("Address", Section.Address);
   IO.mapOptional("Link", Section.Link, StringRef());
   IO.mapOptional("AddressAlign", Section.AddressAlign, Hex64(0));
   IO.mapOptional("EntSize", Section.EntSize);
   IO.mapOptional("Offset", Section.Offset);
 
   // obj2yaml does not dump these fields. They are expected to be empty when we
   // are producing YAML, because yaml2obj sets appropriate values for them
   // automatically when they are not explicitly defined.
   assert(!IO.outputting() ||
          (!Section.ShOffset.hasValue() && !Section.ShSize.hasValue() &&
           !Section.ShName.hasValue() && !Section.ShFlags.hasValue()));
-  IO.mapOptional("ShName", Section.ShName);
-  IO.mapOptional("ShOffset", Section.ShOffset);
-  IO.mapOptional("ShSize", Section.ShSize);
+  maybeMapHex64(IO, "ShName", Section.ShName);
+  maybeMapHex64(IO, "ShOffset", Section.ShOffset);
+  maybeMapHex64(IO, "ShSize", Section.ShSize);
+
+  // ShFlags is used when we want to work around the Flag values validation.
+  // Also, Flag and ShFlags can't be used together. It doesn't make sense to
+  // support the "DEFAULT" value for it.
   IO.mapOptional("ShFlags", Section.ShFlags);
 }
 
 static void sectionMapping(IO &IO, ELFYAML::DynamicSection &Section) {
   commonSectionMapping(IO, Section);
   IO.mapOptional("Entries", Section.Entries);
   IO.mapOptional("Content", Section.Content);
 }

llvm/test/tools/llvm-readobj/ELF/mips-abiflags.test

FileHeader:
   Machine: EM_MIPS
 Sections: []
 
 ## Check we report a warning when we are unable to read the content of the .MIPS.abiflags section.
 # RUN: yaml2obj --docnum=3 -DSHOFFSET=0xffffffff %s -o %t.err1
 # RUN: llvm-readelf -A %t.err1 2>&1 | FileCheck %s -DFILE=%t.err1 --check-prefix=CONTENT-ERR
 # RUN: llvm-readobj -A %t.err1 2>&1 | FileCheck %s -DFILE=%t.err1 --check-prefix=CONTENT-ERR
 
-# CONTENT-ERR:      warning: '[[FILE]]': unable to read the .MIPS.abiflags section: section [index 1] has a sh_offset (0xffffffff) + sh_size (0x18) that is greater than the file size (0x240)
+# CONTENT-ERR:      warning: '[[FILE]]': unable to read the .MIPS.abiflags section: section [index 1] has a sh_offset (0xffffffff) + sh_size (0x18) that is greater than the file size (0x180)
 # CONTENT-ERR-NEXT: There is no .MIPS.options section in the file.
 # CONTENT-ERR-NEXT: There is no .reginfo section in the file.
 
 --- !ELF
 FileHeader:
   Class:   ELFCLASS64
   Data:    ELFDATA2LSB
   Type:    ET_REL
   Machine: EM_MIPS
 Sections:
   - Name:     .MIPS.abiflags
     Type:     SHT_MIPS_ABIFLAGS
     ISA:      MIPS32
-    Offset:   0x100
-    ShOffset: [[SHOFFSET=0x100]]
-    ShSize:   [[SHSIZE=24]]
+    ShOffset: [[SHOFFSET=DEFAULT]]
+    ShSize:   [[SHSIZE=DEFAULT]]
 
 ## Check we report a warning when the .MIPS.abiflags section has an unexpected size.
 # RUN: yaml2obj --docnum=3 -DSHSIZE=23 %s -o %t.err2
 # RUN: llvm-readelf -A %t.err2 2>&1 | FileCheck %s -DFILE=%t.err2 --check-prefix=SIZE-ERR
 # RUN: llvm-readobj -A %t.err2 2>&1 | FileCheck %s -DFILE=%t.err2 --check-prefix=SIZE-ERR
 
 # SIZE-ERR:      warning: '[[FILE]]': unable to read the .MIPS.abiflags section: it has a wrong size (23)
 # SIZE-ERR-NEXT: There is no .MIPS.options section in the file.
 # SIZE-ERR-NEXT: There is no .reginfo section in the file.

llvm/test/tools/yaml2obj/ELF/override-shname.yaml

+## Test how the ShName can be used to override the sh_name field of a section.
+
 ## Check we are able to set a custom sh_name field for different sections
 ## and that doing this does not affect the names stored in the string table.
 
 # RUN: yaml2obj %s -o %t1
 # RUN: llvm-readobj --sections --section-data %t1 | FileCheck %s
 
 # CHECK:      Index: 1
 # CHECK-NEXT: Name:{{.* }}(1)
- Name: .gnu.version_d
     ShName: 0x000000009
     Entries:
   - Name: .regular
     Type: SHT_PROGBITS
     ShName: 0x00000000A
   - Name: .strtab
     Type: SHT_STRTAB
     ShName: 0x00000000B
+
+## Check we can set the ShName to DEFAULT and this is a no-op.
+# RUN: yaml2obj %s --docnum=2 -o %t2
+# RUN: llvm-readobj --sections %t2 | FileCheck %s --check-prefix=DEFAULT
+# RUN: yaml2obj %s --docnum=3 -o %t3
+# RUN: llvm-readobj --sections %t3 | FileCheck %s --check-prefix=DEFAULT
+
+# DEFAULT: Name: .aaa (29)
+# DEFAULT: Name: .ccc (1)
+# DEFAULT: Name: .bbb (6)
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Name: .aaa
+    Type: SHT_PROGBITS
+  - Name: .ccc
+    Type: SHT_PROGBITS
+  - Name: .bbb
+    Type: SHT_PROGBITS
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Name:   .aaa
+    Type:   SHT_PROGBITS
+    ShName: DEFAULT
+  - Name:   .ccc
+    Type:   SHT_PROGBITS
+    ShName: DEFAULT
+## We do not use the TEST macro. It is exist to
+## demonstrate the expected use case for the DEFAULT word.
+  - Name:   .bbb
+    Type:   SHT_PROGBITS
+    ShName: [[TEST=DEFAULT]]

llvm/test/tools/yaml2obj/ELF/override-shoffset.yaml

FileHeader:
   Data:    ELFDATA2LSB
   Type:    ET_REL
   Machine: EM_X86_64
 Sections:
   - Name: .foo
     Type: SHT_PROGBITS
     ShOffset: 0xFFFF0000
     Content: "fefefefefefefefe"
+
+## Check we can set the ShOffset to DEFAULT and this is a no-op.
+# RUN: yaml2obj %s --docnum=4 -o %t4
+# RUN: llvm-readelf --sections %t4 | FileCheck %s --check-prefix=DEFAULT
+# RUN: yaml2obj %s --docnum=5 -o %t5
+# RUN: llvm-readelf --sections %t5 | FileCheck %s --check-prefix=DEFAULT
+
+# DEFAULT: [Nr] Name  Type     Address          Off
+# DEFAULT: [ 1] .aaa  PROGBITS 0000000000000000 000040
+# DEFAULT: [ 2] .bbb  PROGBITS 0000000000000000 000050
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Name: .aaa
+    Type: SHT_PROGBITS
+    Size: 0x10
+  - Name: .bbb
+    Type: SHT_PROGBITS
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Name:     .aaa
+    Type:     SHT_PROGBITS
+    Size:     0x10
+    ShOffset: DEFAULT
+## We do not use the TEST macro. It is exist to
+## demonstrate the expected use case for the DEFAULT word.
+  - Name:     .bbb
+    Type:     SHT_PROGBITS
+    ShOffset: [[TEST=DEFAULT]]

llvm/test/tools/yaml2obj/ELF/override-shsize.yaml

- Name: .foo
     Content: "aaaa"
   - Name: .bar
     Type: SHT_PROGBITS
     Size: 2
     ShSize: 4
   - Name: .zed
     Type: SHT_PROGBITS
     Content: "bbbb"
+
+## Check we can set the ShSize to DEFAULT and this is a no-op.
+# RUN: yaml2obj %s --docnum=6 -o %t6
+# RUN: llvm-readelf --sections %t6 | FileCheck %s --check-prefix=DEFAULT
+
+# DEFAULT: [Nr] Name  Type     Address          Off    Size
+# DEFAULT: [ 1] .aaa  PROGBITS 0000000000000000 000040 000001
+# DEFAULT: [ 2] .bbb  PROGBITS 0000000000000000 000041 000002
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Name:     .aaa
+    Type:     SHT_PROGBITS
+    Size:     0x1
+    ShOffset: DEFAULT
+## We do not use the TEST macro. It is exist to
+## demonstrate the expected use case for the DEFAULT word.
+  - Name:     .bbb
+    Type:     SHT_PROGBITS
+    Size:     0x2
+    ShOffset: [[TEST=DEFAULT]]