[ObjectYAML] Update obj2yaml and yaml2obj for .debug$H section

[zturner]

Summary

This teaches yaml2obj and obj2yaml to be able to dump and generate the experimental COFF .debug$H section. Emitting this section from clang and using it in LLD is still unimplemented, as I want to get all the proper testing infrastructure in place first.

[majnemer]

llvm/lib/ObjectYAML/CodeViewYAMLTypeHashing.cpp

51–54 ↗

(On Diff #125559)

Seeing as how nothing stops MSFT from using the debug$H section for something of their own invention, maybe we should have a magic four byte sequence at the section start to identify that we made it?

[zturner]

llvm/lib/ObjectYAML/CodeViewYAMLTypeHashing.cpp

51–54 ↗

(On Diff #125559)

Once we have this all working, I plan to send a modified copy of the PE/COFF spec to MS and see about getting it officially in the spec. That said, wouldn't your concern already be addressed by the 16-bit Version field?

[ruiu]

llvm/lib/ObjectYAML/CodeViewYAMLTypeHashing.cpp

51–54 ↗

(On Diff #125559)

If Microsoft adds the same section with a different meaning, and if they decide to add a version field at the beginning of the section (it's not an unreasonable assumption), they would conflict, no?

[zturner]

llvm/lib/ObjectYAML/CodeViewYAMLTypeHashing.cpp

51–54 ↗

(On Diff #125559)

If they add the same section with different meaning then it doesn't matter what we add to this section, because it could still conflict. If we're going to use the same section name at all, we have to at least agree on how to parse the header.

So if they add the same section name with different meaning, and we can't agree on a format that doesn't lead to a conflict, we would probably just have to change the section name entirely. I'm not really that opposed to adding another 4-byte magic field to the beginning, but it seems unnecessary since we already have 4 bytes at the beginning which we can treat as "magic".

[ruiu]

llvm/lib/ObjectYAML/CodeViewYAMLTypeHashing.cpp

51–54 ↗

(On Diff #125559)

I mildly prefer the idea of adding the magic number. It's not a strong preference. If you have that one, when Microsoft starts using .debug$H in a different meaning, we can check whether a section starts with the magic number or not and then dispatch. Version number 0 wouldn't work as a magic number as it is too common, but for example 0x16E9CB37 (I made up this number from $RANDOM) is very unlikely to conflict with what Microsoft would add to the beginning of the section.

[zturner]

I added an extra 4-byte field at the beginning of our stream which expects to be set to 0x133C9C5.

llvm/trunk/include/llvm/BinaryFormat/COFF.h

struct ImportHeader {
 
   ImportNameType getNameType() const {
     return static_cast<ImportNameType>((TypeInfo & 0x1C) >> 2);
   }
 };
 
 enum CodeViewIdentifiers {
   DEBUG_SECTION_MAGIC = 0x4,
+  DEBUG_HASHES_SECTION_MAGIC = 0x133C9C5
 };
 
 inline bool isReservedSectionNumber(int32_t SectionNumber) {
   return SectionNumber <= 0;
 }
 
 } // End namespace COFF.
 } // End namespace llvm.
 
 #endif

llvm/trunk/include/llvm/ObjectYAML/COFFYAML.h

 
 #ifndef LLVM_OBJECTYAML_COFFYAML_H
 #define LLVM_OBJECTYAML_COFFYAML_H
 
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/BinaryFormat/COFF.h"
 #include "llvm/ObjectYAML/CodeViewYAMLDebugSections.h"
+#include "llvm/ObjectYAML/CodeViewYAMLTypeHashing.h"
 #include "llvm/ObjectYAML/CodeViewYAMLTypes.h"
 #include "llvm/ObjectYAML/YAML.h"
 #include <cstdint>
 #include <vector>
 
 namespace llvm {
 
 namespace COFF {

 };
 
 struct Section {
   COFF::section Header;
   unsigned Alignment = 0;
   yaml::BinaryRef SectionData;
   std::vector<CodeViewYAML::YAMLDebugSubsection> DebugS;
   std::vector<CodeViewYAML::LeafRecord> DebugT;
+  Optional<CodeViewYAML::DebugHSection> DebugH;
   std::vector<Relocation> Relocations;
   StringRef Name;
 
   Section();
 };
 
 struct Symbol {
   COFF::symbol Header;

llvm/trunk/include/llvm/ObjectYAML/CodeViewYAMLTypeHashing.h

+//==- CodeViewYAMLTypeHashing.h - CodeView YAMLIO Type hashing ----*- C++-*-==//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines classes for handling the YAML representation of CodeView
+// Debug Info.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_OBJECTYAML_CODEVIEWYAMLTYPEHASHING_H
+#define LLVM_OBJECTYAML_CODEVIEWYAMLTYPEHASHING_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/DebugInfo/CodeView/TypeHashing.h"
+#include "llvm/ObjectYAML/YAML.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/YAMLTraits.h"
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+namespace llvm {
+
+namespace CodeViewYAML {
+
+struct GlobalHash {
+  GlobalHash() = default;
+  explicit GlobalHash(StringRef S) : Hash(S) {
+    assert(S.size() == 20 && "Invalid hash size!");
+  }
+  explicit GlobalHash(ArrayRef<uint8_t> S) : Hash(S) {
+    assert(S.size() == 20 && "Invalid hash size!");
+  }
+  yaml::BinaryRef Hash;
+};
+
+struct DebugHSection {
+  uint32_t Magic;
+  uint16_t Version;
+  uint16_t HashAlgorithm;
+  std::vector<GlobalHash> Hashes;
+};
+
+DebugHSection fromDebugH(ArrayRef<uint8_t> DebugT);
+ArrayRef<uint8_t> toDebugH(const DebugHSection &DebugH,
+                           BumpPtrAllocator &Alloc);
+
+} // end namespace CodeViewYAML
+
+} // end namespace llvm
+
+LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::DebugHSection)
+LLVM_YAML_DECLARE_SCALAR_TRAITS(CodeViewYAML::GlobalHash, false)
+LLVM_YAML_IS_SEQUENCE_VECTOR(CodeViewYAML::GlobalHash)
+
+#endif // LLVM_OBJECTYAML_CODEVIEWYAMLTYPES_H

llvm/trunk/lib/ObjectYAML/CMakeLists.txt

 add_llvm_library(LLVMObjectYAML
-  CodeViewYAMLTypes.cpp
-  CodeViewYAMLSymbols.cpp
   CodeViewYAMLDebugSections.cpp
+  CodeViewYAMLSymbols.cpp
+  CodeViewYAMLTypeHashing.cpp
+  CodeViewYAMLTypes.cpp
   COFFYAML.cpp
   DWARFEmitter.cpp
   DWARFVisitor.cpp
   DWARFYAML.cpp
   ELFYAML.cpp
   MachOYAML.cpp
   ObjectYAML.cpp
   WasmYAML.cpp
   YAML.cpp
   )

llvm/trunk/lib/ObjectYAML/COFFYAML.cpp

void MappingTraits<COFFYAML::Section>::mapping(IO &IO, COFFYAML::Section &Sec) {
   MappingNormalization<NSectionCharacteristics, uint32_t> NC(
       IO, Sec.Header.Characteristics);
   IO.mapRequired("Name", Sec.Name);
   IO.mapRequired("Characteristics", NC->Characteristics);
   IO.mapOptional("VirtualAddress", Sec.Header.VirtualAddress, 0U);
   IO.mapOptional("VirtualSize", Sec.Header.VirtualSize, 0U);
   IO.mapOptional("Alignment", Sec.Alignment, 0U);
 
-  // If this is a .debug$S or .debug$T section parse the semantic representation
-  // of the symbols/types.  If it is any other kind of section, just deal in raw
-  // bytes.
+  // If this is a .debug$S .debug$T, or .debug$H section parse the semantic
+  // representation of the symbols/types.  If it is any other kind of section,
+  // just deal in raw bytes.
   IO.mapOptional("SectionData", Sec.SectionData);
   if (Sec.Name == ".debug$S")
     IO.mapOptional("Subsections", Sec.DebugS);
   else if (Sec.Name == ".debug$T")
     IO.mapOptional("Types", Sec.DebugT);
+  else if (Sec.Name == ".debug$H")
+    IO.mapOptional("GlobalHashes", Sec.DebugH);
 
   IO.mapOptional("Relocations", Sec.Relocations);
 }
 
 void MappingTraits<COFFYAML::Object>::mapping(IO &IO, COFFYAML::Object &Obj) {
   IO.mapTag("!COFF", true);
   IO.mapOptional("OptionalHeader", Obj.OptionalHeader);
   IO.mapRequired("header", Obj.Header);
   IO.mapRequired("sections", Obj.Sections);
   IO.mapRequired("symbols", Obj.Symbols);
 }
 
 } // end namespace yaml
 
 } // end namespace llvm

llvm/trunk/lib/ObjectYAML/CodeViewYAMLTypeHashing.cpp

+//===- CodeViewYAMLTypeHashing.cpp - CodeView YAMLIO type hashing ---------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines classes for handling the YAML representation of CodeView
+// Debug Info.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ObjectYAML/CodeViewYAMLTypeHashing.h"
+#include "llvm/Support/BinaryByteStream.h"
+#include "llvm/Support/BinaryStreamReader.h"
+#include "llvm/Support/BinaryStreamWriter.h"
+
+using namespace llvm;
+using namespace llvm::codeview;
+using namespace llvm::CodeViewYAML;
+using namespace llvm::yaml;
+
+namespace llvm {
+namespace yaml {
+
+void MappingTraits<DebugHSection>::mapping(IO &io, DebugHSection &DebugH) {
+  io.mapRequired("Version", DebugH.Version);
+  io.mapRequired("HashAlgorithm", DebugH.HashAlgorithm);
+  io.mapOptional("HashValues", DebugH.Hashes);
+}
+
+void ScalarTraits<GlobalHash>::output(const GlobalHash &GH, void *Ctx,
+                                      raw_ostream &OS) {
+  ScalarTraits<BinaryRef>::output(GH.Hash, Ctx, OS);
+}
+
+StringRef ScalarTraits<GlobalHash>::input(StringRef Scalar, void *Ctx,
+                                          GlobalHash &GH) {
+  return ScalarTraits<BinaryRef>::input(Scalar, Ctx, GH.Hash);
+}
+
+} // end namespace yaml
+} // end namespace llvm
+
+DebugHSection llvm::CodeViewYAML::fromDebugH(ArrayRef<uint8_t> DebugT) {
+  assert(DebugT.size() >= 8);
+  assert((DebugT.size() - 8) % 20 == 0);
+
+  BinaryStreamReader Reader(DebugT, llvm::support::little);
+  DebugHSection DHS;
+  cantFail(Reader.readInteger(DHS.Magic));
+  cantFail(Reader.readInteger(DHS.Version));
+  cantFail(Reader.readInteger(DHS.HashAlgorithm));
+  while (Reader.bytesRemaining() != 0) {
+    ArrayRef<uint8_t> S;
+    cantFail(Reader.readBytes(S, 20));
+    DHS.Hashes.emplace_back(S);
+  }
+  assert(Reader.bytesRemaining() == 0);
+  return DHS;
+}
+
+ArrayRef<uint8_t> llvm::CodeViewYAML::toDebugH(const DebugHSection &DebugH,
+                                               BumpPtrAllocator &Alloc) {
+  uint32_t Size = 8 + 20 * DebugH.Hashes.size();
+  uint8_t *Data = Alloc.Allocate<uint8_t>(Size);
+  MutableArrayRef<uint8_t> Buffer(Data, Size);
+  BinaryStreamWriter Writer(Buffer, llvm::support::little);
+  cantFail(Writer.writeInteger(DebugH.Magic));
+  cantFail(Writer.writeInteger(DebugH.Version));
+  cantFail(Writer.writeInteger(DebugH.HashAlgorithm));
+  SmallString<20> Hash;
+  for (const auto &H : DebugH.Hashes) {
+    Hash.clear();
+    raw_svector_ostream OS(Hash);
+    H.Hash.writeAsBinary(OS);
+    assert((Hash.size() == 20) && "Invalid hash size!");
+    cantFail(Writer.writeFixedString(Hash));
+  }
+  assert(Writer.bytesRemaining() == 0);
+  return Buffer;
+}

llvm/trunk/test/ObjectYAML/CodeView/guid.yaml

-# RUN: yaml2obj %s | obj2yaml | FileCheck %s
-
---- !COFF
-header:
-  Machine:         IMAGE_FILE_MACHINE_AMD64
-  Characteristics: [  ]
-sections:
-  - Name:            '.debug$T'
-    Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
-    Alignment:       1
-    Types:
-      - Kind:            LF_TYPESERVER2
-        TypeServer2:
-          Guid:            '{01DF191B-22BF-6B42-96CE-5258B8329FE5}'
-          Age:             24
-          Name:            'C:\src\llvm-project\build\vc140.pdb'
-symbols:
-  - Name:            '.debug$T'
-    Value:           0
-    SectionNumber:   1
-    SimpleType:      IMAGE_SYM_TYPE_NULL
-    ComplexType:     IMAGE_SYM_DTYPE_NULL
-    StorageClass:    IMAGE_SYM_CLASS_STATIC
-    SectionDefinition:
-      Length:          64
-      NumberOfRelocations: 0
-      NumberOfLinenumbers: 0
-      CheckSum:        0
-      Number:          0
-...
-
-# CHECK: --- !COFF
-# CHECK: header:
-# CHECK:   Machine:         IMAGE_FILE_MACHINE_AMD64
-# CHECK:   Characteristics: [  ]
-# CHECK: sections:
-# CHECK:   - Name:            '.debug$T'
-# CHECK:     Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
-# CHECK:     Alignment:       1
-# CHECK:     Types:
-# CHECK:       - Kind:            LF_TYPESERVER2
-# CHECK:         TypeServer2:
-# CHECK:           Guid:            '{01DF191B-22BF-6B42-96CE-5258B8329FE5}'
-# CHECK:           Age:             24
-# CHECK:           Name:            'C:\src\llvm-project\build\vc140.pdb'
-# CHECK: symbols:
-# CHECK:   - Name:            '.debug$T'
-# CHECK:     Value:           0
-# CHECK:     SectionNumber:   1
-# CHECK:     SimpleType:      IMAGE_SYM_TYPE_NULL
-# CHECK:     ComplexType:     IMAGE_SYM_DTYPE_NULL
-# CHECK:     StorageClass:    IMAGE_SYM_CLASS_STATIC
-# CHECK:     SectionDefinition:
-# CHECK:       Length:          64
-# CHECK:       NumberOfRelocations: 0
-# CHECK:       NumberOfLinenumbers: 0
-# CHECK:       CheckSum:        0
-# CHECK:       Number:          0
-# CHECK: ...

llvm/trunk/test/ObjectYAML/CodeView/sections.yaml

+# RUN: yaml2obj %s > %t.obj
+# RUN: obj2yaml %t.obj | FileCheck --check-prefix=CHECK %s
+# RUN: llvm-objdump -section-headers %t.obj | FileCheck --check-prefix=HEADERS %s
+
+--- !COFF
+header:
+  Machine:         IMAGE_FILE_MACHINE_AMD64
+  Characteristics: [  ]
+sections:
+  - Name:            '.debug$T'
+    Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
+    Alignment:       1
+    Types:
+      - Kind:            LF_TYPESERVER2
+        TypeServer2:
+          Guid:            '{01DF191B-22BF-6B42-96CE-5258B8329FE5}'
+          Age:             24
+          Name:            'C:\src\llvm-project\build\vc140.pdb'
+  - Name:            '.debug$H'
+    Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
+    Alignment:       4
+    GlobalHashes:
+      Version:          0
+      HashAlgorithm:    0
+      HashValues:
+        - 1522A98D88FAF71B618D97BCAC2B89A424EC4805
+        - 8B2BA87CC27BF9D290A31A6070FA296AAA577E53
+        - EC11CE9F78D6BF61F8D913A9E2C98293782A7EB4
+        - 1088AD64CEBC88D9E015058A159516AF20B79286
+        - 457ABCB8AB70407594B5D72BF471B6BDECC99BC9
+symbols:
+  - Name:            '.debug$T'
+    Value:           0
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_STATIC
+    SectionDefinition:
+      Length:          64
+      NumberOfRelocations: 0
+      NumberOfLinenumbers: 0
+      CheckSum:        0
+      Number:          0
+  - Name:            '.debug$H'
+    Value:           0
+    SectionNumber:   2
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_STATIC
+    SectionDefinition:
+      Length:          108
+      NumberOfRelocations: 0
+      NumberOfLinenumbers: 0
+      CheckSum:        2189213922
+      Number:          1
+...
+
+# CHECK: --- !COFF
+# CHECK: header:
+# CHECK:   Machine:         IMAGE_FILE_MACHINE_AMD64
+# CHECK:   Characteristics: [  ]
+# CHECK: sections:
+# CHECK:   - Name:            '.debug$T'
+# CHECK:     Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
+# CHECK:     Alignment:       1
+# CHECK:     Types:
+# CHECK:       - Kind:            LF_TYPESERVER2
+# CHECK:         TypeServer2:
+# CHECK:           Guid:            '{01DF191B-22BF-6B42-96CE-5258B8329FE5}'
+# CHECK:           Age:             24
+# CHECK:           Name:            'C:\src\llvm-project\build\vc140.pdb'
+# CHECK:   - Name:            '.debug$H'
+# CHECK:     Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
+# CHECK:     Alignment:       4
+# CHECK:     GlobalHashes:
+# CHECK:       Version:          0
+# CHECK:       HashAlgorithm:    0
+# CHECK:       HashValues:
+# CHECK:         - 1522A98D88FAF71B618D97BCAC2B89A424EC4805
+# CHECK:         - 8B2BA87CC27BF9D290A31A6070FA296AAA577E53
+# CHECK:         - EC11CE9F78D6BF61F8D913A9E2C98293782A7EB4
+# CHECK:         - 1088AD64CEBC88D9E015058A159516AF20B79286
+# CHECK:         - 457ABCB8AB70407594B5D72BF471B6BDECC99BC9
+# CHECK: symbols:
+# CHECK:   - Name:            '.debug$T'
+# CHECK:     Value:           0
+# CHECK:     SectionNumber:   1
+# CHECK:     SimpleType:      IMAGE_SYM_TYPE_NULL
+# CHECK:     ComplexType:     IMAGE_SYM_DTYPE_NULL
+# CHECK:     StorageClass:    IMAGE_SYM_CLASS_STATIC
+# CHECK:     SectionDefinition:
+# CHECK:       Length:          64
+# CHECK:       NumberOfRelocations: 0
+# CHECK:       NumberOfLinenumbers: 0
+# CHECK:       CheckSum:        0
+# CHECK:       Number:          0
+# CHECK:   - Name:            '.debug$H'
+# CHECK:     Value:           0
+# CHECK:     SectionNumber:   2
+# CHECK:     SimpleType:      IMAGE_SYM_TYPE_NULL
+# CHECK:     ComplexType:     IMAGE_SYM_DTYPE_NULL
+# CHECK:     StorageClass:    IMAGE_SYM_CLASS_STATIC
+# CHECK:     SectionDefinition:
+# CHECK:       Length:          108
+# CHECK:       NumberOfRelocations: 0
+# CHECK:       NumberOfLinenumbers: 0
+# CHECK:       CheckSum:        2189213922
+# CHECK:       Number:          1
+# CHECK: ...
+
+# HEADERS:   0 .debug$T      00000040 0000000000000000 DATA
+# HEADERS:   1 .debug$H      0000006c 0000000000000000 DATA

llvm/trunk/tools/obj2yaml/coff2yaml.cpp

for (const auto &ObjSection : Obj.sections()) {
     if (!ObjSection.isBSS())
       Obj.getSectionContents(COFFSection, sectionData);
     NewYAMLSection.SectionData = yaml::BinaryRef(sectionData);
 
     if (NewYAMLSection.Name == ".debug$S")
       NewYAMLSection.DebugS = CodeViewYAML::fromDebugS(sectionData, SC);
     else if (NewYAMLSection.Name == ".debug$T")
       NewYAMLSection.DebugT = CodeViewYAML::fromDebugT(sectionData);
+    else if (NewYAMLSection.Name == ".debug$H")
+      NewYAMLSection.DebugH = CodeViewYAML::fromDebugH(sectionData);
 
     std::vector<COFFYAML::Relocation> Relocations;
     for (const auto &Reloc : ObjSection.relocations()) {
       const object::coff_relocation *reloc = Obj.getCOFFRelocation(Reloc);
       COFFYAML::Relocation Rel;
       object::symbol_iterator Sym = Reloc.getSymbol();
       Expected<StringRef> SymbolNameOrErr = Sym->getName();
       if (!SymbolNameOrErr) {

llvm/trunk/tools/yaml2obj/yaml2coff.cpp

if (S.Name == ".debug$S") {
                "Object file does not have debug string table!");
 
         S.SectionData =
             toDebugS(S.DebugS, CP.StringsAndChecksums, CP.Allocator);
       }
     } else if (S.Name == ".debug$T") {
       if (S.SectionData.binary_size() == 0)
         S.SectionData = CodeViewYAML::toDebugT(S.DebugT, CP.Allocator);
+    } else if (S.Name == ".debug$H") {
+      if (S.DebugH.hasValue() && S.SectionData.binary_size() == 0)
+        S.SectionData = CodeViewYAML::toDebugH(*S.DebugH, CP.Allocator);
     }
 
     if (S.SectionData.binary_size() > 0) {
       CurrentSectionDataOffset = alignTo(CurrentSectionDataOffset,
                                          CP.isPE() ? CP.getFileAlignment() : 4);
       S.Header.SizeOfRawData = S.SectionData.binary_size();
       if (CP.isPE())
         S.Header.SizeOfRawData =