[PDB] Add symbols to the PDB

[rnk]

Summary

The main complexity in adding symbol records is that we need to
"relocate" all the type indices. Type indices do not have anything like
relocations, an opaque data structure describing where to find existing
type indices for fixups. The linker just has to "know" where the type
references are in the symbol records. I added an overload of
discoverTypeIndices that works on symbol records, and it seems to be
able to link the standard library.

[zturner]

Since we already merge type streams, can you write a test that verifies that these are re-written correctly after linking? Right now the only symbols in the test output have simple type indices (e.g. int), it would be nice to see some symbols in there that refer to non-simple types that have moved around during the type merging process.

[rnk]

In D34432#786167, @zturner wrote:

Since we already merge type streams, can you write a test that verifies that these are re-written correctly after linking? Right now the only symbols in the test output have simple type indices (e.g. int), it would be nice to see some symbols in there that refer to non-simple types that have moved around during the type merging process.

In theory the S_GPROC32_ID record should exercise that, but the minimal dumper doesn't print item ids. I can hack on that first, and then we don't need another test.

[rnk]

In D34432#786170, @rnk wrote:

In D34432#786167, @zturner wrote:

Since we already merge type streams, can you write a test that verifies that these are re-written correctly after linking? Right now the only symbols in the test output have simple type indices (e.g. int), it would be nice to see some symbols in there that refer to non-simple types that have moved around during the type merging process.

In theory the S_GPROC32_ID record should exercise that, but the minimal dumper doesn't print item ids. I can hack on that first, and then we don't need another test.

Eh, who am I kidding. We should have a test that goes on and exercises as many symbol records as possible. I'll put that together.

[rnk]

• add S_UDT test

[ruiu]

lld/COFF/PDB.cpp

137 ↗

(On Diff #103300)

Use the concrete type instead of auto.

142–144 ↗

(On Diff #103300)

You probably want to use fatal as this code handles broken input files.

152–158 ↗

(On Diff #103300)

I guess you are adding more cases in future, but for now please avoid using switch as we have only two choices. It can be written as if (...) return S_END; return Kind;.

168 ↗

(On Diff #103300)

Can you break a line here

172 ↗

(On Diff #103300)

and here so that this doesn't look like one big chunk of code.

[rnk]

lld/COFF/PDB.cpp

142–144 ↗

(On Diff #103300)

It's more likely that our understanding of the record is broken that it is for the record to actually be corrupt. This line actually fires today for files compiled with /Zi, because there's something wrong with our type server handling. libcmt.lib was compiled with /Zi, so any program using the CRT hits this.

I'd rather recover by skipping the record for now. It doesn't result in a broken program, just a less useful PDB.

[rnk]

• formatting

[ruiu]

lld/COFF/PDB.cpp

134 ↗

(On Diff #103406)

Looks like this should be an error too.

192 ↗

(On Diff #103406)

error?

142–144 ↗

(On Diff #103300)

Then can you use error(), which does not call exit?

[rnk]

lld/COFF/PDB.cpp

142–144 ↗

(On Diff #103300)

I don't want to fail the link, though, so this should ultimately be a warning. It's a "log" for now so that it only shows up under verbose output, because you get 100s of these messages when linking against the STL. We can promote it to a warning later.

The scenario I'm imagining is that MSVC starts emitting a new optimized debug info record that we don't understand. IMO, LLD should ignore the record and emit a warning.

[ruiu]

LGTM

lld/trunk/COFF/PDB.cpp

 #include "Error.h"
 #include "SymbolTable.h"
 #include "Symbols.h"
 #include "llvm/DebugInfo/CodeView/CVDebugRecord.h"
 #include "llvm/DebugInfo/CodeView/CVTypeVisitor.h"
 #include "llvm/DebugInfo/CodeView/DebugSubsectionRecord.h"
 #include "llvm/DebugInfo/CodeView/DebugSubsectionVisitor.h"
 #include "llvm/DebugInfo/CodeView/LazyRandomTypeCollection.h"
-#include "llvm/DebugInfo/CodeView/SymbolDumper.h"
 #include "llvm/DebugInfo/CodeView/TypeDumpVisitor.h"
+#include "llvm/DebugInfo/CodeView/TypeIndexDiscovery.h"
 #include "llvm/DebugInfo/CodeView/TypeStreamMerger.h"
 #include "llvm/DebugInfo/CodeView/TypeTableBuilder.h"
 #include "llvm/DebugInfo/MSF/MSFBuilder.h"
 #include "llvm/DebugInfo/MSF/MSFCommon.h"
 #include "llvm/DebugInfo/PDB/Native/DbiStream.h"
 #include "llvm/DebugInfo/PDB/Native/DbiStreamBuilder.h"
 #include "llvm/DebugInfo/PDB/Native/InfoStream.h"
 #include "llvm/DebugInfo/PDB/Native/InfoStreamBuilder.h"
static void mergeDebugT(ObjectFile *File,
   Handler.addSearchPath(sys::path::parent_path(File->getName()));
   if (auto EC = Reader.readArray(Types, Reader.getLength()))
     fatal(EC, "Reader::readArray failed");
   if (auto Err = mergeTypeAndIdRecords(IDTable, TypeTable,
                                                  TypeIndexMap, &Handler, Types))
     fatal(Err, "codeview::mergeTypeStreams failed");
 }
 
+static bool remapTypeIndex(TypeIndex &TI, ArrayRef<TypeIndex> TypeIndexMap) {
+  if (TI.isSimple())
+    return true;
+  if (TI.toArrayIndex() >= TypeIndexMap.size())
+    return false;
+  TI = TypeIndexMap[TI.toArrayIndex()];
+  return true;
+}
+
+static bool remapTypesInSymbolRecord(ObjectFile *File,
+                                     MutableArrayRef<uint8_t> Contents,
+                                     ArrayRef<TypeIndex> TypeIndexMap,
+                                     ArrayRef<TiReference> TypeRefs) {
+  for (const TiReference &Ref : TypeRefs) {
+    unsigned ByteSize = Ref.Count * sizeof(TypeIndex);
+    if (Contents.size() < Ref.Offset + ByteSize) {
+      log("ignoring short symbol record");
+      return false;
+    }
+    MutableArrayRef<TypeIndex> TIs(
+        reinterpret_cast<TypeIndex *>(Contents.data() + Ref.Offset), Ref.Count);
+    for (TypeIndex &TI : TIs)
+      if (!remapTypeIndex(TI, TypeIndexMap)) {
+        log("ignoring symbol record in " + File->getName() +
+            " with bad type index 0x" + utohexstr(TI.getIndex()));
+        return false;
+      }
+  }
+  return true;
+}
+
+/// MSVC translates S_PROC_ID_END to S_END.
+uint16_t canonicalizeSymbolKind(SymbolKind Kind) {
+  if (Kind == SymbolKind::S_PROC_ID_END)
+    return SymbolKind::S_END;
+  return Kind;
+}
+
+/// Copy the symbol record. In a PDB, symbol records must be 4 byte aligned.
+/// The object file may not be aligned.
+static MutableArrayRef<uint8_t> copySymbolForPdb(const CVSymbol &Sym,
+                                                 BumpPtrAllocator &Alloc) {
+  size_t Size = alignTo(Sym.length(), alignOf(CodeViewContainer::Pdb));
+  assert(Size >= 4 && "record too short");
+  assert(Size <= MaxRecordLength && "record too long");
+  void *Mem = Alloc.Allocate(Size, 4);
+
+  // Copy the symbol record and zero out any padding bytes.
+  MutableArrayRef<uint8_t> NewData(reinterpret_cast<uint8_t *>(Mem), Size);
+  memcpy(NewData.data(), Sym.data().data(), Sym.length());
+  memset(NewData.data() + Sym.length(), 0, Size - Sym.length());
+
+  // Update the record prefix length. It should point to the beginning of the
+  // next record. MSVC does some canonicalization of the record kind, so we do
+  // that as well.
+  auto *Prefix = reinterpret_cast<RecordPrefix *>(Mem);
+  Prefix->RecordKind = canonicalizeSymbolKind(Sym.kind());
+  Prefix->RecordLen = Size - 2;
+  return NewData;
+}
+
+static void mergeSymbolRecords(BumpPtrAllocator &Alloc, ObjectFile *File,
+                               ArrayRef<TypeIndex> TypeIndexMap,
+                               BinaryStreamRef SymData) {
+  // FIXME: Improve error recovery by warning and skipping records when
+  // possible.
+  CVSymbolArray Syms;
+  BinaryStreamReader Reader(SymData);
+  ExitOnErr(Reader.readArray(Syms, Reader.getLength()));
+  for (const CVSymbol &Sym : Syms) {
+    // Discover type index references in the record. Skip it if we don't know
+    // where they are.
+    SmallVector<TiReference, 32> TypeRefs;
+    if (!discoverTypeIndices(Sym, TypeRefs)) {
+      log("ignoring unknown symbol record with kind 0x" + utohexstr(Sym.kind()));
+      continue;
+    }
+
+    // Copy the symbol record so we can mutate it.
+    MutableArrayRef<uint8_t> NewData = copySymbolForPdb(Sym, Alloc);
+
+    // Re-map all the type index references.
+    MutableArrayRef<uint8_t> Contents =
+        NewData.drop_front(sizeof(RecordPrefix));
+    if (!remapTypesInSymbolRecord(File, Contents, TypeIndexMap, TypeRefs))
+      continue;
+
+    // FIXME: Fill in "Parent" and "End" fields by maintaining a stack of
+    // scopes.
+
+    // Add the symbol to the module.
+    File->ModuleDBI->addSymbol(CVSymbol(Sym.kind(), NewData));
+  }
+}
+
 // Allocate memory for a .debug$S section and relocate it.
 static ArrayRef<uint8_t> relocateDebugChunk(BumpPtrAllocator &Alloc,
                                             SectionChunk *DebugChunk) {
   uint8_t *Buffer = Alloc.Allocate<uint8_t>(DebugChunk->getSize());
   assert(DebugChunk->OutputSectionOff == 0 &&
          "debug sections should not be in output sections");
   DebugChunk->writeTo(Buffer);
   return consumeDebugMagic(makeArrayRef(Buffer, DebugChunk->getSize()),
for (SectionChunk *DebugChunk : File->getDebugChunks()) {
         case DebugSubsectionKind::FileChecksums:
           ExitOnErr(Checksums.initialize(SS.getRecordData()));
           break;
         case DebugSubsectionKind::Lines:
           // We can add the relocated line table directly to the PDB without
           // modification because the file checksum offsets will stay the same.
           File->ModuleDBI->addDebugSubsection(SS);
           break;
+        case DebugSubsectionKind::Symbols:
+          mergeSymbolRecords(Alloc, File, TypeIndexMap, SS.getRecordData());
+          break;
         default:
           // FIXME: Process the rest of the subsections.
           break;
         }
       }
 
       if (Checksums.valid()) {
         // Make a new file checksum table that refers to offsets in the PDB-wide

lld/trunk/test/COFF/pdb-comdat.test

 
 Both object files will contain debug info for foo, but only the debug info from
 pdb_comdat_main.obj should be included in the PDB.
 
 RUN: rm -rf %t && mkdir -p %t && cd %t
 RUN: yaml2obj %S/Inputs/pdb_comdat_main.yaml -o pdb_comdat_main.obj
 RUN: yaml2obj %S/Inputs/pdb_comdat_bar.yaml -o pdb_comdat_bar.obj
 RUN: lld-link pdb_comdat_main.obj pdb_comdat_bar.obj -out:t.exe -debug -pdb:t.pdb -nodefaultlib -entry:main
-RUN: llvm-pdbutil raw -l t.pdb | FileCheck %s
+RUN: llvm-pdbutil raw -l -symbols t.pdb | FileCheck %s
 
 CHECK:                            Lines
 CHECK: ============================================================
 CHECK-LABEL:   Mod 0000 | `{{.*}}pdb_comdat_main.obj`:
 CHECK:       c:\src\llvm-project\build\pdb_comdat_main.c (MD5: F969E51BBE373436D81492EB61387F36)
 CHECK:       c:\src\llvm-project\build\foo.h (MD5: D74D834EFAC3AE2B45E606A8320B1D5C)
 CHECK-LABEL:   Mod 0001 | `{{.*}}pdb_comdat_bar.obj`:
 CHECK:       c:\src\llvm-project\build\pdb_comdat_bar.c (MD5: 365279DB4FCBEDD721BBFC3B14A953C2)
 CHECK-NOT:       c:\src\llvm-project\build\foo.h
 CHECK-LABEL:   Mod 0002 | `* Linker *`:
 
+CHECK:                           Symbols
+CHECK: ============================================================
+CHECK-LABEL:   Mod 0000 | `{{.*}}pdb_comdat_main.obj`:
+CHECK:   - S_OBJNAME [size = 56] sig=0, `C:\src\llvm-project\build\pdb_comdat_main.obj`
+CHECK:   - S_COMPILE3 [size = 60]
+CHECK:       machine = intel x86-x64, Ver = Microsoft (R) Optimizing Compiler, language = c
+CHECK:       frontend = 19.0.24215.1, backend = 19.0.24215.1
+CHECK:       flags = security checks | hot patchable
+CHECK:   - S_GPROC32_ID [size = 44] `main`
+    FIXME: We need to fill in "end".
+CHECK:       parent = 0, addr = 0002:0000, code size = 24, end = 0
+CHECK:       debug start = 4, debug end = 19, flags = none
+CHECK:   - S_FRAMEPROC [size = 32]
+CHECK:       size = 40, padding size = 0, offset to padding = 0
+CHECK:       bytes of callee saved registers = 0, exception handler addr = 0000:0000
+CHECK:       flags = has async eh | opt speed
+CHECK:   - S_END [size = 4]
+CHECK:   - S_GDATA32 [size = 24] `global`
+CHECK:       type = 0x0074 (int), addr = 0000:0000
+CHECK:   - S_BUILDINFO [size = 8] BuildId = `4106`
+CHECK:   - S_GPROC32_ID [size = 44] `foo`
+CHECK:       parent = 0, addr = 0002:0032, code size = 15, end = 0
+CHECK:       debug start = 0, debug end = 14, flags = none
+CHECK:   - S_FRAMEPROC [size = 32]
+CHECK:       size = 0, padding size = 0, offset to padding = 0
+CHECK:       bytes of callee saved registers = 0, exception handler addr = 0000:0000
+CHECK:       flags = marked inline | has async eh | opt speed
+CHECK:   - S_END [size = 4]
+CHECK-LABEL:   Mod 0001 | `{{.*}}pdb_comdat_bar.obj`:
+CHECK:   - S_OBJNAME [size = 56] sig=0, `C:\src\llvm-project\build\pdb_comdat_bar.obj`
+CHECK:   - S_COMPILE3 [size = 60]
+CHECK:       machine = intel x86-x64, Ver = Microsoft (R) Optimizing Compiler, language = c
+CHECK:       frontend = 19.0.24215.1, backend = 19.0.24215.1
+CHECK:       flags = security checks | hot patchable
+CHECK:   - S_GPROC32_ID [size = 44] `bar`
+CHECK:       parent = 0, addr = 0002:0048, code size = 14, end = 0
+CHECK:       debug start = 4, debug end = 9, flags = none
+CHECK:   - S_FRAMEPROC [size = 32]
+CHECK:       size = 40, padding size = 0, offset to padding = 0
+CHECK:       bytes of callee saved registers = 0, exception handler addr = 0000:0000
+CHECK:       flags = has async eh | opt speed
+CHECK:   - S_END [size = 4]
+CHECK:   - S_GDATA32 [size = 24] `global`
+CHECK:       type = 0x0074 (int), addr = 0000:0000
+CHECK:   - S_BUILDINFO [size = 8] BuildId = `4109`
+CHECK-NOT:   - S_GPROC32_ID {{.*}} `foo`
+CHECK-LABEL:   Mod 0002 | `* Linker *`:
+
 Reorder the object files and verify that the other table is selected.
 
 RUN: lld-link pdb_comdat_bar.obj pdb_comdat_main.obj -out:t.exe -debug -pdb:t.pdb -nodefaultlib -entry:main
 RUN: llvm-pdbutil raw -l t.pdb | FileCheck %s --check-prefix=REORDER
 
 REORDER-LABEL:   Mod 0000 | `{{.*}}pdb_comdat_bar.obj`:
 REORDER:       c:\src\llvm-project\build\pdb_comdat_bar.c (MD5: 365279DB4FCBEDD721BBFC3B14A953C2)
 REORDER:       c:\src\llvm-project\build\foo.h (MD5: D74D834EFAC3AE2B45E606A8320B1D5C)
 REORDER-LABEL:   Mod 0001 | `{{.*}}pdb_comdat_main.obj`:
 REORDER:       c:\src\llvm-project\build\pdb_comdat_main.c
 REORDER-NOT:       c:\src\llvm-project\build\foo.h
 REORDER-LABEL:   Mod 0002 | `* Linker *`:

lld/trunk/test/COFF/pdb-symbol-types.yaml

+# RUN: yaml2obj %s -o %t.obj
+# RUN: lld-link %t.obj -nodefaultlib -entry:main -debug -out:%t.exe -pdb:%t.pdb
+# RUN: llvm-pdbutil raw -symbols %t.pdb | FileCheck %s
+
+# To regenerate the object file:
+# $ cat symbol-types.c
+# struct Foo { int x; };
+# typedef struct Foo UDT_Foo;
+# UDT_Foo global_foo = {42};
+# int main() { return global_foo.x; }
+# $ cl -c -Z7 symbol-types.c
+
+# Note that the type of 'global' goes from 0x1005 in the object file to 0x1004
+# in the PDB because the LF_FUNC_ID is moved to the id stream.
+
+# CHECK:                           Symbols
+# CHECK: ============================================================
+# CHECK-LABEL:   Mod 0000 | `C:\src\llvm-project\build\tools\lld\test\COFF\Output\pdb-symbol-types.yaml.tmp.obj`:
+# CHECK:   - S_OBJNAME [size = 52] sig=0, `C:\src\llvm-project\build\symbol-types.obj`
+# CHECK:   - S_COMPILE3 [size = 60]
+# CHECK:       machine = intel x86-x64, Ver = Microsoft (R) Optimizing Compiler, language = c
+# CHECK:       frontend = 19.0.24215.1, backend = 19.0.24215.1
+# CHECK:       flags = security checks | hot patchable
+# CHECK:   - S_GPROC32_ID [size = 44] `main`
+# CHECK:       parent = 0, addr = 0002:0000, code size = 7, end = 0
+# CHECK:       debug start = 0, debug end = 6, flags = none
+# CHECK:   - S_FRAMEPROC [size = 32]
+# CHECK:       size = 0, padding size = 0, offset to padding = 0
+# CHECK:       bytes of callee saved registers = 0, exception handler addr = 0000:0000
+# CHECK:       flags = has async eh | opt speed
+# CHECK:   - S_END [size = 4]
+# CHECK:   - S_GDATA32 [size = 28] `global_foo`
+# CHECK:       type = 0x1004 (Foo), addr = 0001:0000
+# CHECK:   - S_UDT [size = 16] `UDT_Foo`
+# CHECK:       original type = 0x1004
+# CHECK:   - S_UDT [size = 12] `Foo`
+# CHECK:       original type = 0x1004
+# CHECK:   - S_BUILDINFO [size = 8] BuildId = `4106`
+# CHECK-LABEL:   Mod 0001 | `* Linker *`:
+
+--- !COFF
+header:          
+  Machine:         IMAGE_FILE_MACHINE_AMD64
+  Characteristics: [  ]
+sections:        
+  - Name:            .drectve
+    Characteristics: [ IMAGE_SCN_LNK_INFO, IMAGE_SCN_LNK_REMOVE ]
+    Alignment:       1
+    SectionData:     2020202F44454641554C544C49423A224C4942434D5422202F44454641554C544C49423A224F4C444E414D45532220
+  - Name:            '.debug$S'
+    Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
+    Alignment:       1
+    Subsections:     
+      - !Symbols
+        Records:         
+          - Kind:            S_OBJNAME
+            ObjNameSym:      
+              Signature:       0
+              ObjectName:      'C:\src\llvm-project\build\symbol-types.obj'
+          - Kind:            S_COMPILE3
+            Compile3Sym:     
+              Flags:           [ SecurityChecks, HotPatch ]
+              Machine:         X64
+              FrontendMajor:   19
+              FrontendMinor:   0
+              FrontendBuild:   24215
+              FrontendQFE:     1
+              BackendMajor:    19
+              BackendMinor:    0
+              BackendBuild:    24215
+              BackendQFE:      1
+              Version:         'Microsoft (R) Optimizing Compiler'
+      - !Symbols
+        Records:         
+          - Kind:            S_GPROC32_ID
+            ProcSym:         
+              CodeSize:        7
+              DbgStart:        0
+              DbgEnd:          6
+              FunctionType:    4098
+              Flags:           [  ]
+              DisplayName:     main
+          - Kind:            S_FRAMEPROC
+            FrameProcSym:    
+              TotalFrameBytes: 0
+              PaddingFrameBytes: 0
+              OffsetToPadding: 0
+              BytesOfCalleeSavedRegisters: 0
+              OffsetOfExceptionHandler: 0
+              SectionIdOfExceptionHandler: 0
+              Flags:           [ AsynchronousExceptionHandling, OptimizedForSpeed ]
+          - Kind:            S_PROC_ID_END
+            ScopeEndSym:     
+      - !Lines
+        CodeSize:        7
+        Flags:           [  ]
+        RelocOffset:     0
+        RelocSegment:    0
+        Blocks:          
+          - FileName:        'c:\src\llvm-project\build\symbol-types.c'
+            Lines:           
+              - Offset:          0
+                LineStart:       4
+                IsStatement:     true
+                EndDelta:        0
+              - Offset:          0
+                LineStart:       5
+                IsStatement:     true
+                EndDelta:        0
+              - Offset:          6
+                LineStart:       6
+                IsStatement:     true
+                EndDelta:        0
+            Columns:         
+      - !Symbols
+        Records:         
+          - Kind:            S_GDATA32
+            DataSym:         
+              Type:            4101
+              DisplayName:     global_foo
+          - Kind:            S_UDT
+            UDTSym:          
+              Type:            4101
+              UDTName:         UDT_Foo
+          - Kind:            S_UDT
+            UDTSym:          
+              Type:            4101
+              UDTName:         Foo
+      - !FileChecksums
+        Checksums:       
+          - FileName:        'c:\src\llvm-project\build\symbol-types.c'
+            Kind:            MD5
+            Checksum:        F833E1A4909FF6FEC5689A664F3BE725
+      - !StringTable
+        Strings:         
+          - 'c:\src\llvm-project\build\symbol-types.c'
+      - !Symbols
+        Records:         
+          - Kind:            S_BUILDINFO
+            BuildInfoSym:    
+              BuildId:         4111
+    Relocations:     
+      - VirtualAddress:  164
+        SymbolName:      main
+        Type:            IMAGE_REL_AMD64_SECREL
+      - VirtualAddress:  168
+        SymbolName:      main
+        Type:            IMAGE_REL_AMD64_SECTION
+      - VirtualAddress:  220
+        SymbolName:      main
+        Type:            IMAGE_REL_AMD64_SECREL
+      - VirtualAddress:  224
+        SymbolName:      main
+        Type:            IMAGE_REL_AMD64_SECTION
+      - VirtualAddress:  284
+        SymbolName:      global_foo
+        Type:            IMAGE_REL_AMD64_SECREL
+      - VirtualAddress:  288
+        SymbolName:      global_foo
+        Type:            IMAGE_REL_AMD64_SECTION
+  - Name:            '.debug$T'
+    Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]
+    Alignment:       1
+    Types:           
+      - Kind:            LF_ARGLIST
+        ArgList:         
+          ArgIndices:      [ 0 ]
+      - Kind:            LF_PROCEDURE
+        Procedure:       
+          ReturnType:      116
+          CallConv:        NearC
+          Options:         [ None ]
+          ParameterCount:  0
+          ArgumentList:    4096
+      - Kind:            LF_FUNC_ID
+        FuncId:          
+          ParentScope:     0
+          FunctionType:    4097
+          Name:            main
+      - Kind:            LF_STRUCTURE
+        Class:           
+          MemberCount:     0
+          Options:         [ None, ForwardReference, HasUniqueName ]
+          FieldList:       0
+          Name:            Foo
+          UniqueName:      '.?AUFoo@@'
+          DerivationList:  0
+          VTableShape:     0
+          Size:            0
+      - Kind:            LF_FIELDLIST
+        FieldList:       
+          - Kind:            LF_MEMBER
+            DataMember:      
+              Attrs:           3
+              Type:            116
+              FieldOffset:     0
+              Name:            x
+      - Kind:            LF_STRUCTURE
+        Class:           
+          MemberCount:     1
+          Options:         [ None, HasUniqueName ]
+          FieldList:       4100
+          Name:            Foo
+          UniqueName:      '.?AUFoo@@'
+          DerivationList:  0
+          VTableShape:     0
+          Size:            4
+      - Kind:            LF_STRING_ID
+        StringId:        
+          Id:              0
+          String:          'c:\src\llvm-project\build\symbol-types.c'
+      - Kind:            LF_UDT_SRC_LINE
+        UdtSourceLine:   
+          UDT:             4101
+          SourceFile:      4102
+          LineNumber:      1
+      - Kind:            LF_STRING_ID
+        StringId:        
+          Id:              0
+          String:          'C:\src\llvm-project\build'
+      - Kind:            LF_STRING_ID
+        StringId:        
+          Id:              0
+          String:          'C:\PROGRA~2\MICROS~1.0\VC\Bin\amd64\cl.exe'
+      - Kind:            LF_STRING_ID
+        StringId:        
+          Id:              0
+          String:          '-c -Z7 -MT -IC:\PROGRA~2\MICROS~1.0\VC\include -IC:\PROGRA~2\MICROS~1.0\VC\atlmfc\include -IC:\PROGRA~2\WI3CF2~1\10\include\10.0.14393.0\ucrt -IC:\PROGRA~2\WI3CF2~1\10\include\10.0.14393.0\shared -IC:\PROGRA~2\WI3CF2~1\10\include\10.0.14393.0\um'
+      - Kind:            LF_SUBSTR_LIST
+        StringList:      
+          StringIndices:   [ 4106 ]
+      - Kind:            LF_STRING_ID
+        StringId:        
+          Id:              4107
+          String:          ' -IC:\PROGRA~2\WI3CF2~1\10\include\10.0.14393.0\winrt -TC -X'
+      - Kind:            LF_STRING_ID
+        StringId:        
+          Id:              0
+          String:          symbol-types.c
+      - Kind:            LF_STRING_ID
+        StringId:        
+          Id:              0
+          String:          'C:\src\llvm-project\build\vc140.pdb'
+      - Kind:            LF_BUILDINFO
+        BuildInfo:       
+          ArgIndices:      [ 4104, 4105, 4109, 4110, 4108 ]
+  - Name:            .data
+    Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA, IMAGE_SCN_MEM_READ, IMAGE_SCN_MEM_WRITE ]
+    Alignment:       4
+    SectionData:     2A000000
+  - Name:            '.text$mn'
+    Characteristics: [ IMAGE_SCN_CNT_CODE, IMAGE_SCN_MEM_EXECUTE, IMAGE_SCN_MEM_READ ]
+    Alignment:       16
+    SectionData:     8B0500000000C3
+    Relocations:     
+      - VirtualAddress:  2
+        SymbolName:      global_foo
+        Type:            IMAGE_REL_AMD64_REL32
+symbols:         
+  - Name:            '@comp.id'
+    Value:           17063575
+    SectionNumber:   -1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_STATIC
+  - Name:            '@feat.00'
+    Value:           2147484048
+    SectionNumber:   -1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_STATIC
+  - Name:            .drectve
+    Value:           0
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_STATIC
+    SectionDefinition: 
+      Length:          47
+      NumberOfRelocations: 0
+      NumberOfLinenumbers: 0
+      CheckSum:        0
+      Number:          0
+  - Name:            '.debug$S'
+    Value:           0
+    SectionNumber:   2
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_STATIC
+    SectionDefinition: 
+      Length:          432
+      NumberOfRelocations: 6
+      NumberOfLinenumbers: 0
+      CheckSum:        0
+      Number:          0
+  - Name:            '.debug$T'
+    Value:           0
+    SectionNumber:   3
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_STATIC
+    SectionDefinition: 
+      Length:          732
+      NumberOfRelocations: 0
+      NumberOfLinenumbers: 0
+      CheckSum:        0
+      Number:          0
+  - Name:            .data
+    Value:           0
+    SectionNumber:   4
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_STATIC
+    SectionDefinition: 
+      Length:          4
+      NumberOfRelocations: 0
+      NumberOfLinenumbers: 0
+      CheckSum:        3482275674
+      Number:          0
+  - Name:            global_foo
+    Value:           0
+    SectionNumber:   4
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_EXTERNAL
+  - Name:            '.text$mn'
+    Value:           0
+    SectionNumber:   5
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_NULL
+    StorageClass:    IMAGE_SYM_CLASS_STATIC
+    SectionDefinition: 
+      Length:          7
+      NumberOfRelocations: 1
+      NumberOfLinenumbers: 0
+      CheckSum:        3635526833
+      Number:          0
+  - Name:            main
+    Value:           0
+    SectionNumber:   5
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_FUNCTION
+    StorageClass:    IMAGE_SYM_CLASS_EXTERNAL
+...

llvm/trunk/include/llvm/DebugInfo/CodeView/TypeIndexDiscovery.h

 //===- TypeIndexDiscovery.h -------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_DEBUGINFO_CODEVIEW_TYPEINDEXDISCOVERY_H
 #define LLVM_DEBUGINFO_CODEVIEW_TYPEINDEXDISCOVERY_H
 
 #include "llvm/ADT/SmallVector.h"
+#include "llvm/DebugInfo/CodeView/SymbolRecord.h"
 #include "llvm/DebugInfo/CodeView/TypeRecord.h"
 #include "llvm/Support/Error.h"
 
 namespace llvm {
 namespace codeview {
 enum class TiRefKind { TypeRef, IndexRef };
 struct TiReference {
   TiRefKind Kind;
   uint32_t Offset;
   uint32_t Count;
 };
 
 void discoverTypeIndices(ArrayRef<uint8_t> RecordData,
                          SmallVectorImpl<TiReference> &Refs);
 void discoverTypeIndices(const CVType &Type,
                          SmallVectorImpl<TiReference> &Refs);
+
+/// Discover type indices in symbol records. Returns false if this is an unknown
+/// record.
+bool discoverTypeIndices(const CVSymbol &Symbol,
+                         SmallVectorImpl<TiReference> &Refs);
 }
 }
 
 #endif

llvm/trunk/lib/DebugInfo/CodeView/TypeIndexDiscovery.cpp

static void discoverTypeIndices(ArrayRef<uint8_t> Content, TypeLeafKind Kind,
   case TypeLeafKind::LF_POINTER:
     handlePointer(Content, Refs);
     break;
   default:
     break;
   }
 }
 
+static bool discoverTypeIndices(ArrayRef<uint8_t> Content, SymbolKind Kind,
+                                SmallVectorImpl<TiReference> &Refs) {
+  uint32_t Count;
+  // FIXME: In the future it would be nice if we could avoid hardcoding these
+  // values.  One idea is to define some structures representing these types
+  // that would allow the use of offsetof().
+  switch (Kind) {
+  case SymbolKind::S_GPROC32:
+  case SymbolKind::S_LPROC32:
+  case SymbolKind::S_GPROC32_ID:
+  case SymbolKind::S_LPROC32_ID:
+  case SymbolKind::S_LPROC32_DPC:
+  case SymbolKind::S_LPROC32_DPC_ID:
+    Refs.push_back({TiRefKind::IndexRef, 24, 1}); // LF_FUNC_ID
+    break;
+  case SymbolKind::S_UDT:
+    Refs.push_back({TiRefKind::TypeRef, 0, 1}); // UDT
+    break;
+  case SymbolKind::S_GDATA32:
+  case SymbolKind::S_LDATA32:
+    Refs.push_back({TiRefKind::TypeRef, 0, 1}); // Type
+    break;
+  case SymbolKind::S_BUILDINFO:
+    Refs.push_back({TiRefKind::IndexRef, 0, 1}); // Compile flags
+    break;
+  case SymbolKind::S_LOCAL:
+    Refs.push_back({TiRefKind::TypeRef, 0, 1}); // Type
+    break;
+  case SymbolKind::S_CONSTANT:
+    Refs.push_back({TiRefKind::TypeRef, 0, 1}); // Type
+    break;
+  case SymbolKind::S_REGREL32:
+    Refs.push_back({TiRefKind::TypeRef, 4, 1}); // Type
+    break;
+  case SymbolKind::S_CALLSITEINFO:
+    Refs.push_back({TiRefKind::TypeRef, 8, 1}); // Call signature
+    break;
+  case SymbolKind::S_CALLERS:
+  case SymbolKind::S_CALLEES:
+    // The record is a count followed by an array of type indices.
+    Count = *reinterpret_cast<const ulittle32_t *>(Content.data());
+    Refs.push_back({TiRefKind::IndexRef, 4, Count}); // Callees
+    break;
+  case SymbolKind::S_INLINESITE:
+    Refs.push_back({TiRefKind::IndexRef, 8, 1}); // ID of inlinee
+    break;
+
+  // Defranges don't have types, just registers and code offsets.
+  case SymbolKind::S_DEFRANGE_REGISTER:
+  case SymbolKind::S_DEFRANGE_REGISTER_REL:
+  case SymbolKind::S_DEFRANGE_FRAMEPOINTER_REL:
+  case SymbolKind::S_DEFRANGE_FRAMEPOINTER_REL_FULL_SCOPE:
+  case SymbolKind::S_DEFRANGE_SUBFIELD_REGISTER:
+  case SymbolKind::S_DEFRANGE_SUBFIELD:
+    break;
+
+  // No type refernces.
+  case SymbolKind::S_LABEL32:
+  case SymbolKind::S_OBJNAME:
+  case SymbolKind::S_COMPILE:
+  case SymbolKind::S_COMPILE2:
+  case SymbolKind::S_COMPILE3:
+  case SymbolKind::S_BLOCK32:
+  case SymbolKind::S_FRAMEPROC:
+    break;
+  // Scope ending symbols.
+  case SymbolKind::S_END:
+  case SymbolKind::S_INLINESITE_END:
+  case SymbolKind::S_PROC_ID_END:
+    break;
+  default:
+    return false; // Unknown symbol.
+  }
+  return true;
+}
+
 void llvm::codeview::discoverTypeIndices(const CVType &Type,
                                          SmallVectorImpl<TiReference> &Refs) {
   ::discoverTypeIndices(Type.content(), Type.kind(), Refs);
 }
 
 void llvm::codeview::discoverTypeIndices(ArrayRef<uint8_t> RecordData,
                                          SmallVectorImpl<TiReference> &Refs) {
   const RecordPrefix *P =
       reinterpret_cast<const RecordPrefix *>(RecordData.data());
   TypeLeafKind K = static_cast<TypeLeafKind>(uint16_t(P->RecordKind));
   ::discoverTypeIndices(RecordData.drop_front(sizeof(RecordPrefix)), K, Refs);
 }
+
+bool llvm::codeview::discoverTypeIndices(const CVSymbol &Sym,
+                                         SmallVectorImpl<TiReference> &Refs) {
+  SymbolKind K = Sym.kind();
+  return ::discoverTypeIndices(Sym.content(), K, Refs);
+}

llvm/trunk/unittests/DebugInfo/CodeView/TypeIndexDiscoveryTest.cpp

 //===- llvm/unittest/DebugInfo/CodeView/TypeIndexDiscoveryTest.cpp --------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/DebugInfo/CodeView/TypeIndexDiscovery.h"
 
 #include "llvm/DebugInfo/CodeView/TypeTableBuilder.h"
+#include "llvm/DebugInfo/CodeView/SymbolSerializer.h"
 #include "llvm/Support/Allocator.h"
 
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 
 using namespace llvm;
 using namespace llvm::codeview;
 
 class TypeIndexIteratorTest : public testing::Test {
 public:
   TypeIndexIteratorTest() {}
 
   void SetUp() override {
     Refs.clear();
     TTB = make_unique<TypeTableBuilder>(Storage);
     FLRB = make_unique<FieldListRecordBuilder>(*TTB);
+    Symbols.clear();
   }
 
   void TearDown() override {
     FLRB.reset();
     TTB.reset();
   }
 
 protected:
   template <typename... Indices>
   bool checkTypeReferences(uint32_t RecordIndex, Indices &&... TIs) const {
     EXPECT_EQ(sizeof...(Indices), countRefs(RecordIndex));
-    return checkTypeReferencesImpl(RecordIndex, std::forward<Indices>(TIs)...);
+
+    // Choose between type or symbol records. The checking code doesn't care
+    // which we have.
+    std::vector<ArrayRef<uint8_t>> CVRecords;
+    if (Symbols.empty()) {
+      CVRecords = TTB->records();
+    } else {
+      for (const CVSymbol &S : Symbols)
+        CVRecords.push_back(S.data());
+    }
+
+    return checkTypeReferencesImpl(RecordIndex, CVRecords,
+                                   std::forward<Indices>(TIs)...);
   }
 
   template <typename... T> void writeFieldList(T &&... MemberRecords) {
     FLRB->begin();
     writeFieldListImpl(std::forward<T>(MemberRecords)...);
     FLRB->end(true);
     ASSERT_EQ(1u, TTB->records().size());
     discoverAllTypeIndices();
   }
 
   template <typename... T> void writeTypeRecords(T &&... Records) {
     writeTypeRecordsImpl(std::forward<T>(Records)...);
     ASSERT_EQ(sizeof...(T), TTB->records().size());
     discoverAllTypeIndices();
   }
 
+  template <typename... T> void writeSymbolRecords(T &&... Records) {
+    writeSymbolRecordsImpl(std::forward<T>(Records)...);
+    ASSERT_EQ(sizeof...(T), Symbols.size());
+    discoverTypeIndicesInSymbols();
+  }
+
+
   std::unique_ptr<TypeTableBuilder> TTB;
 
 private:
   uint32_t countRefs(uint32_t RecordIndex) const {
     auto &R = Refs[RecordIndex];
     uint32_t Count = 0;
     for (auto &Ref : R) {
       Count += Ref.Count;
for (auto &Ref : RefList) {
       if (llvm::any_of(Indices,
                        [TI](const TypeIndex &Other) { return Other == TI; }))
         return true;
     }
     return false;
   }
 
   template <typename... Indices>
-  bool checkTypeReferencesImpl(uint32_t RecordIndex) const {
+  bool checkTypeReferencesImpl(uint32_t RecordIndex,
+                               ArrayRef<ArrayRef<uint8_t>> CVRecords) const {
     return true;
   }
 
   template <typename... Indices>
-  bool checkTypeReferencesImpl(uint32_t RecordIndex, TypeIndex TI,
-                               Indices &&... Rest) const {
-    ArrayRef<uint8_t> Record = TTB->records()[RecordIndex];
+  bool checkTypeReferencesImpl(uint32_t RecordIndex,
+                               ArrayRef<ArrayRef<uint8_t>> CVRecords,
+                               TypeIndex TI, Indices &&... Rest) const {
+    ArrayRef<uint8_t> Record = CVRecords[RecordIndex];
     bool Success = checkOneTypeReference(RecordIndex, Record, TI);
     EXPECT_TRUE(Success);
-    return Success &
-           checkTypeReferencesImpl(RecordIndex, std::forward<Indices>(Rest)...);
+    return Success & checkTypeReferencesImpl(RecordIndex, CVRecords,
+                                             std::forward<Indices>(Rest)...);
   }
 
   void discoverAllTypeIndices() {
     Refs.resize(TTB->records().size());
     for (uint32_t I = 0; I < TTB->records().size(); ++I) {
       ArrayRef<uint8_t> Data = TTB->records()[I];
       discoverTypeIndices(Data, Refs[I]);
     }
   }
 
+  void discoverTypeIndicesInSymbols() {
+    Refs.resize(Symbols.size());
+    for (uint32_t I = 0; I < Symbols.size(); ++I)
+      discoverTypeIndices(Symbols[I], Refs[I]);
+  }
+
   // Helper function to write out a field list record with the given list
   // of member records.
   void writeFieldListImpl() {}
 
   template <typename RecType, typename... Rest>
   void writeFieldListImpl(RecType &&Record, Rest &&... Records) {
     FLRB->writeMemberType(Record);
     writeFieldListImpl(std::forward<Rest>(Records)...);
   }
 
   // Helper function to write out a list of type records.
   void writeTypeRecordsImpl() {}
 
   template <typename RecType, typename... Rest>
   void writeTypeRecordsImpl(RecType &&Record, Rest &&... Records) {
     TTB->writeKnownType(Record);
     writeTypeRecordsImpl(std::forward<Rest>(Records)...);
   }
 
+  // Helper function to write out a list of symbol records.
+  void writeSymbolRecordsImpl() {}
+
+  template <typename RecType, typename... Rest>
+  void writeSymbolRecordsImpl(RecType &&Record, Rest &&... Records) {
+    Symbols.push_back(SymbolSerializer::writeOneSymbol(Record, Storage,
+                                                       CodeViewContainer::Pdb));
+    writeSymbolRecordsImpl(std::forward<Rest>(Records)...);
+  }
+
   std::vector<SmallVector<TiReference, 4>> Refs;
   std::unique_ptr<FieldListRecordBuilder> FLRB;
+  std::vector<CVSymbol> Symbols;
   BumpPtrAllocator Storage;
 };
 
 namespace leafs {
 static FuncIdRecord FuncId(TypeIndex(1), TypeIndex(2), "FuncId");
 static MemberFuncIdRecord MemFuncId(TypeIndex(3), TypeIndex(4), "FuncId");
 static StringIdRecord StringId(TypeIndex(5), "TheString");
 static struct {
writeFieldList(BaseClass, Enumerator, DataMember, OverloadedMethod,
                  NestedType, StaticDataMember, VirtualBaseClass, VFPtr,
                  Continuation);
 
   checkTypeReferences(
       0, BaseClass.Type, DataMember.Type, OverloadedMethod.MethodList,
       OneMethod.T1, OneMethod.T2, OneMethod.T3, OneMethod.T4, NestedType.Type,
       StaticDataMember.Type, VirtualBaseClass.BaseType,
       VirtualBaseClass.VBPtrType, VFPtr.Type, Continuation.ContinuationIndex);
-}
+}
+
+TEST_F(TypeIndexIteratorTest, ProcSym) {
+  ProcSym GS(SymbolRecordKind::GlobalProcSym);
+  GS.FunctionType = TypeIndex(0x40);
+  ProcSym LS(SymbolRecordKind::ProcSym);
+  LS.FunctionType = TypeIndex(0x41);
+  writeSymbolRecords(GS, LS);
+  checkTypeReferences(0, GS.FunctionType);
+  checkTypeReferences(1, LS.FunctionType);
+}
+
+TEST_F(TypeIndexIteratorTest, DataSym) {
+  DataSym DS(SymbolRecordKind::GlobalData);
+  DS.Type = TypeIndex(0x40);
+  writeSymbolRecords(DS);
+  checkTypeReferences(0, DS.Type);
+}
+
+TEST_F(TypeIndexIteratorTest, CallerSym) {
+  CallerSym Callees(SymbolRecordKind::CalleeSym);
+  Callees.Indices.push_back(TypeIndex(1));
+  Callees.Indices.push_back(TypeIndex(2));
+  Callees.Indices.push_back(TypeIndex(3));
+  CallerSym Callers(SymbolRecordKind::CallerSym);
+  Callers.Indices.push_back(TypeIndex(4));
+  Callers.Indices.push_back(TypeIndex(5));
+  Callers.Indices.push_back(TypeIndex(6));
+  writeSymbolRecords(Callees, Callers);
+  checkTypeReferences(0, TypeIndex(1), TypeIndex(2), TypeIndex(3));
+  checkTypeReferences(1, TypeIndex(4), TypeIndex(5), TypeIndex(6));
+}