[codeview] Expose -gcodeview-ghash for global type hashing

[rnk]

Summary

Experience has shown that the functionality is useful. It makes linking
optimized clang with debug info for me a lot faster, 20s to 13s. The
type merging phase of PDB writing goes from 10s to 3s.

This removes the LLVM cl::opt and replaces it with a metadata flag.

After this change, users can do the following to use ghash:

• add -gcodeview-ghash to compiler flags
• replace /DEBUG with /DEBUG:GHASH in linker flags

[takuto.ikuta]

This change makes sense to me.
But I want chromium config will be updated in next clang roll.
https://cs.chromium.org/chromium/src/build/config/compiler/BUILD.gn?l=2210&rcl=c1582e69def506093836a7cc6ba1e9c77c3b7a69

[aganea]

@rnk Could you possibly post build metrics for Chromium (for the whole build process, compilation and link) with and without ghash?
In my early tests, it seemed that on a large project, compilation was a bit slower overall, but linking was faster.
If that still holds, enabling ghash seems only useful when iterating, or if you have a good caching mecanism in your build system (which would avoid rebuilding all).

[rnk]

In D54370#1294648, @aganea wrote:

@rnk Could you possibly post build metrics for Chromium (for the whole build process, compilation and link) with and without ghash?

I think we'll put the complete analysis here: https://crbug.com/904324 You should be able to star the issue with any google account to get updates by mail.

In my early tests, it seemed that on a large project, compilation was a bit slower overall, but linking was faster.
If that still holds, enabling ghash seems only useful when iterating, or if you have a good caching mecanism in your build system (which would avoid rebuilding all).

That sounds right. It effectively moves hashing work from link time to compile time, which you can mitigate by parallelizing the build with lots of cores or a fancy distributed build system. The extra object file size is crummy, though.

[rnk]

I collected statistics:
https://bugs.chromium.org/p/chromium/issues/detail?id=904324#c4
The summary is that links in Chromium get 32% faster, but object files grow by 7%. There is no compile time impact. Check it out for more details.

[dmajor]

Should the directions in http://blog.llvm.org/2018/01/improving-link-time-on-windows-with.html be updated with the new flags?

clang/include/clang/Driver/Options.td

 def gdwarf_2 : Flag<["-"], "gdwarf-2">, Group<g_Group>,
   HelpText<"Generate source-level debug information with dwarf version 2">;
 def gdwarf_3 : Flag<["-"], "gdwarf-3">, Group<g_Group>,
   HelpText<"Generate source-level debug information with dwarf version 3">;
 def gdwarf_4 : Flag<["-"], "gdwarf-4">, Group<g_Group>,
   HelpText<"Generate source-level debug information with dwarf version 4">;
 def gdwarf_5 : Flag<["-"], "gdwarf-5">, Group<g_Group>,
   HelpText<"Generate source-level debug information with dwarf version 5">;
+
 def gcodeview : Flag<["-"], "gcodeview">,
   HelpText<"Generate CodeView debug information">,
   Flags<[CC1Option, CC1AsOption, CoreOption]>;
+def gcodeview_ghash : Flag<["-"], "gcodeview-ghash">,
+  HelpText<"Emit type record hashes in a .debug$H section">,
+  Flags<[CC1Option, CoreOption]>;
+def gno_codeview_ghash : Flag<["-"], "gno-codeview-ghash">, Flags<[CoreOption]>;
+
 // Equivalent to our default dwarf version. Forces usual dwarf emission when
 // CodeView is enabled.
 def gdwarf : Flag<["-"], "gdwarf">, Alias<gdwarf_4>, Flags<[CoreOption]>;
 
 def gfull : Flag<["-"], "gfull">, Group<g_Group>;
 def gused : Flag<["-"], "gused">, Group<g_Group>;
 def gstabs : Joined<["-"], "gstabs">, Group<g_Group>, Flags<[Unsupported]>;
 def gcoff : Joined<["-"], "gcoff">, Group<g_Group>, Flags<[Unsupported]>;

clang/include/clang/Frontend/CodeGenOptions.def

 /// Dwarf version. Version zero indicates to LLVM that no DWARF should be
 /// emitted.
 VALUE_CODEGENOPT(DwarfVersion, 3, 0)
 
 /// Whether we should emit CodeView debug information. It's possible to emit
 /// CodeView and DWARF into the same object.
 CODEGENOPT(EmitCodeView, 1, 0)
 
+/// Whether to emit the .debug$H section containing hashes of CodeView types.
+CODEGENOPT(CodeViewGHash, 1, 0)
+
 /// The kind of inlining to perform.
 ENUM_CODEGENOPT(Inlining, InliningMethod, 2, NormalInlining)
 
 // Vector functions library to use.
 ENUM_CODEGENOPT(VecLib, VectorLibrary, 2, NoLibrary)
 
 /// The default TLS model to use.
 ENUM_CODEGENOPT(DefaultTLSModel, TLSModel, 2, GeneralDynamicTLSModel)

clang/lib/CodeGen/CodeGenModule.cpp

if (CodeGenOpts.DwarfVersion) {
     // We can change from Warning to Latest if such mode is supported.
     getModule().addModuleFlag(llvm::Module::Warning, "Dwarf Version",
                               CodeGenOpts.DwarfVersion);
   }
   if (CodeGenOpts.EmitCodeView) {
     // Indicate that we want CodeView in the metadata.
     getModule().addModuleFlag(llvm::Module::Warning, "CodeView", 1);
   }
+  if (CodeGenOpts.CodeViewGHash) {
+    getModule().addModuleFlag(llvm::Module::Warning, "CodeViewGHash", 1);
+  }
   if (CodeGenOpts.ControlFlowGuard) {
     // We want function ID tables for Control Flow Guard.
     getModule().addModuleFlag(llvm::Module::Warning, "cfguardtable", 1);
   }
   if (CodeGenOpts.OptimizationLevel > 0 && CodeGenOpts.StrictVTablePointers) {
     // We don't support LTO with 2 with different StrictVTablePointers
     // FIXME: we could support it by stripping all the information introduced
     // by StrictVTablePointers.

clang/lib/Driver/ToolChains/Clang.cpp

static void RenderDebugOptions(const ToolChain &TC, const Driver &D,
   if (EmitCodeView) {
     if (const Arg *A = Args.getLastArg(options::OPT_gcodeview)) {
       EmitCodeView = checkDebugInfoOption(A, Args, D, TC);
       if (EmitCodeView) {
         // DWARFVersion remains at 0 if no explicit choice was made.
         CmdArgs.push_back("-gcodeview");
       }
     }
+
+    // Emit codeview type hashes if requested.
+    if (Args.hasFlag(options::OPT_gcodeview_ghash,
+                     options::OPT_gno_codeview_ghash, false)) {
+      CmdArgs.push_back("-gcodeview-ghash");
+    }
   }
 
   if (!EmitCodeView && DWARFVersion == 0 &&
       DebugInfoKind != codegenoptions::NoDebugInfo)
     DWARFVersion = TC.GetDefaultDwarfVersion();
 
   // -gline-directives-only supported only for the DWARF debug info.
   if (DWARFVersion == 0 && DebugInfoKind == codegenoptions::DebugDirectivesOnly)

clang/lib/Frontend/CompilerInvocation.cpp

if (Val == ~0U)
       Diags.Report(diag::err_drv_invalid_value) << A->getAsString(Args)
                                                 << A->getValue();
     else
       Opts.setDebuggerTuning(static_cast<llvm::DebuggerKind>(Val));
   }
   Opts.DwarfVersion = getLastArgIntValue(Args, OPT_dwarf_version_EQ, 0, Diags);
   Opts.DebugColumnInfo = Args.hasArg(OPT_dwarf_column_info);
   Opts.EmitCodeView = Args.hasArg(OPT_gcodeview);
+  Opts.CodeViewGHash = Args.hasArg(OPT_gcodeview_ghash);
   Opts.MacroDebugInfo = Args.hasArg(OPT_debug_info_macro);
   Opts.WholeProgramVTables = Args.hasArg(OPT_fwhole_program_vtables);
   Opts.LTOVisibilityPublicStd = Args.hasArg(OPT_flto_visibility_public_std);
   Opts.EnableSplitDwarf = Args.hasArg(OPT_enable_split_dwarf);
   Opts.SplitDwarfFile = Args.getLastArgValue(OPT_split_dwarf_file);
   Opts.SplitDwarfInlining = !Args.hasArg(OPT_fno_split_dwarf_inlining);
   Opts.DebugTypeExtRefs = Args.hasArg(OPT_dwarf_ext_refs);
   Opts.DebugExplicitImport = Args.hasArg(OPT_dwarf_explicit_import);

clang/test/CodeGenCXX/debug-info-ms-abi.cpp

 // RUN: %clang_cc1 %s -triple=i686-pc-windows-msvc -debug-info-kind=limited -gcodeview -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple=i686-pc-windows-msvc -debug-info-kind=limited -gcodeview -gcodeview-ghash -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,GHASH
 
 // Tests that certain miscellaneous features work in the MS ABI.
 
 struct Foo {
   virtual void f();
   virtual void g();
   virtual void h();
   static void i(int, int);

 // CHECK-SAME: containingType: ![[Foo]], virtuality: DW_VIRTUALITY_virtual, virtualIndex: 2,
 // CHECK-SAME: flags: DIFlagPrototyped | DIFlagIntroducedVirtual,
 
 // CHECK: ![[i]] = !DISubprogram(name: "i",
 // CHECK-SAME: flags: DIFlagPrototyped | DIFlagStaticMember
 // CHECK-NEXT: ![[dummy:[0-9]+]] = !DISubroutineType(types: ![[Signature:[0-9]+]])
 // CHECK: ![[Signature]] = !{null, ![[BasicInt:[0-9]+]], ![[BasicInt]]}
 // CHECK: ![[BasicInt]] = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+
+// CHECK: !{{[0-9]+}} = !{i32 2, !"CodeView", i32 1}
+// GHASH: !{{[0-9]+}} = !{i32 2, !"CodeViewGHash", i32 1}

clang/test/Driver/gcodeview-ghash.c

+// Note: %s must be preceded by --, otherwise it may be interpreted as a
+// command-line option, e.g. on Mac where %s is commonly under /Users.
+
+// GHASH: "-gcodeview-ghash"
+// NO_GHASH-NOT: "-gcodeview-ghash"
+
+// default
+// RUN: %clang_cl /Z7 -### -- %s 2>&1 | FileCheck -check-prefix=NO_GHASH %s
+// enabled
+// RUN: %clang_cl /Z7 -gcodeview-ghash -### -- %s 2>&1 | FileCheck -check-prefix=GHASH %s
+// disabled
+// RUN: %clang_cl /Z7 -gcodeview-ghash -gno-codeview-ghash -### -- %s 2>&1 | FileCheck -check-prefix=NO_GHASH %s
+
+// enabled, no /Z7
+// RUN: %clang_cl -gcodeview-ghash -### -- %s 2>&1 | FileCheck -check-prefix=NO_GHASH %s
+
+// GCC-style driver
+// RUN: %clang -g -gcodeview -gcodeview-ghash -### -- %s 2>&1 | FileCheck -check-prefix=GHASH %s
+// RUN: %clang -g -gcodeview -gcodeview-ghash -gno-codeview-ghash -### -- %s 2>&1 | FileCheck -check-prefix=NO_GHASH %s

llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h

 class MachineFunction;
 
 /// Collects and handles line tables information in a CodeView format.
 class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   MCStreamer &OS;
   BumpPtrAllocator Allocator;
   codeview::GlobalTypeTableBuilder TypeTable;
 
+  /// Whether to emit type record hashes into .debug$H.
+  bool EmitDebugGlobalHashes = false;
+
   /// The codeview CPU type used by the translation unit.
   codeview::CPUType TheCPU;
 
   /// Represents the most general definition range.
   struct LocalVarDefRange {
     /// Indicates that variable data is stored in memory relative to the
     /// specified register.
     int InMemory : 1;

llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp

 #include <limits>
 #include <string>
 #include <utility>
 #include <vector>
 
 using namespace llvm;
 using namespace llvm::codeview;
 
-static cl::opt<bool> EmitDebugGlobalHashes("emit-codeview-ghash-section",
-                                           cl::ReallyHidden, cl::init(false));
-
 static CPUType mapArchToCVCPUType(Triple::ArchType Type) {
   switch (Type) {
   case Triple::ArchType::x86:
     return CPUType::Pentium3;
   case Triple::ArchType::x86_64:
     return CPUType::X64;
   case Triple::ArchType::thumb:
     return CPUType::Thumb;
if (!MMI->getModule()->getNamedMetadata("llvm.dbg.cu") ||
     MMI->setDebugInfoAvailability(false);
     return;
   }
   // Tell MMI that we have debug info.
   MMI->setDebugInfoAvailability(true);
 
   TheCPU =
       mapArchToCVCPUType(Triple(MMI->getModule()->getTargetTriple()).getArch());
+
+  // Check if we should emit type record hashes.
+  ConstantInt *GH = mdconst::extract_or_null<ConstantInt>(
+      MMI->getModule()->getModuleFlag("CodeViewGHash"));
+  EmitDebugGlobalHashes = GH && !GH->isZero();
 }
 
 StringRef CodeViewDebug::getFullFilepath(const DIFile *File) {
   std::string &Filepath = FileToFilepathMap[File];
   if (!Filepath.empty())
     return Filepath;
 
   StringRef Dir = File->getDirectory(), Filename = File->getFilename();

llvm/test/DebugInfo/COFF/global-type-hashes.ll

-; RUN: llc -filetype=obj -emit-codeview-ghash-section < %s > %t.obj
+; RUN: llc -filetype=obj < %s > %t.obj
 ; RUN: obj2yaml %t.obj | FileCheck %s --check-prefix=YAML
-; RUN: llc -filetype=asm -emit-codeview-ghash-section < %s \
-; RUN:   | FileCheck %s --check-prefix=ASM
+; RUN: llc -filetype=asm < %s | FileCheck %s --check-prefix=ASM
 
 ; C++ source to regenerate:
 ; $ cat t.cpp
 ; struct Foo {
 ;   Foo(int x, int y) : X(x), Y(y) {}
 ;   int method() { return X + Y; }
 ;   int X;
 ;   int Y;
entry:
   ret i32 %add, !dbg !43
 }
 
 attributes #0 = { noinline norecurse nounwind optnone "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "no-frame-pointer-elim"="false" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-features"="+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
 attributes #1 = { nounwind readnone speculatable }
 attributes #2 = { noinline nounwind optnone "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "no-frame-pointer-elim"="false" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-features"="+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
 
 !llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!3, !4, !5, !6}
+!llvm.module.flags = !{!3, !4, !5, !6, !100}
 !llvm.ident = !{!7}
 
+
 !0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 6.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !2)
 !1 = !DIFile(filename: "<stdin>", directory: "D:\5Csrc\5Cllvmbuild\5Cclang\5CDebug\5Cx86", checksumkind: CSK_MD5, checksum: "6279449503d9075c38e615e8387667c3")
 !2 = !{}
 !3 = !{i32 1, !"NumRegisterParameters", i32 0}
 !4 = !{i32 2, !"CodeView", i32 1}
+!100 = !{i32 2, !"CodeViewGHash", i32 1}
 !5 = !{i32 2, !"Debug Info Version", i32 3}
 !6 = !{i32 1, !"wchar_size", i32 2}
 !7 = !{!"clang version 6.0.0 "}
 !8 = distinct !DISubprogram(name: "main", scope: !9, file: !9, line: 8, type: !10, isLocal: false, isDefinition: true, scopeLine: 8, flags: DIFlagPrototyped, isOptimized: false, unit: !0, retainedNodes: !2)
 !9 = !DIFile(filename: "foo.cpp", directory: "D:\5Csrc\5Cllvmbuild\5Cclang\5CDebug\5Cx86", checksumkind: CSK_MD5, checksum: "6279449503d9075c38e615e8387667c3")
 !10 = !DISubroutineType(types: !11)
 !11 = !{!12, !12, !13}
 !12 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)