Debug prefix map for machine code emission

[starsid]

Summary

This is useful to omit the debug compilation dir when compiling assembly
files through the clang driver.

See https://bugs.llvm.org/show_bug.cgi?id=38050 for more details.

[starsid]

Align comments

[compnerd]

include/llvm/MC/MCContext.h

503 ↗

(On Diff #154282)

clang-format please - your & are bound to the wrong side

lib/MC/MCContext.cpp

538 ↗

(On Diff #154282)

please clang-format this - your & are bound to the wrong side.

lib/MC/MCDwarf.cpp

405 ↗

(On Diff #154282)

mmm, should this really apply to the compilation directory?

406 ↗

(On Diff #154282)

Since you are changing the line, please make this const auto &.

[starsid]

use LLVM style for clang-format

[starsid]

lib/MC/MCDwarf.cpp

405 ↗

(On Diff #154282)

I think so, the point is to remap all paths so that the binary can be made to have the same digest irrespective of the source root.

I checked with gcc and they do the same for assembly files.

curl -O https://golang.org/src/runtime/cgo/gcc_amd64.S
gcc -g -c gcc_amd64.S -o gcc_amd64 && strings gcc_amd64 | grep $PWD 
gcc -g -c gcc_amd64.S -o gcc_amd64 -fdebug-prefix-map=$PWD= && strings gcc_amd64 | grep $PWD

[starsid]

add flag to llvm-mc and test logic through llvm-mc

[starsid]

Simplify code.

• Remap DWARF dirs in place before calling Emit.
• Remap compilation dir in place before calling Emit.

In-place editing ensures we did not miss a place, and simplifies lifecycle
management of the strings.

The option is available only through llvm-mc and cc1as.

All regression tests pass.

[100%] Running the LLVM regression tests
Testing Time: 171.72s
  Expected Passes    : 25615
  Expected Failures  : 149
  Unsupported Tests  : 667

[probinson]

Some minor things. clang-format-diff is your friend. :-)

include/llvm/MC/MCContext.h

497 ↗

(On Diff #154481)

This looks longer than 80 columns. clang-format-diff will fix this kind of thing for you.
Or use a typedef for the map.

lib/MC/MCContext.cpp

538 ↗

(On Diff #154481)

80-column violation?

test/MC/ELF/debug-prefix-map.s

5 ↗

(On Diff #154481)

Can you omit the -dwarf-version 5 as I think this should work for any version.

[starsid]

Line length limit to 80. Remove -dwarf-version from test.

[starsid]

include/llvm/MC/MCContext.h

497 ↗

(On Diff #154481)

Oh I did not know about clang-format-diff. Thank you.

It could be useful to configure clang-format as a linter for arcanist, if most people use arcanist for code reviews. I can send in a patch for that if you want.

We started using 100 as the line length limit in my organization, so I tend to not notice the 80 column violations anymore. Thank you for pointing these out.

[probinson]

LGTM.

If you have the interest and knowledge to automate clang-format-diff with arcanist, I'm sure there are many people who would be very happy to see a patch. I don't use arcanist but I know others do.

[starsid]

In D48988#1156370, @probinson wrote:

LGTM.

If you have the interest and knowledge to automate clang-format-diff with arcanist, I'm sure there are many people who would be very happy to see a patch. I don't use arcanist but I know others do.

Thank you. I do not have write permissions for the repo, so will you please commit this for me.

For the linter, I have sent D49116 to you.

llvm/trunk/include/llvm/MC/MCContext.h

private:
     /// Boolean toggled when .secure_log_unique / .secure_log_reset is seen to
     /// catch errors if .secure_log_unique appears twice without
     /// .secure_log_reset appearing between them.
     bool SecureLogUsed = false;
 
     /// The compilation directory to use for DW_AT_comp_dir.
     SmallString<128> CompilationDir;
 
+    /// Prefix replacement map for source file information.
+    std::map<const std::string, const std::string> DebugPrefixMap;
+
     /// The main file name if passed in explicitly.
     std::string MainFileName;
 
     /// The dwarf file and directory tables from the dwarf .file directive.
     /// We now emit a line table for each compile unit. To reduce the prologue
     /// size of each line table, the files and directories used by each compile
     /// unit are separated.
     std::map<unsigned, MCDwarfLineTable> MCDwarfLineTablesCUMap;
public:
     /// Get the compilation directory for DW_AT_comp_dir
     /// The compilation directory should be set with \c setCompilationDir before
     /// calling this function. If it is unset, an empty string will be returned.
     StringRef getCompilationDir() const { return CompilationDir; }
 
     /// Set the compilation directory for DW_AT_comp_dir
     void setCompilationDir(StringRef S) { CompilationDir = S.str(); }
 
+    /// Get the debug prefix map.
+    const std::map<const std::string, const std::string> &
+    getDebugPrefixMap() const {
+      return DebugPrefixMap;
+    }
+
+    /// Add an entry to the debug prefix map.
+    void addDebugPrefixMapEntry(const std::string &From, const std::string &To);
+
+    // Remaps the given path in-place as per the debug prefix map.
+    void RemapDebugPath(std::string *Path);
+
+    // Remaps the compilation dir as per the debug prefix map.
+    void RemapCompilationDir();
+
     /// Get the main file name for use in error messages and debug
     /// info. This can be set to ensure we've got the correct file name
     /// after preprocessing or for -save-temps.
     const std::string &getMainFileName() const { return MainFileName; }
 
     /// Set the main file name and override the default.
     void setMainFileName(StringRef S) { MainFileName = S; }
 

llvm/trunk/include/llvm/MC/MCDwarf.h

Expected<unsigned> tryGetFile(StringRef &Directory, StringRef &FileName,
                                 unsigned FileNumber = 0);
   unsigned getFile(StringRef &Directory, StringRef &FileName,
                    MD5::MD5Result *Checksum, Optional<StringRef> &Source,
                    unsigned FileNumber = 0) {
     return cantFail(tryGetFile(Directory, FileName, Checksum, Source,
                                FileNumber));
   }
 
+  void RemapDwarfDirs(MCContext &Context);
+
   void setRootFile(StringRef Directory, StringRef FileName,
                    MD5::MD5Result *Checksum, Optional<StringRef> Source) {
     Header.CompilationDir = Directory;
     Header.RootFile.Name = FileName;
     Header.RootFile.DirIndex = 0;
     Header.RootFile.Checksum = Checksum;
     Header.RootFile.Source = Source;
     Header.trackMD5Usage(Checksum);

llvm/trunk/lib/MC/MCContext.cpp

MCSectionWasm *MCContext::getWasmSection(const Twine &Section, SectionKind Kind,
 
   return Result;
 }
 
 MCSubtargetInfo &MCContext::getSubtargetCopy(const MCSubtargetInfo &STI) {
   return *new (MCSubtargetAllocator.Allocate()) MCSubtargetInfo(STI);
 }
 
+void MCContext::addDebugPrefixMapEntry(const std::string &From,
+                                       const std::string &To) {
+  DebugPrefixMap.insert(std::make_pair(From, To));
+}
+
+void MCContext::RemapDebugPath(std::string *Path) {
+  for (const auto &Entry : DebugPrefixMap)
+    if (StringRef(*Path).startswith(Entry.first)) {
+      std::string RemappedPath =
+          (Twine(Entry.second) + Path->substr(Entry.first.size())).str();
+      Path->swap(RemappedPath);
+    }
+}
+
+void MCContext::RemapCompilationDir() {
+  std::string CompDir = CompilationDir.str();
+  RemapDebugPath(&CompDir);
+  CompilationDir = CompDir;
+}
+
 //===----------------------------------------------------------------------===//
 // Dwarf Management
 //===----------------------------------------------------------------------===//
 
 /// getDwarfFile - takes a file name and number to place in the dwarf file and
 /// directory tables.  If the file number has already been allocated it is an
 /// error and zero is returned and the client reports the error, else the
 /// allocated file number is returned.  The file numbers may be in any order.

llvm/trunk/lib/MC/MCDwarf.cpp

void MCDwarfLineTable::Emit(MCObjectStreamer *MCOS,
   Optional<MCDwarfLineStr> LineStr;
   if (context.getDwarfVersion() >= 5)
     LineStr = MCDwarfLineStr(context);
 
   // Switch to the section where the table will be emitted into.
   MCOS->SwitchSection(context.getObjectFileInfo()->getDwarfLineSection());
 
   // Handle the rest of the Compile Units.
-  for (const auto &CUIDTablePair : LineTables)
-    CUIDTablePair.second.EmitCU(MCOS, Params, LineStr);
+  for (const auto &CUIDTablePair : LineTables) {
+    auto &LineTable = context.getMCDwarfLineTable(CUIDTablePair.first);
+    LineTable.RemapDwarfDirs(MCOS->getContext());
+    LineTable.EmitCU(MCOS, Params, LineStr);
+  }
 
   if (LineStr)
     LineStr->emitSection(MCOS);
 }
 
 void MCDwarfDwoLineTable::Emit(MCStreamer &MCOS, MCDwarfLineTableParams Params,
                                MCSection *Section) const {
   if (Header.MCDwarfFiles.empty())
MCOS->EmitULEB128IntValue(LineStr ? dwarf::DW_FORM_line_strp
                                     : dwarf::DW_FORM_string);
   MCOS->EmitULEB128IntValue(MCDwarfDirs.size() + 1);
   // Try not to emit an empty compilation directory.
   const StringRef CompDir =
       CompilationDir.empty() ? CtxCompilationDir : StringRef(CompilationDir);
   if (LineStr) {
     // Record path strings, emit references here.
     LineStr->emitRef(MCOS, CompDir);
-    for (auto &Dir : MCDwarfDirs)
+    for (const auto &Dir : MCDwarfDirs)
       LineStr->emitRef(MCOS, Dir);
   } else {
     // The list of directory paths.  Compilation directory comes first.
     MCOS->EmitBytes(CompDir);
     MCOS->EmitBytes(StringRef("\0", 1));
-    for (auto &Dir : MCDwarfDirs) {
+    for (const auto &Dir : MCDwarfDirs) {
       MCOS->EmitBytes(Dir);                // The DirectoryName, and...
       MCOS->EmitBytes(StringRef("\0", 1)); // its null terminator.
     }
   }
 
   // The file format, which is the inline null-terminated filename and a
   // directory index.  We don't track file size/timestamp so don't emit them
   // in the v5 table.  Emit MD5 checksums and source if we have them.
MCDwarfLineTableHeader::tryGetFile(StringRef &Directory,
   File.Source = Source;
   if (Source)
     HasSource = true;
 
   // return the allocated FileNumber.
   return FileNumber;
 }
 
+void MCDwarfLineTable::RemapDwarfDirs(MCContext &Context) {
+  for (auto &Dir : Header.MCDwarfDirs)
+    Context.RemapDebugPath(&Dir);
+}
+
 /// Utility function to emit the encoding to a streamer.
 void MCDwarfLineAddr::Emit(MCStreamer *MCOS, MCDwarfLineTableParams Params,
                            int64_t LineDelta, uint64_t AddrDelta) {
   MCContext &Context = MCOS->getContext();
   SmallString<256> Tmp;
   raw_svector_ostream OS(Tmp);
   MCDwarfLineAddr::Encode(Context, Params, LineDelta, AddrDelta, OS);
   MCOS->EmitBytes(OS.str());

llvm/trunk/lib/MC/MCObjectStreamer.cpp

void MCObjectStreamer::emitFill(const MCExpr &NumValues, int64_t Size,
   insert(new MCFillFragment(Expr, Size, NumValues, Loc));
 }
 
 void MCObjectStreamer::EmitFileDirective(StringRef Filename) {
   getAssembler().addFileName(Filename);
 }
 
 void MCObjectStreamer::FinishImpl() {
+  // Remap the compilation directory before emitting.
+  getContext().RemapCompilationDir();
+
   // If we are generating dwarf for assembly source files dump out the sections.
   if (getContext().getGenDwarfForAssembly())
     MCGenDwarfInfo::Emit(this);
 
   // Dump out the dwarf file & directory tables and line tables.
   MCDwarfLineTable::Emit(this, getAssembler().getDWARFLinetableParams());
 
   flushPendingLabels(nullptr);
   getAssembler().Finish();
 }

llvm/trunk/test/MC/ELF/debug-prefix-map.s

+// RUN: mkdir -p %t.foo
+// RUN: cp %s %t.foo/src.s
+// RUN: cd %t.foo
+
+// RUN: llvm-mc -triple=x86_64-linux-unknown -g src.s -filetype=obj -o out.o
+// RUN: llvm-dwarfdump -v -debug-info out.o | FileCheck --check-prefix=NO_MAP %s
+// RUN: llvm-mc -triple=x86_64-linux-unknown -g src.s -filetype=obj -o out.o -fdebug-prefix-map=%t.foo=src_root
+// RUN: llvm-dwarfdump -v -debug-info out.o | FileCheck --check-prefix=MAP --implicit-check-not ".foo" %s
+
+f:
+  nop
+
+// NO_MAP: DW_AT_comp_dir [DW_FORM_string] ("{{.*}}.foo")
+
+// MAP: DW_AT_name [DW_FORM_string] ("src.s")
+// MAP: DW_AT_comp_dir [DW_FORM_string] ("src_root")
+// MAP: DW_AT_decl_file [DW_FORM_data4] ("src_root/src.s")

llvm/trunk/tools/llvm-mc/llvm-mc.cpp

 static cl::opt<bool>
 GenDwarfForAssembly("g", cl::desc("Generate dwarf debugging info for assembly "
                                   "source files"));
 
 static cl::opt<std::string>
 DebugCompilationDir("fdebug-compilation-dir",
                     cl::desc("Specifies the debug info's compilation dir"));
 
+static cl::list<std::string>
+DebugPrefixMap("fdebug-prefix-map",
+               cl::desc("Map file source paths in debug info"),
+               cl::value_desc("= separated key-value pairs"));
+
 static cl::opt<std::string>
 MainFileName("main-file-name",
              cl::desc("Specifies the name we should consider the input file"));
 
 static cl::opt<bool> SaveTempLabels("save-temp-labels",
                                     cl::desc("Don't discard temporary labels"));
 
 static cl::opt<bool> NoExecStack("no-exec-stack",
int main(int argc, char **argv) {
   if (!DebugCompilationDir.empty())
     Ctx.setCompilationDir(DebugCompilationDir);
   else {
     // If no compilation dir is set, try to use the current directory.
     SmallString<128> CWD;
     if (!sys::fs::current_path(CWD))
       Ctx.setCompilationDir(CWD);
   }
+  for (const auto &Arg : DebugPrefixMap) {
+    const auto &KV = StringRef(Arg).split('=');
+    Ctx.addDebugPrefixMapEntry(KV.first, KV.second);
+  }
   if (!MainFileName.empty())
     Ctx.setMainFileName(MainFileName);
   if (GenDwarfForAssembly && DwarfVersion >= 5) {
     // DWARF v5 needs the root file as well as the compilation directory.
     // If we find a '.file 0' directive that will supersede these values.
     MD5 Hash;
     MD5::MD5Result *Cksum =
         (MD5::MD5Result *)Ctx.allocate(sizeof(MD5::MD5Result), 1);