[WIP][ASan] Apply -ffile-prefix-map mappings to ASan instrumentation

[leonardchan]

Summary

Some asan-instrumented binaries that are distributed as part of a clang toolchain result in different sizes depending on the length of the build path. This is because ASan is including the path to the source file from the module ID and sanitizer metadata. To have a deterministic build independent of the source file's location, we can extend the usage of -ffile-prefix-map to also apply to sanitizers.

In this patch:

• Add a StringMap that can be used by anyone with access to CodeGenOptions that holds all mappings passed to -ffile-prefix-map.
• Replace path prefixes for module IDs on module creation.
• Replace path prefixes for sanitizer metadata source locations when creating a GlobalsMetadata object.

Design considerations:

• We avoid adding a new flag for sanitizers to prevent adding an extra knob. We would also expect that -ffile-prefix-map generally apply to all instances of paths we would see emitted into a stripped object file.
• I applied the mapping on GlobalsMetadata creation when the ASanGlobalsMetadataWrapperPass runs because I couldn't think of a better place to apply the prefix mapping. The path was originally passed down as a StringRef all the way from a PresumedLoc in SanitizerMetadata::getLocationMetadata(), which cannot be edited with replace_path_prefix.
• I added the mapping as metadata (llvm.asan.file.prefix.map) to prevent having to pass the original StringMap from clang all the way down to ASan and iterating over the map whenever ASan creates a private string. Adding it to metadata that ASanGlobalsMetadataWrapperPass can read should allow us to do this once when that pass is created.

[leonardchan]

Putting this on hold for now. Although this implementation works for ASan, it would be have to be repeated for other tools like SourceBasedCoverage or other sanitizers.

After discussing with @phosek and @mcgrathr, a more general approach would be to develop some internal API that allows users to get SourceLocations for metadata or other paths that will be embeded in IR. Then once we have a single source for retrieving these paths, we could apply -ffile-prefix-mappings once to prevent having to propagate the mapping down to IR.

[MaskRay]

Putting this on hold for now. Although this implementation works for ASan, it would be have to be repeated for other tools like SourceBasedCoverage or other sanitizers.

(You can click "Add Action..."->"Plan Changes" so that this Differential will disappear from others' review list for a while..)

clang/include/clang/Basic/CodeGenOptions.h

 
 #ifndef LLVM_CLANG_BASIC_CODEGENOPTIONS_H
 #define LLVM_CLANG_BASIC_CODEGENOPTIONS_H
 
 #include "clang/Basic/DebugInfoOptions.h"
 #include "clang/Basic/Sanitizers.h"
 #include "clang/Basic/XRayInstr.h"
 #include "llvm/ADT/FloatingPointMode.h"
+#include "llvm/ADT/StringMap.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Regex.h"
 #include "llvm/Target/TargetOptions.h"
 #include <map>
 #include <memory>
 #include <string>
 #include <vector>
 
public:
   std::string DwarfDebugFlags;
 
   /// The string containing the commandline for the llvm.commandline metadata,
   /// if non-empty.
   std::string RecordCommandLine;
 
   std::map<std::string, std::string> DebugPrefixMap;
 
+  /// Prefix mapping for file prefixes.
+  llvm::StringMap<std::string> FilePrefixMap;
+
   /// The ABI to use for passing floating point arguments.
   std::string FloatABI;
 
   /// The floating-point denormal mode to use.
   llvm::DenormalMode FPDenormalMode = llvm::DenormalMode::getIEEE();
 
   /// The floating-point denormal mode to use, for float.
   llvm::DenormalMode FP32DenormalMode = llvm::DenormalMode::getIEEE();

clang/lib/CodeGen/ModuleBuilder.cpp

namespace {
 
   protected:
     std::unique_ptr<llvm::Module> M;
     std::unique_ptr<CodeGen::CodeGenModule> Builder;
 
   private:
     SmallVector<FunctionDecl *, 8> DeferredInlineMemberFuncDefs;
 
-    static llvm::StringRef ExpandModuleName(llvm::StringRef ModuleName,
+    static llvm::SmallString<64> ExpandModuleName(llvm::StringRef ModuleName,
                                                   const CodeGenOptions &CGO) {
-      if (ModuleName == "-" && !CGO.MainFileName.empty())
-        return CGO.MainFileName;
-      return ModuleName;
+      llvm::SmallString<64> Name =
+          (ModuleName == "-" && !CGO.MainFileName.empty()) ? CGO.MainFileName
+                                                           : ModuleName;
+      for (const auto &Entry : CGO.FilePrefixMap) {
+        if (llvm::sys::path::replace_path_prefix(
+                Name, Entry.getKey(), Entry.getValue(),
+                llvm::sys::path::Style::native, true))
+          break;
+      }
+      return Name;
     }
 
   public:
     CodeGeneratorImpl(DiagnosticsEngine &diags, llvm::StringRef ModuleName,
                       const HeaderSearchOptions &HSO,
                       const PreprocessorOptions &PPO, const CodeGenOptions &CGO,
                       llvm::LLVMContext &C,
                       CoverageSourceInfo *CoverageInfo = nullptr)
         : Diags(diags), Ctx(nullptr), HeaderSearchOpts(HSO),
           PreprocessorOpts(PPO), CodeGenOpts(CGO), HandlingTopLevelDecls(0),
-          CoverageInfo(CoverageInfo),
-          M(new llvm::Module(ExpandModuleName(ModuleName, CGO), C)) {
+          CoverageInfo(CoverageInfo) {
+      auto ExpandedName = ExpandModuleName(ModuleName, CGO);
+      M.reset(new llvm::Module(ExpandedName, C));
       C.setDiscardValueNames(CGO.DiscardValueNames);
     }
 
     ~CodeGeneratorImpl() override {
       // There should normally not be any leftover inline method definitions.
       assert(DeferredInlineMemberFuncDefs.empty() ||
              Diags.hasErrorOccurred());
     }
public:
 
     llvm::Constant *GetAddrOfGlobal(GlobalDecl global, bool isForDefinition) {
       return Builder->GetAddrOfGlobal(global, ForDefinition_t(isForDefinition));
     }
 
     llvm::Module *StartModule(llvm::StringRef ModuleName,
                               llvm::LLVMContext &C) {
       assert(!M && "Replacing existing Module?");
-      M.reset(new llvm::Module(ExpandModuleName(ModuleName, CodeGenOpts), C));
+      auto ExpandedName = ExpandModuleName(ModuleName, CodeGenOpts);
+      M.reset(new llvm::Module(ExpandedName, C));
       Initialize(*Ctx);
       return M.get();
     }
 
     void Initialize(ASTContext &Context) override {
       Ctx = &Context;
 
       M->setTargetTriple(Ctx->getTargetInfo().getTriple().getTriple());

clang/lib/CodeGen/SanitizerMetadata.cpp

 #include "clang/AST/Type.h"
 #include "clang/Basic/SourceManager.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/IR/Constants.h"
 
 using namespace clang;
 using namespace CodeGen;
 
+static constexpr char kASanFilePrefixMapMetadata[] =
+    "llvm.asan.file.prefix.map";
+
 SanitizerMetadata::SanitizerMetadata(CodeGenModule &CGM) : CGM(CGM) {}
 
 static bool isAsanHwasanOrMemTag(const SanitizerSet& SS) {
   return SS.hasOneOf(SanitizerKind::Address | SanitizerKind::KernelAddress |
                      SanitizerKind::HWAddress | SanitizerKind::KernelHWAddress |
                      SanitizerKind::MemTag);
 }
 
llvm::Metadata *GlobalMetadata[] = {
           llvm::ConstantInt::get(llvm::Type::getInt1Ty(VMContext), IsDynInit)),
       llvm::ConstantAsMetadata::get(llvm::ConstantInt::get(
           llvm::Type::getInt1Ty(VMContext), IsBlacklisted))};
 
   llvm::MDNode *ThisGlobal = llvm::MDNode::get(VMContext, GlobalMetadata);
   llvm::NamedMDNode *AsanGlobals =
       CGM.getModule().getOrInsertNamedMetadata("llvm.asan.globals");
   AsanGlobals->addOperand(ThisGlobal);
+
+  // Add metadata to propgate file prefix mappings down to ASan. We can't edit
+  // the existing metadata passed to llvm.asan.globals because the metadata
+  // contains StringRefs that we can't edit. These refs point to strings all the
+  // way back to strings from the SourceManager.
+  //
+  // This will be an array of key-value pairs from the prefix mapping stored in
+  // the codegen options. These can be used to adjust the paths used during the
+  // ASanGlobalsMetadata analysis.
+
+  // Make sure not to add the prefix metadata more than once.
+  if (CGM.getModule().getNamedMetadata(kASanFilePrefixMapMetadata))
+    return;
+
+  llvm::NamedMDNode *AsanPrefixMapGlobals =
+      CGM.getModule().getOrInsertNamedMetadata(kASanFilePrefixMapMetadata);
+  for (const auto &Entry : CGM.getCodeGenOpts().FilePrefixMap) {
+    llvm::Metadata *PrefixPairMetadata[] = {
+        llvm::MDString::get(VMContext, Entry.getKey()),
+        llvm::MDString::get(VMContext, Entry.getValue()),
+    };
+    AsanPrefixMapGlobals->addOperand(
+        llvm::MDNode::get(VMContext, PrefixPairMetadata));
+  }
 }
 
 void SanitizerMetadata::reportGlobalToASan(llvm::GlobalVariable *GV,
                                            const VarDecl &D, bool IsDynInit) {
   if (!isAsanHwasanOrMemTag(CGM.getLangOpts().Sanitize))
     return;
   std::string QualName;
   llvm::raw_string_ostream OS(QualName);

clang/lib/Driver/ToolChains/Clang.cpp

if (Map.find('=') == StringRef::npos)
       D.Diag(diag::err_drv_invalid_argument_to_option)
           << Map << A->getOption().getName();
     else
       CmdArgs.push_back(Args.MakeArgString("-fmacro-prefix-map=" + Map));
     A->claim();
   }
 }
 
+/// Add a CC1 option to specify the file path prefix map.
+static void addFilePrefixMapArgMapArg(const Driver &D, const ArgList &Args,
+                                      ArgStringList &CmdArgs) {
+  for (const Arg *A : Args.filtered(options::OPT_ffile_prefix_map_EQ)) {
+    StringRef Map = A->getValue();
+    if (Map.find('=') == StringRef::npos)
+      D.Diag(diag::err_drv_invalid_argument_to_option)
+          << Map << A->getOption().getName();
+    else
+      CmdArgs.push_back(Args.MakeArgString("-ffile-prefix-map=" + Map));
+    A->claim();
+  }
+}
+
 /// Vectorize at all optimization levels greater than 1 except for -Oz.
 /// For -Oz the loop vectorizer is disabled, while the slp vectorizer is
 /// enabled.
 static bool shouldEnableVectorizerAtOLevel(const ArgList &Args, bool isSlpVec) {
   if (Arg *A = Args.getLastArg(options::OPT_O_Group)) {
     if (A->getOption().matches(options::OPT_O4) ||
         A->getOption().matches(options::OPT_Ofast))
       return true;
#endif
 
   if (!ShouldEnableAutolink(Args, TC, JA))
     CmdArgs.push_back("-fno-autolink");
 
   // Add in -fdebug-compilation-dir if necessary.
   addDebugCompDirArg(Args, CmdArgs, D.getVFS());
 
   addDebugPrefixMapArg(D, Args, CmdArgs);
+  addFilePrefixMapArgMapArg(D, Args, CmdArgs);
 
   if (Arg *A = Args.getLastArg(options::OPT_ftemplate_depth_,
                                options::OPT_ftemplate_depth_EQ)) {
     CmdArgs.push_back("-ftemplate-depth");
     CmdArgs.push_back(A->getValue());
   }
 
   if (Arg *A = Args.getLastArg(options::OPT_foperator_arrow_depth_EQ)) {

clang/lib/Frontend/CompilerInvocation.cpp

Opts.ForceDwarfFrameSection =
       Args.hasFlag(OPT_fforce_dwarf_frame, OPT_fno_force_dwarf_frame, false);
 
   for (const auto &Arg : Args.getAllArgValues(OPT_fdebug_prefix_map_EQ)) {
     auto Split = StringRef(Arg).split('=');
     Opts.DebugPrefixMap.insert(
         {std::string(Split.first), std::string(Split.second)});
   }
 
+  for (const auto &Arg : Args.getAllArgValues(OPT_ffile_prefix_map_EQ)) {
+    auto Split = StringRef(Arg).split('=');
+    Opts.FilePrefixMap.insert(
+        {std::string(Split.first), std::string(Split.second)});
+  }
+
   if (const Arg *A =
           Args.getLastArg(OPT_emit_llvm_uselists, OPT_no_emit_llvm_uselists))
     Opts.EmitLLVMUseLists = A->getOption().getID() == OPT_emit_llvm_uselists;
 
   Opts.DisableLLVMPasses = Args.hasArg(OPT_disable_llvm_passes);
   Opts.DisableLifetimeMarkers = Args.hasArg(OPT_disable_lifetimemarkers);
 
   const llvm::Triple::ArchType DebugEntryValueArchs[] = {

clang/test/CodeGen/asan-globals-file-prefix-map.cpp

+// RUN: %clang_cc1 %s -fsanitize=address -ffile-prefix-map=%S=/tmp -S -emit-llvm -o - %s -triple x86_64-unknown-linux-gnu| FileCheck %s
+
+int global;
+
+// Check that the __asan_gen_* contants use the prefix map.
+// CHECK: @global = global { i32, [60 x i8] } zeroinitializer, align 32
+// CHECK: @___asan_gen_{{\.?[0-9]*}} = private constant [38 x i8] c"/tmp/asan-globals-file-prefix-map.cpp\00", align 1
+// CHECK: @___asan_gen_{{\.?[0-9]*}} = private unnamed_addr constant [38 x i8] c"/tmp/asan-globals-file-prefix-map.cpp\00", align 1
+
+// Check that the `llvm.asan.file.prefix.map` metadata is emitted.
+// CHECK: !llvm.asan.globals = !{[[GLOBAL:\![0-9]+]]}
+// CHECK: !llvm.asan.file.prefix.map = !{[[MAPPING:\![0-9]+]]}
+
+// CHECK: [[GLOBAL]] = !{i32* getelementptr{{.*}} [[MDLOC:![0-9]+]], !"global"
+// CHECK: [[MDLOC]] = !{!"[[PATH:.+]]/asan-globals-file-prefix-map.cpp"
+// CHECK: [[MAPPING]] = !{!"[[PATH]]", !"/tmp"}

llvm/include/llvm/Transforms/Instrumentation/AddressSanitizer.h

 #include "llvm/IR/Function.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/PassManager.h"
 
 namespace llvm {
 
 /// Frontend-provided metadata for source location.
 struct LocationMetadata {
-  StringRef Filename;
+  llvm::SmallString<64> Filename;
   int LineNo = 0;
   int ColumnNo = 0;
 
   LocationMetadata() = default;
 
   bool empty() const { return Filename.empty(); }
   void parse(MDNode *MDN);
 };

llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp

for (auto MDN : Globals->operands()) {
     if (auto *Name = cast_or_null<MDString>(MDN->getOperand(2)))
       E.Name = Name->getString();
     ConstantInt *IsDynInit = mdconst::extract<ConstantInt>(MDN->getOperand(3));
     E.IsDynInit |= IsDynInit->isOne();
     ConstantInt *IsBlacklisted =
         mdconst::extract<ConstantInt>(MDN->getOperand(4));
     E.IsBlacklisted |= IsBlacklisted->isOne();
   }
+
+  // Go through the file prefix map to replace prefixes in metadata that may
+  // contain filepaths. Do this once in the globals metadata pass so we don't
+  // need to run over the map in ASan for each metadata instance.
+  NamedMDNode *PrefixMap = M.getNamedMetadata("llvm.asan.file.prefix.map");
+  if (!PrefixMap)
+    return;
+  for (auto &Entry : Entries) {
+    for (auto *MDN : PrefixMap->operands()) {
+      assert(MDN->getNumOperands() == 2 &&
+             "Each node in the prefix map should be a pair");
+      StringRef Prefix = cast<MDString>(MDN->getOperand(0))->getString();
+      StringRef Replacement = cast<MDString>(MDN->getOperand(1))->getString();
+      if (llvm::sys::path::replace_path_prefix(
+              Entry.second.SourceLoc.Filename, Prefix, Replacement,
+              llvm::sys::path::Style::native, true)) {
+        break;
+      }
+    }
+  }
 }
 
 AnalysisKey ASanGlobalsMetadataAnalysis::Key;
 
 GlobalsMetadata ASanGlobalsMetadataAnalysis::run(Module &M,
                                                  ModuleAnalysisManager &AM) {
   return GlobalsMetadata(M);
 }

llvm/test/Instrumentation/AddressSanitizer/asan_file_prefix_map.ll

+; RUN: opt < %s -asan -asan-module -S | FileCheck %s
+
+target triple = "x86_64-unknown-linux-gnu"
+
+@global = global i32 0, align 4
+@global2 = global i32 0, align 4
+
+; Check that the string used for the source location has it's prefix replaced if llvm.asan.file.prefix.map is present.
+; CHECK: @___asan_gen_.{{[0-9]+}} = private unnamed_addr constant [33 x i8] c"/some/other/tmp/asan-globals.cpp\00", align 1
+; CHECK: @___asan_gen_.{{[0-9]+}} = private unnamed_addr constant [36 x i8] c"../some/other/blah/asan-globals.cpp\00", align 1
+
+!llvm.asan.globals = !{!0, !1}
+!llvm.asan.file.prefix.map = !{!101, !102}
+!101 = !{!"/tmp", !"/some/other/tmp"}
+!102 = !{!"blah", !"../some/other/blah"}
+
+!0 = !{i32* @global, !6, !"global", i1 false, i1 false}
+!1 = !{i32* @global2, !7, !"global2", i1 false, i1 false}
+
+!6 = !{!"/tmp/asan-globals.cpp", i32 5, i32 5}
+!7 = !{!"blah/asan-globals.cpp", i32 5, i32 5}

llvm/test/Instrumentation/AddressSanitizer/asan_no_file_prefix_map.ll

+; RUN: opt < %s -asan -asan-module -S | FileCheck %s
+
+target triple = "x86_64-unknown-linux-gnu"
+
+@global = global i32 0, align 4
+@global2 = global i32 0, align 4
+
+; Check that the string used for the source location has its original prefix.
+; CHECK: @___asan_gen_.{{[0-9]+}} = private unnamed_addr constant [22 x i8] c"/tmp/asan-globals.cpp\00", align 1
+; CHECK: @___asan_gen_.{{[0-9]+}} = private unnamed_addr constant [22 x i8] c"blah/asan-globals.cpp\00", align 1
+
+!llvm.asan.globals = !{!0, !1}
+!101 = !{!"/tmp", !"/some/other/tmp"}
+!102 = !{!"blah", !"../some/other/blah"}
+
+!0 = !{i32* @global, !6, !"global", i1 false, i1 false}
+!1 = !{i32* @global2, !7, !"global2", i1 false, i1 false}
+
+!6 = !{!"/tmp/asan-globals.cpp", i32 5, i32 5}
+!7 = !{!"blah/asan-globals.cpp", i32 5, i32 5}