Initial implementation of -fmacro-prefix-map and -ffile-prefix-map

[dankm]

Summary

GCC 8 implements -fmacro-prefix-map. Like -fdebug-prefix-map, it replaces a string prefix for the FILE macro.
-ffile-prefix-map is the union of -fdebug-prefix-map and -fmacro-prefix-map

[erichkeane]

include/clang/Lex/Preprocessor.h

155 ↗

(On Diff #156003)

Scrolling back up, put this implementation as close to the debug implementation as you can, they are so ridiculously related that having them far enough apart that future changes could cause them to divert is troublesome to me.

include/clang/Lex/PreprocessorOptions.h

174

It seems this can be StringRefs as well.

lib/Driver/ToolChains/Clang.cpp

616–617

filtered can take multiple options, you should be able to not add anything here except adding OPT_ffile_prefix_map_EQ to the filtered line, plus a ternary in the Diag line.

628

See advice above.

Additionally/alternatively, I wonder if these two functions could be trivially combined.

1279

Is there a good reason for this to not live with the call to addDebugPrefixMapArg?

lib/Frontend/CompilerInvocation.cpp

3094

Again, this is so much like the debug-prefix otpion, it should probably just be right next to it.

Additionally, we don't use curley brackets on single-line bodies.

lib/Lex/PPMacroExpansion.cpp

1462

make MacroPrefixMap const.

1533

This change shouldn't be necessary, SmallString is still likely the right answer here.

lib/Lex/Preprocessor.cpp

160 ↗

(On Diff #156003)

I'm unconvinced that this is necessary. ExpandBuiltinMacro is in Preprocessor, so it has access to PPOpts directly.

[joerg]

lib/Driver/ToolChains/Clang.cpp

616

I find it confusing that -ffile_prefix_map implies -fdebug-prefix-map. I'm not sure that is desirable in every case. It seems better to have a combined option that explicitly does both.

619

I'd prefer the bailout style here, i.e. if (...) { D.diag(...); continue }

628

Or at least the for loop could be refactored into a small helper function that takes the option name, output option and error as argument.

lib/Lex/PPMacroExpansion.cpp

1465

This doesn't handle directory vs string prefix prefix correctly, does it? At the very least, it should have a test case :)

[dankm]

lib/Driver/ToolChains/Clang.cpp

616

-ffile-prefix-map is the combined option. -fmacro-prefix-map is the preprocessor option, and -fdebug-prefix-map is the codegen option.

628

Good ideas. I'll look into them.

1279

Mostly because this is the function that adds preprocessor specific options. There's no other reason why it couldn't be done alongside addDebugPrefixMapArg in this file.

lib/Frontend/CompilerInvocation.cpp

3094

This is here because it's a preprocessor option. This function handles those. The DebugPrefixMap handling is a codegen option.

lib/Lex/PPMacroExpansion.cpp

1462

I shall.

1465

Good catch. I expect not. But on the other hand, it's exactly what debug-prefix-map does :)

I'll add test cases in a future review. My first goal was getting something sort-of working.

1533

I tried that long ago. It didn't work, I don't remember exactly why. But I agree that SmallString should be enough. I'll dig more.

lib/Lex/Preprocessor.cpp

160 ↗

(On Diff #156003)

It has access to PPOpts, but the implementation file doesn't have a full definition of PreprocessorOptions. I could add that to this file, then this becomes redundant.

[dankm]

Address some of the comments by erichkeane and joerg.

[erichkeane]

include/clang/Basic/DiagnosticDriverKinds.td

118

Since these are otherwise identical, perhaps a %select{...|...} for the flag name?

include/clang/Lex/PreprocessorOptions.h

174

Did you miss this one? Or is there a good reason these cannot be stringrefs?

lib/Driver/ToolChains/Clang.cpp

621

With the continue above, 'else' is unnecessary/against coding standard.

636

See above.

lib/Lex/PPMacroExpansion.cpp

1461

Did clang-format do this formatting? It looks REALLY weird...

1533

Just noting to handle this before approval.

1536

First, comments end in a period. Second, isn't that what the next line does?

[dankm]

include/clang/Lex/PreprocessorOptions.h

174

I didn't miss it. StringRefs here don't survive. The function that adds them to the map creates temporary strings, that go away once that function ends causing StringRefs to dangle. std::string keeps copies.

lib/Driver/ToolChains/Clang.cpp

621

Next diff will have that.

lib/Lex/PPMacroExpansion.cpp

1461

No, that's my text editor. I'll fix it.

1465

There should be a test, but apparently the debug prefix map code also does this.

What do you think the correct behaviour should be? a string prefix, or a directory prefix?

1533

Yup, with some changes to remapMacroPath SmallString works fine.

1536

Yes, old comment is old ;)

[erichkeane]

include/clang/Lex/PreprocessorOptions.h

174

Oh! I hadn't realized that getAllArgValues gives a vector<string>. That is actually pretty odd for our codebase. Looking into it, there is no reason that function cannot return a vector of StringRef...

Alright, at one point someone should likely fix that, but that person should change this type.

[alxu]

FYI, gcc uses the stupid and bad string prefix matching approach. if clang supports the same option, it should have the same behavior. you could decide that it's a bad idea, but then the option should be called something else, otherwise people will have to go back to testing compiler name and versions again which is :(.

[rnk]

lgtm

[Lekensteyn]

The functionality looks correct to me, but could you include some tests in test/Driver/ and test/Preprocessor/ just to be sure?
test/Driver/debug-prefix-map.c and test/CodeGen/debug-prefix-map.c could serve as inspiration.

The documentation should probable be updated too: docs/ClangCommandLineReference.rst

(It would be nice to have this feature for Reproducible Builds)

lib/Lex/PPMacroExpansion.cpp

1465

It should be a string prefix (like GCC)

[joerg]

lib/Lex/PPMacroExpansion.cpp

1465

I disagree. I consider it a bug in GCC that it is a string prefix. It's quite inconsistent as well.

[Lekensteyn]

lib/Lex/PPMacroExpansion.cpp

1465

I agree with you, it should have been a directory prefix but GCC implements it as a string prefix although the GCC documents it as:
"-fdebug-prefix-map=old=new When compiling files residing in directory old, record debugging information describing them as if the files resided in directory new instead."

If you decide to fix -fmacro-prefix-map to use a directory prefix match, then the -fdebug-prefix-map should also be fixed for consistency. What about implementing the (buggy) GCC-compatible behavior first and then fixing both cases in a future patch? (I don't mind when the buggy behavior is fixed, I just want to see this functionality moving forward.)

Another edge case that I ran into is when using the option to drop directories. When using -ffile-prefix-map=/src=, the command cd /src && cc /src/foo.c would have __FILE__ equal to /foo.c. As a native "fix", one would try -ffile-prefix-map=/src/= which indeed produces __FILE__ equal to foo.c.

Matching with a trailing slash however fails to correctly remap some debug information, namely DW_AT_comp_dir. This contains the working directory (/src) which is not matched by /src/. By using a proper directory prefix match, this would be nicely fixed.

[ChristophBerg]

PostgreSQL 11 is now using LLVM to do JITing of SQL expressions. We'd need this feature to strip the build directory off the .bc bitcode files so the .deb packages build reproducibly.
@dankm: Are you still working on this? What can we do to help getting this move forward?

[dankm]

In D49466#1296790, @ChristophBerg wrote:

PostgreSQL 11 is now using LLVM to do JITing of SQL expressions. We'd need this feature to strip the build directory off the .bc bitcode files so the .deb packages build reproducibly.
@dankm: Are you still working on this? What can we do to help getting this move forward?

I am. I'm about to push a new review. Sorry I missed this earlier.

[dankm]

lib/Lex/PPMacroExpansion.cpp

1465

Yes, I'm going to submit my code with tests, and hoist the prefix remapping (for debug-prefix-map and macro-prefix-map) into a common location. Most probably part of Path.

[dankm]

Added unit tests for the prefix remapping.

Switched the sorting on the prefix map, so that <somepath>/sub gets remapped before <somepath> if both are specified.

I intend to do a more invasive change after this review to unify path prefix remapping.

[alxu]

FYI, according to my comment on D49652, assuming I checked it correctly, gcc applies the maps in reverse order of command line specification, not sorted order. It seems unlikely that anyone is actually depending on the order though.

[dankm]

In D49466#1353437, @alxu wrote:

FYI, according to my comment on D49652, assuming I checked it correctly, gcc applies the maps in reverse order of command line specification, not sorted order. It seems unlikely that anyone is actually depending on the order though.

Yeah, I noticed that, but it appears to be undefined by GCC's documentation. I agree with review D49652, but I also want to get this in before 8.0 branches, even if it's not ideal.

Right now we apply them in strict alphabetical order, switching to reverse order lets one map /objdir/<sysroot> to / while remapping /objdir/ to /some/other/dir, which is my use case.

[joerg]

That's the other reason why I find the GCC specification as string prefix confusing. I still say we should just go with mapping of path names and then the order question mostly goes away.

[Lekensteyn]

It would be nice to have this for Clang 8.0, the branch date is within 5 days :)

lib/Driver/ToolChains/Clang.cpp

620

For clang -ffile-prefix-map=foo, wouldn't this report invalid argument 'foo' to -fdebug-prefix-map? If so, perhaps some method of A or A->getOption() can be used?

633

Same concern here about -ffile-prefix-map=foo showing an error message about -fmacro-prefix-map.

test/Preprocessor/file_test.c

6

Any reason to keep this comment?

[dankm]

In D49466#1354238, @Lekensteyn wrote:

It would be nice to have this for Clang 8.0, the branch date is within 5 days :)

Yup, that's why I'm ignoring a new baby for this :)

lib/Driver/ToolChains/Clang.cpp

620

Yes, it would seem so. It looks like A->getOption().getName() can be used.

test/Preprocessor/file_test.c

6

Ha. No. That's from when I started writing this test. It can go away.

[dankm]

Made diagnostics for file-prefix-map display the actual option name.

[Lekensteyn]

Could you add more tests to check the error message for bad options (missing =):

-fdebug-prefix-map=bad
-fmacro-prefix-map=bad
-ffile-prefix-map=bad

FWIW, GCC emits two errors for -ffile-prefix-map=bad.

Another edge case is -ffile-prefix-map==foo/, GCC currently uses this to prepend foo/ to every path. Not sure if that is intentional, but that is the current behavior (one which is also replicated by this patch I believe).

Could you also mark review comments that are completed as "done"? It should make the diff easier to read (I hope) :)

include/clang/Basic/DiagnosticDriverKinds.td

118–119

Maybe rename _to_prefix_map to _to_option? (And maybe swap the order of parameters so %0 comes before %1?)

[dankm]

renamed err_drv_invalid_argument_to_prefix_map to err_drv_invalid_argument_to_option
added more frontend tests for macro-prefix-map and file-prefix-map.

[dankm]

In D49466#1354468, @Lekensteyn wrote:

Could you add more tests to check the error message for bad options (missing =):

-fdebug-prefix-map=bad
-fmacro-prefix-map=bad
-ffile-prefix-map=bad

Some more got added with the latest diff

FWIW, GCC emits two errors for -ffile-prefix-map=bad.

Yes, this does too. It looked odd to me, but it's not a huge deal.

Another edge case is -ffile-prefix-map==foo/, GCC currently uses this to prepend foo/ to every path. Not sure if that is intentional, but that is the current behavior (one which is also replicated by this patch I believe).

Yes, with this patch it does that for file-prefix-map and macro-prefix-map. It already did that (sort-of) for debug-prefix-map, but seems to add it twice for some debugging information, but I'll fix that later since it's done that since at least version 5.0.

Could you also mark review comments that are completed as "done"? It should make the diff easier to read (I hope) :)

Yes, I tried to do that with this comment. I'm new to phabricator.

[Lekensteyn]

Except one thing, it looks reasonable to me. I'll try to run some tests and report back tomorrow.

(Not very familiar with Phabricator either. I still see some comments, hopefully the "Collapse" function does something useful here.)

test/Driver/prefix-map.S

7 ↗

(On Diff #181363)

Maybe restore the old file name (debug-prefix-map.S) since this still tests the debug prefix functionality? And otherwise this comment needs to be updated.

[Lekensteyn]

Tests pass here, using it on a large CMake project with a CMAKE_BUILD_TYPE=Debug and c/cxxflags -ffile-prefix-map=$builddir= -ffile-prefix-map=$srcdir/= -fuse-ld=lld successfully strips all traces of $builddir and $srcdir.

If you could take care of the previous comment (undo the rename or rename debug-prefix.map.c), then I've no further comments.

If @joerg or someone else could give the final review/pass, that would be great :)

lib/Driver/ToolChains/Clang.cpp

619

Wouldn't using if (...) { D.diag(...); continue; } also skip the A->claim() call? Presumably that could result in spurious errors as well about unused arguments?

[dankm]

Restored original test case file names.

[dankm]

lib/Driver/ToolChains/Clang.cpp

619

It would, and did. I had @joerg's suggestion in an earlier patch on this review.

[Lekensteyn]

Still fine by me, thanks!

As for the commit message, perhaps reference:
https://bugs.llvm.org/show_bug.cgi?id=38135

[joerg]

As discussed with dankm on IRC, I still would like to see the correct behavior going into 8.0, i.e. not change it later. Since this also matters for potential faster implementations later, it seems like a good idea to do it now. The changes are well-localized.

(1) Do path prefix matching and not string prefix matching. The difference is that the file name must be longer than the prefix and the prefix must be followed by a path separator.
(2) The longest prefix match wins. Substituation is applied only once per file name, independent of the rules. This gives more predictable output and allows switching to a tree-lookup later.

[dankm]

Enforce path mapping. This requires LLVM review D56769.

[Lekensteyn]

Changes still look reasonable, but the preceding path (https://reviews.llvm.org/D56769) needs some work.

lib/CodeGen/CGDebugInfo.cpp

607

Any reason for dropping remapDIPath here? Wouldn't this result in the full path being included even when using:

clang -fdebug-prefix-map=/full/path/= /full/path/source.c

lib/Lex/PPMacroExpansion.cpp

1466

Style: space between if and (

[dankm]

I'll update the style nit, and spend some non-tired time on the string remapping. Thanks

lib/CodeGen/CGDebugInfo.cpp

607

Whoops. That probably shouldn't have been included this round. This is a bugfix. MainFileName is already remapped from earlier in this function, this keeps it from remapping twice if you have an empty old prefix.

[dankm]

Update style.

[Lekensteyn]

lib/CodeGen/CGDebugInfo.cpp

607

The remapping was done here:

c
  std::string MainFileDir;
  if (const FileEntry *MainFile = SM.getFileEntryForID(SM.getMainFileID())) {
    MainFileDir = remapDIPath(MainFile->getDir()->getName());

(Observation: the declaration could probably be moved inside the if block since it is not used outside.)

What about the second case though? For example, assume /tmp/testdir/mytest.ii:

# 1 "/tmp/mytest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/mytest.c"
int main(int argc, const char *argv[])
{
    return 0;
}

What happens if you now compile with clang -fdebug-prefix-map=/tmp/=/bla/ /tmp/testdir/mytest.ii from /tmp/testdir?

Unless this affects the current patch, consider moving it to a separate change.

[dankm]

Undo accidental change.

[dankm]

Sure, I'll (eventually) make a separate review.

[dankm]

Move trailing path separator stripping back to Clang.

[raj.khem]

lib/CodeGen/CGDebugInfo.cpp

476–478

looking at llvm/lib/Support/Path.cpp replace_path_prefix() returns void but here inside if() it will expect a bool return value

[raj.khem]

lib/CodeGen/CGDebugInfo.cpp

476–478

nm I guess I needed to look into https://reviews.llvm.org/D56769 as well.

[Lekensteyn]

Hi @dankm, any progress on this feature? The proposed branch off date for Clang 9.0.0 is 18 July 2019: https://lists.llvm.org/pipermail/cfe-dev/2019-June/062628.html

[manojgupta]

@dankm are you still working on this patch?

[dankm]

Latest changes. I've been sitting on these for months, so I don't remember all that changed. The path remapping contract changed somewhat, and it's now based on the git monorepo.

[dankm]

In D49466#1586140, @manojgupta wrote:

@dankm are you still working on this patch?

Yes, I've been afk for a bit due to family circumstances, but I just uploaded more.

[Lekensteyn]

Thanks for picking this up again. I've left some nitpicks below in a quick review.

The "strict" parameter is not precisely defined, if that is fixed I think this would be ready for merge.

clang/test/Driver/debug-prefix-map.c

8 ↗

(On Diff #212723)

What about combining these two tests? The command is the same, maybe you could have a new -check-prefix to reduce the number of invocations? Likewise for the cases below.

llvm/include/llvm/Support/Path.h

172 ↗

(On Diff #212723)

"strict checking" is ambiguous on its own. What about something like:

If strict is true, a directory separator following \a OldPrefix will also be stripped. Otherwise, directory separators will only be matched and stripped when present in \a OldPrefix.

Or whatever semantics you would like to assign to "strict mode".

181 ↗

(On Diff #212723)

Why have a variant with the parameters swapped, is it common in LLVM to have such convenience wrappers?

Why not require callers to pass Style::native whenever they want to modify "strict"?

llvm/lib/Support/Path.cpp

512 ↗

(On Diff #212723)

this condition is duplicated above

[phosek]

@dankm do you still plan to work on this? We would really like to see this landed and we could help if needed.

[MaskRay]

It'd be nice to see this land.

[phosek]

@dankm is it OK if we take over this change to push it forward?

[dankm]

llvm/include/llvm/Support/Path.h

172 ↗

(On Diff #212723)

Thanks. I like your wording, and I've used it in the incoming patch.

181 ↗

(On Diff #212723)

Works for me. I did that to make my call sites somewhat more readable. It'll be changed too.

[dankm]

clang/test/Driver/debug-prefix-map.c

8 ↗

(On Diff #212723)

The tests will need more thinking, you're probably right, but I don't have much time to work on this at the moment. How do you feel about addressing this in the future?

[dankm]

• Address feedback from @Lekensteyn

[dankm]

Whoops. There are extra changes here I didn't mean to submit :/

[dankm]

• Address feedback from @Lekensteyn

[dankm]

In D49466#1681534, @phosek wrote:

@dankm is it OK if we take over this change to push it forward?

At this point sure. Unless it's accepted as-is now, then I don't have a commit bit to finish it off.

[dankm]

Ping?

[MaskRay]

In D49466#1760765, @dankm wrote:

Ping?

The tests need fixing... I can commit it. Now that we've migrated to the llvm monorepo, the git commit message can retain the author info properly...

[MaskRay]

Add back remapDIPath that was unintentionally deleted by D69213, caught by a test.

Small adjustment of the code

[MaskRay]

Minimize diff in Options.td
Properly rebase remapDIPath on top of D69213

[dankm]

In D49466#1760860, @MaskRay wrote:

Add back remapDIPath that was unintentionally deleted by D69213, caught by a test.

Small adjustment of the code

Right. Can't believe I missed that. Thanks.

[thakis]

There's still one failing test on Windows after the fix attempt: http://45.33.8.238/win/3052/step_6.txt

Please take a look and revert if it's not an easy fix. (And please watch bots after committing stuff.)

[MaskRay]

In D49466#1761044, @thakis wrote:

There's still one failing test on Windows after the fix attempt: http://45.33.8.238/win/3052/step_6.txt

Please take a look and revert if it's not an easy fix. (And please watch bots after committing stuff.)

XFAIL'ed clang/test/Preprocessor/file_test.c. I didn't receive an email from http://45.33.8.238/win/3052/step_6.txt not sure if that was because the author email was @dankm's, not mine. If there is a console. please tell me :)

[MaskRay]

Does this work on Windows?

--- i/clang/test/Preprocessor/file_test.c
+++ w/clang/test/Preprocessor/file_test.c
@@ -1,8 +1,7 @@
-// XFAIL: system-windows
 // RUN: %clang -E -ffile-prefix-map=%p=/UNLIKELY_PATH/empty -c -o - %s | FileCheck %s
 // RUN: %clang -E -fmacro-prefix-map=%p=/UNLIKELY_PATH/empty -c -o - %s | FileCheck %s
 // RUN: %clang -E -fmacro-prefix-map=%p=/UNLIKELY_PATH=empty -c -o - %s | FileCheck %s -check-prefix CHECK-EVIL
-// RUN: %clang -E -fmacro-prefix-map=%p/= -c -o - %s | FileCheck %s --check-prefix CHECK-REMOVE
+// RUN: %clang -E -fmacro-prefix-map=%/p/= -c -o - %/s | FileCheck %s --check-prefix CHECK-REMOVE

startswith is not ideal because /t will match /tmp. However, gcc appears to use something similar to startswith, see:

% gcc -c -g -fdebug-prefix-map=/t=x /tmp/c/a.c -o /tmp/c/a.o
% llvm-dwarfdump /tmp/c/a.o | grep -m 1 DW_AT_name
              DW_AT_name        ("xmp/c/a.c")

[MaskRay]

The ugly path separator pattern {{(/|\\\\)}} appears in 60+ tests. Can we teach clang and other tools to

1. accept both / and \ input
2. but only output /

on Windows? We can probably remove llvm::sys::path::Style::{windows,posix,native} from include/llvm/Support/Path.h and only keep the posix form.

[probinson]

In D49466#1761156, @MaskRay wrote:

The ugly path separator pattern {{(/|\\\\)}} appears in 60+ tests. Can we teach clang and other tools to

1. accept both / and \ input

In general they do, AFAIK, although it's not feasible in cases where / is the character that introduces an option, which is common on standard Windows utilities.

2. but only output /

on Windows?

This is often actually incorrect for Windows.

We can probably remove llvm::sys::path::Style::{windows,posix,native} from include/llvm/Support/Path.h and only keep the posix form.

It's highly unlikely that will be correct for all cases, and certainly will not match users' expectations. Debug info, for example, will want normal Windows syntax file paths with \.

include/clang/Basic/DiagnosticDriverKinds.td

 def err_drv_invalid_mfloat_abi : Error<
   "invalid float ABI '%0'">;
 def err_drv_invalid_mtp : Error<
   "invalid thread pointer reading mode '%0'">;
 def err_drv_missing_arg_mtp : Error<
   "missing argument to '%0'">;
 def err_drv_invalid_libcxx_deployment : Error<
   "invalid deployment target for -stdlib=libc++ (requires %0 or later)">;
-def err_drv_invalid_argument_to_fdebug_prefix_map : Error<
-  "invalid argument '%0' to -fdebug-prefix-map">;
+def err_drv_invalid_argument_to_option : Error<
+  "invalid argument '%0' to -%1">;
 def err_drv_malformed_sanitizer_blacklist : Error<
   "malformed sanitizer blacklist: '%0'">;
 def err_drv_duplicate_config : Error<
   "no more than one option '--config' is allowed">;
 def err_drv_config_file_not_exist : Error<
   "configuration file '%0' does not exist">;
 def err_drv_config_file_not_found : Error<
   "configuration file '%0' cannot be found">;

include/clang/Driver/Options.td

 def fdebug_ranges_base_address: Flag <["-"], "fdebug-ranges-base-address">, Group<f_Group>,
   Flags<[CC1Option]>, HelpText<"Use DWARF base address selection entries in debug_ranges">;
 def fno_debug_ranges_base_address: Flag <["-"], "fno-debug-ranges-base-address">, Group<f_Group>,
   Flags<[CC1Option]>;
 def fsplit_dwarf_inlining: Flag <["-"], "fsplit-dwarf-inlining">, Group<f_Group>,
   Flags<[CC1Option]>, HelpText<"Provide minimal debug info in the object/executable to facilitate online symbolication/stack traces in the absence of .dwo/.dwp files when using Split DWARF">;
 def fno_split_dwarf_inlining: Flag<["-"], "fno-split-dwarf-inlining">, Group<f_Group>,
   Flags<[CC1Option]>;
+def ffile_prefix_map_EQ
+  : Joined<["-"], "ffile-prefix-map=">, Group<f_Group>, Flags<[CC1Option]>,
+    HelpText<"remap file source paths in debug info and predefined preprocessor macros">;
 def fdebug_prefix_map_EQ
   : Joined<["-"], "fdebug-prefix-map=">, Group<f_Group>,
     Flags<[CC1Option,CC1AsOption]>,
     HelpText<"remap file source paths in debug info">;
+def fmacro_prefix_map_EQ
+  : Joined<["-"], "fmacro-prefix-map=">, Group<Preprocessor_Group>, Flags<[CC1Option]>,
+    HelpText<"remap file source paths in predefined preprocessor macros">;
 def g_Flag : Flag<["-"], "g">, Group<g_Group>,
   HelpText<"Generate source-level debug information">;
 def gline_tables_only : Flag<["-"], "gline-tables-only">, Group<gN_Group>,
   Flags<[CoreOption]>, HelpText<"Emit debug line number tables only">;
 def gline_directives_only : Flag<["-"], "gline-directives-only">, Group<gN_Group>,
   Flags<[CoreOption]>, HelpText<"Emit debug line info directives only">;
 def gmlt : Flag<["-"], "gmlt">, Alias<gline_tables_only>;
 def g0 : Flag<["-"], "g0">, Group<gN_Group>;

include/clang/Lex/PreprocessorOptions.h

public:
   /// The set of modules that failed to build.
   ///
   /// This pointer will be shared among all of the compiler instances created
   /// to (re)build modules, so that once a module fails to build anywhere,
   /// other instances will see that the module has failed and won't try to
   /// build it again.
   std::shared_ptr<FailedModulesSet> FailedModules;
 
+  /// A prefix map for __FILE__ and __BASE_FILE__
+  std::map<std::string, std::string, std::greater<std::string>> MacroPrefixMap;
+
 public:
   PreprocessorOptions() : PrecompiledPreambleBytes(0, false) {}
 
   void addMacroDef(StringRef Name) { Macros.emplace_back(Name, false); }
   void addMacroUndef(StringRef Name) { Macros.emplace_back(Name, true); }
 
   void addRemappedFile(StringRef From, StringRef To) {
     RemappedFiles.emplace_back(From, To);

lib/CodeGen/CGDebugInfo.h

#include "clang/Basic/OpenCLImageTypes.def"
   llvm::DIType *OCLReserveIDDITy = nullptr;
 #define EXT_OPAQUE_TYPE(ExtType, Id, Ext) \
   llvm::DIType *Id##Ty = nullptr;
 #include "clang/Basic/OpenCLExtensionTypes.def"
 
   /// Cache of previously constructed Types.
   llvm::DenseMap<const void *, llvm::TrackingMDRef> TypeCache;
 
-  llvm::SmallDenseMap<llvm::StringRef, llvm::StringRef> DebugPrefixMap;
+  std::map<llvm::StringRef, llvm::StringRef, std::greater<llvm::StringRef>> DebugPrefixMap;
 
   /// Cache that maps VLA types to size expressions for that type,
   /// represented by instantiated Metadata nodes.
   llvm::SmallDenseMap<QualType, llvm::Metadata *> SizeExprCache;
 
   struct ObjCInterfaceCacheEntry {
     const ObjCInterfaceType *Type;
     llvm::DIType *Decl;

lib/CodeGen/CGDebugInfo.cpp

if (llvm::sys::path::is_absolute(RemappedFile)) {
     File = RemappedFile;
   }
   llvm::DIFile *F = DBuilder.createFile(File, Dir, CSInfo, Source);
   DIFileCache[FileName.data()].reset(F);
   return F;
 }
 
 std::string CGDebugInfo::remapDIPath(StringRef Path) const {
+  SmallString<256> p = Path;
   for (const auto &Entry : DebugPrefixMap)
-    if (Path.startswith(Entry.first))
-      return (Twine(Entry.second) + Path.substr(Entry.first.size())).str();
-  return Path.str();
+    if (llvm::sys::path::replace_path_prefix(p, Entry.first, Entry.second))
+      break;
+  return p.str();
 }
 
 unsigned CGDebugInfo::getLineNumber(SourceLocation Loc) {
   if (Loc.isInvalid() && CurLoc.isInvalid())
     return 0;
   SourceManager &SM = CGM.getContext().getSourceManager();
   PresumedLoc PLoc = SM.getPresumedLoc(Loc.isValid() ? Loc : CurLoc);
   return PLoc.isValid() ? PLoc.getLine() : 0;
void CGDebugInfo::CreateCompileUnit() {
   auto &CGOpts = CGM.getCodeGenOpts();
   // The DIFile used by the CU is distinct from the main source
   // file. Its directory part specifies what becomes the
   // DW_AT_comp_dir (the compilation directory), even if the source
   // file was specified with an absolute path.
   if (CSKind)
     CSInfo.emplace(*CSKind, Checksum);
   llvm::DIFile *CUFile = DBuilder.createFile(
-      remapDIPath(MainFileName), remapDIPath(getCurrentDirname()), CSInfo,
+      MainFileName, remapDIPath(getCurrentDirname()), CSInfo,
       getSource(SM, SM.getMainFileID()));
 
   // Create new compile unit.
   TheCU = DBuilder.createCompileUnit(
       LangTag, CUFile, CGOpts.EmitVersionIdentMetadata ? Producer : "",
       LO.Optimize || CGOpts.PrepareForLTO || CGOpts.PrepareForThinLTO,
       CGOpts.DwarfDebugFlags, RuntimeVers,
       (CGOpts.getSplitDwarfMode() != CodeGenOptions::NoFission)

lib/Driver/ToolChains/Clang.cpp

static void addDebugCompDirArg(const ArgList &Args, ArgStringList &CmdArgs) {
   if (!llvm::sys::fs::current_path(cwd)) {
     CmdArgs.push_back("-fdebug-compilation-dir");
     CmdArgs.push_back(Args.MakeArgString(cwd));
   }
 }
 
 /// Add a CC1 and CC1AS option to specify the debug file path prefix map.
 static void addDebugPrefixMapArg(const Driver &D, const ArgList &Args, ArgStringList &CmdArgs) {
-  for (const Arg *A : Args.filtered(options::OPT_fdebug_prefix_map_EQ)) {
+  for (const Arg *A : Args.filtered(options::OPT_ffile_prefix_map_EQ,
+                                    options::OPT_fdebug_prefix_map_EQ)) {
     StringRef Map = A->getValue();
     if (Map.find('=') == StringRef::npos)
-      D.Diag(diag::err_drv_invalid_argument_to_fdebug_prefix_map) << Map;
+      D.Diag(diag::err_drv_invalid_argument_to_option) << Map << A->getOption().getName();
     else
       CmdArgs.push_back(Args.MakeArgString("-fdebug-prefix-map=" + Map));
     A->claim();
   }
 }
 
+/// Add a CC1 and CC1AS option to specify the macro file path prefix map.
+static void addMacroPrefixMapArg(const Driver &D, const ArgList &Args, ArgStringList &CmdArgs) {
+  for (const Arg *A : Args.filtered(options::OPT_ffile_prefix_map_EQ,
+                                    options::OPT_fmacro_prefix_map_EQ)) {
+    StringRef Map = A->getValue();
+    if (Map.find('=') == StringRef::npos)
+      D.Diag(diag::err_drv_invalid_argument_to_option) << Map << A->getOption().getName();
+    else
+      CmdArgs.push_back(Args.MakeArgString("-fmacro-prefix-map=" + Map));
+    A->claim();
+  }
+}
+
 /// Vectorize at all optimization levels greater than 1 except for -Oz.
 /// For -Oz the loop vectorizer is disable, while the slp vectorizer is enabled.
 static bool shouldEnableVectorizerAtOLevel(const ArgList &Args, bool isSlpVec) {
   if (Arg *A = Args.getLastArg(options::OPT_O_Group)) {
     if (A->getOption().matches(options::OPT_O4) ||
         A->getOption().matches(options::OPT_Ofast))
       return true;
 
if (!IsIAMCU)
     forAllAssociatedToolChains(C, JA, getToolChain(),
                                [&Args, &CmdArgs](const ToolChain &TC) {
                                  TC.AddClangSystemIncludeArgs(Args, CmdArgs);
                                });
   else {
     // For IAMCU add special include arguments.
     getToolChain().AddIAMCUIncludeArgs(Args, CmdArgs);
   }
+
+  addMacroPrefixMapArg(D, Args, CmdArgs);
 }
 
 // FIXME: Move to target hook.
 static bool isSignedCharDefault(const llvm::Triple &Triple) {
   switch (Triple.getArch()) {
   default:
     return true;
 

lib/Driver/ToolChains/FreeBSD.cpp

 //===--- FreeBSD.cpp - FreeBSD ToolChain Implementations --------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "FreeBSD.h"
 #include "Arch/ARM.h"
 #include "Arch/Mips.h"
 #include "Arch/Sparc.h"
 #include "CommonArgs.h"
 #include "clang/Driver/Compilation.h"
+#include "clang/Driver/DriverDiagnostic.h"
 #include "clang/Driver/Options.h"
 #include "clang/Driver/SanitizerArgs.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Support/VirtualFileSystem.h"
 
 using namespace clang::driver;
 using namespace clang::driver::tools;
 using namespace clang::driver::toolchains;
 using namespace clang;
 using namespace llvm::opt;
 
 void freebsd::Assembler::ConstructJob(Compilation &C, const JobAction &JA,
                                       const InputInfo &Output,
                                       const InputInfoList &Inputs,
                                       const ArgList &Args,
                                       const char *LinkingOutput) const {
   claimNoWarnArgs(Args);
   ArgStringList CmdArgs;
+  const auto &D = getToolChain().getDriver();
 
   // When building 32-bit code on FreeBSD/amd64, we have to explicitly
   // instruct as in the base system to assemble 32-bit code.
   switch (getToolChain().getArch()) {
   default:
     break;
   case llvm::Triple::x86:
     CmdArgs.push_back("--32");
void freebsd::Assembler::ConstructJob(Compilation &C, const JobAction &JA,
   case llvm::Triple::sparcv9: {
     std::string CPU = getCPUName(Args, getToolChain().getTriple());
     CmdArgs.push_back(sparc::getSparcAsmModeForCPU(CPU, getToolChain().getTriple()));
     AddAssemblerKPIC(getToolChain(), Args, CmdArgs);
     break;
   }
   }
 
+  for (const Arg *A : Args.filtered(options::OPT_ffile_prefix_map_EQ,
+                                    options::OPT_fdebug_prefix_map_EQ)) {
+    StringRef Map = A->getValue();
+    if (Map.find('=') == StringRef::npos)
+      D.Diag(diag::err_drv_invalid_argument_to_option) << Map << A->getOption().getName();
+    else {
+      CmdArgs.push_back(Args.MakeArgString("--debug-prefix-map"));
+      CmdArgs.push_back(Args.MakeArgString(Map));
+    }
+    A->claim();
+  }
+
   Args.AddAllArgValues(CmdArgs, options::OPT_Wa_COMMA, options::OPT_Xassembler);
 
   CmdArgs.push_back("-o");
   CmdArgs.push_back(Output.getFilename());
 
   for (const auto &II : Inputs)
     CmdArgs.push_back(II.getFilename());
 

lib/Driver/ToolChains/Gnu.cpp

case llvm::Triple::systemz: {
     // Always pass an -march option, since our default of z10 is later
     // than the GNU assembler's default.
     StringRef CPUName = systemz::getSystemZTargetCPU(Args);
     CmdArgs.push_back(Args.MakeArgString("-march=" + CPUName));
     break;
   }
   }
 
+  for (const Arg *A : Args.filtered(options::OPT_ffile_prefix_map_EQ,
+                                    options::OPT_fdebug_prefix_map_EQ)) {
+    StringRef Map = A->getValue();
+    if (Map.find('=') == StringRef::npos)
+      D.Diag(diag::err_drv_invalid_argument_to_option) << Map << A->getOption().getName();
+    else {
+      CmdArgs.push_back(Args.MakeArgString("--debug-prefix-map"));
+      CmdArgs.push_back(Args.MakeArgString(Map));
+    }
+    A->claim();
+  }
+
   Args.AddAllArgs(CmdArgs, options::OPT_I);
   Args.AddAllArgValues(CmdArgs, options::OPT_Wa_COMMA, options::OPT_Xassembler);
 
   CmdArgs.push_back("-o");
   CmdArgs.push_back(Output.getFilename());
 
   for (const auto &II : Inputs)
     CmdArgs.push_back(II.getFilename());

lib/Frontend/CompilerInvocation.cpp

static void ParsePreprocessorArgs(PreprocessorOptions &Opts, ArgList &Args,
   Opts.DetailedRecord = Args.hasArg(OPT_detailed_preprocessing_record);
   Opts.DisablePCHValidation = Args.hasArg(OPT_fno_validate_pch);
   Opts.AllowPCHWithCompilerErrors = Args.hasArg(OPT_fallow_pch_with_errors);
 
   Opts.DumpDeserializedPCHDecls = Args.hasArg(OPT_dump_deserialized_pch_decls);
   for (const auto *A : Args.filtered(OPT_error_on_deserialized_pch_decl))
     Opts.DeserializedPCHDeclsToErrorOn.insert(A->getValue());
 
+  for (const auto &A : Args.getAllArgValues(OPT_fmacro_prefix_map_EQ))
+    Opts.MacroPrefixMap.insert(StringRef(A).split('='));
+
   if (const Arg *A = Args.getLastArg(OPT_preamble_bytes_EQ)) {
     StringRef Value(A->getValue());
     size_t Comma = Value.find(',');
     unsigned Bytes = 0;
     unsigned EndOfLine = 0;
 
     if (Comma == StringRef::npos ||
         Value.substr(0, Comma).getAsInteger(10, Bytes) ||

lib/Lex/PPMacroExpansion.cpp

 #include "clang/Lex/DirectoryLookup.h"
 #include "clang/Lex/ExternalPreprocessorSource.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/LexDiagnostic.h"
 #include "clang/Lex/MacroArgs.h"
 #include "clang/Lex/MacroInfo.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorLexer.h"
+#include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Lex/Token.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/FoldingSet.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/Format.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cassert>
 #include <cstddef>
 #include <cstring>
 #include <ctime>
 #include <string>
 #include <tuple>

 /// Implements the __is_target_environment builtin macro.
 static bool isTargetEnvironment(const TargetInfo &TI,
                                 const IdentifierInfo *II) {
   std::string EnvName = (llvm::Twine("---") + II->getName().lower()).str();
   llvm::Triple Env(EnvName);
   return TI.getTriple().getEnvironment() == Env.getEnvironment();
 }
 
+static void remapMacroPath(
+    SmallString<256> &Path,
+    const std::map<std::string, std::string, std::greater<std::string>>
+        &MacroPrefixMap) {
+  for (const auto &Entry : MacroPrefixMap)
+    if (llvm::sys::path::replace_path_prefix(Path, Entry.first, Entry.second))
+      break;
+}
+
 /// ExpandBuiltinMacro - If an identifier token is read that is to be expanded
 /// as a builtin macro, handle it and return the next token as 'Tok'.
 void Preprocessor::ExpandBuiltinMacro(Token &Tok) {
   // Figure out which token this is.
   IdentifierInfo *II = Tok.getIdentifierInfo();
   assert(II && "Can't be a macro without id info!");
 
   // If this is an _Pragma or Microsoft __pragma directive, expand it,
if (II == Ident__BASE_FILE__ && PLoc.isValid()) {
         if (PLoc.isInvalid())
           break;
 
         NextLoc = PLoc.getIncludeLoc();
       }
     }
 
     // Escape this filename.  Turn '\' -> '\\' '"' -> '\"'
-    SmallString<128> FN;
+    SmallString<256> FN;
     if (PLoc.isValid()) {
       FN += PLoc.getFilename();
       Lexer::Stringify(FN);
+      remapMacroPath(FN, PPOpts->MacroPrefixMap);
       OS << '"' << FN << '"';
     }
     Tok.setKind(tok::string_literal);
   } else if (II == Ident__DATE__) {
     Diag(Tok.getLocation(), diag::warn_pp_date_time);
     if (!DATELoc.isValid())
       ComputeDATE_TIME(DATELoc, TIMELoc, *this);
     Tok.setKind(tok::string_literal);

test/CodeGen/debug-prefix-map.c

 // RUN: %clang_cc1 -debug-info-kind=standalone -fdebug-prefix-map=%p=/UNLIKELY_PATH/empty %s -emit-llvm -o - | FileCheck %s -check-prefix CHECK-NO-MAIN-FILE-NAME
 // RUN: %clang_cc1 -debug-info-kind=standalone -fdebug-prefix-map=%p=/UNLIKELY_PATH=empty %s -emit-llvm -o - | FileCheck %s -check-prefix CHECK-EVIL
 // RUN: %clang_cc1 -debug-info-kind=standalone -fdebug-prefix-map=%p=/UNLIKELY_PATH/empty %s -emit-llvm -o - -main-file-name debug-prefix-map.c | FileCheck %s
 // RUN: %clang_cc1 -debug-info-kind=standalone -fdebug-prefix-map=%p=/UNLIKELY_PATH/empty %s -emit-llvm -o - -fdebug-compilation-dir %p | FileCheck %s -check-prefix CHECK-COMPILATION-DIR
+// RUN: %clang -g -fdebug-prefix-map=%p=/UNLIKELY_PATH/empty -S -c %s -emit-llvm -o - | FileCheck %s
+// RUN: %clang -g -ffile-prefix-map=%p=/UNLIKELY_PATH/empty -S -c %s -emit-llvm -o - | FileCheck %s
 
 #include "Inputs/stdio.h"
 
 int main(int argc, char **argv) {
   (void)argc;
   (void)argv;
   return 0;
 }

test/Driver/debug-prefix-map.S

 // RUN: %clang -### -g -fdebug-prefix-map=old=new %s 2>&1 | FileCheck %s
+// RUN: %clang -### -g -ffile-prefix-map=old=new %s 2>&1 | FileCheck %s
 
 // CHECK: cc1as
 // CHECK-SAME: -fdebug-prefix-map=old=new
 
 // More tests for this flag in debug-prefix-map.c.

test/Driver/debug-prefix-map.c

-// RUN: %clang -### -fdebug-prefix-map=old %s 2>&1 | FileCheck %s -check-prefix CHECK-INVALID
-// RUN: %clang -### -fdebug-prefix-map=old=new %s 2>&1 | FileCheck %s -check-prefix CHECK-SIMPLE
-// RUN: %clang -### -fdebug-prefix-map=old=n=ew %s 2>&1 | FileCheck %s -check-prefix CHECK-COMPLEX
-// RUN: %clang -### -fdebug-prefix-map=old= %s 2>&1 | FileCheck %s -check-prefix CHECK-EMPTY
+// RUN: %clang -### -fdebug-prefix-map=old %s 2>&1 | FileCheck %s -check-prefix CHECK-DEBUG-INVALID
+// RUN: %clang -### -fmacro-prefix-map=old %s 2>&1 | FileCheck %s -check-prefix CHECK-MACRO-INVALID
+// RUN: %clang -### -ffile-prefix-map=old %s 2>&1 | FileCheck %s -check-prefix CHECK-FILE-INVALID
 
-// CHECK-INVALID: error: invalid argument 'old' to -fdebug-prefix-map
-// CHECK-SIMPLE: fdebug-prefix-map=old=new
-// CHECK-COMPLEX: fdebug-prefix-map=old=n=ew
-// CHECK-EMPTY: fdebug-prefix-map=old=
+// RUN: %clang -### -fdebug-prefix-map=old=new %s 2>&1 | FileCheck %s -check-prefix CHECK-DEBUG-SIMPLE
+// RUN: %clang -### -fmacro-prefix-map=old=new %s 2>&1 | FileCheck %s -check-prefix CHECK-MACRO-SIMPLE
+// RUN: %clang -### -ffile-prefix-map=old=new %s 2>&1 | FileCheck %s -check-prefix CHECK-DEBUG-SIMPLE
+// RUN: %clang -### -ffile-prefix-map=old=new %s 2>&1 | FileCheck %s -check-prefix CHECK-MACRO-SIMPLE
+
+// RUN: %clang -### -fdebug-prefix-map=old=n=ew %s 2>&1 | FileCheck %s -check-prefix CHECK-DEBUG-COMPLEX
+// RUN: %clang -### -fmacro-prefix-map=old=n=ew %s 2>&1 | FileCheck %s -check-prefix CHECK-MACRO-COMPLEX
+// RUN: %clang -### -ffile-prefix-map=old=n=ew %s 2>&1 | FileCheck %s -check-prefix CHECK-DEBUG-COMPLEX
+// RUN: %clang -### -ffile-prefix-map=old=n=ew %s 2>&1 | FileCheck %s -check-prefix CHECK-MACRO-COMPLEX
+
+// RUN: %clang -### -fdebug-prefix-map=old= %s 2>&1 | FileCheck %s -check-prefix CHECK-DEBUG-EMPTY
+// RUN: %clang -### -fmacro-prefix-map=old= %s 2>&1 | FileCheck %s -check-prefix CHECK-MACRO-EMPTY
+// RUN: %clang -### -ffile-prefix-map=old= %s 2>&1 | FileCheck %s -check-prefix CHECK-DEBUG-EMPTY
+// RUN: %clang -### -ffile-prefix-map=old= %s 2>&1 | FileCheck %s -check-prefix CHECK-MACRO-EMPTY
+
+// CHECK-DEBUG-INVALID: error: invalid argument 'old' to -fdebug-prefix-map
+// CHECK-MACRO-INVALID: error: invalid argument 'old' to -fmacro-prefix-map
+// CHECK-FILE-INVALID: error: invalid argument 'old' to -ffile-prefix-map
+// CHECK-DEBUG-SIMPLE: fdebug-prefix-map=old=new
+// CHECK-MACRO-SIMPLE: fmacro-prefix-map=old=new
+// CHECK-DEBUG-COMPLEX: fdebug-prefix-map=old=n=ew
+// CHECK-MACRO-COMPLEX: fmacro-prefix-map=old=n=ew
+// CHECK-DEBUG-EMPTY: fdebug-prefix-map=old=
+// CHECK-MACRO-EMPTY: fmacro-prefix-map=old=

test/Preprocessor/file_test.h

+filename: __FILE__
+basefile: __BASE_FILE__

test/Preprocessor/file_test.c

+// RUN: %clang -E -ffile-prefix-map=%p=/UNLIKELY_PATH/empty -c -o - %s | FileCheck %s
+// RUN: %clang -E -fmacro-prefix-map=%p=/UNLIKELY_PATH/empty -c -o - %s | FileCheck %s
+// RUN: %clang -E -fmacro-prefix-map=%p=/UNLIKELY_PATH=empty -c -o - %s | FileCheck %s -check-prefix CHECK-EVIL
+// RUN: %clang -E -fmacro-prefix-map=%p/= -c -o - %s | FileCheck %s --check-prefix CHECK-REMOVE
+
+filename: __FILE__
+#include "file_test.h"
+
+// CHECK: filename: "/UNLIKELY_PATH/empty{{[/\\]}}file_test.c"
+// CHECK: filename: "/UNLIKELY_PATH/empty{{[/\\]}}file_test.h"
+// CHECK: basefile: "/UNLIKELY_PATH/empty{{[/\\]}}file_test.c"
+// CHECK-NOT: filename:
+
+// CHECK-EVIL: filename: "/UNLIKELY_PATH=empty{{[/\\]}}file_test.c"
+// CHECK-EVIL: filename: "/UNLIKELY_PATH=empty{{[/\\]}}file_test.h"
+// CHECK-EVIL: basefile: "/UNLIKELY_PATH=empty{{[/\\]}}file_test.c"
+// CHECK-EVIL-NOT: filename:
+
+// CHECK-REMOVE: filename: "file_test.c"
+// CHECK-REMOVE: filename: "file_test.h"
+// CHECK-REMOVE: basefile: "file_test.c"
+// CHECK-REMOVE-NOT: filename: