[LTO][COFF] Use bitcode file names in lto native object file names.

[zequanwu]

Summary

Currently the lto native object files have names like main.exe.lto.1.obj. In
PDB, those names are used as names for each compiland. Microsoft’s tool
SizeBench uses those names to present to users the size of each object files.
So, names like main.exe.lto.1.obj is not user friendly.

This patch makes the lto native object file names more readable by using
the bitcode file names as part of the file names. For example, if the input
bitcode file has path like "path/to/foo.obj", its corresponding lto native
object file path would be "path/to/main.exe.lto.foo.obj". Since the lto native
object file name only bothers PDB, this patch only changes the lld-linker's
behavior.

[zequanwu]

Ping.

[tejohnson]

The naming convention used here for COFF is actually very similar to what is done for other ThinLTO save-temps output files, which are placed at the input module locations. We may want to just do this across the board. @MaskRay wdyt? A few other questions/comments below.

clang/lib/CodeGen/BackendUtil.cpp

1113

I think you might need to mark the new parameter as unused here and in other cases where it isn't used via LLVM_ATTRIBUTE_UNUSED, to avoid build warnings - I can't recall how strictly that is enforced.

lld/COFF/LTO.cpp

231

This case should always be i==0 I think?

247

The above changes affect both the MemoryBufferRef name as well as the saveTemps output file name. I assume the change to the former is what is required for PDB, is that correct?

lld/MachO/LTO.cpp

134

In some files this parameter is called "filed" and in others "originFile". Better to be consistent, and perhaps to use a more descriptive name, something like moduleName ?

llvm/include/llvm/Support/Caching.h

55

This and possibly other header file descriptions need updates.

[zequanwu]

Update.

clang/lib/CodeGen/BackendUtil.cpp

1113

LLVM_ATTRIBUTE_UNUSED is used to suppress unused functions. For unused parameter, I don't see any build warnings when compiling with this patch. I feel like it's very common to have unused parameter.

lld/COFF/LTO.cpp

231

IIUC, "ld-temp.o" is the name of combined module. Do you mean there will be only 1 combined module and it will always be the first task?

247

Yes, it changes both the MemoryBufferRef and the saveTemps output file name otherwise the saveTemps output file name won't match with the the names in pdb. The former is what's written into PDB.

[tejohnson]

@MaskRay wondering if this is a good change to make for ELF as well, wdyt?

clang/lib/CodeGen/BackendUtil.cpp

1113

LLVM_ATTRIBUTE_UNUSED maps to attribute((unused)) which works for either functions or parameters. However, I see where the macro is defined in llvm/include/llvm/Support/Compiler.h that using "(void)unused;" is preferred. Either one works (see below). However, it must be the case that there are no bots building with -Wunused-parameter, since I see Task is also unused here. So nevermind this suggestion, since what you have currently is consistent with what is already done here.

$ cat unused.cc
int foo(int x1, int x2 __attribute__((__unused__)), int x3) {
  (void)x3;
  return 1;
}
$ clang++ -c unused.cc -Wunused-parameter
unused.cc:1:13: warning: unused parameter 'x1' [-Wunused-parameter]
int foo(int x1, int x2 __attribute__((__unused__)), int x3) {
            ^
1 warning generated.

lld/COFF/LTO.cpp

231

Yes. So you don't need the handling for i==0 in the name in this case (you could probably assert that i==0).

llvm/include/llvm/Support/Caching.h

55

Can you add a note that ModuleName is the unique module identifier for the bitcode module the cache is being checked for.

[zequanwu]

Update.

[MaskRay]

In D137217#3926601, @tejohnson wrote:

@MaskRay wondering if this is a good change to make for ELF as well, wdyt?

Yes, I think this is a good idea and improves debuggability. The linker side change is non-trivial (see lld/COFF changes), so making this patch focus on the COFF part is good. I am happy to do the ELF side.

lld/COFF/LTO.cpp

231

This looks like a hack. assert(i == 0) will fail with -opt:lldltopartitions=2: buf[1] will be called ld-temp.o as well.

In addition, if an input bitcode file is called ld-temp.o (for some reason they don't use .obj, just ld-temp.o), assert(i == 0) will fail as well.

240

What if two input bitcode files have the same basename, e.g. dir1/a.obj and dir2/a.obj?

[MaskRay]

lld/COFF/LTO.cpp

231

I guess if (i < config->ltoPartitions) may fix the issue.

[tejohnson]

lld/COFF/LTO.cpp

231

Ah ok, forgot about the lto partitions case. Sorry, @zequanwu, looks like you will need to go back to your old handling that appends i if it is non-zero.

240

I think that should be ok as the output file path created here includes the directory. So you should get dir1/a.out.lto.a.obj and dir2/a.out.lto.a.obj.

[zequanwu]

Address comments.

lld/COFF/LTO.cpp

231

I reverted this part back. Since input bitcode file can be called ld-temp.o, i could be any number smaller than maxTasks, removed assertion.

240

Yes, you will get dir1/a.out.lto.a.obj and dir2/a.out.lto.a.obj.

[tejohnson]

lgtm

[MaskRay]

clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp

847

Twine should only be used as const references as arguments.

[zequanwu]

Update Twine argument to const Twine&.

[thakis]

This doesn't build:
http://45.33.8.238/win/70474/step_4.txt
http://45.33.8.238/linux/92146/step_4.txt

Please take a look and revert for now if it takes a while to fix.

[zequanwu]

Relanded here to match api changes at https://reviews.llvm.org/D135590

[glandium]

This broke the gold plugin:

[task 2022-11-22T21:03:29.486Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1108:19: error: no matching function for call to 'localCache'
[task 2022-11-22T21:03:29.486Z]     Cache = check(localCache("ThinLTO", "Thin", options::cache_dir, AddBuffer));
[task 2022-11-22T21:03:29.487Z]                   ^~~~~~~~~~
[task 2022-11-22T21:03:29.487Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/Support/Caching.h:72:21: note: candidate function not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1102:20)' to 'llvm::AddBufferFn' (aka 'function<void (unsigned int, const llvm::Twine &, std::unique_ptr<MemoryBuffer>)>') for 4th argument
[task 2022-11-22T21:03:29.487Z] Expected<FileCache> localCache(
[task 2022-11-22T21:03:29.488Z]                     ^
[task 2022-11-22T21:03:29.488Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1110:18: error: no viable conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'llvm::AddStreamFn' (aka 'function<Expected<std::unique_ptr<CachedFileStream>> (unsigned int, const llvm::Twine &)>')
[task 2022-11-22T21:03:29.488Z]   check(Lto->run(AddStream, Cache));
[task 2022-11-22T21:03:29.488Z]                  ^~~~~~~~~
[task 2022-11-22T21:03:29.488Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:421:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::nullptr_t' (aka 'nullptr_t') for 1st argument
[task 2022-11-22T21:03:29.488Z]       function(nullptr_t) noexcept
[task 2022-11-22T21:03:29.488Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:432:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'const std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &' for 1st argument
[task 2022-11-22T21:03:29.489Z]       function(const function& __x);
[task 2022-11-22T21:03:29.489Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:441:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &&' for 1st argument
[task 2022-11-22T21:03:29.489Z]       function(function&& __x) noexcept : _Function_base()
[task 2022-11-22T21:03:29.489Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:465:2: note: candidate template ignored: substitution failure [with _Functor = (lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20), $1 = void]: no type named 'type' in 'std::result_of<(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &(unsigned int, const llvm::Twine &)>'
[task 2022-11-22T21:03:29.489Z]         function(_Functor);
[task 2022-11-22T21:03:29.489Z]         ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/LTO/LTO.h:278:25: note: passing argument to parameter 'AddStream' here
[task 2022-11-22T21:03:29.489Z]   Error run(AddStreamFn AddStream, FileCache Cache = nullptr);
[task 2022-11-22T21:03:29.489Z]                         ^
[task 2022-11-22T21:03:29.489Z] 2 errors generated.

[zequanwu]

In D137217#3944952, @glandium wrote:

This broke the gold plugin:

[task 2022-11-22T21:03:29.486Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1108:19: error: no matching function for call to 'localCache'
[task 2022-11-22T21:03:29.486Z]     Cache = check(localCache("ThinLTO", "Thin", options::cache_dir, AddBuffer));
[task 2022-11-22T21:03:29.487Z]                   ^~~~~~~~~~
[task 2022-11-22T21:03:29.487Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/Support/Caching.h:72:21: note: candidate function not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1102:20)' to 'llvm::AddBufferFn' (aka 'function<void (unsigned int, const llvm::Twine &, std::unique_ptr<MemoryBuffer>)>') for 4th argument
[task 2022-11-22T21:03:29.487Z] Expected<FileCache> localCache(
[task 2022-11-22T21:03:29.488Z]                     ^
[task 2022-11-22T21:03:29.488Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1110:18: error: no viable conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'llvm::AddStreamFn' (aka 'function<Expected<std::unique_ptr<CachedFileStream>> (unsigned int, const llvm::Twine &)>')
[task 2022-11-22T21:03:29.488Z]   check(Lto->run(AddStream, Cache));
[task 2022-11-22T21:03:29.488Z]                  ^~~~~~~~~
[task 2022-11-22T21:03:29.488Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:421:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::nullptr_t' (aka 'nullptr_t') for 1st argument
[task 2022-11-22T21:03:29.488Z]       function(nullptr_t) noexcept
[task 2022-11-22T21:03:29.488Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:432:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'const std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &' for 1st argument
[task 2022-11-22T21:03:29.489Z]       function(const function& __x);
[task 2022-11-22T21:03:29.489Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:441:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &&' for 1st argument
[task 2022-11-22T21:03:29.489Z]       function(function&& __x) noexcept : _Function_base()
[task 2022-11-22T21:03:29.489Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:465:2: note: candidate template ignored: substitution failure [with _Functor = (lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20), $1 = void]: no type named 'type' in 'std::result_of<(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &(unsigned int, const llvm::Twine &)>'
[task 2022-11-22T21:03:29.489Z]         function(_Functor);
[task 2022-11-22T21:03:29.489Z]         ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/LTO/LTO.h:278:25: note: passing argument to parameter 'AddStream' here
[task 2022-11-22T21:03:29.489Z]   Error run(AddStreamFn AddStream, FileCache Cache = nullptr);
[task 2022-11-22T21:03:29.489Z]                         ^
[task 2022-11-22T21:03:29.489Z] 2 errors generated.

Thanks for noticing. I have fixed at https://reviews.llvm.org/rG10a43c4641c20e0a50edc0ff99915c837a507cc1. Unfortunately, this patch is reverted right before that fix pushed. I relanded this at https://reviews.llvm.org/rG84be92d26fcb1ddad533c0c614a79a81c59f795d.

[glandium]

Still broken:

task 2022-11-22T22:09:00.912Z] /usr/lib/llvm-11/bin/clang++ --sysroot=/builds/worker/fetches/sysroot -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/gold -I/builds/worker/fetches/llvm-project/llvm/tools/gold -Iinclude -I/builds/worker/fetches/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -fPIC  -fno-exceptions -fno-rtti -std=c++17 -MD -MT tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o -MF tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o.d -o tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o -c /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1111:18: error: no viable conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'llvm::AddStreamFn' (aka 'function<Expected<std::unique_ptr<CachedFileStream>> (unsigned int, const llvm::Twine &)>')
[task 2022-11-22T22:09:00.912Z]   check(Lto->run(AddStream, Cache));
[task 2022-11-22T22:09:00.912Z]                  ^~~~~~~~~
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:421:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::nullptr_t' (aka 'nullptr_t') for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(nullptr_t) noexcept
[task 2022-11-22T22:09:00.912Z]       ^
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:432:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'const std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &' for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(const function& __x);
[task 2022-11-22T22:09:00.912Z]       ^
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:441:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &&' for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(function&& __x) noexcept : _Function_base()
[task 2022-11-22T22:09:00.913Z]       ^
[task 2022-11-22T22:09:00.913Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:465:2: note: candidate template ignored: substitution failure [with _Functor = (lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20), $1 = void]: no type named 'type' in 'std::result_of<(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &(unsigned int, const llvm::Twine &)>'
[task 2022-11-22T22:09:00.913Z]         function(_Functor);
[task 2022-11-22T22:09:00.913Z]         ^
[task 2022-11-22T22:09:00.913Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/LTO/LTO.h:278:25: note: passing argument to parameter 'AddStream' here
[task 2022-11-22T22:09:00.913Z]   Error run(AddStreamFn AddStream, FileCache Cache = nullptr);
[task 2022-11-22T22:09:00.913Z]                         ^
[task 2022-11-22T22:09:00.913Z] 1 error generated.

[zequanwu]

In D137217#3945136, @glandium wrote:

Still broken:

task 2022-11-22T22:09:00.912Z] /usr/lib/llvm-11/bin/clang++ --sysroot=/builds/worker/fetches/sysroot -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/gold -I/builds/worker/fetches/llvm-project/llvm/tools/gold -Iinclude -I/builds/worker/fetches/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -fPIC  -fno-exceptions -fno-rtti -std=c++17 -MD -MT tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o -MF tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o.d -o tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o -c /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1111:18: error: no viable conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'llvm::AddStreamFn' (aka 'function<Expected<std::unique_ptr<CachedFileStream>> (unsigned int, const llvm::Twine &)>')
[task 2022-11-22T22:09:00.912Z]   check(Lto->run(AddStream, Cache));
[task 2022-11-22T22:09:00.912Z]                  ^~~~~~~~~
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:421:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::nullptr_t' (aka 'nullptr_t') for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(nullptr_t) noexcept
[task 2022-11-22T22:09:00.912Z]       ^
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:432:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'const std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &' for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(const function& __x);
[task 2022-11-22T22:09:00.912Z]       ^
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:441:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &&' for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(function&& __x) noexcept : _Function_base()
[task 2022-11-22T22:09:00.913Z]       ^
[task 2022-11-22T22:09:00.913Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:465:2: note: candidate template ignored: substitution failure [with _Functor = (lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20), $1 = void]: no type named 'type' in 'std::result_of<(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &(unsigned int, const llvm::Twine &)>'
[task 2022-11-22T22:09:00.913Z]         function(_Functor);
[task 2022-11-22T22:09:00.913Z]         ^
[task 2022-11-22T22:09:00.913Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/LTO/LTO.h:278:25: note: passing argument to parameter 'AddStream' here
[task 2022-11-22T22:09:00.913Z]   Error run(AddStreamFn AddStream, FileCache Cache = nullptr);
[task 2022-11-22T22:09:00.913Z]                         ^
[task 2022-11-22T22:09:00.913Z] 1 error generated.

Oh, sorry, I missed another place. Hope this will fix it.

[glandium]

Almost there, but not quite:

[task 2022-11-22T23:55:36.341Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1106:6: error: no matching function for call to object of type '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1095:7)'
[task 2022-11-22T23:55:36.341Z]     *AddStream(Task)->OS << MB->getBuffer();
[task 2022-11-22T23:55:36.341Z]      ^~~~~~~~~
[task 2022-11-22T23:55:36.341Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1095:7: note: candidate function not viable: requires 2 arguments, but 1 was provided
[task 2022-11-22T23:55:36.341Z]       [&](size_t Task,
[task 2022-11-22T23:55:36.341Z]       ^
[task 2022-11-22T23:55:36.341Z] 1 error generated.

[MaskRay]

In D137217#3945366, @glandium wrote:

Almost there, but not quite:

[task 2022-11-22T23:55:36.341Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1106:6: error: no matching function for call to object of type '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1095:7)'
[task 2022-11-22T23:55:36.341Z]     *AddStream(Task)->OS << MB->getBuffer();
[task 2022-11-22T23:55:36.341Z]      ^~~~~~~~~
[task 2022-11-22T23:55:36.341Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1095:7: note: candidate function not viable: requires 2 arguments, but 1 was provided
[task 2022-11-22T23:55:36.341Z]       [&](size_t Task,
[task 2022-11-22T23:55:36.341Z]       ^
[task 2022-11-22T23:55:36.341Z] 1 error generated.

ninja LLVMgold.so works for me now. I think @zequanwu fixed this in 10a43c4641c20e0a50edc0ff99915c837a507cc1/5d140dc2c0f3068fae7127b85df861c420848078/d23b63ceccfe526c3a9b17d35032b3666e6816a1

[int3]

I'm trying to add similar support to lld-macho, hence the question :)

lld/COFF/LTO.cpp

183

Any reason why this doesn't instead store the module name in the file_names[task] vector that the cache callback uses? Are the task IDs not unique across each run of ltoObj, regardless of whether we end up using the cache or not?

[zequanwu]

lld/COFF/LTO.cpp

183

Users have to explicitly uses the flag /opt:lldltocache=path to let lld to search for cache files at the given folder. If it's not given, the callback of localCache won't be invoked.

clang/lib/CodeGen/BackendUtil.cpp

static void runThinLTOBackend(
 
   // We can simply import the values mentioned in the combined index, since
   // we should only invoke this using the individual indexes written out
   // via a WriteIndexesThinBackend.
   FunctionImporter::ImportMapTy ImportList;
   if (!lto::initImportList(*M, *CombinedIndex, ImportList))
     return;
 
-  auto AddStream = [&](size_t Task) {
+  auto AddStream = [&](size_t Task, const Twine &ModuleName) {
     return std::make_unique<CachedFileStream>(std::move(OS),
                                               CGOpts.ObjectFilenameForDebug);
   };
   lto::Config Conf;
   if (CGOpts.SaveTempsFilePrefix != "") {
     if (Error E = Conf.addSaveTemps(CGOpts.SaveTempsFilePrefix + ".",
                                     /* UseInputModulePath */ false)) {
       handleAllErrors(std::move(E), [&](ErrorInfoBase &EIB) {

clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp

for (auto &BitcodeInput : BitcodeInputFiles) {
     // Add the bitcode file with its resolved symbols to the LTO job.
     if (Error Err = LTOBackend->add(std::move(*BitcodeFileOrErr), Resolutions))
       return Err;
   }
 
   // Run the LTO job to compile the bitcode.
   size_t MaxTasks = LTOBackend->getMaxTasks();
   SmallVector<StringRef> Files(MaxTasks);
-  auto AddStream = [&](size_t Task) -> std::unique_ptr<CachedFileStream> {
+  auto AddStream =
+      [&](size_t Task,
+          const Twine &ModuleName) -> std::unique_ptr<CachedFileStream> {
     int FD = -1;
     auto &TempFile = Files[Task];
     StringRef Extension = (Triple.isNVPTX()) ? "s" : "o";
     std::string TaskStr = Task ? "." + std::to_string(Task) : "";
     auto TempFileOrErr =
         createOutputFile(sys::path::filename(ExecutableName) + "-device-" +
                              Triple.getTriple() + TaskStr,
                          Extension);

lld/COFF/LTO.h

public:
   BitcodeCompiler();
   ~BitcodeCompiler();
 
   void add(BitcodeFile &f);
   std::vector<InputFile *> compile(COFFLinkerContext &ctx);
 
 private:
   std::unique_ptr<llvm::lto::LTO> ltoObj;
-  std::vector<SmallString<0>> buf;
-  std::vector<std::unique_ptr<MemoryBuffer>> files;
+  std::vector<std::pair<std::string, SmallString<0>>> buf;
+  std::vector<std::pair<std::string, std::unique_ptr<MemoryBuffer>>> files;
   std::unique_ptr<llvm::raw_fd_ostream> indexFile;
   llvm::DenseSet<StringRef> thinIndices;
 };
 }
 
 #endif

lld/COFF/LTO.cpp

std::vector<InputFile *> BitcodeCompiler::compile(COFFLinkerContext &ctx) {
   buf.resize(maxTasks);
   files.resize(maxTasks);
 
   // The /lldltocache option specifies the path to a directory in which to cache
   // native object files for ThinLTO incremental builds. If a path was
   // specified, configure LTO to use it as the cache directory.
   FileCache cache;
   if (!config->ltoCache.empty())
-    cache =
-        check(localCache("ThinLTO", "Thin", config->ltoCache,
-                         [&](size_t task, std::unique_ptr<MemoryBuffer> mb) {
-                           files[task] = std::move(mb);
+    cache = check(localCache("ThinLTO", "Thin", config->ltoCache,
+                             [&](size_t task, const Twine &moduleName,
+                                 std::unique_ptr<MemoryBuffer> mb) {
+                               files[task].first = moduleName.str();
+                               files[task].second = std::move(mb);
                              }));
 
   checkError(ltoObj->run(
-      [&](size_t task) {
+      [&](size_t task, const Twine &moduleName) {
+        buf[task].first = moduleName.str();
         return std::make_unique<CachedFileStream>(
-            std::make_unique<raw_svector_ostream>(buf[task]));
+            std::make_unique<raw_svector_ostream>(buf[task].second));
       },
       cache));
 
   // Emit empty index files for non-indexed files
   for (StringRef s : thinIndices) {
     std::string path = getThinLTOOutputFile(s);
     openFile(path + ".thinlto.bc");
     if (config->thinLTOEmitImportsFiles)
       openFile(path + ".imports");
   }
 
   // ThinLTO with index only option is required to generate only the index
   // files. After that, we exit from linker and ThinLTO backend runs in a
   // distributed environment.
   if (config->thinLTOIndexOnly) {
     if (!config->ltoObjPath.empty())
-      saveBuffer(buf[0], config->ltoObjPath);
+      saveBuffer(buf[0].second, config->ltoObjPath);
     if (indexFile)
       indexFile->close();
     return {};
   }
 
   if (!config->ltoCache.empty())
     pruneCache(config->ltoCache, config->ltoCachePolicy, files);
 
   std::vector<InputFile *> ret;
   for (unsigned i = 0; i != maxTasks; ++i) {
-    // Assign unique names to LTO objects. This ensures they have unique names
-    // in the PDB if one is produced. The names should look like:
-    // - foo.exe.lto.obj
-    // - foo.exe.lto.1.obj
-    // - ...
-    StringRef ltoObjName =
-        saver().save(Twine(config->outputFile) + ".lto" +
-                     (i == 0 ? Twine("") : Twine('.') + Twine(i)) + ".obj");
-
+    StringRef bitcodeFilePath;
     // Get the native object contents either from the cache or from memory.  Do
     // not use the cached MemoryBuffer directly, or the PDB will not be
     // deterministic.
     StringRef objBuf;
-    if (files[i])
-      objBuf = files[i]->getBuffer();
-    else
-      objBuf = buf[i];
+    if (files[i].second) {
+      objBuf = files[i].second->getBuffer();
+      bitcodeFilePath = files[i].first;
+    } else {
+      objBuf = buf[i].second;
+      bitcodeFilePath = buf[i].first;
+    }
     if (objBuf.empty())
       continue;
 
+    // If the input bitcode file is path/to/a.obj, then the corresponding lto
+    // object file name will soemthing like: path/to/main.exe.lto.a.obj.
+    StringRef ltoObjName;
+    if (bitcodeFilePath == "ld-temp.o") {
+      ltoObjName =
+          saver().save(Twine(config->outputFile) + ".lto" +
+                       (i == 0 ? Twine("") : Twine('.') + Twine(i)) + ".obj");
+    } else {
+      StringRef directory = sys::path::parent_path(bitcodeFilePath);
+      StringRef baseName = sys::path::filename(bitcodeFilePath);
+      StringRef outputFileBaseName = sys::path::filename(config->outputFile);
+      SmallString<64> path;
+      sys::path::append(path, directory,
+                        outputFileBaseName + ".lto." + baseName);
+      sys::path::remove_dots(path, true);
+      ltoObjName = saver().save(path.str());
+    }
     if (config->saveTemps)
-      saveBuffer(buf[i], ltoObjName);
+      saveBuffer(buf[i].second, ltoObjName);
     ret.push_back(make<ObjFile>(ctx, MemoryBufferRef(objBuf, ltoObjName)));
   }
 
   return ret;
 }

lld/ELF/LTO.cpp

std::vector<InputFile *> BitcodeCompiler::compile() {
   buf.resize(maxTasks);
   files.resize(maxTasks);
 
   // The --thinlto-cache-dir option specifies the path to a directory in which
   // to cache native object files for ThinLTO incremental builds. If a path was
   // specified, configure LTO to use it as the cache directory.
   FileCache cache;
   if (!config->thinLTOCacheDir.empty())
-    cache =
-        check(localCache("ThinLTO", "Thin", config->thinLTOCacheDir,
-                         [&](size_t task, std::unique_ptr<MemoryBuffer> mb) {
+    cache = check(localCache("ThinLTO", "Thin", config->thinLTOCacheDir,
+                             [&](size_t task, const Twine &moduleName,
+                                 std::unique_ptr<MemoryBuffer> mb) {
                                files[task] = std::move(mb);
                              }));
 
   if (!ctx.bitcodeFiles.empty())
     checkError(ltoObj->run(
-        [&](size_t task) {
+        [&](size_t task, const Twine &moduleName) {
           return std::make_unique<CachedFileStream>(
               std::make_unique<raw_svector_ostream>(buf[task]));
         },
         cache));
 
   // Emit empty index files for non-indexed files but not in single-module mode.
   if (config->thinLTOModulesToCompile.empty()) {
     for (StringRef s : thinIndices) {

lld/MachO/LTO.cpp

std::vector<ObjFile *> BitcodeCompiler::compile() {
   buf.resize(maxTasks);
   files.resize(maxTasks);
 
   // The -cache_path_lto option specifies the path to a directory in which
   // to cache native object files for ThinLTO incremental builds. If a path was
   // specified, configure LTO to use it as the cache directory.
   FileCache cache;
   if (!config->thinLTOCacheDir.empty())
-    cache =
-        check(localCache("ThinLTO", "Thin", config->thinLTOCacheDir,
-                         [&](size_t task, std::unique_ptr<MemoryBuffer> mb) {
+    cache = check(localCache("ThinLTO", "Thin", config->thinLTOCacheDir,
+                             [&](size_t task, const Twine &moduleName,
+                                 std::unique_ptr<MemoryBuffer> mb) {
                                files[task] = std::move(mb);
                              }));
 
   checkError(ltoObj->run(
-      [&](size_t task) {
+      [&](size_t task, const Twine &moduleName) {
         return std::make_unique<CachedFileStream>(
             std::make_unique<raw_svector_ostream>(buf[task]));
       },
       cache));
 
   if (!config->thinLTOCacheDir.empty())
     pruneCache(config->thinLTOCacheDir, config->thinLTOCachePolicy, files);
 

lld/test/COFF/lto-obj-path.ll

 ; REQUIRES: x86
 
 ; Test to ensure that thinlto-index-only with lto-obj-path creates
 ; the native object file.
-; RUN: opt -module-summary %s -o %t1.obj
-; RUN: opt -module-summary %p/Inputs/thinlto.ll -o %t2.obj
-; RUN: rm -f %t4.obj
-; RUN: lld-link -thinlto-index-only -lto-obj-path:%t4.obj -out:t3.exe \
-; RUN:     -entry:main %t1.obj %t2.obj
-; RUN: llvm-readobj -h %t4.obj | FileCheck %s
-; RUN: llvm-nm %t4.obj 2>&1 | FileCheck %s -check-prefix=SYMBOLS
-; RUN: llvm-nm %t4.obj 2>&1 | count 1
+; RUN: rm -rf %t.dir/objpath && mkdir -p %t.dir/objpath
+; RUN: opt -module-summary %s -o %t.dir/objpath/t1.obj
+; RUN: opt -module-summary %p/Inputs/thinlto.ll -o %t.dir/objpath/t2.obj
+; RUN: rm -f %t.dir/objpath/t4.obj
+; RUN: lld-link -thinlto-index-only -lto-obj-path:%t.dir/objpath/t4.obj \
+; RUN:     -out:%t.dir/objpath/t3.exe -entry:main %t.dir/objpath/t1.obj \
+; RUN:     %t.dir/objpath/t2.obj
+; RUN: llvm-readobj -h %t.dir/objpath/t4.obj | FileCheck %s
+; RUN: llvm-nm %t.dir/objpath/t4.obj 2>&1 | FileCheck %s -check-prefix=SYMBOLS
+; RUN: llvm-nm %t.dir/objpath/t4.obj 2>&1 | count 1
 
 ;; Ensure lld emits empty combined module if specific obj-path.
-; RUN: rm -fr %t.dir/objpath && mkdir -p %t.dir/objpath
-; RUN: lld-link /out:%t.dir/objpath/a.exe -lto-obj-path:%t4.obj \
-; RUN:     -entry:main %t1.obj %t2.obj -lldsavetemps
-; RUN: ls %t.dir/objpath/a.exe.lto.* | count 3
+; RUN: lld-link /out:%t.dir/objpath/a.exe -lto-obj-path:%t.dir/objpath/t4.obj \
+; RUN:     -entry:main %t.dir/objpath/t1.obj %t.dir/objpath/t2.obj -lldsavetemps
+; RUN: ls %t.dir/objpath/a.exe.lto.obj
+; RUN: ls %t.dir/objpath/a.exe.lto.t1.obj
+; RUN: ls %t.dir/objpath/a.exe.lto.t2.obj
 
 ;; Ensure lld does not emit empty combined module in default.
-; RUN: rm -fr %t.dir/objpath && mkdir -p %t.dir/objpath
+; RUN: rm %t.dir/objpath/a.exe.lto.*
 ; RUN: lld-link /out:%t.dir/objpath/a.exe \
-; RUN:     -entry:main %t1.obj %t2.obj -lldsavetemps
-; RUN: ls %t.dir/objpath/a.exe.lto.* | count 2
+; RUN:     -entry:main %t.dir/objpath/t1.obj %t.dir/objpath/t2.obj -lldsavetemps
+; RUN: ls %t.dir/objpath/a.exe.lto.t1.obj
+; RUN: ls %t.dir/objpath/a.exe.lto.t2.obj
+; RUN: not ls %t.dir/objpath/a.exe.lto.obj
 
 ; CHECK: Format: COFF-x86-64
 ; SYMBOLS: @feat.00
 
 target datalayout = "e-m:w-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-pc-windows-msvc19.0.24215"
 
 declare void @g(...)
 
 define void @main() {
   call void (...) @g()
   ret void
 }

lld/test/COFF/pdb-thinlto.ll

define i32 @main() {
   call void @foo()
   ret i32 0
 }
 
 declare void @foo()
 
 ; CHECK:                           Modules
 ; CHECK: ============================================================
-; CHECK: Mod 0000 | `{{.*}}main.exe.lto.1.obj`:
-; CHECK: Obj: `{{.*}}main.exe.lto.1.obj`:
-; CHECK: Mod 0001 | `{{.*}}main.exe.lto.2.obj`:
-; CHECK: Obj: `{{.*}}main.exe.lto.2.obj`:
+; CHECK: Mod 0000 | `{{.*}}main.exe.lto.main.bc`:
+; CHECK: Obj: `{{.*}}main.exe.lto.main.bc`:
+; CHECK: Mod 0001 | `{{.*}}main.exe.lto.foo.bc`:
+; CHECK: Obj: `{{.*}}main.exe.lto.foo.bc`:
 ; CHECK: Mod 0002 | `* Linker *`:

lld/test/COFF/thinlto.ll

 ; REQUIRES: x86
 ; RUN: rm -fr %T/thinlto
 ; RUN: mkdir %T/thinlto
 ; RUN: opt -thinlto-bc -o %T/thinlto/main.obj %s
 ; RUN: opt -thinlto-bc -o %T/thinlto/foo.obj %S/Inputs/lto-dep.ll
 ; RUN: lld-link /lldsavetemps /out:%T/thinlto/main.exe /entry:main /subsystem:console %T/thinlto/main.obj %T/thinlto/foo.obj
-; RUN: llvm-nm %T/thinlto/main.exe.lto.1.obj | FileCheck %s
+; RUN: llvm-nm %T/thinlto/main.exe.lto.foo.obj | FileCheck %s
 
 ; Test various possible options for /opt:lldltojobs
 ; RUN: lld-link /lldsavetemps /out:%T/thinlto/main.exe /entry:main /subsystem:console %T/thinlto/main.obj %T/thinlto/foo.obj /opt:lldltojobs=1
-; RUN: llvm-nm %T/thinlto/main.exe.lto.1.obj | FileCheck %s
+; RUN: llvm-nm %T/thinlto/main.exe.lto.foo.obj | FileCheck %s
 ; RUN: lld-link /lldsavetemps /out:%T/thinlto/main.exe /entry:main /subsystem:console %T/thinlto/main.obj %T/thinlto/foo.obj /opt:lldltojobs=all
-; RUN: llvm-nm %T/thinlto/main.exe.lto.1.obj | FileCheck %s
+; RUN: llvm-nm %T/thinlto/main.exe.lto.foo.obj | FileCheck %s
 ; RUN: lld-link /lldsavetemps /out:%T/thinlto/main.exe /entry:main /subsystem:console %T/thinlto/main.obj %T/thinlto/foo.obj /opt:lldltojobs=100
-; RUN: llvm-nm %T/thinlto/main.exe.lto.1.obj | FileCheck %s
+; RUN: llvm-nm %T/thinlto/main.exe.lto.foo.obj | FileCheck %s
 ; RUN: not lld-link /lldsavetemps /out:%T/thinlto/main.exe /entry:main /subsystem:console %T/thinlto/main.obj %T/thinlto/foo.obj /opt:lldltojobs=foo 2>&1 | FileCheck %s --check-prefix=BAD-JOBS
 ; BAD-JOBS: error: /opt:lldltojobs: invalid job count: foo
 
 ; This command will store full path to foo.obj in the archive %t.lib
 ; Check that /lldsavetemps is still usable in such case.
 ; RUN: lld-link /lib %T/thinlto/foo.obj /out:%t.lib
 ; RUN: lld-link /lldsavetemps /out:%t.exe /entry:main /subsystem:console %T/thinlto/main.obj %t.lib
 

lld/wasm/LTO.cpp

std::vector<StringRef> BitcodeCompiler::compile() {
   buf.resize(maxTasks);
   files.resize(maxTasks);
 
   // The --thinlto-cache-dir option specifies the path to a directory in which
   // to cache native object files for ThinLTO incremental builds. If a path was
   // specified, configure LTO to use it as the cache directory.
   FileCache cache;
   if (!config->thinLTOCacheDir.empty())
-    cache =
-        check(localCache("ThinLTO", "Thin", config->thinLTOCacheDir,
-                         [&](size_t task, std::unique_ptr<MemoryBuffer> mb) {
+    cache = check(localCache("ThinLTO", "Thin", config->thinLTOCacheDir,
+                             [&](size_t task, const Twine &moduleName,
+                                 std::unique_ptr<MemoryBuffer> mb) {
                                files[task] = std::move(mb);
                              }));
 
   checkError(ltoObj->run(
-      [&](size_t task) {
+      [&](size_t task, const Twine &moduleName) {
         return std::make_unique<CachedFileStream>(
             std::make_unique<raw_svector_ostream>(buf[task]));
       },
       cache));
 
   if (!config->thinLTOCacheDir.empty())
     pruneCache(config->thinLTOCacheDir, config->thinLTOCachePolicy, files);
 

lldb/source/Core/DataFileCache.cpp

DataFileCache::DataFileCache(llvm::StringRef path, llvm::CachePruningPolicy policy) {
   pruneCache(path, policy);
 
   // This lambda will get called when the data is gotten from the cache and
   // also after the data was set for a given key. We only need to take
   // ownership of the data if we are geting the data, so we use the
   // m_take_ownership member variable to indicate if we need to take
   // ownership.
 
-  auto add_buffer = [this](unsigned task, std::unique_ptr<llvm::MemoryBuffer> m) {
+  auto add_buffer = [this](unsigned task, const llvm::Twine &moduleName,
+                           std::unique_ptr<llvm::MemoryBuffer> m) {
     if (m_take_ownership)
       m_mem_buff_up = std::move(m);
   };
   llvm::Expected<llvm::FileCache> cache_or_err =
       llvm::localCache("LLDBModuleCache", "lldb-module", path, add_buffer);
   if (cache_or_err)
     m_cache_callback = std::move(*cache_or_err);
   else {
DataFileCache::GetCachedData(llvm::StringRef key) {
 
   const unsigned task = 1;
   m_take_ownership = true;
   // If we call the "m_cache_callback" function and the data is cached, it will
   // call the "add_buffer" lambda function from the constructor which will in
   // turn take ownership of the member buffer that is passed to the callback and
   // put it into a member variable.
   llvm::Expected<llvm::AddStreamFn> add_stream_or_err =
-      m_cache_callback(task, key);
+      m_cache_callback(task, key, "");
   m_take_ownership = false;
   // At this point we either already called the "add_buffer" lambda with
   // the data or we haven't. We can tell if we got the cached data by checking
   // the add_stream function pointer value below.
   if (add_stream_or_err) {
     llvm::AddStreamFn &add_stream = *add_stream_or_err;
     // If the "add_stream" is nullptr, then the data was cached and we already
     // called the "add_buffer" lambda. If it is valid, then if we were to call

 bool DataFileCache::SetCachedData(llvm::StringRef key,
                                   llvm::ArrayRef<uint8_t> data) {
   std::lock_guard<std::mutex> guard(m_mutex);
   const unsigned task = 2;
   // If we call this function and the data is cached, it will call the
   // add_buffer lambda function from the constructor which will ignore the
   // data.
   llvm::Expected<llvm::AddStreamFn> add_stream_or_err =
-      m_cache_callback(task, key);
+      m_cache_callback(task, key, "");
   // If we reach this code then we either already called the callback with
   // the data or we haven't. We can tell if we had the cached data by checking
   // the CacheAddStream function pointer value below.
   if (add_stream_or_err) {
     llvm::AddStreamFn &add_stream = *add_stream_or_err;
     // If the "add_stream" is nullptr, then the data was cached. If it is
     // valid, then if we call the add_stream function with a task it will
     // cause the file to get generated, but we only want to check if the data
     // is cached here, so we don't want to call it here. Note that the
     // add_buffer will also get called in this case after the data has been
     // provided, but we won't take ownership of the memory buffer as we just
     // want to write the data.
     if (add_stream) {
       llvm::Expected<std::unique_ptr<llvm::CachedFileStream>> file_or_err =
-          add_stream(task);
+          add_stream(task, "");
       if (file_or_err) {
         llvm::CachedFileStream *cfs = file_or_err->get();
         cfs->OS->write((const char *)data.data(), data.size());
         return true;
       } else {
         Log *log = GetLog(LLDBLog::Modules);
         LLDB_LOG_ERROR(log, file_or_err.takeError(),
                        "failed to get the cache file stream for key: {0}");

llvm/include/llvm/Support/Caching.h

public:
   std::unique_ptr<raw_pwrite_stream> OS;
   std::string ObjectPathName;
   virtual ~CachedFileStream() = default;
 };
 
 /// This type defines the callback to add a file that is generated on the fly.
 ///
 /// Stream callbacks must be thread safe.
-using AddStreamFn =
-    std::function<Expected<std::unique_ptr<CachedFileStream>>(unsigned Task)>;
+using AddStreamFn = std::function<Expected<std::unique_ptr<CachedFileStream>>(
+    unsigned Task, const Twine &ModuleName)>;
 
 /// This is the type of a file cache. To request an item from the cache, pass a
 /// unique string as the Key. For hits, the cached file will be added to the
 /// link and this function will return AddStreamFn(). For misses, the cache will
 /// return a stream callback which must be called at most once to produce
 /// content for the stream. The file stream produced by the stream callback will
-/// add the file to the link after the stream is written to.
+/// add the file to the link after the stream is written to. ModuleName is the
+/// unique module identifier for the bitcode module the cache is being checked
+/// for.
 ///
 /// Clients generally look like this:
 ///
-/// if (AddStreamFn AddStream = Cache(Task, Key))
+/// if (AddStreamFn AddStream = Cache(Task, Key, ModuleName))
 ///   ProduceContent(AddStream);
-using FileCache =
-    std::function<Expected<AddStreamFn>(unsigned Task, StringRef Key)>;
+using FileCache = std::function<Expected<AddStreamFn>(
+    unsigned Task, StringRef Key, const Twine &ModuleName)>;
 
 /// This type defines the callback to add a pre-existing file (e.g. in a cache).
 ///
 /// Buffer callbacks must be thread safe.
-using AddBufferFn =
-    std::function<void(unsigned Task, std::unique_ptr<MemoryBuffer> MB)>;
+using AddBufferFn = std::function<void(unsigned Task, const Twine &ModuleName,
+                                       std::unique_ptr<MemoryBuffer> MB)>;
 
 /// Create a local file system cache which uses the given cache name, temporary
 /// file prefix, cache directory and file callback.  This function does not
 /// immediately create the cache directory if it does not yet exist; this is
 /// done lazily the first time a file is added.  The cache name appears in error
 /// messages for errors during caching. The temporary file prefix is used in the
 /// temporary file naming scheme used when writing files atomically.
 Expected<FileCache> localCache(
-    Twine CacheNameRef, Twine TempFilePrefixRef, Twine CacheDirectoryPathRef,
-    AddBufferFn AddBuffer = [](size_t Task, std::unique_ptr<MemoryBuffer> MB) {
-    });
+    const Twine &CacheNameRef, const Twine &TempFilePrefixRef,
+    const Twine &CacheDirectoryPathRef,
+    AddBufferFn AddBuffer = [](size_t Task, const Twine &ModuleName,
+                               std::unique_ptr<MemoryBuffer> MB) {});
 } // namespace llvm
 
 #endif

llvm/lib/Debuginfod/Debuginfod.cpp

Expected<std::string> getCachedOrDownloadArtifact(
   Expected<FileCache> CacheOrErr =
       localCache("Debuginfod-client", ".debuginfod-client", CacheDirectoryPath);
   if (!CacheOrErr)
     return CacheOrErr.takeError();
 
   FileCache Cache = *CacheOrErr;
   // We choose an arbitrary Task parameter as we do not make use of it.
   unsigned Task = 0;
-  Expected<AddStreamFn> CacheAddStreamOrErr = Cache(Task, UniqueKey);
+  Expected<AddStreamFn> CacheAddStreamOrErr = Cache(Task, UniqueKey, "");
   if (!CacheAddStreamOrErr)
     return CacheAddStreamOrErr.takeError();
   AddStreamFn &CacheAddStream = *CacheAddStreamOrErr;
   if (!CacheAddStream)
     return std::string(AbsCachedArtifactPath);
   // The artifact was not found in the local cache, query the debuginfod
   // servers.
   if (!HTTPClient::isAvailable())
Expected<std::string> getCachedOrDownloadArtifact(
   HTTPClient Client;
   Client.setTimeout(Timeout);
   for (StringRef ServerUrl : DebuginfodUrls) {
     SmallString<64> ArtifactUrl;
     sys::path::append(ArtifactUrl, sys::path::Style::posix, ServerUrl, UrlPath);
 
     // Perform the HTTP request and if successful, write the response body to
     // the cache.
-    StreamedHTTPResponseHandler Handler([&]() { return CacheAddStream(Task); },
-                                        Client);
+    StreamedHTTPResponseHandler Handler(
+        [&]() { return CacheAddStream(Task, ""); }, Client);
     HTTPRequest Request(ArtifactUrl);
     Request.Headers = getHeaders();
     Error Err = Client.perform(Request, Handler);
     if (Err)
       return std::move(Err);
 
     if (Client.responseCode() != 200)
       continue;

llvm/lib/LTO/LTO.cpp

if (!Cache || !CombinedIndex.modulePaths().count(ModuleID) ||
       // no module hash.
       return RunThinBackend(AddStream);
 
     SmallString<40> Key;
     // The module may be cached, this helps handling it.
     computeLTOCacheKey(Key, Conf, CombinedIndex, ModuleID, ImportList,
                        ExportList, ResolvedODR, DefinedGlobals, CfiFunctionDefs,
                        CfiFunctionDecls);
-    Expected<AddStreamFn> CacheAddStreamOrErr = Cache(Task, Key);
+    Expected<AddStreamFn> CacheAddStreamOrErr = Cache(Task, Key, ModuleID);
     if (Error Err = CacheAddStreamOrErr.takeError())
       return Err;
     AddStreamFn &CacheAddStream = *CacheAddStreamOrErr;
     if (CacheAddStream)
       return RunThinBackend(CacheAddStream);
 
     return Error::success();
   }

llvm/lib/LTO/LTOBackend.cpp

static void codegen(const Config &Conf, TargetMachine *TM,
   if (!DwoFile.empty()) {
     std::error_code EC;
     DwoOut = std::make_unique<ToolOutputFile>(DwoFile, EC, sys::fs::OF_None);
     if (EC)
       report_fatal_error(Twine("Failed to open ") + DwoFile + ": " +
                          EC.message());
   }
 
-  Expected<std::unique_ptr<CachedFileStream>> StreamOrErr = AddStream(Task);
+  Expected<std::unique_ptr<CachedFileStream>> StreamOrErr =
+      AddStream(Task, Mod.getModuleIdentifier());
   if (Error Err = StreamOrErr.takeError())
     report_fatal_error(std::move(Err));
   std::unique_ptr<CachedFileStream> &Stream = *StreamOrErr;
   TM->Options.ObjectFilenameForDebug = Stream->ObjectPathName;
 
   legacy::PassManager CodeGenPasses;
   TargetLibraryInfoImpl TLII(Triple(Mod.getTargetTriple()));
   CodeGenPasses.add(new TargetLibraryInfoWrapperPass(TLII));

llvm/lib/LTO/LTOCodeGenerator.cpp

 
 bool LTOCodeGenerator::compileOptimizedToFile(const char **Name) {
   if (useAIXSystemAssembler())
     setFileType(CGFT_AssemblyFile);
 
   // make unique temp output file to put generated code
   SmallString<128> Filename;
 
-  auto AddStream = [&](size_t Task) -> std::unique_ptr<CachedFileStream> {
+  auto AddStream =
+      [&](size_t Task,
+          const Twine &ModuleName) -> std::unique_ptr<CachedFileStream> {
     StringRef Extension(Config.CGFileType == CGFT_AssemblyFile ? "s" : "o");
 
     int FD;
     std::error_code EC =
         sys::fs::createTemporaryFile("lto-llvm", Extension, FD, Filename);
     if (EC)
       emitError(EC.message());
 

llvm/lib/Support/Caching.cpp

 #if !defined(_MSC_VER) && !defined(__MINGW32__)
 #include <unistd.h>
 #else
 #include <io.h>
 #endif
 
 using namespace llvm;
 
-Expected<FileCache> llvm::localCache(Twine CacheNameRef,
-                                     Twine TempFilePrefixRef,
-                                     Twine CacheDirectoryPathRef,
+Expected<FileCache> llvm::localCache(const Twine &CacheNameRef,
+                                     const Twine &TempFilePrefixRef,
+                                     const Twine &CacheDirectoryPathRef,
                                      AddBufferFn AddBuffer) {
 
   // Create local copies which are safely captured-by-copy in lambdas
   SmallString<64> CacheName, TempFilePrefix, CacheDirectoryPath;
   CacheNameRef.toVector(CacheName);
   TempFilePrefixRef.toVector(TempFilePrefix);
   CacheDirectoryPathRef.toVector(CacheDirectoryPath);
 
-  return [=](unsigned Task, StringRef Key) -> Expected<AddStreamFn> {
+  return [=](unsigned Task, StringRef Key,
+             const Twine &ModuleName) -> Expected<AddStreamFn> {
     // This choice of file name allows the cache to be pruned (see pruneCache()
     // in include/llvm/Support/CachePruning.h).
     SmallString<64> EntryPath;
     sys::path::append(EntryPath, CacheDirectoryPath, "llvmcache-" + Key);
     // First, see if we have a cache hit.
     SmallString<64> ResultPath;
     Expected<sys::fs::file_t> FDOrErr = sys::fs::openNativeFileForRead(
         Twine(EntryPath), sys::fs::OF_UpdateAtime, &ResultPath);
     std::error_code EC;
     if (FDOrErr) {
       ErrorOr<std::unique_ptr<MemoryBuffer>> MBOrErr =
           MemoryBuffer::getOpenFile(*FDOrErr, EntryPath,
                                     /*FileSize=*/-1,
                                     /*RequiresNullTerminator=*/false);
       sys::fs::closeFile(*FDOrErr);
       if (MBOrErr) {
-        AddBuffer(Task, std::move(*MBOrErr));
+        AddBuffer(Task, ModuleName, std::move(*MBOrErr));
         return AddStreamFn();
       }
       EC = MBOrErr.getError();
     } else {
       EC = errorToErrorCode(FDOrErr.takeError());
     }
 
     // On Windows we can fail to open a cache file with a permission denied
     // error. This generally means that another process has requested to delete
     // the file while it is still open, but it could also mean that another
     // process has opened the file without the sharing permissions we need.
     // Since the file is probably being deleted we handle it in the same way as
     // if the file did not exist at all.
     if (EC != errc::no_such_file_or_directory && EC != errc::permission_denied)
       return createStringError(EC, Twine("Failed to open cache file ") +
                                        EntryPath + ": " + EC.message() + "\n");
 
     // This file stream is responsible for commiting the resulting file to the
     // cache and calling AddBuffer to add it to the link.
     struct CacheStream : CachedFileStream {
       AddBufferFn AddBuffer;
       sys::fs::TempFile TempFile;
+      std::string ModuleName;
       unsigned Task;
 
       CacheStream(std::unique_ptr<raw_pwrite_stream> OS, AddBufferFn AddBuffer,
                   sys::fs::TempFile TempFile, std::string EntryPath,
-                  unsigned Task)
+                  std::string ModuleName, unsigned Task)
           : CachedFileStream(std::move(OS), std::move(EntryPath)),
             AddBuffer(std::move(AddBuffer)), TempFile(std::move(TempFile)),
-            Task(Task) {}
+            ModuleName(ModuleName), Task(Task) {}
 
       ~CacheStream() {
         // TODO: Manually commit rather than using non-trivial destructor,
         // allowing to replace report_fatal_errors with a return Error.
 
         // Make sure the stream is closed before committing it.
         OS.reset();
 
struct CacheStream : CachedFileStream {
           return Error::success();
         });
 
         if (E)
           report_fatal_error(Twine("Failed to rename temporary file ") +
                              TempFile.TmpName + " to " + ObjectPathName + ": " +
                              toString(std::move(E)) + "\n");
 
-        AddBuffer(Task, std::move(*MBOrErr));
+        AddBuffer(Task, ModuleName, std::move(*MBOrErr));
       }
     };
 
-    return [=](size_t Task) -> Expected<std::unique_ptr<CachedFileStream>> {
+    return [=](size_t Task, const Twine &ModuleName)
+               -> Expected<std::unique_ptr<CachedFileStream>> {
       // Create the cache directory if not already done. Doing this lazily
       // ensures the filesystem isn't mutated until the cache is.
       if (std::error_code EC = sys::fs::create_directories(
               CacheDirectoryPath, /*IgnoreExisting=*/true))
         return errorCodeToError(EC);
 
       // Write to a temporary to avoid race condition
       SmallString<64> TempFilenameModel;
       sys::path::append(TempFilenameModel, CacheDirectoryPath,
                         TempFilePrefix + "-%%%%%%.tmp.o");
       Expected<sys::fs::TempFile> Temp = sys::fs::TempFile::create(
           TempFilenameModel, sys::fs::owner_read | sys::fs::owner_write);
       if (!Temp)
         return createStringError(errc::io_error,
                                  toString(Temp.takeError()) + ": " + CacheName +
                                      ": Can't get a temporary file");
 
       // This CacheStream will move the temporary file into the cache when done.
       return std::make_unique<CacheStream>(
           std::make_unique<raw_fd_ostream>(Temp->FD, /* ShouldClose */ false),
-          AddBuffer, std::move(*Temp), std::string(EntryPath.str()), Task);
+          AddBuffer, std::move(*Temp), std::string(EntryPath.str()),
+          ModuleName.str(), Task);
     };
   };
 }

llvm/tools/llvm-lto/llvm-lto.cpp

bool handleDiagnostics(const DiagnosticInfo &DI) override {
         break;
       case DS_Note:
         OS << "note";
         break;
       }
       if (!CurrentActivity.empty())
         OS << ' ' << CurrentActivity;
       OS << ": ";
-  
       DiagnosticPrinterRawOStream DP(OS);
       DI.print(DP);
       OS << '\n';
-  
       if (DI.getSeverity() == DS_Error)
         exit(1);
       return true;
     }
   };
   }
 
 static void error(const Twine &Msg) {
if (SaveModuleFile) {
       std::string ModuleFilename = OutputFilename;
       ModuleFilename += ".merged.bc";
       std::string ErrMsg;
 
       if (!CodeGen.writeMergedModules(ModuleFilename))
         error("writing merged module failed.");
     }
 
-    auto AddStream = [&](size_t Task) -> std::unique_ptr<CachedFileStream> {
+    auto AddStream =
+        [&](size_t Task,
+            const Twine &ModuleName) -> std::unique_ptr<CachedFileStream> {
       std::string PartFilename = OutputFilename;
       if (Parallelism != 1)
         PartFilename += "." + utostr(Task);
 
       std::error_code EC;
       auto S =
           std::make_unique<raw_fd_ostream>(PartFilename, EC, sys::fs::OF_None);
       if (EC)

llvm/tools/llvm-lto2/llvm-lto2.cpp

if (!CommandLineResolutions.empty()) {
     for (auto UnusedRes : CommandLineResolutions)
       llvm::errs() << argv[0] << ": unused symbol resolution for "
                    << UnusedRes.first.first << ',' << UnusedRes.first.second
                    << '\n';
   }
   if (HasErrors)
     return 1;
 
-  auto AddStream = [&](size_t Task) -> std::unique_ptr<CachedFileStream> {
+  auto AddStream =
+      [&](size_t Task,
+          const Twine &ModuleName) -> std::unique_ptr<CachedFileStream> {
     std::string Path = OutputFilename + "." + utostr(Task);
 
     std::error_code EC;
     auto S = std::make_unique<raw_fd_ostream>(Path, EC, sys::fs::OF_None);
     check(EC, Path);
     return std::make_unique<CachedFileStream>(std::move(S), Path);
   };
 
-  auto AddBuffer = [&](size_t Task, std::unique_ptr<MemoryBuffer> MB) {
-    *AddStream(Task)->OS << MB->getBuffer();
+  auto AddBuffer = [&](size_t Task, const Twine &ModuleName,
+                       std::unique_ptr<MemoryBuffer> MB) {
+    *AddStream(Task, ModuleName)->OS << MB->getBuffer();
   };
 
   FileCache Cache;
   if (!CacheDir.empty())
     Cache = check(localCache("ThinLTO", "Thin", CacheDir, AddBuffer),
                   "failed to create cache");
 
   check(Lto.run(AddStream, Cache), "LTO::run failed");