This is an archive of the discontinued LLVM Phabricator instance.

[LTO][COFF] Use bitcode file names in lto native object file names.
ClosedPublic

Authored by zequanwu on Nov 1 2022, 5:14 PM.

Details

Summary

Currently the lto native object files have names like main.exe.lto.1.obj. In
PDB, those names are used as names for each compiland. Microsoft’s tool
SizeBench uses those names to present to users the size of each object files.
So, names like main.exe.lto.1.obj is not user friendly.

This patch makes the lto native object file names more readable by using
the bitcode file names as part of the file names. For example, if the input
bitcode file has path like "path/to/foo.obj", its corresponding lto native
object file path would be "path/to/main.exe.lto.foo.obj". Since the lto native
object file name only bothers PDB, this patch only changes the lld-linker's
behavior.

Diff Detail

Event Timeline

zequanwu created this revision.Nov 1 2022, 5:14 PM
Herald added projects: Restricted Project, Restricted Project. · View Herald Transcript
Herald added a reviewer: Restricted Project. · View Herald Transcript
Herald added subscribers: pmatos, asb, ormris and 6 others. · View Herald Transcript
zequanwu requested review of this revision.Nov 1 2022, 5:14 PM
Herald added projects: Restricted Project, Restricted Project, Restricted Project. · View Herald TranscriptNov 1 2022, 5:14 PM

The naming convention used here for COFF is actually very similar to what is done for other ThinLTO save-temps output files, which are placed at the input module locations. We may want to just do this across the board. @MaskRay wdyt? A few other questions/comments below.

clang/lib/CodeGen/BackendUtil.cpp
1104

I think you might need to mark the new parameter as unused here and in other cases where it isn't used via LLVM_ATTRIBUTE_UNUSED, to avoid build warnings - I can't recall how strictly that is enforced.

lld/COFF/LTO.cpp
229

This case should always be i==0 I think?

244

The above changes affect both the MemoryBufferRef name as well as the saveTemps output file name. I assume the change to the former is what is required for PDB, is that correct?

lld/MachO/LTO.cpp
132

In some files this parameter is called "filed" and in others "originFile". Better to be consistent, and perhaps to use a more descriptive name, something like moduleName ?

llvm/include/llvm/Support/Caching.h
55

This and possibly other header file descriptions need updates.

zequanwu updated this revision to Diff 475296.Nov 14 2022, 3:29 PM
zequanwu marked 2 inline comments as done.

Update.

clang/lib/CodeGen/BackendUtil.cpp
1104

LLVM_ATTRIBUTE_UNUSED is used to suppress unused functions. For unused parameter, I don't see any build warnings when compiling with this patch. I feel like it's very common to have unused parameter.

lld/COFF/LTO.cpp
229

IIUC, "ld-temp.o" is the name of combined module. Do you mean there will be only 1 combined module and it will always be the first task?

244

Yes, it changes both the MemoryBufferRef and the saveTemps output file name otherwise the saveTemps output file name won't match with the the names in pdb. The former is what's written into PDB.

@MaskRay wondering if this is a good change to make for ELF as well, wdyt?

clang/lib/CodeGen/BackendUtil.cpp
1104

LLVM_ATTRIBUTE_UNUSED maps to attribute((unused)) which works for either functions or parameters. However, I see where the macro is defined in llvm/include/llvm/Support/Compiler.h that using "(void)unused;" is preferred. Either one works (see below). However, it must be the case that there are no bots building with -Wunused-parameter, since I see Task is also unused here. So nevermind this suggestion, since what you have currently is consistent with what is already done here.

$ cat unused.cc
int foo(int x1, int x2 __attribute__((__unused__)), int x3) {
  (void)x3;
  return 1;
}
$ clang++ -c unused.cc -Wunused-parameter
unused.cc:1:13: warning: unused parameter 'x1' [-Wunused-parameter]
int foo(int x1, int x2 __attribute__((__unused__)), int x3) {
            ^
1 warning generated.
lld/COFF/LTO.cpp
229

Yes. So you don't need the handling for i==0 in the name in this case (you could probably assert that i==0).

llvm/include/llvm/Support/Caching.h
55

Can you add a note that ModuleName is the unique module identifier for the bitcode module the cache is being checked for.

zequanwu updated this revision to Diff 475538.Nov 15 2022, 11:26 AM
zequanwu marked 2 inline comments as done.

Update.

MaskRay added a comment.EditedNov 15 2022, 8:04 PM

@MaskRay wondering if this is a good change to make for ELF as well, wdyt?

Yes, I think this is a good idea and improves debuggability. The linker side change is non-trivial (see lld/COFF changes), so making this patch focus on the COFF part is good. I am happy to do the ELF side.

lld/COFF/LTO.cpp
229

This looks like a hack. assert(i == 0) will fail with -opt:lldltopartitions=2: buf[1] will be called ld-temp.o as well.

In addition, if an input bitcode file is called ld-temp.o (for some reason they don't use .obj, just ld-temp.o), assert(i == 0) will fail as well.

238

What if two input bitcode files have the same basename, e.g. dir1/a.obj and dir2/a.obj?

MaskRay added inline comments.Nov 15 2022, 8:06 PM
lld/COFF/LTO.cpp
229

I guess if (i < config->ltoPartitions) may fix the issue.

tejohnson added inline comments.Nov 16 2022, 7:37 AM
lld/COFF/LTO.cpp
229

Ah ok, forgot about the lto partitions case. Sorry, @zequanwu, looks like you will need to go back to your old handling that appends i if it is non-zero.

238

I think that should be ok as the output file path created here includes the directory. So you should get dir1/a.out.lto.a.obj and dir2/a.out.lto.a.obj.

zequanwu updated this revision to Diff 475895.Nov 16 2022, 12:39 PM
zequanwu marked an inline comment as done.

Address comments.

lld/COFF/LTO.cpp
229

I reverted this part back. Since input bitcode file can be called ld-temp.o, i could be any number smaller than maxTasks, removed assertion.

238

Yes, you will get dir1/a.out.lto.a.obj and dir2/a.out.lto.a.obj.

This revision is now accepted and ready to land.Nov 18 2022, 8:56 AM
MaskRay added inline comments.Nov 19 2022, 2:12 PM
clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp
846

Twine should only be used as const references as arguments.

MaskRay requested changes to this revision.Nov 19 2022, 2:13 PM
This revision now requires changes to proceed.Nov 19 2022, 2:13 PM
zequanwu updated this revision to Diff 477014.Nov 21 2022, 3:01 PM

Update Twine argument to const Twine&.

zequanwu marked an inline comment as done.Nov 21 2022, 3:01 PM
MaskRay accepted this revision as: MaskRay.Nov 21 2022, 6:39 PM
MaskRay accepted this revision.
This revision is now accepted and ready to land.Nov 21 2022, 6:40 PM
thakis added a subscriber: thakis.Nov 22 2022, 10:32 AM

This doesn't build:
http://45.33.8.238/win/70474/step_4.txt
http://45.33.8.238/linux/92146/step_4.txt

Please take a look and revert for now if it takes a while to fix.

This broke the gold plugin:

[task 2022-11-22T21:03:29.486Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1108:19: error: no matching function for call to 'localCache'
[task 2022-11-22T21:03:29.486Z]     Cache = check(localCache("ThinLTO", "Thin", options::cache_dir, AddBuffer));
[task 2022-11-22T21:03:29.487Z]                   ^~~~~~~~~~
[task 2022-11-22T21:03:29.487Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/Support/Caching.h:72:21: note: candidate function not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1102:20)' to 'llvm::AddBufferFn' (aka 'function<void (unsigned int, const llvm::Twine &, std::unique_ptr<MemoryBuffer>)>') for 4th argument
[task 2022-11-22T21:03:29.487Z] Expected<FileCache> localCache(
[task 2022-11-22T21:03:29.488Z]                     ^
[task 2022-11-22T21:03:29.488Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1110:18: error: no viable conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'llvm::AddStreamFn' (aka 'function<Expected<std::unique_ptr<CachedFileStream>> (unsigned int, const llvm::Twine &)>')
[task 2022-11-22T21:03:29.488Z]   check(Lto->run(AddStream, Cache));
[task 2022-11-22T21:03:29.488Z]                  ^~~~~~~~~
[task 2022-11-22T21:03:29.488Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:421:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::nullptr_t' (aka 'nullptr_t') for 1st argument
[task 2022-11-22T21:03:29.488Z]       function(nullptr_t) noexcept
[task 2022-11-22T21:03:29.488Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:432:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'const std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &' for 1st argument
[task 2022-11-22T21:03:29.489Z]       function(const function& __x);
[task 2022-11-22T21:03:29.489Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:441:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &&' for 1st argument
[task 2022-11-22T21:03:29.489Z]       function(function&& __x) noexcept : _Function_base()
[task 2022-11-22T21:03:29.489Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:465:2: note: candidate template ignored: substitution failure [with _Functor = (lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20), $1 = void]: no type named 'type' in 'std::result_of<(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &(unsigned int, const llvm::Twine &)>'
[task 2022-11-22T21:03:29.489Z]         function(_Functor);
[task 2022-11-22T21:03:29.489Z]         ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/LTO/LTO.h:278:25: note: passing argument to parameter 'AddStream' here
[task 2022-11-22T21:03:29.489Z]   Error run(AddStreamFn AddStream, FileCache Cache = nullptr);
[task 2022-11-22T21:03:29.489Z]                         ^
[task 2022-11-22T21:03:29.489Z] 2 errors generated.

This broke the gold plugin:

[task 2022-11-22T21:03:29.486Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1108:19: error: no matching function for call to 'localCache'
[task 2022-11-22T21:03:29.486Z]     Cache = check(localCache("ThinLTO", "Thin", options::cache_dir, AddBuffer));
[task 2022-11-22T21:03:29.487Z]                   ^~~~~~~~~~
[task 2022-11-22T21:03:29.487Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/Support/Caching.h:72:21: note: candidate function not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1102:20)' to 'llvm::AddBufferFn' (aka 'function<void (unsigned int, const llvm::Twine &, std::unique_ptr<MemoryBuffer>)>') for 4th argument
[task 2022-11-22T21:03:29.487Z] Expected<FileCache> localCache(
[task 2022-11-22T21:03:29.488Z]                     ^
[task 2022-11-22T21:03:29.488Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1110:18: error: no viable conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'llvm::AddStreamFn' (aka 'function<Expected<std::unique_ptr<CachedFileStream>> (unsigned int, const llvm::Twine &)>')
[task 2022-11-22T21:03:29.488Z]   check(Lto->run(AddStream, Cache));
[task 2022-11-22T21:03:29.488Z]                  ^~~~~~~~~
[task 2022-11-22T21:03:29.488Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:421:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::nullptr_t' (aka 'nullptr_t') for 1st argument
[task 2022-11-22T21:03:29.488Z]       function(nullptr_t) noexcept
[task 2022-11-22T21:03:29.488Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:432:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'const std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &' for 1st argument
[task 2022-11-22T21:03:29.489Z]       function(const function& __x);
[task 2022-11-22T21:03:29.489Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:441:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &&' for 1st argument
[task 2022-11-22T21:03:29.489Z]       function(function&& __x) noexcept : _Function_base()
[task 2022-11-22T21:03:29.489Z]       ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:465:2: note: candidate template ignored: substitution failure [with _Functor = (lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20), $1 = void]: no type named 'type' in 'std::result_of<(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &(unsigned int, const llvm::Twine &)>'
[task 2022-11-22T21:03:29.489Z]         function(_Functor);
[task 2022-11-22T21:03:29.489Z]         ^
[task 2022-11-22T21:03:29.489Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/LTO/LTO.h:278:25: note: passing argument to parameter 'AddStream' here
[task 2022-11-22T21:03:29.489Z]   Error run(AddStreamFn AddStream, FileCache Cache = nullptr);
[task 2022-11-22T21:03:29.489Z]                         ^
[task 2022-11-22T21:03:29.489Z] 2 errors generated.

Thanks for noticing. I have fixed at https://reviews.llvm.org/rG10a43c4641c20e0a50edc0ff99915c837a507cc1. Unfortunately, this patch is reverted right before that fix pushed. I relanded this at https://reviews.llvm.org/rG84be92d26fcb1ddad533c0c614a79a81c59f795d.

Still broken:

task 2022-11-22T22:09:00.912Z] /usr/lib/llvm-11/bin/clang++ --sysroot=/builds/worker/fetches/sysroot -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/gold -I/builds/worker/fetches/llvm-project/llvm/tools/gold -Iinclude -I/builds/worker/fetches/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -fPIC  -fno-exceptions -fno-rtti -std=c++17 -MD -MT tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o -MF tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o.d -o tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o -c /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1111:18: error: no viable conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'llvm::AddStreamFn' (aka 'function<Expected<std::unique_ptr<CachedFileStream>> (unsigned int, const llvm::Twine &)>')
[task 2022-11-22T22:09:00.912Z]   check(Lto->run(AddStream, Cache));
[task 2022-11-22T22:09:00.912Z]                  ^~~~~~~~~
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:421:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::nullptr_t' (aka 'nullptr_t') for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(nullptr_t) noexcept
[task 2022-11-22T22:09:00.912Z]       ^
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:432:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'const std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &' for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(const function& __x);
[task 2022-11-22T22:09:00.912Z]       ^
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:441:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &&' for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(function&& __x) noexcept : _Function_base()
[task 2022-11-22T22:09:00.913Z]       ^
[task 2022-11-22T22:09:00.913Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:465:2: note: candidate template ignored: substitution failure [with _Functor = (lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20), $1 = void]: no type named 'type' in 'std::result_of<(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &(unsigned int, const llvm::Twine &)>'
[task 2022-11-22T22:09:00.913Z]         function(_Functor);
[task 2022-11-22T22:09:00.913Z]         ^
[task 2022-11-22T22:09:00.913Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/LTO/LTO.h:278:25: note: passing argument to parameter 'AddStream' here
[task 2022-11-22T22:09:00.913Z]   Error run(AddStreamFn AddStream, FileCache Cache = nullptr);
[task 2022-11-22T22:09:00.913Z]                         ^
[task 2022-11-22T22:09:00.913Z] 1 error generated.

Still broken:

task 2022-11-22T22:09:00.912Z] /usr/lib/llvm-11/bin/clang++ --sysroot=/builds/worker/fetches/sysroot -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/gold -I/builds/worker/fetches/llvm-project/llvm/tools/gold -Iinclude -I/builds/worker/fetches/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -fPIC  -fno-exceptions -fno-rtti -std=c++17 -MD -MT tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o -MF tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o.d -o tools/gold/CMakeFiles/LLVMgold.dir/gold-plugin.cpp.o -c /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1111:18: error: no viable conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'llvm::AddStreamFn' (aka 'function<Expected<std::unique_ptr<CachedFileStream>> (unsigned int, const llvm::Twine &)>')
[task 2022-11-22T22:09:00.912Z]   check(Lto->run(AddStream, Cache));
[task 2022-11-22T22:09:00.912Z]                  ^~~~~~~~~
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:421:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::nullptr_t' (aka 'nullptr_t') for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(nullptr_t) noexcept
[task 2022-11-22T22:09:00.912Z]       ^
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:432:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'const std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &' for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(const function& __x);
[task 2022-11-22T22:09:00.912Z]       ^
[task 2022-11-22T22:09:00.912Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:441:7: note: candidate constructor not viable: no known conversion from '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, const llvm::Twine &)> &&' for 1st argument
[task 2022-11-22T22:09:00.912Z]       function(function&& __x) noexcept : _Function_base()
[task 2022-11-22T22:09:00.913Z]       ^
[task 2022-11-22T22:09:00.913Z] /builds/worker/fetches/sysroot/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:465:2: note: candidate template ignored: substitution failure [with _Functor = (lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20), $1 = void]: no type named 'type' in 'std::result_of<(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &(unsigned int, const llvm::Twine &)>'
[task 2022-11-22T22:09:00.913Z]         function(_Functor);
[task 2022-11-22T22:09:00.913Z]         ^
[task 2022-11-22T22:09:00.913Z] /builds/worker/fetches/llvm-project/llvm/include/llvm/LTO/LTO.h:278:25: note: passing argument to parameter 'AddStream' here
[task 2022-11-22T22:09:00.913Z]   Error run(AddStreamFn AddStream, FileCache Cache = nullptr);
[task 2022-11-22T22:09:00.913Z]                         ^
[task 2022-11-22T22:09:00.913Z] 1 error generated.

Oh, sorry, I missed another place. Hope this will fix it.

Almost there, but not quite:

[task 2022-11-22T23:55:36.341Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1106:6: error: no matching function for call to object of type '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1095:7)'
[task 2022-11-22T23:55:36.341Z]     *AddStream(Task)->OS << MB->getBuffer();
[task 2022-11-22T23:55:36.341Z]      ^~~~~~~~~
[task 2022-11-22T23:55:36.341Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1095:7: note: candidate function not viable: requires 2 arguments, but 1 was provided
[task 2022-11-22T23:55:36.341Z]       [&](size_t Task,
[task 2022-11-22T23:55:36.341Z]       ^
[task 2022-11-22T23:55:36.341Z] 1 error generated.

Almost there, but not quite:

[task 2022-11-22T23:55:36.341Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1106:6: error: no matching function for call to object of type '(lambda at /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1095:7)'
[task 2022-11-22T23:55:36.341Z]     *AddStream(Task)->OS << MB->getBuffer();
[task 2022-11-22T23:55:36.341Z]      ^~~~~~~~~
[task 2022-11-22T23:55:36.341Z] /builds/worker/fetches/llvm-project/llvm/tools/gold/gold-plugin.cpp:1095:7: note: candidate function not viable: requires 2 arguments, but 1 was provided
[task 2022-11-22T23:55:36.341Z]       [&](size_t Task,
[task 2022-11-22T23:55:36.341Z]       ^
[task 2022-11-22T23:55:36.341Z] 1 error generated.

ninja LLVMgold.so works for me now. I think @zequanwu fixed this in 10a43c4641c20e0a50edc0ff99915c837a507cc1/5d140dc2c0f3068fae7127b85df861c420848078/d23b63ceccfe526c3a9b17d35032b3666e6816a1

int3 added a subscriber: int3.Mar 23 2023, 3:51 PM

I'm trying to add similar support to lld-macho, hence the question :)

lld/COFF/LTO.cpp
181

Any reason why this doesn't instead store the module name in the file_names[task] vector that the cache callback uses? Are the task IDs not unique across each run of ltoObj, regardless of whether we end up using the cache or not?

zequanwu added inline comments.Mar 30 2023, 1:02 PM
lld/COFF/LTO.cpp
181

Users have to explicitly uses the flag /opt:lldltocache=path to let lld to search for cache files at the given folder. If it's not given, the callback of localCache won't be invoked.