Add `--dependency-files` option, which is equivalent to compiler option -MD.

[MaskRay]

Summary

Clang and GCC have a feature (-MD flag) to create a dependency file
in a format that make command can read, so that you don't have to
manually maintain dependencies between .c files and .h files.

There was no corresponding feature in the linker. With this patch, you
can now do the same thing by passing --dependency-files=<path> to lld.
I believe this option is worth adding.

This is an example of a generated file:
https://gist.github.com/rui314/f5e2c669c70bec31f260bfd4b8ef41a6

Fixes https://bugs.llvm.org/show_bug.cgi?id=42806.

[pcc]

lld/ELF/Driver.cpp

1408

Clang has a bunch of complicated logic to handle escaping here: http://llvm-cs.pcc.me.uk/tools/clang/lib/Frontend/DependencyFile.cpp#299

Would it be worth extracting it into a function and using it from here? Or maybe there's something simpler that we can do that would be understood by ninja et al.?

[MaskRay]

lld/ELF/Config.h

89

Move dependencyFiles below chroot?

lld/ELF/Driver.cpp

1393

statically-linked -> ``

a.o/b.a/c.so are not statically linked..

[ruiu]

lld/ELF/Driver.cpp

1393

Yeah, and in that case, c.so is not part of your program, and it is not wrong to say that your executable does not really depend on c.so at link-time as long as c.so maintains a stable API.

[ruiu]

• quote a filename if it contains a space character
• add a test

[ruiu]

lld/ELF/Driver.cpp

1408

I read through the code and honestly my feeling is that we probably wouldn't have to make our code that bullet-proof. That logic is complicated because it tries to handle pathnames containing # or a special three-letter sequence \ #. If someone actually tried to use a filename that starts with a space followed by '#', that person is intentionally trying to break the build system.

So I made a change to quote a string if it contains a space character, which I believe should be enough.

[grimar]

lld/test/ELF/write-dependencies.s

4 ↗

(On Diff #212277)

This probably a bit simpler:

llvm-mc -filetype=obj -triple=x86_64-unknown-linux /dev/null -o "%t/bar baz.o"

6 ↗

(On Diff #212277)

Since you're testing the output in a particular format, what about adding --strict-whitespace --match-full-lines to FileCheck invocation?

[MaskRay]

If you need a makefile snippet: { echo -n 'a: '; clang a.o -o a -Wl,-t | sed 's/(.*//' | sort -u | sed '$!s/$/ \\/';} > a.d (assuming there is no special character)...

If there is some special needs, maybe we can improve -t to add more customization?

lld/ELF/Driver.cpp

1393

I think the phrase "statically-linked system libraries" is not accurate. Neither a system .a nor a system .so is statically linked.

[ruiu]

lld/ELF/Driver.cpp

1393

I feel it is colloquially OK, but precisely speaking, you are right. How about this? "even though system libraries that are linked to your executable statically are a part of your program."

[MaskRay]

lld/ELF/Driver.cpp

1393

"even though system libraries that are linked to your executable statically are a part of your program."

This looks good.

The code looks good but my main question is if we should add a new option, rather than improve an existing option (-t/--trace), which has a fairly similar feature. What --write-dependencies does can be easily implemented with some sed commands. With a small modification, it can be adjusted to Ninja output (and probably dependency format used by other build systems).

[peter.smith]

No objections to the idea in principle. I think it may need a bit of wider discussion to reach its full potential though. My understanding is that many developers use makefile/ninja generation systems such as cmake rather than hand-write the file themselves. As such would this get much use unless it was integrated into these generators? May be worth approaching them to see if they have any requirements/observations about the option?

I also think PCC has a good point about special characters and quoting, I'm no expert myself, and have been bitten by edge cases in the past, albeit these were usually routines that had to cope with Unix and Windows paths.

[emaste]

In D65430#1605980, @peter.smith wrote:

My understanding is that many developers use makefile/ninja generation systems such as cmake rather than hand-write the file themselves. As such would this get much use unless it was integrated into these generators? May be worth approaching them to see if they have any requirements/observations about the option?

Having it integrated into e.g. cmake would be a great project for someone to take on, but for us (FreeBSD) it would be useful by itself. Our build is about 175K lines of hand-written Makefiles and we could plug it in with a small change in a couple of places. In any case we shouldn't hold up lld support waiting on prospective cmake changes IMO.

[ruiu]

In D65430#1605980, @peter.smith wrote:

No objections to the idea in principle. I think it may need a bit of wider discussion to reach its full potential though. My understanding is that many developers use makefile/ninja generation systems such as cmake rather than hand-write the file themselves. As such would this get much use unless it was integrated into these generators? May be worth approaching them to see if they have any requirements/observations about the option?

These generators like cmake don't know the complete picture of the dependency list -- for example, if you pass -fsanitize to the compiler, it links libasan, and if you update libasan, you may want to rebuild your executable again, but that dependency cannot be tracked by cmake.

You are right that this probably needs wider discussion. I'll start a thread at llvm-dev.

[phosek]

lld/ELF/Options.td

419

Have you considered naming the option -dependency-file? That's the name of the CC1 flag (see https://github.com/llvm/llvm-project/blob/master/clang/include/clang/Driver/Options.td#L606) and I think it'd be nice to make those consistent across different tools.

[ruiu]

lld/ELF/Options.td

419

No, but that's a good suggestion. --write-dependencies, which is an alias for -MD, is also taken from clang, but in clang, that option doesn't take an argument. Clang's -dependency-file takes an argument. So, for consistency, it seems like -dependency-file is a better name.

[ruiu]

lld/test/ELF/write-dependencies.s

6 ↗

(On Diff #212277)

Since whitespace characters are not significant, I don't think we need to test that, but adding --match-full-lines is a good idea.

[ruiu]

• address review comments

[MaskRay]

In D65430#1606209, @emaste wrote:

In D65430#1605980, @peter.smith wrote:

My understanding is that many developers use makefile/ninja generation systems such as cmake rather than hand-write the file themselves. As such would this get much use unless it was integrated into these generators? May be worth approaching them to see if they have any requirements/observations about the option?

Having it integrated into e.g. cmake would be a great project for someone to take on, but for us (FreeBSD) it would be useful by itself. Our build is about 175K lines of hand-written Makefiles and we could plug it in with a small change in a couple of places. In any case we shouldn't hold up lld support waiting on prospective cmake changes IMO.

Ed, does ld.lld a.o -o /dev/null | sed 's/(.*//' | sort -u | sed '$!s/$/ \\/';} generate a usable Makefile fragment? I think the scenarios where people write Makefile are not very common now (think cmake/bazel/meson/scons/buck/...). Makefile and build.ninja are mostly used as an "assembly language". I think a dependency graph feature will be useful, but a Makefile fragment is probably not the best format. (I actually like Makefile more than the alternatives but I know the trend is that people are moving away from hand-written Makefile) At least it looks like the output can be easily generated from ld.lld -t output.

[ruiu]

In D65430#1614557, @MaskRay wrote:

In D65430#1606209, @emaste wrote:

In D65430#1605980, @peter.smith wrote:

My understanding is that many developers use makefile/ninja generation systems such as cmake rather than hand-write the file themselves. As such would this get much use unless it was integrated into these generators? May be worth approaching them to see if they have any requirements/observations about the option?

Having it integrated into e.g. cmake would be a great project for someone to take on, but for us (FreeBSD) it would be useful by itself. Our build is about 175K lines of hand-written Makefiles and we could plug it in with a small change in a couple of places. In any case we shouldn't hold up lld support waiting on prospective cmake changes IMO.

Ed, does ld.lld a.o -o /dev/null | sed 's/(.*//' | sort -u | sed '$!s/$/ \\/';} generate a usable Makefile fragment? I think the scenarios where people write Makefile are not very common now (think cmake/bazel/meson/scons/buck/...). Makefile and build.ninja are mostly used as an "assembly language". I think a dependency graph feature will be useful, but a Makefile fragment is probably not the best format. (I actually like Makefile more than the alternatives but I know the trend is that people are moving away from hand-written Makefile) At least it looks like the output can be easily generated from ld.lld -t output.

I mentioned Makefile because -MD was created with that usage in mind (at least originally), but I guess that file format can be understood by other build tools. Am I missing something? Even though some build tools can understand C/C++ file dependencies without using a compiler, there's still some languages that the build systems cannot read, but still they can read auto-generated Makefile-compatible dependency file, no?

[MaskRay]

In D65430#1614683, @ruiu wrote:

In D65430#1614557, @MaskRay wrote:

In D65430#1606209, @emaste wrote:

In D65430#1605980, @peter.smith wrote:

My understanding is that many developers use makefile/ninja generation systems such as cmake rather than hand-write the file themselves. As such would this get much use unless it was integrated into these generators? May be worth approaching them to see if they have any requirements/observations about the option?

Having it integrated into e.g. cmake would be a great project for someone to take on, but for us (FreeBSD) it would be useful by itself. Our build is about 175K lines of hand-written Makefiles and we could plug it in with a small change in a couple of places. In any case we shouldn't hold up lld support waiting on prospective cmake changes IMO.

Ed, does ld.lld a.o -o /dev/null | sed 's/(.*//' | sort -u | sed '$!s/$/ \\/';} generate a usable Makefile fragment? I think the scenarios where people write Makefile are not very common now (think cmake/bazel/meson/scons/buck/...). Makefile and build.ninja are mostly used as an "assembly language". I think a dependency graph feature will be useful, but a Makefile fragment is probably not the best format. (I actually like Makefile more than the alternatives but I know the trend is that people are moving away from hand-written Makefile) At least it looks like the output can be easily generated from ld.lld -t output.

I mentioned Makefile because -MD was created with that usage in mind (at least originally), but I guess that file format can be understood by other build tools. Am I missing something? Even though some build tools can understand C/C++ file dependencies without using a compiler, there's still some languages that the build systems cannot read, but still they can read auto-generated Makefile-compatible dependency file, no?

Different build systems use different syntax :/

Some are represented as a function call, e.g. executable('main', 'main.c'). Some are cc_binary(name="main", srcs=["main.c"]). llvm has LLVMBuild.txt. A Makefile fragment cannot be consumed by any of the build systems ;-)

In a hand-written Makefile, a.d is usually used in the following pattern:

a.o a.d: a.c
  gcc -MD -c $<

-include a.d   # suppress the error if a.d does not exist

a.c has many direct and indirect includes, e.g. stdio.h, stddef.h, a.h.
When any of the dependency is updated, a.o will be considered stale and will get rebuilt next time make is executed.

-MD is useful in that it gives the information that isn't available before: we can't tell what files a.c includes.

In the ld.lld --write-dependencies= case, the dependencies (bitcode files, object files, shared objects, and archive members that get pulled out) are already known. If it isn't, we'll get a "undefined reference" error. So I don't know who will be the consumer of the dependency output.

[phosek]

In D65430#1614759, @MaskRay wrote:

In D65430#1614683, @ruiu wrote:

In D65430#1614557, @MaskRay wrote:

In D65430#1606209, @emaste wrote:

In D65430#1605980, @peter.smith wrote:

My understanding is that many developers use makefile/ninja generation systems such as cmake rather than hand-write the file themselves. As such would this get much use unless it was integrated into these generators? May be worth approaching them to see if they have any requirements/observations about the option?

Having it integrated into e.g. cmake would be a great project for someone to take on, but for us (FreeBSD) it would be useful by itself. Our build is about 175K lines of hand-written Makefiles and we could plug it in with a small change in a couple of places. In any case we shouldn't hold up lld support waiting on prospective cmake changes IMO.

Ed, does ld.lld a.o -o /dev/null | sed 's/(.*//' | sort -u | sed '$!s/$/ \\/';} generate a usable Makefile fragment? I think the scenarios where people write Makefile are not very common now (think cmake/bazel/meson/scons/buck/...). Makefile and build.ninja are mostly used as an "assembly language". I think a dependency graph feature will be useful, but a Makefile fragment is probably not the best format. (I actually like Makefile more than the alternatives but I know the trend is that people are moving away from hand-written Makefile) At least it looks like the output can be easily generated from ld.lld -t output.

I mentioned Makefile because -MD was created with that usage in mind (at least originally), but I guess that file format can be understood by other build tools. Am I missing something? Even though some build tools can understand C/C++ file dependencies without using a compiler, there's still some languages that the build systems cannot read, but still they can read auto-generated Makefile-compatible dependency file, no?

Different build systems use different syntax :/

Some are represented as a function call, e.g. executable('main', 'main.c'). Some are cc_binary(name="main", srcs=["main.c"]). llvm has LLVMBuild.txt. A Makefile fragment cannot be consumed by any of the build systems ;-)

In a hand-written Makefile, a.d is usually used in the following pattern:

a.o a.d: a.c
  gcc -MD -c $<

-include a.d   # suppress the error if a.d does not exist

a.c has many direct and indirect includes, e.g. stdio.h, stddef.h, a.h.
When any of the dependency is updated, a.o will be considered stale and will get rebuilt next time make is executed.

-MD is useful in that it gives the information that isn't available before: we can't tell what files a.c includes.

In the ld.lld --write-dependencies= case, the dependencies (bitcode files, object files, shared objects, and archive members that get pulled out) are already known. If it isn't, we'll get a "undefined reference" error. So I don't know who will be the consumer of the dependency output.

One use case for ld.lld --dependency-file= is auto-linking which is supported by lld for both ELF and COFF. In case of auto-linking, you don't know which additional dependencies are going to be included until you process all input files. Another case that's specific to ELF are linker scripts which allows specifying additional inputs using the INPUT statement e.g. libc++ installs the following linker script INPUT(libc++.so.1 -lunwind -lc++abi) as libc++.so.

Regarding the depfile format, the simplified Makefile format is the de-facto standard for depfiles and is emitted and supported by build systems like Make or Ninja (which is generated by number of other high-level build systems like CMake, Meson, GN, etc.). Using this format would make this functionality immediately usable.

[MaskRay]

In D65430#1615429, @phosek wrote:

One use case for ld.lld --dependency-file= is auto-linking which is supported by lld for both ELF and COFF. In case of auto-linking, you don't know which additional dependencies are going to be included until you process all input files. Another case that's specific to ELF are linker scripts which allows specifying additional inputs using the INPUT statement e.g. libc++ installs the following linker script INPUT(libc++.so.1 -lunwind -lc++abi) as libc++.so.

Regarding the depfile format, the simplified Makefile format is the de-facto standard for depfiles and is emitted and supported by build systems like Make or Ninja (which is generated by number of other high-level build systems like CMake, Meson, GN, etc.). Using this format would make this functionality immediately usable.

I carefully think about this. deplibs is probably the only reasonable use case. Let me give an example to ensure I understand it correctly:

touch d.h

cat > a.c <<e
#include "d.h"
#pragma comment(lib, "b.a")
#pragma comment(lib, "c.a")
extern int b, c;
int main() { return b+c; }
e

cat > b.c <<e
int b;
e

cat > c.c <<e
int c;
e

cc -c b.c c.c; ar r b.a b.o; ar r c.a b.o

build.ninja:

rule cc
  depfile = $out.d
  command = clang -MD -MT $out -MF $out.d -c $in -o $out

rule link
  depfile = $out.link.d
  command = clang $in -fuse-ld=lld -Wl,--dependency-file=$out.link.d -o $out

build a: link a.o
build a.o: cc a.c

Alternatively, Makefile:

CFLAGS = -MD -MP -MF $@.d  # note -MP is not required in build.ninja
CC = clang
LDFLAGS = -fuse-ld=lld -Wl,--dependency-file=$@.link.d
a_deps = a.o  # explicit deps (.deplibs are excluded)

# implicit rule of a cannot be used because it would link all files specified in a.link.d together
a: $(a_deps)
	$(LINK.c) $(a_deps) $(LDLIBS) -o $@

-include a.o.d
-include a.link.d

This dependency file approach has an assumption: a.o.d is not older than a.o, a.link.d is not older than a.

1. if a.o.d does not exist, according to the assumption, a.o does not exist as well. ninja or make generates a.o and a.o.d. a and a.link.d are similar.
2. if a.c or d.h is updated, a get generated due to the dependency described in a.o.d.
3. if d.h is deleted and #include "d.h" is deleted from a.c. Note there is no rule to generate d.h. ninja seems to ignore its nonexistence, while Makefile needs a force target. That is why in the Makefile version we use -MP.
4. if b.a is deleted and #pragma comment(lib, "b.a") is deleted from a.c. ninja ignores b.a and generates a. However, GNU make will error: make: *** No rule to make target 'b.a', needed by 'a'. Stop.

I think ld.lld --write-dependencies= can be plugged into a ninja based build system (including cmake -G Ninja) without change, but not into a Makefile based build system.
To use this feature in meson/bazel/buck/..., a post-processing tool is definitely needed.
However, if a post-processing tool is ever needed, ld.lld -t will be just as simple (or difficult) to use.

Let's revisit this ninja fragment:

rule link
  depfile = $out.link.d
  command = clang $in -fuse-ld=lld -Wl,--dependency-file=$out.link.d -o $out

#pragma comment(lib, "b.a") is not common. If the build system goes through the trouble to use deplibs,
there may likely be other post processing steps. I wonder if it can just point -fuse-ld= to a custom wrapper that processes ld.lld -t output and generates $out.link.d.

So the question is whether we consider this feature too application-specific.
For simple dependency information, we can already use -t.
For richer information, there is a proposal for dependency analysis (https://lists.llvm.org/pipermail/llvm-dev/2019-February/130565.html)

I'd also like to hear from @bd1976llvm, who implemented the deplibs feature in the first place.

lld/ELF/Config.h

87

This probably should be

llvm::SetVector<std::string, std::vector<std::string>,
                llvm::StringSet<llvm::MallocAllocator>>
    dependencyFiles; // for --dependency-file

Some StringRef elements are not internalized.

lld/ELF/Driver.cpp

1377

--dependency-file

[ruiu]

As to post-process -verbose output, I think that that is too fragile to depend on, as -verbose is mainly for inspection and debugging. Any arbitrary output may be added to the -verbose output that can confuse post-process scripts. I also don't want to recommend users post-process -verbose output because of the same reason -- if people rely on the current -verbose output too much, any change to the file may break scripts in the wild which could prevent us from adding new messages there.

[MaskRay]

In D65430#1616713, @ruiu wrote:

As to post-process -verbose output, I think that that is too fragile to depend on, as -verbose is mainly for inspection and debugging. Any arbitrary output may be added to the -verbose output that can confuse post-process scripts. I also don't want to recommend users post-process -verbose output because of the same reason -- if people rely on the current -verbose output too much, any change to the file may break scripts in the wild which could prevent us from adding new messages there.

It is -t | --trace in ELF, not --verbose. (In COFF there is no --trace counterpart so I asked whether we should add a /trace on the mailing list.) The output format of --trace is stable.

[phosek]

Has there been any progress on this? This need for this came out again in our build today: we currently generate input linker scripts that pull in the right library at link time (e.g. this would like INPUT(/path/to/real/library.so)). Our distributed build needs to know about all the files that are needed as input to send them over to the server, and these input linker scripts are currently posing a problem. We were hoping to use the depfile support to find all such files; the system already knows how to parse depfiles to handle header dependencies so there isn't any additional support needed there. We would really like to see this landed as it'd unblock our progress or having to come up with some custom solution like manually parsing the input linker scripts.

[phosek]

In D65430#1616208, @MaskRay wrote:

#pragma comment(lib, "b.a") is not common. If the build system goes through the trouble to use deplibs,

These are already being used by libc++ and libc++abi so I wouldn't say it's uncommon.

there may likely be other post processing steps. I wonder if it can just point -fuse-ld= to a custom wrapper that processes ld.lld -t output and generates $out.link.d.

This sounds like a lot of extra effort that would have to be replicated in every build system that would like to use the depfile support (i.e. every build that wants to use linker depfiles would need such wrapper and integrate it into their build).

The implementation of this feature is very minimal and doesn't significantly increase the linker complexity so I don't understand why there's so much pushback?

[MaskRay]

In D65430#1739325, @phosek wrote:

In D65430#1616208, @MaskRay wrote:

#pragma comment(lib, "b.a") is not common. If the build system goes through the trouble to use deplibs,

These are already being used by libc++ and libc++abi so I wouldn't say it's uncommon.

there may likely be other post processing steps. I wonder if it can just point -fuse-ld= to a custom wrapper that processes ld.lld -t output and generates $out.link.d.

This sounds like a lot of extra effort that would have to be replicated in every build system that would like to use the depfile support (i.e. every build that wants to use linker depfiles would need such wrapper and integrate it into their build).

See my analysis at https://reviews.llvm.org/D65430#1616208 , I don't think the in-tree C++ implementation reduces complexity for other build systems. It works for Ninja, but not even for Make.

The implementation of this feature is very minimal and doesn't significantly increase the linker complexity so I don't understand why there's so much pushback?

We've been innovative in introducing useful options. That is a good thing. But we've already gotten similar features in this area (e.g. --trace, --verbose). The pushback is about the generality of a feature. The question raised in https://reviews.llvm.org/D65430#1616208 is never answered. I hope we can be aware of feature creep. Ideally we just implement powerful and orthogonal features that can meet a wide range of demand, instead of inventing a new option for every feature request that only differs from the previous ones in some nuance.

[phosek]

In D65430#1739394, @MaskRay wrote:

In D65430#1739325, @phosek wrote:

In D65430#1616208, @MaskRay wrote:

#pragma comment(lib, "b.a") is not common. If the build system goes through the trouble to use deplibs,

These are already being used by libc++ and libc++abi so I wouldn't say it's uncommon.

there may likely be other post processing steps. I wonder if it can just point -fuse-ld= to a custom wrapper that processes ld.lld -t output and generates $out.link.d.

This sounds like a lot of extra effort that would have to be replicated in every build system that would like to use the depfile support (i.e. every build that wants to use linker depfiles would need such wrapper and integrate it into their build).

See my analysis at https://reviews.llvm.org/D65430#1616208 , I don't think the in-tree C++ implementation reduces complexity for other build systems. It works for Ninja, but not even for Make.

The implementation of this feature is very minimal and doesn't significantly increase the linker complexity so I don't understand why there's so much pushback?

We've been innovative in introducing useful options. That is a good thing. But we've already gotten similar features in this area (e.g. --trace, --verbose). The pushback is about the generality of a feature. The question raised in https://reviews.llvm.org/D65430#1616208 is never answered. I hope we can be aware of feature creep. Ideally we just implement powerful and orthogonal features that can meet a wide range of demand, instead of inventing a new option for every feature request that only differs from the previous ones in some nuance.

That's fair, but I don't feel like we've been upholding that principle in the past and I don't understand why we should start right now. For example, we've introduced a number of features that are specific to Android: -use-android-relr-tags, or Partitions. We've also added IDE specific features: -vs-diagnostics. While it's true that depfiles could only be utilized by Ninja at the moment, Ninja is being used as a backend for multiple build systems: CMake, GN, Meson, Blueprint, kati, to name the most popular ones. This already seems like a much higher bar than for the other features.

To try and answer your questions from https://reviews.llvm.org/D65430#1616208, there are two different use cases I mentioned before, deplibs is one of them, input linker scripts is the other one. Regarding the issue with Make, could we address that using the same approach as -MP does by generating a phony target for each dependency other than the main file? For your example we would generate the following depfile:

a: b.a
b.a:

I do agree that Makefile format is not great, but it's what existing compilers and build systems already use and support so it seems like the most pragmatic option for now. I know there's a discussion about introducing a new (possibly JSON based) depfile format for C++ modules, and if that indeed happens and build system implement support for this format, we could extend lld to support the new format as well, but I don't think we should be blocking this feature on that.

[bd1976llvm]

Hi All, Apologies that I missed the request for input here.

Great summary @MaskRay. I like the examples. Worth being careful as dependency graph problems are tricky (I find).

Firstly, the problem that you have raised here is already addressed for header files with -MP, no? Therefore, I think that in lld we simply need to implement the following set: -M, -MM, -MF, -MG, -MP, -MQ, -MD, -MMD. We can presumably reuse the clang code for this so it won't be too bad.

Secondly, you have a point about these being make specific options; however, that is the well established standard. I would support implementing make specific output now and then implementing whatever more generic format build systems standardize on later. Just my 2 cents.

Incidentally, autolinking was modeled on the Microsoft feature where this problem does not exist - thanks to the infamous tracker.exe.

Finally, I don't think that this is as useful a feature (as header file dep generation) because, unlike header files which are not build outputs (usually), libraries are often build outputs. So, although implementing the -M options in LLD would handle incremental builds, you still need to add explicit edges to your makefiles to handle the initial build. If you are having to modify your makefiles then "comment lib" pragmas seem less useful. As I understand it autolinking is intended for linking libraries that are not built by your makefiles e.g. system libraries, or libraries supplied by thirdparty vendors.

[phosek]

The requirement for this feature came up again recently in our build so I'd like to revive this discussion, is there any way we could get this change landed?

In D65430#1755644, @bd1976llvm wrote:

Hi All, Apologies that I missed the request for input here.

Great summary @MaskRay. I like the examples. Worth being careful as dependency graph problems are tricky (I find).

Firstly, the problem that you have raised here is already addressed for header files with -MP, no? Therefore, I think that in lld we simply need to implement the following set: -M, -MM, -MF, -MG, -MP, -MQ, -MD, -MMD. We can presumably reuse the clang code for this so it won't be too bad.

Secondly, you have a point about these being make specific options; however, that is the well established standard. I would support implementing make specific output now and then implementing whatever more generic format build systems standardize on later. Just my 2 cents.

Incidentally, autolinking was modeled on the Microsoft feature where this problem does not exist - thanks to the infamous tracker.exe.

Finally, I don't think that this is as useful a feature (as header file dep generation) because, unlike header files which are not build outputs (usually), libraries are often build outputs. So, although implementing the -M options in LLD would handle incremental builds, you still need to add explicit edges to your makefiles to handle the initial build. If you are having to modify your makefiles then "comment lib" pragmas seem less useful. As I understand it autolinking is intended for linking libraries that are not built by your makefiles e.g. system libraries, or libraries supplied by thirdparty vendors.

This intended use case for this feature is for libraries that are not part of the build, i.e. system libraries and libraries supplied by vendors. We would like to relink the target if the library changes, but today there's no way to do that. When you pass e.g. -lz or use #pragma comment(lib, "z"), linker might resolve this to /path/to/libz.a. Whenever this library gets updated we would like to relink any target that depends on this library. This is a very similar use case to header files where depfiles are typically used to track dependency on system headers.

[MaskRay]

I have mentioned this several times. You can just process -t output like { echo -n 'a: '; clang a.o -o a -Wl,-t | sed 's/(.*//' | sort -u | sed '$!s/$/ \\/';} > a.d

The problems are summarized previously and I don't see they are addressed. Given an existing simple way listing dependencies, I don't think we need more -M* like options.

[phosek]

In D65430#2089961, @MaskRay wrote:

I have mentioned this several times. You can just process -t output like { echo -n 'a: '; clang a.o -o a -Wl,-t | sed 's/(.*//' | sort -u | sed '$!s/$/ \\/';} > a.d

Your workaround isn't a replacement for --dependency-file because -t has a different semantics. Proper dependency file tracking has to include files opened and not included in the link after examination, e.g. archives whose contents might influence the link if changed but didn't influence the link this time around.

Also this workaround is deeply unappealing from the perspective of integrating into the build. It's not portable and it assumes you can just invoke arbitrary shell script from your build system which is generally not the case, and certainly not the case for us. It also requires invoking 4 external commands which adds additional overhead. You could address it by creating a wrapper around lld (I'm ignoring the -t semantics issue here), but that's almost certainly going to be more complicated and more expensive than this implementation which is 20 lines of code and is unlikely to require any future maintenance.

The problems are summarized previously and I don't see they are addressed. Given an existing simple way listing dependencies, I don't think we need more -M* like options.

That's easy to address, we can modify the semantics of --dependency-file to behave like -MP by default which makes this feature usable from both Ninja and Make. The proposed BFD ld and gold implementation already behaves that way, I'm happy to update this change to match that.

[MaskRay]

Closed by D82437

lld/ELF/Config.h

 //===- Config.h -------------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_ELF_CONFIG_H
 #define LLD_ELF_CONFIG_H
 
 #include "lld/Common/ErrorHandler.h"
 #include "llvm/ADT/MapVector.h"
+#include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSet.h"
 #include "llvm/BinaryFormat/ELF.h"
 #include "llvm/Support/CachePruning.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Endian.h"
 #include <atomic>
 #include <vector>

 // This struct contains the global configuration for the linker.
 // Most fields are direct mapping from the command line options
 // and such fields have the same name as the corresponding options.
 // Most fields are initialized by the driver.
 struct Configuration {
   uint8_t osabi = 0;
   uint32_t andFeatures = 0;
   llvm::CachePruningPolicy thinLTOCachePolicy;
+  llvm::SetVector<StringRef> dependencyFiles; // for --dependency-file
   llvm::StringMap<uint64_t> sectionStartMap;
   llvm::StringRef chroot;
-  llvm::StringRef dynamicLinker;
+  llvm::StringRef dependencyFile;
   llvm::StringRef dwoDir;
+  llvm::StringRef dynamicLinker;
   llvm::StringRef entry;
   llvm::StringRef emulation;
   llvm::StringRef fini;
   llvm::StringRef init;
   llvm::StringRef ltoAAPipeline;
   llvm::StringRef ltoCSProfileFile;
   llvm::StringRef ltoNewPmPasses;
   llvm::StringRef ltoObjPath;

lld/ELF/Driver.cpp

static void readConfigs(opt::InputArgList &args) {
   config->checkSections =
       args.hasFlag(OPT_check_sections, OPT_no_check_sections, true);
   config->chroot = args.getLastArgValue(OPT_chroot);
   config->compressDebugSections = getCompressDebugSections(args);
   config->cref = args.hasFlag(OPT_cref, OPT_no_cref, false);
   config->defineCommon = args.hasFlag(OPT_define_common, OPT_no_define_common,
                                       !args.hasArg(OPT_relocatable));
   config->demangle = args.hasFlag(OPT_demangle, OPT_no_demangle, true);
+  config->dependencyFile = args.getLastArgValue(OPT_dependency_file);
   config->dependentLibraries = args.hasFlag(OPT_dependent_libraries, OPT_no_dependent_libraries, true);
   config->disableVerify = args.hasArg(OPT_disable_verify);
   config->discard = getDiscard(args);
   config->dwoDir = args.getLastArgValue(OPT_plugin_opt_dwo_dir_eq);
   config->dynamicLinker = getDynamicLinker(args);
   config->ehFrameHdr =
       args.hasFlag(OPT_eh_frame_hdr, OPT_no_eh_frame_hdr, false);
   config->emitLLVM = args.hasArg(OPT_plugin_opt_emit_llvm, false);
if (auto *lo = dyn_cast<LazyObject>(sym))
     mb = lo->file->mb;
   else
     mb = cast<LazyArchive>(sym)->getMemberBuffer();
 
   if (isBitcode(mb))
     sym->fetch();
 }
 
+// Handle --write-dependencies=<path>. If that option is given, lld creates a
+// file at a given path with the following contents:
+//
+//   <output-file>: <input-file> ...
+//
+// where <output-file> is a pathname of an output file and <input-file>
+// ... is a list of pathnames of all input files. `make` command can read a
+// file in the above format and interpret it as a dependency info.
+//
+// This option is useful if you want to make your final executable to depend
+// on all input files including system libraries. Here is why.
+//
+// When you write a Makefile, you usually write it so that the final
+// executable depends on all user-generated object files. Normally, you
+// don't make your executable to depend on system libraries (such as libc)
+// because you don't know the exact paths of libraries, even though system
+// libraries that are linked to your executable statically are technically a
+// part of your program. By using --write-dependencies option, you can make
+// lld to dump dependency info so that you can maintain exact dependencies
+// easily.
+static void writeDependencyFile() {
+  std::error_code ec;
+  raw_fd_ostream os(config->dependencyFile, ec, sys::fs::F_None);
+  if (ec) {
+    error("cannot open " + config->dependencyFile + ": " + ec.message());
+    return;
+  }
+
+  os << config->outputFile << ":";
+  for (StringRef path : config->dependencyFiles) {
+    SmallString<128> s = path;
+    sys::path::remove_dots(s, /*remove_dot_dot=*/true);
+    if (s.str().contains(' '))
+      os << " \\\n  \"" << s << '"';
+    else
+      os << " \\\n  " << s;
+  }
+  os << "\n";
+}
+
 // Replaces common symbols with defined symbols reside in .bss sections.
 // This function is called after all symbol names are resolved. As a
 // result, the passes after the symbol resolution won't see any
 // symbols of type CommonSymbol.
 static void replaceCommonSymbols() {
   symtab->forEachSymbol([](Symbol *sym) {
     auto *s = dyn_cast<CommonSymbol>(sym);
     if (!s)
llvm::erase_if(inputSections, [](InputSectionBase *s) {
     }
 
     // We do not want to emit debug sections if --strip-all
     // or -strip-debug are given.
     return config->strip != StripPolicy::None &&
            (s->name.startswith(".debug") || s->name.startswith(".zdebug"));
   });
 
+  // Since we now have a complete set of input files, we can create
+  // a .d file to record build dependencies.
+  if (!config->dependencyFile.empty())
+    writeDependencyFile();
+
   // Now that the number of partitions is fixed, save a pointer to the main
   // partition.
   mainPart = &partitions[0];
 
   // Read .note.gnu.property sections from input object files which
   // contain a hint to tweak linker's and loader's behaviors.
   config->andFeatures = getAndFeatures<ELFT>();
 

lld/ELF/InputFiles.cpp

 
 Optional<MemoryBufferRef> elf::readFile(StringRef path) {
   // The --chroot option changes our virtual root directory.
   // This is useful when you are dealing with files created by --reproduce.
   if (!config->chroot.empty() && path.startswith("/"))
     path = saver.save(config->chroot + path);
 
   log(path);
+  config->dependencyFiles.insert(path);
 
   auto mbOrErr = MemoryBuffer::getFile(path, -1, false);
   if (auto ec = mbOrErr.getError()) {
     error("cannot open " + path + ": " + ec.message());
     return None;
   }
 
   std::unique_ptr<MemoryBuffer> &mb = *mbOrErr;

lld/ELF/Options.td

 defm define_common: B<"define-common",
     "Assign space to common symbols",
     "Do not assign space to common symbols">;
 
 defm demangle: B<"demangle",
     "Demangle symbol names (default)",
     "Do not demangle symbol names">;
 
+defm dependency_file: Eq<"dependency-file", "Write a dependency file">,
+  MetaVarName<"<path>">;
+
 def disable_new_dtags: F<"disable-new-dtags">,
   HelpText<"Disable new dynamic tags">;
 
 def discard_all: F<"discard-all">, HelpText<"Delete all local symbols">;
 
 def discard_locals: F<"discard-locals">,
   HelpText<"Delete temporary local symbols">;
 

 
 defm whole_archive: B<"whole-archive",
     "Force load of all members in a static library",
     "Do not force load of all members in a static library (default)">;
 
 defm wrap: Eq<"wrap", "Use wrapper functions for symbol">,
   MetaVarName<"<symbol>=<symbol>">;
 
 def z: JoinedOrSeparate<["-"], "z">, MetaVarName<"<option>">,
   HelpText<"Linker option extensions">;
 
 def visual_studio_diagnostics_format : F<"vs-diagnostics">,
 HelpText<"Format diagnostics for Visual Studio compatiblity">;
 
 // Aliases
 def: Separate<["-"], "f">, Alias<auxiliary>, HelpText<"Alias for --auxiliary">;
 def: F<"call_shared">, Alias<Bdynamic>, HelpText<"Alias for --Bdynamic">;

lld/test/ELF/dependency-file.s

+// REQUIRES: x86
+// RUN: mkdir -p %t
+// RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux %s -o %t/foo.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux /dev/null -o "%t/bar baz.o"
+// RUN: ld.lld -o %t/foo.exe %t/foo.o %t/"bar baz.o" --dependency-file=%t/foo.d
+// RUN: FileCheck --match-full-lines %s < %t/foo.d
+
+// CHECK:      {{.*}}foo.exe: \
+// CHECK-NEXT:   {{.*}}foo.o \
+// CHECK-NEXT:   "{{.*}}bar baz.o"
+
+.global _start
+_start: