This is an archive of the discontinued LLVM Phabricator instance.

Allow CompilerInvocations to generate .d files.
ClosedPublic

Authored by saugustine on Jun 16 2017, 4:41 PM.

Details

Summary

Most clang tools should ignore the -M
family of options because one wouldn't want them
to generate a new dependency (.d) file. However,
some tools may want this dependency file. This
patch creates a mechanism for them to do this.

This implementation just plumbs a boolean down
several layers of calls. Each of the modified calls
has several call sites, and so a single member
variable or new API entry point won't work.

An alternative would be to write a function to filter
the -M family of arguments out of CC1Args, and have
each caller call that function by hand before calling
newInvocation, Invocation::run, or buildAstFromCodeWithArgs.
This is a more complicated and error-prone solution.
Why burden all the callers to remember to use
this function?

But I could rewrite this patch to use that method if
that is deemed more appropriate.

Event Timeline

saugustine created this revision.Jun 16 2017, 4:41 PM
saugustine edited subscribers, added: cfe-commits; removed: klimek.
echristo edited reviewers, added: bkramer; removed: echristo.Jun 16 2017, 5:19 PM
echristo edited edge metadata.

Going to let Ben review this. I'd rather not pass the bool down and he might know a way to avoid that here by knowing the code more :)

I think a better way might be to generally leave dependency options alone, add a default argument adapter to filter out all deps related flags, and allow users to add their own argument adapters that don't do that.

I think a better way might be to generally leave dependency options alone, add a default argument adapter to filter out all deps related flags, and allow users to add their own argument adapters that don't do that.

This argument adapter would have to be passed down in a similar way, no?

buildASTFromCodeWithArgs, toolIinvocation::Run, and newInvocation are all entry-points that would need this behavior, and all are called by themselves in one place or another.

I'm happy to do that, as it is quite a bit more flexible, but it doesn't seem any cleaner.

I think a better way might be to generally leave dependency options alone, add a default argument adapter to filter out all deps related flags, and allow users to add their own argument adapters that don't do that.

This argument adapter would have to be passed down in a similar way, no?

buildASTFromCodeWithArgs, toolIinvocation::Run, and newInvocation are all entry-points that would need this behavior, and all are called by themselves in one place or another.

I think it's cleaner, because it uses a concept we already have (argument adapters).

I'm happy to do that, as it is quite a bit more flexible, but it doesn't seem any cleaner.

I think it's cleaner, because it uses a concept we already have (argument adapters).

Will you point me to an example of these. Google is coming up empty.

I think it's cleaner, because it uses a concept we already have (argument adapters).

Will you point me to an example of these. Google is coming up empty.

Actually, now that I figured out you mean ArgumentAdjusters, I am making progress.

Actually, now that I figured out you mean ArgumentAdjusters, I am making progress.

Unfortunately, ArgumentAdjusters only work on vector<std::string>, and while ToolInvocation::Invocation takes its arguments in that form, tooling::newInvocation (which returns a CompilerInvocation) takes a SmallVector<const char *, 16>, so we either need to change one side's interface, or write two ArgumentAdjusters, but with similar semantics.

What is your preferred solution?

I mean, arguments need to be adjusted before converting to ArgStringList and calling newInvocation? I'm not sure I fully understand the problem, can you elaborate?

I mean, arguments need to be adjusted before converting to ArgStringList and calling newInvocation? I'm not sure I fully understand the problem, can you elaborate?

This gets back to why the original patch plumbed the boolean all the way down to newInvocation.

newInvocation is the function that actually discards the dependency file options--today unconditionally. But there is code that calls newInvocation directly (ClangFuzzer is one), without going through a higher-level API. So I can't adjust the arguments at a higher level and still preserve the old behavior.

Unfortunately, newInvocation's argument list type is incompatible with ArgumentAdjusters, so something else will need to be done. What do you recommend?

I mean, arguments need to be adjusted before converting to ArgStringList and calling newInvocation? I'm not sure I fully understand the problem, can you elaborate?

This gets back to why the original patch plumbed the boolean all the way down to newInvocation.

newInvocation is the function that actually discards the dependency file options--today unconditionally.

Correct, and it shouldn't.

But there is code that calls newInvocation directly (ClangFuzzer is one), without going through a higher-level API. So I can't adjust the arguments at a higher level and still preserve the old behavior.

Can't the call sites that use it themselves fix the arguments themselves? We should be able to provide a convenience wrapper for those use cases, I'd guess they all look basically the same?

Unfortunately, newInvocation's argument list type is incompatible with ArgumentAdjusters, so something else will need to be done. What do you recommend?

I'd have expected something like:

  1. create ArgumentAdjuster that filters out deps file creation args
  2. make that the default in the higher level APIs that already use ArgumentAdjuster
  3. for each call site that uses the lower level API, look into what data they have (probably just argv from main?) and create some nice ArgumentAdjuster wrapper so they can use the default argument adjuster with a 1 line change

Rework this patch to use argument adjusters. It turns out that
the call to newInvocation from ClangFuzzer has a very limited
set of hard-coded arguments, so I don't think it is necessary
to do the hand-adjusting there. Any new callers would be on their
own.

bkramer edited reviewers, added: klimek; removed: bkramer.Jun 30 2017, 3:16 AM
klimek accepted this revision.Jun 30 2017, 3:20 AM

LG, thx for bearing with me, this looks great.
(and sorry if I didn't send this earlier, just noticed I forgot to hit submit :( )

lib/Tooling/ArgumentsAdjusters.cpp
66 ↗(On Diff #104307)

s/ also/, too/?

This revision is now accepted and ready to land.Jun 30 2017, 3:20 AM
saugustine marked an inline comment as done.Jul 6 2017, 2:01 PM
saugustine closed this revision.Jul 6 2017, 2:03 PM