Page MenuHomePhabricator

Move detection of libc++ include dirs to Driver on MacOS
ClosedPublic

Authored by ilya-biryukov on Nov 16 2018, 5:46 AM.

Details

Summary

The intention is to make the tools replaying compilations from 'compile_commands.json'
(clang-tidy, clangd, etc.) find the same standard library as the original compiler
specified in 'compile_commands.json'.

Previously, the library detection logic was in the frontend (InitHeaderSearch.cpp) and relied
on the value of resource dir as an approximation of the compiler install dir. The new logic
uses the actual compiler install dir and is performed in the driver. This is consistent with
the C++ standard library detection on other platforms and allows to override the resource dir
in the tools using the compile_commands.json without altering the
standard library detection mechanism. The tools have to override the resource dir to make sure
they use a consistent version of the builtin headers.

There is still logic in InitHeaderSearch that attemps to add the absolute includes for the
the C++ standard library, so we keep passing the -stdlib=libc++ from the driver to the frontend
via cc1 args to avoid breaking that. In the long run, we should move this logic to the driver too,
but it could potentially break the library detection on other systems, so we don't tackle it in this
patch to keep its scope manageable.

This is a second attempt to fix the issue, first one was commited in r346652 and reverted in r346675.
The original fix relied on an ad-hoc propagation (bypassing the cc1 flags) of the install dir from the
driver to the frontend's HeaderSearchOptions. Unsurpisingly, the propagation was incomplete, it broke
the libc++ detection in clang itself, which caused LLDB tests to break.

The LLDB tests pass with new fix.

Diff Detail

Repository
rC Clang

Event Timeline

ilya-biryukov created this revision.Nov 16 2018, 5:46 AM

I don't think we want to move the logic to add a libc++ path to the driver. -cc1 with -resource-dir and -stdlib=libc++ should still work as before. In this case the previous patch is better, except it should not set InstalledDir except when needed (e.g. for tooling when working with a CDB that has an absolute path to the compiler), so when InstalledDir is empty it should fallback to the current logic in InitHeaderSearch.cpp. That should solve the issues we had.

I don't think we want to move the logic to add a libc++ path to the driver.

My opinion is not very educated because I don't have a lot of experience/knowledge of Clang, but from my perspective it does make sense to limit how much knowledge of the standard library path the front end needs to have. IOW, having the driver figure it out and then pass include paths to the front-end just like for any other library does make sense from my limited point of view. Again -- this is not a practical argument, it's just from a conceptual perspective.

I don't think we want to move the logic to add a libc++ path to the driver. -cc1 with -resource-dir and -stdlib=libc++ should still work as before. In this case the previous patch is better, except it should not set InstalledDir except when needed (e.g. for tooling when working with a CDB that has an absolute path to the compiler), so when InstalledDir is empty it should fallback to the current logic in InitHeaderSearch.cpp. That should solve the issues we had.

Why not?
@ldionne made a good point: there is value in keeping the frontend separate from the logic to detect system libraries and other toolchain-specific things.

This is how other toolchains do it (Linux, etc.). It's more straightforward to the user (one can inspect the -cc1 args produced by the driver), this is where the logic to detect linker dependencies is. It seems that was also the plan of some developers looking in that area: the test I've edited suggests this should be moved to the driver (darwin-stdlib.cpp, it says: If and when we change to finding them in the driver ...), there's a bug corresponding bug to do this: http://llvm.org/PR30548.
I might not have full information, but my understanding is that -resource-dir has a pretty well-defined semantics: it should be used to find the compiler resources, i.e. only the buit-in headers and not the C++ standard library.

What are the reasons we need to keep the old behavior?

My two cents:

I needed something like this when trying to automatically detect if libc++ supported things like sized deallocation, so I would like to see this change go through.
(My understanding was that libc++ added a __libcpp_version file for just this reason, but it's hard to detect because the include path computation is done after we initialize the language options)

I don't think we want to move the logic to add a libc++ path to the driver. -cc1 with -resource-dir and -stdlib=libc++ should still work as before. In this case the previous patch is better, except it should not set InstalledDir except when needed (e.g. for tooling when working with a CDB that has an absolute path to the compiler), so when InstalledDir is empty it should fallback to the current logic in InitHeaderSearch.cpp. That should solve the issues we had.

Why not?
@ldionne made a good point: there is value in keeping the frontend separate from the logic to detect system libraries and other toolchain-specific things.

This is how other toolchains do it (Linux, etc.). It's more straightforward to the user (one can inspect the -cc1 args produced by the driver), this is where the logic to detect linker dependencies is. It seems that was also the plan of some developers looking in that area: the test I've edited suggests this should be moved to the driver (darwin-stdlib.cpp, it says: If and when we change to finding them in the driver ...), there's a bug corresponding bug to do this: http://llvm.org/PR30548.
I might not have full information, but my understanding is that -resource-dir has a pretty well-defined semantics: it should be used to find the compiler resources, i.e. only the buit-in headers and not the C++ standard library.

What are the reasons we need to keep the old behavior?

Sounds convincing.
@dexonsmith What do you think?

@ilya-biryukov I'm going to do some internal testing to see if we uncover any issues.

dexonsmith added 1 blocking reviewer(s): arphaman.Nov 26 2018, 1:44 PM

Sounds convincing.
@dexonsmith What do you think?

Besides maintaining correct behaviour, I think the most important thing here is that the code organization is logical. Header search is complicated and we should be trying to make/keep it simple. I'm a little skeptical that splitting this logic between cc1 and the driver will simplify things, but I haven't looked in detail and I'll defer to your (collective) judgement.

@ilya-biryukov I'm going to do some internal testing to see if we uncover any issues.

I've added you as a blocking reviewer since I think we should work through any uncovered issues pre-commit.

I'm a little skeptical that splitting this logic between cc1 and the driver will simplify things, but I haven't looked in detail and I'll defer to your (collective) judgement.

Most of the toolchain search logic is in the driver, so moving from the frontend to the driver is actually a step in moving the logic into one place.

@ilya-biryukov I'm going to do some internal testing to see if we uncover any issues.

Many thanks, will be waiting for the results of the internal testing.

ilya-biryukov added a comment.EditedNov 30 2018, 9:14 AM

@arphaman, did you have a chance to run the tests? There's no rush, just wanted to know whether we have any data if this change breaks anything in practice?

So far everything looks fine, but I have to rerun a couple of things due to infrastructural issues. I should have the final results next Monday.

So far everything looks fine, but I have to rerun a couple of things due to infrastructural issues. I should have the final results next Monday.

Thanks for the update. Will be waiting for the final results.

arphaman accepted this revision.Dec 3 2018, 1:40 PM

LGTM

This revision is now accepted and ready to land.Dec 3 2018, 1:40 PM

Please mention this change in the release notes though.

This revision was automatically updated to reflect the committed changes.

D55322 is a review for release notes

ormris added a subscriber: ormris.Jan 15 2019, 1:09 PM

Hi Ilya,

This commit is causing a test failure on the PS4 bot. Could you take a look?

http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast/builds/42251

Thanks,
Matthew

Will take a look.
For reference: the particular failing test was added in r351222 (D56680), not in this change.

r351334 should fix this.
However, this is an indication that libc++ living alongside the compiler (i.e. in /bin/../include/c++/v1) cannot be found on PS4.
I am not familiar with PS4 myself, but I believe it would be reasonable to assume finding libc++ living alongside the clang itself should work on all platforms, including PS4.