This is an archive of the discontinued LLVM Phabricator instance.

[clang] Don't look into <sysroot> for C++ headers if they are found alongside the toolchain
ClosedPublic

Authored by ldionne on Oct 7 2020, 12:29 PM.

Details

Summary

Currently, Clang looks for libc++ headers alongside the installation
directory of Clang, and it also adds a search path for headers in the
-isysroot. This is problematic if headers are found in both the toolchain
and in the sysroot, since #include_next will end up finding the libc++
headers in the sysroot instead of the intended system headers.

This patch changes the logic such that if the toolchain contains libc++
headers, no C++ header paths are added in the sysroot. However, if the
toolchain does *not* contain libc++ headers, the sysroot is searched as
usual.

This should not be a breaking change, since any code that previously
relied on some libc++ headers being found in the sysroot suffered from
the #include_next issue described above, which renders any libc++ header
basically useless.

Diff Detail

Event Timeline

ldionne created this revision.Oct 7 2020, 12:29 PM
Herald added a project: Restricted Project. · View Herald TranscriptOct 7 2020, 12:29 PM
ldionne requested review of this revision.Oct 7 2020, 12:29 PM
dexonsmith accepted this revision.Oct 8 2020, 8:21 AM
This revision is now accepted and ready to land.Oct 8 2020, 8:21 AM
ldionne updated this revision to Diff 297000.Oct 8 2020, 9:47 AM

Try to fix test failure on Linux

@ldionne and @dexonsmith, it seems that this diff breaks clang-tools like clang-tidy and clangd if -isysroot is specified in the compilation database. Moreover, clang from Xcode 14 and 15 uses paths specified in -sysroot and ignores headers alongside the clang binary. I don’t know if it is new behavior or it was this way when this diff was committed. Minimal reproducer is following:

$ cat compile_commands.json
[
    {
        "directory": "/Users/test/test",
        "file": "test.cpp",
        "arguments": [
            "/Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++",
            "-isysroot",
            "/Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk",
            "-c",
            "test.cpp",
            "-v"
        ]
    }
]

$ <upstream>/build/bin/clang-tidy -p compile_commands.json test.cpp
...
#include <...> search starts here:
 /Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1
 <upstream>/build/lib/clang/18/include
 /Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
 /Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
1 error generated.
Error while processing test.cpp.
test.cpp:1:10: error: 'cxxabi.h' file not found [clang-diagnostic-error]
    1 | #include <cxxabi.h>
      |          ^~~~~~~~~~
Found compiler error(s).

$ /Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -isysroot /Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -c test.cpp -v
...
#include <...> search starts here:
 /Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1
 /Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/14.0.3/include
 /Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
 /Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /Applications/Xcode_14.3.1_14E300b.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.

So upstream clang-tools uses a path to the compiler binary specified in CDB as an installation dir and searches in ".../XcodeDefault.xctoolchain/usr/bin/../include/c++/v1" but the Apple compiler from CDB searches in the path specified in sysroot ".../MacOSX.sdk/usr/include/c++/v1". I also checked clangd from Apple toolchain, it also uses path species in -sysroot. I have a local patch that honors -isysroot if it is explicitly specified in the command line but prefers headers alongside the binary in other cases and it seems that it matches the behavior of Apple compiler. But it breaks all tests that specifically test that -isysroot is ignored. Should I change the test and send the patch for review?

Herald added a project: Restricted Project. · View Herald TranscriptAug 4 2023, 10:24 AM
Herald added a subscriber: MaskRay. · View Herald Transcript

I don't have access to rdar these days to look into the current state or to refresh my memory.

My memory is that the high-level goal was to move the libc++ headers from the toolchain to the SDK. For the transition, this was staged in such a way that they were in both locations for some period of time (internally, the transition may be ongoing, for all I know).

Apple clang needed to prefer the SDK, since that was the "new" place. I don't remember the reasoning for inverting this behaviour for LLVM clang.

  • Maybe, because this way an LLVM toolchain could still override the C++ headers, like it could previously?
  • Maybe, there was a plan to eventually switch the default in LLVM clang, but that got dropped/forgotten?
  • Maybe, there was a plan to eventually switch the default in Apple clang, but that was dropped/forgotten, or the internal transition is ongoing? (This rings a bell. "Change LLVM clang to the desired end state, and we'll flip the direction temporarily in Apple clang for the duration of the transition...")

Anyway, I'm not sure the best way forward here. Perhaps clang should have a flag to decide which is preferred. Or perhaps LLVM clang should just change to match Apple clang for now.

@ldionne, thoughts?

@dexonsmith thank you for quick response. I think we need to update upstream behaviour to match system compiler behaviour otherwise clang-tools that uses path to compiler binary form CDB will not work as expected and extra options won't help much because it is not easy to inject in some cases. I sent https://reviews.llvm.org/D157283 for review. I got this change by running test agains system compiler and fixing it to match the real behaviour, after that I guessed how exactly this behaviour could be achieved.

I just did some digging. We completed the transition to the libcxx-headers-in-the-SDK already, and now upstream and downstream Clang are the same with respect to those search paths. I think it might just be the case that you need the very latest AppleClang for that to be reflected (as you know AppleClang always lags a bit behind). So I think there's nothing special to do here, upstream behaves properly and downstream will catch up automatically.