[Driver] Update GCC libraries detection logic for Gentoo.
ClosedPublic

Authored by manojgupta on Apr 3 2018, 3:47 PM.

Details

Summary
  1. Find GCC's LDPATH from the actual GCC config file.
  2. Avoid picking libraries from a similar named tuple if the exact tuple is installed.

Diff Detail

Repository
rC Clang
manojgupta created this revision.Apr 3 2018, 3:47 PM

Michał, will appreciate if you can test this on your (multiple?) Gentoo configurations.
Will work on updating the testcases after that.

To test it locally, I used the following :

$ cat test.cpp

#include <iostream>
#include <string>

int main() {
  std::cout << " Hello, World!" << std::endl;
  return 0;
}

Setup cross compilers for x86_64 and aarch64.

$ sudo crossdev -S x86_64-gentoo-linux-gnu
$ sudo crossdev -S aarch64-gentoo-linux-gnu

$ Build and install clang in a location other than /usr/bin e.g. $HOME/clang-testing/bin
Verify that correct gcc libs are picked.
$ /path/to/clang++ --sysroot=/usr/aarch64-gentoo-linux-gnu -B/usr/libexec/gcc/aarch64-gentoo-linux-gnu -target aarch64-gentoo-linux-gnu test.cpp -o main -Wl,-t -v

Further verify that any lib/headers from x86_64-pc-linux-gnu are NOT picked after the patch is applied.
$ /path/to/clang++ --sysroot=/usr/x86_64-gentoo-linux-gnu -B/usr/libexec/gcc/x86_64-gentoo-linux-gnu -target x86_64-gentoo-linux-gnu test.cpp -o main -Wl,-t -v

Chandler, I recall that you are also a Gentoo user so please take a look.

mgorny requested changes to this revision.Apr 4 2018, 12:09 AM
mgorny added inline comments.
lib/Driver/ToolChains/Gnu.cpp
2287

I'm not sure if there's a point in keeping this if you actually parse the config file. I can't think of a really valid case when GCC would be installed without LDPATH in the config file. Not saying it's wrong. You may want at least to push it after the configfile paths though.

2290

Typo: 'libaries'

2299

Misindent.

2303

This probably doesn't harm but I'd prefer if quotes weren't considered an obligatory part of the string. I'd rather grab it by LDPATH= and strip quotes afterwards if present on both sides. But I won't insist on this.

2306

Here you seem to assume that there would be at most 2 paths. That's a wrong assumption — there are triple-ABI targets (e.g. amd64 with x32 variant), and there is no reason prohibiting more LDPATHs. So please use the 'full' split, and iterate over all paths.

This revision now requires changes to proceed.Apr 4 2018, 12:09 AM

Address requested changes.

manojgupta marked 3 inline comments as done.Apr 4 2018, 12:24 PM
manojgupta added inline comments.
lib/Driver/ToolChains/Gnu.cpp
2306

I had forgotten about x32. Splitting to get all paths is the right thing to do anyway.

mgorny added a comment.Apr 4 2018, 1:13 PM

Thanks. Besides that one tiny nit, looks good at a first glance. I'll test it tomorrow or the next day (but only for the most basic use, sorry).

However, I do not feel confident enough with Clang code to ack change this large on my own. So let's wait for @chandlerc to take a second look, or maybe try to add more reviewers.

lib/Driver/ToolChains/Gnu.cpp
2303

Well, I meant stripping them if both are present. However, this is not a biggie.

Adding more reviewers.

rnk accepted this revision.Apr 4 2018, 2:43 PM

Thousands of lines of C++ to answer one of life's most simple questions: where are the libraries and where are the headers. =p

lgtm

lib/Driver/ToolChains/Gnu.cpp
2229–2278

Can you break the loop here to reduce indentation of the config parsing logic below?

manojgupta updated this revision to Diff 141084.Apr 4 2018, 4:19 PM

Fix some typos.
Reduced indent in the loop.

manojgupta updated this revision to Diff 141096.Apr 4 2018, 5:44 PM

Fix a bug in passing the actual path when libraries are present in sysroot.

Modified an existing gentoo sysroot as the test case. Please let me know if you want me checkin a new sysroot.

manojgupta updated this revision to Diff 141099.Apr 4 2018, 7:28 PM

Avoid double appending of sysroot prefix to base path.

If that's not a problem, then the more tests, the merrier ;-). Preferably something specific to crossdev would be helpful, given this is a new use case, and/or something that would actually have directory mismatches with CURRENT entry name (i.e. that wouldn't have worked before).

Updated tests and added a sysroot to test that LDLIBS are picked by
parsing /etc/env.d/gcc/tuple-version file.

Ok, I've tried it on top of clang-6.0.0 and I get the following test failures:

Failing Tests (8):
    Clang :: Driver/android-ndk-standalone.cpp
    Clang :: Driver/constructors.c
    Clang :: Driver/cuda-detect.cu
    Clang :: Driver/env.c
    Clang :: Driver/gcc-version-debug.c
    Clang :: Driver/linux-header-search.cpp
    Clang :: Driver/linux-ld.c
    Clang :: Driver/pic.c

Could you check whether the tests pass for you? Me using 6.0.0 may be the culprit here but I don't think it should have such significant effect.

My understanding is that some of the tests are failing since we are scanning for Gentoo specific GCC installation first in both sysroot and "/". So, clang succeeds in find the libraries in "/usr/lib/gcc" even when sysroot argument is passed.
Previously, "/usr/lib/gcc" was not scanned when sysroot was passed.

Maybe, it is better to add /usr to list of paths to search instead of scanning in "/"?

Basically, Remove the newly added line 2245: SysRootPrefixes.push_back("");
and instead do Prefixes.push_back("/usr") ?

mgorny added a comment.Apr 5 2018, 2:23 PM

Ok, that's a problem. I think we really ought to consider all possibilites of sysroot first, and either do not fall back to main system at all or do that only if sysroot doesn't have any install at all. Basically, it is important that we don't break non-Gentoo sysroots, even when running on top of Gentoo.

I don't really understand what you mean with the prefixes stuff but it doesn't sound like it's going to solve the problem.

manojgupta added a comment.EditedApr 5 2018, 6:10 PM

I think the tests are already broken in Gentoo when clang is installed in /usr/bin even without this patch. The tests only work if clang binary is not installed in /usr/bin.

RootCause is the existing lines in Gnu.cpp:

// Then look for gcc installed alongside clang.
Prefixes.push_back(D.InstalledDir + "/..");

e.g. Specified debian sysroot is not picked as expected for the following command line.

$ clang -v --target=i386-unknown-linux --gcc-toolchain="" --sysroot=test/Driver/Inputs/debian_multiarch_tree 2>&1
clang version 7.0.0
Target: i386-unknown-linux
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-pc-linux-gnu/4.9.x
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/4.9.x
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/4.9.x
Found candidate GCC installation: test/Driver/Inputs/debian_multiarch_tree/usr/lib/gcc/i686-linux-gnu/4.5
Found candidate GCC installation: test/Driver/Inputs/debian_multiarch_tree/usr/lib/gcc/x86_64-linux-gnu/4.5
Selected GCC installation: /usr/bin/../lib/gcc/i686-pc-linux-gnu/4.9.x
Candidate multilib: .;@m32
Selected multilib: .;@m32

To be honest, I don't really know. But since we're not installing it straight to /usr, I suppose that's not a problem we need to solve right now.

However, they do work with clang installed in /usr/lib/llvm/*/bin, and this patch must not regress that. So if your /usr hack solves that, I don't see a problem with it.

manojgupta updated this revision to Diff 141295.Apr 6 2018, 2:00 AM
manojgupta edited the summary of this revision. (Show Details)

Can solve crossdev issue by passing -gcc-toolchain=/usr
so just detect gcc libs as an offset to sysroot for now.

Michał, can you give the latest change a try? Hopefully it should not break any tests.

mgorny added a comment.Apr 6 2018, 2:44 AM

Well, it's better:

Failing Tests (1):
    Clang :: Driver/linux-header-search.cpp

It is apparently the new test failing:

+ /var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src-abi_x86_32.x86/bin/clang -no-canonical-prefixes /var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/linux-header-search.cpp -### -fsyntax-only -target i386-unknown-linux-gnu -stdlib=libstdc++ --sysroot=/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/Inputs/gentoo_linux_gcc_multi_version_tree --gcc-toolchain=
+ /var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src-abi_x86_32.x86/bin/clang -no-canonical-prefixes /var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/linux-header-search.cpp -### -fsyntax-only -target x86_64-unknown-linux-gnu -stdlib=libstdc++ --sysroot=/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/Inputs/gentoo_linux_gcc_4.9.x_tree --gcc-toolchain=
+ /usr/lib64/llvm/6/bin/FileCheck --check-prefix=CHECK-GENTOO-4-9-X /var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/linux-header-search.cpp
/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/linux-header-search.cpp:359:24: error: expected string not found in input
// CHECK-GENTOO-4-9-X: "-internal-isystem" "[[SYSROOT]]/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.x/include/g++-v4.9.3"
                       ^
<stdin>:5:752: note: scanning from here
 "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src-abi_x86_32.x86/bin/clang" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-fsyntax-only" "-disable-free" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "linux-header-search.cpp" "-mrelocation-model" "static" "-mthread-model" "posix" "-mdisable-fp-elim" "-fmath-errno" "-masm-verbose" "-mconstructor-aliases" "-munwind-tables" "-fuse-init-array" "-target-cpu" "x86-64" "-dwarf-column-info" "-debugger-tuning=gdb" "-resource-dir" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src-abi_x86_32.x86/bin/../../../../lib/clang/6.0.0" "-isysroot" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/Inputs/gentoo_linux_gcc_4.9.x_tree" "-internal-isystem" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/Inputs/gentoo_linux_gcc_4.9.x_tree/usr/local/include" "-internal-isystem" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src-abi_x86_32.x86/bin/../../../../lib/clang/6.0.0/include" "-internal-externc-isystem" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/Inputs/gentoo_linux_gcc_4.9.x_tree/include" "-internal-externc-isystem" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/Inputs/gentoo_linux_gcc_4.9.x_tree/usr/include" "-fdeprecated-macro" "-fdebug-compilation-dir" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src-abi_x86_32.x86/test/Driver" "-ferror-limit" "19" "-fmessage-length" "0" "-fobjc-runtime=gcc" "-fcxx-exceptions" "-fexceptions" "-fdiagnostics-show-option" "-x" "c++" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/linux-header-search.cpp"
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ^
<stdin>:5:752: note: with variable "SYSROOT" equal to "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/Inputs/gentoo_linux_gcc_4.9.x_tree"
 "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src-abi_x86_32.x86/bin/clang" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-fsyntax-only" "-disable-free" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "linux-header-search.cpp" "-mrelocation-model" "static" "-mthread-model" "posix" "-mdisable-fp-elim" "-fmath-errno" "-masm-verbose" "-mconstructor-aliases" "-munwind-tables" "-fuse-init-array" "-target-cpu" "x86-64" "-dwarf-column-info" "-debugger-tuning=gdb" "-resource-dir" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src-abi_x86_32.x86/bin/../../../../lib/clang/6.0.0" "-isysroot" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/Inputs/gentoo_linux_gcc_4.9.x_tree" "-internal-isystem" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/Inputs/gentoo_linux_gcc_4.9.x_tree/usr/local/include" "-internal-isystem" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src-abi_x86_32.x86/bin/../../../../lib/clang/6.0.0/include" "-internal-externc-isystem" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/Inputs/gentoo_linux_gcc_4.9.x_tree/include" "-internal-externc-isystem" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/Inputs/gentoo_linux_gcc_4.9.x_tree/usr/include" "-fdeprecated-macro" "-fdebug-compilation-dir" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src-abi_x86_32.x86/test/Driver" "-ferror-limit" "19" "-fmessage-length" "0" "-fobjc-runtime=gcc" "-fcxx-exceptions" "-fexceptions" "-fdiagnostics-show-option" "-x" "c++" "/var/tmp/portage/sys-devel/clang-6.0.0-r1/work/x/y/cfe-6.0.0.src/test/Driver/linux-header-search.cpp"
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ^

--

********************

I can't reproduce this using the trunk llvm. Does it pass for you if %clang is replaced by %clang++ in the test case?

mgorny added a comment.Apr 7 2018, 4:03 AM

I'm sorry, I see the problem now — the diff generated by Phabricator does not include the empty files x_x (seriously, this thing keeps surprising me in how broken it could be). I'm going to try again with correct file set tonight or tomorrow. If you could send the complete patch (preferably -p1 if you have one) to mgorny AT gentoo.org, that would also be helpful.

mgorny accepted this revision.Apr 7 2018, 12:26 PM

Works fine, thanks a lot! Note that I haven't tested crossdev or anything special, just regular multilib.

This revision is now accepted and ready to land.Apr 7 2018, 12:26 PM
This revision was automatically updated to reflect the committed changes.