[clang][driver] Add gcc-toolset/devtoolset 12 to prefixes

[tbaeder]

Summary

This makes it possible to use the newest gcc-toolset/devtoolset on RHEL.

[MaskRay]

It may be time to add unittests to clang/unittests/Driver/ToolChainTest.cpp for /opt/rh detection. You may use TEST(ToolChainTest, VFSGCCInstallation) as an example creating a pseudo file hierarchy in memory.

[MaskRay]

In addition, if every new GCC release requires addition to /opt/rh/gcc-toolset-$major (if I understand correctly), we probably want to switch to llvm::vfs::directory_iterator iteration and using the largest version.

[MaskRay]

clang/lib/Driver/ToolChains/Gnu.cpp

2164

delete blank line

2170

The two if can be combined

clang/unittests/Driver/ToolChainTest.cpp

623

Add a comment that these directories are ignored.

650

512 is arbitrary. It's good to test something close to the real.

[MaskRay]

Thanks!

clang/unittests/Driver/ToolChainTest.cpp

618

End full sentences with a period .

[MaskRay]

Harbormaster completed remote builds in B165047: Diff 430277.

You can get Windows bot results in the link.

[tbaeder]

In D125862#3532630, @MaskRay wrote:

Harbormaster completed remote builds in B165047: Diff 430277.

You can get Windows bot results in the link.

Hm, the windows pre-merge checks don't fail. Just the buildbots. I wonder if the BuildCompilation() calls simply have the wrong parameters. The other calls in the file seem to always create a fake foo.cpp even if they only check the verbose info.

Also not great that printVerboseInfo() seems to print nothing in case of an error :/

[tbaeder]

@MaskRay Ignored the new test on Windows hosts now like you suggested. Does this look good?

[tbaeder]

Ping. @MaskRay Can you take a quick look and check if this is what you had in mind?

[MaskRay]

In D125862#3552465, @tbaeder wrote:

Ping. @MaskRay Can you take a quick look and check if this is what you had in mind?

LG

clang/unittests/Driver/ToolChainTest.cpp

614

delete blank line

622

delete public.

Can you avoid using a derived DiagnosticConsumer?

697

This should be removed if we exclude Windows.

[tbaeder]

clang/unittests/Driver/ToolChainTest.cpp

622

I can use the SimpleDiagnosticConsumer class created earlier in the file.

[nemanjai]

The original toolchain detection added root/usr to the paths which was certainly necessary on PowerPC. Since that suffix is removed in this patch, our Redhat buildbot is now broken (https://lab.llvm.org/buildbot/#/builders/57/builds/18577). @quinnp has posted an update to this at https://reviews.llvm.org/D127310.

[ebevhan]

Hi! This test is failing in some of our downstream builds.

clang/unittests/Driver/ToolChainTest.cpp

646

There's a binary name missing here:

{"clang", "--gcc-toolchain="}

Driver strips the first argument, so the behavior of this test becomes dependent on GCC_INSTALL_PREFIX which can cause it to fail in some cases.

686

Here too.

[tbaeder]

clang/unittests/Driver/ToolChainTest.cpp

646

Thanks, I pushed a commit adding the driver name.

clang/lib/Driver/ToolChains/Gnu.cpp

for (llvm::vfs::directory_iterator LI = D.getVFS().dir_begin(PrefixDir, EC),
       Prefixes.push_back(CandidatePrefix);
     }
     return;
   }
 
   // Non-Solaris is much simpler - most systems just go with "/usr".
   if (SysRoot.empty() && TargetTriple.getOS() == llvm::Triple::Linux) {
     // Yet, still look for RHEL/CentOS devtoolsets and gcc-toolsets.
-    Prefixes.push_back("/opt/rh/gcc-toolset-11/root/usr");
-    Prefixes.push_back("/opt/rh/gcc-toolset-10/root/usr");
-    Prefixes.push_back("/opt/rh/devtoolset-11/root/usr");
-    Prefixes.push_back("/opt/rh/devtoolset-10/root/usr");
-    Prefixes.push_back("/opt/rh/devtoolset-9/root/usr");
-    Prefixes.push_back("/opt/rh/devtoolset-8/root/usr");
-    Prefixes.push_back("/opt/rh/devtoolset-7/root/usr");
-    Prefixes.push_back("/opt/rh/devtoolset-6/root/usr");
-    Prefixes.push_back("/opt/rh/devtoolset-4/root/usr");
-    Prefixes.push_back("/opt/rh/devtoolset-3/root/usr");
-    Prefixes.push_back("/opt/rh/devtoolset-2/root/usr");
+    // Find the directory in /opt/rh/ starting with gcc-toolset-* or
+    // devtoolset-* with the highest version number and add that
+    // one to our prefixes.
+    std::string PrefixDir = SysRoot.str() + "/opt/rh";
+    std::string ChosenToolsetDir;
+    unsigned ChosenToolsetVersion = 0;
+    std::error_code EC;
+    for (llvm::vfs::directory_iterator LI = D.getVFS().dir_begin(PrefixDir, EC),
+                                       LE;
+         !EC && LI != LE; LI = LI.increment(EC)) {
+      StringRef ToolsetDir = llvm::sys::path::filename(LI->path());
+
+      if (!ToolsetDir.startswith("gcc-toolset-") &&
+          !ToolsetDir.startswith("devtoolset-"))
+        continue;
+
+      unsigned ToolsetVersion;
+      if (ToolsetDir.substr(ToolsetDir.rfind('-') + 1)
+              .getAsInteger(10, ToolsetVersion))
+        continue;
+
+      if (ToolsetVersion > ChosenToolsetVersion) {
+        ChosenToolsetVersion = ToolsetVersion;
+        ChosenToolsetDir = PrefixDir + "/" + ToolsetDir.str();
+      }
+    }
+
+    if (ChosenToolsetVersion > 0)
+      Prefixes.push_back(ChosenToolsetDir);
   }
   Prefixes.push_back(SysRoot.str() + "/usr");
 }
 
 /*static*/ void Generic_GCC::GCCInstallationDetector::CollectLibDirsAndTriples(
     const llvm::Triple &TargetTriple, const llvm::Triple &BiarchTriple,
     SmallVectorImpl<StringRef> &LibDirs,
     SmallVectorImpl<StringRef> &TripleAliases,

clang/unittests/Driver/ToolChainTest.cpp

TEST(DxcModeTest, ValidatorVersionValidation) {
   EXPECT_STREQ(
       DiagConsumer->Errors.back().c_str(),
       "invalid validator version : foo\nFormat of validator version is "
       "\"<major>.<minor>\" (ex:\"1.4\").");
   Diags.Clear();
   DiagConsumer->clear();
 }
 
+TEST(ToolChainTest, Toolsets) {
+  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions();
+  IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
+  struct TestDiagnosticConsumer : public DiagnosticConsumer {};
+
+  // Check (newer) GCC toolset installation
+  {
+    IntrusiveRefCntPtr<llvm::vfs::InMemoryFileSystem> InMemoryFileSystem(
+        new llvm::vfs::InMemoryFileSystem);
+
+    InMemoryFileSystem->addFile("/opt/rh/gcc-toolset-", 0,
+                                llvm::MemoryBuffer::getMemBuffer("\n"));
+    InMemoryFileSystem->addFile("/opt/rh/gcc-toolset--", 0,
+                                llvm::MemoryBuffer::getMemBuffer("\n"));
+    InMemoryFileSystem->addFile("/opt/rh/gcc-toolset-2", 0,
+                                llvm::MemoryBuffer::getMemBuffer("\n"));
+    InMemoryFileSystem->addFile("/opt/rh/gcc-toolset--1", 0,
+                                llvm::MemoryBuffer::getMemBuffer("\n"));
+
+    // File needed for GCC installation detection
+    InMemoryFileSystem->addFile(
+        "/opt/rh/gcc-toolset-512/lib/gcc/x86_64-redhat-linux/11/crtbegin.o", 0,
+        llvm::MemoryBuffer::getMemBuffer("\n"));
+
+    DiagnosticsEngine Diags(DiagID, &*DiagOpts, new TestDiagnosticConsumer);
+    Driver TheDriver("/bin/clang", "x86_64-redhat-linux", Diags,
+                     "clang LLVM compiler", InMemoryFileSystem);
+    std::unique_ptr<Compilation> C(TheDriver.BuildCompilation({"-v"}));
+    ASSERT_TRUE(C);
+    std::string S;
+    {
+      llvm::raw_string_ostream OS(S);
+      C->getDefaultToolChain().printVerboseInfo(OS);
+    }
+    if (is_style_windows(llvm::sys::path::Style::native))
+      std::replace(S.begin(), S.end(), '\\', '/');
+    EXPECT_EQ("Found candidate GCC installation: "
+              "/opt/rh/gcc-toolset-512/lib/gcc/x86_64-redhat-linux/11\n"
+              "Selected GCC installation: "
+              "/opt/rh/gcc-toolset-512/lib/gcc/x86_64-redhat-linux/11\n"
+              "Candidate multilib: .;@m64\n"
+              "Selected multilib: .;@m64\n",
+              S);
+  }
+
+  // And older devtoolset
+  {
+    IntrusiveRefCntPtr<llvm::vfs::InMemoryFileSystem> InMemoryFileSystem(
+        new llvm::vfs::InMemoryFileSystem);
+
+    InMemoryFileSystem->addFile("/opt/rh/devtoolset-", 0,
+                                llvm::MemoryBuffer::getMemBuffer("\n"));
+    InMemoryFileSystem->addFile("/opt/rh/devtoolset--", 0,
+                                llvm::MemoryBuffer::getMemBuffer("\n"));
+    InMemoryFileSystem->addFile("/opt/rh/devtoolset-2", 0,
+                                llvm::MemoryBuffer::getMemBuffer("\n"));
+    InMemoryFileSystem->addFile("/opt/rh/devtoolset--1", 0,
+                                llvm::MemoryBuffer::getMemBuffer("\n"));
+
+    // File needed for GCC installation detection
+    InMemoryFileSystem->addFile(
+        "/opt/rh/devtoolset-512/lib/gcc/x86_64-redhat-linux/11/crtbegin.o", 0,
+        llvm::MemoryBuffer::getMemBuffer("\n"));
+
+    DiagnosticsEngine Diags(DiagID, &*DiagOpts, new TestDiagnosticConsumer);
+    Driver TheDriver("/bin/clang", "x86_64-redhat-linux", Diags,
+                     "clang LLVM compiler", InMemoryFileSystem);
+    std::unique_ptr<Compilation> C(TheDriver.BuildCompilation({"-v"}));
+    ASSERT_TRUE(C);
+    std::string S;
+    {
+      llvm::raw_string_ostream OS(S);
+      C->getDefaultToolChain().printVerboseInfo(OS);
+    }
+    if (is_style_windows(llvm::sys::path::Style::native))
+      std::replace(S.begin(), S.end(), '\\', '/');
+    EXPECT_EQ("Found candidate GCC installation: "
+              "/opt/rh/devtoolset-512/lib/gcc/x86_64-redhat-linux/11\n"
+              "Selected GCC installation: "
+              "/opt/rh/devtoolset-512/lib/gcc/x86_64-redhat-linux/11\n"
+              "Candidate multilib: .;@m64\n"
+              "Selected multilib: .;@m64\n",
+              S);
+  }
+}
+
 } // end anonymous namespace.