Not for review -- test for not spawning subprocess for -cc1

[thakis]

Summary

Instead, let driver directly call Frontend's main()

[aganea]

This patch doesn't link on Windows/VS2017 as main() is not in the same library as clangDriver. So I went ahead and added the following in clang/trunk/tools/driver/driver.cpp:

int main(int argc_, const char **argv_) {
  llvm::InitLLVM X(argc_, argv_);
  SmallVector<const char *, 256> argv(argv_, argv_ + argc_);

  if (llvm::sys::Process::FixupStandardFileDescriptors())
    return (int)llvm::sys::ProgramReturnCode::Failure;

  llvm::InitializeAllTargets();
  auto TargetAndMode = ToolChain::getTargetAndModeFromProgramName(argv[0]);

  llvm::BumpPtrAllocator A;
  llvm::StringSaver Saver(A);

  llvm::sys::DynamicLibrary::AddSymbol("main", &main); /////////////////////////////////////// here

  // Parse response files using the GNU syntax, unless we're in CL mode. There
  // are two ways to put clang in CL compatibility mode: argv[0] is either
  // clang-cl or cl, or --driver-mode=cl is on the command line. The normal
  // command line parsing can't happen until after response file parsing, so we
  // have to manually search for a --driver-mode=cl argument the hard way.
  // Finally, our -cc1 tools don't care which tokenization mode we use because
  // response files written by clang will tokenize the same way in either mode.

Then I've called it from Compilation.cpp with:

void Compilation::ExecuteJobs(const JobList &Jobs,
                              FailingCommandList &FailingCommands) const {
  if (Jobs.size() == 1) {
    Command &C = *Jobs.begin();
    if (C.getExecutable() == getDriver().getClangProgramPath()) {
      ArgStringList Args = C.getArguments();
      Args.insert(Args.begin(), C.getExecutable());
      //main(Args.size(), const_cast<const char **>(&Args[0]));

      MainFn M = FuncPtr<MainFn>(llvm::sys::DynamicLibrary::SearchForAddressOfSymbol("main"));
      M(Args.size(), const_cast<const char **>(&Args[0]));
    }
    return;
  }

[aganea]

Cross referencing https://reviews.llvm.org/D52193#1260825 here:

Updated tests again. This time I've closed all applications, all unused services, and tried to have a "clean" machine.

New timings without the child clang-cl.exe being invoked (hacked from D52411). The test consists in building Clang+LLVM+LLD at r343846 using the compiler specified on each line of the table.

Config 1: Intel Xeon Haswell 6 cores / 12 HW threads, 3.5 GHz, 15 MB cache, 128 GB RAM, SSD 550 MB/s (Windows 10 Fall Creators update 1709)

Ninja:

MSVC 15.8.6 (cl + link)	29 min 4 sec
clang-cl + lld-link 7.0 official release	25 min 45 sec
clang-cl + lld-link 8.0 r343846 built with clang-cl 7.0 official (+D52411, no child)	23 min 27 sec

Config 2: Intel Xeon Skylake 18 cores / 36 HW threads, x2 (Dual CPU), 72 HW threads total, 2.3 GHz, 24.75 MB cache each, 128 GB RAM, NVMe SSD 4.6 GB/s (Windows 10 Fall Creators update 1709)

Ninja:

MSVC 15.8.6 (cl + link)	6 min 40 sec
clang-cl + lld-link 7.0 official release	6 min 24 sec
clang-cl + lld-link 8.0 r343846 built with clang-cl 7.0 official (+D52411, no child)	5 min 10 sec

I'm wondering if the improvement is of the same order on Linux. I'd be interested to see your build times, out of curiosity? Can any of you try it with and without D52411?

[thakis]

In D52411#1260822, @aganea wrote:

This patch doesn't link on Windows/VS2017 as main() is not in the same library as clangDriver

Huh, I would've expected this to not matter. Are you building all the parts of clang as different .dlls? clangDriver should just become a .lib, and then stuff gets all linked together in a single executable at the end.

[aganea]

In D52411#1274604, @thakis wrote:

In D52411#1260822, @aganea wrote:

This patch doesn't link on Windows/VS2017 as main() is not in the same library as clangDriver

Huh, I would've expected this to not matter. Are you building all the parts of clang as different .dlls? clangDriver should just become a .lib, and then stuff gets all linked together in a single executable at the end.

It's actually libclang.dll that doesn't compile - not a big deal.

121>   Creating library F:/svn/build/Release/lib/libclang.lib and object F:/svn/build/Release/lib/libclang.exp
121>clangDriver.lib(Compilation.obj) : error LNK2019: unresolved external symbol main referenced in function "public: void __cdecl clang::driver::Compilation::executeJobs(class llvm::SmallVectorImpl<struct clang::driver::JobExecutionState> &)" (?executeJobs@Compilation@driver@clang@@QEAAXAEAV?$SmallVectorImpl@UJobExecutionState@driver@clang@@@llvm@@@Z)
121>F:\svn\build\Release\bin\libclang.dll : fatal error LNK1120: 1 unresolved externals

lib/Driver/Compilation.cpp

 using namespace clang;
 using namespace llvm::opt;
 
+int main(int argc_, const char **argv_);
+
 Compilation::Compilation(const Driver &D, const ToolChain &_DefaultToolChain,
                          InputArgList *_Args, DerivedArgList *_TranslatedArgs)
     : TheDriver(D), DefaultToolChain(_DefaultToolChain), Args(_Args),

   if (const Command *C = dyn_cast<Command>(&J)) {
     if (!InputsOk(*C, FailingCommands))
       return;
-    const Command *FailingCommand = nullptr;
-    if (int Res = ExecuteCommand(*C, FailingCommand))
-      FailingCommands.push_back(std::make_pair(Res, FailingCommand));
+    if (C->getExecutable() == getDriver().getClangProgramPath()) {
+      ArgStringList Args = C->getArguments();
+Args.insert(Args.begin(), C->getExecutable());
+//fprintf(stderr, "in-process\n");
+// XXX response files
+      main(Args.size(), const_cast<const char**>(&Args[0]));
+// XXX this breaks /fallback
+exit(0);
+    } else {
+      const Command *FailingCommand = nullptr;
+      if (int Res = ExecuteCommand(*C, FailingCommand))
+        FailingCommands.push_back(std::make_pair(Res, FailingCommand));
+    }
   } else {
     const JobList *Jobs = cast<JobList>(&J);
     for (const auto &Job : *Jobs)

tools/driver/driver.cpp

   if (llvm::sys::Process::FixupStandardFileDescriptors())
     return 1;
 
+//for (int i = 0; i < argc_; ++i) fprintf(stderr, " %s", argv_[i]);
+//fprintf(stderr, "\n");
+
   SmallVector<const char *, 256> argv;
   llvm::SpecificBumpPtrAllocator<char> ArgAllocator;
-  std::error_code EC = llvm::sys::Process::GetArgumentVector(
-      argv, llvm::makeArrayRef(argv_, argc_), ArgAllocator);
-  if (EC) {
-    llvm::errs() << "error: couldn't get arguments: " << EC.message() << '\n';
-    return 1;
-  }
+// XXX :-(
+  //std::error_code EC = llvm::sys::Process::GetArgumentVector(
+      //argv, llvm::makeArrayRef(argv_, argc_), ArgAllocator);
+for (int i = 0; i < argc_; ++i) argv.push_back(argv_[i]);
+  //if (EC) {
+    //llvm::errs() << "error: couldn't get arguments: " << EC.message() << '\n';
+    //return 1;
+  //}
 
   std::set<std::string> SavedStrings;
   StringSetSaver Saver(SavedStrings);