[ThinLTO] Optionally ignore empty index file

[tejohnson]

Summary

In order to simplify distributed build system integration, where actions
may be scheduled before the Thin Link which determines the list of
objects selected by the linker. The gold plugin currently will emit
0-sized index files for objects not selected by the link, to enable
checking for expected output files by the build system. If the build
system then schedules a backend action for these bitcode files, we want
to be able to fall back to normal compilation instead of failing.

Fallback is enabled under an option in LLVM (D28410), in which case a nullptr
is returned from llvm::getModuleSummaryIndexForFile. Clang can just proceed
with non-ThinLTO compilation in that case.

I am investigating whether this can be addressed in our build system, but that
is a longer term fix and so this enables a workaround in the meantime.

[mehdi_amini]

lib/CodeGen/BackendUtil.cpp

65

Is it common to do this in clang?

[tejohnson]

lib/CodeGen/BackendUtil.cpp

65

Doesn't look common, but I see one other LLVM internal option (in CodeGenPGO.cpp), and a few other uses in tests. Is there a better way to do internal options in clang?

[tejohnson]

lib/CodeGen/BackendUtil.cpp

65

Since this doesn't look like something commonly done in clang, I am splitting this into 2 patches, one in LLVM and one here. LLVM will instead contain the option, and return a nullptr from llvm::getModuleSummaryIndexForFile when the option is enabled and the file is empty. The clang side can just proceed to normal compilation if a nullptr CombinedIndex is returned (if the option is not enabled, LLVM would return an error instead).

[tejohnson]

Move option to LLVM.

[mehdi_amini]

LGTM

[tejohnson]

In D28362#638203, @mehdi_amini wrote:

LGTM

Thanks, let me know if the llvm companion patch (D28410) is ok as well.

[mehdi_amini]

In order to simplify distributed build system integration, where actions
may be scheduled before the Thin Link which determines the list of
objects selected by the linker. The gold plugin currently will emit
0-sized index files for objects not selected by the link, to enable
checking for expected output files by the build system. If the build
system then schedules a backend action for these bitcode files, we want
to be able to fall back to normal compilation instead of failing.

Just thought about it: why not emit an empty *index* instead of an empty file?

[tejohnson]

In D28362#638260, @mehdi_amini wrote:

In order to simplify distributed build system integration, where actions
may be scheduled before the Thin Link which determines the list of
objects selected by the linker. The gold plugin currently will emit
0-sized index files for objects not selected by the link, to enable
checking for expected output files by the build system. If the build
system then schedules a backend action for these bitcode files, we want
to be able to fall back to normal compilation instead of failing.

Just thought about it: why not emit an empty *index* instead of an empty file?

I thought about that initially. But it will also need some special handling as e.g. FunctionImportGlobalProcessing::shouldPromoteLocalToGlobal on the module being compiled will complain when it doesn't find summaries for the GVs in that module. There might be other places too, that's the first one I found when I was thinking about this approach.

lib/CodeGen/BackendUtil.cpp

 #include "llvm/IR/Module.h"
 #include "llvm/IR/ModuleSummaryIndex.h"
 #include "llvm/IR/Verifier.h"
 #include "llvm/LTO/LTOBackend.h"
 #include "llvm/MC/SubtargetFeature.h"
 #include "llvm/Object/ModuleSummaryIndexObjectFile.h"
 #include "llvm/Passes/PassBuilder.h"
 #include "llvm/Support/CommandLine.h"
+#include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/PrettyStackTrace.h"
 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/Timer.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
 #include "llvm/Target/TargetSubtargetInfo.h"
 #include "llvm/Transforms/Coroutines.h"
 #include "llvm/Transforms/IPO.h"
 #include "llvm/Transforms/IPO/AlwaysInliner.h"
 #include "llvm/Transforms/IPO/PassManagerBuilder.h"
 #include "llvm/Transforms/Instrumentation.h"
 #include "llvm/Transforms/ObjCARC.h"
 #include "llvm/Transforms/Scalar.h"
 #include "llvm/Transforms/Scalar/GVN.h"
 #include "llvm/Transforms/Utils/SymbolRewriter.h"
 #include <memory>
 using namespace clang;
 using namespace llvm;
 
+static llvm::cl::opt<bool> IgnoreEmptyThinLTOIndexFile(
+    "ignore-empty-index-file", llvm::cl::ZeroOrMore,
+    llvm::cl::desc(
+        "Ignore an empty index file and perform non-ThinLTO compilation"),
+    llvm::cl::init(false));
+
 namespace {
 
 class EmitAssemblyHelper {
   DiagnosticsEngine &Diags;
   const CodeGenOptions &CodeGenOpts;
   const clang::TargetOptions &TargetOpts;
   const LangOptions &LangOpts;
   Module *TheModule;

 
 void clang::EmitBackendOutput(DiagnosticsEngine &Diags,
                               const CodeGenOptions &CGOpts,
                               const clang::TargetOptions &TOpts,
                               const LangOptions &LOpts, const llvm::DataLayout &TDesc,
                               Module *M, BackendAction Action,
                               std::unique_ptr<raw_pwrite_stream> OS) {
   if (!CGOpts.ThinLTOIndexFile.empty()) {
+    bool DoThinLTOBackend = true;
+    if (IgnoreEmptyThinLTOIndexFile) {
+      uint64_t IndexFileSize;
+      llvm::sys::fs::file_size(CGOpts.ThinLTOIndexFile, IndexFileSize);
+      DoThinLTOBackend = IndexFileSize > 0;
+    }
+    if (DoThinLTOBackend) {
       runThinLTOBackend(CGOpts, M, std::move(OS));
       return;
     }
+  }
 
   EmitAssemblyHelper AsmHelper(Diags, CGOpts, TOpts, LOpts, M);
 
   if (CGOpts.ExperimentalNewPassManager)
     AsmHelper.EmitAssemblyWithNewPassManager(Action, std::move(OS));
   else
     AsmHelper.EmitAssembly(Action, std::move(OS));
 

test/CodeGen/thinlto_backend.ll

 ; REQUIRES: x86-registered-target
 
 ; RUN: opt -module-summary -o %t1.o %s
 ; RUN: opt -module-summary -o %t2.o %S/Inputs/thinlto_backend.ll
 ; RUN: llvm-lto -thinlto -o %t %t1.o %t2.o
 
 ; Ensure clang -cc1 give expected error for incorrect input type
 ; RUN: not %clang_cc1 -O2 -o %t1.o -x c %s -c -fthinlto-index=%t.thinlto.bc 2>&1 | FileCheck %s -check-prefix=CHECK-WARNING
 ; CHECK-WARNING: error: invalid argument '-fthinlto-index={{.*}}' only allowed with '-x ir'
 
 ; Ensure we get expected error for missing index file
 ; RUN: %clang -O2 -o %t4.o -x ir %t1.o -c -fthinlto-index=bad.thinlto.bc 2>&1 | FileCheck %s -check-prefix=CHECK-ERROR1
 ; CHECK-ERROR1: Error loading index file 'bad.thinlto.bc'
 
+; Ensure we ignore empty index file under -ignore-empty-index-file
+; RUN: touch %t4.thinlto.bc
+; RUN: %clang -O2 -o %t4.o -x ir %t1.o -c -fthinlto-index=%t4.thinlto.bc -ignore-empty-index-file 2>&1 | FileCheck %s -check-prefix=CHECK-NOERROR
+; CHECK-NOERROR-NOT: Error loading index file 'bad.thinlto.bc'
+
 ; Ensure f2 was imported
 ; RUN: %clang -target x86_64-unknown-linux-gnu -O2 -o %t3.o -x ir %t1.o -c -fthinlto-index=%t.thinlto.bc
 ; RUN: llvm-nm %t3.o | FileCheck --check-prefix=CHECK-OBJ %s
 ; CHECK-OBJ: T f1
 ; CHECK-OBJ-NOT: U f2
 
 ; Ensure we get expected error for input files without summaries
 ; RUN: opt -o %t2.o %s