[LTO] Added flag to generate assembly file with after LTO passes

[bunty2020]

Summary

save temps option currently doesn't generate assembly file with -flto flag.
To generate assembly file after LTO passes use -Wl,-plugin-opt=-save-temps

[tejohnson]

Thanks for the patch. Looks like there wasn't a way to dump assembly before codegen even before the restructuring of this out of gold-plugin.

Your patch does not follow LLVM coding conventions. Please clang-format it.

Please also add a test.

lib/LTO/LTOBackend.cpp

45

Just "lto-print-asm" (drop leading 'f' since this is an internal option)

46

s/in//

172

The -Wl,-plugin-opt is gold specific, and this LTO support is used by multiple clients (with more coming). Just remove the "-Wl,-plugin-opt=" (and update option name.

173

This should be a static function. But in any case I am suggesting a different approach below where this would go away.

178

Better to set this up in addSaveTemps (but keep optional) and use the same file naming used there (and remove the FLTOAsmFilename option). You could then do it via a new variant of setHook (e.g. setAsmHook) and create a new config hook like PrintAsmHook.

E.g. something like

auto setAsmHook = [&](ModuleHookFn &Hook) {

 // Keep track of the hook created earlier, which also needs to run.
 ModuleHookFn ExistingHook = Hook;
 Hook = [=](unsigned Task, const Module &M, const TargetMachine *TM) {
... create the filename as in setHook but with .s extension and invoke addPassesToEmitFile...

}
...
setAsmHook(PrintAsmHook);

Then you would replace where you are calling genAssemblyFile with an invocation of that new hook.

[mehdi_amini]

lib/LTO/LTOBackend.cpp

49

I rather not have any cl::opt controlling the behavior of the API.

178

I agree that this would be better, however usual hooks don't have access to the targetMachine, and more important, the CodeGen actually modifies the Module, which forces it to be cloned entirely before dumping the ASM.

[bunty2020]

I have modified save temps to generate assembly file too. But, I had to clone the module for assembly generation. I would like to know if there is a work around to generate assembly without cloning module.

[mehdi_amini]

In D24020#531164, @bunty2020 wrote:

I have modified save temps to generate assembly file too. But, I had to clone the module for assembly generation. I would like to know if there is a work around to generate assembly without cloning module.

I already mentioned it in my previous comments: the codegen is modifying the IR. So running it two times on the same module is likely to give different results (Depending on your use-case, it may or may not matter to you, but it can be surprising).

include/llvm/LTO/Config.h

150 ↗

(On Diff #69949)

Comment says "before", name is "after".

Also you shouldn't describe what a hook is actually doing, this is up to the client.

lib/LTO/LTOBackend.cpp

135

This will be "costly", we don't want this by default with "addSaveTemps".
And since it hasn't been needed till now, it is likely a "rare" need. Most of the time (i.e. always till now), using clang -O2 -disable-llvm-optzns output.lto.bc -S is enough to reproduce the codegen and produce an assembly file out of the save-temps bitcode if needed.

[tejohnson]

include/llvm/LTO/Config.h

150 ↗

(On Diff #69949)

Part of the issue is that there is already a PreCodeGenModuleHook. That can't be used here though because of the need for passing the TM. So either name it something specific to asm generation like PreCodeGenAsmHook, or specific to taking the TM like PreCodeGenTMHook. Mehdi - WDYT?

248 ↗

(On Diff #69949)

Remove the whitespace change.

lib/LTO/LTOBackend.cpp

103

The block below here that computes the name and opens the file can be refactored out into a named lambda or helper function.

135

Right, I had suggested keeping this optional in my original suggestion about moving this to a hook. I think it should go back under the option from the original patch.

230

Remove whitespace change

lib/LTO/LTOBackend.cpp

 
 #include "llvm/LTO/LTOBackend.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/Bitcode/ReaderWriter.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/LTO/LTO.h"
 #include "llvm/MC/SubtargetFeature.h"
+#include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/ThreadPool.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Transforms/IPO.h"
 #include "llvm/Transforms/IPO/PassManagerBuilder.h"
 #include "llvm/Transforms/Utils/FunctionImportUtils.h"
 #include "llvm/Transforms/Utils/SplitModule.h"
 
 using namespace llvm;
 using namespace lto;
 
+static cl::opt<std::string>
+FLTOAsmFilename("flto-asm-file",
+             cl::desc("Assembly file name to print assembly code after "
+                      "LTO phase. Works only with flto-print-asm enabled."),
+             cl::Hidden);
+
+static cl::opt<bool>
+FLTOPrintAsm("flto-print-asm",
+             cl::desc("Print assembly file in after LTO phase"),
+             cl::init(false), cl::Hidden);
+
+
 Error Config::addSaveTemps(std::string OutputFileName,
                            bool UseInputModulePath) {
   ShouldDiscardValueNames = false;
 
   std::error_code EC;
   ResolutionFile = llvm::make_unique<raw_fd_ostream>(
       OutputFileName + "resolution.txt", EC, sys::fs::OpenFlags::F_Text);
   if (EC)
Error Config::addSaveTemps(std::string OutputFileName,
   setHook("3.import", PostImportModuleHook);
   setHook("4.opt", PostOptModuleHook);
   setHook("5.precodegen", PreCodeGenModuleHook);
 
   CombinedIndexHook = [=](const ModuleSummaryIndex &Index) {
     std::string Path = OutputFileName + "index.bc";
     std::error_code EC;
     raw_fd_ostream OS(Path, EC, sys::fs::OpenFlags::F_None);
     if (EC) {
       // Because -save-temps is a debugging feature, we report the error
       // directly and exit.
       llvm::errs() << "failed to open " << Path << ": " << EC.message() << '\n';
       exit(1);
     }
     WriteIndexToFile(Index, OS);
     return true;
   };
return std::unique_ptr<TargetMachine>(TheTarget->createTargetMachine(
       C.CodeModel, C.CGOptLevel));
 }
 
 bool opt(Config &C, TargetMachine *TM, unsigned Task, Module &M,
          bool IsThinLto) {
   M.setDataLayout(TM->createDataLayout());
 
   legacy::PassManager passes;
   passes.add(createTargetTransformInfoWrapperPass(TM->getTargetIRAnalysis()));
 
   PassManagerBuilder PMB;
   PMB.LibraryInfo = new TargetLibraryInfoImpl(Triple(TM->getTargetTriple()));
   PMB.Inliner = createFunctionInliningPass();
   // Unconditionally verify input since it is not verified before this
   // point and has unknown origin.
   PMB.VerifyInput = true;
   PMB.VerifyOutput = !C.DisableVerify;
return [Task, &AddOutput](unsigned TaskId) {
     auto Output = AddOutput(Task);
     if (Output->isCachingEnabled() && Output->tryLoadFromCache(""))
       report_fatal_error("Cache hit without a valid key?");
     assert(Task == TaskId && "Unexpexted TaskId mismatch");
     return Output;
   };
 }
 
+// Generates assembly file with -Wl,-plugin-opt=-flto-print-asm
+void genAssemblyFile(TargetMachine *TM, Module &M){
+  int FD;
+  std::error_code EC;
+  SmallString<32> OutputFile;
+
+  if(FLTOAsmFilename == ""){
+    EC = sys::fs::createTemporaryFile("llvm-flto", "s", FD, OutputFile);
+    errs() << "Wrtiting to file : " << OutputFile << "\n";
+  }else{
+    EC = sys::fs::openFileForWrite(StringRef(FLTOAsmFilename),
+                                                 FD, sys::fs::F_None);
+  }
+
+  if (EC)
+    report_fatal_error( "Could not open file: %s", EC.message().c_str());
+
+  std::unique_ptr<raw_pwrite_stream>
+    AsmStream = llvm::make_unique<llvm::raw_fd_ostream>(FD, true);
+
+  legacy::PassManager CodeGenPasses;
+
+  if (TM->addPassesToEmitFile(CodeGenPasses, *AsmStream,
+                              TargetMachine::CGFT_AssemblyFile))
+    report_fatal_error("Failed to setup assembly gen in codegen");
+
+  CodeGenPasses.run(M);
+}
 void codegen(Config &C, TargetMachine *TM, AddOutputFn AddOutput, unsigned Task,
              Module &M) {
   if (C.PreCodeGenModuleHook && !C.PreCodeGenModuleHook(Task, M))
     return;
 
   auto Output = AddOutput(Task);
   std::unique_ptr<raw_pwrite_stream> OS = Output->getStream();
   legacy::PassManager CodeGenPasses;
+
+  // Generate assembly file
+  if(FLTOPrintAsm){
+    genAssemblyFile(TM, M);
+  }
+
   if (TM->addPassesToEmitFile(CodeGenPasses, *OS,
                               TargetMachine::CGFT_ObjectFile))
     report_fatal_error("Failed to setup codegen");
   CodeGenPasses.run(M);
 }
 
 void splitCodeGen(Config &C, TargetMachine *TM, AddOutputFn AddOutput,
                   unsigned ParallelCodeGenParallelismLevel,
else if (M.getTargetTriple().empty())
     M.setTargetTriple(C.DefaultTriple);
 
   std::string Msg;
   const Target *T = TargetRegistry::lookupTarget(M.getTargetTriple(), Msg);
   if (!T)
     return make_error<StringError>(Msg, inconvertibleErrorCode());
   return T;
 }
 
 }
 
 Error lto::backend(Config &C, AddOutputFn AddOutput,
                    unsigned ParallelCodeGenParallelismLevel,
                    std::unique_ptr<Module> M) {
   Expected<const Target *> TOrErr = initAndLookupTarget(C, *M);
   if (!TOrErr)
     return TOrErr.takeError();