[PoC][RISCV] enable LTO/ThinLTO on RISCV

[khchen]

Summary

This patch enable LTO/ThinLTO on RISCV based on TargetMachine Hook in D72624.

[MaskRay]

llvm/test/LTO/RISCV/mabi.ll

6

llvm-lto is for the old LTO API and llvm-lto2 is for the new API. The file does not test llvm-lto2 for full LTO.

I am not clear about how tests are organized in this directory. Hope @tejohnson can clarify... and point out the ideal directory structure..

[tejohnson]

llvm/lib/LTO/LTO.cpp

1029 ↗

(On Diff #257010)

We definitely don't want to be doing this. We by design do not parse IR of ThinLTO modules during the LTO link, only in the ThinLTO backends. This is what helps keep ThinLTO fast. I'm not sure what the value of reading it in only one of potentially many ThinLTO modules is anyway? If you want to detect mismatches between ThinLTO modules, it will have to either be done naturally in the ThinLTO backends when we (selectively) merge IR there (i.e. caught by the IRMover). Otherwise, if you want to do it during the thin link, then you will need to record this information in the ThinLTO summary. See my comment here (the two big paragraphs towards the end of my comment): https://reviews.llvm.org/D72624#1820580

llvm/test/LTO/RISCV/mabi.ll

6

As @MaskRay notes, llvm-lto tests the "old" LTO API, e.g. ThinLTOCodeGenerator.cpp and LTOCodeGenerator.cpp. The code in LTO.cpp for example is the "new" linker-resolution based LTO, which is tested by llvm-lto2.

The Resolution subdirectory of llvm/test/LTO was originally meant for testing the "new" resolution based LTO. But we have subsequently added a lot of new LTO based testing via llvm-lto2 in the other directories, for things that we might want to test for both APIs (as they are both still in active use by different linkers).

This is a good place to add this test, since this is based on the support in D72624 which adds the hook to both the old and new LTO APIs. Ideally you should test both regular and ThinLTO with both llvm-lto and llvm-lto2 to get all combinations.

[khchen]

llvm/lib/LTO/LTO.cpp

1029 ↗

(On Diff #257010)

@tejohnson Thanks, it has been very helpful! I would try summary info approach to avoid to parse IR.

Different target-abi flag generates different elf objects and they can not linked together, but runRegularLTO generates empty object from CombinedModule (empty module) without any target-abi module flag, it means in some case this CombinedModule object can not link with other input module.
This is why I copy module flag from input to CombinedModule.
In LTO, the only legal way is all input modules have same target-abi flag, so CombinedModule does not care where is the target-abi flag value come from.

llvm/test/LTO/RISCV/mabi.ll

6

@MaskRay Thanks for your reminder.

@tejohnson

Ideally you should test both regular and ThinLTO with both llvm-lto and llvm-lto2 to get all combinations.

Do you mean if I claim RISCV supports thinLTO, ideally I need to add more testcases like X86 in test/LTO and test/ThinLTO/X86?
I saw there are just few testcase in test/LTO/ARM, for example lto-linking-metadata.ll" also tests llvm-lto and llvm-lto2, so I did the similar thing in RISCV...

[tejohnson]

llvm/lib/LTO/LTO.cpp

1029 ↗

(On Diff #257010)

In the ThinLTO case, as you note the CombinedModule is empty. So why does the target-abi matter? Does the linker complain? If so, we should probably just suppress emission of this empty object file. I think it is always emitted because that helps in some build systems, like ours, which want to know a priori all possible output files, but perhaps we could put this under an option. But I'd like to understand whether this is resulting in a link failure first.

llvm/test/LTO/RISCV/mabi.ll

6

Not necessarily now, but I would imagine if RISCV ThinLTO is supported, then eventually more RISCV-specific test cases will be added. For now something simple is probably ok, to test the basic cases. You can add the llvm-lto and llvm-lto2 testing into the same .ll test - that's what I meant. Test all 4 combinations (new/old LTO api and regular/thin LTO), rather than just the two you have now (regular+old, thin+new).

[khchen]

llvm/lib/LTO/LTO.cpp

1029 ↗

(On Diff #257010)

Yes, linker emits an error, this situation is similar to linking ARM's softfp object with hardfp object, but RISCV have more combinations and flexibility with abi in the same platform.

ps. In other targets (ex. ARM, MIPS), the abi (target-abi) info for a file is derived from target triple, -mcpu or -mabi, but in RISC-V, abi is only derived from -mabi or -mattr (and both are encoded in IR now). It's why ARM doesn't have this issue.
BTW, GCC LTO also encodes the abi info into the ELF.

llvm/test/LTO/RISCV/mabi.ll

6

Got it, thanks.

[khchen]

I plan to suppress to emit empty module or avoid ld to link empty module's object file.

llvm/lib/LTO/LTO.cpp

1029 ↗

(On Diff #257010)

Sorry, I didn't make it clear enough. This linking error only happen in gold ld (I had not tested lld yet) due to the file list for linking includes empty CombinedModule object.

ps. CombinedModule is not always empty in mixed-mode LTO (ex. gold/X86/mixed_lto.ll). It means my original patch has bug. :S

[tejohnson]

In D78035#1991591, @khchen wrote:

I plan to suppress to emit empty module or avoid ld to link empty module's object file.

This will need to be dependent on linker options. Our build system relies on always getting this object file when the -Wl,-plugin-opt,obj-path=${file} (gold-plugin.cpp). or -Wl,-lto-obj-path=${file} (lld) options are passed. Presumably you could add a boolean flag to the lto::Config to communicate whether this option was specified and use that to control creation of the output even when it is an empty module.

If you create a new patch for this work please make sure I am a reviewer so I can make sure we aren't broken.

llvm/lib/LTO/LTO.cpp

1029 ↗

(On Diff #257010)

This linking error only happen in gold ld (I had not tested lld yet) due to the file list for linking includes empty CombinedModule object.

Interesting. The other option then is to figure out how lld is ignoring this empty file and add similar logic into gold-plugin.cpp. Basically you would want to avoid the call to add_input_file for any empty objects (this is the callback into gold to add the object files to be linked). At the point where we call that we just have the file name and a boolean indicating whether it should be cleaned up or not after the link. You would need to do some kind of restructuring or possibly make the Files array a tuple with another bool flag indicating whether the obj file buffer was empty.

ps. CombinedModule is not always empty in mixed-mode LTO (ex. gold/X86/mixed_lto.ll). It means my original patch has bug. :S

Right, in that case it gets the ABI flag naturally from the regular LTO modules. It's not even just mixed lto mode, we can have split hybrid LTO modules in some cases for CFI and WPD (one regular LTO module and one ThinLTO module within one .o bitcode file), where you will end up with a non-empty combined module when building with ThinLTO.

[khchen]

rebase on D78988

[tejohnson]

llvm/test/LTO/RISCV/mabi-invalid.ll

11

Here's another case where the regular LTO testing is using the old LTO API (via llvm-lto) and ThinLTO is using the new LTO API (via llvm-lto2). As noted earlier, all 4 combinations should be tested.

llvm/test/tools/gold/RISCV/mixed_lto.ll

20

There's a typo in the input name "miaxed_lto.ll". If the test worked despite that then it probably needs some improvement.

[khchen]

update testcases

[khchen]

I'm not working on RISC-V now and please reference https://reviews.llvm.org/D132843#3770454 to see the follow-up work.

clang/lib/Driver/ToolChains/RISCVToolchain.h

 public:
   RISCVToolChain(const Driver &D, const llvm::Triple &Triple,
                  const llvm::opt::ArgList &Args);
 
   bool IsIntegratedAssemblerDefault() const override { return true; }
   void addClangTargetOptions(const llvm::opt::ArgList &DriverArgs,
                              llvm::opt::ArgStringList &CC1Args,
                              Action::OffloadKind) const override;
+  bool HasNativeLLVMSupport() const override { return true; }
   RuntimeLibType GetDefaultRuntimeLibType() const override;
   UnwindLibType
   GetUnwindLibType(const llvm::opt::ArgList &Args) const override;
   void
   AddClangSystemIncludeArgs(const llvm::opt::ArgList &DriverArgs,
                             llvm::opt::ArgStringList &CC1Args) const override;
   void
   addLibStdCxxIncludePaths(const llvm::opt::ArgList &DriverArgs,

clang/lib/Driver/ToolChains/RISCVToolchain.cpp

void RISCV::Linker::ConstructJob(Compilation &C, const JobAction &JA,
   if (IsRV64) {
     CmdArgs.push_back("elf64lriscv");
   } else {
     CmdArgs.push_back("elf32lriscv");
   }
 
   std::string Linker = getToolChain().GetLinkerPath();
 
+  if (D.isUsingLTO()) {
+    assert(!Inputs.empty() && "Must have at least one input.");
+    addLTOOptions(ToolChain, Args, CmdArgs, Output, Inputs[0],
+                  D.getLTOMode() == LTOK_Thin);
+  }
+
   bool WantCRTs =
       !Args.hasArg(options::OPT_nostdlib, options::OPT_nostartfiles);
 
   const char *crtbegin, *crtend;
   auto RuntimeLib = ToolChain.GetRuntimeLibType(Args);
   if (RuntimeLib == ToolChain::RLT_Libgcc) {
     crtbegin = "crtbegin.o";
     crtend = "crtend.o";

llvm/lib/Target/RISCV/RISCVTargetMachine.h

public:
   const RISCVSubtarget *getSubtargetImpl() const = delete;
 
   TargetPassConfig *createPassConfig(PassManagerBase &PM) override;
 
   TargetLoweringObjectFile *getObjFileLowering() const override {
     return TLOF.get();
   }
 
+  void setTargetOptionsWithModuleMetadata(
+      const Module &M LLVM_ATTRIBUTE_UNUSED) const override;
+
   TargetTransformInfo getTargetTransformInfo(const Function &F) override;
 };
 }
 
 #endif

llvm/lib/Target/RISCV/RISCVTargetMachine.cpp

if (const MDString *ModuleTargetABI = dyn_cast_or_null<MDString>(
       }
       ABIName = ModuleTargetABI->getString();
     }
     I = std::make_unique<RISCVSubtarget>(TargetTriple, CPU, FS, ABIName, *this);
   }
   return I.get();
 }
 
+void RISCVTargetMachine::setTargetOptionsWithModuleMetadata(
+    const Module &M LLVM_ATTRIBUTE_UNUSED) const {
+  StringRef ABIName = Options.MCOptions.getABIName();
+  if (const MDString *ModuleTargetABI =
+          dyn_cast_or_null<MDString>(M.getModuleFlag("target-abi"))) {
+    StringRef ModuleABIName = ModuleTargetABI->getString();
+    if (!ABIName.empty() && ModuleABIName != ABIName)
+      report_fatal_error("-target-abi option != target-abi module flag");
+    if (ABIName.empty())
+      Options.MCOptions.ABIName = ModuleABIName.str();
+  }
+}
+
 TargetTransformInfo
 RISCVTargetMachine::getTargetTransformInfo(const Function &F) {
   return TargetTransformInfo(RISCVTTIImpl(this, F));
 }
 
 namespace {
 class RISCVPassConfig : public TargetPassConfig {
 public:

llvm/test/CodeGen/RISCV/module-target-abi2.ll

 ; RUN: llc -mtriple=riscv32 < %s 2>&1 \
 ; RUN:   | FileCheck -check-prefix=DEFAULT %s
 ; RUN: not --crash llc -mtriple=riscv32 -target-abi ilp32 < %s 2>&1 \
 ; RUN:   | FileCheck -check-prefix=RV32IF-ILP32 %s
 ; RUN: llc -mtriple=riscv32 -target-abi ilp32f < %s 2>&1 \
 ; RUN:   | FileCheck -check-prefix=RV32IF-ILP32F %s
 ; RUN: llc -mtriple=riscv32 -filetype=obj < %s | llvm-readelf -h - | FileCheck -check-prefixes=FLAGS %s
 
 ; RV32IF-ILP32: -target-abi option != target-abi module flag
 
-; FLAGS: Flags: 0x0
-; // this should be "Flags :0x2, single-float ABI", it will be fixed later.
+; FLAGS: Flags: 0x2, single-float ABI
 
 define float @foo(i32 %a) nounwind #0 {
 ; DEFAULT: # %bb.0:
 ; DEFAULT-NEXT: fcvt.s.w fa0, a0
 ; DEFAULT-NEXT: ret
 ; RV32IF-ILP32F: # %bb.0:
 ; RV32IF-ILP32F: fcvt.s.w fa0, a0
 ; RV32IF-ILP32F: ret
   %conv = sitofp i32 %a to float
   ret float %conv
 }
 
 attributes #0 = { "target-features"="+f"}
 !llvm.module.flags = !{!0}
 !0 = !{i32 1, !"target-abi", !"ilp32f"}

llvm/test/LTO/RISCV/Inputs/mabi-invalid.ll

+target datalayout = "e-m:e-p:32:32-i64:64-n32-S128"
+target triple = "riscv32-unknown-linux-gnu"
+
+define float @foo(float %x) #0 {
+  %conv = fpext float %x to double
+  %add = fadd double %conv, 0x400921FD80C9BEFB
+  %conv1 = fptrunc double %add to float
+  ret float %conv1
+}
+
+attributes #0 = { nounwind "target-features"="+a,+c,+f,+m,+relax" }
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"target-abi", !"ilp32"}

llvm/test/LTO/RISCV/Inputs/mabi.ll

+target datalayout = "e-m:e-p:32:32-i64:64-n32-S128"
+target triple = "riscv32-unknown-linux-gnu"
+
+define float @foo(float %x) #0 {
+  %conv = fpext float %x to double
+  %add = fadd double %conv, 0x400921FD80C9BEFB
+  %conv1 = fptrunc double %add to float
+  ret float %conv1
+}
+
+attributes #0 = { nounwind "target-features"="+a,+c,+f,+m,+relax" }
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"target-abi", !"ilp32f"}

llvm/test/LTO/RISCV/lit.local.cfg

+if not 'RISCV' in config.root.targets:
+  config.unsupported = True

llvm/test/LTO/RISCV/mabi-invalid.ll

+; Check with regular LTO
+; RUN: rm -f %t*
+; RUN: llvm-as < %s >%t1
+; RUN: llvm-as < %p/Inputs/mabi-invalid.ll >%t2
+; Check old API
+; RUN: not llvm-lto -exported-symbol=main  -o %t3 %t1 %t2 2>&1 | FileCheck %s
+; Check new API
+; RUN: not llvm-lto2 run -r %t1,foo, -r %t1,main,plx -r %t2,foo,plx -o %t3.o %t1 %t2 2>&1 | FileCheck %s
+
+; Check with ThinLTO.
+; RUN: rm -f %t*
+; RUN: opt -module-summary -o %t1 %s
+; RUN: opt -module-summary -o %t2 %p/Inputs/mabi-invalid.ll
+; Check old API
+; RUN: not --crash llvm-lto -thinlto -thinlto-action=run %t1 %t2 -exported-symbol=main 2>&1 | FileCheck %s
+; Check new API
+; RUN: not --crash llvm-lto2 run -r %t1,foo, -r %t1,main,plx -r %t2,foo,plx -o %t3.o %t1 %t2 2>&1 | FileCheck %s
+
+; CHECK: 'target-abi': IDs have conflicting values
+
+target datalayout = "e-m:e-p:32:32-i64:64-n32-S128"
+target triple = "riscv32-unknown-linux-gnu"
+
+declare float @foo(float) #1
+
+define float @main(float %x) #0 {
+  %retval = call float @foo(float 10.0)
+  ret float %retval
+}
+
+attributes #0 = { nounwind "target-features"="+a,+c,+f,+m,+relax" }
+attributes #1 = { nounwind "target-features"="+a,+c,+f,+m,+relax" }
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"target-abi", !"ilp32f"}

llvm/test/LTO/RISCV/mabi.ll

+; Test target-abi module flag generate correct elf flags.
+
+; Check with regular LTO
+; RUN: rm -f %t*
+; RUN: llvm-as < %s >%t1
+; RUN: llvm-as < %p/Inputs/mabi.ll >%t2
+; Check old API
+; RUN: llvm-lto -exported-symbol=main  -o %t3 %t1 %t2
+; RUN: llvm-readelf -h %t3 | FileCheck %s
+; Check new API
+; RUN: llvm-lto2 run -r %t1,foo, -r %t1,main,plx -r %t2,foo,plx -o %t3.o %t1 %t2
+; RUN: llvm-readelf -h %t3.o.0 | FileCheck %s
+
+; Check with ThinLTO.
+; RUN: rm -f %t*
+; RUN: opt -module-summary -o %t1 %s
+; RUN: opt -module-summary -o %t2 %p/Inputs/mabi.ll
+; Check old API
+; RUN: llvm-lto -thinlto -thinlto-action=run %t1 %t2 -exported-symbol=main
+; RUN: llvm-readelf -h %t1.thinlto.o | FileCheck %s
+; RUN: llvm-readelf -h %t2.thinlto.o | FileCheck %s
+; Check new API
+; RUN: llvm-lto2 run -r %t1,foo, -r %t1,main,plx -r %t2,foo,plx -o %t3.o %t1 %t2
+; RUN: llvm-readelf -h %t3.o.1 | FileCheck %s
+; RUN: llvm-readelf -h %t3.o.2 | FileCheck %s
+
+; CHECK: Flags: 0x2, single-float ABI
+
+target datalayout = "e-m:e-p:32:32-i64:64-n32-S128"
+target triple = "riscv32-unknown-linux-gnu"
+
+declare float @foo(float) #1
+
+define float @main(float %x) #0 {
+  %retval = call float @foo(float 10.0)
+  ret float %retval
+}
+
+attributes #0 = { nounwind "target-features"="+a,+c,+f,+m,+relax" }
+attributes #1 = { nounwind "target-features"="+a,+c,+f,+m,+relax" }
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"target-abi", !"ilp32f"}

llvm/test/lit.cfg.py

def have_ld_plugin_support():
     emu_line = [l for l in ld_out.split('\n') if 'supported emulations' in l]
     if len(emu_line) != 1:
         return False
     emu_line = emu_line[0]
     fields = emu_line.split(':')
     if len(fields) != 3:
         return False
     emulations = fields[2].split()
+    # default RISC-V ld supports linker plugin
+    if 'elf32lriscv' in emulations:
+        config.available_features.add('ld_emu_elf32lriscv')
+        return True
     if 'elf_x86_64' not in emulations:
         return False
     if 'elf32ppc' in emulations:
         config.available_features.add('ld_emu_elf32ppc')
 
     ld_version = subprocess.Popen(
         [config.gold_executable, '--version'], stdout=subprocess.PIPE, env={'LANG': 'C'})
     if not 'GNU gold' in ld_version.stdout.read().decode():

llvm/test/tools/gold/RISCV/Inputs/mixed_lto.ll

+target datalayout = "e-m:e-p:32:32-i64:64-n32-S128"
+target triple = "riscv32-unknown-linux-gnu"
+
+declare float @g() #1
+define i32 @main() #0 {
+  call float @g()
+  ret i32 0
+}
+
+attributes #0 = { nounwind "target-features"="+a,+c,+f,+m,+relax" }
+attributes #1 = { nounwind "target-features"="+a,+c,+f,+m,+relax" }
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"target-abi", !"ilp32f"}

llvm/test/tools/gold/RISCV/lit.local.cfg

+if (not 'ld_plugin' in  config.available_features):
+   config.unsupported = True

llvm/test/tools/gold/RISCV/mixed_lto.ll

+; REQUIRES: ld_emu_elf32lriscv
+
+; RUN: rm -f %t*.o
+;; Test mixed-mode LTO (mix of regular and thin LTO objects)
+; RUN: opt %s -o %t.o
+; RUN: opt -module-summary %p/Inputs/mixed_lto.ll -o %t2.o
+
+; RUN: %gold -m elf32lriscv -plugin %llvmshlibdir/LLVMgold%shlibext \
+; RUN:     --plugin-opt=thinlto \
+; RUN:     -shared --plugin-opt=-import-instr-limit=0 \
+; RUN:     -o %t3.o %t2.o %t.o
+; RUN: llvm-nm %t3.o | FileCheck %s
+;; ChecK target ABI info
+; RUN: llvm-readelf -h %t3.o | FileCheck %s --check-prefix=CHECK-ABI
+
+; RUN: opt %s -module-summary -o %thin0.o
+; RUN: opt %p/Inputs/mixed_lto.ll -module-summary -o %thin1.o
+; RUN: %gold -m elf32lriscv -plugin %llvmshlibdir/LLVMgold%shlibext \
+; RUN:     --plugin-opt=thinlto \
+; RUN:     -shared --plugin-opt=-import-instr-limit=0 \
+; RUN:     -o %thin2.o %thin0.o %thin1.o
+; RUN: llvm-nm %t3.o | FileCheck %s
+;; ChecK target ABI info
+; RUN: llvm-readelf -h %t3.o | FileCheck %s --check-prefix=CHECK-ABI
+; RUN: llvm-readelf -h %thin2.o | FileCheck %s --check-prefix=CHECK-ABI
+
+;; Test ThinLTO
+; CHECK-DAG: T main
+; CHECK-DAG: T g
+
+; CHECK-ABI: Flags: 0x2, single-float ABI
+
+
+target datalayout = "e-m:e-p:32:32-i64:64-n32-S128"
+target triple = "riscv32-unknown-linux-gnu"
+define i32 @g() #0 {
+  ret i32 0
+}
+
+attributes #0 = { nounwind "target-features"="+a,+c,+f,+m,+relax" }
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"target-abi", !"ilp32f"}