[llvm-exegesis] Cross compile all enabled targets

[reames]

Summary

llvm-exegesis is rather odd in the LLVM ecosystem in code is selectively compiled based on the native machine. LLVM is cross compiler by default, so this stands out as odd. It's also less then helpful when working on code for a target other than your native dev environment.

This change only changes the build setup. A later change will enable -march support to allow actual benchmarking under e.g. simulators in a cross compilation environment.

This is mostly cmake foo and my cmake knowledge is pretty much non-existent. Please don't hesitate to suggest cleaner ways of doing something.

[courbet]

I'm not really qualified to comment on the CMake but this sounds conceptually fine apart from my comment on initialization.

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

301–304

I think building all targets makes sens so that you can run an llvm-exegesis compiled on (say) X86 on AArch64. However at runtime we should only need to initialize the native target as we're going to jit compile and execute only for the host anyway.

[RKSimon]

Does this affect which tests we can run for each target? For x86 at least we split the jit/runtime tests off into subfolders from the analysis-only tests so we could at least test analysis on builds that didn't have PFM.

[reames]

In D133150#3766354, @RKSimon wrote:

Does this affect which tests we can run for each target? For x86 at least we split the jit/runtime tests off into subfolders from the analysis-only tests so we could at least test analysis on builds that didn't have PFM.

At the moment, no, it should not. It does run the target initialization code for all targets now, but that shouldn't be a visible change in behavior.

Thank you for pointing out the analysis tests being separated for x86, I hadn't noticed that. Getting those enabled on all targets is likely a reasonable initial goal for the next change or two.

My eventual plan is to enable testing of the code gen in a target independent manner, and have only the benchmark execution be restricted to native target. (Well, even then I want to support cross builds, but step by step.)

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

301–304

My eventual goal for this is to be able to exercise the jitting part in a cross compilation environment. So, initializing all targets will be necessary.

For this particular patch, I need the All variants to check linkage. Without them, you can skip compiling a target specific library and not notice. If you really want, I can remove them and do just the compile step in this patch, but I don't really see why.

[courbet]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

301–304

I see, SG.

[RKSimon]

LGTM as a first step - any other comments?

[courbet]

LGTM as far as I'm concerned.

[reames]

After landing this, I got a couple dozen build bot failure notices. A large fraction of them are warnings in unrelated files - I'm guessing the cmake change confused incremental bots - but at least one looks potentially relevant. Specifically, a polly bot (https://lab.llvm.org/buildbot#builders/56/builds/3201) is failing to link in the ARM target. I don't have a good theory on what might be happening, and can't (yet) reproduce.

[efriedma]

Maybe missing AllTargetsDisassemblers? But not sure why that would only show up on that specific buildbot.

[reames]

In D133150#3780865, @efriedma wrote:

Maybe missing AllTargetsDisassemblers? But not sure why that would only show up on that specific buildbot.

That's my guess too. I'm going to land it as a speculative fix and see what the bots do.

It looks like disassembly is only initialized on one path, and that's still using the native calls. So if we didn't have the disassembler linked in, we'd see an error like this. No idea why we apparently getting a different link set in this case..

[thakis]

FWIW, lli also uses this pattern I think.

[reames]

In D133150#3781066, @thakis wrote:

FWIW, lli also uses this pattern I think.

I don't follow. LLI doesn't appear to have any target dependent code.

[thakis]

In D133150#3781166, @reames wrote:

In D133150#3781066, @thakis wrote:

FWIW, lli also uses this pattern I think.

I don't follow. LLI doesn't appear to have any target dependent code.

I mean that it also depends on native.

Unrelatedly, the "native" target in llvm-exegesis gets initialized due to the LLVM_EXEGESIS_INITIALIZE_NATIVE_TARGET=Initialize${LLVM_EXEGESIS_NATIVE_ARCH}ExegesisTarget in llvm/tools/llvm-exegesis/CMakeLists.txt. How do the non-native targets get initialized?

[reames]

In D133150#3784677, @thakis wrote:

In D133150#3781166, @reames wrote:

In D133150#3781066, @thakis wrote:

FWIW, lli also uses this pattern I think.

I don't follow. LLI doesn't appear to have any target dependent code.

I mean that it also depends on native.

Unrelatedly, the "native" target in llvm-exegesis gets initialized due to the LLVM_EXEGESIS_INITIALIZE_NATIVE_TARGET=Initialize${LLVM_EXEGESIS_NATIVE_ARCH}ExegesisTarget in llvm/tools/llvm-exegesis/CMakeLists.txt. How do the non-native targets get initialized?

In tree? They don't. See D133605

llvm/tools/llvm-exegesis/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
+  AllTargetsAsmParsers
+  AllTargetsCodeGens
+  AllTargetsDescs
+  AllTargetsInfos
   MC
   MCParser
   Support
-  native
   )
 
 add_llvm_tool(llvm-exegesis
   DISABLE_LLVM_LINK_LLVM_DYLIB
   llvm-exegesis.cpp
 
   DEPENDS
   intrinsics_gen
   )
 
+# Has side effect of defining LLVM_EXEGESIS_TARGETS
 add_subdirectory(lib)
 
-# Link the native exegesis target if compiled and on the right host.
-if ((LLVM_TARGETS_TO_BUILD MATCHES "${LLVM_NATIVE_ARCH}") AND (LLVM_EXEGESIS_TARGETS MATCHES "${LLVM_NATIVE_ARCH}"))
+# Register the native target (we don't yet support -march)
+if (LLVM_EXEGESIS_TARGETS MATCHES "${LLVM_NATIVE_ARCH}")
   set(LLVM_EXEGESIS_NATIVE_ARCH "${LLVM_NATIVE_ARCH}")
-endif()
-
-if (LLVM_EXEGESIS_NATIVE_ARCH)
-  set(LLVM_EXEGESIS_NATIVE_TARGET "LLVMExegesis${LLVM_EXEGESIS_NATIVE_ARCH}")
   set_source_files_properties(llvm-exegesis.cpp PROPERTIES COMPILE_FLAGS "-DLLVM_EXEGESIS_INITIALIZE_NATIVE_TARGET=Initialize${LLVM_EXEGESIS_NATIVE_ARCH}ExegesisTarget")
 endif()
 
+# Link the native exegesis targets
+set(libs)
+foreach(t ${LLVM_EXEGESIS_TARGETS})
+  string(STRIP ${t} t)
+  list(APPEND libs "LLVMExegesis${t}")
+endforeach()
+
 target_link_libraries(llvm-exegesis PRIVATE
   LLVMExegesis
-  ${LLVM_EXEGESIS_NATIVE_TARGET}
+  ${libs}
   )

llvm/tools/llvm-exegesis/lib/CMakeLists.txt

-set(TARGETS_TO_APPEND "")
 
+set(LLVM_EXEGESIS_TARGETS)
 if (LLVM_TARGETS_TO_BUILD MATCHES "X86")
-  add_subdirectory(X86)
-  set(TARGETS_TO_APPEND "${TARGETS_TO_APPEND} X86")
+  list(APPEND LLVM_EXEGESIS_TARGETS "X86")
 endif()
 if (LLVM_TARGETS_TO_BUILD MATCHES "AArch64")
-  add_subdirectory(AArch64)
-  set(TARGETS_TO_APPEND "${TARGETS_TO_APPEND} AArch64")
+  list(APPEND LLVM_EXEGESIS_TARGETS "AArch64")
 endif()
 if (LLVM_TARGETS_TO_BUILD MATCHES "PowerPC")
-  add_subdirectory(PowerPC)
-  set(TARGETS_TO_APPEND "${TARGETS_TO_APPEND} PowerPC")
+  list(APPEND LLVM_EXEGESIS_TARGETS "PowerPC")
 endif()
 if (LLVM_TARGETS_TO_BUILD MATCHES "Mips")
-  add_subdirectory(Mips)
-  set(TARGETS_TO_APPEND "${TARGETS_TO_APPEND} Mips")
+  list(APPEND LLVM_EXEGESIS_TARGETS "Mips")
 endif()
 
-set(LLVM_EXEGESIS_TARGETS "${LLVM_EXEGESIS_TARGETS} ${TARGETS_TO_APPEND}" PARENT_SCOPE)
+set(LLVM_EXEGESIS_TARGETS ${LLVM_EXEGESIS_TARGETS} PARENT_SCOPE)
+
+foreach(t ${LLVM_EXEGESIS_TARGETS})
+  add_subdirectory(${t})
+endforeach()
 
 set(LLVM_LINK_COMPONENTS
   Analysis
   CodeGen
   Core
   ExecutionEngine
   GlobalISel
   MC

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

 void benchmarkMain() {
 #ifndef HAVE_LIBPFM
   ExitWithError("benchmarking unavailable, LLVM was built without libpfm.");
 #endif
 
   if (exegesis::pfm::pfmInitialize())
     ExitWithError("cannot initialize libpfm");
 
-  InitializeNativeTarget();
-  InitializeNativeTargetAsmPrinter();
-  InitializeNativeTargetAsmParser();
+  InitializeAllTargets();
+  InitializeAllTargetMCs();
+  InitializeAllAsmPrinters();
+  InitializeAllAsmParsers();
   InitializeNativeExegesisTarget();
 
   const LLVMState State(CpuName);
 
   // Preliminary check to ensure features needed for requested
   // benchmark mode are present on target CPU and/or OS.
   ExitOnErr(State.getExegesisTarget().checkFeatureSupport());