Introduce `llvm-min-tblgen` to build public header files

[chapuni]

Summary

llvm-min-tblgen is capable of building llvm/include/llvm;

• -gen-attrs
• -gen-directive-*
• -gen-intrinsics-*
• -gen-riscv-target-def
• -gen-vt

llvm-min-tblgen is built and used only when llvm-tblgen is built in-tree.
This is not installed.

llvm-tblgen is built with complete set and may be installed.
check-llvm uses not llvm-min-tblgen but llvm-tblgen.

LLVM_TABLEGEN_PROJECT overrides the definition of tablegen(project).
LLVM_HEADERS is used as the overridden prefix for LLVM header generators.

If EXPORT is not specified in add_tablegen, its tablegen is treated as internal.

Let llvm-tblgen depend on intrinsics_gen

Depends on D149072

[chapuni]

Changes in test files can be committed separately in advance.

I have just confirmed building. Let me know if there would be issues around installation and customization.

I am planning to move llvm-cg-tblgen to other place, possibly llvm/lib/CodeGen/TableGen.
See also D144351 and https://discourse.llvm.org/t/issues-in-llvm-tblgen-high-parallelized-build/68037/30

[mstorsjo]

llvm/utils/TableGen/CMakeLists.txt

69

I think there's a bit of conflict here, when you're calling add_tablegen twice with project set to LLVM.

This parameter is used for setting up the ${project}_TABLEGEN variable, which allows the user to specify a path to an existing such binary when e.g. cross compiling, instead of building a new one - see e.g. https://github.com/llvm/llvm-project/blob/llvmorg-16.0.0/llvm/cmake/modules/TableGen.cmake#L156-L169.

Currently, this can be set via LLVM_TABLEGEN, CLANG_TABLEGEN etc. I'm not sure if we should rename the project parameter there - I think it might work by passing LLVM_CG for that parameter for the new executable, making LLVM_CG_TABLEGEN.

For easier use, one can implicitly set LLVM_NATIVE_TOOL_DIR=<directory> these days, pointing at a directory containing all these *-tblgen binaries. But despite this, the individual setting variables kinda need to work too (and I'm afraid this current form, with clashing options, might cause trouble in build configs where that variable is used, e.g. when cross compiling LLVM).

[chapuni]

@mstorsjo How about it?

• llvm-minimal-tblgen -- Build only in the tree.
• llvm-tblgen includes codegen emitters. It may be exported. It may be used to check-llvm (my patches to test/TableGen would be unneeded)

[mstorsjo]

In D146352#4205131, @chapuni wrote:

@mstorsjo How about it?

• llvm-minimal-tblgen -- Build only in the tree.
• llvm-tblgen includes codegen emitters. It may be exported. It may be used to check-llvm (my patches to test/TableGen would be unneeded)

Hmm, unsure... It would hide the fact that there's two separate tools more, making it mostly a concern of the llvm build itself internally, but I wonder how much more complex it would make the cmake logic; it makes more reasoning about the tblgen executables less straightforward...

[chapuni]

• Adopt D144351. Changes to TableGen.cpp is not needed.

[chapuni]

• Use llvm-min-tblgen for llvm/include/llvm

[chapuni]

@mstorsjo I have introduced llvm-min-tblgen as the special tablegen for headers.
How about it? I think less intrusive.

[chapuni]

• Bazel

[chapuni]

I have confirmed this works fine for the simple bootstrap.
I expect this wouldn't prevent external llvm-tblgen.

[chapuni]

I knew this won't work with build_native_tool.
I supposed I could avoid building host's llvm-min-tblgen.

I will introduce like subproject in add_tablegen.

[chapuni]

• Revise

[chapuni]

• Restore LLVM_LINK_COMPONENTS

[chapuni]

• Use llvm-min-tblgen for SupportTests

[chapuni]

• bazel: s/LLVMTableGenGlobalISel/TableGenGlobalISel/

[chapuni]

• Rebase

[chapuni]

• Introduce D149072
• Use llvm-min-tblgen for SupportTests

[chapuni]

• Update comment line

[mstorsjo]

I think this looks quite neat overall, good stuff!

I measured the benefit from this patch in another build case; where I just want to build a small set of executables (for testing compiller-rt) in a new build tree; building tools like FileCheck is quite quick in general, but llvm-config has got surprisingly large numbers of dependencies. With this patch, the number of build steps to build llvm-config drops from 248 to 188, and the total cpu time for compiling it drops from ~11 minutes to ~5 minutes.

llvm/cmake/modules/TableGen.cmake

185 ↗

(On Diff #516432)

I guess these two lines could be consider a standalone NFC refactoring/cleanup on top of the current code as well?

[chapuni]

• Rebase
• Let llvm-tblgen depend on intrinsics_gen
• Cleanup llvm/modmap

[chapuni]

• [Bazel] Let llvm-tblgen depend on :intrinsic_enums_gen

[chapuni]

In D146352#4296422, @mstorsjo wrote:

I measured the benefit from this patch in another build case; where I just want to build a small set of executables (for testing compiller-rt) in a new build tree; building tools like FileCheck is quite quick in general, but llvm-config has got surprisingly large numbers of dependencies. With this patch, the number of build steps to build llvm-config drops from 248 to 188, and the total cpu time for compiling it drops from ~11 minutes to ~5 minutes.

I don't guess your background. Does it come from the bottleneck in the full version of llvm-tblgen?

llvm/cmake/modules/TableGen.cmake

185 ↗

(On Diff #516432)

It is leftover in my attempts.
Would it be better to rewind it? I prefer this.

[mstorsjo]

In D146352#4299284, @chapuni wrote:

In D146352#4296422, @mstorsjo wrote:

I measured the benefit from this patch in another build case; where I just want to build a small set of executables (for testing compiller-rt) in a new build tree; building tools like FileCheck is quite quick in general, but llvm-config has got surprisingly large numbers of dependencies. With this patch, the number of build steps to build llvm-config drops from 248 to 188, and the total cpu time for compiling it drops from ~11 minutes to ~5 minutes.

I don't guess your background. Does it come from the bottleneck in the full version of llvm-tblgen?

Yes; building llvm-config seems to require building some files that currently require the full llvm-tblgen, but after this patch can make do with llvm-min-tblgen.

llvm/cmake/modules/TableGen.cmake

185 ↗

(On Diff #516432)

I think the refactoring is good, but I think you could split it out from this patch and apply it directly (or send a separate patch which should be quick and easy to review+approve), and rebase this on top of it.

[chapuni]

• Restore ${project}-tablegen-host

[chapuni]

llvm/cmake/modules/TableGen.cmake

185 ↗

(On Diff #516432)

Okay, I've split it to D149343.

[chapuni]

• Add comment lines

[chapuni]

This is the last diff for the series of D148770.

@dblaikie Could you please take a look as a view of layering?

[dblaikie]

Don't know that I'm a great reviewer for this - I'm really bad at all the build file stuff (but thanks a bunch for having a go at updating the bazel build files - much appreciated). But, at a very rough glance it looks OK to me.

[mehdi_amini]

I just see this popping up in the recent hours:

[ 20%] Building RISCVTargetParserDef.inc...
llvm-min-tblgen: Unknown command line argument '-gen-riscv-target-def'.  Try: '../../../bin/llvm-min-tblgen --help'
llvm-min-tblgen: Did you mean '--help-list-hidden'?
make[3]: *** [include/llvm/TargetParser/CMakeFiles/RISCVTargetParserTableGen.dir/build.make:118: include/llvm/TargetParser/RISCVTargetParserDef.inc] Error 1
make[2]: *** [CMakeFiles/Makefile2:17904: include/llvm/TargetParser/CMakeFiles/RISCVTargetParserTableGen.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
[ 20%] Linking CXX static library ../../../../../lib/libMLIRTblgenLib.a

Seems related?

[chapuni]

In D146352#4311572, @mehdi_amini wrote:

I just see this popping up in the recent hours:

It was an issue in Makefile generator. Resolved in rG73fbe9c9291a.
Sorry for inconvenience.

Besides, I see a few still failing builders, for example
https://lab.llvm.org/buildbot/#/builders/77/builds/26531

I will revert this later if I didn't find any solutions.

[chapuni]

I have fixed external LLVM_TABLEGEN in rG95d4506dda79 as a quick fix. I will rewrite it later.

llvm/lib/Target/CMakeLists.txt

 list(APPEND LLVM_COMMON_DEPENDS intrinsics_gen)
 
 list(APPEND LLVM_TABLEGEN_FLAGS -I ${LLVM_MAIN_SRC_DIR}/lib/Target)
 
+set(LLVM_TABLEGEN_EXE llvm-cg-tblgen)
+
 add_llvm_component_library(LLVMTarget
   Target.cpp
   TargetIntrinsicInfo.cpp
   TargetLoweringObjectFile.cpp
   TargetMachine.cpp
   TargetMachineC.cpp
 
   ADDITIONAL_HEADER_DIRS

llvm/unittests/TableGen/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   TableGen
   Support
   )
 
 set(LLVM_TARGET_DEFINITIONS Automata.td)
 
+set(LLVM_TABLEGEN_EXE llvm-cg-tblgen)
+
 tablegen(LLVM AutomataTables.inc -gen-searchable-tables)
 tablegen(LLVM AutomataAutomata.inc -gen-automata)
 add_public_tablegen_target(AutomataTestTableGen)
 
 add_llvm_unittest(TableGenTests DISABLE_LLVM_LINK_LLVM_DYLIB
   AutomataTest.cpp
   CodeExpanderTest.cpp
   ParserEntryPointTest.cpp
   )
 
 target_link_libraries(TableGenTests PRIVATE LLVMTableGenGlobalISel LLVMTableGen)

llvm/utils/TableGen/CMakeLists.txt

 add_subdirectory(GlobalISel)
 
 set(LLVM_LINK_COMPONENTS Support)
 
-add_tablegen(llvm-tblgen LLVM
+add_llvm_library(TableGenCommon EXCLUDE_FROM_ALL
+  CodeGenIntrinsics.cpp
+  SDNodeProperties.cpp
+  TableGen.cpp
+  PARTIAL_SOURCES_INTENDED
+  )
+
+add_tablegen(llvm-cg-tblgen LLVM
   DESTINATION "${LLVM_TOOLS_INSTALL_DIR}"
   EXPORT LLVM
   AsmMatcherEmitter.cpp
   AsmWriterEmitter.cpp
   AsmWriterInst.cpp
-  Attributes.cpp
   CallingConvEmitter.cpp
   CodeEmitterGen.cpp
   CodeGenDAGPatterns.cpp
   CodeGenHwModes.cpp
   CodeGenInstAlias.cpp
   CodeGenInstruction.cpp
-  CodeGenIntrinsics.cpp
   CodeGenMapTable.cpp
   CodeGenRegisters.cpp
   CodeGenSchedule.cpp
   CodeGenTarget.cpp
   DAGISelEmitter.cpp
   DAGISelMatcherEmitter.cpp
   DAGISelMatcherGen.cpp
   DAGISelMatcherOpt.cpp
   DAGISelMatcher.cpp
   DecoderEmitter.cpp
   DFAEmitter.cpp
   DFAPacketizerEmitter.cpp
-  DirectiveEmitter.cpp
   DisassemblerEmitter.cpp
   DXILEmitter.cpp
   ExegesisEmitter.cpp
   FastISelEmitter.cpp
   GICombinerEmitter.cpp
   GlobalISelEmitter.cpp
   InfoByHwMode.cpp
   InstrInfoEmitter.cpp
   InstrDocsEmitter.cpp
-  IntrinsicEmitter.cpp
-  OptEmitter.cpp
-  OptParserEmitter.cpp
-  OptRSTEmitter.cpp
   PredicateExpander.cpp
   PseudoLoweringEmitter.cpp
   CompressInstEmitter.cpp
   RegisterBankEmitter.cpp
   RegisterInfoEmitter.cpp
-  RISCVTargetDefEmitter.cpp
-  SDNodeProperties.cpp
   SearchableTableEmitter.cpp
   SubtargetEmitter.cpp
   SubtargetFeatureInfo.cpp
-  TableGen.cpp
   Types.cpp
   VarLenCodeEmitterGen.cpp
   X86DisassemblerTables.cpp
   X86EVEX2VEXTablesEmitter.cpp
   X86FoldTablesEmitter.cpp
   X86MnemonicTables.cpp
   X86ModRMFilters.cpp
   X86RecognizableInstr.cpp
   WebAssemblyDisassemblerEmitter.cpp
+  PARTIAL_SOURCES_INTENDED
+  $<TARGET_OBJECTS:TableGenCommon>
+  )
+target_link_libraries(llvm-cg-tblgen PRIVATE LLVMTableGenGlobalISel)
+set_target_properties(llvm-cg-tblgen PROPERTIES FOLDER "Tablegenning")
+
+add_tablegen(llvm-tblgen LLVM
+  DESTINATION "${LLVM_TOOLS_INSTALL_DIR}"
+  EXPORT LLVM
+  Attributes.cpp
   CTagsEmitter.cpp
+  DirectiveEmitter.cpp
+  IntrinsicEmitter.cpp
+  OptEmitter.cpp
+  OptParserEmitter.cpp
+  OptRSTEmitter.cpp
+  RISCVTargetDefEmitter.cpp
+  $<TARGET_OBJECTS:TableGenCommon>
   )
-target_link_libraries(llvm-tblgen PRIVATE LLVMTableGenGlobalISel)
 set_target_properties(llvm-tblgen PROPERTIES FOLDER "Tablegenning")