Out-of-tree machine IR passes

[aguinet]

Summary

This patch add the possibilities to create out-of-tree MIR passes. This is based on how IR out-of-tree passes work.
If this is something that could be upstreamed, I could factorize the code between the two system, as they are very similar!

[aguinet]

As a side remark, I have tested this with an out of tree MIR pass that dump MIR, and it works well. Should I add this test within the test suite? I didn't find such test for the current out-of-tree IR system (but chances are I missed it!).

[serge-sans-paille]

Could you add some documentation, probably in docs/MIRLangRef.rst? Also you lack a test case, probably in ./CodeGen/MIR/Generic/

[aguinet]

Thanks Serge for the comments !

I added examples, a test that test these examples, and a mention of them in the documentation (not sure it's in the right place though!).

[aguinet]

Cleaning up some commented CMake code...

[lkail]

examples/OutOfTreeMIR/ModulePass/ModulePass.cpp

1 ↗

(On Diff #230380)

Please add copyright information.

[aguinet]

Copyright added on new cpp files.

[serge-sans-paille]

examples/CMakeLists.txt

10 ↗

(On Diff #230383)

Why don't you always build it?

examples/OutOfTreeMIR/FunctionPass/CMakeLists.txt

1 ↗

(On Diff #230383)

based on ../lib/Transforms/Hello/CMakeLists.txt, you could pass BUILDTREE_ONLY as an extra flag

lib/CodeGen/TargetPassConfig.cpp

229 ↗

(On Diff #230383)

This code is very similar to the legacy pass manager machinery, which is very different from the new PM one.

Adding the feature here totally makes sense to me, but I'm unsure if the MIR passes management is going to be reworked just like IR passes was, @chandlerc ?

[aguinet]

examples/CMakeLists.txt

10 ↗

(On Diff #230383)

Because I test them as out-of-tree passes. My understanding is that, if LLVM isn't built with shared libraries, or with "LINK_LLVM_DYLIB", I won't be able to build a shared library for the passes that llc would load.

[aguinet]

examples/OutOfTreeMIR/FunctionPass/CMakeLists.txt

1 ↗

(On Diff #230383)

Indeed. Will add it in the next patch!

lib/CodeGen/TargetPassConfig.cpp

229 ↗

(On Diff #230383)

That's a good question :)

[aguinet]

Use BUILDTREE_ONLY, as suggested by @serge-sans-paille .

[aguinet]

lib/CodeGen/TargetPassConfig.cpp

229 ↗

(On Diff #230383)

@chandlerc any insights of what will happen with the MIR passes management? Does this patch conflict with any current refactoring on the subject?

Thanks!

[aguinet]

• Update to monorepo and master
• Add forgotten RegisterMIRPasses.h
• Compile and test even in static mode

cc @serge-sans-paille

[serge-sans-paille]

This all LGTM, but I'd like an other eye on it. Maybe @Meinersbur ?

[shilonic]

In D67258#2248838, @serge-sans-paille wrote:

This all LGTM, but I'd like an other eye on it. Maybe @Meinersbur ?

Any update on that? it's a great solution, filling in the gap with GCC, which enhances dynamic intervention even in RTL phase.
I think this can be utilized for many upgrades and platform adjustments.

[Meinersbur]

RegisterMIRPasses uses the "old" mechanism using a static initializer to register passes. For some time now the llvm-project it trying to get rid of static initializers because of their problems (static initialization order fiasco, more difficult to use as a library, etc.) AFAIU, especially MLIR was designed to avoid static initializers from the ground up, e.g. to register a dialect one has to call e.g. registerAffinePasses explicitly instead of relying on static initializers. I feel the patch re-introduces what we were trying to avoid.

For LLVM, the new mechanism used for NPM is Passes/PassPlugin.h which use dlopen and dlsym to execute a defined entry-point (llvmGetPassPluginInfo).

llvm/docs/WritingAnLLVMPass.rst

cl::opt<RegisterMyPasses::FunctionPassCtor, false,
           RegisterPassParser<RegisterMyPasses> >
   MyPassOpt("mypass",
             cl::init(&createDefaultMyPass),
             cl::desc("my pass option help"));
 
 Here the command option is "``mypass``", with ``createDefaultMyPass`` as the
 default creator.
 
+Out-of-tree MachineIR passes
+----------------------------
+
+MachineIR passes can be implemented out-of-tree and loaded using the ``--load``
+flag of ``llc``, or ``clang`` thanks to ``-Xclang -load -Xclang Pass.so``.
+Examples of such passes are provided in the ``examples/OutOfTreeMIR``
+directory.
+
 Using GDB with dynamically loaded passes
 ----------------------------------------
 
 Unfortunately, using GDB with dynamically loaded passes is not as easy as it
 should be.  First of all, you can't set a breakpoint in a shared object that
 has not been loaded yet, and second of all there are problems with inlined
 functions in shared objects.  Here are some suggestions to debugging your pass
 with GDB.

llvm/examples/CMakeLists.txt

 add_subdirectory(BrainF)
 add_subdirectory(Fibonacci)
 add_subdirectory(HowToUseJIT)
 add_subdirectory(HowToUseLLJIT)
 add_subdirectory(IRTransforms)
 add_subdirectory(Kaleidoscope)
 add_subdirectory(ModuleMaker)
 add_subdirectory(OrcV2Examples)
 add_subdirectory(SpeculativeJIT)
 add_subdirectory(Bye)
 add_subdirectory(ThinLtoJIT)
+add_subdirectory(OutOfTreeMIR)
 
 if(LLVM_ENABLE_EH AND (NOT WIN32) AND (NOT "${LLVM_NATIVE_ARCH}" STREQUAL "ARM"))
     add_subdirectory(ExceptionDemo)
 endif()
 
 if( HAVE_PTHREAD_H )
   add_subdirectory(ParallelJIT)
 endif( HAVE_PTHREAD_H )

llvm/examples/OutOfTreeMIR/CMakeLists.txt

+add_subdirectory(FunctionPass)
+add_subdirectory(ModulePass)

llvm/examples/OutOfTreeMIR/FunctionPass/CMakeLists.txt

+add_llvm_library(MIRFunctionPass MODULE
+  FunctionPass.cpp
+  BUILDTREE_ONLY
+)

llvm/examples/OutOfTreeMIR/FunctionPass/FunctionPass.cpp

+//===-- FunctionPass.cpp - Out-of-tree MachineIR pass example -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include <llvm/CodeGen/MIRPrinter.h>
+#include <llvm/CodeGen/MachineFunctionPass.h>
+#include <llvm/CodeGen/RegisterMIRPasses.h>
+#include <llvm/CodeGen/TargetPassConfig.h>
+#include <llvm/IR/LegacyPassManager.h>
+#include <llvm/Support/raw_ostream.h>
+
+using namespace llvm;
+
+namespace {
+struct MIRPrinter : public MachineFunctionPass {
+  static char ID;
+
+  MIRPrinter() : MachineFunctionPass(ID) {}
+
+  bool runOnMachineFunction(MachineFunction &MF) override {
+    printMIR(errs(), MF);
+    return true;
+  }
+};
+
+char MIRPrinter::ID = 0;
+} // namespace
+
+MachineFunctionPass *createMIRPrinterPass() { return new MIRPrinter{}; }
+
+static void callback(TargetPassConfig const &, legacy::PassManagerBase &PM) {
+  PM.add(createMIRPrinterPass());
+}
+
+static RegisterMIRPasses Register(TargetPassConfig::EP_OptimizerLast, callback);

llvm/examples/OutOfTreeMIR/ModulePass/CMakeLists.txt

+add_llvm_library(MIRModulePass MODULE
+  ModulePass.cpp
+  BUILDTREE_ONLY
+)

llvm/examples/OutOfTreeMIR/ModulePass/ModulePass.cpp

+//===-- ModulePass.cpp - Out-of-tree MachineIR pass example ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include <llvm/CodeGen/MIRPrinter.h>
+#include <llvm/CodeGen/MachineModuleInfo.h>
+#include <llvm/CodeGen/RegisterMIRPasses.h>
+#include <llvm/CodeGen/TargetPassConfig.h>
+#include <llvm/IR/LegacyPassManager.h>
+#include <llvm/IR/Module.h>
+#include <llvm/Support/raw_ostream.h>
+
+using namespace llvm;
+
+namespace {
+struct MIRPrinter : public ModulePass {
+  static char ID;
+
+  MIRPrinter() : ModulePass(ID) {}
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.addRequired<MachineModuleInfoWrapperPass>();
+    AU.addPreserved<MachineModuleInfoWrapperPass>();
+    AU.setPreservesAll();
+    ModulePass::getAnalysisUsage(AU);
+  }
+
+  bool runOnModule(Module &M) override {
+    MachineModuleInfo &MMI =
+        getAnalysis<MachineModuleInfoWrapperPass>().getMMI();
+
+    for (Function &F : M) {
+      if (MachineFunction *MF = MMI.getMachineFunction(F)) {
+        printMIR(errs(), *MF);
+      }
+    }
+    return true;
+  }
+};
+
+char MIRPrinter::ID = 0;
+} // namespace
+
+ModulePass *createMIRPrinterPass() { return new MIRPrinter{}; }
+
+static void callback(TargetPassConfig const &, legacy::PassManagerBase &PM) {
+  PM.add(createMIRPrinterPass());
+}
+
+static RegisterMIRPasses Register(TargetPassConfig::EP_OptimizerLast, callback);

llvm/include/llvm/CodeGen/RegisterMIRPasses.h

+//==- include/llvm/CodeGen/RegisterMIRPasses.h - OOT MIR passes ---*- C++
+//-*-==//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains utilities to register out-of-tree MIR passes.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CODEGEN_REGISTERMIRPASSES_H
+#define LLVM_CODEGEN_REGISTERMIRPASSES_H
+
+#include "llvm/CodeGen/TargetPassConfig.h"
+
+namespace llvm {
+
+class RegisterMIRPasses {
+public:
+  RegisterMIRPasses(TargetPassConfig::ExtensionPointTy Ty,
+                    TargetPassConfig::ExtensionFn Fn) {
+    TargetPassConfig::addGlobalExtension(Ty, Fn);
+  }
+};
+
+} // namespace llvm
+
+#endif

llvm/include/llvm/CodeGen/TargetPassConfig.h

 //===- TargetPassConfig.h - Code Generation pass options --------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 /// Target-Independent Code Generator Pass Configuration Options pass.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_TARGETPASSCONFIG_H
 #define LLVM_CODEGEN_TARGETPASSCONFIG_H
 
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/CodeGen.h"
 #include <cassert>
 #include <string>
 
 namespace llvm {
 
 class LLVMTargetMachine;

 };
 
 
 /// Target-Independent Code Generator Pass Configuration Options.
 ///
 /// This is an ImmutablePass solely for the purpose of exposing CodeGen options
 /// to the internals of other CodeGen passes.
 class TargetPassConfig : public ImmutablePass {
+public:
+  typedef void (*ExtensionFn)(const TargetPassConfig &Builder,
+                              legacy::PassManagerBase &PM);
+
+  enum ExtensionPointTy {
+    /// This extension point allows adding passes before
+    /// any other transformations.
+    EP_EarlyAsPossible,
+
+    /// This extension point allows adding passes before register allocation.
+    EP_BeforeRA,
+    /// This extension point allows adding passes after register allocation.
+    EP_AfterRA,
+    /// This extension point allows adding passes that / run after everything
+    // else.
+    EP_OptimizerLast,
+  };
+  static constexpr size_t EP_EnumSize = EP_OptimizerLast + 1;
+
+  static void addGlobalExtension(ExtensionPointTy Ty, ExtensionFn Fn);
+  void addExtension(ExtensionPointTy Ty, ExtensionFn Fn);
+  void addExtensionsToPM(ExtensionPointTy ETy) const;
+
 private:
+  SmallVector<ExtensionFn, 4> Extensions[EP_EnumSize];
+
   PassManagerBase *PM = nullptr;
   AnalysisID StartBefore = nullptr;
   AnalysisID StartAfter = nullptr;
   AnalysisID StopBefore = nullptr;
   AnalysisID StopAfter = nullptr;
 
   unsigned StartBeforeInstanceNum = 0;
   unsigned StartBeforeCount = 0;

llvm/lib/CodeGen/TargetPassConfig.cpp

StopAfterOpt(StringRef(StopAfterOptName),
                  cl::desc("Stop compilation after a specific pass"),
                  cl::value_desc("pass-name"), cl::init(""), cl::Hidden);
 
 static cl::opt<std::string>
     StopBeforeOpt(StringRef(StopBeforeOptName),
                   cl::desc("Stop compilation before a specific pass"),
                   cl::value_desc("pass-name"), cl::init(""), cl::Hidden);
 
+static ManagedStatic<std::array<SmallVector<TargetPassConfig::ExtensionFn, 4>,
+                                TargetPassConfig::EP_EnumSize>>
+    GlobalExtensions;
+
+void TargetPassConfig::addGlobalExtension(TargetPassConfig::ExtensionPointTy Ty,
+                                          TargetPassConfig::ExtensionFn Fn) {
+  (*GlobalExtensions)[Ty].push_back(Fn);
+}
+
+void TargetPassConfig::addExtension(ExtensionPointTy Ty, ExtensionFn Fn) {
+  Extensions[Ty].push_back(Fn);
+}
+
+void TargetPassConfig::addExtensionsToPM(ExtensionPointTy ETy) const {
+  if (GlobalExtensions.isConstructed()) {
+    for (auto Fn : (*GlobalExtensions)[ETy]) {
+      Fn(*this, *PM);
+    }
+  }
+  for (auto Fn : Extensions[ETy]) {
+    Fn(*this, *PM);
+  }
+}
+
 /// Allow standard passes to be disabled by command line options. This supports
 /// simple binary flags that either suppress the pass or do nothing.
 /// i.e. -disable-mypass=false has no effect.
 /// These should be converted to boolOrDefault in order to use applyOverride.
 static IdentifyingPassPtr applyDisable(IdentifyingPassPtr PassID,
                                        bool Override) {
   if (Override)
     return IdentifyingPassPtr();

 /// tied to a common pass. But if it has subtle dependencies on multiple passes,
 /// the target should override the stage instead.
 ///
 /// TODO: We could use a single addPre/Post(ID) hook to allow pass injection
 /// before/after any target-independent pass. But it's currently overkill.
 void TargetPassConfig::addMachinePasses() {
   AddingMachinePasses = true;
 
+  addExtensionsToPM(EP_EarlyAsPossible);
+
   // Add passes that optimize machine instructions in SSA form.
   if (getOptLevel() != CodeGenOpt::None) {
     addMachineSSAOptimization();
   } else {
     // If the target requests it, assign local variables to stack slots relative
     // to one another and simplify frame index references where possible.
     addPass(&LocalStackSlotAllocationID);
   }
 
   if (TM->Options.EnableIPRA)
     addPass(createRegUsageInfoPropPass());
 
+  addExtensionsToPM(EP_BeforeRA);
+
   // Run pre-ra passes.
   addPreRegAlloc();
 
   // Debugifying the register allocator passes seems to provoke some
   // non-determinism that affects CodeGen and there doesn't seem to be a point
   // where it becomes safe again so stop debugifying here.
   DebugifyIsSafe = false;
 
   // Run register allocation and passes that are tightly coupled with it,
   // including phi elimination and scheduling.
   if (getOptimizeRegAlloc())
     addOptimizedRegAlloc();
   else
     addFastRegAlloc();
 
   // Run post-ra passes.
   addPostRegAlloc();
 
   addPass(&FixupStatepointCallerSavedID);
 
+  addExtensionsToPM(EP_AfterRA);
+
   // Insert prolog/epilog code.  Eliminate abstract frame index references...
   if (getOptLevel() != CodeGenOpt::None) {
     addPass(&PostRAMachineSinkingID);
     addPass(&ShrinkWrapID);
   }
 
   // Prolog/Epilog inserter needs a TargetMachine to instantiate. But only
   // do so if it hasn't been disabled, substituted, or overridden.
void TargetPassConfig::addMachinePasses() {
   }
 
   if (TM->getBBSectionsType() != llvm::BasicBlockSection::None)
     addPass(llvm::createBasicBlockSectionsPass(TM->getBBSectionsFuncListBuf()));
 
   // Add passes that directly emit MI after all other MI passes.
   addPreEmitPass2();
 
+  addExtensionsToPM(EP_OptimizerLast);
+
   AddingMachinePasses = false;
 }
 
 /// Add passes that optimize machine instructions in SSA form.
 void TargetPassConfig::addMachineSSAOptimization() {
   // Pre-ra tail duplication.
   addPass(&EarlyTailDuplicateID);
 

llvm/test/Examples/OutOfTreeMIR/llc.test

+; RUN: llc %s --load=%llvmshlibdir/MIRFunctionPass%shlibext -o /dev/null 2>&1 | FileCheck %s
+; RUN: llc %s --load=%llvmshlibdir/MIRModulePass%shlibext -o /dev/null 2>&1 | FileCheck %s
+
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define dso_local i32 @foo() {
+entry:
+  ; CHECK: MOV32ri 42
+  ret i32 42
+}