[Matrix] Add LowerMatrixIntrinsics to the NPM

[SjoerdMeijer]

Summary

LowerMatrixIntrinsics wasn't running yet running under the new pass manager, and this adds LowerMatrixIntrinsics to the pipeline. It's added early so that these matrix intrinsics, which are produced by Clang, to give other optimiser the opportunity to clean up / optimise this further.

[fhahn]

llvm/lib/Passes/PassBuilder.cpp

935

I think we should keep this under the same enable-matrix flag like in the old pass manager for now and remove the flag for both when the pass only gets run on functions that contain matrix intrinsics.

[SjoerdMeijer]

llvm/lib/Passes/PassBuilder.cpp

935

Ah yeah, cheers, forgot about that! Will add that.

[SjoerdMeijer]

• Check EnableMatrix for adding the pass, and
• also add it to the -O0 pipeline.

[fhahn]

clang/lib/CodeGen/BackendUtil.cpp

1379

IIUC this references the LLVM option. Does this actually work? Shouldn't that use LangOpts.MatrixTypes, which is set when passing -fenable-matrix to Clang.

clang/test/CodeGen/matrix-lowering-opt-levels.c

8

Might be good to add new-pm run-lines as well.

llvm/lib/Passes/PassBuilder.cpp

935

Is there reason this is run at a different position to the legacy pass manager? If not, I think we should run them at roughly the same location, to minimize differences between new and old PM. In the old pass manager, we run before the VectorizerStart callbacks. Here this would be around line 1102 I think.

[SjoerdMeijer]

clang/lib/CodeGen/BackendUtil.cpp

1379

It does actually :) But must indeed be a layering violation...will fix.

clang/test/CodeGen/matrix-lowering-opt-levels.c

8

The many mysteries of the NPM..... Now looking I guess you mean adding -fexperimental-new-pass-manager (I was working with a build with the NPM enabled/built).

llvm/lib/Passes/PassBuilder.cpp

935

Since the intrinsics are generated by Clang I thought lowering them the earlier the better. And in the old PM, EarlyCSE was run after it, so that's why I put it right here just before EarlyCSE, but wouldn't mind moving it to where you suggested.

[fhahn]

clang/test/CodeGen/matrix-lowering-opt-levels.c

8

adding lines with -fexperimental-new-pass-manager is what I mean. Not sure if that works with NPM enabled builds though.

llvm/lib/Passes/PassBuilder.cpp

935

Yes, EarlyCSE runs after LowerMatrixIntrinsics in addition, if EnableMatrix. I think it's not worth diverging in the new PM. If there's a better position, we should use it for both new PM/LegacyPM.

[SjoerdMeijer]

Agreed, and comments addressed.

[fhahn]

LGTM, thanks! It would be good to also run EarlyCSE as mentioned inline.

llvm/lib/Passes/PassBuilder.cpp

1099

We should probably also run EarlyCSE afterwards (if EnableMatrix is true), to keep things in sync with the legacy PM.

[SjoerdMeijer]

Many thanks for reviewing again. Will fix that before committing.

[fhahn]

Awesome, thanks!

clang/lib/CodeGen/BackendUtil.cpp

 #include "llvm/Transforms/Instrumentation/HWAddressSanitizer.h"
 #include "llvm/Transforms/Instrumentation/InstrProfiling.h"
 #include "llvm/Transforms/Instrumentation/MemorySanitizer.h"
 #include "llvm/Transforms/Instrumentation/SanitizerCoverage.h"
 #include "llvm/Transforms/Instrumentation/ThreadSanitizer.h"
 #include "llvm/Transforms/ObjCARC.h"
 #include "llvm/Transforms/Scalar.h"
 #include "llvm/Transforms/Scalar/GVN.h"
+#include "llvm/Transforms/Scalar/LowerMatrixIntrinsics.h"
 #include "llvm/Transforms/Utils.h"
 #include "llvm/Transforms/Utils/CanonicalizeAliases.h"
 #include "llvm/Transforms/Utils/EntryExitInstrumenter.h"
 #include "llvm/Transforms/Utils/NameAnonGlobals.h"
 #include "llvm/Transforms/Utils/SymbolRewriter.h"
 #include "llvm/Transforms/Utils/UniqueInternalLinkageNames.h"
 #include <memory>
 using namespace clang;
if (LangOpts.Sanitize.has(SanitizerKind::HWAddress)) {
           /*CompileKernel=*/false, Recover));
     }
     if (LangOpts.Sanitize.has(SanitizerKind::KernelHWAddress)) {
       MPM.addPass(HWAddressSanitizerPass(
           /*CompileKernel=*/true, /*Recover=*/true));
     }
 
     if (CodeGenOpts.OptimizationLevel == 0) {
+      // FIXME: the backends do not handle matrix intrinsics currently. Make
+      // sure they are also lowered in O0. A lightweight version of the pass
+      // should run in the backend pipeline on demand.
+      if (LangOpts.MatrixTypes)
+        MPM.addPass(
+            createModuleToFunctionPassAdaptor(LowerMatrixIntrinsicsPass()));
+
       addCoroutinePassesAtO0(MPM, LangOpts, CodeGenOpts);
       addSanitizersAtO0(MPM, TargetTriple, LangOpts, CodeGenOpts);
     }
   }
 
   // FIXME: We still use the legacy pass manager to do code generation. We
   // create that pass manager here and use it as needed below.
   legacy::PassManager CodeGenPasses;

clang/test/CodeGen/matrix-lowering-opt-levels.c

+// RUN: %clang -O0 -fenable-matrix -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -O1 -fenable-matrix -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -O2 -fenable-matrix -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -O3 -fenable-matrix -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -Ofast -fenable-matrix -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -Os -fenable-matrix -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -Oz -fenable-matrix -S -emit-llvm %s -o - | FileCheck  %s
+
+// RUN: %clang -O0 -fenable-matrix -fexperimental-new-pass-manager -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -O1 -fenable-matrix -fexperimental-new-pass-manager -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -O2 -fenable-matrix -fexperimental-new-pass-manager -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -O3 -fenable-matrix -fexperimental-new-pass-manager -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -Ofast -fenable-matrix -fexperimental-new-pass-manager -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -Os -fenable-matrix -fexperimental-new-pass-manager -S -emit-llvm %s -o - | FileCheck  %s
+// RUN: %clang -Oz -fenable-matrix -fexperimental-new-pass-manager -S -emit-llvm %s -o - | FileCheck  %s
+
+// Smoke test that the matrix intrinsics are lowered at any optimisation level.
+
+typedef float m4x4_t __attribute__((matrix_type(4, 4)));
+
+m4x4_t f(m4x4_t a, m4x4_t b, m4x4_t c) {
+  //
+  // CHECK-LABEL: f(
+  // CHECK-NOT:     @llvm.matrix
+  // CHECK:       }
+  //
+  return a + b * c;
+}

llvm/lib/Passes/PassBuilder.cpp

 extern cl::opt<bool> EnableHotColdSplit;
 extern cl::opt<bool> EnableOrderFileInstrumentation;
 
 extern cl::opt<bool> FlattenedProfileUsed;
 
 extern cl::opt<AttributorRunOption> AttributorRun;
 extern cl::opt<bool> EnableKnowledgeRetention;
 
+extern cl::opt<bool> EnableMatrix;
+
 const PassBuilder::OptimizationLevel PassBuilder::OptimizationLevel::O0 = {
     /*SpeedLevel*/ 0,
     /*SizeLevel*/ 0};
 const PassBuilder::OptimizationLevel PassBuilder::OptimizationLevel::O1 = {
     /*SpeedLevel*/ 1,
     /*SizeLevel*/ 0};
 const PassBuilder::OptimizationLevel PassBuilder::OptimizationLevel::O2 = {
     /*SpeedLevel*/ 2,
ModulePassManager PassBuilder::buildModuleSimplificationPipeline(
   // Do basic inference of function attributes from known properties of system
   // libraries and other oracles.
   MPM.addPass(InferFunctionAttrsPass());
 
   // Create an early function pass manager to cleanup the output of the
   // frontend.
   FunctionPassManager EarlyFPM(DebugLogging);
   EarlyFPM.addPass(SimplifyCFGPass());
   EarlyFPM.addPass(SROA());
   EarlyFPM.addPass(EarlyCSEPass());
   EarlyFPM.addPass(LowerExpectIntrinsicPass());
   if (PTO.Coroutines)
     EarlyFPM.addPass(CoroEarlyPass());
   if (Level == OptimizationLevel::O3)
     EarlyFPM.addPass(CallSiteSplittingPass());
 
   // In SamplePGO ThinLTO backend, we need instcombine before profile annotation
ModulePassManager PassBuilder::buildModuleOptimizationPipeline(
   // the vectorizer will be able to use them to help recognize vectorizable
   // memory operations.
   MPM.addPass(RequireAnalysisPass<GlobalsAA, Module>());
 
   FunctionPassManager OptimizePM(DebugLogging);
   OptimizePM.addPass(Float2IntPass());
   OptimizePM.addPass(LowerConstantIntrinsicsPass());
 
+  if (EnableMatrix) {
+    OptimizePM.addPass(LowerMatrixIntrinsicsPass());
+    OptimizePM.addPass(EarlyCSEPass());
+  }
+
   // FIXME: We need to run some loop optimizations to re-rotate loops after
   // simplify-cfg and others undo their rotation.
 
   // Optimize the loop execution. These passes operate on entire loop nests
   // rather than on each loop in an inside-out manner, and so they are actually
   // function passes.
 
   for (auto &C : VectorizerStartEPCallbacks)

llvm/lib/Transforms/IPO/PassManagerBuilder.cpp

cl::opt<bool> FlattenedProfileUsed(
     "flattened-profile-used", cl::init(false), cl::Hidden,
     cl::desc("Indicate the sample profile being used is flattened, i.e., "
              "no inline hierachy exists in the profile. "));
 
 cl::opt<bool> EnableOrderFileInstrumentation(
     "enable-order-file-instrumentation", cl::init(false), cl::Hidden,
     cl::desc("Enable order file instrumentation (default = off)"));
 
-static cl::opt<bool>
-    EnableMatrix("enable-matrix", cl::init(false), cl::Hidden,
-                 cl::desc("Enable lowering of the matrix intrinsics"));
+cl::opt<bool> EnableMatrix(
+    "enable-matrix", cl::init(false), cl::Hidden,
 
 cl::opt<AttributorRunOption> AttributorRun(
     "attributor-enable", cl::Hidden, cl::init(AttributorRunOption::NONE),
     cl::desc("Enable the attributor inter-procedural deduction pass."),
     cl::values(clEnumValN(AttributorRunOption::ALL, "all",
                           "enable all attributor runs"),
                clEnumValN(AttributorRunOption::MODULE, "module",
                           "enable module-wide attributor runs"),