[ScalarizeMaskedMemIntrin] Add new PM support

[anna]

Summary

This patch adds new PM support for the pass (can be now used during
middle-end transforms). The old pass is remamed to
ScalarizeMaskedMemIntrinLegacyPass.

Tests-Run: code gen tests.

[anna]

missed adding new header file to diff.

[skatkov]

llvm/include/llvm/LinkAllPasses.h

227

space. please be uniform with the current formatting.

[skatkov]

Please consider separate renaming to Legacy to NFC patch.

llvm/lib/Transforms/Scalar/ScalarizeMaskedMemIntrin.cpp

849

You can get DL inside runImpl as it works identical in both legacy and new pass manager and does not depend on analysis.

[anna]

In D92743#2436311, @skatkov wrote:

Please consider separate renaming to Legacy to NFC patch.

I thought about it and saw that the renaming was done as part of same patch for new PM support for other passes as well (and I think it was intentional, in showing the need for the name update). :) If you feel strongly about it, I can do so.

[anna]

llvm/lib/Transforms/Scalar/ScalarizeMaskedMemIntrin.cpp

849

yeah, good point. will do.

[skatkov]

In D92743#2437846, @anna wrote:

In D92743#2436311, @skatkov wrote:

Please consider separate renaming to Legacy to NFC patch.

I thought about it and saw that the renaming was done as part of same patch for new PM support for other passes as well (and I think it was intentional, in showing the need for the name update). :) If you feel strongly about it, I can do so.

I do not insist on it.

[aeubanks]

can you remove this from the list in https://github.com/llvm/llvm-project/blob/6e614b0c7ed3a9a66428f342bf2a4b3700525395/llvm/tools/opt/opt.cpp#L463 and make sure that no check-llvm tests regress when -enable-new-pm is turned on by default? (I should rename that function)

[aeubanks]

Oh and add a -passes=scalarize-masked-mem-intrin RUN line to one of the tests just to make sure it works

[skatkov]

In D92743#2438780, @aeubanks wrote:

Oh and add a -passes=scalarize-masked-mem-intrin RUN line to one of the tests just to make sure it works

agreed. makes sense.

[anna]

In D92743#2438777, @aeubanks wrote:

can you remove this from the list in https://github.com/llvm/llvm-project/blob/6e614b0c7ed3a9a66428f342bf2a4b3700525395/llvm/tools/opt/opt.cpp#L463 and make sure that no check-llvm tests regress when -enable-new-pm is turned on by default? (I should rename that function)

Yes, pls rename the function since it actually represents which passes are not yet ported to NewPM.
Also, I have run the tests with enable-new-pm=true. However, there are 36 failures which are already failing with new-PM without my change (had to build both with and without my change to confirm this). Is there some buildbot that tracks the NewPM option? It will make it easier to catch if any testcases regress...

[anna]

addressed review comments

[aeubanks]

In D92743#2440868, @anna wrote:

In D92743#2438777, @aeubanks wrote:

can you remove this from the list in https://github.com/llvm/llvm-project/blob/6e614b0c7ed3a9a66428f342bf2a4b3700525395/llvm/tools/opt/opt.cpp#L463 and make sure that no check-llvm tests regress when -enable-new-pm is turned on by default? (I should rename that function)

Yes, pls rename the function since it actually represents which passes are not yet ported to NewPM.

https://reviews.llvm.org/D92818

Also, I have run the tests with enable-new-pm=true. However, there are 36 failures which are already failing with new-PM without my change (had to build both with and without my change to confirm this). Is there some buildbot that tracks the NewPM option? It will make it easier to catch if any testcases regress...

I'm working on reducing it to 0 so that we can have a bot that checks if it regresses.

[anna]

missed one review comment: move DL initialization to runImpl.

[anna]

In D92743#2440881, @aeubanks wrote:

Also, I have run the tests with enable-new-pm=true. However, there are 36 failures which are already failing with new-PM without my change (had to build both with and without my change to confirm this). Is there some buildbot that tracks the NewPM option? It will make it easier to catch if any testcases regress...

I'm working on reducing it to 0 so that we can have a bot that checks if it regresses.

Great, thanks!

llvm/include/llvm/CodeGen/Passes.h

MachineFunctionPass *createResetMachineFunctionPass(bool EmitFallbackDiag,
                                                       bool AbortOnFailedISel);
 
   /// createCodeGenPreparePass - Transform the code to expose more pattern
   /// matching during instruction selection.
   FunctionPass *createCodeGenPreparePass();
 
   /// createScalarizeMaskedMemIntrinPass - Replace masked load, store, gather
   /// and scatter intrinsics with scalar code when target doesn't support them.
-  FunctionPass *createScalarizeMaskedMemIntrinPass();
+  FunctionPass *createScalarizeMaskedMemIntrinLegacyPass();
 
   /// AtomicExpandID -- Lowers atomic operations in terms of either cmpxchg
   /// load-linked/store-conditional loops.
   extern char &AtomicExpandID;
 
   /// MachineLoopInfo - This pass is a loop analysis pass.
   extern char &MachineLoopInfoID;
 

llvm/include/llvm/InitializePasses.h

 void initializeSCEVAAWrapperPassPass(PassRegistry&);
 void initializeSLPVectorizerPass(PassRegistry&);
 void initializeSROALegacyPassPass(PassRegistry&);
 void initializeSafeStackLegacyPassPass(PassRegistry&);
 void initializeSafepointIRVerifierPass(PassRegistry&);
 void initializeSampleProfileLoaderLegacyPassPass(PassRegistry&);
 void initializeModuleSanitizerCoverageLegacyPassPass(PassRegistry &);
 void initializeScalarEvolutionWrapperPassPass(PassRegistry&);
-void initializeScalarizeMaskedMemIntrinPass(PassRegistry&);
+void initializeScalarizeMaskedMemIntrinLegacyPassPass(PassRegistry &);
 void initializeScalarizerLegacyPassPass(PassRegistry&);
 void initializeScavengerTestPass(PassRegistry&);
 void initializeScopedNoAliasAAWrapperPassPass(PassRegistry&);
 void initializeSeparateConstOffsetFromGEPLegacyPassPass(PassRegistry &);
 void initializeShadowStackGCLoweringPass(PassRegistry&);
 void initializeShrinkWrapPass(PassRegistry&);
 void initializeSimpleInlinerPass(PassRegistry&);
 void initializeSimpleLoopUnswitchLegacyPassPass(PassRegistry&);

llvm/include/llvm/LinkAllPasses.h

ForcePassLinking() {
       (void) llvm::createSpeculativeExecutionIfHasBranchDivergencePass();
       (void) llvm::createRewriteSymbolsPass();
       (void) llvm::createStraightLineStrengthReducePass();
       (void) llvm::createMemDerefPrinter();
       (void) llvm::createMustExecutePrinter();
       (void) llvm::createMustBeExecutedContextPrinter();
       (void) llvm::createFloat2IntPass();
       (void) llvm::createEliminateAvailableExternallyPass();
-      (void) llvm::createScalarizeMaskedMemIntrinPass();
+      (void)llvm::createScalarizeMaskedMemIntrinLegacyPass();
       (void) llvm::createWarnMissedTransformationsPass();
       (void) llvm::createHardwareLoopsPass();
       (void) llvm::createInjectTLIMappingsLegacyPass();
       (void) llvm::createUnifyLoopExitsPass();
       (void) llvm::createFixIrreduciblePass();
 
       (void)new llvm::IntervalPartition();
       (void)new llvm::ScalarEvolutionWrapperPass();

llvm/lib/CodeGen/TargetPassConfig.cpp

if (getOptLevel() != CodeGenOpt::None && !DisablePartialLibcallInlining)
     addPass(createPartiallyInlineLibCallsPass());
 
   // Instrument function entry and exit, e.g. with calls to mcount().
   addPass(createPostInlineEntryExitInstrumenterPass());
 
   // Add scalarization of target's unsupported masked memory intrinsics pass.
   // the unsupported intrinsic will be replaced with a chain of basic blocks,
   // that stores/loads element one-by-one if the appropriate mask bit is set.
-  addPass(createScalarizeMaskedMemIntrinPass());
+  addPass(createScalarizeMaskedMemIntrinLegacyPass());
 
   // Expand reduction intrinsics into shuffle sequences if the target wants to.
   // Allow disabling it for testing purposes.
   if (!DisableExpandReductions)
     addPass(createExpandReductionsPass());
 }
 
 /// Turn exception handling constructs into something the code generators can

llvm/lib/Passes/PassBuilder.cpp

 #include "llvm/Transforms/Scalar/NaryReassociate.h"
 #include "llvm/Transforms/Scalar/NewGVN.h"
 #include "llvm/Transforms/Scalar/PartiallyInlineLibCalls.h"
 #include "llvm/Transforms/Scalar/Reassociate.h"
 #include "llvm/Transforms/Scalar/Reg2Mem.h"
 #include "llvm/Transforms/Scalar/RewriteStatepointsForGC.h"
 #include "llvm/Transforms/Scalar/SCCP.h"
 #include "llvm/Transforms/Scalar/SROA.h"
+#include "llvm/Transforms/Scalar/ScalarizeMaskedMemIntrin.h"
 #include "llvm/Transforms/Scalar/Scalarizer.h"
 #include "llvm/Transforms/Scalar/SeparateConstOffsetFromGEP.h"
 #include "llvm/Transforms/Scalar/SimpleLoopUnswitch.h"
 #include "llvm/Transforms/Scalar/SimplifyCFG.h"
 #include "llvm/Transforms/Scalar/Sink.h"
 #include "llvm/Transforms/Scalar/SpeculateAroundPHIs.h"
 #include "llvm/Transforms/Scalar/SpeculativeExecution.h"
 #include "llvm/Transforms/Scalar/StraightLineStrengthReduce.h"

llvm/lib/Passes/PassRegistry.def

 // TODO: rename to print<foo> after NPM switch
 FUNCTION_PASS("print-alias-sets", AliasSetsPrinterPass(dbgs()))
 FUNCTION_PASS("print-predicateinfo", PredicateInfoPrinterPass(dbgs()))
 FUNCTION_PASS("print-mustexecute", MustExecutePrinterPass(dbgs()))
 FUNCTION_PASS("print-memderefs", MemDerefPrinterPass(dbgs()))
 FUNCTION_PASS("reassociate", ReassociatePass())
 FUNCTION_PASS("redundant-dbg-inst-elim", RedundantDbgInstEliminationPass())
 FUNCTION_PASS("reg2mem", RegToMemPass())
+FUNCTION_PASS("scalarize-masked-mem-intrin", ScalarizeMaskedMemIntrinPass())
 FUNCTION_PASS("scalarizer", ScalarizerPass())
 FUNCTION_PASS("separate-const-offset-from-gep", SeparateConstOffsetFromGEPPass())
 FUNCTION_PASS("sccp", SCCPPass())
 FUNCTION_PASS("simplifycfg", SimplifyCFGPass())
 FUNCTION_PASS("sink", SinkingPass())
 FUNCTION_PASS("slp-vectorizer", SLPVectorizerPass())
 FUNCTION_PASS("slsr", StraightLineStrengthReducePass())
 FUNCTION_PASS("speculative-execution", SpeculativeExecutionPass())

llvm/lib/Transforms/Scalar/Scalar.cpp

void llvm::initializeScalarOpts(PassRegistry &Registry) {
   initializeMergeICmpsLegacyPassPass(Registry);
   initializeMergedLoadStoreMotionLegacyPassPass(Registry);
   initializeNaryReassociateLegacyPassPass(Registry);
   initializePartiallyInlineLibCallsLegacyPassPass(Registry);
   initializeReassociateLegacyPassPass(Registry);
   initializeRedundantDbgInstEliminationPass(Registry);
   initializeRegToMemLegacyPass(Registry);
   initializeRewriteStatepointsForGCLegacyPassPass(Registry);
-  initializeScalarizeMaskedMemIntrinPass(Registry);
+  initializeScalarizeMaskedMemIntrinLegacyPassPass(Registry);
   initializeSCCPLegacyPassPass(Registry);
   initializeSROALegacyPassPass(Registry);
   initializeCFGSimplifyPassPass(Registry);
   initializeStructurizeCFGLegacyPassPass(Registry);
   initializeSimpleLoopUnswitchLegacyPassPass(Registry);
   initializeSinkingLegacyPassPass(Registry);
   initializeTailCallElimPass(Registry);
   initializeSeparateConstOffsetFromGEPLegacyPassPass(Registry);

llvm/lib/Transforms/Scalar/ScalarizeMaskedMemIntrin.cpp

 //===- ScalarizeMaskedMemIntrin.cpp - Scalarize unsupported masked mem ----===//
 //                                    instrinsics
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This pass replaces masked memory intrinsics - when unsupported by the target
 // - with a chain of basic blocks, that deal with the elements one-by-one if the
 // appropriate mask bit is set.
 //
 //===----------------------------------------------------------------------===//
 
+#include "llvm/Transforms/Scalar/ScalarizeMaskedMemIntrin.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Constant.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Function.h"

 #include <cassert>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "scalarize-masked-mem-intrin"
 
 namespace {
 
-class ScalarizeMaskedMemIntrin : public FunctionPass {
+class ScalarizeMaskedMemIntrinLegacyPass : public FunctionPass {
 public:
   static char ID; // Pass identification, replacement for typeid
 
-  explicit ScalarizeMaskedMemIntrin() : FunctionPass(ID) {
-    initializeScalarizeMaskedMemIntrinPass(*PassRegistry::getPassRegistry());
+  explicit ScalarizeMaskedMemIntrinLegacyPass() : FunctionPass(ID) {
+    initializeScalarizeMaskedMemIntrinLegacyPassPass(
+        *PassRegistry::getPassRegistry());
   }
 
   bool runOnFunction(Function &F) override;
 
   StringRef getPassName() const override {
     return "Scalarize Masked Memory Intrinsics";
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addRequired<TargetTransformInfoWrapperPass>();
   }
 };
 
 } // end anonymous namespace
 
 static bool optimizeBlock(BasicBlock &BB, bool &ModifiedDT,
                           const TargetTransformInfo &TTI, const DataLayout &DL);
 static bool optimizeCallInst(CallInst *CI, bool &ModifiedDT,
                              const TargetTransformInfo &TTI,
                              const DataLayout &DL);
 
-char ScalarizeMaskedMemIntrin::ID = 0;
+char ScalarizeMaskedMemIntrinLegacyPass::ID = 0;
 
-INITIALIZE_PASS(ScalarizeMaskedMemIntrin, DEBUG_TYPE,
+INITIALIZE_PASS(ScalarizeMaskedMemIntrinLegacyPass, DEBUG_TYPE,
                 "Scalarize unsupported masked memory intrinsics", false, false)
 
-FunctionPass *llvm::createScalarizeMaskedMemIntrinPass() {
-  return new ScalarizeMaskedMemIntrin();
+FunctionPass *llvm::createScalarizeMaskedMemIntrinLegacyPass() {
+  return new ScalarizeMaskedMemIntrinLegacyPass();
 }
 
 static bool isConstantIntVector(Value *Mask) {
   Constant *C = dyn_cast<Constant>(Mask);
   if (!C)
     return false;
 
   unsigned NumElts = cast<FixedVectorType>(Mask->getType())->getNumElements();
if ((Idx + 1) != VectorWidth) {
       Ptr = PtrPhi;
     }
   }
   CI->eraseFromParent();
 
   ModifiedDT = true;
 }
 
-bool ScalarizeMaskedMemIntrin::runOnFunction(Function &F) {
+static bool runImpl(Function &F, const TargetTransformInfo &TTI,
+                    const DataLayout &DL) {
   bool EverMadeChange = false;
-
-  auto &TTI = getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
-  auto &DL = F.getParent()->getDataLayout();
-
   bool MadeChange = true;
   while (MadeChange) {
     MadeChange = false;
     for (Function::iterator I = F.begin(); I != F.end();) {
       BasicBlock *BB = &*I++;
       bool ModifiedDTOnIteration = false;
       MadeChange |= optimizeBlock(*BB, ModifiedDTOnIteration, TTI, DL);
 
       // Restart BB iteration if the dominator tree of the Function was changed
       if (ModifiedDTOnIteration)
         break;
     }
 
     EverMadeChange |= MadeChange;
   }
-
   return EverMadeChange;
 }
 
+bool ScalarizeMaskedMemIntrinLegacyPass::runOnFunction(Function &F) {
+  auto &TTI = getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
+  auto &DL = F.getParent()->getDataLayout();
+  return runImpl(F, TTI, DL);
+}
+
+PreservedAnalyses
+ScalarizeMaskedMemIntrinPass::run(Function &F, FunctionAnalysisManager &AM) {
+  auto &TTI = AM.getResult<TargetIRAnalysis>(F);
+  auto &DL = F.getParent()->getDataLayout();
+  if (!runImpl(F, TTI, DL))
+    return PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserve<TargetIRAnalysis>();
+  return PA;
+}
+
 static bool optimizeBlock(BasicBlock &BB, bool &ModifiedDT,
                           const TargetTransformInfo &TTI,
                           const DataLayout &DL) {
   bool MadeChange = false;
 
   BasicBlock::iterator CurInstIterator = BB.begin();
   while (CurInstIterator != BB.end()) {
     if (CallInst *CI = dyn_cast<CallInst>(&*CurInstIterator++))

llvm/tools/llc/llc.cpp

int main(int argc, char **argv) {
   initializeLoopStrengthReducePass(*Registry);
   initializeLowerIntrinsicsPass(*Registry);
   initializeEntryExitInstrumenterPass(*Registry);
   initializePostInlineEntryExitInstrumenterPass(*Registry);
   initializeUnreachableBlockElimLegacyPassPass(*Registry);
   initializeConstantHoistingLegacyPassPass(*Registry);
   initializeScalarOpts(*Registry);
   initializeVectorization(*Registry);
-  initializeScalarizeMaskedMemIntrinPass(*Registry);
+  initializeScalarizeMaskedMemIntrinLegacyPassPass(*Registry);
   initializeExpandReductionsPass(*Registry);
   initializeHardwareLoopsPass(*Registry);
   initializeTransformUtils(*Registry);
 
   // Initialize debugging passes.
   initializeScavengerTestPass(*Registry);
 
   // Register the target printer for --version.

llvm/tools/opt/opt.cpp

int main(int argc, char **argv) {
   initializeTransformUtils(Registry);
   initializeInstCombine(Registry);
   initializeAggressiveInstCombine(Registry);
   initializeInstrumentation(Registry);
   initializeTarget(Registry);
   // For codegen passes, only passes that do IR to IR transformation are
   // supported.
   initializeExpandMemCmpPassPass(Registry);
-  initializeScalarizeMaskedMemIntrinPass(Registry);
+  initializeScalarizeMaskedMemIntrinLegacyPassPass(Registry);
   initializeCodeGenPreparePass(Registry);
   initializeAtomicExpandPass(Registry);
   initializeRewriteSymbolsLegacyPassPass(Registry);
   initializeWinEHPreparePass(Registry);
   initializeDwarfEHPreparePass(Registry);
   initializeSafeStackLegacyPassPass(Registry);
   initializeSjLjEHPreparePass(Registry);
   initializePreISelIntrinsicLoweringLegacyPassPass(Registry);