[VP,Integer,#2] ExpandVectorPredication pass

[simoll]

Summary

This patch implements expansion of llvm.vp.* intrinsics (https://llvm.org/docs/LangRef.html#vector-predication-intrinsics).

VP expansion is required for targets that do not implement VP code generation. Since expansion is controllable with TTI, targets can switch on the VP intrinsics they do support in their backend offering a smooth transition strategy for VP code generation (VE, RISC-V V, ARM SVE, AVX512, ..).

ExpandVectorPredication

The ExpandVectorPredication pass that folds the EVL parameter into the vector mask and/or lowers a VP intrinsics to a standard IR instruction.

[simoll]

Since this is bound to come up: Yes, there is no expansion for scalable types yet because i could not find a target-agnostic way to generate a step vector. I only found this seemingly abandoned patch: https://reviews.llvm.org/D47774. Any suggestion for a (portable) evl-to-mask lowering or patches for stepvector are appreciated.

[arsenm]

llvm/include/llvm/CodeGen/ExpandVectorPredication.h

2

C++ mode comment

llvm/lib/CodeGen/ExpandVectorPredication.cpp

198–199

Shouldn't be true for FP divisions

272–273

Combine into one LLVM_DEBUG

[simoll]

• fixed sanitizeStrategy
• arsenm's comments (thx!)
• camelCaseForFunctionNames
• rebased

[simoll]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

198–199

Yep. At least not before we have vector-predicated, constrained fdiv.

[simoll]

This will use llvm.get.active.lane.mask to expand VP intrinsics for scalable vectors on targets that do not support the %evl parameter (D79100).

[simoll]

• Use llvm.get.active.lane.mask to fold the %evl parameter into the vector mask parameter for scalable VP ops.
• Never discard %evl if the operation may cause UB (eg for sdiv, udiv).
• Added tests for VP ops with scalable types.

[craig.topper]

llvm/include/llvm/IR/PredicatedInst.h

35

Can this inherit from Instruction? Its a little surprising that it has Instruction in its name but has to be casted to Instruction.

51

Can we move this above copyIRFlags to keep all the deleted things together and away from code that does stuff.

llvm/lib/CodeGen/ExpandVectorPredication.cpp

46

Should this start with lower case 'i'?

71

Add /* Vector */ comment to the true.

77

Save the register bit width instead of querying it again?

99

This isn't safe for sdiv/srem is it? INT_MIN/-1 is overflow and will trap.

248

Do we need the nullptr here? I had to lookup what argument that is.

llvm/lib/IR/PredicatedInst.cpp

29

I think we prefer "auto *" for pointers?

[simoll]

@craig.topper Thanks for your comments! I'll update shortly.

llvm/include/llvm/IR/PredicatedInst.h

35

Conceptually, PredicatedInstruction is a super class of Instruction: An Instruction is a PredicatedInstruction with the mask/evl parameters hard-wired to "all lanes true". That's why we can't make Instruction an ancestor of this one.
I'll clarify this with a comment.

llvm/lib/CodeGen/ExpandVectorPredication.cpp

71

This really should have its own enum.

77

Moved this function in to the CachingVPExpander class and added a lane bits cache there.

99

Good point. I'll fix this.

llvm/lib/IR/PredicatedInst.cpp

29

Turns out we do. Fixed.

[simoll]

Addressed @craig.topper 's comments

[simoll]

llvm/lib/IR/IntrinsicInst.cpp

309

This unrelated change fixes a bug (ParamFactor does not make sense as a return value here and is not initialized!). I'll push a fix in a separate commit along with a test for this case.

[simoll]

Fixed VPIntrinsic::canIgnoreVectorLength separately (a0dfdda4e5e8) and rebased onto that patch.

[craig.topper]

There are still more "auto" variables that are pointers in this patch.

llvm/include/llvm/IR/PredicatedInst.h

35

But a VPIntrinsic is also an Instruction isn't it? At least the CallInst for the vp intrinsic is an Instruction.

[simoll]

Will fix remaining auto* cases.

llvm/include/llvm/IR/PredicatedInst.h

35

The whole reason for having the PredicatedInstruction class is that we want to add a new perspective on the IR where all instructions may be predicated (through a mask and evl). There are no VPIntrinsics in that perspective.

In the standard IR perspective, using the Instruction type, VPIntrinsics are just regular CallInsts calling intrinsics.

[simoll]

• Use auto*
• Rebased

[craig.topper]

llvm/include/llvm/IR/PredicatedInst.h

35

I was just wanting to get rid of the need to cast to Instruction in copyIRFlags and getParent and replaceOperation etc. Everything this class represents must be an Instruction whether its predicated or not. No type safety comes from this class inheriting from User instead of Instruction since a CallInst to a VP intrinsic is also an Instruction.

[simoll]

Will rephrase this patch with VPIntrinsic instead of PredicatedInstruction since the purpose of the latter will only become clear with generalized pattern matching/lifting optimizations to VP.

llvm/include/llvm/IR/PredicatedInst.h

35

That's exactly the point: a PredicatedInstruction cannot implicitly cast to an Instruction because that would be an implicit change of perspective.

[simoll]

• Removed PredicatedInstruction, which isn't strictly necessary for this patch.
• Rebased.

[simoll]

ping

[simoll]

• adapted to get.active.mask changes (ult instead of ule)
• rebased

[simoll]

fixed for privatized ElementCount members.

[simoll]

Any more feedback for this patch?

[frasercrmck]

Aside from my nitpicks, I was wondering if it's possible to add tests for the other strategies. Even if that involved a strategy that you could force on the command-line, since I know we don't have real TTIs just now. It would be nice to see that the pass behaves in the various ways it's meant to.

llvm/lib/CodeGen/ExpandVectorPredication.cpp

313

stray lower-case l in EVL (or should it be %evl?)

318

typo (reassess), also preferably with a capital R to match the style of the other comments

352

typo: strictly

365

Comment trails off here.

387

You shouldn't need the parentheses in these switch cases, also the formatting with the breaks looks a little off to me.

[simoll]

Linking this to the SelectionDAG patch: D91441

[simoll]

In D78203#2355876, @frasercrmck wrote:

Aside from my nitpicks, I was wondering if it's possible to add tests for the other strategies. Even if that involved a strategy that you could force on the command-line, since I know we don't have real TTIs just now. It would be nice to see that the pass behaves in the various ways it's meant to.

I like that idea! I'll add some cl::opts to override the target strategy for tests. Thx :)

[simoll]

• Added override options to override the expansion strategy.
• Added expansion tests for fixed and scalable types and all possible expansion strategies.
• Rebased.

[simoll]

llvm/include/llvm/IR/Intrinsics.td

1302 ↗

(On Diff #306066)

FYI I'll commit the attribute changes to the VP intrinsics separately.

[simoll]

Rebased onto "speculatable" VP intrinsic patch.

[simoll]

Ping. Where do we stand with this patch, can we LGTM or are there comments?

[ctetreau]

This all seams fairly reasonable to me, just some minor nitpicks. I'm not confident in giving this the final approval, but hopefully this can serve as a ping to those who are.

llvm/include/llvm/Analysis/TargetTransformInfo.h

1221

BIKESHED: "doNothing" sounds like a nop, not a test

llvm/lib/CodeGen/ExpandVectorPredication.cpp

138

This could conceivably work for scalable vectors after https://lists.llvm.org/pipermail/llvm-dev/2021-January/147943.html is merged

226–229

please name the type.

249

please name the type

255

please name types where the typename does not appear on the RHS.

303–304

please name the type

323–324

should these get undefined after the include?

[simoll]

Addressed @ctetreau 's comments. Thx!

[simoll]

Modernize tests, replace undef -> poison

[simoll]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

138

Yep. We can switch to stepvector once its merged.

323–324

These get undefined in the included file.

[frasercrmck]

Looks good to me in general, but I'm similarly not comfortable giving the final approval.

llvm/lib/CodeGen/ExpandVectorPredication.cpp

227

super nit: I'm not sure FirstOp and SndOp reads very well. I see that MemoryBuiltins uses FstParam and SndParam, but more common would be something like Op0 and Op1?

[simoll]

NFC

• Addressed super nit
• Formatting
• Rebased

[craig.topper]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

68

Is this supposed to be a std::string& ?

76

Would !EVLTransformOverride.empty() || !MaskTransformOverride.empty() be better?

91

return ConstVec && ConstVec->isAllOnesValues();

96

No message

166

Is this supposed to be a doxygen comment? It only uses 2 slashes

175

Does this comment go to the constructor? the \p references don't apply to this function. But it's not a doxygen comment.

186

Drop curly braces.

327

Can this use Instruction::isBinaryOp?

393

Drop curly braces

401

This might read better as

"\n::::Transforming " << Worklist.size() << " Instructions"

Then it's obvious the number in parens is a count.

llvm/lib/IR/IntrinsicInst.cpp

201

getValue() will assert itself won't it?

214

Same

[rogfer01]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

72

I think this should be VPINTERNAL_VPLEGAL_CASES

llvm/test/CodeGen/Generic/expand-vp.ll

2

Just a suggestion: llvm/utils/update_test_checks.py can reduce the cost of maintaining this test (I understand we'll have to extend it when VP FP lands).

[simoll]

llvm/test/CodeGen/Generic/expand-vp.ll

2

How about we switch to an auto-generated test once caching is in and the code has stabilized?
Without caching the output is very verbose.. also auto-generated tests are really sentinels for change not actually tests

[simoll]

• NFC
• worked through reviewers' comments (thx!)
• rebased

[craig.topper]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

180

Why int32_t here? The caller uses unsigned and LLVM doesn't usually use specific sized types for integers unless the bit width is important.

[rogfer01]

I understand this is a legacy pass manager. Would it make sense to adapt it to the New PM as well?

llvm/test/CodeGen/Generic/expand-vp.ll

2

Yep. Agreed.

[simoll]

In D78203#2707363, @rogfer01 wrote:

I understand this is a legacy pass manager. Would it make sense to adapt it to the New PM as well?

AFAIK SelectionDAG lives and dies with the old pm.
In any case, there is a wrapper pass for the new pm in this patch modeled after the one for reduction expansion. It's just not being used atm. I guess that's good enough?

llvm/include/llvm/CodeGen/ExpandVectorPredication.h

17–22

Wrapper for the new pm

[rogfer01]

I don't think there is anything controversial left to address (and if found we can always fix it later). In the light of the scenarios that will be enabled by this pass I believe there is no reason not to land this change.

[aprantl]

This commit may have broken the -DLLVM_ENABLE_MODULES=1 build. See http://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/31603/consoleFull#2136199809a1ca8a51-895e-46c6-af87-ce24fa4cd561

FAILED: lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/ExpandVectorPredication.cpp.o 
/Users/buildslave/jenkins/workspace/lldb-cmake/host-compiler/bin/clang++  -DGTEST_HAS_RTTI=0 -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/CodeGen -I/Users/buildslave/jenkins/workspace/lldb-cmake/llvm-project/llvm/lib/CodeGen -Iinclude -I/Users/buildslave/jenkins/workspace/lldb-cmake/llvm-project/llvm/include -Wdocumentation -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -fmodules -fmodules-cache-path=/Users/buildslave/jenkins/workspace/lldb-cmake/lldb-build/module.cache -fcxx-modules -Xclang -fmodules-local-submodule-visibility -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -fdiagnostics-color -O3 -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk    -fno-exceptions -fno-rtti -UNDEBUG -std=c++14 -MD -MT lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/ExpandVectorPredication.cpp.o -MF lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/ExpandVectorPredication.cpp.o.d -o lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/ExpandVectorPredication.cpp.o -c /Users/buildslave/jenkins/workspace/lldb-cmake/llvm-project/llvm/lib/CodeGen/ExpandVectorPredication.cpp
/Users/buildslave/jenkins/workspace/lldb-cmake/llvm-project/llvm/lib/CodeGen/ExpandVectorPredication.cpp:461:30: error: redefinition of 'run'
ExpandVectorPredicationPass::run(Function &F, FunctionAnalysisManager &AM) {
                             ^
/Users/buildslave/jenkins/workspace/lldb-cmake/llvm-project/llvm/include/llvm/CodeGen/MachinePassRegistry.def:106:1: note: previous definition is here
DUMMY_FUNCTION_PASS("expandvp", ExpandVectorPredicationPass, ())
^
/Users/buildslave/jenkins/workspace/lldb-cmake/llvm-project/llvm/include/llvm/CodeGen/CodeGenPassBuilder.h:68:23: note: expanded from macro 'DUMMY_FUNCTION_PASS'
    PreservedAnalyses run(Function &, FunctionAnalysisManager &) {             \
                      ^
1 error generated.

Can you please investigate?

[aprantl]

Since it's already late on Friday I took the liberty to revert the commit so the bots can recover and you can investigate the failure at your own pace. Let me know if I can help with the fix!

[simoll]

In D78203#2730888, @aprantl wrote:

Since it's already late on Friday I took the liberty to revert the commit so the bots can recover and you can investigate the failure at your own pace. Let me know if I can help with the fix!

Thanks for taking action! I'll look into how include/llvm/CodeGen/MachinePassRegistry.def actually works.. i suppose moving the entry from DUMMY_FUNCTION_PASS to FUNCTION_PASS should fix this (will reproduce and fix locally). Never had a revert, what is the usual procedure to recommit after revert? (update/new diff, review, or just commit with the fix?)

[aprantl]

In D78203#2732842, @simoll wrote:

In D78203#2730888, @aprantl wrote:

Since it's already late on Friday I took the liberty to revert the commit so the bots can recover and you can investigate the failure at your own pace. Let me know if I can help with the fix!

Thanks for taking action! I'll look into how include/llvm/CodeGen/MachinePassRegistry.def actually works.. i suppose moving the entry from DUMMY_FUNCTION_PASS to FUNCTION_PASS should fix this (will reproduce and fix locally). Never had a revert, what is the usual procedure to recommit after revert? (update/new diff, review, or just commit with the fix?)

You can just re-commit with a comment in the commit message explaining how the re-commit differs from the original one. Unless you make substantial changes or you are unsure about your solution, there is no need to re-review the patch.

llvm/include/llvm/Analysis/TargetTransformInfo.h

 class CallBase;
 class Function;
 class GlobalValue;
 class IntrinsicInst;
 class LoadInst;
 class LoopAccessInfo;
 class Loop;
 class LoopInfo;
+class PredicatedInstruction;
 class ProfileSummaryInfo;
 class SCEV;
 class ScalarEvolution;
 class StoreInst;
 class SwitchInst;
 class TargetLibraryInfo;
 class Type;
 class User;

 
   /// \name Vector Predication Information
   /// @{
   /// Whether the target supports the %evl parameter of VP intrinsic efficiently
   /// in hardware. (see LLVM Language Reference - "Vector Predication
   /// Intrinsics") Use of %evl is discouraged when that is not the case.
   bool hasActiveVectorLength() const;
 
+  struct VPLegalization {
+    enum VPTransform {
+      // keep the predicating parameter
+      Legal = 0,
+      // where legal, discard the predicate parameter
+      Discard = 1,
+      // transform into something else that is also predicating
+      Convert = 2
+    };
+
+    // How to transform the EVL parameter.
+    // Legal:   keep the EVL parameter as it is.
+    // Discard: Ignore the EVL parameter where it is safe to do so.
+    // Convert: Fold the EVL into the mask parameter.
+    VPTransform EVLParamStrategy;
+
+    // How to transform the operator.
+    // Legal:   The target supports this operator.
+    // Convert: Convert this to a non-VP operation.
+    // The 'Discard' strategy is invalid.
+    VPTransform OpStrategy;
+
+    bool doNothing() const {
+      return (EVLParamStrategy == Legal) && (OpStrategy == Legal);
+    }
+    VPLegalization(VPTransform EVLParamStrategy, VPTransform OpStrategy)
+        : EVLParamStrategy(EVLParamStrategy), OpStrategy(OpStrategy) {}
+  };
+
+  /// This will only be called for vector-predicated instructions.
+  /// \returns How the target needs this vector-predicated operation to be
+  /// transformed.
+  VPLegalization
+  getVPLegalizationStrategy(const PredicatedInstruction &PI) const;
   /// @}
 
   /// @}
 
 private:
   /// Estimate the latency of specified instruction.
   /// Returns 1 as the default value.
   int getInstructionLatency(const Instruction *I) const;
public:
   virtual unsigned getStoreVectorFactor(unsigned VF, unsigned StoreSize,
                                         unsigned ChainSizeInBytes,
                                         VectorType *VecTy) const = 0;
   virtual bool useReductionIntrinsic(unsigned Opcode, Type *Ty,
                                      ReductionFlags) const = 0;
   virtual bool shouldExpandReduction(const IntrinsicInst *II) const = 0;
   virtual unsigned getGISelRematGlobalCost() const = 0;
   virtual bool hasActiveVectorLength() const = 0;
+  virtual VPLegalization
+  getVPLegalizationStrategy(const PredicatedInstruction &PI) const = 0;
   virtual int getInstructionLatency(const Instruction *I) = 0;
 };
 
 template <typename T>
 class TargetTransformInfo::Model final : public TargetTransformInfo::Concept {
   T Impl;
 
 public:
public:
   unsigned getGISelRematGlobalCost() const override {
     return Impl.getGISelRematGlobalCost();
   }
 
   bool hasActiveVectorLength() const override {
     return Impl.hasActiveVectorLength();
   }
 
+  VPLegalization
+  getVPLegalizationStrategy(const PredicatedInstruction &PI) const override {
+    return Impl.getVPLegalizationStrategy(PI);
+  }
+
   int getInstructionLatency(const Instruction *I) override {
     return Impl.getInstructionLatency(I);
   }
 };
 
 template <typename T>
 TargetTransformInfo::TargetTransformInfo(T Impl)
     : TTIImpl(new Model<T>(Impl)) {}

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

public:
   }
 
   bool shouldExpandReduction(const IntrinsicInst *II) const { return true; }
 
   unsigned getGISelRematGlobalCost() const { return 1; }
 
   bool hasActiveVectorLength() const { return false; }
 
+  TargetTransformInfo::VPLegalization
+  getVPLegalizationStrategy(const PredicatedInstruction &PI) const {
+    return TargetTransformInfo::VPLegalization(
+        /* EVLParamStrategy */ TargetTransformInfo::VPLegalization::Discard,
+        /* OperatorStrategy */ TargetTransformInfo::VPLegalization::Convert);
+  }
+
 protected:
   // Obtain the minimum required size to hold the value (without the sign)
   // In case of a vector it returns the min required size for one element.
   unsigned minRequiredElementSize(const Value *Val, bool &isSigned) {
     if (isa<ConstantDataVector>(Val) || isa<ConstantVector>(Val)) {
       const auto *VectorValue = cast<Constant>(Val);
 
       // In case of a vector need to pick the max between the min

llvm/include/llvm/CodeGen/ExpandVectorPredication.h

+//===-- ExpandVectorPredication.h - Expand vector predication ---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CODEGEN_EXPANDVECTORPREDICATION_H
+#define LLVM_CODEGEN_EXPANDVECTORPREDICATION_H
+
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+class ExpandVectorPredicationPass
+    : public PassInfoMixin<ExpandVectorPredicationPass> {
+public:
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
+};
+} // end namespace llvm
+
+#endif // LLVM_CODEGEN_EXPANDVECTORPREDICATION_H

llvm/include/llvm/CodeGen/Passes.h

/// MachineDominanaceFrontier - This pass is a machine dominators analysis pass.
   /// This pass performs outlining on machine instructions directly before
   /// printing assembly.
   ModulePass *createMachineOutlinerPass(bool RunOnAllFunctions = true);
 
   /// This pass expands the experimental reduction intrinsics into sequences of
   /// shuffles.
   FunctionPass *createExpandReductionsPass();
 
+  /// This pass expands the vector predication intrinsics into unpredicated
+  /// instructions with selects or just the explicit vector length into the
+  /// predicate mask.
+  FunctionPass *createExpandVectorPredicationPass();
+
   // This pass expands memcmp() to load/stores.
   FunctionPass *createExpandMemCmpPass();
 
   /// Creates Break False Dependencies pass. \see BreakFalseDeps.cpp
   FunctionPass *createBreakFalseDeps();
 
   // This pass expands indirectbr instructions.
   FunctionPass *createIndirectBrExpandPass();

llvm/include/llvm/IR/IntrinsicInst.h

public:
   /// The llvm.vp.* intrinsics for this instruction Opcode
   static Intrinsic::ID GetForOpcode(unsigned OC);
 
   // Whether \p ID is a VP intrinsic ID.
   static bool IsVPIntrinsic(Intrinsic::ID);
 
   /// \return the mask parameter or nullptr.
   Value *getMaskParam() const;
+  void setMaskParam(Value *);
 
   /// \return the vector length parameter or nullptr.
   Value *getVectorLengthParam() const;
+  void setVectorLengthParam(Value *);
 
   /// \return whether the vector length param can be ignored.
   bool canIgnoreVectorLengthParam() const;
 
   /// \return the static element count (vector number of elements) the vector
   /// length parameter applies to.
   ElementCount getStaticVectorLength() const;
 

llvm/include/llvm/IR/PredicatedInst.h

+//===-- PredicatedInst.h - Utility for predicated instructions --*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines various classes for working with vector-predicated
+// instructions. Predicated instructions are either regular instructions or
+// calls to Vector Predication (VP) intrinsics that have a mask and an explicit
+// vector length argument.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_IR_PREDICATEDINST_H
+#define LLVM_IR_PREDICATEDINST_H
+
+#include "llvm/ADT/None.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/Instruction.h"
+#include "llvm/IR/Operator.h"
+#include "llvm/IR/Type.h"
+#include "llvm/IR/Value.h"
+#include "llvm/Support/Casting.h"
+
+#include <cstddef>
+
+namespace llvm {
+
+class BasicBlock;
+
+/// The PredicatedInstruction class makes VPIntrinsics appear as regular
+/// instructions with an additional mask and evl parameter. Regular instructions
+/// appear as if they had their predication parameters hard-wired to "all lanes
+/// enabled". `PredicatedInstruction` is a conceptual superclass of
+/// `Instruction`.
+///
+/// For example, for the intrinsic call:
+///
+///    %x = call @llvm.vp.add.v256i32(%y,%z,%mask,%evl)
+///
+/// `PredicatedInstruction` reports the Opcode as Instruction::Add, just as
+/// for:
+///
+///    %x = add <256 x i32> %y, %z
+///
+class PredicatedInstruction : public User {
+public:
+  // The PredicatedInstruction class is intended to be used as a utility, and is
+  // never itself instantiated.
+  PredicatedInstruction() = delete;
+  ~PredicatedInstruction() = delete;
+
+  void *operator new(size_t s) = delete;
+
+  void copyIRFlags(const Value *V, bool IncludeWrapFlags) {
+    cast<Instruction>(this)->copyIRFlags(V, IncludeWrapFlags);
+  }
+
+  BasicBlock *getParent() { return cast<Instruction>(this)->getParent(); }
+  const BasicBlock *getParent() const {
+    return cast<const Instruction>(this)->getParent();
+  }
+
+  /// \returns the Mask parameter of this instruction or nullptr if there is
+  /// none.
+  Value *getMaskParam() const;
+
+  /// \returns the Explicit Vector Length parameter of this instruction or
+  /// nullptr if there is none.
+  Value *getVectorLengthParam() const;
+
+  /// \returns True if the passed vector length value has no predicating effect
+  /// on the operation.
+  bool canIgnoreVectorLengthParam() const;
+
+  /// \returns True if the static operator of this instruction has a mask or
+  /// vector length parameter.
+  bool isVectorPredicatedOp() const;
+
+  /// \returns Whether this is functionally a binary operator.
+  bool isBinaryOp() const;
+
+  /// \returns the effective Opcode of this operation (ignoring the mask and
+  /// vector length param).
+  unsigned getOpcode() const;
+
+  static bool classof(const Instruction *I) { return isa<Instruction>(I); }
+  static bool classof(const ConstantExpr *CE) { return false; }
+  static bool classof(const Value *V) { return isa<Instruction>(V); }
+
+  /// Convenience function for getting all the fast-math flags, which must be an
+  /// operator which supports these flags. See LangRef.html for the meaning of
+  /// these flags.
+  FastMathFlags getFastMathFlags() const;
+};
+
+} // namespace llvm
+
+#endif // LLVM_IR_PREDICATEDINST_H

llvm/include/llvm/InitializePasses.h

 void initializeEarlyMachineLICMPass(PassRegistry&);
 void initializeEarlyTailDuplicatePass(PassRegistry&);
 void initializeEdgeBundlesPass(PassRegistry&);
 void initializeEliminateAvailableExternallyLegacyPassPass(PassRegistry&);
 void initializeEntryExitInstrumenterPass(PassRegistry&);
 void initializeExpandMemCmpPassPass(PassRegistry&);
 void initializeExpandPostRAPass(PassRegistry&);
 void initializeExpandReductionsPass(PassRegistry&);
+void initializeExpandVectorPredicationPass(PassRegistry &);
 void initializeMakeGuardsExplicitLegacyPassPass(PassRegistry&);
 void initializeExternalAAWrapperPassPass(PassRegistry&);
 void initializeFEntryInserterPass(PassRegistry&);
 void initializeFinalizeISelPass(PassRegistry&);
 void initializeFinalizeMachineBundlesPass(PassRegistry&);
 void initializeFixIrreduciblePass(PassRegistry &);
 void initializeFixupStatepointCallerSavedPass(PassRegistry&);
 void initializeFlattenCFGPassPass(PassRegistry&);

llvm/lib/Analysis/TargetTransformInfo.cpp

unsigned TargetTransformInfo::getStoreVectorFactor(unsigned VF,
   return TTIImpl->getStoreVectorFactor(VF, StoreSize, ChainSizeInBytes, VecTy);
 }
 
 bool TargetTransformInfo::useReductionIntrinsic(unsigned Opcode, Type *Ty,
                                                 ReductionFlags Flags) const {
   return TTIImpl->useReductionIntrinsic(Opcode, Ty, Flags);
 }
 
+TargetTransformInfo::VPLegalization
+TargetTransformInfo::getVPLegalizationStrategy(
+    const PredicatedInstruction &PI) const {
+  return TTIImpl->getVPLegalizationStrategy(PI);
+}
+
 bool TargetTransformInfo::shouldExpandReduction(const IntrinsicInst *II) const {
   return TTIImpl->shouldExpandReduction(II);
 }
 
 unsigned TargetTransformInfo::getGISelRematGlobalCost() const {
   return TTIImpl->getGISelRematGlobalCost();
 }
 

llvm/lib/CodeGen/CMakeLists.txt

add_llvm_component_library(LLVMCodeGen
   DFAPacketizer.cpp
   DwarfEHPrepare.cpp
   EarlyIfConversion.cpp
   EdgeBundles.cpp
   ExecutionDomainFix.cpp
   ExpandMemCmp.cpp
   ExpandPostRAPseudos.cpp
   ExpandReductions.cpp
+  ExpandVectorPredication.cpp
   FaultMaps.cpp
   FEntryInserter.cpp
   FinalizeISel.cpp
   FixupStatepointCallerSaved.cpp
   FuncletLayout.cpp
   GCMetadata.cpp
   GCMetadataPrinter.cpp
   GCRootLowering.cpp

llvm/lib/CodeGen/ExpandVectorPredication.cpp

+//===--- CodeGen/ExpandVectorPredication.cpp - Expand VP intrinsics -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass implements IR expansion for vector predication intrinsics, allowing
+// targets to enable vector predication until just before codegen.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/CodeGen/ExpandVectorPredication.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/Analysis/TargetTransformInfo.h"
+#include "llvm/Analysis/ValueTracking.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Intrinsics.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PredicatedInst.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/MathExtras.h"
+
+using namespace llvm;
+
+using VPLegalization = TargetTransformInfo::VPLegalization;
+
+#define DEBUG_TYPE "expand-vec-pred"
+
+STATISTIC(NumFoldedVL, "Number of folded vector length params");
+STATISTIC(NumLoweredVPOps, "Number of folded vector predication operations");
+
+///// Helpers {
+
+/// \returns Whether the vector mask \p MaskVal has all lane bits set.
+static bool isAllTrueMask(Value *MaskVal) {
+  auto ConstVec = dyn_cast<ConstantVector>(MaskVal);
+  if (!ConstVec)
+    return false;
+  return ConstVec->isAllOnesValue();
+}
+
+/// Computes the smallest integer bit width to hold the step vector <0, ..,
+/// NumVectorElements - 1>
+static unsigned getLeastLaneBitsForStepVector(unsigned NumVectorElements) {
+  unsigned MostSignificantOne =
+      llvm::countLeadingZeros<uint64_t>(NumVectorElements, ZB_Undefined);
+  return std::max<unsigned>(IntegerType::MIN_INT_BITS, 64 - MostSignificantOne);
+}
+
+/// \returns A non-excepting divisor constant for this type.
+static Constant *getSafeDivisor(Type *DivTy) {
+  assert(DivTy->isIntOrIntVectorTy());
+  return ConstantInt::get(DivTy, 1u, false);
+}
+
+/// Transfer operation properties from \p OldVPI to \p NewVal.
+static void transferDecorations(Value &NewVal, PredicatedInstruction &OldPI) {
+  auto NewInst = dyn_cast<Instruction>(&NewVal);
+  if (!NewInst || !isa<FPMathOperator>(NewVal))
+    return;
+
+  auto OldFMOp = dyn_cast<FPMathOperator>(&OldPI);
+  if (!OldFMOp)
+    return;
+
+  NewInst->setFastMathFlags(OldFMOp->getFastMathFlags());
+}
+
+/// Transfer all properties from \p OldOp to \p NewOp and replace all uses.
+/// OldVP gets erased.
+static void replaceOperation(Value &NewOp, PredicatedInstruction &OldOp) {
+  transferDecorations(NewOp, OldOp);
+  OldOp.replaceAllUsesWith(&NewOp);
+  cast<Instruction>(OldOp).eraseFromParent();
+}
+
+//// } Helpers
+
+namespace {
+
+// Expansion pass state at function scope.
+struct CachingVPExpander {
+  Function &F;
+  const TargetTransformInfo &TTI;
+
+  /// \returns A (fixed length) vector with ascending integer indices
+  /// (<0, 1, ..., NumElems-1>).
+  Value *createStepVector(IRBuilder<> &Builder, int32_t ElemBits,
+                          int32_t NumElems);
+
+  /// \returns A bitmask that is true where the lane position is less-than \p
+  /// EVLParam
+  ///
+  /// \p Builder
+  ///    Used for instruction creation.
+  /// \p VLParam
+  ///    The explicit vector length parameter to test against the lane
+  ///    positions.
+  /// \p ElemCount
+  ///    Static (potentially scalable) number of vector elements
+  Value *convertEVLToMask(IRBuilder<> &Builder, Value *EVLParam,
+                          ElementCount ElemCount);
+
+  Value *foldEVLIntoMask(PredicatedInstruction &OldPI);
+
+  /// "Remove" the %evl parameter of \p PI by setting it to the static vector
+  /// length of the operation.
+  void discardEVLParameter(PredicatedInstruction &PI);
+
+  /// \brief Lower this VP binary operator to a non-VP binary operator.
+  Value *expandPredicationInBinaryOperator(IRBuilder<> &Builder,
+                                           PredicatedInstruction &PI);
+
+  /// \brief query TTI and expand the vector predication in \p P accordingly.
+  Value *expandPredication(PredicatedInstruction &PI);
+
+  /// \brief return a good (for fast icmp) integer bit width to expand
+  /// the EVL comparison against the stepvector in.
+  std::map<unsigned, unsigned> StaticVLToBitsCache; // TODO 'SmallMap' class
+  unsigned getLaneBitsForEVLCompare(unsigned StaticVL);
+
+public:
+  CachingVPExpander(Function &F, const TargetTransformInfo &TTI)
+      : F(F), TTI(TTI) {}
+
+  // expand VP ops in \p F according to \p TTI.
+  bool expandVectorPredication();
+};
+
+//// CachingVPExpander {
+
+unsigned CachingVPExpander::getLaneBitsForEVLCompare(unsigned StaticVL) {
+  auto ItCached = StaticVLToBitsCache.find(StaticVL);
+  if (ItCached != StaticVLToBitsCache.end())
+    return ItCached->second;
+
+  // The smallest integer to hold <0, .., ElemCount.Min -1>
+  // Cannot choose less bits than this or the expansion will be invalid.
+  unsigned MinLaneBits = getLeastLaneBitsForStepVector(StaticVL);
+  LLVM_DEBUG(dbgs() << "Least lane bits for " << StaticVL << " is "
+                    << MinLaneBits << "\n";);
+
+  // If  the EVL compare will be expanded into scalar code, choose the
+  // smallest integer type.
+  if (TTI.getRegisterBitWidth(/* Vector */ true) == 0)
+    return MinLaneBits;
+
+  // Otw, the generated vector operation will likely map to vector instructions.
+  // The largest bit width to fit the EVL expansion in one vector register.
+  unsigned MaxLaneBits = std::min<unsigned>(
+      IntegerType::MAX_INT_BITS, TTI.getRegisterBitWidth(true) / StaticVL);
+
+  // Many SIMD instruction are restricted in their supported lane bit widths.
+  // We choose the bit width that gives us the cheapest vector compare.
+  int Cheapest = std::numeric_limits<int>::max();
+  auto &Ctx = F.getContext();
+  unsigned CheapestLaneBits = MinLaneBits;
+  for (auto LaneBits = MinLaneBits; LaneBits < MaxLaneBits; ++LaneBits) {
+    int VecCmpCost = TTI.getCmpSelInstrCost(
+        Instruction::ICmp, VectorType::get(Type::getIntNTy(Ctx, LaneBits),
+                                           StaticVL, /* Scalable */ false));
+    if (VecCmpCost < Cheapest) {
+      Cheapest = VecCmpCost;
+      CheapestLaneBits = LaneBits;
+    }
+  }
+
+  StaticVLToBitsCache[StaticVL] = CheapestLaneBits;
+  return CheapestLaneBits;
+}
+
+Value *CachingVPExpander::createStepVector(IRBuilder<> &Builder,
+                                           int32_t ElemBits, int32_t NumElems) {
+  // TODO add caching
+  SmallVector<Constant *, 16> ConstElems;
+
+  Type *LaneTy = Builder.getIntNTy(ElemBits);
+
+  for (int32_t Idx = 0; Idx < NumElems; ++Idx) {
+    ConstElems.push_back(ConstantInt::get(LaneTy, Idx, false));
+  }
+
+  return ConstantVector::get(ConstElems);
+}
+
+Value *CachingVPExpander::convertEVLToMask(IRBuilder<> &Builder,
+                                           Value *EVLParam,
+                                           ElementCount ElemCount) {
+  // TODO add caching
+  if (ElemCount.Scalable) {
+    auto M = Builder.GetInsertBlock()->getModule();
+    auto BoolVecTy = VectorType::get(Builder.getInt1Ty(), ElemCount);
+    auto ActiveMaskFunc = Intrinsic::getDeclaration(
+        M, Intrinsic::get_active_lane_mask, {BoolVecTy, EVLParam->getType()});
+    // `get_active_lane_mask` performs an implicit less-equal-than comparison.
+    // Offset the lane index by one accordingly.
+    auto ConstZero = Builder.getInt32(1); // Use the %evl param type
+    return Builder.CreateCall(ActiveMaskFunc, {ConstZero, EVLParam});
+  }
+
+  assert(!ElemCount.Scalable && "code path not applicable to scalable types");
+  unsigned NumElems = ElemCount.Min;
+
+  unsigned ElemBits = getLaneBitsForEVLCompare(NumElems);
+
+  Type *LaneTy = Builder.getIntNTy(ElemBits);
+
+  auto ExtVLParam = Builder.CreateZExtOrTrunc(EVLParam, LaneTy);
+  auto VLSplat = Builder.CreateVectorSplat(NumElems, ExtVLParam);
+
+  auto IdxVec = createStepVector(Builder, ElemBits, NumElems);
+
+  return Builder.CreateICmp(CmpInst::ICMP_ULT, IdxVec, VLSplat);
+}
+
+Value *CachingVPExpander::expandPredicationInBinaryOperator(
+    IRBuilder<> &Builder, PredicatedInstruction &PI) {
+  assert(PI.canIgnoreVectorLengthParam());
+  assert(PI.isBinaryOp());
+
+  auto FirstOp = PI.getOperand(0);
+  auto SndOp = PI.getOperand(1);
+
+  auto Mask = PI.getMaskParam();
+
+  // Blend in safe operands
+  if (Mask && !isAllTrueMask(Mask)) {
+    switch (PI.getOpcode()) {
+    default:
+      // can safely ignore the predicate
+      break;
+
+    // Division operators need a safe divisor on masked-off lanes (1)
+    case Instruction::UDiv:
+    case Instruction::SDiv:
+    case Instruction::URem:
+    case Instruction::SRem:
+      // 2nd operand must not be zero
+      auto SafeDivisor = getSafeDivisor(PI.getType());
+      SndOp = Builder.CreateSelect(Mask, SndOp, SafeDivisor);
+    }
+  }
+
+  auto NewOC = static_cast<Instruction::BinaryOps>(PI.getOpcode());
+  auto NewBinOp = Builder.CreateBinOp(NewOC, FirstOp, SndOp, PI.getName());
+
+  replaceOperation(*NewBinOp, PI);
+  return NewBinOp;
+}
+
+void CachingVPExpander::discardEVLParameter(PredicatedInstruction &PI) {
+  LLVM_DEBUG(dbgs() << "Discard EVL parameter in " << PI << "\n");
+
+  if (PI.canIgnoreVectorLengthParam())
+    return;
+
+  Value *EVLParam = PI.getVectorLengthParam();
+  if (!EVLParam)
+    return;
+
+  VPIntrinsic &VPI = cast<VPIntrinsic>(PI);
+  ElementCount StaticElemCount = VPI.getStaticVectorLength();
+  Value *MaxEVL = nullptr;
+  auto Int32Ty = Type::getInt32Ty(PI.getContext());
+  if (StaticElemCount.Scalable) {
+    // TODO add caching
+    auto M = VPI.getModule();
+    auto VScaleFunc = Intrinsic::getDeclaration(M, Intrinsic::vscale, Int32Ty);
+    IRBuilder<> Builder(VPI.getParent(), VPI.getIterator());
+    auto FactorConst = Builder.getInt32(StaticElemCount.Min);
+    auto VScale = Builder.CreateCall(VScaleFunc, {}, "vscale");
+    MaxEVL = Builder.CreateMul(VScale, FactorConst, "scalable_size",
+                               /*NUW*/ true, /*NSW*/ false);
+  } else {
+    MaxEVL = ConstantInt::get(Int32Ty, StaticElemCount.Min, false);
+  }
+  VPI.setVectorLengthParam(MaxEVL);
+}
+
+Value *CachingVPExpander::foldEVLIntoMask(PredicatedInstruction &OldPI) {
+  LLVM_DEBUG(dbgs() << "Folding vlen for " << OldPI << '\n');
+
+  IRBuilder<> Builder(cast<Instruction>(&OldPI));
+
+  // No %evl parameter and so nothing to do here
+  if (OldPI.canIgnoreVectorLengthParam()) {
+    return &OldPI;
+  }
+
+  // Only VP intrinsics can have a %evl parameter
+  VPIntrinsic &VPI = cast<VPIntrinsic>(OldPI);
+  Value *OldMaskParam = VPI.getMaskParam();
+  Value *OldEVLParam = VPI.getVectorLengthParam();
+  assert(OldMaskParam && "no mask param to fold the vl param into");
+  assert(OldEVLParam && "no EVL param to fold away");
+
+  LLVM_DEBUG(dbgs() << "OLD evl: " << *OldEVLParam << '\n');
+  LLVM_DEBUG(dbgs() << "OLD mask: " << *OldMaskParam << '\n');
+
+  // Convert the %evl predication into vector mask predication.
+  ElementCount ElemCount = VPI.getStaticVectorLength();
+  auto *VLMask = convertEVLToMask(Builder, OldEVLParam, ElemCount);
+  auto NewMaskParam = Builder.CreateAnd(VLMask, OldMaskParam);
+  VPI.setMaskParam(NewMaskParam);
+
+  // Drop the EVl parameter
+  discardEVLParameter(OldPI);
+  assert(VPI.canIgnoreVectorLengthParam() &&
+         "transformation did not render the evl param ineffective!");
+
+  // re-asses the modified instruction
+  return &VPI;
+}
+
+Value *CachingVPExpander::expandPredication(PredicatedInstruction &OldPI) {
+  LLVM_DEBUG(dbgs() << "Lowering to unpredicated op: " << OldPI << '\n');
+
+  IRBuilder<> Builder(cast<Instruction>(&OldPI));
+
+  // Try lowering to a LLVM instruction first.
+  unsigned OC = OldPI.getOpcode();
+#define FIRST_BINARY_INST(X) unsigned FirstBinOp = X;
+#define LAST_BINARY_INST(X) unsigned LastBinOp = X;
+#include "llvm/IR/Instruction.def"
+
+  if (FirstBinOp <= OC && OC <= LastBinOp) {
+    return expandPredicationInBinaryOperator(Builder, OldPI);
+  }
+
+  return &OldPI;
+}
+
+//// } CachingVPExpander
+
+struct TransformJob {
+  PredicatedInstruction *PI;
+  TargetTransformInfo::VPLegalization Strategy;
+  TransformJob(PredicatedInstruction *PI,
+               TargetTransformInfo::VPLegalization InitStrat)
+      : PI(PI), Strategy(InitStrat) {}
+
+  bool isDone() const { return Strategy.doNothing(); }
+};
+
+void sanitizeStrategy(Instruction &I, VPLegalization &LegalizeStrat) {
+  // Speculatable instructions do not strictle need predication.
+  if (isSafeToSpeculativelyExecute(&I))
+    return;
+
+  // Preserve the predication effect of the EVL parameter by folding
+  // it into the predicate.
+  if (LegalizeStrat.EVLParamStrategy == VPLegalization::Discard) {
+    LegalizeStrat.EVLParamStrategy = VPLegalization::Convert;
+  }
+}
+
+/// \brief Expand llvm.vp.* intrinsics as requested by \p TTI.
+bool CachingVPExpander::expandVectorPredication() {
+  // Holds all vector-predicated ops with an effective vector length param that
+  SmallVector<TransformJob, 16> Worklist;
+
+  for (auto &I : instructions(F)) {
+    auto &PI = cast<PredicatedInstruction>(I);
+    auto VPStrat = TTI.getVPLegalizationStrategy(PI);
+    sanitizeStrategy(I, VPStrat);
+    if (!VPStrat.doNothing()) {
+      Worklist.emplace_back(&PI, VPStrat);
+    }
+  }
+  if (Worklist.empty())
+    return false;
+
+  LLVM_DEBUG(dbgs() << "\n:::: Transforming instructions. ::::\n");
+  for (TransformJob Job : Worklist) {
+    // Transform the EVL parameter
+    switch (Job.Strategy.EVLParamStrategy) {
+    case VPLegalization::Legal:
+      break;
+    case VPLegalization::Discard: {
+      discardEVLParameter(*Job.PI);
+    } break;
+    case VPLegalization::Convert: {
+      if (foldEVLIntoMask(*Job.PI)) {
+        ++NumFoldedVL;
+      }
+    } break;
+    }
+    Job.Strategy.EVLParamStrategy = VPLegalization::Legal;
+
+    // Replace the operator
+    switch (Job.Strategy.OpStrategy) {
+    case VPLegalization::Legal:
+      break;
+    case VPLegalization::Discard:
+      llvm_unreachable("Invalid strategy for operators.");
+    case VPLegalization::Convert: {
+      expandPredication(*Job.PI);
+      ++NumLoweredVPOps;
+    } break;
+    }
+    Job.Strategy.OpStrategy = VPLegalization::Legal;
+
+    assert(Job.isDone() && "incomplete transformation");
+  }
+
+  return true;
+}
+class ExpandVectorPredication : public FunctionPass {
+public:
+  static char ID;
+  ExpandVectorPredication() : FunctionPass(ID) {
+    initializeExpandVectorPredicationPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnFunction(Function &F) override {
+    const auto *TTI = &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
+    CachingVPExpander VPExpander(F, *TTI);
+    return VPExpander.expandVectorPredication();
+  }
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.addRequired<TargetTransformInfoWrapperPass>();
+    AU.setPreservesCFG();
+  }
+};
+} // namespace
+
+char ExpandVectorPredication::ID;
+INITIALIZE_PASS_BEGIN(ExpandVectorPredication, "expand-vec-pred",
+                      "Expand vector predication intrinsics", false, false)
+INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
+INITIALIZE_PASS_END(ExpandVectorPredication, "expand-vec-pred",
+                    "Expand vector predication intrinsics", false, false)
+
+FunctionPass *llvm::createExpandVectorPredicationPass() {
+  return new ExpandVectorPredication();
+}
+
+PreservedAnalyses
+ExpandVectorPredicationPass::run(Function &F, FunctionAnalysisManager &AM) {
+  const auto &TTI = AM.getResult<TargetIRAnalysis>(F);
+  CachingVPExpander VPExpander(F, TTI);
+  if (!VPExpander.expandVectorPredication())
+    return PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
+}

llvm/lib/CodeGen/TargetPassConfig.cpp

if (getOptLevel() != CodeGenOpt::None && !DisableConstantHoisting)
     addPass(createConstantHoistingPass());
 
   if (getOptLevel() != CodeGenOpt::None && !DisablePartialLibcallInlining)
     addPass(createPartiallyInlineLibCallsPass());
 
   // Instrument function entry and exit, e.g. with calls to mcount().
   addPass(createPostInlineEntryExitInstrumenterPass());
 
+  // Expand vector predication intrinsics into standard IR instructions.
+  // This pass has to run before ScalarizeMaskedMemIntrin and ExpandReduction
+  // passes since it emits those kinds of intrinsics.
+  addPass(createExpandVectorPredicationPass());
+
   // Add scalarization of target's unsupported masked memory intrinsics pass.
   // the unsupported intrinsic will be replaced with a chain of basic blocks,
   // that stores/loads element one-by-one if the appropriate mask bit is set.
   addPass(createScalarizeMaskedMemIntrinPass());
 
   // Expand reduction intrinsics into shuffle sequences if the target wants to.
   addPass(createExpandReductionsPass());
 }

llvm/lib/IR/CMakeLists.txt

add_llvm_component_library(LLVMCore
   ModuleSummaryIndex.cpp
   Operator.cpp
   OptBisect.cpp
   Pass.cpp
   PassInstrumentation.cpp
   PassManager.cpp
   PassRegistry.cpp
   PassTimingInfo.cpp
+  PredicatedInst.cpp
   SafepointIRVerifier.cpp
   ProfileSummary.cpp
   Statepoint.cpp
   Type.cpp
   TypeFinder.cpp
   Use.cpp
   User.cpp
   Value.cpp

llvm/lib/IR/IntrinsicInst.cpp

 
 Value *VPIntrinsic::getMaskParam() const {
   auto maskPos = GetMaskParamPos(getIntrinsicID());
   if (maskPos)
     return getArgOperand(maskPos.getValue());
   return nullptr;
 }
 
+void VPIntrinsic::setMaskParam(Value *NewMask) {
+  auto MaskPos = GetMaskParamPos(getIntrinsicID());
+  assert(MaskPos.hasValue());
+  setArgOperand(MaskPos.getValue(), NewMask);
+}
+
 Value *VPIntrinsic::getVectorLengthParam() const {
   auto vlenPos = GetVectorLengthParamPos(getIntrinsicID());
   if (vlenPos)
     return getArgOperand(vlenPos.getValue());
   return nullptr;
 }
 
+void VPIntrinsic::setVectorLengthParam(Value *NewEVL) {
+  auto EVLPos = GetVectorLengthParamPos(getIntrinsicID());
+  assert(EVLPos.hasValue());
+  setArgOperand(EVLPos.getValue(), NewEVL);
+}
+
 Optional<int> VPIntrinsic::GetMaskParamPos(Intrinsic::ID IntrinsicID) {
   switch (IntrinsicID) {
   default:
     return None;
 
 #define REGISTER_VP_INTRINSIC(VPID, MASKPOS, VLENPOS)                          \
   case Intrinsic::VPID:                                                        \
     return MASKPOS;
if (EC.Scalable) {
     const auto &DL = ParMod->getDataLayout();
 
     // Compare vscale patterns
     uint64_t ParamFactor;
     if (EC.Min > 1 &&
         match(VLParam, m_c_BinOp(m_ConstantInt(ParamFactor), m_VScale(DL)))) {
       return ParamFactor >= EC.Min;
     }
-    if (match(VLParam, m_VScale(DL))) {
-      return ParamFactor;
-    }
-    return false;
+    return match(VLParam, m_VScale(DL));
   }
 
   // standard SIMD operation
   auto VLConst = dyn_cast<ConstantInt>(VLParam);
   if (!VLConst)
     return false;
 
   uint64_t VLNum = VLConst->getZExtValue();

llvm/lib/IR/PredicatedInst.cpp

+//===--- llvm/PredicatedInst.cpp - Vector-predication utility subclass ----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the PredicationInstruction class.
+//
+//===----------------------------------------------------------------------===//
+
+#include <llvm/IR/InstrTypes.h>
+#include <llvm/IR/Instruction.h>
+#include <llvm/IR/Instructions.h>
+#include <llvm/IR/IntrinsicInst.h>
+#include <llvm/IR/PredicatedInst.h>
+
+namespace {
+#define FIRST_BINARY_INST(X) const unsigned FirstBinOp = X;
+#define LAST_BINARY_INST(X) const unsigned LastBinOp = X;
+#include "llvm/IR/Instruction.def"
+} // namespace
+
+namespace llvm {
+
+bool PredicatedInstruction::canIgnoreVectorLengthParam() const {
+  auto *VPI = dyn_cast<VPIntrinsic>(this);
+  if (!VPI)
+    return true;
+
+  return VPI->canIgnoreVectorLengthParam();
+}
+
+FastMathFlags PredicatedInstruction::getFastMathFlags() const {
+  return cast<Instruction>(this)->getFastMathFlags();
+}
+
+bool PredicatedInstruction::isBinaryOp() const {
+  unsigned FuncOC = getOpcode();
+
+  return FirstBinOp <= FuncOC && FuncOC <= LastBinOp;
+}
+
+Value *PredicatedInstruction::getMaskParam() const {
+  auto *thisVP = dyn_cast<VPIntrinsic>(this);
+  if (!thisVP)
+    return nullptr;
+  return thisVP->getMaskParam();
+}
+
+Value *PredicatedInstruction::getVectorLengthParam() const {
+  auto *thisVP = dyn_cast<VPIntrinsic>(this);
+  if (!thisVP)
+    return nullptr;
+  return thisVP->getVectorLengthParam();
+}
+
+bool PredicatedInstruction::isVectorPredicatedOp() const {
+  return isa<VPIntrinsic>(this);
+}
+
+unsigned PredicatedInstruction::getOpcode() const {
+  auto *VPInst = dyn_cast<VPIntrinsic>(this);
+
+  if (!VPInst)
+    return cast<Instruction>(this)->getOpcode();
+
+  return VPInst->getFunctionalOpcode();
+}
+
+} // namespace llvm

llvm/test/CodeGen/AArch64/O0-pipeline.ll

 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Expand Atomic instructions
 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       Lower Garbage Collection Instructions
 ; CHECK-NEXT:       Shadow Stack GC Lowering
 ; CHECK-NEXT:       Lower constant intrinsics
 ; CHECK-NEXT:       Remove unreachable blocks from the CFG
 ; CHECK-NEXT:       Instrument function entry/exit with calls to e.g. mcount() (post inlining)
+; CHECK-NEXT:       Expand vector predication intrinsics
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:       AArch64 Stack Tagging
 ; CHECK-NEXT:     Rewrite Symbols
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Exception handling preparation
 ; CHECK-NEXT:       Safe Stack instrumentation pass
 ; CHECK-NEXT:       Insert stack protectors

llvm/test/CodeGen/AArch64/O3-pipeline.ll

 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Post-Dominator Tree Construction
 ; CHECK-NEXT:       Branch Probability Analysis
 ; CHECK-NEXT:       Block Frequency Analysis
 ; CHECK-NEXT:       Constant Hoisting
 ; CHECK-NEXT:       Partially inline calls to library functions
 ; CHECK-NEXT:       Instrument function entry/exit with calls to e.g. mcount() (post inlining)
+; CHECK-NEXT:       Expand vector predication intrinsics
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:     Stack Safety Analysis
 ; CHECK-NEXT:       FunctionPass Manager
 ; CHECK-NEXT:         Dominator Tree Construction
 ; CHECK-NEXT:         Natural Loop Information
 ; CHECK-NEXT:         Scalar Evolution Analysis
 ; CHECK-NEXT:         Stack Safety Local Analysis

llvm/test/CodeGen/ARM/O3-pipeline.ll

 ; CHECK-NEXT:      Dominator Tree Construction
 ; CHECK-NEXT:      Natural Loop Information
 ; CHECK-NEXT:      Post-Dominator Tree Construction
 ; CHECK-NEXT:      Branch Probability Analysis
 ; CHECK-NEXT:      Block Frequency Analysis
 ; CHECK-NEXT:      Constant Hoisting
 ; CHECK-NEXT:      Partially inline calls to library functions
 ; CHECK-NEXT:      Instrument function entry/exit with calls to e.g. mcount() (post inlining)
+; CHECK-NEXT:      Expand vector predication intrinsics
 ; CHECK-NEXT:      Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:      Expand reduction intrinsics
 ; CHECK-NEXT:      Dominator Tree Construction
 ; CHECK-NEXT:      Natural Loop Information
 ; CHECK-NEXT:      Scalar Evolution Analysis
 ; CHECK-NEXT:      Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:      Function Alias Analysis Results
 ; CHECK-NEXT:      Transform functions to use DSP intrinsics

llvm/test/CodeGen/Generic/expand-vp.ll

+; RUN: opt --expand-vec-pred -S < %s | FileCheck %s
+
+; All VP intrinsics have to be lowered into non-VP ops
+; CHECK-NOT: {{call.* @llvm.vp.add}}
+; CHECK-NOT: {{call.* @llvm.vp.sub}}
+; CHECK-NOT: {{call.* @llvm.vp.mul}}
+; CHECK-NOT: {{call.* @llvm.vp.sdiv}}
+; CHECK-NOT: {{call.* @llvm.vp.srem}}
+; CHECK-NOT: {{call.* @llvm.vp.udiv}}
+; CHECK-NOT: {{call.* @llvm.vp.urem}}
+; CHECK-NOT: {{call.* @llvm.vp.and}}
+; CHECK-NOT: {{call.* @llvm.vp.or}}
+; CHECK-NOT: {{call.* @llvm.vp.xor}}
+; CHECK-NOT: {{call.* @llvm.vp.ashr}}
+; CHECK-NOT: {{call.* @llvm.vp.lshr}}
+; CHECK-NOT: {{call.* @llvm.vp.shl}}
+
+define void @test_vp_int_v8(<8 x i32> %i0, <8 x i32> %i1, <8 x i32> %i2, <8 x i32> %f3, <8 x i1> %m, i32 %n) {
+  %r0 = call <8 x i32> @llvm.vp.add.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r1 = call <8 x i32> @llvm.vp.sub.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r2 = call <8 x i32> @llvm.vp.mul.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r3 = call <8 x i32> @llvm.vp.sdiv.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r4 = call <8 x i32> @llvm.vp.srem.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r5 = call <8 x i32> @llvm.vp.udiv.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r6 = call <8 x i32> @llvm.vp.urem.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r7 = call <8 x i32> @llvm.vp.and.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r8 = call <8 x i32> @llvm.vp.or.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r9 = call <8 x i32> @llvm.vp.xor.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %rA = call <8 x i32> @llvm.vp.ashr.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %rB = call <8 x i32> @llvm.vp.lshr.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %rC = call <8 x i32> @llvm.vp.shl.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  ret void
+}
+
+; fixed-width vectors
+; integer arith
+declare <8 x i32> @llvm.vp.add.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.sub.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.mul.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.sdiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.srem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.udiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.urem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+; bit arith
+declare <8 x i32> @llvm.vp.and.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.xor.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.or.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.ashr.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.lshr.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.shl.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+
+define void @test_vp_int_vscale(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i32> %i2, <vscale x 4 x i32> %f3, <vscale x 4 x i1> %m, i32 %n) {
+  %r0 = call <vscale x 4 x i32> @llvm.vp.add.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r1 = call <vscale x 4 x i32> @llvm.vp.sub.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r2 = call <vscale x 4 x i32> @llvm.vp.mul.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r3 = call <vscale x 4 x i32> @llvm.vp.sdiv.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r4 = call <vscale x 4 x i32> @llvm.vp.srem.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r5 = call <vscale x 4 x i32> @llvm.vp.udiv.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r6 = call <vscale x 4 x i32> @llvm.vp.urem.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r7 = call <vscale x 4 x i32> @llvm.vp.and.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r8 = call <vscale x 4 x i32> @llvm.vp.or.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r9 = call <vscale x 4 x i32> @llvm.vp.xor.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %rA = call <vscale x 4 x i32> @llvm.vp.ashr.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %rB = call <vscale x 4 x i32> @llvm.vp.lshr.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %rC = call <vscale x 4 x i32> @llvm.vp.shl.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  ret void
+}
+
+; scalable-width vectors
+; integer arith
+declare <vscale x 4 x i32> @llvm.vp.add.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.sub.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.mul.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.sdiv.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.srem.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.udiv.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.urem.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+; bit arith
+declare <vscale x 4 x i32> @llvm.vp.and.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.xor.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.or.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.ashr.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.lshr.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.shl.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)

llvm/test/CodeGen/X86/O0-pipeline.ll

 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Expand Atomic instructions
 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       Lower Garbage Collection Instructions
 ; CHECK-NEXT:       Shadow Stack GC Lowering
 ; CHECK-NEXT:       Lower constant intrinsics
 ; CHECK-NEXT:       Remove unreachable blocks from the CFG
 ; CHECK-NEXT:       Instrument function entry/exit with calls to e.g. mcount() (post inlining)
+; CHECK-NEXT:       Expand vector predication intrinsics
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:       Expand indirectbr instructions
 ; CHECK-NEXT:     Rewrite Symbols
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Exception handling preparation
 ; CHECK-NEXT:       Safe Stack instrumentation pass
 ; CHECK-NEXT:       Insert stack protectors

llvm/test/CodeGen/X86/O3-pipeline.ll

 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Post-Dominator Tree Construction
 ; CHECK-NEXT:       Branch Probability Analysis
 ; CHECK-NEXT:       Block Frequency Analysis
 ; CHECK-NEXT:       Constant Hoisting
 ; CHECK-NEXT:       Partially inline calls to library functions
 ; CHECK-NEXT:       Instrument function entry/exit with calls to e.g. mcount() (post inlining)
+; CHECK-NEXT:       Expand vector predication intrinsics
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Interleaved Access Pass
 ; CHECK-NEXT:       X86 Partial Reduction
 ; CHECK-NEXT:       Expand indirectbr instructions
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Natural Loop Information

llvm/tools/llc/llc.cpp

int main(int argc, char **argv) {
   initializeEntryExitInstrumenterPass(*Registry);
   initializePostInlineEntryExitInstrumenterPass(*Registry);
   initializeUnreachableBlockElimLegacyPassPass(*Registry);
   initializeConstantHoistingLegacyPassPass(*Registry);
   initializeScalarOpts(*Registry);
   initializeVectorization(*Registry);
   initializeScalarizeMaskedMemIntrinPass(*Registry);
   initializeExpandReductionsPass(*Registry);
+  initializeExpandVectorPredicationPass(*Registry);
   initializeHardwareLoopsPass(*Registry);
   initializeTransformUtils(*Registry);
 
   // Initialize debugging passes.
   initializeScavengerTestPass(*Registry);
 
   // Register the target printer for --version.
   cl::AddExtraVersionPrinter(TargetRegistry::printRegisteredTargetsForVersion);

llvm/tools/opt/opt.cpp

int main(int argc, char **argv) {
   initializeGlobalMergePass(Registry);
   initializeIndirectBrExpandPassPass(Registry);
   initializeInterleavedLoadCombinePass(Registry);
   initializeInterleavedAccessPass(Registry);
   initializeEntryExitInstrumenterPass(Registry);
   initializePostInlineEntryExitInstrumenterPass(Registry);
   initializeUnreachableBlockElimLegacyPassPass(Registry);
   initializeExpandReductionsPass(Registry);
+  initializeExpandVectorPredicationPass(Registry);
   initializeWasmEHPreparePass(Registry);
   initializeWriteBitcodePassPass(Registry);
   initializeHardwareLoopsPass(Registry);
   initializeTypePromotionPass(Registry);
 
 #ifdef BUILD_EXAMPLES
   initializeExampleIRTransforms(Registry);
 #endif