[Clang] Allow "vector_size" applied to Booleans

[simoll]

Summary

This is the vector_size alternative to D88905.

This patch extends Clang to allow 'bool' as a valid vector element type (attribute vector_size).

This is intended as the canonical type for SIMD masks and facilitates clean vector intrinsic declarations.
Vectors of i1 are supported on IR level and below down to many SIMD ISAs, such as AVX512, ARM SVE (fixed vector length) and the VE target (NEC SX-Aurora TSUBASA).

The RFC on cfe-dev: https://lists.llvm.org/pipermail/cfe-dev/2020-May/065434.html

[simoll]

NFC. Undid stray change.

[simoll]

clang/lib/CodeGen/CGExprScalar.cpp

4710 ↗

(On Diff #268140)

Accidental change. Will undo.

clang/lib/CodeGen/CGRecordLayoutBuilder.cpp

590 ↗

(On Diff #268140)

This is an old comment that is no longer helpful. Will remove.

[lenary]

clang/docs/LanguageExtensions.rst

473 ↗

(On Diff #268140)

It would be nice if this aside about non-bool vectors was more prominently displayed - it's something I hadn't realised before.

[simoll]

clang/docs/LanguageExtensions.rst

473 ↗

(On Diff #268140)

Yep. that caught me by surprise too. I'll move that sentence to the paragraph about GCC vectors above.

[simoll]

• Improved documentation for the size argument of vector_size.
• Rebased

[simoll]

• Rebased
• Implement EmitFromMemory

[simoll]

Updated clang test

[simoll]

NFC

• Style updates
• Rebased

[kaz7]

Thank you for implementing EmitFromMemory. We are locally trying to use this patch to implement vector mask intrinsic instructions on Aurora VE. It is working well with regression tests. We will try test-suite next.

[rsandifo-arm]

clang/docs/LanguageExtensions.rst

473 ↗

(On Diff #268140)

Sorry for the extremely late comment. Like @lenary I hadn't thought about this. I'd assumed that the vector woiuld still be a multiple of 8 bits in size, but I agree that's probably too restrictive to be the only option available.

In that case, would it make sense to add a separate attribute instead? I think it's too surprising to change the units of the existing attribute based on the element type. Perhaps we should even make it take two parameters: the total number of elements, and the number of bits per element. That might be more natural for some AVX and SVE combinations. We wouldn't need to supporrt all combinations from the outset, it's just a question whether we should make the syntax general enough to support it in future.

Perhaps we could do both: support vector_size for bool using byte sizes (and not allowing subbyte vector lengths), and add a new, more general attribute that allows subbyte lengths and explicit subbyte element sizes.

[simoll]

clang/docs/LanguageExtensions.rst

473 ↗

(On Diff #268140)

In that case, would it make sense to add a separate attribute instead? I think it's too surprising to change the units of the existing attribute based on the element type. Perhaps we should even make it take two parameters: the total number of elements, and the number of bits per element. That might be more natural for some AVX and SVE combinations. We wouldn't need to supporrt all combinations from the outset, it's just a question whether we should make the syntax general enough to support it in future.

I guess adding a new attribute makes sense mid to long term. For now, i'd want something that just does the job... ie, what is proposed here. We should absolutely document the semantics of vector_size properly.. it already is counterintuitive (broken, if you ask me).

Perhaps we could do both: support vector_size for bool using byte sizes (and not allowing subbyte vector lengths), and add a new, more general attribute that allows subbyte lengths and explicit subbyte element sizes.

Disallowing subbyte bool vectors actually makes this more complicated because these types are produced implicitly by compares of (legal) vector types. Consider this:

typedef int int3 __attribute__((vector_size(3 * sizeof(int))));
int3 A = ...;
int3 B = ...;
auto Z = A < B; // vector compare yielding a `bool vector_size(3)`-typed value.

Regarding your proposal for a separate subbyte element size and subbyte length, that may or may not make sense but it is surely something that should be discussed more broadly with its own proposal.

[rsandifo-arm]

clang/docs/LanguageExtensions.rst

473 ↗

(On Diff #268140)

Perhaps we could do both: support vector_size for bool using byte sizes (and not allowing subbyte vector lengths), and add a new, more general attribute that allows subbyte lengths and explicit subbyte element sizes.

Disallowing subbyte bool vectors actually makes this more complicated because these types are produced implicitly by compares of (legal) vector types. Consider this:

typedef int int3 __attribute__((vector_size(3 * sizeof(int))));
int3 A = ...;
int3 B = ...;
auto Z = A < B; // vector compare yielding a `bool vector_size(3)`-typed value.

Yeah, I understand the need for some way of creating subbyte vectors. I'm just not sure using the existing vector_size attribute would be the best choice for that case. I.e. the objection wasn't based on “keeping things simple” but more “keeping things consistent“.

That doesn't mean that the above code should be invalid. It just means that it wouldn't be possible to write the type of Z using the existing vector_size attribute.

(FWIW, vector_size was originally a GNUism and GCC stil requires vector sizes to be a power of 2, but I realise that isn't relevant here. And the same principle applies with s/3/4 in the above example anyway.)

Regarding your proposal for a separate subbyte element size and subbyte length, that may or may not make sense but it is surely something that should be discussed more broadly with its own proposal.

Yeah, I agree any new attribute would need to be discussed more widely.

[simoll]

clang/docs/LanguageExtensions.rst

473 ↗

(On Diff #268140)

Perhaps we could do both: support vector_size for bool using byte sizes (and not allowing subbyte vector lengths), and add a new, more general attribute that allows subbyte lengths and explicit subbyte element sizes.

Disallowing subbyte bool vectors actually makes this more complicated because these types are produced implicitly by compares of (legal) vector types. Consider this:

typedef int int3 __attribute__((vector_size(3 * sizeof(int))));
int3 A = ...;
int3 B = ...;
auto Z = A < B; // vector compare yielding a `bool vector_size(3)`-typed value.

Yeah, I understand the need for some way of creating subbyte vectors. I'm just not sure using the existing vector_size attribute would be the best choice for that case. I.e. the objection wasn't based on “keeping things simple” but more “keeping things consistent“.

That doesn't mean that the above code should be invalid. It just means that it wouldn't be possible to write the type of Z using the existing vector_size attribute.

IMHO being able to spell out every type ranks higher than being consistent with regards to a non-standard extension. That is, you could not do the assignment of A < B in C because there is no way to specify the type without auto or other C++ machinery.

(FWIW, vector_size was originally a GNUism and GCC stil requires vector sizes to be a power of 2, but I realise that isn't relevant here. And the same principle applies with s/3/4 in the above example anyway.)

Right, i overlooked the power-of-two constraint.

How much of a blocker are the subbyte sizes and the power-of-two constraint to you? I am asking because vector_size with those constraints would be good enough for us. Keeping the patch as it is mostly makes this extension potentially more useful to other SIMD/Vector users (and of course saves the work of changing it).
We could still lift that constraint (or switch to a new attribute) should the need arise.

[rsandifo-arm]

clang/docs/LanguageExtensions.rst

473 ↗

(On Diff #268140)

How much of a blocker are the subbyte sizes and the power-of-two constraint to you? I am asking because vector_size with those constraints would be good enough for us. Keeping the patch as it is mostly makes this extension potentially more useful to other SIMD/Vector users (and of course saves the work of changing it).
We could still lift that constraint (or switch to a new attribute) should the need arise.

The non-power-of-2 thing seems fine. It's simply removing a constraint and giving non-power-of-2 sizes their obvious meaning.

But I think changing the units in the vector_size is too surprising. I think a good indication is that we'd never have designed it like that if we were adding vector_size now. I think it's something that would often trip users up and that we'd have to keep explaining away.

It's just a personal opinion though.

[simoll]

clang/docs/LanguageExtensions.rst

473 ↗

(On Diff #268140)

But I think changing the units in the vector_size is too surprising. I think a good indication is that we'd never have designed it like that if we were adding vector_size now. I think it's something that would often trip users up and that we'd have to keep explaining away.

Ok. To make sure we are talking about the same thing here, you are suggesting:

typedef bool bool16 __attribute__((vector_size(2)));

Would be a vector of 16 bits stored in two bytes. Correct? If so, that's fine with me and i'll change the patch right away.

[rsandifo-arm]

clang/docs/LanguageExtensions.rst

473 ↗

(On Diff #268140)

But I think changing the units in the vector_size is too surprising. I think a good indication is that we'd never have designed it like that if we were adding vector_size now. I think it's something that would often trip users up and that we'd have to keep explaining away.

Ok. To make sure we are talking about the same thing here, you are suggesting:

typedef bool bool16 __attribute__((vector_size(2)));

Would be a vector of 16 bits stored in two bytes. Correct?

Yeah. And if anyone wants something like a vector of 4 bools in a 4-bit vector (which would certainly be reasonable in principle), we could leave that to some future, more general attribute syntax that doesn't have the same historical baggage as vector_size.

[simoll]

Thanks for the feedback!
Planned change: interpret the N in vector_size(N) as the number of bytes in the type, such that, for example, a vector_size(2) bool vector holds 16 bits

[simoll]

• Fixed debug info representation for bool vectors (and added debug info test).
• Interpret 8*N with the N of vector_size(N) as the numbers of bool elements of the vector.

[simoll]

NFC. Cleanup.

[simoll]

• Fixed type printing & added type printing test.
• Rebased.

[kaz7]

Thank you preparing this i1 patch and doing it on clang side only. We were testing this patch and sending problem reports. Now, we can use this patch without modifying llvm code. We can define vector mask types like below. Then, we can define intrinsic functions using this type. Those are converted to mask type, e.g v256i1, in the back end without adding bitcasts in ISelLowering.cpp. It helps developers very much. I hope this extension is accepted.

typedef double __vr __attribute__((__vector_size__(2048)));
#if __STDC_VERSION__ >= 199901L
// For C99
typedef _Bool __vm    __attribute__((__vector_size__(32)));
typedef _Bool __vm256 __attribute__((__vector_size__(32)));
typedef _Bool __vm512 __attribute__((__vector_size__(64)));
#else
#ifdef __cplusplus
// For C++
typedef bool __vm    __attribute__((__vector_size__(32)));
typedef bool __vm256 __attribute__((__vector_size__(32)));
typedef bool __vm512 __attribute__((__vector_size__(64)));
#else
#error need C++ or C99 to use vector intrinsics for VE
#endif
#endif

[rsandifo-arm]

I'm not qualified to review the CodeGen stuff (or accept the patch, obvs. :-)) but FWIW, here are some comments on the doc and Sema side.

It might be good to have more Sema tests for valid and invalid usage, e.g. for which operators are valid and which aren't.

Thanks again for doing this btw.

clang/docs/LanguageExtensions.rst

459 ↗

(On Diff #285282)

Maybe “Unlike GCC,”? Not much in it though.

489 ↗

(On Diff #285282)

~ is listed twice. I guess “operators” (without qualification) might make people think of the C++ keyword, in which case assignment and [] are allowed too.

492 ↗

(On Diff #285282)

It might be worth clarifying this. With the alignment referring specifically to “integer type”, I wasn't sure what something like:

bool __attribute__((vector_size(256)))

would mean on targets that don't provide 256-byte integer types. Is the type still 256-byte aligned?

clang/include/clang/AST/Type.h

3248 ↗

(On Diff #285282)

Maybe isGenericBooleanVector(), so that the terminology is consistent? Just a suggestion though, not sure if it's an improvement.

clang/lib/Sema/SemaExpr.cpp

9779 ↗

(On Diff #285282)

Is this deliberately wider than isVectorSizeBoolean(), or does it amount to the same thing?

11929 ↗

(On Diff #285282)

In practice, won't this either be a no-op (e.g. for 4-byte-per-bool targets) or always trump the fallback CharTy case? I wasn't sure why the case was needed.

11956 ↗

(On Diff #285282)

Seems like it might be useful to support comparisons too (with false < true, as for scalars). E.g. I guess x == y would otherwise need to be written ~(x ^ y), which seems less readable. Supporting more operators would also help in templated contexts.

clang/lib/Sema/SemaExprCXX.cpp

6271 ↗

(On Diff #285282)

Any reason not to support this for booleans? ?: seems like a natural operation for all vector element types.

[simoll]

clang/docs/LanguageExtensions.rst

492 ↗

(On Diff #285282)

Not exactly: It is the alignment of the corresponding integer type.
For example the following C code:

typedef bool bool256 __attribute__((vector_size(256)));
bool256 P;

gives you (x86_64 host, no machine flags specified):

%P = alloca i2048, align 32

clang/include/clang/AST/Type.h

3248 ↗

(On Diff #285282)

I gave it this specific name so where ever it is used, it documents that whatever depends on it is specific to vector_size bool. This should be cleaned up when a better boolean vector type is introduced.

clang/lib/Sema/SemaExpr.cpp

9779 ↗

(On Diff #285282)

That's an artifact. Will narrow this down to isVectorSizeBoolean().

11929 ↗

(On Diff #285282)

I guess so. Will remove this case.

11956 ↗

(On Diff #285282)

Agreed.

clang/lib/Sema/SemaExprCXX.cpp

6271 ↗

(On Diff #285282)

Sure, will allow it.

[rsandifo-arm]

clang/docs/LanguageExtensions.rst

492 ↗

(On Diff #285282)

I guess my point is pedantic, but what I meant was: does the concept of “the corresponding integer type” necessarily exist at the C/C++ level? E.g. how would you end up with the same LLVM IR statement using C integer types instead of vectors?

My worry was that “the alignment of the corresponding integer type” would only be meaningful to the user if they could identify what the corresponding (C) integer type actually was, and be able to determine its alignment that way.

[simoll]

clang/docs/LanguageExtensions.rst

492 ↗

(On Diff #285282)

To be honest, i hadn't given alignment much thought before. Actually, i am not too happy about bringing in integer types here.

I guess it's sensible to do no different than for the other vector types: take the size rounded up to the next power-ot-two capped by the target's maximum vector alignment.

[simoll]

• Rebased.
• Allow comparisons on boolean vectors.
• Restored result type for vector comparisons on other types.
• Added operator, alignment and constexpr tests.

[simoll]

In D81083#2227760, @rsandifo-arm wrote:

I'm not qualified to review the CodeGen stuff (or accept the patch, obvs. :-)) but FWIW, here are some comments on the doc and Sema side.

Thanks for your comments! Maybe you know people who could review the CodeGen side of this..?

[simoll]

The Hexagon builtins explicitly require bool vectors to have 8-bit wide bool elements. That clashes with our new vector type because bool vectors in builtin functions and vector_size bool vectors have the same internal representation in Clang. However, AFAIK Hexagon is the only target that actually need 8-bit bools. I see two possible ways out of this:

1. Introduce a Bit type in Clang that is only constructible as a vector element type. This way, any existing users of generic bool vectors are undisturbed. VE can use Bit in its builtin declarations to match the vector_size bool type. We may re-purpose the Bit type for a proper boolean vector type in the future.
2. Add a new target property: BoolInVectorSize that is the size of a bool element in boolean vectors. Hexagon, as most targets, would define BoolInVectorSize = BoolSize. VE would define BoolInVectorSize = 1 for the intended behavior.

My preference is for 2. since it means small changes to the current patch and less surprises than the other option. Opinions?

[simoll]

• made bool-vector layout target dependent (one byte per bool element on Hexagon, one bit everywhere else).
• Added hvx-specific debug info test.
• Updated docs.

[simoll]

NFC. Make clang-tidy happy.

[rsmith]

Have you considered using the ext_vector_type attribute instead of vector_size for this? That would have a couple of advantages:

• It's not a GCC attribute, so there's no risk that GCC would give different meaning to the same syntax (such as using one byte per vector element, which would be the natural meaning for a vector of bool as an extension of existing functionality).
• ext_vector_type takes as an argument a number of elements, not a size in bytes, so you could support vectors of non-multiple-of-8 bools.

However, using either an ext_vector_type of bool or a vector_size of bool would still seem problematic in a couple of other ways:

• Those types always produce a vector whose element type is the specified type; you can form an lvalue denoting the element, the element is always represented the same as the underlying type, and you can pun between a vector of N T and an array of N T. GCC even permits forming a pointer or reference to a vector element, and you should assume that Clang will eventually support that too. That would not be possible for bool if packed as a bit-vector.
• You can already perform logical operations between our existing vector types -- for example, comparing two vectors of 4 ints. If vectors of bool are valid, it would be logical and natural to expect a vector of 4 bools to have some connection to the result of such a comparison, but it won't: the type of a comparison of two vectors of 4 ints is a vector of 4 ints each of which is either 0 or -1, and we can't convert that to a vector of 4 bools represented as our existing vector types, because such conversions would be interpreted as bitcasts, and we'd reject because the bitwidth of the vectors is different.
• Vectors of integer types (such as bool) generally support arithmetic operators, which you presumably do not want to support here.
• Our existing vector types carry a lot of baggage (for example, lax vector conversions, and the egregious attribute-on-typedef-changes-the-type hack) that we would not want a new type to include.

So I think you should seriously consider using a new syntax to form a bit-vector type. It seems to me that an N-bit bit-vector is quite similar to an _ExtInt(N), but without the arithmetic support and with a different expected IR type. Maybe following that syntactic path and adding _BitVector(N) syntax would work out better?

[simoll]

In D81083#2237284, @rsmith wrote:

Thanks for dropping by!

Have you considered using the ext_vector_type attribute instead of vector_size for this? That would have a couple of advantages:

• It's not a GCC attribute, so there's no risk that GCC would give different meaning to the same syntax (such as using one byte per vector element, which would be the natural meaning for a vector of bool as an extension of existing functionality).
• ext_vector_type takes as an argument a number of elements, not a size in bytes, so you could support vectors of non-multiple-of-8 bools.

Whichever solution is going to land in Clang, we will end up in the situation that Clang supports bool vectors and GCC does not. Any code that uses this feature and is supposed to compile with GCC will need some #ifdef guard. This is why i am not too worried about (ab-)using the vector_size attribute for this.

However, using either an ext_vector_type of bool or a vector_size of bool would still seem problematic in a couple of other ways:

• Those types always produce a vector whose element type is the specified type; you can form an lvalue denoting the element, the element is always represented the same as the underlying type, and you can pun between a vector of N T and an array of N T. GCC even permits forming a pointer or reference to a vector element, and you should assume that Clang will eventually support that too. That would not be possible for bool if packed as a bit-vector.
• You can already perform logical operations between our existing vector types -- for example, comparing two vectors of 4 ints. If vectors of bool are valid, it would be logical and natural to expect a vector of 4 bools to have some connection to the result of such a comparison, but it won't: the type of a comparison of two vectors of 4 ints is a vector of 4 ints each of which is either 0 or -1, and we can't convert that to a vector of 4 bools represented as our existing vector types, because such conversions would be interpreted as bitcasts, and we'd reject because the bitwidth of the vectors is different.
• Vectors of integer types (such as bool) generally support arithmetic operators, which you presumably do not want to support here.
• Our existing vector types carry a lot of baggage (for example, lax vector conversions, and the egregious attribute-on-typedef-changes-the-type hack) that we would not want a new type to include.

I agree that the vector_size syntax for bool vectors raises some expectations on the behavior of the bool vector. However, i see this more as an issue of proper documentation. The moment somebody uses vector_size attributes they know they are deep in compiler-extension land and far from standard C/C++ semantics.
Now about the vector representation issues in Clang and how we could make this work for bool vectors: earlier, i suggested that we could introduce a Clang-internal Bit datatype that can only be used as a vector element type. By doing so, we wouldn't violate the invariant that bits(vector) == size_of_vector * bits(element_type).

So I think you should seriously consider using a new syntax to form a bit-vector type. It seems to me that an N-bit bit-vector is quite similar to an _ExtInt(N), but without the arithmetic support and with a different expected IR type. Maybe following that syntactic path and adding _BitVector(N) syntax would work out better?

I am open to a different syntax, if piggy backing on the vector_size syntax is the main point of contention. Up to now, the plan was to have vector_size bool now and work on a proper bool vector type to replace it.

[simoll]

Rebased

[simoll]

I'd like to hear your feedback on how to proceed with this patch.

Here is what i would do:

For this patch, we switch to ext_vector_type for bool vectors to not clash head on with GCC should GCC decide to allow vector_size bool with a different semantics in the future.
For the future, we propose a _BitVector(N) extension possibly aiming for a C/C++ TS. We can build on our experience with the ext_vector_type bool for this.

[simoll]

This comment was removed by simoll.

[simoll]

(wrong Diff attached in earlier update)

• rebased

[simoll]

D88905 uses ext_vector_type instead of vector_size following up on @rsmith 's suggestion. The review should continue there.

I'll keep D81083 around for reference.

[simoll]

Taking this off the review queues - we will use the ext_vector_type attribute instead (D88905).

llvm/include/llvm/Analysis/TargetTransformInfo.h

 class SCEV;
 class ScalarEvolution;
 class StoreInst;
 class SwitchInst;
 class TargetLibraryInfo;
 class Type;
 class User;
 class Value;
+class VPIntrinsic;
 struct KnownBits;
 template <typename T> class Optional;
 
 /// Information about a load/store intrinsic defined by the target.
 struct MemIntrinsicInfo {
   /// This is the pointer that the intrinsic is loading from or storing to.
   /// If this is non-null, then analysis/optimization passes can assume that
   /// this intrinsic is functionally equivalent to a load/store from this

 
   /// \name Vector Predication Information
   /// @{
   /// Whether the target supports the %evl parameter of VP intrinsic efficiently
   /// in hardware. (see LLVM Language Reference - "Vector Predication
   /// Intrinsics") Use of %evl is discouraged when that is not the case.
   bool hasActiveVectorLength() const;
 
+  struct VPLegalization {
+    enum VPTransform {
+      // keep the predicating parameter
+      Legal = 0,
+      // where legal, discard the predicate parameter
+      Discard = 1,
+      // transform into something else that is also predicating
+      Convert = 2
+    };
+
+    // How to transform the EVL parameter.
+    // Legal:   keep the EVL parameter as it is.
+    // Discard: Ignore the EVL parameter where it is safe to do so.
+    // Convert: Fold the EVL into the mask parameter.
+    VPTransform EVLParamStrategy;
+
+    // How to transform the operator.
+    // Legal:   The target supports this operator.
+    // Convert: Convert this to a non-VP operation.
+    // The 'Discard' strategy is invalid.
+    VPTransform OpStrategy;
+
+    bool doNothing() const {
+      return (EVLParamStrategy == Legal) && (OpStrategy == Legal);
+    }
+    VPLegalization(VPTransform EVLParamStrategy, VPTransform OpStrategy)
+        : EVLParamStrategy(EVLParamStrategy), OpStrategy(OpStrategy) {}
+  };
+
+  /// \returns How the target needs this vector-predicated operation to be
+  /// transformed.
+  VPLegalization getVPLegalizationStrategy(const VPIntrinsic &PI) const;
   /// @}
 
   /// @}
 
 private:
   /// Estimate the latency of specified instruction.
   /// Returns 1 as the default value.
   int getInstructionLatency(const Instruction *I) const;
virtual bool useReductionIntrinsic(unsigned Opcode, Type *Ty,
                                      ReductionFlags) const = 0;
   virtual bool preferInLoopReduction(unsigned Opcode, Type *Ty,
                                      ReductionFlags) const = 0;
   virtual bool preferPredicatedReductionSelect(unsigned Opcode, Type *Ty,
                                                ReductionFlags) const = 0;
   virtual bool shouldExpandReduction(const IntrinsicInst *II) const = 0;
   virtual unsigned getGISelRematGlobalCost() const = 0;
   virtual bool hasActiveVectorLength() const = 0;
+  virtual VPLegalization
+  getVPLegalizationStrategy(const VPIntrinsic &PI) const = 0;
   virtual int getInstructionLatency(const Instruction *I) = 0;
 };
 
 template <typename T>
 class TargetTransformInfo::Model final : public TargetTransformInfo::Concept {
   T Impl;
 
 public:
public:
   unsigned getGISelRematGlobalCost() const override {
     return Impl.getGISelRematGlobalCost();
   }
 
   bool hasActiveVectorLength() const override {
     return Impl.hasActiveVectorLength();
   }
 
+  VPLegalization
+  getVPLegalizationStrategy(const VPIntrinsic &PI) const override {
+    return Impl.getVPLegalizationStrategy(PI);
+  }
+
   int getInstructionLatency(const Instruction *I) override {
     return Impl.getInstructionLatency(I);
   }
 };
 
 template <typename T>
 TargetTransformInfo::TargetTransformInfo(T Impl)
     : TTIImpl(new Model<T>(Impl)) {}

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

public:
   }
 
   bool shouldExpandReduction(const IntrinsicInst *II) const { return true; }
 
   unsigned getGISelRematGlobalCost() const { return 1; }
 
   bool hasActiveVectorLength() const { return false; }
 
+  TargetTransformInfo::VPLegalization
+  getVPLegalizationStrategy(const VPIntrinsic &PI) const {
+    return TargetTransformInfo::VPLegalization(
+        /* EVLParamStrategy */ TargetTransformInfo::VPLegalization::Discard,
+        /* OperatorStrategy */ TargetTransformInfo::VPLegalization::Convert);
+  }
+
 protected:
   // Obtain the minimum required size to hold the value (without the sign)
   // In case of a vector it returns the min required size for one element.
   unsigned minRequiredElementSize(const Value *Val, bool &isSigned) {
     if (isa<ConstantDataVector>(Val) || isa<ConstantVector>(Val)) {
       const auto *VectorValue = cast<Constant>(Val);
 
       // In case of a vector need to pick the max between the min

llvm/include/llvm/CodeGen/ExpandVectorPredication.h

+//===-- ExpandVectorPredication.h - Expand vector predication ---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CODEGEN_EXPANDVECTORPREDICATION_H
+#define LLVM_CODEGEN_EXPANDVECTORPREDICATION_H
+
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+class ExpandVectorPredicationPass
+    : public PassInfoMixin<ExpandVectorPredicationPass> {
+public:
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
+};
+} // end namespace llvm
+
+#endif // LLVM_CODEGEN_EXPANDVECTORPREDICATION_H

llvm/include/llvm/CodeGen/Passes.h

/// MachineDominanaceFrontier - This pass is a machine dominators analysis pass.
   /// This pass performs outlining on machine instructions directly before
   /// printing assembly.
   ModulePass *createMachineOutlinerPass(bool RunOnAllFunctions = true);
 
   /// This pass expands the experimental reduction intrinsics into sequences of
   /// shuffles.
   FunctionPass *createExpandReductionsPass();
 
+  /// This pass expands the vector predication intrinsics into unpredicated
+  /// instructions with selects or just the explicit vector length into the
+  /// predicate mask.
+  FunctionPass *createExpandVectorPredicationPass();
+
   // This pass expands memcmp() to load/stores.
   FunctionPass *createExpandMemCmpPass();
 
   /// Creates Break False Dependencies pass. \see BreakFalseDeps.cpp
   FunctionPass *createBreakFalseDeps();
 
   // This pass expands indirectbr instructions.
   FunctionPass *createIndirectBrExpandPass();

llvm/include/llvm/IR/IntrinsicInst.h

public:
   /// The llvm.vp.* intrinsics for this instruction Opcode
   static Intrinsic::ID GetForOpcode(unsigned OC);
 
   // Whether \p ID is a VP intrinsic ID.
   static bool IsVPIntrinsic(Intrinsic::ID);
 
   /// \return the mask parameter or nullptr.
   Value *getMaskParam() const;
+  void setMaskParam(Value *);
 
   /// \return the vector length parameter or nullptr.
   Value *getVectorLengthParam() const;
+  void setVectorLengthParam(Value *);
 
   /// \return whether the vector length param can be ignored.
   bool canIgnoreVectorLengthParam() const;
 
   /// \return the static element count (vector number of elements) the vector
   /// length parameter applies to.
   ElementCount getStaticVectorLength() const;
 

llvm/include/llvm/InitializePasses.h

 void initializeEarlyMachineLICMPass(PassRegistry&);
 void initializeEarlyTailDuplicatePass(PassRegistry&);
 void initializeEdgeBundlesPass(PassRegistry&);
 void initializeEliminateAvailableExternallyLegacyPassPass(PassRegistry&);
 void initializeEntryExitInstrumenterPass(PassRegistry&);
 void initializeExpandMemCmpPassPass(PassRegistry&);
 void initializeExpandPostRAPass(PassRegistry&);
 void initializeExpandReductionsPass(PassRegistry&);
+void initializeExpandVectorPredicationPass(PassRegistry &);
 void initializeMakeGuardsExplicitLegacyPassPass(PassRegistry&);
 void initializeExternalAAWrapperPassPass(PassRegistry&);
 void initializeFEntryInserterPass(PassRegistry&);
 void initializeFinalizeISelPass(PassRegistry&);
 void initializeFinalizeMachineBundlesPass(PassRegistry&);
 void initializeFixIrreduciblePass(PassRegistry &);
 void initializeFixupStatepointCallerSavedPass(PassRegistry&);
 void initializeFlattenCFGPassPass(PassRegistry&);

llvm/lib/Analysis/TargetTransformInfo.cpp

bool TargetTransformInfo::preferInLoopReduction(unsigned Opcode, Type *Ty,
   return TTIImpl->preferInLoopReduction(Opcode, Ty, Flags);
 }
 
 bool TargetTransformInfo::preferPredicatedReductionSelect(
     unsigned Opcode, Type *Ty, ReductionFlags Flags) const {
   return TTIImpl->preferPredicatedReductionSelect(Opcode, Ty, Flags);
 }
 
+TargetTransformInfo::VPLegalization
+TargetTransformInfo::getVPLegalizationStrategy(const VPIntrinsic &VPI) const {
+  return TTIImpl->getVPLegalizationStrategy(VPI);
+}
+
 bool TargetTransformInfo::shouldExpandReduction(const IntrinsicInst *II) const {
   return TTIImpl->shouldExpandReduction(II);
 }
 
 unsigned TargetTransformInfo::getGISelRematGlobalCost() const {
   return TTIImpl->getGISelRematGlobalCost();
 }
 

llvm/lib/CodeGen/CMakeLists.txt

add_llvm_component_library(LLVMCodeGen
   DFAPacketizer.cpp
   DwarfEHPrepare.cpp
   EarlyIfConversion.cpp
   EdgeBundles.cpp
   ExecutionDomainFix.cpp
   ExpandMemCmp.cpp
   ExpandPostRAPseudos.cpp
   ExpandReductions.cpp
+  ExpandVectorPredication.cpp
   FaultMaps.cpp
   FEntryInserter.cpp
   FinalizeISel.cpp
   FixupStatepointCallerSaved.cpp
   FuncletLayout.cpp
   GCMetadata.cpp
   GCMetadataPrinter.cpp
   GCRootLowering.cpp

llvm/lib/CodeGen/ExpandVectorPredication.cpp

+//===--- CodeGen/ExpandVectorPredication.cpp - Expand VP intrinsics -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass implements IR expansion for vector predication intrinsics, allowing
+// targets to enable vector predication until just before codegen.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/CodeGen/ExpandVectorPredication.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/Analysis/TargetTransformInfo.h"
+#include "llvm/Analysis/ValueTracking.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Intrinsics.h"
+#include "llvm/IR/Module.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/MathExtras.h"
+
+using namespace llvm;
+
+using VPLegalization = TargetTransformInfo::VPLegalization;
+
+#define DEBUG_TYPE "expand-vec-pred"
+
+STATISTIC(NumFoldedVL, "Number of folded vector length params");
+STATISTIC(NumLoweredVPOps, "Number of folded vector predication operations");
+
+///// Helpers {
+
+/// \returns Whether the vector mask \p MaskVal has all lane bits set.
+static bool isAllTrueMask(Value *MaskVal) {
+  auto *ConstVec = dyn_cast<ConstantVector>(MaskVal);
+  if (!ConstVec)
+    return false;
+  return ConstVec->isAllOnesValue();
+}
+
+/// Computes the smallest integer bit width to hold the step vector <0, ..,
+/// NumVectorElements - 1>
+static unsigned getLeastLaneBitsForStepVector(unsigned NumVectorElements) {
+  unsigned MostSignificantOne =
+      llvm::countLeadingZeros<uint64_t>(NumVectorElements, ZB_Undefined);
+  return std::max<unsigned>(IntegerType::MIN_INT_BITS, 64 - MostSignificantOne);
+}
+
+/// \returns A non-excepting divisor constant for this type.
+static Constant *getSafeDivisor(Type *DivTy) {
+  assert(DivTy->isIntOrIntVectorTy());
+  return ConstantInt::get(DivTy, 1u, false);
+}
+
+/// Transfer operation properties from \p OldVPI to \p NewVal.
+static void transferDecorations(Value &NewVal, VPIntrinsic &VPI) {
+  auto *NewInst = dyn_cast<Instruction>(&NewVal);
+  if (!NewInst || !isa<FPMathOperator>(NewVal))
+    return;
+
+  auto *OldFMOp = dyn_cast<FPMathOperator>(&VPI);
+  if (!OldFMOp)
+    return;
+
+  NewInst->setFastMathFlags(OldFMOp->getFastMathFlags());
+}
+
+/// Transfer all properties from \p OldOp to \p NewOp and replace all uses.
+/// OldVP gets erased.
+static void replaceOperation(Value &NewOp, VPIntrinsic &OldOp) {
+  transferDecorations(NewOp, OldOp);
+  OldOp.replaceAllUsesWith(&NewOp);
+  OldOp.eraseFromParent();
+}
+
+//// } Helpers
+
+namespace {
+
+// Expansion pass state at function scope.
+struct CachingVPExpander {
+  Function &F;
+  const TargetTransformInfo &TTI;
+
+  /// \returns A (fixed length) vector with ascending integer indices
+  /// (<0, 1, ..., NumElems-1>).
+  Value *createStepVector(IRBuilder<> &Builder, int32_t ElemBits,
+                          int32_t NumElems);
+
+  /// \returns A bitmask that is true where the lane position is less-than \p
+  /// EVLParam
+  ///
+  /// \p Builder
+  ///    Used for instruction creation.
+  /// \p VLParam
+  ///    The explicit vector length parameter to test against the lane
+  ///    positions.
+  /// \p ElemCount
+  ///    Static (potentially scalable) number of vector elements
+  Value *convertEVLToMask(IRBuilder<> &Builder, Value *EVLParam,
+                          ElementCount ElemCount);
+
+  Value *foldEVLIntoMask(VPIntrinsic &VPI);
+
+  /// "Remove" the %evl parameter of \p PI by setting it to the static vector
+  /// length of the operation.
+  void discardEVLParameter(VPIntrinsic &PI);
+
+  /// \brief Lower this VP binary operator to a non-VP binary operator.
+  Value *expandPredicationInBinaryOperator(IRBuilder<> &Builder,
+                                           VPIntrinsic &PI);
+
+  /// \brief query TTI and expand the vector predication in \p P accordingly.
+  Value *expandPredication(VPIntrinsic &PI);
+
+  /// \brief return a good (for fast icmp) integer bit width to expand
+  /// the EVL comparison against the stepvector in.
+  std::map<unsigned, unsigned> StaticVLToBitsCache; // TODO 'SmallMap' class
+  unsigned getLaneBitsForEVLCompare(unsigned StaticVL);
+
+public:
+  CachingVPExpander(Function &F, const TargetTransformInfo &TTI)
+      : F(F), TTI(TTI) {}
+
+  // expand VP ops in \p F according to \p TTI.
+  bool expandVectorPredication();
+};
+
+//// CachingVPExpander {
+
+unsigned CachingVPExpander::getLaneBitsForEVLCompare(unsigned StaticVL) {
+  auto ItCached = StaticVLToBitsCache.find(StaticVL);
+  if (ItCached != StaticVLToBitsCache.end())
+    return ItCached->second;
+
+  // The smallest integer to hold <0, .., ElemCount.Min -1>
+  // Cannot choose less bits than this or the expansion will be invalid.
+  unsigned MinLaneBits = getLeastLaneBitsForStepVector(StaticVL);
+  LLVM_DEBUG(dbgs() << "Least lane bits for " << StaticVL << " is "
+                    << MinLaneBits << "\n";);
+
+  // If  the EVL compare will be expanded into scalar code, choose the
+  // smallest integer type.
+  if (TTI.getRegisterBitWidth(/* Vector */ true) == 0)
+    return MinLaneBits;
+
+  // Otw, the generated vector operation will likely map to vector instructions.
+  // The largest bit width to fit the EVL expansion in one vector register.
+  unsigned MaxLaneBits = std::min<unsigned>(
+      IntegerType::MAX_INT_BITS, TTI.getRegisterBitWidth(true) / StaticVL);
+
+  // Many SIMD instruction are restricted in their supported lane bit widths.
+  // We choose the bit width that gives us the cheapest vector compare.
+  int Cheapest = std::numeric_limits<int>::max();
+  auto &Ctx = F.getContext();
+  unsigned CheapestLaneBits = MinLaneBits;
+  for (auto LaneBits = MinLaneBits; LaneBits < MaxLaneBits; ++LaneBits) {
+    int VecCmpCost = TTI.getCmpSelInstrCost(
+        Instruction::ICmp, VectorType::get(Type::getIntNTy(Ctx, LaneBits),
+                                           StaticVL, /* Scalable */ false));
+    if (VecCmpCost < Cheapest) {
+      Cheapest = VecCmpCost;
+      CheapestLaneBits = LaneBits;
+    }
+  }
+
+  StaticVLToBitsCache[StaticVL] = CheapestLaneBits;
+  return CheapestLaneBits;
+}
+
+Value *CachingVPExpander::createStepVector(IRBuilder<> &Builder,
+                                           int32_t ElemBits, int32_t NumElems) {
+  // TODO add caching
+  SmallVector<Constant *, 16> ConstElems;
+
+  Type *LaneTy = Builder.getIntNTy(ElemBits);
+
+  for (int32_t Idx = 0; Idx < NumElems; ++Idx) {
+    ConstElems.push_back(ConstantInt::get(LaneTy, Idx, false));
+  }
+
+  return ConstantVector::get(ConstElems);
+}
+
+Value *CachingVPExpander::convertEVLToMask(IRBuilder<> &Builder,
+                                           Value *EVLParam,
+                                           ElementCount ElemCount) {
+  // TODO add caching
+  if (ElemCount.isScalable()) {
+    auto *M = Builder.GetInsertBlock()->getModule();
+    auto *BoolVecTy = VectorType::get(Builder.getInt1Ty(), ElemCount);
+    auto *ActiveMaskFunc = Intrinsic::getDeclaration(
+        M, Intrinsic::get_active_lane_mask, {BoolVecTy, EVLParam->getType()});
+    // `get_active_lane_mask` performs an implicit less-than comparison.
+    auto *ConstZero = Builder.getInt32(0);
+    return Builder.CreateCall(ActiveMaskFunc, {ConstZero, EVLParam});
+  }
+
+  unsigned NumElems = ElemCount.getFixedValue();
+  unsigned ElemBits = getLaneBitsForEVLCompare(NumElems);
+
+  Type *LaneTy = Builder.getIntNTy(ElemBits);
+
+  auto *ExtVLParam = Builder.CreateZExtOrTrunc(EVLParam, LaneTy);
+  auto *VLSplat = Builder.CreateVectorSplat(NumElems, ExtVLParam);
+
+  auto *IdxVec = createStepVector(Builder, ElemBits, NumElems);
+
+  return Builder.CreateICmp(CmpInst::ICMP_ULT, IdxVec, VLSplat);
+}
+
+Value *
+CachingVPExpander::expandPredicationInBinaryOperator(IRBuilder<> &Builder,
+                                                     VPIntrinsic &VPI) {
+  assert(VPI.canIgnoreVectorLengthParam());
+
+  auto OC = static_cast<Instruction::BinaryOps>(VPI.getFunctionalOpcode());
+  assert(Instruction::isBinaryOp(OC));
+
+  auto *FirstOp = VPI.getOperand(0);
+  auto *SndOp = VPI.getOperand(1);
+
+  auto *Mask = VPI.getMaskParam();
+
+  // Blend in safe operands
+  if (Mask && !isAllTrueMask(Mask)) {
+    switch (OC) {
+    default:
+      // can safely ignore the predicate
+      break;
+
+    // Division operators need a safe divisor on masked-off lanes (1)
+    case Instruction::UDiv:
+    case Instruction::SDiv:
+    case Instruction::URem:
+    case Instruction::SRem:
+      // 2nd operand must not be zero
+      auto *SafeDivisor = getSafeDivisor(VPI.getType());
+      SndOp = Builder.CreateSelect(Mask, SndOp, SafeDivisor);
+    }
+  }
+
+  auto *NewBinOp = Builder.CreateBinOp(OC, FirstOp, SndOp, VPI.getName());
+
+  replaceOperation(*NewBinOp, VPI);
+  return NewBinOp;
+}
+
+void CachingVPExpander::discardEVLParameter(VPIntrinsic &VPI) {
+  LLVM_DEBUG(dbgs() << "Discard EVL parameter in " << VPI << "\n");
+
+  if (VPI.canIgnoreVectorLengthParam())
+    return;
+
+  Value *EVLParam = VPI.getVectorLengthParam();
+  if (!EVLParam)
+    return;
+
+  ElementCount StaticElemCount = VPI.getStaticVectorLength();
+  Value *MaxEVL = nullptr;
+  auto *Int32Ty = Type::getInt32Ty(VPI.getContext());
+  if (StaticElemCount.isScalable()) {
+    // TODO add caching
+    auto *M = VPI.getModule();
+    auto *VScaleFunc = Intrinsic::getDeclaration(M, Intrinsic::vscale, Int32Ty);
+    IRBuilder<> Builder(VPI.getParent(), VPI.getIterator());
+    auto *FactorConst = Builder.getInt32(StaticElemCount.getKnownMinValue());
+    auto *VScale = Builder.CreateCall(VScaleFunc, {}, "vscale");
+    MaxEVL = Builder.CreateMul(VScale, FactorConst, "scalable_size",
+                               /*NUW*/ true, /*NSW*/ false);
+  } else {
+    MaxEVL = ConstantInt::get(Int32Ty, StaticElemCount.getFixedValue(), false);
+  }
+  VPI.setVectorLengthParam(MaxEVL);
+}
+
+Value *CachingVPExpander::foldEVLIntoMask(VPIntrinsic &VPI) {
+  LLVM_DEBUG(dbgs() << "Folding vlen for " << VPI << '\n');
+
+  IRBuilder<> Builder(&VPI);
+
+  // No %evl parameter and so nothing to do here
+  if (VPI.canIgnoreVectorLengthParam()) {
+    return &VPI;
+  }
+
+  // Only VP intrinsics can have a %evl parameter
+  Value *OldMaskParam = VPI.getMaskParam();
+  Value *OldEVLParam = VPI.getVectorLengthParam();
+  assert(OldMaskParam && "no mask param to fold the vl param into");
+  assert(OldEVLParam && "no EVL param to fold away");
+
+  LLVM_DEBUG(dbgs() << "OLD evl: " << *OldEVLParam << '\n');
+  LLVM_DEBUG(dbgs() << "OLD mask: " << *OldMaskParam << '\n');
+
+  // Convert the %evl predication into vector mask predication.
+  ElementCount ElemCount = VPI.getStaticVectorLength();
+  auto *VLMask = convertEVLToMask(Builder, OldEVLParam, ElemCount);
+  auto *NewMaskParam = Builder.CreateAnd(VLMask, OldMaskParam);
+  VPI.setMaskParam(NewMaskParam);
+
+  // Drop the EVl parameter
+  discardEVLParameter(VPI);
+  assert(VPI.canIgnoreVectorLengthParam() &&
+         "transformation did not render the evl param ineffective!");
+
+  // re-asses the modified instruction
+  return &VPI;
+}
+
+Value *CachingVPExpander::expandPredication(VPIntrinsic &VPI) {
+  LLVM_DEBUG(dbgs() << "Lowering to unpredicated op: " << VPI << '\n');
+
+  IRBuilder<> Builder(&VPI);
+
+  // Try lowering to a LLVM instruction first.
+  unsigned OC = VPI.getFunctionalOpcode();
+#define FIRST_BINARY_INST(X) unsigned FirstBinOp = X;
+#define LAST_BINARY_INST(X) unsigned LastBinOp = X;
+#include "llvm/IR/Instruction.def"
+
+  if (FirstBinOp <= OC && OC <= LastBinOp) {
+    return expandPredicationInBinaryOperator(Builder, VPI);
+  }
+
+  return &VPI;
+}
+
+//// } CachingVPExpander
+
+struct TransformJob {
+  VPIntrinsic *PI;
+  TargetTransformInfo::VPLegalization Strategy;
+  TransformJob(VPIntrinsic *PI, TargetTransformInfo::VPLegalization InitStrat)
+      : PI(PI), Strategy(InitStrat) {}
+
+  bool isDone() const { return Strategy.doNothing(); }
+};
+
+void sanitizeStrategy(Instruction &I, VPLegalization &LegalizeStrat) {
+  // Speculatable instructions do not strictle need predication.
+  if (isSafeToSpeculativelyExecute(&I))
+    return;
+
+  // Preserve the predication effect of the EVL parameter by folding
+  // it into the predicate.
+  if (LegalizeStrat.EVLParamStrategy == VPLegalization::Discard) {
+    LegalizeStrat.EVLParamStrategy = VPLegalization::Convert;
+  }
+}
+
+/// \brief Expand llvm.vp.* intrinsics as requested by \p TTI.
+bool CachingVPExpander::expandVectorPredication() {
+  // Holds all vector-predicated ops with an effective vector length param that
+  SmallVector<TransformJob, 16> Worklist;
+
+  for (auto &I : instructions(F)) {
+    auto *VPI = dyn_cast<VPIntrinsic>(&I);
+    if (!VPI)
+      continue;
+    auto VPStrat = TTI.getVPLegalizationStrategy(*VPI);
+    sanitizeStrategy(I, VPStrat);
+    if (!VPStrat.doNothing()) {
+      Worklist.emplace_back(VPI, VPStrat);
+    }
+  }
+  if (Worklist.empty())
+    return false;
+
+  LLVM_DEBUG(dbgs() << "\n:::: Transforming instructions. ::::\n");
+  for (TransformJob Job : Worklist) {
+    // Transform the EVL parameter
+    switch (Job.Strategy.EVLParamStrategy) {
+    case VPLegalization::Legal:
+      break;
+    case VPLegalization::Discard: {
+      discardEVLParameter(*Job.PI);
+    } break;
+    case VPLegalization::Convert: {
+      if (foldEVLIntoMask(*Job.PI)) {
+        ++NumFoldedVL;
+      }
+    } break;
+    }
+    Job.Strategy.EVLParamStrategy = VPLegalization::Legal;
+
+    // Replace the operator
+    switch (Job.Strategy.OpStrategy) {
+    case VPLegalization::Legal:
+      break;
+    case VPLegalization::Discard:
+      llvm_unreachable("Invalid strategy for operators.");
+    case VPLegalization::Convert: {
+      expandPredication(*Job.PI);
+      ++NumLoweredVPOps;
+    } break;
+    }
+    Job.Strategy.OpStrategy = VPLegalization::Legal;
+
+    assert(Job.isDone() && "incomplete transformation");
+  }
+
+  return true;
+}
+class ExpandVectorPredication : public FunctionPass {
+public:
+  static char ID;
+  ExpandVectorPredication() : FunctionPass(ID) {
+    initializeExpandVectorPredicationPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnFunction(Function &F) override {
+    const auto *TTI = &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
+    CachingVPExpander VPExpander(F, *TTI);
+    return VPExpander.expandVectorPredication();
+  }
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.addRequired<TargetTransformInfoWrapperPass>();
+    AU.setPreservesCFG();
+  }
+};
+} // namespace
+
+char ExpandVectorPredication::ID;
+INITIALIZE_PASS_BEGIN(ExpandVectorPredication, "expand-vec-pred",
+                      "Expand vector predication intrinsics", false, false)
+INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
+INITIALIZE_PASS_END(ExpandVectorPredication, "expand-vec-pred",
+                    "Expand vector predication intrinsics", false, false)
+
+FunctionPass *llvm::createExpandVectorPredicationPass() {
+  return new ExpandVectorPredication();
+}
+
+PreservedAnalyses
+ExpandVectorPredicationPass::run(Function &F, FunctionAnalysisManager &AM) {
+  const auto &TTI = AM.getResult<TargetIRAnalysis>(F);
+  CachingVPExpander VPExpander(F, TTI);
+  if (!VPExpander.expandVectorPredication())
+    return PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
+}

llvm/lib/CodeGen/TargetPassConfig.cpp

if (getOptLevel() != CodeGenOpt::None && !DisableConstantHoisting)
     addPass(createConstantHoistingPass());
 
   if (getOptLevel() != CodeGenOpt::None && !DisablePartialLibcallInlining)
     addPass(createPartiallyInlineLibCallsPass());
 
   // Instrument function entry and exit, e.g. with calls to mcount().
   addPass(createPostInlineEntryExitInstrumenterPass());
 
+  // Expand vector predication intrinsics into standard IR instructions.
+  // This pass has to run before ScalarizeMaskedMemIntrin and ExpandReduction
+  // passes since it emits those kinds of intrinsics.
+  addPass(createExpandVectorPredicationPass());
+
   // Add scalarization of target's unsupported masked memory intrinsics pass.
   // the unsupported intrinsic will be replaced with a chain of basic blocks,
   // that stores/loads element one-by-one if the appropriate mask bit is set.
   addPass(createScalarizeMaskedMemIntrinPass());
 
   // Expand reduction intrinsics into shuffle sequences if the target wants to.
   addPass(createExpandReductionsPass());
 }

llvm/lib/IR/IntrinsicInst.cpp

 
 Value *VPIntrinsic::getMaskParam() const {
   auto maskPos = GetMaskParamPos(getIntrinsicID());
   if (maskPos)
     return getArgOperand(maskPos.getValue());
   return nullptr;
 }
 
+void VPIntrinsic::setMaskParam(Value *NewMask) {
+  auto MaskPos = GetMaskParamPos(getIntrinsicID());
+  assert(MaskPos.hasValue());
+  setArgOperand(MaskPos.getValue(), NewMask);
+}
+
 Value *VPIntrinsic::getVectorLengthParam() const {
   auto vlenPos = GetVectorLengthParamPos(getIntrinsicID());
   if (vlenPos)
     return getArgOperand(vlenPos.getValue());
   return nullptr;
 }
 
+void VPIntrinsic::setVectorLengthParam(Value *NewEVL) {
+  auto EVLPos = GetVectorLengthParamPos(getIntrinsicID());
+  assert(EVLPos.hasValue());
+  setArgOperand(EVLPos.getValue(), NewEVL);
+}
+
 Optional<int> VPIntrinsic::GetMaskParamPos(Intrinsic::ID IntrinsicID) {
   switch (IntrinsicID) {
   default:
     return None;
 
 #define REGISTER_VP_INTRINSIC(VPID, MASKPOS, VLENPOS)                          \
   case Intrinsic::VPID:                                                        \
     return MASKPOS;

llvm/test/CodeGen/AArch64/O0-pipeline.ll

 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Expand Atomic instructions
 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       Lower Garbage Collection Instructions
 ; CHECK-NEXT:       Shadow Stack GC Lowering
 ; CHECK-NEXT:       Lower constant intrinsics
 ; CHECK-NEXT:       Remove unreachable blocks from the CFG
 ; CHECK-NEXT:       Instrument function entry/exit with calls to e.g. mcount() (post inlining)
+; CHECK-NEXT:       Expand vector predication intrinsics
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:       AArch64 Stack Tagging
 ; CHECK-NEXT:     Rewrite Symbols
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Exception handling preparation
 ; CHECK-NEXT:       Safe Stack instrumentation pass
 ; CHECK-NEXT:       Insert stack protectors

llvm/test/CodeGen/AArch64/O3-pipeline.ll

 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Post-Dominator Tree Construction
 ; CHECK-NEXT:       Branch Probability Analysis
 ; CHECK-NEXT:       Block Frequency Analysis
 ; CHECK-NEXT:       Constant Hoisting
 ; CHECK-NEXT:       Partially inline calls to library functions
 ; CHECK-NEXT:       Instrument function entry/exit with calls to e.g. mcount() (post inlining)
+; CHECK-NEXT:       Expand vector predication intrinsics
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:     Stack Safety Analysis
 ; CHECK-NEXT:       FunctionPass Manager
 ; CHECK-NEXT:         Dominator Tree Construction
 ; CHECK-NEXT:         Natural Loop Information
 ; CHECK-NEXT:         Scalar Evolution Analysis
 ; CHECK-NEXT:         Stack Safety Local Analysis

llvm/test/CodeGen/ARM/O3-pipeline.ll

 ; CHECK-NEXT:      Dominator Tree Construction
 ; CHECK-NEXT:      Natural Loop Information
 ; CHECK-NEXT:      Post-Dominator Tree Construction
 ; CHECK-NEXT:      Branch Probability Analysis
 ; CHECK-NEXT:      Block Frequency Analysis
 ; CHECK-NEXT:      Constant Hoisting
 ; CHECK-NEXT:      Partially inline calls to library functions
 ; CHECK-NEXT:      Instrument function entry/exit with calls to e.g. mcount() (post inlining)
+; CHECK-NEXT:      Expand vector predication intrinsics
 ; CHECK-NEXT:      Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:      Expand reduction intrinsics
 ; CHECK-NEXT:      Dominator Tree Construction
 ; CHECK-NEXT:      Natural Loop Information
 ; CHECK-NEXT:      Scalar Evolution Analysis
 ; CHECK-NEXT:      Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:      Function Alias Analysis Results
 ; CHECK-NEXT:      Transform functions to use DSP intrinsics

llvm/test/CodeGen/Generic/expand-vp.ll

+; RUN: opt --expand-vec-pred -S < %s | FileCheck %s
+
+; All VP intrinsics have to be lowered into non-VP ops
+; CHECK-NOT: {{call.* @llvm.vp.add}}
+; CHECK-NOT: {{call.* @llvm.vp.sub}}
+; CHECK-NOT: {{call.* @llvm.vp.mul}}
+; CHECK-NOT: {{call.* @llvm.vp.sdiv}}
+; CHECK-NOT: {{call.* @llvm.vp.srem}}
+; CHECK-NOT: {{call.* @llvm.vp.udiv}}
+; CHECK-NOT: {{call.* @llvm.vp.urem}}
+; CHECK-NOT: {{call.* @llvm.vp.and}}
+; CHECK-NOT: {{call.* @llvm.vp.or}}
+; CHECK-NOT: {{call.* @llvm.vp.xor}}
+; CHECK-NOT: {{call.* @llvm.vp.ashr}}
+; CHECK-NOT: {{call.* @llvm.vp.lshr}}
+; CHECK-NOT: {{call.* @llvm.vp.shl}}
+
+define void @test_vp_int_v8(<8 x i32> %i0, <8 x i32> %i1, <8 x i32> %i2, <8 x i32> %f3, <8 x i1> %m, i32 %n) {
+  %r0 = call <8 x i32> @llvm.vp.add.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r1 = call <8 x i32> @llvm.vp.sub.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r2 = call <8 x i32> @llvm.vp.mul.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r3 = call <8 x i32> @llvm.vp.sdiv.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r4 = call <8 x i32> @llvm.vp.srem.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r5 = call <8 x i32> @llvm.vp.udiv.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r6 = call <8 x i32> @llvm.vp.urem.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r7 = call <8 x i32> @llvm.vp.and.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r8 = call <8 x i32> @llvm.vp.or.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %r9 = call <8 x i32> @llvm.vp.xor.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %rA = call <8 x i32> @llvm.vp.ashr.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %rB = call <8 x i32> @llvm.vp.lshr.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  %rC = call <8 x i32> @llvm.vp.shl.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
+  ret void
+}
+
+; fixed-width vectors
+; integer arith
+declare <8 x i32> @llvm.vp.add.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.sub.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.mul.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.sdiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.srem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.udiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.urem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+; bit arith
+declare <8 x i32> @llvm.vp.and.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.xor.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.or.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.ashr.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.lshr.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+declare <8 x i32> @llvm.vp.shl.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+
+define void @test_vp_int_vscale(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i32> %i2, <vscale x 4 x i32> %f3, <vscale x 4 x i1> %m, i32 %n) {
+  %r0 = call <vscale x 4 x i32> @llvm.vp.add.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r1 = call <vscale x 4 x i32> @llvm.vp.sub.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r2 = call <vscale x 4 x i32> @llvm.vp.mul.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r3 = call <vscale x 4 x i32> @llvm.vp.sdiv.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r4 = call <vscale x 4 x i32> @llvm.vp.srem.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r5 = call <vscale x 4 x i32> @llvm.vp.udiv.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r6 = call <vscale x 4 x i32> @llvm.vp.urem.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r7 = call <vscale x 4 x i32> @llvm.vp.and.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r8 = call <vscale x 4 x i32> @llvm.vp.or.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %r9 = call <vscale x 4 x i32> @llvm.vp.xor.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %rA = call <vscale x 4 x i32> @llvm.vp.ashr.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %rB = call <vscale x 4 x i32> @llvm.vp.lshr.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  %rC = call <vscale x 4 x i32> @llvm.vp.shl.nxv4i32(<vscale x 4 x i32> %i0, <vscale x 4 x i32> %i1, <vscale x 4 x i1> %m, i32 %n)
+  ret void
+}
+
+; scalable-width vectors
+; integer arith
+declare <vscale x 4 x i32> @llvm.vp.add.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.sub.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.mul.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.sdiv.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.srem.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.udiv.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.urem.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+; bit arith
+declare <vscale x 4 x i32> @llvm.vp.and.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.xor.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.or.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.ashr.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.lshr.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 4 x i32> @llvm.vp.shl.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i1>, i32)

llvm/test/CodeGen/X86/O0-pipeline.ll

 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Expand Atomic instructions
 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       Lower Garbage Collection Instructions
 ; CHECK-NEXT:       Shadow Stack GC Lowering
 ; CHECK-NEXT:       Lower constant intrinsics
 ; CHECK-NEXT:       Remove unreachable blocks from the CFG
 ; CHECK-NEXT:       Instrument function entry/exit with calls to e.g. mcount() (post inlining)
+; CHECK-NEXT:       Expand vector predication intrinsics
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:       Expand indirectbr instructions
 ; CHECK-NEXT:     Rewrite Symbols
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Exception handling preparation
 ; CHECK-NEXT:       Safe Stack instrumentation pass
 ; CHECK-NEXT:       Insert stack protectors

llvm/tools/llc/llc.cpp

int main(int argc, char **argv) {
   initializeEntryExitInstrumenterPass(*Registry);
   initializePostInlineEntryExitInstrumenterPass(*Registry);
   initializeUnreachableBlockElimLegacyPassPass(*Registry);
   initializeConstantHoistingLegacyPassPass(*Registry);
   initializeScalarOpts(*Registry);
   initializeVectorization(*Registry);
   initializeScalarizeMaskedMemIntrinPass(*Registry);
   initializeExpandReductionsPass(*Registry);
+  initializeExpandVectorPredicationPass(*Registry);
   initializeHardwareLoopsPass(*Registry);
   initializeTransformUtils(*Registry);
 
   // Initialize debugging passes.
   initializeScavengerTestPass(*Registry);
 
   // Register the target printer for --version.
   cl::AddExtraVersionPrinter(TargetRegistry::printRegisteredTargetsForVersion);

llvm/tools/opt/opt.cpp

int main(int argc, char **argv) {
   initializeGlobalMergePass(Registry);
   initializeIndirectBrExpandPassPass(Registry);
   initializeInterleavedLoadCombinePass(Registry);
   initializeInterleavedAccessPass(Registry);
   initializeEntryExitInstrumenterPass(Registry);
   initializePostInlineEntryExitInstrumenterPass(Registry);
   initializeUnreachableBlockElimLegacyPassPass(Registry);
   initializeExpandReductionsPass(Registry);
+  initializeExpandVectorPredicationPass(Registry);
   initializeWasmEHPreparePass(Registry);
   initializeWriteBitcodePassPass(Registry);
   initializeHardwareLoopsPass(Registry);
   initializeTypePromotionPass(Registry);
 
 #ifdef BUILD_EXAMPLES
   initializeExampleIRTransforms(Registry);
 #endif