[LV, VP] Add CLI option to support predicated vectorization

[vkmr]

Summary

This patch adds a hidden command line option to enable predicated vectorization using VP intrinsics.
It is the 3rd patch in the series of patches to enable vector predication support (D99750).

[simoll]

A link to the VP intrinsics section in the LangRef would be nice.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

238

This is a lot of text. Maybe restructure this to make clear what the default is right now (NoPredication), what the default should be in the future (IfAVLSupported) and mention the other two explicitly as experimental/test settings.

[vkmr]

Ping!

[simoll]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

238

@vkmr pong: I stand by that comment ;-)

[vkmr]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

238

Oops my bad! @simoll I 100% stand by your comment too!
I totally forgot about it before the ping. I will address it in an upcoming update.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

cl::values(clEnumValN(PreferPredicateTy::ScalarEpilogue,
                          "predicate-else-scalar-epilogue",
                          "prefer tail-folding, create scalar epilogue if tail "
                          "folding fails."),
               clEnumValN(PreferPredicateTy::PredicateOrDontVectorize,
                          "predicate-dont-vectorize",
                          "prefers tail-folding, don't attempt vectorization if "
                          "tail-folding fails.")));
 
+// Option prefer-predicate-with-vp-intrinsics is an experimental switch to
+// indicate that the loop vectorizer should try to generate VP intrinsics if
+// tail-folding is enabled (note that this option is dependent on the
+// prefer-predicate-over-epilogue option being set to predicate-dont-vectorize).
+// This can be particularly useful for targets like RISC-V and SX-Aurora that
+// support vector length predication.
+// Currently this switch takes four possible values:
+// 0. no-predication: Do not generate VP intrinsics.
+// 1. if-active-vector-length-supported: Only generate VP intrinsics if the
+// target supports active vector length based predication.
+// 2. without-avl-support: Generate VP intrinsics even if vector length based
+// predication is not supported. This will behave a bit like existing
+// tail-folding by using a mask for predication, except all instructions are
+// widened to VP intrinsics and not just memory instructions. Use of this
+// options is discouraged and is only meant for experimental/testing purpose.
+// 3. force-active-vector-length-support: This is purely an experimental/testing
+// option which will be removed in future. It forces the loop vectorizer to
+// assume that the target supports vector length predication.
+namespace PreferVPIntrinsicsTy {
+enum Option {
+  NoPredication = 0,
+  IfAVLSupported,
+  WithoutAVLSupport,
+  ForceAVLSupport
+};
+} // namespace PreferVPIntrinsicsTy
+
+static cl::opt<PreferVPIntrinsicsTy::Option> PreferPredicateWithVPIntrinsics(
+    "prefer-predicate-with-vp-intrinsics",
+    cl::init(PreferVPIntrinsicsTy::NoPredication), cl::Hidden,
+    cl::desc("When vectorizing with tail-folding, generate vector predication "
+             "intrinsics."),
+    cl::values(
+        clEnumValN(PreferVPIntrinsicsTy::NoPredication, "no-predication",
+                   "Do not generate VP intrinsics."),
+        clEnumValN(PreferVPIntrinsicsTy::IfAVLSupported,
+                   "if-active-vector-length-support",
+                   "Only generate VP intrinsics if the target supports vector "
+                   "length predication."),
+        clEnumValN(PreferVPIntrinsicsTy::WithoutAVLSupport,
+                   "without-active-vector-length-support",
+                   "Generate VP intrinsics even if vector length predication "
+                   "is not supported. This option is discouraged."),
+        clEnumValN(PreferVPIntrinsicsTy::ForceAVLSupport,
+                   "force-active-vector-length-support",
+                   "Assume that the target supports vector length predication "
+                   "and generate VP intrinsics accordingly.")));
+
 static cl::opt<bool> MaximizeBandwidth(
     "vectorizer-maximize-bandwidth", cl::init(false), cl::Hidden,
     cl::desc("Maximize bandwidth when selecting vectorization factor which "
              "will be determined by the smallest type in loop."));
 
 static cl::opt<bool> EnableInterleavedMemAccesses(
     "enable-interleaved-mem-accesses", cl::init(false), cl::Hidden,
     cl::desc("Enable vectorization on interleaved memory accesses in a loop"));
public:
 
   /// Returns true if all loop blocks should be masked to fold tail loop.
   bool foldTailByMasking() const { return FoldTailByMasking; }
 
   bool blockNeedsPredication(BasicBlock *BB) const {
     return foldTailByMasking() || Legal->blockNeedsPredication(BB);
   }
 
+  /// Returns true if VP intrinsics should be generated in the tail folded loop.
+  bool preferVPIntrinsics() const {
+    return foldTailByMasking() && PreferVPIntrinsics;
+  }
+
   /// A SmallMapVector to store the InLoop reduction op chains, mapping phi
   /// nodes to the chain of instructions representing the reductions. Uses a
   /// MapVector to ensure deterministic iteration order.
   using ReductionChainMap =
       SmallMapVector<PHINode *, SmallVector<Instruction *, 4>, 4>;
 
   /// Return the chain of instructions representing an inloop reduction.
   const ReductionChainMap &getInLoopReductionChains() const {
private:
   /// or as a peel-loop to handle gaps in interleave-groups.
   /// Under optsize and when the trip count is very small we don't allow any
   /// iterations to execute in the scalar loop.
   ScalarEpilogueLowering ScalarEpilogueStatus = CM_ScalarEpilogueAllowed;
 
   /// All blocks of loop are to be masked to fold tail of scalar iterations.
   bool FoldTailByMasking = false;
 
+  /// Control whether to generate VP intrinsics in vectorized code.
+  bool PreferVPIntrinsics = false;
+
   /// A map holding scalar costs for different vectorization factors. The
   /// presence of a cost for an instruction in the mapping indicates that the
   /// instruction will be scalarized when vectorizing with the associated
   /// vectorization factor. The entries are VF-ScalarCostTy pairs.
   DenseMap<ElementCount, ScalarCostsTy> InstsToScalarize;
 
   /// Holds the instructions known to be uniform after vectorization.
   /// The data is collected per VF.
LoopVectorizationCostModel::computeMaxVF(ElementCount UserVF, unsigned UserIC) {
   }
 
   // If we don't know the precise trip count, or if the trip count that we
   // found modulo the vectorization factor is not zero, try to fold the tail
   // by masking.
   // FIXME: look for a smaller MaxVF that does divide TC rather than masking.
   if (Legal->prepareToFoldTailByMasking()) {
     FoldTailByMasking = true;
+    if (!PreferPredicateWithVPIntrinsics)
+      return MaxFactors;
+
+    if (UserIC > 1) {
+      LLVM_DEBUG(dbgs() << "LV: Preference for VP intrinsics indicated. Will "
+                           "not generate VP intrinsics since interleave count "
+                           "specified is greater than 1.\n");
+      return MaxFactors;
+    }
+
+    if (PreferPredicateWithVPIntrinsics ==
+        PreferVPIntrinsicsTy::IfAVLSupported) {
+      PreferVPIntrinsics = TTI.hasActiveVectorLength();
+      LLVM_DEBUG(dbgs() << "LV: Preference for VP intrinsics indicated. Will "
+                           "try to generate VP Intrinsics if the target "
+                           "support vector length predication.\n");
+    } else {
+      PreferVPIntrinsics = true;
+      LLVM_DEBUG(dbgs() << "LV: Preference for VP intrinsics indicated. Will "
+                           "try to generate VP Intrinsics.\n");
+    }
+
     return MaxFactors;
   }
 
   // If there was a tail-folding hint/switch, but we can't fold the tail by
   // masking, fallback to a vectorization with a scalar epilogue.
   if (ScalarEpilogueStatus == CM_ScalarEpilogueNotNeededUsePredicate) {
     LLVM_DEBUG(dbgs() << "LV: Cannot fold tail by masking: vectorize with a "
                          "scalar epilogue instead.\n");
unsigned LoopVectorizationCostModel::selectInterleaveCount(ElementCount VF,
   // 2. If the loop is really small, then we interleave to reduce the loop
   // overhead.
   // 3. We don't interleave if we think that we will spill registers to memory
   // due to the increased register pressure.
 
   if (!isScalarEpilogueAllowed())
     return 1;
 
+  // Do not interleave if VP intrinsics are preferred and no User IC is
+  // specified.
+  if (preferVPIntrinsics())
+    return 1;
+
   // We used the distance for the interleave count.
   if (Legal->getMaxSafeDepDistBytes() != -1U)
     return 1;
 
   auto BestKnownTC = getSmallBestKnownTC(*PSE.getSE(), TheLoop);
   const bool HasReductions = !Legal->getReductionVars().empty();
   // Do not interleave loops with a relatively small known or estimated trip
   // count. But we will interleave when InterleaveSmallLoopScalarReduction is