[LV] Add -scalable-vectorization=<option> flag.

[sdesmalen]

Summary

This patch adds a new option to the LoopVectorizer to control how
scalable vectors can be used.

Initially, this suggests three levels to control scalable
vectorization, although other more aggressive options can be added in
the future.

The possible options are:

• Disabled: Disables vectorization with scalable vectors.
• Enabled: Vectorize loops using scalable vectors or fixed-width vectors, whichever is most profitable. Favours fixed-width vectors when the cost is a tie.
• Preferred: Like 'Enabled', but favoring scalable vectors when the cost-model is inconclusive.

[CarolineConcatto]

Hey @sdesmalen
Thank you for the patch. Nice to see the famous flag in Phabricator.
I have some comments.
I hope they are not too non-sense.

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

132

I don't understand where is this function is used, because you've replaced it by isScalableVectorizationPreferred, but you did not changed in any other place. Maybe I am missing something.
And none of them are used.

151

What will happened when Scalable.Value is SK_Enabled?
It will not be scalable?

155

nit: isScalableVectorizationDisable()

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

108

Do we needs this?
Does this line do the same:
Scalable("vectorize.scalable.enable", ScalableVectorization, HK_SCALABLE),?

[sdesmalen]

Hi @CarolineConcatto thanks for having a look.

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

132

I'm not entirely sure what you mean.
isScalableVectorizationPreferred is used in disableScalableVectorization, which is used in getWidth and in LoopVectorize.cpp.

151

SK_Enabled means that scalable vectorization can be used, but isn't necessarily preferred over fixed-width. In contrast, the purpose of this method is to answer if scalable vectorization is actually preferred.

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

108

This is necessary when both the flag and the attribute are set (through code pragmas). After this change, if the user-code contains pragmas such as vectorize_width(4, scalable), then having -scalable-vectorization=off will cause the LV to override the hint so that scalable vectors are not considered.

[sdesmalen]

• Added SK_Undefined to fix the case where 'vectorize.width' is specified, but 'vectorize.scalable.enable' is not. In the previous revision of this patch, it would default to the value of the flag, which when set to 'preferred' would cause loops with #pragma clang loop vectorize_width(4) to be vectorized with VF=vscale x 4. This is now resolved (and tested).
• Also renamed disableScalableVectorization -> isScalableVectorizationDisabled.

[CarolineConcatto]

Hey @sdesmalen
I have more questions about the implementation.

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

150

This function is not used.

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

86

Should it be all the possible solutions here:
Val == SK_Disabled
Val == SK_Preferred
Val == SK_Enabled
Val == SK_Undefined

97–98

Can vectorize.scalable.enable have the same values as scalable-vectorization?
Which values should LoopVectorizeHints::Hint::validate check?

115

Just to double check:
if Scalable.Value != SK_Undefined and ScalableVectorization != SK_Disabled
Scalable.value will have the value set in vectorize.scalable.enable, is that correct?
and not in
scalable-vectorization

llvm/test/Transforms/LoopVectorize/AArch64/scalable-vf-analysis.ll

14

Is there a way to tell that VF scalable will not be used?
Only this:
LV: Found feasible scalable VF
is quite odd, because it does not explicitly says it will not use scalable.

[sdesmalen]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

150

Okay, I can move this to the other patch.

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

86

Perhaps at some point, but at the moment the value is interpeted as a boolean, and Clang sets it only to '0' or '1', so if we want to expand it to support other values in the metadata, we'll need to update Clang as well (which is something or another patch).

97–98

So first the variable Scalable is initialized with SK_Undefined.

• Then it looks through the metadata to see if the front-end has set the value (which can only be 'disabled' or 'preferred', which is checked by LoopVectorizeHints::Hint::validate)
• If it is not set by the metadata, then it will see if the flag is set to force scalable vectorization one way or another.
• Otherwise, if it is set by the metadata, it can be overriden by the flag only if the value is 'disable'.

This means it is possible to disable scalable vectorization using a single flag. But otherwise, metadata (because it is more specific) takes precedence over the flag (which is more generic).

115

Yes, that's correct. In that case it will have the value from the metadata.

llvm/test/Transforms/LoopVectorize/AArch64/scalable-vf-analysis.ll

14

It has the -NOT as part of the check prefix though.

e.g. CHECK-NOT: foo
will cause FileCheck to fail if foo occurs.

In this case, the prefix is CHECK_SCALABLE_DISABLED instead of CHECK, so CHECK_SCALABLE_DISABLED-NOT fails if LV: Found feasible scalable VF occurs.

[vkmr]

Thanks @sdesmalen for the patch!
Just a couple of clarifying questions.
Also, all the non-target specific test cases only check for -scalable-vectorization=on and with metadata hint enabled (except for one that focuses on width hint). Just for the sake of completeness, more tests with different values for the flag and its combination with the metadata hint would be nice to have.

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

108

The ordering of SK_Enabled and SK_Preferred feels non-intuitive. Is it because of the check:

case HK_SCALABLE:
    return (Val == SK_Disabled || Val == SK_Preferred);

that can only check Val to be 0 or 1?

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

97–98

If I understand correctly, metadata hints take precedence over this flag unless the flag is set to off?
If this is the case and the default value of the flag (for now) is set to SK_Disabled, that means the metadata hints won't have an effect if the flag is not explicitly set to on or preferred, right?

[paulwalker-arm]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

99–100

It looks weird to have two Kind's that have different definition/handling for Undefined.

99–108

Given the way the code is written, for example isScalableVectorizationPreferred() and the comments attached to the enum values I think the patch will be simpler if the enum names reflected their intent. For example:

Unspecified,
FixedOnly,
PreferFixed,
PreferScalable

[sdesmalen]

• Moved isScalableVectorizationPreferred out of this patch, since it wasn't used.
• Renamed enum values and made it signed again.
• Fixed order of enum values (0 (disabled), 1 (fixed-width only), 2 (fixed-width preferred), 3 (scalable preferred), and remapped boolean i1 from metadata to from 1 -> 3.
• Rebased patch.

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

99–108

That makes the code a bit clearer indeed, so I've adopted your suggestion. I have left the external user-facing option the same though.

108

Yes the LLVM IR Metadata attribute is a boolean 0 or 1. I agree this is a bit non-intuitive, so I've fixed this now by passing the argument to Hint::validate as a reference, so that we can remap the value.

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

97–98

Yes, that's right. This allows people to start experimenting with scalable vectors and using the pragma in their code, without it possibly breaking things unless they compile with the flag -scalable-vectorization=on|preferred.

[paulwalker-arm]

A few stylistic things to consider that I'll not hold you to, but the change to validate is the main thing stopping me from accepting the patch.

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

65–66

See comment on implementation.

99–111

Personally I think the following is prettier and more in keeping of the coding style used by other well documented enums:

enum ScalableForceKind {
  /// Not selected.
  SK_Unspecified = -1,   
  /// Disables vectorization with scalable vectors.
  SK_FixedWidthOnly = 0,
  /// Vectorize loops using scalable vectors or fixed-width
  /// vectors, but favor fixed-width vectors when the cost is
  /// inconclusive.
  SK_PreferFixedWidth = 1, 
  /// Vectorize loops using scalable vectors or fixed-width
  /// vectors, but favor scalable vectors when the cost-model is
  /// inconclusive. This is the default when the scalable.enable
  ///  hint is enabled through a pragma.
  SK_PreferScalable = 2

149–150

Perhaps nit picking but given the function says ExplicitlyEnabled, would it make more sense to explicitly compare against SK_PreferFixedWidth and SK_PreferScalable?

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

60

What about Scalable vectorisation is disabled.

62–64

What about Scalable vectorisation available, but favor fixed-width vectorisation when the cost is inconclusive.?

66–67

What about Scalable vectorisation available and favored when the cost is inconclusive.?

73

I realise you've changed this in response to a previous comment but having a validation function change the value it's validating seems wrong. You could change the function name or perhaps return an optional or something but given the enum values have been renamed I wonder if the original comment has been nullified?

That's to say, whilst I agreed with the comment's original context, I don't see any loss of intuition if the renamed enum order became:

SK_FixedWidthOnly = 0
SK_PreferScalable = 1
SK_PreferFixedWidth = 2

and thus you would no longer need these changes.

[sdesmalen]

• Addressed nits.
• Changed validate to take argument by value, not by ref.

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

73

That's a good point, with the new names of the enum values I agree the change in order is equally intuitive.

[paulwalker-arm]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

149–150

Now that you're not comparing against SK_Unspecified you probably don't need the (ScalableForceKind) casting. If you prefer to keep the casting then I suggest isScalableVectorizationDisabled() should also have it.

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

60

Sorry my bad :) you probably want s/vectorisation/vectorization/

[fhahn]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

99

can this be an enum class or is the code converting from ints to enum values somewhere?

[paulwalker-arm]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

99

I prefer that we remain consistent with the other Kind enums.

[fhahn]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

99

Using an enum class enforces stricter guarantees and allows the compiler the emit better warnings, which seem like tangible practical benefits. I'm not sure if consistency with other enums outweighs the practical benefits? It's quite common for parts of the codebase to evolve step-by-step.

Using an enum until the other enums are migrated leaves the door open for changes that make switching to an enum class harder.

[sdesmalen]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

99

There are several conversions between unsigned and the enum, so this requires adding casts to several places.

[paulwalker-arm]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

99

Sure I get that. I'm just saying I prefer all the enums associated with the LoopVectorizeHints class to be handled the same way.

[fhahn]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

99

There are several conversions between unsigned and the enum, so this requires adding casts to several places.

I missed that comment earlier. If there's the need for casting, it's a moot point anyways.

[sdesmalen]

• Addressed nits.

[vkmr]

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

60

Perhaps I am bike-shedding a bit much here (and I am not too sure about this), but the "on" and "off" terminology indicates binary choice, not making it readily clear that there is a possible third choice "preferred". Also reading -scalable-vectorization=on doesn't make it clear that it in fact prefers fixed vectorization.
May be something like "off", "possible" and "preferred" is clearer.
Thoughts?

[sdesmalen]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

99

Np, thanks for the suggestion anyway!

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

60

It's a good question! From my perspective, having the flag set to 'on' kind of does what it promises, it enables scalable vectorization. It doesn't really suggest to what extend it enables scalable vectorization, which leaves us some room to play with what a good default is. Whereas 'preferred' suggests that scalable vectorization is the preferred way to vectorize loops, instead of fixed-width. I guess the interpretation of 'on' isn't too dissimilar to 'possible', but while the latter is more accurate for now, the former may be more accurate going forward. Unless you feel strongly about this, I have a slight preference to keep 'on' as this feels a bit more future-proof as a default.

I'm not too worried about having much confusion about this, because --help-hidden shows all three options with their meaning, and it's not really meant to be a user-facing option in the sense that end-users will see it, but rather an experimental option for now.

[vkmr]

LGTM.

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

60

I agree with the future-proof reasoning. I am fine with the way it is.

[sdesmalen]

Thanks for the feedback and suggestions @paulwalker-arm, @vkmr and @CarolineConcatto!

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

class LoopVectorizeHints {
   struct Hint {
     const char *Name;
     unsigned Value; // This may have to change for non-numeric values.
     HintKind Kind;
 
     Hint(const char *Name, unsigned Value, HintKind Kind)
         : Name(Name), Value(Value), Kind(Kind) {}
 
     bool validate(unsigned Val);
   };
 
   /// Vectorization width.
   Hint Width;
 
   /// Vectorization interleave factor.
   Hint Interleave;
 
   /// Vectorization forced

 
 public:
   enum ForceKind {
     FK_Undefined = -1, ///< Not selected.
     FK_Disabled = 0,   ///< Forcing disabled.
     FK_Enabled = 1,    ///< Forcing enabled.
   };
 
+  enum ScalableForceKind {
+    /// Not selected.
+    SK_Unspecified = -1,
+    /// Disables vectorization with scalable vectors.
+    SK_FixedWidthOnly = 0,
+    /// Vectorize loops using scalable vectors or fixed-width vectors, but favor
+    /// scalable vectors when the cost-model is inconclusive. This is the
+    /// default when the scalable.enable hint is enabled through a pragma.
+    SK_PreferScalable = 1,
+    /// Vectorize loops using scalable vectors or fixed-width  vectors, but
+    /// favor fixed-width vectors when the cost is inconclusive.
+    SK_PreferFixedWidth = 2,
+  };
+
   LoopVectorizeHints(const Loop *L, bool InterleaveOnlyWhenForced,
                      OptimizationRemarkEmitter &ORE);
 
   /// Mark the loop L as already vectorized by setting the width to 1.
   void setAlreadyVectorized();
 
   bool allowVectorization(Function *F, Loop *L,
                           bool VectorizeOnlyWhenForced) const;
 
   /// Dumps all the hint information.
   void emitRemarkWithHints() const;
 
   ElementCount getWidth() const {
-    return ElementCount::get(Width.Value, isScalable());
+    return ElementCount::get(Width.Value,
+                             isScalableVectorizationExplicitlyEnabled());
   }
   unsigned getInterleave() const {
     if (Interleave.Value)
       return Interleave.Value;
     // If interleaving is not explicitly set, assume that if we do not want
     // unrolling, we also don't want any interleaving.
     if (llvm::hasUnrollTransformation(TheLoop) & TM_Disable)
       return 1;
     return 0;
   }
   unsigned getIsVectorized() const { return IsVectorized.Value; }
   unsigned getPredicate() const { return Predicate.Value; }
   enum ForceKind getForce() const {
     if ((ForceKind)Force.Value == FK_Undefined &&
         hasDisableAllTransformsHint(TheLoop))
       return FK_Disabled;
     return (ForceKind)Force.Value;
   }
 
-  bool isScalable() const { return Scalable.Value; }
+  /// \return true if scalable vectorization has been explicitly enabled.
+  bool isScalableVectorizationExplicitlyEnabled() const {
+    return (ScalableForceKind)Scalable.Value == SK_PreferFixedWidth ||
+           (ScalableForceKind)Scalable.Value == SK_PreferScalable;
+  }
+
+  /// \return true if scalable vectorization has been explicitly disabled.
+  bool isScalableVectorizationDisabled() const {
+    return Scalable.Value == SK_FixedWidthOnly;
+  }
 
   /// If hints are provided that force vectorization, use the AlwaysPrint
   /// pass name to force the frontend to print the diagnostic.
   const char *vectorizeAnalysisPassName() const;
 
   bool allowReordering() const {
     // When enabling loop hints are provided we allow the vectorizer to change
     // the order of operations that is given by the scalar loop. This is not

llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp

static cl::opt<unsigned> VectorizeSCEVCheckThreshold(
     "vectorize-scev-check-threshold", cl::init(16), cl::Hidden,
     cl::desc("The maximum number of SCEV checks allowed."));
 
 static cl::opt<unsigned> PragmaVectorizeSCEVCheckThreshold(
     "pragma-vectorize-scev-check-threshold", cl::init(128), cl::Hidden,
     cl::desc("The maximum number of SCEV checks allowed with a "
              "vectorize(enable) pragma"));
 
+// FIXME: When scalable vectorization is stable enough, change the default
+// to SK_PreferFixedWidth.
+static cl::opt<LoopVectorizeHints::ScalableForceKind> ScalableVectorization(
+    "scalable-vectorization", cl::init(LoopVectorizeHints::SK_FixedWidthOnly),
+    cl::Hidden,
+    cl::desc("Control whether the compiler can use scalable vectors to "
+             "vectorize a loop"),
+    cl::values(
+        clEnumValN(LoopVectorizeHints::SK_FixedWidthOnly, "off",
+                   "Scalable vectorisation is disabled."),
+        clEnumValN(LoopVectorizeHints::SK_PreferFixedWidth, "on",
+                   "Scalable vectorisation is available, but favor fixed-width "
+                   "vectorisation when the cost is inconclusive."),
+        clEnumValN(LoopVectorizeHints::SK_PreferScalable, "preferred",
+                   "Scalable vectorisation is available and favored when the "
+                   "cost is inconclusive.")));
+
 /// Maximum vectorization interleave count.
 static const unsigned MaxInterleaveFactor = 16;
 
 namespace llvm {
 
 bool LoopVectorizeHints::Hint::validate(unsigned Val) {
   switch (Kind) {
   case HK_WIDTH:
     return isPowerOf2_32(Val) && Val <= VectorizerParams::MaxVectorWidth;
   case HK_INTERLEAVE:
     return isPowerOf2_32(Val) && Val <= MaxInterleaveFactor;
   case HK_FORCE:
     return (Val <= 1);
   case HK_ISVECTORIZED:
   case HK_PREDICATE:
   case HK_SCALABLE:
     return (Val == 0 || Val == 1);
   }
   return false;
 }
 
 LoopVectorizeHints::LoopVectorizeHints(const Loop *L,
                                        bool InterleaveOnlyWhenForced,
                                        OptimizationRemarkEmitter &ORE)
     : Width("vectorize.width", VectorizerParams::VectorizationFactor, HK_WIDTH),
       Interleave("interleave.count", InterleaveOnlyWhenForced, HK_INTERLEAVE),
       Force("vectorize.enable", FK_Undefined, HK_FORCE),
       IsVectorized("isvectorized", 0, HK_ISVECTORIZED),
       Predicate("vectorize.predicate.enable", FK_Undefined, HK_PREDICATE),
-      Scalable("vectorize.scalable.enable", false, HK_SCALABLE), TheLoop(L),
-      ORE(ORE) {
+      Scalable("vectorize.scalable.enable", SK_Unspecified, HK_SCALABLE),
+      TheLoop(L), ORE(ORE) {
   // Populate values with existing loop metadata.
   getHintsFromMetadata();
 
   // force-vector-interleave overrides DisableInterleaving.
   if (VectorizerParams::isInterleaveForced())
     Interleave.Value = VectorizerParams::VectorizationInterleave;
 
+  if (Scalable.Value == SK_Unspecified)
+    // If the width is set, but the metadata says nothing about the scalable
+    // property, then assume it concerns only a fixed-width UserVF.
+    // If width is not set, the flag takes precedence.
+    Scalable.Value = Width.Value ? SK_FixedWidthOnly : ScalableVectorization;
+  else if (ScalableVectorization == SK_FixedWidthOnly)
+    // If the flag is set to disable any use of scalable vectors, override the
+    // loop hint.
+    Scalable.Value = SK_FixedWidthOnly;
+
   if (IsVectorized.Value != 1)
     // If the vectorization width and interleaving count are both 1 then
     // consider the loop to have been already vectorized because there's
     // nothing more that we can do.
     IsVectorized.Value =
         getWidth() == ElementCount::getFixed(1) && getInterleave() == 1;
   LLVM_DEBUG(if (InterleaveOnlyWhenForced && getInterleave() == 1) dbgs()
              << "LV: Interleaving disabled by the pass manager\n");

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

LoopVectorizationCostModel::getMaxLegalScalableVF(unsigned MaxSafeElements) {
   if (!TTI.supportsScalableVectors() && !ForceTargetSupportsScalableVectors) {
     reportVectorizationInfo(
         "Disabling scalable vectorization, because target does not "
         "support scalable vectors.",
         "ScalableVectorsUnsupported", ORE, TheLoop);
     return ElementCount::getScalable(0);
   }
 
+  if (Hints->isScalableVectorizationDisabled()) {
+    reportVectorizationInfo("Scalable vectorization is explicitly disabled",
+                            "ScalableVectorizationDisabled", ORE, TheLoop);
+    return ElementCount::getScalable(0);
+  }
+
   auto MaxScalableVF = ElementCount::getScalable(
       std::numeric_limits<ElementCount::ScalarTy>::max());
 
   // Disable scalable vectorization if the loop contains unsupported reductions.
   // Test that the loop-vectorizer can legalize all operations for this MaxVF.
   // FIXME: While for scalable vectors this is currently sufficient, this should
   // be replaced by a more detailed mechanism that filters out specific VFs,
   // instead of invalidating vectorization for a whole set of VFs based on the

llvm/test/Transforms/LoopVectorize/AArch64/first-order-recurrence.ll

-; RUN: opt -loop-vectorize -force-vector-width=4 -force-vector-interleave=1 -mtriple aarch64-unknown-linux-gnu -mattr=+sve -S < %s | FileCheck %s --check-prefix=CHECK-VF4UF1
-; RUN: opt -loop-vectorize -force-vector-width=4 -force-vector-interleave=2 -mtriple aarch64-unknown-linux-gnu -mattr=+sve -S < %s | FileCheck %s --check-prefix=CHECK-VF4UF2
+; RUN: opt -loop-vectorize -scalable-vectorization=on -force-vector-width=4 -force-vector-interleave=1 -mtriple aarch64-unknown-linux-gnu -mattr=+sve -S < %s | FileCheck %s --check-prefix=CHECK-VF4UF1
+; RUN: opt -loop-vectorize -scalable-vectorization=on -force-vector-width=4 -force-vector-interleave=2 -mtriple aarch64-unknown-linux-gnu -mattr=+sve -S < %s | FileCheck %s --check-prefix=CHECK-VF4UF2
 
 ; We vectorize this first order recurrence, with a set of insertelements for
 ; each unrolled part. Make sure these insertelements are generated in-order,
 ; because the shuffle of the first order recurrence will be added after the
 ; insertelement of the last part UF - 1, assuming the latter appears after the
 ; insertelements of all other parts.
 ;
 ; int PR33613(double *b, double j, int d) {

llvm/test/Transforms/LoopVectorize/AArch64/masked-op-cost.ll

 ; REQUIRES: asserts
-; RUN: opt -loop-vectorize -force-vector-interleave=1 -S -debug < %s 2>%t | FileCheck %s
+; RUN: opt -loop-vectorize -scalable-vectorization=on -force-vector-interleave=1 -S -debug-only=loop-vectorize < %s 2>%t | FileCheck %s
 ; RUN: cat %t | FileCheck %s --check-prefix=CHECK-COST
 
 target triple = "aarch64-unknown-linux-gnu"
 
 ; CHECK-COST: Checking a loop in "fixed_width"
 ; CHECK-COST: Found an estimated cost of 11 for VF 2 For instruction:   store i32 2, i32* %arrayidx1, align 4
 ; CHECK-COST: Found an estimated cost of 25 for VF 4 For instruction:   store i32 2, i32* %arrayidx1, align 4
 ; CHECK-COST: Selecting VF: 1.

llvm/test/Transforms/LoopVectorize/AArch64/scalable-call.ll

-; RUN: opt -S -loop-vectorize -force-vector-interleave=1 -instcombine -mattr=+sve -mtriple aarch64-unknown-linux-gnu < %s | FileCheck %s
+; RUN: opt -S -loop-vectorize -force-vector-interleave=1 -instcombine -mattr=+sve -mtriple aarch64-unknown-linux-gnu -scalable-vectorization=on < %s | FileCheck %s
 
 define void @vec_load(i64 %N, double* nocapture %a, double* nocapture readonly %b) {
 ; CHECK-LABEL: @vec_load
 ; CHECK: vector.body:
 ; CHECK: %[[LOAD:.*]] = load <vscale x 2 x double>, <vscale x 2 x double>*
 ; CHECK: call <vscale x 2 x double> @foo_vec(<vscale x 2 x double> %[[LOAD]])
 entry:
   %cmp7 = icmp sgt i64 %N, 0

llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll

-; RUN: opt < %s -loop-vectorize -pass-remarks=loop-vectorize -pass-remarks-analysis=loop-vectorize -pass-remarks-missed=loop-vectorize -mtriple aarch64-unknown-linux-gnu -mattr=+sve -S 2>%t | FileCheck %s -check-prefix=CHECK
+; RUN: opt < %s -loop-vectorize -pass-remarks=loop-vectorize -pass-remarks-analysis=loop-vectorize -pass-remarks-missed=loop-vectorize -mtriple aarch64-unknown-linux-gnu -mattr=+sve -S -scalable-vectorization=on 2>%t | FileCheck %s -check-prefix=CHECK
 ; RUN: cat %t | FileCheck %s -check-prefix=CHECK-REMARK
 
 ; Reduction can be vectorized
 
 ; ADD
 
 ; CHECK-REMARK: vectorized loop (vectorization width: vscale x 8, interleaved count: 2)
 define i32 @add(i32* nocapture %a, i32* nocapture readonly %b, i64 %n) {

llvm/test/Transforms/LoopVectorize/AArch64/scalable-strict-fadd.ll

-; RUN: opt < %s -loop-vectorize -mtriple aarch64-unknown-linux-gnu -mattr=+sve -enable-strict-reductions -S | FileCheck %s -check-prefix=CHECK
+; RUN: opt < %s -loop-vectorize -scalable-vectorization=on -mtriple aarch64-unknown-linux-gnu -mattr=+sve -enable-strict-reductions -S | FileCheck %s -check-prefix=CHECK
 
 define float @fadd_strict(float* noalias nocapture readonly %a, i64 %n) {
 ; CHECK-LABEL: @fadd_strict
 ; CHECK: vector.body:
 ; CHECK: %[[VEC_PHI:.*]] = phi float [ 0.000000e+00, %vector.ph ], [ %[[RDX:.*]], %vector.body ]
 ; CHECK: %[[LOAD:.*]] = load <vscale x 8 x float>, <vscale x 8 x float>*
 ; CHECK: %[[RDX]] = call float @llvm.vector.reduce.fadd.nxv8f32(float %[[VEC_PHI]], <vscale x 8 x float> %[[LOAD]])
 ; CHECK: for.end

llvm/test/Transforms/LoopVectorize/AArch64/scalable-vf-analysis.ll

 ; REQUIRES: asserts
-; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -force-target-instruction-cost=1 -loop-vectorize -S -debug-only=loop-vectorize < %s 2>&1 | FileCheck %s --check-prefixes=CHECK,CHECK_SCALABLE_ON
-; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -force-target-instruction-cost=1 -loop-vectorize -S -debug-only=loop-vectorize -vectorizer-maximize-bandwidth < %s 2>&1 | FileCheck %s --check-prefixes=CHECK,CHECK_SCALABLE_ON_MAXBW
+; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -force-target-instruction-cost=1 -loop-vectorize -S -debug-only=loop-vectorize -scalable-vectorization=on        < %s 2>&1 | FileCheck %s --check-prefixes=CHECK,CHECK_SCALABLE_ON
+; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -force-target-instruction-cost=1 -loop-vectorize -S -debug-only=loop-vectorize -scalable-vectorization=preferred < %s 2>&1 | FileCheck %s --check-prefixes=CHECK,CHECK_SCALABLE_PREFERRED
+; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -force-target-instruction-cost=1 -loop-vectorize -S -debug-only=loop-vectorize -scalable-vectorization=off       < %s 2>&1 | FileCheck %s --check-prefixes=CHECK,CHECK_SCALABLE_DISABLED
+; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -force-target-instruction-cost=1 -loop-vectorize -S -debug-only=loop-vectorize -vectorizer-maximize-bandwidth -scalable-vectorization=on < %s 2>&1 | FileCheck %s --check-prefixes=CHECK,CHECK_SCALABLE_ON_MAXBW
 
 ; Test that the MaxVF for the following loop, that has no dependence distances,
 ; is calculated as vscale x 4 (max legal SVE vector size) or vscale x 16
 ; (maximized bandwidth for i8 in the loop).
 define void @test0(i32* %a, i8* %b, i32* %c) {
 ; CHECK: LV: Checking a loop in "test0"
 ; CHECK_SCALABLE_ON: LV: Found feasible scalable VF = vscale x 4
+; CHECK_SCALABLE_PREFERRED: LV: Found feasible scalable VF = vscale x 4
+; CHECK_SCALABLE_DISABLED-NOT: LV: Found feasible scalable VF
 ; CHECK_SCALABLE_ON_MAXBW: LV: Found feasible scalable VF = vscale x 16
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %arrayidx = getelementptr inbounds i32, i32* %c, i64 %iv
   %0 = load i32, i32* %arrayidx, align 4
exit:
   ret void
 }
 
 ; Test that the MaxVF for the following loop, with a dependence distance
 ; of 64 elements, is calculated as (maxvscale = 16) * 4.
 define void @test1(i32* %a, i8* %b) {
 ; CHECK: LV: Checking a loop in "test1"
 ; CHECK_SCALABLE_ON: LV: Found feasible scalable VF = vscale x 4
+; CHECK_SCALABLE_PREFERRED: LV: Found feasible scalable VF = vscale x 4
+; CHECK_SCALABLE_DISABLED-NOT: LV: Found feasible scalable VF
 ; CHECK_SCALABLE_ON_MAXBW: LV: Found feasible scalable VF = vscale x 4
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
   %0 = load i32, i32* %arrayidx, align 4
exit:
   ret void
 }
 
 ; Test that the MaxVF for the following loop, with a dependence distance
 ; of 32 elements, is calculated as (maxvscale = 16) * 2.
 define void @test2(i32* %a, i8* %b) {
 ; CHECK: LV: Checking a loop in "test2"
 ; CHECK_SCALABLE_ON: LV: Found feasible scalable VF = vscale x 2
+; CHECK_SCALABLE_PREFERRED: LV: Found feasible scalable VF = vscale x 2
+; CHECK_SCALABLE_DISABLED-NOT: LV: Found feasible scalable VF
 ; CHECK_SCALABLE_ON_MAXBW: LV: Found feasible scalable VF = vscale x 2
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
   %0 = load i32, i32* %arrayidx, align 4
exit:
   ret void
 }
 
 ; Test that the MaxVF for the following loop, with a dependence distance
 ; of 16 elements, is calculated as (maxvscale = 16) * 1.
 define void @test3(i32* %a, i8* %b) {
 ; CHECK: LV: Checking a loop in "test3"
 ; CHECK_SCALABLE_ON: LV: Found feasible scalable VF = vscale x 1
+; CHECK_SCALABLE_PREFERRED: LV: Found feasible scalable VF = vscale x 1
+; CHECK_SCALABLE_DISABLED-NOT: LV: Found feasible scalable VF
 ; CHECK_SCALABLE_ON_MAXBW: LV: Found feasible scalable VF = vscale x 1
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
   %0 = load i32, i32* %arrayidx, align 4

 }
 
 ; Test the fallback mechanism when scalable vectors are not feasible due
 ; to e.g. dependence distance. For the '-scalable-vectorization=exclusive'
 ; it shouldn't try to vectorize with fixed-width vectors.
 define void @test4(i32* %a, i32* %b) {
 ; CHECK: LV: Checking a loop in "test4"
 ; CHECK_SCALABLE_ON-NOT: LV: Found feasible scalable VF
+; CHECK_SCALABLE_PREFERRED-NOT: LV: Found feasible scalable VF
+; CHECK_SCALABLE_DISABLED-NOT: LV: Found feasible scalable VF
 ; CHECK_SCALABLE_ON_MAXBW-NOT: LV: Found feasible scalable VF
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
   %0 = load i32, i32* %arrayidx, align 4

llvm/test/Transforms/LoopVectorize/AArch64/scalable-vf-hint.ll

 ; REQUIRES: asserts
-; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -loop-vectorize -S < %s 2>&1 | FileCheck %s
-; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -loop-vectorize -pass-remarks-analysis=loop-vectorize -debug-only=loop-vectorize -S < %s 2>&1 | FileCheck --check-prefix=CHECK-DBG %s
-; RUN: opt -mtriple=aarch64-none-linux-gnu -loop-vectorize -pass-remarks-analysis=loop-vectorize -debug-only=loop-vectorize -S < %s 2>&1 | FileCheck --check-prefix=CHECK-NO-SVE %s
-; RUN: opt -mtriple=aarch64-none-linux-gnu -loop-vectorize -force-target-supports-scalable-vectors=true -pass-remarks-analysis=loop-vectorize -debug-only=loop-vectorize -S < %s 2>&1 | FileCheck --check-prefix=CHECK-NO-MAX-VSCALE %s
+; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -loop-vectorize -S -scalable-vectorization=on < %s 2>&1 | FileCheck %s
+; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -loop-vectorize -pass-remarks-analysis=loop-vectorize -debug-only=loop-vectorize -S -scalable-vectorization=on < %s 2>&1 | FileCheck --check-prefix=CHECK-DBG %s
+; RUN: opt -mtriple=aarch64-none-linux-gnu -loop-vectorize -pass-remarks-analysis=loop-vectorize -debug-only=loop-vectorize -S -scalable-vectorization=on < %s 2>&1 | FileCheck --check-prefix=CHECK-NO-SVE %s
+; RUN: opt -mtriple=aarch64-none-linux-gnu -loop-vectorize -force-target-supports-scalable-vectors=true -pass-remarks-analysis=loop-vectorize -debug-only=loop-vectorize -S -scalable-vectorization=on < %s 2>&1 | FileCheck --check-prefix=CHECK-NO-MAX-VSCALE %s
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 
 ; These tests validate the behaviour of scalable vectorization factor hints,
 ; where the following applies:
 ;
 ; * If the backend does not support scalable vectors, ignore the hint and let
 ;   the vectorizer pick a VF.

llvm/test/Transforms/LoopVectorize/AArch64/sve-basic-vec.ll

-; RUN: opt -loop-vectorize -dce -instcombine -mtriple aarch64-linux-gnu -mattr=+sve < %s -S | FileCheck %s
+; RUN: opt -loop-vectorize -dce -instcombine -mtriple aarch64-linux-gnu -mattr=+sve -scalable-vectorization=on < %s -S | FileCheck %s
 
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-unknown-linux-gnu"
 
 define void @cmpsel_i32(i32* noalias nocapture %a, i32* noalias nocapture readonly %b, i64 %n) {
 ; CHECK-LABEL: @cmpsel_i32(
 ; CHECK-NEXT:  entry:

llvm/test/Transforms/LoopVectorize/AArch64/sve-cond-inv-loads.ll

-; RUN: opt -loop-vectorize -dce -instcombine -mtriple aarch64-linux-gnu -mattr=+sve -S %s -o - | FileCheck %s
+; RUN: opt -loop-vectorize -scalable-vectorization=on -dce -instcombine -mtriple aarch64-linux-gnu -mattr=+sve -S %s -o - | FileCheck %s
 
 define void @cond_inv_load_i32i32i16(i32* noalias nocapture %a, i32* noalias nocapture readonly %cond, i16* noalias nocapture readonly %inv, i64 %n) {
 ; CHECK-LABEL: @cond_inv_load_i32i32i16
 ; CHECK:     vector.ph:
 ; CHECK:       %[[INVINS:.*]] = insertelement <vscale x 4 x i16*> poison, i16* %inv, i32 0
 ; CHECK:       %[[INVSPLAT:.*]] = shufflevector <vscale x 4 x i16*> %[[INVINS]], <vscale x 4 x i16*> poison, <vscale x 4 x i32> zeroinitializer
 ; CHECK:     vector.body:
 ; CHECK:       %[[GEPCOND:.*]] = getelementptr inbounds i32, i32* %cond, i64 %index

llvm/test/Transforms/LoopVectorize/AArch64/sve-extract-last-veclane.ll

-; RUN: opt -loop-vectorize -dce -instcombine -mtriple aarch64-linux-gnu -S < %s | FileCheck %s
+; RUN: opt -loop-vectorize -scalable-vectorization=on -dce -instcombine -mtriple aarch64-linux-gnu -S < %s | FileCheck %s
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-unknown-linux-gnu"
 
 define void @inv_store_last_lane(i32* noalias nocapture %a, i32* noalias nocapture %inv, i32* noalias nocapture readonly %b, i64 %n) #0 {
 ; CHECK-LABEL: @inv_store_last_lane
 ; CHECK: vector.body:
 ; CHECK:  store <vscale x 4 x i32> %[[VEC_VAL:.*]], <

llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter.ll

-; RUN: opt -loop-vectorize -dce -instcombine -mtriple aarch64-linux-gnu -mattr=+sve -S %s -o - | FileCheck %s
+; RUN: opt -loop-vectorize -dce -instcombine -mtriple aarch64-linux-gnu -mattr=+sve -S %s -scalable-vectorization=on -o - | FileCheck %s
 
 define void @gather_nxv4i32_ind64(float* noalias nocapture readonly %a, i64* noalias nocapture readonly %b, float* noalias nocapture %c, i64 %n) {
 ; CHECK-LABEL: @gather_nxv4i32_ind64
 ; CHECK: vector.body:
 ; CHECK:   %[[IND:.*]] = load <vscale x 4 x i64>, <vscale x 4 x i64>*
 ; CHECK:   %[[PTRS:.*]] = getelementptr inbounds float, float* %a, <vscale x 4 x i64> %[[IND]]
 ; CHECK:   %[[GLOAD:.*]] = call <vscale x 4 x float> @llvm.masked.gather.nxv4f32.nxv4p0f32(<vscale x 4 x float*> %[[PTRS]]
 ; CHECK:   store <vscale x 4 x float> %[[GLOAD]], <vscale x 4 x float>*

llvm/test/Transforms/LoopVectorize/AArch64/sve-inductions.ll

-; RUN: opt -mtriple aarch64-linux-gnu -mattr=+sve -loop-vectorize -dce -instcombine < %s -S | FileCheck %s
+; RUN: opt -mtriple aarch64-linux-gnu -mattr=+sve -loop-vectorize -scalable-vectorization=on -dce -instcombine < %s -S | FileCheck %s
 
 ; Test that we can add on the induction variable
 ;   for (long long i = 0; i < n; i++) {
 ;     a[i] = b[i] + i;
 ;   }
 ; with an unroll factor (interleave count) of 2.
 
 define void @add_ind64_unrolled(i64* noalias nocapture %a, i64* noalias nocapture readonly %b, i64 %n) {

llvm/test/Transforms/LoopVectorize/AArch64/sve-inv-loads.ll

-; RUN: opt -S -loop-vectorize -mattr=+sve -mtriple aarch64-linux-gnu < %s | FileCheck %s
+; RUN: opt -S -loop-vectorize -scalable-vectorization=on -mattr=+sve -mtriple aarch64-linux-gnu < %s | FileCheck %s
 
 define void @invariant_load(i64 %n, i32* noalias nocapture %a, i32* nocapture readonly %b) {
 ; CHECK-LABEL: @invariant_load
 ; CHECK: vector.body:
 ; CHECK: %[[GEP:.*]] = getelementptr inbounds i32, i32* %b, i64 42
 ; CHECK-NEXT: %[[INVLOAD:.*]] = load i32, i32* %[[GEP]]
 ; CHECK-NEXT: %[[SPLATINS:.*]] = insertelement <vscale x 4 x i32> poison, i32 %[[INVLOAD]], i32 0
 ; CHECK-NEXT: %[[SPLAT:.*]] = shufflevector <vscale x 4 x i32> %[[SPLATINS]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer

llvm/test/Transforms/LoopVectorize/AArch64/sve-large-strides.ll

-; RUN: opt -mtriple aarch64-linux-gnu -mattr=+sve -loop-vectorize -dce -instcombine -S <%s | FileCheck %s
+; RUN: opt -mtriple aarch64-linux-gnu -mattr=+sve -loop-vectorize -scalable-vectorization=on -dce -instcombine -S <%s | FileCheck %s
 
 define void @stride7_i32(i32* noalias nocapture %dst, i64 %n) {
 ; CHECK-LABEL: @stride7_i32(
 ; CHECK:      vector.body
 ; CHECK:        %[[VEC_IND:.*]] = phi <vscale x 4 x i64> [ %{{.*}}, %vector.ph ], [ %{{.*}}, %vector.body ]
 ; CHECK-NEXT:   %[[PTR_INDICES:.*]] = mul nuw nsw <vscale x 4 x i64> %[[VEC_IND]], shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 7, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
 ; CHECK-NEXT:   %[[PTRS:.*]] = getelementptr inbounds i32, i32* %dst, <vscale x 4 x i64> %[[PTR_INDICES]]
 ; CHECK-NEXT:   %[[GLOAD:.*]] = call <vscale x 4 x i32> @llvm.masked.gather.nxv4i32.nxv4p0i32(<vscale x 4 x i32*> %[[PTRS]]

llvm/test/Transforms/LoopVectorize/AArch64/sve-masked-loadstore.ll

-; RUN: opt -loop-vectorize -dce -instcombine -mtriple aarch64-linux-gnu -mattr=+sve -S %s -o - | FileCheck %s
+; RUN: opt -loop-vectorize -dce -instcombine -mtriple aarch64-linux-gnu -mattr=+sve -S %s -scalable-vectorization=on -o - | FileCheck %s
 
 define void @mloadstore_f32(float* noalias nocapture %a, float* noalias nocapture readonly %b, i64 %n) {
 ; CHECK-LABEL: @mloadstore_f32
 ; CHECK: vector.body:
 ; CHECK:       %[[LOAD1:.*]] = load <vscale x 4 x float>, <vscale x 4 x float>*
 ; CHECK-NEXT:  %[[MASK:.*]] = fcmp ogt <vscale x 4 x float> %[[LOAD1]],
 ; CHECK-NEXT:  %[[GEPA:.*]] = getelementptr inbounds float, float* %a,
 ; CHECK-NEXT:  %[[MLOAD_PTRS:.*]] = bitcast float* %[[GEPA]] to <vscale x 4 x float>*

llvm/test/Transforms/LoopVectorize/AArch64/sve-type-conv.ll

-; RUN: opt -loop-vectorize -dce -instcombine < %s -S | FileCheck %s
+; RUN: opt -loop-vectorize -scalable-vectorization=on -dce -instcombine < %s -S | FileCheck %s
 
 target triple = "aarch64-unknown-linux-gnu"
 
 
 define void @f16_to_f32(float* noalias nocapture %dst, half* noalias nocapture readonly %src, i64 %N) #0 {
 ; CHECK-LABEL: @f16_to_f32(
 ; CHECK: vector.body
 ; CHECK:   %{{.*}} = fpext <vscale x 8 x half> %{{.*}} to <vscale x 8 x float>

llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll

 ; This is the loop in c++ being vectorize in this file with 
 ; experimental.vector.reverse
 
 ;#pragma clang loop vectorize_width(4, scalable)
 ;  for (long int i = N - 1; i >= 0; i--)
 ;  {
 ;    if (cond[i])
 ;      a[i] += 1;
 ;  }
 
 ; The test checks if the mask is being correctly created, reverted and used
 
-; RUN: opt -loop-vectorize -dce -instcombine -mtriple aarch64-linux-gnu -S < %s | FileCheck %s
+; RUN: opt -loop-vectorize -scalable-vectorization=on -dce -instcombine -mtriple aarch64-linux-gnu -S < %s | FileCheck %s
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-unknown-linux-gnu"
 
 define void @vector_reverse_mask_nxv4i1(double* %a, double* %cond, i64 %N) #0 {
 ; CHECK-LABEL: vector.body:
 ; CHECK: %[[REVERSE6:.*]] = call <vscale x 4 x i1> @llvm.experimental.vector.reverse.nxv4i1(<vscale x 4 x i1> %{{.*}})
 ; CHECK: %[[WIDEMSKLOAD:.*]] = call <vscale x 4 x double> @llvm.masked.load.nxv4f64.p0nxv4f64(<vscale x 4 x double>* nonnull %{{.*}}, i32 8, <vscale x 4 x i1> %[[REVERSE6]], <vscale x 4 x double> poison)

llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll

 ; This is the loop in c++ being vectorize in this file with
 ;experimental.vector.reverse
 ;  #pragma clang loop vectorize_width(8, scalable)
 ;  for (int i = N-1; i >= 0; --i)
 ;    a[i] = b[i] + 1.0;
 
-; RUN: opt -loop-vectorize -dce -instcombine -mtriple aarch64-linux-gnu -S < %s | FileCheck %s
+; RUN: opt -loop-vectorize -scalable-vectorization=on -dce -instcombine -mtriple aarch64-linux-gnu -S < %s | FileCheck %s
 
 define void @vector_reverse_f64(i64 %N, double* %a, double* %b) #0{
 ; CHECK-LABEL: @vector_reverse_f64
 ; CHECK-LABEL: vector.body:
 ; CHECK: %[[ADD:.*]] = add i64 %{{.*}}, %N
 ; CHECK-NEXT: %[[GEP:.*]] = getelementptr inbounds double, double* %b, i64 %[[ADD]]
 ; CHECK-NEXT: %[[VSCALE:.*]] = call i32 @llvm.vscale.i32()
 ; CHECK-NEXT: %[[MUL:.*]] = mul i32 %[[VSCALE]], -8

llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll

-; RUN: opt -mtriple aarch64-linux-gnu -mattr=+sve -loop-vectorize -dce -instcombine -S < %s | FileCheck %s
+; RUN: opt -mtriple aarch64-linux-gnu -mattr=+sve -loop-vectorize -scalable-vectorization=on -dce -instcombine -S < %s | FileCheck %s
 
 ; Ensure that we can vectorize loops such as:
 ;   int *ptr = c;
 ;   for (long long i = 0; i < n; i++) {
 ;     int X1 = *ptr++;
 ;     int X2 = *ptr++;
 ;     a[i] = X1 + 1;
 ;     b[i] = X2 + 1;

llvm/test/Transforms/LoopVectorize/RISCV/scalable-vf-hint.ll

 ; RUN: opt -mtriple=riscv64 -mattr=+m,+experimental-v -loop-vectorize \
-; RUN:   -riscv-v-vector-bits-max=512 -S < %s 2>&1 \
+; RUN:   -riscv-v-vector-bits-max=512 -S -scalable-vectorization=on < %s 2>&1 \
 ; RUN:   | FileCheck %s
 
 ; void test(int *a, int *b, int N) {
 ;   #pragma clang loop vectorize(enable) vectorize_width(2, scalable)
 ;   for (int i=0; i<N; ++i) {
 ;     a[i + 64] = a[i] + b[i];
 ;   }
 ; }

llvm/test/Transforms/LoopVectorize/metadata-width.ll

-; RUN: opt < %s  -loop-vectorize -force-vector-interleave=1 -force-target-supports-scalable-vectors=true -dce -instcombine -S | FileCheck %s
+; RUN: opt < %s  -loop-vectorize -force-vector-interleave=1 -force-target-supports-scalable-vectors=true -dce -instcombine -S -scalable-vectorization=on | FileCheck %s
+; RUN: opt < %s  -loop-vectorize -force-vector-interleave=1 -force-target-supports-scalable-vectors=true -dce -instcombine -S -scalable-vectorization=preferred | FileCheck %s
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 
 ; CHECK-LABEL: @test1(
 ; CHECK: store <8 x i32>
 ; CHECK: ret void
 define void @test1(i32* nocapture %a, i32 %n) #0 {
 entry:
for.body:                                         ; preds = %entry, %for.body
   %lftr.wideiv = trunc i64 %indvars.iv.next to i32
   %exitcond = icmp eq i32 %lftr.wideiv, %n
   br i1 %exitcond, label %for.end, label %for.body, !llvm.loop !4
 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
+; Test that when only the width property is specified, and not the scalable.enable flag,
+; that the hint defaults to 'scalable.enable = false'.
+
+; CHECK-LABEL: @test4(
+; CHECK: store <8 x i32>
+; CHECK: ret void
+define void @test4(i32* nocapture %a, i32 %n) #0 {
+entry:
+  %cmp4 = icmp sgt i32 %n, 0
+  br i1 %cmp4, label %for.body, label %for.end
+
+for.body:                                         ; preds = %entry, %for.body
+  %indvars.iv = phi i64 [ %indvars.iv.next, %for.body ], [ 0, %entry ]
+  %arrayidx = getelementptr inbounds i32, i32* %a, i64 %indvars.iv
+  store i32 42, i32* %arrayidx, align 4
+  %indvars.iv.next = add i64 %indvars.iv, 1
+  %lftr.wideiv = trunc i64 %indvars.iv.next to i32
+  %exitcond = icmp eq i32 %lftr.wideiv, %n
+  br i1 %exitcond, label %for.end, label %for.body, !llvm.loop !6
+
+for.end:                                          ; preds = %for.body, %entry
+  ret void
+}
+
 attributes #0 = { nounwind uwtable "less-precise-fpmad"="false" "frame-pointer"="none" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "unsafe-fp-math"="false" "use-soft-float"="false" }
 
-!0 = !{!0, !1}
+!0 = !{!0, !1, !5}
 !1 = !{!"llvm.loop.vectorize.width", i32 8}
 !2 = !{!2, !1, !3}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i32 1}
 !4 = !{!4, !1, !5}
 !5 = !{!"llvm.loop.vectorize.scalable.enable", i32 0}
+!6 = !{!6, !7}
+!7 = !{!"llvm.loop.vectorize.width", i32 8}

llvm/test/Transforms/LoopVectorize/optimal-epilog-vectorization-scalable.ll

 ; REQUIRES: asserts
-; RUN: opt < %s  -passes='loop-vectorize' -force-vector-width=2 -force-target-supports-scalable-vectors=true -enable-epilogue-vectorization -epilogue-vectorization-force-VF=2 --debug-only=loop-vectorize -S 2>&1 | FileCheck %s
+; RUN: opt < %s  -passes='loop-vectorize' -force-vector-width=2 -force-target-supports-scalable-vectors=true -enable-epilogue-vectorization -epilogue-vectorization-force-VF=2 --debug-only=loop-vectorize -S -scalable-vectorization=on 2>&1 | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-n32:64-v256:256:256-v512:512:512"
 
 ; Currently we cannot handle scalable vectorization factors.
 ; CHECK: LV: Checking a loop in "f1"
 ; CHECK: LEV: Epilogue vectorization for scalable vectors not yet supported.
 
 define void @f1(i8* %A) {

llvm/test/Transforms/LoopVectorize/scalable-first-order-recurrence.ll

-; RUN: opt -loop-vectorize -force-vector-width=4 -force-vector-interleave=1 -force-target-supports-scalable-vectors=true -S < %s | FileCheck %s --check-prefix=CHECK-VF4UF1
-; RUN: opt -loop-vectorize -force-vector-width=4 -force-vector-interleave=2 -force-target-supports-scalable-vectors=true -S < %s | FileCheck %s --check-prefix=CHECK-VF4UF2
+; RUN: opt -loop-vectorize -scalable-vectorization=on -force-vector-width=4 -force-vector-interleave=1 -force-target-supports-scalable-vectors=true -S < %s | FileCheck %s --check-prefix=CHECK-VF4UF1
+; RUN: opt -loop-vectorize -scalable-vectorization=on -force-vector-width=4 -force-vector-interleave=2 -force-target-supports-scalable-vectors=true -S < %s | FileCheck %s --check-prefix=CHECK-VF4UF2
 
 ; void recurrence_1(int *a, int *b, int n) {
 ;   for(int i = 0; i < n; i++)
 ;     b[i] =  a[i] + a[i - 1]
 ; }
 ;
 define void @recurrence_1(i32* nocapture readonly %a, i32* nocapture %b, i32 %n) {
 ; CHECK-VF4UF1-LABEL: @recurrence_1

llvm/test/Transforms/LoopVectorize/scalable-loop-unpredicated-body-scalar-tail.ll

-; RUN: opt -S -loop-vectorize -instcombine -force-vector-interleave=1 -force-vector-width=4 -force-target-supports-scalable-vectors=true < %s | FileCheck %s --check-prefix=CHECKUF1
-; RUN: opt -S -loop-vectorize -instcombine -force-vector-interleave=2 -force-vector-width=4 -force-target-supports-scalable-vectors=true < %s | FileCheck %s --check-prefix=CHECKUF2
+; RUN: opt -S -loop-vectorize -instcombine -force-vector-interleave=1 -force-vector-width=4 -force-target-supports-scalable-vectors=true -scalable-vectorization=on < %s | FileCheck %s --check-prefix=CHECKUF1
+; RUN: opt -S -loop-vectorize -instcombine -force-vector-interleave=2 -force-vector-width=4 -force-target-supports-scalable-vectors=true -scalable-vectorization=on < %s | FileCheck %s --check-prefix=CHECKUF2
 
 ; CHECKUF1: for.body.preheader:
 ; CHECKUF1-DAG: %wide.trip.count = zext i32 %N to i64
 ; CHECKUF1-DAG: %[[VSCALE:.*]] = call i64 @llvm.vscale.i64()
 ; CHECKUF1-DAG: %[[VSCALEX4:.*]] = shl i64 %[[VSCALE]], 2
 ; CHECKUF1-DAG: %min.iters.check = icmp ugt i64 %[[VSCALEX4]], %wide.trip.count
 
 ; CHECKUF1: vector.ph:

llvm/test/Transforms/LoopVectorize/scalable-reduction-inloop.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -loop-vectorize -force-target-supports-scalable-vectors=true -S | FileCheck %s
+; RUN: opt < %s -loop-vectorize -force-target-supports-scalable-vectors=true -scalable-vectorization=on -S | FileCheck %s
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 
 define i8 @reduction_add_trunc(i8* noalias nocapture %A) {
 ; CHECK-LABEL: @reduction_add_trunc(
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, %vector.ph ], [ [[INDEX_NEXT:%.*]], %vector.body ]
 ; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <vscale x 8 x i32> [ insertelement (<vscale x 8 x i32> zeroinitializer, i32 255, i32 0), %vector.ph ], [ [[TMP34:%.*]], %vector.body ]

llvm/test/Transforms/LoopVectorize/scalable-vf-hint.ll

 ; REQUIRES: asserts
-; RUN: opt -loop-vectorize -pass-remarks-analysis=loop-vectorize -debug-only=loop-vectorize -S < %s 2>&1 | FileCheck %s
+; RUN: opt -loop-vectorize -pass-remarks-analysis=loop-vectorize -debug-only=loop-vectorize -S -scalable-vectorization=on < %s 2>&1 | FileCheck %s
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 
 ; CHECK: LV: Disabling scalable vectorization, because target does not support scalable vectors.
 ; CHECK: remark: <unknown>:0:0: Disabling scalable vectorization, because target does not support scalable vectors.
 ; CHECK: LV: The Widest register safe to use is: 32 bits.
 define void @test1(i32* %a, i32* %b) {
 entry: