[LV] Calculate max feasible scalable VF.

[sdesmalen]

Summary

This patch also refactors the way the feasible max VF is calculated,
although this is NFC for fixed-width vectors.

After this change scalable VF hints are no longer truncated/clamped
to a shorter scalable VF, nor does it drop the 'scalable flag' from
the suggested VF to vectorize with a similar VF that is fixed.

Instead, the hint is ignored which means the vectorizer is free
to find a more suitable VF, using the CostModel to determine the
best possible VF.

[sdesmalen]

Following discussion on D96021, this patch combines functionality in D96021, D96022 and D96023, and is aligns more closely with the approach outlined by @fhahn.
It does mean the patch is quite a bit bigger and therefore more difficult to review. I'm happy to split off parts of it into separate patches if that's useful.

[paulwalker-arm]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

5818–5821

This doesn't look particularly clean to my eye. I guess my immediate question is why WidestRegister is not a TypeSize given that would truly represent the size.

The same question can be asked of getRegisterBitWidth but perhaps that is a heavily used function? If so then what about introducing an explicit getVectorRegisterBitWidth that returns TypeSize plus allowing it to take a parameter that specifies whether the query is against fixed or scalable vectors.

I can see why you're asking for ScalableBitsPerBlock because you want to make sure you're able to safely divide this by the largest element count, but here I think using TypeSize.isKnownMultipleOf(SizeOfLargestElt) makes that intent clearer for all vector types.

[sdesmalen]

Rebased onto D98874, which changes getRegisterBitWidth to return a TypeSize.

[david-arm]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

138 ↗

(On Diff #331592)

Hi @sdesmalen, to be honest I'm not sure I fully understand this. If the user has set the pragma:

#pragma clang loop vectorize_width(4, scalable)

then haven't they also explicitly disabled fixed width? Maybe it's just the name of the function that confuses me a little, since from the user's perspective it feels like

#pragma clang loop vectorize_width(4, scalable)

and

#pragma clang loop vectorize_width(scalable)

are both disabling fixed width. I thought they were both hints that can be dropped by the compiler if necessary?

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

1624

Should this be element count, since that's what the function returns here?

5641

Isn't this always going to fire for cases where UserVF is scalable and we cannot vectorise reductions? For example, getMaxLegalScalableVF returns ElementCount::getScalable(0) in this case.

5684

Are you deliberately ignoring the scalable case for now because this will be addressed in a later patch?

5839

You could also do

if (MaxVectorSize.isZero())

[c-rhodes]

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

126 ↗

(On Diff #331592)

nit: explicitly disabled (?)

llvm/include/llvm/Transforms/Vectorize/LoopVectorize.h

177

nit: Reports an informative vectorization message:

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

1067–1068

drop failure

5581

nit: comma can be dropped

5678

nit: MaxVF?

5832–5835

MaxVF?

llvm/test/Transforms/LoopVectorize/AArch64/scalable-vf-hint.ll

185–188

Should this select the feasible scalable VF? I suppose this is related to the issue @david-arm pointed out in LoopVectorizationCostModel::computeFeasibleMaxVF

[fhahn]

llvm/include/llvm/Analysis/TargetTransformInfo.h

933 ↗

(On Diff #331592)

This is not needed any longer, right?

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

138 ↗

(On Diff #331592)

Would it be possible to decouple the change to compute the max VFs from the ones adding a new option & tweaking the handling here?

[sdesmalen]

• Dropped -scalable-vectorization= LV option from this patch.
• Addressed other comments.

llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h

138 ↗

(On Diff #331592)

I've removed this code for now, as I'll have a follow-up patch that adds an option to enable/disable scalable vectorization separately.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

5641

The statement above if (ElementCount::isKnownLE(UserVF, MaxSafeUserVF)) return UserVF should make sure that this assert is always true.

5684

Correct!

llvm/test/Transforms/LoopVectorize/AArch64/scalable-vf-hint.ll

185–188

I don't think it should do that by default. Personally I think it's best if the vectorizer just ignores hints that are not valid, and calculate a better answer instead, rather than blindly opting for something that seems "close enough", but may not be profitable.

That's akin to what happens for fixed-width vectors; if the UserVF is not legal (e.g. VF=16), and VF=8 would be legal, it will pick the most profitable VF <= 8, which isn't necessarily 8. Extending this to scalable vectors, we can argue that if VF=vscale x 8 is not legal, it should consider all VFs (fixed and scalable) up to whatever is legal. We can then add an LV option to favour scalable VFs if we know the cost-model doesn't have all the information to make the right decision (although that's something to consider for another patch).

[sdesmalen]

Thanks for all the comments and feedback!

llvm/include/llvm/Analysis/TargetTransformInfo.h

933 ↗

(On Diff #331592)

You're right, I've removed that now.

[c-rhodes]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

5579

Flip this and return early?

5596–5598

should this be moved closer to where it's used?

5666–5668

could you add a comment stating this is ignored for now but will be addressed in a later patch?

5835

I know this came before your patch but I find it a bit confusing VF and VectorSize are use interchangeably

llvm/test/Transforms/LoopVectorize/AArch64/scalable-vf-analysis.ll

3

nit: multiple spaces

33

nit: add newline above

[sdesmalen]

• Addressed nits.
• s/MaxVectorSize/MaxVectorElementCount/ (because it's not actually a size, but an element count).

[sdesmalen]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

5596–5598

This is actually moved as close as possible, because the first use is WidestType (line 5591).

5835

I see how this is confusing, especially because the VectorSize is actually not a size, but rather an element count. The subtle distinction between the two is that the VF is more of a loop-vectorizer concept (the number of lanes handled per vector iteration), whereas the MaxVectorSize (or better yet, MaxVectorElementCount) is the maximum number of elements in a target's vector register.

Given that I'm changing a lot of lines in this function, it probably doesn't really make the diff much worse if I at least remove part of the confusion by renaming VectorSize -> VectorElementCount.

[c-rhodes]

LGTM

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

5596–5598

This is actually moved as close as possible, because the first use is WidestType (line 5591).

[fhahn]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

1624

the other function refer to as VF factor as well, even though they return ElementCount. Using VF (or vectorization factor) seems more consistent with the rest of the code and also more descriptive I think, as it conveys some information on what the intended use is.

1626

The other related functions refer to VF instead of element count in their name. Is there a reason to use ElementCount instead of VF here? Perhaps the name could be made a bit more descriptive in general, including the fact that the function computes the max VF based on target-specific properties?

[sdesmalen]

• s/getMaxVectorElementCount/getMaximizedVFForTarget/
• Removed redundant call to isSafeForAnyVectorWidth

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

1626

You're right that using VF is probably better/more consistent, I mostly chose ElementCount so not to avoid it with other MaxVF-related functions. I've now changed it to getMaximizedVFForTarget, does that make sense to you?

I've chosen the term 'Maximized' instead of 'Max' to distinguish it from 'Maximum', as in:
The LV tries to Maximize the VF based on the target's register width, but with a Maximum of MaxSafeVF.

[sdesmalen]

I think I've addressed all comments now. Unless there are any other, I'll commit the patch in the coming days.

[fhahn]

LGTM, thanks for the updates!

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

1626

Sounds good to me, but I am no expert of the English language :)

[sdesmalen]

Thank you for all the input/feedback/reviews @fhahn and @c-rhodes!

[hubert.reinterpretcast]

Reverting this patch locally resolves the following failure (and many more) for me:
https://lab.llvm.org/buildbot/#/builders/140/builds/2/steps/5/logs/FAIL__Clang__loop-vectorize_c

[hubert.reinterpretcast]

In D98509#2724684, @hubert.reinterpretcast wrote:

Reverting this patch locally resolves the following failure (and many more) for me:
https://lab.llvm.org/buildbot/#/builders/140/builds/2/steps/5/logs/FAIL__Clang__loop-vectorize_c

Trying to figure out if this patch is running into UB. Disabling optimizations in the build seems to resolve these failures.

[sdesmalen]

In D98509#2724752, @hubert.reinterpretcast wrote:

In D98509#2724684, @hubert.reinterpretcast wrote:

Reverting this patch locally resolves the following failure (and many more) for me:
https://lab.llvm.org/buildbot/#/builders/140/builds/2/steps/5/logs/FAIL__Clang__loop-vectorize_c

Trying to figure out if this patch is running into UB. Disabling optimizations in the build seems to resolve these failure

Hi @hubert.reinterpretcast, I've built this patch with UBSan but that didn't point out any issues. Do you have any suggestions on how to reproduce this locally? (I see that buildbot is using some IBM xlclang (proprietary?) compiler to build)

[hubert.reinterpretcast]

In D98509#2725326, @sdesmalen wrote:

Hi @hubert.reinterpretcast, I've built this patch with UBSan but that didn't point out any issues. Do you have any suggestions on how to reproduce this locally? (I see that buildbot is using some IBM xlclang (proprietary?) compiler to build)

I'm actively working on diagnosing this. Would reverting this patch for a few days be much trouble for you? We're hoping to keep our CI green so it's easier to notice new problems.

[sdesmalen]

In D98509#2725542, @hubert.reinterpretcast wrote:

In D98509#2725326, @sdesmalen wrote:

Hi @hubert.reinterpretcast, I've built this patch with UBSan but that didn't point out any issues. Do you have any suggestions on how to reproduce this locally? (I see that buildbot is using some IBM xlclang (proprietary?) compiler to build)

I'm actively working on diagnosing this.

Thanks for looking into this!

Would reverting this patch for a few days be much trouble for you? We're hoping to keep our CI green so it's easier to notice new problems.

Reverting the patch for a few days is fine, I've pushed rG51d648c119d7773ce6fb809353bd6bd14bca8818 for now.

[hubert.reinterpretcast]

In D98509#2725672, @sdesmalen wrote:

Would reverting this patch for a few days be much trouble for you? We're hoping to keep our CI green so it's easier to notice new problems.

Reverting the patch for a few days is fine, I've pushed rG51d648c119d7773ce6fb809353bd6bd14bca8818 for now.

Thank you! Our internal CI is green again. I'll update with the investigation progress once it gets to a meaningful point.

[hubert.reinterpretcast]

In D98509#2725892, @hubert.reinterpretcast wrote:

Thank you! Our internal CI is green again. I'll update with the investigation progress once it gets to a meaningful point.

Maybe a paranoid step, but I confirmed that the impact is from the compilation of lib/Transforms/Vectorize/CMakeFiles/LLVMVectorize.dir/LoopVectorize.cpp.o. Proceeding with cutting down from there.

[hubert.reinterpretcast]

In D98509#2725892, @hubert.reinterpretcast wrote:

In D98509#2725672, @sdesmalen wrote:

Reverting the patch for a few days is fine, I've pushed rG51d648c119d7773ce6fb809353bd6bd14bca8818 for now.

Thank you! Our internal CI is green again. I'll update with the investigation progress once it gets to a meaningful point.

I haven't reached the root cause yet, but it appears that the issue we're hitting can be avoided if the MaxSafeVF parameter is made pass-by-reference instead of pass-by-value.
@sdesmalen, I hope the following is not too intrusive a change to your patch.

diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 2b413fc4950..ec3eb32cbe6 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -1639,7 +1639,7 @@ private:
   ElementCount getMaximizedVFForTarget(unsigned ConstTripCount,
                                        unsigned SmallestType,
                                        unsigned WidestType,
-                                       ElementCount MaxSafeVF);
+                                       const ElementCount &MaxSafeVF);

   /// \return the maximum legal scalable VF, based on the safe max number
   /// of elements.
@@ -5863,7 +5863,7 @@ LoopVectorizationCostModel::computeMaxVF(ElementCount UserVF, unsigned UserIC) {

 ElementCount LoopVectorizationCostModel::getMaximizedVFForTarget(
     unsigned ConstTripCount, unsigned SmallestType, unsigned WidestType,
-    ElementCount MaxSafeVF) {
+    const ElementCount &MaxSafeVF) {
   bool ComputeScalableMaxVF = MaxSafeVF.isScalable();
   TypeSize WidestRegister = TTI.getRegisterBitWidth(
       ComputeScalableMaxVF ? TargetTransformInfo::RGK_ScalableVector

[sdesmalen]

In D98509#2732628, @hubert.reinterpretcast wrote:

I haven't reached the root cause yet, but it appears that the issue we're hitting can be avoided if the MaxSafeVF parameter is made pass-by-reference instead of pass-by-value.
@sdesmalen, I hope the following is not too intrusive a change to your patch.

Hi @hubert.reinterpretcast, thanks for your investigation so far! I have gone through the code several times, but don't really understand why your suggestion fixes the issue. I've also built it with both GCC and Clang, with- and without sanitizers, and with valgrind.

Having said that, I'd be okay with applying your suggestion for the sake of re-landing the patch and not breaking a specific buildbot, but I would like to aim for a concrete date to remove the work-around. What do you think is sensible here?

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

5824

@hubert.reinterpretcast Here MinVF returns a const ElementCount & instead of ElementCount, not sure if that could make a difference?

[fhahn]

In D98509#2732628, @hubert.reinterpretcast wrote:

In D98509#2725892, @hubert.reinterpretcast wrote:

In D98509#2725672, @sdesmalen wrote:

Reverting the patch for a few days is fine, I've pushed rG51d648c119d7773ce6fb809353bd6bd14bca8818 for now.

Thank you! Our internal CI is green again. I'll update with the investigation progress once it gets to a meaningful point.

I haven't reached the root cause yet, but it appears that the issue we're hitting can be avoided if the MaxSafeVF parameter is made pass-by-reference instead of pass-by-value.
@sdesmalen, I hope the following is not too intrusive a change to your patch.

@hubert.reinterpretcast can you provide an IR test case that triggers the failure with sanitizers?

[hubert.reinterpretcast]

In D98509#2736317, @fhahn wrote:

In D98509#2732628, @hubert.reinterpretcast wrote:

In D98509#2725892, @hubert.reinterpretcast wrote:

In D98509#2725672, @sdesmalen wrote:

Reverting the patch for a few days is fine, I've pushed rG51d648c119d7773ce6fb809353bd6bd14bca8818 for now.

Thank you! Our internal CI is green again. I'll update with the investigation progress once it gets to a meaningful point.

I haven't reached the root cause yet, but it appears that the issue we're hitting can be avoided if the MaxSafeVF parameter is made pass-by-reference instead of pass-by-value.
@sdesmalen, I hope the following is not too intrusive a change to your patch.

@hubert.reinterpretcast can you provide an IR test case that triggers the failure with sanitizers?

The build compiler on the specific bot (due to what is currently shipping on the platform) is a proprietary compiler that does not use LLVM IR. We have not reached that point where we can confirm that this is a miscompile issue in the build compiler, but I think there is every chance that would be the conclusion (now that the reduction has reached the point where it is -- we have also encountered latent UB in LLVM source in the past). Prior issues of this sort have taken around two weeks to fix after it is reported to the appropriate team (which is delayed while I'm putting together a reduction). I had spent my whole weekend narrowing this down. I am not sure how much time I will have during the week to make further progress. I think we will be able to deploy a fix to the buildbot in early June.

[hubert.reinterpretcast]

In D98509#2736263, @sdesmalen wrote:

In D98509#2732628, @hubert.reinterpretcast wrote:

I haven't reached the root cause yet, but it appears that the issue we're hitting can be avoided if the MaxSafeVF parameter is made pass-by-reference instead of pass-by-value.
@sdesmalen, I hope the following is not too intrusive a change to your patch.

Hi @hubert.reinterpretcast, thanks for your investigation so far! I have gone through the code several times, but don't really understand why your suggestion fixes the issue. I've also built it with both GCC and Clang, with- and without sanitizers, and with valgrind.

Thanks for looking into this from your end. I feel terrible that I wasn't more clear that a build compiler issue is looking to be likely.

[sdesmalen]

In D98509#2737470, @hubert.reinterpretcast wrote:

Thanks for looking into this from your end. I feel terrible that I wasn't more clear that a build compiler issue is looking to be likely.

No worries, at least we have come up with a practical workaround for now. Please do let me know when this issue will be fixed in the compiler used by Buildbot, so that I can remove the workaround.

Thanks for all the effort you have put into investigating this!

[MaskRay]

Disabling scalable vectorization, because target does not support scalable vectors. should not be emitted for non-aarch64 non-riscv.
Probably also a good idea not to emit it when sve is disabled on aarch64.

% cat a.cc
float A[128], B[128], C[128];

float foo() {
  float sum = 0;
  for (int i = 0; i < 128; i++)
    C[i] = A[i] + B[i];
  return sum;
}
% clang++ -O2 --target=x86_64-linux-gnu -S a.cc -Rpass=loop-vectorize -Rpass-missed=loop-vectorize -Rpass-analysis=loop-vectorize
a.cc:5:3: remark: Disabling scalable vectorization, because target does not support scalable vectors. [-Rpass-analysis=loop-vectorize]
  for (int i = 0; i < 128; i++)
  ^
a.cc:5:3: remark: vectorized loop (vectorization width: 4, interleaved count: 2) [-Rpass=loop-vectorize]

Patch: D108004

llvm/include/llvm/Transforms/Vectorize/LoopVectorize.h

 /// Reports a vectorization failure: print \p DebugMsg for debugging
 /// purposes along with the corresponding optimization remark \p RemarkName.
 /// If \p I is passed, it is an instruction that prevents vectorization.
 /// Otherwise, the loop \p TheLoop is used for the location of the remark.
 void reportVectorizationFailure(const StringRef DebugMsg,
     const StringRef OREMsg, const StringRef ORETag,
     OptimizationRemarkEmitter *ORE, Loop *TheLoop, Instruction *I = nullptr);
 
+/// Reports an informative message: print \p Msg for debugging purposes as well
+/// as an optimization remark. Uses either \p I as location of the remark, or
+/// otherwise \p TheLoop.
+void reportVectorizationInfo(const StringRef OREMsg, const StringRef ORETag,
+                             OptimizationRemarkEmitter *ORE, Loop *TheLoop,
+                             Instruction *I = nullptr);
+
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_VECTORIZE_LOOPVECTORIZE_H

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

if (DIL && Inst->getFunction()->isDebugInfoForProfiling() &&
                    << "Failed to create new discriminator: "
                    << DIL->getFilename() << " Line: " << DIL->getLine());
     }
     else
       B.SetCurrentDebugLocation(DIL);
   } else
     B.SetCurrentDebugLocation(DebugLoc());
 }
 
-/// Write a record \p DebugMsg about vectorization failure to the debug
-/// output stream. If \p I is passed, it is an instruction that prevents
-/// vectorization.
+/// Write a \p DebugMsg about vectorization to the debug output stream. If \p I
+/// is passed, the message relates to that particular instruction.
 #ifndef NDEBUG
-static void debugVectorizationFailure(const StringRef DebugMsg,
+static void debugVectorizationMessage(const StringRef Prefix,
+                                      const StringRef DebugMsg,
                                       Instruction *I) {
-  dbgs() << "LV: Not vectorizing: " << DebugMsg;
+  dbgs() << "LV: " << Prefix << DebugMsg;
   if (I != nullptr)
     dbgs() << " " << *I;
   else
     dbgs() << '.';
   dbgs() << '\n';
 }
 #endif
 
static OptimizationRemarkAnalysis createLVAnalysis(const char *PassName,
   if (I) {
     CodeRegion = I->getParent();
     // If there is no debug location attached to the instruction, revert back to
     // using the loop's.
     if (I->getDebugLoc())
       DL = I->getDebugLoc();
   }
 
-  OptimizationRemarkAnalysis R(PassName, RemarkName, DL, CodeRegion);
-  R << "loop not vectorized: ";
-  return R;
+  return OptimizationRemarkAnalysis(PassName, RemarkName, DL, CodeRegion);
 }
 
 /// Return a value for Step multiplied by VF.
 static Value *createStepForVF(IRBuilder<> &B, Constant *Step, ElementCount VF) {
   assert(isa<ConstantInt>(Step) && "Expected an integer step");
   Constant *StepVal = ConstantInt::get(
       Step->getType(),
       cast<ConstantInt>(Step)->getSExtValue() * VF.getKnownMinValue());
   return VF.isScalable() ? B.CreateVScale(StepVal) : StepVal;
 }
 
 namespace llvm {
 
 void reportVectorizationFailure(const StringRef DebugMsg,
                                 const StringRef OREMsg, const StringRef ORETag,
-    OptimizationRemarkEmitter *ORE, Loop *TheLoop, Instruction *I) {
-  LLVM_DEBUG(debugVectorizationFailure(DebugMsg, I));
+                                OptimizationRemarkEmitter *ORE, Loop *TheLoop,
+                                Instruction *I) {
+  LLVM_DEBUG(debugVectorizationMessage("Not vectorizing: ", DebugMsg, I));
   LoopVectorizeHints Hints(TheLoop, true /* doesn't matter */, *ORE);
-  ORE->emit(createLVAnalysis(Hints.vectorizeAnalysisPassName(),
-                ORETag, TheLoop, I) << OREMsg);
+  ORE->emit(
+      createLVAnalysis(Hints.vectorizeAnalysisPassName(), ORETag, TheLoop, I)
+      << "loop not vectorized: " << OREMsg);
+}
+
+void reportVectorizationInfo(const StringRef Msg, const StringRef ORETag,
+                             OptimizationRemarkEmitter *ORE, Loop *TheLoop,
+                             Instruction *I) {
+  LLVM_DEBUG(debugVectorizationMessage("", Msg, I));
+  LoopVectorizeHints Hints(TheLoop, true /* doesn't matter */, *ORE);
+  ORE->emit(
+      createLVAnalysis(Hints.vectorizeAnalysisPassName(), ORETag, TheLoop, I)
+      << Msg);
 }
 
 } // end namespace llvm
 
 #ifndef NDEBUG
 /// \return string containing a file name and a line # for the given loop.
 static std::string getDebugLocString(const Loop *L) {
   std::string Result;
private:
   unsigned NumPredStores = 0;
 
   /// \return An upper bound for the vectorization factor, a power-of-2 larger
   /// than zero. One is returned if vectorization should best be avoided due
   /// to cost.
   ElementCount computeFeasibleMaxVF(unsigned ConstTripCount,
                                     ElementCount UserVF);
 
+  /// \return the maximum element count based on the target's vector registers,
+  /// limited to MaxVF. This is a helper function of computeFeasibleMaxVF.
+  ElementCount getMaxVectorElementCount(unsigned ConstTripCount,
+                                        unsigned SmallestType,
+                                        unsigned WidestType,
+                                        ElementCount MaxVF);
+
+  /// \return the maximum legal scalable VF, based on the safe max number
+  /// of elements.
+  ElementCount getMaxLegalScalableVF(unsigned MaxSafeElements);
+
   /// The vectorization cost is a combination of the cost itself and a boolean
   /// indicating whether any of the contributing operations will actually
   /// operate on
   /// vector values after type legalization in the backend. If this latter value
   /// is
   /// false, then all operations will be scalarized (i.e. no vectorization has
   /// actually taken place).
   using VectorizationCostTy = std::pair<InstructionCost, bool>;
reportVectorizationFailure("Runtime stride check for small trip count",
         "this loop without such check by compiling with -Os/-Oz",
         "CantVersionLoopWithOptForSize", ORE, TheLoop);
     return true;
   }
 
   return false;
 }
 
+ElementCount
+LoopVectorizationCostModel::getMaxLegalScalableVF(unsigned MaxSafeElements) {
+  if (!TTI.supportsScalableVectors() && !ForceTargetSupportsScalableVectors) {
+    reportVectorizationInfo(
+        "Disabling scalable vectorization, because target does not "
+        "support scalable vectors.",
+        "ScalableVectorsUnsupported", ORE, TheLoop);
+    return ElementCount::getScalable(0);
+  }
+
+  auto MaxScalableVF = ElementCount::getScalable(1u << 16);
+
+  // Disable scalable vectorization if the loop contains unsupported reductions.
+  // Test that the loop-vectorizer can legalize all operations for this MaxVF.
+  // FIXME: While for scalable vectors this is currently sufficient, this should
+  // be replaced by a more detailed mechanism that filters out specific VFs,
+  // instead of invalidating vectorization for a whole set of VFs based on the
+  // MaxVF.
+  if (!canVectorizeReductions(MaxScalableVF)) {
+    reportVectorizationInfo(
+        "Scalable vectorization not supported for the reduction "
+        "operations found in this loop.",
+        "ScalableVFUnfeasible", ORE, TheLoop);
+    return ElementCount::getScalable(0);
+  }
+
+  if (Legal->isSafeForAnyVectorWidth())
+    return MaxScalableVF;
+
+  // Limit MaxScalableVF by the maximum safe dependence distance.
+  Optional<unsigned> MaxVScale = TTI.getMaxVScale();
+  MaxScalableVF = ElementCount::getScalable(
+      MaxVScale ? (MaxSafeElements / MaxVScale.getValue()) : 0);
+  if (!MaxScalableVF)
+    reportVectorizationInfo(
+        "Max legal vector width too small, scalable vectorization "
+        "unfeasible.",
+        "ScalableVFUnfeasible", ORE, TheLoop);
+
+  return MaxScalableVF;
+}
+
+ElementCount
+LoopVectorizationCostModel::computeFeasibleMaxVF(unsigned ConstTripCount,
+                                                 ElementCount UserVF) {
+  MinBWs = computeMinimumValueSizes(TheLoop->getBlocks(), *DB, &TTI);
+  unsigned SmallestType, WidestType;
+  std::tie(SmallestType, WidestType) = getSmallestAndWidestTypes();
+
+  // Get the maximum safe dependence distance in bits computed by LAA.
+  // It is computed by MaxVF * sizeOf(type) * 8, where type is taken from
+  // the memory accesses that is most restrictive (involved in the smallest
+  // dependence distance).
+  unsigned MaxSafeElements =
+      PowerOf2Floor(Legal->getMaxSafeVectorWidthInBits() / WidestType);
+
+  auto MaxSafeFixedVF = ElementCount::getFixed(MaxSafeElements);
+  auto MaxSafeScalableVF = getMaxLegalScalableVF(MaxSafeElements);
+
+  LLVM_DEBUG(dbgs() << "LV: The max safe fixed VF is: " << MaxSafeFixedVF
+                    << ".\n");
+  LLVM_DEBUG(dbgs() << "LV: The max safe scalable VF is: " << MaxSafeScalableVF
+                    << ".\n");
+
+  // First analyze the UserVF, fall back if the UserVF should be ignored.
+  if (UserVF) {
+    auto MaxSafeUserVF =
+        UserVF.isScalable() ? MaxSafeScalableVF : MaxSafeFixedVF;
+
+    if (ElementCount::isKnownLE(UserVF, MaxSafeUserVF))
+      return UserVF;
+
+    assert(ElementCount::isKnownGT(UserVF, MaxSafeUserVF));
+
+    // Only clamp if the UserVF is not scalable. If the UserVF is scalable, it
+    // is better to ignore the hint and let the compiler choose a suitable VF.
+    if (!UserVF.isScalable()) {
+      LLVM_DEBUG(dbgs() << "LV: User VF=" << UserVF
+                        << " is unsafe, clamping to max safe VF="
+                        << MaxSafeFixedVF << ".\n");
+      ORE->emit([&]() {
+        return OptimizationRemarkAnalysis(DEBUG_TYPE, "VectorizationFactor",
+                                          TheLoop->getStartLoc(),
+                                          TheLoop->getHeader())
+               << "User-specified vectorization factor "
+               << ore::NV("UserVectorizationFactor", UserVF)
+               << " is unsafe, clamping to maximum safe vectorization factor "
+               << ore::NV("VectorizationFactor", MaxSafeFixedVF);
+      });
+      return MaxSafeFixedVF;
+    }
+
+    LLVM_DEBUG(dbgs() << "LV: User VF=" << UserVF
+                      << " is unsafe. Ignoring scalable UserVF.\n");
+    ORE->emit([&]() {
+      return OptimizationRemarkAnalysis(DEBUG_TYPE, "VectorizationFactor",
+                                        TheLoop->getStartLoc(),
+                                        TheLoop->getHeader())
+             << "User-specified vectorization factor "
+             << ore::NV("UserVectorizationFactor", UserVF)
+             << " is unsafe. Ignoring the hint to let the compiler pick a "
+                "suitable VF.";
+    });
+  }
+
+  LLVM_DEBUG(dbgs() << "LV: The Smallest and Widest types: " << SmallestType
+                    << " / " << WidestType << " bits.\n");
+
+  ElementCount MaxFixedVF = ElementCount::getFixed(1);
+  if (auto MaxVF = getMaxVectorElementCount(ConstTripCount, SmallestType,
+                                            WidestType, MaxSafeFixedVF))
+    MaxFixedVF = MaxVF;
+
+  if (auto MaxVF = getMaxVectorElementCount(ConstTripCount, SmallestType,
+                                            WidestType, MaxSafeScalableVF))
+    // FIXME: Return scalable VF as well (to be added in future patch).
+    if (MaxVF.isScalable())
+      LLVM_DEBUG(dbgs() << "LV: Found feasible scalable VF = " << MaxVF
+                        << "\n");
+
+  return MaxFixedVF;
+}
+
 Optional<ElementCount>
 LoopVectorizationCostModel::computeMaxVF(ElementCount UserVF, unsigned UserIC) {
   if (Legal->getRuntimePointerChecking()->Need && TTI.hasBranchDivergence()) {
     // TODO: It may by useful to do since it's still likely to be dynamically
     // uniform if the target can skip.
     reportVectorizationFailure(
         "Not inserting runtime ptr check for divergent target",
         "runtime pointer checks needed. Not enabled for divergent target",
         "CantVersionLoopWithDivergentTarget", ORE, TheLoop);
     return None;
   }
 
   unsigned TC = PSE.getSE()->getSmallConstantTripCount(TheLoop);
   LLVM_DEBUG(dbgs() << "LV: Found trip count: " << TC << '\n');
   if (TC == 1) {
     reportVectorizationFailure("Single iteration (non) loop",
         "loop trip count is one, irrelevant for vectorization",
         "SingleIterationLoop", ORE, TheLoop);
reportVectorizationFailure(
       "Cannot optimize for size and vectorize at the same time.",
       "cannot optimize for size and vectorize at the same time. "
       "Enable vectorization of this loop with '#pragma clang loop "
       "vectorize(enable)' when compiling with -Os/-Oz",
       "NoTailLoopWithOptForSize", ORE, TheLoop);
   return None;
 }
 
-ElementCount
-LoopVectorizationCostModel::computeFeasibleMaxVF(unsigned ConstTripCount,
-                                                 ElementCount UserVF) {
-  bool IgnoreScalableUserVF = UserVF.isScalable() &&
-                              !TTI.supportsScalableVectors() &&
-                              !ForceTargetSupportsScalableVectors;
-  if (IgnoreScalableUserVF) {
-    LLVM_DEBUG(
-        dbgs() << "LV: Ignoring VF=" << UserVF
-               << " because target does not support scalable vectors.\n");
-    ORE->emit([&]() {
-      return OptimizationRemarkAnalysis(DEBUG_TYPE, "IgnoreScalableUserVF",
-                                        TheLoop->getStartLoc(),
-                                        TheLoop->getHeader())
-             << "Ignoring VF=" << ore::NV("UserVF", UserVF)
-             << " because target does not support scalable vectors.";
-    });
-  }
-
-  // Beyond this point two scenarios are handled. If UserVF isn't specified
-  // then a suitable VF is chosen. If UserVF is specified and there are
-  // dependencies, check if it's legal. However, if a UserVF is specified and
-  // there are no dependencies, then there's nothing to do.
-  if (UserVF.isNonZero() && !IgnoreScalableUserVF) {
-    if (!canVectorizeReductions(UserVF)) {
-      reportVectorizationFailure(
-          "LV: Scalable vectorization not supported for the reduction "
-          "operations found in this loop. Using fixed-width "
-          "vectorization instead.",
-          "Scalable vectorization not supported for the reduction operations "
-          "found in this loop. Using fixed-width vectorization instead.",
-          "ScalableVFUnfeasible", ORE, TheLoop);
-      return computeFeasibleMaxVF(
-          ConstTripCount, ElementCount::getFixed(UserVF.getKnownMinValue()));
-    }
-
-    if (Legal->isSafeForAnyVectorWidth())
-      return UserVF;
-  }
-
-  MinBWs = computeMinimumValueSizes(TheLoop->getBlocks(), *DB, &TTI);
-  unsigned SmallestType, WidestType;
-  std::tie(SmallestType, WidestType) = getSmallestAndWidestTypes();
-  unsigned WidestRegister =
-      TTI.getRegisterBitWidth(TargetTransformInfo::RGK_FixedWidthVector)
-          .getFixedSize();
-
-  // Get the maximum safe dependence distance in bits computed by LAA.
-  // It is computed by MaxVF * sizeOf(type) * 8, where type is taken from
-  // the memory accesses that is most restrictive (involved in the smallest
-  // dependence distance).
-  unsigned MaxSafeVectorWidthInBits = Legal->getMaxSafeVectorWidthInBits();
-
-  // If the user vectorization factor is legally unsafe, clamp it to a safe
-  // value. Otherwise, return as is.
-  if (UserVF.isNonZero() && !IgnoreScalableUserVF) {
-    unsigned MaxSafeElements =
-        PowerOf2Floor(MaxSafeVectorWidthInBits / WidestType);
-    ElementCount MaxSafeVF = ElementCount::getFixed(MaxSafeElements);
-
-    if (UserVF.isScalable()) {
-      Optional<unsigned> MaxVScale = TTI.getMaxVScale();
-
-      // Scale VF by vscale before checking if it's safe.
-      MaxSafeVF = ElementCount::getScalable(
-          MaxVScale ? (MaxSafeElements / MaxVScale.getValue()) : 0);
-
-      if (MaxSafeVF.isZero()) {
-        // The dependence distance is too small to use scalable vectors,
-        // fallback on fixed.
-        LLVM_DEBUG(
-            dbgs()
-            << "LV: Max legal vector width too small, scalable vectorization "
-               "unfeasible. Using fixed-width vectorization instead.\n");
-        ORE->emit([&]() {
-          return OptimizationRemarkAnalysis(DEBUG_TYPE, "ScalableVFUnfeasible",
-                                            TheLoop->getStartLoc(),
-                                            TheLoop->getHeader())
-                 << "Max legal vector width too small, scalable vectorization "
-                 << "unfeasible. Using fixed-width vectorization instead.";
-        });
-        return computeFeasibleMaxVF(
-            ConstTripCount, ElementCount::getFixed(UserVF.getKnownMinValue()));
-      }
-    }
-
-    LLVM_DEBUG(dbgs() << "LV: The max safe VF is: " << MaxSafeVF << ".\n");
-
-    if (ElementCount::isKnownLE(UserVF, MaxSafeVF))
-      return UserVF;
-
-    LLVM_DEBUG(dbgs() << "LV: User VF=" << UserVF
-                      << " is unsafe, clamping to max safe VF=" << MaxSafeVF
-                      << ".\n");
-    ORE->emit([&]() {
-      return OptimizationRemarkAnalysis(DEBUG_TYPE, "VectorizationFactor",
-                                        TheLoop->getStartLoc(),
-                                        TheLoop->getHeader())
-             << "User-specified vectorization factor "
-             << ore::NV("UserVectorizationFactor", UserVF)
-             << " is unsafe, clamping to maximum safe vectorization factor "
-             << ore::NV("VectorizationFactor", MaxSafeVF);
-    });
-    return MaxSafeVF;
-  }
-
-  WidestRegister = std::min(WidestRegister, MaxSafeVectorWidthInBits);
+ElementCount LoopVectorizationCostModel::getMaxVectorElementCount(
+    unsigned ConstTripCount, unsigned SmallestType, unsigned WidestType,
+    ElementCount MaxSafeVF) {
+  bool ComputeScalableMaxVF = MaxSafeVF.isScalable();
+  TypeSize WidestRegister = TTI.getRegisterBitWidth(
+      ComputeScalableMaxVF ? TargetTransformInfo::RGK_ScalableVector
+                           : TargetTransformInfo::RGK_FixedWidthVector);
+
+  // Convenience function to return the minimum of two ElementCounts.
+  auto MinVF = [](const ElementCount &LHS, const ElementCount &RHS) {
+    assert((LHS.isScalable() == RHS.isScalable()) &&
+           "Scalable flags must match");
+    return ElementCount::isKnownLT(LHS, RHS) ? LHS : RHS;
+  };
 
   // Ensure MaxVF is a power of 2; the dependence distance bound may not be.
   // Note that both WidestRegister and WidestType may not be a powers of 2.
-  auto MaxVectorSize =
-      ElementCount::getFixed(PowerOf2Floor(WidestRegister / WidestType));
-
-  LLVM_DEBUG(dbgs() << "LV: The Smallest and Widest types: " << SmallestType
-                    << " / " << WidestType << " bits.\n");
+  auto MaxVectorElementCount = ElementCount::get(
+      PowerOf2Floor(WidestRegister.getKnownMinSize() / WidestType),
+      ComputeScalableMaxVF);
+  MaxVectorElementCount = MinVF(MaxVectorElementCount, MaxSafeVF);
   LLVM_DEBUG(dbgs() << "LV: The Widest register safe to use is: "
-                    << WidestRegister << " bits.\n");
+                    << (MaxVectorElementCount * WidestType) << " bits.\n");
 
-  assert(MaxVectorSize.getFixedValue() <= WidestRegister &&
-         "Did not expect to pack so many elements"
-         " into one vector!");
-  if (MaxVectorSize.getFixedValue() == 0) {
+  if (!MaxVectorElementCount) {
     LLVM_DEBUG(dbgs() << "LV: The target has no vector registers.\n");
     return ElementCount::getFixed(1);
-  } else if (ConstTripCount && ConstTripCount < MaxVectorSize.getFixedValue() &&
-             isPowerOf2_32(ConstTripCount)) {
+  }
+
+  const auto TripCountEC = ElementCount::getFixed(ConstTripCount);
+  if (ConstTripCount &&
+      ElementCount::isKnownLE(TripCountEC, MaxVectorElementCount) &&
     // We need to clamp the VF to be the ConstTripCount. There is no point in
-    // choosing a higher viable VF as done in the loop below.
+    // choosing a higher viable VF as done in the loop below. If
+    // MaxVectorElementCount is scalable, we only fall back on a fixed VF when
+    // the TC is less than or equal to the known number of lanes.
     LLVM_DEBUG(dbgs() << "LV: Clamping the MaxVF to the constant trip count: "
                       << ConstTripCount << "\n");
-    return ElementCount::getFixed(ConstTripCount);
+    return TripCountEC;
   }
 
-  ElementCount MaxVF = MaxVectorSize;
+  ElementCount MaxVF = MaxVectorElementCount;
   if (TTI.shouldMaximizeVectorBandwidth(!isScalarEpilogueAllowed()) ||
       (MaximizeBandwidth && isScalarEpilogueAllowed())) {
+    auto MaxVectorElementCountMaxBW = ElementCount::get(
+        PowerOf2Floor(WidestRegister.getKnownMinSize() / SmallestType),
+        ComputeScalableMaxVF);
+    if (!Legal->isSafeForAnyVectorWidth())
+      MaxVectorElementCountMaxBW = MinVF(MaxVectorElementCountMaxBW, MaxSafeVF);
+
     // Collect all viable vectorization factors larger than the default MaxVF
-    // (i.e. MaxVectorSize).
+    // (i.e. MaxVectorElementCount).
     SmallVector<ElementCount, 8> VFs;
-    auto MaxVectorSizeMaxBW =
-        ElementCount::getFixed(WidestRegister / SmallestType);
-    for (ElementCount VS = MaxVectorSize * 2;
-         ElementCount::isKnownLE(VS, MaxVectorSizeMaxBW); VS *= 2)
+    for (ElementCount VS = MaxVectorElementCount * 2;
+         ElementCount::isKnownLE(VS, MaxVectorElementCountMaxBW); VS *= 2)
       VFs.push_back(VS);
 
     // For each VF calculate its register usage.
     auto RUs = calculateRegisterUsage(VFs);
 
     // Select the largest VF which doesn't require more registers than existing
     // ones.
     for (int i = RUs.size() - 1; i >= 0; --i) {
       bool Selected = true;
       for (auto &pair : RUs[i].MaxLocalUsers) {
         unsigned TargetNumRegisters = TTI.getNumberOfRegisters(pair.first);
         if (pair.second > TargetNumRegisters)
           Selected = false;
       }
       if (Selected) {
         MaxVF = VFs[i];
         break;
       }
     }
     if (ElementCount MinVF =
-            TTI.getMinimumVF(SmallestType, /*IsScalable=*/false)) {
+            TTI.getMinimumVF(SmallestType, ComputeScalableMaxVF)) {
       if (ElementCount::isKnownLT(MaxVF, MinVF)) {
         LLVM_DEBUG(dbgs() << "LV: Overriding calculated MaxVF(" << MaxVF
                           << ") with target's minimum: " << MinVF << '\n');
         MaxVF = MinVF;
       }
     }
   }
   return MaxVF;

llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll

for.body:
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond.not = icmp eq i64 %iv.next, %n
   br i1 %exitcond.not, label %for.end, label %for.body, !llvm.loop !0
 
 for.end:
   ret float %add
 }
 
-; CHECK-REMARK: Scalable vectorization not supported for the reduction operations found in this loop. Using fixed-width vectorization instead.
+; CHECK-REMARK: Scalable vectorization not supported for the reduction operations found in this loop.
 ; CHECK-REMARK: vectorized loop (vectorization width: 8, interleaved count: 2)
 define bfloat @fadd_fast_bfloat(bfloat* noalias nocapture readonly %a, i64 %n) {
 ; CHECK-LABEL: @fadd_fast_bfloat
 ; CHECK: vector.body:
 ; CHECK: %[[LOAD1:.*]] = load <8 x bfloat>
 ; CHECK: %[[LOAD2:.*]] = load <8 x bfloat>
 ; CHECK: %[[FADD1:.*]] = fadd fast <8 x bfloat> %[[LOAD1]]
 ; CHECK: %[[FADD2:.*]] = fadd fast <8 x bfloat> %[[LOAD2]]

 for.end:
   ret float %.sroa.speculated
 }
 
 ; Reduction cannot be vectorized
 
 ; MUL
 
-; CHECK-REMARK: Scalable vectorization not supported for the reduction operations found in this loop. Using fixed-width vectorization instead.
-; CHECK-REMARK: vectorized loop (vectorization width: 8, interleaved count: 2)
+; CHECK-REMARK: Scalable vectorization not supported for the reduction operations found in this loop.
+; CHECK-REMARK: vectorized loop (vectorization width: 4, interleaved count: 2)
 define i32 @mul(i32* nocapture %a, i32* nocapture readonly %b, i64 %n) {
 ; CHECK-LABEL: @mul
 ; CHECK: vector.body:
-; CHECK: %[[LOAD1:.*]] = load <8 x i32>
-; CHECK: %[[LOAD2:.*]] = load <8 x i32>
-; CHECK: %[[MUL1:.*]] = mul <8 x i32> %[[LOAD1]]
-; CHECK: %[[MUL2:.*]] = mul <8 x i32> %[[LOAD2]]
+; CHECK: %[[LOAD1:.*]] = load <4 x i32>
+; CHECK: %[[LOAD2:.*]] = load <4 x i32>
+; CHECK: %[[MUL1:.*]] = mul <4 x i32> %[[LOAD1]]
+; CHECK: %[[MUL2:.*]] = mul <4 x i32> %[[LOAD2]]
 ; CHECK: middle.block:
-; CHECK: %[[RDX:.*]] = mul <8 x i32> %[[MUL2]], %[[MUL1]]
-; CHECK: call i32 @llvm.vector.reduce.mul.v8i32(<8 x i32> %[[RDX]])
+; CHECK: %[[RDX:.*]] = mul <4 x i32> %[[MUL2]], %[[MUL1]]
+; CHECK: call i32 @llvm.vector.reduce.mul.v4i32(<4 x i32> %[[RDX]])
 entry:
   br label %for.body
 
 for.body:                                         ; preds = %entry, %for.body
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
   %sum.07 = phi i32 [ 2, %entry ], [ %mul, %for.body ]
   %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
   %0 = load i32, i32* %arrayidx, align 4
   %mul = mul nsw i32 %0, %sum.07
   %iv.next = add nuw nsw i64 %iv, 1
   %exitcond.not = icmp eq i64 %iv.next, %n
   br i1 %exitcond.not, label %for.end, label %for.body, !llvm.loop !0
 
 for.end:                                 ; preds = %for.body, %entry
   ret i32 %mul
 }
 
 ; Note: This test was added to ensure we always check the legality of reductions (end emit a warning if necessary) before checking for memory dependencies
-; CHECK-REMARK: Scalable vectorization not supported for the reduction operations found in this loop. Using fixed-width vectorization instead.
-; CHECK-REMARK: vectorized loop (vectorization width: 8, interleaved count: 2)
+; CHECK-REMARK: Scalable vectorization not supported for the reduction operations found in this loop.
+; CHECK-REMARK: vectorized loop (vectorization width: 4, interleaved count: 2)
 define i32 @memory_dependence(i32* noalias nocapture %a, i32* noalias nocapture readonly %b, i64 %n) {
 ; CHECK-LABEL: @memory_dependence
 ; CHECK: vector.body:
-; CHECK: %[[LOAD1:.*]] = load <8 x i32>
-; CHECK: %[[LOAD2:.*]] = load <8 x i32>
-; CHECK: %[[LOAD3:.*]] = load <8 x i32>
-; CHECK: %[[LOAD4:.*]] = load <8 x i32>
-; CHECK: %[[ADD1:.*]] = add nsw <8 x i32> %[[LOAD3]], %[[LOAD1]]
-; CHECK: %[[ADD2:.*]] = add nsw <8 x i32> %[[LOAD4]], %[[LOAD2]]
-; CHECK: %[[MUL1:.*]] = mul <8 x i32> %[[LOAD3]]
-; CHECK: %[[MUL2:.*]] = mul <8 x i32> %[[LOAD4]]
+; CHECK: %[[LOAD1:.*]] = load <4 x i32>
+; CHECK: %[[LOAD2:.*]] = load <4 x i32>
+; CHECK: %[[LOAD3:.*]] = load <4 x i32>
+; CHECK: %[[LOAD4:.*]] = load <4 x i32>
+; CHECK: %[[ADD1:.*]] = add nsw <4 x i32> %[[LOAD3]], %[[LOAD1]]
+; CHECK: %[[ADD2:.*]] = add nsw <4 x i32> %[[LOAD4]], %[[LOAD2]]
+; CHECK: %[[MUL1:.*]] = mul <4 x i32> %[[LOAD3]]
+; CHECK: %[[MUL2:.*]] = mul <4 x i32> %[[LOAD4]]
 ; CHECK: middle.block:
-; CHECK: %[[RDX:.*]] = mul <8 x i32> %[[MUL2]], %[[MUL1]]
-; CHECK: call i32 @llvm.vector.reduce.mul.v8i32(<8 x i32> %[[RDX]])
+; CHECK: %[[RDX:.*]] = mul <4 x i32> %[[MUL2]], %[[MUL1]]
+; CHECK: call i32 @llvm.vector.reduce.mul.v4i32(<4 x i32> %[[RDX]])
 entry:
   br label %for.body
 
 for.body:
   %i = phi i64 [ %inc, %for.body ], [ 0, %entry ]
   %sum = phi i32 [ %mul, %for.body ], [ 2, %entry ]
   %arrayidx = getelementptr inbounds i32, i32* %a, i64 %i
   %0 = load i32, i32* %arrayidx, align 4

llvm/test/Transforms/LoopVectorize/AArch64/scalable-vf-analysis.ll

+; REQUIRES: asserts
+; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -force-target-instruction-cost=1 -loop-vectorize -S -debug-only=loop-vectorize < %s 2>&1 | FileCheck %s --check-prefixes=CHECK,CHECK_SCALABLE_ON
+; RUN: opt -mtriple=aarch64-none-linux-gnu -mattr=+sve -force-target-instruction-cost=1 -loop-vectorize -S -debug-only=loop-vectorize -vectorizer-maximize-bandwidth < %s 2>&1 | FileCheck %s --check-prefixes=CHECK,CHECK_SCALABLE_ON_MAXBW
+
+; Test that the MaxVF for the following loop, that has no dependence distances,
+; is calculated as vscale x 4 (max legal SVE vector size) or vscale x 16
+; (maximized bandwidth for i8 in the loop).
+define void @test0(i32* %a, i8* %b, i32* %c) {
+; CHECK: LV: Checking a loop in "test0"
+; CHECK_SCALABLE_ON: LV: Found feasible scalable VF = vscale x 4
+; CHECK_SCALABLE_ON_MAXBW: LV: Found feasible scalable VF = vscale x 16
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+  %arrayidx = getelementptr inbounds i32, i32* %c, i64 %iv
+  %0 = load i32, i32* %arrayidx, align 4
+  %arrayidx2 = getelementptr inbounds i8, i8* %b, i64 %iv
+  %1 = load i8, i8* %arrayidx2, align 4
+  %zext = zext i8 %1 to i32
+  %add = add nsw i32 %zext, %0
+  %arrayidx5 = getelementptr inbounds i32, i32* %a, i64 %iv
+  store i32 %add, i32* %arrayidx5, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond.not = icmp eq i64 %iv.next, 1024
+  br i1 %exitcond.not, label %exit, label %loop, !llvm.loop !0
+
+exit:
+  ret void
+}
+
+; Test that the MaxVF for the following loop, with a dependence distance
+; of 64 elements, is calculated as (maxvscale = 16) * 4.
+define void @test1(i32* %a, i8* %b) {
+; CHECK: LV: Checking a loop in "test1"
+; CHECK_SCALABLE_ON: LV: Found feasible scalable VF = vscale x 4
+; CHECK_SCALABLE_ON_MAXBW: LV: Found feasible scalable VF = vscale x 4
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+  %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
+  %0 = load i32, i32* %arrayidx, align 4
+  %arrayidx2 = getelementptr inbounds i8, i8* %b, i64 %iv
+  %1 = load i8, i8* %arrayidx2, align 4
+  %zext = zext i8 %1 to i32
+  %add = add nsw i32 %zext, %0
+  %2 = add nuw nsw i64 %iv, 64
+  %arrayidx5 = getelementptr inbounds i32, i32* %a, i64 %2
+  store i32 %add, i32* %arrayidx5, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond.not = icmp eq i64 %iv.next, 1024
+  br i1 %exitcond.not, label %exit, label %loop, !llvm.loop !0
+
+exit:
+  ret void
+}
+
+; Test that the MaxVF for the following loop, with a dependence distance
+; of 32 elements, is calculated as (maxvscale = 16) * 2.
+define void @test2(i32* %a, i8* %b) {
+; CHECK: LV: Checking a loop in "test2"
+; CHECK_SCALABLE_ON: LV: Found feasible scalable VF = vscale x 2
+; CHECK_SCALABLE_ON_MAXBW: LV: Found feasible scalable VF = vscale x 2
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+  %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
+  %0 = load i32, i32* %arrayidx, align 4
+  %arrayidx2 = getelementptr inbounds i8, i8* %b, i64 %iv
+  %1 = load i8, i8* %arrayidx2, align 4
+  %zext = zext i8 %1 to i32
+  %add = add nsw i32 %zext, %0
+  %2 = add nuw nsw i64 %iv, 32
+  %arrayidx5 = getelementptr inbounds i32, i32* %a, i64 %2
+  store i32 %add, i32* %arrayidx5, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond.not = icmp eq i64 %iv.next, 1024
+  br i1 %exitcond.not, label %exit, label %loop, !llvm.loop !0
+
+exit:
+  ret void
+}
+
+; Test that the MaxVF for the following loop, with a dependence distance
+; of 16 elements, is calculated as (maxvscale = 16) * 1.
+define void @test3(i32* %a, i8* %b) {
+; CHECK: LV: Checking a loop in "test3"
+; CHECK_SCALABLE_ON: LV: Found feasible scalable VF = vscale x 1
+; CHECK_SCALABLE_ON_MAXBW: LV: Found feasible scalable VF = vscale x 1
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+  %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
+  %0 = load i32, i32* %arrayidx, align 4
+  %arrayidx2 = getelementptr inbounds i8, i8* %b, i64 %iv
+  %1 = load i8, i8* %arrayidx2, align 4
+  %zext = zext i8 %1 to i32
+  %add = add nsw i32 %zext, %0
+  %2 = add nuw nsw i64 %iv, 16
+  %arrayidx5 = getelementptr inbounds i32, i32* %a, i64 %2
+  store i32 %add, i32* %arrayidx5, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond.not = icmp eq i64 %iv.next, 1024
+  br i1 %exitcond.not, label %exit, label %loop, !llvm.loop !0
+
+exit:
+  ret void
+}
+
+; Test the fallback mechanism when scalable vectors are not feasible due
+; to e.g. dependence distance. For the '-scalable-vectorization=exclusive'
+; it shouldn't try to vectorize with fixed-width vectors.
+define void @test4(i32* %a, i32* %b) {
+; CHECK: LV: Checking a loop in "test4"
+; CHECK_SCALABLE_ON-NOT: LV: Found feasible scalable VF
+; CHECK_SCALABLE_ON_MAXBW-NOT: LV: Found feasible scalable VF
+entry:
+  br label %loop
+
+loop:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+  %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
+  %0 = load i32, i32* %arrayidx, align 4
+  %arrayidx2 = getelementptr inbounds i32, i32* %b, i64 %iv
+  %1 = load i32, i32* %arrayidx2, align 4
+  %add = add nsw i32 %1, %0
+  %2 = add nuw nsw i64 %iv, 8
+  %arrayidx5 = getelementptr inbounds i32, i32* %a, i64 %2
+  store i32 %add, i32* %arrayidx5, align 4
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond.not = icmp eq i64 %iv.next, 1024
+  br i1 %exitcond.not, label %exit, label %loop, !llvm.loop !2
+
+exit:
+  ret void
+}
+
+!0 = distinct !{!0, !1}
+!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = distinct !{!2, !3, !4}
+!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}

llvm/test/Transforms/LoopVectorize/AArch64/scalable-vf-hint.ll

 ;     a[i + 8] = a[i] + b[i];
 ;   }
 ; }
 ;
 ; For scalable vectorization 'vscale' has to be considered, for this example
 ; unless max(vscale)=2 it's unsafe to vectorize. For SVE max(vscale)=16, check
 ; fixed-width vectorization is used instead.
 
-; CHECK-DBG: LV: Max legal vector width too small, scalable vectorization unfeasible. Using fixed-width vectorization instead.
-; CHECK-DBG: remark: <unknown>:0:0: Max legal vector width too small, scalable vectorization unfeasible. Using fixed-width vectorization instead.
-; CHECK-DBG: LV: The max safe VF is: 8.
+; CHECK-DBG: LV: Checking a loop in "test1"
+; CHECK-DBG: LV: Max legal vector width too small, scalable vectorization unfeasible.
+; CHECK-DBG: remark: <unknown>:0:0: Max legal vector width too small, scalable vectorization unfeasible.
+; CHECK-DBG: LV: The max safe fixed VF is: 8.
 ; CHECK-DBG: LV: Selecting VF: 4.
 ; CHECK-LABEL: @test1
 ; CHECK: <4 x i32>
 define void @test1(i32* %a, i32* %b) {
 entry:
   br label %loop
 
 loop:

 ;
 ; void test2(int *a, int *b, int N) {
 ;   #pragma clang loop vectorize(enable) vectorize_width(8, scalable)
 ;   for (int i=0; i<N; ++i) {
 ;     a[i + 4] = a[i] + b[i];
 ;   }
 ; }
 
-; CHECK-DBG: LV: Max legal vector width too small, scalable vectorization unfeasible. Using fixed-width vectorization instead.
-; CHECK-DBG: LV: The max safe VF is: 4.
-; CHECK-DBG: LV: User VF=8 is unsafe, clamping to max safe VF=4.
+; CHECK-DBG: LV: Checking a loop in "test2"
+; CHECK-DBG: LV: Max legal vector width too small, scalable vectorization unfeasible.
+; CHECK-DBG: LV: The max safe fixed VF is: 4.
+; CHECK-DBG: LV: User VF=vscale x 8 is unsafe. Ignoring scalable UserVF.
 ; CHECK-DBG: LV: Selecting VF: 4.
 ; CHECK-LABEL: @test2
 ; CHECK: <4 x i32>
 define void @test2(i32* %a, i32* %b) {
 entry:
   br label %loop
 
 loop:

 ;   for (int i=0; i<N; ++i) {
 ;     a[i + 32] = a[i] + b[i];
 ;   }
 ; }
 ;
 ; Max fixed VF=32, Max scalable VF=2, safe to vectorize.
 
 ; CHECK-DBG-LABEL: LV: Checking a loop in "test3"
-; CHECK-DBG: LV: The max safe VF is: vscale x 2.
+; CHECK-DBG: LV: The max safe scalable VF is: vscale x 2.
 ; CHECK-DBG: LV: Using user VF vscale x 2.
 ; CHECK-LABEL: @test3
 ; CHECK: <vscale x 2 x i32>
 define void @test3(i32* %a, i32* %b) {
 entry:
   br label %loop
 
 loop:

 }
 
 !6 = !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.width", i32 2}
 !8 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 
 ; test4
 ;
-; Scalable vectorization feasible, but the VF is unsafe. Should clamp.
+; Scalable vectorization feasible, but the given VF is unsafe. Should ignore
+; the hint and leave it to the vectorizer to pick a more suitable VF.
 ;
 ; Specifies a vector of <vscale x 4 x i32>, i.e. maximum of 64 x i32 with 4
 ; words per 128-bits (packed).
 ;
 ; void test4(int *a, int *b, int N) {
 ;   #pragma clang loop vectorize(enable) vectorize_width(4, scalable)
 ;   for (int i=0; i<N; ++i) {
 ;     a[i + 32] = a[i] + b[i];
 ;   }
 ; }
 ;
-; Max fixed VF=32, Max scalable VF=2, unsafe to vectorize. Should clamp to 2.
+; Max fixed VF=32, Max scalable VF=2, unsafe to vectorize.
 
 ; CHECK-DBG-LABEL: LV: Checking a loop in "test4"
-; CHECK-DBG: LV: The max safe VF is: vscale x 2.
-; CHECK-DBG: LV: User VF=vscale x 4 is unsafe, clamping to max safe VF=vscale x 2.
-; CHECK-DBG: remark: <unknown>:0:0: User-specified vectorization factor vscale x 4 is unsafe, clamping to maximum safe vectorization factor vscale x 2
-; CHECK-DBG: LV: Using max VF vscale x 2
+; CHECK-DBG: LV: The max safe scalable VF is: vscale x 2.
+; CHECK-DBG: LV: User VF=vscale x 4 is unsafe. Ignoring scalable UserVF.
+; CHECK-DBG: remark: <unknown>:0:0: User-specified vectorization factor vscale x 4 is unsafe. Ignoring the hint to let the compiler pick a suitable VF.
+; CHECK-DBG: Found feasible scalable VF = vscale x 2
+; CHECK-DBG: LV: Selecting VF: 4.
 ; CHECK-LABEL: @test4
-; CHECK: <vscale x 2 x i32>
+; CHECK: <4 x i32>
 define void @test4(i32* %a, i32* %b) {
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
   %0 = load i32, i32* %arrayidx, align 4

 ;   for (int i=0; i<N; ++i) {
 ;     a[i + 128] = a[i] + b[i];
 ;   }
 ; }
 ;
 ; Max fixed VF=128, Max scalable VF=8, safe to vectorize.
 
 ; CHECK-DBG-LABEL: LV: Checking a loop in "test5"
-; CHECK-DBG: LV: The max safe VF is: vscale x 8.
+; CHECK-DBG: LV: The max safe scalable VF is: vscale x 8.
 ; CHECK-DBG: LV: Using user VF vscale x 4
 ; CHECK-LABEL: @test5
 ; CHECK: <vscale x 4 x i32>
 define void @test5(i32* %a, i32* %b) {
 entry:
   br label %loop
 
 loop:

 }
 
 !12 = !{!12, !13, !14}
 !13 = !{!"llvm.loop.vectorize.width", i32 4}
 !14 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 
 ; test6
 ;
-; Scalable vectorization feasible, but the VF is unsafe. Should clamp.
+; Scalable vectorization feasible, but the VF is unsafe. Should ignore
+; the hint and leave it to the vectorizer to pick a more suitable VF.
 ;
 ; Specifies a vector of <vscale x 16 x i32>, i.e. maximum of 256 x i32.
 ;
 ; void test6(int *a, int *b, int N) {
 ;   #pragma clang loop vectorize(enable) vectorize_width(16, scalable)
 ;   for (int i=0; i<N; ++i) {
 ;     a[i + 128] = a[i] + b[i];
 ;   }
 ; }
 ;
-; Max fixed VF=128, Max scalable VF=8, unsafe to vectorize. Should clamp to 8.
+; Max fixed VF=128, Max scalable VF=8, unsafe to vectorize.
 
 ; CHECK-DBG-LABEL: LV: Checking a loop in "test6"
-; CHECK-DBG: LV: The max safe VF is: vscale x 8.
-; CHECK-DBG: LV: User VF=vscale x 16 is unsafe, clamping to max safe VF=vscale x 8.
-; CHECK-DBG: remark: <unknown>:0:0: User-specified vectorization factor vscale x 16 is unsafe, clamping to maximum safe vectorization factor vscale x 8
-; CHECK-DBG: LV: Using max VF vscale x 8
+; CHECK-DBG: LV: The max safe scalable VF is: vscale x 8.
+; CHECK-DBG: LV: User VF=vscale x 16 is unsafe. Ignoring scalable UserVF.
+; CHECK-DBG: remark: <unknown>:0:0: User-specified vectorization factor vscale x 16 is unsafe. Ignoring the hint to let the compiler pick a suitable VF.
+; CHECK-DBG: LV: Found feasible scalable VF = vscale x 4
+; CHECK-DBG: Selecting VF: 4.
 ; CHECK-LABEL: @test6
-; CHECK: <vscale x 8 x i32>
+; CHECK: <4 x i32>
 define void @test6(i32* %a, i32* %b) {
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv
   %0 = load i32, i32* %arrayidx, align 4
exit:
   ret void
 }
 
 !15 = !{!15, !16, !17}
 !16 = !{!"llvm.loop.vectorize.width", i32 16}
 !17 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 
 ; CHECK-NO-SVE-LABEL: LV: Checking a loop in "test_no_sve"
-; CHECK-NO-SVE: LV: Ignoring VF=vscale x 4 because target does not support scalable vectors.
-; CHECK-NO-SVE: remark: <unknown>:0:0: Ignoring VF=vscale x 4 because target does not support scalable vectors.
+; CHECK-NO-SVE: LV: Disabling scalable vectorization, because target does not support scalable vectors.
+; CHECK-NO-SVE: remark: <unknown>:0:0: Disabling scalable vectorization, because target does not support scalable vectors.
+; CHECK-NO-SVE: LV: User VF=vscale x 4 is unsafe. Ignoring scalable UserVF.
 ; CHECK-NO-SVE: LV: Selecting VF: 4.
 ; CHECK-NO-SVE: <4 x i32>
 ; CHECK-NO-SVE-NOT: <vscale x 4 x i32>
 define void @test_no_sve(i32* %a, i32* %b) {
 entry:
   br label %loop
 
 loop:

 !18 = !{!18, !19, !20}
 !19 = !{!"llvm.loop.vectorize.width", i32 4}
 !20 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 
 ; Test the LV falls back to fixed-width vectorization if scalable vectors are
 ; supported but max vscale is undefined.
 ;
 ; CHECK-NO-MAX-VSCALE-LABEL: LV: Checking a loop in "test_no_max_vscale"
-; CHECK-NO-MAX-VSCALE: LV: Max legal vector width too small, scalable vectorization unfeasible. Using fixed-width vectorization instead.
-; CEHCK-NO-MAX-VSCALE: The max safe VF is: 4.
+; CEHCK-NO-MAX-VSCALE: The max safe fixed VF is: 4.
+; CHECK-NO-MAX-VSCALE: LV: User VF=vscale x 4 is unsafe. Ignoring scalable UserVF.
 ; CHECK-NO-MAX-VSCALE: LV: Selecting VF: 4.
 ; CHECK-NO-MAX-VSCALE: <4 x i32>
 define void @test_no_max_vscale(i32* %a, i32* %b) {
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]

llvm/test/Transforms/LoopVectorize/scalable-vf-hint.ll

 ; REQUIRES: asserts
 ; RUN: opt -loop-vectorize -pass-remarks-analysis=loop-vectorize -debug-only=loop-vectorize -S < %s 2>&1 | FileCheck %s
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 
-; CHECK: LV: Ignoring VF=vscale x 4 because target does not support scalable vectors.
-; CHECK: remark: <unknown>:0:0: Ignoring VF=vscale x 4 because target does not support scalable vectors.
+; CHECK: LV: Disabling scalable vectorization, because target does not support scalable vectors.
+; CHECK: remark: <unknown>:0:0: Disabling scalable vectorization, because target does not support scalable vectors.
 ; CHECK: LV: The Widest register safe to use is: 32 bits.
 define void @test1(i32* %a, i32* %b) {
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %arrayidx = getelementptr inbounds i32, i32* %a, i64 %iv