[LoopVectorize] Pass a vector type to isLegalMaskedGather/Scatter

[RosieSumpter]

Summary

This is required to query the legality more precisely in the LoopVectorizer.

This adds another TTI function named 'forceScalarizeMaskedGather/Scatter'
function to work around the hack introduced for MVE, where
isLegalMaskedGather/Scatter would return an answer by second-guessing
where the function was called from, based on the Type passed in (vector
vs scalar). The new interface makes this explicit. It is also used by
X86 to check for vector widths where gather/scatters aren't profitable
(or don't exist) for certain subtargets.

[david-arm]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

1590

According to the default VF = ElementCount::getFixed(1) it looks like we can still have an inconsistency during vectorisation where we get different answers? I think it's also called from collectElementTypesForWidening so we could potentially say a gather is legal when the type is scalar, but then say it's illegal in setCostBasedWideningDecision. I wonder if it's worth adding a comment there saying something like:

// We're simply querying at this point if the target even supports any vector
// gathers and scatters for the given element type? Certain vector forms may
// still be illegal, but setCostBasedWideningDecision will distinguish between
// the legal behaviour for different VFs at that point and generate costs accordingly.

Also, does it matter that LoopVectorizationCostModel::isScalarWithPredication is still passing in an element type rather than a vector? For example, collectLoopUniforms calls isScalarWithPredication and at that point knows the VF. It may be fine, but I'm just a bit worried we might be swapping one inconsistency for another that's all.

[sdesmalen]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

1590

The code in collectElementTypesForWidening seems like a hack to get the LV to choose a wider VF and it seems specific to X86. If I replace that whole expression with if (!T->isSized()) continue, I only get a single test that fails because it explicitly tests for it to use a wider VF. I doubt this is the right mechanism to use for that kind of purpose going forward.

In any case, if collectElementTypesForWidening would consider using a wider VF, it's still the cost-model that will then narrow that down (after this patch), because passing in the VF will result in a more restricted answer for isLegalMaskedGather.

Also, does it matter that LoopVectorizationCostModel::isScalarWithPredication is still passing in an element type rather than a vector? For example, collectLoopUniforms calls isScalarWithPredication and at that point knows the VF. It may be fine, but I'm just a bit worried we might be swapping one inconsistency for another that's all.

I don't know if there's an actual problem atm, but I agree it makes sense to pass it to isScalarWithPredication as well. It should be possible since VF is available in all functions that call it.

[dmgreen]

MVE part sounds OK to me. Is the X86 part expected to be the same or better?

[lebedev.ri]

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

5122–5139

Can you move this code into forceScalarizeMaskedGather()?
This should not have been here, LV should rely on cost model.

[RosieSumpter]

• Moved code which checks if masked gathers/scatters are legal for the given vector width to forceScalarizeMaskedGather/Scatter for X86
• Pass VF to isScalarWithPredication so that we are consistently passing a vector type to isLegalMaskedGather/Scatter
• Add minimum vscale attribute to the necessary AArch64 tests
• Remove changes to X86 tests (X86 behaviour is now unchanged by this patch)

[sdesmalen]

@lebedev.ri/@fhahn given that there exists a pass that scalarizes the masked load/store/gather/scatter intrinsics, is there a reason why we wouldn't always want to choose the intrinsic representation over scalarizing it in the LV? It seems a bit odd to want to trick the LV to think that a masked gather is legal, but then still scalarize it in the end. This means it is not using the right cost-model, unless it reimplements the scalarization costs, like is done by X86TargetTransformInfo::getGSScalarCost.

llvm/lib/Target/ARM/ARMTargetTransformInfo.h

194

nit: s/can to/can lower to/

195–196

nit: s/, so if we are here, we know we want to expand//

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

5125

This is casting to FixedVectorType, so I think we should we make the interface take VectorType instead of Type.

[lebedev.ri]

In D115329#3188834, @sdesmalen wrote:

@lebedev.ri/@fhahn given that there exists a pass that scalarizes the masked load/store/gather/scatter intrinsics,
is there a reason why we wouldn't always want to choose the intrinsic representation over scalarizing it in the LV?
It seems a bit odd to want to trick the LV to think that a masked gather is legal, but then still scalarize it in the end.
This means it is not using the right cost-model, unless it reimplements the scalarization costs,
like is done by X86TargetTransformInfo::getGSScalarCost.

The fact that something is/isn't legal does not mean that LV will pick *that* particular path.
It merely allows LV to consider this path, among the others,
and out of all the legal paths, pick the most profitable one.

We don't "trick LV to think it's legal and then scalarize anyway", it *is* legal.
But for those particular VF's, it's more costly than the native scalarized path.
So LV won't actually produce gather intrinsics, and naturally, they won't be scalarized.

Furthermore, no, while we could just always prefer the intrinsics (from correctness viewpoint),
that is detrimental to the performance. See e.g. my recent patches, namely D111220 / D111363 / D111546.

So no, what happens is correct.

[sdesmalen]

In D115329#3188963, @lebedev.ri wrote:

The fact that something is/isn't legal does not mean that LV will pick *that* particular path.
It merely allows LV to consider this path, among the others,
and out of all the legal paths, pick the most profitable one.

The LV makes the decision to use gathers/scatters because the cost-model said the gather/scatter cost was lower than the scalarization cost, but that only works if the gather/scatter cost is correct. If it will be scalarized, then I'd expect this costs to be no different than the scalarization cost.

We don't "trick LV to think it's legal and then scalarize anyway", it *is* legal.

I'm not sure what the meaning of "legal" is in that case. My understanding of the term is that it means that the target can handle the operation directly without legalization, e.g. using a special instruction or a sequence of instructions. From that perspective, saying that <operation> is legal, means it will not need to be scalarized. (conversely, the ScalarizeMaskedOperations pass only scalarizes the intrinsic if it is *not* marked as legal). If the LV asks the question "is a masked gather operation legal?" and it returns true, it seems wrong to me to query the cost as if it were a gather rather than asking for the scalarization cost, but then later still decide to scalarize the operation.

But for those particular VF's, it's more costly than the native scalarized path.
So LV won't actually produce gather intrinsics, and naturally, they won't be scalarized.

Furthermore, no, while we could just always prefer the intrinsics (from correctness viewpoint),
that is detrimental to the performance. See e.g. my recent patches, namely D111220 / D111363 / D111546.

This is the answer I was after, thanks for pointing me to those patches!

[RosieSumpter]

• Change data type parameter for forceScalarizeMaskedGather/Scatter to be a VectorType instead of a Type
• Address nits

[lebedev.ri]

In D115329#3189215, @sdesmalen wrote:

In D115329#3188963, @lebedev.ri wrote:

The fact that something is/isn't legal does not mean that LV will pick *that* particular path.
It merely allows LV to consider this path, among the others,
and out of all the legal paths, pick the most profitable one.

The LV makes the decision to use gathers/scatters because the cost-model said the gather/scatter cost was lower than the scalarization cost, but that only works if the gather/scatter cost is correct. If it will be scalarized, then I'd expect this costs to be no different than the scalarization cost.

If the cost model is wrong then lots of weird shit will go wrong, yes.

We don't "trick LV to think it's legal and then scalarize anyway", it *is* legal.

I'm not sure what the meaning of "legal" is in that case.

That the gather with VF=2 can be done by the same instruction as for the VF=4, but after padding the mask with zeros.

My understanding of the term is that it means that the target can handle the operation directly without legalization, e.g. using a special instruction or a sequence of instructions.

From that perspective, saying that <operation> is legal, means it will not need to be scalarized.

Define "need".
It is not much different from what VectorCombine does ("would scalar cost be lower than vector cost?
if so, scalarize" or the other way around), just implemented in an another pass.

(conversely, the ScalarizeMaskedOperations pass only scalarizes the intrinsic if it is *not* marked as legal). If the LV asks the question "is a masked gather operation legal?" and it returns true, it seems wrong to me to query the cost as if it were a gather rather than asking for the scalarization cost, but then later still decide to scalarize the operation.

But for those particular VF's, it's more costly than the native scalarized path.
So LV won't actually produce gather intrinsics, and naturally, they won't be scalarized.

Furthermore, no, while we could just always prefer the intrinsics (from correctness viewpoint),
that is detrimental to the performance. See e.g. my recent patches, namely D111220 / D111363 / D111546.

This is the answer I was after, thanks for pointing me to those patches!

Cheers.

[sdesmalen]

In D115329#3189241, @lebedev.ri wrote:

In D115329#3189215, @sdesmalen wrote:

From that perspective, saying that <operation> is legal, means it will not need to be scalarized.

Define "need".
It is not much different from what VectorCombine does ("would scalar cost be lower than vector cost?
if so, scalarize" or the other way around), just implemented in an another pass.

I see your point, it just seemed odd to me that for X86 this required reimplementing a different scalarisation cost rather than relying on the mechanisms already present in LoopVectorize, which tries to cost as accurately as possible whether to scalarize or use the intrinsics.
To me that suggests that the scalarization cost in LoopVectorize is inaccurate or there's improvements to be made to the code for scalarizing in the LoopVectorizer (or both) to make sure there is no disparity in performance.

As you explained, it's currently doing the right thing in the sense that it leads to the desired result.

[sdesmalen]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

3377 ↗

(On Diff #393907)

Hi @RosieSumpter this check should probably not be here anymore. Only isLegalMaskedGather should be used to determine whether or not to use an intrinsic for the gather operation.

[RosieSumpter]

• Removed forceScalarizeMaskedGather check from SLPVectorizer.cpp (since we don't check it in LoopVectorize.cpp)
• Updated necessary SLPVectorizer/X86 tests

[RosieSumpter]

Hi @lebedev.ri, the SLPVectorizer/X86/pr47629 tests have changed because isLegalMaskedGather now returns true for certain cases where it didn't before (due to the check on the number of vector elements now being in forceScalarizeMaskedGather as requested). It then calculates the gather/scatter cost, and because forceScalarizeMaskedGather returns true, it calculates the cost using getGSScalarCost. This cost is higher than before and so it chooses not to vectorize. An alternative approach would be to check forceScalarizeMaskedGather in isLegalMaskedGather instead of when calculating the cost, but this will then mean LoopVectorize will assume these operations need to be scalarized so will cause test failures there. What do you think the preferred option is here?

[sdesmalen]

Hi @RosieSumpter, just wanted to let you know I'm happy with the patch. This looks like a good improvement to me. It conceptually makes more sense and it also allows AArch64TTI to better distinguish between Neon/SVE when using fixed-width vectors.
I want to accept the patch, but I'll defer this to @lebedev.ri because there's still an open question about the X86 cost-model.

[RosieSumpter]

Friendly ping @lebedev.ri, do you have any further comments for this patch?

[lebedev.ri]

No concerns from my side, i guess this is one of the reasonable solutions.

[fhahn]

Are the X86 behavior changes intentional?

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

1608

documentation needs to be updated.

1613

VF should be documented.

[RosieSumpter]

• Updated descriptions of isScalarWithPredication and isPredicatedInst
• rebased

[RosieSumpter]

In D115329#3219230, @fhahn wrote:

Are the X86 behavior changes intentional?

Hi @fhahn, yes these changes are intentional - the cost model for X86 is now more accurate, and in some cases it ends up with a higher cost through getGSScalarCost because it knows the cost of the gather/scatter will be expanded later.

llvm/include/llvm/Analysis/TargetTransformInfo.h

public:
   bool isLegalNTStore(Type *DataType, Align Alignment) const;
   /// Return true if the target supports nontemporal load.
   bool isLegalNTLoad(Type *DataType, Align Alignment) const;
 
   /// Return true if the target supports masked scatter.
   bool isLegalMaskedScatter(Type *DataType, Align Alignment) const;
   /// Return true if the target supports masked gather.
   bool isLegalMaskedGather(Type *DataType, Align Alignment) const;
+  /// Return true if the target forces scalarizing of llvm.masked.gather
+  /// intrinsics.
+  bool forceScalarizeMaskedGather(Type *Type, Align Alignment) const;
+  /// Return true if the target forces scalarizing of llvm.masked.scatter
+  /// intrinsics.
+  bool forceScalarizeMaskedScatter(Type *Type, Align Alignment) const;
 
   /// Return true if the target supports masked compress store.
   bool isLegalMaskedCompressStore(Type *DataType) const;
   /// Return true if the target supports masked expand load.
   bool isLegalMaskedExpandLoad(Type *DataType) const;
 
   /// Return true if we should be enabling ordered reductions for the target.
   bool enableOrderedReductions() const;
public:
   virtual AddressingModeKind
     getPreferredAddressingMode(const Loop *L, ScalarEvolution *SE) const = 0;
   virtual bool isLegalMaskedStore(Type *DataType, Align Alignment) = 0;
   virtual bool isLegalMaskedLoad(Type *DataType, Align Alignment) = 0;
   virtual bool isLegalNTStore(Type *DataType, Align Alignment) = 0;
   virtual bool isLegalNTLoad(Type *DataType, Align Alignment) = 0;
   virtual bool isLegalMaskedScatter(Type *DataType, Align Alignment) = 0;
   virtual bool isLegalMaskedGather(Type *DataType, Align Alignment) = 0;
+  virtual bool forceScalarizeMaskedGather(Type *DataType, Align Alignment) = 0;
+  virtual bool forceScalarizeMaskedScatter(Type *DataType, Align Alignment) = 0;
   virtual bool isLegalMaskedCompressStore(Type *DataType) = 0;
   virtual bool isLegalMaskedExpandLoad(Type *DataType) = 0;
   virtual bool enableOrderedReductions() = 0;
   virtual bool hasDivRemOp(Type *DataType, bool IsSigned) = 0;
   virtual bool hasVolatileVariant(Instruction *I, unsigned AddrSpace) = 0;
   virtual bool prefersVectorizedAddressing() = 0;
   virtual InstructionCost getScalingFactorCost(Type *Ty, GlobalValue *BaseGV,
                                                int64_t BaseOffset,
bool isLegalNTLoad(Type *DataType, Align Alignment) override {
     return Impl.isLegalNTLoad(DataType, Alignment);
   }
   bool isLegalMaskedScatter(Type *DataType, Align Alignment) override {
     return Impl.isLegalMaskedScatter(DataType, Alignment);
   }
   bool isLegalMaskedGather(Type *DataType, Align Alignment) override {
     return Impl.isLegalMaskedGather(DataType, Alignment);
   }
+  bool forceScalarizeMaskedGather(Type *DataType, Align Alignment) override {
+    return Impl.forceScalarizeMaskedGather(DataType, Alignment);
+  }
+  bool forceScalarizeMaskedScatter(Type *DataType, Align Alignment) override {
+    return Impl.forceScalarizeMaskedScatter(DataType, Alignment);
+  }
   bool isLegalMaskedCompressStore(Type *DataType) override {
     return Impl.isLegalMaskedCompressStore(DataType);
   }
   bool isLegalMaskedExpandLoad(Type *DataType) override {
     return Impl.isLegalMaskedExpandLoad(DataType);
   }
   bool enableOrderedReductions() override {
     return Impl.enableOrderedReductions();

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

public:
   bool isLegalMaskedScatter(Type *DataType, Align Alignment) const {
     return false;
   }
 
   bool isLegalMaskedGather(Type *DataType, Align Alignment) const {
     return false;
   }
 
+  bool forceScalarizeMaskedGather(Type *DataType, Align Alignment) const {
+    return false;
+  }
+
+  bool forceScalarizeMaskedScatter(Type *DataType, Align Alignment) const {
+    return false;
+  }
+
   bool isLegalMaskedCompressStore(Type *DataType) const { return false; }
 
   bool isLegalMaskedExpandLoad(Type *DataType) const { return false; }
 
   bool enableOrderedReductions() const { return false; }
 
   bool hasDivRemOp(Type *DataType, bool IsSigned) const { return false; }
 

llvm/lib/Analysis/TargetTransformInfo.cpp

bool TargetTransformInfo::isLegalMaskedGather(Type *DataType,
   return TTIImpl->isLegalMaskedGather(DataType, Alignment);
 }
 
 bool TargetTransformInfo::isLegalMaskedScatter(Type *DataType,
                                                Align Alignment) const {
   return TTIImpl->isLegalMaskedScatter(DataType, Alignment);
 }
 
+bool TargetTransformInfo::forceScalarizeMaskedGather(Type *DataType,
+                                                     Align Alignment) const {
+  return TTIImpl->forceScalarizeMaskedGather(DataType, Alignment);
+}
+
+bool TargetTransformInfo::forceScalarizeMaskedScatter(Type *DataType,
+                                                      Align Alignment) const {
+  return TTIImpl->forceScalarizeMaskedScatter(DataType, Alignment);
+}
+
 bool TargetTransformInfo::isLegalMaskedCompressStore(Type *DataType) const {
   return TTIImpl->isLegalMaskedCompressStore(DataType);
 }
 
 bool TargetTransformInfo::isLegalMaskedExpandLoad(Type *DataType) const {
   return TTIImpl->isLegalMaskedExpandLoad(DataType);
 }
 

llvm/lib/Target/ARM/ARMTargetTransformInfo.h

public:
   bool isProfitableLSRChainElement(Instruction *I);
 
   bool isLegalMaskedLoad(Type *DataTy, Align Alignment);
 
   bool isLegalMaskedStore(Type *DataTy, Align Alignment) {
     return isLegalMaskedLoad(DataTy, Alignment);
   }
 
+  bool forceScalarizeMaskedGather(Type *VTy, Align Alignment) {
+    // For MVE, we have a custom lowering pass that will already have custom
+    // legalised any gathers that we can to MVE intrinsics, and want to expand
+    // all the rest. The pass runs before the masked intrinsic lowering pass, so
+    // if we are here, we know we want to expand.
+    return true;
+  }
+
+  bool forceScalarizeMaskedScatter(Type *VTy, Align Alignment) {
+    return forceScalarizeMaskedGather(VTy, Alignment);
+  }
+
   bool isLegalMaskedGather(Type *Ty, Align Alignment);
 
   bool isLegalMaskedScatter(Type *Ty, Align Alignment) {
     return isLegalMaskedGather(Ty, Alignment);
   }
 
   InstructionCost getMemcpyCost(const Instruction *I);
 

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

bool ARMTTIImpl::isLegalMaskedLoad(Type *DataTy, Align Alignment) {
   return (EltWidth == 32 && Alignment >= 4) ||
          (EltWidth == 16 && Alignment >= 2) || (EltWidth == 8);
 }
 
 bool ARMTTIImpl::isLegalMaskedGather(Type *Ty, Align Alignment) {
   if (!EnableMaskedGatherScatters || !ST->hasMVEIntegerOps())
     return false;
 
-  // This method is called in 2 places:
-  //  - from the vectorizer with a scalar type, in which case we need to get
-  //  this as good as we can with the limited info we have (and rely on the cost
-  //  model for the rest).
-  //  - from the masked intrinsic lowering pass with the actual vector type.
-  // For MVE, we have a custom lowering pass that will already have custom
-  // legalised any gathers that we can to MVE intrinsics, and want to expand all
-  // the rest. The pass runs before the masked intrinsic lowering pass, so if we
-  // are here, we know we want to expand.
-  if (isa<VectorType>(Ty))
-    return false;
-
   unsigned EltWidth = Ty->getScalarSizeInBits();
   return ((EltWidth == 32 && Alignment >= 4) ||
           (EltWidth == 16 && Alignment >= 2) || EltWidth == 8);
 }
 
 /// Given a memcpy/memset/memmove instruction, return the number of memory
 /// operations performed, via querying findOptimalMemOpLowering. Returns -1 if a
 /// call is used.

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

bool X86TTIImpl::supportsGather() const {
   // enable gather with a -march.
   return ST->hasAVX512() || (ST->hasFastGather() && ST->hasAVX2());
 }
 
 bool X86TTIImpl::isLegalMaskedGather(Type *DataTy, Align Alignment) {
   if (!supportsGather())
     return false;
 
   // This function is called now in two cases: from the Loop Vectorizer
-  // and from the Scalarizer.
-  // When the Loop Vectorizer asks about legality of the feature,
-  // the vectorization factor is not calculated yet. The Loop Vectorizer
-  // sends a scalar type and the decision is based on the width of the
-  // scalar element.
-  // Later on, the cost model will estimate usage this intrinsic based on
-  // the vector type.
-  // The Scalarizer asks again about legality. It sends a vector type.
-  // In this case we can reject non-power-of-2 vectors.
-  // We also reject single element vectors as the type legalizer can't
+  // and from the Scalarizer to ask about legality of the feature.
+  // In the case of a vector type being sent we can reject non-power-of-2
+  // vectors. We also reject single element vectors as the type legalizer can't
   // scalarize it.
   if (auto *DataVTy = dyn_cast<FixedVectorType>(DataTy)) {
     unsigned NumElts = DataVTy->getNumElements();
     if (NumElts == 1)
       return false;
     // Gather / Scatter for vector 2 is not profitable on KNL / SKX
     // Vector-4 of gather/scatter instruction does not exist on KNL.
     // We can extend it to 8 elements, but zeroing upper bits of
     // the mask vector will add more instructions. Right now we give the scalar
     // cost of vector-4 for KNL. TODO: Check, maybe the gather/scatter
     // instruction is better in the VariableMask case.
     if (ST->hasAVX512() && (NumElts == 2 || (NumElts == 4 && !ST->hasVLX())))
       return false;
   }
   Type *ScalarTy = DataTy->getScalarType();
   if (ScalarTy->isPointerTy())
     return true;
 
   if (ScalarTy->isFloatTy() || ScalarTy->isDoubleTy())
     return true;
 
   if (!ScalarTy->isIntegerTy())

llvm/lib/Transforms/Scalar/ScalarizeMaskedMemIntrin.cpp

case Intrinsic::masked_store:
       scalarizeMaskedStore(DL, CI, DTU, ModifiedDT);
       return true;
     case Intrinsic::masked_gather: {
       MaybeAlign MA =
           cast<ConstantInt>(CI->getArgOperand(1))->getMaybeAlignValue();
       Type *LoadTy = CI->getType();
       Align Alignment = DL.getValueOrABITypeAlignment(MA,
                                                       LoadTy->getScalarType());
-      if (TTI.isLegalMaskedGather(LoadTy, Alignment))
+      if (TTI.isLegalMaskedGather(LoadTy, Alignment) &&
+          !TTI.forceScalarizeMaskedGather(LoadTy, Alignment))
         return false;
       scalarizeMaskedGather(DL, CI, DTU, ModifiedDT);
       return true;
     }
     case Intrinsic::masked_scatter: {
       MaybeAlign MA =
           cast<ConstantInt>(CI->getArgOperand(2))->getMaybeAlignValue();
       Type *StoreTy = CI->getArgOperand(0)->getType();
       Align Alignment = DL.getValueOrABITypeAlignment(MA,
                                                       StoreTy->getScalarType());
-      if (TTI.isLegalMaskedScatter(StoreTy, Alignment))
+      if (TTI.isLegalMaskedScatter(StoreTy, Alignment) &&
+          !TTI.forceScalarizeMaskedScatter(StoreTy, Alignment))
         return false;
       scalarizeMaskedScatter(DL, CI, DTU, ModifiedDT);
       return true;
     }
     case Intrinsic::masked_expandload:
       if (TTI.isLegalMaskedExpandLoad(CI->getType()))
         return false;
       scalarizeMaskedExpandLoad(DL, CI, DTU, ModifiedDT);

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

public:
   /// for the given \p DataType and kind of access to \p Ptr.
   bool isLegalMaskedLoad(Type *DataType, Value *Ptr, Align Alignment) const {
     return Legal->isConsecutivePtr(DataType, Ptr) &&
            TTI.isLegalMaskedLoad(DataType, Alignment);
   }
 
   /// Returns true if the target machine can represent \p V as a masked gather
   /// or scatter operation.
-  bool isLegalGatherOrScatter(Value *V) {
+  bool isLegalGatherOrScatter(Value *V,
+                              ElementCount VF = ElementCount::getFixed(1)) {
     bool LI = isa<LoadInst>(V);
     bool SI = isa<StoreInst>(V);
     if (!LI && !SI)
       return false;
     auto *Ty = getLoadStoreType(V);
     Align Align = getLoadStoreAlignment(V);
+    if (VF.isVector())
+      Ty = VectorType::get(Ty, VF);
     return (LI && TTI.isLegalMaskedGather(Ty, Align)) ||
            (SI && TTI.isLegalMaskedScatter(Ty, Align));
   }
 
   /// Returns true if the target machine supports all of the reduction
   /// variables found for the given VF.
   bool canVectorizeReductions(ElementCount VF) const {
     return (all_of(Legal->getReductionVars(), [&](auto &Reduction) -> bool {
       const RecurrenceDescriptor &RdxDesc = Reduction.second;
       return TTI.isLegalToVectorizeReduction(RdxDesc, VF);
     }));
   }
 
   /// Returns true if \p I is an instruction that will be scalarized with
   /// predication. Such instructions include conditional stores and
   /// instructions that may divide by zero.
   /// If a non-zero VF has been calculated, we check if I will be scalarized
   /// predication for that VF.
   bool isScalarWithPredication(Instruction *I) const;
 
   // Returns true if \p I is an instruction that will be predicated either
   // through scalar predication or masked load/store or masked gather/scatter.
   // Superset of instructions that return true for isScalarWithPredication.
   bool isPredicatedInst(Instruction *I, bool IsKnownUniform = false) {
     // When we know the load is uniform and the original scalar loop was not
     // predicated we don't need to mark it as a predicated instruction. Any
     // vectorised blocks created when tail-folding are something artificial we
     // have introduced and we know there is always at least one active lane.
     // That's why we call Legal->blockNeedsPredication here because it doesn't
     // query tail-folding.
     if (IsKnownUniform && isa<LoadInst>(I) &&
         !Legal->blockNeedsPredication(I->getParent()))
for (Instruction &I : *BB) {
 
       if (Legal->isUniformMemOp(I)) {
         // TODO: Avoid replicating loads and stores instead of
         // relying on instcombine to remove them.
         // Load: Scalar load + broadcast
         // Store: Scalar store + isLoopInvariantStoreValue ? 0 : extract
         InstructionCost Cost;
         if (isa<StoreInst>(&I) && VF.isScalable() &&
-            isLegalGatherOrScatter(&I)) {
+            isLegalGatherOrScatter(&I, VF)) {
           Cost = getGatherScatterCost(&I, VF);
           setWideningDecision(&I, VF, CM_GatherScatter, Cost);
         } else {
           assert((isa<LoadInst>(&I) || !VF.isScalable()) &&
                  "Cannot yet scalarize uniform stores");
           Cost = getUniformMemOpCost(&I, VF);
           setWideningDecision(&I, VF, CM_Scalarize, Cost);
         }
for (Instruction &I : *BB) {
           continue;
 
         NumAccesses = Group->getNumMembers();
         if (interleavedAccessCanBeWidened(&I, VF))
           InterleaveCost = getInterleaveGroupCost(&I, VF);
       }
 
       InstructionCost GatherScatterCost =
-          isLegalGatherOrScatter(&I)
+          isLegalGatherOrScatter(&I, VF)
               ? getGatherScatterCost(&I, VF) * NumAccesses
               : InstructionCost::getInvalid();
 
       InstructionCost ScalarizationCost =
           getMemInstScalarizationCost(&I, VF) * NumAccesses;
 
       // Choose better solution for the current VF,
       // write down this decision and use it during vectorization.

llvm/test/Analysis/CostModel/X86/masked-gather-i32-with-i8-index.ll

 ; AVX2-FASTGATHER: LV: Found an estimated cost of 1 for VF 1 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 ; AVX2-FASTGATHER: LV: Found an estimated cost of 4 for VF 2 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 ; AVX2-FASTGATHER: LV: Found an estimated cost of 6 for VF 4 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 ; AVX2-FASTGATHER: LV: Found an estimated cost of 12 for VF 8 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 ; AVX2-FASTGATHER: LV: Found an estimated cost of 24 for VF 16 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 ; AVX2-FASTGATHER: LV: Found an estimated cost of 48 for VF 32 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 ;
 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
-; AVX512: LV: Found an estimated cost of 10 for VF 2 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
-; AVX512: LV: Found an estimated cost of 22 for VF 4 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
+; AVX512: LV: Found an estimated cost of 3000000 for VF 2 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
+; AVX512: LV: Found an estimated cost of 3000000 for VF 4 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 ; AVX512: LV: Found an estimated cost of 10 for VF 8 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 ; AVX512: LV: Found an estimated cost of 18 for VF 16 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 ; AVX512: LV: Found an estimated cost of 36 for VF 32 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 ; AVX512: LV: Found an estimated cost of 72 for VF 64 For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 ;
 ; CHECK-NOT: LV: Found an estimated cost of {{[0-9]+}} for VF {{[0-9]+}} For instruction:   %valB.loaded = load i32, i32* %inB, align 4
 define void @test() {
 entry:

llvm/test/Analysis/CostModel/X86/masked-gather-i64-with-i8-index.ll

 ; AVX2-FASTGATHER: LV: Found an estimated cost of 1 for VF 1 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 ; AVX2-FASTGATHER: LV: Found an estimated cost of 4 for VF 2 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 ; AVX2-FASTGATHER: LV: Found an estimated cost of 6 for VF 4 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 ; AVX2-FASTGATHER: LV: Found an estimated cost of 12 for VF 8 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 ; AVX2-FASTGATHER: LV: Found an estimated cost of 24 for VF 16 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 ; AVX2-FASTGATHER: LV: Found an estimated cost of 48 for VF 32 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 ;
 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
-; AVX512: LV: Found an estimated cost of 10 for VF 2 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
-; AVX512: LV: Found an estimated cost of 24 for VF 4 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
+; AVX512: LV: Found an estimated cost of 3000000 for VF 2 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
+; AVX512: LV: Found an estimated cost of 3000000 for VF 4 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 ; AVX512: LV: Found an estimated cost of 10 for VF 8 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 ; AVX512: LV: Found an estimated cost of 20 for VF 16 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 ; AVX512: LV: Found an estimated cost of 40 for VF 32 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 ; AVX512: LV: Found an estimated cost of 80 for VF 64 For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 ;
 ; CHECK-NOT: LV: Found an estimated cost of {{[0-9]+}} for VF {{[0-9]+}} For instruction:   %valB.loaded = load i64, i64* %inB, align 8
 define void @test() {
 entry:

llvm/test/Transforms/LoopVectorize/AArch64/tail-fold-uniform-memops.ll

if.end:                                           ; preds = %if.then, %for.body
   %index.next = add nuw i64 %index, 1
   %exitcond.not = icmp eq i64 %index.next, %n
   br i1 %exitcond.not, label %for.end, label %for.body
 
 for.end:                                          ; preds = %for.inc, %entry
   ret void
 }
 
-attributes #0 = { "target-features"="+neon,+sve,+v8.1a" }
+attributes #0 = { "target-features"="+neon,+sve,+v8.1a" vscale_range(2, 0) }

llvm/test/Transforms/LoopVectorize/X86/gather_scatter.ll

 ; AVX512-NEXT:    br i1 [[TMP40]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; AVX512:       for.end:
 ; AVX512-NEXT:    ret void
 ;
 ; FVW2-LABEL: @foo1(
 ; FVW2-NEXT:  entry:
 ; FVW2-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; FVW2:       vector.body:
-; FVW2-NEXT:    [[INDEX7:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; FVW2-NEXT:    [[INDEX7:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT_1:%.*]], [[PRED_LOAD_CONTINUE9_1:%.*]] ]
 ; FVW2-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER:%.*]], i64 [[INDEX7]]
 ; FVW2-NEXT:    [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <2 x i32>*
 ; FVW2-NEXT:    [[WIDE_LOAD:%.*]] = load <2 x i32>, <2 x i32>* [[TMP1]], align 4
-; FVW2-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[TMP0]], i64 2
-; FVW2-NEXT:    [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <2 x i32>*
-; FVW2-NEXT:    [[WIDE_LOAD8:%.*]] = load <2 x i32>, <2 x i32>* [[TMP3]], align 4
-; FVW2-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i32, i32* [[TMP0]], i64 4
-; FVW2-NEXT:    [[TMP5:%.*]] = bitcast i32* [[TMP4]] to <2 x i32>*
-; FVW2-NEXT:    [[WIDE_LOAD9:%.*]] = load <2 x i32>, <2 x i32>* [[TMP5]], align 4
-; FVW2-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i32, i32* [[TMP0]], i64 6
-; FVW2-NEXT:    [[TMP7:%.*]] = bitcast i32* [[TMP6]] to <2 x i32>*
-; FVW2-NEXT:    [[WIDE_LOAD10:%.*]] = load <2 x i32>, <2 x i32>* [[TMP7]], align 4
-; FVW2-NEXT:    [[TMP8:%.*]] = icmp sgt <2 x i32> [[WIDE_LOAD]], zeroinitializer
-; FVW2-NEXT:    [[TMP9:%.*]] = icmp sgt <2 x i32> [[WIDE_LOAD8]], zeroinitializer
-; FVW2-NEXT:    [[TMP10:%.*]] = icmp sgt <2 x i32> [[WIDE_LOAD9]], zeroinitializer
-; FVW2-NEXT:    [[TMP11:%.*]] = icmp sgt <2 x i32> [[WIDE_LOAD10]], zeroinitializer
-; FVW2-NEXT:    [[TMP12:%.*]] = getelementptr i32, i32* [[INDEX:%.*]], i64 [[INDEX7]]
-; FVW2-NEXT:    [[TMP13:%.*]] = bitcast i32* [[TMP12]] to <2 x i32>*
-; FVW2-NEXT:    [[WIDE_MASKED_LOAD:%.*]] = call <2 x i32> @llvm.masked.load.v2i32.p0v2i32(<2 x i32>* [[TMP13]], i32 4, <2 x i1> [[TMP8]], <2 x i32> poison)
-; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr i32, i32* [[TMP12]], i64 2
-; FVW2-NEXT:    [[TMP15:%.*]] = bitcast i32* [[TMP14]] to <2 x i32>*
-; FVW2-NEXT:    [[WIDE_MASKED_LOAD11:%.*]] = call <2 x i32> @llvm.masked.load.v2i32.p0v2i32(<2 x i32>* [[TMP15]], i32 4, <2 x i1> [[TMP9]], <2 x i32> poison)
-; FVW2-NEXT:    [[TMP16:%.*]] = getelementptr i32, i32* [[TMP12]], i64 4
-; FVW2-NEXT:    [[TMP17:%.*]] = bitcast i32* [[TMP16]] to <2 x i32>*
-; FVW2-NEXT:    [[WIDE_MASKED_LOAD12:%.*]] = call <2 x i32> @llvm.masked.load.v2i32.p0v2i32(<2 x i32>* [[TMP17]], i32 4, <2 x i1> [[TMP10]], <2 x i32> poison)
-; FVW2-NEXT:    [[TMP18:%.*]] = getelementptr i32, i32* [[TMP12]], i64 6
-; FVW2-NEXT:    [[TMP19:%.*]] = bitcast i32* [[TMP18]] to <2 x i32>*
-; FVW2-NEXT:    [[WIDE_MASKED_LOAD13:%.*]] = call <2 x i32> @llvm.masked.load.v2i32.p0v2i32(<2 x i32>* [[TMP19]], i32 4, <2 x i1> [[TMP11]], <2 x i32> poison)
-; FVW2-NEXT:    [[TMP20:%.*]] = sext <2 x i32> [[WIDE_MASKED_LOAD]] to <2 x i64>
-; FVW2-NEXT:    [[TMP21:%.*]] = sext <2 x i32> [[WIDE_MASKED_LOAD11]] to <2 x i64>
-; FVW2-NEXT:    [[TMP22:%.*]] = sext <2 x i32> [[WIDE_MASKED_LOAD12]] to <2 x i64>
-; FVW2-NEXT:    [[TMP23:%.*]] = sext <2 x i32> [[WIDE_MASKED_LOAD13]] to <2 x i64>
-; FVW2-NEXT:    [[TMP24:%.*]] = getelementptr inbounds float, float* [[IN:%.*]], <2 x i64> [[TMP20]]
-; FVW2-NEXT:    [[TMP25:%.*]] = getelementptr inbounds float, float* [[IN]], <2 x i64> [[TMP21]]
-; FVW2-NEXT:    [[TMP26:%.*]] = getelementptr inbounds float, float* [[IN]], <2 x i64> [[TMP22]]
-; FVW2-NEXT:    [[TMP27:%.*]] = getelementptr inbounds float, float* [[IN]], <2 x i64> [[TMP23]]
-; FVW2-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <2 x float> @llvm.masked.gather.v2f32.v2p0f32(<2 x float*> [[TMP24]], i32 4, <2 x i1> [[TMP8]], <2 x float> undef)
-; FVW2-NEXT:    [[WIDE_MASKED_GATHER14:%.*]] = call <2 x float> @llvm.masked.gather.v2f32.v2p0f32(<2 x float*> [[TMP25]], i32 4, <2 x i1> [[TMP9]], <2 x float> undef)
-; FVW2-NEXT:    [[WIDE_MASKED_GATHER15:%.*]] = call <2 x float> @llvm.masked.gather.v2f32.v2p0f32(<2 x float*> [[TMP26]], i32 4, <2 x i1> [[TMP10]], <2 x float> undef)
-; FVW2-NEXT:    [[WIDE_MASKED_GATHER16:%.*]] = call <2 x float> @llvm.masked.gather.v2f32.v2p0f32(<2 x float*> [[TMP27]], i32 4, <2 x i1> [[TMP11]], <2 x float> undef)
-; FVW2-NEXT:    [[TMP28:%.*]] = fadd <2 x float> [[WIDE_MASKED_GATHER]], <float 5.000000e-01, float 5.000000e-01>
-; FVW2-NEXT:    [[TMP29:%.*]] = fadd <2 x float> [[WIDE_MASKED_GATHER14]], <float 5.000000e-01, float 5.000000e-01>
-; FVW2-NEXT:    [[TMP30:%.*]] = fadd <2 x float> [[WIDE_MASKED_GATHER15]], <float 5.000000e-01, float 5.000000e-01>
-; FVW2-NEXT:    [[TMP31:%.*]] = fadd <2 x float> [[WIDE_MASKED_GATHER16]], <float 5.000000e-01, float 5.000000e-01>
-; FVW2-NEXT:    [[TMP32:%.*]] = getelementptr float, float* [[OUT:%.*]], i64 [[INDEX7]]
-; FVW2-NEXT:    [[TMP33:%.*]] = bitcast float* [[TMP32]] to <2 x float>*
-; FVW2-NEXT:    call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP28]], <2 x float>* [[TMP33]], i32 4, <2 x i1> [[TMP8]])
-; FVW2-NEXT:    [[TMP34:%.*]] = getelementptr float, float* [[TMP32]], i64 2
-; FVW2-NEXT:    [[TMP35:%.*]] = bitcast float* [[TMP34]] to <2 x float>*
-; FVW2-NEXT:    call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP29]], <2 x float>* [[TMP35]], i32 4, <2 x i1> [[TMP9]])
-; FVW2-NEXT:    [[TMP36:%.*]] = getelementptr float, float* [[TMP32]], i64 4
-; FVW2-NEXT:    [[TMP37:%.*]] = bitcast float* [[TMP36]] to <2 x float>*
-; FVW2-NEXT:    call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP30]], <2 x float>* [[TMP37]], i32 4, <2 x i1> [[TMP10]])
-; FVW2-NEXT:    [[TMP38:%.*]] = getelementptr float, float* [[TMP32]], i64 6
-; FVW2-NEXT:    [[TMP39:%.*]] = bitcast float* [[TMP38]] to <2 x float>*
-; FVW2-NEXT:    call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP31]], <2 x float>* [[TMP39]], i32 4, <2 x i1> [[TMP11]])
-; FVW2-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX7]], 8
-; FVW2-NEXT:    [[TMP40:%.*]] = icmp eq i64 [[INDEX_NEXT]], 4096
-; FVW2-NEXT:    br i1 [[TMP40]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; FVW2-NEXT:    [[TMP2:%.*]] = icmp sgt <2 x i32> [[WIDE_LOAD]], zeroinitializer
+; FVW2-NEXT:    [[TMP3:%.*]] = getelementptr i32, i32* [[INDEX:%.*]], i64 [[INDEX7]]
+; FVW2-NEXT:    [[TMP4:%.*]] = bitcast i32* [[TMP3]] to <2 x i32>*
+; FVW2-NEXT:    [[WIDE_MASKED_LOAD:%.*]] = call <2 x i32> @llvm.masked.load.v2i32.p0v2i32(<2 x i32>* [[TMP4]], i32 4, <2 x i1> [[TMP2]], <2 x i32> poison)
+; FVW2-NEXT:    [[TMP5:%.*]] = sext <2 x i32> [[WIDE_MASKED_LOAD]] to <2 x i64>
+; FVW2-NEXT:    [[TMP6:%.*]] = extractelement <2 x i1> [[TMP2]], i32 0
+; FVW2-NEXT:    br i1 [[TMP6]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; FVW2:       pred.load.if:
+; FVW2-NEXT:    [[TMP7:%.*]] = extractelement <2 x i64> [[TMP5]], i32 0
+; FVW2-NEXT:    [[TMP8:%.*]] = getelementptr inbounds float, float* [[IN:%.*]], i64 [[TMP7]]
+; FVW2-NEXT:    [[TMP9:%.*]] = load float, float* [[TMP8]], align 4
+; FVW2-NEXT:    [[TMP10:%.*]] = insertelement <2 x float> poison, float [[TMP9]], i32 0
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE]]
+; FVW2:       pred.load.continue:
+; FVW2-NEXT:    [[TMP11:%.*]] = phi <2 x float> [ poison, [[VECTOR_BODY]] ], [ [[TMP10]], [[PRED_LOAD_IF]] ]
+; FVW2-NEXT:    [[TMP12:%.*]] = extractelement <2 x i1> [[TMP2]], i32 1
+; FVW2-NEXT:    br i1 [[TMP12]], label [[PRED_LOAD_IF8:%.*]], label [[PRED_LOAD_CONTINUE9:%.*]]
+; FVW2:       pred.load.if8:
+; FVW2-NEXT:    [[TMP13:%.*]] = extractelement <2 x i64> [[TMP5]], i32 1
+; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr inbounds float, float* [[IN]], i64 [[TMP13]]
+; FVW2-NEXT:    [[TMP15:%.*]] = load float, float* [[TMP14]], align 4
+; FVW2-NEXT:    [[TMP16:%.*]] = insertelement <2 x float> [[TMP11]], float [[TMP15]], i32 1
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE9]]
+; FVW2:       pred.load.continue9:
+; FVW2-NEXT:    [[TMP17:%.*]] = phi <2 x float> [ [[TMP11]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP16]], [[PRED_LOAD_IF8]] ]
+; FVW2-NEXT:    [[TMP18:%.*]] = fadd <2 x float> [[TMP17]], <float 5.000000e-01, float 5.000000e-01>
+; FVW2-NEXT:    [[TMP19:%.*]] = getelementptr float, float* [[OUT:%.*]], i64 [[INDEX7]]
+; FVW2-NEXT:    [[TMP20:%.*]] = bitcast float* [[TMP19]] to <2 x float>*
+; FVW2-NEXT:    call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP18]], <2 x float>* [[TMP20]], i32 4, <2 x i1> [[TMP2]])
+; FVW2-NEXT:    [[INDEX_NEXT:%.*]] = or i64 [[INDEX7]], 2
+; FVW2-NEXT:    [[TMP21:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER]], i64 [[INDEX_NEXT]]
+; FVW2-NEXT:    [[TMP22:%.*]] = bitcast i32* [[TMP21]] to <2 x i32>*
+; FVW2-NEXT:    [[WIDE_LOAD_1:%.*]] = load <2 x i32>, <2 x i32>* [[TMP22]], align 4
+; FVW2-NEXT:    [[TMP23:%.*]] = icmp sgt <2 x i32> [[WIDE_LOAD_1]], zeroinitializer
+; FVW2-NEXT:    [[TMP24:%.*]] = getelementptr i32, i32* [[INDEX]], i64 [[INDEX_NEXT]]
+; FVW2-NEXT:    [[TMP25:%.*]] = bitcast i32* [[TMP24]] to <2 x i32>*
+; FVW2-NEXT:    [[WIDE_MASKED_LOAD_1:%.*]] = call <2 x i32> @llvm.masked.load.v2i32.p0v2i32(<2 x i32>* [[TMP25]], i32 4, <2 x i1> [[TMP23]], <2 x i32> poison)
+; FVW2-NEXT:    [[TMP26:%.*]] = sext <2 x i32> [[WIDE_MASKED_LOAD_1]] to <2 x i64>
+; FVW2-NEXT:    [[TMP27:%.*]] = extractelement <2 x i1> [[TMP23]], i32 0
+; FVW2-NEXT:    br i1 [[TMP27]], label [[PRED_LOAD_IF_1:%.*]], label [[PRED_LOAD_CONTINUE_1:%.*]]
+; FVW2:       pred.load.if.1:
+; FVW2-NEXT:    [[TMP28:%.*]] = extractelement <2 x i64> [[TMP26]], i32 0
+; FVW2-NEXT:    [[TMP29:%.*]] = getelementptr inbounds float, float* [[IN]], i64 [[TMP28]]
+; FVW2-NEXT:    [[TMP30:%.*]] = load float, float* [[TMP29]], align 4
+; FVW2-NEXT:    [[TMP31:%.*]] = insertelement <2 x float> poison, float [[TMP30]], i32 0
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE_1]]
+; FVW2:       pred.load.continue.1:
+; FVW2-NEXT:    [[TMP32:%.*]] = phi <2 x float> [ poison, [[PRED_LOAD_CONTINUE9]] ], [ [[TMP31]], [[PRED_LOAD_IF_1]] ]
+; FVW2-NEXT:    [[TMP33:%.*]] = extractelement <2 x i1> [[TMP23]], i32 1
+; FVW2-NEXT:    br i1 [[TMP33]], label [[PRED_LOAD_IF8_1:%.*]], label [[PRED_LOAD_CONTINUE9_1]]
+; FVW2:       pred.load.if8.1:
+; FVW2-NEXT:    [[TMP34:%.*]] = extractelement <2 x i64> [[TMP26]], i32 1
+; FVW2-NEXT:    [[TMP35:%.*]] = getelementptr inbounds float, float* [[IN]], i64 [[TMP34]]
+; FVW2-NEXT:    [[TMP36:%.*]] = load float, float* [[TMP35]], align 4
+; FVW2-NEXT:    [[TMP37:%.*]] = insertelement <2 x float> [[TMP32]], float [[TMP36]], i32 1
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE9_1]]
+; FVW2:       pred.load.continue9.1:
+; FVW2-NEXT:    [[TMP38:%.*]] = phi <2 x float> [ [[TMP32]], [[PRED_LOAD_CONTINUE_1]] ], [ [[TMP37]], [[PRED_LOAD_IF8_1]] ]
+; FVW2-NEXT:    [[TMP39:%.*]] = fadd <2 x float> [[TMP38]], <float 5.000000e-01, float 5.000000e-01>
+; FVW2-NEXT:    [[TMP40:%.*]] = getelementptr float, float* [[OUT]], i64 [[INDEX_NEXT]]
+; FVW2-NEXT:    [[TMP41:%.*]] = bitcast float* [[TMP40]] to <2 x float>*
+; FVW2-NEXT:    call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP39]], <2 x float>* [[TMP41]], i32 4, <2 x i1> [[TMP23]])
+; FVW2-NEXT:    [[INDEX_NEXT_1]] = add nuw nsw i64 [[INDEX7]], 4
+; FVW2-NEXT:    [[TMP42:%.*]] = icmp eq i64 [[INDEX_NEXT_1]], 4096
+; FVW2-NEXT:    br i1 [[TMP42]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; FVW2:       for.end:
 ; FVW2-NEXT:    ret void
 ;
 entry:
   %in.addr = alloca float*, align 8
   %out.addr = alloca float*, align 8
   %trigger.addr = alloca i32*, align 8
   %index.addr = alloca i32*, align 8

 ; AVX512-NEXT:    [[TMP79:%.*]] = getelementptr inbounds float, float* [[OUT]], <16 x i64> <i64 3840, i64 3856, i64 3872, i64 3888, i64 3904, i64 3920, i64 3936, i64 3952, i64 3968, i64 3984, i64 4000, i64 4016, i64 4032, i64 4048, i64 4064, i64 4080>
 ; AVX512-NEXT:    call void @llvm.masked.scatter.v16f32.v16p0f32(<16 x float> [[TMP78]], <16 x float*> [[TMP79]], i32 4, <16 x i1> [[TMP76]])
 ; AVX512-NEXT:    ret void
 ;
 ; FVW2-LABEL: @foo2(
 ; FVW2-NEXT:  entry:
 ; FVW2-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; FVW2:       vector.body:
-; FVW2-NEXT:    [[INDEX7:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE9:%.*]] ]
-; FVW2-NEXT:    [[VEC_IND:%.*]] = phi <2 x i64> [ <i64 0, i64 16>, [[ENTRY]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE9]] ]
+; FVW2-NEXT:    [[INDEX7:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE11:%.*]] ]
 ; FVW2-NEXT:    [[OFFSET_IDX:%.*]] = shl i64 [[INDEX7]], 4
 ; FVW2-NEXT:    [[TMP0:%.*]] = or i64 [[OFFSET_IDX]], 16
 ; FVW2-NEXT:    [[TMP1:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER:%.*]], i64 [[OFFSET_IDX]]
 ; FVW2-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER]], i64 [[TMP0]]
 ; FVW2-NEXT:    [[TMP3:%.*]] = load i32, i32* [[TMP1]], align 4
 ; FVW2-NEXT:    [[TMP4:%.*]] = load i32, i32* [[TMP2]], align 4
 ; FVW2-NEXT:    [[TMP5:%.*]] = insertelement <2 x i32> poison, i32 [[TMP3]], i32 0
 ; FVW2-NEXT:    [[TMP6:%.*]] = insertelement <2 x i32> [[TMP5]], i32 [[TMP4]], i32 1
 ; FVW2-NEXT:    [[TMP7:%.*]] = icmp sgt <2 x i32> [[TMP6]], zeroinitializer
-; FVW2-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT_IN:%.*]], %struct.In* [[IN:%.*]], <2 x i64> [[VEC_IND]], i32 1
-; FVW2-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <2 x float> @llvm.masked.gather.v2f32.v2p0f32(<2 x float*> [[TMP8]], i32 4, <2 x i1> [[TMP7]], <2 x float> undef)
-; FVW2-NEXT:    [[TMP9:%.*]] = fadd <2 x float> [[WIDE_MASKED_GATHER]], <float 5.000000e-01, float 5.000000e-01>
-; FVW2-NEXT:    [[TMP10:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
-; FVW2-NEXT:    br i1 [[TMP10]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
+; FVW2-NEXT:    [[TMP8:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
+; FVW2-NEXT:    br i1 [[TMP8]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; FVW2:       pred.load.if:
+; FVW2-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT_IN:%.*]], %struct.In* [[IN:%.*]], i64 [[OFFSET_IDX]], i32 1
+; FVW2-NEXT:    [[TMP10:%.*]] = load float, float* [[TMP9]], align 4
+; FVW2-NEXT:    [[TMP11:%.*]] = insertelement <2 x float> poison, float [[TMP10]], i32 0
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE]]
+; FVW2:       pred.load.continue:
+; FVW2-NEXT:    [[TMP12:%.*]] = phi <2 x float> [ poison, [[VECTOR_BODY]] ], [ [[TMP11]], [[PRED_LOAD_IF]] ]
+; FVW2-NEXT:    [[TMP13:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
+; FVW2-NEXT:    br i1 [[TMP13]], label [[PRED_LOAD_IF8:%.*]], label [[PRED_LOAD_CONTINUE9:%.*]]
+; FVW2:       pred.load.if8:
+; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [[STRUCT_IN]], %struct.In* [[IN]], i64 [[TMP0]], i32 1
+; FVW2-NEXT:    [[TMP15:%.*]] = load float, float* [[TMP14]], align 4
+; FVW2-NEXT:    [[TMP16:%.*]] = insertelement <2 x float> [[TMP12]], float [[TMP15]], i32 1
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE9]]
+; FVW2:       pred.load.continue9:
+; FVW2-NEXT:    [[TMP17:%.*]] = phi <2 x float> [ [[TMP12]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP16]], [[PRED_LOAD_IF8]] ]
+; FVW2-NEXT:    [[TMP18:%.*]] = fadd <2 x float> [[TMP17]], <float 5.000000e-01, float 5.000000e-01>
+; FVW2-NEXT:    [[TMP19:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
+; FVW2-NEXT:    br i1 [[TMP19]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
 ; FVW2:       pred.store.if:
-; FVW2-NEXT:    [[TMP11:%.*]] = getelementptr inbounds float, float* [[OUT:%.*]], i64 [[OFFSET_IDX]]
-; FVW2-NEXT:    [[TMP12:%.*]] = extractelement <2 x float> [[TMP9]], i32 0
-; FVW2-NEXT:    store float [[TMP12]], float* [[TMP11]], align 4
+; FVW2-NEXT:    [[TMP20:%.*]] = getelementptr inbounds float, float* [[OUT:%.*]], i64 [[OFFSET_IDX]]
+; FVW2-NEXT:    [[TMP21:%.*]] = extractelement <2 x float> [[TMP18]], i32 0
+; FVW2-NEXT:    store float [[TMP21]], float* [[TMP20]], align 4
 ; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE]]
 ; FVW2:       pred.store.continue:
-; FVW2-NEXT:    [[TMP13:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
-; FVW2-NEXT:    br i1 [[TMP13]], label [[PRED_STORE_IF8:%.*]], label [[PRED_STORE_CONTINUE9]]
-; FVW2:       pred.store.if8:
-; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr inbounds float, float* [[OUT]], i64 [[TMP0]]
-; FVW2-NEXT:    [[TMP15:%.*]] = extractelement <2 x float> [[TMP9]], i32 1
-; FVW2-NEXT:    store float [[TMP15]], float* [[TMP14]], align 4
-; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE9]]
-; FVW2:       pred.store.continue9:
+; FVW2-NEXT:    [[TMP22:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
+; FVW2-NEXT:    br i1 [[TMP22]], label [[PRED_STORE_IF10:%.*]], label [[PRED_STORE_CONTINUE11]]
+; FVW2:       pred.store.if10:
+; FVW2-NEXT:    [[TMP23:%.*]] = getelementptr inbounds float, float* [[OUT]], i64 [[TMP0]]
+; FVW2-NEXT:    [[TMP24:%.*]] = extractelement <2 x float> [[TMP18]], i32 1
+; FVW2-NEXT:    store float [[TMP24]], float* [[TMP23]], align 4
+; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE11]]
+; FVW2:       pred.store.continue11:
 ; FVW2-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX7]], 2
-; FVW2-NEXT:    [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 32, i64 32>
-; FVW2-NEXT:    [[TMP16:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
-; FVW2-NEXT:    br i1 [[TMP16]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP2:![0-9]+]]
+; FVW2-NEXT:    [[TMP25:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
+; FVW2-NEXT:    br i1 [[TMP25]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; FVW2:       for.end:
 ; FVW2-NEXT:    ret void
 ;
 entry:
   %in.addr = alloca %struct.In*, align 8
   %out.addr = alloca float*, align 8
   %trigger.addr = alloca i32*, align 8
   %index.addr = alloca i32*, align 8

 ; AVX512-NEXT:    [[TMP79:%.*]] = getelementptr inbounds [[STRUCT_OUT]], %struct.Out* [[OUT]], <16 x i64> <i64 3840, i64 3856, i64 3872, i64 3888, i64 3904, i64 3920, i64 3936, i64 3952, i64 3968, i64 3984, i64 4000, i64 4016, i64 4032, i64 4048, i64 4064, i64 4080>, i32 1
 ; AVX512-NEXT:    call void @llvm.masked.scatter.v16f32.v16p0f32(<16 x float> [[TMP78]], <16 x float*> [[TMP79]], i32 4, <16 x i1> [[TMP76]])
 ; AVX512-NEXT:    ret void
 ;
 ; FVW2-LABEL: @foo3(
 ; FVW2-NEXT:  entry:
 ; FVW2-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; FVW2:       vector.body:
-; FVW2-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE8:%.*]] ]
-; FVW2-NEXT:    [[VEC_IND:%.*]] = phi <2 x i64> [ <i64 0, i64 16>, [[ENTRY]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE8]] ]
+; FVW2-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE10:%.*]] ]
 ; FVW2-NEXT:    [[OFFSET_IDX:%.*]] = shl i64 [[INDEX]], 4
 ; FVW2-NEXT:    [[TMP0:%.*]] = or i64 [[OFFSET_IDX]], 16
 ; FVW2-NEXT:    [[TMP1:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER:%.*]], i64 [[OFFSET_IDX]]
 ; FVW2-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER]], i64 [[TMP0]]
 ; FVW2-NEXT:    [[TMP3:%.*]] = load i32, i32* [[TMP1]], align 4
 ; FVW2-NEXT:    [[TMP4:%.*]] = load i32, i32* [[TMP2]], align 4
 ; FVW2-NEXT:    [[TMP5:%.*]] = insertelement <2 x i32> poison, i32 [[TMP3]], i32 0
 ; FVW2-NEXT:    [[TMP6:%.*]] = insertelement <2 x i32> [[TMP5]], i32 [[TMP4]], i32 1
 ; FVW2-NEXT:    [[TMP7:%.*]] = icmp sgt <2 x i32> [[TMP6]], zeroinitializer
-; FVW2-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT_IN:%.*]], %struct.In* [[IN:%.*]], <2 x i64> [[VEC_IND]], i32 1
-; FVW2-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <2 x float> @llvm.masked.gather.v2f32.v2p0f32(<2 x float*> [[TMP8]], i32 4, <2 x i1> [[TMP7]], <2 x float> undef)
-; FVW2-NEXT:    [[TMP9:%.*]] = fadd <2 x float> [[WIDE_MASKED_GATHER]], <float 5.000000e-01, float 5.000000e-01>
-; FVW2-NEXT:    [[TMP10:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
-; FVW2-NEXT:    br i1 [[TMP10]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
+; FVW2-NEXT:    [[TMP8:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
+; FVW2-NEXT:    br i1 [[TMP8]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; FVW2:       pred.load.if:
+; FVW2-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT_IN:%.*]], %struct.In* [[IN:%.*]], i64 [[OFFSET_IDX]], i32 1
+; FVW2-NEXT:    [[TMP10:%.*]] = load float, float* [[TMP9]], align 4
+; FVW2-NEXT:    [[TMP11:%.*]] = insertelement <2 x float> poison, float [[TMP10]], i32 0
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE]]
+; FVW2:       pred.load.continue:
+; FVW2-NEXT:    [[TMP12:%.*]] = phi <2 x float> [ poison, [[VECTOR_BODY]] ], [ [[TMP11]], [[PRED_LOAD_IF]] ]
+; FVW2-NEXT:    [[TMP13:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
+; FVW2-NEXT:    br i1 [[TMP13]], label [[PRED_LOAD_IF7:%.*]], label [[PRED_LOAD_CONTINUE8:%.*]]
+; FVW2:       pred.load.if7:
+; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [[STRUCT_IN]], %struct.In* [[IN]], i64 [[TMP0]], i32 1
+; FVW2-NEXT:    [[TMP15:%.*]] = load float, float* [[TMP14]], align 4
+; FVW2-NEXT:    [[TMP16:%.*]] = insertelement <2 x float> [[TMP12]], float [[TMP15]], i32 1
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE8]]
+; FVW2:       pred.load.continue8:
+; FVW2-NEXT:    [[TMP17:%.*]] = phi <2 x float> [ [[TMP12]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP16]], [[PRED_LOAD_IF7]] ]
+; FVW2-NEXT:    [[TMP18:%.*]] = fadd <2 x float> [[TMP17]], <float 5.000000e-01, float 5.000000e-01>
+; FVW2-NEXT:    [[TMP19:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
+; FVW2-NEXT:    br i1 [[TMP19]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
 ; FVW2:       pred.store.if:
-; FVW2-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT_OUT:%.*]], %struct.Out* [[OUT:%.*]], i64 [[OFFSET_IDX]], i32 1
-; FVW2-NEXT:    [[TMP12:%.*]] = extractelement <2 x float> [[TMP9]], i32 0
-; FVW2-NEXT:    store float [[TMP12]], float* [[TMP11]], align 4
+; FVW2-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [[STRUCT_OUT:%.*]], %struct.Out* [[OUT:%.*]], i64 [[OFFSET_IDX]], i32 1
+; FVW2-NEXT:    [[TMP21:%.*]] = extractelement <2 x float> [[TMP18]], i32 0
+; FVW2-NEXT:    store float [[TMP21]], float* [[TMP20]], align 4
 ; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE]]
 ; FVW2:       pred.store.continue:
-; FVW2-NEXT:    [[TMP13:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
-; FVW2-NEXT:    br i1 [[TMP13]], label [[PRED_STORE_IF7:%.*]], label [[PRED_STORE_CONTINUE8]]
-; FVW2:       pred.store.if7:
-; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [[STRUCT_OUT]], %struct.Out* [[OUT]], i64 [[TMP0]], i32 1
-; FVW2-NEXT:    [[TMP15:%.*]] = extractelement <2 x float> [[TMP9]], i32 1
-; FVW2-NEXT:    store float [[TMP15]], float* [[TMP14]], align 4
-; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE8]]
-; FVW2:       pred.store.continue8:
+; FVW2-NEXT:    [[TMP22:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
+; FVW2-NEXT:    br i1 [[TMP22]], label [[PRED_STORE_IF9:%.*]], label [[PRED_STORE_CONTINUE10]]
+; FVW2:       pred.store.if9:
+; FVW2-NEXT:    [[TMP23:%.*]] = getelementptr inbounds [[STRUCT_OUT]], %struct.Out* [[OUT]], i64 [[TMP0]], i32 1
+; FVW2-NEXT:    [[TMP24:%.*]] = extractelement <2 x float> [[TMP18]], i32 1
+; FVW2-NEXT:    store float [[TMP24]], float* [[TMP23]], align 4
+; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE10]]
+; FVW2:       pred.store.continue10:
 ; FVW2-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 2
-; FVW2-NEXT:    [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 32, i64 32>
-; FVW2-NEXT:    [[TMP16:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
-; FVW2-NEXT:    br i1 [[TMP16]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; FVW2-NEXT:    [[TMP25:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
+; FVW2-NEXT:    br i1 [[TMP25]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; FVW2:       for.end:
 ; FVW2-NEXT:    ret void
 ;
 entry:
   %in.addr = alloca %struct.In*, align 8
   %out.addr = alloca %struct.Out*, align 8
   %trigger.addr = alloca i32*, align 8
   %i = alloca i32, align 4

 ; AVX512-NEXT:    [[TMP79:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT]], <16 x i64> <i64 3840, i64 3856, i64 3872, i64 3888, i64 3904, i64 3920, i64 3936, i64 3952, i64 3968, i64 3984, i64 4000, i64 4016, i64 4032, i64 4048, i64 4064, i64 4080>
 ; AVX512-NEXT:    call void @llvm.masked.scatter.v16f32.v16p1f32(<16 x float> [[TMP78]], <16 x float addrspace(1)*> [[TMP79]], i32 4, <16 x i1> [[TMP76]])
 ; AVX512-NEXT:    ret void
 ;
 ; FVW2-LABEL: @foo2_addrspace(
 ; FVW2-NEXT:  entry:
 ; FVW2-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; FVW2:       vector.body:
-; FVW2-NEXT:    [[INDEX7:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE9:%.*]] ]
-; FVW2-NEXT:    [[VEC_IND:%.*]] = phi <2 x i64> [ <i64 0, i64 16>, [[ENTRY]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE9]] ]
+; FVW2-NEXT:    [[INDEX7:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE11:%.*]] ]
 ; FVW2-NEXT:    [[OFFSET_IDX:%.*]] = shl i64 [[INDEX7]], 4
 ; FVW2-NEXT:    [[TMP0:%.*]] = or i64 [[OFFSET_IDX]], 16
 ; FVW2-NEXT:    [[TMP1:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER:%.*]], i64 [[OFFSET_IDX]]
 ; FVW2-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER]], i64 [[TMP0]]
 ; FVW2-NEXT:    [[TMP3:%.*]] = load i32, i32* [[TMP1]], align 4
 ; FVW2-NEXT:    [[TMP4:%.*]] = load i32, i32* [[TMP2]], align 4
 ; FVW2-NEXT:    [[TMP5:%.*]] = insertelement <2 x i32> poison, i32 [[TMP3]], i32 0
 ; FVW2-NEXT:    [[TMP6:%.*]] = insertelement <2 x i32> [[TMP5]], i32 [[TMP4]], i32 1
 ; FVW2-NEXT:    [[TMP7:%.*]] = icmp sgt <2 x i32> [[TMP6]], zeroinitializer
-; FVW2-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT_IN:%.*]], [[STRUCT_IN]] addrspace(1)* [[IN:%.*]], <2 x i64> [[VEC_IND]], i32 1
-; FVW2-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <2 x float> @llvm.masked.gather.v2f32.v2p1f32(<2 x float addrspace(1)*> [[TMP8]], i32 4, <2 x i1> [[TMP7]], <2 x float> undef)
-; FVW2-NEXT:    [[TMP9:%.*]] = fadd <2 x float> [[WIDE_MASKED_GATHER]], <float 5.000000e-01, float 5.000000e-01>
-; FVW2-NEXT:    [[TMP10:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
-; FVW2-NEXT:    br i1 [[TMP10]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
+; FVW2-NEXT:    [[TMP8:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
+; FVW2-NEXT:    br i1 [[TMP8]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; FVW2:       pred.load.if:
+; FVW2-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT_IN:%.*]], [[STRUCT_IN]] addrspace(1)* [[IN:%.*]], i64 [[OFFSET_IDX]], i32 1
+; FVW2-NEXT:    [[TMP10:%.*]] = load float, float addrspace(1)* [[TMP9]], align 4
+; FVW2-NEXT:    [[TMP11:%.*]] = insertelement <2 x float> poison, float [[TMP10]], i32 0
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE]]
+; FVW2:       pred.load.continue:
+; FVW2-NEXT:    [[TMP12:%.*]] = phi <2 x float> [ poison, [[VECTOR_BODY]] ], [ [[TMP11]], [[PRED_LOAD_IF]] ]
+; FVW2-NEXT:    [[TMP13:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
+; FVW2-NEXT:    br i1 [[TMP13]], label [[PRED_LOAD_IF8:%.*]], label [[PRED_LOAD_CONTINUE9:%.*]]
+; FVW2:       pred.load.if8:
+; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [[STRUCT_IN]], [[STRUCT_IN]] addrspace(1)* [[IN]], i64 [[TMP0]], i32 1
+; FVW2-NEXT:    [[TMP15:%.*]] = load float, float addrspace(1)* [[TMP14]], align 4
+; FVW2-NEXT:    [[TMP16:%.*]] = insertelement <2 x float> [[TMP12]], float [[TMP15]], i32 1
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE9]]
+; FVW2:       pred.load.continue9:
+; FVW2-NEXT:    [[TMP17:%.*]] = phi <2 x float> [ [[TMP12]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP16]], [[PRED_LOAD_IF8]] ]
+; FVW2-NEXT:    [[TMP18:%.*]] = fadd <2 x float> [[TMP17]], <float 5.000000e-01, float 5.000000e-01>
+; FVW2-NEXT:    [[TMP19:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
+; FVW2-NEXT:    br i1 [[TMP19]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
 ; FVW2:       pred.store.if:
-; FVW2-NEXT:    [[TMP11:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT:%.*]], i64 [[OFFSET_IDX]]
-; FVW2-NEXT:    [[TMP12:%.*]] = extractelement <2 x float> [[TMP9]], i32 0
-; FVW2-NEXT:    store float [[TMP12]], float addrspace(1)* [[TMP11]], align 4
+; FVW2-NEXT:    [[TMP20:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT:%.*]], i64 [[OFFSET_IDX]]
+; FVW2-NEXT:    [[TMP21:%.*]] = extractelement <2 x float> [[TMP18]], i32 0
+; FVW2-NEXT:    store float [[TMP21]], float addrspace(1)* [[TMP20]], align 4
 ; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE]]
 ; FVW2:       pred.store.continue:
-; FVW2-NEXT:    [[TMP13:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
-; FVW2-NEXT:    br i1 [[TMP13]], label [[PRED_STORE_IF8:%.*]], label [[PRED_STORE_CONTINUE9]]
-; FVW2:       pred.store.if8:
-; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT]], i64 [[TMP0]]
-; FVW2-NEXT:    [[TMP15:%.*]] = extractelement <2 x float> [[TMP9]], i32 1
-; FVW2-NEXT:    store float [[TMP15]], float addrspace(1)* [[TMP14]], align 4
-; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE9]]
-; FVW2:       pred.store.continue9:
+; FVW2-NEXT:    [[TMP22:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
+; FVW2-NEXT:    br i1 [[TMP22]], label [[PRED_STORE_IF10:%.*]], label [[PRED_STORE_CONTINUE11]]
+; FVW2:       pred.store.if10:
+; FVW2-NEXT:    [[TMP23:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT]], i64 [[TMP0]]
+; FVW2-NEXT:    [[TMP24:%.*]] = extractelement <2 x float> [[TMP18]], i32 1
+; FVW2-NEXT:    store float [[TMP24]], float addrspace(1)* [[TMP23]], align 4
+; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE11]]
+; FVW2:       pred.store.continue11:
 ; FVW2-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX7]], 2
-; FVW2-NEXT:    [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 32, i64 32>
-; FVW2-NEXT:    [[TMP16:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
-; FVW2-NEXT:    br i1 [[TMP16]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
+; FVW2-NEXT:    [[TMP25:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
+; FVW2-NEXT:    br i1 [[TMP25]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; FVW2:       for.end:
 ; FVW2-NEXT:    ret void
 ;
 entry:
   %in.addr = alloca %struct.In addrspace(1)*, align 8
   %out.addr = alloca float addrspace(1)*, align 8
   %trigger.addr = alloca i32*, align 8
   %index.addr = alloca i32*, align 8

 ; AVX512-NEXT:    [[TMP79:%.*]] = getelementptr inbounds float, float* [[OUT]], <16 x i64> <i64 3840, i64 3856, i64 3872, i64 3888, i64 3904, i64 3920, i64 3936, i64 3952, i64 3968, i64 3984, i64 4000, i64 4016, i64 4032, i64 4048, i64 4064, i64 4080>
 ; AVX512-NEXT:    call void @llvm.masked.scatter.v16f32.v16p0f32(<16 x float> [[TMP78]], <16 x float*> [[TMP79]], i32 4, <16 x i1> [[TMP76]])
 ; AVX512-NEXT:    ret void
 ;
 ; FVW2-LABEL: @foo2_addrspace2(
 ; FVW2-NEXT:  entry:
 ; FVW2-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; FVW2:       vector.body:
-; FVW2-NEXT:    [[INDEX7:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE9:%.*]] ]
-; FVW2-NEXT:    [[VEC_IND:%.*]] = phi <2 x i64> [ <i64 0, i64 16>, [[ENTRY]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE9]] ]
+; FVW2-NEXT:    [[INDEX7:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE11:%.*]] ]
 ; FVW2-NEXT:    [[OFFSET_IDX:%.*]] = shl i64 [[INDEX7]], 4
 ; FVW2-NEXT:    [[TMP0:%.*]] = or i64 [[OFFSET_IDX]], 16
 ; FVW2-NEXT:    [[TMP1:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER:%.*]], i64 [[OFFSET_IDX]]
 ; FVW2-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER]], i64 [[TMP0]]
 ; FVW2-NEXT:    [[TMP3:%.*]] = load i32, i32* [[TMP1]], align 4
 ; FVW2-NEXT:    [[TMP4:%.*]] = load i32, i32* [[TMP2]], align 4
 ; FVW2-NEXT:    [[TMP5:%.*]] = insertelement <2 x i32> poison, i32 [[TMP3]], i32 0
 ; FVW2-NEXT:    [[TMP6:%.*]] = insertelement <2 x i32> [[TMP5]], i32 [[TMP4]], i32 1
 ; FVW2-NEXT:    [[TMP7:%.*]] = icmp sgt <2 x i32> [[TMP6]], zeroinitializer
-; FVW2-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT_IN:%.*]], [[STRUCT_IN]] addrspace(1)* [[IN:%.*]], <2 x i64> [[VEC_IND]], i32 1
-; FVW2-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <2 x float> @llvm.masked.gather.v2f32.v2p1f32(<2 x float addrspace(1)*> [[TMP8]], i32 4, <2 x i1> [[TMP7]], <2 x float> undef)
-; FVW2-NEXT:    [[TMP9:%.*]] = fadd <2 x float> [[WIDE_MASKED_GATHER]], <float 5.000000e-01, float 5.000000e-01>
-; FVW2-NEXT:    [[TMP10:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
-; FVW2-NEXT:    br i1 [[TMP10]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
+; FVW2-NEXT:    [[TMP8:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
+; FVW2-NEXT:    br i1 [[TMP8]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; FVW2:       pred.load.if:
+; FVW2-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT_IN:%.*]], [[STRUCT_IN]] addrspace(1)* [[IN:%.*]], i64 [[OFFSET_IDX]], i32 1
+; FVW2-NEXT:    [[TMP10:%.*]] = load float, float addrspace(1)* [[TMP9]], align 4
+; FVW2-NEXT:    [[TMP11:%.*]] = insertelement <2 x float> poison, float [[TMP10]], i32 0
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE]]
+; FVW2:       pred.load.continue:
+; FVW2-NEXT:    [[TMP12:%.*]] = phi <2 x float> [ poison, [[VECTOR_BODY]] ], [ [[TMP11]], [[PRED_LOAD_IF]] ]
+; FVW2-NEXT:    [[TMP13:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
+; FVW2-NEXT:    br i1 [[TMP13]], label [[PRED_LOAD_IF8:%.*]], label [[PRED_LOAD_CONTINUE9:%.*]]
+; FVW2:       pred.load.if8:
+; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [[STRUCT_IN]], [[STRUCT_IN]] addrspace(1)* [[IN]], i64 [[TMP0]], i32 1
+; FVW2-NEXT:    [[TMP15:%.*]] = load float, float addrspace(1)* [[TMP14]], align 4
+; FVW2-NEXT:    [[TMP16:%.*]] = insertelement <2 x float> [[TMP12]], float [[TMP15]], i32 1
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE9]]
+; FVW2:       pred.load.continue9:
+; FVW2-NEXT:    [[TMP17:%.*]] = phi <2 x float> [ [[TMP12]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP16]], [[PRED_LOAD_IF8]] ]
+; FVW2-NEXT:    [[TMP18:%.*]] = fadd <2 x float> [[TMP17]], <float 5.000000e-01, float 5.000000e-01>
+; FVW2-NEXT:    [[TMP19:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
+; FVW2-NEXT:    br i1 [[TMP19]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
 ; FVW2:       pred.store.if:
-; FVW2-NEXT:    [[TMP11:%.*]] = getelementptr inbounds float, float* [[OUT:%.*]], i64 [[OFFSET_IDX]]
-; FVW2-NEXT:    [[TMP12:%.*]] = extractelement <2 x float> [[TMP9]], i32 0
-; FVW2-NEXT:    store float [[TMP12]], float* [[TMP11]], align 4
+; FVW2-NEXT:    [[TMP20:%.*]] = getelementptr inbounds float, float* [[OUT:%.*]], i64 [[OFFSET_IDX]]
+; FVW2-NEXT:    [[TMP21:%.*]] = extractelement <2 x float> [[TMP18]], i32 0
+; FVW2-NEXT:    store float [[TMP21]], float* [[TMP20]], align 4
 ; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE]]
 ; FVW2:       pred.store.continue:
-; FVW2-NEXT:    [[TMP13:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
-; FVW2-NEXT:    br i1 [[TMP13]], label [[PRED_STORE_IF8:%.*]], label [[PRED_STORE_CONTINUE9]]
-; FVW2:       pred.store.if8:
-; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr inbounds float, float* [[OUT]], i64 [[TMP0]]
-; FVW2-NEXT:    [[TMP15:%.*]] = extractelement <2 x float> [[TMP9]], i32 1
-; FVW2-NEXT:    store float [[TMP15]], float* [[TMP14]], align 4
-; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE9]]
-; FVW2:       pred.store.continue9:
+; FVW2-NEXT:    [[TMP22:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
+; FVW2-NEXT:    br i1 [[TMP22]], label [[PRED_STORE_IF10:%.*]], label [[PRED_STORE_CONTINUE11]]
+; FVW2:       pred.store.if10:
+; FVW2-NEXT:    [[TMP23:%.*]] = getelementptr inbounds float, float* [[OUT]], i64 [[TMP0]]
+; FVW2-NEXT:    [[TMP24:%.*]] = extractelement <2 x float> [[TMP18]], i32 1
+; FVW2-NEXT:    store float [[TMP24]], float* [[TMP23]], align 4
+; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE11]]
+; FVW2:       pred.store.continue11:
 ; FVW2-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX7]], 2
-; FVW2-NEXT:    [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 32, i64 32>
-; FVW2-NEXT:    [[TMP16:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
-; FVW2-NEXT:    br i1 [[TMP16]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
+; FVW2-NEXT:    [[TMP25:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
+; FVW2-NEXT:    br i1 [[TMP25]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; FVW2:       for.end:
 ; FVW2-NEXT:    ret void
 ;
 entry:
   %in.addr = alloca %struct.In addrspace(1)*, align 8
   %out.addr = alloca float addrspace(0)*, align 8
   %trigger.addr = alloca i32*, align 8
   %index.addr = alloca i32*, align 8

 ; AVX512-NEXT:    [[TMP79:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT]], <16 x i64> <i64 3840, i64 3856, i64 3872, i64 3888, i64 3904, i64 3920, i64 3936, i64 3952, i64 3968, i64 3984, i64 4000, i64 4016, i64 4032, i64 4048, i64 4064, i64 4080>
 ; AVX512-NEXT:    call void @llvm.masked.scatter.v16f32.v16p1f32(<16 x float> [[TMP78]], <16 x float addrspace(1)*> [[TMP79]], i32 4, <16 x i1> [[TMP76]])
 ; AVX512-NEXT:    ret void
 ;
 ; FVW2-LABEL: @foo2_addrspace3(
 ; FVW2-NEXT:  entry:
 ; FVW2-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; FVW2:       vector.body:
-; FVW2-NEXT:    [[INDEX7:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE9:%.*]] ]
-; FVW2-NEXT:    [[VEC_IND:%.*]] = phi <2 x i64> [ <i64 0, i64 16>, [[ENTRY]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE9]] ]
+; FVW2-NEXT:    [[INDEX7:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE11:%.*]] ]
 ; FVW2-NEXT:    [[OFFSET_IDX:%.*]] = shl i64 [[INDEX7]], 4
 ; FVW2-NEXT:    [[TMP0:%.*]] = or i64 [[OFFSET_IDX]], 16
 ; FVW2-NEXT:    [[TMP1:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER:%.*]], i64 [[OFFSET_IDX]]
 ; FVW2-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[TRIGGER]], i64 [[TMP0]]
 ; FVW2-NEXT:    [[TMP3:%.*]] = load i32, i32* [[TMP1]], align 4
 ; FVW2-NEXT:    [[TMP4:%.*]] = load i32, i32* [[TMP2]], align 4
 ; FVW2-NEXT:    [[TMP5:%.*]] = insertelement <2 x i32> poison, i32 [[TMP3]], i32 0
 ; FVW2-NEXT:    [[TMP6:%.*]] = insertelement <2 x i32> [[TMP5]], i32 [[TMP4]], i32 1
 ; FVW2-NEXT:    [[TMP7:%.*]] = icmp sgt <2 x i32> [[TMP6]], zeroinitializer
-; FVW2-NEXT:    [[TMP8:%.*]] = getelementptr inbounds [[STRUCT_IN:%.*]], %struct.In* [[IN:%.*]], <2 x i64> [[VEC_IND]], i32 1
-; FVW2-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <2 x float> @llvm.masked.gather.v2f32.v2p0f32(<2 x float*> [[TMP8]], i32 4, <2 x i1> [[TMP7]], <2 x float> undef)
-; FVW2-NEXT:    [[TMP9:%.*]] = fadd <2 x float> [[WIDE_MASKED_GATHER]], <float 5.000000e-01, float 5.000000e-01>
-; FVW2-NEXT:    [[TMP10:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
-; FVW2-NEXT:    br i1 [[TMP10]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
+; FVW2-NEXT:    [[TMP8:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
+; FVW2-NEXT:    br i1 [[TMP8]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; FVW2:       pred.load.if:
+; FVW2-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [[STRUCT_IN:%.*]], %struct.In* [[IN:%.*]], i64 [[OFFSET_IDX]], i32 1
+; FVW2-NEXT:    [[TMP10:%.*]] = load float, float* [[TMP9]], align 4
+; FVW2-NEXT:    [[TMP11:%.*]] = insertelement <2 x float> poison, float [[TMP10]], i32 0
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE]]
+; FVW2:       pred.load.continue:
+; FVW2-NEXT:    [[TMP12:%.*]] = phi <2 x float> [ poison, [[VECTOR_BODY]] ], [ [[TMP11]], [[PRED_LOAD_IF]] ]
+; FVW2-NEXT:    [[TMP13:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
+; FVW2-NEXT:    br i1 [[TMP13]], label [[PRED_LOAD_IF8:%.*]], label [[PRED_LOAD_CONTINUE9:%.*]]
+; FVW2:       pred.load.if8:
+; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr inbounds [[STRUCT_IN]], %struct.In* [[IN]], i64 [[TMP0]], i32 1
+; FVW2-NEXT:    [[TMP15:%.*]] = load float, float* [[TMP14]], align 4
+; FVW2-NEXT:    [[TMP16:%.*]] = insertelement <2 x float> [[TMP12]], float [[TMP15]], i32 1
+; FVW2-NEXT:    br label [[PRED_LOAD_CONTINUE9]]
+; FVW2:       pred.load.continue9:
+; FVW2-NEXT:    [[TMP17:%.*]] = phi <2 x float> [ [[TMP12]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP16]], [[PRED_LOAD_IF8]] ]
+; FVW2-NEXT:    [[TMP18:%.*]] = fadd <2 x float> [[TMP17]], <float 5.000000e-01, float 5.000000e-01>
+; FVW2-NEXT:    [[TMP19:%.*]] = extractelement <2 x i1> [[TMP7]], i32 0
+; FVW2-NEXT:    br i1 [[TMP19]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
 ; FVW2:       pred.store.if:
-; FVW2-NEXT:    [[TMP11:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT:%.*]], i64 [[OFFSET_IDX]]
-; FVW2-NEXT:    [[TMP12:%.*]] = extractelement <2 x float> [[TMP9]], i32 0
-; FVW2-NEXT:    store float [[TMP12]], float addrspace(1)* [[TMP11]], align 4
+; FVW2-NEXT:    [[TMP20:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT:%.*]], i64 [[OFFSET_IDX]]
+; FVW2-NEXT:    [[TMP21:%.*]] = extractelement <2 x float> [[TMP18]], i32 0
+; FVW2-NEXT:    store float [[TMP21]], float addrspace(1)* [[TMP20]], align 4
 ; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE]]
 ; FVW2:       pred.store.continue:
-; FVW2-NEXT:    [[TMP13:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
-; FVW2-NEXT:    br i1 [[TMP13]], label [[PRED_STORE_IF8:%.*]], label [[PRED_STORE_CONTINUE9]]
-; FVW2:       pred.store.if8:
-; FVW2-NEXT:    [[TMP14:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT]], i64 [[TMP0]]
-; FVW2-NEXT:    [[TMP15:%.*]] = extractelement <2 x float> [[TMP9]], i32 1
-; FVW2-NEXT:    store float [[TMP15]], float addrspace(1)* [[TMP14]], align 4
-; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE9]]
-; FVW2:       pred.store.continue9:
+; FVW2-NEXT:    [[TMP22:%.*]] = extractelement <2 x i1> [[TMP7]], i32 1
+; FVW2-NEXT:    br i1 [[TMP22]], label [[PRED_STORE_IF10:%.*]], label [[PRED_STORE_CONTINUE11]]
+; FVW2:       pred.store.if10:
+; FVW2-NEXT:    [[TMP23:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT]], i64 [[TMP0]]
+; FVW2-NEXT:    [[TMP24:%.*]] = extractelement <2 x float> [[TMP18]], i32 1
+; FVW2-NEXT:    store float [[TMP24]], float addrspace(1)* [[TMP23]], align 4
+; FVW2-NEXT:    br label [[PRED_STORE_CONTINUE11]]
+; FVW2:       pred.store.continue11:
 ; FVW2-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX7]], 2
-; FVW2-NEXT:    [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 32, i64 32>
-; FVW2-NEXT:    [[TMP16:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
-; FVW2-NEXT:    br i1 [[TMP16]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
+; FVW2-NEXT:    [[TMP25:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
+; FVW2-NEXT:    br i1 [[TMP25]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; FVW2:       for.end:
 ; FVW2-NEXT:    ret void
 ;
 entry:
   %in.addr = alloca %struct.In addrspace(0)*, align 8
   %out.addr = alloca float addrspace(1)*, align 8
   %trigger.addr = alloca i32*, align 8
   %index.addr = alloca i32*, align 8