[SVE][LoopVectorize] Add support for scalable vectorization of loops with vector reverse

[CarolineConcatto]

Summary

This patch adds support for reverse loop vectorization.
It is possible to vectorize the following loop:

for (int i = n-1; i >= 0; --i)
  a[i] = b[i] + 1.0;

with fixed or scalable vector.
The loop-vectorizer will use 'reverse' on the loads/stores to make
sure the lanes themselves are also handled in the right order.
This patch adds support for scalable vector on IRBuilder interface to
create a reverse vector. The IR function
CreateVectorReverse lowers to experimental.vector.reverse for scalable vector
and keedp the original behavior for fixed vector using shuffle reverse.

Depends on D94883

Depends on D95603

Depends on D95139

[CarolineConcatto]

-s/scalabe/scalable

[david-arm]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

451

Hi Carol, is the reason why you bail out here because we are casting between at least one illegal type? If so, perhaps you could add a TODO here, something like:

// TODO : For scalable vectors we avoid calling the BaseT version for now because it doesn't yet work for illegal vector types as it assumes these are fixed width.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

2801

Hi Carol. I don't think this is right here. The number of cached lanes in my patch (D95139) simply refers to the number of values we have managed to cache. It's just an optimisation the vectoriser uses to avoid constantly calculating the same lane index expressions. However, what you're after here is a runtime calculation for the last lane index - you can use getRuntimeVF in my patch for this. I think the code below will probably need splitting out into fixed width vs scalable cases, i.e.

Builder.getInt32(1 - NumElts)

would need to be something like:

Builder.CreateSub(Builder.getInt32(1), getRuntimeVF())

for scalable vectors. Hope that makes sense!

[sdesmalen]

llvm/include/llvm/IR/IRBuilder.h

2538

It would be nice if the IRBuilder interface supports both the scalable and fixed-width case, i.e. for scalable vectors it creates the intrinsic, for fixed-width vectors it creates a shufflevector.
Then in the LoopVectorizer it can just call CreateVectorReverse(Vec) and get the same behaviour.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

451–456

I don't think you should return just some random value here. Is there anything in broken in BasicTTIImpl that needs fixing?

If you run into any asserts caused by the cast from FP -> Int, you can simplify the test so that it doesn't need that cast (and then fix this up in a separate patch)

1190

Can you instead add cases to the table?

e.g.

{ TTI::SK_Reverse, MVT::nxv16i8, 1 },
{ TTI::SK_Reverse, MVT::nxv8i16, 1 },
 ...

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

2800

NumElts can't be unsigned here, because it has to be a value that's evaluated at runtime.
This means it needs to multiply VF.getKnownMinValue by the runtime value of VScale, making NumElts a Value *.

For the first GEP you can also do the arithmetic on a pointer of the vector type, and index by -Part, e.g.

getelementptr <vscale x 8 x double>, <vscale x 8 x double> *%ptr, i64 -<part>

<=>

%vscale = call i64 @llvm.vscale()
%part = mul i64 %vscale, 8
getelementptr double, double *%ptr, i64 %part

The second GEP indexes a specific element, so that would still need to be a getelementptr on a double*.

llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll

23

It is difficult to verify what this code is generating, and the GEPs here are actually really important to get right, so can you just check for explicit values?

[fhahn]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1191

Also, can this have a separate cost model test? If so, can this be split off as a separate improvement?

[CarolineConcatto]

• Move cost function for Shuffle reverse to D93639
• Use getRunTimeVF to compute the runtime vector size for scalable vector
• add tests for fixed vector

[CarolineConcatto]

Thank you all for the review.
I believe I had addressed your comments.
I've created another patch to compute the cost for vector reverse with scalable vector.
I've removed the cast from int to double in one of the tests, so we don't need cast fixed for this feature.
And the IR builder now has a reverse function for fixed and scalable vectors.

llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll

23

I don't know if I understood what you want.
I had to change the checks.
Do you mind to show what do you like to see being checked in these tests?

[david-arm]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

2813

Maybe better to use Builder.getInt32(-Part) to be consistent with how it's done for the fixed width case?

llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll

15

If you use a i64 variable here then it makes the tests simpler as the variable doesn't need sign-extending with sext

22

Hi @CarolineConcatto something doesn't look right here because the result will be zero.

51

See comment about N - this line can be killed if you have "i64 N", then everywhere that uses %0 can just use %N instead. If you do this then the for.body.preheader block can be killed too and the branch above can jump straight to for.body

71

Same comment as in previous function.

[david-arm]

llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll

22

I think I may have made a mistake here and the '0' is there because this is the 0th Part. Looking at the way GEPs are created it doesn't really optimise the case when Part=0. Adding "-instcombine" to the RUN line would clear up any dead code here.

[CarolineConcatto]

• change how to get NumElt inside LoopVectorize
• change the index to be 64 bits

[CarolineConcatto]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

2813

Thank you David, you are correct things should be coherent.

llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll

51

@david-arm
I've replaced the index to be 64 bits, but I could not remove the for.body.preheader.

[CarolineConcatto]

-removing redundant lines in the test

[CarolineConcatto]

• change index 'i' in the tests to be 64 bits instead of 32bits

[CarolineConcatto]

Thank you @david-arm.
I've removed the pre.header.
It is indeed better.

[david-arm]

Hi @CarolineConcatto, it's looking much better now and thanks for dealing with all the review comments!

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

2812

Hi @CarolineConcatto, I just realised there isn't a test for this for scalable vectors. Would you be able to add a test for blocks with masks too? I think it should be something like:

void foo(int * __restrict__ a, int * __restrict__ cond, long long N) {
  for (int i = N - 1; i >= 0; i--)
  {
    if (cond[i])
      a[i] += 1;
  }
}

llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll

22 ↗

(On Diff #322962)

I think that since you're adding tests for this it's probably a good idea to expand all the shufflevector CHECK lines in this file to make sure we have a reverse mask here too?

[bin.cheng-ali]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

2825

Sorry for one nitpicking.
IIUC the difference for scalable/fixed cases is the two operands of GEPs, is it better to factor out common code by doing below?
if (VF.isScalable()) {

// build GEP operands for scalable case

}
else {

// build GEP operands for fixed case

}
// common code building VecPtr with above GEP operands.

[david-arm]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

2825

I guess alternatively we could commonise them into one block where we always calculate

Value *LastLane =
    Builder.CreateSub(Builder.getInt32(1),
                      getRuntimeVF(Builder, Builder.getInt32Ty(), VF));

for both fixed-width and scalable vectors, and just let instcombine and other passes fold the constants? I'm not sure if that breaks lots of tests though ...

[CarolineConcatto]

• Add tests for when vector.reverse needs to reverse a mask
• improve code in LoopVectorize.cpp
• C&P function getRunTimeVF from D95139

[CarolineConcatto]

Thank you all for the review.
I C&P the function getRunTime from D95139.
If D95139 is merged before this patch I will remove it.
All the other dependencies are already in main, and getRunTimeVF is the only function that this patch needs from D95139.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

2812

Thank you @david-arm for pointing that.
Yes, indeed it was missing test for that.
The cost model for i1 was missing with vector reverse.

2825

So @david-arm 's suggestion works for fixed width and scalable vectors.
At least no test has failed.
I could also have done the way @bin.cheng-ali suggested:

if (scalable){
  NumElts =   Builder.CreateMul(Builder.getInt32(1),
                      getRuntimeVF(Builder, Builder.getInt32Ty(), VF));
  LastLane =   Builder.CreateSub(Builder.getInt32(1),
                      getRuntimeVF(Builder, Builder.getInt32Ty(), VF));
}else{
   NumElts = 1 * VF.getKnownMinValue()
  LastLane = 1 - VF.getKnownMinValue()
}

But as all work with getRunTime I thought it was code duplication and this way the code looks simpler.
I added a comment to explain how it works if it is fixed-width vector for me in the future too

[CarolineConcatto]

llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll

15

Hi @david-arm I
think I forgot to answer that suggestion.

This file has 2 tests, for 2 variable types, double and integer variables

@vector_reverse_f64(i64 %N, double* %a, double* %b)
 @vector_reverse_i64(i64 %N, i64* %a, i64* %b)

So I believe it is ok to leave this as it is. Unless I am missing something. (Which is completely ok for me)
I can see for @vector_reverse_i64 that sext instruction is gone.

[david-arm]

llvm/include/llvm/IR/IRBuilder.h

2533

nit: This function looks a bit too complex to live in a header - maybe better defined in the .cpp file?

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

1146

nit: One of my patches has already landed with this function so I think you'll need to remove it when committing.

2811

nit: Maybe instead of calling getRuntimeVF twice you can call it once and assign it to a variable?

llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll

24 ↗

(On Diff #328434)

nit: Perhaps better renamed to nxv4i1 since that's what the generated code looks like?

llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll

24 ↗

(On Diff #328434)

nit: Perhaps better renamed to v4i1 since the CHECK lines have that type?

27 ↗

(On Diff #328434)

This seems a bit less well tested than the sve version - perhaps it's worth adding CHECK lines for the store too? I assume the store will reuse the same mask?

llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll

29 ↗

(On Diff #328434)

nit: %[[REVERSE6]]?

64 ↗

(On Diff #328434)

nit: %[[REVERSE6]]

[CarolineConcatto]

• Fix nit in the .ll tests and LoopVectorize.cpp
• Remote copied getRunTimeVF from LoopVectorize.cpp
• Move CreateVectorReverse implementation from IRBuilder.h to IRBuilder.cpp

[CarolineConcatto]

Hey @david-arm sorry for the nit in the tests.
Usually happens when I do copy and paste many times.
I've updated the patch and rebased it with the main.

Carol

llvm/include/llvm/IR/IRBuilder.h

2533

Thank you David.
I hadn't noticed that this could be also implemented in IRBuilder.cpp.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

1146

Wonderfull! Good to know!

2811

Good idea David, so I can reduce code duplication

llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll

24 ↗

(On Diff #328434)

That is what happens when I do C&P. I've creates tests for all masks sizes, but only submitted one size.
I was wondering, I did not add tests for all mask sizes here.
Do you think it is valid?
I guess that one size fits all, if one mask size does not break it means that the rest will not break too.

llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll

27 ↗

(On Diff #328434)

Ok, I think it is valid to have similar check's for scalable and fixed vectors

llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll

29 ↗

(On Diff #328434)

Good catch!

[david-arm]

LGTM!

llvm/lib/IR/IRBuilder.cpp

996 ↗

(On Diff #328984)

nit: I think in this case it's worth fixing the clang-tidy comment. It looks like the function above also uses "auto *..."

llvm/include/llvm/IR/IRBuilder.h

public:
 
   /// Create a unary shuffle. The second vector operand of the IR instruction
   /// is poison.
   Value *CreateShuffleVector(Value *V, ArrayRef<int> Mask,
                              const Twine &Name = "") {
     return CreateShuffleVector(V, PoisonValue::get(V->getType()), Mask, Name);
   }
 
+  Value *CreateVectorReverse(Value *V, const Twine &Name = "") {
+    auto Ty = cast<VectorType>(V->getType());
+    Module *M = BB->getParent()->getParent();
+    Function *F = Intrinsic::getDeclaration(
+        M, Intrinsic::experimental_vector_reverse, Ty);
+    return Insert(CallInst::Create(F, V), Name);
+  }
+
   Value *CreateExtractValue(Value *Agg,
                             ArrayRef<unsigned> Idxs,
                             const Twine &Name = "") {
     if (auto *AggC = dyn_cast<Constant>(Agg))
       return Insert(Folder.CreateExtractValue(AggC, Idxs), Name);
     return Insert(ExtractValueInst::Create(Agg, Idxs), Name);
   }
 

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

ConversionTbl[] = {
     { ISD::FP_TO_UINT, MVT::v2i8,  MVT::v2f64, 2 },
   };
 
   if (const auto *Entry = ConvertCostTableLookup(ConversionTbl, ISD,
                                                  DstTy.getSimpleVT(),
                                                  SrcTy.getSimpleVT()))
     return AdjustCost(Entry->Cost);
 
+  // Avoid to call BasicTTImpl for scalable vector and then assert
+  // ATM in the BasicTTImpl the cost for ISD that accepts scalable
+  // vector is 0
+  // but not all of them accept scalable vector.
+  if (isa<ScalableVectorType>(Src))
+    return 0;
+
   return AdjustCost(
       BaseT::getCastInstrCost(Opcode, Dst, Src, CCH, CostKind, I));
 }
 
 int AArch64TTIImpl::getExtractWithExtendCost(unsigned Opcode, Type *Dst,
                                              VectorType *VecTy,
                                              unsigned Index) {
 
static const CostTblEntry ShuffleTbl[] = {
       { TTI::SK_PermuteSingleSrc, MVT::v4f32, 3 }, // perfectshuffle worst case.
       { TTI::SK_PermuteSingleSrc, MVT::v2f64, 1 }, // mov.
     };
     std::pair<int, MVT> LT = TLI->getTypeLegalizationCost(DL, Tp);
     if (const auto *Entry = CostTableLookup(ShuffleTbl, Kind, LT.second))
       return LT.first * Entry->Cost;
   }
 
+  // Scalable vector type use the new intrinsic vector_reverse
+  // for TTI::SK_Reverse. Fixed vector types keep using shuffle vector
+  if (Kind == TTI::SK_Reverse && isa<ScalableVectorType>(Tp)) {
+    std::pair<int, MVT> LT = TLI->getTypeLegalizationCost(DL, Tp);
+    return LT.first + 1;
+  }
+
   return BaseT::getShuffleCost(Kind, Tp, Index, SubTp);
 }

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

 static Value *createStepForVF(IRBuilder<> &B, Constant *Step, ElementCount VF) {
   assert(isa<ConstantInt>(Step) && "Expected an integer step");
   Constant *StepVal = ConstantInt::get(
       Step->getType(),
       cast<ConstantInt>(Step)->getSExtValue() * VF.getKnownMinValue());
   return VF.isScalable() ? B.CreateVScale(StepVal) : StepVal;
 }
 
 /// Return the runtime value for VF.
 static Value *getRuntimeVF(IRBuilder<> &B, Type *Ty, ElementCount VF) {
   Constant *EC = ConstantInt::get(Ty, VF.getKnownMinValue());
   return VF.isScalable() ? B.CreateVScale(EC) : EC;
 }
 
 namespace llvm {
 
 void reportVectorizationFailure(const StringRef DebugMsg,
void InnerLoopVectorizer::packScalarIntoVectorValue(
   assert(!Instance.isNonConstLane() && "Only constant lane indices supported");
   VectorValue = Builder.CreateInsertElement(VectorValue, ScalarInst,
                                             Builder.getInt32(Instance.Lane));
   VectorLoopValueMap.resetVectorValue(V, Instance.Part, VectorValue);
 }
 
 Value *InnerLoopVectorizer::reverseVector(Value *Vec) {
   assert(Vec->getType()->isVectorTy() && "Invalid type");
-  assert(!VF.isScalable() && "Cannot reverse scalable vectors");
+
+  // Call the instrinsic vector_reverse for scalable vector
+  if (VF.isScalable())
+    return Builder.CreateVectorReverse(Vec, "reverse");
+
+  // Keep the original behaviour for fixed vector
   SmallVector<int, 8> ShuffleMask;
   for (unsigned i = 0; i < VF.getKnownMinValue(); ++i)
     ShuffleMask.push_back(VF.getKnownMinValue() - i - 1);
 
   return Builder.CreateShuffleVector(Vec, ShuffleMask, "reverse");
 }
 
 // Return whether we allow using masked interleave-groups (for dealing with
const auto CreateVecPtr = [&](unsigned Part, Value *Ptr) -> Value * {
     // Calculate the pointer for the specific unroll-part.
     GetElementPtrInst *PartPtr = nullptr;
 
     bool InBounds = false;
     if (auto *gep = dyn_cast<GetElementPtrInst>(Ptr->stripPointerCasts()))
       InBounds = gep->isInBounds();
 
     if (Reverse) {
-      assert(!VF.isScalable() &&
-             "Reversing vectors is not yet supported for scalable vectors.");
-
+      unsigned NumElts = VF.isScalable()
+                             ? VectorLoopValueMap.getNumCachedLanes()
+                             : VF.getKnownMinValue();
       // If the address is consecutive but reversed, then the
       // wide store needs to start at the last vector element.
       PartPtr = cast<GetElementPtrInst>(Builder.CreateGEP(
-          ScalarDataTy, Ptr, Builder.getInt32(-Part * VF.getKnownMinValue())));
+          ScalarDataTy, Ptr, Builder.getInt32(-Part * NumElts)));
       PartPtr->setIsInBounds(InBounds);
       PartPtr = cast<GetElementPtrInst>(Builder.CreateGEP(
-          ScalarDataTy, PartPtr, Builder.getInt32(1 - VF.getKnownMinValue())));
+          ScalarDataTy, PartPtr, Builder.getInt32(1 - NumElts)));
       PartPtr->setIsInBounds(InBounds);
       if (isMaskRequired) // Reverse of a null all-one mask is a null mask.
         BlockInMaskParts[Part] = reverseVector(BlockInMaskParts[Part]);
     } else {
       Value *Increment = createStepForVF(Builder, Builder.getInt32(Part), VF);
       PartPtr = cast<GetElementPtrInst>(
           Builder.CreateGEP(ScalarDataTy, Ptr, Increment));
       PartPtr->setIsInBounds(InBounds);
     }
 
     unsigned AddressSpace = Ptr->getType()->getPointerAddressSpace();
     return Builder.CreateBitCast(PartPtr, DataTy->getPointerTo(AddressSpace));
   };
 
   // Handle Stores:
   if (SI) {
     setDebugLocFromInst(Builder, SI);
 
     for (unsigned Part = 0; Part < UF; ++Part) {
       Instruction *NewSI = nullptr;
       Value *StoredVal = State.get(StoredValue, Part);
       if (CreateGatherScatter) {
         Value *MaskPart = isMaskRequired ? BlockInMaskParts[Part] : nullptr;
         Value *VectorGep = State.get(Addr, Part);
if (canTruncateToMinimalBitwidth(I, VF)) {
         VectorTy =
             largestIntegerVectorType(ToVectorTy(I->getType(), VF), MinVecTy);
       } else if (Opcode == Instruction::ZExt || Opcode == Instruction::SExt) {
         SrcVecTy = largestIntegerVectorType(SrcVecTy, MinVecTy);
         VectorTy =
             smallestIntegerVectorType(ToVectorTy(I->getType(), VF), MinVecTy);
       }
     }
-
-    assert(!VF.isScalable() && "VF is assumed to be non scalable");
     unsigned N = isScalarAfterVectorization(I, VF) ? VF.getKnownMinValue() : 1;
     return N *
            TTI.getCastInstrCost(Opcode, VectorTy, SrcVecTy, CCH, CostKind, I);
   }
   case Instruction::Call: {
     bool NeedToScalarize;
     CallInst *CI = cast<CallInst>(I);
     unsigned CallCost = getVectorCallCost(CI, VF, NeedToScalarize);

llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll

+; RUN: opt -loop-vectorize -dce -instcombine -mtriple aarch64-linux-gnu -S < %s 2>%t | FileCheck %s
+
+; RUN: FileCheck --check-prefix=WARN --allow-empty %s <%t
+
+; This is the loop in c++ being vectorize in this file with
+;experimental.vector.reverse
+;  #pragma clang loop vectorize_width(4, scalable)
+;  for (int i = N-1; i >= 0; --i)
+;    a[i] = b[i] + 1.0;
+
+; If this check fails please read test/CodeGen/AArch64/README for instructions on how to resolve it.
+; WARN-NOT: warning
+
+target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+
+define void @vecReverse_vla(i32 %N, double* nocapture %a, double* nocapture readonly %b) #0{
+; CHECK-LABEL: @vecReverse_vla
+; CHECK: %[[WIDE:.*]] = load <vscale x 8 x double>, <vscale x 8 x double>* %{{.*}}, align 8
+; CHECK-NEXT: %[[REVERSE:.*]] = call <vscale x 8 x double> @llvm.experimental.vector.reverse.nxv8f64(<vscale x 8 x double> %[[WIDE]])
+; CHECK-NEXT:  %[[FADD:.*]] = fadd <vscale x 8 x double> %[[REVERSE]], shufflevector (<vscale x 8 x double> insertelement (<vscale x 8 x double> poison, double 1.000000e+00, i32 0), <vscale x 8 x double> poison, <vscale x 8 x i32> zeroinitializer)
+; CHECK-NEXT:  %{{.*}} = getelementptr inbounds double, double* %{{.*}}, i64 -8
+; CHECK-NEXT:  %[[REVERSE6:.*]] = call <vscale x 8 x double> @llvm.experimental.vector.reverse.nxv8f64(<vscale x 8 x double> %[[FADD]])
+; CHECK-NEXT:  %{{.*}} = getelementptr inbounds double, double* %{{.*}}, i64 %{{.*}}
+; CHECK-NEXT:  %[[CAST:.*]] = bitcast double* %{{.*}} to <vscale x 8 x double>*
+; CHECK-NEXT:  store <vscale x 8 x double> %[[REVERSE6]], <vscale x 8 x double>* %[[CAST]], align 8
+; CHECK-NEXT:  %{{.*}} = call i64 @llvm.vscale.i64()
+
+
+entry:
+  %cmp8 = icmp sgt i32 %N, 0
+  br i1 %cmp8, label %for.body.preheader, label %for.cond.cleanup.loopexit
+
+for.body.preheader:                               ; preds = %entry
+  %0 = zext i32 %N to i64
+  br label %for.body
+
+for.cond.cleanup.loopexit:                        ; preds = %for.body
+  ret void
+
+
+for.body:                                         ; preds = %for.body.preheader, %for.body
+  %indvars.iv = phi i64 [ %0, %for.body.preheader ], [ %indvars.iv.next, %for.body ]
+  %indvars.iv.next = add nsw i64 %indvars.iv, -1
+  %arrayidx = getelementptr inbounds double, double* %b, i64 %indvars.iv.next
+  %1 = load double, double* %arrayidx, align 8
+  %add = fadd double %1, 1.000000e+00
+  %arrayidx2 = getelementptr inbounds double, double* %a, i64 %indvars.iv.next
+  store double %add, double* %arrayidx2, align 8
+  %cmp = icmp sgt i64 %indvars.iv, 1
+  br i1 %cmp, label %for.body, label %for.cond.cleanup.loopexit, !llvm.loop !0
+}
+
+
+; Function Attrs: nofree norecurse nounwind uwtable mustprogress
+define void @vecReverse_i64(i32 %N, i64* nocapture %a, i64* nocapture readonly %b) #0 {
+; CHECK-LABEL: @vecReverse_i64
+; CHECK:  %[[WIDE:.*]] = load <vscale x 8 x i64>, <vscale x 8 x i64>* %{{.*}}, align 8
+; CHECK-NEXT:  %[[REVERSE:.*]] = call <vscale x 8 x i64> @llvm.experimental.vector.reverse.nxv8i64(<vscale x 8 x i64> %[[WIDE]])
+; CHECK-NEXT:  %[[CAST:.*]] = sitofp <vscale x 8 x i64> %[[REVERSE]] to <vscale x 8 x double>
+; CHECK-NEXT:  %{{.*}} = fadd <vscale x 8 x double> %[[CAST]], shufflevector (<vscale x 8 x double> insertelement (<vscale x 8 x double> poison, double 1.000000e+00, i32 0), <vscale x 8 x double> poison, <vscale x 8 x i32> zeroinitializer)
+; CHECK-NEXT:  %[[CAST2:.*]] = fptosi <vscale x 8 x double> %{{.*}} to <vscale x 8 x i64>
+; CHECK-NEXT:  %[[GETPTR:.*]] = getelementptr inbounds i64, i64* %{{.*}}, i64 -8
+; CHECK-NEXT:  %[[REVERSE6:.*]] = call <vscale x 8 x i64> @llvm.experimental.vector.reverse.nxv8i64(<vscale x 8 x i64> %[[CAST2]])
+; CHECK-NEXT:  %{{.*}} = getelementptr inbounds i64, i64* %[[GETPTR]], i64 %{{.*}}
+; CHECK-NEXT:  %[[BITCAST:.*]] = bitcast i64* %{{.*}} to <vscale x 8 x i64>*
+; CHECK-NEXT:  store <vscale x 8 x i64> %[[REVERSE6]], <vscale x 8 x i64>* %[[BITCAST]], align 8
+; CHECK-NEXT:  %{{.*}} = call i64 @llvm.vscale.i64()
+
+entry:
+  %cmp9 = icmp sgt i32 %N, 0
+  br i1 %cmp9, label %for.body.preheader, label %for.cond.cleanup.loopexit
+
+for.body.preheader:                               ; preds = %entry
+  %0 = zext i32 %N to i64
+  br label %for.body
+
+for.cond.cleanup.loopexit:                        ; preds = %for.body
+  ret void
+
+for.body:                                         ; preds = %for.body.preheader, %for.body
+  %indvars.iv = phi i64 [ %0, %for.body.preheader ], [ %indvars.iv.next, %for.body ]
+  %indvars.iv.next = add nsw i64 %indvars.iv, -1
+  %arrayidx = getelementptr inbounds i64, i64* %b, i64 %indvars.iv.next
+  %1 = load i64, i64* %arrayidx, align 8
+  %conv = sitofp i64 %1 to double
+  %add = fadd double %conv, 1.000000e+00
+  %conv1 = fptosi double %add to i64
+  %arrayidx3 = getelementptr inbounds i64, i64* %a, i64 %indvars.iv.next
+  store i64 %conv1, i64* %arrayidx3, align 8
+  %cmp = icmp sgt i64 %indvars.iv, 1
+  br i1 %cmp, label %for.body, label %for.cond.cleanup.loopexit, !llvm.loop !0
+}
+
+attributes #0 = { "target-cpu"="generic" "target-features"="+neon,+sve" }
+
+
+!0 = distinct !{!0, !1, !2, !3, !4}
+!1 = !{!"llvm.loop.mustprogress"}
+!2 = !{!"llvm.loop.vectorize.width", i32 8}
+!3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+
+
+