[AArch64][SVE] Allow to lower WHILEop operations with constant operands to PTRUE

[dtemirbulatov]

Summary

This change allows to lower WHILEop operations with constant operands to PTRUE instructions, it also fixes issue with unsigned integer overflow for WHILELO support already commited in https://reviews.llvm.org/D137547.

[dtemirbulatov]

Unified handling of WHILEop.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

4817–4828

This isn't quite what I had in mind when suggesting you can increase commonality. I had in mind a helper function that takes parameters like IsSigned and IsLess or perhaps even makes use of the existing AArch64CC::CondCode enum?

[dtemirbulatov]

Addressed comments, added signed overflow handling while subtraction/addintion.

[sdesmalen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

4327

nit: Move ElementSize closer to its use.

4329

nit: Can you move these out into separate variables, e.g. X and Y?

4339

Should this Overflow test be moved below the if (IsOpEqualOrSame) { ... } condition?

4811–4950

Is it worth just moving all the behaviour into that function, and passing IsSigned, IsLess and IsEqual as parameters, such that you get:

case Intrinsic::aarch64_sve_whilelo:
  return optimizeWhile(Op.getOperand(1), Op.getOperand(2), /*IsSigned=*/false, /*IsLess=*/true, /*IsEqual=*/false);
case Intrinsic::aarch64_sve_whilels:
  return optimizeWhile(Op.getOperand(1), Op.getOperand(2), /*IsSigned=*/false, /*IsLess=*/true, /*IsEqual=*/true);
case Intrinsic::aarch64_sve_whilelt:
  return optimizeWhile(Op.getOperand(1), Op.getOperand(2), /*IsSigned=*/true, /*IsLess=*/true, /*IsEqual=*/false);
case Intrinsic::aarch64_sve_whilele:
  return optimizeWhile(Op.getOperand(1), Op.getOperand(2), /*IsSigned=*/true, /*IsLess=*/true, /*IsEqual=*/true);
...

[dtemirbulatov]

Addressed remarks, avoided to check overflow while adding one to the range for equal or same conditions.

[sdesmalen]

Thanks that's looking a lot better!

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

4602

nit: add a newline before the if?

4605

I guess this can still overflow right? I think you had the right code for that before, it only needed moving the if (Overflow) return SDValue(); after the increment.

It would also be good to add a test for the overflow case (both signed and unsigned, and both 'less' and 'less or equal')

[dtemirbulatov]

Added tests for signed/unsigned overflow.

[dtemirbulatov]

This comment was removed by dtemirbulatov.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

4605

With the increment case, I don't think we have to worry about overflow/underflow. it either world be 0 or too large/too small number that are not representable with PTRUE VL instruction.

[dtemirbulatov]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

4605

With the increment case, I don't think we have to worry about overflow/underflow. it either world be 0 or too large/too small number that are not representable with PTRUE VL instruction.

[dtemirbulatov]

Added overflow check for the increment.

[sdesmalen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

4605

nit: add newline above.

4609

You didn't add any tests for this case, so it's currently untested.

[sdesmalen]

llvm/test/CodeGen/AArch64/sve-intrinsics-while.ll

87

I guess this doesn't match:

ptrue p0.d, vl4

because NumActiveElems.getZExtValue() <= (MinSVEVectorSize / ElementSize) evaluates to false.

Can you clarify the reason these intrinsics don't fold in the name of these tests (e.g. whilele_d_ii_dont_fold_to_ptrue_larger_than_minvec) and/or add a comment to explain it?

101

Same question here: rename this to whilele_b_ii_dont_fold_to_ptrue_nonexistent_vl9 or something?

121

same question about renaming this test, e.g. whilele_b_ii_dont_fold_to_ptrue_overflow ?

257

For whilelo (unsigned), this is the same as -1, and is therefore not much different than the test above.
I would suggest keeping this test (because it's unsigned) and removing the test above

385

For whilels (unsigned), this is the same as -1, and is therefore not much different than the test above.
I would suggest keeping this test (because it's unsigned) and removing the test above.

[dtemirbulatov]

Addressed suggestions.

[dtemirbulatov]

Missed two comments, addressed those as well.

[sdesmalen]

llvm/test/CodeGen/AArch64/sve2-intrinsics-while.ll

141

Similar comment as for whilegt, it's better to pick some numbers that actually end up as a valid vl if you remove the overflow check, e.g.

@llvm.aarch64.sve.whilege.nxv16i1.i32(i32 -2147483641, i32 2147483647)

245

Similar comment as for whilehi, this doesn't overflow. You could use

@llvm.aarch64.sve.whilehs.nxv16i1.i32(i32 6, i32 4294967295)    ; this would wrap around to vl8

374

It's better to pick some numbers that actually end up as a valid vl if you remove the overflow check, e.g.

@llvm.aarch64.sve.whilegt.nxv16i1.i32(i32 -2147483641, i32 2147483647)

Can you use similar numbers whilege?

449

This doesn't overflow right? 4294967295 - 6 => 4294967289 (no overflow here).

For whilehi, it would overflow if the start value would be lower than the end value, e.g.

@llvm.aarch64.sve.whilehi.nxv16i1.i32(i32 7, i32 4294967295)    ; this would wrap around to vl8

[dtemirbulatov]

Fixed remarks.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (OrigTy.getSizeInBits() >= 64)
     return N;
 
   // Must extend size to at least 64 bits to be used as an operand for VMULL.
   EVT NewVT = getExtensionTo64Bits(OrigTy);
 
   return DAG.getNode(ExtOpcode, SDLoc(N), NewVT, N);
 }
 
 // Returns lane if Op extracts from a two-element vector and lane is constant
 // (i.e., extractelt(<2 x Ty> %v, ConstantLane)), and None otherwise.
 static Optional<uint64_t> getConstantLaneNumOfExtractHalfOperand(SDValue &Op) {
   SDNode *OpNode = Op.getNode();
   if (OpNode->getOpcode() != ISD::EXTRACT_VECTOR_ELT)
     return None;
 
   EVT VT = OpNode->getOperand(0).getValueType();
   ConstantSDNode *C = dyn_cast<ConstantSDNode>(OpNode->getOperand(1));
   if (!VT.isFixedLengthVector() || VT.getVectorNumElements() != 2 || !C)
     return None;
 
   return C->getZExtValue();
 }
 
 static bool isExtendedBUILD_VECTOR(SDNode *N, SelectionDAG &DAG,
                                    bool isSigned) {
   EVT VT = N->getValueType(0);
 
   if (N->getOpcode() != ISD::BUILD_VECTOR)
     return false;
assert(InVT.getVectorElementType() == MVT::i1 &&
          VT.getVectorElementType() == MVT::i1 &&
          "Expected a predicate-to-predicate bitcast");
   assert(VT.isScalableVector() && DAG.getTargetLoweringInfo().isTypeLegal(VT) &&
          InVT.isScalableVector() &&
          DAG.getTargetLoweringInfo().isTypeLegal(InVT) &&
          "Only expect to cast between legal scalable predicate types!");
 
   // Return the operand if the cast isn't changing type,
   // e.g. <n x 16 x i1> -> <n x 16 x i1>
   if (InVT == VT)
     return Op;
 
   SDValue Reinterpret = DAG.getNode(AArch64ISD::REINTERPRET_CAST, DL, VT, Op);
 
   // We only have to zero the lanes if new lanes are being defined, e.g. when
   // casting from <vscale x 2 x i1> to <vscale x 16 x i1>. If this is not the
   // case (e.g. when casting from <vscale x 16 x i1> -> <vscale x 2 x i1>) then
   // we can return here.
   if (InVT.bitsGT(VT))
     return Reinterpret;
 
   // Check if the other lanes are already known to be zeroed by
   // construction.
   if (isZeroingInactiveLanes(Op))
if (Op.getValueType() == MVT::i32)
       return DAG.getNode(ISD::BITCAST, dl, MVT::i32,
                          DAG.getNode(ISD::INTRINSIC_WO_CHAIN, dl, MVT::f32,
                                      Op.getOperand(0),
                                      DAG.getNode(ISD::BITCAST, dl, MVT::f64,
                                                  Op.getOperand(1))));
     return SDValue();
   }
   case Intrinsic::aarch64_sve_whilelo: {
     if (isa<ConstantSDNode>(Op.getOperand(1)) &&
         isa<ConstantSDNode>(Op.getOperand(2))) {
       unsigned MinSVEVectorSize =
           std::max(Subtarget->getMinSVEVectorSizeInBits(), 128u);
       unsigned ElementSize = 128 / Op.getValueType().getVectorMinNumElements();
-      unsigned NumActiveElems =
-          Op.getConstantOperandVal(2) - Op.getConstantOperandVal(1);
+      bool Overflow;
+      APInt NumActiveElems = Op.getConstantOperandAPInt(2).usub_ov(
+          Op.getConstantOperandAPInt(1), Overflow);
       Optional<unsigned> PredPattern =
-          getSVEPredPatternFromNumElements(NumActiveElems);
+          getSVEPredPatternFromNumElements(NumActiveElems.getZExtValue());
+      if ((PredPattern != None) && !Overflow &&
+          NumActiveElems.getZExtValue() <= (MinSVEVectorSize / ElementSize))
+        return getPTrue(DAG, dl, Op.getValueType(), *PredPattern);
+    }
+    return SDValue();
+  }
+  case Intrinsic::aarch64_sve_whilelt: {
+    if (isa<ConstantSDNode>(Op.getOperand(1)) &&
+        isa<ConstantSDNode>(Op.getOperand(2))) {
+      unsigned MinSVEVectorSize =
+          std::max(Subtarget->getMinSVEVectorSizeInBits(), 128u);
+      unsigned ElementSize = 128 / Op.getValueType().getVectorMinNumElements();
+      APInt NumActiveElems =
+          Op.getConstantOperandAPInt(2) - Op.getConstantOperandAPInt(1);
+      Optional<unsigned> PredPattern =
+          getSVEPredPatternFromNumElements(NumActiveElems.getZExtValue());
+      if ((PredPattern != None) &&
+          NumActiveElems.getZExtValue() <= (MinSVEVectorSize / ElementSize))
+        return getPTrue(DAG, dl, Op.getValueType(), *PredPattern);
+    }
+    return SDValue();
+  }
+  case Intrinsic::aarch64_sve_whilels: {
+    if (isa<ConstantSDNode>(Op.getOperand(1)) &&
+        isa<ConstantSDNode>(Op.getOperand(2))) {
+      unsigned MinSVEVectorSize =
+          std::max(Subtarget->getMinSVEVectorSizeInBits(), 128u);
+      unsigned ElementSize = 128 / Op.getValueType().getVectorMinNumElements();
+      bool Overflow;
+      APInt Op1 = Op.getConstantOperandAPInt(1);
+      APInt NumActiveElems =
+          Op.getConstantOperandAPInt(2).usub_ov(Op1, Overflow);
+      if (Overflow)
+        return SDValue();
+      APInt One(NumActiveElems.getBitWidth(), 1, false);
+      NumActiveElems = NumActiveElems.uadd_ov(One, Overflow);
+      Optional<unsigned> PredPattern =
+          getSVEPredPatternFromNumElements(NumActiveElems.getZExtValue());
+      if ((PredPattern != None) && !Overflow &&
+          NumActiveElems.getZExtValue() <= (MinSVEVectorSize / ElementSize))
+        return getPTrue(DAG, dl, Op.getValueType(), *PredPattern);
+    }
+    return SDValue();
+  }
+  case Intrinsic::aarch64_sve_whilele: {
+    if (isa<ConstantSDNode>(Op.getOperand(1)) &&
+        isa<ConstantSDNode>(Op.getOperand(2))) {
+      unsigned MinSVEVectorSize =
+          std::max(Subtarget->getMinSVEVectorSizeInBits(), 128u);
+      unsigned ElementSize = 128 / Op.getValueType().getVectorMinNumElements();
+      APInt NumActiveElems =
+          Op.getConstantOperandAPInt(2) - Op.getConstantOperandAPInt(1) + 1;
+      Optional<unsigned> PredPattern =
+          getSVEPredPatternFromNumElements(NumActiveElems.getZExtValue());
       if ((PredPattern != None) &&
-          NumActiveElems <= (MinSVEVectorSize / ElementSize))
+          NumActiveElems.getZExtValue() <= (MinSVEVectorSize / ElementSize))
+        return getPTrue(DAG, dl, Op.getValueType(), *PredPattern);
+    }
+    return SDValue();
+  }
+  case Intrinsic::aarch64_sve_whilege: {
+    if (isa<ConstantSDNode>(Op.getOperand(1)) &&
+        isa<ConstantSDNode>(Op.getOperand(2))) {
+      unsigned MinSVEVectorSize =
+          std::max(Subtarget->getMinSVEVectorSizeInBits(), 128u);
+      unsigned ElementSize = 128 / Op.getValueType().getVectorMinNumElements();
+      APInt NumActiveElems =
+          Op.getConstantOperandAPInt(1) - Op.getConstantOperandAPInt(2) + 1;
+      Optional<unsigned> PredPattern =
+          getSVEPredPatternFromNumElements(NumActiveElems.getZExtValue());
+      if ((PredPattern != None) &&
+          NumActiveElems.getZExtValue() <= (MinSVEVectorSize / ElementSize))
+        return getPTrue(DAG, dl, Op.getValueType(), *PredPattern);
+    }
+    return SDValue();
+  }
+  case Intrinsic::aarch64_sve_whilegt: {
+    if (isa<ConstantSDNode>(Op.getOperand(1)) &&
+        isa<ConstantSDNode>(Op.getOperand(2))) {
+      unsigned MinSVEVectorSize =
+          std::max(Subtarget->getMinSVEVectorSizeInBits(), 128u);
+      unsigned ElementSize = 128 / Op.getValueType().getVectorMinNumElements();
+      APInt NumActiveElems =
+          Op.getConstantOperandAPInt(1) - Op.getConstantOperandAPInt(2);
+      Optional<unsigned> PredPattern =
+          getSVEPredPatternFromNumElements(NumActiveElems.getZExtValue());
+      if ((PredPattern != None) &&
+          NumActiveElems.getZExtValue() <= (MinSVEVectorSize / ElementSize))
+        return getPTrue(DAG, dl, Op.getValueType(), *PredPattern);
+    }
+    return SDValue();
+  }
+  case Intrinsic::aarch64_sve_whilehs: {
+    if (isa<ConstantSDNode>(Op.getOperand(1)) &&
+        isa<ConstantSDNode>(Op.getOperand(2))) {
+      unsigned MinSVEVectorSize =
+          std::max(Subtarget->getMinSVEVectorSizeInBits(), 128u);
+      unsigned ElementSize = 128 / Op.getValueType().getVectorMinNumElements();
+      bool Overflow;
+      APInt Op2 = Op.getConstantOperandAPInt(2);
+      APInt NumActiveElems =
+          Op.getConstantOperandAPInt(1).usub_ov(Op2, Overflow);
+      if (Overflow)
+        return SDValue();
+      APInt One(NumActiveElems.getBitWidth(), 1, false);
+      NumActiveElems = NumActiveElems.uadd_ov(One, Overflow);
+      Optional<unsigned> PredPattern =
+          getSVEPredPatternFromNumElements(NumActiveElems.getZExtValue());
+      if ((PredPattern != None) && !Overflow &&
+          NumActiveElems.getZExtValue() <= (MinSVEVectorSize / ElementSize))
+        return getPTrue(DAG, dl, Op.getValueType(), *PredPattern);
+    }
+    return SDValue();
+  }
+  case Intrinsic::aarch64_sve_whilehi: {
+    if (isa<ConstantSDNode>(Op.getOperand(1)) &&
+        isa<ConstantSDNode>(Op.getOperand(2))) {
+      unsigned MinSVEVectorSize =
+          std::max(Subtarget->getMinSVEVectorSizeInBits(), 128u);
+      unsigned ElementSize = 128 / Op.getValueType().getVectorMinNumElements();
+      bool Overflow;
+      APInt Op2 = Op.getConstantOperandAPInt(2);
+      APInt NumActiveElems =
+          Op.getConstantOperandAPInt(1).usub_ov(Op2, Overflow);
+      Optional<unsigned> PredPattern =
+          getSVEPredPatternFromNumElements(NumActiveElems.getZExtValue());
+      if ((PredPattern != None) && !Overflow &&
+          NumActiveElems.getZExtValue() <= (MinSVEVectorSize / ElementSize))
         return getPTrue(DAG, dl, Op.getValueType(), *PredPattern);
     }
     return SDValue();
   }
   case Intrinsic::aarch64_sve_sunpkhi:
     return DAG.getNode(AArch64ISD::SUNPKHI, dl, Op.getValueType(),
                        Op.getOperand(1));
   case Intrinsic::aarch64_sve_sunpklo:
     return DAG.getNode(AArch64ISD::SUNPKLO, dl, Op.getValueType(),
                        Op.getOperand(1));

llvm/test/CodeGen/AArch64/sve-intrinsics-while.ll

 ; CHECK-LABEL: whilele_d_xx:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilele p0.d, x0, x1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilele.nxv2i1.i64(i64 %a, i64 %b)
   ret <vscale x 2 x i1> %out
 }
 
+define <vscale x 2 x i1> @whilele_d_ii() {
+; CHECK-LABEL: whilele_d_ii:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, #3
+; CHECK-NEXT:    whilele p0.d, xzr, x8
+; CHECK-NEXT:    ret
+  %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilele.nxv2i1.i64(i64 0, i64 3)
+  ret <vscale x 2 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilele_b_ii1() {
+; CHECK-LABEL: whilele_b_ii1:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.b, vl6
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilele.nxv16i1.i64(i64 -2, i64 3)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilele_b_ii2() {
+; CHECK-LABEL: whilele_b_ii2:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov w8, #9
+; CHECK-NEXT:    whilele p0.b, xzr, x8
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilele.nxv16i1.i64(i64 0, i64 9)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilele_b_ii3() vscale_range(16, 16) {
+; CHECK-LABEL: whilele_b_ii3:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b, vl256
+; CHECK-NEXT:    ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilele.nxv16i1.i64(i64 0, i64 255)
+  ret <vscale x 16 x i1> %out
+}
+
 ;
 ; WHILELO
 ;
 
 define <vscale x 16 x i1> @whilelo_b_ww(i32 %a, i32 %b) {
 ; CHECK-LABEL: whilelo_b_ww:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilelo p0.b, w0, w1
 ; CHECK-NEXT:    ret

 ; CHECK-LABEL: whilelo_d_xx:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilelo p0.d, x0, x1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilelo.nxv2i1.i64(i64 %a, i64 %b)
   ret <vscale x 2 x i1> %out
 }
 
+define <vscale x 2 x i1> @whilelo_d_ii() {
+; CHECK-LABEL: whilelo_d_ii:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, #3
+; CHECK-NEXT:    whilelo p0.d, xzr, x8
+; CHECK-NEXT:    ret
+  %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilelo.nxv2i1.i64(i64 0, i64 3)
+  ret <vscale x 2 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilelo_b_ii1() {
+; CHECK-LABEL: whilelo_b_ii1:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.b, vl6
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilelo.nxv16i1.i64(i64 2, i64 8)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilelo_b_ii2() {
+; CHECK-LABEL: whilelo_b_ii2:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov w8, #9
+; CHECK-NEXT:    whilelo p0.b, xzr, x8
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilelo.nxv16i1.i64(i64 0, i64 9)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilelo_b_ii3() vscale_range(16, 16) {
+; CHECK-LABEL: whilelo_b_ii3:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b, vl256
+; CHECK-NEXT:    ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilelo.nxv16i1.i64(i64 0, i64 256)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilelo_b_ii4() {
+; CHECK-LABEL: whilelo_b_ii4:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov w8, #3
+; CHECK-NEXT:    mov x9, #-2
+; CHECK-NEXT:    whilelo p0.b, x9, x8
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilelo.nxv16i1.i64(i64 -2, i64 3)
+  ret <vscale x 16 x i1> %out
+}
+
 ;
 ; WHILELS
 ;
 
 define <vscale x 16 x i1> @whilels_b_ww(i32 %a, i32 %b) {
 ; CHECK-LABEL: whilels_b_ww:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilels p0.b, w0, w1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilels.nxv16i1.i32(i32 %a, i32 %b)
   ret <vscale x 16 x i1> %out
 }
 
 define <vscale x 16 x i1> @whilels_b_xx(i64 %a, i64 %b) {
 ; CHECK-LABEL: whilels_b_xx:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilels p0.b, x0, x1

 ; CHECK-LABEL: whilels_d_xx:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilels p0.d, x0, x1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilels.nxv2i1.i64(i64 %a, i64 %b)
   ret <vscale x 2 x i1> %out
 }
 
+define <vscale x 2 x i1> @whilels_d_ii() {
+; CHECK-LABEL: whilels_d_ii:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, #3
+; CHECK-NEXT:    whilels p0.d, xzr, x8
+; CHECK-NEXT:    ret
+  %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilels.nxv2i1.i64(i64 0, i64 3)
+  ret <vscale x 2 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilels_b_ii1() {
+; CHECK-LABEL: whilels_b_ii1:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.b, vl7
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilels.nxv16i1.i64(i64 2, i64 8)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilels_b_ii2() {
+; CHECK-LABEL: whilels_b_ii2:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov w8, #9
+; CHECK-NEXT:    whilels p0.b, xzr, x8
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilels.nxv16i1.i64(i64 0, i64 9)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilels_b_ii3() vscale_range(16, 16) {
+; CHECK-LABEL: whilels_b_ii3:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b, vl256
+; CHECK-NEXT:    ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilels.nxv16i1.i64(i64 0, i64 255)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilels_b_ii4() {
+; CHECK-LABEL: whilels_b_ii4:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov w8, #3
+; CHECK-NEXT:    mov x9, #-2
+; CHECK-NEXT:    whilels p0.b, x9, x8
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilels.nxv16i1.i64(i64 -2, i64 3)
+  ret <vscale x 16 x i1> %out
+}
+
 ;
 ; WHILELT
 ;
 
 define <vscale x 16 x i1> @whilelt_b_ww(i32 %a, i32 %b) {
 ; CHECK-LABEL: whilelt_b_ww:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilelt p0.b, w0, w1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilelt.nxv16i1.i32(i32 %a, i32 %b)
   ret <vscale x 16 x i1> %out
 }
 
 define <vscale x 16 x i1> @whilelt_b_xx(i64 %a, i64 %b) {
 ; CHECK-LABEL: whilelt_b_xx:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilelt p0.b, x0, x1

 ; CHECK-LABEL: whilelt_d_xx:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilelt p0.d, x0, x1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilelt.nxv2i1.i64(i64 %a, i64 %b)
   ret <vscale x 2 x i1> %out
 }
 
+define <vscale x 2 x i1> @whilelt_d_ii() {
+; CHECK-LABEL: whilelt_d_ii:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, #3
+; CHECK-NEXT:    whilelt p0.d, xzr, x8
+; CHECK-NEXT:    ret
+  %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilelt.nxv2i1.i64(i64 0, i64 3)
+  ret <vscale x 2 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilelt_b_ii1() {
+; CHECK-LABEL: whilelt_b_ii1:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.b, vl5
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilelt.nxv16i1.i64(i64 -2, i64 3)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilelt_b_ii2() {
+; CHECK-LABEL: whilelt_b_ii2:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov w8, #9
+; CHECK-NEXT:    whilelt p0.b, xzr, x8
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilelt.nxv16i1.i64(i64 0, i64 9)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilelt_b_ii3() vscale_range(16, 16) {
+; CHECK-LABEL: whilelt_b_ii3:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b, vl256
+; CHECK-NEXT:    ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilelt.nxv16i1.i64(i64 0, i64 256)
+  ret <vscale x 16 x i1> %out
+}
+
 declare <vscale x 16 x i1> @llvm.aarch64.sve.whilele.nxv16i1.i32(i32, i32)
 declare <vscale x 16 x i1> @llvm.aarch64.sve.whilele.nxv16i1.i64(i64, i64)
 declare <vscale x 8 x i1> @llvm.aarch64.sve.whilele.nxv8i1.i32(i32, i32)
 declare <vscale x 8 x i1> @llvm.aarch64.sve.whilele.nxv8i1.i64(i64, i64)
 declare <vscale x 4 x i1> @llvm.aarch64.sve.whilele.nxv4i1.i32(i32, i32)
 declare <vscale x 4 x i1> @llvm.aarch64.sve.whilele.nxv4i1.i64(i64, i64)
 declare <vscale x 2 x i1> @llvm.aarch64.sve.whilele.nxv2i1.i32(i32, i32)
 declare <vscale x 2 x i1> @llvm.aarch64.sve.whilele.nxv2i1.i64(i64, i64)

llvm/test/CodeGen/AArch64/sve2-intrinsics-while.ll

 ; CHECK-LABEL: whilege_d_xx:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilege p0.d, x0, x1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilege.nxv2i1.i64(i64 %a, i64 %b)
   ret <vscale x 2 x i1> %out
 }
 
+define <vscale x 2 x i1> @whilege_d_ii() {
+; CHECK-LABEL: whilege_d_ii:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, #3
+; CHECK-NEXT:    whilege p0.d, x8, xzr
+; CHECK-NEXT:    ret
+  %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilege.nxv2i1.i64(i64 3, i64 0)
+  ret <vscale x 2 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilege_b_ii1() {
+; CHECK-LABEL: whilege_b_ii1:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.b, vl6
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilege.nxv16i1.i32(i32 3, i32 -2)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilege_b_ii2() {
+; CHECK-LABEL: whilege_b_ii2:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov w8, #9
+; CHECK-NEXT:    whilege p0.b, x8, xzr
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilege.nxv16i1.i64(i64 9, i64 0)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilege_b_ii3() vscale_range(16, 16) {
+; CHECK-LABEL: whilege_b_ii3:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b, vl256
+; CHECK-NEXT:    ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilege.nxv16i1.i64(i64 255, i64 0)
+  ret <vscale x 16 x i1> %out
+}
+
 ;
 ; WHILEHS
 ;
 
 define <vscale x 16 x i1> @whilehs_b_ww(i32 %a, i32 %b) {
 ; CHECK-LABEL: whilehs_b_ww:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilehs p0.b, w0, w1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilehs.nxv16i1.i32(i32 %a, i32 %b)
   ret <vscale x 16 x i1> %out
 }
 
 define <vscale x 16 x i1> @whilehs_b_xx(i64 %a, i64 %b) {
 ; CHECK-LABEL: whilehs_b_xx:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilehs p0.b, x0, x1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilehs.nxv16i1.i64(i64 %a, i64 %b)
   ret <vscale x 16 x i1> %out
 }
 
 define <vscale x 8 x i1> @whilehs_h_ww(i32 %a, i32 %b) {
 ; CHECK-LABEL: whilehs_h_ww:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilehs p0.h, w0, w1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 8 x i1> @llvm.aarch64.sve.whilehs.nxv8i1.i32(i32 %a, i32 %b)
   ret <vscale x 8 x i1> %out

 ; CHECK-LABEL: whilehs_d_xx:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilehs p0.d, x0, x1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilehs.nxv2i1.i64(i64 %a, i64 %b)
   ret <vscale x 2 x i1> %out
 }
 
+define <vscale x 2 x i1> @whilehs_d_ii() {
+; CHECK-LABEL: whilehs_d_ii:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, #3
+; CHECK-NEXT:    whilehs p0.d, x8, xzr
+; CHECK-NEXT:    ret
+  %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilehs.nxv2i1.i64(i64 3, i64 0)
+  ret <vscale x 2 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilehs_b_ii1() {
+; CHECK-LABEL: whilehs_b_ii1:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.b, vl7
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilehs.nxv16i1.i64(i64 8, i64 2)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilehs_b_ii2() {
+; CHECK-LABEL: whilehs_b_ii2:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov w8, #9
+; CHECK-NEXT:    whilehs p0.b, x8, xzr
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilehs.nxv16i1.i64(i64 9, i64 0)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilehs_b_ii3() vscale_range(16, 16) {
+; CHECK-LABEL: whilehs_b_ii3:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b, vl256
+; CHECK-NEXT:    ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilehs.nxv16i1.i64(i64 255, i64 0)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilehs_b_ii4() {
+; CHECK-LABEL: whilehs_b_ii4:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov w8, #3
+; CHECK-NEXT:    mov x9, #-2
+; CHECK-NEXT:    whilehs p0.b, x9, x8
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilehs.nxv16i1.i64(i64 -2, i64 3)
+  ret <vscale x 16 x i1> %out
+}
+
 ;
 ; WHILEGT
 ;
 
 define <vscale x 16 x i1> @whilegt_b_ww(i32 %a, i32 %b) {
 ; CHECK-LABEL: whilegt_b_ww:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilegt p0.b, w0, w1

 ; CHECK-LABEL: whilehi_s_ww:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilehi p0.s, w0, w1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 4 x i1> @llvm.aarch64.sve.whilehi.nxv4i1.i32(i32 %a, i32 %b)
   ret <vscale x 4 x i1> %out
 }
 
 define <vscale x 4 x i1> @whilehi_s_xx(i64 %a, i64 %b) {
 ; CHECK-LABEL: whilehi_s_xx:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilehi p0.s, x0, x1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 4 x i1> @llvm.aarch64.sve.whilehi.nxv4i1.i64(i64 %a, i64 %b)
   ret <vscale x 4 x i1> %out
 }
 

 ; CHECK-LABEL: whilehi_d_xx:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    whilehi p0.d, x0, x1
 ; CHECK-NEXT:    ret
   %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilehi.nxv2i1.i64(i64 %a, i64 %b)
   ret <vscale x 2 x i1> %out
 }
 
+define <vscale x 2 x i1> @whilehi_d_ii() {
+; CHECK-LABEL: whilehi_d_ii:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, #3
+; CHECK-NEXT:    whilehi p0.d, x8, xzr
+; CHECK-NEXT:    ret
+  %out = call <vscale x 2 x i1> @llvm.aarch64.sve.whilehi.nxv2i1.i64(i64 3, i64 0)
+  ret <vscale x 2 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilehi_b_ii1() {
+; CHECK-LABEL: whilehi_b_ii1:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ptrue p0.b, vl6
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilehi.nxv16i1.i64(i64 8, i64 2)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilehi_b_ii2() {
+; CHECK-LABEL: whilehi_b_ii2:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov w8, #9
+; CHECK-NEXT:    whilehi p0.b, x8, xzr
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilehi.nxv16i1.i64(i64 9, i64 0)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilehi_b_ii3() vscale_range(16, 16) {
+; CHECK-LABEL: whilehi_b_ii3:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b, vl256
+; CHECK-NEXT:    ret
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilehi.nxv16i1.i64(i64 256, i64 0)
+  ret <vscale x 16 x i1> %out
+}
+
+define <vscale x 16 x i1> @whilehi_b_ii4() {
+; CHECK-LABEL: whilehi_b_ii4:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    mov w8, #3
+; CHECK-NEXT:    mov x9, #-2
+; CHECK-NEXT:    whilehi p0.b, x9, x8
+; CHECK-NEXT:    ret
+entry:
+  %out = call <vscale x 16 x i1> @llvm.aarch64.sve.whilehi.nxv16i1.i64(i64 -2, i64 3)
+  ret <vscale x 16 x i1> %out
+}
+
 declare <vscale x 16 x i1> @llvm.aarch64.sve.whilege.nxv16i1.i32(i32, i32)
 declare <vscale x 16 x i1> @llvm.aarch64.sve.whilege.nxv16i1.i64(i64, i64)
 declare <vscale x 8 x i1> @llvm.aarch64.sve.whilege.nxv8i1.i32(i32, i32)
 declare <vscale x 8 x i1> @llvm.aarch64.sve.whilege.nxv8i1.i64(i64, i64)
 declare <vscale x 4 x i1> @llvm.aarch64.sve.whilege.nxv4i1.i32(i32, i32)
 declare <vscale x 4 x i1> @llvm.aarch64.sve.whilege.nxv4i1.i64(i64, i64)
 declare <vscale x 2 x i1> @llvm.aarch64.sve.whilege.nxv2i1.i32(i32, i32)
 declare <vscale x 2 x i1> @llvm.aarch64.sve.whilege.nxv2i1.i64(i64, i64)