[DAG] Enable scalable vectors handling in computeKnownBits

[dmgreen]

Summary

The computeKnownBits functions do not currently handle known bits of scalable vectors, as the meaning of DemandedElts is unclear for a vector that is scalable. This changes that so that the majority of knownBits is run, with the check for scalable vectors being pushed into the individual cases that a not trivially lane-wise independent. The DemandedElts size is set to getVectorMinNumElements and asserted to be all-ones, although this patch doesn't make a strong opinion as to what that means, exactly. Nothing currently makes use of it in a way that varies between elements.

In order to keep the tests working this also:

• Adds known bits for ISD::SPLAT_VECTOR, which is the same bits as the input element (possibly truncated).
• Adds some simple known bits for ISD::STEP_VECTOR, which for power-two increments we know lower bit zeroes.
• The patterns for INDEX were made to work with adds or add-like-ors, to keep them working as intended.

[RKSimon]

SGTM but I'll defer to people who are actively working with scalable vectors whether the getVectorMinNumElements approach will work.

I had been wondering whether we're going have to move from APInt to some kind of 'DemandedMask' wrapper class to handle everything,

llvm/lib/Target/AArch64/AArch64InstrFormats.td

61 ↗

(On Diff #438268)

Just move this as a pre-commit NFC?

[paulwalker-arm]

I don't like the path this patch is taking. In it current form it is changing what the current interface means without documenting such, nor paying much heed to what all the code paths might do. This is why during initial scalable vector bring up, we decided to go the safest route and essentially bail for scalable vectors. Sure this means we implicitly changed the meaning of the "all known" value but there was little else we could do.

It was discussed during the SVE sync calls we ran a while back and whilst no firm decision was made it was pretty clear that KnownBits will need to undergo the same transition as TypeSize and ElementCount. I don't think this is something we can nor should short cut.

[dmgreen]

For now this shores up the meaning of DemandedElts a little - making it always getVectorMinNumElements and assert that it is always all-ones.

I have versions of this patch for ComputeNumSignBits and SimplifyDemandedBits (and some patches on top), but have not sent them for review until we can work through how DemandedElts would work for scalable vectors.

[dmgreen]

In D128159#3596025, @paulwalker-arm wrote:

I don't like the path this patch is taking. In it current form it is changing what the current interface means without documenting such, nor paying much heed to what all the code paths might do. This is why during initial scalable vector bring up, we decided to go the safest route and essentially bail for scalable vectors. Sure this means we implicitly changed the meaning of the "all known" value but there was little else we could do.

It was discussed during the SVE sync calls we ran a while back and whilst no firm decision was made it was pretty clear that KnownBits will need to undergo the same transition as TypeSize and ElementCount. I don't think this is something we can nor should short cut.

I'm not sure that's entirely necessary in order to get a lot of good improvement out of computeKnownBits/computeSignBits/SimplifyDemandedBits. A large part of the optimizations they perform are identical for all lanes, and with scalable vectors that is much more common than demanded elt's that vary by lane. This would at least be a nice first step forward without trying to solve everything all at once.

What kind of optimizations would you hope to be solved with scalable demanded elts? That might help define what kind of interface it would need.

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

2906–2907

should we assert here?

assert((!Op.getValueType().isScalableVector() || DemandedElts.isAllOnes()) && "All scalable vector elements must be demanded");

[david-arm]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

3462

I'm surprised we need so few bail-outs for all the opcodes for scalable vectors? For example, ISD::INTRINSIC_VOID, etc. call into target functions that may not yet deal with this properly. @paulwalker-arm is right that there was discussion about this during SVE sync calls with @efriedma and Chris Tetreault and I remember distinctly that we were steered away (for what seemed like sensible reasons) from this approach. It's certainly worth having this discussion again now though. I think part of the problem is that you have to make sure every existing, and new, opcode takes sensible action for scalable vectors. Without a new interface to make it clear that DemandedElts now work with scalable vectors there is possibly a danger of introducing subtle bugs without realising it. At the very least it might be safer to explicitly call out all supported opcodes at the start of the function and bail out for any supported ones, before even getting to the switch statement?

[dmgreen]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

3462

When I looked through all the opcodes I had not see anything that would cause problems, including into the AArch64 and RISCV computeKnownBitsForTargetNode methods. Did you have anything in particular that you think would be a problem?

Most interesting operations are lane-wise-identical, just passing DemandedElts through unchanged to other calls. That is the part that gives us a lot of benefit for SVE.

[david-arm]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

3462

It's more that in general not teaching users that DemandedElts for this function now works for scalable vectors could lead to the silent introduction of bugs. For example, does it only refer to the first known minimum number of elements, or does it refer to a repeating pattern of elements for each chunk? I think we should define explicitly what this means.

Also, here you're only changing the meaning of DemandedElts for one function, but there are lots of functions scattered throughout codegen and IR that have similar interfaces that take a DemandedElts. So there is now an inconsistency and is potentially confusing for users as to what it means. A quick grep of llvm/include/llvm showed around 65 declared functions that take a parameter of exactly the same name.

SelectionDAG::computeKnownBits is called by SelectionDAG::MaskedValueIsZero, SelectionDAG::MaskedVectorIsZero, TargetLowering::SimplifyMultipleUseDemandedBits, etc, which also take a DemandedElts parameter and pass it straight on to computeKnownBits. However, the comments on those interfaces haven't been updated to explain they now work for scalable vectors. Rather than go to the trouble of updating the comments on all those functions, I personally think it's easier to just create a new simple wrapper class to represent demanded elements in a similar way to what we did for ElementCount and TypeSize.

[dmgreen]

Sorry for the delay - I became busy with some other things. I think I might put up the other patches I have so we can all take a look, as I don't see any huge reasons why not to go with this. We (for the moment) define DemandedElts for scalable vectors that all lanes must be demanded, which we represent as a DemandElts mask of VectorMinNumElements with all ones, and have asserts to make sure that is always true. It is simple, efficient, and gets us a lot of the benefits of KnownBits/SimplifyDemandedBits.

It's more that in general not teaching users that DemandedElts for this function now works for scalable vectors could lead to the silent introduction of bugs. For example, does it only refer to the first known minimum number of elements, or does it refer to a repeating pattern of elements for each chunk? I think we should define explicitly what this means.

It is defined to require that all lanes of the scalable vector are demanded.

Also, here you're only changing the meaning of DemandedElts for one function, but there are lots of functions scattered throughout codegen and IR that have similar interfaces that take a DemandedElts. So there is now an inconsistency and is potentially confusing for users as to what it means. A quick grep of llvm/include/llvm showed around 65 declared functions that take a parameter of exactly the same name.

Like I said, I have patches for ComputeNumSignBits and SimplifyDemandedBits. This covers almost all of the uses of DemandedElts inside SDAG. There are other IR level functions, but they are separate.

SelectionDAG::computeKnownBits is called by SelectionDAG::MaskedValueIsZero, SelectionDAG::MaskedVectorIsZero, TargetLowering::SimplifyMultipleUseDemandedBits, etc, which also take a DemandedElts parameter and pass it straight on to computeKnownBits. However, the comments on those interfaces haven't been updated to explain they now work for scalable vectors.

MaskedVectorIsZero and MaskedValueIsZero with DemandedElts are only actually called from the X86 backend. The others I have patches for, and haven't found any problems with them.

Rather than go to the trouble of updating the comments on all those functions, I personally think it's easier to just create a new simple wrapper class to represent demanded elements in a similar way to what we did for ElementCount and TypeSize.

A DemandedElts will not really be like ElementCount or TypeSize. It is not one thing (a count or type) optionally made scalable. It is a bitmask for each lane in a possibly effectively-infinite vector. It can't just be specified with a Bitmask that gets repeated. As far as I can tell, adding that might only be useful for BITCAST.

The ZERO/SIGN_EXTEND_VECTOR_INREG don't seem to be used for scalable vectors. Which really leaves EXTRACT_SUBVECTOR, EXTRACT_VECTOR_ELT and INSERT_SUBVECTOR/INSERT_VECTOR_ELEMENT, which are specific to the lane. They would probably need to be represented as "all lanes are 0 except these" or "all lanes are 1 except these", which are both useful for different places.

So I think DemandedElts would need to be a repeated pattern with some way to represent lanes that deviate from it, that needs to grow as far as needed for any constant lane indices. Plus all the logic needed to make the anding and oring and extracting work, but I've not looked into seeing how to implement them yet. I'm not against making something like that, but it needs to be added for a reason. Just handling BITCAST and the EXTRACT/INSERTS from scalable vectors doesn't feel like a huge reason to me compared to what else these functions provide.

[dmgreen]

A little bit of rewording.

[dmgreen]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

2906–2907

I was keeping the assert with the existing one for the length of DemandedElts. I can move it or both of them if you think that's best. (But I don't think that ConstantSDNode/ConstantFPSDNode can be scalable, and the MaxRecursionDepth didn't seem important for whether we assert or not). Let me know what you think. Happy to move it.

[paulwalker-arm]

Reading through the comments, has anything changed since my first comment? The main response seems to be "but hay it works". I'm sure that's true, but we've had several such moments when adding scalable support and each time we've been told to implement things properly. Hence the new vector type, element count, type size and instruction cost. I'm not hearing good reasons why demanded elements is any different.

To call out specific cases I'm worried about:

1. Target's that implement fixed length operations via the scalable types. It seems reasonable to be able to track the real elements within their container.
2. Ensuring operations on <vscale x 1 x types do not get misrepresented as being scalar like.

Whilst I'm sure it's possible to show these changes function correctly, it's my belief that such validation is based on either today's limited implementation or plain luck, rather than the design being solid.

Perhaps others have a different view? To be honest this is an area that I've largely ignored so I'll need to dig deeper but I just don't see a safe implementation without either a new type or new set of functions that are designed specifically for the limited use case you want to handle.

[reames]

FYI, D137140 so this patch is now mostly redundant.

I went back to look at the two parts I left out. From the original commit message that was:
"Adds some simple known bits for ISD::STEP_VECTOR, which for power-two increments we know lower bit zeroes."
"The patterns for INDEX were made to work with adds or add-like-ors, to keep them working as intended."

I applied both of these changes locally, and saw no test difference. Either individually or together. Given that, I'm not going to push those further until we have a motivating example or a test in tree. Feel free to split into it's own patch - with appropriate test - if desired. I'd be happy to review.

[dmgreen]

OK. There might be more to do, but that can be done in other reviews. Closing.

llvm/include/llvm/CodeGen/SelectionDAG.h

public:
   /// them in Known. For vectors, the known bits are those that are shared by
   /// every vector element.
   /// Targets can implement the computeKnownBitsForTargetNode method in the
   /// TargetLowering class to allow target nodes to be understood.
   KnownBits computeKnownBits(SDValue Op, unsigned Depth = 0) const;
 
   /// Determine which bits of Op are known to be either zero or one and return
   /// them in Known. The DemandedElts argument allows us to only collect the
-  /// known bits that are shared by the requested vector elements.
-  /// Targets can implement the computeKnownBitsForTargetNode method in the
-  /// TargetLowering class to allow target nodes to be understood.
+  /// known bits that are shared by the requested vector elements. For scalable
+  /// vectors the DemandedElts must be getVectorMinNumElements in size and all
+  /// lanes must be demanded. Targets can implement the
+  /// computeKnownBitsForTargetNode method in the TargetLowering class to allow
+  /// target nodes to be understood.
   KnownBits computeKnownBits(SDValue Op, const APInt &DemandedElts,
                              unsigned Depth = 0) const;
 
   /// Used to represent the possible overflow behavior of an operation.
   /// Never: the operation cannot overflow.
   /// Always: the operation will always overflow.
   /// Sometime: the operation may or may not overflow.
   enum OverflowKind {

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

 }
 
 /// Determine which bits of Op are known to be either zero or one and return
 /// them in Known. For vectors, the known bits are those that are shared by
 /// every vector element.
 KnownBits SelectionDAG::computeKnownBits(SDValue Op, unsigned Depth) const {
   EVT VT = Op.getValueType();
 
-  // TOOD: Until we have a plan for how to represent demanded elements for
-  // scalable vectors, we can just bail out for now.
-  if (Op.getValueType().isScalableVector()) {
-    unsigned BitWidth = Op.getScalarValueSizeInBits();
-    return KnownBits(BitWidth);
-  }
-
   APInt DemandedElts = VT.isVector()
-                           ? APInt::getAllOnes(VT.getVectorNumElements())
+                           ? APInt::getAllOnes(VT.getVectorMinNumElements())
                            : APInt(1, 1);
   return computeKnownBits(Op, DemandedElts, Depth);
 }
 
 /// Determine which bits of Op are known to be either zero or one and return
 /// them in Known. The DemandedElts argument allows us to only collect the known
 /// bits that are shared by the requested vector elements.
 KnownBits SelectionDAG::computeKnownBits(SDValue Op, const APInt &DemandedElts,
                                          unsigned Depth) const {
   unsigned BitWidth = Op.getScalarValueSizeInBits();
 
   KnownBits Known(BitWidth);   // Don't know anything.
 
-  // TOOD: Until we have a plan for how to represent demanded elements for
-  // scalable vectors, we can just bail out for now.
-  if (Op.getValueType().isScalableVector())
-    return Known;
-
   if (auto *C = dyn_cast<ConstantSDNode>(Op)) {
     // We know all of the bits for a constant!
     return KnownBits::makeConstant(C->getAPIntValue());
   }
   if (auto *C = dyn_cast<ConstantFPSDNode>(Op)) {
     // We know all of the bits for a constant fp!
     return KnownBits::makeConstant(C->getValueAPF().bitcastToAPInt());
   }
 
   if (Depth >= MaxRecursionDepth)
     return Known;  // Limit search depth.
 
   KnownBits Known2;
   unsigned NumElts = DemandedElts.getBitWidth();
   assert((!Op.getValueType().isVector() ||
-          NumElts == Op.getValueType().getVectorNumElements()) &&
+          NumElts == Op.getValueType().getVectorMinNumElements()) &&
          "Unexpected vector size");
+  assert((!Op.getValueType().isScalableVector() || DemandedElts.isAllOnes()) &&
+         "Expected all demanded lanes from scalable vectors");
 
   if (!DemandedElts)
     return Known;  // No demanded elts, better to assume we don't know anything.
 
   unsigned Opcode = Op.getOpcode();
   switch (Opcode) {
   case ISD::BUILD_VECTOR:
     // Collect the known bits that are shared by every demanded vector element.
for (unsigned i = 0, e = Op.getNumOperands(); i != e; ++i) {
       // Known bits are the values that are shared by every demanded element.
       Known = KnownBits::commonBits(Known, Known2);
 
       // If we don't know any bits, early out.
       if (Known.isUnknown())
         break;
     }
     break;
+  case ISD::SPLAT_VECTOR: {
+    SDValue SrcOp = Op.getOperand(0);
+    Known = computeKnownBits(SrcOp, Depth + 1);
+    if (SrcOp.getValueSizeInBits() != BitWidth) {
+      assert(SrcOp.getValueSizeInBits() > BitWidth &&
+             "Expected SPLAT_VECTOR implicit truncation");
+      Known = Known.trunc(BitWidth);
+    }
+    break;
+  }
+  case ISD::STEP_VECTOR: {
+    const APInt &Step = Op.getConstantOperandAPInt(0);
+    if (Step.isPowerOf2())
+      Known.Zero.setLowBits(Step.logBase2());
+    break;
+  }
   case ISD::VECTOR_SHUFFLE: {
     // Collect the known bits that are shared by every vector element referenced
     // by the shuffle.
     APInt DemandedLHS(NumElts, 0), DemandedRHS(NumElts, 0);
     Known.Zero.setAllBits(); Known.One.setAllBits();
     const ShuffleVectorSDNode *SVN = cast<ShuffleVectorSDNode>(Op);
     assert(NumElts == SVN->getMask().size() && "Unexpected vector size");
     for (unsigned i = 0; i != NumElts; ++i) {
case ISD::VECTOR_SHUFFLE: {
     if (!!DemandedRHS) {
       SDValue RHS = Op.getOperand(1);
       Known2 = computeKnownBits(RHS, DemandedRHS, Depth + 1);
       Known = KnownBits::commonBits(Known, Known2);
     }
     break;
   }
   case ISD::CONCAT_VECTORS: {
+    if (Op.getValueType().isScalableVector())
+      return Known;
+
     // Split DemandedElts and test each of the demanded subvectors.
     Known.Zero.setAllBits(); Known.One.setAllBits();
     EVT SubVectorVT = Op.getOperand(0).getValueType();
     unsigned NumSubVectorElts = SubVectorVT.getVectorNumElements();
     unsigned NumSubVectors = Op.getNumOperands();
     for (unsigned i = 0; i != NumSubVectors; ++i) {
       APInt DemandedSub =
           DemandedElts.extractBits(NumSubVectorElts, i * NumSubVectorElts);
       if (!!DemandedSub) {
         SDValue Sub = Op.getOperand(i);
         Known2 = computeKnownBits(Sub, DemandedSub, Depth + 1);
         Known = KnownBits::commonBits(Known, Known2);
       }
       // If we don't know any bits, early out.
       if (Known.isUnknown())
         break;
     }
     break;
   }
   case ISD::INSERT_SUBVECTOR: {
+    if (Op.getValueType().isScalableVector())
+      return Known;
+
     // Demand any elements from the subvector and the remainder from the src its
     // inserted into.
     SDValue Src = Op.getOperand(0);
     SDValue Sub = Op.getOperand(1);
     uint64_t Idx = Op.getConstantOperandVal(2);
     unsigned NumSubElts = Sub.getValueType().getVectorNumElements();
     APInt DemandedSubElts = DemandedElts.extractBits(NumSubElts, Idx);
     APInt DemandedSrcElts = DemandedElts;
     DemandedSrcElts.insertBits(APInt::getZero(NumSubElts), Idx);
 
     Known.One.setAllBits();
     Known.Zero.setAllBits();
     if (!!DemandedSubElts) {
       Known = computeKnownBits(Sub, DemandedSubElts, Depth + 1);
       if (Known.isUnknown())
         break; // early-out.
     }
     if (!!DemandedSrcElts) {
       Known2 = computeKnownBits(Src, DemandedSrcElts, Depth + 1);
       Known = KnownBits::commonBits(Known, Known2);
     }
     break;
   }
   case ISD::EXTRACT_SUBVECTOR: {
+    if (Op.getValueType().isScalableVector())
+      return Known;
+
     // Offset the demanded elts by the subvector index.
     SDValue Src = Op.getOperand(0);
     // Bail until we can represent demanded elements for scalable vectors.
     if (Src.getValueType().isScalableVector())
       break;
     uint64_t Idx = Op.getConstantOperandVal(1);
     unsigned NumSrcElts = Src.getValueType().getVectorNumElements();
     APInt DemandedSrcElts = DemandedElts.zext(NumSrcElts).shl(Idx);
     Known = computeKnownBits(Src, DemandedSrcElts, Depth + 1);
     break;
   }
   case ISD::SCALAR_TO_VECTOR: {
+    if (Op.getValueType().isScalableVector())
+      return Known;
+
     // We know about scalar_to_vector as much as we know about it source,
     // which becomes the first element of otherwise unknown vector.
     if (DemandedElts != 1)
       break;
 
     SDValue N0 = Op.getOperand(0);
     Known = computeKnownBits(N0, Depth + 1);
     if (N0.getValueSizeInBits() != BitWidth)
       Known = Known.trunc(BitWidth);
 
     break;
   }
   case ISD::BITCAST: {
+    if (Op.getValueType().isScalableVector())
+      return Known;
+
     SDValue N0 = Op.getOperand(0);
     EVT SubVT = N0.getValueType();
     unsigned SubBitWidth = SubVT.getScalarSizeInBits();
 
     // Ignore bitcasts from unsupported types.
     if (!(SubVT.isInteger() || SubVT.isFloatingPoint()))
       break;
 
if (ISD::isNON_EXTLoad(LD) && Cst) {
       Known.Zero.setBitsFrom(MemBits);
     } else if (const MDNode *Ranges = LD->getRanges()) {
       if (LD->getExtensionType() == ISD::NON_EXTLOAD)
         computeKnownBitsFromRangeMetadata(*Ranges, Known);
     }
     break;
   }
   case ISD::ZERO_EXTEND_VECTOR_INREG: {
+    if (Op.getValueType().isScalableVector())
+      return Known;
+
     EVT InVT = Op.getOperand(0).getValueType();
     APInt InDemandedElts = DemandedElts.zext(InVT.getVectorNumElements());
     Known = computeKnownBits(Op.getOperand(0), InDemandedElts, Depth + 1);
     Known = Known.zext(BitWidth);
     break;
   }
   case ISD::ZERO_EXTEND: {
     Known = computeKnownBits(Op.getOperand(0), DemandedElts, Depth + 1);
     Known = Known.zext(BitWidth);
     break;
   }
   case ISD::SIGN_EXTEND_VECTOR_INREG: {
+    if (Op.getValueType().isScalableVector())
+      return Known;
+
     EVT InVT = Op.getOperand(0).getValueType();
     APInt InDemandedElts = DemandedElts.zext(InVT.getVectorNumElements());
     Known = computeKnownBits(Op.getOperand(0), InDemandedElts, Depth + 1);
     // If the sign bit is known to be zero or one, then sext will extend
     // it to the top bits, else it will just zext.
     Known = Known.sext(BitWidth);
     break;
   }
   case ISD::SIGN_EXTEND: {
     Known = computeKnownBits(Op.getOperand(0), DemandedElts, Depth + 1);
     // If the sign bit is known to be zero or one, then sext will extend
     // it to the top bits, else it will just zext.
     Known = Known.sext(BitWidth);
     break;
   }
   case ISD::ANY_EXTEND_VECTOR_INREG: {
+    if (Op.getValueType().isScalableVector())
+      return Known;
+
     EVT InVT = Op.getOperand(0).getValueType();
     APInt InDemandedElts = DemandedElts.zext(InVT.getVectorNumElements());
     Known = computeKnownBits(Op.getOperand(0), InDemandedElts, Depth + 1);
     Known = Known.anyext(BitWidth);
     break;
   }
   case ISD::ANY_EXTEND: {
     Known = computeKnownBits(Op.getOperand(0), DemandedElts, Depth + 1);
if (ConstEltNo && ConstEltNo->getAPIntValue().ult(NumSrcElts))
           APInt::getOneBitSet(NumSrcElts, ConstEltNo->getZExtValue());
 
     Known = computeKnownBits(InVec, DemandedSrcElts, Depth + 1);
     if (BitWidth > EltBitWidth)
       Known = Known.anyext(BitWidth);
     break;
   }
   case ISD::INSERT_VECTOR_ELT: {
+    if (Op.getValueType().isScalableVector())
+      return Known;
+
     // If we know the element index, split the demand between the
     // source vector and the inserted element, otherwise assume we need
     // the original demanded vector elements and the value.
     SDValue InVec = Op.getOperand(0);
     SDValue InVal = Op.getOperand(1);
     SDValue EltNo = Op.getOperand(2);
     bool DemandedVal = true;
     APInt DemandedVecElts = DemandedElts;

llvm/lib/Target/AArch64/SVEInstrFormats.td

multiclass sve_int_index_ii<string asm> {
   def : Pat<(nxv8i16 (step_vector simm5_16b_tgt:$imm5b)),
             (!cast<Instruction>(NAME # "_H") (i32 0), (!cast<SDNodeXForm>("trunc_imm") $imm5b))>;
   def : Pat<(nxv4i32 (step_vector simm5_32b_tgt:$imm5b)),
             (!cast<Instruction>(NAME # "_S") (i32 0), simm5_32b:$imm5b)>;
   def : Pat<(nxv2i64 (step_vector simm5_64b_tgt:$imm5b)),
             (!cast<Instruction>(NAME # "_D") (i64 0), simm5_64b:$imm5b)>;
 
   // add(step_vector(step), dup(X)) -> index(X, step).
-  def : Pat<(add (nxv16i8 (step_vector_oneuse simm5_8b_tgt:$imm5b)), (nxv16i8 (splat_vector(simm5_8b:$imm5)))),
+  def : Pat<(add_and_or_is_add (nxv16i8 (step_vector_oneuse simm5_8b_tgt:$imm5b)), (nxv16i8 (splat_vector(simm5_8b:$imm5)))),
             (!cast<Instruction>(NAME # "_B") simm5_8b:$imm5, (!cast<SDNodeXForm>("trunc_imm") $imm5b))>;
-  def : Pat<(add (nxv8i16 (step_vector_oneuse simm5_16b_tgt:$imm5b)), (nxv8i16 (splat_vector(simm5_16b:$imm5)))),
+  def : Pat<(add_and_or_is_add (nxv8i16 (step_vector_oneuse simm5_16b_tgt:$imm5b)), (nxv8i16 (splat_vector(simm5_16b:$imm5)))),
             (!cast<Instruction>(NAME # "_H") simm5_16b:$imm5, (!cast<SDNodeXForm>("trunc_imm") $imm5b))>;
-  def : Pat<(add (nxv4i32 (step_vector_oneuse simm5_32b_tgt:$imm5b)), (nxv4i32 (splat_vector(simm5_32b:$imm5)))),
+  def : Pat<(add_and_or_is_add (nxv4i32 (step_vector_oneuse simm5_32b_tgt:$imm5b)), (nxv4i32 (splat_vector(simm5_32b:$imm5)))),
             (!cast<Instruction>(NAME # "_S") simm5_32b:$imm5, simm5_32b:$imm5b)>;
-  def : Pat<(add (nxv2i64 (step_vector_oneuse simm5_64b_tgt:$imm5b)), (nxv2i64 (splat_vector(simm5_64b:$imm5)))),
+  def : Pat<(add_and_or_is_add (nxv2i64 (step_vector_oneuse simm5_64b_tgt:$imm5b)), (nxv2i64 (splat_vector(simm5_64b:$imm5)))),
             (!cast<Instruction>(NAME # "_D") simm5_64b:$imm5, simm5_64b:$imm5b)>;
 }
 
 class sve_int_index_ir<bits<2> sz8_64, string asm, ZPRRegOp zprty,
                        RegisterClass srcRegType, Operand imm_ty>
 : I<(outs zprty:$Zd), (ins imm_ty:$imm5, srcRegType:$Rm),
   asm, "\t$Zd, $imm5, $Rm",
   "", []>, Sched<[]> {

llvm/test/CodeGen/AArch64/sve-intrinsics-perm-select.ll

 }
 
 ; NOTE: Index out of range (0-3)
 define <vscale x 2 x i64> @dupq_i64_range(<vscale x 2 x i64> %a) {
 ; CHECK-LABEL: dupq_i64_range:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    index z1.d, #0, #1
 ; CHECK-NEXT:    and z1.d, z1.d, #0x1
-; CHECK-NEXT:    add z1.d, z1.d, #8 // =0x8
+; CHECK-NEXT:    orr z1.d, z1.d, #0x8
 ; CHECK-NEXT:    tbl z0.d, { z0.d }, z1.d
 ; CHECK-NEXT:    ret
   %out = call <vscale x 2 x i64> @llvm.aarch64.sve.dupq.lane.nxv2i64(<vscale x 2 x i64> %a, i64 4)
   ret <vscale x 2 x i64> %out
 }
 
 define dso_local <vscale x 2 x double> @dupq_ld1rqd_f64() {
 ; CHECK-LABEL: dupq_ld1rqd_f64:

llvm/test/CodeGen/AArch64/sve-knownbits.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve -verify-machineinstrs < %s | FileCheck %s
 
 define <vscale x 8 x i16> @test_knownzero(<vscale x 8 x i16> %x) {
 ; CHECK-LABEL: test_knownzero:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    lsl z0.h, z0.h, #8
-; CHECK-NEXT:    and z0.h, z0.h, #0x8
+; CHECK-NEXT:    mov z0.h, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a1 = shl <vscale x 8 x i16> %x, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 8, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
   %a2 = and <vscale x 8 x i16> %a1, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 8, i32 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
   ret <vscale x 8 x i16> %a2
 }
 
 define <vscale x 4 x i32> @asrlsr(<vscale x 4 x i64> %va) {
 ; CHECK-LABEL: asrlsr:

llvm/test/CodeGen/AArch64/sve-umulo-sdnode.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve -verify-machineinstrs < %s | FileCheck %s
 
 declare { <vscale x 2 x i8>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i8(<vscale x 2 x i8>, <vscale x 2 x i8>)
 
 define <vscale x 2 x i8> @umulo_nxv2i8(<vscale x 2 x i8> %x, <vscale x 2 x i8> %y) {
 ; CHECK-LABEL: umulo_nxv2i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    and z1.d, z1.d, #0xff
 ; CHECK-NEXT:    and z0.d, z0.d, #0xff
-; CHECK-NEXT:    movprfx z2, z0
-; CHECK-NEXT:    mul z2.d, p0/m, z2.d, z1.d
-; CHECK-NEXT:    umulh z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    lsr z1.d, z2.d, #8
-; CHECK-NEXT:    cmpne p1.d, p0/z, z0.d, #0
+; CHECK-NEXT:    mul z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    lsr z1.d, z0.d, #8
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z1.d, #0
-; CHECK-NEXT:    sel p0.b, p0, p0.b, p1.b
-; CHECK-NEXT:    mov z2.d, p0/m, #0 // =0x0
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mov z0.d, p0/m, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a = call { <vscale x 2 x i8>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i8(<vscale x 2 x i8> %x, <vscale x 2 x i8> %y)
   %b = extractvalue { <vscale x 2 x i8>, <vscale x 2 x i1> } %a, 0
   %c = extractvalue { <vscale x 2 x i8>, <vscale x 2 x i1> } %a, 1
   %d = select <vscale x 2 x i1> %c, <vscale x 2 x i8> zeroinitializer, <vscale x 2 x i8> %b
   ret <vscale x 2 x i8> %d
 }
 
 declare { <vscale x 4 x i8>, <vscale x 4 x i1> } @llvm.umul.with.overflow.nxv4i8(<vscale x 4 x i8>, <vscale x 4 x i8>)
 
 define <vscale x 4 x i8> @umulo_nxv4i8(<vscale x 4 x i8> %x, <vscale x 4 x i8> %y) {
 ; CHECK-LABEL: umulo_nxv4i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    and z1.s, z1.s, #0xff
 ; CHECK-NEXT:    and z0.s, z0.s, #0xff
-; CHECK-NEXT:    movprfx z2, z0
-; CHECK-NEXT:    mul z2.s, p0/m, z2.s, z1.s
-; CHECK-NEXT:    umulh z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    lsr z1.s, z2.s, #8
-; CHECK-NEXT:    cmpne p1.s, p0/z, z0.s, #0
+; CHECK-NEXT:    mul z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    lsr z1.s, z0.s, #8
 ; CHECK-NEXT:    cmpne p0.s, p0/z, z1.s, #0
-; CHECK-NEXT:    sel p0.b, p0, p0.b, p1.b
-; CHECK-NEXT:    mov z2.s, p0/m, #0 // =0x0
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mov z0.s, p0/m, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a = call { <vscale x 4 x i8>, <vscale x 4 x i1> } @llvm.umul.with.overflow.nxv4i8(<vscale x 4 x i8> %x, <vscale x 4 x i8> %y)
   %b = extractvalue { <vscale x 4 x i8>, <vscale x 4 x i1> } %a, 0
   %c = extractvalue { <vscale x 4 x i8>, <vscale x 4 x i1> } %a, 1
   %d = select <vscale x 4 x i1> %c, <vscale x 4 x i8> zeroinitializer, <vscale x 4 x i8> %b
   ret <vscale x 4 x i8> %d
 }
 
 declare { <vscale x 8 x i8>, <vscale x 8 x i1> } @llvm.umul.with.overflow.nxv8i8(<vscale x 8 x i8>, <vscale x 8 x i8>)
 
 define <vscale x 8 x i8> @umulo_nxv8i8(<vscale x 8 x i8> %x, <vscale x 8 x i8> %y) {
 ; CHECK-LABEL: umulo_nxv8i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h
 ; CHECK-NEXT:    and z1.h, z1.h, #0xff
 ; CHECK-NEXT:    and z0.h, z0.h, #0xff
-; CHECK-NEXT:    movprfx z2, z0
-; CHECK-NEXT:    mul z2.h, p0/m, z2.h, z1.h
-; CHECK-NEXT:    umulh z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT:    lsr z1.h, z2.h, #8
-; CHECK-NEXT:    cmpne p1.h, p0/z, z0.h, #0
+; CHECK-NEXT:    mul z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    lsr z1.h, z0.h, #8
 ; CHECK-NEXT:    cmpne p0.h, p0/z, z1.h, #0
-; CHECK-NEXT:    sel p0.b, p0, p0.b, p1.b
-; CHECK-NEXT:    mov z2.h, p0/m, #0 // =0x0
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mov z0.h, p0/m, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a = call { <vscale x 8 x i8>, <vscale x 8 x i1> } @llvm.umul.with.overflow.nxv8i8(<vscale x 8 x i8> %x, <vscale x 8 x i8> %y)
   %b = extractvalue { <vscale x 8 x i8>, <vscale x 8 x i1> } %a, 0
   %c = extractvalue { <vscale x 8 x i8>, <vscale x 8 x i1> } %a, 1
   %d = select <vscale x 8 x i1> %c, <vscale x 8 x i8> zeroinitializer, <vscale x 8 x i8> %b
   ret <vscale x 8 x i8> %d
 }
 

 declare { <vscale x 2 x i16>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i16(<vscale x 2 x i16>, <vscale x 2 x i16>)
 
 define <vscale x 2 x i16> @umulo_nxv2i16(<vscale x 2 x i16> %x, <vscale x 2 x i16> %y) {
 ; CHECK-LABEL: umulo_nxv2i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    and z1.d, z1.d, #0xffff
 ; CHECK-NEXT:    and z0.d, z0.d, #0xffff
-; CHECK-NEXT:    movprfx z2, z0
-; CHECK-NEXT:    mul z2.d, p0/m, z2.d, z1.d
-; CHECK-NEXT:    umulh z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    lsr z1.d, z2.d, #16
-; CHECK-NEXT:    cmpne p1.d, p0/z, z0.d, #0
+; CHECK-NEXT:    mul z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    lsr z1.d, z0.d, #16
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z1.d, #0
-; CHECK-NEXT:    sel p0.b, p0, p0.b, p1.b
-; CHECK-NEXT:    mov z2.d, p0/m, #0 // =0x0
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mov z0.d, p0/m, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a = call { <vscale x 2 x i16>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i16(<vscale x 2 x i16> %x, <vscale x 2 x i16> %y)
   %b = extractvalue { <vscale x 2 x i16>, <vscale x 2 x i1> } %a, 0
   %c = extractvalue { <vscale x 2 x i16>, <vscale x 2 x i1> } %a, 1
   %d = select <vscale x 2 x i1> %c, <vscale x 2 x i16> zeroinitializer, <vscale x 2 x i16> %b
   ret <vscale x 2 x i16> %d
 }
 
 declare { <vscale x 4 x i16>, <vscale x 4 x i1> } @llvm.umul.with.overflow.nxv4i16(<vscale x 4 x i16>, <vscale x 4 x i16>)
 
 define <vscale x 4 x i16> @umulo_nxv4i16(<vscale x 4 x i16> %x, <vscale x 4 x i16> %y) {
 ; CHECK-LABEL: umulo_nxv4i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    and z1.s, z1.s, #0xffff
 ; CHECK-NEXT:    and z0.s, z0.s, #0xffff
-; CHECK-NEXT:    movprfx z2, z0
-; CHECK-NEXT:    mul z2.s, p0/m, z2.s, z1.s
-; CHECK-NEXT:    umulh z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    lsr z1.s, z2.s, #16
-; CHECK-NEXT:    cmpne p1.s, p0/z, z0.s, #0
+; CHECK-NEXT:    mul z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    lsr z1.s, z0.s, #16
 ; CHECK-NEXT:    cmpne p0.s, p0/z, z1.s, #0
-; CHECK-NEXT:    sel p0.b, p0, p0.b, p1.b
-; CHECK-NEXT:    mov z2.s, p0/m, #0 // =0x0
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mov z0.s, p0/m, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a = call { <vscale x 4 x i16>, <vscale x 4 x i1> } @llvm.umul.with.overflow.nxv4i16(<vscale x 4 x i16> %x, <vscale x 4 x i16> %y)
   %b = extractvalue { <vscale x 4 x i16>, <vscale x 4 x i1> } %a, 0
   %c = extractvalue { <vscale x 4 x i16>, <vscale x 4 x i1> } %a, 1
   %d = select <vscale x 4 x i1> %c, <vscale x 4 x i16> zeroinitializer, <vscale x 4 x i16> %b
   ret <vscale x 4 x i16> %d
 }
 

 declare { <vscale x 2 x i32>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i32(<vscale x 2 x i32>, <vscale x 2 x i32>)
 
 define <vscale x 2 x i32> @umulo_nxv2i32(<vscale x 2 x i32> %x, <vscale x 2 x i32> %y) {
 ; CHECK-LABEL: umulo_nxv2i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    and z1.d, z1.d, #0xffffffff
 ; CHECK-NEXT:    and z0.d, z0.d, #0xffffffff
-; CHECK-NEXT:    movprfx z2, z0
-; CHECK-NEXT:    mul z2.d, p0/m, z2.d, z1.d
-; CHECK-NEXT:    umulh z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    lsr z1.d, z2.d, #32
-; CHECK-NEXT:    cmpne p1.d, p0/z, z0.d, #0
+; CHECK-NEXT:    mul z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    lsr z1.d, z0.d, #32
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z1.d, #0
-; CHECK-NEXT:    sel p0.b, p0, p0.b, p1.b
-; CHECK-NEXT:    mov z2.d, p0/m, #0 // =0x0
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mov z0.d, p0/m, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a = call { <vscale x 2 x i32>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i32(<vscale x 2 x i32> %x, <vscale x 2 x i32> %y)
   %b = extractvalue { <vscale x 2 x i32>, <vscale x 2 x i1> } %a, 0
   %c = extractvalue { <vscale x 2 x i32>, <vscale x 2 x i1> } %a, 1
   %d = select <vscale x 2 x i1> %c, <vscale x 2 x i32> zeroinitializer, <vscale x 2 x i32> %b
   ret <vscale x 2 x i32> %d
 }