[AArch64][CodeGen] Always use SVE (when enabled) to lower integer divides

[david-arm]

Summary

This patch adds custom lowering support for ISD::SDIV and ISD::UDIV
when SVE is enabled, regardless of the minimum SVE vector length. We do
this because NEON simply does not have vector integer divide support, so
we want to take advantage of these instructions in SVE.

As part of this patch I've also simplified LowerToPredicatedOp to avoid
re-asking the same question about whether we should be using SVE for
fixed length vectors. Once we've made the decision to call
LowerToPredicatedOp, then we should simply assert we should be using SVE.

I've pulled the 128-bit min SVE vector bits tests out of

CodeGen/AArch64/sve-fixed-length-int-div.ll
CodeGen/AArch64/sve-fixed-length-int-rem.ll

and moved them here instead

CodeGen/AArch64/sve-fixed-length-int-div-128.ll
CodeGen/AArch64/sve-fixed-length-int-rem-128.ll

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11183

Here I think VT.isFixedLengthVector() && hasSVE() is all you need. This is operation legalisation so the types must already be legal. The current useSVEForFixedLengthVectorVT call exists purely to prevent the one case you no longer want to prevent.

llvm/lib/Target/AArch64/AArch64ISelLowering.h

497–501

My immediate reaction is that I'm not sure this enum is required. It exists so that you can pass Always to useSVEForFixedLengthVectorVT but under those circumstances I don't believe there's a need to ever call useSVEForFixedLengthVectorVT.

[david-arm]

• Removed the NeonOverride enum from previous patch.

[sdesmalen]

llvm/test/CodeGen/AArch64/sve-fixed-length-int-div.ll

1

Instead of duplicating these tests and moving these check lines to a different file, I'd rather have a RUN line with a check-prefix=VBITS_EQ_128, where you only add check lines for the cases where it currently already has CHECK and CHECK-NEXT lines.

For all places where the expected CHECK lines are different from the expected output for VBITS_EQ_128, please add a FIXME: The code should be equivalent. Remove CHECK lines in favour of VBITS_EQ_128. or something like that. When we fix the issue that having a larger vector width generates different code for the same fixed-width vectors, it will be obvious which lines to remove, and we don't end up with a stale test file (e.g. sve-fixed-length-int-div-128.ll)

[david-arm]

• Moved the 128-bit tests back into the original file.

[david-arm]

llvm/test/CodeGen/AArch64/sve-fixed-length-int-rem.ll

781

NOTE: The codegen here would be improved if we also used SVE for 64-bit element vector multiplies!

[paulwalker-arm]

The code looks fine to me. For the tests I would prefer it if you removed the new VBITS_EQ_128 lines for the >128bit vector tests as I don't believe they're testing anything new that isn't already being tested by the 64 and 128 bit vector tests. More than that, because you're hardwiring all the output I can see it just being a source of pain as the tests are more likely to need to be manually updated after unrelated code generation changes, which is something that has previously annoyed people. If you think the checks offer real value then as a minimum please consider at least removing them from the v32i8 variants as they're likely to be the worst offenders.

llvm/test/CodeGen/AArch64/sve-fixed-length-int-div.ll

186

Do you need these CHECK lines? I would have thought it's only worth VBITS_EQ_128 CHECk lines for the 128bit tests given I'm sure by now type legalisation is already well tested for NEON.

966

As above I'm not sure of the value of these CHECK lines.

[sdesmalen]

In D117871#3289061, @paulwalker-arm wrote:

The code looks fine to me. For the tests I would prefer it if you removed the new VBITS_EQ_128 lines for the >128bit vector tests as I don't believe they're testing anything new that isn't already being tested by the 64 and 128 bit vector tests. More than that, because you're hardwiring all the output I can see it just being a source of pain as the tests are more likely to need to be manually updated after unrelated code generation changes, which is something that has previously annoyed people. If you think the checks offer real value then as a minimum please consider at least removing them from the v32i8 variants as they're likely to be the worst offenders.

+1

llvm/lib/Target/AArch64/AArch64ISelLowering.h

enum Rounding {
   RZ = 3,    // Round towards Zero
   rmMask = 3 // Bit mask selecting rounding mode
 };
 
 // Bit position of rounding mode bits in FPCR.
 const unsigned RoundingBitsPos = 22;
 } // namespace AArch64
 
 class AArch64Subtarget;
 
 class AArch64TargetLowering : public TargetLowering {
 public:
   explicit AArch64TargetLowering(const TargetMachine &TM,
                                  const AArch64Subtarget &STI);
 
   /// Selects the correct CCAssignFn for a given CallingConvention value.
   CCAssignFn *CCAssignFnForCall(CallingConv::ID CC, bool IsVarArg) const;
 
   /// Selects the correct CCAssignFn for a given CallingConvention value.
   CCAssignFn *CCAssignFnForReturn(CallingConv::ID CC) const;
 
private:
   SDValue LowerSET_ROUNDING(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerINSERT_VECTOR_ELT(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerEXTRACT_VECTOR_ELT(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSCALAR_TO_VECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerBUILD_VECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVECTOR_SHUFFLE(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSPLAT_VECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerDUPQLane(SDValue Op, SelectionDAG &DAG) const;
-  SDValue LowerToPredicatedOp(SDValue Op, SelectionDAG &DAG, unsigned NewOp,
-                              bool OverrideNEON = false) const;
+  SDValue LowerToPredicatedOp(SDValue Op, SelectionDAG &DAG,
+                              unsigned NewOp) const;
   SDValue LowerToScalableOp(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVECTOR_SPLICE(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerEXTRACT_SUBVECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerINSERT_SUBVECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerDIV(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerMUL(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVectorSRA_SRL_SHL(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerShiftParts(SDValue Op, SelectionDAG &DAG) const;

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

for (auto VT : {MVT::nxv2bf16, MVT::nxv4bf16, MVT::nxv8bf16}) {
       setOperationAction(ISD::INSERT_SUBVECTOR, VT, Custom);
     }
 
     setOperationAction(ISD::SPLAT_VECTOR, MVT::nxv8bf16, Custom);
 
     setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::i8, Custom);
     setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::i16, Custom);
 
+    // NEON doesn't support integer divides, but SVE does
+    for (auto VT : {MVT::v8i8, MVT::v16i8, MVT::v4i16, MVT::v8i16, MVT::v2i32,
+                    MVT::v4i32, MVT::v1i64, MVT::v2i64}) {
+      setOperationAction(ISD::SDIV, VT, Custom);
+      setOperationAction(ISD::UDIV, VT, Custom);
+    }
+
     // NOTE: Currently this has to happen after computeRegisterProperties rather
     // than the preferred option of combining it with the addRegisterClass call.
     if (Subtarget->useSVEForFixedLengthVectors()) {
       for (MVT VT : MVT::integer_fixedlen_vector_valuetypes())
         if (useSVEForFixedLengthVectorVT(VT))
           addTypeForFixedLengthSVE(VT);
       for (MVT VT : MVT::fp_fixedlen_vector_valuetypes())
         if (useSVEForFixedLengthVectorVT(VT))
if (Subtarget->useSVEForFixedLengthVectors()) {
       setOperationAction(ISD::CTLZ, MVT::v2i64, Custom);
       setOperationAction(ISD::CTTZ, MVT::v1i64, Custom);
       setOperationAction(ISD::MUL, MVT::v1i64, Custom);
       setOperationAction(ISD::MUL, MVT::v2i64, Custom);
       setOperationAction(ISD::MULHS, MVT::v1i64, Custom);
       setOperationAction(ISD::MULHS, MVT::v2i64, Custom);
       setOperationAction(ISD::MULHU, MVT::v1i64, Custom);
       setOperationAction(ISD::MULHU, MVT::v2i64, Custom);
-      setOperationAction(ISD::SDIV, MVT::v8i8, Custom);
-      setOperationAction(ISD::SDIV, MVT::v16i8, Custom);
-      setOperationAction(ISD::SDIV, MVT::v4i16, Custom);
-      setOperationAction(ISD::SDIV, MVT::v8i16, Custom);
-      setOperationAction(ISD::SDIV, MVT::v2i32, Custom);
-      setOperationAction(ISD::SDIV, MVT::v4i32, Custom);
-      setOperationAction(ISD::SDIV, MVT::v1i64, Custom);
-      setOperationAction(ISD::SDIV, MVT::v2i64, Custom);
       setOperationAction(ISD::SMAX, MVT::v1i64, Custom);
       setOperationAction(ISD::SMAX, MVT::v2i64, Custom);
       setOperationAction(ISD::SMIN, MVT::v1i64, Custom);
       setOperationAction(ISD::SMIN, MVT::v2i64, Custom);
-      setOperationAction(ISD::UDIV, MVT::v8i8, Custom);
-      setOperationAction(ISD::UDIV, MVT::v16i8, Custom);
-      setOperationAction(ISD::UDIV, MVT::v4i16, Custom);
-      setOperationAction(ISD::UDIV, MVT::v8i16, Custom);
-      setOperationAction(ISD::UDIV, MVT::v2i32, Custom);
-      setOperationAction(ISD::UDIV, MVT::v4i32, Custom);
-      setOperationAction(ISD::UDIV, MVT::v1i64, Custom);
-      setOperationAction(ISD::UDIV, MVT::v2i64, Custom);
       setOperationAction(ISD::UMAX, MVT::v1i64, Custom);
       setOperationAction(ISD::UMAX, MVT::v2i64, Custom);
       setOperationAction(ISD::UMIN, MVT::v1i64, Custom);
       setOperationAction(ISD::UMIN, MVT::v2i64, Custom);
       setOperationAction(ISD::VECREDUCE_SMAX, MVT::v2i64, Custom);
       setOperationAction(ISD::VECREDUCE_SMIN, MVT::v2i64, Custom);
       setOperationAction(ISD::VECREDUCE_UMAX, MVT::v2i64, Custom);
       setOperationAction(ISD::VECREDUCE_UMIN, MVT::v2i64, Custom);

 
 SDValue AArch64TargetLowering::LowerMUL(SDValue Op, SelectionDAG &DAG) const {
   EVT VT = Op.getValueType();
 
   // If SVE is available then i64 vector multiplications can also be made legal.
   bool OverrideNEON = VT == MVT::v2i64 || VT == MVT::v1i64;
 
   if (VT.isScalableVector() || useSVEForFixedLengthVectorVT(VT, OverrideNEON))
-    return LowerToPredicatedOp(Op, DAG, AArch64ISD::MUL_PRED, OverrideNEON);
+    return LowerToPredicatedOp(Op, DAG, AArch64ISD::MUL_PRED);
 
   // Multiplications are only custom-lowered for 128-bit vectors so that
   // VMULL can be detected.  Otherwise v2i64 multiplications are not legal.
   assert(VT.is128BitVector() && VT.isInteger() &&
          "unexpected type for custom-lowering ISD::MUL");
   SDNode *N0 = Op.getOperand(0).getNode();
   SDNode *N1 = Op.getOperand(1).getNode();
   unsigned NewOpc = 0;
case ISD::FSINCOS:
     return LowerFSINCOS(Op, DAG);
   case ISD::FLT_ROUNDS_:
     return LowerFLT_ROUNDS_(Op, DAG);
   case ISD::SET_ROUNDING:
     return LowerSET_ROUNDING(Op, DAG);
   case ISD::MUL:
     return LowerMUL(Op, DAG);
   case ISD::MULHS:
-    return LowerToPredicatedOp(Op, DAG, AArch64ISD::MULHS_PRED,
-                               /*OverrideNEON=*/true);
+    return LowerToPredicatedOp(Op, DAG, AArch64ISD::MULHS_PRED);
   case ISD::MULHU:
-    return LowerToPredicatedOp(Op, DAG, AArch64ISD::MULHU_PRED,
-                               /*OverrideNEON=*/true);
+    return LowerToPredicatedOp(Op, DAG, AArch64ISD::MULHU_PRED);
   case ISD::INTRINSIC_W_CHAIN:
     return LowerINTRINSIC_W_CHAIN(Op, DAG);
   case ISD::INTRINSIC_WO_CHAIN:
     return LowerINTRINSIC_WO_CHAIN(Op, DAG);
   case ISD::ATOMIC_STORE:
     if (cast<MemSDNode>(Op)->getMemoryVT() == MVT::i128) {
       assert(Subtarget->hasLSE2());
       return LowerStore128(Op, DAG);
case ISD::ABDS:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::ABDS_PRED);
   case ISD::ABDU:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::ABDU_PRED);
   case ISD::BITREVERSE:
     return LowerBitreverse(Op, DAG);
   case ISD::BSWAP:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::BSWAP_MERGE_PASSTHRU);
   case ISD::CTLZ:
-    return LowerToPredicatedOp(Op, DAG, AArch64ISD::CTLZ_MERGE_PASSTHRU,
-                               /*OverrideNEON=*/true);
+    return LowerToPredicatedOp(Op, DAG, AArch64ISD::CTLZ_MERGE_PASSTHRU);
   case ISD::CTTZ:
     return LowerCTTZ(Op, DAG);
   case ISD::VECTOR_SPLICE:
     return LowerVECTOR_SPLICE(Op, DAG);
   }
 }
 
 bool AArch64TargetLowering::mergeStoresAfterLegalization(EVT VT) const {
SDValue AArch64TargetLowering::LowerMinMax(SDValue Op,
   }
 
   if (VT.isScalableVector() ||
       useSVEForFixedLengthVectorVT(VT, /*OverrideNEON=*/true)) {
     switch (Opcode) {
     default:
       llvm_unreachable("Wrong instruction");
     case ISD::SMAX:
-      return LowerToPredicatedOp(Op, DAG, AArch64ISD::SMAX_PRED,
-                                 /*OverrideNEON=*/true);
+      return LowerToPredicatedOp(Op, DAG, AArch64ISD::SMAX_PRED);
     case ISD::SMIN:
-      return LowerToPredicatedOp(Op, DAG, AArch64ISD::SMIN_PRED,
-                                 /*OverrideNEON=*/true);
+      return LowerToPredicatedOp(Op, DAG, AArch64ISD::SMIN_PRED);
     case ISD::UMAX:
-      return LowerToPredicatedOp(Op, DAG, AArch64ISD::UMAX_PRED,
-                                 /*OverrideNEON=*/true);
+      return LowerToPredicatedOp(Op, DAG, AArch64ISD::UMAX_PRED);
     case ISD::UMIN:
-      return LowerToPredicatedOp(Op, DAG, AArch64ISD::UMIN_PRED,
-                                 /*OverrideNEON=*/true);
+      return LowerToPredicatedOp(Op, DAG, AArch64ISD::UMIN_PRED);
     }
   }
 
   SDValue Op0 = Op.getOperand(0);
   SDValue Op1 = Op.getOperand(1);
   SDValue Cond = DAG.getSetCC(DL, VT, Op0, Op1, CC);
   return DAG.getSelect(DL, VT, Cond, Op0, Op1);
 }
 
 SDValue AArch64TargetLowering::LowerBitreverse(SDValue Op,
                                                SelectionDAG &DAG) const {
   EVT VT = Op.getValueType();
 
   if (VT.isScalableVector() ||
       useSVEForFixedLengthVectorVT(VT, /*OverrideNEON=*/true))
-    return LowerToPredicatedOp(Op, DAG, AArch64ISD::BITREVERSE_MERGE_PASSTHRU,
-                               true);
+    return LowerToPredicatedOp(Op, DAG, AArch64ISD::BITREVERSE_MERGE_PASSTHRU);
 
   SDLoc DL(Op);
   SDValue REVB;
   MVT VST;
 
   switch (VT.getSimpleVT().SimpleTy) {
   default:
     llvm_unreachable("Invalid type for bitreverse!");
static bool isPow2Splat(SDValue Op, uint64_t &SplatVal, bool &Negated) {
 
   return false;
 }
 
 SDValue AArch64TargetLowering::LowerDIV(SDValue Op, SelectionDAG &DAG) const {
   EVT VT = Op.getValueType();
   SDLoc dl(Op);
 
-  if (useSVEForFixedLengthVectorVT(VT, /*OverrideNEON=*/true))
+  if (VT.isFixedLengthVector() && Subtarget->hasSVE())
     return LowerFixedLengthVectorIntDivideToSVE(Op, DAG);
 
   assert(VT.isScalableVector() && "Expected a scalable vector.");
 
   bool Signed = Op.getOpcode() == ISD::SDIV;
   unsigned PredOpcode = Signed ? AArch64ISD::SDIV_PRED : AArch64ISD::UDIV_PRED;
 
   bool Negated;
if (Signed && isPow2Splat(Op.getOperand(1), SplatVal, Negated)) {
     if (Negated)
       Res = DAG.getNode(ISD::SUB, dl, VT, DAG.getConstant(0, dl, VT), Res);
 
     return convertFromScalableVector(DAG, VT, Res);
   }
 
   // Scalable vector i32/i64 DIV is supported.
   if (EltVT == MVT::i32 || EltVT == MVT::i64)
-    return LowerToPredicatedOp(Op, DAG, PredOpcode, /*OverrideNEON=*/true);
+    return LowerToPredicatedOp(Op, DAG, PredOpcode);
 
   // Scalable vector i8/i16 DIV is not supported. Promote it to i32.
   EVT ContainerVT = getContainerForFixedLengthVector(DAG, VT);
   EVT HalfVT = VT.getHalfNumVectorElementsVT(*DAG.getContext());
   EVT FixedWidenedVT = HalfVT.widenIntegerVectorElementType(*DAG.getContext());
   EVT ScalableWidenedVT = getContainerForFixedLengthVector(DAG, FixedWidenedVT);
 
   // If this is not a full vector, extend, div, and truncate it.
SDValue AArch64TargetLowering::LowerFixedLengthInsertVectorElt(
   return convertFromScalableVector(DAG, VT, ScalableRes);
 }
 
 // Convert vector operation 'Op' to an equivalent predicated operation whereby
 // the original operation's type is used to construct a suitable predicate.
 // NOTE: The results for inactive lanes are undefined.
 SDValue AArch64TargetLowering::LowerToPredicatedOp(SDValue Op,
                                                    SelectionDAG &DAG,
-                                                   unsigned NewOp,
-                                                   bool OverrideNEON) const {
+                                                   unsigned NewOp) const {
   EVT VT = Op.getValueType();
   SDLoc DL(Op);
   auto Pg = getPredicateForVector(DAG, DL, VT);
 
-  if (useSVEForFixedLengthVectorVT(VT, OverrideNEON)) {
+  if (VT.isFixedLengthVector()) {
+    assert(VT.getFixedSizeInBits() <= Subtarget->getMinSVEVectorSizeInBits() &&
+           "Cannot use SVE to lower fixed length predicated op!");
     EVT ContainerVT = getContainerForFixedLengthVector(DAG, VT);
 
     // Create list of operands by converting existing ones to scalable types.
     SmallVector<SDValue, 4> Operands = {Pg};
     for (const SDValue &V : Op->op_values()) {
       if (isa<CondCodeSDNode>(V)) {
         Operands.push_back(V);
         continue;
       }
 
       if (const VTSDNode *VTNode = dyn_cast<VTSDNode>(V)) {
         EVT VTArg = VTNode->getVT().getVectorElementType();
         EVT NewVTArg = ContainerVT.changeVectorElementType(VTArg);
         Operands.push_back(DAG.getValueType(NewVTArg));
         continue;
       }
 
-      assert(useSVEForFixedLengthVectorVT(V.getValueType(), OverrideNEON) &&
+      assert(V.getValueType().getFixedSizeInBits() <=
+                 Subtarget->getMinSVEVectorSizeInBits() &&
              "Only fixed length vectors are supported!");
       Operands.push_back(convertToScalableVector(DAG, ContainerVT, V));
     }
 
     if (isMergePassthruOpcode(NewOp))
       Operands.push_back(DAG.getUNDEF(ContainerVT));
 
     auto ScalableRes = DAG.getNode(NewOp, DL, ContainerVT, Operands);

llvm/test/CodeGen/AArch64/sve-fixed-length-int-div.ll

-; RUN: llc -aarch64-sve-vector-bits-min=128  -asm-verbose=0 < %s | FileCheck %s -check-prefix=NO_SVE
+; RUN: llc -aarch64-sve-vector-bits-min=128  -asm-verbose=0 < %s | FileCheck %s -check-prefix=VBITS_EQ_128
 ; RUN: llc -aarch64-sve-vector-bits-min=256  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_EQ_256
 ; RUN: llc -aarch64-sve-vector-bits-min=384  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_EQ_256
 ; RUN: llc -aarch64-sve-vector-bits-min=512  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_EQ_512
 ; RUN: llc -aarch64-sve-vector-bits-min=640  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_EQ_512
 ; RUN: llc -aarch64-sve-vector-bits-min=768  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_EQ_512
 ; RUN: llc -aarch64-sve-vector-bits-min=896  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_EQ_512
 ; RUN: llc -aarch64-sve-vector-bits-min=1024 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1152 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1280 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1408 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1536 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1664 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1792 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1920 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=2048 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_GE_2048,VBITS_EQ_2048
 
 target triple = "aarch64-unknown-linux-gnu"
 
-; Don't use SVE when its registers are no bigger than NEON.
-; NO_SVE-NOT: ptrue
-
 ;
 ; SDIV
 ;
 
 ; Vector vXi8 sdiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for >= 256 bits here.
 define <8 x i8> @sdiv_v8i8(<8 x i8> %op1, <8 x i8> %op2) #0 {
 ; CHECK-LABEL: sdiv_v8i8:
 ; CHECK: ptrue [[PG0:p[0-9]+]].s, vl8
 ; CHECK-NEXT: sunpklo [[OP2_LO:z[0-9]+]].h, z1.b
 ; CHECK-NEXT: sunpklo [[OP1_LO:z[0-9]+]].h, z0.b
 ; CHECK-NEXT: sunpklo [[OP2_LO_LO:z[0-9]+]].s, z1.h
 ; CHECK-NEXT: sunpklo [[OP1_LO_LO:z[0-9]+]].s, z0.h
 ; CHECK-NEXT: sdiv [[DIV:z[0-9]+]].s, [[PG0]]/m, [[OP1_LO_LO]].s, [[OP2_LO_LO]].s

 ; CHECK-NEXT: mov [[FINAL]].b[4], [[SCALAR4]]
 ; CHECK-NEXT: umov [[SCALAR5:w[0-9]+]], [[VEC]].h[5]
 ; CHECK-NEXT: mov [[FINAL]].b[5], [[SCALAR5]]
 ; CHECK-NEXT: umov [[SCALAR6:w[0-9]+]], [[VEC]].h[6]
 ; CHECK-NEXT: mov [[FINAL]].b[6], [[SCALAR6]]
 ; CHECK-NEXT: umov [[SCALAR7:w[0-9]+]], [[VEC]].h[7]
 ; CHECK-NEXT: mov [[FINAL]].b[7], [[SCALAR7]]
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v8i8:
+; VBITS_EQ_128:         sshll v1.8h, v1.8b, #0
+; VBITS_EQ_128-NEXT:    sshll v0.8h, v0.8b, #0
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    sunpkhi z2.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpkhi z3.s, z0.h
+; VBITS_EQ_128-NEXT:    sunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpklo z0.s, z0.h
+; VBITS_EQ_128-NEXT:    sdivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    sdiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    uzp1 z0.h, z0.h, z2.h
+; VBITS_EQ_128-NEXT:    xtn v0.8b, v0.8h
+; VBITS_EQ_128-NEXT:    ret
+
   %res = sdiv <8 x i8> %op1, %op2
   ret <8 x i8> %res
 }
 
 define <16 x i8> @sdiv_v16i8(<16 x i8> %op1, <16 x i8> %op2) #0 {
 ; CHECK-LABEL: sdiv_v16i8:
 
 ; HALF VECTOR:

 ; VBITS_GE_512-NEXT: sunpklo [[OP2_LO:z[0-9]+]].h, z1.b
 ; VBITS_GE_512-NEXT: sunpklo [[OP1_LO:z[0-9]+]].h, z0.b
 ; VBITS_GE_512-NEXT: sunpklo [[OP2_LO_LO:z[0-9]+]].s, z1.h
 ; VBITS_GE_512-NEXT: sunpklo [[OP1_LO_LO:z[0-9]+]].s, z0.h
 ; VBITS_GE_512-NEXT: sdiv [[DIV:z[0-9]+]].s, [[PG]]/m, [[OP1_LO_LO]].s, [[OP2_LO_LO]].s
 ; VBITS_GE_512-NEXT: uzp1 [[RES1:z[0-9]+]].h, [[DIV]].h, [[DIV]].h
 ; VBITS_GE_512-NEXT: uzp1 [[RES2:z[0-9]+]].b, [[RES1]].b, [[RES1]].b
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v16i8:
+; VBITS_EQ_128:         sunpkhi z2.h, z1.b
+; VBITS_EQ_128-NEXT:    sunpkhi z3.h, z0.b
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    sunpklo z1.h, z1.b
+; VBITS_EQ_128-NEXT:    sunpkhi z4.s, z2.h
+; VBITS_EQ_128-NEXT:    sunpkhi z5.s, z3.h
+; VBITS_EQ_128-NEXT:    sunpklo z2.s, z2.h
+; VBITS_EQ_128-NEXT:    sunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    sunpklo z0.h, z0.b
+; VBITS_EQ_128-NEXT:    sdivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    sdivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    sunpkhi z3.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpkhi z5.s, z0.h
+; VBITS_EQ_128-NEXT:    sunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpklo z0.s, z0.h
+; VBITS_EQ_128-NEXT:    sdivr z3.s, p0/m, z3.s, z5.s
+; VBITS_EQ_128-NEXT:    sdiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    uzp1 z1.h, z2.h, z4.h
+; VBITS_EQ_128-NEXT:    uzp1 z0.h, z0.h, z3.h
+; VBITS_EQ_128-NEXT:    uzp1 z0.b, z0.b, z1.b
+; VBITS_EQ_128-NEXT:    ret
+
   %res = sdiv <16 x i8> %op1, %op2
   ret <16 x i8> %res
 }
 
 define void @sdiv_v32i8(<32 x i8>* %a, <32 x i8>* %b) #0 {
 ; CHECK-LABEL: sdiv_v32i8:
 
 ; FULL VECTOR:

 ; VBITS_GE_1024-NEXT: sunpklo [[OP1_LO:z[0-9]+]].h, [[OP1]].b
 ; VBITS_GE_1024-NEXT: sunpklo [[OP2_LO_LO:z[0-9]+]].s, [[OP2_LO]].h
 ; VBITS_GE_1024-NEXT: sunpklo [[OP1_LO_LO:z[0-9]+]].s, [[OP1_LO]].h
 ; VBITS_GE_1024-NEXT: sdiv [[DIV1:z[0-9]+]].s, [[PG2]]/m, [[OP1_LO_LO]].s, [[OP2_LO_LO]].s
 ; VBITS_GE_1024-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; VBITS_GE_1024-NEXT: uzp1 [[UZP2:z[0-9]+]].b, [[UZP1]].b, [[UZP1]].b
 ; VBITS_GE_1024-NEXT: st1b { [[UZP2:z[0-9]+]].b }, [[PG1]], [x0]
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v32i8:
+; VBITS_EQ_128:         ldp q3, q0, [x1]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    ldp q2, q1, [x0]
+; VBITS_EQ_128-NEXT:    sunpkhi z4.h, z0.b
+; VBITS_EQ_128-NEXT:    sunpklo z0.h, z0.b
+; VBITS_EQ_128-NEXT:    sunpkhi z6.s, z4.h
+; VBITS_EQ_128-NEXT:    sunpklo z4.s, z4.h
+; VBITS_EQ_128-NEXT:    sunpkhi z16.s, z0.h
+; VBITS_EQ_128-NEXT:    sunpklo z0.s, z0.h
+; VBITS_EQ_128-NEXT:    sunpkhi z5.h, z1.b
+; VBITS_EQ_128-NEXT:    sunpklo z1.h, z1.b
+; VBITS_EQ_128-NEXT:    sunpkhi z7.s, z5.h
+; VBITS_EQ_128-NEXT:    sunpklo z5.s, z5.h
+; VBITS_EQ_128-NEXT:    sdivr z6.s, p0/m, z6.s, z7.s
+; VBITS_EQ_128-NEXT:    sdivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    sunpkhi z5.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    uzp1 z4.h, z4.h, z6.h
+; VBITS_EQ_128-NEXT:    sdivr z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    sunpkhi z1.h, z3.b
+; VBITS_EQ_128-NEXT:    sunpkhi z6.h, z2.b
+; VBITS_EQ_128-NEXT:    sdiv z5.s, p0/m, z5.s, z16.s
+; VBITS_EQ_128-NEXT:    sunpkhi z7.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpkhi z16.s, z6.h
+; VBITS_EQ_128-NEXT:    sunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpklo z6.s, z6.h
+; VBITS_EQ_128-NEXT:    sunpklo z3.h, z3.b
+; VBITS_EQ_128-NEXT:    sunpklo z2.h, z2.b
+; VBITS_EQ_128-NEXT:    sdivr z7.s, p0/m, z7.s, z16.s
+; VBITS_EQ_128-NEXT:    sdivr z1.s, p0/m, z1.s, z6.s
+; VBITS_EQ_128-NEXT:    sunpkhi z6.s, z3.h
+; VBITS_EQ_128-NEXT:    sunpkhi z16.s, z2.h
+; VBITS_EQ_128-NEXT:    sunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    sunpklo z2.s, z2.h
+; VBITS_EQ_128-NEXT:    sdivr z6.s, p0/m, z6.s, z16.s
+; VBITS_EQ_128-NEXT:    sdiv z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    uzp1 z1.h, z1.h, z7.h
+; VBITS_EQ_128-NEXT:    uzp1 z2.h, z2.h, z6.h
+; VBITS_EQ_128-NEXT:    uzp1 z0.h, z0.h, z5.h
+; VBITS_EQ_128-NEXT:    uzp1 z1.b, z2.b, z1.b
+; VBITS_EQ_128-NEXT:    uzp1 z0.b, z0.b, z4.b
+; VBITS_EQ_128-NEXT:    stp q1, q0, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <32 x i8>, <32 x i8>* %a
   %op2 = load <32 x i8>, <32 x i8>* %b
   %res = sdiv <32 x i8> %op1, %op2
   store <32 x i8> %res, <32 x i8>* %a
   ret void
 }
 
 define void @sdiv_v64i8(<64 x i8>* %a, <64 x i8>* %b) #0 {
; CHECK: ret
   %op1 = load <256 x i8>, <256 x i8>* %a
   %op2 = load <256 x i8>, <256 x i8>* %b
   %res = sdiv <256 x i8> %op1, %op2
   store <256 x i8> %res, <256 x i8>* %a
   ret void
 }
 
 ; Vector vXi16 sdiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for >= 256 bits here.
 define <4 x i16> @sdiv_v4i16(<4 x i16> %op1, <4 x i16> %op2) #0 {
 ; CHECK-LABEL: sdiv_v4i16:
 ; CHECK: sshll v1.4s, v1.4h, #0
 ; CHECK-NEXT: sshll v0.4s, v0.4h, #0
 ; CHECK-NEXT: ptrue [[PG1:p[0-9]+]].s, vl4
 ; CHECK-NEXT: sdivr [[DIV1:z[0-9]+]].s, [[PG1]]/m, [[OP2:z[0-9]+]].s, [[OP1:z[0-9]+]].s
 ; CHECK-NEXT: mov w8, v1.s[1]
 ; CHECK-NEXT: mov w9, v1.s[2]
 ; CHECK-NEXT: mov v0.16b, v1.16b
 ; CHECK-NEXT: mov v0.h[1], w8
 ; CHECK-NEXT: mov w8, v1.s[3]
 ; CHECK-NEXT: mov v0.h[2], w9
 ; CHECK-NEXT: mov v0.h[3], w8
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v4i16:
+; VBITS_EQ_128:         sshll v1.4s, v1.4h, #0
+; VBITS_EQ_128-NEXT:    sshll v0.4s, v0.4h, #0
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    sdiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    xtn v0.4h, v0.4s
+; VBITS_EQ_128-NEXT:    ret
+
   %res = sdiv <4 x i16> %op1, %op2
   ret <4 x i16> %res
 }
 
 define <8 x i16> @sdiv_v8i16(<8 x i16> %op1, <8 x i16> %op2) #0 {
 ; CHECK-LABEL: sdiv_v8i16:
 ; CHECK: ptrue [[PG1:p[0-9]+]].s, vl8
 ; CHECK-NEXT: sunpklo [[OP2_LO:z[0-9]+]].s, [[OP2]].h
 ; CHECK-NEXT: sunpklo [[OP1_LO:z[0-9]+]].s, [[OP1]].h
 ; CHECK-NEXT: sdiv [[DIV1:z[0-9]+]].s, [[PG1]]/m, [[OP1_LO]].s, [[OP2_LO]].s
 ; CHECK-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v8i16:
+; VBITS_EQ_128:         ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    sunpkhi z2.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpkhi z3.s, z0.h
+; VBITS_EQ_128-NEXT:    sunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpklo z0.s, z0.h
+; VBITS_EQ_128-NEXT:    sdivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    sdiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    uzp1 z0.h, z0.h, z2.h
+; VBITS_EQ_128-NEXT:    ret
+
   %res = sdiv <8 x i16> %op1, %op2
   ret <8 x i16> %res
 }
 
 define void @sdiv_v16i16(<16 x i16>* %a, <16 x i16>* %b) #0 {
 ; CHECK-LABEL: sdiv_v16i16:
 
 ; FULL VECTOR:

 ; VBITS_GE_512-NEXT: ld1h { [[OP1:z[0-9]+]].h }, [[PG1]]/z, [x0]
 ; VBITS_GE_512-NEXT: ld1h { [[OP2:z[0-9]+]].h }, [[PG1]]/z, [x1]
 ; VBITS_GE_512-NEXT: sunpklo [[OP2_LO:z[0-9]+]].s, [[OP2]].h
 ; VBITS_GE_512-NEXT: sunpklo [[OP1_LO:z[0-9]+]].s, [[OP1]].h
 ; VBITS_GE_512-NEXT: sdiv [[DIV1:z[0-9]+]].s, [[PG2]]/m, [[OP1_LO]].s, [[OP2_LO]].s
 ; VBITS_GE_512-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; VBITS_GE_512-NEXT: st1h { [[UZP1:z[0-9]+]].h }, [[PG1]], [x0]
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v16i16:
+; VBITS_EQ_128:         ldp q0, q1, [x1]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    sunpkhi z6.s, z0.h
+; VBITS_EQ_128-NEXT:    sunpklo z0.s, z0.h
+; VBITS_EQ_128-NEXT:    ldp q3, q2, [x0]
+; VBITS_EQ_128-NEXT:    sunpkhi z4.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpkhi z5.s, z2.h
+; VBITS_EQ_128-NEXT:    sunpklo z2.s, z2.h
+; VBITS_EQ_128-NEXT:    sdivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    sunpkhi z5.s, z3.h
+; VBITS_EQ_128-NEXT:    sunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    sdiv z5.s, p0/m, z5.s, z6.s
+; VBITS_EQ_128-NEXT:    sdivr z0.s, p0/m, z0.s, z3.s
+; VBITS_EQ_128-NEXT:    sdivr z1.s, p0/m, z1.s, z2.s
+; VBITS_EQ_128-NEXT:    uzp1 z0.h, z0.h, z5.h
+; VBITS_EQ_128-NEXT:    uzp1 z1.h, z1.h, z4.h
+; VBITS_EQ_128-NEXT:    stp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <16 x i16>, <16 x i16>* %a
   %op2 = load <16 x i16>, <16 x i16>* %b
   %res = sdiv <16 x i16> %op1, %op2
   store <16 x i16> %res, <16 x i16>* %a
   ret void
 }
 
 define void @sdiv_v32i16(<32 x i16>* %a, <32 x i16>* %b) #0 {

 }
 
 ; Vector v2i32 sdiv are not legal for NEON so use SVE when available.
 define <2 x i32> @sdiv_v2i32(<2 x i32> %op1, <2 x i32> %op2) #0 {
 ; CHECK-LABEL: sdiv_v2i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl2
 ; CHECK: sdiv z0.s, [[PG]]/m, z0.s, z1.s
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v2i32:
+; VBITS_EQ_128:         ptrue p0.s, vl2
+; VBITS_EQ_128-NEXT:    sdiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    ret
+
   %res = sdiv <2 x i32> %op1, %op2
   ret <2 x i32> %res
 }
 
 ; Vector v4i32 sdiv are not legal for NEON so use SVE when available.
 define <4 x i32> @sdiv_v4i32(<4 x i32> %op1, <4 x i32> %op2) #0 {
 ; CHECK-LABEL: sdiv_v4i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl4
 ; CHECK: sdiv z0.s, [[PG]]/m, z0.s, z1.s
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v4i32:
+; VBITS_EQ_128:         ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    sdiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    ret
+
   %res = sdiv <4 x i32> %op1, %op2
   ret <4 x i32> %res
 }
 
 define void @sdiv_v8i32(<8 x i32>* %a, <8 x i32>* %b) #0 {
 ; CHECK-LABEL: sdiv_v8i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl8
 ; CHECK-NEXT: ld1w { [[OP1:z[0-9]+]].s }, [[PG]]/z, [x0]
 ; CHECK-NEXT: ld1w { [[OP2:z[0-9]+]].s }, [[PG]]/z, [x1]
 ; CHECK-NEXT: sdiv [[RES:z[0-9]+]].s, [[PG]]/m, [[OP1]].s, [[OP2]].s
 ; CHECK-NEXT: st1w { [[RES]].s }, [[PG]], [x0]
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v8i32:
+; VBITS_EQ_128:         ldp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    ldp q2, q3, [x1]
+; VBITS_EQ_128-NEXT:    sdiv z0.s, p0/m, z0.s, z2.s
+; VBITS_EQ_128-NEXT:    sdiv z1.s, p0/m, z1.s, z3.s
+; VBITS_EQ_128-NEXT:    stp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <8 x i32>, <8 x i32>* %a
   %op2 = load <8 x i32>, <8 x i32>* %b
   %res = sdiv <8 x i32> %op1, %op2
   store <8 x i32> %res, <8 x i32>* %a
   ret void
 }
 
 define void @sdiv_v16i32(<16 x i32>* %a, <16 x i32>* %b) #0 {

 }
 
 ; Vector i64 sdiv are not legal for NEON so use SVE when available.
 define <1 x i64> @sdiv_v1i64(<1 x i64> %op1, <1 x i64> %op2) #0 {
 ; CHECK-LABEL: sdiv_v1i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl1
 ; CHECK: sdiv z0.d, [[PG]]/m, z0.d, z1.d
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v1i64:
+; VBITS_EQ_128:         ptrue p0.d, vl1
+; VBITS_EQ_128-NEXT:    sdiv z0.d, p0/m, z0.d, z1.d
+; VBITS_EQ_128-NEXT:    ret
+
   %res = sdiv <1 x i64> %op1, %op2
   ret <1 x i64> %res
 }
 
 ; Vector i64 sdiv are not legal for NEON so use SVE when available.
 define <2 x i64> @sdiv_v2i64(<2 x i64> %op1, <2 x i64> %op2) #0 {
 ; CHECK-LABEL: sdiv_v2i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl2
 ; CHECK: sdiv z0.d, [[PG]]/m, z0.d, z1.d
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v2i64:
+; VBITS_EQ_128:         ptrue p0.d, vl2
+; VBITS_EQ_128-NEXT:    sdiv z0.d, p0/m, z0.d, z1.d
+; VBITS_EQ_128-NEXT:    ret
+
   %res = sdiv <2 x i64> %op1, %op2
   ret <2 x i64> %res
 }
 
 define void @sdiv_v4i64(<4 x i64>* %a, <4 x i64>* %b) #0 {
 ; CHECK-LABEL: sdiv_v4i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl4
 ; CHECK-NEXT: ld1d { [[OP1:z[0-9]+]].d }, [[PG]]/z, [x0]
 ; CHECK-NEXT: ld1d { [[OP2:z[0-9]+]].d }, [[PG]]/z, [x1]
 ; CHECK-NEXT: sdiv [[RES:z[0-9]+]].d, [[PG]]/m, [[OP1]].d, [[OP2]].d
 ; CHECK-NEXT: st1d { [[RES]].d }, [[PG]], [x0]
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: sdiv_v4i64:
+; VBITS_EQ_128:         ldp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ptrue p0.d, vl2
+; VBITS_EQ_128-NEXT:    ldp q2, q3, [x1]
+; VBITS_EQ_128-NEXT:    sdiv z0.d, p0/m, z0.d, z2.d
+; VBITS_EQ_128-NEXT:    sdiv z1.d, p0/m, z1.d, z3.d
+; VBITS_EQ_128-NEXT:    stp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <4 x i64>, <4 x i64>* %a
   %op2 = load <4 x i64>, <4 x i64>* %b
   %res = sdiv <4 x i64> %op1, %op2
   store <4 x i64> %res, <4 x i64>* %a
   ret void
 }
 
 define void @sdiv_v8i64(<8 x i64>* %a, <8 x i64>* %b) #0 {
; VBITS_GE_2048-NEXT: ret
   ret void
 }
 
 ;
 ; UDIV
 ;
 
 ; Vector vXi8 udiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for >= 256 bits here.
 define <8 x i8> @udiv_v8i8(<8 x i8> %op1, <8 x i8> %op2) #0 {
 ; CHECK-LABEL: udiv_v8i8:
 ; CHECK: ptrue [[PG0:p[0-9]+]].s, vl8
 ; CHECK-NEXT: uunpklo [[OP2_LO:z[0-9]+]].h, z1.b
 ; CHECK-NEXT: uunpklo [[OP1_LO:z[0-9]+]].h, z0.b
 ; CHECK-NEXT: uunpklo [[OP2_LO_LO:z[0-9]+]].s, z1.h
 ; CHECK-NEXT: uunpklo [[OP1_LO_LO:z[0-9]+]].s, z0.h
 ; CHECK-NEXT: udiv [[DIV:z[0-9]+]].s, [[PG0]]/m, [[OP1_LO_LO]].s, [[OP2_LO_LO]].s

 ; CHECK-NEXT: mov [[FINAL]].b[4], [[SCALAR4]]
 ; CHECK-NEXT: umov [[SCALAR5:w[0-9]+]], [[VEC]].h[5]
 ; CHECK-NEXT: mov [[FINAL]].b[5], [[SCALAR5]]
 ; CHECK-NEXT: umov [[SCALAR6:w[0-9]+]], [[VEC]].h[6]
 ; CHECK-NEXT: mov [[FINAL]].b[6], [[SCALAR6]]
 ; CHECK-NEXT: umov [[SCALAR7:w[0-9]+]], [[VEC]].h[7]
 ; CHECK-NEXT: mov [[FINAL]].b[7], [[SCALAR7]]
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: udiv_v8i8:
+; VBITS_EQ_128:         ushll v1.8h, v1.8b, #0
+; VBITS_EQ_128-NEXT:    ushll v0.8h, v0.8b, #0
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    uunpkhi z2.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpkhi z3.s, z0.h
+; VBITS_EQ_128-NEXT:    uunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpklo z0.s, z0.h
+; VBITS_EQ_128-NEXT:    udivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    udiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    uzp1 z0.h, z0.h, z2.h
+; VBITS_EQ_128-NEXT:    xtn v0.8b, v0.8h
+; VBITS_EQ_128-NEXT:    ret
+
   %res = udiv <8 x i8> %op1, %op2
   ret <8 x i8> %res
 }
 
 define <16 x i8> @udiv_v16i8(<16 x i8> %op1, <16 x i8> %op2) #0 {
 ; CHECK-LABEL: udiv_v16i8:
 
 ; HALF VECTOR:

 ; VBITS_GE_512-NEXT: uunpklo [[OP2_LO:z[0-9]+]].h, z1.b
 ; VBITS_GE_512-NEXT: uunpklo [[OP1_LO:z[0-9]+]].h, z0.b
 ; VBITS_GE_512-NEXT: uunpklo [[OP2_LO_LO:z[0-9]+]].s, z1.h
 ; VBITS_GE_512-NEXT: uunpklo [[OP1_LO_LO:z[0-9]+]].s, z0.h
 ; VBITS_GE_512-NEXT: udiv [[DIV:z[0-9]+]].s, [[PG]]/m, [[OP1_LO_LO]].s, [[OP2_LO_LO]].s
 ; VBITS_GE_512-NEXT: uzp1 [[RES1:z[0-9]+]].h, [[DIV]].h, [[DIV]].h
 ; VBITS_GE_512-NEXT: uzp1 [[RES2:z[0-9]+]].b, [[RES1]].b, [[RES1]].b
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: udiv_v16i8:
+; VBITS_EQ_128:         uunpkhi z2.h, z1.b
+; VBITS_EQ_128-NEXT:    uunpkhi z3.h, z0.b
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    uunpklo z1.h, z1.b
+; VBITS_EQ_128-NEXT:    uunpkhi z4.s, z2.h
+; VBITS_EQ_128-NEXT:    uunpkhi z5.s, z3.h
+; VBITS_EQ_128-NEXT:    uunpklo z2.s, z2.h
+; VBITS_EQ_128-NEXT:    uunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    uunpklo z0.h, z0.b
+; VBITS_EQ_128-NEXT:    udivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    udivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    uunpkhi z3.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpkhi z5.s, z0.h
+; VBITS_EQ_128-NEXT:    uunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpklo z0.s, z0.h
+; VBITS_EQ_128-NEXT:    udivr z3.s, p0/m, z3.s, z5.s
+; VBITS_EQ_128-NEXT:    udiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    uzp1 z1.h, z2.h, z4.h
+; VBITS_EQ_128-NEXT:    uzp1 z0.h, z0.h, z3.h
+; VBITS_EQ_128-NEXT:    uzp1 z0.b, z0.b, z1.b
+; VBITS_EQ_128-NEXT:    ret
+
   %res = udiv <16 x i8> %op1, %op2
   ret <16 x i8> %res
 }
 
 define void @udiv_v32i8(<32 x i8>* %a, <32 x i8>* %b) #0 {
 ; CHECK-LABEL: udiv_v32i8:
 
 ; FULL VECTOR:

 ; VBITS_GE_1024-NEXT: uunpklo [[OP1_LO:z[0-9]+]].h, [[OP1]].b
 ; VBITS_GE_1024-NEXT: uunpklo [[OP2_LO_LO:z[0-9]+]].s, [[OP2_LO]].h
 ; VBITS_GE_1024-NEXT: uunpklo [[OP1_LO_LO:z[0-9]+]].s, [[OP1_LO]].h
 ; VBITS_GE_1024-NEXT: udiv [[DIV1:z[0-9]+]].s, [[PG2]]/m, [[OP1_LO_LO]].s, [[OP2_LO_LO]].s
 ; VBITS_GE_1024-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; VBITS_GE_1024-NEXT: uzp1 [[UZP2:z[0-9]+]].b, [[UZP1]].b, [[UZP1]].b
 ; VBITS_GE_1024-NEXT: st1b { [[UZP2:z[0-9]+]].b }, [[PG1]], [x0]
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: udiv_v32i8:
+; VBITS_EQ_128:         ldp q3, q0, [x1]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    ldp q2, q1, [x0]
+; VBITS_EQ_128-NEXT:    uunpkhi z4.h, z0.b
+; VBITS_EQ_128-NEXT:    uunpklo z0.h, z0.b
+; VBITS_EQ_128-NEXT:    uunpkhi z6.s, z4.h
+; VBITS_EQ_128-NEXT:    uunpklo z4.s, z4.h
+; VBITS_EQ_128-NEXT:    uunpkhi z16.s, z0.h
+; VBITS_EQ_128-NEXT:    uunpklo z0.s, z0.h
+; VBITS_EQ_128-NEXT:    uunpkhi z5.h, z1.b
+; VBITS_EQ_128-NEXT:    uunpklo z1.h, z1.b
+; VBITS_EQ_128-NEXT:    uunpkhi z7.s, z5.h
+; VBITS_EQ_128-NEXT:    uunpklo z5.s, z5.h
+; VBITS_EQ_128-NEXT:    udivr z6.s, p0/m, z6.s, z7.s
+; VBITS_EQ_128-NEXT:    udivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    uunpkhi z5.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    uzp1 z4.h, z4.h, z6.h
+; VBITS_EQ_128-NEXT:    udivr z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    uunpkhi z1.h, z3.b
+; VBITS_EQ_128-NEXT:    uunpkhi z6.h, z2.b
+; VBITS_EQ_128-NEXT:    udiv z5.s, p0/m, z5.s, z16.s
+; VBITS_EQ_128-NEXT:    uunpkhi z7.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpkhi z16.s, z6.h
+; VBITS_EQ_128-NEXT:    uunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpklo z6.s, z6.h
+; VBITS_EQ_128-NEXT:    uunpklo z3.h, z3.b
+; VBITS_EQ_128-NEXT:    uunpklo z2.h, z2.b
+; VBITS_EQ_128-NEXT:    udivr z7.s, p0/m, z7.s, z16.s
+; VBITS_EQ_128-NEXT:    udivr z1.s, p0/m, z1.s, z6.s
+; VBITS_EQ_128-NEXT:    uunpkhi z6.s, z3.h
+; VBITS_EQ_128-NEXT:    uunpkhi z16.s, z2.h
+; VBITS_EQ_128-NEXT:    uunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    uunpklo z2.s, z2.h
+; VBITS_EQ_128-NEXT:    udivr z6.s, p0/m, z6.s, z16.s
+; VBITS_EQ_128-NEXT:    udiv z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    uzp1 z1.h, z1.h, z7.h
+; VBITS_EQ_128-NEXT:    uzp1 z2.h, z2.h, z6.h
+; VBITS_EQ_128-NEXT:    uzp1 z0.h, z0.h, z5.h
+; VBITS_EQ_128-NEXT:    uzp1 z1.b, z2.b, z1.b
+; VBITS_EQ_128-NEXT:    uzp1 z0.b, z0.b, z4.b
+; VBITS_EQ_128-NEXT:    stp q1, q0, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <32 x i8>, <32 x i8>* %a
   %op2 = load <32 x i8>, <32 x i8>* %b
   %res = udiv <32 x i8> %op1, %op2
   store <32 x i8> %res, <32 x i8>* %a
   ret void
 }
 
 define void @udiv_v64i8(<64 x i8>* %a, <64 x i8>* %b) #0 {
; CHECK: ret
   %op1 = load <256 x i8>, <256 x i8>* %a
   %op2 = load <256 x i8>, <256 x i8>* %b
   %res = udiv <256 x i8> %op1, %op2
   store <256 x i8> %res, <256 x i8>* %a
   ret void
 }
 
 ; Vector vXi16 udiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for >= 256 bits here.
 define <4 x i16> @udiv_v4i16(<4 x i16> %op1, <4 x i16> %op2) #0 {
 ; CHECK-LABEL: udiv_v4i16:
 ; CHECK: ushll v1.4s, v1.4h, #0
 ; CHECK-NEXT: ushll v0.4s, v0.4h, #0
 ; CHECK-NEXT: ptrue [[PG1:p[0-9]+]].s, vl4
 ; CHECK-NEXT: udivr [[DIV1:z[0-9]+]].s, [[PG1]]/m, [[OP2:z[0-9]+]].s, [[OP1:z[0-9]+]].s
 ; CHECK-NEXT: mov w8, v1.s[1]
 ; CHECK-NEXT: mov w9, v1.s[2]
 ; CHECK-NEXT: mov v0.16b, v1.16b
 ; CHECK-NEXT: mov v0.h[1], w8
 ; CHECK-NEXT: mov w8, v1.s[3]
 ; CHECK-NEXT: mov v0.h[2], w9
 ; CHECK-NEXT: mov v0.h[3], w8
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: udiv_v4i16:
+; VBITS_EQ_128:         ushll v1.4s, v1.4h, #0
+; VBITS_EQ_128-NEXT:    ushll v0.4s, v0.4h, #0
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    udiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    xtn v0.4h, v0.4s
+; VBITS_EQ_128-NEXT:    ret
+
   %res = udiv <4 x i16> %op1, %op2
   ret <4 x i16> %res
 }
 
 define <8 x i16> @udiv_v8i16(<8 x i16> %op1, <8 x i16> %op2) #0 {
 ; CHECK-LABEL: udiv_v8i16:
 ; CHECK: ptrue [[PG1:p[0-9]+]].s, vl8
 ; CHECK-NEXT: uunpklo [[OP2_LO:z[0-9]+]].s, [[OP2]].h
 ; CHECK-NEXT: uunpklo [[OP1_LO:z[0-9]+]].s, [[OP1]].h
 ; CHECK-NEXT: udiv [[DIV1:z[0-9]+]].s, [[PG1]]/m, [[OP1_LO]].s, [[OP2_LO]].s
 ; CHECK-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: udiv_v8i16:
+; VBITS_EQ_128:         ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    uunpkhi z2.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpkhi z3.s, z0.h
+; VBITS_EQ_128-NEXT:    uunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpklo z0.s, z0.h
+; VBITS_EQ_128-NEXT:    udivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    udiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    uzp1 z0.h, z0.h, z2.h
+; VBITS_EQ_128-NEXT:    ret
+
   %res = udiv <8 x i16> %op1, %op2
   ret <8 x i16> %res
 }
 
 define void @udiv_v16i16(<16 x i16>* %a, <16 x i16>* %b) #0 {
 ; CHECK-LABEL: udiv_v16i16:
 
 ; FULL VECTOR:

 ; VBITS_GE_512-NEXT: ld1h { [[OP1:z[0-9]+]].h }, [[PG1]]/z, [x0]
 ; VBITS_GE_512-NEXT: ld1h { [[OP2:z[0-9]+]].h }, [[PG1]]/z, [x1]
 ; VBITS_GE_512-NEXT: uunpklo [[OP2_LO:z[0-9]+]].s, [[OP2]].h
 ; VBITS_GE_512-NEXT: uunpklo [[OP1_LO:z[0-9]+]].s, [[OP1]].h
 ; VBITS_GE_512-NEXT: udiv [[DIV1:z[0-9]+]].s, [[PG2]]/m, [[OP1_LO]].s, [[OP2_LO]].s
 ; VBITS_GE_512-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; VBITS_GE_512-NEXT: st1h { [[UZP1:z[0-9]+]].h }, [[PG1]], [x0]
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: udiv_v16i16:
+; VBITS_EQ_128:         ldp q0, q1, [x1]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    uunpkhi z6.s, z0.h
+; VBITS_EQ_128-NEXT:    uunpklo z0.s, z0.h
+; VBITS_EQ_128-NEXT:    ldp q3, q2, [x0]
+; VBITS_EQ_128-NEXT:    uunpkhi z4.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpklo z1.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpkhi z5.s, z2.h
+; VBITS_EQ_128-NEXT:    uunpklo z2.s, z2.h
+; VBITS_EQ_128-NEXT:    udivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    uunpkhi z5.s, z3.h
+; VBITS_EQ_128-NEXT:    uunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    udiv z5.s, p0/m, z5.s, z6.s
+; VBITS_EQ_128-NEXT:    udivr z0.s, p0/m, z0.s, z3.s
+; VBITS_EQ_128-NEXT:    udivr z1.s, p0/m, z1.s, z2.s
+; VBITS_EQ_128-NEXT:    uzp1 z0.h, z0.h, z5.h
+; VBITS_EQ_128-NEXT:    uzp1 z1.h, z1.h, z4.h
+; VBITS_EQ_128-NEXT:    stp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <16 x i16>, <16 x i16>* %a
   %op2 = load <16 x i16>, <16 x i16>* %b
   %res = udiv <16 x i16> %op1, %op2
   store <16 x i16> %res, <16 x i16>* %a
   ret void
 }
 
 define void @udiv_v32i16(<32 x i16>* %a, <32 x i16>* %b) #0 {

 }
 
 ; Vector v2i32 udiv are not legal for NEON so use SVE when available.
 define <2 x i32> @udiv_v2i32(<2 x i32> %op1, <2 x i32> %op2) #0 {
 ; CHECK-LABEL: udiv_v2i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl2
 ; CHECK: udiv z0.s, [[PG]]/m, z0.s, z1.s
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: udiv_v2i32:
+; VBITS_EQ_128:         ptrue p0.s, vl2
+; VBITS_EQ_128-NEXT:    udiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    ret
+
   %res = udiv <2 x i32> %op1, %op2
   ret <2 x i32> %res
 }
 
 ; Vector v4i32 udiv are not legal for NEON so use SVE when available.
 define <4 x i32> @udiv_v4i32(<4 x i32> %op1, <4 x i32> %op2) #0 {
 ; CHECK-LABEL: udiv_v4i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl4
 ; CHECK: udiv z0.s, [[PG]]/m, z0.s, z1.s
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: udiv_v4i32:
+; VBITS_EQ_128:         ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    udiv z0.s, p0/m, z0.s, z1.s
+; VBITS_EQ_128-NEXT:    ret
+
   %res = udiv <4 x i32> %op1, %op2
   ret <4 x i32> %res
 }
 
 define void @udiv_v8i32(<8 x i32>* %a, <8 x i32>* %b) #0 {
 ; CHECK-LABEL: udiv_v8i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl8
 ; CHECK-NEXT: ld1w { [[OP1:z[0-9]+]].s }, [[PG]]/z, [x0]
 ; CHECK-NEXT: ld1w { [[OP2:z[0-9]+]].s }, [[PG]]/z, [x1]
 ; CHECK-NEXT: udiv [[RES:z[0-9]+]].s, [[PG]]/m, [[OP1]].s, [[OP2]].s
 ; CHECK-NEXT: st1w { [[RES]].s }, [[PG]], [x0]
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: udiv_v8i32:
+; VBITS_EQ_128:         ldp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    ldp q2, q3, [x1]
+; VBITS_EQ_128-NEXT:    udiv z0.s, p0/m, z0.s, z2.s
+; VBITS_EQ_128-NEXT:    udiv z1.s, p0/m, z1.s, z3.s
+; VBITS_EQ_128-NEXT:    stp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <8 x i32>, <8 x i32>* %a
   %op2 = load <8 x i32>, <8 x i32>* %b
   %res = udiv <8 x i32> %op1, %op2
   store <8 x i32> %res, <8 x i32>* %a
   ret void
 }
 
 define void @udiv_v16i32(<16 x i32>* %a, <16 x i32>* %b) #0 {

 }
 
 ; Vector i64 udiv are not legal for NEON so use SVE when available.
 define <1 x i64> @udiv_v1i64(<1 x i64> %op1, <1 x i64> %op2) #0 {
 ; CHECK-LABEL: udiv_v1i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl1
 ; CHECK: udiv z0.d, [[PG]]/m, z0.d, z1.d
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: udiv_v1i64:
+; VBITS_EQ_128:         ptrue p0.d, vl1
+; VBITS_EQ_128-NEXT:    udiv z0.d, p0/m, z0.d, z1.d
+; VBITS_EQ_128-NEXT:    ret
+
   %res = udiv <1 x i64> %op1, %op2
   ret <1 x i64> %res
 }
 
 ; Vector i64 udiv are not legal for NEON so use SVE when available.
 define <2 x i64> @udiv_v2i64(<2 x i64> %op1, <2 x i64> %op2) #0 {
 ; CHECK-LABEL: udiv_v2i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl2
 ; CHECK: udiv z0.d, [[PG]]/m, z0.d, z1.d
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: udiv_v2i64:
+; VBITS_EQ_128:         ptrue p0.d, vl2
+; VBITS_EQ_128-NEXT:    udiv z0.d, p0/m, z0.d, z1.d
+; VBITS_EQ_128-NEXT:    ret
+
   %res = udiv <2 x i64> %op1, %op2
   ret <2 x i64> %res
 }
 
 define void @udiv_v4i64(<4 x i64>* %a, <4 x i64>* %b) #0 {
 ; CHECK-LABEL: udiv_v4i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl4
 ; CHECK-NEXT: ld1d { [[OP1:z[0-9]+]].d }, [[PG]]/z, [x0]
 ; CHECK-NEXT: ld1d { [[OP2:z[0-9]+]].d }, [[PG]]/z, [x1]
 ; CHECK-NEXT: udiv [[RES:z[0-9]+]].d, [[PG]]/m, [[OP1]].d, [[OP2]].d
 ; CHECK-NEXT: st1d { [[RES]].d }, [[PG]], [x0]
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: udiv_v4i64:
+; VBITS_EQ_128:         ldp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ptrue p0.d, vl2
+; VBITS_EQ_128-NEXT:    ldp q2, q3, [x1]
+; VBITS_EQ_128-NEXT:    udiv z0.d, p0/m, z0.d, z2.d
+; VBITS_EQ_128-NEXT:    udiv z1.d, p0/m, z1.d, z3.d
+; VBITS_EQ_128-NEXT:    stp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <4 x i64>, <4 x i64>* %a
   %op2 = load <4 x i64>, <4 x i64>* %b
   %res = udiv <4 x i64> %op1, %op2
   store <4 x i64> %res, <4 x i64>* %a
   ret void
 }
 
 define void @udiv_v8i64(<8 x i64>* %a, <8 x i64>* %b) #0 {

llvm/test/CodeGen/AArch64/sve-fixed-length-int-rem.ll

-; RUN: llc -aarch64-sve-vector-bits-min=128  -asm-verbose=0 < %s | FileCheck %s -check-prefix=NO_SVE
+; RUN: llc -aarch64-sve-vector-bits-min=128  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=VBITS_EQ_128
 ; RUN: llc -aarch64-sve-vector-bits-min=256  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_EQ_256
 ; RUN: llc -aarch64-sve-vector-bits-min=384  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_EQ_256
 ; RUN: llc -aarch64-sve-vector-bits-min=512  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_EQ_512
 ; RUN: llc -aarch64-sve-vector-bits-min=640  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_EQ_512
 ; RUN: llc -aarch64-sve-vector-bits-min=768  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_EQ_512
 ; RUN: llc -aarch64-sve-vector-bits-min=896  -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_EQ_512
 ; RUN: llc -aarch64-sve-vector-bits-min=1024 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1152 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1280 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1408 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1536 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1664 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1792 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1920 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_EQ_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=2048 -asm-verbose=0 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_GE_2048,VBITS_EQ_2048
 
 target triple = "aarch64-unknown-linux-gnu"
 
-; Don't use SVE when its registers are no bigger than NEON.
-; NO_SVE-NOT: ptrue
-
 ;
 ; SREM
 ;
 
 ; Vector vXi8 sdiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for >= 256 bits here.
 define <8 x i8> @srem_v8i8(<8 x i8> %op1, <8 x i8> %op2) #0 {
 ; CHECK-LABEL: srem_v8i8:
 ; CHECK: sunpklo [[OP2_LO:z[0-9]+]].h, [[OP2:z[0-9]+]].b
 ; CHECK-NEXT: sunpklo [[OP1_LO:z[0-9]+]].h, [[OP1:z[0-9]+]].b
 ; CHECK-NEXT: ptrue [[PG1:p[0-9]+]].s, vl8
 ; CHECK-NEXT: sunpklo [[OP2_LO_LO:z[0-9]+]].s, [[OP2_LO]].h
 ; CHECK-NEXT: sunpklo [[OP1_LO_LO:z[0-9]+]].s, [[OP1_LO]].h
 ; CHECK-NEXT: sdivr [[DIV1:z[0-9]+]].s, [[PG1]]/m, [[OP2_LO_LO]].s, [[OP1_LO_LO]].s

 ; CHECK-NEXT: umov [[SCALAR6:w[0-9]+]], [[VEC]].h[5]
 ; CHECK-NEXT: mov [[FINAL]].b[5], [[SCALAR6]]
 ; CHECK-NEXT: umov [[SCALAR7:w[0-9]+]], [[VEC]].h[6]
 ; CHECK-NEXT: mov [[FINAL]].b[6], [[SCALAR7]]
 ; CHECK-NEXT: umov [[SCALAR8:w[0-9]+]], [[VEC]].h[7]
 ; CHECK-NEXT: mov [[FINAL]].b[7], [[SCALAR8]]
 ; CHECK-NEXT: mls v0.8b, [[FINAL]].8b, v1.8b
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: srem_v8i8:
+; VBITS_EQ_128:         sshll v2.8h, v1.8b, #0
+; VBITS_EQ_128-NEXT:    sshll v3.8h, v0.8b, #0
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    sunpkhi z4.s, z2.h
+; VBITS_EQ_128-NEXT:    sunpkhi z5.s, z3.h
+; VBITS_EQ_128-NEXT:    sunpklo z2.s, z2.h
+; VBITS_EQ_128-NEXT:    sunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    sdivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    sdivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    uzp1 z2.h, z2.h, z4.h
+; VBITS_EQ_128-NEXT:    xtn v2.8b, v2.8h
+; VBITS_EQ_128-NEXT:    mls v0.8b, v2.8b, v1.8b
+; VBITS_EQ_128-NEXT:    ret
+
   %res = srem <8 x i8> %op1, %op2
   ret <8 x i8> %res
 }
 
 define <16 x i8> @srem_v16i8(<16 x i8> %op1, <16 x i8> %op2) #0 {
 ; CHECK-LABEL: srem_v16i8:
 
 ; HALF VECTOR

 ; VBITS_GE_512-NEXT: ptrue [[PG1:p[0-9]+]].s, vl16
 ; VBITS_GE_512-NEXT: sunpklo [[OP2_LO_LO:z[0-9]+]].s, [[OP2_LO]].h
 ; VBITS_GE_512-NEXT: sunpklo [[OP1_LO_LO:z[0-9]+]].s, [[OP1_LO]].h
 ; VBITS_GE_512-NEXT: sdivr [[DIV1:z[0-9]+]].s, [[PG1]]/m, [[OP2_LO_LO]].s, [[OP1_LO_LO]].s
 ; VBITS_GE_512-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; VBITS_GE_512-NEXT: uzp1 [[UZP2:z[0-9]+]].b, [[UZP1]].b, [[UZP1]].b
 ; VBITS_GE_512-NEXT: mls v0.16b, v2.16b, v1.16b
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: srem_v16i8:
+; VBITS_EQ_128:         sunpkhi z2.h, z1.b
+; VBITS_EQ_128-NEXT:    sunpkhi z3.h, z0.b
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    sunpkhi z5.s, z2.h
+; VBITS_EQ_128-NEXT:    sunpkhi z6.s, z3.h
+; VBITS_EQ_128-NEXT:    sunpklo z2.s, z2.h
+; VBITS_EQ_128-NEXT:    sunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    sunpklo z4.h, z1.b
+; VBITS_EQ_128-NEXT:    sdivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    sunpklo z3.h, z0.b
+; VBITS_EQ_128-NEXT:    sdivr z5.s, p0/m, z5.s, z6.s
+; VBITS_EQ_128-NEXT:    sunpkhi z6.s, z4.h
+; VBITS_EQ_128-NEXT:    sunpkhi z7.s, z3.h
+; VBITS_EQ_128-NEXT:    sunpklo z4.s, z4.h
+; VBITS_EQ_128-NEXT:    sunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    sdivr z6.s, p0/m, z6.s, z7.s
+; VBITS_EQ_128-NEXT:    sdiv z3.s, p0/m, z3.s, z4.s
+; VBITS_EQ_128-NEXT:    uzp1 z2.h, z2.h, z5.h
+; VBITS_EQ_128-NEXT:    uzp1 z3.h, z3.h, z6.h
+; VBITS_EQ_128-NEXT:    uzp1 z2.b, z3.b, z2.b
+; VBITS_EQ_128-NEXT:    mls v0.16b, v2.16b, v1.16b
+; VBITS_EQ_128-NEXT:    ret
+
   %res = srem <16 x i8> %op1, %op2
   ret <16 x i8> %res
 }
 
 define void @srem_v32i8(<32 x i8>* %a, <32 x i8>* %b) #0 {
 ; CHECK-LABEL: srem_v32i8:
 
 ; FULL VECTOR:

 ; VBITS_GE_1024-NEXT: sunpklo [[OP1_LO_LO:z[0-9]+]].s, [[OP1_LO]].h
 ; VBITS_GE_1024-NEXT: sdivr [[DIV1:z[0-9]+]].s, [[PG2]]/m, [[OP2_LO_LO]].s, [[OP1_LO_LO]].s
 ; VBITS_GE_1024-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; VBITS_GE_1024-NEXT: uzp1 [[UZP2:z[0-9]+]].b, [[UZP1]].b, [[UZP1]].b
 ; VBITS_GE_1024-NEXT: mul [[OP2]].b, [[PG1]]/m, [[OP2]].b, [[UZP2]].b
 ; VBITS_GE_1024-NEXT: sub [[OP1]].b, [[PG1]]/m, [[OP1]].b, [[OP2]].b
 ; VBITS_GE_1024-NEXT: st1b { [[OP1:z[0-9]+]].b }, [[PG1]], [x0]
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: srem_v32i8:
+; VBITS_EQ_128:         ldp q2, q0, [x0]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    ldp q3, q1, [x1]
+; VBITS_EQ_128-NEXT:    sunpkhi z5.h, z0.b
+; VBITS_EQ_128-NEXT:    sunpklo z7.h, z0.b
+; VBITS_EQ_128-NEXT:    sunpkhi z17.s, z5.h
+; VBITS_EQ_128-NEXT:    sunpklo z5.s, z5.h
+; VBITS_EQ_128-NEXT:    sunpkhi z4.h, z1.b
+; VBITS_EQ_128-NEXT:    sunpklo z6.h, z1.b
+; VBITS_EQ_128-NEXT:    sunpkhi z16.s, z4.h
+; VBITS_EQ_128-NEXT:    sunpklo z4.s, z4.h
+; VBITS_EQ_128-NEXT:    sunpkhi z18.s, z6.h
+; VBITS_EQ_128-NEXT:    sdivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    sunpkhi z5.s, z7.h
+; VBITS_EQ_128-NEXT:    sunpklo z6.s, z6.h
+; VBITS_EQ_128-NEXT:    sunpklo z7.s, z7.h
+; VBITS_EQ_128-NEXT:    sdiv z5.s, p0/m, z5.s, z18.s
+; VBITS_EQ_128-NEXT:    sdivr z6.s, p0/m, z6.s, z7.s
+; VBITS_EQ_128-NEXT:    sdivr z16.s, p0/m, z16.s, z17.s
+; VBITS_EQ_128-NEXT:    uzp1 z5.h, z6.h, z5.h
+; VBITS_EQ_128-NEXT:    sunpkhi z6.h, z3.b
+; VBITS_EQ_128-NEXT:    sunpkhi z7.h, z2.b
+; VBITS_EQ_128-NEXT:    uzp1 z4.h, z4.h, z16.h
+; VBITS_EQ_128-NEXT:    sunpkhi z16.s, z6.h
+; VBITS_EQ_128-NEXT:    sunpkhi z17.s, z7.h
+; VBITS_EQ_128-NEXT:    sunpklo z6.s, z6.h
+; VBITS_EQ_128-NEXT:    sunpklo z7.s, z7.h
+; VBITS_EQ_128-NEXT:    sdivr z16.s, p0/m, z16.s, z17.s
+; VBITS_EQ_128-NEXT:    sdivr z6.s, p0/m, z6.s, z7.s
+; VBITS_EQ_128-NEXT:    sunpklo z7.h, z3.b
+; VBITS_EQ_128-NEXT:    sunpklo z17.h, z2.b
+; VBITS_EQ_128-NEXT:    sunpkhi z18.s, z7.h
+; VBITS_EQ_128-NEXT:    sunpkhi z19.s, z17.h
+; VBITS_EQ_128-NEXT:    sunpklo z7.s, z7.h
+; VBITS_EQ_128-NEXT:    sunpklo z17.s, z17.h
+; VBITS_EQ_128-NEXT:    sdivr z18.s, p0/m, z18.s, z19.s
+; VBITS_EQ_128-NEXT:    sdivr z7.s, p0/m, z7.s, z17.s
+; VBITS_EQ_128-NEXT:    uzp1 z6.h, z6.h, z16.h
+; VBITS_EQ_128-NEXT:    uzp1 z7.h, z7.h, z18.h
+; VBITS_EQ_128-NEXT:    uzp1 z4.b, z5.b, z4.b
+; VBITS_EQ_128-NEXT:    uzp1 z5.b, z7.b, z6.b
+; VBITS_EQ_128-NEXT:    mls v2.16b, v5.16b, v3.16b
+; VBITS_EQ_128-NEXT:    mls v0.16b, v4.16b, v1.16b
+; VBITS_EQ_128-NEXT:    stp q2, q0, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <32 x i8>, <32 x i8>* %a
   %op2 = load <32 x i8>, <32 x i8>* %b
   %res = srem <32 x i8> %op1, %op2
   store <32 x i8> %res, <32 x i8>* %a
   ret void
 }
 
 define void @srem_v64i8(<64 x i8>* %a, <64 x i8>* %b) #0 {
; VBITS_EQ_2048-NEXT: ret
   %op1 = load <256 x i8>, <256 x i8>* %a
   %op2 = load <256 x i8>, <256 x i8>* %b
   %res = srem <256 x i8> %op1, %op2
   store <256 x i8> %res, <256 x i8>* %a
   ret void
 }
 
 ; Vector vXi16 sdiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for >= 256 bits here.
 define <4 x i16> @srem_v4i16(<4 x i16> %op1, <4 x i16> %op2) #0 {
 ; CHECK-LABEL: srem_v4i16:
 ; CHECK: sshll v2.4s, v1.4h, #0
 ; CHECK-NEXT: sshll v3.4s, v0.4h, #0
 ; CHECK-NEXT: ptrue [[PG1:p[0-9]+]].s, vl4
 ; CHECK-NEXT: sdivr [[DIV1:z[0-9]+]].s, [[PG1]]/m, z2.s, z3.s
 ; CHECK-NEXT: mov [[SCALAR1:w[0-9]+]], [[VEC:v[0-9]+]].s[1]
 ; CHECK-NEXT: mov [[SCALAR2:w[0-9]+]], [[VEC]].s[2]
 ; CHECK-NEXT: mov [[VEC2:v[0-9]+]].16b, [[VEC]].16b
 ; CHECK-NEXT: mov [[VEC2]].h[1], [[SCALAR1]]
 ; CHECK-NEXT: mov [[SCALAR3:w[0-9]+]], [[VEC]].s[3]
 ; CHECK-NEXT: mov [[VEC2]].h[2], [[SCALAR2]]
 ; CHECK-NEXT: mov [[VEC2]].h[3], [[SCALAR3]]
 ; CHECK-NEXT: mls v0.4h, [[VEC2]].4h, v1.4h
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: srem_v4i16:
+; VBITS_EQ_128:         sshll v2.4s, v1.4h, #0
+; VBITS_EQ_128-NEXT:    sshll v3.4s, v0.4h, #0
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    sdivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    xtn v2.4h, v2.4s
+; VBITS_EQ_128-NEXT:    mls v0.4h, v2.4h, v1.4h
+; VBITS_EQ_128-NEXT:    ret
+
   %res = srem <4 x i16> %op1, %op2
   ret <4 x i16> %res
 }
 
 define <8 x i16> @srem_v8i16(<8 x i16> %op1, <8 x i16> %op2) #0 {
 ; CHECK-LABEL: srem_v8i16:
 ; CHECK: ptrue [[PG1:p[0-9]+]].s, vl8
 ; CHECK-NEXT: sunpklo [[OP2_LO:z[0-9]+]].s, [[OP2]].h
 ; CHECK-NEXT: sunpklo [[OP1_LO:z[0-9]+]].s, [[OP1]].h
 ; CHECK-NEXT: sdivr [[DIV1:z[0-9]+]].s, [[PG1]]/m, [[OP2_LO]].s, [[OP1_LO]].s
 ; CHECK-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; CHECK-NEXT: mls v0.8h, v2.8h, v1.8h
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: srem_v8i16:
+; VBITS_EQ_128:         ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    sunpkhi z2.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpkhi z3.s, z0.h
+; VBITS_EQ_128-NEXT:    sunpklo z4.s, z1.h
+; VBITS_EQ_128-NEXT:    sdivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    sunpklo z5.s, z0.h
+; VBITS_EQ_128-NEXT:    movprfx z3, z5
+; VBITS_EQ_128-NEXT:    sdiv z3.s, p0/m, z3.s, z4.s
+; VBITS_EQ_128-NEXT:    uzp1 z2.h, z3.h, z2.h
+; VBITS_EQ_128-NEXT:    mls v0.8h, v2.8h, v1.8h
+; VBITS_EQ_128-NEXT:    ret
+
   %res = srem <8 x i16> %op1, %op2
   ret <8 x i16> %res
 }
 
 define void @srem_v16i16(<16 x i16>* %a, <16 x i16>* %b) #0 {
 ; CHECK-LABEL: srem_v16i16:
 
 ; FULL VECTOR:

 ; VBITS_GE_512-NEXT: sunpklo [[OP2_LO:z[0-9]+]].s, [[OP2]].h
 ; VBITS_GE_512-NEXT: sunpklo [[OP1_LO:z[0-9]+]].s, [[OP1]].h
 ; VBITS_GE_512-NEXT: sdivr [[DIV1:z[0-9]+]].s, [[PG2]]/m, [[OP2_LO]].s, [[OP1_LO]].s
 ; VBITS_GE_512-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; VBITS_GE_512-NEXT: mul [[OP2]].h, [[PG1]]/m, [[OP2]].h, [[UZP1]].h
 ; VBITS_GE_512-NEXT: sub [[OP1]].h, [[PG1]]/m, [[OP1]].h, [[OP2]].h
 ; VBITS_GE_512-NEXT: st1h { [[OP1:z[0-9]+]].h }, [[PG1]], [x0]
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: srem_v16i16:
+; VBITS_EQ_128:         ldp q2, q0, [x0]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    sunpkhi z17.s, z2.h
+; VBITS_EQ_128-NEXT:    ldp q3, q1, [x1]
+; VBITS_EQ_128-NEXT:    sunpkhi z5.s, z0.h
+; VBITS_EQ_128-NEXT:    sunpklo z7.s, z0.h
+; VBITS_EQ_128-NEXT:    sunpkhi z16.s, z3.h
+; VBITS_EQ_128-NEXT:    sdivr z16.s, p0/m, z16.s, z17.s
+; VBITS_EQ_128-NEXT:    sunpkhi z4.s, z1.h
+; VBITS_EQ_128-NEXT:    sunpklo z6.s, z1.h
+; VBITS_EQ_128-NEXT:    sdivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    sunpklo z5.s, z3.h
+; VBITS_EQ_128-NEXT:    sdivr z6.s, p0/m, z6.s, z7.s
+; VBITS_EQ_128-NEXT:    sunpklo z7.s, z2.h
+; VBITS_EQ_128-NEXT:    sdivr z5.s, p0/m, z5.s, z7.s
+; VBITS_EQ_128-NEXT:    uzp1 z4.h, z6.h, z4.h
+; VBITS_EQ_128-NEXT:    uzp1 z5.h, z5.h, z16.h
+; VBITS_EQ_128-NEXT:    mls v2.8h, v5.8h, v3.8h
+; VBITS_EQ_128-NEXT:    mls v0.8h, v4.8h, v1.8h
+; VBITS_EQ_128-NEXT:    stp q2, q0, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <16 x i16>, <16 x i16>* %a
   %op2 = load <16 x i16>, <16 x i16>* %b
   %res = srem <16 x i16> %op1, %op2
   store <16 x i16> %res, <16 x i16>* %a
   ret void
 }
 
 define void @srem_v32i16(<32 x i16>* %a, <32 x i16>* %b) #0 {

 ; Vector v2i32 sdiv are not legal for NEON so use SVE when available.
 define <2 x i32> @srem_v2i32(<2 x i32> %op1, <2 x i32> %op2) #0 {
 ; CHECK-LABEL: srem_v2i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl2
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], z0
 ; CHECK-NEXT: sdiv z2.s, [[PG]]/m, [[PFX]].s, z1.s
 ; CHECK-NEXT: mls v0.2s, v2.2s, v1.2s
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: srem_v2i32:
+; VBITS_EQ_128:         ptrue p0.s, vl2
+; VBITS_EQ_128-NEXT:    movprfx z2, z0
+; VBITS_EQ_128-NEXT:    sdiv z2.s, p0/m, z2.s, z1.s
+; VBITS_EQ_128-NEXT:    mls v0.2s, v2.2s, v1.2s
+; VBITS_EQ_128-NEXT:    ret
+
   %res = srem <2 x i32> %op1, %op2
   ret <2 x i32> %res
 }
 
 ; Vector v4i32 sdiv are not legal for NEON so use SVE when available.
 define <4 x i32> @srem_v4i32(<4 x i32> %op1, <4 x i32> %op2) #0 {
 ; CHECK-LABEL: srem_v4i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl4
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], z0
 ; CHECK-NEXT: sdiv z2.s, [[PG]]/m, [[PFX]].s, z1.s
 ; CHECK-NEXT: mls v0.4s, v2.4s, v1.4s
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: srem_v4i32:
+; VBITS_EQ_128:         ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    movprfx z2, z0
+; VBITS_EQ_128-NEXT:    sdiv z2.s, p0/m, z2.s, z1.s
+; VBITS_EQ_128-NEXT:    mls v0.4s, v2.4s, v1.4s
+; VBITS_EQ_128-NEXT:    ret
+
   %res = srem <4 x i32> %op1, %op2
   ret <4 x i32> %res
 }
 
 define void @srem_v8i32(<8 x i32>* %a, <8 x i32>* %b) #0 {
 ; CHECK-LABEL: srem_v8i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl8
 ; CHECK-NEXT: ld1w { [[OP1:z[0-9]+]].s }, [[PG]]/z, [x0]
 ; CHECK-NEXT: ld1w { [[OP2:z[0-9]+]].s }, [[PG]]/z, [x1]
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], [[OP1]]
 ; CHECK-NEXT: sdiv [[DIV:z[0-9]+]].s, [[PG]]/m, [[PFX]].s, [[OP2]].s
 ; CHECK-NEXT: mul [[MUL:z[0-9]+]].s, [[PG]]/m, [[OP2]].s, [[DIV]].s
 ; CHECK-NEXT: sub [[RES:z[0-9]+]].s, [[PG]]/m, [[OP1]].s, [[MUL]].s
 ; CHECK-NEXT: st1w { [[RES]].s }, [[PG]], [x0]
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: srem_v8i32:
+; VBITS_EQ_128:         ldp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    ldp q2, q3, [x1]
+; VBITS_EQ_128-NEXT:    movprfx z4, z1
+; VBITS_EQ_128-NEXT:    sdiv z4.s, p0/m, z4.s, z3.s
+; VBITS_EQ_128-NEXT:    movprfx z5, z0
+; VBITS_EQ_128-NEXT:    sdiv z5.s, p0/m, z5.s, z2.s
+; VBITS_EQ_128-NEXT:    mls v0.4s, v5.4s, v2.4s
+; VBITS_EQ_128-NEXT:    mls v1.4s, v4.4s, v3.4s
+; VBITS_EQ_128-NEXT:    stp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <8 x i32>, <8 x i32>* %a
   %op2 = load <8 x i32>, <8 x i32>* %b
   %res = srem <8 x i32> %op1, %op2
   store <8 x i32> %res, <8 x i32>* %a
   ret void
 }
 
 define void @srem_v16i32(<16 x i32>* %a, <16 x i32>* %b) #0 {
; VBITS_GE_2048-NEXT: ret
   %op1 = load <64 x i32>, <64 x i32>* %a
   %op2 = load <64 x i32>, <64 x i32>* %b
   %res = srem <64 x i32> %op1, %op2
   store <64 x i32> %res, <64 x i32>* %a
   ret void
 }
 
 ; Vector i64 sdiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for the 128 bits case here.
 define <1 x i64> @srem_v1i64(<1 x i64> %op1, <1 x i64> %op2) #0 {
 ; CHECK-LABEL: srem_v1i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl1
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], [[OP1]]
 ; CHECK-NEXT: sdiv [[DIV:z[0-9]+]].d, [[PG]]/m, [[PFX]].d, z1.d
 ; CHECK-NEXT: mul z1.d, [[PG]]/m, [[OP2]].d, [[DIV]].d
 ; CHECK-NEXT: sub d0, d0, d1
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: srem_v1i64:
+; VBITS_EQ_128:         ptrue p0.d, vl1
+; VBITS_EQ_128-NEXT:    movprfx z2, z0
+; VBITS_EQ_128-NEXT:    sdiv z2.d, p0/m, z2.d, z1.d
+; VBITS_EQ_128-NEXT:    fmov x8, d2
+; VBITS_EQ_128-NEXT:    fmov x9, d1
+; VBITS_EQ_128-NEXT:    mul x8, x8, x9
+; VBITS_EQ_128-NEXT:    fmov d1, x8
+; VBITS_EQ_128-NEXT:    sub d0, d0, d1
+; VBITS_EQ_128-NEXT:    ret
+
   %res = srem <1 x i64> %op1, %op2
   ret <1 x i64> %res
 }
 
 ; Vector i64 sdiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for the 128 bits case here.
 define <2 x i64> @srem_v2i64(<2 x i64> %op1, <2 x i64> %op2) #0 {
 ; CHECK-LABEL: srem_v2i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl2
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], [[OP1]]
 ; CHECK-NEXT: sdiv [[DIV:z[0-9]+]].d, [[PG]]/m, [[PFX]].d, z1.d
 ; CHECK-NEXT: mul z1.d, [[PG]]/m, [[OP2]].d, [[DIV]].d
 ; CHECK-NEXT: sub v0.2d, v0.2d, v1.2d
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: srem_v2i64:
+; VBITS_EQ_128:         ptrue p0.d, vl2
+; VBITS_EQ_128-NEXT:    movprfx z2, z0
+; VBITS_EQ_128-NEXT:    sdiv z2.d, p0/m, z2.d, z1.d
+; VBITS_EQ_128-NEXT:    fmov x9, d2
+; VBITS_EQ_128-NEXT:    fmov x10, d1
+; VBITS_EQ_128-NEXT:    mov x8, v2.d[1]
+; VBITS_EQ_128-NEXT:    mov x11, v1.d[1]
+; VBITS_EQ_128-NEXT:    mul x9, x9, x10
+; VBITS_EQ_128-NEXT:    mul x8, x8, x11
+; VBITS_EQ_128-NEXT:    fmov d1, x9
+; VBITS_EQ_128-NEXT:    mov v1.d[1], x8
+; VBITS_EQ_128-NEXT:    sub v0.2d, v0.2d, v1.2d
+; VBITS_EQ_128-NEXT:    ret
+
   %res = srem <2 x i64> %op1, %op2
   ret <2 x i64> %res
 }
 
 define void @srem_v4i64(<4 x i64>* %a, <4 x i64>* %b) #0 {
 ; CHECK-LABEL: srem_v4i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl4
 ; CHECK-NEXT: ld1d { [[OP1:z[0-9]+]].d }, [[PG]]/z, [x0]
 ; CHECK-NEXT: ld1d { [[OP2:z[0-9]+]].d }, [[PG]]/z, [x1]
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], [[OP1]]
 ; CHECK-NEXT: sdiv [[DIV:z[0-9]+]].d, [[PG]]/m, [[PFX]].d, [[OP2]].d
 ; CHECK-NEXT: mul [[MUL:z[0-9]+]].d, [[PG]]/m, [[OP2]].d, [[DIV]].d
 ; CHECK-NEXT: sub [[RES:z[0-9]+]].d, [[PG]]/m, [[OP1]].d, [[MUL]].d
 ; CHECK-NEXT: st1d { [[RES]].d }, [[PG]], [x0]
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: srem_v4i64:
+; VBITS_EQ_128:         ldp q0, q3, [x1]
+; VBITS_EQ_128-NEXT:    ptrue p0.d, vl2
+; VBITS_EQ_128-NEXT:    mov x8, v0.d[1]
+; VBITS_EQ_128-NEXT:    fmov x9, d0
+; VBITS_EQ_128-NEXT:    ldp q1, q2, [x0]
+; VBITS_EQ_128-NEXT:    movprfx z4, z1
+; VBITS_EQ_128-NEXT:    sdiv z4.d, p0/m, z4.d, z0.d
+; VBITS_EQ_128-NEXT:    movprfx z0, z2
+; VBITS_EQ_128-NEXT:    sdiv z0.d, p0/m, z0.d, z3.d
+; VBITS_EQ_128-NEXT:    fmov x12, d4
+; VBITS_EQ_128-NEXT:    fmov x13, d3
+; VBITS_EQ_128-NEXT:    fmov x14, d0
+; VBITS_EQ_128-NEXT:    mov x10, v3.d[1]
+; VBITS_EQ_128-NEXT:    mov x11, v4.d[1]
+; VBITS_EQ_128-NEXT:    mul x9, x12, x9
+; VBITS_EQ_128-NEXT:    mov x15, v0.d[1]
+; VBITS_EQ_128-NEXT:    mul x12, x14, x13
+; VBITS_EQ_128-NEXT:    mul x8, x11, x8
+; VBITS_EQ_128-NEXT:    fmov d0, x9
+; VBITS_EQ_128-NEXT:    mul x10, x15, x10
+; VBITS_EQ_128-NEXT:    fmov d3, x12
+; VBITS_EQ_128-NEXT:    mov v0.d[1], x8
+; VBITS_EQ_128-NEXT:    mov v3.d[1], x10
+; VBITS_EQ_128-NEXT:    sub v0.2d, v1.2d, v0.2d
+; VBITS_EQ_128-NEXT:    sub v1.2d, v2.2d, v3.2d
+; VBITS_EQ_128-NEXT:    stp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <4 x i64>, <4 x i64>* %a
   %op2 = load <4 x i64>, <4 x i64>* %b
   %res = srem <4 x i64> %op1, %op2
   store <4 x i64> %res, <4 x i64>* %a
   ret void
 }
 
 define void @srem_v8i64(<8 x i64>* %a, <8 x i64>* %b) #0 {
; VBITS_GE_2048-NEXT: ret
   ret void
 }
 
 ;
 ; UREM
 ;
 
 ; Vector vXi8 udiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for >= 256 bits here.
 define <8 x i8> @urem_v8i8(<8 x i8> %op1, <8 x i8> %op2) #0 {
 ; CHECK-LABEL: urem_v8i8:
 ; CHECK: uunpklo [[OP2_LO:z[0-9]+]].h, [[OP2:z[0-9]+]].b
 ; CHECK-NEXT: uunpklo [[OP1_LO:z[0-9]+]].h, [[OP1:z[0-9]+]].b
 ; CHECK-NEXT: ptrue [[PG1:p[0-9]+]].s, vl8
 ; CHECK-NEXT: uunpklo [[OP2_LO_LO:z[0-9]+]].s, [[OP2_LO]].h
 ; CHECK-NEXT: uunpklo [[OP1_LO_LO:z[0-9]+]].s, [[OP1_LO]].h
 ; CHECK-NEXT: udivr [[DIV1:z[0-9]+]].s, [[PG1]]/m, [[OP2_LO_LO]].s, [[OP1_LO_LO]].s

 ; CHECK-NEXT: umov [[SCALAR5:w[0-9]+]], [[VEC]].h[5]
 ; CHECK-NEXT: mov [[FINAL]].b[5], [[SCALAR5]]
 ; CHECK-NEXT: umov [[SCALAR6:w[0-9]+]], [[VEC]].h[6]
 ; CHECK-NEXT: mov [[FINAL]].b[6], [[SCALAR6]]
 ; CHECK-NEXT: umov [[SCALAR7:w[0-9]+]], [[VEC]].h[7]
 ; CHECK-NEXT: mov [[FINAL]].b[7], [[SCALAR7]]
 ; CHECK-NEXT: mls v0.8b, [[FINAL]].8b, v1.8b
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: urem_v8i8:
+; VBITS_EQ_128:         ushll v2.8h, v1.8b, #0
+; VBITS_EQ_128-NEXT:    ushll v3.8h, v0.8b, #0
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    uunpkhi z4.s, z2.h
+; VBITS_EQ_128-NEXT:    uunpkhi z5.s, z3.h
+; VBITS_EQ_128-NEXT:    uunpklo z2.s, z2.h
+; VBITS_EQ_128-NEXT:    uunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    udivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    udivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    uzp1 z2.h, z2.h, z4.h
+; VBITS_EQ_128-NEXT:    xtn v2.8b, v2.8h
+; VBITS_EQ_128-NEXT:    mls v0.8b, v2.8b, v1.8b
+; VBITS_EQ_128-NEXT:    ret
+
   %res = urem <8 x i8> %op1, %op2
   ret <8 x i8> %res
 }
 
 define <16 x i8> @urem_v16i8(<16 x i8> %op1, <16 x i8> %op2) #0 {
 ; CHECK-LABEL: urem_v16i8:
 
 ; HALF VECTOR

 ; VBITS_GE_512-NEXT: ptrue [[PG1:p[0-9]+]].s, vl16
 ; VBITS_GE_512-NEXT: uunpklo [[OP2_LO_LO:z[0-9]+]].s, [[OP2_LO]].h
 ; VBITS_GE_512-NEXT: uunpklo [[OP1_LO_LO:z[0-9]+]].s, [[OP1_LO]].h
 ; VBITS_GE_512-NEXT: udivr [[DIV1:z[0-9]+]].s, [[PG1]]/m, [[OP2_LO_LO]].s, [[OP1_LO_LO]].s
 ; VBITS_GE_512-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; VBITS_GE_512-NEXT: uzp1 [[UZP2:z[0-9]+]].b, [[UZP1]].b, [[UZP1]].b
 ; VBITS_GE_512-NEXT: mls v0.16b, v2.16b, v1.16b
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: urem_v16i8:
+; VBITS_EQ_128:         uunpkhi z2.h, z1.b
+; VBITS_EQ_128-NEXT:    uunpkhi z3.h, z0.b
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    uunpkhi z5.s, z2.h
+; VBITS_EQ_128-NEXT:    uunpkhi z6.s, z3.h
+; VBITS_EQ_128-NEXT:    uunpklo z2.s, z2.h
+; VBITS_EQ_128-NEXT:    uunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    uunpklo z4.h, z1.b
+; VBITS_EQ_128-NEXT:    udivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    uunpklo z3.h, z0.b
+; VBITS_EQ_128-NEXT:    udivr z5.s, p0/m, z5.s, z6.s
+; VBITS_EQ_128-NEXT:    uunpkhi z6.s, z4.h
+; VBITS_EQ_128-NEXT:    uunpkhi z7.s, z3.h
+; VBITS_EQ_128-NEXT:    uunpklo z4.s, z4.h
+; VBITS_EQ_128-NEXT:    uunpklo z3.s, z3.h
+; VBITS_EQ_128-NEXT:    udivr z6.s, p0/m, z6.s, z7.s
+; VBITS_EQ_128-NEXT:    udiv z3.s, p0/m, z3.s, z4.s
+; VBITS_EQ_128-NEXT:    uzp1 z2.h, z2.h, z5.h
+; VBITS_EQ_128-NEXT:    uzp1 z3.h, z3.h, z6.h
+; VBITS_EQ_128-NEXT:    uzp1 z2.b, z3.b, z2.b
+; VBITS_EQ_128-NEXT:    mls v0.16b, v2.16b, v1.16b
+; VBITS_EQ_128-NEXT:    ret
+
   %res = urem <16 x i8> %op1, %op2
   ret <16 x i8> %res
 }
 
 define void @urem_v32i8(<32 x i8>* %a, <32 x i8>* %b) #0 {
 ; CHECK-LABEL: urem_v32i8:
 
 ; FULL VECTOR:

 ; VBITS_GE_1024-NEXT: uunpklo [[OP1_LO_LO:z[0-9]+]].s, [[OP1_LO]].h
 ; VBITS_GE_1024-NEXT: udivr [[DIV1:z[0-9]+]].s, [[PG2]]/m, [[OP2_LO_LO]].s, [[OP1_LO_LO]].s
 ; VBITS_GE_1024-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; VBITS_GE_1024-NEXT: uzp1 [[UZP2:z[0-9]+]].b, [[UZP1]].b, [[UZP1]].b
 ; VBITS_GE_1024-NEXT: mul [[OP2]].b, [[PG1]]/m, [[OP2]].b, [[UZP2]].b
 ; VBITS_GE_1024-NEXT: sub [[OP1]].b, [[PG1]]/m, [[OP1]].b, [[OP2]].b
 ; VBITS_GE_1024-NEXT: st1b { [[OP1:z[0-9]+]].b }, [[PG1]], [x0]
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: urem_v32i8:
+; VBITS_EQ_128:         ldp q2, q0, [x0]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    ldp q3, q1, [x1]
+; VBITS_EQ_128-NEXT:    uunpkhi z5.h, z0.b
+; VBITS_EQ_128-NEXT:    uunpklo z7.h, z0.b
+; VBITS_EQ_128-NEXT:    uunpkhi z17.s, z5.h
+; VBITS_EQ_128-NEXT:    uunpklo z5.s, z5.h
+; VBITS_EQ_128-NEXT:    uunpkhi z4.h, z1.b
+; VBITS_EQ_128-NEXT:    uunpklo z6.h, z1.b
+; VBITS_EQ_128-NEXT:    uunpkhi z16.s, z4.h
+; VBITS_EQ_128-NEXT:    uunpklo z4.s, z4.h
+; VBITS_EQ_128-NEXT:    uunpkhi z18.s, z6.h
+; VBITS_EQ_128-NEXT:    udivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    uunpkhi z5.s, z7.h
+; VBITS_EQ_128-NEXT:    uunpklo z6.s, z6.h
+; VBITS_EQ_128-NEXT:    uunpklo z7.s, z7.h
+; VBITS_EQ_128-NEXT:    udiv z5.s, p0/m, z5.s, z18.s
+; VBITS_EQ_128-NEXT:    udivr z6.s, p0/m, z6.s, z7.s
+; VBITS_EQ_128-NEXT:    udivr z16.s, p0/m, z16.s, z17.s
+; VBITS_EQ_128-NEXT:    uzp1 z5.h, z6.h, z5.h
+; VBITS_EQ_128-NEXT:    uunpkhi z6.h, z3.b
+; VBITS_EQ_128-NEXT:    uunpkhi z7.h, z2.b
+; VBITS_EQ_128-NEXT:    uzp1 z4.h, z4.h, z16.h
+; VBITS_EQ_128-NEXT:    uunpkhi z16.s, z6.h
+; VBITS_EQ_128-NEXT:    uunpkhi z17.s, z7.h
+; VBITS_EQ_128-NEXT:    uunpklo z6.s, z6.h
+; VBITS_EQ_128-NEXT:    uunpklo z7.s, z7.h
+; VBITS_EQ_128-NEXT:    udivr z16.s, p0/m, z16.s, z17.s
+; VBITS_EQ_128-NEXT:    udivr z6.s, p0/m, z6.s, z7.s
+; VBITS_EQ_128-NEXT:    uunpklo z7.h, z3.b
+; VBITS_EQ_128-NEXT:    uunpklo z17.h, z2.b
+; VBITS_EQ_128-NEXT:    uunpkhi z18.s, z7.h
+; VBITS_EQ_128-NEXT:    uunpkhi z19.s, z17.h
+; VBITS_EQ_128-NEXT:    uunpklo z7.s, z7.h
+; VBITS_EQ_128-NEXT:    uunpklo z17.s, z17.h
+; VBITS_EQ_128-NEXT:    udivr z18.s, p0/m, z18.s, z19.s
+; VBITS_EQ_128-NEXT:    udivr z7.s, p0/m, z7.s, z17.s
+; VBITS_EQ_128-NEXT:    uzp1 z6.h, z6.h, z16.h
+; VBITS_EQ_128-NEXT:    uzp1 z7.h, z7.h, z18.h
+; VBITS_EQ_128-NEXT:    uzp1 z4.b, z5.b, z4.b
+; VBITS_EQ_128-NEXT:    uzp1 z5.b, z7.b, z6.b
+; VBITS_EQ_128-NEXT:    mls v2.16b, v5.16b, v3.16b
+; VBITS_EQ_128-NEXT:    mls v0.16b, v4.16b, v1.16b
+; VBITS_EQ_128-NEXT:    stp q2, q0, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <32 x i8>, <32 x i8>* %a
   %op2 = load <32 x i8>, <32 x i8>* %b
   %res = urem <32 x i8> %op1, %op2
   store <32 x i8> %res, <32 x i8>* %a
   ret void
 }
 
 define void @urem_v64i8(<64 x i8>* %a, <64 x i8>* %b) #0 {
; VBITS_EQ_2048-NEXT: ret
   %op1 = load <256 x i8>, <256 x i8>* %a
   %op2 = load <256 x i8>, <256 x i8>* %b
   %res = urem <256 x i8> %op1, %op2
   store <256 x i8> %res, <256 x i8>* %a
   ret void
 }
 
 ; Vector vXi16 udiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for >= 256 bits here.
 define <4 x i16> @urem_v4i16(<4 x i16> %op1, <4 x i16> %op2) #0 {
 ; CHECK-LABEL: urem_v4i16:
 ; CHECK: ushll v2.4s, v1.4h, #0
 ; CHECK-NEXT: ushll v3.4s, v0.4h, #0
 ; CHECK-NEXT: ptrue [[PG1:p[0-9]+]].s, vl4
 ; CHECK-NEXT: udivr [[DIV1:z[0-9]+]].s, [[PG1]]/m, z2.s, z3.s
 ; CHECK-NEXT: mov [[SCALAR1:w[0-9]+]], [[VEC:v[0-9]+]].s[1]
 ; CHECK-NEXT: mov [[SCALAR2:w[0-9]+]], [[VEC]].s[2]
 ; CHECK-NEXT: mov v3.16b, v2.16b
 ; CHECK-NEXT: mov [[VECO:v[0-9]+]].h[1], [[SCALAR1]]
 ; CHECK-NEXT: mov [[SCALAR3:w[0-9]+]], [[VEC]].s[3]
 ; CHECK-NEXT: mov [[VECO]].h[2], [[SCALAR2]]
 ; CHECK-NEXT: mov [[VECO]].h[3], [[SCALAR3]]
 ; CHECK-NEXT: mls v0.4h, [[VECO]].4h, v1.4h
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: urem_v4i16:
+; VBITS_EQ_128:         ushll v2.4s, v1.4h, #0
+; VBITS_EQ_128-NEXT:    ushll v3.4s, v0.4h, #0
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    udivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    xtn v2.4h, v2.4s
+; VBITS_EQ_128-NEXT:    mls v0.4h, v2.4h, v1.4h
+; VBITS_EQ_128-NEXT:    ret
+
   %res = urem <4 x i16> %op1, %op2
   ret <4 x i16> %res
 }
 
 define <8 x i16> @urem_v8i16(<8 x i16> %op1, <8 x i16> %op2) #0 {
 ; CHECK-LABEL: urem_v8i16:
 ; CHECK: ptrue [[PG1:p[0-9]+]].s, vl8
 ; CHECK-NEXT: uunpklo [[OP2_LO:z[0-9]+]].s, [[OP2]].h
 ; CHECK-NEXT: uunpklo [[OP1_LO:z[0-9]+]].s, [[OP1]].h
 ; CHECK-NEXT: udivr [[DIV1:z[0-9]+]].s, [[PG1]]/m, [[OP2_LO]].s, [[OP1_LO]].s
 ; CHECK-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; CHECK-NEXT: mls v0.8h, v2.8h, v1.8h
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: urem_v8i16:
+; VBITS_EQ_128:         ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    uunpkhi z2.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpkhi z3.s, z0.h
+; VBITS_EQ_128-NEXT:    uunpklo z4.s, z1.h
+; VBITS_EQ_128-NEXT:    udivr z2.s, p0/m, z2.s, z3.s
+; VBITS_EQ_128-NEXT:    uunpklo z5.s, z0.h
+; VBITS_EQ_128-NEXT:    movprfx z3, z5
+; VBITS_EQ_128-NEXT:    udiv z3.s, p0/m, z3.s, z4.s
+; VBITS_EQ_128-NEXT:    uzp1 z2.h, z3.h, z2.h
+; VBITS_EQ_128-NEXT:    mls v0.8h, v2.8h, v1.8h
+; VBITS_EQ_128-NEXT:    ret
+
   %res = urem <8 x i16> %op1, %op2
   ret <8 x i16> %res
 }
 
 define void @urem_v16i16(<16 x i16>* %a, <16 x i16>* %b) #0 {
 ; CHECK-LABEL: urem_v16i16:
 
 ; FULL VECTOR:

 ; VBITS_GE_512-NEXT: uunpklo [[OP2_LO:z[0-9]+]].s, [[OP2]].h
 ; VBITS_GE_512-NEXT: uunpklo [[OP1_LO:z[0-9]+]].s, [[OP1]].h
 ; VBITS_GE_512-NEXT: udivr [[DIV1:z[0-9]+]].s, [[PG2]]/m, [[OP2_LO]].s, [[OP1_LO]].s
 ; VBITS_GE_512-NEXT: uzp1 [[UZP1:z[0-9]+]].h, [[DIV1]].h, [[DIV1]].h
 ; VBITS_GE_512-NEXT: mul [[OP2]].h, [[PG1]]/m, [[OP2]].h, [[UZP1]].h
 ; VBITS_GE_512-NEXT: sub [[OP1]].h, [[PG1]]/m, [[OP1]].h, [[OP2]].h
 ; VBITS_GE_512-NEXT: st1h { [[OP1:z[0-9]+]].h }, [[PG1]], [x0]
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: urem_v16i16:
+; VBITS_EQ_128:         ldp q2, q0, [x0]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    uunpkhi z17.s, z2.h
+; VBITS_EQ_128-NEXT:    ldp q3, q1, [x1]
+; VBITS_EQ_128-NEXT:    uunpkhi z5.s, z0.h
+; VBITS_EQ_128-NEXT:    uunpklo z7.s, z0.h
+; VBITS_EQ_128-NEXT:    uunpkhi z16.s, z3.h
+; VBITS_EQ_128-NEXT:    udivr z16.s, p0/m, z16.s, z17.s
+; VBITS_EQ_128-NEXT:    uunpkhi z4.s, z1.h
+; VBITS_EQ_128-NEXT:    uunpklo z6.s, z1.h
+; VBITS_EQ_128-NEXT:    udivr z4.s, p0/m, z4.s, z5.s
+; VBITS_EQ_128-NEXT:    uunpklo z5.s, z3.h
+; VBITS_EQ_128-NEXT:    udivr z6.s, p0/m, z6.s, z7.s
+; VBITS_EQ_128-NEXT:    uunpklo z7.s, z2.h
+; VBITS_EQ_128-NEXT:    udivr z5.s, p0/m, z5.s, z7.s
+; VBITS_EQ_128-NEXT:    uzp1 z4.h, z6.h, z4.h
+; VBITS_EQ_128-NEXT:    uzp1 z5.h, z5.h, z16.h
+; VBITS_EQ_128-NEXT:    mls v2.8h, v5.8h, v3.8h
+; VBITS_EQ_128-NEXT:    mls v0.8h, v4.8h, v1.8h
+; VBITS_EQ_128-NEXT:    stp q2, q0, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <16 x i16>, <16 x i16>* %a
   %op2 = load <16 x i16>, <16 x i16>* %b
   %res = urem <16 x i16> %op1, %op2
   store <16 x i16> %res, <16 x i16>* %a
   ret void
 }
 
 define void @urem_v32i16(<32 x i16>* %a, <32 x i16>* %b) #0 {

 ; Vector v2i32 udiv are not legal for NEON so use SVE when available.
 define <2 x i32> @urem_v2i32(<2 x i32> %op1, <2 x i32> %op2) #0 {
 ; CHECK-LABEL: urem_v2i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl2
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], z0
 ; CHECK-NEXT: udiv z2.s, [[PG]]/m, [[PFX]].s, z1.s
 ; CHECK-NEXT: mls v0.2s, v2.2s, v1.2s
 ; CHECK: ret
+
+; VBITS_EQ_128-LABEL: urem_v2i32:
+; VBITS_EQ_128:         ptrue p0.s, vl2
+; VBITS_EQ_128-NEXT:    movprfx z2, z0
+; VBITS_EQ_128-NEXT:    udiv z2.s, p0/m, z2.s, z1.s
+; VBITS_EQ_128-NEXT:    mls v0.2s, v2.2s, v1.2s
+; VBITS_EQ_128-NEXT:    ret
+
   %res = urem <2 x i32> %op1, %op2
   ret <2 x i32> %res
 }
 
 ; Vector v4i32 udiv are not legal for NEON so use SVE when available.
 define <4 x i32> @urem_v4i32(<4 x i32> %op1, <4 x i32> %op2) #0 {
 ; CHECK-LABEL: urem_v4i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl4
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], z0
 ; CHECK-NEXT: udiv z2.s, [[PG]]/m, [[PFX]].s, z1.s
 ; CHECK-NEXT: mls v0.4s, v2.4s, v1.4s
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: urem_v4i32:
+; VBITS_EQ_128:         ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    movprfx z2, z0
+; VBITS_EQ_128-NEXT:    udiv z2.s, p0/m, z2.s, z1.s
+; VBITS_EQ_128-NEXT:    mls v0.4s, v2.4s, v1.4s
+; VBITS_EQ_128-NEXT:    ret
+
   %res = urem <4 x i32> %op1, %op2
   ret <4 x i32> %res
 }
 
 define void @urem_v8i32(<8 x i32>* %a, <8 x i32>* %b) #0 {
 ; CHECK-LABEL: urem_v8i32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl8
 ; CHECK-NEXT: ld1w { [[OP1:z[0-9]+]].s }, [[PG]]/z, [x0]
 ; CHECK-NEXT: ld1w { [[OP2:z[0-9]+]].s }, [[PG]]/z, [x1]
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], [[OP1]]
 ; CHECK-NEXT: udiv [[DIV:z[0-9]+]].s, [[PG]]/m, [[PFX]].s, [[OP2]].s
 ; CHECK-NEXT: mul [[MUL:z[0-9]+]].s, [[PG]]/m, [[OP2]].s, [[DIV]].s
 ; CHECK-NEXT: sub [[RES:z[0-9]+]].s, [[PG]]/m, [[OP1]].s, [[MUL]].s
 ; CHECK-NEXT: st1w { [[RES]].s }, [[PG]], [x0]
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: urem_v8i32:
+; VBITS_EQ_128:         ldp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ptrue p0.s, vl4
+; VBITS_EQ_128-NEXT:    ldp q2, q3, [x1]
+; VBITS_EQ_128-NEXT:    movprfx z4, z1
+; VBITS_EQ_128-NEXT:    udiv z4.s, p0/m, z4.s, z3.s
+; VBITS_EQ_128-NEXT:    movprfx z5, z0
+; VBITS_EQ_128-NEXT:    udiv z5.s, p0/m, z5.s, z2.s
+; VBITS_EQ_128-NEXT:    mls v0.4s, v5.4s, v2.4s
+; VBITS_EQ_128-NEXT:    mls v1.4s, v4.4s, v3.4s
+; VBITS_EQ_128-NEXT:    stp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <8 x i32>, <8 x i32>* %a
   %op2 = load <8 x i32>, <8 x i32>* %b
   %res = urem <8 x i32> %op1, %op2
   store <8 x i32> %res, <8 x i32>* %a
   ret void
 }
 
 define void @urem_v16i32(<16 x i32>* %a, <16 x i32>* %b) #0 {
; VBITS_GE_2048-NEXT: ret
   %op1 = load <64 x i32>, <64 x i32>* %a
   %op2 = load <64 x i32>, <64 x i32>* %b
   %res = urem <64 x i32> %op1, %op2
   store <64 x i32> %res, <64 x i32>* %a
   ret void
 }
 
 ; Vector i64 udiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for the 128 bits case here.
 define <1 x i64> @urem_v1i64(<1 x i64> %op1, <1 x i64> %op2) #0 {
 ; CHECK-LABEL: urem_v1i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl1
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], [[OP1]]
 ; CHECK-NEXT: udiv [[DIV:z[0-9]+]].d, [[PG]]/m, [[PFX]].d, z1.d
 ; CHECK-NEXT: mul z1.d, [[PG]]/m, [[OP2]].d, [[DIV]].d
 ; CHECK-NEXT: sub d0, d0, d1
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: urem_v1i64:
+; VBITS_EQ_128:         ptrue p0.d, vl1
+; VBITS_EQ_128-NEXT:    movprfx z2, z0
+; VBITS_EQ_128-NEXT:    udiv z2.d, p0/m, z2.d, z1.d
+; VBITS_EQ_128-NEXT:    fmov x8, d2
+; VBITS_EQ_128-NEXT:    fmov x9, d1
+; VBITS_EQ_128-NEXT:    mul x8, x8, x9
+; VBITS_EQ_128-NEXT:    fmov d1, x8
+; VBITS_EQ_128-NEXT:    sub d0, d0, d1
+; VBITS_EQ_128-NEXT:    ret
+
   %res = urem <1 x i64> %op1, %op2
   ret <1 x i64> %res
 }
 
 ; Vector i64 udiv are not legal for NEON so use SVE when available.
+; FIXME: We should be able to improve the codegen for the 128 bits case here.
 define <2 x i64> @urem_v2i64(<2 x i64> %op1, <2 x i64> %op2) #0 {
 ; CHECK-LABEL: urem_v2i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl2
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], [[OP1]]
 ; CHECK-NEXT: udiv [[DIV:z[0-9]+]].d, [[PG]]/m, [[PFX]].d, z1.d
 ; CHECK-NEXT: mul z1.d, [[PG]]/m, [[OP2]].d, [[DIV]].d
 ; CHECK-NEXT: sub v0.2d, v0.2d, v1.2d
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: urem_v2i64:
+; VBITS_EQ_128:         ptrue p0.d, vl2
+; VBITS_EQ_128-NEXT:    movprfx z2, z0
+; VBITS_EQ_128-NEXT:    udiv z2.d, p0/m, z2.d, z1.d
+; VBITS_EQ_128-NEXT:    fmov x9, d2
+; VBITS_EQ_128-NEXT:    fmov x10, d1
+; VBITS_EQ_128-NEXT:    mov x8, v2.d[1]
+; VBITS_EQ_128-NEXT:    mov x11, v1.d[1]
+; VBITS_EQ_128-NEXT:    mul x9, x9, x10
+; VBITS_EQ_128-NEXT:    mul x8, x8, x11
+; VBITS_EQ_128-NEXT:    fmov d1, x9
+; VBITS_EQ_128-NEXT:    mov v1.d[1], x8
+; VBITS_EQ_128-NEXT:    sub v0.2d, v0.2d, v1.2d
+; VBITS_EQ_128-NEXT:    ret
+
   %res = urem <2 x i64> %op1, %op2
   ret <2 x i64> %res
 }
 
 define void @urem_v4i64(<4 x i64>* %a, <4 x i64>* %b) #0 {
 ; CHECK-LABEL: urem_v4i64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl4
 ; CHECK-NEXT: ld1d { [[OP1:z[0-9]+]].d }, [[PG]]/z, [x0]
 ; CHECK-NEXT: ld1d { [[OP2:z[0-9]+]].d }, [[PG]]/z, [x1]
 ; CHECK-NEXT: movprfx [[PFX:z[0-9]+]], [[OP1]]
 ; CHECK-NEXT: udiv [[DIV:z[0-9]+]].d, [[PG]]/m, [[PFX]].d, [[OP2]].d
 ; CHECK-NEXT: mul [[MUL:z[0-9]+]].d, [[PG]]/m, [[OP2]].d, [[DIV]].d
 ; CHECK-NEXT: sub [[RES:z[0-9]+]].d, [[PG]]/m, [[OP1]].d, [[MUL]].d
 ; CHECK-NEXT: st1d { [[RES]].d }, [[PG]], [x0]
 ; CHECK-NEXT: ret
+
+; VBITS_EQ_128-LABEL: urem_v4i64:
+; VBITS_EQ_128:         ldp q0, q3, [x1]
+; VBITS_EQ_128-NEXT:    ptrue p0.d, vl2
+; VBITS_EQ_128-NEXT:    mov x8, v0.d[1]
+; VBITS_EQ_128-NEXT:    fmov x9, d0
+; VBITS_EQ_128-NEXT:    ldp q1, q2, [x0]
+; VBITS_EQ_128-NEXT:    movprfx z4, z1
+; VBITS_EQ_128-NEXT:    udiv z4.d, p0/m, z4.d, z0.d
+; VBITS_EQ_128-NEXT:    movprfx z0, z2
+; VBITS_EQ_128-NEXT:    udiv z0.d, p0/m, z0.d, z3.d
+; VBITS_EQ_128-NEXT:    fmov x12, d4
+; VBITS_EQ_128-NEXT:    fmov x13, d3
+; VBITS_EQ_128-NEXT:    fmov x14, d0
+; VBITS_EQ_128-NEXT:    mov x10, v3.d[1]
+; VBITS_EQ_128-NEXT:    mov x11, v4.d[1]
+; VBITS_EQ_128-NEXT:    mul x9, x12, x9
+; VBITS_EQ_128-NEXT:    mov x15, v0.d[1]
+; VBITS_EQ_128-NEXT:    mul x12, x14, x13
+; VBITS_EQ_128-NEXT:    mul x8, x11, x8
+; VBITS_EQ_128-NEXT:    fmov d0, x9
+; VBITS_EQ_128-NEXT:    mul x10, x15, x10
+; VBITS_EQ_128-NEXT:    fmov d3, x12
+; VBITS_EQ_128-NEXT:    mov v0.d[1], x8
+; VBITS_EQ_128-NEXT:    mov v3.d[1], x10
+; VBITS_EQ_128-NEXT:    sub v0.2d, v1.2d, v0.2d
+; VBITS_EQ_128-NEXT:    sub v1.2d, v2.2d, v3.2d
+; VBITS_EQ_128-NEXT:    stp q0, q1, [x0]
+; VBITS_EQ_128-NEXT:    ret
+
   %op1 = load <4 x i64>, <4 x i64>* %a
   %op2 = load <4 x i64>, <4 x i64>* %b
   %res = urem <4 x i64> %op1, %op2
   store <4 x i64> %res, <4 x i64>* %a
   ret void
 }
 
 define void @urem_v8i64(<8 x i64>* %a, <8 x i64>* %b) #0 {