[CodeGen][SVE] Legalisation of extends with scalable types

[kmclaughlin]

Summary

This patch adds legalisation of extensions where the operand
of the extend is a legal scalable type but the result is not.

EXTRACT_SUBVECTOR is used to split the result, before
being replaced by target-specific [S|U]UNPK[HI|LO] operations.

For example:

zext <vscale x 16 x i8> %a to <vscale x 16 x i16>

should emit:

uunpklo z2.h, z0.b
uunpkhi z1.h, z0.b

[efriedma]

Can you explain why target-independent legalization isn't suitable here? I'd prefer not to have target-specific type legalization for every operation on vscale'ed types. (The target-independent name of AArch64ISD::SUNPKLO is ISD::SIGN_EXTEND_VECTOR_INREG.)

[david-arm]

Hi Kerry, I think we do already have some suitable test files where perhaps you could add your tests instead of creating new files? For example, there are:

CodeGen/AArch64/sve-int-arith.ll (perhaps integer divides and shifts could live there?)
CodeGen/AArch64/sve-int-div-pred.ll (some divides already in here)
CodeGen/AArch64/sve-sext-zext.ll (extend tests recently added)

Thanks!

[david-arm]

Hi @efriedma, is there a target-independent equivalent of SUNPKHI? From a quick glance at the codebase where X86 uses ISD::SIGN_EXTEND_VECTOR_INREG it seems vector shuffles are still required for the Hi part, which is fine for fixed length vectors I guess.

[efriedma]

For sunpckhi... no, not really. You'd need to either add a new opcode, or add a new shuffle operation of some sort.

[kmclaughlin]

• Removed ReplaceExtensionResults and instead try to use extract_subvector as much as possible to legalise the result
• Added ReplaceExtractSubVectorResults, which replaces extract_subvector with unpack operations
• Changed performSignExtendInRegCombine to replace a sign extend of an unsigned unpack with a signed unpack
• Removed changes to unrelated test files

[david-arm]

Hi Kerry, just a couple of comments about the use of getVectorNumElements() - we're trying to remove calls to this function so it would be good if you could use getVectorElementCount() instead. Thanks!

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13718

I think we want to move away from calling getVectorNumElements(), so might need to change this to something like

ElementCount ResEC = VT.getVectorElementCount();
if (InVT.getVectorElementCount() != (ResEC * 2))

13726

And here you could then change this to:

if ((Index != 0) && (Index != ResEC.Min))

[david-arm]

Sorry I forgot to mention I think we have an existing test file for extends:

llvm/test/CodeGen/AArch64/sve-sext-zext.ll

It might be worth adding these cases to that file instead of creating a new one?

[kmclaughlin]

• Replaced uses of getVectorNumElements() with getVectorElementCount()
• Moved the new tests into the existing sve-sext-zext.ll file

[efriedma]

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

4337 ↗

(On Diff #267280)

In general, there are four possibilities for legalization of the operand vector:

1. Legal. In this case, I guess the operation you want is essentially ANY_EXTEND_VECTOR_INREG. You're currently handling this in target-specific code; I guess that's okay for now.
2. Promote. In this case, you can promote both the operand and the result, what you're doing here.
3. Widen. In this case, you just widen the operand.
4. Split. In this case, you pick the correct half (or construct it with a shuffle), and extract from that.

---

Currently, you're assuming the the result of GetPromotedInteger() has an element type narrower or equal to the element type of NOutVT. I'm not sure that's true on all targets, but I guess it's true for SVE. Better to explicitly check, I think.

4344 ↗

(On Diff #267280)

Please put a report_fatal_error here for scalable vectors, so we don't end up with some obscure invalid BUILD_VECTOR error.

Alternatively, I guess you could call ExpandExtractFromVectorThroughStack().

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

895

Can you restrict this to specifically the types you're interested in handling? It looks like you only implemented handling for the following types: nxv8i8, nxv4i16, nxv2i32.

8507

I think you can just assert the type isn't scalable? It shouldn't be possible to get here. You're only marking EXTRACT_SUBVECTOR Custom for illegal types.

10588

Do you need to truncate ExtVal somewhere, so you don't end up with a DUP with an over-wide constant?

13161

It seems a little fragile to assume the inner VT of the SIGN_EXTEND_INREG is exactly the type you're expecting here. Probably worth at least adding an assertion to encode the assumptions you're making.

13697

"Bubble truncates to illegal types to the surface" is an optimization?

llvm/test/CodeGen/AArch64/sve-sext-zext.ll

195 ↗

(On Diff #267280)

Please just generate the checks with update_llc_test_checks.py.

[kmclaughlin]

• Restricted the illegal types which should be lowered for EXTRACT_SUBVECTOR to those handled in this patch (nxv8i8, nxv4i16 & nxv2i32)
• Removed unnecessary changes in ReplaceExtractSubVectorResults
• Updated the tests with update_llc_test_checks.py
• Added a check on expected types in performSignExtendInRegCombine & PromoteIntRes_EXTRACT_SUBVECTOR

[kmclaughlin]

Thanks for taking another look at this, @efriedma!

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

10588

I've changed the call to getNode below that creates the DUP to use DAG.getAnyExtOrTrunc (similar to what we do in LowerSPLAT_VECTOR)

13161

I've added an assert above here to make sure the sign_extend_inreg and unpack types match, is this the assumption you were referring to?

13697

Removed - this was not required for this patch.

[efriedma]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

10588

I'm specifically concerned that you could end up with something like (nxv16i8 (dup (i32 0x12345678))).

13161

We assert that SIGN_EXTEND_INREG has valid operand/result types elsewhere.

I was more concerned about the inner VT (cast<VTSDNode>(N->getOperand(1))->getVT();). You could end up creating an invalid SIGN_EXTEND_INREG if the type is something weird, like a non-byte-size integer type.

[kmclaughlin]

• Added a truncate of ExtVal in performSVEAndCombine
• Changed the assert added to performSignExtendInRegCombine in the previous revision

[kmclaughlin]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

10588

I see what you mean - I've added a truncate of Dup->getOperand(0) for this, which will truncate the constant to the type of UnpkOp

13161

Removed my previous check on the operand & result types and added an assert for the type of VT.

[efriedma]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

10599

It's not legal to generate operations with type EltTy after legalization. You get away with it here because it's guaranteed to constant-fold... but probably less confusing to use APInt::trunc instead.

[kmclaughlin]

• Use APInt::trunc to truncate the constant in performSVEAndCombine

[efriedma]

LGTM

llvm/lib/Target/AArch64/AArch64ISelLowering.h

bool getIndexedAddressParts(SDNode *Op, SDValue &Base, SDValue &Offset,
                               SelectionDAG &DAG) const;
   bool getPreIndexedAddressParts(SDNode *N, SDValue &Base, SDValue &Offset,
                                  ISD::MemIndexedMode &AM,
                                  SelectionDAG &DAG) const override;
   bool getPostIndexedAddressParts(SDNode *N, SDNode *Op, SDValue &Base,
                                   SDValue &Offset, ISD::MemIndexedMode &AM,
                                   SelectionDAG &DAG) const override;
 
+  void ReplaceExtensionResults(SDNode *N, SmallVectorImpl<SDValue> &Results,
+                               SelectionDAG &DAG, unsigned HiOpcode,
+                               unsigned LoOpcode) const;
+
   void ReplaceNodeResults(SDNode *N, SmallVectorImpl<SDValue> &Results,
                           SelectionDAG &DAG) const override;
 
   bool shouldNormalizeToSelectSequence(LLVMContext &, EVT) const override;
 
   void finalizeLowering(MachineFunction &MF) const override;
 
   bool shouldLocalize(const MachineInstr &MI,

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

for (MVT VT : MVT::integer_scalable_vector_valuetypes()) {
         setOperationAction(ISD::UDIV, VT, Custom);
         setOperationAction(ISD::SMIN, VT, Custom);
         setOperationAction(ISD::UMIN, VT, Custom);
         setOperationAction(ISD::SMAX, VT, Custom);
         setOperationAction(ISD::UMAX, VT, Custom);
         setOperationAction(ISD::SHL, VT, Custom);
         setOperationAction(ISD::SRL, VT, Custom);
         setOperationAction(ISD::SRA, VT, Custom);
+      } else {
+        if (VT.getVectorElementType() != MVT::i1) {
+          // Use UNPK{LO,HI} sequences to lower extensions from legal SVE
+          // types to wider-than-legal types.
+          setOperationAction(ISD::SIGN_EXTEND, VT, Custom);
+          setOperationAction(ISD::ZERO_EXTEND, VT, Custom);
+          setOperationAction(ISD::ANY_EXTEND, VT, Custom);
+        }
       }
     }
     setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::i8, Custom);
     setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::i16, Custom);
 
     for (MVT VT : MVT::fp_scalable_vector_valuetypes()) {
       if (isTypeLegal(VT)) {
         setOperationAction(ISD::SPLAT_VECTOR, VT, Custom);
SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
                                               SelectionDAG &DAG) const {
   LLVM_DEBUG(dbgs() << "Custom lowering: ");
   LLVM_DEBUG(Op.dump());
 
   switch (Op.getOpcode()) {
   default:
     llvm_unreachable("unimplemented operand");
     return SDValue();
+  case ISD::ANY_EXTEND:
+  case ISD::SIGN_EXTEND:
+  case ISD::ZERO_EXTEND:
+    // Needed because we have selected custom lowering for illegal SVE types.
+    // The cases we actually want to handle are where the operand is legal
+    // and the result isn't, which go through ReplaceNodeResults instead.
+    // This code only sees cases where the result is legal and the operand
+    // isn't.
+    return SDValue();
   case ISD::BITCAST:
     return LowerBITCAST(Op, DAG);
   case ISD::GlobalAddress:
     return LowerGlobalAddress(Op, DAG);
   case ISD::GlobalTLSAddress:
     return LowerGlobalTLSAddress(Op, DAG);
   case ISD::SETCC:
   case ISD::STRICT_FSETCC:
AArch64TargetLowering::LowerEXTRACT_VECTOR_ELT(SDValue Op,
   return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, ExtrTy, WideVec,
                      Op.getOperand(1));
 }
 
 SDValue AArch64TargetLowering::LowerEXTRACT_SUBVECTOR(SDValue Op,
                                                       SelectionDAG &DAG) const {
   EVT VT = Op.getOperand(0).getValueType();
   SDLoc dl(Op);
   // Just in case...
   if (!VT.isVector())
     return SDValue();
 
   ConstantSDNode *Cst = dyn_cast<ConstantSDNode>(Op.getOperand(1));
   if (!Cst)
     return SDValue();
   unsigned Val = Cst->getZExtValue();
 
static SDValue performSVEAndCombine(SDNode *N,
 
   EVT MemVT;
 
   // SVE load instructions perform an implicit zero-extend, which makes them
   // perfect candidates for combining.
   switch (Src->getOpcode()) {
   case AArch64ISD::LD1:
   case AArch64ISD::LDNF1:
   case AArch64ISD::LDFF1:
     MemVT = cast<VTSDNode>(Src->getOperand(3))->getVT();
     break;
   case AArch64ISD::GLD1:
   case AArch64ISD::GLD1_SCALED:
   case AArch64ISD::GLD1_SXTW:
   case AArch64ISD::GLD1_SXTW_SCALED:
   case AArch64ISD::GLD1_UXTW:
   case AArch64ISD::GLD1_UXTW_SCALED:
   case AArch64ISD::GLD1_IMM:
   case AArch64ISD::GLDFF1:
   case AArch64ISD::GLDFF1_SCALED:
   case AArch64ISD::GLDFF1_SXTW:
   case AArch64ISD::GLDFF1_SXTW_SCALED:
   case AArch64ISD::GLDFF1_UXTW:
   case AArch64ISD::GLDFF1_UXTW_SCALED:
   case AArch64ISD::GLDFF1_IMM:
   case AArch64ISD::GLDNT1:
     MemVT = cast<VTSDNode>(Src->getOperand(4))->getVT();
     break;
performSignExtendInRegCombine(SDNode *N, TargetLowering::DAGCombinerInfo &DCI,
   // SVE load nodes (e.g. AArch64ISD::GLD1) are straightforward candidates
   // for DAG Combine with SIGN_EXTEND_INREG. Bail out for all other nodes.
   unsigned NewOpc;
   unsigned MemVTOpNum = 4;
   switch (Opc) {
   case AArch64ISD::LD1:
     NewOpc = AArch64ISD::LD1S;
     MemVTOpNum = 3;
     break;
   case AArch64ISD::LDNF1:
     NewOpc = AArch64ISD::LDNF1S;
     MemVTOpNum = 3;
     break;
   case AArch64ISD::LDFF1:
     NewOpc = AArch64ISD::LDFF1S;
     MemVTOpNum = 3;
     break;
static void ReplaceReductionResults(SDNode *N,
   SDLoc dl(N);
   std::tie(LoVT, HiVT) = DAG.GetSplitDestVTs(N->getValueType(0));
   std::tie(Lo, Hi) = DAG.SplitVectorOperand(N, 0);
   SDValue InterVal = DAG.getNode(InterOp, dl, LoVT, Lo, Hi);
   SDValue SplitVal = DAG.getNode(AcrossOp, dl, LoVT, InterVal);
   Results.push_back(SplitVal);
 }
 
+// If the node is an extension from a legal SVE type to something wider,
+// use HiOpcode and LoOpcode to extend each half individually, then
+// concatenate them together.
+void AArch64TargetLowering::ReplaceExtensionResults(
+    SDNode *N, SmallVectorImpl<SDValue> &Results, SelectionDAG &DAG,
+    unsigned HiOpcode, unsigned LoOpcode) const {
+  SDLoc DL(N);
+  SDValue In = N->getOperand(0);
+  EVT InVT = In.getValueType();
+
+  assert(InVT.isScalableVector() && "Can only lower scalable vectors");
+
+  if (!isTypeLegal(InVT))
+    return;
+
+  EVT InEltVT = InVT.getVectorElementType();
+  auto EltCnt = InVT.getVectorElementCount();
+  unsigned InEltBits = InEltVT.getSizeInBits();
+
+  if (InEltBits != 8 && InEltBits != 16 && InEltBits != 32)
+    return;
+
+  // The result must be at least twice as wide as the input in order for
+  // this to work.
+  EVT VT = N->getValueType(0);
+  EVT EltVT = VT.getVectorElementType();
+  if (EltVT.getSizeInBits() < InEltBits * 2)
+    return;
+
+  // Extend In to a double-width vector.
+  EVT NewEltVT = EVT::getIntegerVT(*DAG.getContext(), InEltBits * 2);
+  EVT NewVT = EVT::getVectorVT(*DAG.getContext(), NewEltVT, EltCnt / 2);
+  assert(isTypeLegal(NewVT) && "Extension result should be legal");
+
+  SDValue Lo = DAG.getNode(LoOpcode, DL, NewVT, In);
+  SDValue Hi = DAG.getNode(HiOpcode, DL, NewVT, In);
+
+  // If necessary, extend again using the original code.  Such extensions
+  // will also need legalizing, but at least we're making forward progress.
+  NewVT = EVT::getVectorVT(*DAG.getContext(), EltVT, EltCnt / 2);
+  Lo = DAG.getNode(N->getOpcode(), DL, NewVT, Lo);
+  Hi = DAG.getNode(N->getOpcode(), DL, NewVT, Hi);
+
+  Results.push_back(DAG.getNode(ISD::CONCAT_VECTORS, DL, VT, Lo, Hi));
+}
+
 static std::pair<SDValue, SDValue> splitInt128(SDValue N, SelectionDAG &DAG) {
   SDLoc DL(N);
   SDValue Lo = DAG.getNode(ISD::TRUNCATE, DL, MVT::i64, N);
   SDValue Hi = DAG.getNode(ISD::TRUNCATE, DL, MVT::i64,
                            DAG.getNode(ISD::SRL, DL, MVT::i128, N,
                                        DAG.getConstant(64, DL, MVT::i64)));
   return std::make_pair(Lo, Hi);
 }
 
 // Create an even/odd pair of X registers holding integer value V.
 static SDValue createGPRPairNode(SelectionDAG &DAG, SDValue V) {
   SDLoc dl(V.getNode());
   SDValue VLo = DAG.getAnyExtOrTrunc(V, dl, MVT::i64);
   SDValue VHi = DAG.getAnyExtOrTrunc(
       DAG.getNode(ISD::SRL, dl, MVT::i128, V, DAG.getConstant(64, dl, MVT::i64)),
       dl, MVT::i64);
   if (DAG.getDataLayout().isBigEndian())
     std::swap (VLo, VHi);
   SDValue RegClass =
       DAG.getTargetConstant(AArch64::XSeqPairsClassRegClassID, dl, MVT::i32);
   SDValue SubReg0 = DAG.getTargetConstant(AArch64::sube64, dl, MVT::i32);
   SDValue SubReg1 = DAG.getTargetConstant(AArch64::subo64, dl, MVT::i32);
   const SDValue Ops[] = { RegClass, VLo, SubReg0, VHi, SubReg1 };
   return SDValue(
       DAG.getMachineNode(TargetOpcode::REG_SEQUENCE, dl, MVT::Untyped, Ops), 0);
 }
 
 static void ReplaceCMP_SWAP_128Results(SDNode *N,
                                        SmallVectorImpl<SDValue> &Results,
                                        SelectionDAG &DAG,
                                        const AArch64Subtarget *Subtarget) {
   assert(N->getValueType(0) == MVT::i128 &&
          "AtomicCmpSwap on types less than 128 should be legal");
 
   if (Subtarget->hasLSE()) {
     // LSE has a 128-bit compare and swap (CASP), but i128 is not a legal type,
     // so lower it here, wrapped in REG_SEQUENCE and EXTRACT_SUBREG.
     SDValue Ops[] = {
         createGPRPairNode(DAG, N->getOperand(2)), // Compare value
         createGPRPairNode(DAG, N->getOperand(3)), // Store value
         N->getOperand(1), // Ptr
         N->getOperand(0), // Chain in
     };
 
     MachineMemOperand *MemOp = cast<MemSDNode>(N)->getMemOperand();
 
     unsigned Opcode;
     switch (MemOp->getOrdering()) {
     case AtomicOrdering::Monotonic:
       Opcode = AArch64::CASPX;
       break;
     case AtomicOrdering::Acquire:
       Opcode = AArch64::CASPAX;
       break;
     case AtomicOrdering::Release:
       Opcode = AArch64::CASPLX;
       break;
     case AtomicOrdering::AcquireRelease:
     case AtomicOrdering::SequentiallyConsistent:
       Opcode = AArch64::CASPALX;
void AArch64TargetLowering::ReplaceNodeResults(
   case ISD::FP_TO_UINT:
   case ISD::FP_TO_SINT:
     assert(N->getValueType(0) == MVT::i128 && "unexpected illegal conversion");
     // Let normal code take care of it by not adding anything to Results.
     return;
   case ISD::ATOMIC_CMP_SWAP:
     ReplaceCMP_SWAP_128Results(N, Results, DAG, Subtarget);
     return;
+  case ISD::SIGN_EXTEND:
+    ReplaceExtensionResults(N, Results, DAG,
+                            AArch64ISD::SUNPKHI, AArch64ISD::SUNPKLO);
+    return;
+  case ISD::ZERO_EXTEND:
+  case ISD::ANY_EXTEND:
+    ReplaceExtensionResults(N, Results, DAG,
+                            AArch64ISD::UUNPKHI, AArch64ISD::UUNPKLO);
+    return;
   case ISD::LOAD: {
     assert(SDValue(N, 0).getValueType() == MVT::i128 &&
            "unexpected load's value type");
     LoadSDNode *LoadNode = cast<LoadSDNode>(N);
     if (!LoadNode->isVolatile() || LoadNode->getMemoryVT() != MVT::i128) {
       // Non-volatile loads are optimized later in AArch64's load/store
       // optimizer.
       return;

llvm/test/CodeGen/AArch64/llvm-ir-to-intrinsic.ll

-; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
-
-;
-; SDIV
-;
-
-define <vscale x 4 x i32> @sdiv_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
-; CHECK-LABEL: @sdiv_i32
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: sdiv z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %div = sdiv <vscale x 4 x i32> %a, %b
-  ret <vscale x 4 x i32> %div
-}
-
-define <vscale x 2 x i64> @sdiv_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
-; CHECK-LABEL: @sdiv_i64
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: sdiv z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %div = sdiv <vscale x 2 x i64> %a, %b
-  ret <vscale x 2 x i64> %div
-}
-
-define <vscale x 8 x i32> @sdiv_split_i32(<vscale x 8 x i32> %a, <vscale x 8 x i32> %b) {
-; CHECK-LABEL: @sdiv_split_i32
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: sdiv z0.s, p0/m, z0.s, z2.s
-; CHECK-DAG: sdiv z1.s, p0/m, z1.s, z3.s
-; CHECK-NEXT: ret
-  %div = sdiv <vscale x 8 x i32> %a, %b
-  ret <vscale x 8 x i32> %div
-}
-
-define <vscale x 2 x i32> @sdiv_promote_i32(<vscale x 2 x i32> %a, <vscale x 2 x i32> %b) {
-; CHECK-LABEL: @sdiv_promote_i32
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: sxtw z1.d, p0/m, z1.d
-; CHECK-DAG: sxtw z0.d, p0/m, z0.d
-; CHECK-DAG: sdiv z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %div = sdiv <vscale x 2 x i32> %a, %b
-  ret <vscale x 2 x i32> %div
-}
-
-define <vscale x 4 x i64> @sdiv_split_i64(<vscale x 4 x i64> %a, <vscale x 4 x i64> %b) {
-; CHECK-LABEL: @sdiv_split_i64
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: sdiv z0.d, p0/m, z0.d, z2.d
-; CHECK-DAG: sdiv z1.d, p0/m, z1.d, z3.d
-; CHECK-NEXT: ret
-  %div = sdiv <vscale x 4 x i64> %a, %b
-  ret <vscale x 4 x i64> %div
-}
-
-;
-; UDIV
-;
-
-define <vscale x 4 x i32> @udiv_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
-; CHECK-LABEL: @udiv_i32
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: udiv z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %div = udiv <vscale x 4 x i32> %a, %b
-  ret <vscale x 4 x i32> %div
-}
-
-define <vscale x 2 x i64> @udiv_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
-; CHECK-LABEL: @udiv_i64
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: udiv z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %div = udiv <vscale x 2 x i64> %a, %b
-  ret <vscale x 2 x i64> %div
-}
-
-define <vscale x 8 x i32> @udiv_split_i32(<vscale x 8 x i32> %a, <vscale x 8 x i32> %b) {
-; CHECK-LABEL: @udiv_split_i32
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: udiv z0.s, p0/m, z0.s, z2.s
-; CHECK-DAG: udiv z1.s, p0/m, z1.s, z3.s
-; CHECK-NEXT: ret
-  %div = udiv <vscale x 8 x i32> %a, %b
-  ret <vscale x 8 x i32> %div
-}
-
-define <vscale x 2 x i32> @udiv_promote_i32(<vscale x 2 x i32> %a, <vscale x 2 x i32> %b) {
-; CHECK-LABEL: @udiv_promote_i32
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: and z1.d, z1.d, #0xffffffff
-; CHECK-DAG: and z0.d, z0.d, #0xffffffff
-; CHECK-DAG: udiv z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %div = udiv <vscale x 2 x i32> %a, %b
-  ret <vscale x 2 x i32> %div
-}
-
-define <vscale x 4 x i64> @udiv_split_i64(<vscale x 4 x i64> %a, <vscale x 4 x i64> %b) {
-; CHECK-LABEL: @udiv_split_i64
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: udiv z0.d, p0/m, z0.d, z2.d
-; CHECK-DAG: udiv z1.d, p0/m, z1.d, z3.d
-; CHECK-NEXT: ret
-  %div = udiv <vscale x 4 x i64> %a, %b
-  ret <vscale x 4 x i64> %div
-}
-
-;
-; SMIN
-;
-
-define <vscale x 16 x i8> @smin_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
-; CHECK-LABEL: @smin_i8
-; CHECK-DAG: ptrue p0.b
-; CHECK-DAG: smin z0.b, p0/m, z0.b, z1.b
-; CHECK-NEXT: ret
-  %cmp = icmp slt <vscale x 16 x i8> %a, %b
-  %min = select <vscale x 16 x i1> %cmp, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b
-  ret <vscale x 16 x i8> %min
-}
-
-define <vscale x 8 x i16> @smin_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
-; CHECK-LABEL: @smin_i16
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: smin z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT: ret
-  %cmp = icmp slt <vscale x 8 x i16> %a, %b
-  %min = select <vscale x 8 x i1> %cmp, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b
-  ret <vscale x 8 x i16> %min
-}
-
-define <vscale x 4 x i32> @smin_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
-; CHECK-LABEL: smin_i32:
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: smin z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %cmp = icmp slt <vscale x 4 x i32> %a, %b
-  %min = select <vscale x 4 x i1> %cmp, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b
-  ret <vscale x 4 x i32> %min
-}
-
-define <vscale x 2 x i64> @smin_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
-; CHECK-LABEL: smin_i64:
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: smin z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %cmp = icmp slt <vscale x 2 x i64> %a, %b
-  %min = select <vscale x 2 x i1> %cmp, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b
-  ret <vscale x 2 x i64> %min
-}
-
-define <vscale x 32 x i8> @smin_split_i8(<vscale x 32 x i8> %a, <vscale x 32 x i8> %b, <vscale x 32 x i8> %c) {
-; CHECK-LABEL: @smin_split_i8
-; CHECK-DAG: ptrue p0.b
-; CHECK-DAG: smin z0.b, p0/m, z0.b, z2.b
-; CHECK-DAG: smin z1.b, p0/m, z1.b, z3.b
-; CHECK-NEXT: ret
-  %cmp = icmp slt <vscale x 32 x i8> %a, %b
-  %min = select <vscale x 32 x i1> %cmp, <vscale x 32 x i8> %a, <vscale x 32 x i8> %b
-  ret <vscale x 32 x i8> %min
-}
-
-define <vscale x 32 x i16> @smin_split_i16(<vscale x 32 x i16> %a, <vscale x 32 x i16> %b, <vscale x 32 x i16> %c) {
-; CHECK-LABEL: smin_split_i16:
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: smin z0.h, p0/m, z0.h, z4.h
-; CHECK-DAG: smin z1.h, p0/m, z1.h, z5.h
-; CHECK-DAG: smin z2.h, p0/m, z2.h, z6.h
-; CHECK-DAG: smin z3.h, p0/m, z3.h, z7.h
-; CHECK-NEXT: ret
-  %cmp = icmp slt <vscale x 32 x i16> %a, %b
-  %min = select <vscale x 32 x i1> %cmp, <vscale x 32 x i16> %a, <vscale x 32 x i16> %b
-  ret <vscale x 32 x i16> %min
-}
-
-define <vscale x 8 x i32> @smin_split_i32(<vscale x 8 x i32> %a, <vscale x 8 x i32> %b, <vscale x 8 x i32> %c) {
-; CHECK-LABEL: smin_split_i32:
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: smin z0.s, p0/m, z0.s, z2.s
-; CHECK-DAG: smin z1.s, p0/m, z1.s, z3.s
-; CHECK-NEXT: ret
-  %cmp = icmp slt <vscale x 8 x i32> %a, %b
-  %min = select <vscale x 8 x i1> %cmp, <vscale x 8 x i32> %a, <vscale x 8 x i32> %b
-  ret <vscale x 8 x i32> %min
-}
-
-define <vscale x 4 x i64> @smin_split_i64(<vscale x 4 x i64> %a, <vscale x 4 x i64> %b, <vscale x 4 x i64> %c) {
-; CHECK-LABEL: smin_split_i64:
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: smin z0.d, p0/m, z0.d, z2.d
-; CHECK-DAG: smin z1.d, p0/m, z1.d, z3.d
-; CHECK-NEXT: ret
-  %cmp = icmp slt <vscale x 4 x i64> %a, %b
-  %min = select <vscale x 4 x i1> %cmp, <vscale x 4 x i64> %a, <vscale x 4 x i64> %b
-  ret <vscale x 4 x i64> %min
-}
-
-define <vscale x 8 x i8> @smin_promote_i8(<vscale x 8 x i8> %a, <vscale x 8 x i8> %b, <vscale x 8 x i8> %c) {
-; CHECK-LABEL: @smin_promote_i8
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: sxtb z1.h, p0/m, z1.h
-; CHECK-DAG: sxtb z0.h, p0/m, z0.h
-; CHECK-DAG: smin z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT: ret
-  %cmp = icmp slt <vscale x 8 x i8> %a, %b
-  %min = select <vscale x 8 x i1> %cmp, <vscale x 8 x i8> %a, <vscale x 8 x i8> %b
-  ret <vscale x 8 x i8> %min
-}
-
-define <vscale x 4 x i16> @smin_promote_i16(<vscale x 4 x i16> %a, <vscale x 4 x i16> %b, <vscale x 4 x i16> %c) {
-; CHECK-LABEL: @smin_promote_i16
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: sxth z1.s, p0/m, z1.s
-; CHECK-DAG: sxth z0.s, p0/m, z0.s
-; CHECK-DAG: smin z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %cmp = icmp slt <vscale x 4 x i16> %a, %b
-  %min = select <vscale x 4 x i1> %cmp, <vscale x 4 x i16> %a, <vscale x 4 x i16> %b
-  ret <vscale x 4 x i16> %min
-}
-
-define <vscale x 2 x i32> @smin_promote_i32(<vscale x 2 x i32> %a, <vscale x 2 x i32> %b, <vscale x 2 x i32> %c) {
-; CHECK-LABEL: @smin_promote_i32
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: sxtw z1.d, p0/m, z1.d
-; CHECK-DAG: sxtw z0.d, p0/m, z0.d
-; CHECK-DAG: smin z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %cmp = icmp slt <vscale x 2 x i32> %a, %b
-  %min = select <vscale x 2 x i1> %cmp, <vscale x 2 x i32> %a, <vscale x 2 x i32> %b
-  ret <vscale x 2 x i32> %min
-}
-
-;
-; UMIN
-;
-
-define <vscale x 16 x i8> @umin_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
-; CHECK-LABEL: @umin_i8
-; CHECK-DAG: ptrue p0.b
-; CHECK-DAG: umin z0.b, p0/m, z0.b, z1.b
-; CHECK-NEXT: ret
-  %cmp = icmp ult <vscale x 16 x i8> %a, %b
-  %min = select <vscale x 16 x i1> %cmp, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b
-  ret <vscale x 16 x i8> %min
-}
-
-define <vscale x 8 x i16> @umin_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
-; CHECK-LABEL: @umin_i16
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: umin z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT: ret
-  %cmp = icmp ult <vscale x 8 x i16> %a, %b
-  %min = select <vscale x 8 x i1> %cmp, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b
-  ret <vscale x 8 x i16> %min
-}
-
-define <vscale x 4 x i32> @umin_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
-; CHECK-LABEL: umin_i32:
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: umin z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %cmp = icmp ult <vscale x 4 x i32> %a, %b
-  %min = select <vscale x 4 x i1> %cmp, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b
-  ret <vscale x 4 x i32> %min
-}
-
-define <vscale x 2 x i64> @umin_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
-; CHECK-LABEL: umin_i64:
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: umin z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %cmp = icmp ult <vscale x 2 x i64> %a, %b
-  %min = select <vscale x 2 x i1> %cmp, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b
-  ret <vscale x 2 x i64> %min
-}
-
-define <vscale x 4 x i64> @umin_split_i64(<vscale x 4 x i64> %a, <vscale x 4 x i64> %b, <vscale x 4 x i64> %c) {
-; CHECK-LABEL: umin_split_i64:
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: umin z0.d, p0/m, z0.d, z2.d
-; CHECK-DAG: umin z1.d, p0/m, z1.d, z3.d
-; CHECK-NEXT: ret
-  %cmp = icmp ult <vscale x 4 x i64> %a, %b
-  %min = select <vscale x 4 x i1> %cmp, <vscale x 4 x i64> %a, <vscale x 4 x i64> %b
-  ret <vscale x 4 x i64> %min
-}
-
-define <vscale x 8 x i8> @umin_promote_i8(<vscale x 8 x i8> %a, <vscale x 8 x i8> %b, <vscale x 8 x i8> %c) {
-; CHECK-LABEL: @umin_promote_i8
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: umin z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT: ret
-  %cmp = icmp ult <vscale x 8 x i8> %a, %b
-  %min = select <vscale x 8 x i1> %cmp, <vscale x 8 x i8> %a, <vscale x 8 x i8> %b
-  ret <vscale x 8 x i8> %min
-}
-
-;
-; SMAX
-;
-
-define <vscale x 16 x i8> @smax_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
-; CHECK-LABEL: @smax_i8
-; CHECK-DAG: ptrue p0.b
-; CHECK-DAG: smax z0.b, p0/m, z0.b, z1.b
-; CHECK-NEXT: ret
-  %cmp = icmp sgt <vscale x 16 x i8> %a, %b
-  %max = select <vscale x 16 x i1> %cmp, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b
-  ret <vscale x 16 x i8> %max
-}
-
-define <vscale x 8 x i16> @smax_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
-; CHECK-LABEL: @smax_i16
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: smax z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT: ret
-  %cmp = icmp sgt <vscale x 8 x i16> %a, %b
-  %max = select <vscale x 8 x i1> %cmp, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b
-  ret <vscale x 8 x i16> %max
-}
-
-define <vscale x 4 x i32> @smax_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
-; CHECK-LABEL: smax_i32:
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: smax z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %cmp = icmp sgt <vscale x 4 x i32> %a, %b
-  %max = select <vscale x 4 x i1> %cmp, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b
-  ret <vscale x 4 x i32> %max
-}
-
-define <vscale x 2 x i64> @smax_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
-; CHECK-LABEL: smax_i64:
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: smax z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %cmp = icmp sgt <vscale x 2 x i64> %a, %b
-  %max = select <vscale x 2 x i1> %cmp, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b
-  ret <vscale x 2 x i64> %max
-}
-
-define <vscale x 8 x i32> @smax_split_i32(<vscale x 8 x i32> %a, <vscale x 8 x i32> %b, <vscale x 8 x i32> %c) {
-; CHECK-LABEL: smax_split_i32:
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: smax z0.s, p0/m, z0.s, z2.s
-; CHECK-DAG: smax z1.s, p0/m, z1.s, z3.s
-; CHECK-NEXT: ret
-  %cmp = icmp sgt <vscale x 8 x i32> %a, %b
-  %max = select <vscale x 8 x i1> %cmp, <vscale x 8 x i32> %a, <vscale x 8 x i32> %b
-  ret <vscale x 8 x i32> %max
-}
-
-define <vscale x 4 x i16> @smax_promote_i16(<vscale x 4 x i16> %a, <vscale x 4 x i16> %b, <vscale x 4 x i16> %c) {
-; CHECK-LABEL: @smax_promote_i16
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: smax z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %cmp = icmp sgt <vscale x 4 x i16> %a, %b
-  %max = select <vscale x 4 x i1> %cmp, <vscale x 4 x i16> %a, <vscale x 4 x i16> %b
-  ret <vscale x 4 x i16> %max
-}
-
-;
-; UMAX
-;
-
-define <vscale x 16 x i8> @umax_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
-; CHECK-LABEL: @umax_i8
-; CHECK-DAG: ptrue p0.b
-; CHECK-DAG: umax z0.b, p0/m, z0.b, z1.b
-; CHECK-NEXT: ret
-  %cmp = icmp ugt <vscale x 16 x i8> %a, %b
-  %max = select <vscale x 16 x i1> %cmp, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b
-  ret <vscale x 16 x i8> %max
-}
-
-define <vscale x 8 x i16> @umax_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
-; CHECK-LABEL: @umax_i16
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: umax z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT: ret
-  %cmp = icmp ugt <vscale x 8 x i16> %a, %b
-  %max = select <vscale x 8 x i1> %cmp, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b
-  ret <vscale x 8 x i16> %max
-}
-
-define <vscale x 4 x i32> @umax_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
-; CHECK-LABEL: umax_i32:
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: umax z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %cmp = icmp ugt <vscale x 4 x i32> %a, %b
-  %max = select <vscale x 4 x i1> %cmp, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b
-  ret <vscale x 4 x i32> %max
-}
-
-define <vscale x 2 x i64> @umax_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
-; CHECK-LABEL: umax_i64:
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: umax z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %cmp = icmp ugt <vscale x 2 x i64> %a, %b
-  %max = select <vscale x 2 x i1> %cmp, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b
-  ret <vscale x 2 x i64> %max
-}
-
-define <vscale x 16 x i16> @umax_split_i16(<vscale x 16 x i16> %a, <vscale x 16 x i16> %b, <vscale x 16 x i16> %c) {
-; CHECK-LABEL: umax_split_i16:
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: umax z0.h, p0/m, z0.h, z2.h
-; CHECK-DAG: umax z1.h, p0/m, z1.h, z3.h
-; CHECK-NEXT: ret
-  %cmp = icmp ugt <vscale x 16 x i16> %a, %b
-  %max = select <vscale x 16 x i1> %cmp, <vscale x 16 x i16> %a, <vscale x 16 x i16> %b
-  ret <vscale x 16 x i16> %max
-}
-
-define <vscale x 2 x i32> @umax_promote_i32(<vscale x 2 x i32> %a, <vscale x 2 x i32> %b, <vscale x 2 x i32> %c) {
-; CHECK-LABEL: @umax_promote_i32
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: umax z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %cmp = icmp ugt <vscale x 2 x i32> %a, %b
-  %max = select <vscale x 2 x i1> %cmp, <vscale x 2 x i32> %a, <vscale x 2 x i32> %b
-  ret <vscale x 2 x i32> %max
-}
-
-;
-; ASR
-;
-
-define <vscale x 16 x i8> @asr_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b){
-; CHECK-LABEL: @asr_i8
-; CHECK-DAG: ptrue p0.b
-; CHECK-DAG: asr z0.b, p0/m, z0.b, z1.b
-; CHECK-NEXT: ret
-  %shr = ashr <vscale x 16 x i8> %a, %b
-  ret <vscale x 16 x i8> %shr
-}
-
-define <vscale x 8 x i16> @asr_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b){
-; CHECK-LABEL: @asr_i16
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: asr z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT: ret
-  %shr = ashr <vscale x 8 x i16> %a, %b
-  ret <vscale x 8 x i16> %shr
-}
-
-define <vscale x 4 x i32> @asr_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b){
-; CHECK-LABEL: @asr_i32
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: asr z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %shr = ashr <vscale x 4 x i32> %a, %b
-  ret <vscale x 4 x i32> %shr
-}
-
-define <vscale x 2 x i64> @asr_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b){
-; CHECK-LABEL: @asr_i64
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: asr z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %shr = ashr <vscale x 2 x i64> %a, %b
-  ret <vscale x 2 x i64> %shr
-}
-
-define <vscale x 16 x i16> @asr_split_i16(<vscale x 16 x i16> %a, <vscale x 16 x i16> %b){
-; CHECK-LABEL: @asr_split_i16
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: asr z0.h, p0/m, z0.h, z2.h
-; CHECK-DAG: asr z1.h, p0/m, z1.h, z3.h
-; CHECK-NEXT: ret
-  %shr = ashr <vscale x 16 x i16> %a, %b
-  ret <vscale x 16 x i16> %shr
-}
-
-define <vscale x 2 x i32> @asr_promote_i32(<vscale x 2 x i32> %a, <vscale x 2 x i32> %b){
-; CHECK-LABEL: @asr_promote_i32
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: and z1.d, z1.d, #0xffffffff
-; CHECK-DAG: asr z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %shr = ashr <vscale x 2 x i32> %a, %b
-  ret <vscale x 2 x i32> %shr
-}
-
-;
-; LSL
-;
-
-define <vscale x 16 x i8> @lsl_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b){
-; CHECK-LABEL: @lsl_i8
-; CHECK-DAG: ptrue p0.b
-; CHECK-DAG: lsl z0.b, p0/m, z0.b, z1.b
-; CHECK-NEXT: ret
-  %shl = shl <vscale x 16 x i8> %a, %b
-  ret <vscale x 16 x i8> %shl
-}
-
-define <vscale x 8 x i16> @lsl_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b){
-; CHECK-LABEL: @lsl_i16
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: lsl z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT: ret
-  %shl = shl <vscale x 8 x i16> %a, %b
-  ret <vscale x 8 x i16> %shl
-}
-
-define <vscale x 4 x i32> @lsl_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b){
-; CHECK-LABEL: @lsl_i32
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: lsl z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %shl = shl <vscale x 4 x i32> %a, %b
-  ret <vscale x 4 x i32> %shl
-}
-
-define <vscale x 2 x i64> @lsl_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b){
-; CHECK-LABEL: @lsl_i64
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: lsl z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %shl = shl <vscale x 2 x i64> %a, %b
-  ret <vscale x 2 x i64> %shl
-}
-
-define <vscale x 4 x i64> @lsl_split_i64(<vscale x 4 x i64> %a, <vscale x 4 x i64> %b){
-; CHECK-LABEL: @lsl_split_i64
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: lsl z0.d, p0/m, z0.d, z2.d
-; CHECK-DAG: lsl z1.d, p0/m, z1.d, z3.d
-; CHECK-NEXT: ret
-  %shl = shl <vscale x 4 x i64> %a, %b
-  ret <vscale x 4 x i64> %shl
-}
-
-define <vscale x 4 x i16> @lsl_promote_i16(<vscale x 4 x i16> %a, <vscale x 4 x i16> %b){
-; CHECK-LABEL: @lsl_promote_i16
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: and z1.s, z1.s, #0xffff
-; CHECK-DAG: lsl z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %shl = shl <vscale x 4 x i16> %a, %b
-  ret <vscale x 4 x i16> %shl
-}
-
-;
-; LSR
-;
-
-define <vscale x 16 x i8> @lsr_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b){
-; CHECK-LABEL: @lsr_i8
-; CHECK-DAG: ptrue p0.b
-; CHECK-DAG: lsr z0.b, p0/m, z0.b, z1.b
-; CHECK-NEXT: ret
-  %shr = lshr <vscale x 16 x i8> %a, %b
-  ret <vscale x 16 x i8> %shr
-}
-
-define <vscale x 8 x i16> @lsr_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b){
-; CHECK-LABEL: @lsr_i16
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: lsr z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT: ret
-  %shr = lshr <vscale x 8 x i16> %a, %b
-  ret <vscale x 8 x i16> %shr
-}
-
-define <vscale x 4 x i32> @lsr_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b){
-; CHECK-LABEL: @lsr_i32
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: lsr z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT: ret
-  %shr = lshr <vscale x 4 x i32> %a, %b
-  ret <vscale x 4 x i32> %shr
-}
-
-define <vscale x 2 x i64> @lsr_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b){
-; CHECK-LABEL: @lsr_i64
-; CHECK-DAG: ptrue p0.d
-; CHECK-DAG: lsr z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT: ret
-  %shr = lshr <vscale x 2 x i64> %a, %b
-  ret <vscale x 2 x i64> %shr
-}
-
-define <vscale x 8 x i8> @lsr_promote_i8(<vscale x 8 x i8> %a, <vscale x 8 x i8> %b){
-; CHECK-LABEL: @lsr_promote_i8
-; CHECK-DAG: ptrue p0.h
-; CHECK-DAG: and z1.h, z1.h, #0xff
-; CHECK-DAG: lsr z0.h, p0/m, z0.h, z1.h
-; CHECK-NEXT: ret
-  %shr = lshr <vscale x 8 x i8> %a, %b
-  ret <vscale x 8 x i8> %shr
-}
-
-define <vscale x 8 x i32> @lsr_split_i32(<vscale x 8 x i32> %a, <vscale x 8 x i32> %b){
-; CHECK-LABEL: @lsr_split_i32
-; CHECK-DAG: ptrue p0.s
-; CHECK-DAG: lsr z0.s, p0/m, z0.s, z2.s
-; CHECK-DAG: lsr z1.s, p0/m, z1.s, z3.s
-; CHECK-NEXT: ret
-  %shr = lshr <vscale x 8 x i32> %a, %b
-  ret <vscale x 8 x i32> %shr
-}

llvm/test/CodeGen/AArch64/sve-arith.ll

+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
+
+;
+; SDIV
+;
+
+define <vscale x 4 x i32> @sdiv_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
+; CHECK-LABEL: @sdiv_i32
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: sdiv z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %div = sdiv <vscale x 4 x i32> %a, %b
+  ret <vscale x 4 x i32> %div
+}
+
+define <vscale x 2 x i64> @sdiv_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
+; CHECK-LABEL: @sdiv_i64
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: sdiv z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %div = sdiv <vscale x 2 x i64> %a, %b
+  ret <vscale x 2 x i64> %div
+}
+
+define <vscale x 8 x i32> @sdiv_split_i32(<vscale x 8 x i32> %a, <vscale x 8 x i32> %b) {
+; CHECK-LABEL: @sdiv_split_i32
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: sdiv z0.s, p0/m, z0.s, z2.s
+; CHECK-DAG: sdiv z1.s, p0/m, z1.s, z3.s
+; CHECK-NEXT: ret
+  %div = sdiv <vscale x 8 x i32> %a, %b
+  ret <vscale x 8 x i32> %div
+}
+
+define <vscale x 2 x i32> @sdiv_promote_i32(<vscale x 2 x i32> %a, <vscale x 2 x i32> %b) {
+; CHECK-LABEL: @sdiv_promote_i32
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: sxtw z1.d, p0/m, z1.d
+; CHECK-DAG: sxtw z0.d, p0/m, z0.d
+; CHECK-DAG: sdiv z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %div = sdiv <vscale x 2 x i32> %a, %b
+  ret <vscale x 2 x i32> %div
+}
+
+define <vscale x 4 x i64> @sdiv_split_i64(<vscale x 4 x i64> %a, <vscale x 4 x i64> %b) {
+; CHECK-LABEL: @sdiv_split_i64
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: sdiv z0.d, p0/m, z0.d, z2.d
+; CHECK-DAG: sdiv z1.d, p0/m, z1.d, z3.d
+; CHECK-NEXT: ret
+  %div = sdiv <vscale x 4 x i64> %a, %b
+  ret <vscale x 4 x i64> %div
+}
+
+;
+; UDIV
+;
+
+define <vscale x 4 x i32> @udiv_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
+; CHECK-LABEL: @udiv_i32
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: udiv z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %div = udiv <vscale x 4 x i32> %a, %b
+  ret <vscale x 4 x i32> %div
+}
+
+define <vscale x 2 x i64> @udiv_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
+; CHECK-LABEL: @udiv_i64
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: udiv z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %div = udiv <vscale x 2 x i64> %a, %b
+  ret <vscale x 2 x i64> %div
+}
+
+define <vscale x 8 x i32> @udiv_split_i32(<vscale x 8 x i32> %a, <vscale x 8 x i32> %b) {
+; CHECK-LABEL: @udiv_split_i32
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: udiv z0.s, p0/m, z0.s, z2.s
+; CHECK-DAG: udiv z1.s, p0/m, z1.s, z3.s
+; CHECK-NEXT: ret
+  %div = udiv <vscale x 8 x i32> %a, %b
+  ret <vscale x 8 x i32> %div
+}
+
+define <vscale x 2 x i32> @udiv_promote_i32(<vscale x 2 x i32> %a, <vscale x 2 x i32> %b) {
+; CHECK-LABEL: @udiv_promote_i32
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: and z1.d, z1.d, #0xffffffff
+; CHECK-DAG: and z0.d, z0.d, #0xffffffff
+; CHECK-DAG: udiv z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %div = udiv <vscale x 2 x i32> %a, %b
+  ret <vscale x 2 x i32> %div
+}
+
+define <vscale x 4 x i64> @udiv_split_i64(<vscale x 4 x i64> %a, <vscale x 4 x i64> %b) {
+; CHECK-LABEL: @udiv_split_i64
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: udiv z0.d, p0/m, z0.d, z2.d
+; CHECK-DAG: udiv z1.d, p0/m, z1.d, z3.d
+; CHECK-NEXT: ret
+  %div = udiv <vscale x 4 x i64> %a, %b
+  ret <vscale x 4 x i64> %div
+}
+
+;
+; SMIN
+;
+
+define <vscale x 16 x i8> @smin_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
+; CHECK-LABEL: @smin_i8
+; CHECK-DAG: ptrue p0.b
+; CHECK-DAG: smin z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT: ret
+  %cmp = icmp slt <vscale x 16 x i8> %a, %b
+  %min = select <vscale x 16 x i1> %cmp, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b
+  ret <vscale x 16 x i8> %min
+}
+
+define <vscale x 8 x i16> @smin_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
+; CHECK-LABEL: @smin_i16
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: smin z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT: ret
+  %cmp = icmp slt <vscale x 8 x i16> %a, %b
+  %min = select <vscale x 8 x i1> %cmp, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b
+  ret <vscale x 8 x i16> %min
+}
+
+define <vscale x 4 x i32> @smin_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
+; CHECK-LABEL: smin_i32:
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: smin z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %cmp = icmp slt <vscale x 4 x i32> %a, %b
+  %min = select <vscale x 4 x i1> %cmp, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b
+  ret <vscale x 4 x i32> %min
+}
+
+define <vscale x 2 x i64> @smin_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
+; CHECK-LABEL: smin_i64:
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: smin z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %cmp = icmp slt <vscale x 2 x i64> %a, %b
+  %min = select <vscale x 2 x i1> %cmp, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b
+  ret <vscale x 2 x i64> %min
+}
+
+define <vscale x 32 x i8> @smin_split_i8(<vscale x 32 x i8> %a, <vscale x 32 x i8> %b, <vscale x 32 x i8> %c) {
+; CHECK-LABEL: @smin_split_i8
+; CHECK-DAG: ptrue p0.b
+; CHECK-DAG: smin z0.b, p0/m, z0.b, z2.b
+; CHECK-DAG: smin z1.b, p0/m, z1.b, z3.b
+; CHECK-NEXT: ret
+  %cmp = icmp slt <vscale x 32 x i8> %a, %b
+  %min = select <vscale x 32 x i1> %cmp, <vscale x 32 x i8> %a, <vscale x 32 x i8> %b
+  ret <vscale x 32 x i8> %min
+}
+
+define <vscale x 32 x i16> @smin_split_i16(<vscale x 32 x i16> %a, <vscale x 32 x i16> %b, <vscale x 32 x i16> %c) {
+; CHECK-LABEL: smin_split_i16:
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: smin z0.h, p0/m, z0.h, z4.h
+; CHECK-DAG: smin z1.h, p0/m, z1.h, z5.h
+; CHECK-DAG: smin z2.h, p0/m, z2.h, z6.h
+; CHECK-DAG: smin z3.h, p0/m, z3.h, z7.h
+; CHECK-NEXT: ret
+  %cmp = icmp slt <vscale x 32 x i16> %a, %b
+  %min = select <vscale x 32 x i1> %cmp, <vscale x 32 x i16> %a, <vscale x 32 x i16> %b
+  ret <vscale x 32 x i16> %min
+}
+
+define <vscale x 8 x i32> @smin_split_i32(<vscale x 8 x i32> %a, <vscale x 8 x i32> %b, <vscale x 8 x i32> %c) {
+; CHECK-LABEL: smin_split_i32:
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: smin z0.s, p0/m, z0.s, z2.s
+; CHECK-DAG: smin z1.s, p0/m, z1.s, z3.s
+; CHECK-NEXT: ret
+  %cmp = icmp slt <vscale x 8 x i32> %a, %b
+  %min = select <vscale x 8 x i1> %cmp, <vscale x 8 x i32> %a, <vscale x 8 x i32> %b
+  ret <vscale x 8 x i32> %min
+}
+
+define <vscale x 4 x i64> @smin_split_i64(<vscale x 4 x i64> %a, <vscale x 4 x i64> %b, <vscale x 4 x i64> %c) {
+; CHECK-LABEL: smin_split_i64:
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: smin z0.d, p0/m, z0.d, z2.d
+; CHECK-DAG: smin z1.d, p0/m, z1.d, z3.d
+; CHECK-NEXT: ret
+  %cmp = icmp slt <vscale x 4 x i64> %a, %b
+  %min = select <vscale x 4 x i1> %cmp, <vscale x 4 x i64> %a, <vscale x 4 x i64> %b
+  ret <vscale x 4 x i64> %min
+}
+
+define <vscale x 8 x i8> @smin_promote_i8(<vscale x 8 x i8> %a, <vscale x 8 x i8> %b, <vscale x 8 x i8> %c) {
+; CHECK-LABEL: @smin_promote_i8
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: sxtb z1.h, p0/m, z1.h
+; CHECK-DAG: sxtb z0.h, p0/m, z0.h
+; CHECK-DAG: smin z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT: ret
+  %cmp = icmp slt <vscale x 8 x i8> %a, %b
+  %min = select <vscale x 8 x i1> %cmp, <vscale x 8 x i8> %a, <vscale x 8 x i8> %b
+  ret <vscale x 8 x i8> %min
+}
+
+define <vscale x 4 x i16> @smin_promote_i16(<vscale x 4 x i16> %a, <vscale x 4 x i16> %b, <vscale x 4 x i16> %c) {
+; CHECK-LABEL: @smin_promote_i16
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: sxth z1.s, p0/m, z1.s
+; CHECK-DAG: sxth z0.s, p0/m, z0.s
+; CHECK-DAG: smin z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %cmp = icmp slt <vscale x 4 x i16> %a, %b
+  %min = select <vscale x 4 x i1> %cmp, <vscale x 4 x i16> %a, <vscale x 4 x i16> %b
+  ret <vscale x 4 x i16> %min
+}
+
+define <vscale x 2 x i32> @smin_promote_i32(<vscale x 2 x i32> %a, <vscale x 2 x i32> %b, <vscale x 2 x i32> %c) {
+; CHECK-LABEL: @smin_promote_i32
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: sxtw z1.d, p0/m, z1.d
+; CHECK-DAG: sxtw z0.d, p0/m, z0.d
+; CHECK-DAG: smin z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %cmp = icmp slt <vscale x 2 x i32> %a, %b
+  %min = select <vscale x 2 x i1> %cmp, <vscale x 2 x i32> %a, <vscale x 2 x i32> %b
+  ret <vscale x 2 x i32> %min
+}
+
+;
+; UMIN
+;
+
+define <vscale x 16 x i8> @umin_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
+; CHECK-LABEL: @umin_i8
+; CHECK-DAG: ptrue p0.b
+; CHECK-DAG: umin z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT: ret
+  %cmp = icmp ult <vscale x 16 x i8> %a, %b
+  %min = select <vscale x 16 x i1> %cmp, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b
+  ret <vscale x 16 x i8> %min
+}
+
+define <vscale x 8 x i16> @umin_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
+; CHECK-LABEL: @umin_i16
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: umin z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT: ret
+  %cmp = icmp ult <vscale x 8 x i16> %a, %b
+  %min = select <vscale x 8 x i1> %cmp, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b
+  ret <vscale x 8 x i16> %min
+}
+
+define <vscale x 4 x i32> @umin_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
+; CHECK-LABEL: umin_i32:
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: umin z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %cmp = icmp ult <vscale x 4 x i32> %a, %b
+  %min = select <vscale x 4 x i1> %cmp, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b
+  ret <vscale x 4 x i32> %min
+}
+
+define <vscale x 2 x i64> @umin_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
+; CHECK-LABEL: umin_i64:
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: umin z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %cmp = icmp ult <vscale x 2 x i64> %a, %b
+  %min = select <vscale x 2 x i1> %cmp, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b
+  ret <vscale x 2 x i64> %min
+}
+
+define <vscale x 4 x i64> @umin_split_i64(<vscale x 4 x i64> %a, <vscale x 4 x i64> %b, <vscale x 4 x i64> %c) {
+; CHECK-LABEL: umin_split_i64:
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: umin z0.d, p0/m, z0.d, z2.d
+; CHECK-DAG: umin z1.d, p0/m, z1.d, z3.d
+; CHECK-NEXT: ret
+  %cmp = icmp ult <vscale x 4 x i64> %a, %b
+  %min = select <vscale x 4 x i1> %cmp, <vscale x 4 x i64> %a, <vscale x 4 x i64> %b
+  ret <vscale x 4 x i64> %min
+}
+
+define <vscale x 8 x i8> @umin_promote_i8(<vscale x 8 x i8> %a, <vscale x 8 x i8> %b, <vscale x 8 x i8> %c) {
+; CHECK-LABEL: @umin_promote_i8
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: umin z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT: ret
+  %cmp = icmp ult <vscale x 8 x i8> %a, %b
+  %min = select <vscale x 8 x i1> %cmp, <vscale x 8 x i8> %a, <vscale x 8 x i8> %b
+  ret <vscale x 8 x i8> %min
+}
+
+;
+; SMAX
+;
+
+define <vscale x 16 x i8> @smax_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
+; CHECK-LABEL: @smax_i8
+; CHECK-DAG: ptrue p0.b
+; CHECK-DAG: smax z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT: ret
+  %cmp = icmp sgt <vscale x 16 x i8> %a, %b
+  %max = select <vscale x 16 x i1> %cmp, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b
+  ret <vscale x 16 x i8> %max
+}
+
+define <vscale x 8 x i16> @smax_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
+; CHECK-LABEL: @smax_i16
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: smax z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT: ret
+  %cmp = icmp sgt <vscale x 8 x i16> %a, %b
+  %max = select <vscale x 8 x i1> %cmp, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b
+  ret <vscale x 8 x i16> %max
+}
+
+define <vscale x 4 x i32> @smax_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
+; CHECK-LABEL: smax_i32:
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: smax z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %cmp = icmp sgt <vscale x 4 x i32> %a, %b
+  %max = select <vscale x 4 x i1> %cmp, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b
+  ret <vscale x 4 x i32> %max
+}
+
+define <vscale x 2 x i64> @smax_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
+; CHECK-LABEL: smax_i64:
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: smax z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %cmp = icmp sgt <vscale x 2 x i64> %a, %b
+  %max = select <vscale x 2 x i1> %cmp, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b
+  ret <vscale x 2 x i64> %max
+}
+
+define <vscale x 8 x i32> @smax_split_i32(<vscale x 8 x i32> %a, <vscale x 8 x i32> %b, <vscale x 8 x i32> %c) {
+; CHECK-LABEL: smax_split_i32:
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: smax z0.s, p0/m, z0.s, z2.s
+; CHECK-DAG: smax z1.s, p0/m, z1.s, z3.s
+; CHECK-NEXT: ret
+  %cmp = icmp sgt <vscale x 8 x i32> %a, %b
+  %max = select <vscale x 8 x i1> %cmp, <vscale x 8 x i32> %a, <vscale x 8 x i32> %b
+  ret <vscale x 8 x i32> %max
+}
+
+define <vscale x 4 x i16> @smax_promote_i16(<vscale x 4 x i16> %a, <vscale x 4 x i16> %b, <vscale x 4 x i16> %c) {
+; CHECK-LABEL: @smax_promote_i16
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: smax z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %cmp = icmp sgt <vscale x 4 x i16> %a, %b
+  %max = select <vscale x 4 x i1> %cmp, <vscale x 4 x i16> %a, <vscale x 4 x i16> %b
+  ret <vscale x 4 x i16> %max
+}
+
+;
+; UMAX
+;
+
+define <vscale x 16 x i8> @umax_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b, <vscale x 16 x i8> %c) {
+; CHECK-LABEL: @umax_i8
+; CHECK-DAG: ptrue p0.b
+; CHECK-DAG: umax z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT: ret
+  %cmp = icmp ugt <vscale x 16 x i8> %a, %b
+  %max = select <vscale x 16 x i1> %cmp, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b
+  ret <vscale x 16 x i8> %max
+}
+
+define <vscale x 8 x i16> @umax_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b, <vscale x 8 x i16> %c) {
+; CHECK-LABEL: @umax_i16
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: umax z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT: ret
+  %cmp = icmp ugt <vscale x 8 x i16> %a, %b
+  %max = select <vscale x 8 x i1> %cmp, <vscale x 8 x i16> %a, <vscale x 8 x i16> %b
+  ret <vscale x 8 x i16> %max
+}
+
+define <vscale x 4 x i32> @umax_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b, <vscale x 4 x i32> %c) {
+; CHECK-LABEL: umax_i32:
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: umax z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %cmp = icmp ugt <vscale x 4 x i32> %a, %b
+  %max = select <vscale x 4 x i1> %cmp, <vscale x 4 x i32> %a, <vscale x 4 x i32> %b
+  ret <vscale x 4 x i32> %max
+}
+
+define <vscale x 2 x i64> @umax_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b, <vscale x 2 x i64> %c) {
+; CHECK-LABEL: umax_i64:
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: umax z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %cmp = icmp ugt <vscale x 2 x i64> %a, %b
+  %max = select <vscale x 2 x i1> %cmp, <vscale x 2 x i64> %a, <vscale x 2 x i64> %b
+  ret <vscale x 2 x i64> %max
+}
+
+define <vscale x 16 x i16> @umax_split_i16(<vscale x 16 x i16> %a, <vscale x 16 x i16> %b, <vscale x 16 x i16> %c) {
+; CHECK-LABEL: umax_split_i16:
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: umax z0.h, p0/m, z0.h, z2.h
+; CHECK-DAG: umax z1.h, p0/m, z1.h, z3.h
+; CHECK-NEXT: ret
+  %cmp = icmp ugt <vscale x 16 x i16> %a, %b
+  %max = select <vscale x 16 x i1> %cmp, <vscale x 16 x i16> %a, <vscale x 16 x i16> %b
+  ret <vscale x 16 x i16> %max
+}
+
+define <vscale x 2 x i32> @umax_promote_i32(<vscale x 2 x i32> %a, <vscale x 2 x i32> %b, <vscale x 2 x i32> %c) {
+; CHECK-LABEL: @umax_promote_i32
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: umax z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %cmp = icmp ugt <vscale x 2 x i32> %a, %b
+  %max = select <vscale x 2 x i1> %cmp, <vscale x 2 x i32> %a, <vscale x 2 x i32> %b
+  ret <vscale x 2 x i32> %max
+}
+
+;
+; ASR
+;
+
+define <vscale x 16 x i8> @asr_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b){
+; CHECK-LABEL: @asr_i8
+; CHECK-DAG: ptrue p0.b
+; CHECK-DAG: asr z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT: ret
+  %shr = ashr <vscale x 16 x i8> %a, %b
+  ret <vscale x 16 x i8> %shr
+}
+
+define <vscale x 8 x i16> @asr_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b){
+; CHECK-LABEL: @asr_i16
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT: ret
+  %shr = ashr <vscale x 8 x i16> %a, %b
+  ret <vscale x 8 x i16> %shr
+}
+
+define <vscale x 4 x i32> @asr_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b){
+; CHECK-LABEL: @asr_i32
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: asr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %shr = ashr <vscale x 4 x i32> %a, %b
+  ret <vscale x 4 x i32> %shr
+}
+
+define <vscale x 2 x i64> @asr_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b){
+; CHECK-LABEL: @asr_i64
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: asr z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %shr = ashr <vscale x 2 x i64> %a, %b
+  ret <vscale x 2 x i64> %shr
+}
+
+define <vscale x 16 x i16> @asr_split_i16(<vscale x 16 x i16> %a, <vscale x 16 x i16> %b){
+; CHECK-LABEL: @asr_split_i16
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: asr z0.h, p0/m, z0.h, z2.h
+; CHECK-DAG: asr z1.h, p0/m, z1.h, z3.h
+; CHECK-NEXT: ret
+  %shr = ashr <vscale x 16 x i16> %a, %b
+  ret <vscale x 16 x i16> %shr
+}
+
+define <vscale x 2 x i32> @asr_promote_i32(<vscale x 2 x i32> %a, <vscale x 2 x i32> %b){
+; CHECK-LABEL: @asr_promote_i32
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: and z1.d, z1.d, #0xffffffff
+; CHECK-DAG: asr z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %shr = ashr <vscale x 2 x i32> %a, %b
+  ret <vscale x 2 x i32> %shr
+}
+
+;
+; LSL
+;
+
+define <vscale x 16 x i8> @lsl_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b){
+; CHECK-LABEL: @lsl_i8
+; CHECK-DAG: ptrue p0.b
+; CHECK-DAG: lsl z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT: ret
+  %shl = shl <vscale x 16 x i8> %a, %b
+  ret <vscale x 16 x i8> %shl
+}
+
+define <vscale x 8 x i16> @lsl_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b){
+; CHECK-LABEL: @lsl_i16
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT: ret
+  %shl = shl <vscale x 8 x i16> %a, %b
+  ret <vscale x 8 x i16> %shl
+}
+
+define <vscale x 4 x i32> @lsl_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b){
+; CHECK-LABEL: @lsl_i32
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: lsl z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %shl = shl <vscale x 4 x i32> %a, %b
+  ret <vscale x 4 x i32> %shl
+}
+
+define <vscale x 2 x i64> @lsl_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b){
+; CHECK-LABEL: @lsl_i64
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: lsl z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %shl = shl <vscale x 2 x i64> %a, %b
+  ret <vscale x 2 x i64> %shl
+}
+
+define <vscale x 4 x i64> @lsl_split_i64(<vscale x 4 x i64> %a, <vscale x 4 x i64> %b){
+; CHECK-LABEL: @lsl_split_i64
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: lsl z0.d, p0/m, z0.d, z2.d
+; CHECK-DAG: lsl z1.d, p0/m, z1.d, z3.d
+; CHECK-NEXT: ret
+  %shl = shl <vscale x 4 x i64> %a, %b
+  ret <vscale x 4 x i64> %shl
+}
+
+define <vscale x 4 x i16> @lsl_promote_i16(<vscale x 4 x i16> %a, <vscale x 4 x i16> %b){
+; CHECK-LABEL: @lsl_promote_i16
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: and z1.s, z1.s, #0xffff
+; CHECK-DAG: lsl z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %shl = shl <vscale x 4 x i16> %a, %b
+  ret <vscale x 4 x i16> %shl
+}
+
+;
+; LSR
+;
+
+define <vscale x 16 x i8> @lsr_i8(<vscale x 16 x i8> %a, <vscale x 16 x i8> %b){
+; CHECK-LABEL: @lsr_i8
+; CHECK-DAG: ptrue p0.b
+; CHECK-DAG: lsr z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT: ret
+  %shr = lshr <vscale x 16 x i8> %a, %b
+  ret <vscale x 16 x i8> %shr
+}
+
+define <vscale x 8 x i16> @lsr_i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b){
+; CHECK-LABEL: @lsr_i16
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: lsr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT: ret
+  %shr = lshr <vscale x 8 x i16> %a, %b
+  ret <vscale x 8 x i16> %shr
+}
+
+define <vscale x 4 x i32> @lsr_i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b){
+; CHECK-LABEL: @lsr_i32
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: lsr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT: ret
+  %shr = lshr <vscale x 4 x i32> %a, %b
+  ret <vscale x 4 x i32> %shr
+}
+
+define <vscale x 2 x i64> @lsr_i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b){
+; CHECK-LABEL: @lsr_i64
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: lsr z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT: ret
+  %shr = lshr <vscale x 2 x i64> %a, %b
+  ret <vscale x 2 x i64> %shr
+}
+
+define <vscale x 8 x i8> @lsr_promote_i8(<vscale x 8 x i8> %a, <vscale x 8 x i8> %b){
+; CHECK-LABEL: @lsr_promote_i8
+; CHECK-DAG: ptrue p0.h
+; CHECK-DAG: and z1.h, z1.h, #0xff
+; CHECK-DAG: lsr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT: ret
+  %shr = lshr <vscale x 8 x i8> %a, %b
+  ret <vscale x 8 x i8> %shr
+}
+
+define <vscale x 8 x i32> @lsr_split_i32(<vscale x 8 x i32> %a, <vscale x 8 x i32> %b){
+; CHECK-LABEL: @lsr_split_i32
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: lsr z0.s, p0/m, z0.s, z2.s
+; CHECK-DAG: lsr z1.s, p0/m, z1.s, z3.s
+; CHECK-NEXT: ret
+  %shr = lshr <vscale x 8 x i32> %a, %b
+  ret <vscale x 8 x i32> %shr
+}

llvm/test/CodeGen/AArch64/sve-ext.ll

+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
+
+;
+; SEXT
+;
+
+define <vscale x 16 x i16> @sext_b_to_h(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: sext_b_to_h:
+; CHECK-DAG: sunpklo z2.h, z0.b
+; CHECK-DAG: sunpkhi z1.h, z0.b
+; CHECK-DAG: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %ext = sext <vscale x 16 x i8> %a to <vscale x 16 x i16>
+  ret <vscale x 16 x i16> %ext
+}
+
+define <vscale x 8 x i32> @sext_h_to_s(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: sext_h_to_s:
+; CHECK-DAG: sunpklo z2.s, z0.h
+; CHECK-DAG: sunpkhi z1.s, z0.h
+; CHECK-DAG: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %ext = sext <vscale x 8 x i16> %a to <vscale x 8 x i32>
+  ret <vscale x 8 x i32> %ext
+}
+
+define <vscale x 4 x i64> @sext_s_to_d(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: sext_s_to_d:
+; CHECK-DAG: sunpklo z2.d, z0.s
+; CHECK-DAG: sunpkhi z1.d, z0.s
+; CHECK-DAG: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %ext = sext <vscale x 4 x i32> %a to <vscale x 4 x i64>
+  ret <vscale x 4 x i64> %ext
+}
+
+define <vscale x 16 x i32> @sext_b_to_s(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: sext_b_to_s:
+; CHECK-DAG: sunpklo [[LO:z[0-9]+]].h, z0.b
+; CHECK-DAG: sunpkhi [[HI:z[0-9]+]].h, z0.b
+; CHECK-DAG: sunpklo [[LOLO:z[0-9]+]].s, [[LO]].h
+; CHECK-DAG: sunpkhi {{z[0-9]+}}.s, [[LO]].h
+; CHECK-DAG: sunpklo {{z[0-9]+}}.s, [[HI]].h
+; CHECK-DAG: sunpkhi {{z[0-9]+}}.s, [[HI]].h
+; CHECK: ret
+  %ext = sext <vscale x 16 x i8> %a to <vscale x 16 x i32>
+  ret <vscale x 16 x i32> %ext
+}
+
+define <vscale x 4 x i16> @sext_promote_b_to_s(<vscale x 4 x i8> %in) {
+; CHECK-LABEL: @sext_promote
+; CHECK-DAG: ptrue p0.s
+; CHECK-DAG: sxtb z0.s, p0/m, z0.s
+; CHECK-NEXT: ret
+  %out = sext <vscale x 4 x i8> %in to <vscale x 4 x i16>
+  ret <vscale x 4 x i16> %out
+}
+
+define <vscale x 2 x i32> @sext_promote_h_to_d(<vscale x 2 x i16> %in) {
+; CHECK-LABEL: @sext_promote_h_to_d
+; CHECK-DAG: ptrue p0.d
+; CHECK-DAG: sxth z0.d, p0/m, z0.d
+; CHECK-NEXT: ret
+  %out = sext <vscale x 2 x i16> %in to <vscale x 2 x i32>
+  ret <vscale x 2 x i32> %out
+}
+
+; ZEXT
+
+define <vscale x 16 x i16> @zext_b_to_h(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: zext_b_to_h:
+; CHECK-DAG: uunpklo z2.h, z0.b
+; CHECK-DAG: uunpkhi z1.h, z0.b
+; CHECK-DAG: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %ext = zext <vscale x 16 x i8> %a to <vscale x 16 x i16>
+  ret <vscale x 16 x i16> %ext
+}
+
+define <vscale x 8 x i32> @zext_h_to_s(<vscale x 8 x i16> %a) {
+; CHECK-LABEL: zext_h_to_s:
+; CHECK-DAG: uunpklo z2.s, z0.h
+; CHECK-DAG: uunpkhi z1.s, z0.h
+; CHECK-DAG: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %ext = zext <vscale x 8 x i16> %a to <vscale x 8 x i32>
+  ret <vscale x 8 x i32> %ext
+}
+
+define <vscale x 4 x i64> @zext_s_to_d(<vscale x 4 x i32> %a) {
+; CHECK-LABEL: zext_s_to_d:
+; CHECK-DAG: uunpklo z2.d, z0.s
+; CHECK-DAG: uunpkhi z1.d, z0.s
+; CHECK-DAG: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %ext = zext <vscale x 4 x i32> %a to <vscale x 4 x i64>
+  ret <vscale x 4 x i64> %ext
+}
+
+define <vscale x 16 x i32> @zext_b_to_s(<vscale x 16 x i8> %a) {
+; CHECK-LABEL: zext_b_to_s:
+; CHECK-DAG: uunpklo [[LO:z[0-9]+]].h, z0.b
+; CHECK-DAG: uunpkhi [[HI:z[0-9]+]].h, z0.b
+; CHECK-DAG: uunpklo z0.s, [[LO]].h
+; CHECK-DAG: uunpkhi z1.s, [[LO]].h
+; CHECK-DAG: uunpklo z2.s, [[HI]].h
+; CHECK-DAG: uunpkhi z3.s, [[HI]].h
+; CHECK: ret
+  %ext = zext <vscale x 16 x i8> %a to <vscale x 16 x i32>
+  ret <vscale x 16 x i32> %ext
+}
+
+define <vscale x 4 x i16> @zext_promote_b_to_s(<vscale x 4 x i8> %in) {
+; CHECK-LABEL: @zext_promote
+; CHECK-DAG: and z0.s, z0.s, #0xff
+; CHECK-NEXT: ret
+  %out = zext <vscale x 4 x i8> %in to <vscale x 4 x i16>
+  ret <vscale x 4 x i16> %out
+}
+
+define <vscale x 2 x i32> @zext_promote_h_to_d(<vscale x 2 x i16> %in) {
+; CHECK-LABEL: @zext_promote_h_to_d
+; CHECK-DAG: and z0.d, z0.d, #0xffff
+; CHECK-NEXT: ret
+  %out = zext <vscale x 2 x i16> %in to <vscale x 2 x i32>
+  ret <vscale x 2 x i32> %out
+}