[AArch64][SVE2] Combine add+lsr to rshrnb for stores

[MattDevereau]

Summary

[AArch64][SVE2] Combine add+lsr to rshrnb for stores

The example sequence

add z0.h, z0.h, #32
lsr z0.h, #6
st1b z0.h, x1

can be replaced with

rshrnb z0.b, #6
st1b z0.h, x1

As the top half of the destination elements are truncated.

In similar fashion,

add z0.s, z0.s, #32
lsr z1.s, z1.s, #6
add z1.s, z1.s, #32
lsr z0.s, z0.s, #6
uzp1 z0.h, z0.h, z1.h

Can be replaced with

rshrnb z1.h, z1.s, #6
rshrnb z0.h, z0.s, #6
uzp1 z0.h, z0.h, z1.h

[david-arm]

This is a nice optimisation @MattDevereau, thanks! I found there is also another case we could support with loops like this where the store doesn't come straight afterwards:

void foo(unsigned short *dest, unsigned short *src, long n) {
  for (long i = 0; i < n; i++)
    dest[i] += ((src[i] + 32) >> 6);
}

In this case the IR sequence is add, lshr, trunc since the truncate doesn't get absorbed into the store. Maybe it's worth seeing if you can reuse your code in tryCombineStoredNarrowShift for this case too?

[kmclaughlin]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20103

Hi @MattDevereau,
You might need to check the value returned by DAG.getSplatValue here as I think if the operand is not a splat as expected, the dyn_cast will fail.
Please can you add a negative test for this scenario as well?

[MattDevereau]

@kmclaughlin Thank you. I've added two tests @neg_trunc_lsr_add_op1_not_splat and @neg_trunc_lsr_op1_not_splat to bail out of emitting rshrnb when the RHS operands are not splat values.

I've also added a test @neg_add_has_two_uses and a check that the add does not have more than one use. Without this it is possible to generate this regression which costs 2 extra cycles

neg_add_two_use:
  ptrue p0.h
  ld1h { z0.h }, p0/z, [x0]
  rshrnb z1.b, z0.h, #6
  add z0.h, z0.h, #32 // =0x20
  add z0.h, z0.h, z0.h
  st1h { z0.h }, p0, [x2, x3, lsl #1]
  st1b { z1.h }, p0, [x1, x3]
  ret

[kmclaughlin]

Thank you for adding the new tests @MattDevereau, I just have a couple of small suggestions in the trySimplifySrlAddToRshrnb function.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20096

I think if you change this to dyn_cast_or_null you can remove the additional isSplatValue check at the beginning of the function.

20105

Similarly here, I think you can remove the isSplatValue above by using dyn_cast_or_null

[MattDevereau]

Replaced explicit checks for splat values with dyn_cast_or_null

[kmclaughlin]

Thanks @MattDevereau, there's one nit but otherwise LGTM!

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20113–20119

nit: The braces here aren't necessary

20126–20132

As above :)

[dmgreen]

Hello. When adding these for NEON we were able to the transform via tablegen patterns at selection time, which has some benefits about leaving the nodes generic for as long as possible. In this case because of the differences in the instructions I would expect that it need demanded bits, so doing it as a combine probably makes sense.

It's best not to generate a MachineNode directly though if we can.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20115

We should avoid creating machine nodes in SelectionDAG combines. It's like a layering violation. Can you change this to either generate an intrinsic, or preferably add a new AArch64ISD node for it?

[hassnaa-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20110

Hi Mat,
I think that check doesn't consider the case when the AddValue has enabled bits other than the most significant bit.
ex: the ShiftValue is 6, and the AddValue is 33 (100001)b
Is that correct ?

20110

Do you handle the case when the destination of the AddOperation has an enabled bit in the same index of the enabled bit in the add Value ?
In that case the Add operation will has side affect, and by combining that case into rshrnb, you cancel the side effect of the AddOperation.
Do you agree on that ? or I'm missing something ??

[MattDevereau]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20110

I think that check doesn't consider the case when the AddValue has enabled bits other than the most significant bit.
ex: the ShiftValue is 6, and the AddValue is 33 (100001)b

This check would correctly reject that example, which is not a case for this combine. As per the documentation for rshrnb the operation is

integer res = (UInt(element) + (1 << (shift-1))) >> shift;

As 33 is not possible to get from left shifting 1, the combine will correctly bail. The lower bits in this case would be kept as the combine wouldn't fire.

20110

What side effect do you mean exactly?

Do you handle the case when the destination of the AddOperation has an enabled bit in the same index of the enabled bit in the add Value ?

I don't think the bits in the destination register of the add matter, register allocation should handle that fine and give us a non conflicting destination register.

20115

Thanks for pointing that out. I'm working on emitting a new AArch64ISD node now, however it adds a bit more complexity to the patch.

[hassnaa-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20110

Sorry I didn't express it clearly.
For example,
The add operation is src + (100000)b
I mean that what if the src has an enabled bit at the index corresponding to the '1' in the addedVal, index 5
so that means there will be 1+1 and then index 6 will be affected.
How that case is handled ?

[MattDevereau]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20110

Sorry but I don't understand what you mean by case. I think what you are referring to is the rounding behaviour of rshrnb which is the intended calculation and not a side effect.

[MattDevereau]

Emit an AArch64ISD node instead of the machine node directly.
Include D->S addressing mode of RSHRNB

[dmgreen]

Thanks. This looks good, but it might need to be using 1ULL for the shift value and there are few extra suggestions below. Otherwise LGTM

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20089

Can you either add a check that the VT is one that is valid for RSHRNB, or an assert that we only handle those types?

20109

This may need to be 1ULL << (ShiftValue - 1), as the number could be a 64bit value.

20112

trySimplifySrlAddToRshrnb could take a SDValue, instead of needing the cast.

20113–20115

SDValue Rshrnb = DAG.getNode(
      AArch64ISD::RSHRNB_I, DL, VT,
      Add->getOperand(0), DAG.getTargetConstant(ShiftValue, DL, MVT::i32));

(Or just return it directly)

20780

I think RshrnbVT is just ValueVT?

llvm/lib/Target/AArch64/AArch64ISelLowering.h

217 ↗

(On Diff #546405)

This might deserve to be in it's own little section. There may be a number of T/B nodes we end up adding, if we treat them in the same way.

llvm/lib/Target/AArch64/SVEInstrFormats.td

4309 ↗

(On Diff #546405)

It might be better to generate a nxv16i8->nxv8i16 RSHRN with a bitcast/nvcast back to nxv8i16.

[MattDevereau]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20089

Since were matching the Rshrnb ISD node emitted with the patterns in tablegen that match nvx8i16 -> nxv16i8, nxv4i32 -> nxv8i16, nxv2i64 -> nxv4i32 I've added an explicit block for this now:

EVT ResVT;
if (VT == MVT::nxv8i16)
  ResVT = MVT::nxv16i8;
else if (VT == MVT::nxv4i32)
  ResVT = MVT::nxv8i16;
else if (VT == MVT::nxv2i64)
  ResVT = MVT::nxv4i32;
else
  return SDValue();

We can use ResVT for the SDNode creation of Rshrnb and use VT for the bitcast back to the type that fits the DAG correctly.

20109

Done, and changed int64_t AddValue to uint64_t AddValue to line up with the change.

20112

I get the feeling this is a side-grade since I need to pass in SDLoc as an extra parameter this way, but I can see the cast looks a bit ugly. I'm not fussy on this so I'll go with your suggestion.

20113–20115

This is directly returning a bitcast SDNode back to the original VT now.

[dmgreen]

Thanks. LGTM

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20087

SDValue is usually passed by value, not as a pointer. It might also be able to generate the DL from SDLoc DL(Srl);, depending on where it is best for the debug loc to come from.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

static SDValue performUzpCombine(SDNode *N, SelectionDAG &DAG) {
   SDLoc DL(N);
   SDValue Op0 = N->getOperand(0);
   SDValue Op1 = N->getOperand(1);
   EVT ResVT = N->getValueType(0);
 
   // uzp1(x, undef) -> concat(truncate(x), undef)
   if (Op1.getOpcode() == ISD::UNDEF) {
     EVT BCVT = MVT::Other, HalfVT = MVT::Other;
     switch (ResVT.getSimpleVT().SimpleTy) {
     default:
       break;
     case MVT::v16i8:
       BCVT = MVT::v8i16;
       HalfVT = MVT::v8i8;
       break;
     case MVT::v8i16:
       BCVT = MVT::v4i32;
       HalfVT = MVT::v4i16;
       break;
     case MVT::v4i32:
       BCVT = MVT::v2i64;
       HalfVT = MVT::v2i32;
       break;
     }
     if (BCVT != MVT::Other) {
       SDValue BC = DAG.getBitcast(BCVT, Op0);
       SDValue Trunc = DAG.getNode(ISD::TRUNCATE, DL, HalfVT, BC);
       return DAG.getNode(ISD::CONCAT_VECTORS, DL, ResVT, Trunc,
                          DAG.getUNDEF(HalfVT));
     }
   }
 
   // uzp1(unpklo(uzp1(x, y)), z) => uzp1(x, z)
   if (Op0.getOpcode() == AArch64ISD::UUNPKLO) {
     if (Op0.getOperand(0).getOpcode() == AArch64ISD::UZP1) {
       SDValue X = Op0.getOperand(0).getOperand(0);
       return DAG.getNode(AArch64ISD::UZP1, DL, ResVT, X, Op1);
     }
   }
 
   // uzp1(x, unpkhi(uzp1(y, z))) => uzp1(x, z)
   if (Op1.getOpcode() == AArch64ISD::UUNPKHI) {
     if (Op1.getOperand(0).getOpcode() == AArch64ISD::UZP1) {
       SDValue Z = Op1.getOperand(0).getOperand(1);
       return DAG.getNode(AArch64ISD::UZP1, DL, ResVT, Op0, Z);
     }
   }
 
   // uzp1(xtn x, xtn y) -> xtn(uzp1 (x, y))
   // Only implemented on little-endian subtargets.
   bool IsLittleEndian = DAG.getDataLayout().isLittleEndian();
 
   // This optimization only works on little endian.
   if (!IsLittleEndian)
     return SDValue();
 
   if (ResVT != MVT::v2i32 && ResVT != MVT::v4i16 && ResVT != MVT::v8i8)
     return SDValue();
 
   auto getSourceOp = [](SDValue Operand) -> SDValue {
     const unsigned Opcode = Operand.getOpcode();
     if (Opcode == ISD::TRUNCATE)
static SDValue combineBoolVectorAndTruncateStore(SelectionDAG &DAG,
 
   EVT StoreVT =
       EVT::getIntegerVT(*DAG.getContext(), MemVT.getStoreSizeInBits());
   SDValue ExtendedBits = DAG.getZExtOrTrunc(VectorBits, DL, StoreVT);
   return DAG.getStore(Store->getChain(), DL, ExtendedBits, Store->getBasePtr(),
                       Store->getMemOperand());
 }
 
+static SDValue tryCombineStoredNarrowShift(StoreSDNode *N, SelectionDAG &DAG,
+                                           const AArch64Subtarget *Subtarget) {
+  if (!Subtarget->hasSVE2())
+    return SDValue();
+
+  EVT MemVT = N->getMemoryVT();
+  EVT MemVTEltTy = MemVT.getVectorElementType();
+  EVT VT = N->getValue().getValueType();
+  EVT VTEltTy = VT.getVectorElementType();
+
+  unsigned Opc;
+  if (MemVTEltTy == MVT::i8 && VTEltTy == MVT::i16)
+    Opc = AArch64::RSHRNB_ZZI_B;
+  else if (MemVTEltTy == MVT::i16 && VTEltTy == MVT::i32)
+    Opc = AArch64::RSHRNB_ZZI_H;
+  else
+    return SDValue();
+
+  SDValue Srl = N->getOperand(1);
+  if (Srl->getOpcode() == ISD::SRL) {
+    auto SrlOp1 =
+        dyn_cast<ConstantSDNode>(DAG.getSplatValue(Srl->getOperand(1)));
+    if (!SrlOp1)
+      return SDValue();
+    unsigned ShiftValue = SrlOp1->getZExtValue();
+
+    SDValue Add = Srl->getOperand(0);
+    if (Add->getOpcode() != ISD::ADD)
+      return SDValue();
+    auto AddOp1 =
+        dyn_cast<ConstantSDNode>(DAG.getSplatValue(Add->getOperand(1)));
+    if (!AddOp1)
+      return SDValue();
+    int64_t AddValue = AddOp1->getZExtValue();
+
+    if (AddValue != 1 << (ShiftValue - 1))
+      return SDValue();
+
+    SDLoc DL(N);
+    SDValue RSHRNB = SDValue(
+        DAG.getMachineNode(Opc, DL, VT,
+                           {Add->getOperand(0),
+                            DAG.getTargetConstant(ShiftValue, DL, MVT::i32)}),
+        0);
+    return DAG.getTruncStore(N->getChain(), N, RSHRNB, N->getBasePtr(),
+                             N->getMemoryVT(), N->getMemOperand());
+  }
+  return SDValue();
+}
+
 static SDValue performSTORECombine(SDNode *N,
                                    TargetLowering::DAGCombinerInfo &DCI,
                                    SelectionDAG &DAG,
                                    const AArch64Subtarget *Subtarget) {
   StoreSDNode *ST = cast<StoreSDNode>(N);
   SDValue Chain = ST->getChain();
   SDValue Value = ST->getValue();
   SDValue Ptr = ST->getBasePtr();
if (Subtarget->supportsAddressTopByteIgnored() &&
     return SDValue(N, 0);
 
   if (SDValue Store = foldTruncStoreOfExt(DAG, N))
     return Store;
 
   if (SDValue Store = combineBoolVectorAndTruncateStore(DAG, ST))
     return Store;
 
+  if (ST->isTruncatingStore())
+    if (SDValue Store = tryCombineStoredNarrowShift(ST, DAG, Subtarget))
+      return Store;
+
   return SDValue();
 }
 
 static SDValue performMSTORECombine(SDNode *N,
                                     TargetLowering::DAGCombinerInfo &DCI,
                                     SelectionDAG &DAG,
                                     const AArch64Subtarget *Subtarget) {
   MaskedStoreSDNode *MST = cast<MaskedStoreSDNode>(N);

llvm/test/CodeGen/AArch64/sve2-intrinsics-binary-narrowing-shr.ll

-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve2 < %s | FileCheck %s
-; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sme < %s | FileCheck %s
-
-;
-; SHRNB
-;
-
-define <vscale x 16 x i8> @shrnb_h(<vscale x 8 x i16> %a) {
-; CHECK-LABEL: shrnb_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    shrnb z0.b, z0.h, #8
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.shrnb.nxv8i16(<vscale x 8 x i16> %a,
-                                                                 i32 8)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @shrnb_s(<vscale x 4 x i32> %a) {
-; CHECK-LABEL: shrnb_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    shrnb z0.h, z0.s, #16
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.shrnb.nxv4i32(<vscale x 4 x i32> %a,
-                                                                 i32 16)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @shrnb_d(<vscale x 2 x i64> %a) {
-; CHECK-LABEL: shrnb_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    shrnb z0.s, z0.d, #32
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.shrnb.nxv2i64(<vscale x 2 x i64> %a,
-                                                                 i32 32)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; RSHRNB
-;
-
-define <vscale x 16 x i8> @rshrnb_h(<vscale x 8 x i16> %a) {
-; CHECK-LABEL: rshrnb_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    rshrnb z0.b, z0.h, #2
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.rshrnb.nxv8i16(<vscale x 8 x i16> %a,
-                                                                  i32 2)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @rshrnb_s(<vscale x 4 x i32> %a) {
-; CHECK-LABEL: rshrnb_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    rshrnb z0.h, z0.s, #2
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.rshrnb.nxv4i32(<vscale x 4 x i32> %a,
-                                                                  i32 2)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @rshrnb_d(<vscale x 2 x i64> %a) {
-; CHECK-LABEL: rshrnb_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    rshrnb z0.s, z0.d, #2
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.rshrnb.nxv2i64(<vscale x 2 x i64> %a,
-                                                                  i32 2)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; UQSHRNB
-;
-
-define <vscale x 16 x i8> @uqshrnb_h(<vscale x 8 x i16> %a) {
-; CHECK-LABEL: uqshrnb_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqshrnb z0.b, z0.h, #1
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.uqshrnb.nxv8i16(<vscale x 8 x i16> %a,
-                                                                   i32 1)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @uqshrnb_s(<vscale x 4 x i32> %a) {
-; CHECK-LABEL: uqshrnb_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqshrnb z0.h, z0.s, #1
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.uqshrnb.nxv4i32(<vscale x 4 x i32> %a,
-                                                                   i32 1)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @uqshrnb_d(<vscale x 2 x i64> %a) {
-; CHECK-LABEL: uqshrnb_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqshrnb z0.s, z0.d, #1
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.uqshrnb.nxv2i64(<vscale x 2 x i64> %a,
-                                                                   i32 1)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; SQSHRNB
-;
-
-define <vscale x 16 x i8> @sqshrnb_h(<vscale x 8 x i16> %a) {
-; CHECK-LABEL: sqshrnb_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrnb z0.b, z0.h, #1
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.sqshrnb.nxv8i16(<vscale x 8 x i16> %a,
-                                                                   i32 1)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @sqshrnb_s(<vscale x 4 x i32> %a) {
-; CHECK-LABEL: sqshrnb_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrnb z0.h, z0.s, #1
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.sqshrnb.nxv4i32(<vscale x 4 x i32> %a,
-                                                                   i32 1)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @sqshrnb_d(<vscale x 2 x i64> %a) {
-; CHECK-LABEL: sqshrnb_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrnb z0.s, z0.d, #1
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.sqshrnb.nxv2i64(<vscale x 2 x i64> %a,
-                                                                   i32 1)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; SQSHRUNB
-;
-
-define <vscale x 16 x i8> @sqshrunb_h(<vscale x 8 x i16> %a) {
-; CHECK-LABEL: sqshrunb_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrunb z0.b, z0.h, #7
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.sqshrunb.nxv8i16(<vscale x 8 x i16> %a,
-                                                                    i32 7)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @sqshrunb_s(<vscale x 4 x i32> %a) {
-; CHECK-LABEL: sqshrunb_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrunb z0.h, z0.s, #15
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.sqshrunb.nxv4i32(<vscale x 4 x i32> %a,
-                                                                    i32 15)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @sqshrunb_d(<vscale x 2 x i64> %a) {
-; CHECK-LABEL: sqshrunb_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrunb z0.s, z0.d, #31
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.sqshrunb.nxv2i64(<vscale x 2 x i64> %a,
-                                                                    i32 31)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; UQRSHRNB
-;
-
-define <vscale x 16 x i8> @uqrshrnb_h(<vscale x 8 x i16> %a) {
-; CHECK-LABEL: uqrshrnb_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqrshrnb z0.b, z0.h, #2
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.uqrshrnb.nxv8i16(<vscale x 8 x i16> %a,
-                                                                    i32 2)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @uqrshrnb_s(<vscale x 4 x i32> %a) {
-; CHECK-LABEL: uqrshrnb_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqrshrnb z0.h, z0.s, #2
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.uqrshrnb.nxv4i32(<vscale x 4 x i32> %a,
-                                                                    i32 2)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @uqrshrnb_d(<vscale x 2 x i64> %a) {
-; CHECK-LABEL: uqrshrnb_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqrshrnb z0.s, z0.d, #2
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.uqrshrnb.nxv2i64(<vscale x 2 x i64> %a,
-                                                                    i32 2)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; SQRSHRNB
-;
-
-define <vscale x 16 x i8> @sqrshrnb_h(<vscale x 8 x i16> %a) {
-; CHECK-LABEL: sqrshrnb_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrnb z0.b, z0.h, #2
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.sqrshrnb.nxv8i16(<vscale x 8 x i16> %a,
-                                                                    i32 2)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @sqrshrnb_s(<vscale x 4 x i32> %a) {
-; CHECK-LABEL: sqrshrnb_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrnb z0.h, z0.s, #2
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.sqrshrnb.nxv4i32(<vscale x 4 x i32> %a,
-                                                                    i32 2)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @sqrshrnb_d(<vscale x 2 x i64> %a) {
-; CHECK-LABEL: sqrshrnb_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrnb z0.s, z0.d, #2
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.sqrshrnb.nxv2i64(<vscale x 2 x i64> %a,
-                                                                    i32 2)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; SQRSHRUNB
-;
-
-define <vscale x 16 x i8> @sqrshrunb_h(<vscale x 8 x i16> %a) {
-; CHECK-LABEL: sqrshrunb_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrunb z0.b, z0.h, #6
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.sqrshrunb.nxv8i16(<vscale x 8 x i16> %a,
-                                                                     i32 6)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @sqrshrunb_s(<vscale x 4 x i32> %a) {
-; CHECK-LABEL: sqrshrunb_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrunb z0.h, z0.s, #14
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.sqrshrunb.nxv4i32(<vscale x 4 x i32> %a,
-                                                                     i32 14)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @sqrshrunb_d(<vscale x 2 x i64> %a) {
-; CHECK-LABEL: sqrshrunb_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrunb z0.s, z0.d, #30
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.sqrshrunb.nxv2i64(<vscale x 2 x i64> %a,
-                                                                     i32 30)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; SHRNT
-;
-
-define <vscale x 16 x i8> @shrnt_h(<vscale x 16 x i8> %a, <vscale x 8 x i16> %b) {
-; CHECK-LABEL: shrnt_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    shrnt z0.b, z1.h, #3
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.shrnt.nxv8i16(<vscale x 16 x i8> %a,
-                                                                 <vscale x 8 x i16> %b,
-                                                                 i32 3)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @shrnt_s(<vscale x 8 x i16> %a, <vscale x 4 x i32> %b) {
-; CHECK-LABEL: shrnt_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    shrnt z0.h, z1.s, #3
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.shrnt.nxv4i32(<vscale x 8 x i16> %a,
-                                                                 <vscale x 4 x i32> %b,
-                                                                 i32 3)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @shrnt_d(<vscale x 4 x i32> %a, <vscale x 2 x i64> %b) {
-; CHECK-LABEL: shrnt_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    shrnt z0.s, z1.d, #3
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.shrnt.nxv2i64(<vscale x 4 x i32> %a,
-                                                                 <vscale x 2 x i64> %b,
-                                                                 i32 3)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; RSHRNT
-;
-
-define <vscale x 16 x i8> @rshrnt_h(<vscale x 16 x i8> %a, <vscale x 8 x i16> %b) {
-; CHECK-LABEL: rshrnt_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    rshrnt z0.b, z1.h, #1
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.rshrnt.nxv8i16(<vscale x 16 x i8> %a,
-                                                                  <vscale x 8 x i16> %b,
-                                                                  i32 1)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @rshrnt_s(<vscale x 8 x i16> %a, <vscale x 4 x i32> %b) {
-; CHECK-LABEL: rshrnt_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    rshrnt z0.h, z1.s, #5
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.rshrnt.nxv4i32(<vscale x 8 x i16> %a,
-                                                                  <vscale x 4 x i32> %b,
-                                                                  i32 5)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @rshrnt_d(<vscale x 4 x i32> %a, <vscale x 2 x i64> %b) {
-; CHECK-LABEL: rshrnt_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    rshrnt z0.s, z1.d, #5
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.rshrnt.nxv2i64(<vscale x 4 x i32> %a,
-                                                                  <vscale x 2 x i64> %b,
-                                                                  i32 5)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; UQSHRNT
-;
-
-define <vscale x 16 x i8> @uqshrnt_h(<vscale x 16 x i8> %a, <vscale x 8 x i16> %b) {
-; CHECK-LABEL: uqshrnt_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqshrnt z0.b, z1.h, #5
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.uqshrnt.nxv8i16(<vscale x 16 x i8> %a,
-                                                                   <vscale x 8 x i16> %b,
-                                                                   i32 5)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @uqshrnt_s(<vscale x 8 x i16> %a, <vscale x 4 x i32> %b) {
-; CHECK-LABEL: uqshrnt_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqshrnt z0.h, z1.s, #13
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.uqshrnt.nxv4i32(<vscale x 8 x i16> %a,
-                                                                   <vscale x 4 x i32> %b,
-                                                                   i32 13)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @uqshrnt_d(<vscale x 4 x i32> %a, <vscale x 2 x i64> %b) {
-; CHECK-LABEL: uqshrnt_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqshrnt z0.s, z1.d, #29
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.uqshrnt.nxv2i64(<vscale x 4 x i32> %a,
-                                                                   <vscale x 2 x i64> %b,
-                                                                   i32 29)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; SQSHRNT
-;
-
-define <vscale x 16 x i8> @sqshrnt_h(<vscale x 16 x i8> %a, <vscale x 8 x i16> %b) {
-; CHECK-LABEL: sqshrnt_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrnt z0.b, z1.h, #5
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.sqshrnt.nxv8i16(<vscale x 16 x i8> %a,
-                                                                   <vscale x 8 x i16> %b,
-                                                                   i32 5)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @sqshrnt_s(<vscale x 8 x i16> %a, <vscale x 4 x i32> %b) {
-; CHECK-LABEL: sqshrnt_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrnt z0.h, z1.s, #13
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.sqshrnt.nxv4i32(<vscale x 8 x i16> %a,
-                                                                   <vscale x 4 x i32> %b,
-                                                                   i32 13)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @sqshrnt_d(<vscale x 4 x i32> %a, <vscale x 2 x i64> %b) {
-; CHECK-LABEL: sqshrnt_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrnt z0.s, z1.d, #29
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.sqshrnt.nxv2i64(<vscale x 4 x i32> %a,
-                                                                   <vscale x 2 x i64> %b,
-                                                                   i32 29)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; SQSHRUNT
-;
-
-define <vscale x 16 x i8> @sqshrunt_h(<vscale x 16 x i8> %a, <vscale x 8 x i16> %b) {
-; CHECK-LABEL: sqshrunt_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrunt z0.b, z1.h, #4
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.sqshrunt.nxv8i16(<vscale x 16 x i8> %a,
-                                                                    <vscale x 8 x i16> %b,
-                                                                    i32 4)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @sqshrunt_s(<vscale x 8 x i16> %a, <vscale x 4 x i32> %b) {
-; CHECK-LABEL: sqshrunt_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrunt z0.h, z1.s, #4
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.sqshrunt.nxv4i32(<vscale x 8 x i16> %a,
-                                                                    <vscale x 4 x i32> %b,
-                                                                    i32 4)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @sqshrunt_d(<vscale x 4 x i32> %a, <vscale x 2 x i64> %b) {
-; CHECK-LABEL: sqshrunt_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqshrunt z0.s, z1.d, #4
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.sqshrunt.nxv2i64(<vscale x 4 x i32> %a,
-                                                                    <vscale x 2 x i64> %b,
-                                                                    i32 4)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; UQRSHRNT
-;
-
-define <vscale x 16 x i8> @uqrshrnt_h(<vscale x 16 x i8> %a, <vscale x 8 x i16> %b) {
-; CHECK-LABEL: uqrshrnt_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqrshrnt z0.b, z1.h, #8
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.uqrshrnt.nxv8i16(<vscale x 16 x i8> %a,
-                                                                    <vscale x 8 x i16> %b,
-                                                                    i32 8)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @uqrshrnt_s(<vscale x 8 x i16> %a, <vscale x 4 x i32> %b) {
-; CHECK-LABEL: uqrshrnt_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqrshrnt z0.h, z1.s, #12
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.uqrshrnt.nxv4i32(<vscale x 8 x i16> %a,
-                                                                    <vscale x 4 x i32> %b,
-                                                                    i32 12)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @uqrshrnt_d(<vscale x 4 x i32> %a, <vscale x 2 x i64> %b) {
-; CHECK-LABEL: uqrshrnt_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    uqrshrnt z0.s, z1.d, #28
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.uqrshrnt.nxv2i64(<vscale x 4 x i32> %a,
-                                                                    <vscale x 2 x i64> %b,
-                                                                    i32 28)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; SQRSHRNT
-;
-
-define <vscale x 16 x i8> @sqrshrnt_h(<vscale x 16 x i8> %a, <vscale x 8 x i16> %b) {
-; CHECK-LABEL: sqrshrnt_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrnt z0.b, z1.h, #8
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.sqrshrnt.nxv8i16(<vscale x 16 x i8> %a,
-                                                                    <vscale x 8 x i16> %b,
-                                                                    i32 8)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @sqrshrnt_s(<vscale x 8 x i16> %a, <vscale x 4 x i32> %b) {
-; CHECK-LABEL: sqrshrnt_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrnt z0.h, z1.s, #12
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.sqrshrnt.nxv4i32(<vscale x 8 x i16> %a,
-                                                                    <vscale x 4 x i32> %b,
-                                                                    i32 12)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @sqrshrnt_d(<vscale x 4 x i32> %a, <vscale x 2 x i64> %b) {
-; CHECK-LABEL: sqrshrnt_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrnt z0.s, z1.d, #28
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.sqrshrnt.nxv2i64(<vscale x 4 x i32> %a,
-                                                                    <vscale x 2 x i64> %b,
-                                                                    i32 28)
-  ret <vscale x 4 x i32> %out
-}
-
-;
-; SQRSHRUNT
-;
-
-define <vscale x 16 x i8> @sqrshrunt_h(<vscale x 16 x i8> %a, <vscale x 8 x i16> %b) {
-; CHECK-LABEL: sqrshrunt_h:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrunt z0.b, z1.h, #1
-; CHECK-NEXT:    ret
-  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.sqrshrunt.nxv8i16(<vscale x 16 x i8> %a,
-                                                                     <vscale x 8 x i16> %b,
-                                                                     i32 1)
-  ret <vscale x 16 x i8> %out
-}
-
-define <vscale x 8 x i16> @sqrshrunt_s(<vscale x 8 x i16> %a, <vscale x 4 x i32> %b) {
-; CHECK-LABEL: sqrshrunt_s:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrunt z0.h, z1.s, #5
-; CHECK-NEXT:    ret
-  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.sqrshrunt.nxv4i32(<vscale x 8 x i16> %a,
-                                                                     <vscale x 4 x i32> %b,
-                                                                     i32 5)
-  ret <vscale x 8 x i16> %out
-}
-
-define <vscale x 4 x i32> @sqrshrunt_d(<vscale x 4 x i32> %a, <vscale x 2 x i64> %b) {
-; CHECK-LABEL: sqrshrunt_d:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sqrshrunt z0.s, z1.d, #5
-; CHECK-NEXT:    ret
-  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.sqrshrunt.nxv2i64(<vscale x 4 x i32> %a,
-                                                                     <vscale x 2 x i64> %b,
-                                                                     i32 5)
-  ret <vscale x 4 x i32> %out
-}
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.shrnb.nxv8i16(<vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.shrnb.nxv4i32(<vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.shrnb.nxv2i64(<vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.rshrnb.nxv8i16(<vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.rshrnb.nxv4i32(<vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.rshrnb.nxv2i64(<vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.uqshrnb.nxv8i16(<vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.uqshrnb.nxv4i32(<vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.uqshrnb.nxv2i64(<vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.sqshrnb.nxv8i16(<vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.sqshrnb.nxv4i32(<vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.sqshrnb.nxv2i64(<vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.uqrshrnb.nxv8i16(<vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.uqrshrnb.nxv4i32(<vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.uqrshrnb.nxv2i64(<vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.sqrshrnb.nxv8i16(<vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.sqrshrnb.nxv4i32(<vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.sqrshrnb.nxv2i64(<vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.sqshrunb.nxv8i16(<vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.sqshrunb.nxv4i32(<vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.sqshrunb.nxv2i64(<vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.sqrshrunb.nxv8i16(<vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.sqrshrunb.nxv4i32(<vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.sqrshrunb.nxv2i64(<vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.shrnt.nxv8i16(<vscale x 16 x i8>, <vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.shrnt.nxv4i32(<vscale x 8 x i16>, <vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.shrnt.nxv2i64(<vscale x 4 x i32>, <vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.rshrnt.nxv8i16(<vscale x 16 x i8>, <vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.rshrnt.nxv4i32(<vscale x 8 x i16>, <vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.rshrnt.nxv2i64(<vscale x 4 x i32>, <vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.uqshrnt.nxv8i16(<vscale x 16 x i8>, <vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.uqshrnt.nxv4i32(<vscale x 8 x i16>, <vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.uqshrnt.nxv2i64(<vscale x 4 x i32>, <vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.sqshrnt.nxv8i16(<vscale x 16 x i8>, <vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.sqshrnt.nxv4i32(<vscale x 8 x i16>, <vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.sqshrnt.nxv2i64(<vscale x 4 x i32>, <vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.sqshrunt.nxv8i16(<vscale x 16 x i8>, <vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.sqshrunt.nxv4i32(<vscale x 8 x i16>, <vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.sqshrunt.nxv2i64(<vscale x 4 x i32>, <vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.uqrshrnt.nxv8i16(<vscale x 16 x i8>, <vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.uqrshrnt.nxv4i32(<vscale x 8 x i16>, <vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.uqrshrnt.nxv2i64(<vscale x 4 x i32>, <vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.sqrshrnt.nxv8i16(<vscale x 16 x i8>, <vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.sqrshrnt.nxv4i32(<vscale x 8 x i16>, <vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.sqrshrnt.nxv2i64(<vscale x 4 x i32>, <vscale x 2 x i64>, i32)
-
-declare <vscale x 16 x i8> @llvm.aarch64.sve.sqrshrunt.nxv8i16(<vscale x 16 x i8>, <vscale x 8 x i16>, i32)
-declare <vscale x 8 x i16> @llvm.aarch64.sve.sqrshrunt.nxv4i32(<vscale x 8 x i16>, <vscale x 4 x i32>, i32)
-declare <vscale x 4 x i32> @llvm.aarch64.sve.sqrshrunt.nxv2i64(<vscale x 4 x i32>, <vscale x 2 x i64>, i32)

llvm/test/CodeGen/AArch64/sve2-intrinsics-combine-rshrnb.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve2 < %s | FileCheck %s
+
+define void @add_lshr_rshrnb_b_6(ptr %ptr, ptr %dst, i64 %index12){
+; CHECK-LABEL: add_lshr_rshrnb_b_6:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    rshrnb z0.b, z0.h, #6
+; CHECK-NEXT:    st1b { z0.h }, p0, [x1, x2]
+; CHECK-NEXT:    ret
+  %wide.load13 = load <vscale x 8 x i16>, ptr %ptr, align 2
+  %1 = add <vscale x 8 x i16> %wide.load13, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 32, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %2 = lshr <vscale x 8 x i16> %1, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 6, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %3 = trunc <vscale x 8 x i16> %2 to <vscale x 8 x i8>
+  %4 = getelementptr inbounds i8, ptr %dst, i64 %index12
+  store <vscale x 8 x i8> %3, ptr %4, align 1
+  ret void
+}
+
+define void @neg_add_lshr_rshrnb_b_6(ptr %ptr, ptr %dst, i64 %index12){
+; CHECK-LABEL: neg_add_lshr_rshrnb_b_6:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    add z0.h, z0.h, #1 // =0x1
+; CHECK-NEXT:    lsr z0.h, z0.h, #6
+; CHECK-NEXT:    st1b { z0.h }, p0, [x1, x2]
+; CHECK-NEXT:    ret
+  %wide.load13 = load <vscale x 8 x i16>, ptr %ptr, align 2
+  %1 = add <vscale x 8 x i16> %wide.load13, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %2 = lshr <vscale x 8 x i16> %1, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 6, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %3 = trunc <vscale x 8 x i16> %2 to <vscale x 8 x i8>
+  %4 = getelementptr inbounds i8, ptr %dst, i64 %index12
+  store <vscale x 8 x i8> %3, ptr %4, align 1
+  ret void
+}
+
+define void @add_lshr_rshrnb_h_7(ptr %ptr, ptr %dst, i64 %index12){
+; CHECK-LABEL: add_lshr_rshrnb_h_7:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    rshrnb z0.b, z0.h, #7
+; CHECK-NEXT:    st1b { z0.h }, p0, [x1, x2]
+; CHECK-NEXT:    ret
+  %wide.load13 = load <vscale x 8 x i16>, ptr %ptr, align 2
+  %1 = add <vscale x 8 x i16> %wide.load13, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 64, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %2 = lshr <vscale x 8 x i16> %1, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 7, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %3 = trunc <vscale x 8 x i16> %2 to <vscale x 8 x i8>
+  %4 = getelementptr inbounds i8, ptr %dst, i64 %index12
+  store <vscale x 8 x i8> %3, ptr %4, align 1
+  ret void
+}
+
+define void @add_lshr_rshrn_h_6(ptr %ptr, ptr %dst, i64 %index12){
+; CHECK-LABEL: add_lshr_rshrn_h_6:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    rshrnb z0.h, z0.s, #6
+; CHECK-NEXT:    st1h { z0.s }, p0, [x1, x2, lsl #1]
+; CHECK-NEXT:    ret
+  %wide.load13 = load <vscale x 4 x i32>, ptr %ptr, align 2
+  %1 = add <vscale x 4 x i32> %wide.load13, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 32, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %2 = lshr <vscale x 4 x i32> %1, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 6, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %3 = trunc <vscale x 4 x i32> %2 to <vscale x 4 x i16>
+  %4 = getelementptr inbounds i16, ptr %dst, i64 %index12
+  store <vscale x 4 x i16> %3, ptr %4, align 1
+  ret void
+}
+
+define void @add_lshr_rshrnb_h_2(ptr %ptr, ptr %dst, i64 %index12){
+; CHECK-LABEL: add_lshr_rshrnb_h_2:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    rshrnb z0.h, z0.s, #2
+; CHECK-NEXT:    st1h { z0.s }, p0, [x1, x2, lsl #1]
+; CHECK-NEXT:    ret
+  %wide.load13 = load <vscale x 4 x i32>, ptr %ptr, align 2
+  %1 = add <vscale x 4 x i32> %wide.load13, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 2, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %2 = lshr <vscale x 4 x i32> %1, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 2, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %3 = trunc <vscale x 4 x i32> %2 to <vscale x 4 x i16>
+  %4 = getelementptr inbounds i16, ptr %dst, i64 %index12
+  store <vscale x 4 x i16> %3, ptr %4, align 1
+  ret void
+}
+
+define void @neg_add_lshr_rshrnb_h_0(ptr %ptr, ptr %dst, i64 %index12){
+; CHECK-LABEL: neg_add_lshr_rshrnb_h_0:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ret
+  %wide.load13 = load <vscale x 4 x i32>, ptr %ptr, align 2
+  %1 = add <vscale x 4 x i32> %wide.load13, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %2 = lshr <vscale x 4 x i32> %1, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 -1, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %3 = trunc <vscale x 4 x i32> %2 to <vscale x 4 x i16>
+  %4 = getelementptr inbounds i16, ptr %dst, i64 %index12
+  store <vscale x 4 x i16> %3, ptr %4, align 1
+  ret void
+}