[AArch64][SVE2] Combine add+lsr to rshrnb for stores

[MattDevereau]

Summary

[AArch64][SVE2] Combine add+lsr to rshrnb for stores

The example sequence

add z0.h, z0.h, #32
lsr z0.h, #6
st1b z0.h, x1

can be replaced with

rshrnb z0.b, #6
st1b z0.h, x1

As the top half of the destination elements are truncated.

In similar fashion,

add z0.s, z0.s, #32
lsr z1.s, z1.s, #6
add z1.s, z1.s, #32
lsr z0.s, z0.s, #6
uzp1 z0.h, z0.h, z1.h

Can be replaced with

rshrnb z1.h, z1.s, #6
rshrnb z0.h, z0.s, #6
uzp1 z0.h, z0.h, z1.h

[david-arm]

This is a nice optimisation @MattDevereau, thanks! I found there is also another case we could support with loops like this where the store doesn't come straight afterwards:

void foo(unsigned short *dest, unsigned short *src, long n) {
  for (long i = 0; i < n; i++)
    dest[i] += ((src[i] + 32) >> 6);
}

In this case the IR sequence is add, lshr, trunc since the truncate doesn't get absorbed into the store. Maybe it's worth seeing if you can reuse your code in tryCombineStoredNarrowShift for this case too?

[kmclaughlin]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20107

Hi @MattDevereau,
You might need to check the value returned by DAG.getSplatValue here as I think if the operand is not a splat as expected, the dyn_cast will fail.
Please can you add a negative test for this scenario as well?

[MattDevereau]

@kmclaughlin Thank you. I've added two tests @neg_trunc_lsr_add_op1_not_splat and @neg_trunc_lsr_op1_not_splat to bail out of emitting rshrnb when the RHS operands are not splat values.

I've also added a test @neg_add_has_two_uses and a check that the add does not have more than one use. Without this it is possible to generate this regression which costs 2 extra cycles

neg_add_two_use:
  ptrue p0.h
  ld1h { z0.h }, p0/z, [x0]
  rshrnb z1.b, z0.h, #6
  add z0.h, z0.h, #32 // =0x20
  add z0.h, z0.h, z0.h
  st1h { z0.h }, p0, [x2, x3, lsl #1]
  st1b { z1.h }, p0, [x1, x3]
  ret

[kmclaughlin]

Thank you for adding the new tests @MattDevereau, I just have a couple of small suggestions in the trySimplifySrlAddToRshrnb function.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20100

I think if you change this to dyn_cast_or_null you can remove the additional isSplatValue check at the beginning of the function.

20109

Similarly here, I think you can remove the isSplatValue above by using dyn_cast_or_null

[MattDevereau]

Replaced explicit checks for splat values with dyn_cast_or_null

[kmclaughlin]

Thanks @MattDevereau, there's one nit but otherwise LGTM!

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20169–20175

nit: The braces here aren't necessary

20182–20188

As above :)

[dmgreen]

Hello. When adding these for NEON we were able to the transform via tablegen patterns at selection time, which has some benefits about leaving the nodes generic for as long as possible. In this case because of the differences in the instructions I would expect that it need demanded bits, so doing it as a combine probably makes sense.

It's best not to generate a MachineNode directly though if we can.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20119

We should avoid creating machine nodes in SelectionDAG combines. It's like a layering violation. Can you change this to either generate an intrinsic, or preferably add a new AArch64ISD node for it?

[hassnaa-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20114

Hi Mat,
I think that check doesn't consider the case when the AddValue has enabled bits other than the most significant bit.
ex: the ShiftValue is 6, and the AddValue is 33 (100001)b
Is that correct ?

20114

Do you handle the case when the destination of the AddOperation has an enabled bit in the same index of the enabled bit in the add Value ?
In that case the Add operation will has side affect, and by combining that case into rshrnb, you cancel the side effect of the AddOperation.
Do you agree on that ? or I'm missing something ??

[MattDevereau]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20114

I think that check doesn't consider the case when the AddValue has enabled bits other than the most significant bit.
ex: the ShiftValue is 6, and the AddValue is 33 (100001)b

This check would correctly reject that example, which is not a case for this combine. As per the documentation for rshrnb the operation is

integer res = (UInt(element) + (1 << (shift-1))) >> shift;

As 33 is not possible to get from left shifting 1, the combine will correctly bail. The lower bits in this case would be kept as the combine wouldn't fire.

20114

What side effect do you mean exactly?

Do you handle the case when the destination of the AddOperation has an enabled bit in the same index of the enabled bit in the add Value ?

I don't think the bits in the destination register of the add matter, register allocation should handle that fine and give us a non conflicting destination register.

20119

Thanks for pointing that out. I'm working on emitting a new AArch64ISD node now, however it adds a bit more complexity to the patch.

[hassnaa-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20114

Sorry I didn't express it clearly.
For example,
The add operation is src + (100000)b
I mean that what if the src has an enabled bit at the index corresponding to the '1' in the addedVal, index 5
so that means there will be 1+1 and then index 6 will be affected.
How that case is handled ?

[MattDevereau]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20114

Sorry but I don't understand what you mean by case. I think what you are referring to is the rounding behaviour of rshrnb which is the intended calculation and not a side effect.

[MattDevereau]

Emit an AArch64ISD node instead of the machine node directly.
Include D->S addressing mode of RSHRNB

[dmgreen]

Thanks. This looks good, but it might need to be using 1ULL for the shift value and there are few extra suggestions below. Otherwise LGTM

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20093

Can you either add a check that the VT is one that is valid for RSHRNB, or an assert that we only handle those types?

20113

This may need to be 1ULL << (ShiftValue - 1), as the number could be a 64bit value.

20117–20119

SDValue Rshrnb = DAG.getNode(
      AArch64ISD::RSHRNB_I, DL, VT,
      Add->getOperand(0), DAG.getTargetConstant(ShiftValue, DL, MVT::i32));

(Or just return it directly)

20168

trySimplifySrlAddToRshrnb could take a SDValue, instead of needing the cast.

20792

I think RshrnbVT is just ValueVT?

llvm/lib/Target/AArch64/AArch64ISelLowering.h

217

This might deserve to be in it's own little section. There may be a number of T/B nodes we end up adding, if we treat them in the same way.

llvm/lib/Target/AArch64/SVEInstrFormats.td

4309 ↗

(On Diff #546405)

It might be better to generate a nxv16i8->nxv8i16 RSHRN with a bitcast/nvcast back to nxv8i16.

[MattDevereau]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20093

Since were matching the Rshrnb ISD node emitted with the patterns in tablegen that match nvx8i16 -> nxv16i8, nxv4i32 -> nxv8i16, nxv2i64 -> nxv4i32 I've added an explicit block for this now:

EVT ResVT;
if (VT == MVT::nxv8i16)
  ResVT = MVT::nxv16i8;
else if (VT == MVT::nxv4i32)
  ResVT = MVT::nxv8i16;
else if (VT == MVT::nxv2i64)
  ResVT = MVT::nxv4i32;
else
  return SDValue();

We can use ResVT for the SDNode creation of Rshrnb and use VT for the bitcast back to the type that fits the DAG correctly.

20113

Done, and changed int64_t AddValue to uint64_t AddValue to line up with the change.

20117–20119

This is directly returning a bitcast SDNode back to the original VT now.

20168

I get the feeling this is a side-grade since I need to pass in SDLoc as an extra parameter this way, but I can see the cast looks a bit ugly. I'm not fussy on this so I'll go with your suggestion.

[dmgreen]

Thanks. LGTM

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

20091

SDValue is usually passed by value, not as a pointer. It might also be able to generate the DL from SDLoc DL(Srl);, depending on where it is best for the debug loc to come from.

llvm/lib/Target/AArch64/AArch64ISelLowering.h

enum NodeType : unsigned {
   VASHR,
 
   // Vector shift by scalar (again)
   SQSHL_I,
   UQSHL_I,
   SQSHLU_I,
   SRSHR_I,
   URSHR_I,
 
+  // Vector narrowing shift by immediate (bottom)
+  RSHRNB_I,
+
   // Vector shift by constant and insert
   VSLI,
   VSRI,
 
   // Vector comparisons
   CMEQ,
   CMGE,
   CMGT,

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

case AArch64ISD::FIRST_NUMBER:
     MAKE_CASE(AArch64ISD::ASSERT_ZEXT_BOOL)
     MAKE_CASE(AArch64ISD::MOPS_MEMSET)
     MAKE_CASE(AArch64ISD::MOPS_MEMSET_TAGGING)
     MAKE_CASE(AArch64ISD::MOPS_MEMCOPY)
     MAKE_CASE(AArch64ISD::MOPS_MEMMOVE)
     MAKE_CASE(AArch64ISD::CALL_BTI)
     MAKE_CASE(AArch64ISD::MRRS)
     MAKE_CASE(AArch64ISD::MSRR)
+    MAKE_CASE(AArch64ISD::RSHRNB_I)
   }
 #undef MAKE_CASE
   return nullptr;
 }
 
 MachineBasicBlock *
 AArch64TargetLowering::EmitF128CSEL(MachineInstr &MI,
                                     MachineBasicBlock *MBB) const {
if (MLD->isUnindexed() && MLD->getExtensionType() != ISD::SEXTLOAD &&
         return NewLoad;
       }
     }
   }
 
   return SDValue();
 }
 
-static SDValue performUzpCombine(SDNode *N, SelectionDAG &DAG) {
+// Try to simplify:
+//    t1 = nxv8i16 add(X, 1 << (ShiftValue - 1))
+//    t2 = nxv8i16 srl(t1, ShiftValue)
+// to
+//    t1 = nxv8i16 rshrnb(X, shiftvalue).
+// rshrnb will zero the top half bits of each element. Therefore, this combine
+// should only be performed when a following instruction with the rshrnb
+// as an operand does not care about the top half of each element. For example,
+// a uzp1 or a truncating store.
+static SDValue trySimplifySrlAddToRshrnb(SDValue Srl, SelectionDAG &DAG,
+                                         const AArch64Subtarget *Subtarget) {
+  EVT VT = Srl->getValueType(0);
+
+  if (!VT.isScalableVector() || !Subtarget->hasSVE2() ||
+      Srl->getOpcode() != ISD::SRL)
+    return SDValue();
+
+  EVT ResVT;
+  if (VT == MVT::nxv8i16)
+    ResVT = MVT::nxv16i8;
+  else if (VT == MVT::nxv4i32)
+    ResVT = MVT::nxv8i16;
+  else if (VT == MVT::nxv2i64)
+    ResVT = MVT::nxv4i32;
+  else
+    return SDValue();
+
+  auto SrlOp1 =
+      dyn_cast_or_null<ConstantSDNode>(DAG.getSplatValue(Srl->getOperand(1)));
+  if (!SrlOp1)
+    return SDValue();
+  unsigned ShiftValue = SrlOp1->getZExtValue();
+
+  SDValue Add = Srl->getOperand(0);
+  if (Add->getOpcode() != ISD::ADD || !Add->hasOneUse())
+    return SDValue();
+  auto AddOp1 =
+      dyn_cast_or_null<ConstantSDNode>(DAG.getSplatValue(Add->getOperand(1)));
+  if (!AddOp1)
+    return SDValue();
+  uint64_t AddValue = AddOp1->getZExtValue();
+  if (AddValue != 1ULL << (ShiftValue - 1))
+    return SDValue();
+
+  SDLoc DL(Srl);
+  SDValue Rshrnb = DAG.getNode(
+      AArch64ISD::RSHRNB_I, DL, ResVT,
+      {Add->getOperand(0), DAG.getTargetConstant(ShiftValue, DL, MVT::i32)});
+  return DAG.getNode(ISD::BITCAST, DL, VT, Rshrnb);
+}
+
+static SDValue performUzpCombine(SDNode *N, SelectionDAG &DAG,
+                                 const AArch64Subtarget *Subtarget) {
   SDLoc DL(N);
   SDValue Op0 = N->getOperand(0);
   SDValue Op1 = N->getOperand(1);
   EVT ResVT = N->getValueType(0);
 
   // uzp1(x, undef) -> concat(truncate(x), undef)
   if (Op1.getOpcode() == ISD::UNDEF) {
     EVT BCVT = MVT::Other, HalfVT = MVT::Other;
if (Op1.getOpcode() == ISD::UNDEF) {
     if (BCVT != MVT::Other) {
       SDValue BC = DAG.getBitcast(BCVT, Op0);
       SDValue Trunc = DAG.getNode(ISD::TRUNCATE, DL, HalfVT, BC);
       return DAG.getNode(ISD::CONCAT_VECTORS, DL, ResVT, Trunc,
                          DAG.getUNDEF(HalfVT));
     }
   }
 
+  if (SDValue Rshrnb = trySimplifySrlAddToRshrnb(Op0, DAG, Subtarget))
+    return DAG.getNode(AArch64ISD::UZP1, DL, ResVT, Rshrnb, Op1);
+
+  if (SDValue Rshrnb = trySimplifySrlAddToRshrnb(Op1, DAG, Subtarget))
+    return DAG.getNode(AArch64ISD::UZP1, DL, ResVT, Op0, Rshrnb);
+
   // uzp1(unpklo(uzp1(x, y)), z) => uzp1(x, z)
   if (Op0.getOpcode() == AArch64ISD::UUNPKLO) {
     if (Op0.getOperand(0).getOpcode() == AArch64ISD::UZP1) {
       SDValue X = Op0.getOperand(0).getOperand(0);
       return DAG.getNode(AArch64ISD::UZP1, DL, ResVT, X, Op1);
     }
   }
 
   // uzp1(x, unpkhi(uzp1(y, z))) => uzp1(x, z)
   if (Op1.getOpcode() == AArch64ISD::UUNPKHI) {
     if (Op1.getOperand(0).getOpcode() == AArch64ISD::UZP1) {
       SDValue Z = Op1.getOperand(0).getOperand(1);
       return DAG.getNode(AArch64ISD::UZP1, DL, ResVT, Op0, Z);
     }
   }
 
   // uzp1(xtn x, xtn y) -> xtn(uzp1 (x, y))
   // Only implemented on little-endian subtargets.
   bool IsLittleEndian = DAG.getDataLayout().isLittleEndian();
 
   // This optimization only works on little endian.
   if (!IsLittleEndian)
     return SDValue();
 
if (Subtarget->supportsAddressTopByteIgnored() &&
     return SDValue(N, 0);
 
   if (SDValue Store = foldTruncStoreOfExt(DAG, N))
     return Store;
 
   if (SDValue Store = combineBoolVectorAndTruncateStore(DAG, ST))
     return Store;
 
+  if (ST->isTruncatingStore())
+    if (SDValue Rshrnb =
+            trySimplifySrlAddToRshrnb(ST->getOperand(1), DAG, Subtarget)) {
+      EVT StoreVT = ST->getMemoryVT();
+      if ((ValueVT == MVT::nxv8i16 && StoreVT == MVT::nxv8i8) ||
+          (ValueVT == MVT::nxv4i32 && StoreVT == MVT::nxv4i16) ||
+          (ValueVT == MVT::nxv2i64 && StoreVT == MVT::nxv2i32))
+        return DAG.getTruncStore(ST->getChain(), ST, Rshrnb, ST->getBasePtr(),
+                                 StoreVT, ST->getMemOperand());
+    }
+
   return SDValue();
 }
 
 static SDValue performMSTORECombine(SDNode *N,
                                     TargetLowering::DAGCombinerInfo &DCI,
                                     SelectionDAG &DAG,
                                     const AArch64Subtarget *Subtarget) {
   MaskedStoreSDNode *MST = cast<MaskedStoreSDNode>(N);
SDValue AArch64TargetLowering::PerformDAGCombine(SDNode *N,
   case AArch64ISD::NVCAST:
     return performNVCASTCombine(N);
   case AArch64ISD::SPLICE:
     return performSpliceCombine(N, DAG);
   case AArch64ISD::UUNPKLO:
   case AArch64ISD::UUNPKHI:
     return performUnpackCombine(N, DAG, Subtarget);
   case AArch64ISD::UZP1:
-    return performUzpCombine(N, DAG);
+    return performUzpCombine(N, DAG, Subtarget);
   case AArch64ISD::SETCC_MERGE_ZERO:
     return performSetccMergeZeroCombine(N, DCI);
   case AArch64ISD::REINTERPRET_CAST:
     return performReinterpretCastCombine(N);
   case AArch64ISD::GLD1_MERGE_ZERO:
   case AArch64ISD::GLD1_SCALED_MERGE_ZERO:
   case AArch64ISD::GLD1_UXTW_MERGE_ZERO:
   case AArch64ISD::GLD1_SXTW_MERGE_ZERO:

llvm/lib/Target/AArch64/AArch64InstrInfo.td

 def AArch64stilp : SDNode<"AArch64ISD::STILP", SDT_AArch64stilp, [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
 def AArch64stnp : SDNode<"AArch64ISD::STNP", SDT_AArch64stnp, [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
 
 def AArch64tbl : SDNode<"AArch64ISD::TBL", SDT_AArch64TBL>;
 def AArch64mrs : SDNode<"AArch64ISD::MRS",
                         SDTypeProfile<1, 1, [SDTCisVT<0, i64>, SDTCisVT<1, i32>]>,
                         [SDNPHasChain, SDNPOutGlue]>;
 
+def SD_AArch64rshrnb : SDTypeProfile<1, 2, [SDTCisVec<0>, SDTCisVec<1>, SDTCisInt<2>]>;
+def AArch64rshrnb : SDNode<"AArch64ISD::RSHRNB_I", SD_AArch64rshrnb>;
+def AArch64rshrnb_pf : PatFrags<(ops node:$rs, node:$i),
+                            [(AArch64rshrnb node:$rs, node:$i),
+                            (int_aarch64_sve_rshrnb node:$rs, node:$i)]>;
+
 // Match add node and also treat an 'or' node is as an 'add' if the or'ed operands
 // have no common bits.
 def add_and_or_is_add : PatFrags<(ops node:$lhs, node:$rhs),
                          [(add node:$lhs, node:$rhs), (or node:$lhs, node:$rhs)],[{
    if (N->getOpcode() == ISD::ADD)
      return true;
    return CurDAG->haveNoCommonBitsSet(N->getOperand(0), N->getOperand(1));
 }]> {

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

let Predicates = [HasSVE2orSME] in {
   defm ADCLT_ZZZ : sve2_int_addsub_long_carry<0b01, "adclt", int_aarch64_sve_adclt>;
   defm SBCLB_ZZZ : sve2_int_addsub_long_carry<0b10, "sbclb", int_aarch64_sve_sbclb>;
   defm SBCLT_ZZZ : sve2_int_addsub_long_carry<0b11, "sbclt", int_aarch64_sve_sbclt>;
 
   // SVE2 bitwise shift right narrow (bottom)
   defm SQSHRUNB_ZZI  : sve2_int_bin_shift_imm_right_narrow_bottom<0b000, "sqshrunb",  int_aarch64_sve_sqshrunb>;
   defm SQRSHRUNB_ZZI : sve2_int_bin_shift_imm_right_narrow_bottom<0b001, "sqrshrunb", int_aarch64_sve_sqrshrunb>;
   defm SHRNB_ZZI     : sve2_int_bin_shift_imm_right_narrow_bottom<0b010, "shrnb",     int_aarch64_sve_shrnb>;
-  defm RSHRNB_ZZI    : sve2_int_bin_shift_imm_right_narrow_bottom<0b011, "rshrnb",    int_aarch64_sve_rshrnb>;
+  defm RSHRNB_ZZI    : sve2_int_bin_shift_imm_right_narrow_bottom<0b011, "rshrnb",    AArch64rshrnb_pf>;
   defm SQSHRNB_ZZI   : sve2_int_bin_shift_imm_right_narrow_bottom<0b100, "sqshrnb",   int_aarch64_sve_sqshrnb>;
   defm SQRSHRNB_ZZI  : sve2_int_bin_shift_imm_right_narrow_bottom<0b101, "sqrshrnb",  int_aarch64_sve_sqrshrnb>;
   defm UQSHRNB_ZZI   : sve2_int_bin_shift_imm_right_narrow_bottom<0b110, "uqshrnb",   int_aarch64_sve_uqshrnb>;
   defm UQRSHRNB_ZZI  : sve2_int_bin_shift_imm_right_narrow_bottom<0b111, "uqrshrnb",  int_aarch64_sve_uqrshrnb>;
 
   // SVE2 bitwise shift right narrow (top)
   defm SQSHRUNT_ZZI  : sve2_int_bin_shift_imm_right_narrow_top<0b000, "sqshrunt",  int_aarch64_sve_sqshrunt>;
   defm SQRSHRUNT_ZZI : sve2_int_bin_shift_imm_right_narrow_top<0b001, "sqrshrunt", int_aarch64_sve_sqrshrunt>;

llvm/test/CodeGen/AArch64/sve2-intrinsics-combine-rshrnb.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve2 < %s | FileCheck %s
+
+define void @add_lshr_rshrnb_b_6(ptr %ptr, ptr %dst, i64 %index){
+; CHECK-LABEL: add_lshr_rshrnb_b_6:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    rshrnb z0.b, z0.h, #6
+; CHECK-NEXT:    st1b { z0.h }, p0, [x1, x2]
+; CHECK-NEXT:    ret
+  %load = load <vscale x 8 x i16>, ptr %ptr, align 2
+  %1 = add <vscale x 8 x i16> %load, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 32, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %2 = lshr <vscale x 8 x i16> %1, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 6, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %3 = trunc <vscale x 8 x i16> %2 to <vscale x 8 x i8>
+  %4 = getelementptr inbounds i8, ptr %dst, i64 %index
+  store <vscale x 8 x i8> %3, ptr %4, align 1
+  ret void
+}
+
+define void @neg_add_lshr_rshrnb_b_6(ptr %ptr, ptr %dst, i64 %index){
+; CHECK-LABEL: neg_add_lshr_rshrnb_b_6:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    add z0.h, z0.h, #1 // =0x1
+; CHECK-NEXT:    lsr z0.h, z0.h, #6
+; CHECK-NEXT:    st1b { z0.h }, p0, [x1, x2]
+; CHECK-NEXT:    ret
+  %load = load <vscale x 8 x i16>, ptr %ptr, align 2
+  %1 = add <vscale x 8 x i16> %load, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 1, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %2 = lshr <vscale x 8 x i16> %1, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 6, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %3 = trunc <vscale x 8 x i16> %2 to <vscale x 8 x i8>
+  %4 = getelementptr inbounds i8, ptr %dst, i64 %index
+  store <vscale x 8 x i8> %3, ptr %4, align 1
+  ret void
+}
+
+define void @add_lshr_rshrnb_h_7(ptr %ptr, ptr %dst, i64 %index){
+; CHECK-LABEL: add_lshr_rshrnb_h_7:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    rshrnb z0.b, z0.h, #7
+; CHECK-NEXT:    st1b { z0.h }, p0, [x1, x2]
+; CHECK-NEXT:    ret
+  %load = load <vscale x 8 x i16>, ptr %ptr, align 2
+  %1 = add <vscale x 8 x i16> %load, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 64, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %2 = lshr <vscale x 8 x i16> %1, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 7, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %3 = trunc <vscale x 8 x i16> %2 to <vscale x 8 x i8>
+  %4 = getelementptr inbounds i8, ptr %dst, i64 %index
+  store <vscale x 8 x i8> %3, ptr %4, align 1
+  ret void
+}
+
+define void @add_lshr_rshrn_h_6(ptr %ptr, ptr %dst, i64 %index){
+; CHECK-LABEL: add_lshr_rshrn_h_6:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    rshrnb z0.h, z0.s, #6
+; CHECK-NEXT:    st1h { z0.s }, p0, [x1, x2, lsl #1]
+; CHECK-NEXT:    ret
+  %load = load <vscale x 4 x i32>, ptr %ptr, align 2
+  %1 = add <vscale x 4 x i32> %load, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 32, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %2 = lshr <vscale x 4 x i32> %1, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 6, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %3 = trunc <vscale x 4 x i32> %2 to <vscale x 4 x i16>
+  %4 = getelementptr inbounds i16, ptr %dst, i64 %index
+  store <vscale x 4 x i16> %3, ptr %4, align 1
+  ret void
+}
+
+define void @add_lshr_rshrnb_h_2(ptr %ptr, ptr %dst, i64 %index){
+; CHECK-LABEL: add_lshr_rshrnb_h_2:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    rshrnb z0.h, z0.s, #2
+; CHECK-NEXT:    st1h { z0.s }, p0, [x1, x2, lsl #1]
+; CHECK-NEXT:    ret
+  %load = load <vscale x 4 x i32>, ptr %ptr, align 2
+  %1 = add <vscale x 4 x i32> %load, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 2, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %2 = lshr <vscale x 4 x i32> %1, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 2, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %3 = trunc <vscale x 4 x i32> %2 to <vscale x 4 x i16>
+  %4 = getelementptr inbounds i16, ptr %dst, i64 %index
+  store <vscale x 4 x i16> %3, ptr %4, align 1
+  ret void
+}
+
+define void @neg_add_lshr_rshrnb_h_0(ptr %ptr, ptr %dst, i64 %index){
+; CHECK-LABEL: neg_add_lshr_rshrnb_h_0:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ret
+  %load = load <vscale x 4 x i32>, ptr %ptr, align 2
+  %1 = add <vscale x 4 x i32> %load, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %2 = lshr <vscale x 4 x i32> %1, trunc (<vscale x 4 x i64> shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 -1, i64 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer) to <vscale x 4 x i32>)
+  %3 = trunc <vscale x 4 x i32> %2 to <vscale x 4 x i16>
+  %4 = getelementptr inbounds i16, ptr %dst, i64 %index
+  store <vscale x 4 x i16> %3, ptr %4, align 1
+  ret void
+}
+
+define void @wide_add_shift_add_rshrnb_b(ptr %dest, i64 %index, <vscale x 16 x i16> %arg1){
+; CHECK-LABEL: wide_add_shift_add_rshrnb_b:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    rshrnb z1.b, z1.h, #6
+; CHECK-NEXT:    ld1b { z2.b }, p0/z, [x0, x1]
+; CHECK-NEXT:    rshrnb z0.b, z0.h, #6
+; CHECK-NEXT:    uzp1 z0.b, z0.b, z1.b
+; CHECK-NEXT:    add z0.b, z2.b, z0.b
+; CHECK-NEXT:    st1b { z0.b }, p0, [x0, x1]
+; CHECK-NEXT:    ret
+  %1 = add <vscale x 16 x i16> %arg1, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 32, i64 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
+  %2 = lshr <vscale x 16 x i16> %1, shufflevector (<vscale x 16 x i16> insertelement (<vscale x 16 x i16> poison, i16 6, i64 0), <vscale x 16 x i16> poison, <vscale x 16 x i32> zeroinitializer)
+  %3 = getelementptr inbounds i8, ptr %dest, i64 %index
+  %load = load <vscale x 16 x i8>, ptr %3, align 2
+  %4 = trunc <vscale x 16 x i16> %2 to <vscale x 16 x i8>
+  %5 = add <vscale x 16 x i8> %load, %4
+  store <vscale x 16 x i8> %5, ptr %3, align 2
+  ret void
+}
+
+define void @wide_add_shift_add_rshrnb_h(ptr %dest, i64 %index, <vscale x 8 x i32> %arg1){
+; CHECK-LABEL: wide_add_shift_add_rshrnb_h:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    rshrnb z1.h, z1.s, #6
+; CHECK-NEXT:    ld1h { z2.h }, p0/z, [x0, x1, lsl #1]
+; CHECK-NEXT:    rshrnb z0.h, z0.s, #6
+; CHECK-NEXT:    uzp1 z0.h, z0.h, z1.h
+; CHECK-NEXT:    add z0.h, z2.h, z0.h
+; CHECK-NEXT:    st1h { z0.h }, p0, [x0, x1, lsl #1]
+; CHECK-NEXT:    ret
+  %1 = add <vscale x 8 x i32> %arg1, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 32, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
+  %2 = lshr <vscale x 8 x i32> %1, shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 6, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer)
+  %3 = getelementptr inbounds i16, ptr %dest, i64 %index
+  %load = load <vscale x 8 x i16>, ptr %3, align 2
+  %4 = trunc <vscale x 8 x i32> %2 to <vscale x 8 x i16>
+  %5 = add <vscale x 8 x i16> %load, %4
+  store <vscale x 8 x i16> %5, ptr %3, align 2
+  ret void
+}
+
+define void @neg_trunc_lsr_add_op1_not_splat(ptr %ptr, ptr %dst, i64 %index, <vscale x 8 x i16> %add_op1){
+; CHECK-LABEL: neg_trunc_lsr_add_op1_not_splat:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; CHECK-NEXT:    add z0.h, z1.h, z0.h
+; CHECK-NEXT:    lsr z0.h, z0.h, #6
+; CHECK-NEXT:    st1b { z0.h }, p0, [x1, x2]
+; CHECK-NEXT:    ret
+  %load = load <vscale x 8 x i16>, ptr %ptr, align 2
+  %1 = add <vscale x 8 x i16> %load, %add_op1
+  %2 = lshr <vscale x 8 x i16> %1, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 6, i64 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  %3 = trunc <vscale x 8 x i16> %2 to <vscale x 8 x i8>
+  %4 = getelementptr inbounds i8, ptr %dst, i64 %index
+  store <vscale x 8 x i8> %3, ptr %4, align 1
+  ret void
+}
+
+define void @neg_trunc_lsr_op1_not_splat(ptr %ptr, ptr %dst, i64 %index, <vscale x 8 x i16> %lshr_op1){
+; CHECK-LABEL: neg_trunc_lsr_op1_not_splat:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; CHECK-NEXT:    add z1.h, z1.h, #32 // =0x20
+; CHECK-NEXT:    lsrr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    st1b { z0.h }, p0, [x1, x2]
+; CHECK-NEXT:    ret
+  %load = load <vscale x 8 x i16>, ptr %ptr, align 2
+  %1 = add <vscale x 8 x i16> %load, shufflevector (<vscale x 8 x i16> insertelement (<vscale x 8 x i16> poison, i16 32, i64 0), <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer)
+  %2 = lshr <vscale x 8 x i16> %1, %lshr_op1
+  %3 = trunc <vscale x 8 x i16> %2 to <vscale x 8 x i8>
+  %4 = getelementptr inbounds i8, ptr %dst, i64 %index
+  store <vscale x 8 x i8> %3, ptr %4, align 1
+  ret void
+}
+
+define void @neg_add_has_two_uses(ptr %ptr, ptr %dst, ptr %dst2, i64 %index){
+; CHECK-LABEL: neg_add_has_two_uses:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    add z0.h, z0.h, #32 // =0x20
+; CHECK-NEXT:    lsr z1.h, z0.h, #6
+; CHECK-NEXT:    add z0.h, z0.h, z0.h
+; CHECK-NEXT:    st1h { z0.h }, p0, [x2, x3, lsl #1]
+; CHECK-NEXT:    st1b { z1.h }, p0, [x1, x3]
+; CHECK-NEXT:    ret
+  %load = load <vscale x 8 x i16>, ptr %ptr, align 2
+  %1 = add <vscale x 8 x i16> %load, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 32, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %2 = lshr <vscale x 8 x i16> %1, trunc (<vscale x 8 x i32> shufflevector (<vscale x 8 x i32> insertelement (<vscale x 8 x i32> poison, i32 6, i64 0), <vscale x 8 x i32> poison, <vscale x 8 x i32> zeroinitializer) to <vscale x 8 x i16>)
+  %3 = add <vscale x 8 x i16> %1, %1
+  %4 = getelementptr inbounds i16, ptr %dst2, i64 %index
+  %5 = trunc <vscale x 8 x i16> %2 to <vscale x 8 x i8>
+  %6 = getelementptr inbounds i8, ptr %dst, i64 %index
+  store <vscale x 8 x i16> %3, ptr %4, align 1
+  store <vscale x 8 x i8> %5, ptr %6, align 1
+  ret void
+}
+
+define void @add_lshr_rshrnb_s(ptr %ptr, ptr %dst, i64 %index){
+; CHECK-LABEL: add_lshr_rshrnb_s:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    rshrnb z0.s, z0.d, #6
+; CHECK-NEXT:    st1w { z0.d }, p0, [x1, x2, lsl #2]
+; CHECK-NEXT:    ret
+  %load = load <vscale x 2 x i64>, ptr %ptr, align 2
+  %1 = add <vscale x 2 x i64> %load, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 32, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %2 = lshr <vscale x 2 x i64> %1, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 6, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %3 = trunc <vscale x 2 x i64> %2 to <vscale x 2 x i32>
+  %4 = getelementptr inbounds i32, ptr %dst, i64 %index
+  store <vscale x 2 x i32> %3, ptr %4, align 1
+  ret void
+}
+
+define void @neg_add_lshr_rshrnb_s(ptr %ptr, ptr %dst, i64 %index){
+; CHECK-LABEL: neg_add_lshr_rshrnb_s:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    add z0.d, z0.d, #32 // =0x20
+; CHECK-NEXT:    lsr z0.d, z0.d, #6
+; CHECK-NEXT:    st1h { z0.d }, p0, [x1, x2, lsl #1]
+; CHECK-NEXT:    ret
+  %load = load <vscale x 2 x i64>, ptr %ptr, align 2
+  %1 = add <vscale x 2 x i64> %load, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 32, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %2 = lshr <vscale x 2 x i64> %1, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> poison, i64 6, i64 0), <vscale x 2 x i64> poison, <vscale x 2 x i32> zeroinitializer)
+  %3 = trunc <vscale x 2 x i64> %2 to <vscale x 2 x i16>
+  %4 = getelementptr inbounds i16, ptr %dst, i64 %index
+  store <vscale x 2 x i16> %3, ptr %4, align 1
+  ret void
+}