[WIP][SVE] Prototype for general merging MOVPRFX support.

[cameron.mcinally]

Summary

This is a prototype for general merging MOVPRFX support. It ties the pseudo's destination register to the passthru register, which seems to work for simple test cases.***

The intention is to use this general merging mechanism to also handle the zero merging case, but that would take some more cleanup work to get there.

***From an offline discussion with Sander, this implementation has limitations that I have not hit yet. Posting this patch to start a discussion.

[sdesmalen]

Hi @cameron.mcinally thanks for this patch! The approach makes sense and nicely extends the mechanism we have for the zeroing forms. We can use similar pseudos for the zeroing case as well. Today I played around a bit with your patch and shared some changes for your reference in D80410. I'm not really planning to land it, but it at least highlights the bug for the zeroing-pseudos that currently exists in master. Feel free to use for reference or ignore/discard if you've already worked on something similar.

We should probably think about where we want to write down the design for this mechanism somewhere, as we use these pseudos to solve multiple problems.

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

382

This comment suggests that this is only possible for _ZERO and _UNDEF variants, but I'm not sure if that comment is still correct.

402

Can you update the description of expand_DestructiveOp to describe the new style of pseudos like FSUB_ZPZZ_MERGE_B?

408

nit: s/PassIdx/PassthruIdx/ ?

llvm/lib/Target/AArch64/SVEInstrFormats.td

471

I'm happy to keep the constraint for now, but I don't think it is strictly necessary, because if $Zd != $Zpt it is still possible to generate a valid instruction sequence.

For:

define <vscale x 4 x float> @foo(<vscale x 4 x i1> %p, <vscale x 4 x float> %z0, <vscale x 4 x float> %z1, <vscale x 4 x float> %passthru) {
  %z0_in = select <vscale x 4 x i1> %p, <vscale x 4 x float> %z0, <vscale x 4 x float> %passthru
  %sub = call <vscale x 4 x float> @llvm.aarch64.sve.fsub.nxv4f32(<vscale x 4 x i1> %p, <vscale x 4 x float> %z0_in, <vscale x 4 x float> %z0)
  ret <vscale x 4 x float> %sub
}

LLVM will generate:

movprfx z2.s, p0/m, z0.s
fsub    z2.s, p0/m, z2.s, z0.s
mov     z0.d, z2.d
ret

Where the last mov is needed because the register allocator chose to allocate the pseudo as:

z2 = FSUB_PSEUDO p0, z0, z0, z2

with the result operand z2 tied to the merge value. I think we could alternatively fall back on using a select instruction if the register allocator didn't have the tied operand restriction and instead generate:

sel z0.s, p0, z0.s, z2.s
fsub z0.s, p0/m, z0.s, z0.s
ret

llvm/test/CodeGen/AArch64/sve-intrinsics-int-arith-merging.ll

62

nit: the %a_z name implies that the false lanes are zeroed, perhaps %a_m is more appropriate.

76

I think we should add tests where the %passthru value is %b as that should cause it to use the reverse instruction for e.g. sub -> subr (or swap the operands in case of add).

[cameron.mcinally]

Thanks, Sander.

I have a nagging feeling that I forced this implementation on you. My intention was to use this patch as an intuition pump, not necessarily as the path forward. I don't have a lot of intuition around MOVPRFX today, so I'm hoping for guidance on the best way to proceed.

I definitely see the desired flexibility of having a lowering pass pre-regalloc. If you think that's the better solution, I'll work on it. I just don't have a strong opinions on where in the pipeline that pass should live.

Also, I plan to look at your zeroing Diff (D80410) next week. Thanks for that.

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

382

Massaged this a bit. Let me know if anything sounds off.

402

Updated. Please let me know if that's what you had in mine. Advertising copy is not my strong suit...

408

Updated, but I just realized that this isn't needed yet. With DstReg tied to the Passthru reg, I can replace MI.getOperand(PassIdx).getReg() with DestReg below.

I think I'll leave it for now in case the untied/zeroing case needs it. Can tear it out later if it's not needed.

llvm/lib/Target/AArch64/SVEInstrFormats.td

471

Ah, ok, I think I see the problem now. In this case we can clobber z0 since it only has the one use. I'll have to explore this further.

I think I'll look at the ConditionalEarlyClobberPass to see if this case can be guarded against. If you know it can't, or are aware of other cases that may be problematic, please let me know.

llvm/test/CodeGen/AArch64/sve-intrinsics-int-arith-merging.ll

62

Good catch. Fixed.

76

Added.

Also added a guard to prevent MOVPRFX Zx, p0/m, Zx from being generated. We might not want that for the p0/z zero merging case, but I haven't hit that yet.

[sdesmalen]

In D80260#2051652, @cameron.mcinally wrote:

I have a nagging feeling that I forced this implementation on you.

Not at all, it may be more the other way around :-) We initially went the pseudo-route downstream so we could use the reverse instructions, but we later used this for the cheaper zeroing as well.
For the former (using reverse instructions) there probably isn't much else we can do then go the pseudo route, but for the latter (cheaper merging) there may also be other alternatives to consider.

My intention was to use this patch as an intuition pump, not necessarily as the path forward. I don't have a lot of intuition around MOVPRFX today, so I'm hoping for guidance on the best way to proceed.

Are you happy to bring this topic up in tomorrow's meeting? If there's time, we can talk through the approach in this patch and maybe get some more input.

I definitely see the desired flexibility of having a lowering pass pre-regalloc. If you think that's the better solution, I'll work on it. I just don't have a strong opinions on where in the pipeline that pass should live.

I'm a bit confused by what you mean with 'pre-regalloc lowering pass'? (Do you mean something like the ConditionalEarlyClobber pass mentioned in D80410?)

[cameron.mcinally]

Sorry for the delay, we've been a bit distracted over here...

Here's the patch that we briefly discussed at last week's Sync-up meeting. The patch attempts to rewrite the MachineInstrs in the general zero-merging case. I agree with @paulwalker-arm that MachineInstr isn't a great place to rewrite a sequence of instructions, but this case is fairly limited, so it might not be too bad. I'll elaborate about that a little now.

The general passthru merging case is largely uninteresting. Unless I've made a mistake, register allocation seems optimal for it (in the cases I've come across, at least). The zeroing case is a bit trickier. We start with a pattern like this:

class SVE_3_Op_Pat_SelZero_Passthru<ValueType vtd, SDPatternOperator op, ValueType vt1,
                   ValueType vt2, ValueType vt3, Instruction inst>
: Pat<(vtd (vtd (op vt1:$Op1, (vselect vt1:$Op1, vt2:$Op2, (vt2 SVEDup0:$Dup)), vt3:$Op3))),
      (inst $Op1, $Op2, $Op3, $Dup)>;

The SVEDup0 is the problem. It's accounted for in the Pseudo generated, but the original DUP also hangs around. In cases where the DUP only has one use, it can be removed. That's the hang up, i.e. can we safely remove that superfluous DUP when expanding the Pseudo.

You'll find the proposed solution under the FalseLanes == AArch64::FalseLanesZero block in this Diff. It's not an ideal solution, but I'm not convinced that it's overly risky either. The sequence we're looking for is fairly constrained, *I think*.

You'll also noticed that the test cases I've included are overkill. I've included them to save effort on the reviewer side. Just wanted the register allocation decisions to be easily seen.

[cameron.mcinally]

In D80260#2058560, @sdesmalen wrote:

I definitely see the desired flexibility of having a lowering pass pre-regalloc. If you think that's the better solution, I'll work on it. I just don't have a strong opinions on where in the pipeline that pass should live.

I'm a bit confused by what you mean with 'pre-regalloc lowering pass'? (Do you mean something like the ConditionalEarlyClobber pass mentioned in D80410?)

I'm probably the one confused. :D

We definitely need an extra register available in some of the MOVPRFX cases. Scavenging that reg probably isn't a good fix. So I was thinking it would be easier to generate the MOVPRFX when we're still at the virtual register phase. It's not clear to me how to make this work though, so I might be way off.

@paulwalker-arm mentioned something at last week's Sync-up meeting about generating the MOVPRFX directly during DAGCombine or similar. That sounds promising, but I'm not sure if I see that path clearly yet.

[sdesmalen]

Hi @cameron.mcinally, sorry for my delayed reponse, I was OoO part of last week and didn't have much bandwidth to reply.

The general passthru merging case is largely uninteresting. Unless I've made a mistake, register allocation seems optimal for it (in the cases I've come across, at least).

Yes, that seems a fair assessment.

The zeroing case is a bit trickier. We start with a pattern like this:

class SVE_3_Op_Pat_SelZero_Passthru<ValueType vtd, SDPatternOperator op, ValueType vt1,
                   ValueType vt2, ValueType vt3, Instruction inst>
: Pat<(vtd (vtd (op vt1:$Op1, (vselect vt1:$Op1, vt2:$Op2, (vt2 SVEDup0:$Dup)), vt3:$Op3))),
      (inst $Op1, $Op2, $Op3, $Dup)>;

The SVEDup0 is the problem. It's accounted for in the Pseudo generated, but the original DUP also hangs around. In cases where the DUP only has one use, it can be removed. That's the hang up, i.e. can we safely remove that superfluous DUP when expanding the Pseudo.

I don't think the compiler will always be able to remove the superfluous DUP (it may have been hoisted out of the loop for example), but I'm not sure how much of a problem that is in practice. Maybe this is fine for a first implementation. Did you get a chance to try this out on real code?

You'll find the proposed solution under the FalseLanes == AArch64::FalseLanesZero block in this Diff. It's not an ideal solution, but I'm not convinced that it's overly risky either. The sequence we're looking for is fairly constrained, *I think*.

While it may seem trivial to remove that instruction this way, I'd rather leave this to a dedicated pass like DeadMachineInstructionElim that could be run directly after the pseudo-expand pass. There's details and special cases to take into account, such as defs/uses on subregisters (instead of the full register) or the use being a DEBUG instruction. The former could lead to the DUP being removed while it shouldn't, and the latter can prevent it from being removed while it should. The dedicated pass is probably better suited to handle those.

We definitely need an extra register available in some of the MOVPRFX cases. Scavenging that reg probably isn't a good fix. So I was thinking it would be easier to generate the MOVPRFX when we're still at the virtual register phase. It's not clear to me how to make this work though, so I might be way off.

When passing dup(0) to the pseudo, the compiler can always fall back on the explicit merge (predicated mov) and there is no longer a need to scavenge a register. I agree it would be better to do this earlier so we can avoid this altogether though.

@paulwalker-arm mentioned something at last week's Sync-up meeting about generating the MOVPRFX directly during DAGCombine or similar. That sounds promising, but I'm not sure if I see that path clearly yet.

What Paul meant is that we ideally want to add a new constraints to TableGen that allow us to model the constraints for using MOVPRFX more precisely. For the pseudo:

Zd = FSUB_PSEUDO_ZEROING Pg, Zs1, Zs2

Instead of forcing "Zd = Zs1", we want to allow any register allocation as long as one of the registers is unique, because the ExpandPseudo pass can always handle that case with movprfx directly or additionally with a reverse/commutative instruction. By adding a new TableGen constraint like "one_of_is_unique(Zd, Zs1, Zs2)" and teaching the register allocator to implement that constraint, we can avoid more ugly solutions like the conditional early-clobber pass (that only does half a job because it forces Zd to be unique, which may not necessarily be the best choice) or having to pass the dup(0) to the instruction and then trying to remove the instruction afterwards (which it may fail to remove). We could instead model the constraint as we want it. Other pseudos (e.g. for ternary instructions) may require slightly different constraints, but the one_of_is_unique() probably solves 90-95% of the cases that need solving.

[cameron.mcinally]

Abandoning this revision for now. @greened isn't afraid of the register allocator, so he's going to be picking up this project.

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

 ///   FSUBR_ZPmZ_B  Z0, Pg/m, Z0, Z1
 ///
 /// Then we implement the zeroing of the false lanes of Z0 by adding
 /// a zeroing MOVPRFX instruction:
 ///
 ///   MOVPRFX_ZPzZ_B Z0, Pg/z, Z0
 ///   FSUBR_ZPmZ_B   Z0, Pg/m, Z0, Z1
 ///
 /// Note that this can only be done for _ZERO or _UNDEF variants where
 /// we can guarantee the false lanes to be zeroed (by implementing this)
 /// or that they are undef (don't care / not used), otherwise the
 /// swapping of operands is illegal because the operation is not
 /// (or cannot be emulated to be) fully commutative.
 bool AArch64ExpandPseudo::expand_DestructiveOp(
                             MachineInstr &MI,
                             MachineBasicBlock &MBB,
                             MachineBasicBlock::iterator MBBI) {
   unsigned Opcode = AArch64::getSVEPseudoMap(MI.getOpcode());
   uint64_t DType = TII->get(Opcode).TSFlags & AArch64::DestructiveInstTypeMask;
   uint64_t FalseLanes = MI.getDesc().TSFlags & AArch64::FalseLanesMask;
-  bool FalseZero = FalseLanes == AArch64::FalseLanesZero;
 
   unsigned DstReg = MI.getOperand(0).getReg();
   bool DstIsDead = MI.getOperand(0).isDead();
 
   if (DType == AArch64::DestructiveBinary)
     assert(DstReg != MI.getOperand(3).getReg());
 
   bool UseRev = false;
-  unsigned PredIdx, DOPIdx, SrcIdx;
+  unsigned PredIdx, DOPIdx, SrcIdx, PassIdx;
   switch (DType) {
   case AArch64::DestructiveBinaryComm:
   case AArch64::DestructiveBinaryCommWithRev:
     if (DstReg == MI.getOperand(3).getReg()) {
       // FSUB Zd, Pg, Zs1, Zd  ==> FSUBR   Zd, Pg/m, Zd, Zs1
-      std::tie(PredIdx, DOPIdx, SrcIdx) = std::make_tuple(1, 3, 2);
+      std::tie(PredIdx, DOPIdx, SrcIdx, PassIdx) = std::make_tuple(1, 3, 2, 4);
       UseRev = true;
       break;
     }
     LLVM_FALLTHROUGH;
   case AArch64::DestructiveBinary:
   case AArch64::DestructiveBinaryImm:
-    std::tie(PredIdx, DOPIdx, SrcIdx) = std::make_tuple(1, 2, 3);
+    std::tie(PredIdx, DOPIdx, SrcIdx, PassIdx) = std::make_tuple(1, 2, 3, 4);
    break;
   default:
     llvm_unreachable("Unsupported Destructive Operand type");
   }
 
 #ifndef NDEBUG
   // MOVPRFX can only be used if the destination operand
   // is the destructive operand, not as any other operand,
   // so the Destructive Operand must be unique.
if (UseRev) {
     if (AArch64::getSVERevInstr(Opcode) != -1)
       Opcode = AArch64::getSVERevInstr(Opcode);
     else if (AArch64::getSVEOrigInstr(Opcode) != -1)
       Opcode = AArch64::getSVEOrigInstr(Opcode);
   }
 
   // Get the right MOVPRFX
   uint64_t ElementSize = TII->getElementSizeForOpcode(Opcode);
-  unsigned MovPrfx, MovPrfxZero;
+  unsigned MovPrfx, MovPrfxZero, MovPrfxMerge;
   switch (ElementSize) {
   case AArch64::ElementSizeNone:
   case AArch64::ElementSizeB:
     MovPrfx = AArch64::MOVPRFX_ZZ;
     MovPrfxZero = AArch64::MOVPRFX_ZPzZ_B;
+    MovPrfxMerge = AArch64::MOVPRFX_ZPmZ_B;
     break;
   case AArch64::ElementSizeH:
     MovPrfx = AArch64::MOVPRFX_ZZ;
     MovPrfxZero = AArch64::MOVPRFX_ZPzZ_H;
+    MovPrfxMerge = AArch64::MOVPRFX_ZPmZ_H;
     break;
   case AArch64::ElementSizeS:
     MovPrfx = AArch64::MOVPRFX_ZZ;
     MovPrfxZero = AArch64::MOVPRFX_ZPzZ_S;
+    MovPrfxMerge = AArch64::MOVPRFX_ZPmZ_S;
     break;
   case AArch64::ElementSizeD:
     MovPrfx = AArch64::MOVPRFX_ZZ;
     MovPrfxZero = AArch64::MOVPRFX_ZPzZ_D;
+    MovPrfxMerge = AArch64::MOVPRFX_ZPmZ_D;
     break;
   default:
     llvm_unreachable("Unsupported ElementSize");
   }
 
   //
   // Create the destructive operation (if required)
   //
   MachineInstrBuilder PRFX, DOP;
-  if (FalseZero) {
+  if (FalseLanes == AArch64::FalseLanesZero) {
     assert(ElementSize != AArch64::ElementSizeNone &&
            "This instruction is unpredicated");
 
     // Merge source operand into destination register
     PRFX = BuildMI(MBB, MBBI, MI.getDebugLoc(), TII->get(MovPrfxZero))
                .addReg(DstReg, RegState::Define)
                .addReg(MI.getOperand(PredIdx).getReg())
                .addReg(MI.getOperand(DOPIdx).getReg());
 
     // After the movprfx, the destructive operand is same as Dst
     DOPIdx = 0;
+  } else if (FalseLanes == AArch64::FalseLanesMerge) {
+    PRFX = BuildMI(MBB, MBBI, MI.getDebugLoc(), TII->get(MovPrfxMerge))
+               .addReg(MI.getOperand(PassIdx).getReg(), RegState::Define)
+               .addReg(MI.getOperand(PassIdx).getReg())
+               .addReg(MI.getOperand(PredIdx).getReg())
+               .addReg(MI.getOperand(DOPIdx).getReg());
+    DOPIdx = 0;
   } else if (DstReg != MI.getOperand(DOPIdx).getReg()) {
     PRFX = BuildMI(MBB, MBBI, MI.getDebugLoc(), TII->get(MovPrfx))
                .addReg(DstReg, RegState::Define)
                .addReg(MI.getOperand(DOPIdx).getReg());
     DOPIdx = 0;
   }
 
   //

llvm/lib/Target/AArch64/AArch64InstrFormats.td

 def DestructiveUnary              : DestructiveInstTypeEnum<2>;
 def DestructiveBinaryImm          : DestructiveInstTypeEnum<3>;
 def DestructiveBinaryShImmUnpred  : DestructiveInstTypeEnum<4>;
 def DestructiveBinary             : DestructiveInstTypeEnum<5>;
 def DestructiveBinaryComm         : DestructiveInstTypeEnum<6>;
 def DestructiveBinaryCommWithRev  : DestructiveInstTypeEnum<7>;
 def DestructiveTernaryCommWithRev : DestructiveInstTypeEnum<8>;
 
-class FalseLanesEnum<bits<2> val> {
-  bits<2> Value = val;
+class FalseLanesEnum<bits<3> val> {
+  bits<3> Value = val;
 }
 def FalseLanesNone  : FalseLanesEnum<0>;
 def FalseLanesZero  : FalseLanesEnum<1>;
 def FalseLanesUndef : FalseLanesEnum<2>;
+def FalseLanesMerge : FalseLanesEnum<3>;
 
 // AArch64 Instruction Format
 class AArch64Inst<Format f, string cstr> : Instruction {
   field bits<32> Inst; // Instruction encoding.
   // Mask of bits that cause an encoding to be UNPREDICTABLE.
   // If a bit is set, then if the corresponding bit in the
   // target encoding differs from its value in the "Inst" field,
   // the instruction is UNPREDICTABLE (SoftFail in abstract parlance).
   field bits<32> Unpredictable = 0;
   // SoftFail is the generic name for this field, but we alias it so
   // as to make it more obvious what it means in ARM-land.
   field bits<32> SoftFail = Unpredictable;
   let Namespace   = "AArch64";
   Format F        = f;
   bits<2> Form    = F.Value;
 
   // Defaults
   FalseLanesEnum FalseLanes = FalseLanesNone;
   DestructiveInstTypeEnum DestructiveInstType = NotDestructive;
   ElementSizeEnum ElementSize = ElementSizeNone;
 
-  let TSFlags{8-7} = FalseLanes.Value;
+  let TSFlags{9-7} = FalseLanes.Value;
   let TSFlags{6-3} = DestructiveInstType.Value;
   let TSFlags{2-0} = ElementSize.Value;
 
   let Pattern     = [];
   let Constraints = cstr;
 }
 
 class InstSubst<string Asm, dag Result, bit EmitPriority = 0>

llvm/lib/Target/AArch64/AArch64InstrInfo.h

enum DestructiveInstType {
   DestructiveBinaryShImmUnpred  = TSFLAG_DESTRUCTIVE_INST_TYPE(0x4),
   DestructiveBinary             = TSFLAG_DESTRUCTIVE_INST_TYPE(0x5),
   DestructiveBinaryComm         = TSFLAG_DESTRUCTIVE_INST_TYPE(0x6),
   DestructiveBinaryCommWithRev  = TSFLAG_DESTRUCTIVE_INST_TYPE(0x7),
   DestructiveTernaryCommWithRev = TSFLAG_DESTRUCTIVE_INST_TYPE(0x8),
 };
 
 enum FalseLaneType {
   FalseLanesMask  = TSFLAG_FALSE_LANE_TYPE(0x3),
   FalseLanesZero  = TSFLAG_FALSE_LANE_TYPE(0x1),
   FalseLanesUndef = TSFLAG_FALSE_LANE_TYPE(0x2),
+  FalseLanesMerge = TSFLAG_FALSE_LANE_TYPE(0x3),
 };
 
 #undef TSFLAG_ELEMENT_SIZE_TYPE
 #undef TSFLAG_DESTRUCTIVE_INST_TYPE
 #undef TSFLAG_FALSE_LANE_TYPE
 
 int getSVEPseudoMap(uint16_t Opcode);
 int getSVERevInstr(uint16_t Opcode);
 int getSVEOrigInstr(uint16_t Opcode);
 }
 
 } // end namespace llvm
 
 #endif

llvm/lib/Target/AArch64/SVEInstrFormats.td

class SVE_4_Op_Imm_Pat<ValueType vtd, SDPatternOperator op, ValueType vt1,
                        Operand ImmTy, Instruction inst>
 : Pat<(vtd (op vt1:$Op1, vt2:$Op2, vt3:$Op3, (vt4 ImmTy:$Op4))),
       (inst $Op1, $Op2, $Op3, ImmTy:$Op4)>;
 
 def SVEDup0 : ComplexPattern<i64, 0, "SelectDupZero", []>;
 def SVEDup0Undef : ComplexPattern<i64, 0, "SelectDupZeroOrUndef", []>;
 
 let AddedComplexity = 1 in {
+class SVE_3_Op_Pat_Sel<ValueType vtd, SDPatternOperator op, ValueType vt1,
+                       ValueType vt2, ValueType vt3, Instruction inst>
+: Pat<(vtd (op vt1:$Op1, (vselect vt1:$Op1, vt2:$Op2, vt2:$Passthru), vt3:$Op3)),
+      (inst $Op1, $Op2, $Op3, $Passthru)>;
+
 class SVE_3_Op_Pat_SelZero<ValueType vtd, SDPatternOperator op, ValueType vt1,
                    ValueType vt2, ValueType vt3, Instruction inst>
 : Pat<(vtd (vtd (op vt1:$Op1, (vselect vt1:$Op1, vt2:$Op2, (SVEDup0)), vt3:$Op3))),
       (inst $Op1, $Op2, $Op3)>;
 
 class SVE_3_Op_Pat_Shift_Imm_SelZero<ValueType vtd, SDPatternOperator op,
                                      ValueType vt1, ValueType vt2,
                                      Operand vt3, Instruction inst>
let hasNoSchedulingInfo = 1 in {
   }
 
   class PredTwoOpImmPseudo<string name, ZPRRegOp zprty, Operand immty,
                            FalseLanesEnum flags = FalseLanesNone>
   : SVEPseudo2Instr<name, 0>,
     Pseudo<(outs zprty:$Zd), (ins PPR3bAny:$Pg, zprty:$Zs1, immty:$imm), []> {
     let FalseLanes = flags;
   }
+
+  class PredTwoOpMergePseudo<string name, ZPRRegOp zprty,
+                             FalseLanesEnum flags = FalseLanesNone>
+  : SVEPseudo2Instr<name, 0>,
+    Pseudo<(outs zprty:$Zd), (ins PPR3bAny:$Pg, zprty:$Zs1, zprty:$Zs2, zprty:$Zpt), []> {
+    let FalseLanes = flags;
+    let Constraints = "$Zd = $Zpt";
+  }
 }
 
 //===----------------------------------------------------------------------===//
 // SVE Predicate Misc Group
 //===----------------------------------------------------------------------===//
 
 class sve_int_pfalse<bits<6> opc, string asm>
 : I<(outs PPR8:$Pd), (ins),
multiclass sve_int_bin_pred_zx<SDPatternOperator op> {
   def _ZERO_H : PredTwoOpPseudo<NAME # _H, ZPR16, FalseLanesZero>;
   def _ZERO_S : PredTwoOpPseudo<NAME # _S, ZPR32, FalseLanesZero>;
   def _ZERO_D : PredTwoOpPseudo<NAME # _D, ZPR64, FalseLanesZero>;
 
   def : SVE_3_Op_Pat_SelZero<nxv16i8, op, nxv16i1, nxv16i8, nxv16i8, !cast<Pseudo>(NAME # _ZERO_B)>;
   def : SVE_3_Op_Pat_SelZero<nxv8i16, op, nxv8i1, nxv8i16, nxv8i16, !cast<Pseudo>(NAME # _ZERO_H)>;
   def : SVE_3_Op_Pat_SelZero<nxv4i32, op, nxv4i1, nxv4i32, nxv4i32, !cast<Pseudo>(NAME # _ZERO_S)>;
   def : SVE_3_Op_Pat_SelZero<nxv2i64, op, nxv2i1, nxv2i64, nxv2i64, !cast<Pseudo>(NAME # _ZERO_D)>;
+
+  def _B : PredTwoOpMergePseudo<NAME # _B, ZPR8, FalseLanesMerge>;
+  def _H : PredTwoOpMergePseudo<NAME # _H, ZPR16, FalseLanesMerge>;
+  def _S : PredTwoOpMergePseudo<NAME # _S, ZPR32, FalseLanesMerge>;
+  def _D : PredTwoOpMergePseudo<NAME # _D, ZPR64, FalseLanesMerge>;
+
+  def : SVE_3_Op_Pat_Sel<nxv16i8, op, nxv16i1, nxv16i8, nxv16i8, !cast<Pseudo>(NAME # _B)>;
+  def : SVE_3_Op_Pat_Sel<nxv8i16, op, nxv8i1, nxv8i16, nxv8i16, !cast<Pseudo>(NAME # _H)>;
+  def : SVE_3_Op_Pat_Sel<nxv4i32, op, nxv4i1, nxv4i32, nxv4i32, !cast<Pseudo>(NAME # _S)>;
+  def : SVE_3_Op_Pat_Sel<nxv2i64, op, nxv2i1, nxv2i64, nxv2i64, !cast<Pseudo>(NAME # _D)>;
 }
 
 multiclass sve_int_bin_pred_shift_wide<bits<3> opc, string asm,
                                   SDPatternOperator op> {
   def _B : sve_int_bin_pred_shift<0b00, 0b1, opc, asm, ZPR8, ZPR64>;
   def _H : sve_int_bin_pred_shift<0b01, 0b1, opc, asm, ZPR16, ZPR64>;
   def _S : sve_int_bin_pred_shift<0b10, 0b1, opc, asm, ZPR32, ZPR64>;
 

llvm/test/CodeGen/AArch64/sve-intrinsics-int-arith-merging.ll

 ; CHECK-NEXT: ret
   %a_z = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> zeroinitializer
   %out = call <vscale x 2 x i64> @llvm.aarch64.sve.add.nxv2i64(<vscale x 2 x i1> %pg,
                                                                <vscale x 2 x i64> %a_z,
                                                                <vscale x 2 x i64> %b)
   ret <vscale x 2 x i64> %out
 }
 
+define <vscale x 16 x i8> @add_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a,
+                                  <vscale x 16 x i8> %b, <vscale x 16 x i8> %passthru) {
+; CHECK-LABEL: add_i8:
+; CHECK:      movprfx z2.b, p0/m, z0.b
+; CHECK-NEXT: add z2.b, p0/m, z2.b, z1.b
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %passthru
+  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.add.nxv16i8(<vscale x 16 x i1> %pg,
+                                                               <vscale x 16 x i8> %a_z,
+                                                               <vscale x 16 x i8> %b)
+  ret <vscale x 16 x i8> %out
+}
+
+define <vscale x 8 x i16> @add_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a,
+                                   <vscale x 8 x i16> %b, <vscale x 8 x i16> %passthru) {
+; CHECK-LABEL: add_i16
+; CHECK:      movprfx z2.h, p0/m, z0.h
+; CHECK-NEXT: add z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %passthru
+  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.add.nxv8i16(<vscale x 8 x i1> %pg,
+                                                               <vscale x 8 x i16> %a_z,
+                                                               <vscale x 8 x i16> %b)
+  ret <vscale x 8 x i16> %out
+}
+
+define <vscale x 4 x i32> @add_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a,
+                                   <vscale x 4 x i32> %b, <vscale x 4 x i32> %passthru) {
+; CHECK-LABEL: add_i32:
+; CHECK:      movprfx z2.s, p0/m, z0.s
+; CHECK-NEXT: add z2.s, p0/m, z2.s, z1.s
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %passthru
+  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.add.nxv4i32(<vscale x 4 x i1> %pg,
+                                                               <vscale x 4 x i32> %a_z,
+                                                               <vscale x 4 x i32> %b)
+  ret <vscale x 4 x i32> %out
+}
+
+define <vscale x 2 x i64> @add_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a,
+                                   <vscale x 2 x i64> %b, <vscale x 2 x i64> %passthru) {
+; CHECK-LABEL: add_i64:
+; CHECK:      movprfx z2.d, p0/m, z0.d
+; CHECK-NEXT: add z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %passthru
+  %out = call <vscale x 2 x i64> @llvm.aarch64.sve.add.nxv2i64(<vscale x 2 x i1> %pg,
+                                                               <vscale x 2 x i64> %a_z,
+                                                               <vscale x 2 x i64> %b)
+  ret <vscale x 2 x i64> %out
+}
+
 ;
 ; SUB
 ;
 
 define <vscale x 16 x i8> @sub_i8_zero(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b) {
 ; CHECK-LABEL: sub_i8_zero:
 ; CHECK:      movprfx z0.b, p0/z, z0.b
 ; CHECK-NEXT: sub z0.b, p0/m, z0.b, z1.b

 ; CHECK-NEXT: ret
   %a_z = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> zeroinitializer
   %out = call <vscale x 2 x i64> @llvm.aarch64.sve.sub.nxv2i64(<vscale x 2 x i1> %pg,
                                                                <vscale x 2 x i64> %a_z,
                                                                <vscale x 2 x i64> %b)
   ret <vscale x 2 x i64> %out
 }
 
+define <vscale x 16 x i8> @sub_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a,
+                                  <vscale x 16 x i8> %b, <vscale x 16 x i8> %passthru) {
+; CHECK-LABEL: sub_i8:
+; CHECK:      movprfx z2.b, p0/m, z0.b
+; CHECK-NEXT: sub z2.b, p0/m, z2.b, z1.b
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %passthru
+  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.sub.nxv16i8(<vscale x 16 x i1> %pg,
+                                                               <vscale x 16 x i8> %a_z,
+                                                               <vscale x 16 x i8> %b)
+  ret <vscale x 16 x i8> %out
+}
+
+define <vscale x 8 x i16> @sub_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a,
+                                   <vscale x 8 x i16> %b, <vscale x 8 x i16> %passthru) {
+; CHECK-LABEL: sub_i16
+; CHECK:      movprfx z2.h, p0/m, z0.h
+; CHECK-NEXT: sub z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %passthru
+  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.sub.nxv8i16(<vscale x 8 x i1> %pg,
+                                                               <vscale x 8 x i16> %a_z,
+                                                               <vscale x 8 x i16> %b)
+  ret <vscale x 8 x i16> %out
+}
+
+define <vscale x 4 x i32> @sub_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a,
+                                   <vscale x 4 x i32> %b, <vscale x 4 x i32> %passthru) {
+; CHECK-LABEL: sub_i32:
+; CHECK:      movprfx z2.s, p0/m, z0.s
+; CHECK-NEXT: sub z2.s, p0/m, z2.s, z1.s
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %passthru
+  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.sub.nxv4i32(<vscale x 4 x i1> %pg,
+                                                               <vscale x 4 x i32> %a_z,
+                                                               <vscale x 4 x i32> %b)
+  ret <vscale x 4 x i32> %out
+}
+
+define <vscale x 2 x i64> @sub_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a,
+                                   <vscale x 2 x i64> %b, <vscale x 2 x i64> %passthru) {
+; CHECK-LABEL: sub_i64:
+; CHECK:      movprfx z2.d, p0/m, z0.d
+; CHECK-NEXT: sub z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %passthru
+  %out = call <vscale x 2 x i64> @llvm.aarch64.sve.sub.nxv2i64(<vscale x 2 x i1> %pg,
+                                                               <vscale x 2 x i64> %a_z,
+                                                               <vscale x 2 x i64> %b)
+  ret <vscale x 2 x i64> %out
+}
+
 ;
 ; SUBR
 ;
 
 define <vscale x 16 x i8> @subr_i8_zero(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b) {
 ; CHECK-LABEL: subr_i8_zero:
 ; CHECK:      movprfx z0.b, p0/z, z0.b
 ; CHECK-NEXT: subr z0.b, p0/m, z0.b, z1.b

 ; CHECK-NEXT: ret
   %a_z = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> zeroinitializer
   %out = call <vscale x 2 x i64> @llvm.aarch64.sve.subr.nxv2i64(<vscale x 2 x i1> %pg,
                                                                 <vscale x 2 x i64> %a_z,
                                                                 <vscale x 2 x i64> %b)
   ret <vscale x 2 x i64> %out
 }
 
+define <vscale x 16 x i8> @subr_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a,
+                                   <vscale x 16 x i8> %b, <vscale x 16 x i8> %passthru) {
+; CHECK-LABEL: subr_i8:
+; CHECK:      movprfx z2.b, p0/m, z0.b
+; CHECK-NEXT: subr z2.b, p0/m, z2.b, z1.b
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %passthru
+  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.subr.nxv16i8(<vscale x 16 x i1> %pg,
+                                                                <vscale x 16 x i8> %a_z,
+                                                                <vscale x 16 x i8> %b)
+  ret <vscale x 16 x i8> %out
+}
+
+define <vscale x 8 x i16> @subr_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a,
+                                   <vscale x 8 x i16> %b, <vscale x 8 x i16> %passthru) {
+; CHECK-LABEL: subr_i16
+; CHECK:      movprfx z2.h, p0/m, z0.h
+; CHECK-NEXT: subr z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %passthru
+  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.subr.nxv8i16(<vscale x 8 x i1> %pg,
+                                                                <vscale x 8 x i16> %a_z,
+                                                                <vscale x 8 x i16> %b)
+  ret <vscale x 8 x i16> %out
+}
+
+define <vscale x 4 x i32> @subr_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a,
+                                    <vscale x 4 x i32> %b, <vscale x 4 x i32> %passthru) {
+; CHECK-LABEL: subr_i32:
+; CHECK:      movprfx z2.s, p0/m, z0.s
+; CHECK-NEXT: subr z2.s, p0/m, z2.s, z1.s
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %passthru
+  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.subr.nxv4i32(<vscale x 4 x i1> %pg,
+                                                                <vscale x 4 x i32> %a_z,
+                                                                <vscale x 4 x i32> %b)
+  ret <vscale x 4 x i32> %out
+}
+
+define <vscale x 2 x i64> @subr_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a,
+                                    <vscale x 2 x i64> %b, <vscale x 2 x i64> %passthru) {
+; CHECK-LABEL: subr_i64:
+; CHECK:      movprfx z2.d, p0/m, z0.d
+; CHECK-NEXT: subr z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %passthru
+  %out = call <vscale x 2 x i64> @llvm.aarch64.sve.subr.nxv2i64(<vscale x 2 x i1> %pg,
+                                                                <vscale x 2 x i64> %a_z,
+                                                                <vscale x 2 x i64> %b)
+  ret <vscale x 2 x i64> %out
+}
+
 declare <vscale x 16 x i8> @llvm.aarch64.sve.add.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x i8>, <vscale x 16 x i8>)
 declare <vscale x 8 x i16> @llvm.aarch64.sve.add.nxv8i16(<vscale x 8 x i1>, <vscale x 8 x i16>, <vscale x 8 x i16>)
 declare <vscale x 4 x i32> @llvm.aarch64.sve.add.nxv4i32(<vscale x 4 x i1>, <vscale x 4 x i32>, <vscale x 4 x i32>)
 declare <vscale x 2 x i64> @llvm.aarch64.sve.add.nxv2i64(<vscale x 2 x i1>, <vscale x 2 x i64>, <vscale x 2 x i64>)
 
 declare <vscale x 16 x i8> @llvm.aarch64.sve.sub.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x i8>, <vscale x 16 x i8>)
 declare <vscale x 8 x i16> @llvm.aarch64.sve.sub.nxv8i16(<vscale x 8 x i1>, <vscale x 8 x i16>, <vscale x 8 x i16>)
 declare <vscale x 4 x i32> @llvm.aarch64.sve.sub.nxv4i32(<vscale x 4 x i1>, <vscale x 4 x i32>, <vscale x 4 x i32>)
 declare <vscale x 2 x i64> @llvm.aarch64.sve.sub.nxv2i64(<vscale x 2 x i1>, <vscale x 2 x i64>, <vscale x 2 x i64>)
 
 declare <vscale x 16 x i8> @llvm.aarch64.sve.subr.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x i8>, <vscale x 16 x i8>)
 declare <vscale x 8 x i16> @llvm.aarch64.sve.subr.nxv8i16(<vscale x 8 x i1>, <vscale x 8 x i16>, <vscale x 8 x i16>)
 declare <vscale x 4 x i32> @llvm.aarch64.sve.subr.nxv4i32(<vscale x 4 x i1>, <vscale x 4 x i32>, <vscale x 4 x i32>)
 declare <vscale x 2 x i64> @llvm.aarch64.sve.subr.nxv2i64(<vscale x 2 x i1>, <vscale x 2 x i64>, <vscale x 2 x i64>)

llvm/test/CodeGen/AArch64/sve-intrinsics-shifts-merging.ll

 ; CHECK: asr z0.s, p0/m, z0.s, z1.d
   %a_z = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> zeroinitializer
   %out = call <vscale x 4 x i32> @llvm.aarch64.sve.asr.wide.nxv4i32(<vscale x 4 x i1> %pg,
                                                                <vscale x 4 x i32> %a_z,
                                                                <vscale x 2 x i64> %b)
   ret <vscale x 4 x i32> %out
 }
 
+define <vscale x 16 x i8> @asr_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a,
+                                  <vscale x 16 x i8> %b, <vscale x 16 x i8> %passthru) {
+; CHECK-LABEL: asr_i8:
+; CHECK:      movprfx z2.b, p0/m, z0.b
+; CHECK-NEXT: asr z2.b, p0/m, z2.b, z1.b
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %passthru
+  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.asr.nxv16i8(<vscale x 16 x i1> %pg,
+                                                          <vscale x 16 x i8> %a_z,
+                                                          <vscale x 16 x i8> %b)
+  ret <vscale x 16 x i8> %out
+}
+
+define <vscale x 8 x i16> @asr_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a,
+                                   <vscale x 8 x i16> %b, <vscale x 8 x i16> %passthru) {
+; CHECK-LABEL: asr_i16:
+; CHECK:      movprfx z2.h, p0/m, z0.h
+; CHECK-NEXT: asr z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %passthru
+  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.asr.nxv8i16(<vscale x 8 x i1> %pg,
+                                                          <vscale x 8 x i16> %a_z,
+                                                          <vscale x 8 x i16> %b)
+  ret <vscale x 8 x i16> %out
+}
+
+define <vscale x 4 x i32> @asr_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a,
+                                   <vscale x 4 x i32> %b, <vscale x 4 x i32> %passthru) {
+; CHECK-LABEL: asr_i32:
+; CHECK:      movprfx z2.s, p0/m, z0.s
+; CHECK-NEXT: asr z2.s, p0/m, z2.s, z1.s
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %passthru
+  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.asr.nxv4i32(<vscale x 4 x i1> %pg,
+                                                          <vscale x 4 x i32> %a_z,
+                                                          <vscale x 4 x i32> %b)
+  ret <vscale x 4 x i32> %out
+}
+
+define <vscale x 2 x i64> @asr_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a,
+                                   <vscale x 2 x i64> %b, <vscale x 2 x i64> %passthru) {
+; CHECK-LABEL: asr_i64:
+; CHECK:      movprfx z2.d, p0/m, z0.d
+; CHECK-NEXT: asr z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %passthru
+  %out = call <vscale x 2 x i64> @llvm.aarch64.sve.asr.nxv2i64(<vscale x 2 x i1> %pg,
+                                                          <vscale x 2 x i64> %a_z,
+                                                          <vscale x 2 x i64> %b)
+  ret <vscale x 2 x i64> %out
+}
+
 ;
 ; ASRD
 ;
 
 define <vscale x 16 x i8> @asrd_i8_zero(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a) {
 ; CHECK-LABEL: asrd_i8_zero:
 ; CHECK:      movprfx z0.b, p0/z, z0.b
 ; CHECK-NEXT: asrd z0.b, p0/m, z0.b, #1

 ; CHECK: lsl z0.s, p0/m, z0.s, z1.d
   %a_z = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> zeroinitializer
   %out = call <vscale x 4 x i32> @llvm.aarch64.sve.lsl.wide.nxv4i32(<vscale x 4 x i1> %pg,
                                                                <vscale x 4 x i32> %a_z,
                                                                <vscale x 2 x i64> %b)
   ret <vscale x 4 x i32> %out
 }
 
+define <vscale x 16 x i8> @lsl_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a,
+                                   <vscale x 16 x i8> %b, <vscale x 16 x i8> %passthru) {
+; CHECK-LABEL: lsl_i8:
+; CHECK:      movprfx z2.b, p0/m, z0.b
+; CHECK-NEXT: lsl z2.b, p0/m, z2.b, z1.b
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %passthru
+  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.lsl.nxv16i8(<vscale x 16 x i1> %pg,
+                                                          <vscale x 16 x i8> %a_z,
+                                                          <vscale x 16 x i8> %b)
+  ret <vscale x 16 x i8> %out
+}
+
+define <vscale x 8 x i16> @lsl_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a,
+                                   <vscale x 8 x i16> %b, <vscale x 8 x i16> %passthru) {
+; CHECK-LABEL: lsl_i16:
+; CHECK:      movprfx z2.h, p0/m, z0.h
+; CHECK-NEXT: lsl z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %passthru
+  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.lsl.nxv8i16(<vscale x 8 x i1> %pg,
+                                                          <vscale x 8 x i16> %a_z,
+                                                          <vscale x 8 x i16> %b)
+  ret <vscale x 8 x i16> %out
+}
+
+define <vscale x 4 x i32> @lsl_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a,
+                                   <vscale x 4 x i32> %b, <vscale x 4 x i32> %passthru) {
+; CHECK-LABEL: lsl_i32:
+; CHECK:      movprfx z2.s, p0/m, z0.s
+; CHECK-NEXT: lsl z2.s, p0/m, z2.s, z1.s
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %passthru
+  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.lsl.nxv4i32(<vscale x 4 x i1> %pg,
+                                                          <vscale x 4 x i32> %a_z,
+                                                          <vscale x 4 x i32> %b)
+  ret <vscale x 4 x i32> %out
+}
+
+define <vscale x 2 x i64> @lsl_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a,
+                                   <vscale x 2 x i64> %b, <vscale x 2 x i64> %passthru) {
+; CHECK-LABEL: lsl_i64:
+; CHECK:      movprfx z2.d, p0/m, z0.d
+; CHECK-NEXT: lsl z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %passthru
+  %out = call <vscale x 2 x i64> @llvm.aarch64.sve.lsl.nxv2i64(<vscale x 2 x i1> %pg,
+                                                          <vscale x 2 x i64> %a_z,
+                                                          <vscale x 2 x i64> %b)
+  ret <vscale x 2 x i64> %out
+}
+
 ;
 ; LSR
 ;
 
 define <vscale x 16 x i8> @lsr_i8_zero(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %b) {
 ; CHECK-LABEL: lsr_i8_zero:
 ; CHECK:      movprfx z0.b, p0/z, z0.b
 ; CHECK-NEXT: lsr z0.b, p0/m, z0.b, z1.b

 ; CHECK: lsr z0.s, p0/m, z0.s, z1.d
   %a_z = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> zeroinitializer
   %out = call <vscale x 4 x i32> @llvm.aarch64.sve.lsr.wide.nxv4i32(<vscale x 4 x i1> %pg,
                                                                <vscale x 4 x i32> %a_z,
                                                                <vscale x 2 x i64> %b)
   ret <vscale x 4 x i32> %out
 }
 
+define <vscale x 16 x i8> @lsr_i8(<vscale x 16 x i1> %pg, <vscale x 16 x i8> %a,
+                                   <vscale x 16 x i8> %b, <vscale x 16 x i8> %passthru) {
+; CHECK-LABEL: lsr_i8:
+; CHECK:      movprfx z2.b, p0/m, z0.b
+; CHECK-NEXT: lsr z2.b, p0/m, z2.b, z1.b
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 16 x i1> %pg, <vscale x 16 x i8> %a, <vscale x 16 x i8> %passthru
+  %out = call <vscale x 16 x i8> @llvm.aarch64.sve.lsr.nxv16i8(<vscale x 16 x i1> %pg,
+                                                          <vscale x 16 x i8> %a_z,
+                                                          <vscale x 16 x i8> %b)
+  ret <vscale x 16 x i8> %out
+}
+
+define <vscale x 8 x i16> @lsr_i16(<vscale x 8 x i1> %pg, <vscale x 8 x i16> %a,
+                                   <vscale x 8 x i16> %b, <vscale x 8 x i16> %passthru) {
+; CHECK-LABEL: lsr_i16:
+; CHECK:      movprfx z2.h, p0/m, z0.h
+; CHECK-NEXT: lsr z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 8 x i1> %pg, <vscale x 8 x i16> %a, <vscale x 8 x i16> %passthru
+  %out = call <vscale x 8 x i16> @llvm.aarch64.sve.lsr.nxv8i16(<vscale x 8 x i1> %pg,
+                                                          <vscale x 8 x i16> %a_z,
+                                                          <vscale x 8 x i16> %b)
+  ret <vscale x 8 x i16> %out
+}
+
+define <vscale x 4 x i32> @lsr_i32(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %a,
+                                   <vscale x 4 x i32> %b, <vscale x 4 x i32> %passthru) {
+; CHECK-LABEL: lsr_i32:
+; CHECK:      movprfx z2.s, p0/m, z0.s
+; CHECK-NEXT: lsr z2.s, p0/m, z2.s, z1.s
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 4 x i1> %pg, <vscale x 4 x i32> %a, <vscale x 4 x i32> %passthru
+  %out = call <vscale x 4 x i32> @llvm.aarch64.sve.lsr.nxv4i32(<vscale x 4 x i1> %pg,
+                                                          <vscale x 4 x i32> %a_z,
+                                                          <vscale x 4 x i32> %b)
+  ret <vscale x 4 x i32> %out
+}
+
+define <vscale x 2 x i64> @lsr_i64(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %a,
+                                   <vscale x 2 x i64> %b, <vscale x 2 x i64> %passthru) {
+; CHECK-LABEL: lsr_i64:
+; CHECK:      movprfx z2.d, p0/m, z0.d
+; CHECK-NEXT: lsr z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT: mov z0.d, z2.d
+; CHECK-NEXT: ret
+  %a_z = select <vscale x 2 x i1> %pg, <vscale x 2 x i64> %a, <vscale x 2 x i64> %passthru
+  %out = call <vscale x 2 x i64> @llvm.aarch64.sve.lsr.nxv2i64(<vscale x 2 x i1> %pg,
+                                                          <vscale x 2 x i64> %a_z,
+                                                          <vscale x 2 x i64> %b)
+  ret <vscale x 2 x i64> %out
+}
+
 declare <vscale x 16 x i8> @llvm.aarch64.sve.asr.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x i8>, <vscale x 16 x i8>)
 declare <vscale x 8 x i16> @llvm.aarch64.sve.asr.nxv8i16(<vscale x 8 x i1>, <vscale x 8 x i16>, <vscale x 8 x i16>)
 declare <vscale x 4 x i32> @llvm.aarch64.sve.asr.nxv4i32(<vscale x 4 x i1>, <vscale x 4 x i32>, <vscale x 4 x i32>)
 declare <vscale x 2 x i64> @llvm.aarch64.sve.asr.nxv2i64(<vscale x 2 x i1>, <vscale x 2 x i64>, <vscale x 2 x i64>)
 
 declare <vscale x 16 x i8> @llvm.aarch64.sve.asr.wide.nxv16i8(<vscale x 16 x i1>, <vscale x 16 x i8>, <vscale x 2 x i64>)
 declare <vscale x 8 x i16> @llvm.aarch64.sve.asr.wide.nxv8i16(<vscale x 8 x i1>, <vscale x 8 x i16>, <vscale x 2 x i64>)
 declare <vscale x 4 x i32> @llvm.aarch64.sve.asr.wide.nxv4i32(<vscale x 4 x i1>, <vscale x 4 x i32>, <vscale x 2 x i64>)