[DAGCombine] Initialize the default operation action for SIGN_EXTEND_INREG for vector type as 'expand' instead of 'legal'

[steven.zhang]

Summary

For now, we didn't set the default operation action for SIGN_EXTEND_INREG for vector type, which is 0 by default, that is legal. However, most target didn't have native instructions to support this opcode. It should be set as expand by default, as what we did for ANY_EXTEND_VECTOR_INREG.

This become critical as some target(i.e. Mips) didn't set it as expand for vector type, which will have problems as they didn't have the native instruction for it.

This patch will set the default operation action for SIGN_EXTEND_INREG for vector type as expand. And grep all the targets that specify its match pattern for vector type and make it legal.

[jsji]

Why do you call this [NFC]? Also why you think this do NOT need any testcases?

[steven.zhang]

In D70000#1738927, @jsji wrote:

Why do you call this [NFC]? Also why you think this do NOT need any testcases?

This patch just change the default action operation from legal to expand, and mark it as legal for each target that supports it. It is nfc in essential. And I agree that, it might change the action from legal to expand for those target mis-set it before.

[steven.zhang]

And yes, I need add test as it indeed change the behavior.

[steven.zhang]

Add the tests.

[steven.zhang]

• llvm/test/CodeGen/ARM/signext-inreg.ll
  • Expose a bug. Because the default action is legal, we have to set it as expand inside addTypeForNEON. And it will override the action set for MVE. Change it to expand by default, and set it to legal for MVE.
• CodeGen/Hexagon/signext-inreg.ll
  • Without this patch, we fail to select the instruction for test3 with option -march=hexagon. This is because all the sext_inreg for the vector type is legal by default if the target didn't set it. And Hexagon is the target. It will generate the sext_inreg node that target didn't support, which hit the assertion in the instruction selection. And this is exactly the case that this patch want to avoid.

[dmgreen]

It seems that this thing has been around since 2010. How come we are changing the default now, as opposed to just fixing the backends that don't support this instruction?

If MIPs is the target that is failing, there should probably be a test to show it is now OK.

llvm/lib/Target/ARM/ARMISelLowering.cpp

368

Nice one.

llvm/test/CodeGen/ARM/signext-inreg.ll

2

There are already tests in Thumb2/mve-sext.ll which cover sexts. The target here is a little odd for mve (mixing A profile and M profile architectures).

[steven.zhang]

In D70000#1740485, @dmgreen wrote:

It seems that this thing has been around since 2010. How come we are changing the default now, as opposed to just fixing the backends that don't support this instruction?

If MIPs is the target that is failing, there should probably be a test to show it is now OK.

Yeah, it could be easy to fix those target that didn’t set as expand. But it is just hiding the problem. The key here is that, for sext_inreg for vector type, the default action should be expand instead of legal, just as what we did for sext_vector_inreg. See the test I gave on hexagon target(we fail to do the instr selection). It is easy to cook this test and expose the problem. I believe we might have more potential issues as more targets are added. The problem of hexagon is exactly the same as mips, as we are marking those op that didn’t have native instruction support as legal.

[steven.zhang]

llvm/test/CodeGen/ARM/signext-inreg.ll

2

I am not familiar with Arm target... From the implementation, it is valid case to show the issue. I will take a look at the case you mentioned. Thank you!

[dmgreen]

In D70000#1740595, @steven.zhang wrote:

In D70000#1740485, @dmgreen wrote:

It seems that this thing has been around since 2010. How come we are changing the default now, as opposed to just fixing the backends that don't support this instruction?

If MIPs is the target that is failing, there should probably be a test to show it is now OK.

Yeah, it could be easy to fix those target that didn’t set as expand. But it is just hiding the problem. The key here is that, for sext_inreg for vector type, the default action should be expand instead of legal, just as what we did for sext_vector_inreg. See the test I gave on hexagon target(we fail to do the instr selection). It is easy to cook this test and expose the problem. I believe we might have more potential issues as more targets are added. The problem of hexagon is exactly the same as mips, as we are marking those op that didn’t have native instruction support as legal.

Righteo. And your argument for changing the default at this late stage is for consistency with the other _inreg nodes? That sounds fair. The counter argument would be that it is also nice to keep consistency with what was already in llvm! Not make changes like this that change the default action for an operation unless we have good reason to do so.

Take something like FLOG10 for example. It defaults to legal and targets should be making it Expand if they do not have a relevant instruction, like most targets do. Something like FPOWI will default to Expand though, because it was added later and would break existing targets if it was not Expand. There doesn't seem to be a lot of consistency other than when the node was added.

I don't really have a strong opinion. If people think that changing the default is good for consistency with the other _inreg nodes, then that's fine.

llvm/test/CodeGen/ARM/signext-inreg.ll

2

The ARM backend has two completely separate vector architectures in it. There is Neon, the older of the two that works with Cortex-A style cpus (think the big-little cores that are in you phone). Recently we have also added the MVE vector extension that works with Cortex-M style cpus, which are smaller microcontrollers designed to go anywhere (like sensors and other low-power devices). Cortex-A can be A32 (arm) or T32 (thumb), Cortex-M can only be T32 (thumb)

So this test has a Cortex-A device (running in A32 mode) with MVE (an T32 extension to M-class cpus).

I'm not sure this test is adding a huge amount for MVE, but if you want to keep it I would suggest that part should be moved to with the other MVE tests. Like I said though, they will already be tested by the mve-sext.ll tests. Leaving Neon here sounds good.

[steven.zhang]

In D70000#1741155, @dmgreen wrote:

In D70000#1740595, @steven.zhang wrote:

In D70000#1740485, @dmgreen wrote:

It seems that this thing has been around since 2010. How come we are changing the default now, as opposed to just fixing the backends that don't support this instruction?

If MIPs is the target that is failing, there should probably be a test to show it is now OK.

Yeah, it could be easy to fix those target that didn’t set as expand. But it is just hiding the problem. The key here is that, for sext_inreg for vector type, the default action should be expand instead of legal, just as what we did for sext_vector_inreg. See the test I gave on hexagon target(we fail to do the instr selection). It is easy to cook this test and expose the problem. I believe we might have more potential issues as more targets are added. The problem of hexagon is exactly the same as mips, as we are marking those op that didn’t have native instruction support as legal.

Righteo. And your argument for changing the default at this late stage is for consistency with the other _inreg nodes? That sounds fair. The counter argument would be that it is also nice to keep consistency with what was already in llvm! Not make changes like this that change the default action for an operation unless we have good reason to do so.

Take something like FLOG10 for example. It defaults to legal and targets should be making it Expand if they do not have a relevant instruction, like most targets do. Something like FPOWI will default to Expand though, because it was added later and would break existing targets if it was not Expand. There doesn't seem to be a lot of consistency other than when the node was added.

I don't really have a strong opinion. If people think that changing the default is good for consistency with the other _inreg nodes, then that's fine.

Agree with you on that, keeping the consistency is NOT the reason. We need to figure out the correct way.

In llvm, we have 18 targets now. But only 2 targets support the sext_inreg for vector type! That is, ARM and Hexagon. Even within these two targets, it only supports the operation for some limited vector type. We might have:

1. sext_inreg for vector type is legal by default.
   • That requires all the targets to mark it as expand if they didn't have the native instruction support. However, most targets didn't do this. And bugs hide. i.e. Hexagon failed to select the instruction. And I believe it is easy to cook the test for most target to expose the issue. And as you mentioned, we could easily fix this by visit all the targets and set it as expand. If we did that, why cannot we just mark it as expand by default ?
2. sext_inreg for vector type is expand by default
   • What we need to do, is set it as legal for some vector type for ARM and Hexagon.

The option 2 makes more sense to me, as it is consistent with the semantics. The op is NOT legal for most platform currently. And that is also what we have done for sext_vector_inreg. My understanding is that, if this operation is NOT supported by most targets, we'd better set it as expand by default. Please correct me if I miss something. And I agree that, sometimes, we are not willing to change the default value which might break the other targets. As little target provides the match pattern, it is a good time to fix it after nearly ten years. What do you think ?

[steven.zhang]

llvm/test/CodeGen/ARM/signext-inreg.ll

2

Thank you for the explanation, dmgree:) I will take a deep look on this.

[steven.zhang]

llvm/test/CodeGen/ARM/signext-inreg.ll

2

So, we have ARM and Thumb2 and MVE is for Thumb2. Is it right ? I will add the test into mve-sext.ll. The testing here is trying to verify that, the Neon won't override the action operation setting for MVE.

[dmgreen]

llvm/test/CodeGen/ARM/signext-inreg.ll

2

Having Neon + MVE at the same time would be an invalid combination.

[steven.zhang]

llvm/test/CodeGen/ARM/signext-inreg.ll

2

ok，get it.Thank you for the clarify.

[steven.zhang]

Remove the NEON + MVE test, and add one test in mve-sext.ll, which will fail without this patch.

LLVM ERROR: Cannot select: t13: v4i32 = sign_extend_inreg t3, ValueType:ch:v4i1
  t3: v4i32 = bitcast t2
    t2: v2f64,ch = CopyFromReg t0, Register:v2f64 %0
      t1: v2f64 = Register %0
In function: sext_v4i32_v4i32

The reason is the same as Hexagon, Mips and other targets. That is, we are setting the default operation action for sext_inreg for vector type as legal.

[dmgreen]

OK. So it's not about consistency with the other INREGs, its to do with them rarely being used for vectors?

I think you reasoning makes a lot of sense. Plus anyone using these nodes downstream and relying on it being Legal will likely have tests for the sext_inreg patterns they added, so should easily catch any differences.

If the hexagon changes are OK here, I'm happy.

llvm/lib/Target/ARM/ARMISelLowering.cpp

183

You can leave this line in. It's not bad to be explicit about this being expand.

llvm/test/CodeGen/Thumb2/mve-sext.ll

4 ↗

(On Diff #229252)

Can you add equivalents for v8i16, v16i8 and maybe v2i64?

Or actually, I can do it later. I don't want to give you more work to do.

[steven.zhang]

llvm/lib/Target/ARM/ARMISelLowering.cpp

183

make sense

llvm/test/CodeGen/Thumb2/mve-sext.ll

4 ↗

(On Diff #229252)

I will add it.

[steven.zhang]

Address dmgreen's comments.

[steven.zhang]

@kparzysz Would you please help me review the Hexagon's part change ?

[kparzysz]

llvm/lib/Target/Hexagon/HexagonISelLowering.cpp

1540

The types should be v2i8, v4i8, and v2i16.

llvm/lib/Target/Hexagon/HexagonISelLoweringHVX.cpp

23

Please do not use globals for this.

64

Please remove this.

203

Please replace this with

if (Use64b) {
  for (MVT T : {MVT::v16i8, MVT::v32i8, MVT::v16i16})
    setOperationAction(ISD::SIGN_EXTEND_INREG, T, Legal);
} else {
  for (MVT T : {MVT::v32i8, MVT::v64i8, MVT::v32i16})
    setOperationAction(ISD::SIGN_EXTEND_INREG, T, Legal);
}

[steven.zhang]

Address comments.

llvm/lib/Target/Hexagon/HexagonISelLowering.cpp

1540

I just only see two match pattern for this ISD Node in the TD file, and no match pattern for v4i8. Did I miss something ?

def: Pat<(v2i32 (sext_inreg V2I32:$Rs, v2i8)),
         (Combinew (A2_sxtb (HiReg $Rs)), (A2_sxtb (LoReg $Rs)))>;

def: Pat<(v2i32 (sext_inreg V2I32:$Rs, v2i16)),
         (Combinew (A2_sxth (HiReg $Rs)), (A2_sxth (LoReg $Rs)))>;

llvm/lib/Target/Hexagon/HexagonISelLoweringHVX.cpp

203

We need to mark the dest type as Legal too, as Vector Op Legalizer will check the operand 0 instead of the Operand 1 is legal or not. That is important for some target to support:

v4i32 sext_inreg Val, v4i8
v4i16 sext_inreg Val, v4i8

[steven.zhang]

Gentle ping ...

[steven.zhang]

@kparzysz Any more comments ? Thank you.

[steven.zhang]

Ping ...

[steven.zhang]

Ping ...

[RKSimon]

LGTM

[steven.zhang]

Rebase the patch due to the AArch64 commit 65651f197a2c which try to legalize sext_inreg for vector type for SVE. We need to specify it explicitly instead of relying on the default value. I will hold on this patch for several days in case someone has comments on the change of "SVE" part.

[dmgreen]

The AArch64 changes look good to me. Thanks

llvm/lib/CodeGen/TargetLoweringBase.cpp

for (MVT VT : MVT::all_valuetypes()) {
 
     // These library functions default to expand.
     setOperationAction(ISD::FROUND, VT, Expand);
     setOperationAction(ISD::FPOWI, VT, Expand);
 
     // These operations default to expand for vector types.
     if (VT.isVector()) {
       setOperationAction(ISD::FCOPYSIGN, VT, Expand);
+      setOperationAction(ISD::SIGN_EXTEND_INREG, VT, Expand);
       setOperationAction(ISD::ANY_EXTEND_VECTOR_INREG, VT, Expand);
       setOperationAction(ISD::SIGN_EXTEND_VECTOR_INREG, VT, Expand);
       setOperationAction(ISD::ZERO_EXTEND_VECTOR_INREG, VT, Expand);
       setOperationAction(ISD::SPLAT_VECTOR, VT, Expand);
     }
 
     // Constrained floating-point operations default to expand.
     setOperationAction(ISD::STRICT_FADD, VT, Expand);

llvm/lib/Target/ARM/ARMISelLowering.cpp

void ARMTargetLowering::addTypeForNEON(MVT VT, MVT PromotedLdStVT,
   }
   setOperationAction(ISD::BUILD_VECTOR,      VT, Custom);
   setOperationAction(ISD::VECTOR_SHUFFLE,    VT, Custom);
   setOperationAction(ISD::CONCAT_VECTORS,    VT, Legal);
   setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Legal);
   setOperationAction(ISD::SELECT,            VT, Expand);
   setOperationAction(ISD::SELECT_CC,         VT, Expand);
   setOperationAction(ISD::VSELECT,           VT, Expand);
-  setOperationAction(ISD::SIGN_EXTEND_INREG, VT, Expand);
   if (VT.isInteger()) {
     setOperationAction(ISD::SHL, VT, Custom);
     setOperationAction(ISD::SRA, VT, Custom);
     setOperationAction(ISD::SRL, VT, Custom);
   }
 
   // Promote all bit-wise operations.
   if (VT.isInteger() && VT != PromotedBitwiseVT) {
void ARMTargetLowering::addMVEVectorTypes(bool HasMVEFP) {
   setOperationAction(ISD::OR, MVT::v2i64, Legal);
   setOperationAction(ISD::XOR, MVT::v2i64, Legal);
 
   // It is legal to extload from v4i8 to v4i16 or v4i32.
   addAllExtLoads(MVT::v8i16, MVT::v8i8, Legal);
   addAllExtLoads(MVT::v4i32, MVT::v4i16, Legal);
   addAllExtLoads(MVT::v4i32, MVT::v4i8, Legal);
 
+  // It is legal to sign extend from v4i8/v4i16 to v4i32 or v8i8 to v8i16.
+  setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::v4i8,  Legal);
+  setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::v4i16, Legal);
+  setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::v4i32, Legal);
+  setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::v8i8,  Legal);
+  setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::v8i16, Legal);
+
   // Some truncating stores are legal too.
   setTruncStoreAction(MVT::v4i32, MVT::v4i16, Legal);
   setTruncStoreAction(MVT::v4i32, MVT::v4i8,  Legal);
   setTruncStoreAction(MVT::v8i16, MVT::v8i8,  Legal);
 
   // Pre and Post inc on these are legal, given the correct extends
   for (unsigned im = (unsigned)ISD::PRE_INC;
        im != (unsigned)ISD::LAST_INDEXED_MODE; ++im) {

llvm/lib/Target/Hexagon/HexagonISelLowering.cpp

HexagonTargetLowering::HexagonTargetLowering(const TargetMachine &TM,
   // are legal.
   setLoadExtAction(ISD::EXTLOAD,  MVT::v2i16, MVT::v2i8, Legal);
   setLoadExtAction(ISD::ZEXTLOAD, MVT::v2i16, MVT::v2i8, Legal);
   setLoadExtAction(ISD::SEXTLOAD, MVT::v2i16, MVT::v2i8, Legal);
   setLoadExtAction(ISD::EXTLOAD,  MVT::v4i16, MVT::v4i8, Legal);
   setLoadExtAction(ISD::ZEXTLOAD, MVT::v4i16, MVT::v4i8, Legal);
   setLoadExtAction(ISD::SEXTLOAD, MVT::v4i16, MVT::v4i8, Legal);
 
+  setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::v2i8,  Legal);
+  setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::v2i16, Legal);
+  setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::v2i32, Legal);
+
   // Types natively supported:
   for (MVT NativeVT : {MVT::v8i1, MVT::v4i1, MVT::v2i1, MVT::v4i8,
                        MVT::v8i8, MVT::v2i16, MVT::v4i16, MVT::v2i32}) {
     setOperationAction(ISD::BUILD_VECTOR,       NativeVT, Custom);
     setOperationAction(ISD::EXTRACT_VECTOR_ELT, NativeVT, Custom);
     setOperationAction(ISD::INSERT_VECTOR_ELT,  NativeVT, Custom);
     setOperationAction(ISD::EXTRACT_SUBVECTOR,  NativeVT, Custom);
     setOperationAction(ISD::INSERT_SUBVECTOR,   NativeVT, Custom);

llvm/lib/Target/Hexagon/HexagonISelLoweringHVX.cpp

 #include "llvm/Support/CommandLine.h"
 
 using namespace llvm;
 
 static const MVT LegalV64[] =  { MVT::v64i8,  MVT::v32i16,  MVT::v16i32 };
 static const MVT LegalW64[] =  { MVT::v128i8, MVT::v64i16,  MVT::v32i32 };
 static const MVT LegalV128[] = { MVT::v128i8, MVT::v64i16,  MVT::v32i32 };
 static const MVT LegalW128[] = { MVT::v256i8, MVT::v128i16, MVT::v64i32 };
-
+static const MVT InRegV64[] = { MVT::v32i8, MVT::v32i16, MVT::v16i8,
+                                MVT::v16i16, MVT::v16i32 };
+static const MVT InRegV128[] = { MVT::v64i8, MVT::v64i16, MVT::v32i8,
+                                 MVT::v32i16, MVT::v32i32 };
 
 void
 HexagonTargetLowering::initializeHVXLowering() {
   if (Subtarget.useHVX64BOps()) {
     addRegisterClass(MVT::v64i8,  &Hexagon::HvxVRRegClass);
     addRegisterClass(MVT::v32i16, &Hexagon::HvxVRRegClass);
     addRegisterClass(MVT::v16i32, &Hexagon::HvxVRRegClass);
     addRegisterClass(MVT::v128i8, &Hexagon::HvxWRRegClass);
if (Subtarget.useHVX64BOps()) {
     addRegisterClass(MVT::v1024i1, &Hexagon::HvxQRRegClass);
   }
 
   // Set up operation actions.
 
   bool Use64b = Subtarget.useHVX64BOps();
   ArrayRef<MVT> LegalV = Use64b ? LegalV64 : LegalV128;
   ArrayRef<MVT> LegalW = Use64b ? LegalW64 : LegalW128;
+  ArrayRef<MVT> InRegV = Use64b ? InRegV64 : InRegV128;
   MVT ByteV = Use64b ?  MVT::v64i8 : MVT::v128i8;
   MVT ByteW = Use64b ? MVT::v128i8 : MVT::v256i8;
 
   auto setPromoteTo = [this] (unsigned Opc, MVT FromTy, MVT ToTy) {
     setOperationAction(Opc, FromTy, Promote);
     AddPromotedToType(Opc, FromTy, ToTy);
   };
 
for (MVT T : LegalV) {
     setOperationAction(ISD::INSERT_VECTOR_ELT,  BoolV, Custom);
     setOperationAction(ISD::EXTRACT_SUBVECTOR,  BoolV, Custom);
     setOperationAction(ISD::EXTRACT_VECTOR_ELT, BoolV, Custom);
     setOperationAction(ISD::AND,                BoolV, Legal);
     setOperationAction(ISD::OR,                 BoolV, Legal);
     setOperationAction(ISD::XOR,                BoolV, Legal);
   }
 
+  for (MVT T : InRegV)
+    setOperationAction(ISD::SIGN_EXTEND_INREG, T,  Legal);
+
   setTargetDAGCombine(ISD::VSELECT);
 }
 
 SDValue
 HexagonTargetLowering::getInt(unsigned IntId, MVT ResTy, ArrayRef<SDValue> Ops,
                               const SDLoc &dl, SelectionDAG &DAG) const {
   SmallVector<SDValue,4> IntOps;
   IntOps.push_back(DAG.getConstant(IntId, dl, MVT::i32));

llvm/test/CodeGen/ARM/signext-inreg.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=armv8 -mattr=+mve | FileCheck %s
 define <4 x i32> @test(<4 x i32> %m) {
 ; CHECK-LABEL: test:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmov d1, r2, r3
 ; CHECK-NEXT:    vmov d0, r0, r1
-; CHECK-NEXT:    vshl.i32 q0, q0, #24
-; CHECK-NEXT:    vshr.s32 q0, q0, #24
+; CHECK-NEXT:    vmovlb.s8 q0, q0
+; CHECK-NEXT:    vmovlb.s16 q0, q0
 ; CHECK-NEXT:    vmov r0, r1, d0
 ; CHECK-NEXT:    vmov r2, r3, d1
 ; CHECK-NEXT:    bx lr
 entry:
   %shl = shl <4 x i32> %m, <i32 24, i32 24, i32 24, i32 24>
   %shr = ashr exact <4 x i32> %shl, <i32 24, i32 24, i32 24, i32 24>
   ret <4 x i32> %shr
 }

llvm/test/CodeGen/Hexagon/signext-inreg.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -march=hexagon | FileCheck %s
+; RUN: llc < %s -march=hexagon -mattr=+hvx,hvx-length64b | FileCheck %s --check-prefix=CHECK-64B
+; RUN: llc < %s -march=hexagon -mattr=+hvx,hvx-length128b | FileCheck %s --check-prefix=CHECK-128B
+define <2 x i32> @test1(<2 x i32> %m) {
+; CHECK-LABEL: test1:
+; CHECK:         .cfi_startproc
+; CHECK-NEXT:  // %bb.0: // %entry
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r1 = extract(r1,#8,#0)
+; CHECK-NEXT:     r0 = sxtb(r0)
+; CHECK-NEXT:     jumpr r31
+; CHECK-NEXT:    }
+;
+; CHECK-64B-LABEL: test1:
+; CHECK-64B:         .cfi_startproc
+; CHECK-64B-NEXT:  // %bb.0: // %entry
+; CHECK-64B-NEXT:    {
+; CHECK-64B-NEXT:     r1 = extract(r1,#8,#0)
+; CHECK-64B-NEXT:     r0 = sxtb(r0)
+; CHECK-64B-NEXT:     jumpr r31
+; CHECK-64B-NEXT:    }
+;
+; CHECK-128B-LABEL: test1:
+; CHECK-128B:         .cfi_startproc
+; CHECK-128B-NEXT:  // %bb.0: // %entry
+; CHECK-128B-NEXT:    {
+; CHECK-128B-NEXT:     r1 = extract(r1,#8,#0)
+; CHECK-128B-NEXT:     r0 = sxtb(r0)
+; CHECK-128B-NEXT:     jumpr r31
+; CHECK-128B-NEXT:    }
+entry:
+  %shl = shl <2 x i32> %m, <i32 24, i32 24>
+  %shr = ashr exact <2 x i32> %shl, <i32 24, i32 24>
+  ret <2 x i32> %shr
+}
+
+define <16 x i32> @test2(<16 x i32> %m) {
+; CHECK-LABEL: test2:
+; CHECK:         .cfi_startproc
+; CHECK-NEXT:  // %bb.0: // %entry
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r3 = extract(r3,#8,#0)
+; CHECK-NEXT:     r29 = add(r29,#-8)
+; CHECK-NEXT:     r2 = sxtb(r2)
+; CHECK-NEXT:     r4 = sxtb(r4)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r5 = extract(r5,#8,#0)
+; CHECK-NEXT:     r13:12 = memd(r29+#48)
+; CHECK-NEXT:     memd(r29+#0) = r17:16
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r13 = extract(r13,#8,#0)
+; CHECK-NEXT:     r12 = sxtb(r12)
+; CHECK-NEXT:     r15:14 = memd(r29+#40)
+; CHECK-NEXT:     r9:8 = memd(r29+#32)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r9 = extract(r9,#8,#0)
+; CHECK-NEXT:     r8 = sxtb(r8)
+; CHECK-NEXT:     r11:10 = memd(r29+#24)
+; CHECK-NEXT:     r7:6 = memd(r29+#16)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r11 = extract(r11,#8,#0)
+; CHECK-NEXT:     r10 = sxtb(r10)
+; CHECK-NEXT:     r14 = sxtb(r14)
+; CHECK-NEXT:     r17:16 = memd(r29+#8)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r15 = extract(r15,#8,#0)
+; CHECK-NEXT:     r17 = extract(r17,#8,#0)
+; CHECK-NEXT:     r16 = sxtb(r16)
+; CHECK-NEXT:     r6 = sxtb(r6)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r7 = extract(r7,#8,#0)
+; CHECK-NEXT:     memd(r0+#56) = r13:12
+; CHECK-NEXT:     memd(r0+#48) = r15:14
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     memd(r0+#40) = r9:8
+; CHECK-NEXT:     memd(r0+#32) = r11:10
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     memd(r0+#24) = r7:6
+; CHECK-NEXT:     memd(r0+#16) = r17:16
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     memd(r0+#8) = r5:4
+; CHECK-NEXT:     memd(r0+#0) = r3:2
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r29 = add(r29,#8)
+; CHECK-NEXT:     r17:16 = memd(r29+#0)
+; CHECK-NEXT:     jumpr r31
+; CHECK-NEXT:    } // 8-byte Folded Reload
+;
+; CHECK-64B-LABEL: test2:
+; CHECK-64B:         .cfi_startproc
+; CHECK-64B-NEXT:  // %bb.0: // %entry
+; CHECK-64B-NEXT:    {
+; CHECK-64B-NEXT:     r0 = #24
+; CHECK-64B-NEXT:    }
+; CHECK-64B-NEXT:    {
+; CHECK-64B-NEXT:     v0.w = vasl(v0.w,r0)
+; CHECK-64B-NEXT:    }
+; CHECK-64B-NEXT:    {
+; CHECK-64B-NEXT:     v0.w = vasr(v0.w,r0)
+; CHECK-64B-NEXT:     jumpr r31
+; CHECK-64B-NEXT:    }
+;
+; CHECK-128B-LABEL: test2:
+; CHECK-128B:         .cfi_startproc
+; CHECK-128B-NEXT:  // %bb.0: // %entry
+; CHECK-128B-NEXT:    {
+; CHECK-128B-NEXT:     r0 = #24
+; CHECK-128B-NEXT:    }
+; CHECK-128B-NEXT:    {
+; CHECK-128B-NEXT:     v0.w = vasl(v0.w,r0)
+; CHECK-128B-NEXT:    }
+; CHECK-128B-NEXT:    {
+; CHECK-128B-NEXT:     v0.w = vasr(v0.w,r0)
+; CHECK-128B-NEXT:     jumpr r31
+; CHECK-128B-NEXT:    }
+entry:
+  %shl = shl <16 x i32> %m, <i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24>
+  %shr = ashr exact <16 x i32> %shl, <i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24, i32 24>
+  ret <16 x i32> %shr
+}
+
+define <64 x i16> @test3(<64 x i16> %m) {
+; CHECK-LABEL: test3:
+; CHECK:         .cfi_startproc
+; CHECK-NEXT:  // %bb.0: // %entry
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r3:2 = vaslh(r3:2,#8)
+; CHECK-NEXT:     r5:4 = vaslh(r5:4,#8)
+; CHECK-NEXT:     r9:8 = memd(r29+#96)
+; CHECK-NEXT:     r11:10 = memd(r29+#88)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r13:12 = vaslh(r9:8,#8)
+; CHECK-NEXT:     r11:10 = vaslh(r11:10,#8)
+; CHECK-NEXT:     r9:8 = memd(r29+#80)
+; CHECK-NEXT:     r7:6 = memd(r29+#104)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r15:14 = vaslh(r7:6,#8)
+; CHECK-NEXT:     r9:8 = vaslh(r9:8,#8)
+; CHECK-NEXT:     r7:6 = memd(r29+#72)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r15:14 = vasrh(r15:14,#8)
+; CHECK-NEXT:     r13:12 = vasrh(r13:12,#8)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r11:10 = vasrh(r11:10,#8)
+; CHECK-NEXT:     r9:8 = vasrh(r9:8,#8)
+; CHECK-NEXT:     r15:14 = memd(r29+#64)
+; CHECK-NEXT:     memd(r0+#120) = r15:14
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r7:6 = vaslh(r7:6,#8)
+; CHECK-NEXT:     r15:14 = vaslh(r15:14,#8)
+; CHECK-NEXT:     r13:12 = memd(r29+#56)
+; CHECK-NEXT:     memd(r0+#112) = r13:12
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r13:12 = vaslh(r13:12,#8)
+; CHECK-NEXT:     r7:6 = vasrh(r7:6,#8)
+; CHECK-NEXT:     r11:10 = memd(r29+#48)
+; CHECK-NEXT:     memd(r0+#104) = r11:10
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r11:10 = vaslh(r11:10,#8)
+; CHECK-NEXT:     r15:14 = vasrh(r15:14,#8)
+; CHECK-NEXT:     r9:8 = memd(r29+#40)
+; CHECK-NEXT:     memd(r0+#96) = r9:8
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r9:8 = vaslh(r9:8,#8)
+; CHECK-NEXT:     r13:12 = vasrh(r13:12,#8)
+; CHECK-NEXT:     r7:6 = memd(r29+#32)
+; CHECK-NEXT:     memd(r0+#88) = r7:6
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r11:10 = vasrh(r11:10,#8)
+; CHECK-NEXT:     r9:8 = vasrh(r9:8,#8)
+; CHECK-NEXT:     r15:14 = memd(r29+#0)
+; CHECK-NEXT:     memd(r0+#80) = r15:14
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r7:6 = vaslh(r7:6,#8)
+; CHECK-NEXT:     r15:14 = vaslh(r15:14,#8)
+; CHECK-NEXT:     r13:12 = memd(r29+#16)
+; CHECK-NEXT:     memd(r0+#72) = r13:12
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r13:12 = vaslh(r13:12,#8)
+; CHECK-NEXT:     r7:6 = vasrh(r7:6,#8)
+; CHECK-NEXT:     r11:10 = memd(r29+#24)
+; CHECK-NEXT:     memd(r0+#64) = r11:10
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r11:10 = vaslh(r11:10,#8)
+; CHECK-NEXT:     r3:2 = vasrh(r3:2,#8)
+; CHECK-NEXT:     r9:8 = memd(r29+#8)
+; CHECK-NEXT:     memd(r0+#56) = r9:8
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r9:8 = vaslh(r9:8,#8)
+; CHECK-NEXT:     r13:12 = vasrh(r13:12,#8)
+; CHECK-NEXT:     memd(r0+#48) = r7:6
+; CHECK-NEXT:     memd(r0+#0) = r3:2
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r11:10 = vasrh(r11:10,#8)
+; CHECK-NEXT:     r7:6 = vasrh(r15:14,#8)
+; CHECK-NEXT:     memd(r0+#32) = r13:12
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     r9:8 = vasrh(r9:8,#8)
+; CHECK-NEXT:     r5:4 = vasrh(r5:4,#8)
+; CHECK-NEXT:     memd(r0+#40) = r11:10
+; CHECK-NEXT:     memd(r0+#16) = r7:6
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     jumpr r31
+; CHECK-NEXT:     memd(r0+#24) = r9:8
+; CHECK-NEXT:     memd(r0+#8) = r5:4
+; CHECK-NEXT:    }
+;
+; CHECK-64B-LABEL: test3:
+; CHECK-64B:         .cfi_startproc
+; CHECK-64B-NEXT:  // %bb.0: // %entry
+; CHECK-64B-NEXT:    {
+; CHECK-64B-NEXT:     r0 = #8
+; CHECK-64B-NEXT:    }
+; CHECK-64B-NEXT:    {
+; CHECK-64B-NEXT:     v0.h = vasl(v0.h,r0)
+; CHECK-64B-NEXT:    }
+; CHECK-64B-NEXT:    {
+; CHECK-64B-NEXT:     v1.h = vasl(v1.h,r0)
+; CHECK-64B-NEXT:    }
+; CHECK-64B-NEXT:    {
+; CHECK-64B-NEXT:     v0.h = vasr(v0.h,r0)
+; CHECK-64B-NEXT:    }
+; CHECK-64B-NEXT:    {
+; CHECK-64B-NEXT:     v1.h = vasr(v1.h,r0)
+; CHECK-64B-NEXT:     jumpr r31
+; CHECK-64B-NEXT:    }
+;
+; CHECK-128B-LABEL: test3:
+; CHECK-128B:         .cfi_startproc
+; CHECK-128B-NEXT:  // %bb.0: // %entry
+; CHECK-128B-NEXT:    {
+; CHECK-128B-NEXT:     r0 = #8
+; CHECK-128B-NEXT:    }
+; CHECK-128B-NEXT:    {
+; CHECK-128B-NEXT:     v0.h = vasl(v0.h,r0)
+; CHECK-128B-NEXT:    }
+; CHECK-128B-NEXT:    {
+; CHECK-128B-NEXT:     v0.h = vasr(v0.h,r0)
+; CHECK-128B-NEXT:     jumpr r31
+; CHECK-128B-NEXT:    }
+entry:
+  %shl = shl <64 x i16> %m, <i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8>
+  %shr = ashr exact <64 x i16> %shl, <i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8>
+  ret <64 x i16> %shr
+}