[AArch64][Global ISel] Add sext/zext improvements

[dmgreen]

Summary

This patch adds improvements for sext/zext in Global ISel.

For example, this piece of code:

define i64 @si64(<4 x i32> %0, i32 %1) {
  %3 = extractelement <4 x i32> %0, i64 1
  %s = sext i32 %3 to i64
  ret i64 %s
}

Used to have this lowering:

si64:                            
	mov	s0, v0.s[1]
	fmov	w8, s0
	sxtw	x0, w8
	ret

Whereas this patch makes it lower to this:

si64:                             
	smov	x0, v0.h[0]
	ret

[paquette]

llvm/include/llvm/CodeGen/GlobalISel/MIPatternMatch.h

296

The boolean parameter on the end is true if this is commutative. I don't think G_EXTRACT_VECTOR_ELT is? Should this be false?

297

Can we use a name that makes this clear that this is matching G_EXTRACT_VECTOR_ELT? I can imagine people reading code using this thinking that it's matching some strange type of vector extend.

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

2148

Why did this code move?

llvm/test/CodeGen/AArch64/GlobalISel/arm64-atomic.ll

324 ↗

(On Diff #366643)

I don't think this test change is necessary.

[Rin]

llvm/include/llvm/CodeGen/GlobalISel/MIPatternMatch.h

296

Hmm, that's a good point. I need to look into that.

297

Yeah, sure thing, I'll rename it

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

2148

For the i64 case for G_SEXT we saw that it was choosing the tablegen patterns instead of the switch(Opcode). To avoid that, this code was moved to earlySelect.

llvm/test/CodeGen/AArch64/GlobalISel/arm64-atomic.ll

324 ↗

(On Diff #366643)

I think the test case was failing without the charge, but I'll switch it back and see if it works.

[dmgreen]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

2148

Do we need to move all of this, or can we get away with just doing the i64 in earlySelect?

[Rin]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

2148

I thought of that, but it would be a lot of code that gets repeated. But I can to do that instead

[Rin]

Address review comments

[Rin]

llvm/test/CodeGen/AArch64/GlobalISel/arm64-atomic.ll

324 ↗

(On Diff #366643)

Nevermind, it seems this was only a change due to clang-format. I will get rid of it

[dmgreen]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

3220

I think this needs to be matching any Integer, not the specific index of 1

3231–3236

And then this needs to use that index found above, not 0.

llvm/test/CodeGen/AArch64/GlobalISel/legalise-sext-zext.mir

3

This should maybe be running -run-pass=instruction-select, not legalizer?

[Rin]

Fix tests and address revie comments.

[Rin]

Remove changes from atomic test.

[Rin]

Remove changes from atomic test

[paquette]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

3160

It looks like the code here and the code in the earlySelect case for G_SEXT is very similar.

Would it be possible to factor some of it out into functions?

I could imagine an emit function which represents this patter, which takes an IsSigned parameter. Then you could choose e.g. UMOV or SMOV based off of that.

llvm/test/CodeGen/AArch64/GlobalISel/legalise-sext-zext.mir

1

Can we rename this test to something like "select-extract-vector-elt-with-extend" or something? It's not really a legalisation test.

[Rin]

Create function to avoid repeating code. Rename test.

This is my last day of placement with Arm so I'm handing this patch over.

[dmgreen]

Yep. Now that @Rin is off to do more important things, I'm happy to take over the patch and get it finished off.

I think some of the opcodes still need updating, which I will look into soon.

[dmgreen]

Update with Opcodes and extended types. I split off the code into a selectUSMovFromExtend function to encapsulate it a bit more in the process.

[dmgreen]

Update a missing test, and add an v8s8 equivalent.

[aemerson]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

4913

Why are we trying to match a copy? These should have been combined away. If there are cases where they aren't, then we should probably manually match the pattern using a helper like getOpcodeDef() which can look through copies.

[dmgreen]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

4913

My understanding is that the COPY is a FPR -> GPR COPY, so is still needed. We can change it to getOpcodeDef, but I preferred the m_GExtractVectorElt version :)

[dmgreen]

Use getOpcodeDef

[aemerson]

LGTM with nit.

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

4940

unfinished comment?

llvm/include/llvm/CodeGen/GlobalISel/MIPatternMatch.h

 
 template <typename LHS, typename RHS>
 inline BinaryOpc_match<LHS, RHS, true>
 m_CommutativeBinOp(unsigned Opcode, const LHS &L, const RHS &R) {
   return BinaryOpc_match<LHS, RHS, true>(Opcode, L, R);
 }
 
 template <typename LHS, typename RHS>
+inline BinaryOp_match<LHS, RHS, TargetOpcode::G_EXTRACT_VECTOR_ELT, false>
+m_GExtVecElt(const LHS &L, const RHS &R) {
+  return BinaryOp_match<LHS, RHS, TargetOpcode::G_EXTRACT_VECTOR_ELT, false>(L,
+                                                                             R);
+}
+
+template <typename LHS, typename RHS>
 inline BinaryOp_match<LHS, RHS, TargetOpcode::G_ADD, true>
 m_GAdd(const LHS &L, const RHS &R) {
   return BinaryOp_match<LHS, RHS, TargetOpcode::G_ADD, true>(L, R);
 }
 
 template <typename LHS, typename RHS>
 inline BinaryOp_match<LHS, RHS, TargetOpcode::G_PTR_ADD, true>
 m_GPtrAdd(const LHS &L, const RHS &R) {

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

auto *CV = ConstantDataVector::getSplat(
         MRI.getType(Dst).getNumElements(),
         ConstantInt::get(Type::getIntNTy(Ctx, MRI.getType(Src).getSizeInBits()),
                          ValAndVReg->Value));
     if (!emitConstantVector(Dst, CV, MIB, MRI))
       return false;
     I.eraseFromParent();
     return true;
   }
+
+  case TargetOpcode::G_SEXT: {
+    unsigned Opcode = I.getOpcode();
+    const Register DefReg = I.getOperand(0).getReg();
+    Register SrcReg = I.getOperand(1).getReg();
+    const LLT DstTy = MRI.getType(DefReg);
+    const LLT SrcTy = MRI.getType(SrcReg);
+    unsigned DstSize = DstTy.getSizeInBits();
+    unsigned SrcSize = SrcTy.getSizeInBits();
+
+    if (DstTy.isVector())
+      return false; // Should be handled by imported patterns.
+
+    assert((*RBI.getRegBank(DefReg, MRI, TRI)).getID() ==
+               AArch64::GPRRegBankID &&
+           "Unexpected ext regbank");
+
+    MachineInstr *ExtI;
+
+    if (DstSize == 64) {
+      // FIXME: Can we avoid manually doing this?
+      if (!RBI.constrainGenericRegister(SrcReg, AArch64::GPR32RegClass, MRI)) {
+        LLVM_DEBUG(dbgs() << "Failed to constrain " << TII.getName(Opcode)
+                          << " operand\n");
+        return false;
+      }
+      SrcReg =
+          MIB.buildInstr(AArch64::SUBREG_TO_REG, {&AArch64::GPR64RegClass}, {})
+              .addImm(0)
+              .addUse(SrcReg)
+              .addImm(AArch64::sub_32)
+              .getReg(0);
+      Register Src0;
+      if (mi_match(I.getOperand(1).getReg(), MRI,
+                   m_Copy(m_GExtVecElt(m_Reg(Src0), m_SpecificICst(1))))) {
+        const LLT &VecTy = MRI.getType(Src0);
+
+        if (VecTy.getSizeInBits() != 128) {
+          MachineInstr *ScalarToVector = emitScalarToVector(
+              VecTy.getSizeInBits(), &AArch64::FPR128RegClass, Src0, MIB);
+          if (!ScalarToVector)
+            return false;
+          Src0 = ScalarToVector->getOperand(0).getReg();
+        }
+        ExtI =
+            MIB.buildInstr(AArch64::SMOVvi16to64, {DefReg}, {Src0}).addImm(0);
+      } else
+        ExtI = MIB.buildInstr(AArch64::SBFMXri, {DefReg}, {SrcReg})
+                   .addImm(0)
+                   .addImm(SrcSize - 1);
+    } else {
+      return false;
+    }
+
+    constrainSelectedInstRegOperands(*ExtI, TII, TRI, RBI);
+    I.eraseFromParent();
+    return true;
+  }
+
   case TargetOpcode::G_BR:
     return false;
   case TargetOpcode::G_SHL:
     return earlySelectSHL(I, MRI);
   case TargetOpcode::G_CONSTANT: {
     bool IsZero = false;
     if (I.getOperand(1).isCImm())
       IsZero = I.getOperand(1).getCImm()->getZExtValue() == 0;
if (!IsSigned) {
         if (!Def || !isDef32(*Def)) {
           // No. Zero out using an OR.
           Register OrDst = MRI.createVirtualRegister(&AArch64::GPR32RegClass);
           const Register ZReg = AArch64::WZR;
           MIB.buildInstr(AArch64::ORRWrs, {OrDst}, {ZReg, SrcReg}).addImm(0);
           SubregToRegSrc = OrDst;
         }
 
+        Register Src0;
+        if (mi_match(I.getOperand(1).getReg(), MRI,
+                     m_Copy(m_GExtVecElt(m_Reg(Src0), m_SpecificICst(1))))) {
+          const LLT &VecTy = MRI.getType(Src0);
+
+          if (VecTy.getSizeInBits() != 128) {
+            MachineInstr *ScalarToVector = emitScalarToVector(
+                VecTy.getSizeInBits(), &AArch64::FPR128RegClass, Src0, MIB);
+            if (!ScalarToVector)
+              return false;
+            Src0 = ScalarToVector->getOperand(0).getReg();
+          }
+
+          Register New = MRI.createVirtualRegister(&AArch64::GPR32RegClass);
+          ExtI = MIB.buildInstr(AArch64::UMOVvi16, {New}, {Src0}).addImm(0);
+          MIB.buildInstr(AArch64::SUBREG_TO_REG, {DefReg}, {})
+              .addImm(0)
+              .addUse(New)
+              .addImm(AArch64::sub_32);
+        } else
           MIB.buildInstr(AArch64::SUBREG_TO_REG, {DefReg}, {})
               .addImm(0)
               .addUse(SubregToRegSrc)
               .addImm(AArch64::sub_32);
 
         if (!RBI.constrainGenericRegister(DefReg, AArch64::GPR64RegClass,
                                           MRI)) {
           LLVM_DEBUG(dbgs() << "Failed to constrain G_ZEXT destination\n");
           return false;
         }
 
         if (!RBI.constrainGenericRegister(SrcReg, AArch64::GPR32RegClass,
if (DstSize == 64) {
       if (Opcode != TargetOpcode::G_SEXT_INREG) {
         // FIXME: Can we avoid manually doing this?
         if (!RBI.constrainGenericRegister(SrcReg, AArch64::GPR32RegClass,
                                           MRI)) {
           LLVM_DEBUG(dbgs() << "Failed to constrain " << TII.getName(Opcode)
                             << " operand\n");
           return false;
         }
+
         SrcReg = MIB.buildInstr(AArch64::SUBREG_TO_REG,
                                 {&AArch64::GPR64RegClass}, {})
                      .addImm(0)
                      .addUse(SrcReg)
                      .addImm(AArch64::sub_32)
                      .getReg(0);
       }
+      Register Src0;
+      if (mi_match(I.getOperand(1).getReg(), MRI,
+                   m_Copy(m_GExtVecElt(m_Reg(Src0), m_SpecificICst(1))))) {
+        const LLT &VecTy = MRI.getType(Src0);
+
+        if (VecTy.getSizeInBits() != 128) {
+          MachineInstr *ScalarToVector = emitScalarToVector(
+              VecTy.getSizeInBits(), &AArch64::FPR128RegClass, Src0, MIB);
+          if (!ScalarToVector)
+            return false;
+          Src0 = ScalarToVector->getOperand(0).getReg();
+        }
 
+        ExtI = MIB.buildInstr(AArch64::UMOVvi16, {DefReg}, {Src0}).addImm(0);
+      } else
         ExtI = MIB.buildInstr(IsSigned ? AArch64::SBFMXri : AArch64::UBFMXri,
                               {DefReg}, {SrcReg})
                    .addImm(0)
                    .addImm(SrcSize - 1);
     } else if (DstSize <= 32) {
+      Register Src0;
+      int64_t Lane;
+      if (mi_match(I.getOperand(1).getReg(), MRI,
+                   m_Copy(m_GExtVecElt(m_Reg(Src0), m_ICst(Lane))))) {
+        const LLT &VecTy = MRI.getType(Src0);
+
+        if (VecTy.getSizeInBits() != 128) {
+          MachineInstr *ScalarToVector = emitScalarToVector(
+              VecTy.getSizeInBits(), &AArch64::FPR128RegClass, Src0, MIB);
+          if (!ScalarToVector)
+            return false;
+          Src0 = ScalarToVector->getOperand(0).getReg();
+        }
+
+        ExtI =
+            MIB.buildInstr(IsSigned ? AArch64::SMOVvi16to32 : AArch64::UMOVvi16,
+                           {DefReg}, {Src0})
+                .addImm(Lane);
+      } else
         ExtI = MIB.buildInstr(IsSigned ? AArch64::SBFMWri : AArch64::UBFMWri,
                               {DefReg}, {SrcReg})
                    .addImm(0)
                    .addImm(SrcSize - 1);
     } else {
       return false;
     }
 
     constrainSelectedInstRegOperands(*ExtI, TII, TRI, RBI);
     I.eraseFromParent();
     return true;
   }
bool AArch64InstructionSelector::selectInsertElt(MachineInstr &I,
   const LLT DstTy = MRI.getType(DstReg);
   unsigned VecSize = DstTy.getSizeInBits();
 
   // Get information on the element we want to insert into the destination.
   Register EltReg = I.getOperand(2).getReg();
   const LLT EltTy = MRI.getType(EltReg);
   unsigned EltSize = EltTy.getSizeInBits();
   if (EltSize < 16 || EltSize > 64)
     return false; // Don't support all element types yet.
 
   // Find the definition of the index. Bail out if it's not defined by a
   // G_CONSTANT.
   Register IdxReg = I.getOperand(3).getReg();
   auto VRegAndVal = getConstantVRegValWithLookThrough(IdxReg, MRI);
   if (!VRegAndVal)
     return false;
   unsigned LaneIdx = VRegAndVal->Value.getSExtValue();
if (VecSize < 128) {
     if (!ScalarToVec)
       return false;
     SrcReg = ScalarToVec->getOperand(0).getReg();
   }
 
   // Create an insert into a new FPR128 register.
   // Note that if our vector is already 128 bits, we end up emitting an extra
   // register.
   MachineInstr *InsMI =
       emitLaneInsert(None, SrcReg, EltReg, LaneIdx, EltRB, MIB);
 
   if (VecSize < 128) {
     // If we had to widen to perform the insert, then we have to demote back to
     // the original size to get the result we want.
     Register DemoteVec = InsMI->getOperand(0).getReg();
     const TargetRegisterClass *RC =
         getMinClassForRegBank(*RBI.getRegBank(DemoteVec, MRI, TRI), VecSize);

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp

getActionDefinitionsBuilder(G_EXTRACT_VECTOR_ELT)
       .unsupportedIf([=](const LegalityQuery &Query) {
         const LLT &EltTy = Query.Types[1].getElementType();
         return Query.Types[0] != EltTy;
       })
       .minScalar(2, s64)
       .legalIf([=](const LegalityQuery &Query) {
         const LLT &VecTy = Query.Types[1];
         return VecTy == v2s16 || VecTy == v4s16 || VecTy == v8s16 ||
-               VecTy == v4s32 || VecTy == v2s64 || VecTy == v2s32 ||
-               VecTy == v16s8 || VecTy == v2s32 || VecTy == v2p0;
+               VecTy == v8s8 || VecTy == v4s32 || VecTy == v2s64 ||
+               VecTy == v2s32 || VecTy == v16s8 || VecTy == v2s32 ||
+               VecTy == v2p0;
       })
       .minScalarOrEltIf(
           [=](const LegalityQuery &Query) {
             // We want to promote to <M x s1> to <M x s64> if that wouldn't
             // cause the total vec size to be > 128b.
             return Query.Types[1].getNumElements() <= 2;
           },
           0, s64)

llvm/test/CodeGen/AArch64/GlobalISel/legalise-sext-zext.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=aarch64-- -run-pass=instruction-select -verify-machineinstrs %s -o - | FileCheck %s
+
+---
+name:            si64
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $q0, $w0
+
+    ; CHECK-LABEL: name: si64
+    ; CHECK: liveins: $q0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr128 = COPY $q0
+    ; CHECK: [[CPYi32_:%[0-9]+]]:fpr32 = CPYi32 [[COPY]], 1
+    ; CHECK: [[COPY1:%[0-9]+]]:gpr32 = COPY [[CPYi32_]]
+    ; CHECK: [[SUBREG_TO_REG:%[0-9]+]]:gpr64 = SUBREG_TO_REG 0, [[COPY1]], %subreg.sub_32
+    ; CHECK: [[SMOVvi16to64_:%[0-9]+]]:gpr64 = SMOVvi16to64 [[COPY]], 0
+    ; CHECK: $x0 = COPY [[SMOVvi16to64_]]
+    ; CHECK: RET_ReallyLR implicit $x0
+    %0:fpr(<4 x s32>) = COPY $q0
+    %3:gpr(s64) = G_CONSTANT i64 1
+    %2:fpr(s32) = G_EXTRACT_VECTOR_ELT %0(<4 x s32>), %3(s64)
+    %5:gpr(s32) = COPY %2(s32)
+    %4:gpr(s64) = G_SEXT %5(s32)
+    $x0 = COPY %4(s64)
+    RET_ReallyLR implicit $x0
+
+...
+---
+name:            si64_2
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $d0, $w0
+
+    ; CHECK-LABEL: name: si64_2
+    ; CHECK: liveins: $d0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr64 = COPY $d0
+    ; CHECK: [[DEF:%[0-9]+]]:fpr128 = IMPLICIT_DEF
+    ; CHECK: [[INSERT_SUBREG:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF]], [[COPY]], %subreg.dsub
+    ; CHECK: [[CPYi32_:%[0-9]+]]:fpr32 = CPYi32 [[INSERT_SUBREG]], 1
+    ; CHECK: [[COPY1:%[0-9]+]]:gpr32 = COPY [[CPYi32_]]
+    ; CHECK: [[SUBREG_TO_REG:%[0-9]+]]:gpr64 = SUBREG_TO_REG 0, [[COPY1]], %subreg.sub_32
+    ; CHECK: [[DEF1:%[0-9]+]]:fpr128 = IMPLICIT_DEF
+    ; CHECK: [[INSERT_SUBREG1:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF1]], [[COPY]], %subreg.dsub
+    ; CHECK: [[SMOVvi16to64_:%[0-9]+]]:gpr64 = SMOVvi16to64 [[INSERT_SUBREG1]], 0
+    ; CHECK: $x0 = COPY [[SMOVvi16to64_]]
+    ; CHECK: RET_ReallyLR implicit $x0
+    %0:fpr(<2 x s32>) = COPY $d0
+    %3:gpr(s64) = G_CONSTANT i64 1
+    %2:fpr(s32) = G_EXTRACT_VECTOR_ELT %0(<2 x s32>), %3(s64)
+    %5:gpr(s32) = COPY %2(s32)
+    %4:gpr(s64) = G_SEXT %5(s32)
+    $x0 = COPY %4(s64)
+    RET_ReallyLR implicit $x0
+
+...
+---
+name:            zi64
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $q0, $w0
+
+    ; CHECK-LABEL: name: zi64
+    ; CHECK: liveins: $q0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr128 = COPY $q0
+    ; CHECK: [[CPYi32_:%[0-9]+]]:fpr32 = CPYi32 [[COPY]], 1
+    ; CHECK: [[COPY1:%[0-9]+]]:gpr32 = COPY [[CPYi32_]]
+    ; CHECK: [[ORRWrs:%[0-9]+]]:gpr32 = ORRWrs $wzr, [[COPY1]], 0
+    ; CHECK: [[UMOVvi16_:%[0-9]+]]:gpr32 = UMOVvi16 [[COPY]], 0
+    ; CHECK: [[SUBREG_TO_REG:%[0-9]+]]:gpr64 = SUBREG_TO_REG 0, [[UMOVvi16_]], %subreg.sub_32
+    ; CHECK: $x0 = COPY [[SUBREG_TO_REG]]
+    ; CHECK: RET_ReallyLR implicit $x0
+    %0:fpr(<4 x s32>) = COPY $q0
+    %3:gpr(s64) = G_CONSTANT i64 1
+    %2:fpr(s32) = G_EXTRACT_VECTOR_ELT %0(<4 x s32>), %3(s64)
+    %5:gpr(s32) = COPY %2(s32)
+    %4:gpr(s64) = G_ZEXT %5(s32)
+    $x0 = COPY %4(s64)
+    RET_ReallyLR implicit $x0
+
+...
+---
+name:            zi64_2
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $d0, $w0
+
+    ; CHECK-LABEL: name: zi64_2
+    ; CHECK: liveins: $d0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr64 = COPY $d0
+    ; CHECK: [[DEF:%[0-9]+]]:fpr128 = IMPLICIT_DEF
+    ; CHECK: [[INSERT_SUBREG:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF]], [[COPY]], %subreg.dsub
+    ; CHECK: [[CPYi32_:%[0-9]+]]:fpr32 = CPYi32 [[INSERT_SUBREG]], 1
+    ; CHECK: [[COPY1:%[0-9]+]]:gpr32 = COPY [[CPYi32_]]
+    ; CHECK: [[ORRWrs:%[0-9]+]]:gpr32 = ORRWrs $wzr, [[COPY1]], 0
+    ; CHECK: [[DEF1:%[0-9]+]]:fpr128 = IMPLICIT_DEF
+    ; CHECK: [[INSERT_SUBREG1:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF1]], [[COPY]], %subreg.dsub
+    ; CHECK: [[UMOVvi16_:%[0-9]+]]:gpr32 = UMOVvi16 [[INSERT_SUBREG1]], 0
+    ; CHECK: [[SUBREG_TO_REG:%[0-9]+]]:gpr64 = SUBREG_TO_REG 0, [[UMOVvi16_]], %subreg.sub_32
+    ; CHECK: $x0 = COPY [[SUBREG_TO_REG]]
+    ; CHECK: RET_ReallyLR implicit $x0
+     %0:fpr(<2 x s32>) = COPY $d0
+    %3:gpr(s64) = G_CONSTANT i64 1
+    %2:fpr(s32) = G_EXTRACT_VECTOR_ELT %0(<2 x s32>), %3(s64)
+    %5:gpr(s32) = COPY %2(s32)
+    %4:gpr(s64) = G_ZEXT %5(s32)
+    $x0 = COPY %4(s64)
+    RET_ReallyLR implicit $x0
+
+...
+---
+name:            si32
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $q0, $w0
+
+    ; CHECK-LABEL: name: si32
+    ; CHECK: liveins: $q0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr128 = COPY $q0
+    ; CHECK: [[SMOVvi16to32_:%[0-9]+]]:gpr32 = SMOVvi16to32 [[COPY]], 1
+    ; CHECK: $w0 = COPY [[SMOVvi16to32_]]
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:fpr(<8 x s16>) = COPY $q0
+    %4:gpr(s64) = G_CONSTANT i64 1
+    %3:fpr(s16) = G_EXTRACT_VECTOR_ELT %0(<8 x s16>), %4(s64)
+    %6:gpr(s16) = COPY %3(s16)
+    %5:gpr(s32) = G_SEXT %6(s16)
+    $w0 = COPY %5(s32)
+    RET_ReallyLR implicit $w0
+
+...
+---
+name:            zi32
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $q0, $w0
+
+    ; CHECK-LABEL: name: zi32
+    ; CHECK: liveins: $q0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr128 = COPY $q0
+    ; CHECK: [[UMOVvi16_:%[0-9]+]]:gpr32 = UMOVvi16 [[COPY]], 1
+    ; CHECK: $w0 = COPY [[UMOVvi16_]]
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:fpr(<8 x s16>) = COPY $q0
+    %4:gpr(s64) = G_CONSTANT i64 1
+    %3:fpr(s16) = G_EXTRACT_VECTOR_ELT %0(<8 x s16>), %4(s64)
+    %6:gpr(s16) = COPY %3(s16)
+    %5:gpr(s32) = G_ZEXT %6(s16)
+    $w0 = COPY %5(s32)
+    RET_ReallyLR implicit $w0
+
+...
+---
+name:            si32_2
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $d0, $w0
+
+    ; CHECK-LABEL: name: si32_2
+    ; CHECK: liveins: $d0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr64 = COPY $d0
+    ; CHECK: [[DEF:%[0-9]+]]:fpr128 = IMPLICIT_DEF
+    ; CHECK: [[INSERT_SUBREG:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF]], [[COPY]], %subreg.dsub
+    ; CHECK: [[SMOVvi16to32_:%[0-9]+]]:gpr32 = SMOVvi16to32 [[INSERT_SUBREG]], 1
+    ; CHECK: $w0 = COPY [[SMOVvi16to32_]]
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:fpr(<4 x s16>) = COPY $d0
+    %4:gpr(s64) = G_CONSTANT i64 1
+    %3:fpr(s16) = G_EXTRACT_VECTOR_ELT %0(<4 x s16>), %4(s64)
+    %6:gpr(s16) = COPY %3(s16)
+    %5:gpr(s32) = G_SEXT %6(s16)
+    $w0 = COPY %5(s32)
+    RET_ReallyLR implicit $w0
+
+...
+---
+name:            zi32_2
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $d0, $w0
+
+    ; CHECK-LABEL: name: zi32_2
+    ; CHECK: liveins: $d0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr64 = COPY $d0
+    ; CHECK: [[DEF:%[0-9]+]]:fpr128 = IMPLICIT_DEF
+    ; CHECK: [[INSERT_SUBREG:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF]], [[COPY]], %subreg.dsub
+    ; CHECK: [[UMOVvi16_:%[0-9]+]]:gpr32 = UMOVvi16 [[INSERT_SUBREG]], 1
+    ; CHECK: $w0 = COPY [[UMOVvi16_]]
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:fpr(<4 x s16>) = COPY $d0
+    %4:gpr(s64) = G_CONSTANT i64 1
+    %3:fpr(s16) = G_EXTRACT_VECTOR_ELT %0(<4 x s16>), %4(s64)
+    %6:gpr(s16) = COPY %3(s16)
+    %5:gpr(s32) = G_ZEXT %6(s16)
+    $w0 = COPY %5(s32)
+    RET_ReallyLR implicit $w0
+
+...
+---
+name:            si16
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $q0, $w0
+
+    ; CHECK-LABEL: name: si16
+    ; CHECK: liveins: $q0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr128 = COPY $q0
+    ; CHECK: [[SMOVvi16to32_:%[0-9]+]]:gpr32 = SMOVvi16to32 [[COPY]], 1
+    ; CHECK: $w0 = COPY [[SMOVvi16to32_]]
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:fpr(<16 x s8>) = COPY $q0
+    %4:gpr(s64) = G_CONSTANT i64 1
+    %3:fpr(s8) = G_EXTRACT_VECTOR_ELT %0(<16 x s8>), %4(s64)
+    %7:gpr(s8) = COPY %3(s8)
+    %6:gpr(s32) = G_SEXT %7(s8)
+    $w0 = COPY %6(s32)
+    RET_ReallyLR implicit $w0
+...
+---
+name:            zi16
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $q0, $w0
+
+    ; CHECK-LABEL: name: zi16
+    ; CHECK: liveins: $q0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr128 = COPY $q0
+    ; CHECK: [[UMOVvi16_:%[0-9]+]]:gpr32 = UMOVvi16 [[COPY]], 1
+    ; CHECK: $w0 = COPY [[UMOVvi16_]]
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:fpr(<16 x s8>) = COPY $q0
+    %4:gpr(s64) = G_CONSTANT i64 1
+    %3:fpr(s8) = G_EXTRACT_VECTOR_ELT %0(<16 x s8>), %4(s64)
+    %7:gpr(s8) = COPY %3(s8)
+    %6:gpr(s32) = G_ZEXT %7(s8)
+    $w0 = COPY %6(s32)
+    RET_ReallyLR implicit $w0
+
+...
+---
+name:            si16_2
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $d0, $w0
+
+    ; CHECK-LABEL: name: si16_2
+    ; CHECK: liveins: $d0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr64 = COPY $d0
+    ; CHECK: [[DEF:%[0-9]+]]:fpr128 = IMPLICIT_DEF
+    ; CHECK: [[INSERT_SUBREG:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF]], [[COPY]], %subreg.dsub
+    ; CHECK: [[SMOVvi16to32_:%[0-9]+]]:gpr32 = SMOVvi16to32 [[INSERT_SUBREG]], 1
+    ; CHECK: $w0 = COPY [[SMOVvi16to32_]]
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:fpr(<8 x s8>) = COPY $d0
+    %4:gpr(s64) = G_CONSTANT i64 1
+    %3:fpr(s8) = G_EXTRACT_VECTOR_ELT %0(<8 x s8>), %4(s64)
+    %7:gpr(s8) = COPY %3(s8)
+    %6:gpr(s32) = G_SEXT %7(s8)
+    $w0 = COPY %6(s32)
+    RET_ReallyLR implicit $w0
+...
+---
+name:            zi16_2
+legalized:       true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $d0, $w0
+
+    ; CHECK-LABEL: name: zi16_2
+    ; CHECK: liveins: $d0, $w0
+    ; CHECK: [[COPY:%[0-9]+]]:fpr64 = COPY $d0
+    ; CHECK: [[DEF:%[0-9]+]]:fpr128 = IMPLICIT_DEF
+    ; CHECK: [[INSERT_SUBREG:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF]], [[COPY]], %subreg.dsub
+    ; CHECK: [[UMOVvi16_:%[0-9]+]]:gpr32 = UMOVvi16 [[INSERT_SUBREG]], 1
+    ; CHECK: $w0 = COPY [[UMOVvi16_]]
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:fpr(<8 x s8>) = COPY $d0
+    %4:gpr(s64) = G_CONSTANT i64 1
+    %3:fpr(s8) = G_EXTRACT_VECTOR_ELT %0(<8 x s8>), %4(s64)
+    %7:gpr(s8) = COPY %3(s8)
+    %6:gpr(s32) = G_ZEXT %7(s8)
+    $w0 = COPY %6(s32)
+    RET_ReallyLR implicit $w0
+
+...

llvm/test/CodeGen/AArch64/GlobalISel/select-int-ext.mir

registers:
   - { id: 1, class: gpr }
 
 body:             |
   bb.0:
     liveins: $w0
 
     ; CHECK-LABEL: name: sext_s64_from_s32
     ; CHECK: [[COPY:%[0-9]+]]:gpr32 = COPY $w0
-    ; CHECK: [[DEF:%[0-9]+]]:gpr64all = IMPLICIT_DEF
-    ; CHECK: [[INSERT_SUBREG:%[0-9]+]]:gpr64 = INSERT_SUBREG [[DEF]], [[COPY]], %subreg.sub_32
-    ; CHECK: [[SBFMXri:%[0-9]+]]:gpr64 = SBFMXri [[INSERT_SUBREG]], 0, 31
+    ; CHECK: [[SUBREG_TO_REG:%[0-9]+]]:gpr64 = SUBREG_TO_REG 0, [[COPY]], %subreg.sub_32
+    ; CHECK: [[SBFMXri:%[0-9]+]]:gpr64 = SBFMXri [[SUBREG_TO_REG]], 0, 31
     ; CHECK: $x0 = COPY [[SBFMXri]]
     %0(s32) = COPY $w0
     %1(s64) = G_SEXT %0
     $x0 = COPY %1(s64)
 ...
 
 ---
 name:            sext_s32_from_s16

llvm/test/CodeGen/AArch64/sext-zext.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-eabi %s -o - | FileCheck %s --check-prefixes=CHECK-ISEL
+; RUN: llc -mtriple=aarch64-eabi -global-isel %s -o - | FileCheck %s --check-prefixes=CHECK-GLOBAL
+
+define i64 @si64(<4 x i32> %0, i32 %1) {
+; CHECK-ISEL-LABEL: si64:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    smov x0, v0.s[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: si64:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    smov x0, v0.h[0]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <4 x i32> %0, i64 1
+  %s = sext i32 %3 to i64
+  ret i64 %s
+}
+
+define i64 @zi64(<4 x i32> %0, i32 %1) {
+; CHECK-ISEL-LABEL: zi64:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    mov w0, v0.s[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: zi64:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    umov w0, v0.h[0]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <4 x i32> %0, i64 1
+  %s = zext i32 %3 to i64
+  ret i64 %s
+}
+
+define i64 @si64_2(<2 x i32> %0, i32 %1) {
+; CHECK-ISEL-LABEL: si64_2:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-ISEL-NEXT:    smov x0, v0.s[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: si64_2:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-GLOBAL-NEXT:    smov x0, v0.h[0]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <2 x i32> %0, i64 1
+  %s = sext i32 %3 to i64
+  ret i64 %s
+}
+
+define i64 @zi64_2(<2 x i32> %0, i32 %1) {
+; CHECK-ISEL-LABEL: zi64_2:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-ISEL-NEXT:    mov w0, v0.s[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: zi64_2:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-GLOBAL-NEXT:    umov w0, v0.h[0]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <2 x i32> %0, i64 1
+  %z = zext i32 %3 to i64
+  ret i64 %z
+}
+
+
+define i32 @si32(<8 x i16> %0, i16 %1) {
+; CHECK-ISEL-LABEL: si32:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    smov w0, v0.h[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: si32:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    smov w0, v0.h[1]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <8 x i16> %0, i32 1
+  %s = sext i16 %3 to i32
+  ret i32 %s
+}
+
+define i32 @si32_4(<4 x i16> %0, i16 %1) {
+; CHECK-ISEL-LABEL: si32_4:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-ISEL-NEXT:    smov w0, v0.h[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: si32_4:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-GLOBAL-NEXT:    smov w0, v0.h[1]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <4 x i16> %0, i32 1
+  %s = sext i16 %3 to i32
+  ret i32 %s
+}
+
+define i32 @zi32(<8 x i16> %0, i16 %1) {
+; CHECK-ISEL-LABEL: zi32:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    umov w0, v0.h[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: zi32:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    umov w0, v0.h[1]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <8 x i16> %0, i32 1
+  %s = zext i16 %3 to i32
+  ret i32 %s
+}
+
+define i32 @zi32_4(<4 x i16> %0, i16 %1) {
+; CHECK-ISEL-LABEL: zi32_4:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-ISEL-NEXT:    umov w0, v0.h[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: zi32_4:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-GLOBAL-NEXT:    umov w0, v0.h[1]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <4 x i16> %0, i32 1
+  %z = zext i16 %3 to i32
+  ret i32 %z
+}
+
+define i16 @si16(<16 x i8> %0, i8 %1) {
+; CHECK-ISEL-LABEL: si16:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    smov w0, v0.b[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: si16:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    smov w0, v0.h[1]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <16 x i8> %0, i16 1
+  %s = sext i8 %3 to i16
+  ret i16 %s
+}
+
+define i16 @zi16(<16 x i8> %0, i8 %1) {
+; CHECK-ISEL-LABEL: zi16:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    umov w0, v0.b[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: zi16:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    umov w0, v0.h[1]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <16 x i8> %0, i16 1
+  %z = zext i8 %3 to i16
+  ret i16 %z
+}
+
+define i16 @si16_8(<8 x i8> %0, i8 %1) {
+; CHECK-ISEL-LABEL: si16_8:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-ISEL-NEXT:    smov w0, v0.b[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: si16_8:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-GLOBAL-NEXT:    smov w0, v0.h[1]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <8 x i8> %0, i16 1
+  %s = sext i8 %3 to i16
+  ret i16 %s
+}
+
+define i16 @zi16_8(<8 x i8> %0, i8 %1) {
+; CHECK-ISEL-LABEL: zi16_8:
+; CHECK-ISEL:       // %bb.0:
+; CHECK-ISEL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-ISEL-NEXT:    umov w0, v0.b[1]
+; CHECK-ISEL-NEXT:    ret
+;
+; CHECK-GLOBAL-LABEL: zi16_8:
+; CHECK-GLOBAL:       // %bb.0:
+; CHECK-GLOBAL-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-GLOBAL-NEXT:    umov w0, v0.h[1]
+; CHECK-GLOBAL-NEXT:    ret
+  %3 = extractelement <8 x i8> %0, i16 1
+  %z = zext i8 %3 to i16
+  ret i16 %z
+}
+