[AArch64] Eliminating the use of integer unit in moving from a Neon scalar result of a uaddlv to a Neon vector

[nilanjana_basu]

Added all the variants of uaddlv, added 9 more unit tests & reduced the code for the pattern match.

[nilanjana_basu]

Added more patterns to match different types of vector insertion for the same set of 'uaddlv' functions

[nilanjana_basu]

The code generated for inserting the scalar result of a 'uaddlv' intrinsic function to a vector exhibits a redundant move to the integer register first, & then moves it back to the destination vector Neon register. This can be done directly. In my understanding, this issue occurs because the selected instruction for vector insertion expects an i32/i64 integer register to hold the value and the selected instruction for the 'uaddlv' function is giving an output of i32/i64 at the instruction selection time, whereas in practice the result of 'uaddlv' goes to a Neon register. Therefore I added patterns to match the combined use of these two instructions (uaddlv & vector_insert), to generate the direct move from the source to destination Neon register.

I am seeking feedback on whether there exists a better/more generic way of doing this, since currently we are matching patterns case by case. Even though I tried to be exhaustive, it's easy to miss a pattern. Please let me know your thoughts on this.

[dmgreen]

Tablegen patterns are always an option - there is no problem with adding those. Like you say the lowering of ADDV reductions is a little odd in places. If you wanted to fix this very generically it could be written as a post-isel peephole optimization, perhaps in AArch64MIPeepholeOpt. I've not looked deeply into the issues, but It could transform fmov w8, s1; mov.s v2[0], w8 to an lane move instruction, which replaces two fairly slow instructions with a fast one. Happening later it might help in more cases than just the reductions.

[efriedma]

The other option is to do some lowering earlier: split the intrinsic into two operations (the reduction, and the vector extract), and select them separately. We already have AArch64ISD opcodes for other reductions; see combineAcrossLanesIntrinsic in AArch64ISelLowering.cpp. This could unlock additional optimizations. (For example, we could combine the insert with another shuffle.)

[nilanjana_basu]

Thank you everyone for the feedback. I am currently working on the post-isel peephole optimization since that can capture the more generic pattern of

%y:gpr = COPY %x:fpr
%z:fpr = INSvi..gpr .., a, %y:gpr

& replace it with
%z:fpr = INSvi..lane .., a, %x:fpr, 0

which should apply to a larger class of code than vector insertion of the uaddlv intrinsic result.
I'll update the patch & problem description once I have a fully working solution.

[nilanjana_basu]

Moved the optimization from ISel phase to Post-ISel peephole optimization

[dmgreen]

Thanks - this looks like it should be pretty generic.

llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp

561–565

Will these ever not be true?

577

Please clang-format. You can also drop the brackets from single-statement if's.
Is it worth checking that the subreg indices are as-expected? I'm not sure they can actually be incorrect.

595

It may not be Killed if it has other uses.

600

If this is removing instructions, does it need to check that the COPYs have 1 use?
Could it just remove MI and let the others be removed naturally if they are no longer used?

llvm/test/CodeGen/AArch64/fptosi-sat-vector.ll

3505

Are you using llvm/utils/update_llc_test_checks.py on the file? I'm not sure the indenting should change.

llvm/test/CodeGen/AArch64/peephole-insvigpr.mir

112

Some of this can often be removed to make the tests a little simple.

[fhahn]

llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp

595

If this is not covered by the existing tests, could you add one that covers this case?

llvm/test/CodeGen/AArch64/peephole-insvigpr.mir

25

you should be able to keep the function here to a minimum, i.e.

define void @insert_vec_v6i64_uaddlv_from_v4i32() {
  ret void
 }

Then you have to remove the references to LLVM IR in the MIR itself, .e.g.

bb.0.entry: -> bb.0:`

STRDui killed %17, %0, 2 :: (store (s64) into %ir.0 + 16) -> STRDui killed %17, %0, 2 :: (store (s64))

86

I think most of those should be removable. Also would be good to add the new test separately.

92

you probably are also able to remove this.

[nilanjana_basu]

Addressed most comments of reviewers - removed kill flag on source register, deleted removal of dangling COPY instructions, removed unnecessary checks, updated test scripts.

[nilanjana_basu]

Removing redundant modification in a file

[nilanjana_basu]

Addressed the reviewer comments.

llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp

561–565

I haven't found a case where it does. Removed it in the latest patch.

577

Please clang-format. You can also drop the brackets from single-statement if's.

Done

Is it worth checking that the subreg indices are as-expected? I'm not sure they can actually be incorrect.

The check for the source register being virtual and of type FP128 is needed. I removed the rest.

595

It may not be Killed if it has other uses.

I replaced the 'kill' flag with the original flag of the source register.

600

Deleted the removal of dangling COPY instructions to allow them to be handled naturally.

llvm/test/CodeGen/AArch64/fptosi-sat-vector.ll

3505

Thank you for mentioning this - wasn't aware of this tool before. Updated the files.

[nilanjana_basu]

Removed redundant lines from MIR test file

[dmgreen]

Thanks. From what I can see this LGTM.

The Peephole optimizations have a habit of causing subtle problems. Please make sure you run a bootstrap at least to see if any problems arise.

[nilanjana_basu]

Merged with latest codebase & updated the tests

[nilanjana_basu]

In D142594#4138899, @dmgreen wrote:

Thanks. From what I can see this LGTM.

The Peephole optimizations have a habit of causing subtle problems. Please make sure you run a bootstrap at least to see if any problems arise.

Thank you for the review. I ran all unit tests, llvm-lit tests with llvm-test-suite & a bootstrap build test.

llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp

 //
 //    If AArch64's 32-bit form of instruction defines the source operand of
 //    ORRWrs, we can remove the ORRWrs because the upper 32 bits of the source
 //    operand are set to zero.
 //
 // 5. %reg = INSERT_SUBREG %reg(tied-def 0), %subreg, subidx
 //     ==> %reg:subidx =  SUBREG_TO_REG 0, %subreg, subidx
 //
+// 6. %intermediate:gpr32 = COPY %src:fpr128
+//    %dst:fpr128 = INSvi32gpr %dst_vec:fpr128, dst_index, %intermediate:gpr32
+//     ==> %dst:fpr128 = INSvi32lane %dst_vec:fpr128, dst_index, %src:fpr128, 0
+//
+//    In cases where a source FPR is copied to a GPR in order to be copied
+//    to a destination FPR, we can directly copy the values between the FPRs,
+//    eliminating the use of the Integer unit. When we match a pattern of
+//    INSvi[X]gpr that is preceded by a chain of COPY instructions from a FPR
+//    source, we use the INSvi[X]lane to replace the COPY & INSvi[X]gpr
+//    instructions.
+//
 //===----------------------------------------------------------------------===//
 
 #include "AArch64ExpandImm.h"
 #include "AArch64InstrInfo.h"
 #include "MCTargetDesc/AArch64AddressingModes.h"
 #include "llvm/CodeGen/MachineDominators.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
 
struct AArch64MIPeepholeOpt : public MachineFunctionPass {
   bool visitADDSUB(unsigned PosOpc, unsigned NegOpc, MachineInstr &MI);
   template <typename T>
   bool visitADDSSUBS(OpcodePair PosOpcs, OpcodePair NegOpcs, MachineInstr &MI);
 
   template <typename T>
   bool visitAND(unsigned Opc, MachineInstr &MI);
   bool visitORR(MachineInstr &MI);
   bool visitINSERT(MachineInstr &MI);
+  bool visitINSviGPR(MachineInstr &MI, unsigned Opc);
   bool runOnMachineFunction(MachineFunction &MF) override;
 
   StringRef getPassName() const override {
     return "AArch64 MI Peephole Optimization pass";
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.setPreservesCFG();
bool AArch64MIPeepholeOpt::splitTwoPartImm(
   MI.eraseFromParent();
   if (SubregToRegMI)
     SubregToRegMI->eraseFromParent();
   MovMI->eraseFromParent();
 
   return true;
 }
 
+bool AArch64MIPeepholeOpt::visitINSviGPR(MachineInstr &MI, unsigned Opc) {
+  // Check if this INSvi[X]gpr comes from COPY of a source FPR128
+  //
+  // From
+  //  %intermediate1:gpr64 = COPY %src:fpr128
+  //  %intermediate2:gpr32 = COPY %intermediate1:gpr64
+  //  %dst:fpr128 = INSvi[X]gpr %dst_vec:fpr128, dst_index, %intermediate2:gpr32
+  // To
+  //  %dst:fpr128 = INSvi[X]lane %dst_vec:fpr128, dst_index, %src:fpr128,
+  //  src_index
+  // where src_index = 0, X = [8|16|32|64]
+
+  MachineInstr *SrcMI = MRI->getUniqueVRegDef(MI.getOperand(3).getReg());
+
+  // For a chain of COPY instructions, find the initial source register
+  // and check if it's an FPR128
+  while (true) {
+    if (!SrcMI || SrcMI->getOpcode() != TargetOpcode::COPY)
+      return false;
+
+    if (!SrcMI->getOperand(1).getReg().isVirtual())
+      return false;
+
+    if (MRI->getRegClass(SrcMI->getOperand(1).getReg()) ==
+        &AArch64::FPR128RegClass) {
+      break;
+    }
+    SrcMI = MRI->getUniqueVRegDef(SrcMI->getOperand(1).getReg());
+  }
+
+  Register DstReg = MI.getOperand(0).getReg();
+  Register SrcReg = SrcMI->getOperand(1).getReg();
+  MachineInstr *INSvilaneMI =
+      BuildMI(*MI.getParent(), MI, MI.getDebugLoc(), TII->get(Opc), DstReg)
+          .add(MI.getOperand(1))
+          .add(MI.getOperand(2))
+          .addUse(SrcReg, getRegState(SrcMI->getOperand(1)))
+          .addImm(0);
+
+  LLVM_DEBUG(dbgs() << MI << "  replace by:\n: " << *INSvilaneMI << "\n");
+  MI.eraseFromParent();
+  return true;
+}
+
 bool AArch64MIPeepholeOpt::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
   TII = static_cast<const AArch64InstrInfo *>(MF.getSubtarget().getInstrInfo());
   TRI = static_cast<const AArch64RegisterInfo *>(
       MF.getSubtarget().getRegisterInfo());
   MLI = &getAnalysis<MachineLoopInfo>();
   MRI = &MF.getRegInfo();
 
   assert(MRI->isSSA() && "Expected to be run on SSA form!");
 
   bool Changed = false;
 
   for (MachineBasicBlock &MBB : MF) {
for (MachineInstr &MI : make_early_inc_range(MBB)) {
                                           {AArch64::SUBXri, AArch64::SUBSXri},
                                           MI);
         break;
       case AArch64::SUBSXrr:
         Changed = visitADDSSUBS<uint64_t>({AArch64::SUBXri, AArch64::SUBSXri},
                                           {AArch64::ADDXri, AArch64::ADDSXri},
                                           MI);
         break;
+      case AArch64::INSvi64gpr:
+        Changed = visitINSviGPR(MI, AArch64::INSvi64lane);
+        break;
+      case AArch64::INSvi32gpr:
+        Changed = visitINSviGPR(MI, AArch64::INSvi32lane);
+        break;
+      case AArch64::INSvi16gpr:
+        Changed = visitINSviGPR(MI, AArch64::INSvi16lane);
+        break;
+      case AArch64::INSvi8gpr:
+        Changed = visitINSviGPR(MI, AArch64::INSvi8lane);
+        break;
       }
     }
   }
 
   return Changed;
 }
 
 FunctionPass *llvm::createAArch64MIPeepholeOptPass() {
   return new AArch64MIPeepholeOpt();
 }

llvm/test/CodeGen/AArch64/aarch64-neon-vector-insert-uaddlv.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=arm64-apple-ios -o - %s | FileCheck %s
 
 declare i32 @llvm.aarch64.neon.uaddlv.i32.v8i8(<8 x i8>) #0
 declare i32 @llvm.aarch64.neon.uaddlv.i32.v16i8(<16 x i8>) #0
 declare i32 @llvm.aarch64.neon.uaddlv.i32.v4i16(<4 x i16>) #0
 declare i32 @llvm.aarch64.neon.uaddlv.i32.v8i16(<8 x i16>) #0
 declare i64 @llvm.aarch64.neon.uaddlv.i64.v4i32(<4 x i32>) #0
 
 define void @insert_vec_v2i32_uaddlv_from_v8i16(ptr %0) {
 ; CHECK-LABEL: insert_vec_v2i32_uaddlv_from_v8i16:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    uaddlv.8h s0, v0
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    mov.s v1[0], w8
-; CHECK-NEXT:    ucvtf.2s v0, v1
-; CHECK-NEXT:    str d0, [x0]
+; CHECK-NEXT:    mov.s v1[0], v0[0]
+; CHECK-NEXT:    ucvtf.2s v1, v1
+; CHECK-NEXT:    str d1, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v8i16(<8 x i16> zeroinitializer)
   %1 = insertelement <2 x i32> zeroinitializer, i32 %vaddlv, i64 0
   %2 = uitofp <2 x i32> %1 to <2 x float>
   store <2 x float> %2, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v4i32_uaddlv_from_v8i16(ptr %0) {
 ; CHECK-LABEL: insert_vec_v4i32_uaddlv_from_v8i16:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    uaddlv.8h s1, v0
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov.s v0[0], w8
+; CHECK-NEXT:    mov.s v0[0], v1[0]
 ; CHECK-NEXT:    ucvtf.4s v0, v0
 ; CHECK-NEXT:    str q0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v8i16(<8 x i16> zeroinitializer)
   %1 = insertelement <4 x i32> zeroinitializer, i32 %vaddlv, i64 0
   %2 = uitofp <4 x i32> %1 to <4 x float>
   store <4 x float> %2, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v16i32_uaddlv_from_v8i16(ptr %0) {
 ; CHECK-LABEL: insert_vec_v16i32_uaddlv_from_v8i16:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
-; CHECK-NEXT:    movi.2d v2, #0000000000000000
-; CHECK-NEXT:    uaddlv.8h s1, v0
+; CHECK-NEXT:    movi.2d v1, #0000000000000000
+; CHECK-NEXT:    uaddlv.8h s2, v0
 ; CHECK-NEXT:    stp q0, q0, [x0, #32]
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov.s v2[0], w8
-; CHECK-NEXT:    ucvtf.4s v1, v2
+; CHECK-NEXT:    mov.s v1[0], v2[0]
+; CHECK-NEXT:    ucvtf.4s v1, v1
 ; CHECK-NEXT:    stp q1, q0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v8i16(<8 x i16> zeroinitializer)
   %1 = insertelement <16 x i32> zeroinitializer, i32 %vaddlv, i64 0
   %2 = uitofp <16 x i32> %1 to <16 x float>
   store <16 x float> %2, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v23i32_uaddlv_from_v8i16(ptr %0) {
 ; CHECK-LABEL: insert_vec_v23i32_uaddlv_from_v8i16:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
-; CHECK-NEXT:    movi.2d v2, #0000000000000000
-; CHECK-NEXT:    uaddlv.8h s1, v0
+; CHECK-NEXT:    add x8, x0, #88
+; CHECK-NEXT:    movi.2d v1, #0000000000000000
+; CHECK-NEXT:    uaddlv.8h s2, v0
 ; CHECK-NEXT:    stp q0, q0, [x0, #16]
 ; CHECK-NEXT:    stp q0, q0, [x0, #48]
-; CHECK-NEXT:    str d0, [x0, #80]
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov.s v2[0], w8
-; CHECK-NEXT:    add x8, x0, #88
 ; CHECK-NEXT:    st1.s { v0 }[2], [x8]
-; CHECK-NEXT:    ucvtf.4s v1, v2
+; CHECK-NEXT:    mov.s v1[0], v2[0]
+; CHECK-NEXT:    str d0, [x0, #80]
+; CHECK-NEXT:    ucvtf.4s v1, v1
 ; CHECK-NEXT:    str q1, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v8i16(<8 x i16> zeroinitializer)
   %1 = insertelement <23 x i32> zeroinitializer, i32 %vaddlv, i64 0
   %2 = uitofp <23 x i32> %1 to <23 x float>
   store <23 x float> %2, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v2i32_uaddlv_from_v16i8(ptr %0) {
 ; CHECK-LABEL: insert_vec_v2i32_uaddlv_from_v16i8:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    uaddlv.16b h0, v0
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    mov.s v1[0], w8
-; CHECK-NEXT:    ucvtf.2s v0, v1
-; CHECK-NEXT:    str d0, [x0]
+; CHECK-NEXT:    mov.s v1[0], v0[0]
+; CHECK-NEXT:    ucvtf.2s v1, v1
+; CHECK-NEXT:    str d1, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v16i8(<16 x i8> zeroinitializer)
   %1 = insertelement <2 x i32> zeroinitializer, i32 %vaddlv, i64 0
   %2 = uitofp <2 x i32> %1 to <2 x float>
   store <2 x float> %2, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v2i32_uaddlv_from_v8i8(ptr %0) {
 ; CHECK-LABEL: insert_vec_v2i32_uaddlv_from_v8i8:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    uaddlv.8b h1, v0
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov.s v0[0], w8
+; CHECK-NEXT:    mov.s v0[0], v1[0]
 ; CHECK-NEXT:    ucvtf.2s v0, v0
 ; CHECK-NEXT:    str d0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v8i8(<8 x i8> zeroinitializer)
   %1 = insertelement <2 x i32> zeroinitializer, i32 %vaddlv, i64 0
   %2 = uitofp <2 x i32> %1 to <2 x float>
   store <2 x float> %2, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v2i32_uaddlv_from_v4i16(ptr %0) {
 ; CHECK-LABEL: insert_vec_v2i32_uaddlv_from_v4i16:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    uaddlv.4h s1, v0
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov.s v0[0], w8
+; CHECK-NEXT:    mov.s v0[0], v1[0]
 ; CHECK-NEXT:    ucvtf.2s v0, v0
 ; CHECK-NEXT:    str d0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v4i16(<4 x i16> zeroinitializer)
   %1 = insertelement <2 x i32> zeroinitializer, i32 %vaddlv, i64 0
   %2 = uitofp <2 x i32> %1 to <2 x float>
   store <2 x float> %2, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v6i64_uaddlv_from_v4i32(ptr %0) {
 ; CHECK-LABEL: insert_vec_v6i64_uaddlv_from_v4i32:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    movi d0, #0000000000000000
+; CHECK-NEXT:    movi.2d v3, #0000000000000000
 ; CHECK-NEXT:    uaddlv.4s d2, v1
-; CHECK-NEXT:    fmov x8, d2
-; CHECK-NEXT:    movi.2d v2, #0000000000000000
-; CHECK-NEXT:    mov.d v1[0], x8
-; CHECK-NEXT:    str d2, [x0, #16]
+; CHECK-NEXT:    str d3, [x0, #16]
+; CHECK-NEXT:    mov.d v1[0], v2[0]
 ; CHECK-NEXT:    ucvtf.2d v1, v1
 ; CHECK-NEXT:    fcvtn v1.2s, v1.2d
 ; CHECK-NEXT:    mov.d v1[1], v0[0]
 ; CHECK-NEXT:    str q1, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i64 @llvm.aarch64.neon.uaddlv.i64.v4i32(<4 x i32> zeroinitializer)
   %1 = insertelement <6 x i64> zeroinitializer, i64 %vaddlv, i64 0
   %2 = uitofp <6 x i64> %1 to <6 x float>
   store <6 x float> %2, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v2i64_uaddlv_from_v4i32(ptr %0) {
 ; CHECK-LABEL: insert_vec_v2i64_uaddlv_from_v4i32:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    uaddlv.4s d1, v0
-; CHECK-NEXT:    fmov x8, d1
-; CHECK-NEXT:    mov.d v0[0], x8
+; CHECK-NEXT:    mov.d v0[0], v1[0]
 ; CHECK-NEXT:    ucvtf.2d v0, v0
 ; CHECK-NEXT:    fcvtn v0.2s, v0.2d
 ; CHECK-NEXT:    str d0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i64 @llvm.aarch64.neon.uaddlv.i64.v4i32(<4 x i32> zeroinitializer)
   %1 = insertelement <2 x i64> zeroinitializer, i64 %vaddlv, i64 0
   %2 = uitofp <2 x i64> %1 to <2 x float>
   store <2 x float> %2, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v5i64_uaddlv_from_v4i32(ptr %0) {
 ; CHECK-LABEL: insert_vec_v5i64_uaddlv_from_v4i32:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    str wzr, [x0, #16]
 ; CHECK-NEXT:    movi d0, #0000000000000000
 ; CHECK-NEXT:    uaddlv.4s d2, v1
-; CHECK-NEXT:    fmov x8, d2
-; CHECK-NEXT:    mov.d v1[0], x8
+; CHECK-NEXT:    mov.d v1[0], v2[0]
 ; CHECK-NEXT:    ucvtf.2d v1, v1
 ; CHECK-NEXT:    fcvtn v1.2s, v1.2d
 ; CHECK-NEXT:    mov.d v1[1], v0[0]
 ; CHECK-NEXT:    str q1, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i64 @llvm.aarch64.neon.uaddlv.i64.v4i32(<4 x i32> zeroinitializer)
   %1 = insertelement <5 x i64> zeroinitializer, i64 %vaddlv, i64 0
   %2 = uitofp <5 x i64> %1 to <5 x float>
   store <5 x float> %2, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v8i16_uaddlv_from_v8i16(ptr %0) {
 ; CHECK-LABEL: insert_vec_v8i16_uaddlv_from_v8i16:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    stp xzr, xzr, [x0, #16]
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    uaddlv.8h s0, v0
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    mov.h v1[0], w8
-; CHECK-NEXT:    ushll.4s v0, v1, #0
-; CHECK-NEXT:    ucvtf.4s v0, v0
-; CHECK-NEXT:    str q0, [x0]
+; CHECK-NEXT:    mov.h v1[0], v0[0]
+; CHECK-NEXT:    ushll.4s v1, v1, #0
+; CHECK-NEXT:    ucvtf.4s v1, v1
+; CHECK-NEXT:    str q1, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v8i16(<8 x i16> zeroinitializer)
   %1 = trunc i32 %vaddlv to i16
   %2 = insertelement <8 x i16> zeroinitializer, i16 %1, i64 0
   %3 = uitofp <8 x i16> %2 to <8 x float>
   store <8 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v3i16_uaddlv_from_v8i16(ptr %0) {
 ; CHECK-LABEL: insert_vec_v3i16_uaddlv_from_v8i16:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
+; CHECK-NEXT:    add x8, x0, #8
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    uaddlv.8h s0, v0
-; CHECK-NEXT:    fmov w8, s0
-; CHECK-NEXT:    mov.h v1[0], w8
-; CHECK-NEXT:    add x8, x0, #8
-; CHECK-NEXT:    ushll.4s v0, v1, #0
-; CHECK-NEXT:    ucvtf.4s v0, v0
-; CHECK-NEXT:    st1.s { v0 }[2], [x8]
-; CHECK-NEXT:    str d0, [x0]
+; CHECK-NEXT:    mov.h v1[0], v0[0]
+; CHECK-NEXT:    ushll.4s v1, v1, #0
+; CHECK-NEXT:    ucvtf.4s v1, v1
+; CHECK-NEXT:    st1.s { v1 }[2], [x8]
+; CHECK-NEXT:    str d1, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v8i16(<8 x i16> zeroinitializer)
   %1 = trunc i32 %vaddlv to i16
   %2 = insertelement <3 x i16> zeroinitializer, i16 %1, i64 0
   %3 = uitofp <3 x i16> %2 to <3 x float>
   store <3 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v16i64_uaddlv_from_v4i16(ptr %0) {
 ; CHECK-LABEL: insert_vec_v16i64_uaddlv_from_v4i16:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    movi d0, #0000000000000000
-; CHECK-NEXT:    movi.2d v3, #0000000000000000
-; CHECK-NEXT:    uaddlv.4h s2, v1
+; CHECK-NEXT:    movi.2d v2, #0000000000000000
+; CHECK-NEXT:    uaddlv.4h s3, v1
 ; CHECK-NEXT:    stp q1, q1, [x0, #32]
-; CHECK-NEXT:    fmov w8, s2
-; CHECK-NEXT:    mov.s v3[0], w8
-; CHECK-NEXT:    ucvtf.2d v2, v3
+; CHECK-NEXT:    mov.s v2[0], v3[0]
+; CHECK-NEXT:    ucvtf.2d v2, v2
 ; CHECK-NEXT:    fcvtn v2.2s, v2.2d
 ; CHECK-NEXT:    mov.d v2[1], v0[0]
 ; CHECK-NEXT:    stp q2, q1, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v4i16(<4 x i16> zeroinitializer)
   %1 = zext i32 %vaddlv to i64
   %2 = insertelement <16 x i64> zeroinitializer, i64 %1, i64 0
   %3 = uitofp <16 x i64> %2 to <16 x float>
   store <16 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v16i8_uaddlv_from_v8i8(ptr %0) {
 ; CHECK-LABEL: insert_vec_v16i8_uaddlv_from_v8i8:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
-; CHECK-NEXT:    movi.2d v2, #0000000000000000
-; CHECK-NEXT:    uaddlv.8b h1, v0
+; CHECK-NEXT:    movi.2d v1, #0000000000000000
+; CHECK-NEXT:    uaddlv.8b h2, v0
 ; CHECK-NEXT:    stp q0, q0, [x0, #32]
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov.b v2[0], w8
-; CHECK-NEXT:    zip1.8b v1, v2, v0
+; CHECK-NEXT:    mov.b v1[0], v2[0]
+; CHECK-NEXT:    zip1.8b v1, v1, v0
 ; CHECK-NEXT:    bic.4h v1, #255, lsl #8
 ; CHECK-NEXT:    ushll.4s v1, v1, #0
 ; CHECK-NEXT:    ucvtf.4s v1, v1
 ; CHECK-NEXT:    stp q1, q0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v8i8(<8 x i8> zeroinitializer)
   %1 = trunc i32 %vaddlv to i8
   %2 = insertelement <16 x i8> zeroinitializer, i8 %1, i64 0
   %3 = uitofp <16 x i8> %2 to <16 x float>
   store <16 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v8i8_uaddlv_from_v8i8(ptr %0) {
 ; CHECK-LABEL: insert_vec_v8i8_uaddlv_from_v8i8:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    stp xzr, xzr, [x0, #16]
 ; CHECK-NEXT:    uaddlv.8b h1, v0
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov.h v0[0], w8
+; CHECK-NEXT:    mov.h v0[0], v1[0]
 ; CHECK-NEXT:    bic.4h v0, #255, lsl #8
 ; CHECK-NEXT:    ushll.4s v0, v0, #0
 ; CHECK-NEXT:    ucvtf.4s v0, v0
 ; CHECK-NEXT:    str q0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v8i8(<8 x i8> zeroinitializer)
   %1 = trunc i32 %vaddlv to i8
   %2 = insertelement <8 x i8> zeroinitializer, i8 %1, i64 0
   %3 = uitofp <8 x i8> %2 to <8 x float>
   store <8 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v12i16_uaddlv_from_v4i16(ptr %0) {
 ; CHECK-LABEL: insert_vec_v12i16_uaddlv_from_v4i16:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    stp xzr, xzr, [x0, #16]
 ; CHECK-NEXT:    stp xzr, xzr, [x0, #32]
 ; CHECK-NEXT:    uaddlv.4h s1, v0
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov.h v0[0], w8
+; CHECK-NEXT:    mov.h v0[0], v1[0]
 ; CHECK-NEXT:    ushll.4s v0, v0, #0
 ; CHECK-NEXT:    ucvtf.4s v0, v0
 ; CHECK-NEXT:    str q0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v4i16(<4 x i16> zeroinitializer)
   %1 = trunc i32 %vaddlv to i16
   %2 = insertelement <12 x i16> zeroinitializer, i16 %1, i64 0
   %3 = uitofp <12 x i16> %2 to <12 x float>
   store <12 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v8i32_uaddlv_from_v4i32(ptr %0) {
 ; CHECK-LABEL: insert_vec_v8i32_uaddlv_from_v4i32:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    stp xzr, xzr, [x0, #16]
 ; CHECK-NEXT:    uaddlv.4s d1, v0
-; CHECK-NEXT:    fmov x8, d1
-; CHECK-NEXT:    mov.s v0[0], w8
+; CHECK-NEXT:    mov.s v0[0], v1[0]
 ; CHECK-NEXT:    ucvtf.4s v0, v0
 ; CHECK-NEXT:    str q0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i64 @llvm.aarch64.neon.uaddlv.i64.v4i32(<4 x i32> zeroinitializer)
   %1 = trunc i64 %vaddlv to i32
   %2 = insertelement <8 x i32> zeroinitializer, i32 %1, i64 0
   %3 = uitofp <8 x i32> %2 to <8 x float>
   store <8 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v16i32_uaddlv_from_v4i32(ptr %0) {
 ; CHECK-LABEL: insert_vec_v16i32_uaddlv_from_v4i32:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
-; CHECK-NEXT:    movi.2d v2, #0000000000000000
-; CHECK-NEXT:    uaddlv.4s d1, v0
+; CHECK-NEXT:    movi.2d v1, #0000000000000000
+; CHECK-NEXT:    uaddlv.4s d2, v0
 ; CHECK-NEXT:    stp q0, q0, [x0, #32]
-; CHECK-NEXT:    fmov x8, d1
-; CHECK-NEXT:    mov.s v2[0], w8
-; CHECK-NEXT:    ucvtf.4s v1, v2
+; CHECK-NEXT:    mov.s v1[0], v2[0]
+; CHECK-NEXT:    ucvtf.4s v1, v1
 ; CHECK-NEXT:    stp q1, q0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i64 @llvm.aarch64.neon.uaddlv.i64.v4i32(<4 x i32> zeroinitializer)
   %1 = trunc i64 %vaddlv to i32
   %2 = insertelement <16 x i32> zeroinitializer, i32 %1, i64 0
   %3 = uitofp <16 x i32> %2 to <16 x float>
   store <16 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v4i16_uaddlv_from_v4i32(ptr %0) {
 ; CHECK-LABEL: insert_vec_v4i16_uaddlv_from_v4i32:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    uaddlv.4s d0, v0
-; CHECK-NEXT:    fmov x8, d0
-; CHECK-NEXT:    mov.h v1[0], w8
+; CHECK-NEXT:    mov.h v1[0], v0[0]
 ; CHECK-NEXT:    ushll.4s v0, v1, #0
 ; CHECK-NEXT:    ucvtf.4s v0, v0
 ; CHECK-NEXT:    str q0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i64 @llvm.aarch64.neon.uaddlv.i64.v4i32(<4 x i32> zeroinitializer)
   %1 = trunc i64 %vaddlv to i16
   %2 = insertelement <4 x i16> zeroinitializer, i16 %1, i64 0
   %3 = uitofp <4 x i16> %2 to <4 x float>
   store <4 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v16i16_uaddlv_from_v4i32(ptr %0) {
 ; CHECK-LABEL: insert_vec_v16i16_uaddlv_from_v4i32:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    uaddlv.4s d0, v0
-; CHECK-NEXT:    fmov x8, d0
+; CHECK-NEXT:    mov.h v1[0], v0[0]
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
-; CHECK-NEXT:    mov.h v1[0], w8
-; CHECK-NEXT:    stp q0, q0, [x0, #32]
 ; CHECK-NEXT:    ushll.4s v1, v1, #0
+; CHECK-NEXT:    stp q0, q0, [x0, #32]
 ; CHECK-NEXT:    ucvtf.4s v1, v1
 ; CHECK-NEXT:    stp q1, q0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i64 @llvm.aarch64.neon.uaddlv.i64.v4i32(<4 x i32> zeroinitializer)
   %1 = trunc i64 %vaddlv to i16
   %2 = insertelement <16 x i16> zeroinitializer, i16 %1, i64 0
   %3 = uitofp <16 x i16> %2 to <16 x float>
   store <16 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v8i8_uaddlv_from_v4i32(ptr %0) {
 ; CHECK-LABEL: insert_vec_v8i8_uaddlv_from_v4i32:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    stp xzr, xzr, [x0, #16]
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    uaddlv.4s d0, v0
-; CHECK-NEXT:    fmov x8, d0
-; CHECK-NEXT:    mov.h v1[0], w8
+; CHECK-NEXT:    mov.h v1[0], v0[0]
 ; CHECK-NEXT:    bic.4h v1, #255, lsl #8
 ; CHECK-NEXT:    ushll.4s v0, v1, #0
 ; CHECK-NEXT:    ucvtf.4s v0, v0
 ; CHECK-NEXT:    str q0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i64 @llvm.aarch64.neon.uaddlv.i64.v4i32(<4 x i32> zeroinitializer)
   %1 = trunc i64 %vaddlv to i8
   %2 = insertelement <8 x i8> zeroinitializer, i8 %1, i64 0
   %3 = uitofp <8 x i8> %2 to <8 x float>
   store <8 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v16i8_uaddlv_from_v4i32(ptr %0) {
 ; CHECK-LABEL: insert_vec_v16i8_uaddlv_from_v4i32:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    uaddlv.4s d0, v0
-; CHECK-NEXT:    fmov x8, d0
-; CHECK-NEXT:    mov.b v1[0], w8
+; CHECK-NEXT:    mov.b v1[0], v0[0]
 ; CHECK-NEXT:    zip1.8b v0, v1, v0
 ; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:    bic.4h v0, #255, lsl #8
 ; CHECK-NEXT:    ushll.4s v0, v0, #0
 ; CHECK-NEXT:    stp q1, q1, [x0, #32]
 ; CHECK-NEXT:    ucvtf.4s v0, v0
 ; CHECK-NEXT:    stp q0, q1, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i64 @llvm.aarch64.neon.uaddlv.i64.v4i32(<4 x i32> zeroinitializer)
   %1 = trunc i64 %vaddlv to i8
   %2 = insertelement <16 x i8> zeroinitializer, i8 %1, i64 0
   %3 = uitofp <16 x i8> %2 to <16 x float>
   store <16 x float> %3, ptr %0, align 8
   ret void
 }
 
 define void @insert_vec_v2i32_uaddlv_from_v8i16_nz_index(ptr %0) {
 ; CHECK-LABEL: insert_vec_v2i32_uaddlv_from_v8i16_nz_index:
 ; CHECK:       ; %bb.0: ; %entry
 ; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    uaddlv.8h s1, v0
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov.s v0[2], w8
+; CHECK-NEXT:    mov.s v0[2], v1[0]
 ; CHECK-NEXT:    ucvtf.4s v0, v0
 ; CHECK-NEXT:    str q0, [x0]
 ; CHECK-NEXT:    ret
 
 entry:
   %vaddlv = tail call i32 @llvm.aarch64.neon.uaddlv.i32.v8i16(<8 x i16> zeroinitializer)
   %1 = insertelement <4 x i32> zeroinitializer, i32 %vaddlv, i64 2
   %2 = uitofp <4 x i32> %1 to <4 x float>
   store <4 x float> %2, ptr %0, align 8
   ret void
 }

llvm/test/CodeGen/AArch64/fptosi-sat-vector.ll

 
 define <16 x i8> @test_signed_v16f64_v16i8(<16 x double> %f) {
 ; CHECK-LABEL: test_signed_v16f64_v16i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov d16, v0.d[1]
 ; CHECK-NEXT:    mov w8, #127
 ; CHECK-NEXT:    fcvtzs w11, d0
 ; CHECK-NEXT:    mov w9, #-128
-; CHECK-NEXT:    fcvtzs w13, d1
 ; CHECK-NEXT:    mov d0, v2.d[1]
-; CHECK-NEXT:    fcvtzs w14, d2
+; CHECK-NEXT:    fcvtzs w13, d1
 ; CHECK-NEXT:    fcvtzs w10, d16
 ; CHECK-NEXT:    mov d16, v1.d[1]
-; CHECK-NEXT:    mov d1, v3.d[1]
-; CHECK-NEXT:    fcvtzs w15, d0
+; CHECK-NEXT:    fcvtzs w14, d0
 ; CHECK-NEXT:    cmp w10, #127
 ; CHECK-NEXT:    csel w10, w10, w8, lt
 ; CHECK-NEXT:    fcvtzs w12, d16
 ; CHECK-NEXT:    cmn w10, #128
 ; CHECK-NEXT:    csel w10, w10, w9, gt
 ; CHECK-NEXT:    cmp w11, #127
 ; CHECK-NEXT:    csel w11, w11, w8, lt
 ; CHECK-NEXT:    cmn w11, #128
 ; CHECK-NEXT:    csel w11, w11, w9, gt
 ; CHECK-NEXT:    cmp w12, #127
 ; CHECK-NEXT:    csel w12, w12, w8, lt
 ; CHECK-NEXT:    cmn w12, #128
 ; CHECK-NEXT:    csel w12, w12, w9, gt
 ; CHECK-NEXT:    cmp w13, #127
-; CHECK-NEXT:    csel w13, w13, w8, lt
 ; CHECK-NEXT:    fmov s0, w11
-; CHECK-NEXT:    cmn w13, #128
-; CHECK-NEXT:    csel w11, w13, w9, gt
-; CHECK-NEXT:    cmp w15, #127
-; CHECK-NEXT:    mov v0.s[1], w10
-; CHECK-NEXT:    csel w10, w15, w8, lt
-; CHECK-NEXT:    cmn w10, #128
-; CHECK-NEXT:    fcvtzs w13, d3
-; CHECK-NEXT:    fmov s2, w11
-; CHECK-NEXT:    csel w10, w10, w9, gt
-; CHECK-NEXT:    cmp w14, #127
-; CHECK-NEXT:    fcvtzs w11, d1
-; CHECK-NEXT:    mov w15, v0.s[1]
-; CHECK-NEXT:    csel w14, w14, w8, lt
-; CHECK-NEXT:    mov v2.s[1], w12
-; CHECK-NEXT:    cmn w14, #128
-; CHECK-NEXT:    csel w12, w14, w9, gt
-; CHECK-NEXT:    cmp w11, #127
-; CHECK-NEXT:    csel w11, w11, w8, lt
-; CHECK-NEXT:    mov d1, v4.d[1]
-; CHECK-NEXT:    mov v0.b[1], w15
+; CHECK-NEXT:    csel w11, w13, w8, lt
 ; CHECK-NEXT:    cmn w11, #128
-; CHECK-NEXT:    fmov w14, s2
+; CHECK-NEXT:    fcvtzs w13, d2
 ; CHECK-NEXT:    csel w11, w11, w9, gt
-; CHECK-NEXT:    fmov s3, w12
-; CHECK-NEXT:    cmp w13, #127
-; CHECK-NEXT:    mov w12, v2.s[1]
-; CHECK-NEXT:    csel w13, w13, w8, lt
-; CHECK-NEXT:    mov v0.b[2], w14
-; CHECK-NEXT:    cmn w13, #128
-; CHECK-NEXT:    mov v3.s[1], w10
-; CHECK-NEXT:    csel w13, w13, w9, gt
-; CHECK-NEXT:    fcvtzs w15, d1
-; CHECK-NEXT:    fcvtzs w14, d4
-; CHECK-NEXT:    mov d1, v5.d[1]
-; CHECK-NEXT:    mov v0.b[3], w12
-; CHECK-NEXT:    fmov s4, w13
-; CHECK-NEXT:    cmp w15, #127
-; CHECK-NEXT:    fmov w13, s3
-; CHECK-NEXT:    csel w10, w15, w8, lt
-; CHECK-NEXT:    mov w12, v3.s[1]
-; CHECK-NEXT:    cmn w10, #128
-; CHECK-NEXT:    fcvtzs w15, d1
-; CHECK-NEXT:    csel w10, w10, w9, gt
 ; CHECK-NEXT:    cmp w14, #127
-; CHECK-NEXT:    mov v0.b[4], w13
+; CHECK-NEXT:    mov v0.s[1], w10
 ; CHECK-NEXT:    csel w14, w14, w8, lt
-; CHECK-NEXT:    mov v4.s[1], w11
 ; CHECK-NEXT:    cmn w14, #128
-; CHECK-NEXT:    csel w14, w14, w9, gt
-; CHECK-NEXT:    fcvtzs w13, d5
-; CHECK-NEXT:    cmp w15, #127
-; CHECK-NEXT:    mov d2, v6.d[1]
-; CHECK-NEXT:    mov v0.b[5], w12
-; CHECK-NEXT:    csel w11, w15, w8, lt
-; CHECK-NEXT:    fmov w12, s4
-; CHECK-NEXT:    cmn w11, #128
-; CHECK-NEXT:    fmov s1, w14
-; CHECK-NEXT:    csel w11, w11, w9, gt
+; CHECK-NEXT:    mov d2, v3.d[1]
+; CHECK-NEXT:    fmov s1, w11
+; CHECK-NEXT:    csel w11, w14, w9, gt
 ; CHECK-NEXT:    cmp w13, #127
-; CHECK-NEXT:    mov w14, v4.s[1]
-; CHECK-NEXT:    mov v0.b[6], w12
+; CHECK-NEXT:    fcvtzs w10, d3
+; CHECK-NEXT:    mov w14, v0.s[1]
 ; CHECK-NEXT:    csel w13, w13, w8, lt
-; CHECK-NEXT:    mov v1.s[1], w10
 ; CHECK-NEXT:    cmn w13, #128
-; CHECK-NEXT:    fcvtzs w15, d2
+; CHECK-NEXT:    mov d3, v4.d[1]
 ; CHECK-NEXT:    csel w13, w13, w9, gt
-; CHECK-NEXT:    fcvtzs w10, d6
-; CHECK-NEXT:    mov v0.b[7], w14
-; CHECK-NEXT:    cmp w15, #127
-; CHECK-NEXT:    fmov w14, s1
-; CHECK-NEXT:    csel w12, w15, w8, lt
+; CHECK-NEXT:    mov v1.s[1], w12
+; CHECK-NEXT:    fcvtzs w12, d2
+; CHECK-NEXT:    mov v0.b[1], w14
 ; CHECK-NEXT:    fmov s2, w13
-; CHECK-NEXT:    mov w13, v1.s[1]
-; CHECK-NEXT:    mov d1, v7.d[1]
+; CHECK-NEXT:    cmp w12, #127
+; CHECK-NEXT:    fcvtzs w13, d3
+; CHECK-NEXT:    csel w12, w12, w8, lt
+; CHECK-NEXT:    fcvtzs w14, d4
 ; CHECK-NEXT:    cmn w12, #128
-; CHECK-NEXT:    fcvtzs w15, d7
+; CHECK-NEXT:    mov d3, v5.d[1]
+; CHECK-NEXT:    mov v2.s[1], w11
+; CHECK-NEXT:    mov w11, v1.s[1]
+; CHECK-NEXT:    mov v0.b[2], v1.b[0]
 ; CHECK-NEXT:    csel w12, w12, w9, gt
 ; CHECK-NEXT:    cmp w10, #127
-; CHECK-NEXT:    mov v0.b[8], w14
+; CHECK-NEXT:    mov d4, v6.d[1]
 ; CHECK-NEXT:    csel w10, w10, w8, lt
-; CHECK-NEXT:    mov v2.s[1], w11
 ; CHECK-NEXT:    cmn w10, #128
-; CHECK-NEXT:    fcvtzs w11, d1
 ; CHECK-NEXT:    csel w10, w10, w9, gt
-; CHECK-NEXT:    mov v0.b[9], w13
-; CHECK-NEXT:    fmov w14, s2
-; CHECK-NEXT:    cmp w11, #127
-; CHECK-NEXT:    fmov s1, w10
-; CHECK-NEXT:    csel w10, w11, w8, lt
+; CHECK-NEXT:    cmp w13, #127
+; CHECK-NEXT:    mov v0.b[3], w11
+; CHECK-NEXT:    csel w13, w13, w8, lt
+; CHECK-NEXT:    cmn w13, #128
+; CHECK-NEXT:    fcvtzs w11, d3
+; CHECK-NEXT:    csel w13, w13, w9, gt
+; CHECK-NEXT:    cmp w14, #127
+; CHECK-NEXT:    fmov s3, w10
+; CHECK-NEXT:    csel w10, w14, w8, lt
+; CHECK-NEXT:    mov w14, v2.s[1]
 ; CHECK-NEXT:    cmn w10, #128
-; CHECK-NEXT:    mov w13, v2.s[1]
-; CHECK-NEXT:    mov v0.b[10], w14
+; CHECK-NEXT:    mov v0.b[4], v2.b[0]
 ; CHECK-NEXT:    csel w10, w10, w9, gt
-; CHECK-NEXT:    cmp w15, #127
-; CHECK-NEXT:    mov v1.s[1], w12
-; CHECK-NEXT:    csel w8, w15, w8, lt
+; CHECK-NEXT:    mov v3.s[1], w12
+; CHECK-NEXT:    cmp w11, #127
+; CHECK-NEXT:    csel w11, w11, w8, lt
+; CHECK-NEXT:    fcvtzs w12, d5
+; CHECK-NEXT:    cmn w11, #128
+; CHECK-NEXT:    mov v0.b[5], w14
+; CHECK-NEXT:    fcvtzs w14, d4
+; CHECK-NEXT:    fmov s4, w10
+; CHECK-NEXT:    csel w10, w11, w9, gt
+; CHECK-NEXT:    mov w11, v3.s[1]
+; CHECK-NEXT:    cmp w12, #127
+; CHECK-NEXT:    csel w12, w12, w8, lt
+; CHECK-NEXT:    mov v0.b[6], v3.b[0]
+; CHECK-NEXT:    cmn w12, #128
+; CHECK-NEXT:    mov v4.s[1], w13
+; CHECK-NEXT:    csel w12, w12, w9, gt
+; CHECK-NEXT:    cmp w14, #127
+; CHECK-NEXT:    csel w13, w14, w8, lt
+; CHECK-NEXT:    mov v0.b[7], w11
+; CHECK-NEXT:    fcvtzs w11, d6
+; CHECK-NEXT:    cmn w13, #128
+; CHECK-NEXT:    fmov s5, w12
+; CHECK-NEXT:    csel w12, w13, w9, gt
+; CHECK-NEXT:    mov w13, v4.s[1]
+; CHECK-NEXT:    cmp w11, #127
+; CHECK-NEXT:    mov d6, v7.d[1]
+; CHECK-NEXT:    mov v0.b[8], v4.b[0]
+; CHECK-NEXT:    csel w11, w11, w8, lt
+; CHECK-NEXT:    cmn w11, #128
+; CHECK-NEXT:    mov v5.s[1], w10
+; CHECK-NEXT:    csel w10, w11, w9, gt
+; CHECK-NEXT:    fcvtzs w11, d6
+; CHECK-NEXT:    mov v0.b[9], w13
+; CHECK-NEXT:    fcvtzs w13, d7
+; CHECK-NEXT:    fmov s6, w10
+; CHECK-NEXT:    mov w10, v5.s[1]
+; CHECK-NEXT:    cmp w11, #127
+; CHECK-NEXT:    csel w11, w11, w8, lt
+; CHECK-NEXT:    mov v0.b[10], v5.b[0]
+; CHECK-NEXT:    cmn w11, #128
+; CHECK-NEXT:    mov v6.s[1], w12
+; CHECK-NEXT:    mov v0.b[11], w10
+; CHECK-NEXT:    csel w10, w11, w9, gt
+; CHECK-NEXT:    cmp w13, #127
+; CHECK-NEXT:    csel w8, w13, w8, lt
 ; CHECK-NEXT:    cmn w8, #128
 ; CHECK-NEXT:    csel w8, w8, w9, gt
-; CHECK-NEXT:    mov v0.b[11], w13
-; CHECK-NEXT:    fmov w9, s1
-; CHECK-NEXT:    fmov s2, w8
-; CHECK-NEXT:    mov w8, v1.s[1]
-; CHECK-NEXT:    mov v0.b[12], w9
-; CHECK-NEXT:    mov v2.s[1], w10
-; CHECK-NEXT:    mov v0.b[13], w8
-; CHECK-NEXT:    fmov w8, s2
-; CHECK-NEXT:    mov w9, v2.s[1]
-; CHECK-NEXT:    mov v0.b[14], w8
-; CHECK-NEXT:    mov v0.b[15], w9
+; CHECK-NEXT:    mov w9, v6.s[1]
+; CHECK-NEXT:    mov v0.b[12], v6.b[0]
+; CHECK-NEXT:    fmov s7, w8
+; CHECK-NEXT:    mov v0.b[13], w9
+; CHECK-NEXT:    mov v7.s[1], w10
+; CHECK-NEXT:    mov v0.b[14], v7.b[0]
+; CHECK-NEXT:    mov w8, v7.s[1]
+; CHECK-NEXT:    mov v0.b[15], w8
 ; CHECK-NEXT:    ret
     %x = call <16 x i8> @llvm.fptosi.sat.v16f64.v16i8(<16 x double> %f)
     ret <16 x i8> %x
 }
 
 define <8 x i16> @test_signed_v8f64_v8i16(<8 x double> %f) {
 ; CHECK-LABEL: test_signed_v8f64_v8i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov d4, v3.d[1]

llvm/test/CodeGen/AArch64/fptoui-sat-vector.ll

 ; CHECK-NEXT:    csel w9, w9, w8, lo
 ; CHECK-NEXT:    cmp w10, #255
 ; CHECK-NEXT:    csel w10, w10, w8, lo
 ; CHECK-NEXT:    cmp w11, #255
 ; CHECK-NEXT:    fmov s0, w10
 ; CHECK-NEXT:    csel w10, w11, w8, lo
 ; CHECK-NEXT:    cmp w12, #255
 ; CHECK-NEXT:    csel w11, w12, w8, lo
+; CHECK-NEXT:    fcvtzu w12, d2
 ; CHECK-NEXT:    mov v0.s[1], w9
 ; CHECK-NEXT:    fcvtzu w9, d1
+; CHECK-NEXT:    mov d2, v3.d[1]
 ; CHECK-NEXT:    fmov s1, w11
-; CHECK-NEXT:    fcvtzu w11, d2
 ; CHECK-NEXT:    cmp w9, #255
-; CHECK-NEXT:    mov d2, v3.d[1]
-; CHECK-NEXT:    mov w12, v0.s[1]
+; CHECK-NEXT:    mov w11, v0.s[1]
 ; CHECK-NEXT:    csel w9, w9, w8, lo
+; CHECK-NEXT:    cmp w12, #255
 ; CHECK-NEXT:    mov v1.s[1], w10
-; CHECK-NEXT:    cmp w11, #255
-; CHECK-NEXT:    csel w11, w11, w8, lo
+; CHECK-NEXT:    csel w12, w12, w8, lo
 ; CHECK-NEXT:    fcvtzu w10, d2
-; CHECK-NEXT:    mov d2, v4.d[1]
-; CHECK-NEXT:    mov v0.b[1], w12
-; CHECK-NEXT:    fmov w13, s1
-; CHECK-NEXT:    mov w12, v1.s[1]
-; CHECK-NEXT:    fmov s1, w11
+; CHECK-NEXT:    mov v0.b[1], w11
 ; CHECK-NEXT:    fcvtzu w11, d3
+; CHECK-NEXT:    fmov s2, w12
+; CHECK-NEXT:    mov w12, v1.s[1]
 ; CHECK-NEXT:    cmp w10, #255
-; CHECK-NEXT:    mov v0.b[2], w13
-; CHECK-NEXT:    mov v1.s[1], w9
-; CHECK-NEXT:    csel w9, w10, w8, lo
+; CHECK-NEXT:    mov d3, v4.d[1]
+; CHECK-NEXT:    csel w10, w10, w8, lo
+; CHECK-NEXT:    mov v0.b[2], v1.b[0]
 ; CHECK-NEXT:    cmp w11, #255
-; CHECK-NEXT:    fcvtzu w10, d2
+; CHECK-NEXT:    mov v2.s[1], w9
 ; CHECK-NEXT:    csel w11, w11, w8, lo
-; CHECK-NEXT:    mov d2, v5.d[1]
+; CHECK-NEXT:    fcvtzu w9, d3
+; CHECK-NEXT:    mov d3, v5.d[1]
 ; CHECK-NEXT:    mov v0.b[3], w12
-; CHECK-NEXT:    fmov w12, s1
-; CHECK-NEXT:    cmp w10, #255
-; CHECK-NEXT:    mov w13, v1.s[1]
-; CHECK-NEXT:    fmov s1, w11
-; CHECK-NEXT:    fcvtzu w11, d4
-; CHECK-NEXT:    mov v0.b[4], w12
-; CHECK-NEXT:    mov v1.s[1], w9
-; CHECK-NEXT:    csel w9, w10, w8, lo
-; CHECK-NEXT:    cmp w11, #255
-; CHECK-NEXT:    csel w10, w11, w8, lo
-; CHECK-NEXT:    mov v0.b[5], w13
-; CHECK-NEXT:    fcvtzu w13, d2
-; CHECK-NEXT:    fmov w11, s1
-; CHECK-NEXT:    mov w12, v1.s[1]
-; CHECK-NEXT:    fmov s1, w10
-; CHECK-NEXT:    fcvtzu w10, d5
-; CHECK-NEXT:    cmp w13, #255
-; CHECK-NEXT:    mov v0.b[6], w11
-; CHECK-NEXT:    mov d2, v6.d[1]
-; CHECK-NEXT:    mov v1.s[1], w9
-; CHECK-NEXT:    csel w9, w13, w8, lo
+; CHECK-NEXT:    fcvtzu w12, d4
+; CHECK-NEXT:    fmov s4, w11
+; CHECK-NEXT:    mov w11, v2.s[1]
+; CHECK-NEXT:    cmp w9, #255
+; CHECK-NEXT:    csel w9, w9, w8, lo
+; CHECK-NEXT:    cmp w12, #255
+; CHECK-NEXT:    mov v0.b[4], v2.b[0]
+; CHECK-NEXT:    csel w12, w12, w8, lo
+; CHECK-NEXT:    mov v4.s[1], w10
+; CHECK-NEXT:    fcvtzu w10, d3
+; CHECK-NEXT:    fmov s3, w12
+; CHECK-NEXT:    mov v0.b[5], w11
+; CHECK-NEXT:    fcvtzu w11, d5
+; CHECK-NEXT:    mov w12, v4.s[1]
 ; CHECK-NEXT:    cmp w10, #255
-; CHECK-NEXT:    fcvtzu w13, d6
 ; CHECK-NEXT:    csel w10, w10, w8, lo
+; CHECK-NEXT:    mov d5, v6.d[1]
+; CHECK-NEXT:    cmp w11, #255
+; CHECK-NEXT:    mov v0.b[6], v4.b[0]
+; CHECK-NEXT:    csel w11, w11, w8, lo
+; CHECK-NEXT:    mov v3.s[1], w9
+; CHECK-NEXT:    fcvtzu w9, d6
+; CHECK-NEXT:    mov d6, v7.d[1]
 ; CHECK-NEXT:    mov v0.b[7], w12
-; CHECK-NEXT:    fcvtzu w12, d2
-; CHECK-NEXT:    fmov w11, s1
-; CHECK-NEXT:    fmov s2, w10
-; CHECK-NEXT:    mov w10, v1.s[1]
+; CHECK-NEXT:    fcvtzu w12, d5
+; CHECK-NEXT:    fmov s5, w11
+; CHECK-NEXT:    mov w11, v3.s[1]
 ; CHECK-NEXT:    cmp w12, #255
-; CHECK-NEXT:    mov d1, v7.d[1]
-; CHECK-NEXT:    mov v0.b[8], w11
-; CHECK-NEXT:    mov v2.s[1], w9
-; CHECK-NEXT:    csel w9, w12, w8, lo
-; CHECK-NEXT:    cmp w13, #255
-; CHECK-NEXT:    csel w11, w13, w8, lo
-; CHECK-NEXT:    fcvtzu w13, d7
-; CHECK-NEXT:    mov v0.b[9], w10
-; CHECK-NEXT:    fmov w10, s2
-; CHECK-NEXT:    fmov s3, w11
-; CHECK-NEXT:    fcvtzu w11, d1
-; CHECK-NEXT:    mov w12, v2.s[1]
-; CHECK-NEXT:    mov v0.b[10], w10
-; CHECK-NEXT:    mov v3.s[1], w9
+; CHECK-NEXT:    mov v0.b[8], v3.b[0]
+; CHECK-NEXT:    csel w12, w12, w8, lo
+; CHECK-NEXT:    cmp w9, #255
+; CHECK-NEXT:    mov v5.s[1], w10
+; CHECK-NEXT:    csel w9, w9, w8, lo
+; CHECK-NEXT:    fcvtzu w10, d6
+; CHECK-NEXT:    mov v0.b[9], w11
+; CHECK-NEXT:    fcvtzu w11, d7
+; CHECK-NEXT:    fmov s16, w9
+; CHECK-NEXT:    mov w9, v5.s[1]
+; CHECK-NEXT:    cmp w10, #255
+; CHECK-NEXT:    mov v0.b[10], v5.b[0]
+; CHECK-NEXT:    mov v16.s[1], w12
+; CHECK-NEXT:    mov v0.b[11], w9
+; CHECK-NEXT:    csel w9, w10, w8, lo
 ; CHECK-NEXT:    cmp w11, #255
-; CHECK-NEXT:    csel w9, w11, w8, lo
-; CHECK-NEXT:    cmp w13, #255
-; CHECK-NEXT:    csel w8, w13, w8, lo
-; CHECK-NEXT:    mov v0.b[11], w12
-; CHECK-NEXT:    fmov w10, s3
-; CHECK-NEXT:    fmov s1, w8
-; CHECK-NEXT:    mov w8, v3.s[1]
-; CHECK-NEXT:    mov v0.b[12], w10
-; CHECK-NEXT:    mov v1.s[1], w9
-; CHECK-NEXT:    mov v0.b[13], w8
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov w9, v1.s[1]
-; CHECK-NEXT:    mov v0.b[14], w8
-; CHECK-NEXT:    mov v0.b[15], w9
+; CHECK-NEXT:    mov w10, v16.s[1]
+; CHECK-NEXT:    csel w8, w11, w8, lo
+; CHECK-NEXT:    mov v0.b[12], v16.b[0]
+; CHECK-NEXT:    fmov s6, w8
+; CHECK-NEXT:    mov v0.b[13], w10
+; CHECK-NEXT:    mov v6.s[1], w9
+; CHECK-NEXT:    mov v0.b[14], v6.b[0]
+; CHECK-NEXT:    mov w8, v6.s[1]
+; CHECK-NEXT:    mov v0.b[15], w8
 ; CHECK-NEXT:    ret
     %x = call <16 x i8> @llvm.fptoui.sat.v16f64.v16i8(<16 x double> %f)
     ret <16 x i8> %x
 }
 
 define <8 x i16> @test_unsigned_v8f64_v8i16(<8 x double> %f) {
 ; CHECK-LABEL: test_unsigned_v8f64_v8i16:
 ; CHECK:       // %bb.0:

llvm/test/CodeGen/AArch64/neon-extracttruncate.ll

 
 define <8 x i8> @extract_2_v4i32(<4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: extract_2_v4i32:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    mov w8, v0.s[1]
 ; CHECK-NEXT:    mov w9, v0.s[2]
 ; CHECK-NEXT:    mov w10, v0.s[3]
 ; CHECK-NEXT:    mov v0.b[1], w8
-; CHECK-NEXT:    fmov w8, s1
+; CHECK-NEXT:    mov w8, v1.s[1]
 ; CHECK-NEXT:    mov v0.b[2], w9
-; CHECK-NEXT:    mov w9, v1.s[1]
+; CHECK-NEXT:    mov w9, v1.s[2]
 ; CHECK-NEXT:    mov v0.b[3], w10
-; CHECK-NEXT:    mov v0.b[4], w8
-; CHECK-NEXT:    mov w8, v1.s[2]
-; CHECK-NEXT:    mov v0.b[5], w9
-; CHECK-NEXT:    mov w9, v1.s[3]
-; CHECK-NEXT:    mov v0.b[6], w8
-; CHECK-NEXT:    mov v0.b[7], w9
+; CHECK-NEXT:    mov v0.b[4], v1.b[0]
+; CHECK-NEXT:    mov v0.b[5], w8
+; CHECK-NEXT:    mov w8, v1.s[3]
+; CHECK-NEXT:    mov v0.b[6], w9
+; CHECK-NEXT:    mov v0.b[7], w8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
 entry:
   %a0 = extractelement <4 x i32> %a, i32 0
   %a1 = extractelement <4 x i32> %a, i32 1
   %a2 = extractelement <4 x i32> %a, i32 2
   %a3 = extractelement <4 x i32> %a, i32 3
   %b0 = extractelement <4 x i32> %b, i32 0

llvm/test/CodeGen/AArch64/peephole-insvigpr.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -run-pass=aarch64-mi-peephole-opt -mtriple=aarch64-unknown-linux -verify-machineinstrs -o - %s | FileCheck %s
 --- |
-  source_filename = "/Users/nilanjana/Documents/code/llvm-project/llvm/test/CodeGen/AArch64/tmp.ll"
-
-  ; Function Attrs: nocallback nofree nosync nounwind willreturn memory(none)
-  declare i32 @llvm.aarch64.neon.uaddlv.i32.v8i16(<8 x i16>) #0
-
-  ; Function Attrs: nocallback nofree nosync nounwind willreturn memory(none)
-  declare i64 @llvm.aarch64.neon.uaddlv.i64.v4i32(<4 x i32>) #0
-
   define void @insert_vec_v6i64_uaddlv_from_v4i32(ptr %0) {
   entry:
     ret void
   }
 
   define void @insert_vec_v2i32_uaddlv_from_v8i16(ptr %0) {
   entry:
     ret void
   }
 
   define void @insert_vec_v8i16_uaddlv_from_v8i16(ptr %0) {
   entry:
     ret void
   }
 
   define void @insert_vec_v16i8_uaddlv_from_v4i32(ptr %0) {
   entry:
     ret void
   }
 
   define void @insert_vec_v2i32_uaddlv_from_v8i16_nz_index(ptr %0) {
   entry:
     ret void
   }
 
   ; The optimization is not applicable when the source is not a virtual register
   define void @insert_vec_from_gpr(i32 %v, ptr %p) {
   entry:
     ret void
   }
bb.0.entry:
     ; CHECK: liveins: $x0
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64common = COPY $x0
     ; CHECK-NEXT: [[MOVIv2d_ns:%[0-9]+]]:fpr128 = MOVIv2d_ns 0
     ; CHECK-NEXT: [[UADDLVv4i32v:%[0-9]+]]:fpr64 = UADDLVv4i32v [[MOVIv2d_ns]]
     ; CHECK-NEXT: [[DEF:%[0-9]+]]:fpr128 = IMPLICIT_DEF
     ; CHECK-NEXT: [[INSERT_SUBREG:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF]], killed [[UADDLVv4i32v]], %subreg.dsub
     ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64 = COPY [[INSERT_SUBREG]].dsub
-    ; CHECK-NEXT: [[INSvi64gpr:%[0-9]+]]:fpr128 = INSvi64gpr [[MOVIv2d_ns]], 0, killed [[COPY1]]
+    ; CHECK-NEXT: [[INSvi64lane:%[0-9]+]]:fpr128 = INSvi64lane [[MOVIv2d_ns]], 0, [[INSERT_SUBREG]], 0
     ; CHECK-NEXT: [[MOVID:%[0-9]+]]:fpr64 = MOVID 0
     ; CHECK-NEXT: [[DEF1:%[0-9]+]]:fpr128 = IMPLICIT_DEF
     ; CHECK-NEXT: [[INSERT_SUBREG1:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF1]], killed [[MOVID]], %subreg.dsub
-    ; CHECK-NEXT: [[UCVTFv2f64_:%[0-9]+]]:fpr128 = nofpexcept UCVTFv2f64 killed [[INSvi64gpr]], implicit $fpcr
+    ; CHECK-NEXT: [[UCVTFv2f64_:%[0-9]+]]:fpr128 = nofpexcept UCVTFv2f64 killed [[INSvi64lane]], implicit $fpcr
     ; CHECK-NEXT: [[FCVTNv2i32_:%[0-9]+]]:fpr64 = nofpexcept FCVTNv2i32 killed [[UCVTFv2f64_]], implicit $fpcr
     ; CHECK-NEXT: [[DEF2:%[0-9]+]]:fpr128 = IMPLICIT_DEF
     ; CHECK-NEXT: [[INSERT_SUBREG2:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF2]], killed [[FCVTNv2i32_]], %subreg.dsub
-    ; CHECK-NEXT: [[INSvi64lane:%[0-9]+]]:fpr128 = INSvi64lane [[INSERT_SUBREG2]], 1, killed [[INSERT_SUBREG1]], 0
+    ; CHECK-NEXT: [[INSvi64lane1:%[0-9]+]]:fpr128 = INSvi64lane [[INSERT_SUBREG2]], 1, killed [[INSERT_SUBREG1]], 0
     ; CHECK-NEXT: [[COPY2:%[0-9]+]]:fpr64 = COPY [[MOVIv2d_ns]].dsub
     ; CHECK-NEXT: STRDui killed [[COPY2]], [[COPY]], 2 :: (store (s64) into %ir.0 + 16)
-    ; CHECK-NEXT: STRQui killed [[INSvi64lane]], [[COPY]], 0 :: (store (s128) into %ir.0, align 8)
+    ; CHECK-NEXT: STRQui killed [[INSvi64lane1]], [[COPY]], 0 :: (store (s128) into %ir.0, align 8)
     ; CHECK-NEXT: RET_ReallyLR
     %0:gpr64common = COPY $x0
     %1:fpr128 = MOVIv2d_ns 0
     %2:fpr64 = UADDLVv4i32v %1
     %4:fpr128 = IMPLICIT_DEF
     %3:fpr128 = INSERT_SUBREG %4, killed %2, %subreg.dsub
     %5:gpr64 = COPY %3.dsub
     %7:fpr128 = INSvi64gpr %1, 0, killed %5
     %8:fpr64 = MOVID 0
     %10:fpr128 = IMPLICIT_DEF
     %9:fpr128 = INSERT_SUBREG %10, killed %8, %subreg.dsub
     %11:fpr128 = nofpexcept UCVTFv2f64 killed %7, implicit $fpcr
     %12:fpr64 = nofpexcept FCVTNv2i32 killed %11, implicit $fpcr
     %14:fpr128 = IMPLICIT_DEF
     %13:fpr128 = INSERT_SUBREG %14, killed %12, %subreg.dsub
     %15:fpr128 = INSvi64lane %13, 1, killed %9, 0
     %17:fpr64 = COPY %1.dsub
     STRDui killed %17, %0, 2 :: (store (s64) into %ir.0 + 16)
     STRQui killed %15, %0, 0 :: (store (s128) into %ir.0, align 8)
     RET_ReallyLR
 
 ...
 ---
 name:            insert_vec_v2i32_uaddlv_from_v8i16
 registers:
   - { id: 0, class: gpr64common, preferred-register: '' }
   - { id: 1, class: fpr128, preferred-register: '' }
   - { id: 2, class: fpr32, preferred-register: '' }
   - { id: 3, class: fpr128, preferred-register: '' }
   - { id: 4, class: fpr128, preferred-register: '' }
   - { id: 5, class: gpr32, preferred-register: '' }
   - { id: 6, class: fpr64, preferred-register: '' }
   - { id: 7, class: fpr128, preferred-register: '' }
   - { id: 8, class: fpr128, preferred-register: '' }
   - { id: 9, class: fpr128, preferred-register: '' }
bb.0.entry:
     ; CHECK-NEXT: [[MOVIv2d_ns:%[0-9]+]]:fpr128 = MOVIv2d_ns 0
     ; CHECK-NEXT: [[UADDLVv8i16v:%[0-9]+]]:fpr32 = UADDLVv8i16v killed [[MOVIv2d_ns]]
     ; CHECK-NEXT: [[DEF:%[0-9]+]]:fpr128 = IMPLICIT_DEF
     ; CHECK-NEXT: [[INSERT_SUBREG:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF]], killed [[UADDLVv8i16v]], %subreg.ssub
     ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY [[INSERT_SUBREG]].ssub
     ; CHECK-NEXT: [[MOVID:%[0-9]+]]:fpr64 = MOVID 0
     ; CHECK-NEXT: [[DEF1:%[0-9]+]]:fpr128 = IMPLICIT_DEF
     ; CHECK-NEXT: [[INSERT_SUBREG1:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF1]], killed [[MOVID]], %subreg.dsub
-    ; CHECK-NEXT: [[INSvi32gpr:%[0-9]+]]:fpr128 = INSvi32gpr [[INSERT_SUBREG1]], 0, killed [[COPY1]]
-    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:fpr64 = COPY [[INSvi32gpr]].dsub
+    ; CHECK-NEXT: [[INSvi32lane:%[0-9]+]]:fpr128 = INSvi32lane [[INSERT_SUBREG1]], 0, [[INSERT_SUBREG]], 0
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:fpr64 = COPY [[INSvi32lane]].dsub
     ; CHECK-NEXT: [[UCVTFv2f32_:%[0-9]+]]:fpr64 = nofpexcept UCVTFv2f32 killed [[COPY2]], implicit $fpcr
     ; CHECK-NEXT: STRDui killed [[UCVTFv2f32_]], [[COPY]], 0 :: (store (s64) into %ir.0)
     ; CHECK-NEXT: RET_ReallyLR
     %0:gpr64common = COPY $x0
     %1:fpr128 = MOVIv2d_ns 0
     %2:fpr32 = UADDLVv8i16v killed %1
     %4:fpr128 = IMPLICIT_DEF
     %3:fpr128 = INSERT_SUBREG %4, killed %2, %subreg.ssub
bb.0.entry:
     ; CHECK-NEXT: [[MOVIv2d_ns:%[0-9]+]]:fpr128 = MOVIv2d_ns 0
     ; CHECK-NEXT: [[UADDLVv8i16v:%[0-9]+]]:fpr32 = UADDLVv8i16v killed [[MOVIv2d_ns]]
     ; CHECK-NEXT: [[DEF:%[0-9]+]]:fpr128 = IMPLICIT_DEF
     ; CHECK-NEXT: [[INSERT_SUBREG:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF]], killed [[UADDLVv8i16v]], %subreg.ssub
     ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY [[INSERT_SUBREG]].ssub
     ; CHECK-NEXT: [[MOVID:%[0-9]+]]:fpr64 = MOVID 0
     ; CHECK-NEXT: [[DEF1:%[0-9]+]]:fpr128 = IMPLICIT_DEF
     ; CHECK-NEXT: [[INSERT_SUBREG1:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF1]], killed [[MOVID]], %subreg.dsub
-    ; CHECK-NEXT: [[INSvi16gpr:%[0-9]+]]:fpr128 = INSvi16gpr [[INSERT_SUBREG1]], 0, killed [[COPY1]]
-    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:fpr64 = COPY [[INSvi16gpr]].dsub
+    ; CHECK-NEXT: [[INSvi16lane:%[0-9]+]]:fpr128 = INSvi16lane [[INSERT_SUBREG1]], 0, [[INSERT_SUBREG]], 0
+    ; CHECK-NEXT: [[COPY2:%[0-9]+]]:fpr64 = COPY [[INSvi16lane]].dsub
     ; CHECK-NEXT: [[USHLLv4i16_shift:%[0-9]+]]:fpr128 = USHLLv4i16_shift killed [[COPY2]], 0
     ; CHECK-NEXT: [[UCVTFv4f32_:%[0-9]+]]:fpr128 = nofpexcept UCVTFv4f32 killed [[USHLLv4i16_shift]], implicit $fpcr
     ; CHECK-NEXT: [[COPY3:%[0-9]+]]:gpr32 = COPY $wzr
     ; CHECK-NEXT: STRWui [[COPY3]], [[COPY]], 7 :: (store (s32) into %ir.0 + 28)
     ; CHECK-NEXT: STRWui [[COPY3]], [[COPY]], 6 :: (store (s32) into %ir.0 + 24, align 8)
     ; CHECK-NEXT: STRWui [[COPY3]], [[COPY]], 5 :: (store (s32) into %ir.0 + 20)
     ; CHECK-NEXT: STRWui [[COPY3]], [[COPY]], 4 :: (store (s32) into %ir.0 + 16, align 8)
     ; CHECK-NEXT: STRQui killed [[UCVTFv4f32_]], [[COPY]], 0 :: (store (s128) into %ir.0, align 8)
bb.0.entry:
     ; CHECK-NEXT: [[UADDLVv4i32v:%[0-9]+]]:fpr64 = UADDLVv4i32v [[MOVIv2d_ns]]
     ; CHECK-NEXT: [[DEF:%[0-9]+]]:fpr128 = IMPLICIT_DEF
     ; CHECK-NEXT: [[INSERT_SUBREG:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF]], killed [[UADDLVv4i32v]], %subreg.dsub
     ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr64all = COPY [[INSERT_SUBREG]].dsub
     ; CHECK-NEXT: [[COPY2:%[0-9]+]]:gpr32 = COPY [[COPY1]].sub_32
     ; CHECK-NEXT: [[MOVID:%[0-9]+]]:fpr64 = MOVID 0
     ; CHECK-NEXT: [[DEF1:%[0-9]+]]:fpr128 = IMPLICIT_DEF
     ; CHECK-NEXT: [[INSERT_SUBREG1:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF1]], killed [[MOVID]], %subreg.dsub
-    ; CHECK-NEXT: [[INSvi8gpr:%[0-9]+]]:fpr128 = INSvi8gpr [[INSERT_SUBREG1]], 0, killed [[COPY2]]
-    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:fpr64 = COPY [[INSvi8gpr]].dsub
+    ; CHECK-NEXT: [[INSvi8lane:%[0-9]+]]:fpr128 = INSvi8lane [[INSERT_SUBREG1]], 0, [[INSERT_SUBREG]], 0
+    ; CHECK-NEXT: [[COPY3:%[0-9]+]]:fpr64 = COPY [[INSvi8lane]].dsub
     ; CHECK-NEXT: [[DEF2:%[0-9]+]]:fpr64 = IMPLICIT_DEF
     ; CHECK-NEXT: [[ZIP1v8i8_:%[0-9]+]]:fpr64 = ZIP1v8i8 killed [[COPY3]], killed [[DEF2]]
     ; CHECK-NEXT: [[BICv4i16_:%[0-9]+]]:fpr64 = BICv4i16 [[ZIP1v8i8_]], 255, 8
     ; CHECK-NEXT: [[USHLLv4i16_shift:%[0-9]+]]:fpr128 = USHLLv4i16_shift killed [[BICv4i16_]], 0
     ; CHECK-NEXT: [[UCVTFv4f32_:%[0-9]+]]:fpr128 = nofpexcept UCVTFv4f32 killed [[USHLLv4i16_shift]], implicit $fpcr
     ; CHECK-NEXT: STRQui [[MOVIv2d_ns]], [[COPY]], 3 :: (store (s128) into %ir.0 + 48, align 8)
     ; CHECK-NEXT: STRQui [[MOVIv2d_ns]], [[COPY]], 2 :: (store (s128) into %ir.0 + 32, align 8)
     ; CHECK-NEXT: STRQui [[MOVIv2d_ns]], [[COPY]], 1 :: (store (s128) into %ir.0 + 16, align 8)
bb.0.entry:
     ; CHECK: liveins: $x0
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64common = COPY $x0
     ; CHECK-NEXT: [[MOVIv2d_ns:%[0-9]+]]:fpr128 = MOVIv2d_ns 0
     ; CHECK-NEXT: [[UADDLVv8i16v:%[0-9]+]]:fpr32 = UADDLVv8i16v [[MOVIv2d_ns]]
     ; CHECK-NEXT: [[DEF:%[0-9]+]]:fpr128 = IMPLICIT_DEF
     ; CHECK-NEXT: [[INSERT_SUBREG:%[0-9]+]]:fpr128 = INSERT_SUBREG [[DEF]], killed [[UADDLVv8i16v]], %subreg.ssub
     ; CHECK-NEXT: [[COPY1:%[0-9]+]]:gpr32 = COPY [[INSERT_SUBREG]].ssub
-    ; CHECK-NEXT: [[INSvi32gpr:%[0-9]+]]:fpr128 = INSvi32gpr [[MOVIv2d_ns]], 2, killed [[COPY1]]
-    ; CHECK-NEXT: [[UCVTFv4f32_:%[0-9]+]]:fpr128 = nofpexcept UCVTFv4f32 killed [[INSvi32gpr]], implicit $fpcr
+    ; CHECK-NEXT: [[INSvi32lane:%[0-9]+]]:fpr128 = INSvi32lane [[MOVIv2d_ns]], 2, [[INSERT_SUBREG]], 0
+    ; CHECK-NEXT: [[UCVTFv4f32_:%[0-9]+]]:fpr128 = nofpexcept UCVTFv4f32 killed [[INSvi32lane]], implicit $fpcr
     ; CHECK-NEXT: STRQui killed [[UCVTFv4f32_]], [[COPY]], 0 :: (store (s128) into %ir.0, align 8)
     ; CHECK-NEXT: RET_ReallyLR
     %0:gpr64common = COPY $x0
     %1:fpr128 = MOVIv2d_ns 0
     %2:fpr32 = UADDLVv8i16v %1
     %4:fpr128 = IMPLICIT_DEF
     %3:fpr128 = INSERT_SUBREG %4, killed %2, %subreg.ssub
     %5:gpr32 = COPY %3.ssub

llvm/test/CodeGen/AArch64/sve-fixed-length-masked-gather.ll

 ;
 
 define void @masked_gather_v2f16(ptr %a, ptr %b) vscale_range(2,0) #0 {
 ; CHECK-LABEL: masked_gather_v2f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr s1, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
+; CHECK-NEXT:    ldr q2, [x1]
 ; CHECK-NEXT:    fcmeq v1.4h, v1.4h, #0.0
 ; CHECK-NEXT:    sshll v1.4s, v1.4h, #0
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov w9, v1.s[1]
-; CHECK-NEXT:    ldr q1, [x1]
-; CHECK-NEXT:    mov v0.h[0], w8
-; CHECK-NEXT:    mov v0.h[1], w9
+; CHECK-NEXT:    mov v0.h[0], v1.h[0]
+; CHECK-NEXT:    mov w8, v1.s[1]
+; CHECK-NEXT:    mov v0.h[1], w8
 ; CHECK-NEXT:    shl v0.4h, v0.4h, #15
 ; CHECK-NEXT:    cmlt v0.4h, v0.4h, #0
 ; CHECK-NEXT:    sunpklo z0.s, z0.h
 ; CHECK-NEXT:    sunpklo z0.d, z0.s
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
-; CHECK-NEXT:    ld1h { z0.d }, p0/z, [z1.d]
+; CHECK-NEXT:    ld1h { z0.d }, p0/z, [z2.d]
 ; CHECK-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; CHECK-NEXT:    uzp1 z0.h, z0.h, z0.h
 ; CHECK-NEXT:    str s0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <2 x half>, ptr %a
   %ptrs = load <2 x ptr>, ptr %b
   %mask = fcmp oeq <2 x half> %cval, zeroinitializer
   %vals = call <2 x half> @llvm.masked.gather.v2f16(<2 x ptr> %ptrs, i32 8, <2 x i1> %mask, <2 x half> undef)

llvm/test/CodeGen/AArch64/sve-fixed-length-masked-loads.ll

 ; CHECK-LABEL: masked_load_v2f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr s1, [x0]
 ; CHECK-NEXT:    ptrue p0.h, vl4
 ; CHECK-NEXT:    ldr s2, [x1]
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    fcmeq v1.4h, v1.4h, v2.4h
 ; CHECK-NEXT:    sshll v1.4s, v1.4h, #0
-; CHECK-NEXT:    fmov w8, s1
-; CHECK-NEXT:    mov w9, v1.s[1]
-; CHECK-NEXT:    mov v0.h[0], w8
-; CHECK-NEXT:    mov v0.h[1], w9
+; CHECK-NEXT:    mov v0.h[0], v1.h[0]
+; CHECK-NEXT:    mov w8, v1.s[1]
+; CHECK-NEXT:    mov v0.h[1], w8
 ; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
 ; CHECK-NEXT:    ret
   %a = load <2 x half>, ptr %ap
   %b = load <2 x half>, ptr %bp
   %mask = fcmp oeq <2 x half> %a, %b
   %load = call <2 x half> @llvm.masked.load.v2f16(ptr %ap, i32 8, <2 x i1> %mask, <2 x half> zeroinitializer)

llvm/test/CodeGen/AArch64/sve-fixed-length-masked-scatter.ll

 ; CHECK-LABEL: masked_scatter_v2f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr s1, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    fcmeq v2.4h, v1.4h, #0.0
 ; CHECK-NEXT:    uunpklo z1.s, z1.h
 ; CHECK-NEXT:    sshll v2.4s, v2.4h, #0
-; CHECK-NEXT:    fmov w8, s2
-; CHECK-NEXT:    mov w9, v2.s[1]
+; CHECK-NEXT:    mov v0.h[0], v2.h[0]
+; CHECK-NEXT:    mov w8, v2.s[1]
 ; CHECK-NEXT:    ldr q2, [x1]
-; CHECK-NEXT:    mov v0.h[0], w8
-; CHECK-NEXT:    mov v0.h[1], w9
+; CHECK-NEXT:    mov v0.h[1], w8
 ; CHECK-NEXT:    shl v0.4h, v0.4h, #15
 ; CHECK-NEXT:    cmlt v0.4h, v0.4h, #0
 ; CHECK-NEXT:    sunpklo z0.s, z0.h
 ; CHECK-NEXT:    sunpklo z0.d, z0.s
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
 ; CHECK-NEXT:    uunpklo z0.d, z1.s
 ; CHECK-NEXT:    st1h { z0.d }, p0, [z2.d]
 ; CHECK-NEXT:    ret

llvm/test/CodeGen/AArch64/sve-fixed-length-masked-stores.ll

 ; CHECK-LABEL: masked_store_v2f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr s1, [x0]
 ; CHECK-NEXT:    ptrue p0.h, vl4
 ; CHECK-NEXT:    ldr s2, [x1]
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    fcmeq v2.4h, v1.4h, v2.4h
 ; CHECK-NEXT:    sshll v2.4s, v2.4h, #0
-; CHECK-NEXT:    fmov w8, s2
-; CHECK-NEXT:    mov w9, v2.s[1]
-; CHECK-NEXT:    mov v0.h[0], w8
-; CHECK-NEXT:    mov v0.h[1], w9
+; CHECK-NEXT:    mov v0.h[0], v2.h[0]
+; CHECK-NEXT:    mov w8, v2.s[1]
+; CHECK-NEXT:    mov v0.h[1], w8
 ; CHECK-NEXT:    shl v0.4h, v0.4h, #15
 ; CHECK-NEXT:    cmlt v0.4h, v0.4h, #0
 ; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
 ; CHECK-NEXT:    st1h { z1.h }, p0, [x1]
 ; CHECK-NEXT:    ret
   %a = load <2 x half>, ptr %ap
   %b = load <2 x half>, ptr %bp
   %mask = fcmp oeq <2 x half> %a, %b