[ARM] Fold VPNOT into vpt blocks

[dmgreen]

Summary

We can fold a VPNOT into a predicate block, inverting the predicate to create "else" predicated instructions. This teaches the MVEVPTBlockPass to do that, updating the mask and instruction predicates of instructions following a VPNOT, and removing the VPNOT.

[dmgreen]

I realised that this should probably be checking that the value produced by VPNOT is not used further along in the function. The VPT block "else" will not be updating the value like VPNOT will.

[samparker]

Would be good to see this rebased once D66577 lands.

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

116

nit: return mask % 2 ==1

[dmgreen]

I had to make this check for Kill flags on VPNOT vpr regs, and it has gotten a bit of a rewrite in the process.

[samparker]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

158

I'm struggling to understand the looping and the logic here. Why is a simple walk forward over the instructions not enough? I'd expect to see some conditions being checked and then just break when an instruction no longer meets the criteria. For instance, why can't the checks for kills and defs be performed in the do-loop above?

167

typos: backwoods and guarenteed.

[dmgreen]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

158

I was originally attempting to make it work something like that, but there was too much backtracking to make it work nicely. For example we were storing the LastMI and going back if an VPNOT was at the end of the block. Or if a VPNOT is found, we want to add it to the list of dead instructions, flip the CurrentModeT bit and inverse the "Then" to "Else" condition on all following instructions. Except if the value of VPR.P0 that the VPNOT produced is needed after the VPT block. Then we have to go back to before the last VPNOT, reversing all that we just did.

So instead, this is trying to keep it simpler. Checking what we want to change and then making the change.

The alternative, I think, would be replicating a lot of the code in the VPNOT forward scan, otherwise we would not know how far we had to look for dead vprs.

[samparker]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

158

How about start by iterating backwards and only begin to add VPNOTs once you've found one that kills VPR? The scanning forward suggestions to me that it would be useful to know what's at the end first.

[dmgreen]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

158

Hmm, I'm still not seeing that being simpler than this. If we scan backwards we wouldn't know where the block started! The current structure tries to get the start and end of the block, before doing the adjustments below.

Can you explain more about what you don't like about the current structure? I (very) initially had this getting a list of instructions in a vector (even though that didn't help much so I took it out). We may want in the future expand this to skip out non-predicate non-important instruction, to combine more into the same vpt block.

[samparker]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

158

I don't like not being able to follow a small piece of code :) There's quite a lot of changes to iterators which is challenging my feeble mind, maybe just renaming them would help. But I guess that there's a few things that I don't understand:

• Why do we need to know the start..? It looks like your search begins when you find something that isn't predicated, so why couldn't your search just halt there instead, if traversing backwards?
• What is the first while loop doing on line 145? Why would there be multiple, consecutive, VPNOT instructions?
• It looks like VPNOTs are counted against NumInstrs, but they'll be removed.

[dmgreen]

It turns out that I do really need this to stop the old version from crashing. But I think this patch is trying to do too much, both fixing the bugs and adding VPNOT's at the same time. I've re-done the fixes as D67219, and will get back to this later.

[samparker]

Can this be abandoned now?

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

enum VPTMaskValue {
   TTEE  =  5, // 0b0101
   TTET  =  7, // 0b0111
   TEEE  =  9, // 0b1001
   TEET  = 11, // 0b1011
   TETT  = 13, // 0b1101
   TETE  = 15  // 0b1111
 };
 
+// Add either a T or an E to an existing mask. Returns true if the 4 instruction
+// limit has been reached.
+static unsigned AddOneToVPTMask(unsigned mask, bool T) {
+  switch (mask) {
+  case 0:
+    assert(T && "VPT mask cannot start with an E!");
+    return VPTMaskValue::T;
+  case VPTMaskValue::T:
+    return T ? VPTMaskValue::TT : VPTMaskValue::TE;
+  case VPTMaskValue::TT:
+    return T ? VPTMaskValue::TTT : VPTMaskValue::TTE;
+  case VPTMaskValue::TE:
+    return T ? VPTMaskValue::TET : VPTMaskValue::TEE;
+  case VPTMaskValue::TTT:
+    return T ? VPTMaskValue::TTTT : VPTMaskValue::TTTE;
+  case VPTMaskValue::TTE:
+    return T ? VPTMaskValue::TTET : VPTMaskValue::TTEE;
+  case VPTMaskValue::TET:
+    return T ? VPTMaskValue::TETT : VPTMaskValue::TETE;
+  case VPTMaskValue::TEE:
+    return T ? VPTMaskValue::TEET : VPTMaskValue::TEEE;
+  default:
+    llvm_unreachable("Unexpected VPT mask");
+  }
+}
+
 bool MVEVPTBlock::InsertVPTBlocks(MachineBasicBlock &Block) {
   bool Modified = false;
-  MachineBasicBlock::iterator MBIter = Block.begin();
-  MachineBasicBlock::iterator EndIter = Block.end();
+  MachineBasicBlock::instr_iterator MBIter = Block.instr_begin();
+  MachineBasicBlock::instr_iterator EndIter = Block.instr_end();
 
   while (MBIter != EndIter) {
     MachineInstr *MI = &*MBIter;
     unsigned PredReg = 0;
     DebugLoc dl = MI->getDebugLoc();
 
     ARMVCC::VPTCodes Pred = getVPTInstrPredicate(*MI, PredReg);
 
     // The idea of the predicate is that None, Then and Else are for use when
     // handling assembly language: they correspond to the three possible
     // suffixes "", "t" and "e" on the mnemonic. So when instructions are read
     // from assembly source or disassembled from object code, you expect to see
     // a mixture whenever there's a long VPT block. But in code generation, we
     // hope we'll never generate an Else as input to this pass.
     assert(Pred != ARMVCC::Else && "VPT block pass does not expect Else preds");
 
     if (Pred == ARMVCC::None) {
       ++MBIter;
       continue;
     }
 
-    MachineInstrBuilder MIBuilder =
-        BuildMI(Block, MBIter, dl, TII->get(ARM::MVE_VPST));
+    LLVM_DEBUG(dbgs() << "Creating VPT block for: "; MI->dump());
 
-    MachineBasicBlock::iterator VPSTInsertPos = MIBuilder.getInstr();
-    int VPTInstCnt = 1;
-    ARMVCC::VPTCodes NextPred;
+    // We collect a series of instructions that look like they should also go
+    // into the same VPT block. This can include other predicated instructions
+    // and VPNOTs.
+    int NumInstrs = 1;
+    MachineBasicBlock::instr_iterator LastVPNOT = EndIter;
+    ARMVCC::VPTCodes NextPred = Pred;
 
     do {
-      ++MBIter;
+      // Add any VPNOTs
+      while (++MBIter != EndIter && MBIter->getOpcode() == ARM::MVE_VPNOT)
+        LastVPNOT = MBIter;
+      if (MBIter == EndIter)
+        break;
+
+      // Check the next instruction is still in the vpt block
       NextPred = getVPTInstrPredicate(*MBIter, PredReg);
-    } while (NextPred != ARMVCC::None && NextPred == Pred && ++VPTInstCnt < 4);
+      assert(NextPred != ARMVCC::Else &&
+             "VPT block pass does not expect Else preds");
+    } while (NextPred == Pred && ++NumInstrs <= 4);
+
+    // MBIter now points to one past the end of the block.
+
+    // Now that we have a list of instructions between MI and MBIter, adjust it
+    // to make sure that it is a valid block. Including making sure that any the
+    // vpr from any VPNOTs does not out-live the block. Otherwise we would
+    // leave the wrong value when it is removed and converted to a "else".
+    if (LastVPNOT != EndIter) {
+      bool Killed =
+          std::any_of(std::next(LastVPNOT), MBIter, [](const MachineInstr &MI) {
+            return MI.killsRegister(ARM::VPR) || MI.definesRegister(ARM::VPR);
+          });
+      // If we do not find a kill of VPR in the block reverse backwards to the
+      // VPNOT. This is guaranteed to redef VPR (and won't be included in the
+      // block).
+      if (!Killed)
+        MBIter = LastVPNOT;
+    }
+    // Remove any VPNOTs from the end of the block.
+    while (std::prev(MBIter)->getOpcode() == ARM::MVE_VPNOT)
+      --MBIter;
+
+    // We now have a sequence of instructions between MI and MBIter that we want
+    // to make into a VPTBlock. Scan through again, adjusting predicates and
+    // block masks, removing the VPNOTs.
+    unsigned BlockMask = VPTMaskValue::T;
+    SmallVector<MachineInstr *, 4> DeadInstr;
+    bool CurrentModeT = true; // T=true or E=false, inverted by a VPNOT
+    MachineBasicBlock::instr_iterator BlockIter(MI);
+
+    for (++BlockIter; BlockIter != MBIter; ++BlockIter) {
+      if (BlockIter->getOpcode() == ARM::MVE_VPNOT) {
+        LLVM_DEBUG(dbgs() << "  folding: "; BlockIter->dump());
+        CurrentModeT = !CurrentModeT;
+        DeadInstr.push_back(&*BlockIter);
+      } else {
+        LLVM_DEBUG(dbgs() << "  adding : "; BlockIter->dump());
+        // Update the VPT mask
+        BlockMask = AddOneToVPTMask(BlockMask, CurrentModeT);
+        // And alter the instructions predicate if we are making an "Else"
+        if (!CurrentModeT) {
+          int PIdx = findFirstVPTPredOperandIdx(*BlockIter);
+          assert(PIdx != -1 && "We should have a predicated instruction!");
+          BlockIter->getOperand(PIdx).setImm(ARMVCC::Else);
+        }
+      }
+    }
 
-    switch (VPTInstCnt) {
-    case 1:
-      MIBuilder.addImm(VPTMaskValue::T);
-      break;
-    case 2:
-      MIBuilder.addImm(VPTMaskValue::TT);
-      break;
-    case 3:
-      MIBuilder.addImm(VPTMaskValue::TTT);
-      break;
-    case 4:
-      MIBuilder.addImm(VPTMaskValue::TTTT);
-      break;
-    default:
-      llvm_unreachable("Unexpected number of instruction in a VPT block");
-    };
+    // Now build the new instructions, and finalize the bundle
+    for (auto &I : DeadInstr)
+      I->eraseFromParent();
 
-    MachineInstr *LastMI = &*MBIter;
-    finalizeBundle(Block, VPSTInsertPos.getInstrIterator(),
-                   ++LastMI->getIterator());
+    MachineInstrBuilder MIBuilder =
+        BuildMI(Block, MI, dl, TII->get(ARM::MVE_VPST));
+    MIBuilder.addImm(BlockMask);
 
-    Modified = true;
-    LLVM_DEBUG(dbgs() << "VPT block created for: "; MI->dump());
+    finalizeBundle(Block, MachineBasicBlock::instr_iterator(MIBuilder.getInstr()), MBIter);
 
-    ++MBIter;
+    Modified = true;
   }
   return Modified;
 }
 
 bool MVEVPTBlock::runOnMachineFunction(MachineFunction &Fn) {
   const ARMSubtarget &STI =
       static_cast<const ARMSubtarget &>(Fn.getSubtarget());
 

llvm/test/CodeGen/Thumb2/mve-masked-load.ll

 }
 
 define arm_aapcs_vfpcc i8* @masked_v4i32_preinc(i8* %x, i8* %y, <4 x i32> %a) {
 ; CHECK-LE-LABEL: masked_v4i32_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrwt.u32 q0, [r0, #4]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v4i32_preinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.32 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s32 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrwt.u32 q0, [r0, #4]
-; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <4 x i32>*
   %c = icmp sgt <4 x i32> %a, zeroinitializer
   %1 = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %0, i32 4, <4 x i1> %c, <4 x i32> undef)
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define arm_aapcs_vfpcc i8* @masked_v4i32_postinc(i8* %x, i8* %y, <4 x i32> %a) {
 ; CHECK-LE-LABEL: masked_v4i32_postinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrwt.u32 q0, [r0]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v4i32_postinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.32 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s32 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrwt.u32 q0, [r0]
-; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <4 x i32>*
   %c = icmp sgt <4 x i32> %a, zeroinitializer
   %1 = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %0, i32 4, <4 x i1> %c, <4 x i32> undef)
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 
 
 define arm_aapcs_vfpcc <8 x i16> @masked_v8i16_align4_zero(<8 x i16> *%dest, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8i16_align4_zero:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0]
 ; CHECK-LE-NEXT:    vpsel q0, q0, q1
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8i16_align4_zero:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-BE-NEXT:    vrev64.16 q2, q0
-; CHECK-BE-NEXT:    vrev32.16 q1, q1
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q2, zr
+; CHECK-BE-NEXT:    vrev32.16 q1, q1
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0]
 ; CHECK-BE-NEXT:    vpsel q1, q0, q1
 ; CHECK-BE-NEXT:    vrev64.16 q0, q1
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %l = call <8 x i16> @llvm.masked.load.v8i16.p0v8i16(<8 x i16>* %dest, i32 2, <8 x i1> %c, <8 x i16> zeroinitializer)

 define i8* @masked_v8i16_preinc(i8* %x, i8* %y, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8i16_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vldr d1, [sp]
 ; CHECK-LE-NEXT:    vmov d0, r2, r3
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0, #4]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8i16_preinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vldr d1, [sp]
 ; CHECK-BE-NEXT:    vmov d0, r3, r2
 ; CHECK-BE-NEXT:    vrev64.16 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0, #4]
-; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <8 x i16>*
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %1 = call <8 x i16> @llvm.masked.load.v8i16.p0v8i16(<8 x i16>* %0, i32 4, <8 x i1> %c, <8 x i16> undef)
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 4
   ret i8* %z
 }
 
 define arm_aapcs_vfpcc i8* @masked_v8i16_postinc(i8* %x, i8* %y, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8i16_postinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8i16_postinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.16 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0]
-; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <8 x i16>*
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %1 = call <8 x i16> @llvm.masked.load.v8i16.p0v8i16(<8 x i16>* %0, i32 4, <8 x i1> %c, <8 x i16> undef)
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 4
   ret i8* %z
 }
 
 
 define arm_aapcs_vfpcc <16 x i8> @masked_v16i8_align4_zero(<16 x i8> *%dest, <16 x i8> %a) {
 ; CHECK-LE-LABEL: masked_v16i8_align4_zero:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vcmp.s8 gt, q0, zr
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrbt.u8 q0, [r0]
 ; CHECK-LE-NEXT:    vpsel q0, q0, q1
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v16i8_align4_zero:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-BE-NEXT:    vrev64.8 q2, q0
-; CHECK-BE-NEXT:    vrev32.8 q1, q1
 ; CHECK-BE-NEXT:    vcmp.s8 gt, q2, zr
+; CHECK-BE-NEXT:    vrev32.8 q1, q1
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrbt.u8 q0, [r0]
 ; CHECK-BE-NEXT:    vpsel q1, q0, q1
 ; CHECK-BE-NEXT:    vrev64.8 q0, q1
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %c = icmp sgt <16 x i8> %a, zeroinitializer
   %l = call <16 x i8> @llvm.masked.load.v16i8.p0v16i8(<16 x i8>* %dest, i32 1, <16 x i1> %c, <16 x i8> zeroinitializer)

 }
 
 define arm_aapcs_vfpcc i8* @masked_v16i8_preinc(i8* %x, i8* %y, <16 x i8> %a) {
 ; CHECK-LE-LABEL: masked_v16i8_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s8 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrbt.u8 q0, [r0, #4]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v16i8_preinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.8 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s8 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrbt.u8 q0, [r0, #4]
-; CHECK-BE-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <16 x i8>*
   %c = icmp sgt <16 x i8> %a, zeroinitializer
   %1 = call <16 x i8> @llvm.masked.load.v16i8.p0v16i8(<16 x i8>* %0, i32 4, <16 x i1> %c, <16 x i8> undef)
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 4
   ret i8* %z
 }
 
 define arm_aapcs_vfpcc i8* @masked_v16i8_postinc(i8* %x, i8* %y, <16 x i8> %a) {
 ; CHECK-LE-LABEL: masked_v16i8_postinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s8 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrbt.u8 q0, [r0]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v16i8_postinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.8 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s8 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrbt.u8 q0, [r0]
-; CHECK-BE-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <16 x i8>*
   %c = icmp sgt <16 x i8> %a, zeroinitializer
   %1 = call <16 x i8> @llvm.masked.load.v16i8.p0v16i8(<16 x i8>* %0, i32 4, <16 x i1> %c, <16 x i8> undef)
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 4
   ret i8* %z
 }
 
 
 define arm_aapcs_vfpcc <4 x float> @masked_v4f32_align4_zero(<4 x float> *%dest, <4 x i32> %a) {
 ; CHECK-LE-LABEL: masked_v4f32_align4_zero:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrwt.u32 q0, [r0]
 ; CHECK-LE-NEXT:    vpsel q0, q0, q1
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v4f32_align4_zero:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.32 q2, q0

 }
 
 define arm_aapcs_vfpcc i8* @masked_v4f32_preinc(i8* %x, i8* %y, <4 x i32> %a) {
 ; CHECK-LE-LABEL: masked_v4f32_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrwt.u32 q0, [r0, #4]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v4f32_preinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.32 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s32 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrwt.u32 q0, [r0, #4]
-; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <4 x float>*
   %c = icmp sgt <4 x i32> %a, zeroinitializer
   %1 = call <4 x float> @llvm.masked.load.v4f32.p0v4f32(<4 x float>* %0, i32 4, <4 x i1> %c, <4 x float> undef)
   %2 = bitcast i8* %y to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 4
   ret i8* %z
 }
 
 define arm_aapcs_vfpcc i8* @masked_v4f32_postinc(i8* %x, i8* %y, <4 x i32> %a) {
 ; CHECK-LE-LABEL: masked_v4f32_postinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrwt.u32 q0, [r0]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v4f32_postinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.32 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s32 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrwt.u32 q0, [r0]
-; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <4 x float>*
   %c = icmp sgt <4 x i32> %a, zeroinitializer
   %1 = call <4 x float> @llvm.masked.load.v4f32.p0v4f32(<4 x float>* %0, i32 4, <4 x i1> %c, <4 x float> undef)
   %2 = bitcast i8* %y to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 4
   ret i8* %z
 }
 
 
 define arm_aapcs_vfpcc <8 x half> @masked_v8f16_align4_zero(<8 x half> *%dest, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8f16_align4_zero:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0]
 ; CHECK-LE-NEXT:    vpsel q0, q0, q1
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8f16_align4_zero:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-BE-NEXT:    vrev64.16 q2, q0
-; CHECK-BE-NEXT:    vrev32.16 q1, q1
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q2, zr
+; CHECK-BE-NEXT:    vrev32.16 q1, q1
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0]
 ; CHECK-BE-NEXT:    vpsel q1, q0, q1
 ; CHECK-BE-NEXT:    vrev64.16 q0, q1
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %l = call <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>* %dest, i32 2, <8 x i1> %c, <8 x half> zeroinitializer)

 }
 
 define arm_aapcs_vfpcc i8* @masked_v8f16_preinc(i8* %x, i8* %y, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8f16_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0, #4]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8f16_preinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.16 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0, #4]
-; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <8 x half>*
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %1 = call <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>* %0, i32 4, <8 x i1> %c, <8 x half> undef)
   %2 = bitcast i8* %y to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 4
   ret i8* %z
 }
 
 define arm_aapcs_vfpcc i8* @masked_v8f16_postinc(i8* %x, i8* %y, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8f16_postinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8f16_postinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.16 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0]
-; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <8 x half>*
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %1 = call <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>* %0, i32 4, <8 x i1> %c, <8 x half> undef)
   %2 = bitcast i8* %y to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 4

llvm/test/CodeGen/Thumb2/mve-pred-not.ll

entry:
   %s = select <2 x i1> %o, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %s
 }
 
 define arm_aapcs_vfpcc <4 x i32> @vpnot_v4i1(<4 x i32> %a, <4 x i32> %b, <4 x i32> %c) {
 ; CHECK-LABEL: vpnot_v4i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vcmp.s32 lt, q0, zr
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vpste
 ; CHECK-NEXT:    vcmpt.s32 gt, q1, zr
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vcmpt.i32 eq, q2, zr
+; CHECK-NEXT:    vcmpe.i32 eq, q2, zr
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c1 = icmp slt <4 x i32> %a, zeroinitializer
   %c2 = icmp sgt <4 x i32> %b, zeroinitializer
   %c3 = icmp eq <4 x i32> %c, zeroinitializer
   %o1 = and <4 x i1> %c1, %c2
   %o2 = xor <4 x i1> %o1, <i1 -1, i1 -1, i1 -1, i1 -1>
   %o = and <4 x i1> %c3, %o2
   %s = select <4 x i1> %o, <4 x i32> %a, <4 x i32> %b
   ret <4 x i32> %s
 }

llvm/test/CodeGen/Thumb2/mve-vpt-block.mir

bb.0.entry:
     liveins: $q0, $q1, $q2, $r0
 
     ; CHECK-LABEL: name: test_vminnmq_m_f32_v2
     ; CHECK: liveins: $q0, $q1, $q2, $r0
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14, $noreg
     ; CHECK: BUNDLE implicit-def $p0, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q1, implicit killed $q2, implicit killed $vpr, implicit killed $q0 {
     ; CHECK:   MVE_VPST 8, implicit-def $p0
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
-    ; CHECK:   tBX_RET 14, $noreg, implicit internal $q0
     ; CHECK: }
+    ; CHECK: tBX_RET 14, $noreg, implicit $q0
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-block2.mir

 constants:       []
 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2, $q3, $r0
 
     ; CHECK-LABEL: name: test_vminnmq_m_f32_v2
     ; CHECK: liveins: $q0, $q1, $q2, $q3, $r0
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit-def dead $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q2, implicit killed $q3, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q2, implicit killed $q3, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
     ; CHECK:   MVE_VPST 4, implicit-def $p0
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q3, 1, renamable $vpr, killed renamable $q0
     ; CHECK:   renamable $q1 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q0, killed renamable $q3, 1, killed renamable $vpr, killed renamable $q1
-    ; CHECK:   $q0 = MVE_VORR internal killed $q1, internal killed $q1, 0, $noreg, internal undef $q0
     ; CHECK: }
+    ; CHECK: $q0 = MVE_VORR killed $q1, killed $q1, 0, $noreg, undef $q0
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q3, 1, renamable $vpr, killed renamable $q0
     renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q0, killed renamable $q3, 1, killed renamable $vpr, killed renamable $q1
     $q0 = MVE_VORR killed $q1, killed $q1, 0, $noreg, undef $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-block4.mir

bb.0.entry:
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14, $noreg
     ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $q2, implicit-def $d4, implicit-def $s8, implicit-def $s9, implicit-def $d5, implicit-def $s10, implicit-def $s11, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q2, implicit $q3, implicit $vpr, implicit killed $q0 {
     ; CHECK:   MVE_VPST 1, implicit-def $p0
     ; CHECK:   renamable $q2 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q3, 1, renamable $vpr, undef renamable $q2
     ; CHECK:   renamable $q2 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q2, internal renamable $q2, 1, renamable $vpr, internal undef renamable $q2
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q2, renamable $q3, 1, renamable $vpr, killed renamable $q0
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q0, renamable $q3, 1, renamable $vpr, internal undef renamable $q0
     ; CHECK: }
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q0, implicit killed $q3, implicit killed $vpr, implicit killed $q1 {
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q0, implicit killed $q3, implicit killed $vpr, implicit killed $q1 {
     ; CHECK:   MVE_VPST 8, implicit-def $p0
     ; CHECK:   renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q0, killed renamable $q3, 1, killed renamable $vpr, killed renamable $q1
-    ; CHECK:   $q0 = MVE_VORR internal killed $q1, internal killed $q1, 0, $noreg, undef $q0
     ; CHECK: }
+    ; CHECK: $q0 = MVE_VORR killed $q1, killed $q1, 0, $noreg, undef $q0
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     renamable $q2 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q3, 1, renamable $vpr, undef renamable $q2
     renamable $q2 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q2, 1, renamable $vpr, undef renamable $q2
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q3, 1, renamable $vpr, killed renamable $q0
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q0, renamable $q3, 1, renamable $vpr, undef renamable $q0
     renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q0, killed renamable $q3, 1, killed renamable $vpr, killed renamable $q1
     $q0 = MVE_VORR killed $q1, killed $q1, 0, $noreg, undef $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-block5.mir

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2, $r0
 
     ; CHECK-LABEL: name: test_vminnmq_m_f32_v2
     ; CHECK: liveins: $q0, $q1, $q2, $r0
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14, $noreg
     ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q1, implicit $q2, implicit $vpr, implicit killed $q3, implicit $q0 {
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q1, implicit $q2, implicit $vpr, implicit killed $q3 {
     ; CHECK:   MVE_VPST 4, implicit-def $p0
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     ; CHECK:   renamable $q1 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, internal renamable $q3, 1, renamable $vpr, undef renamable $q1
-    ; CHECK:   $q3 = MVE_VORR $q0, $q0, 0, $noreg, internal undef $q3
     ; CHECK: }
+    ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q1, implicit killed $q2, implicit killed $vpr, implicit killed $q3, implicit killed $q0 {
     ; CHECK:   MVE_VPST 4, implicit-def $p0
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
-    ; CHECK:   tBX_RET 14, $noreg, implicit internal $q0
     ; CHECK: }
+    ; CHECK: tBX_RET 14, $noreg, implicit $q0
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, renamable $q3, 1, renamable $vpr, undef renamable $q1
     $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-block6.mir

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2, $r0, $r1
 
     ; CHECK-LABEL: name: test_vminnmq_m_f32_v2
     ; CHECK: liveins: $q0, $q1, $q2, $r0, $r1
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14, $noreg
     ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $vpr, implicit killed $q1, implicit $q2, implicit killed $vpr, implicit killed $q3, implicit killed $r1 {
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit killed $q1, implicit $q2, implicit killed $vpr, implicit killed $q3 {
     ; CHECK:   MVE_VPST 8, implicit-def $p0
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, killed renamable $vpr, killed renamable $q3
-    ; CHECK:   $vpr = VMSR_P0 killed $r1, 14, $noreg
     ; CHECK: }
+    ; CHECK: $vpr = VMSR_P0 killed $r1, 14, $noreg
     ; CHECK: BUNDLE implicit-def $p0, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q3, implicit killed $q2, implicit killed $vpr, implicit killed $q0 {
     ; CHECK:   MVE_VPST 8, implicit-def $p0
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
-    ; CHECK:   tBX_RET 14, $noreg, implicit internal $q0
     ; CHECK: }
+    ; CHECK: tBX_RET 14, $noreg, implicit $q0
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, killed renamable $vpr, killed renamable $q3
     $vpr = VMSR_P0 killed $r1, 14, $noreg
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-block7.mir

 constants:       []
 body:             |
   ; CHECK-LABEL: name: test_vminnmq_m_f32_v2
   ; CHECK: bb.0.entry:
   ; CHECK:   successors: %bb.1(0x80000000)
   ; CHECK:   liveins: $q0, $q1, $q2, $r0
   ; CHECK:   $vpr = VMSR_P0 killed $r0, 14, $noreg
   ; CHECK:   $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
-  ; CHECK:   BUNDLE implicit-def $p0, implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit-def $vpr, implicit killed $q1, implicit $q2, implicit killed $vpr, implicit killed $q3 {
+  ; CHECK:   BUNDLE implicit-def $p0, implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q1, implicit $q2, implicit $vpr, implicit killed $q3 {
   ; CHECK:     MVE_VPST 4, implicit-def $p0
   ; CHECK:     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
   ; CHECK:     renamable $q1 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, internal renamable $q3, 1, renamable $vpr, undef renamable $q1
-  ; CHECK:     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
   ; CHECK:   }
   ; CHECK: bb.1.bb2:
   ; CHECK:   liveins: $q0, $q1, $q2, $q3, $vpr
   ; CHECK:   BUNDLE implicit-def $p0, implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q1, implicit killed $q2, implicit killed $vpr, implicit killed $q3, implicit killed $q0 {
   ; CHECK:     MVE_VPST 4, implicit-def $p0
   ; CHECK:     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
   ; CHECK:     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
-  ; CHECK:     tBX_RET 14, $noreg, implicit internal $q0
   ; CHECK:   }
+  ; CHECK:   tBX_RET 14, $noreg, implicit $q0
   bb.0.entry:
     liveins: $q0, $q1, $q2, $r0
 
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, renamable $q3, 1, renamable $vpr, undef renamable $q1
-    renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
 
   bb.1.bb2:
     liveins: $q0, $q1, $q2, $q3, $vpr
 
     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-block8.mir

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2, $r0
 
     ; CHECK-LABEL: name: test_vminnmq_m_f32_v2
     ; CHECK: liveins: $q0, $q1, $q2, $r0
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14, $noreg
     ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $vpr, implicit killed $q1, implicit $q2, implicit killed $vpr, implicit killed $q3 {
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit killed $q1, implicit $q2, implicit $vpr, implicit killed $q3 {
     ; CHECK:   MVE_VPST 8, implicit-def $p0
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
-    ; CHECK:   renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     ; CHECK: }
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit-def $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit killed $q3, implicit $vpr, implicit undef $q1, implicit $q0 {
+    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q3, implicit $vpr, implicit undef $q1 {
     ; CHECK:   MVE_VPST 8, implicit-def $p0
     ; CHECK:   renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, renamable $q3, 1, renamable $vpr, undef renamable $q1
-    ; CHECK:   $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     ; CHECK: }
+    ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q1, implicit killed $q2, implicit killed $vpr, implicit killed $q3, implicit killed $q0 {
     ; CHECK:   MVE_VPST 4, implicit-def $p0
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
-    ; CHECK:   tBX_RET 14, $noreg, implicit internal $q0
     ; CHECK: }
+    ; CHECK: tBX_RET 14, $noreg, implicit $q0
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, renamable $q3, 1, renamable $vpr, undef renamable $q1
     $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-nots.mir

liveins:
   - { reg: '$q2', virtual-reg: '' }
 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr, implicit killed $q2 {
+    ; CHECK:   MVE_VPST 12, implicit-def $p0
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-    ; CHECK:   renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
-    ; CHECK: }
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $vpr, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q2, implicit $zr, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
-    ; CHECK:   renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, internal killed renamable $vpr
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 2, internal killed renamable $vpr
     ; CHECK: }
+    ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     tBX_RET 14, $noreg, implicit $q0
 
body:             |
   ; CHECK-LABEL: name: vpnot_end
   ; CHECK: bb.0.entry:
   ; CHECK:   successors: %bb.1(0x80000000)
   ; CHECK:   liveins: $q0, $q1, $q2
   ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
   ; CHECK:   BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr {
   ; CHECK:     MVE_VPST 8, implicit-def $p0
   ; CHECK:     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-  ; CHECK:     renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
   ; CHECK:   }
+  ; CHECK:   renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
   ; CHECK: bb.1.bb2:
   ; CHECK:   liveins: $q0, $q1, $q2, $vpr
-  ; CHECK:   BUNDLE implicit-def $p0, implicit-def dead $vpr, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q2, implicit $zr, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
+  ; CHECK:   BUNDLE implicit-def $p0, implicit-def $vpr, implicit killed $q2, implicit $zr, implicit killed $vpr {
   ; CHECK:     MVE_VPST 8, implicit-def $p0
   ; CHECK:     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
-  ; CHECK:     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, internal killed renamable $vpr
   ; CHECK:   }
+  ; CHECK:   renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
   ; CHECK:   tBX_RET 14, $noreg, implicit $q0
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
 
liveins:
   - { reg: '$q2', virtual-reg: '' }
 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot_two
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr, implicit killed $q2 {
+    ; CHECK:   MVE_VPST 4, implicit-def $p0
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-    ; CHECK:   renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
-    ; CHECK: }
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $vpr, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q2, implicit $zr, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
-    ; CHECK:   renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, internal killed renamable $vpr
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, internal killed renamable $vpr
     ; CHECK: }
+    ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     tBX_RET 14, $noreg, implicit $q0
liveins:
   - { reg: '$q2', virtual-reg: '' }
 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot_lots
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr, implicit killed $q2 {
+    ; CHECK:   MVE_VPST 12, implicit-def $p0
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-    ; CHECK:   renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
-    ; CHECK: }
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $vpr, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q2, implicit $zr, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
-    ; CHECK:   renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, internal killed renamable $vpr
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 2, internal killed renamable $vpr
     ; CHECK: }
+    ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot_first
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $vpr, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q1, implicit $zr, implicit killed $vpr, implicit killed $q2, implicit killed $q0 {
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr, implicit killed $q2 {
     ; CHECK:   MVE_VPST 4, implicit-def $p0
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, internal killed renamable $vpr
-    ; CHECK:   renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, internal killed renamable $vpr
     ; CHECK: }
+    ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     tBX_RET 14, $noreg, implicit $q0
 

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot_many
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr, implicit killed $q2 {
+    ; CHECK:   MVE_VPST 12, implicit-def $p0
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-    ; CHECK:   renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
-    ; CHECK: }
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit killed $q2, implicit $zr, implicit killed $vpr {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
-    ; CHECK:   renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 2, internal killed renamable $vpr
     ; CHECK: }
+    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     tBX_RET 14, $noreg, implicit $q0
 
 ...