[ARM] Fold VPNOT into vpt blocks

[dmgreen]

Summary

We can fold a VPNOT into a predicate block, inverting the predicate to create "else" predicated instructions. This teaches the MVEVPTBlockPass to do that, updating the mask and instruction predicates of instructions following a VPNOT, and removing the VPNOT.

[dmgreen]

I realised that this should probably be checking that the value produced by VPNOT is not used further along in the function. The VPT block "else" will not be updating the value like VPNOT will.

[samparker]

Would be good to see this rebased once D66577 lands.

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

116

nit: return mask % 2 ==1

[dmgreen]

I had to make this check for Kill flags on VPNOT vpr regs, and it has gotten a bit of a rewrite in the process.

[samparker]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

207

I'm struggling to understand the looping and the logic here. Why is a simple walk forward over the instructions not enough? I'd expect to see some conditions being checked and then just break when an instruction no longer meets the criteria. For instance, why can't the checks for kills and defs be performed in the do-loop above?

216

typos: backwoods and guarenteed.

[dmgreen]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

207

I was originally attempting to make it work something like that, but there was too much backtracking to make it work nicely. For example we were storing the LastMI and going back if an VPNOT was at the end of the block. Or if a VPNOT is found, we want to add it to the list of dead instructions, flip the CurrentModeT bit and inverse the "Then" to "Else" condition on all following instructions. Except if the value of VPR.P0 that the VPNOT produced is needed after the VPT block. Then we have to go back to before the last VPNOT, reversing all that we just did.

So instead, this is trying to keep it simpler. Checking what we want to change and then making the change.

The alternative, I think, would be replicating a lot of the code in the VPNOT forward scan, otherwise we would not know how far we had to look for dead vprs.

[samparker]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

207

How about start by iterating backwards and only begin to add VPNOTs once you've found one that kills VPR? The scanning forward suggestions to me that it would be useful to know what's at the end first.

[dmgreen]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

207

Hmm, I'm still not seeing that being simpler than this. If we scan backwards we wouldn't know where the block started! The current structure tries to get the start and end of the block, before doing the adjustments below.

Can you explain more about what you don't like about the current structure? I (very) initially had this getting a list of instructions in a vector (even though that didn't help much so I took it out). We may want in the future expand this to skip out non-predicate non-important instruction, to combine more into the same vpt block.

[samparker]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

207

I don't like not being able to follow a small piece of code :) There's quite a lot of changes to iterators which is challenging my feeble mind, maybe just renaming them would help. But I guess that there's a few things that I don't understand:

• Why do we need to know the start..? It looks like your search begins when you find something that isn't predicated, so why couldn't your search just halt there instead, if traversing backwards?
• What is the first while loop doing on line 145? Why would there be multiple, consecutive, VPNOT instructions?
• It looks like VPNOTs are counted against NumInstrs, but they'll be removed.

[dmgreen]

It turns out that I do really need this to stop the old version from crashing. But I think this patch is trying to do too much, both fixing the bugs and adding VPNOT's at the same time. I've re-done the fixes as D67219, and will get back to this later.

[samparker]

Can this be abandoned now?

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

enum VPTMaskValue {
   TTEE  =  5, // 0b0101
   TTET  =  7, // 0b0111
   TEEE  =  9, // 0b1001
   TEET  = 11, // 0b1011
   TETT  = 13, // 0b1101
   TETE  = 15  // 0b1111
 };
 
+// Add either a T or an E to an existing mask. Returns true if the 4 instruction
+// limit has been reached.
+static int AddOneToVPTMask(int &mask, bool T) {
+  switch (mask) {
+    case 0:
+      assert(T && "VPT mask cannot start with an E!");
+      mask = VPTMaskValue::T;
+      break;
+    case VPTMaskValue::T:
+      mask = T ? VPTMaskValue::TT : VPTMaskValue::TE;
+      break;
+    case VPTMaskValue::TT:
+      mask = T ? VPTMaskValue::TTT : VPTMaskValue::TTE;
+      break;
+    case VPTMaskValue::TE:
+      mask = T ? VPTMaskValue::TET : VPTMaskValue::TEE;
+      break;
+    case VPTMaskValue::TTT:
+      mask = T ? VPTMaskValue::TTTT : VPTMaskValue::TTTE;
+      break;
+    case VPTMaskValue::TTE:
+      mask = T ? VPTMaskValue::TTET : VPTMaskValue::TTEE;
+      break;
+    case VPTMaskValue::TET:
+      mask = T ? VPTMaskValue::TETT : VPTMaskValue::TETE;
+      break;
+    case VPTMaskValue::TEE:
+      mask = T ? VPTMaskValue::TEET : VPTMaskValue::TEEE;
+      break;
+    default:
+      llvm_unreachable("Unexpected VPT mask");
+  }
+
+  if (mask % 2 == 1)
+    return true;
+  return false;
+}
+
 bool MVEVPTBlock::InsertVPTBlocks(MachineBasicBlock &Block) {
   bool Modified = false;
   MachineBasicBlock::iterator MBIter = Block.begin();
   MachineBasicBlock::iterator EndIter = Block.end();
 
   while (MBIter != EndIter) {
     MachineInstr *MI = &*MBIter;
     unsigned PredReg = 0;
while (MBIter != EndIter) {
     // hope we'll never generate an Else as input to this pass.
     assert(Pred != ARMVCC::Else && "VPT block pass does not expect Else preds");
 
     if (Pred == ARMVCC::None) {
       ++MBIter;
       continue;
     }
 
+    LLVM_DEBUG(dbgs() << "Creating VPT block for: "; MI->dump());
     MachineInstrBuilder MIBuilder =
         BuildMI(Block, MBIter, dl, TII->get(ARM::MVE_VPST));
 
     MachineBasicBlock::iterator VPSTInsertPos = MIBuilder.getInstr();
-    int VPTInstCnt = 1;
-    ARMVCC::VPTCodes NextPred;
+    int BlockMask = VPTMaskValue::T;
+    bool CurrentModeT = true; // T=true or E=false, inverted by a VPNOT
+    MachineInstr *LastMI = &*MBIter;
+    SmallVector<MachineInstr*, 4> DeadInstr;
+    bool LastInstInBlock = false;
 
     do {
       ++MBIter;
-      NextPred = getVPTInstrPredicate(*MBIter, PredReg);
-    } while (NextPred != ARMVCC::None && NextPred == Pred && ++VPTInstCnt < 4);
 
-    switch (VPTInstCnt) {
-    case 1:
-      MIBuilder.addImm(VPTMaskValue::T);
-      break;
-    case 2:
-      MIBuilder.addImm(VPTMaskValue::TT);
-      break;
-    case 3:
-      MIBuilder.addImm(VPTMaskValue::TTT);
+      // Look for VPNOTs, inverting the predicate
+      SmallVector<MachineInstr*, 4> Nots;
+      while (MBIter != EndIter && MBIter->getOpcode() == ARM::MVE_VPNOT) {
+        CurrentModeT = !CurrentModeT;
+        Nots.push_back(&*MBIter);
+        ++MBIter;
+      }
+      if (MBIter == EndIter)
         break;
-    case 4:
-      MIBuilder.addImm(VPTMaskValue::TTTT);
+
+      // Check the next instruction is still in the vpt block
+      ARMVCC::VPTCodes NextPred = getVPTInstrPredicate(*MBIter, PredReg);
+      assert(NextPred != ARMVCC::Else && "VPT block pass does not expect Else preds");
+      if (NextPred == ARMVCC::None || NextPred != Pred)
         break;
-    default:
-      llvm_unreachable("Unexpected number of instruction in a VPT block");
-    };
 
-    MachineInstr *LastMI = &*MBIter;
+      // Update the VPT mask
+      LastInstInBlock = AddOneToVPTMask(BlockMask, CurrentModeT);
+      // And alter the instructions predicate if we are making an "Else"
+      if (!CurrentModeT) {
+        int PIdx = findFirstVPTPredOperandIdx(*MBIter);
+        assert(PIdx != -1 && "We should have a predicate instruction!");
+        MBIter->getOperand(PIdx).setImm(ARMVCC::Else);
+      }
+
+      LastMI = &*MBIter;
+      for (auto I : Nots) {
+        LLVM_DEBUG(dbgs() << "  folding VPNOT: "; I->dump());
+        DeadInstr.push_back(I);
+      }
+      LLVM_DEBUG(dbgs() << "  adding to vptblock: "; MBIter->dump());
+    } while (!LastInstInBlock);
+
+
+    for (auto &I : DeadInstr)
+      I->eraseFromParent();
+
+    // Add the correct VPT Block mask to the VPST
+    MIBuilder.addImm(BlockMask);
+
     finalizeBundle(Block, VPSTInsertPos.getInstrIterator(),
                    ++LastMI->getIterator());
 
     Modified = true;
-    LLVM_DEBUG(dbgs() << "VPT block created for: "; MI->dump());
 
+    if (MBIter != EndIter)
       ++MBIter;
   }
   return Modified;
 }
 
 bool MVEVPTBlock::runOnMachineFunction(MachineFunction &Fn) {
   const ARMSubtarget &STI =
       static_cast<const ARMSubtarget &>(Fn.getSubtarget());
 
   if (!STI.isThumb2() || !STI.hasMVEIntegerOps())
     return false;
 
   TII = static_cast<const Thumb2InstrInfo *>(STI.getInstrInfo());
   TRI = STI.getRegisterInfo();
 
   LLVM_DEBUG(dbgs() << "********** ARM MVE VPT BLOCKS **********\n"
                     << "********** Function: " << Fn.getName() << '\n');
 
   bool Modified = false;

llvm/test/CodeGen/Thumb2/mve-masked-load.ll

 }
 
 define arm_aapcs_vfpcc i8* @masked_v4i32_preinc(i8* %x, i8* %y, <4 x i32> %a) {
 ; CHECK-LE-LABEL: masked_v4i32_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrwt.u32 q0, [r0, #4]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v4i32_preinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.32 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s32 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrwt.u32 q0, [r0, #4]
-; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <4 x i32>*
   %c = icmp sgt <4 x i32> %a, zeroinitializer
   %1 = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %0, i32 4, <4 x i1> %c, <4 x i32> undef)
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 define arm_aapcs_vfpcc i8* @masked_v4i32_postinc(i8* %x, i8* %y, <4 x i32> %a) {
 ; CHECK-LE-LABEL: masked_v4i32_postinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrwt.u32 q0, [r0]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v4i32_postinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.32 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s32 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrwt.u32 q0, [r0]
-; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <4 x i32>*
   %c = icmp sgt <4 x i32> %a, zeroinitializer
   %1 = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %0, i32 4, <4 x i1> %c, <4 x i32> undef)
   %2 = bitcast i8* %y to <4 x i32>*
   store <4 x i32> %1, <4 x i32>* %2, align 4
   ret i8* %z
 }
 
 
 
 define arm_aapcs_vfpcc <8 x i16> @masked_v8i16_align4_zero(<8 x i16> *%dest, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8i16_align4_zero:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0]
 ; CHECK-LE-NEXT:    vpsel q0, q0, q1
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8i16_align4_zero:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-BE-NEXT:    vrev64.16 q2, q0
-; CHECK-BE-NEXT:    vrev32.16 q1, q1
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q2, zr
+; CHECK-BE-NEXT:    vrev32.16 q1, q1
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0]
 ; CHECK-BE-NEXT:    vpsel q1, q0, q1
 ; CHECK-BE-NEXT:    vrev64.16 q0, q1
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %l = call <8 x i16> @llvm.masked.load.v8i16.p0v8i16(<8 x i16>* %dest, i32 2, <8 x i1> %c, <8 x i16> zeroinitializer)

 define i8* @masked_v8i16_preinc(i8* %x, i8* %y, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8i16_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vldr d1, [sp]
 ; CHECK-LE-NEXT:    vmov d0, r2, r3
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0, #4]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8i16_preinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vldr d1, [sp]
 ; CHECK-BE-NEXT:    vmov d0, r3, r2
 ; CHECK-BE-NEXT:    vrev64.16 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0, #4]
-; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <8 x i16>*
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %1 = call <8 x i16> @llvm.masked.load.v8i16.p0v8i16(<8 x i16>* %0, i32 4, <8 x i1> %c, <8 x i16> undef)
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 4
   ret i8* %z
 }
 
 define arm_aapcs_vfpcc i8* @masked_v8i16_postinc(i8* %x, i8* %y, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8i16_postinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8i16_postinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.16 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0]
-; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <8 x i16>*
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %1 = call <8 x i16> @llvm.masked.load.v8i16.p0v8i16(<8 x i16>* %0, i32 4, <8 x i1> %c, <8 x i16> undef)
   %2 = bitcast i8* %y to <8 x i16>*
   store <8 x i16> %1, <8 x i16>* %2, align 4
   ret i8* %z
 }
 
 
 define arm_aapcs_vfpcc <16 x i8> @masked_v16i8_align4_zero(<16 x i8> *%dest, <16 x i8> %a) {
 ; CHECK-LE-LABEL: masked_v16i8_align4_zero:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vcmp.s8 gt, q0, zr
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrbt.u8 q0, [r0]
 ; CHECK-LE-NEXT:    vpsel q0, q0, q1
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v16i8_align4_zero:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-BE-NEXT:    vrev64.8 q2, q0
-; CHECK-BE-NEXT:    vrev32.8 q1, q1
 ; CHECK-BE-NEXT:    vcmp.s8 gt, q2, zr
+; CHECK-BE-NEXT:    vrev32.8 q1, q1
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrbt.u8 q0, [r0]
 ; CHECK-BE-NEXT:    vpsel q1, q0, q1
 ; CHECK-BE-NEXT:    vrev64.8 q0, q1
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %c = icmp sgt <16 x i8> %a, zeroinitializer
   %l = call <16 x i8> @llvm.masked.load.v16i8.p0v16i8(<16 x i8>* %dest, i32 1, <16 x i1> %c, <16 x i8> zeroinitializer)

 }
 
 define arm_aapcs_vfpcc i8* @masked_v16i8_preinc(i8* %x, i8* %y, <16 x i8> %a) {
 ; CHECK-LE-LABEL: masked_v16i8_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s8 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrbt.u8 q0, [r0, #4]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v16i8_preinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.8 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s8 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrbt.u8 q0, [r0, #4]
-; CHECK-BE-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <16 x i8>*
   %c = icmp sgt <16 x i8> %a, zeroinitializer
   %1 = call <16 x i8> @llvm.masked.load.v16i8.p0v16i8(<16 x i8>* %0, i32 4, <16 x i1> %c, <16 x i8> undef)
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 4
   ret i8* %z
 }
 
 define arm_aapcs_vfpcc i8* @masked_v16i8_postinc(i8* %x, i8* %y, <16 x i8> %a) {
 ; CHECK-LE-LABEL: masked_v16i8_postinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s8 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrbt.u8 q0, [r0]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v16i8_postinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.8 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s8 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrbt.u8 q0, [r0]
-; CHECK-BE-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrb.8 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <16 x i8>*
   %c = icmp sgt <16 x i8> %a, zeroinitializer
   %1 = call <16 x i8> @llvm.masked.load.v16i8.p0v16i8(<16 x i8>* %0, i32 4, <16 x i1> %c, <16 x i8> undef)
   %2 = bitcast i8* %y to <16 x i8>*
   store <16 x i8> %1, <16 x i8>* %2, align 4
   ret i8* %z
 }
 
 
 define arm_aapcs_vfpcc <4 x float> @masked_v4f32_align4_zero(<4 x float> *%dest, <4 x i32> %a) {
 ; CHECK-LE-LABEL: masked_v4f32_align4_zero:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrwt.u32 q0, [r0]
 ; CHECK-LE-NEXT:    vpsel q0, q0, q1
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v4f32_align4_zero:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.32 q2, q0

 }
 
 define arm_aapcs_vfpcc i8* @masked_v4f32_preinc(i8* %x, i8* %y, <4 x i32> %a) {
 ; CHECK-LE-LABEL: masked_v4f32_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrwt.u32 q0, [r0, #4]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v4f32_preinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.32 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s32 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrwt.u32 q0, [r0, #4]
-; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <4 x float>*
   %c = icmp sgt <4 x i32> %a, zeroinitializer
   %1 = call <4 x float> @llvm.masked.load.v4f32.p0v4f32(<4 x float>* %0, i32 4, <4 x i1> %c, <4 x float> undef)
   %2 = bitcast i8* %y to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 4
   ret i8* %z
 }
 
 define arm_aapcs_vfpcc i8* @masked_v4f32_postinc(i8* %x, i8* %y, <4 x i32> %a) {
 ; CHECK-LE-LABEL: masked_v4f32_postinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s32 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrwt.u32 q0, [r0]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v4f32_postinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.32 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s32 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrwt.u32 q0, [r0]
-; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <4 x float>*
   %c = icmp sgt <4 x i32> %a, zeroinitializer
   %1 = call <4 x float> @llvm.masked.load.v4f32.p0v4f32(<4 x float>* %0, i32 4, <4 x i1> %c, <4 x float> undef)
   %2 = bitcast i8* %y to <4 x float>*
   store <4 x float> %1, <4 x float>* %2, align 4
   ret i8* %z
 }
 
 
 define arm_aapcs_vfpcc <8 x half> @masked_v8f16_align4_zero(<8 x half> *%dest, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8f16_align4_zero:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
+; CHECK-LE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0]
 ; CHECK-LE-NEXT:    vpsel q0, q0, q1
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8f16_align4_zero:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-BE-NEXT:    vrev64.16 q2, q0
-; CHECK-BE-NEXT:    vrev32.16 q1, q1
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q2, zr
+; CHECK-BE-NEXT:    vrev32.16 q1, q1
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0]
 ; CHECK-BE-NEXT:    vpsel q1, q0, q1
 ; CHECK-BE-NEXT:    vrev64.16 q0, q1
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %l = call <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>* %dest, i32 2, <8 x i1> %c, <8 x half> zeroinitializer)

 }
 
 define arm_aapcs_vfpcc i8* @masked_v8f16_preinc(i8* %x, i8* %y, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8f16_preinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0, #4]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8f16_preinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.16 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0, #4]
-; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %z to <8 x half>*
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %1 = call <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>* %0, i32 4, <8 x i1> %c, <8 x half> undef)
   %2 = bitcast i8* %y to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 4
   ret i8* %z
 }
 
 define arm_aapcs_vfpcc i8* @masked_v8f16_postinc(i8* %x, i8* %y, <8 x i16> %a) {
 ; CHECK-LE-LABEL: masked_v8f16_postinc:
 ; CHECK-LE:       @ %bb.0: @ %entry
 ; CHECK-LE-NEXT:    vcmp.s16 gt, q0, zr
 ; CHECK-LE-NEXT:    vpst
 ; CHECK-LE-NEXT:    vldrht.u16 q0, [r0]
-; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    adds r0, #4
+; CHECK-LE-NEXT:    vstrw.32 q0, [r1]
 ; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: masked_v8f16_postinc:
 ; CHECK-BE:       @ %bb.0: @ %entry
 ; CHECK-BE-NEXT:    vrev64.16 q1, q0
 ; CHECK-BE-NEXT:    vcmp.s16 gt, q1, zr
 ; CHECK-BE-NEXT:    vpst
 ; CHECK-BE-NEXT:    vldrht.u16 q0, [r0]
-; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    adds r0, #4
+; CHECK-BE-NEXT:    vstrh.16 q0, [r1]
 ; CHECK-BE-NEXT:    bx lr
 entry:
   %z = getelementptr inbounds i8, i8* %x, i32 4
   %0 = bitcast i8* %x to <8 x half>*
   %c = icmp sgt <8 x i16> %a, zeroinitializer
   %1 = call <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>* %0, i32 4, <8 x i1> %c, <8 x half> undef)
   %2 = bitcast i8* %y to <8 x half>*
   store <8 x half> %1, <8 x half>* %2, align 4

llvm/test/CodeGen/Thumb2/mve-pred-not.ll

entry:
   %s = select <2 x i1> %o, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %s
 }
 
 define arm_aapcs_vfpcc <4 x i32> @vpnot_v4i1(<4 x i32> %a, <4 x i32> %b, <4 x i32> %c) {
 ; CHECK-LABEL: vpnot_v4i1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vcmp.s32 lt, q0, zr
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vpste
 ; CHECK-NEXT:    vcmpt.s32 gt, q1, zr
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vcmpt.i32 eq, q2, zr
+; CHECK-NEXT:    vcmpe.i32 eq, q2, zr
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c1 = icmp slt <4 x i32> %a, zeroinitializer
   %c2 = icmp sgt <4 x i32> %b, zeroinitializer
   %c3 = icmp eq <4 x i32> %c, zeroinitializer
   %o1 = and <4 x i1> %c1, %c2
   %o2 = xor <4 x i1> %o1, <i1 -1, i1 -1, i1 -1, i1 -1>
   %o = and <4 x i1> %c3, %o2
   %s = select <4 x i1> %o, <4 x i32> %a, <4 x i32> %b
   ret <4 x i32> %s
 }

llvm/test/CodeGen/Thumb2/mve-vpt-block.mir

bb.0.entry:
     liveins: $q0, $q1, $q2, $r0
 
     ; CHECK-LABEL: name: test_vminnmq_m_f32_v2
     ; CHECK: liveins: $q0, $q1, $q2, $r0
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14, $noreg
     ; CHECK: BUNDLE implicit-def $p0, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q1, implicit killed $q2, implicit killed $vpr, implicit killed $q0 {
     ; CHECK:   MVE_VPST 8, implicit-def $p0
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
-    ; CHECK:   tBX_RET 14, $noreg, implicit internal $q0
     ; CHECK: }
+    ; CHECK: tBX_RET 14, $noreg, implicit $q0
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-block2.mir

 constants:       []
 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2, $q3, $r0
 
     ; CHECK-LABEL: name: test_vminnmq_m_f32_v2
     ; CHECK: liveins: $q0, $q1, $q2, $q3, $r0
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit-def dead $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q2, implicit killed $q3, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q2, implicit killed $q3, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
     ; CHECK:   MVE_VPST 4, implicit-def $p0
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q3, 1, renamable $vpr, killed renamable $q0
     ; CHECK:   renamable $q1 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q0, killed renamable $q3, 1, killed renamable $vpr, killed renamable $q1
-    ; CHECK:   $q0 = MVE_VORR internal killed $q1, internal killed $q1, 0, $noreg, internal undef $q0
     ; CHECK: }
+    ; CHECK: $q0 = MVE_VORR killed $q1, killed $q1, 0, $noreg, undef $q0
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q3, 1, renamable $vpr, killed renamable $q0
     renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q0, killed renamable $q3, 1, killed renamable $vpr, killed renamable $q1
     $q0 = MVE_VORR killed $q1, killed $q1, 0, $noreg, undef $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-block4.mir

bb.0.entry:
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14, $noreg
     ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $q2, implicit-def $d4, implicit-def $s8, implicit-def $s9, implicit-def $d5, implicit-def $s10, implicit-def $s11, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q2, implicit $q3, implicit $vpr, implicit killed $q0 {
     ; CHECK:   MVE_VPST 1, implicit-def $p0
     ; CHECK:   renamable $q2 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q3, 1, renamable $vpr, undef renamable $q2
     ; CHECK:   renamable $q2 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q2, internal renamable $q2, 1, renamable $vpr, internal undef renamable $q2
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q2, renamable $q3, 1, renamable $vpr, killed renamable $q0
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q0, renamable $q3, 1, renamable $vpr, internal undef renamable $q0
     ; CHECK: }
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q0, implicit killed $q3, implicit killed $vpr, implicit killed $q1 {
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q0, implicit killed $q3, implicit killed $vpr, implicit killed $q1 {
     ; CHECK:   MVE_VPST 8, implicit-def $p0
     ; CHECK:   renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q0, killed renamable $q3, 1, killed renamable $vpr, killed renamable $q1
-    ; CHECK:   $q0 = MVE_VORR internal killed $q1, internal killed $q1, 0, $noreg, undef $q0
     ; CHECK: }
+    ; CHECK: $q0 = MVE_VORR killed $q1, killed $q1, 0, $noreg, undef $q0
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     renamable $q2 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q3, 1, renamable $vpr, undef renamable $q2
     renamable $q2 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q2, 1, renamable $vpr, undef renamable $q2
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q2, renamable $q3, 1, renamable $vpr, killed renamable $q0
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q0, renamable $q3, 1, renamable $vpr, undef renamable $q0
     renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q0, killed renamable $q3, 1, killed renamable $vpr, killed renamable $q1
     $q0 = MVE_VORR killed $q1, killed $q1, 0, $noreg, undef $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-block5.mir

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2, $r0
 
     ; CHECK-LABEL: name: test_vminnmq_m_f32_v2
     ; CHECK: liveins: $q0, $q1, $q2, $r0
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14, $noreg
     ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q1, implicit $q2, implicit $vpr, implicit killed $q3, implicit $q0 {
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q1, implicit $q2, implicit $vpr, implicit killed $q3 {
     ; CHECK:   MVE_VPST 4, implicit-def $p0
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     ; CHECK:   renamable $q1 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, internal renamable $q3, 1, renamable $vpr, undef renamable $q1
-    ; CHECK:   $q3 = MVE_VORR $q0, $q0, 0, $noreg, internal undef $q3
     ; CHECK: }
+    ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q1, implicit killed $q2, implicit killed $vpr, implicit killed $q3, implicit killed $q0 {
     ; CHECK:   MVE_VPST 4, implicit-def $p0
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
-    ; CHECK:   tBX_RET 14, $noreg, implicit internal $q0
     ; CHECK: }
+    ; CHECK: tBX_RET 14, $noreg, implicit $q0
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, renamable $q3, 1, renamable $vpr, undef renamable $q1
     $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-block6.mir

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2, $r0, $r1
 
     ; CHECK-LABEL: name: test_vminnmq_m_f32_v2
     ; CHECK: liveins: $q0, $q1, $q2, $r0, $r1
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14, $noreg
     ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $vpr, implicit killed $q1, implicit $q2, implicit killed $vpr, implicit killed $q3, implicit killed $r1 {
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit killed $q1, implicit $q2, implicit killed $vpr, implicit killed $q3 {
     ; CHECK:   MVE_VPST 8, implicit-def $p0
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, killed renamable $vpr, killed renamable $q3
-    ; CHECK:   $vpr = VMSR_P0 killed $r1, 14, $noreg
     ; CHECK: }
+    ; CHECK: $vpr = VMSR_P0 killed $r1, 14, $noreg
     ; CHECK: BUNDLE implicit-def $p0, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q3, implicit killed $q2, implicit killed $vpr, implicit killed $q0 {
     ; CHECK:   MVE_VPST 8, implicit-def $p0
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
-    ; CHECK:   tBX_RET 14, $noreg, implicit internal $q0
     ; CHECK: }
+    ; CHECK: tBX_RET 14, $noreg, implicit $q0
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, killed renamable $vpr, killed renamable $q3
     $vpr = VMSR_P0 killed $r1, 14, $noreg
     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
     tBX_RET 14, $noreg, implicit $q0
 
 ...

llvm/test/CodeGen/Thumb2/mve-vpt-block7.mir

 constants:       []
 body:             |
   ; CHECK-LABEL: name: test_vminnmq_m_f32_v2
   ; CHECK: bb.0.entry:
   ; CHECK:   successors: %bb.1(0x80000000)
   ; CHECK:   liveins: $q0, $q1, $q2, $r0
   ; CHECK:   $vpr = VMSR_P0 killed $r0, 14, $noreg
   ; CHECK:   $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
-  ; CHECK:   BUNDLE implicit-def $p0, implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit-def $vpr, implicit killed $q1, implicit $q2, implicit killed $vpr, implicit killed $q3 {
+  ; CHECK:   BUNDLE implicit-def $p0, implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $q1, implicit $q2, implicit $vpr, implicit killed $q3 {
   ; CHECK:     MVE_VPST 4, implicit-def $p0
   ; CHECK:     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
   ; CHECK:     renamable $q1 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, internal renamable $q3, 1, renamable $vpr, undef renamable $q1
-  ; CHECK:     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
   ; CHECK:   }
+  ; CHECK:   renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
   ; CHECK: bb.1.bb2:
   ; CHECK:   liveins: $q0, $q1, $q2, $q3, $vpr
   ; CHECK:   BUNDLE implicit-def $p0, implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q1, implicit killed $q2, implicit killed $vpr, implicit killed $q3, implicit killed $q0 {
   ; CHECK:     MVE_VPST 4, implicit-def $p0
   ; CHECK:     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
   ; CHECK:     renamable $q0 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
-  ; CHECK:     tBX_RET 14, $noreg, implicit internal $q0
   ; CHECK:   }
+  ; CHECK:   tBX_RET 14, $noreg, implicit $q0
   bb.0.entry:
     liveins: $q0, $q1, $q2, $r0
 
     $vpr = VMSR_P0 killed $r0, 14, $noreg
     $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, renamable $q3, 1, renamable $vpr, undef renamable $q1
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg

llvm/test/CodeGen/Thumb2/mve-vpt-nots.mir

liveins:
   - { reg: '$q2', virtual-reg: '' }
 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr, implicit killed $q2 {
+    ; CHECK:   MVE_VPST 12, implicit-def $p0
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-    ; CHECK:   renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
-    ; CHECK: }
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $vpr, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q2, implicit $zr, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
-    ; CHECK:   renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, internal killed renamable $vpr
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 2, internal killed renamable $vpr
     ; CHECK: }
+    ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     tBX_RET 14, $noreg, implicit $q0
 
body:             |
   ; CHECK-LABEL: name: vpnot_end
   ; CHECK: bb.0.entry:
   ; CHECK:   successors: %bb.1(0x80000000)
   ; CHECK:   liveins: $q0, $q1, $q2
   ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
   ; CHECK:   BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr {
   ; CHECK:     MVE_VPST 8, implicit-def $p0
   ; CHECK:     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-  ; CHECK:     renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
   ; CHECK:   }
+  ; CHECK:   renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
   ; CHECK: bb.1.bb2:
   ; CHECK:   liveins: $q0, $q1, $q2, $vpr
-  ; CHECK:   BUNDLE implicit-def $p0, implicit-def dead $vpr, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q2, implicit $zr, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
+  ; CHECK:   BUNDLE implicit-def $p0, implicit-def $vpr, implicit killed $q2, implicit $zr, implicit killed $vpr {
   ; CHECK:     MVE_VPST 8, implicit-def $p0
   ; CHECK:     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
-  ; CHECK:     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, internal killed renamable $vpr
   ; CHECK:   }
+  ; CHECK:   renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
   ; CHECK:   tBX_RET 14, $noreg, implicit $q0
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
 
liveins:
   - { reg: '$q2', virtual-reg: '' }
 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot_two
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr, implicit killed $q2 {
+    ; CHECK:   MVE_VPST 4, implicit-def $p0
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-    ; CHECK:   renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
-    ; CHECK: }
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $vpr, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q2, implicit $zr, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
-    ; CHECK:   renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, internal killed renamable $vpr
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, internal killed renamable $vpr
     ; CHECK: }
+    ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     tBX_RET 14, $noreg, implicit $q0
liveins:
   - { reg: '$q2', virtual-reg: '' }
 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot_lots
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr, implicit killed $q2 {
+    ; CHECK:   MVE_VPST 12, implicit-def $p0
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-    ; CHECK:   renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
-    ; CHECK: }
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $vpr, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q2, implicit $zr, implicit killed $vpr, implicit killed $q0, implicit killed $q1 {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
-    ; CHECK:   renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, internal killed renamable $vpr
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 2, internal killed renamable $vpr
     ; CHECK: }
+    ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot_first
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def dead $vpr, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $q1, implicit $zr, implicit killed $vpr, implicit killed $q2, implicit killed $q0 {
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr, implicit killed $q2 {
     ; CHECK:   MVE_VPST 4, implicit-def $p0
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, internal killed renamable $vpr
-    ; CHECK:   renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, internal killed renamable $vpr
     ; CHECK: }
+    ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     tBX_RET 14, $noreg, implicit $q0
 

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot_many
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
+    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit $q1, implicit $zr, implicit killed $vpr, implicit killed $q2 {
+    ; CHECK:   MVE_VPST 12, implicit-def $p0
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-    ; CHECK:   renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
-    ; CHECK: }
-    ; CHECK: BUNDLE implicit-def $p0, implicit-def $vpr, implicit killed $q2, implicit $zr, implicit killed $vpr {
-    ; CHECK:   MVE_VPST 8, implicit-def $p0
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
-    ; CHECK:   renamable $vpr = MVE_VPNOT internal killed renamable $vpr, 0, $noreg
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 2, internal killed renamable $vpr
     ; CHECK: }
+    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     tBX_RET 14, $noreg, implicit $q0
 
 ...