[Target][ARM] Improvements to the VPT Block Insertion Pass

[Pierre-vh]

Summary

This patch adds a feature to the MVE VPT block insertion patch: It can now remove VPNOT instructions in some circumstances in order to place the instructions predicated by the VPNOT in an else block instead; This results in more compact code.

For example, this pass used to generate this kind of assembly:

vldrw.u32	q1, [r5]
vpt.s32	ge, q1, r2                      ; Added by MVE-VPT block insertion pass
vcmpt.s32	le, q1, r3
vpnot
vpst	                                ; Added by MVE-VPT block insertion pass
vstrwt.32	q0, [r5], #16

Now, when the VPNOT's result (stored in VPR) is not needed, and when the above block has enough room, it'll generate this instead:

vldrw.u32	q1, [r5]
vpte.s32	ge, q1, r2                     ; Added by MVE-VPT block insertion pass - Notice the "te" instead of "t"
vcmpt.s32	le, q1, r3
vstrwe.32	q0, [r5], #16       ; "t" changed to "e", indicating that this instruction is now part of the "else"

This is much shorter and should be more efficient, and the pass will only remove the VPNOT when VPR is used+killed by one instruction or when VPR is written to in its block.
This should be enough to avoid losing the result of a VPNOT if it's needed, however I'm open to suggestions as I don't actually know if it's good enough.

[Pierre-vh]

Running clang-format on the file.

(My IDE was using its own settings instead of clang-format, this is why the style was incorrect)

[dmgreen]

Good to see this getting updated.

Does this handle predicates like TET/TEET/etc, where we go back to a Then predicate? Any instruction that defs the VPR should reset us back to a Then (so a VCMP will reset the predicate to Then for subsequent instructions in the block). Obviously a VPNOT can flip back to Then too.

Can you make sure we have tests for those situations? Best bet nowadays would probably be to use instrinsics to create some interesting testcases.

[Pierre-vh]

In D75993#1918007, @dmgreen wrote:

Good to see this getting updated.

Does this handle predicates like TET/TEET/etc, where we go back to a Then predicate? Any instruction that defs the VPR should reset us back to a Then (so a VCMP will reset the predicate to Then for subsequent instructions in the block). Obviously a VPNOT can flip back to Then too.

Can you make sure we have tests for those situations? Best bet nowadays would probably be to use instrinsics to create some interesting testcases.

This was not a goal for this initial patch, I just focused on handling else blocks.
However, this is certainly something I can do if you want, but it should it be part of this patch or another patch?

[Pierre-vh]

Updated the diff to fix a bug: The pass refused to generate vpsteee blocks because the condition in SkipPredicatedInstrs was wrong (It needed to do one more iteration when MaxSkips == 0, and then stop).

The pass can now correctly generate this kind of assembly:

	vpsteee	
	vcmpt.s32	le, q1, r12
	vcmpe.s32	lt, q1, r3
	vstrwe.32	q0, [r6], #16
	vstrwe.32	q0, [r4], #16

[Pierre-vh]

This revision will need significant changes as we found a few issues with it.

[Pierre-vh]

Here is the refactored version of this patch. I tried to make the structure a bit cleaner and easier to work on.

• Made the function that expands the block masks more generic, it can now add one or more T/Es to any mask.
• Split the code into multiple functions

Note that I still need to add tests for the new supported cases, I'll try to add them later today or on Monday.

[Pierre-vh]

Once again it needs a bit of work before being ready for review. I need to add support for TETE masks and add a few tests for this improved pass.

[Pierre-vh]

• Added support for TETE masks to the pass
• Added multiple MIR tests for every possible mask that can be generated by the pass
• Added a few LLVM IR tests, unfortunately this does not cover every possible cases as spilling prevents other masks from being generated. I'll look into how this can be improved as a separate patch.

I think this should be ready for review.

[Pierre-vh]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

296

This shouldn't be here and has been fixed locally.
However, I won't send a new diff just for a single line, so I'll push it when I have more meaningful changes to accompany it.

[Pierre-vh]

Another patch to fix a bug - the predicate was not switching back to T on the second VPNOT found.

[Pierre-vh]

• The BlockMask enum is now an enum-class, so I had to make some changes to the pass.

[dmgreen]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

97

Is this only called with a Count of 1 now? If so, can it be simplified.

101

Add a message for the assert.

134

Maybe name this "AddPredicatedInstruction", as we conceptually "adding" them to the block, as opposed to skipping them.

161

Can this use killsRegister?

177

Add a message to the assert.

183–184

This doesn't look like it needed to change? Unless the line after is too long too?

192–249

Why use a uint64_t?

201

Was VPTThenInstCnt renamed to something else? BlockSize?

248–251

Maybe just use:

CurrentPredicate = CurrentPredicate == ARMVCC::Then ? ARMVCC::Else : ARMVCC::Then;

[Pierre-vh]

(I've marked some comments as Done because I changed those locally, I'll update the diff once the review is complete (2 comments left))

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

134

I named the function "Skip" because it advances the iterator (it skips the predicated instruction in front of the iterator), this function doesn't really "care" about the block as a whole, so for me calling it "AddPredicatedInstructions" doesn't make sense.

183–184

You are right, it doesn't need to change. It was probably changed by mistake by me or clang-format.

192–249

It's the type of the argument of addImm below.
Of course I can also use unsigned if you prefer, or I can change this to ARM::PredBlockMask and convert to uint64_t when calling addImm.

201

It's indeed "BlockSize" now, I'll change this comment.

[dmgreen]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

134

Hmm. "Skip" implied to me that we would not end up doing anything with these instructions. Like we would skip instructions between VPT blocks that are not predicated. We just leave them alone.

How about "StepOver" instead?

192–249

unsigned is probably fine, it will fit in either case. If the printing is still simple, PredBlockMask would probably be best. Up to you.

[Pierre-vh]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

134

StepOverPredicatedInstrs is good I think. I'll change it.

192–249

I'll do something like this then:

ARM::PredBlockMask BlockMask = CreateVPTBlock(MBIter, EndIter, DeadInstructions);
// ...
      MIBuilder.addImm((uint64_t)BlockMask);
// ...
      MIBuilder.addImm((uint64_t)BlockMask);

[dmgreen]

LGTM

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

230

Find the register -> Find the operand holding the predicate

[Pierre-vh]

The patch had already been approved, but needed to be rebased.
Only llvm/test/CodeGen/Thumb2/mve-pred-threshold.ll caused a merge conflict and needed to be re-updated with utils/update_llc_test_checks.py, so it has changed a bit.

[Pierre-vh]

Please re-review the llvm/test/CodeGen/Thumb2/mve-pred-threshold.ll. Everything else should be good as they didn't change, they were only rebased.

[dmgreen]

Oh yeah. Because we are now sinking float splats. Still LGTM.

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

 #include <cassert>
 #include <new>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "arm-mve-vpt"
 
 namespace {
-  class MVEVPTBlock : public MachineFunctionPass {
-  public:
-    static char ID;
-    const Thumb2InstrInfo *TII;
-    const TargetRegisterInfo *TRI;
 
-    MVEVPTBlock() : MachineFunctionPass(ID) {}
 
-    bool runOnMachineFunction(MachineFunction &Fn) override;
 
-    MachineFunctionProperties getRequiredProperties() const override {
-      return MachineFunctionProperties().set(
-          MachineFunctionProperties::Property::NoVRegs);
-    }
 
-    StringRef getPassName() const override {
-      return "MVE VPT block insertion pass";
-    }
 
-  private:
-    bool InsertVPTBlocks(MachineBasicBlock &MBB);
-  };
 
-  char MVEVPTBlock::ID = 0;
 
 } // end anonymous namespace
 
 INITIALIZE_PASS(MVEVPTBlock, DEBUG_TYPE, "ARM MVE VPT block pass", false, false)
 
 static MachineInstr *findVCMPToFoldIntoVPST(MachineBasicBlock::iterator MI,
                                             const TargetRegisterInfo *TRI,
                                             unsigned &NewOpcode) {
if (registerDefinedBetween(CmpMI->getOperand(1).getReg(), std::next(CmpMI),
                              MI, TRI))
     return nullptr;
   if (registerDefinedBetween(CmpMI->getOperand(2).getReg(), std::next(CmpMI),
                              MI, TRI))
     return nullptr;
   return &*CmpMI;
 }
 
+// Adds "elses" to a T, TT or TTT blockmask.
+static uint64_t AddElsesToARMVPTBlockMask(uint64_t BlockMask,
+                                          unsigned NumElses) {
+  static uint64_t Table[3][3] = {
+      /*T  */ {ARMVCC::TE, ARMVCC::TEE, ARMVCC::TEEE},
+      /*TT */ {ARMVCC::TTE, ARMVCC::TTEE, 0},
+      /*TTT*/ {ARMVCC::TTTE, 0, 0},
+  };
+  assert(NumElses != 0);
+  switch (BlockMask) {
+  case ARMVCC::T:
+    assert(NumElses <= 3 && "Cannot add more than 3 elses to a T BlockMask - "
+                            "Max BlockMask size is 4");
+    return Table[0][NumElses - 1];
+  case ARMVCC::TT:
+    assert(NumElses <= 2 && "Cannot add more than 2 elses to a TT BlockMask - "
+                            "Max BlockMask size is 4");
+    return Table[1][NumElses - 1];
+  case ARMVCC::TTT:
+    assert(NumElses == 1 && "Cannot add more than 1 else to a T BlockMask - "
+                            "Max BlockMask size is 4");
+    return Table[2][NumElses - 1];
+  default:
+    llvm_unreachable("Unsupported BlockMask");
+  }
+}
+
+// Skips a block of predicated instructions.
+// Returns true if this successfully skipped the whole block of predicated
+// instructions. Returns false when it stopped early (due to MaxSkips), or if
+// MBIter didn't point to a predicated instruction.
+static bool SkipPredicatedInstrs(MachineBasicBlock &Block,
+                                 MachineBasicBlock::instr_iterator &MBIter,
+                                 ARMVCC::VPTCodes Pred, unsigned MaxSkips,
+                                 unsigned &NumInstSkipped) {
+  MachineBasicBlock::instr_iterator EndIter = Block.instr_end();
+  ARMVCC::VPTCodes NextPred = ARMVCC::None;
+  unsigned PredReg;
+  NumInstSkipped = 0;
+
+  while (MBIter != EndIter) {
+    NextPred = getVPTInstrPredicate(*MBIter, PredReg);
+    assert(NextPred != ARMVCC::Else &&
+           "VPT block pass does not expect Else preds");
+    if ((NextPred != Pred) || (MaxSkips == 0))
+      break;
+    --MaxSkips;
+    ++MBIter;
+    ++NumInstSkipped;
+  };
+
+  return NumInstSkipped != 0 && (NextPred == ARMVCC::None || MBIter == EndIter);
+}
+
+// Returns true if we can safely optimize the block following a VPNOT.
+//
+// This considers that a VPNOT can be safely removed when at least one of the
+// instructions in its block defines VPR, or uses+kills it.
+static bool CanOptimizeVPNOTBlock(MachineBasicBlock::iterator Iter,
+                                  MachineBasicBlock::iterator End) {
+  for (; Iter != End; ++Iter)
+    if (Iter->definesRegister(ARM::VPR) ||
+        Iter->findRegisterUseOperandIdx(ARM::VPR, true) != -1)
+      return true;
+  return false;
+}
+
 bool MVEVPTBlock::InsertVPTBlocks(MachineBasicBlock &Block) {
   bool Modified = false;
   MachineBasicBlock::instr_iterator MBIter = Block.instr_begin();
   MachineBasicBlock::instr_iterator EndIter = Block.instr_end();
 
+  SmallVector<MachineInstr *, 4> DeadInstructions;
+
   while (MBIter != EndIter) {
     MachineInstr *MI = &*MBIter;
     unsigned PredReg = 0;
-    DebugLoc dl = MI->getDebugLoc();
+    DebugLoc DL = MI->getDebugLoc();
 
     ARMVCC::VPTCodes Pred = getVPTInstrPredicate(*MI, PredReg);
 
     // The idea of the predicate is that None, Then and Else are for use when
     // handling assembly language: they correspond to the three possible
     // suffixes "", "t" and "e" on the mnemonic. So when instructions are read
     // from assembly source or disassembled from object code, you expect to see
     // a mixture whenever there's a long VPT block. But in code generation, we
     // hope we'll never generate an Else as input to this pass.
     assert(Pred != ARMVCC::Else && "VPT block pass does not expect Else preds");
 
-    if (Pred == ARMVCC::None) {
-      ++MBIter;
+    if (Pred == ARMVCC::None)
       continue;
-    }
-
-    LLVM_DEBUG(dbgs() << "VPT block created for: "; MI->dump());
-    int VPTInstCnt = 1;
-    ARMVCC::VPTCodes NextPred;
 
     // Look at subsequent instructions, checking if they can be in the same VPT
     // block.
+    LLVM_DEBUG(dbgs() << "VPT block created for: "; MI->dump());
+
+    // The number of instructions in the "then" part of the VPT block.
+    int VPTThenInstCnt = 1;
+
+    unsigned SkippedInstrs;
+    SkipPredicatedInstrs(Block, MBIter, Pred, 3, SkippedInstrs);
+
+    VPTThenInstCnt += SkippedInstrs;
+    assert(VPTThenInstCnt <= 4);
+
+    LLVM_DEBUG(
+        for (MachineBasicBlock::instr_iterator Iter = ++MI->getIterator(),
+             End = MBIter;
+             Iter != End; ++Iter) {
+          dbgs() << "  adding: ";
+          Iter->dump();
+        });
+
+    // Generate the initial BlockMask
+    unsigned BlockMask = getARMVPTBlockMask(VPTThenInstCnt);
+
+    // If the next instr is VPNOT, we can even add an else block if there's room
+    // for it in the VPT block above (at least one instr available) and if we
+    // can safely remove the VPNOT
+    if (MBIter != EndIter && VPTThenInstCnt < 4 &&
+        MBIter->getOpcode() == ARM::MVE_VPNOT) {
+      unsigned VPTElseInstCnt = 0;
+      MachineBasicBlock::instr_iterator AfterVPNotIter = MBIter;
+      ++AfterVPNotIter;
+      // Skip through the predicated instructions of the VPNOT, stopping after
+      // (4 - VPTThenInstCnt). If the manage to skip a whole block, continue.
+      if (SkipPredicatedInstrs(Block, AfterVPNotIter, ARMVCC::Then,
+                               (4 - VPTThenInstCnt), VPTElseInstCnt)) {
+        // Check if we can optimize the VPNOT block.
+        if (CanOptimizeVPNOTBlock(MBIter, AfterVPNotIter)) {
+          LLVM_DEBUG(dbgs() << "  removing VPNOT: "; MBIter->dump(););
+
+          // Record the VPNot to remove it later.
+          DeadInstructions.push_back(&*MBIter);
           ++MBIter;
-    while (MBIter != EndIter && VPTInstCnt < 4) {
-      NextPred = getVPTInstrPredicate(*MBIter, PredReg);
-      assert(NextPred != ARMVCC::Else &&
-             "VPT block pass does not expect Else preds");
-      if (NextPred != Pred)
-        break;
-      LLVM_DEBUG(dbgs() << "  adding : "; MBIter->dump());
-      ++VPTInstCnt;
-      ++MBIter;
-    };
 
-    unsigned BlockMask = getARMVPTBlockMask(VPTInstCnt);
+          // Update the BlockMask to include the elses.
+          BlockMask = AddElsesToARMVPTBlockMask(BlockMask, VPTElseInstCnt);
+
+          // Replace all Thens with Elses in the predicated instructions block
+          // and move MBIter past the end of the block.
+          for (; MBIter != AfterVPNotIter; ++MBIter) {
+            int OpIdx = findFirstVPTPredOperandIdx(*MBIter);
+            assert(OpIdx != -1);
+            MBIter->getOperand(OpIdx).setImm(ARMVCC::Else);
+
+            LLVM_DEBUG(dbgs() << "    adding else: "; MBIter->dump(););
+          }
+        }
+      }
+    }
 
     // Search back for a VCMP that can be folded to create a VPT, or else create
     // a VPST directly
     MachineInstrBuilder MIBuilder;
     unsigned NewOpcode;
-    MachineInstr *VCMP = findVCMPToFoldIntoVPST(MI, TRI, NewOpcode);
-    if (VCMP) {
+    if (MachineInstr *VCMP = findVCMPToFoldIntoVPST(MI, TRI, NewOpcode)) {
       LLVM_DEBUG(dbgs() << "  folding VCMP into VPST: "; VCMP->dump());
-      MIBuilder = BuildMI(Block, MI, dl, TII->get(NewOpcode));
+      MIBuilder = BuildMI(Block, MI, DL, TII->get(NewOpcode));
       MIBuilder.addImm(BlockMask);
       MIBuilder.add(VCMP->getOperand(1));
       MIBuilder.add(VCMP->getOperand(2));
       MIBuilder.add(VCMP->getOperand(3));
       VCMP->eraseFromParent();
     } else {
-      MIBuilder = BuildMI(Block, MI, dl, TII->get(ARM::MVE_VPST));
+      MIBuilder = BuildMI(Block, MI, DL, TII->get(ARM::MVE_VPST));
       MIBuilder.addImm(BlockMask);
     }
 
     finalizeBundle(
         Block, MachineBasicBlock::instr_iterator(MIBuilder.getInstr()), MBIter);
 
     Modified = true;
   }
+
+  // Erase all dead instructions
+  for (MachineInstr *DeadMI : DeadInstructions) {
+    if (DeadMI->isInsideBundle())
+      DeadMI->eraseFromBundle();
+    else
+      DeadMI->eraseFromParent();
+  }
+
   return Modified;
 }
 
 bool MVEVPTBlock::runOnMachineFunction(MachineFunction &Fn) {
   const ARMSubtarget &STI =
       static_cast<const ARMSubtarget &>(Fn.getSubtarget());
 
   if (!STI.isThumb2() || !STI.hasMVEIntegerOps())
     return false;
 
   TII = static_cast<const Thumb2InstrInfo *>(STI.getInstrInfo());
   TRI = STI.getRegisterInfo();
 
   LLVM_DEBUG(dbgs() << "********** ARM MVE VPT BLOCKS **********\n"
                     << "********** Function: " << Fn.getName() << '\n');
 
   bool Modified = false;
   for (MachineBasicBlock &MBB : Fn)
     Modified |= InsertVPTBlocks(MBB);
 
   LLVM_DEBUG(dbgs() << "**************************************\n");
   return Modified;
 }
 
 /// createMVEVPTBlock - Returns an instance of the MVE VPT block
 /// insertion pass.
 FunctionPass *llvm::createMVEVPTBlockPass() { return new MVEVPTBlock(); }

llvm/test/CodeGen/Thumb2/mve-pred-not.ll

entry:
   %o = xor <2 x i1> %c1, <i1 -1, i1 -1>
   %s = select <2 x i1> %o, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %s
 }
 
 define arm_aapcs_vfpcc <4 x i32> @vpnot_v4i1(<4 x i32> %a, <4 x i32> %b, <4 x i32> %c) {
 ; CHECK-LABEL: vpnot_v4i1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vpt.s32 lt, q0, zr
+; CHECK-NEXT:    vpte.s32 lt, q0, zr
 ; CHECK-NEXT:    vcmpt.s32 gt, q1, zr
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vcmpt.i32 eq, q2, zr
+; CHECK-NEXT:    vcmpe.i32 eq, q2, zr
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c1 = icmp slt <4 x i32> %a, zeroinitializer
   %c2 = icmp sgt <4 x i32> %b, zeroinitializer
   %c3 = icmp eq <4 x i32> %c, zeroinitializer
   %o1 = and <4 x i1> %c1, %c2
   %o2 = xor <4 x i1> %o1, <i1 -1, i1 -1, i1 -1, i1 -1>
   %o = and <4 x i1> %c3, %o2
   %s = select <4 x i1> %o, <4 x i32> %a, <4 x i32> %b
   ret <4 x i32> %s
 }

llvm/test/CodeGen/Thumb2/mve-pred-threshold.ll

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #2
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB0_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q1, [r0]
-; CHECK-NEXT:    vpt.s32 ge, q1, r2
+; CHECK-NEXT:    vpte.s32 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s32 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q0, [r0], #16
+; CHECK-NEXT:    vstrwe.32 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB0_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0
   br i1 %cmp15, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #3
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB1_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrh.u16 q1, [r0]
-; CHECK-NEXT:    vpt.s16 ge, q1, r2
+; CHECK-NEXT:    vpte.s16 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s16 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q0, [r0], #16
+; CHECK-NEXT:    vstrhe.16 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB1_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv2 = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv2, 3
   %cmp22 = icmp eq i16 %N, 0
   br i1 %cmp22, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #4
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB2_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrb.u8 q1, [r0]
-; CHECK-NEXT:    vpt.s8 ge, q1, r2
+; CHECK-NEXT:    vpte.s8 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s8 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrbt.8 q0, [r0], #16
+; CHECK-NEXT:    vstrbe.8 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB2_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv2 = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv2, 4
   %cmp20 = icmp eq i16 %N, 0
   br i1 %cmp20, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    vdup.32 q0, r1
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    vdup.32 q1, r1
 ; CHECK-NEXT:  .LBB3_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q3, [r0]
-; CHECK-NEXT:    vpt.f32 le, q0, q3
+; CHECK-NEXT:    vpte.f32 le, q0, q3
 ; CHECK-NEXT:    vcmpt.f32 le, q3, q1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q2, [r0], #16
+; CHECK-NEXT:    vstrwe.32 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB3_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0
   br i1 %cmp15, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    vneg.f16 s0, s0
 ; CHECK-NEXT:    vdup.16 q1, r1
 ; CHECK-NEXT:    vmov r2, s0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:    vdup.16 q0, r2
 ; CHECK-NEXT:  .LBB4_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrh.u16 q3, [r0]
-; CHECK-NEXT:    vpt.f16 le, q1, q3
+; CHECK-NEXT:    vpte.f16 le, q1, q3
 ; CHECK-NEXT:    vcmpt.f16 le, q3, q0
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q2, [r0], #16
+; CHECK-NEXT:    vstrhe.16 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB4_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %0 = bitcast float %T.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %conv = zext i16 %N to i32

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #2
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB5_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q1, [r0]
-; CHECK-NEXT:    vpt.s32 ge, q1, r2
+; CHECK-NEXT:    vpte.s32 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s32 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q0, [r0], #16
+; CHECK-NEXT:    vstrwe.32 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB5_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0
   br i1 %cmp15, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #3
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB6_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrh.u16 q1, [r0]
-; CHECK-NEXT:    vpt.s16 ge, q1, r2
+; CHECK-NEXT:    vpte.s16 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s16 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q0, [r0], #16
+; CHECK-NEXT:    vstrhe.16 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB6_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv2 = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv2, 3
   %cmp22 = icmp eq i16 %N, 0
   br i1 %cmp22, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #4
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB7_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrb.u8 q1, [r0]
-; CHECK-NEXT:    vpt.s8 ge, q1, r2
+; CHECK-NEXT:    vpte.s8 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s8 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrbt.8 q0, [r0], #16
+; CHECK-NEXT:    vstrbe.8 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB7_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv2 = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv2, 4
   %cmp20 = icmp eq i16 %N, 0
   br i1 %cmp20, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    vdup.32 q0, r1
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    vdup.32 q1, r1
 ; CHECK-NEXT:  .LBB8_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q3, [r0]
-; CHECK-NEXT:    vpt.f32 le, q0, q3
+; CHECK-NEXT:    vpte.f32 le, q0, q3
 ; CHECK-NEXT:    vcmpt.f32 le, q3, q1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q2, [r0], #16
+; CHECK-NEXT:    vstrwe.32 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB8_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0
   br i1 %cmp15, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    vneg.f16 s0, s0
 ; CHECK-NEXT:    vdup.16 q1, r1
 ; CHECK-NEXT:    vmov r2, s0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:    vdup.16 q0, r2
 ; CHECK-NEXT:  .LBB9_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrh.u16 q3, [r0]
-; CHECK-NEXT:    vpt.f16 le, q1, q3
+; CHECK-NEXT:    vpte.f16 le, q1, q3
 ; CHECK-NEXT:    vcmpt.f16 le, q3, q0
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q2, [r0], #16
+; CHECK-NEXT:    vstrhe.16 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB9_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %0 = bitcast float %T.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %conv = zext i16 %N to i32

llvm/test/CodeGen/Thumb2/mve-vpt-3-blocks-kill-vpr.mir

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2, $r0
 
     ; CHECK-LABEL: name: vpt_3_blocks_kill_vpr
     ; CHECK: liveins: $q0, $q1, $q2, $r0
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14 /* CC::al */, $noreg
     ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
-    ; CHECK: BUNDLE implicit-def $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit $vpr, implicit killed $q1, implicit $q2, implicit killed $q3 {
-    ; CHECK:   MVE_VPST 8, implicit $vpr
+    ; CHECK: BUNDLE implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $vpr, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $vpr, implicit killed $q1, implicit $q2, implicit killed $q3 {
+    ; CHECK:   MVE_VPST 12, implicit $vpr
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
-    ; CHECK: }
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit $vpr, implicit killed $q3, implicit undef $q1 {
-    ; CHECK:   MVE_VPST 8, implicit $vpr
-    ; CHECK:   renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, renamable $q3, 1, renamable $vpr, undef renamable $q1
+    ; CHECK:   renamable $q1 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, internal renamable $q3, 2, internal renamable $vpr, undef renamable $q1
     ; CHECK: }
     ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     ; CHECK: BUNDLE implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $vpr, implicit killed $q1, implicit killed $q2, implicit killed $q3, implicit killed $q0 {
     ; CHECK:   MVE_VPST 4, implicit $vpr
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
     ; CHECK: }
     ; CHECK: tBX_RET 14 /* CC::al */, $noreg, implicit $q0

llvm/test/CodeGen/Thumb2/mve-vpt-nots.mir

liveins:
   - { reg: '$q1', virtual-reg: '' }
   - { reg: '$q2', virtual-reg: '' }
 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot
     ; CHECK: liveins: $q0, $q1, $q2
-    ; CHECK: BUNDLE implicit-def $vpr, implicit $q0, implicit $zr, implicit $q1 {
-    ; CHECK:   MVE_VPTv4s32r 8, renamable $q0, $zr, 11, implicit-def $vpr
+    ; CHECK: BUNDLE implicit-def $vpr, implicit $q0, implicit $zr, implicit $q1, implicit killed $q2 {
+    ; CHECK:   MVE_VPTv4s32r 12, renamable $q0, $zr, 11, implicit-def $vpr
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, internal killed renamable $vpr
-    ; CHECK: }
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $vpr, implicit killed $vpr, implicit killed $q2, implicit $zr {
-    ; CHECK:   MVE_VPST 8, implicit $vpr
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 2, internal killed renamable $vpr
     ; CHECK: }
     ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14 /* CC::al */, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot_many
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $vpr, implicit killed $vpr, implicit $q1, implicit $zr {
-    ; CHECK:   MVE_VPST 8, implicit $vpr
+    ; CHECK: BUNDLE implicit-def $vpr, implicit killed $vpr, implicit $q1, implicit $zr, implicit killed $q2 {
+    ; CHECK:   MVE_VPST 12, implicit $vpr
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-    ; CHECK: }
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $vpr, implicit killed $vpr, implicit killed $q2, implicit $zr {
-    ; CHECK:   MVE_VPST 8, implicit $vpr
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 2, internal killed renamable $vpr
     ; CHECK: }
     ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14 /* CC::al */, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     tBX_RET 14, $noreg, implicit $q0
 
 ...