[Target][ARM] Improvements to the VPT Block Insertion Pass

[Pierre-vh]

Summary

This patch adds a feature to the MVE VPT block insertion patch: It can now remove VPNOT instructions in some circumstances in order to place the instructions predicated by the VPNOT in an else block instead; This results in more compact code.

For example, this pass used to generate this kind of assembly:

vldrw.u32	q1, [r5]
vpt.s32	ge, q1, r2                      ; Added by MVE-VPT block insertion pass
vcmpt.s32	le, q1, r3
vpnot
vpst	                                ; Added by MVE-VPT block insertion pass
vstrwt.32	q0, [r5], #16

Now, when the VPNOT's result (stored in VPR) is not needed, and when the above block has enough room, it'll generate this instead:

vldrw.u32	q1, [r5]
vpte.s32	ge, q1, r2                     ; Added by MVE-VPT block insertion pass - Notice the "te" instead of "t"
vcmpt.s32	le, q1, r3
vstrwe.32	q0, [r5], #16       ; "t" changed to "e", indicating that this instruction is now part of the "else"

This is much shorter and should be more efficient, and the pass will only remove the VPNOT when VPR is used+killed by one instruction or when VPR is written to in its block.
This should be enough to avoid losing the result of a VPNOT if it's needed, however I'm open to suggestions as I don't actually know if it's good enough.

[Pierre-vh]

Running clang-format on the file.

(My IDE was using its own settings instead of clang-format, this is why the style was incorrect)

[dmgreen]

Good to see this getting updated.

Does this handle predicates like TET/TEET/etc, where we go back to a Then predicate? Any instruction that defs the VPR should reset us back to a Then (so a VCMP will reset the predicate to Then for subsequent instructions in the block). Obviously a VPNOT can flip back to Then too.

Can you make sure we have tests for those situations? Best bet nowadays would probably be to use instrinsics to create some interesting testcases.

[Pierre-vh]

In D75993#1918007, @dmgreen wrote:

Good to see this getting updated.

Does this handle predicates like TET/TEET/etc, where we go back to a Then predicate? Any instruction that defs the VPR should reset us back to a Then (so a VCMP will reset the predicate to Then for subsequent instructions in the block). Obviously a VPNOT can flip back to Then too.

Can you make sure we have tests for those situations? Best bet nowadays would probably be to use instrinsics to create some interesting testcases.

This was not a goal for this initial patch, I just focused on handling else blocks.
However, this is certainly something I can do if you want, but it should it be part of this patch or another patch?

[Pierre-vh]

Updated the diff to fix a bug: The pass refused to generate vpsteee blocks because the condition in SkipPredicatedInstrs was wrong (It needed to do one more iteration when MaxSkips == 0, and then stop).

The pass can now correctly generate this kind of assembly:

	vpsteee	
	vcmpt.s32	le, q1, r12
	vcmpe.s32	lt, q1, r3
	vstrwe.32	q0, [r6], #16
	vstrwe.32	q0, [r4], #16

[Pierre-vh]

This revision will need significant changes as we found a few issues with it.

[Pierre-vh]

Here is the refactored version of this patch. I tried to make the structure a bit cleaner and easier to work on.

• Made the function that expands the block masks more generic, it can now add one or more T/Es to any mask.
• Split the code into multiple functions

Note that I still need to add tests for the new supported cases, I'll try to add them later today or on Monday.

[Pierre-vh]

Once again it needs a bit of work before being ready for review. I need to add support for TETE masks and add a few tests for this improved pass.

[Pierre-vh]

• Added support for TETE masks to the pass
• Added multiple MIR tests for every possible mask that can be generated by the pass
• Added a few LLVM IR tests, unfortunately this does not cover every possible cases as spilling prevents other masks from being generated. I'll look into how this can be improved as a separate patch.

I think this should be ready for review.

[Pierre-vh]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

319

This shouldn't be here and has been fixed locally.
However, I won't send a new diff just for a single line, so I'll push it when I have more meaningful changes to accompany it.

[Pierre-vh]

Another patch to fix a bug - the predicate was not switching back to T on the second VPNOT found.

[Pierre-vh]

• The BlockMask enum is now an enum-class, so I had to make some changes to the pass.

[dmgreen]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

97

Is this only called with a Count of 1 now? If so, can it be simplified.

101

Add a message for the assert.

134

Maybe name this "AddPredicatedInstruction", as we conceptually "adding" them to the block, as opposed to skipping them.

161

Can this use killsRegister?

177

Add a message to the assert.

201

Was VPTThenInstCnt renamed to something else? BlockSize?

248–251

Maybe just use:

CurrentPredicate = CurrentPredicate == ARMVCC::Then ? ARMVCC::Else : ARMVCC::Then;

260–263

This doesn't look like it needed to change? Unless the line after is too long too?

270–271

Why use a uint64_t?

[Pierre-vh]

(I've marked some comments as Done because I changed those locally, I'll update the diff once the review is complete (2 comments left))

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

134

I named the function "Skip" because it advances the iterator (it skips the predicated instruction in front of the iterator), this function doesn't really "care" about the block as a whole, so for me calling it "AddPredicatedInstructions" doesn't make sense.

201

It's indeed "BlockSize" now, I'll change this comment.

260–263

You are right, it doesn't need to change. It was probably changed by mistake by me or clang-format.

270–271

It's the type of the argument of addImm below.
Of course I can also use unsigned if you prefer, or I can change this to ARM::PredBlockMask and convert to uint64_t when calling addImm.

[dmgreen]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

134

Hmm. "Skip" implied to me that we would not end up doing anything with these instructions. Like we would skip instructions between VPT blocks that are not predicated. We just leave them alone.

How about "StepOver" instead?

270–271

unsigned is probably fine, it will fit in either case. If the printing is still simple, PredBlockMask would probably be best. Up to you.

[Pierre-vh]

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

134

StepOverPredicatedInstrs is good I think. I'll change it.

270–271

I'll do something like this then:

ARM::PredBlockMask BlockMask = CreateVPTBlock(MBIter, EndIter, DeadInstructions);
// ...
      MIBuilder.addImm((uint64_t)BlockMask);
// ...
      MIBuilder.addImm((uint64_t)BlockMask);

[dmgreen]

LGTM

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

230

Find the register -> Find the operand holding the predicate

[Pierre-vh]

The patch had already been approved, but needed to be rebased.
Only llvm/test/CodeGen/Thumb2/mve-pred-threshold.ll caused a merge conflict and needed to be re-updated with utils/update_llc_test_checks.py, so it has changed a bit.

[Pierre-vh]

Please re-review the llvm/test/CodeGen/Thumb2/mve-pred-threshold.ll. Everything else should be good as they didn't change, they were only rebased.

[dmgreen]

Oh yeah. Because we are now sinking float splats. Still LGTM.

llvm/lib/Target/ARM/MVEVPTBlockPass.cpp

 #include <cassert>
 #include <new>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "arm-mve-vpt"
 
 namespace {
-  class MVEVPTBlock : public MachineFunctionPass {
-  public:
-    static char ID;
-    const Thumb2InstrInfo *TII;
-    const TargetRegisterInfo *TRI;
 
-    MVEVPTBlock() : MachineFunctionPass(ID) {}
 
-    bool runOnMachineFunction(MachineFunction &Fn) override;
 
-    MachineFunctionProperties getRequiredProperties() const override {
-      return MachineFunctionProperties().set(
-          MachineFunctionProperties::Property::NoVRegs);
-    }
 
-    StringRef getPassName() const override {
-      return "MVE VPT block insertion pass";
-    }
 
-  private:
-    bool InsertVPTBlocks(MachineBasicBlock &MBB);
-  };
 
-  char MVEVPTBlock::ID = 0;
 
 } // end anonymous namespace
 
 INITIALIZE_PASS(MVEVPTBlock, DEBUG_TYPE, "ARM MVE VPT block pass", false, false)
 
 static MachineInstr *findVCMPToFoldIntoVPST(MachineBasicBlock::iterator MI,
                                             const TargetRegisterInfo *TRI,
                                             unsigned &NewOpcode) {
if (registerDefinedBetween(CmpMI->getOperand(1).getReg(), std::next(CmpMI),
                              MI, TRI))
     return nullptr;
   if (registerDefinedBetween(CmpMI->getOperand(2).getReg(), std::next(CmpMI),
                              MI, TRI))
     return nullptr;
   return &*CmpMI;
 }
 
+static unsigned ExpandBlockMask(unsigned BlockMask, unsigned Count,
+                                ARMVCC::VPTCodes Kind) {
+  assert(Count != 0 && Kind != ARMVCC::None);
+  assert(countTrailingZeros(BlockMask) != 0 && "Mask is already full");
+
+  auto Done = [&](unsigned AddedThen, unsigned AddedElse) {
+    unsigned Mask = (Kind == ARMVCC::Then) ? AddedThen : AddedElse;
+    return (Count == 1) ? Mask : ExpandBlockMask(Mask, Count - 1, Kind);
+  };
+
+  switch (BlockMask) {
+  case ARMVCC::T:
+    return Done(ARMVCC::TT, ARMVCC::TE);
+  case ARMVCC::TT:
+    return Done(ARMVCC::TTT, ARMVCC::TTE);
+  case ARMVCC::TE:
+    return Done(ARMVCC::TET, ARMVCC::TEE);
+  case ARMVCC::TTT:
+    return Done(ARMVCC::TTTT, ARMVCC::TTTE);
+  case ARMVCC::TTE:
+    return Done(ARMVCC::TTET, ARMVCC::TTEE);
+  case ARMVCC::TET:
+    return Done(ARMVCC::TETT, ARMVCC::TETE);
+  case ARMVCC::TEE:
+    return Done(ARMVCC::TEET, ARMVCC::TEEE);
+  default:
+    llvm_unreachable("Unknown Mask");
+  }
+}
+
+// Simply skips a block of predicated instructions.
+// Returns true if this successfully skipped the whole block of predicated
+// instructions. Returns false when it stopped early (due to MaxSkips), or if
+// Iter didn't point to a predicated instruction.
+static bool SkipPredicatedInstrs(MachineBasicBlock::instr_iterator &Iter,
+                                 MachineBasicBlock::instr_iterator EndIter,
+                                 unsigned MaxSkips, unsigned &NumInstSkipped) {
+  ARMVCC::VPTCodes NextPred = ARMVCC::None;
+  unsigned PredReg;
+  NumInstSkipped = 0;
+
+  while (Iter != EndIter) {
+    NextPred = getVPTInstrPredicate(*Iter, PredReg);
+    assert(NextPred != ARMVCC::Else &&
+           "VPT block pass does not expect Else preds");
+    if (NextPred == ARMVCC::None || MaxSkips == 0)
+      break;
+    --MaxSkips;
+    ++Iter;
+    ++NumInstSkipped;
+  };
+
+  return NumInstSkipped != 0 && (NextPred == ARMVCC::None || Iter == EndIter);
+}
+
+// Returns true if at least one instruction in the range [Iter, End) defines
+// or kills VPR.
+static bool IsVPRDefinedOrKilledByBlock(MachineBasicBlock::iterator Iter,
+                                        MachineBasicBlock::iterator End) {
+  for (; Iter != End; ++Iter)
+    if (Iter->definesRegister(ARM::VPR) ||
+        Iter->findRegisterUseOperandIdx(ARM::VPR, true) != -1)
+      return true;
+  return false;
+}
+
+// Given an iterator (Iter) that points at an instruction with a "Then"
+// predicate, tries to create the largest block of continuous predicated
+// instructions possible, and returns the VPT Block Mask of that block.
+//
+// This will try to perform some minor optimization in order to maximize the
+// size of the block.
+static unsigned
+CreateVPTBlock(MachineBasicBlock::instr_iterator &Iter,
+               MachineBasicBlock::instr_iterator EndIter,
+               SmallVectorImpl<MachineInstr *> &DeadInstructions) {
+  MachineBasicBlock::instr_iterator BlockBeg = Iter;
+  assert(getVPTInstrPredicate(*Iter) == ARMVCC::Then);
+
+  LLVM_DEBUG(dbgs() << "VPT block created for: "; Iter->dump());
+
+  unsigned BlockSize;
+  SkipPredicatedInstrs(Iter, EndIter, 4, BlockSize);
+
+  LLVM_DEBUG(for (MachineBasicBlock::instr_iterator AddedInstIter =
+                      std::next(BlockBeg);
+                  AddedInstIter != Iter; ++AddedInstIter) {
+    dbgs() << "  adding: ";
+    AddedInstIter->dump();
+  });
+
+  // Generate the initial BlockMask
+  unsigned BlockMask = getARMVPTBlockMask(BlockSize);
+
+  // If we have a block of 4 predicated instructions, we have reached the end
+  // iterator, or there's no VPNOT in sight, then we're done.
+  if (BlockSize == 4 || Iter == EndIter || Iter->getOpcode() != ARM::MVE_VPNOT)
+    return BlockMask;
+
+  // Try to skip all of the predicated instructions after the VPNOT, stopping
+  // after (4 - VPTThenInstCnt).
+  unsigned ElseInstCnt = 0;
+  MachineBasicBlock::instr_iterator VPNOTBlockEndIter = std::next(Iter);
+  if (!SkipPredicatedInstrs(VPNOTBlockEndIter, EndIter, (4 - BlockSize),
+                            ElseInstCnt))
+    return BlockMask;
+
+  // Check if this VPNOT can be removed or not: It can only be removed if at
+  // least one of the predicated instruction that follows it kills or sets
+  // VPR.
+  if (!IsVPRDefinedOrKilledByBlock(Iter, VPNOTBlockEndIter))
+    return BlockMask;
+
+  LLVM_DEBUG(dbgs() << "  removing VPNOT: "; Iter->dump(););
+
+  // Record the VPNot to remove it later.
+  DeadInstructions.push_back(&*Iter);
+  ++Iter;
+
+  // Replace "then" by "elses" in the block until we find an instruction that
+  // defines VPR, then after that leave everything to "t".
+  // Note that we are using "Iter" to iterate over the block so we can update it
+  // at the same time.
+  bool ChangeToElse = true;
+  for (; Iter != VPNOTBlockEndIter; ++Iter) {
+    // Find the register in which the predicate is
+    int OpIdx = findFirstVPTPredOperandIdx(*Iter);
+    assert(OpIdx != -1);
+
+    // Update the mask + change the predicate to an else if needed.
+    if (ChangeToElse) {
+      // Change the predicate and update the mask
+      Iter->getOperand(OpIdx).setImm(ARMVCC::Else);
+      BlockMask = ExpandBlockMask(BlockMask, 1, ARMVCC::Else);
+      // Reset back to a "then" predicate if this instruction defines VPR.
+      if (Iter->definesRegister(ARM::VPR))
+        ChangeToElse = false;
+    } else
+      BlockMask = ExpandBlockMask(BlockMask, 1, ARMVCC::Then);
+
+    LLVM_DEBUG(dbgs() << "  adding: "; Iter->dump());
+  }
+
+  return BlockMask;
+}
+
 bool MVEVPTBlock::InsertVPTBlocks(MachineBasicBlock &Block) {
   bool Modified = false;
   MachineBasicBlock::instr_iterator MBIter = Block.instr_begin();
   MachineBasicBlock::instr_iterator EndIter = Block.instr_end();
 
+  SmallVector<MachineInstr *, 4> DeadInstructions;
+
   while (MBIter != EndIter) {
     MachineInstr *MI = &*MBIter;
     unsigned PredReg = 0;
-    DebugLoc dl = MI->getDebugLoc();
+    DebugLoc DL = MI->getDebugLoc();
 
     ARMVCC::VPTCodes Pred = getVPTInstrPredicate(*MI, PredReg);
 
     // The idea of the predicate is that None, Then and Else are for use when
     // handling assembly language: they correspond to the three possible
     // suffixes "", "t" and "e" on the mnemonic. So when instructions are read
-    // from assembly source or disassembled from object code, you expect to see
-    // a mixture whenever there's a long VPT block. But in code generation, we
-    // hope we'll never generate an Else as input to this pass.
+    // from assembly source or disassembled from object code, you expect to
+    // see a mixture whenever there's a long VPT block. But in code
+    // generation, we hope we'll never generate an Else as input to this pass.
     assert(Pred != ARMVCC::Else && "VPT block pass does not expect Else preds");
 
     if (Pred == ARMVCC::None) {
       ++MBIter;
       continue;
     }
 
-    LLVM_DEBUG(dbgs() << "VPT block created for: "; MI->dump());
-    int VPTInstCnt = 1;
-    ARMVCC::VPTCodes NextPred;
+    unsigned BlockMask = CreateVPTBlock(MBIter, EndIter, DeadInstructions);
 
-    // Look at subsequent instructions, checking if they can be in the same VPT
-    // block.
-    ++MBIter;
-    while (MBIter != EndIter && VPTInstCnt < 4) {
-      NextPred = getVPTInstrPredicate(*MBIter, PredReg);
-      assert(NextPred != ARMVCC::Else &&
-             "VPT block pass does not expect Else preds");
-      if (NextPred != Pred)
-        break;
-      LLVM_DEBUG(dbgs() << "  adding : "; MBIter->dump());
-      ++VPTInstCnt;
-      ++MBIter;
-    };
-
-    unsigned BlockMask = getARMVPTBlockMask(VPTInstCnt);
-
-    // Search back for a VCMP that can be folded to create a VPT, or else create
-    // a VPST directly
+    // Search back for a VCMP that can be folded to create a VPT, or else
+    // create a VPST directly
     MachineInstrBuilder MIBuilder;
     unsigned NewOpcode;
-    MachineInstr *VCMP = findVCMPToFoldIntoVPST(MI, TRI, NewOpcode);
-    if (VCMP) {
+    LLVM_DEBUG(dbgs() << "  final block mask: " << BlockMask << "\n");
+    if (MachineInstr *VCMP = findVCMPToFoldIntoVPST(MI, TRI, NewOpcode)) {
       LLVM_DEBUG(dbgs() << "  folding VCMP into VPST: "; VCMP->dump());
-      MIBuilder = BuildMI(Block, MI, dl, TII->get(NewOpcode));
+      MIBuilder = BuildMI(Block, MI, DL, TII->get(NewOpcode));
       MIBuilder.addImm(BlockMask);
       MIBuilder.add(VCMP->getOperand(1));
       MIBuilder.add(VCMP->getOperand(2));
       MIBuilder.add(VCMP->getOperand(3));
       VCMP->eraseFromParent();
     } else {
-      MIBuilder = BuildMI(Block, MI, dl, TII->get(ARM::MVE_VPST));
+      MIBuilder = BuildMI(Block, MI, DL, TII->get(ARM::MVE_VPST));
       MIBuilder.addImm(BlockMask);
     }
 
     finalizeBundle(
         Block, MachineBasicBlock::instr_iterator(MIBuilder.getInstr()), MBIter);
 
     Modified = true;
   }
+
+  // Erase all dead instructions
+  for (MachineInstr *DeadMI : DeadInstructions) {
+    if (DeadMI->isInsideBundle())
+      DeadMI->eraseFromBundle();
+    else
+      DeadMI->eraseFromParent();
+  }
+
   return Modified;
 }
 
 bool MVEVPTBlock::runOnMachineFunction(MachineFunction &Fn) {
   const ARMSubtarget &STI =
       static_cast<const ARMSubtarget &>(Fn.getSubtarget());
 
   if (!STI.isThumb2() || !STI.hasMVEIntegerOps())
     return false;
 
   TII = static_cast<const Thumb2InstrInfo *>(STI.getInstrInfo());
   TRI = STI.getRegisterInfo();
 
   LLVM_DEBUG(dbgs() << "********** ARM MVE VPT BLOCKS **********\n"
                     << "********** Function: " << Fn.getName() << '\n');
 
   bool Modified = false;
   for (MachineBasicBlock &MBB : Fn)
     Modified |= InsertVPTBlocks(MBB);
 
   LLVM_DEBUG(dbgs() << "**************************************\n");
   return Modified;
 }
 
 /// createMVEVPTBlock - Returns an instance of the MVE VPT block
 /// insertion pass.
 FunctionPass *llvm::createMVEVPTBlockPass() { return new MVEVPTBlock(); }

llvm/lib/Target/ARM/Thumb2InstrInfo.h

 /// instructions which are "predicated", but are not in IT blocks.
 ARMCC::CondCodes getITInstrPredicate(const MachineInstr &MI, unsigned &PredReg);
 
 // getVPTInstrPredicate: VPT analogue of that, plus a helper function
 // corresponding to MachineInstr::findFirstPredOperandIdx.
 int findFirstVPTPredOperandIdx(const MachineInstr &MI);
 ARMVCC::VPTCodes getVPTInstrPredicate(const MachineInstr &MI,
                                       unsigned &PredReg);
+inline ARMVCC::VPTCodes getVPTInstrPredicate(const MachineInstr &MI) {
+  unsigned PredReg;
+  return getVPTInstrPredicate(MI, PredReg);
 }
+} // namespace llvm
 
 #endif

llvm/test/CodeGen/Thumb2/mve-pred-not.ll

entry:
   %o = xor <2 x i1> %c1, <i1 -1, i1 -1>
   %s = select <2 x i1> %o, <2 x i64> %a, <2 x i64> %b
   ret <2 x i64> %s
 }
 
 define arm_aapcs_vfpcc <4 x i32> @vpnot_v4i1(<4 x i32> %a, <4 x i32> %b, <4 x i32> %c) {
 ; CHECK-LABEL: vpnot_v4i1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vpt.s32 lt, q0, zr
+; CHECK-NEXT:    vpte.s32 lt, q0, zr
 ; CHECK-NEXT:    vcmpt.s32 gt, q1, zr
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vcmpt.i32 eq, q2, zr
+; CHECK-NEXT:    vcmpe.i32 eq, q2, zr
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %c1 = icmp slt <4 x i32> %a, zeroinitializer
   %c2 = icmp sgt <4 x i32> %b, zeroinitializer
   %c3 = icmp eq <4 x i32> %c, zeroinitializer
   %o1 = and <4 x i1> %c1, %c2
   %o2 = xor <4 x i1> %o1, <i1 -1, i1 -1, i1 -1, i1 -1>
   %o = and <4 x i1> %c3, %o2
   %s = select <4 x i1> %o, <4 x i32> %a, <4 x i32> %b
   ret <4 x i32> %s
 }

llvm/test/CodeGen/Thumb2/mve-pred-threshold.ll

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #2
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB0_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q1, [r0]
-; CHECK-NEXT:    vpt.s32 ge, q1, r2
+; CHECK-NEXT:    vpte.s32 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s32 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q0, [r0], #16
+; CHECK-NEXT:    vstrwe.32 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB0_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0
   br i1 %cmp15, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #3
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB1_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrh.u16 q1, [r0]
-; CHECK-NEXT:    vpt.s16 ge, q1, r2
+; CHECK-NEXT:    vpte.s16 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s16 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q0, [r0], #16
+; CHECK-NEXT:    vstrhe.16 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB1_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv2 = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv2, 3
   %cmp22 = icmp eq i16 %N, 0
   br i1 %cmp22, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #4
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB2_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrb.u8 q1, [r0]
-; CHECK-NEXT:    vpt.s8 ge, q1, r2
+; CHECK-NEXT:    vpte.s8 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s8 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrbt.8 q0, [r0], #16
+; CHECK-NEXT:    vstrbe.8 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB2_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv2 = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv2, 4
   %cmp20 = icmp eq i16 %N, 0
   br i1 %cmp20, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    vdup.32 q0, r1
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    vdup.32 q1, r1
 ; CHECK-NEXT:  .LBB3_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q3, [r0]
-; CHECK-NEXT:    vpt.f32 le, q0, q3
+; CHECK-NEXT:    vpte.f32 le, q0, q3
 ; CHECK-NEXT:    vcmpt.f32 le, q3, q1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q2, [r0], #16
+; CHECK-NEXT:    vstrwe.32 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB3_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0
   br i1 %cmp15, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    vneg.f16 s0, s0
 ; CHECK-NEXT:    vdup.16 q1, r1
 ; CHECK-NEXT:    vmov r2, s0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:    vdup.16 q0, r2
 ; CHECK-NEXT:  .LBB4_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrh.u16 q3, [r0]
-; CHECK-NEXT:    vpt.f16 le, q1, q3
+; CHECK-NEXT:    vpte.f16 le, q1, q3
 ; CHECK-NEXT:    vcmpt.f16 le, q3, q0
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q2, [r0], #16
+; CHECK-NEXT:    vstrhe.16 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB4_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %0 = bitcast float %T.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %conv = zext i16 %N to i32

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #2
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB5_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q1, [r0]
-; CHECK-NEXT:    vpt.s32 ge, q1, r2
+; CHECK-NEXT:    vpte.s32 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s32 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q0, [r0], #16
+; CHECK-NEXT:    vstrwe.32 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB5_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0
   br i1 %cmp15, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #3
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB6_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrh.u16 q1, [r0]
-; CHECK-NEXT:    vpt.s16 ge, q1, r2
+; CHECK-NEXT:    vpte.s16 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s16 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q0, [r0], #16
+; CHECK-NEXT:    vstrhe.16 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB6_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv2 = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv2, 3
   %cmp22 = icmp eq i16 %N, 0
   br i1 %cmp22, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    add.w lr, r3, r1, lsr #4
 ; CHECK-NEXT:    rsbs r1, r2, #0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB7_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrb.u8 q1, [r0]
-; CHECK-NEXT:    vpt.s8 ge, q1, r2
+; CHECK-NEXT:    vpte.s8 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.s8 le, q1, r1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrbt.8 q0, [r0], #16
+; CHECK-NEXT:    vstrbe.8 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB7_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv2 = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv2, 4
   %cmp20 = icmp eq i16 %N, 0
   br i1 %cmp20, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    vdup.32 q0, r1
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    vdup.32 q1, r1
 ; CHECK-NEXT:  .LBB8_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q3, [r0]
-; CHECK-NEXT:    vpt.f32 le, q0, q3
+; CHECK-NEXT:    vpte.f32 le, q0, q3
 ; CHECK-NEXT:    vcmpt.f32 le, q3, q1
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q2, [r0], #16
+; CHECK-NEXT:    vstrwe.32 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB8_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0
   br i1 %cmp15, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    vneg.f16 s0, s0
 ; CHECK-NEXT:    vdup.16 q1, r1
 ; CHECK-NEXT:    vmov r2, s0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:    vdup.16 q0, r2
 ; CHECK-NEXT:  .LBB9_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrh.u16 q3, [r0]
-; CHECK-NEXT:    vpt.f16 le, q1, q3
+; CHECK-NEXT:    vpte.f16 le, q1, q3
 ; CHECK-NEXT:    vcmpt.f16 le, q3, q0
-; CHECK-NEXT:    vpnot
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q2, [r0], #16
+; CHECK-NEXT:    vstrhe.16 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB9_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %0 = bitcast float %T.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %conv = zext i16 %N to i32

llvm/test/CodeGen/Thumb2/mve-vpt-3-blocks-kill-vpr.mir

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2, $r0
 
     ; CHECK-LABEL: name: vpt_3_blocks_kill_vpr
     ; CHECK: liveins: $q0, $q1, $q2, $r0
     ; CHECK: $vpr = VMSR_P0 killed $r0, 14 /* CC::al */, $noreg
     ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
-    ; CHECK: BUNDLE implicit-def $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit $vpr, implicit killed $q1, implicit $q2, implicit killed $q3 {
-    ; CHECK:   MVE_VPST 8, implicit $vpr
+    ; CHECK: BUNDLE implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $vpr, implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit killed $vpr, implicit killed $q1, implicit $q2, implicit killed $q3 {
+    ; CHECK:   MVE_VPST 12, implicit $vpr
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
-    ; CHECK: }
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $q1, implicit-def $d2, implicit-def $s4, implicit-def $s5, implicit-def $d3, implicit-def $s6, implicit-def $s7, implicit $vpr, implicit killed $q3, implicit undef $q1 {
-    ; CHECK:   MVE_VPST 8, implicit $vpr
-    ; CHECK:   renamable $q1 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q3, renamable $q3, 1, renamable $vpr, undef renamable $q1
+    ; CHECK:   renamable $q1 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, internal renamable $q3, 2, internal renamable $vpr, undef renamable $q1
     ; CHECK: }
     ; CHECK: $q3 = MVE_VORR $q0, $q0, 0, $noreg, undef $q3
     ; CHECK: BUNDLE implicit-def dead $q3, implicit-def $d6, implicit-def $s12, implicit-def $s13, implicit-def $d7, implicit-def $s14, implicit-def $s15, implicit-def $q0, implicit-def $d0, implicit-def $s0, implicit-def $s1, implicit-def $d1, implicit-def $s2, implicit-def $s3, implicit killed $vpr, implicit killed $q1, implicit killed $q2, implicit killed $q3, implicit killed $q0 {
     ; CHECK:   MVE_VPST 4, implicit $vpr
     ; CHECK:   renamable $q3 = nnan ninf nsz MVE_VMINNMf32 killed renamable $q1, renamable $q2, 1, renamable $vpr, killed renamable $q3
     ; CHECK:   renamable $q0 = nnan ninf nsz MVE_VMINNMf32 internal killed renamable $q3, killed renamable $q2, 1, killed renamable $vpr, killed renamable $q0
     ; CHECK: }
     ; CHECK: tBX_RET 14 /* CC::al */, $noreg, implicit $q0

llvm/test/CodeGen/Thumb2/mve-vpt-nots.mir

liveins:
   - { reg: '$q1', virtual-reg: '' }
   - { reg: '$q2', virtual-reg: '' }
 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot
     ; CHECK: liveins: $q0, $q1, $q2
-    ; CHECK: BUNDLE implicit-def $vpr, implicit $q0, implicit $zr, implicit $q1 {
-    ; CHECK:   MVE_VPTv4s32r 8, renamable $q0, $zr, 11, implicit-def $vpr
+    ; CHECK: BUNDLE implicit-def $vpr, implicit $q0, implicit $zr, implicit $q1, implicit killed $q2 {
+    ; CHECK:   MVE_VPTv4s32r 12, renamable $q0, $zr, 11, implicit-def $vpr
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, internal killed renamable $vpr
-    ; CHECK: }
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $vpr, implicit killed $vpr, implicit killed $q2, implicit $zr {
-    ; CHECK:   MVE_VPST 8, implicit $vpr
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 2, internal killed renamable $vpr
     ; CHECK: }
     ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14 /* CC::al */, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr

 body:             |
   bb.0.entry:
     liveins: $q0, $q1, $q2
 
     ; CHECK-LABEL: name: vpnot_many
     ; CHECK: liveins: $q0, $q1, $q2
     ; CHECK: renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $vpr, implicit killed $vpr, implicit $q1, implicit $zr {
-    ; CHECK:   MVE_VPST 8, implicit $vpr
+    ; CHECK: BUNDLE implicit-def $vpr, implicit killed $vpr, implicit $q1, implicit $zr, implicit killed $q2 {
+    ; CHECK:   MVE_VPST 12, implicit $vpr
     ; CHECK:   renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
-    ; CHECK: }
-    ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
-    ; CHECK: BUNDLE implicit-def $vpr, implicit killed $vpr, implicit killed $q2, implicit $zr {
-    ; CHECK:   MVE_VPST 8, implicit $vpr
-    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
+    ; CHECK:   renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 2, internal killed renamable $vpr
     ; CHECK: }
     ; CHECK: renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     ; CHECK: renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     ; CHECK: tBX_RET 14 /* CC::al */, $noreg, implicit $q0
     renamable $vpr = MVE_VCMPs32r renamable $q0, $zr, 11, 0, $noreg
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPs32r renamable $q1, $zr, 12, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $vpr = MVE_VCMPi32r killed renamable $q2, $zr, 0, 1, killed renamable $vpr
     renamable $vpr = MVE_VPNOT killed renamable $vpr, 0, $noreg
     renamable $q0 = MVE_VPSEL killed renamable $q0, killed renamable $q1, 0, killed renamable $vpr
     tBX_RET 14, $noreg, implicit $q0
 
 ...