[ARM] Allow tail predication of VLDn

[dmgreen]

Summary

VLD2 instructions cannot be predicated, so we cannot tail predicate them from autovec. From intrinsics though, they should be valid as they will just end up loading extra values into off vector lanes, not effecting the on lanes. The same is true for loads in general where so long as we are not using the other vector lanes, an unpredicated load can be converted to a predicated one.

This marks VLD2 and VLD4 instructions as validForTailPredication and allows any unpredicated load in tail predication loop, which I believe will be valid given the other checks we have (but may be mistaken. Some checks I could think of appear to be missing (to do with moving between vector lanes), but that seems to be an orthogonal issue).

[samparker]

This has made me think a bit.. but it seems to make sense and from looking again at the existing checks, they look enough for correctness - to my pleasant surprise!
It feels odd though, that these instructions can't be used in a VPT block but we can use them here... I guess because they actually use mechanisms. But have we got confirmation that both/all registers are always guaranteed to be updated properly in a tail-predicated loop? I don't think there's anything in the reference manual about that limitation but I also haven't found it very clear anyway! One case which we'd have to avoid is if, in a horrible circumstance, LR is used as an address register which is post-indexed and predication could become UNKNOWN, but that isn't specific to VLD2/4.

[dmgreen]

In D86022#2223660, @samparker wrote:

This has made me think a bit.. but it seems to make sense and from looking again at the existing checks, they look enough for correctness - to my pleasant surprise!
It feels odd though, that these instructions can't be used in a VPT block but we can use them here... I guess because they actually use mechanisms. But have we got confirmation that both/all registers are always guaranteed to be updated properly in a tail-predicated loop? I don't think there's anything in the reference manual about that limitation but I also haven't found it very clear anyway!

Yeah. The architectural manual doesn't use the predicate in the VLD2/4 code. So they can't be predicated and always just load the whole vector width of values. This is OK so long as the original code said it was safe to load from that many memory locations, and as this is from intrinsics (not autovec) that seems valid. I even tested it. And for normal loads it's OK to load less values if they are not going to be used.

One case which we'd have to avoid is if, in a horrible circumstance, LR is used as an address register which is post-indexed and predication could become UNKNOWN, but that isn't specific to VLD2/4.

Do you mean:

The behavior of a beat-wise capable instruction that modifies LR and is within a tail predicated low overhead loop
is CONSTRAINED UNPREDICTABLE, the permitted behaviors are either of

That's a new one to me.

[samparker]

Yeah. The architectural manual doesn't use the predicate in the VLD2/4 code. So they can't be predicated and always just load the whole vector width of values.

Okay.

The behavior of a beat-wise capable instruction that modifies LR and is within a tail predicated low overhead loop is CONSTRAINED UNPREDICTABLE

Yeah, me too... but not your problem here. Thanks.

llvm/lib/Target/ARM/ARMInstrMVE.td

class MVE_vldst24_base<bit writeback, bit fourregs, bits<2> stage, bits<2> size,
   let Inst{8-7} = size;
   let Inst{6-5} = stage;
   let Inst{4-1} = 0b0000;
   let Inst{0} = fourregs;
 
   let mayLoad = load;
   let mayStore = !eq(load,0);
   let hasSideEffects = 0;
+  let validForTailPredication = load;
 }
 
 // A parameter class used to encapsulate all the ways the writeback
 // variants of VLD20 and friends differ from the non-writeback ones.
 class MVE_vldst24_writeback<bit b, dag Oo, dag Io,
                             string sy="", string c="", string n=""> {
   bit writeback = b;
   dag Oops = Oo;

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

bool LowOverheadLoop::ValidateLiveOuts() {
   // preventing many instructions from updating their falsely predicated
   // lanes. This analysis assumes that all the instructions perform lane-wise
   // operations and don't perform any exchanges.
   // A masked load, whether through VPT or tail predication, will write zeros
   // to any of the falsely predicated bytes. So, from the loads, we know that
   // the false lanes are zeroed and here we're trying to track that those false
   // lanes remain zero, or where they change, the differences are masked away
   // by their user(s).
-  // All MVE loads and stores have to be predicated, so we know that any load
+  // All MVE stores have to be predicated, so we know that any predicate load
   // operands, or stored results are equivalent already. Other explicitly
   // predicated instructions will perform the same operation in the original
   // loop and the tail-predicated form too. Because of this, we can insert
   // loads, stores and other predicated instructions into our Predicated
   // set and build from there.
   const TargetRegisterClass *QPRs = TRI.getRegClass(ARM::MQPRRegClassID);
   SetVector<MachineInstr *> FalseLanesUnknown;
   SmallPtrSet<MachineInstr *, 4> FalseLanesZero;
bool LowOverheadLoop::ValidateMVEInst(MachineInstr* MI) {
   // predication, only allow the instruction if it's contained within a valid
   // VPT block.
   if ((Flags & ARMII::ValidForTailPredication) == 0 && !IsUse) {
     LLVM_DEBUG(dbgs() << "ARM Loops: Can't tail predicate: " << *MI);
     return false;
   }
 
   // If the instruction is already explicitly predicated, then the conversion
-  // will be fine, but ensure that all memory operations are predicated.
-  return !IsUse && MI->mayLoadOrStore() ? false : true;
+  // will be fine, but ensure that all store operations are predicated.
+  return !IsUse && MI->mayStore() ? false : true;
 }
 
 bool ARMLowOverheadLoops::runOnMachineFunction(MachineFunction &mf) {
   const ARMSubtarget &ST = static_cast<const ARMSubtarget&>(mf.getSubtarget());
   if (!ST.hasLOB())
     return false;
 
   MF = &mf;

llvm/test/CodeGen/Thumb2/LowOverheadLoops/unpredload.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=armv8.1m.main -mattr=+mve -tail-predication=enabled --verify-machineinstrs %s -o - | FileCheck %s
 
 define void @arm_cmplx_mag_squared_q15_mve(i16* %pSrc, i16* %pDst, i32 %blockSize) {
 ; CHECK-LABEL: arm_cmplx_mag_squared_q15_mve:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    subs.w r12, r2, #8
-; CHECK-NEXT:    mov.w r3, #-1
-; CHECK-NEXT:    csinv r3, r3, r12, pl
-; CHECK-NEXT:    add.w r12, r3, r2
-; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    add.w lr, r3, r12, lsr #3
-; CHECK-NEXT:    dls lr, lr
+; CHECK-NEXT:    dlstp.16 lr, r2
 ; CHECK-NEXT:  .LBB0_1: @ %do.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vld20.16 {q0, q1}, [r0]
-; CHECK-NEXT:    vctp.16 r2
-; CHECK-NEXT:    subs r2, #8
 ; CHECK-NEXT:    vld21.16 {q0, q1}, [r0]!
-; CHECK-NEXT:    vpstttt
-; CHECK-NEXT:    vmulht.s16 q2, q1, q1
-; CHECK-NEXT:    vmulht.s16 q0, q0, q0
-; CHECK-NEXT:    vqaddt.s16 q0, q0, q2
-; CHECK-NEXT:    vshrt.s16 q0, q0, #1
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q0, [r1], #16
-; CHECK-NEXT:    le lr, .LBB0_1
+; CHECK-NEXT:    vmulh.s16 q2, q1, q1
+; CHECK-NEXT:    vmulh.s16 q0, q0, q0
+; CHECK-NEXT:    vqadd.s16 q0, q0, q2
+; CHECK-NEXT:    vshr.s16 q0, q0, #1
+; CHECK-NEXT:    vstrh.16 q0, [r1], #16
+; CHECK-NEXT:    letp lr, .LBB0_1
 ; CHECK-NEXT:  @ %bb.2: @ %do.end
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   br label %do.body
 
 do.body:                                          ; preds = %do.body, %entry
   %blockSize.addr.0 = phi i32 [ %blockSize, %entry ], [ %sub, %do.body ]
   %pDst.addr.0 = phi i16* [ %pDst, %entry ], [ %add.ptr7, %do.body ]

 do.end:                                           ; preds = %do.body
   ret i32 %5
 }
 
 define i32 @good2(i32* nocapture readonly %x, i32* nocapture readonly %y, i32 %n) {
 ; CHECK-LABEL: good2:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    mov r3, r2
-; CHECK-NEXT:    cmp r2, #4
-; CHECK-NEXT:    it ge
-; CHECK-NEXT:    movge r3, #4
-; CHECK-NEXT:    subs r3, r2, r3
-; CHECK-NEXT:    add.w r12, r3, #3
-; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    add.w lr, r3, r12, lsr #2
 ; CHECK-NEXT:    mov.w r12, #0
-; CHECK-NEXT:    dls lr, lr
+; CHECK-NEXT:    dlstp.32 lr, r2
 ; CHECK-NEXT:  .LBB3_1: @ %do.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vctp.32 r2
 ; CHECK-NEXT:    vldrw.u32 q0, [r1], #16
 ; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
-; CHECK-NEXT:    subs r2, #4
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vmlavat.s32 r12, q1, q0
-; CHECK-NEXT:    le lr, .LBB3_1
+; CHECK-NEXT:    vmlava.s32 r12, q1, q0
+; CHECK-NEXT:    letp lr, .LBB3_1
 ; CHECK-NEXT:  @ %bb.2: @ %do.end
 ; CHECK-NEXT:    mov r0, r12
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   br label %do.body
 
 do.body:                                          ; preds = %do.body, %entry
   %s.0 = phi i32 [ 0, %entry ], [ %5, %do.body ]

llvm/unittests/Target/ARM/MachineInstrTest.cpp

TEST(MachineInstrValidTailPredication, IsCorrect) {
   using namespace ARM;
 
   auto IsValidTPOpcode = [](unsigned Opcode) {
     switch (Opcode) {
     default:
       return false;
     case MVE_ASRLi:
     case MVE_ASRLr:
-    case MVE_LSRL:  
+    case MVE_LSRL:
     case MVE_SQRSHR:
     case MVE_SQSHL:
     case MVE_SRSHR:
     case MVE_UQRSHL:
     case MVE_UQSHL:
     case MVE_URSHR:
     case MVE_VABDf16:
     case MVE_VABDf32:
     case MVE_VABDs16:
     case MVE_VABDs32:
-    case MVE_VABDs8:  
+    case MVE_VABDs8:
     case MVE_VABDu16:
     case MVE_VABDu32:
     case MVE_VABDu8:
     case MVE_VABSf16:
     case MVE_VABSf32:
     case MVE_VABSs16:
     case MVE_VABSs32:
     case MVE_VABSs8:
auto IsValidTPOpcode = [](unsigned Opcode) {
     case MVE_VHSUBu32:
     case MVE_VHSUBu8:
     case MVE_VIDUPu16:
     case MVE_VIDUPu32:
     case MVE_VIDUPu8:
     case MVE_VIWDUPu16:
     case MVE_VIWDUPu32:
     case MVE_VIWDUPu8:
+    case MVE_VLD20_8:
+    case MVE_VLD21_8:
+    case MVE_VLD20_16:
+    case MVE_VLD21_16:
+    case MVE_VLD20_32:
+    case MVE_VLD21_32:
+    case MVE_VLD20_8_wb:
+    case MVE_VLD21_8_wb:
+    case MVE_VLD20_16_wb:
+    case MVE_VLD21_16_wb:
+    case MVE_VLD20_32_wb:
+    case MVE_VLD21_32_wb:
+    case MVE_VLD40_8:
+    case MVE_VLD41_8:
+    case MVE_VLD42_8:
+    case MVE_VLD43_8:
+    case MVE_VLD40_16:
+    case MVE_VLD41_16:
+    case MVE_VLD42_16:
+    case MVE_VLD43_16:
+    case MVE_VLD40_32:
+    case MVE_VLD41_32:
+    case MVE_VLD42_32:
+    case MVE_VLD43_32:
+    case MVE_VLD40_8_wb:
+    case MVE_VLD41_8_wb:
+    case MVE_VLD42_8_wb:
+    case MVE_VLD43_8_wb:
+    case MVE_VLD40_16_wb:
+    case MVE_VLD41_16_wb:
+    case MVE_VLD42_16_wb:
+    case MVE_VLD43_16_wb:
+    case MVE_VLD40_32_wb:
+    case MVE_VLD41_32_wb:
+    case MVE_VLD42_32_wb:
+    case MVE_VLD43_32_wb:
     case MVE_VLDRBS16:
     case MVE_VLDRBS16_post:
     case MVE_VLDRBS16_pre:
     case MVE_VLDRBS16_rq:
     case MVE_VLDRBS32:
     case MVE_VLDRBS32_post:
     case MVE_VLDRBS32_pre:
     case MVE_VLDRBS32_rq:
auto IsValidTPOpcode = [](unsigned Opcode) {
     case MVE_VLDRWU32_post:
     case MVE_VLDRWU32_pre:
     case MVE_VLDRWU32_qi:
     case MVE_VLDRWU32_qi_pre:
     case MVE_VLDRWU32_rq:
     case MVE_VLDRWU32_rq_u:
     case MVE_VMOVimmf32:
     case MVE_VMOVimmi16:
-    case MVE_VMOVimmi32:  
+    case MVE_VMOVimmi32:
     case MVE_VMOVimmi64:
-    case MVE_VMOVimmi8: 
+    case MVE_VMOVimmi8:
     case MVE_VMOVNi16bh:
     case MVE_VMOVNi16th:
     case MVE_VMOVNi32bh:
     case MVE_VMOVNi32th:
     case MVE_VMULLBp16:
     case MVE_VMULLBp8:
     case MVE_VMULLBs16:
     case MVE_VMULLBs32:
     case MVE_VMULLBs8:
     case MVE_VMULLBu16:
     case MVE_VMULLBu32:
     case MVE_VMULLBu8:
     case MVE_VMULLTp16:
     case MVE_VMULLTp8:
     case MVE_VMULLTs16:
     case MVE_VMULLTs32:
     case MVE_VMULLTs8:
     case MVE_VMULLTu16:
     case MVE_VMULLTu32:
-    case MVE_VMULLTu8:  
+    case MVE_VMULLTu8:
     case MVE_VMUL_qr_f16:
     case MVE_VMUL_qr_f32:
     case MVE_VMUL_qr_i16:
     case MVE_VMUL_qr_i32:
     case MVE_VMUL_qr_i8:
     case MVE_VMULf16:
     case MVE_VMULf32:
     case MVE_VMULi16:
     case MVE_VMULi8:
     case MVE_VMULi32:
     case MVE_VMVN:
     case MVE_VMVNimmi16:
     case MVE_VMVNimmi32:
     case MVE_VNEGf16:
     case MVE_VNEGf32:
     case MVE_VNEGs16:
     case MVE_VNEGs32:
     case MVE_VNEGs8:
     case MVE_VORN:
     case MVE_VORR:
     case MVE_VORRimmi16:
     case MVE_VORRimmi32:
-    case MVE_VPST:  
+    case MVE_VPST:
     case MVE_VQABSs16:
     case MVE_VQABSs32:
     case MVE_VQABSs8:
     case MVE_VQADD_qr_s16:
     case MVE_VQADD_qr_s32:
     case MVE_VQADD_qr_s8:
     case MVE_VQADD_qr_u16:
     case MVE_VQADD_qr_u32:
auto IsValidTPOpcode = [](unsigned Opcode) {
     case MVE_VQSUBs8:
     case MVE_VQSUBu16:
     case MVE_VQSUBu32:
     case MVE_VQSUBu8:
     case MVE_VRHADDs16:
     case MVE_VRHADDs32:
     case MVE_VRHADDs8:
     case MVE_VRHADDu16:
-    case MVE_VRHADDu32: 
+    case MVE_VRHADDu32:
     case MVE_VRHADDu8:
     case MVE_VRINTf16A:
     case MVE_VRINTf16M:
     case MVE_VRINTf16N:
     case MVE_VRINTf16P:
     case MVE_VRINTf16X:
     case MVE_VRINTf16Z:
     case MVE_VRINTf32A:
     case MVE_VRINTf32M:
     case MVE_VRINTf32N:
-    case MVE_VRINTf32P: 
-    case MVE_VRINTf32X: 
+    case MVE_VRINTf32P:
+    case MVE_VRINTf32X:
     case MVE_VRINTf32Z:
     case MVE_VRSHL_by_vecs16:
     case MVE_VRSHL_by_vecs32:
-    case MVE_VRSHL_by_vecs8:  
+    case MVE_VRSHL_by_vecs8:
     case MVE_VRSHL_by_vecu16:
     case MVE_VRSHL_by_vecu32:
     case MVE_VRSHL_by_vecu8:
     case MVE_VRSHL_qrs16:
     case MVE_VRSHL_qrs32:
     case MVE_VRSHL_qrs8:
     case MVE_VRSHL_qru16:
     case MVE_VRSHL_qru32:
auto IsValidTPOpcode = [](unsigned Opcode) {
     case MVE_VSRIimm32:
     case MVE_VSRIimm8:
     case MVE_VSTRB16:
     case MVE_VSTRB16_post:
     case MVE_VSTRB16_pre:
     case MVE_VSTRB16_rq:
     case MVE_VSTRB32:
     case MVE_VSTRB32_post:
-    case MVE_VSTRB32_pre: 
+    case MVE_VSTRB32_pre:
     case MVE_VSTRB32_rq:
     case MVE_VSTRB8_rq:
     case MVE_VSTRBU8:
     case MVE_VSTRBU8_post:
     case MVE_VSTRBU8_pre:
     case MVE_VSTRD64_qi:
     case MVE_VSTRD64_qi_pre:
     case MVE_VSTRD64_rq:
TEST(MachineInstrValidTailPredication, IsCorrect) {
   auto MII = TM->getMCInstrInfo();
 
   for (unsigned i = 0; i < ARM::INSTRUCTION_LIST_END; ++i) {
     const MCInstrDesc &Desc = TII->get(i);
 
     for (auto &Op : Desc.operands()) {
       // Only check instructions that access the MQPR regs.
       if ((Op.OperandType & MCOI::OPERAND_REGISTER) == 0 ||
-          Op.RegClass != ARM::MQPRRegClassID)
+          (Op.RegClass != ARM::MQPRRegClassID &&
+           Op.RegClass != ARM::QQPRRegClassID &&
+           Op.RegClass != ARM::QQQQPRRegClassID))
         continue;
 
       uint64_t Flags = MII->get(i).TSFlags;
       bool Valid = (Flags & ARMII::ValidForTailPredication) != 0;
       ASSERT_EQ(IsValidTPOpcode(i), Valid)
                 << MII->getName(i)
                 << ": mismatched expectation for tail-predicated safety\n";
       break;