[VirtRegRewriter] Insert missing killed flags when tracking subregister liveness

[Conanap]

Summary

When using a register operand for the last time and if all its lanes are defined, virtregrewriter may fail to add the killed flag when tracking liveness of subregisters. On PPC, this may cause the generation of unnecessary code.
For example, for the following code:

0B	bb.0 (%ir-block.0):
	  liveins: $v2, $v3, $v4, $v5
16B	  undef %14.sub_vsx1:vsrprc_with_sub_64_in_vfrc = COPY $v5
32B	  %14.sub_vsx0:vsrprc_with_sub_64_in_vfrc = COPY $v4
48B	  undef %8.sub_vsx1:vsrprc_with_sub_64_in_vfrc = COPY $v3
64B	  %8.sub_vsx0:vsrprc_with_sub_64_in_vfrc = COPY $v2
80B	  %4:g8rc_and_g8rc_nox0 = LD 0, %fixed-stack.0 :: (load 8 from %fixed-stack.0, align 16)
160B	  %8:vsrprc_with_sub_64_in_vfrc = KILL_PAIR %8:vsrprc_with_sub_64_in_vfrc(tied-def 0)
176B	  undef %15.sub_pair0:uaccrc = COPY %8:vsrprc_with_sub_64_in_vfrc
256B	  %14:vsrprc_with_sub_64_in_vfrc = KILL_PAIR %14:vsrprc_with_sub_64_in_vfrc(tied-def 0)
288B	  %15.sub_pair1:uaccrc = COPY %14:vsrprc_with_sub_64_in_vfrc
304B	  %28:accrc = BUILD_UACC %15:uaccrc
336B	  %28:accrc = XXMTACC %28:accrc(tied-def 0)
340B	  SPILL_ACC %28:accrc, 0, %stack.0 :: (store 64 into %stack.0, align 16)
352B	  ADJCALLSTACKDOWN 32, 0, implicit-def dead $r1, implicit $r1
368B	  BL8_NOTOC @foo, <regmask $cr2 $cr3 $cr4 $f14 $f15 $f16 $f17 $f18 $f19 $f20 $f21 $f22 $f23 $f24 $f25 $f26 $f27 $f28 $f29 $f30 $f31 $r14 $r15 $r16 $r17 $r18 $r19 $r20 $r21 $r22 $r23 $r24 $r25 and 60 more...>, implicit-def dead $lr8, implicit $rm, implicit-def $r1
384B	  ADJCALLSTACKUP 32, 0, implicit-def dead $r1, implicit $r1
408B	  %25:accrc = RESTORE_ACC 0, %stack.0 :: (load 64 from %stack.0, align 16)
416B	  %25:accrc = XXMFACC %25:accrc(tied-def 0)
448B	  STXV %25.sub_vsx0:accrc, 48, %4:g8rc_and_g8rc_nox0 :: (store 16 into %ir.2 + 48)
480B	  STXV %25.sub_vsx1:accrc, 32, %4:g8rc_and_g8rc_nox0 :: (store 16 into %ir.2 + 32, align 32)
528B	  STXV %25.sub_pair1_then_sub_vsx0:accrc, 16, %4:g8rc_and_g8rc_nox0 :: (store 16 into %ir.2 + 16)
536B	  undef %26.sub_pair1_then_sub_vsx1:accrc = COPY %25.sub_pair1_then_sub_vsx1:accrc
560B	  STXVX %26.sub_pair1_then_sub_vsx1:accrc, $zero8, %4:g8rc_and_g8rc_nox0 :: (store 16 into %ir.2, align 64)
576B	  BLR8 implicit $lr8, implicit $rm

LiveIntervals::addKillFlags should add a killed flag to the %28 operand of the SPILL_ACC instruction (340B).
The live interval for %28 is:

%28 [304r,336r:0)[336r,340r:1)  0@304r 1@336r L0000000000000080 [304r,336r:0)[336r,340r:1)  0@304r 1@336r L0000000000000040 [304r,336r:0)[336r,340r:1)  0@304r 1@336r L0000000000000002 [304r,336r:0)[336r,340r:1)  0@304r 1@336r L0000000000000100 [304r,336r:0)[336r,340r:1)  0@304r 1@336r weight:INF
RegMasks: 368r

We see that lanes L0000000000000080, L0000000000000040, L0000000000000002 and L0000000000000100 are all live until 340B.
The full lane mask of the accumulator registers (accrc) is L00000000000001C2 meaning that all the lanes used for the SPILL are defined and the flag should be added.

This patch fixes two things:

1. The way the mask for defined lanes is computed. We currently compute the mask L0000000000000000 instead of L00000000000001C2. To fix that, we go through all the segments before the instruction using the register operand to see which lanes are defined and killed at this instruction.

2. The way the mask for used lanes is computed. In case we use a register that is not a subregister, we currently use the conservative default mask LFFFFFFFFFFFFFFFF to represent used lanes. However, this mask is not necessarily accurate. As shown above, on PPC, the mask should be L00000000000001C2 for accumulators. So we try to retrieve this mask from the register class instead of using the default mask.

I also need someone familiar with the AMDGPU backend to check if the test cases are still correct after these changes.

[arsenm]

llvm/lib/CodeGen/LiveIntervals.cpp

706–707

What was the point of copying out the subranges here before?

728–750

This looks like a separate NFC change

793–794

getRegClass should be sufficient. There can't be a vreg without a class at this point

llvm/test/CodeGen/PowerPC/subreg-killed.mir

32–102

Should be able to drop most if not all of this

111–133

Can you simplify this at all? -run-pass=none after deleting the register section will also compact these vreg numbers

[lkail]

llvm/lib/CodeGen/LiveIntervals.cpp

793–794

I think it can be simplified as

unsigned SubReg = MO.getSubReg();
LaneBitmask UseMask = SubReg ? TRI->getSubRegIndexLaneMask(SubReg)
                             : MRI->getMaxLaneMaskForVReg(Reg);

[bsaleil]

Remove unrelated NFC change, simplify lane mask computation and MIR code

[bsaleil]

llvm/lib/CodeGen/LiveIntervals.cpp

706–707

To be honest, I don't understand why we copied the subranges here. @MatzeB, do you have any idea ?

728–750

you're right, I removed that.

793–794

Thanks, good catch @lkail I simplified that.

[bsaleil]

Gentle ping :)

[arsenm]

llvm/test/CodeGen/PowerPC/subreg-killed.mir

18–30

Don't need the IR section

33–77

Can drop most of this (I think you just need tracksRegLiveness: true)

[bsaleil]

Rebase and reduce PPC test case.

[foad]

llvm/test/CodeGen/AMDGPU/spill-scavenge-offset.ll

19

Why do you need this change?

[bsaleil]

llvm/test/CodeGen/AMDGPU/spill-scavenge-offset.ll

19

@foad, If I understand correctly the test case, the generated code is divided in two parts, spills and reloads.
The two first check lines (s_mov_b32 and scratch_store_dwordx4) are supposed to match a spill, the two other check lines (s_movk_i32 and scratch_load_dwordx4) are supposed to match a reload.
Currently, the first two check lines are correctly matched in the spills part.
But s_movk_i32 is actually also matched in the spills section. It turns out that there is a scratch_load_dwordx4 for that same register in the reload part, so the test succeeds.
This patch modifies a bit the register allocation for this test case and now there is no scratch_load_dwordx4 for that register.
By adding the 1 here, we force the s_movk_i32 to be matched in the reload section because no s_movk_i32 uses 0x1 in the spills section.
Then, the scratch_load_dwordx4 for that register is correctly matched.

[foad]

llvm/test/CodeGen/AMDGPU/spill-scavenge-offset.ll

19

Makes sense, thanks.

[stefanp]

LGTM.

[Conanap]

A new test case started failing in the process of upstreaming this patched; updated that test case.

llvm/lib/CodeGen/LiveIntervals.cpp

 
 //===----------------------------------------------------------------------===//
 // Register allocator hooks.
 //
 
 void LiveIntervals::addKillFlags(const VirtRegMap *VRM) {
   // Keep track of regunit ranges.
   SmallVector<std::pair<const LiveRange*, LiveRange::const_iterator>, 8> RU;
-  // Keep track of subregister ranges.
-  SmallVector<std::pair<const LiveInterval::SubRange*,
-                        LiveRange::const_iterator>, 4> SRs;
 
   for (unsigned i = 0, e = MRI->getNumVirtRegs(); i != e; ++i) {
     Register Reg = Register::index2VirtReg(i);
     if (MRI->reg_nodbg_empty(Reg))
       continue;
     const LiveInterval &LI = getInterval(Reg);
     if (LI.empty())
       continue;
 
     // Find the regunit intervals for the assigned register. They may overlap
     // the virtual register live range, cancelling any kills.
     RU.clear();
     for (MCRegUnitIterator Unit(VRM->getPhys(Reg), TRI); Unit.isValid();
          ++Unit) {
       const LiveRange &RURange = getRegUnit(*Unit);
       if (RURange.empty())
         continue;
       RU.push_back(std::make_pair(&RURange, RURange.find(LI.begin()->end)));
     }
-
-    if (MRI->subRegLivenessEnabled()) {
-      SRs.clear();
-      for (const LiveInterval::SubRange &SR : LI.subranges()) {
-        SRs.push_back(std::make_pair(&SR, SR.find(LI.begin()->end)));
-      }
-    }
-
     // Every instruction that kills Reg corresponds to a segment range end
     // point.
     for (LiveInterval::const_iterator RI = LI.begin(), RE = LI.end(); RI != RE;
          ++RI) {
       // A block index indicates an MBB edge.
       if (RI->end.isBlock())
         continue;
       MachineInstr *MI = getInstructionFromIndex(RI->end);
       if (!MI)
         continue;
 
       // Check if any of the regunits are live beyond the end of RI. That could
       // happen when a physreg is defined as a copy of a virtreg:
       //
       //   %eax = COPY %5
       //   FOO %5             <--- MI, cancel kill because %eax is live.
       //   BAR killed %eax
       //
       // There should be no kill flag on FOO when %5 is rewritten as %eax.
       for (auto &RUP : RU) {
         const LiveRange &RURange = *RUP.first;
         LiveRange::const_iterator &I = RUP.second;
         if (I == RURange.end())
           continue;
         I = RURange.advanceTo(I, RI->end);
         if (I == RURange.end() || I->start >= RI->end)
           continue;
         // I is overlapping RI.
         goto CancelKill;
       }
 
       if (MRI->subRegLivenessEnabled()) {
         // When reading a partial undefined value we must not add a kill flag.
         // The regalloc might have used the undef lane for something else.
         // Example:
         //     %1 = ...                  ; R32: %1
         //     %2:high16 = ...           ; R64: %2
         //        = read killed %2        ; R64: %2
         //        = read %1              ; R32: %1
         // The <kill> flag is correct for %2, but the register allocator may
         // assign R0L to %1, and R0 to %2 because the low 32bits of R0
         // are actually never written by %2. After assignment the <kill>
         // flag at the read instruction is invalid.
         LaneBitmask DefinedLanesMask;
-        if (!SRs.empty()) {
+        if (LI.hasSubRanges()) {
           // Compute a mask of lanes that are defined.
           DefinedLanesMask = LaneBitmask::getNone();
-          for (auto &SRP : SRs) {
-            const LiveInterval::SubRange &SR = *SRP.first;
-            LiveRange::const_iterator &I = SRP.second;
-            if (I == SR.end())
-              continue;
-            I = SR.advanceTo(I, RI->end);
-            if (I == SR.end() || I->start >= RI->end)
-              continue;
-            // I is overlapping RI
+          for (const LiveInterval::SubRange &SR : LI.subranges())
+            for (const LiveRange::Segment &Segment: SR.segments) {
+              if (Segment.start >= RI->end)
+                break;
+              if (Segment.end == RI->end) {
                 DefinedLanesMask |= SR.LaneMask;
+                break;
+              }
             }
         } else
           DefinedLanesMask = LaneBitmask::getAll();
 
         bool IsFullWrite = false;
         for (const MachineOperand &MO : MI->operands()) {
           if (!MO.isReg() || MO.getReg() != Reg)
             continue;
           if (MO.isUse()) {
             // Reading any undefined lanes?
-            LaneBitmask UseMask = TRI->getSubRegIndexLaneMask(MO.getSubReg());
+            unsigned SubReg = MO.getSubReg();
+            LaneBitmask UseMask = SubReg ? TRI->getSubRegIndexLaneMask(SubReg)
+                                         : MRI->getMaxLaneMaskForVReg(Reg);
             if ((UseMask & ~DefinedLanesMask).any())
               goto CancelKill;
           } else if (MO.getSubReg() == 0) {
             // Writing to the full register?
             assert(MO.isDef());
             IsFullWrite = true;
           }
         }
 
         // If an instruction writes to a subregister, a new segment starts in
         // the LiveInterval. But as this is only overriding part of the register

llvm/test/CodeGen/AMDGPU/load-global-i16.ll

 ; GCN-NOHSA-SI-NEXT:    s_waitcnt expcnt(2)
 ; GCN-NOHSA-SI-NEXT:    v_and_b32_e32 v19, s0, v4
 ; GCN-NOHSA-SI-NEXT:    s_waitcnt expcnt(0)
 ; GCN-NOHSA-SI-NEXT:    buffer_store_dword v19, off, s[12:15], 0 offset:20 ; 4-byte Folded Spill
 ; GCN-NOHSA-SI-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NOHSA-SI-NEXT:    buffer_store_dword v20, off, s[12:15], 0 offset:24 ; 4-byte Folded Spill
 ; GCN-NOHSA-SI-NEXT:    buffer_store_dword v21, off, s[12:15], 0 offset:28 ; 4-byte Folded Spill
 ; GCN-NOHSA-SI-NEXT:    buffer_store_dword v22, off, s[12:15], 0 offset:32 ; 4-byte Folded Spill
-; GCN-NOHSA-SI-NEXT:    s_waitcnt expcnt(1)
-; GCN-NOHSA-SI-NEXT:    v_and_b32_e32 v21, s0, v3
+; GCN-NOHSA-SI-NEXT:    v_and_b32_e32 v4, s0, v3
 ; GCN-NOHSA-SI-NEXT:    v_lshrrev_b32_e32 v32, 16, v5
 ; GCN-NOHSA-SI-NEXT:    v_and_b32_e32 v30, s0, v5
 ; GCN-NOHSA-SI-NEXT:    v_lshrrev_b32_e32 v36, 16, v6
 ; GCN-NOHSA-SI-NEXT:    v_and_b32_e32 v34, s0, v6
 ; GCN-NOHSA-SI-NEXT:    v_lshrrev_b32_e32 v28, 16, v8
 ; GCN-NOHSA-SI-NEXT:    v_and_b32_e32 v26, s0, v8
 ; GCN-NOHSA-SI-NEXT:    v_lshrrev_b32_e32 v40, 16, v7
 ; GCN-NOHSA-SI-NEXT:    v_and_b32_e32 v38, s0, v7

 ; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v51, v1
 ; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v53, v1
 ; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v43, v1
 ; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v45, v1
 ; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v39, v1
 ; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v41, v1
 ; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v31, v1
 ; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v33, v1
-; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v4, v21
 ; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v6, v23
 ; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v5, v1
 ; GCN-NOHSA-SI-NEXT:    v_mov_b32_e32 v7, v1
 ; GCN-NOHSA-SI-NEXT:    buffer_store_dword v4, off, s[12:15], 0 offset:36 ; 4-byte Folded Spill
 ; GCN-NOHSA-SI-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NOHSA-SI-NEXT:    buffer_store_dword v5, off, s[12:15], 0 offset:40 ; 4-byte Folded Spill
 ; GCN-NOHSA-SI-NEXT:    buffer_store_dword v6, off, s[12:15], 0 offset:44 ; 4-byte Folded Spill
 ; GCN-NOHSA-SI-NEXT:    buffer_store_dword v7, off, s[12:15], 0 offset:48 ; 4-byte Folded Spill

llvm/test/CodeGen/AMDGPU/regbank-reassign-wave64.mir

 # GCN: $vgpr24_vgpr25_vgpr26_vgpr27 = IMPLICIT_DEF
 # GCN: $vgpr28_vgpr29_vgpr30_vgpr31 = IMPLICIT_DEF
 # GCN: $vgpr32_vgpr33_vgpr34_vgpr35 = IMPLICIT_DEF
 # GCN: $vgpr36_vgpr37_vgpr38_vgpr39 = IMPLICIT_DEF
 # GCN: $vgpr40_vgpr41_vgpr42_vgpr43 = IMPLICIT_DEF
 # GCN: $vgpr44_vgpr45_vgpr46_vgpr47 = IMPLICIT_DEF
 # GCN: $vgpr2 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr5, $vcc, implicit $exec
 # GCN: $vgpr2 = V_CNDMASK_B32_e64 0, $vgpr0, 0, $vgpr4, killed $vcc, implicit $exec
-# GCN: $sgpr0_sgpr1 = V_CMP_LT_U64_e64 $vgpr4_vgpr5, $vgpr0_vgpr1, implicit $exec
+# GCN: $sgpr0_sgpr1 = V_CMP_LT_U64_e64 killed $vgpr4_vgpr5, killed $vgpr0_vgpr1, implicit $exec
 ---
 name:            vgpr64_mixed_use
 tracksRegLiveness: true
 registers:
   - { id: 0, class: vreg_64, preferred-register: '$vgpr0_vgpr1' }
   - { id: 1, class: vreg_64, preferred-register: '$vgpr4_vgpr5' }
   - { id: 2, class: sreg_64_xexec, preferred-register: '$vcc' }
   - { id: 3, class: vgpr_32 }

llvm/test/CodeGen/AMDGPU/regbank-reassign.mir

bb.0:
     DS_WRITE2_B32_gfx9 %2, %1.sub10, %1.sub11, 10, 11, 0, implicit $exec
     DS_WRITE2_B32_gfx9 %2, %1.sub12, %1.sub13, 12, 13, 0, implicit $exec
     DS_WRITE2_B32_gfx9 %2, %1.sub14, %1.sub15, 14, 15, 0, implicit $exec
     S_ENDPGM 0
 ...
 
 # GCN-LABEL: vgpr_lo16_sub{{$}}
 # GCN: renamable $vgpr0 = V_AND_B32_e32 killed $vgpr3, killed $vgpr1, implicit $exec
-# GCN: renamable $vgpr1_lo16 = COPY renamable $vgpr0_lo16
+# GCN: renamable $vgpr1_lo16 = COPY killed renamable $vgpr0_lo16
 ---
 name:            vgpr_lo16_sub
 tracksRegLiveness: true
 registers:
   - { id: 0, class: vgpr_32, preferred-register: '$vgpr1' }
   - { id: 1, class: vgpr_32, preferred-register: '$vgpr5' }
   - { id: 2, class: vgpr_32 }
   - { id: 3, class: vgpr_lo16 }
bb.0:
 
     %0 = COPY $vgpr0_lo16
     $vgpr1_lo16 = COPY %0
     SI_RETURN_TO_EPILOG $vgpr1_lo16
 ...
 
 # GCN-LABEL: vgpr_hi16_sub{{$}}
 # GCN: renamable $vgpr0 = V_AND_B32_e32 killed $vgpr3, killed $vgpr1, implicit $exec
-# GCN: renamable $vgpr1_hi16 = COPY renamable $vgpr0_hi16
+# GCN: renamable $vgpr1_hi16 = COPY killed renamable $vgpr0_hi16
 ---
 name:            vgpr_hi16_sub
 tracksRegLiveness: true
 registers:
   - { id: 0, class: vgpr_32, preferred-register: '$vgpr1' }
   - { id: 1, class: vgpr_32, preferred-register: '$vgpr5' }
   - { id: 2, class: vgpr_32 }
   - { id: 3, class: vgpr_hi16 }
bb.0:
 
     %0 = COPY $vgpr0_hi16
     $vgpr1_hi16 = COPY %0
     SI_RETURN_TO_EPILOG $vgpr1_hi16
 ...
 
 # GCN-LABEL: sgpr_lo16_sub{{$}}
 # GCN: renamable $sgpr0 = S_AND_B32 killed renamable $sgpr14, $sgpr0, implicit-def $scc
-# GCN: renamable $sgpr1_lo16 = COPY renamable $sgpr0_lo16
+# GCN: renamable $sgpr1_lo16 = COPY killed renamable $sgpr0_lo16
 ---
 name:            sgpr_lo16_sub
 tracksRegLiveness: true
 registers:
   - { id: 0, class: sgpr_32, preferred-register: '$sgpr16' }
   - { id: 1, class: sgpr_32 }
   - { id: 2, class: sgpr_lo16 }
 body: |
bb.0:
     $vgpr3_lo16 = IMPLICIT_DEF
     %2 = V_AND_B32_e32 %1, %0, implicit $exec
     S_ENDPGM 0
 ...
 
 # Test that bank of subreg is considered during scavenging.
 # If handled incorrectly an infinite loop occurs.
 # GCN-LABEL: s0_vs_s15_16_17_sub1{{$}}
-# GCN: S_AND_B32 renamable $sgpr13, $sgpr0,
+# GCN: S_AND_B32 killed renamable $sgpr13, $sgpr0,
 ---
 name:            s0_vs_s15_16_17_sub1
 tracksRegLiveness: true
 registers:
   - { id: 0, class: sgpr_96, preferred-register: '$sgpr15_sgpr16_sgpr17' }
   - { id: 1, class: sgpr_32 }
 body: |
   bb.0:

 # GCN: $vgpr20_vgpr21_vgpr22_vgpr23 = IMPLICIT_DEF
 # GCN: $vgpr24_vgpr25_vgpr26_vgpr27 = IMPLICIT_DEF
 # GCN: $vgpr28_vgpr29_vgpr30_vgpr31 = IMPLICIT_DEF
 # GCN: $vgpr32_vgpr33_vgpr34_vgpr35 = IMPLICIT_DEF
 # GCN: $vgpr36_vgpr37_vgpr38_vgpr39 = IMPLICIT_DEF
 # GCN: $vgpr40_vgpr41_vgpr42_vgpr43 = IMPLICIT_DEF
 # GCN: $vgpr44_vgpr45_vgpr46_vgpr47 = IMPLICIT_DEF
 # GCN: $vgpr0_vgpr1 = V_ADD_F64 0, $vgpr11_vgpr12, 0, killed $vgpr16_vgpr17, 0, 0, implicit $mode, implicit $exec
-# GCN: $vgpr0_vgpr1 = V_ADD_F64 0, $vgpr9_vgpr10, 0, killed $vgpr14_vgpr15, 0, 0, implicit $mode, implicit $exec
+# GCN: $vgpr0_vgpr1 = V_ADD_F64 0, killed $vgpr9_vgpr10, 0, killed $vgpr14_vgpr15, 0, 0, implicit $mode, implicit $exec
 ---
 name:            vgpr_sub_dependence
 tracksRegLiveness: true
 registers:
   - { id: 0, class: vreg_128, preferred-register: '$vgpr10_vgpr11_vgpr12_vgpr13' }
   - { id: 1, class: vreg_64, preferred-register: '$vgpr16_vgpr17' }
   - { id: 2, class: vreg_64, preferred-register: '$vgpr14_vgpr15' }
   - { id: 3, class: vreg_64 }

llvm/test/CodeGen/AMDGPU/spill-scavenge-offset.ll

 ; is used to calculate the scratch load/store address. Make sure that this
 ; mechanism works even when many spills happen.
 
 ; Just test that it compiles successfully.
 ; CHECK-LABEL: test
 
 ; GFX9-FLATSCR: s_mov_b32 [[SOFF1:s[0-9]+]], 4{{$}}
 ; GFX9-FLATSCR: scratch_store_dwordx4 off, v[{{[0-9:]+}}], [[SOFF1]] ; 16-byte Folded Spill
-; GFX9-FLATSCR: s_movk_i32 [[SOFF2:s[0-9]+]], 0x{{[0-9a-f]+}}{{$}}
+; GFX9-FLATSCR: s_movk_i32 [[SOFF2:s[0-9]+]], 0x1{{[0-9a-f]+}}{{$}}
 ; GFX9-FLATSCR: scratch_load_dwordx4 v[{{[0-9:]+}}], off, [[SOFF2]] ; 16-byte Folded Reload
 
 ; GFX10-FLATSCR: scratch_store_dwordx4 off, v[{{[0-9:]+}}], off offset:{{[0-9]+}} ; 16-byte Folded Spill
 ; GFX10-FLATSCR: scratch_load_dwordx4 v[{{[0-9:]+}}], off, off offset:{{[0-9]+}} ; 16-byte Folded Reload
 define amdgpu_kernel void @test(<1280 x i32> addrspace(1)* %out, <1280 x i32> addrspace(1)* %in) {
 entry:
   %lo = call i32 @llvm.amdgcn.mbcnt.lo(i32 -1, i32 0)
   %tid = call i32 @llvm.amdgcn.mbcnt.hi(i32 -1, i32 %lo)

llvm/test/CodeGen/AMDGPU/splitkit-nolivesubranges.mir

 tracksRegLiveness: true
 body: |
   bb.0:
     liveins: $sgpr0_sgpr1
     ; CHECK-LABEL: name: func0
     ; CHECK: liveins: $sgpr0_sgpr1
     ; CHECK: renamable $sgpr0 = IMPLICIT_DEF
     ; CHECK: renamable $sgpr1 = IMPLICIT_DEF
-    ; CHECK: $sgpr104 = S_AND_B32 renamable $sgpr0, renamable $sgpr1, implicit-def $scc
+    ; CHECK: $sgpr104 = S_AND_B32 killed renamable $sgpr0, renamable $sgpr1, implicit-def $scc
     ; CHECK: KILL implicit-def $vcc, implicit-def $sgpr0_sgpr1_sgpr2_sgpr3_sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19_sgpr20_sgpr21_sgpr22_sgpr23_sgpr24_sgpr25_sgpr26_sgpr27_sgpr28_sgpr29_sgpr30_sgpr31, implicit-def $sgpr32_sgpr33_sgpr34_sgpr35_sgpr36_sgpr37_sgpr38_sgpr39_sgpr40_sgpr41_sgpr42_sgpr43_sgpr44_sgpr45_sgpr46_sgpr47_sgpr48_sgpr49_sgpr50_sgpr51_sgpr52_sgpr53_sgpr54_sgpr55_sgpr56_sgpr57_sgpr58_sgpr59_sgpr60_sgpr61_sgpr62_sgpr63, implicit-def $sgpr64_sgpr65_sgpr66_sgpr67_sgpr68_sgpr69_sgpr70_sgpr71_sgpr72_sgpr73_sgpr74_sgpr75_sgpr76_sgpr77_sgpr78_sgpr79_sgpr80_sgpr81_sgpr82_sgpr83_sgpr84_sgpr85_sgpr86_sgpr87_sgpr88_sgpr89_sgpr90_sgpr91_sgpr92_sgpr93_sgpr94_sgpr95, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99_sgpr100_sgpr101_sgpr102_sgpr103
     ; CHECK: renamable $sgpr0_sgpr1 = IMPLICIT_DEF
     ; CHECK: renamable $sgpr0 = IMPLICIT_DEF
     ; CHECK: renamable $sgpr1 = IMPLICIT_DEF
     ; CHECK: SI_SPILL_S64_SAVE renamable $sgpr0_sgpr1, %stack.0, implicit $exec, implicit $sp_reg :: (store 8 into %stack.0, align 4, addrspace 5)
     ; CHECK: KILL implicit-def $vcc, implicit-def $sgpr0_sgpr1_sgpr2_sgpr3_sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19_sgpr20_sgpr21_sgpr22_sgpr23_sgpr24_sgpr25_sgpr26_sgpr27_sgpr28_sgpr29_sgpr30_sgpr31, implicit-def $sgpr32_sgpr33_sgpr34_sgpr35_sgpr36_sgpr37_sgpr38_sgpr39_sgpr40_sgpr41_sgpr42_sgpr43_sgpr44_sgpr45_sgpr46_sgpr47_sgpr48_sgpr49_sgpr50_sgpr51_sgpr52_sgpr53_sgpr54_sgpr55_sgpr56_sgpr57_sgpr58_sgpr59_sgpr60_sgpr61_sgpr62_sgpr63, implicit-def $sgpr64_sgpr65_sgpr66_sgpr67_sgpr68_sgpr69_sgpr70_sgpr71_sgpr72_sgpr73_sgpr74_sgpr75_sgpr76_sgpr77_sgpr78_sgpr79_sgpr80_sgpr81_sgpr82_sgpr83_sgpr84_sgpr85_sgpr86_sgpr87_sgpr88_sgpr89_sgpr90_sgpr91_sgpr92_sgpr93_sgpr94_sgpr95, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99_sgpr100_sgpr101_sgpr102_sgpr103
     ; CHECK: renamable $sgpr0_sgpr1 = SI_SPILL_S64_RESTORE %stack.0, implicit $exec, implicit $sp_reg :: (load 8 from %stack.0, align 4, addrspace 5)
-    ; CHECK: $sgpr105 = S_AND_B32 renamable $sgpr1, renamable $sgpr1, implicit-def $scc
+    ; CHECK: $sgpr105 = S_AND_B32 killed renamable $sgpr1, renamable $sgpr1, implicit-def $scc
     ; CHECK: S_NOP 0, implicit $sgpr104, implicit $sgpr105
     %0:sreg_64 = COPY $sgpr0_sgpr1
     %0.sub0:sreg_64 = IMPLICIT_DEF
     %0.sub1:sreg_64 = IMPLICIT_DEF
     $sgpr104 = S_AND_B32 %0.sub0, %0.sub1, implicit-def $scc
     KILL implicit-def $vcc, implicit-def $sgpr0_sgpr1_sgpr2_sgpr3_sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19_sgpr20_sgpr21_sgpr22_sgpr23_sgpr24_sgpr25_sgpr26_sgpr27_sgpr28_sgpr29_sgpr30_sgpr31, implicit-def $sgpr32_sgpr33_sgpr34_sgpr35_sgpr36_sgpr37_sgpr38_sgpr39_sgpr40_sgpr41_sgpr42_sgpr43_sgpr44_sgpr45_sgpr46_sgpr47_sgpr48_sgpr49_sgpr50_sgpr51_sgpr52_sgpr53_sgpr54_sgpr55_sgpr56_sgpr57_sgpr58_sgpr59_sgpr60_sgpr61_sgpr62_sgpr63, implicit-def $sgpr64_sgpr65_sgpr66_sgpr67_sgpr68_sgpr69_sgpr70_sgpr71_sgpr72_sgpr73_sgpr74_sgpr75_sgpr76_sgpr77_sgpr78_sgpr79_sgpr80_sgpr81_sgpr82_sgpr83_sgpr84_sgpr85_sgpr86_sgpr87_sgpr88_sgpr89_sgpr90_sgpr91_sgpr92_sgpr93_sgpr94_sgpr95, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99_sgpr100_sgpr101_sgpr102_sgpr103
     %0.sub0:sreg_64 = IMPLICIT_DEF
     %0.sub1:sreg_64 = IMPLICIT_DEF
     KILL implicit-def $vcc, implicit-def $sgpr0_sgpr1_sgpr2_sgpr3_sgpr4_sgpr5_sgpr6_sgpr7_sgpr8_sgpr9_sgpr10_sgpr11_sgpr12_sgpr13_sgpr14_sgpr15_sgpr16_sgpr17_sgpr18_sgpr19_sgpr20_sgpr21_sgpr22_sgpr23_sgpr24_sgpr25_sgpr26_sgpr27_sgpr28_sgpr29_sgpr30_sgpr31, implicit-def $sgpr32_sgpr33_sgpr34_sgpr35_sgpr36_sgpr37_sgpr38_sgpr39_sgpr40_sgpr41_sgpr42_sgpr43_sgpr44_sgpr45_sgpr46_sgpr47_sgpr48_sgpr49_sgpr50_sgpr51_sgpr52_sgpr53_sgpr54_sgpr55_sgpr56_sgpr57_sgpr58_sgpr59_sgpr60_sgpr61_sgpr62_sgpr63, implicit-def $sgpr64_sgpr65_sgpr66_sgpr67_sgpr68_sgpr69_sgpr70_sgpr71_sgpr72_sgpr73_sgpr74_sgpr75_sgpr76_sgpr77_sgpr78_sgpr79_sgpr80_sgpr81_sgpr82_sgpr83_sgpr84_sgpr85_sgpr86_sgpr87_sgpr88_sgpr89_sgpr90_sgpr91_sgpr92_sgpr93_sgpr94_sgpr95, implicit-def $sgpr96_sgpr97_sgpr98_sgpr99_sgpr100_sgpr101_sgpr102_sgpr103
     $sgpr105 = S_AND_B32 %0.sub1, %0.sub1, implicit-def $scc
     S_NOP 0, implicit $sgpr104, implicit $sgpr105
 ...

llvm/test/CodeGen/AMDGPU/splitkit.mir

 # allocated to sgpr0_sgpr1 and the first to something else so we see two copies
 # in between for the two subregisters that are alive.
 # CHECK-LABEL: name: func1
 # CHECK: [[REG0:\$sgpr[0-9]+]] = COPY $sgpr0
 # CHECK: [[REG1:\$sgpr[0-9]+]] = COPY $sgpr2
 # CHECK: S_NOP 0
 # CHECK: S_NOP 0, implicit renamable [[REG0]]
 # CHECK: S_NOP 0, implicit renamable [[REG1]]
-# CHECK: $sgpr0 = COPY renamable [[REG0]]
+# CHECK: $sgpr0 = COPY killed renamable [[REG0]]
 # CHECK: $sgpr2 = COPY renamable [[REG1]]
 # CHECK: S_NOP
 # CHECK: S_NOP 0, implicit renamable $sgpr0
-# CHECK: S_NOP 0, implicit renamable $sgpr2
+# CHECK: S_NOP 0, implicit killed renamable $sgpr2
 name: func1
 tracksRegLiveness: true
 body: |
   bb.0:
     liveins: $sgpr0, $sgpr1, $sgpr2
     undef %0.sub0 : sgpr_128 = COPY $sgpr0
     %0.sub2 = COPY $sgpr2
 

llvm/test/CodeGen/AMDGPU/subreg_interference.mir

 # We should not detect any interference between v0/v1 here and only allocate
 # sgpr0-sgpr3.
 #
 # CHECK-LABEL: func0
 # CHECK: S_NOP 0, implicit-def renamable $sgpr0
 # CHECK: S_NOP 0, implicit-def renamable $sgpr3
 # CHECK: S_NOP 0, implicit-def renamable $sgpr1
 # CHECK: S_NOP 0, implicit-def renamable $sgpr2
-# CHECK: S_NOP 0, implicit renamable $sgpr0, implicit renamable $sgpr3
-# CHECK: S_NOP 0, implicit renamable $sgpr1, implicit renamable $sgpr2
+# CHECK: S_NOP 0, implicit killed renamable $sgpr0, implicit renamable $sgpr3
+# CHECK: S_NOP 0, implicit killed renamable $sgpr1, implicit renamable $sgpr2
 name: func0
 body: |
   bb.0:
     S_NOP 0, implicit-def undef %0.sub0 : sgpr_128
     S_NOP 0, implicit-def %0.sub3
     S_NOP 0, implicit-def undef %1.sub1 : sgpr_128
     S_NOP 0, implicit-def %1.sub2
 
 
     S_NOP 0, implicit %0.sub0, implicit %0.sub3
     S_NOP 0, implicit %1.sub1, implicit %1.sub2
 ...

llvm/test/CodeGen/AMDGPU/virtregrewrite-undef-identity-copy.mir

bb.0:
     ; CHECK: $sgpr4 = COPY $sgpr95
     ; CHECK: dead $sgpr30_sgpr31 = SI_CALL killed renamable $sgpr6_sgpr7, @foo, csr_amdgpu_highregs, implicit $sgpr0_sgpr1_sgpr2_sgpr3, implicit $sgpr4
     ; CHECK: ADJCALLSTACKDOWN 0, 4, implicit-def $scc, implicit-def $sgpr32, implicit $sgpr32, implicit $sgpr95
     ; CHECK: renamable $sgpr6_sgpr7 = SI_PC_ADD_REL_OFFSET target-flags(amdgpu-rel32-lo) @bar + 4, target-flags(amdgpu-rel32-hi) @bar + 4, implicit-def dead $scc
     ; CHECK: ADJCALLSTACKUP 0, 0, implicit-def $scc, implicit-def $sgpr32, implicit $sgpr32, implicit $sgpr95
     ; CHECK: $sgpr4 = COPY $sgpr95
     ; CHECK: $vgpr0 = COPY renamable $vgpr40
     ; CHECK: $vgpr1 = COPY renamable $vgpr41
-    ; CHECK: $vgpr2 = COPY renamable $vgpr42
+    ; CHECK: $vgpr2 = COPY killed renamable $vgpr42
     ; CHECK: $vgpr3 = KILL undef renamable $vgpr3
     ; CHECK: dead $sgpr30_sgpr31 = SI_CALL killed renamable $sgpr6_sgpr7, @bar, csr_amdgpu_highregs, implicit $sgpr0_sgpr1_sgpr2_sgpr3, implicit $sgpr4, implicit $vgpr0, implicit killed $vgpr1, implicit killed $vgpr2, implicit killed $vgpr3, implicit-def $vgpr0
     ; CHECK: ADJCALLSTACKDOWN 0, 4, implicit-def $scc, implicit-def $sgpr32, implicit $sgpr32, implicit $sgpr95
     ; CHECK: FLAT_STORE_DWORD undef renamable $vgpr0_vgpr1, killed renamable $vgpr0, 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 4, addrspace 1)
     ; CHECK: S_ENDPGM 0
     %0:vreg_128 = FLAT_LOAD_DWORDX4 undef %1:vreg_64, 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (load 16, addrspace 1)
     %2:sreg_64 = SI_PC_ADD_REL_OFFSET target-flags(amdgpu-rel32-lo) @foo + 4, target-flags(amdgpu-rel32-hi) @foo + 4, implicit-def dead $scc
     ADJCALLSTACKUP 0, 0, implicit-def $sgpr32, implicit $sgpr32, implicit $sgpr95, implicit-def $scc

llvm/test/CodeGen/PowerPC/subreg-killed.mir

+# RUN: llc -mcpu=pwr10 -O3 -ppc-track-subreg-liveness -verify-machineinstrs \
+# RUN:   -mtriple=powerpc64le-unknown-linux-gnu -run-pass=greedy,virtregrewriter \
+# RUN:   -o - %s | FileCheck %s
+
+# This test case checks that the 'killed' flag is properly added when using
+# subregisters.
+
+# CHECK-LABEL: test
+# CHECK: KILL_PAIR killed
+# CHECK-NEXT: COPY killed
+# CHECK-NEXT: KILL_PAIR killed
+# CHECK-NEXT: COPY killed
+# CHECK-NEXT: BUILD_UACC killed
+# CHECK-NEXT: XXMTACC killed
+# CHECK-NEXT: SPILL_ACC killed
+
+--- |
+  declare <512 x i1> @llvm.ppc.mma.xvf16ger2pp(<512 x i1>, <16 x i8>, <16 x i8>)
+
+  declare <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8>, <16 x i8>, <16 x i8>, <16 x i8>)
+
+  declare void @foo()
+
+  define void @test(<16 x i8> %vc1, <16 x i8> %vc2, <16 x i8> %vc3, <16 x i8> %vc4, i8* %ptr) {
+    %1 = tail call <512 x i1> @llvm.ppc.mma.assemble.acc(<16 x i8> %vc1, <16 x i8> %vc2, <16 x i8> %vc3, <16 x i8> %vc4)
+    tail call void @foo()
+    %2 = bitcast i8* %ptr to <512 x i1>*
+    store <512 x i1> %1, <512 x i1>* %2, align 64
+    ret void
+  }
+...
+---
+name:            test
+alignment:       16
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:
+  - { id: 0, class: vrrc, preferred-register: '' }
+  - { id: 1, class: vrrc, preferred-register: '' }
+  - { id: 2, class: vrrc, preferred-register: '' }
+  - { id: 3, class: vrrc, preferred-register: '' }
+  - { id: 4, class: vsrprc_with_sub_64_in_vfrc, preferred-register: '' }
+  - { id: 5, class: vsrprc_with_sub_64_in_vfrc, preferred-register: '' }
+  - { id: 6, class: g8rc_and_g8rc_nox0, preferred-register: '' }
+  - { id: 7, class: uaccrc, preferred-register: '' }
+  - { id: 8, class: accrc, preferred-register: '' }
+  - { id: 9, class: accrc, preferred-register: '' }
+  - { id: 10, class: accrc, preferred-register: '' }
+liveins:
+  - { reg: '$v2', virtual-reg: '%0' }
+  - { reg: '$v3', virtual-reg: '%1' }
+  - { reg: '$v4', virtual-reg: '%2' }
+  - { reg: '$v5', virtual-reg: '%3' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    16
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:
+  - { id: 0, size: 8 }
+stack:
+  - { id: 0, size: 64 }
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.0):
+    liveins: $v2, $v3, $v4, $v5
+
+    undef %4.sub_vsx1:vsrprc_with_sub_64_in_vfrc = COPY $v5
+    %4.sub_vsx0:vsrprc_with_sub_64_in_vfrc = COPY $v4
+    undef %5.sub_vsx1:vsrprc_with_sub_64_in_vfrc = COPY $v3
+    %5.sub_vsx0:vsrprc_with_sub_64_in_vfrc = COPY $v2
+    %6:g8rc_and_g8rc_nox0 = LD 0, %fixed-stack.0 :: (load 8 from %fixed-stack.0, align 16)
+    %5:vsrprc_with_sub_64_in_vfrc = KILL_PAIR %5
+    undef %7.sub_pair0:uaccrc = COPY %5
+    %4:vsrprc_with_sub_64_in_vfrc = KILL_PAIR %4
+    %7.sub_pair1:uaccrc = COPY %4
+    %8:accrc = BUILD_UACC %7
+    %8:accrc = XXMTACC %8
+    SPILL_ACC %8, 0, %stack.0 :: (store 64 into %stack.0, align 16)
+    ADJCALLSTACKDOWN 32, 0, implicit-def dead $r1, implicit $r1
+    BL8_NOTOC @foo, csr_ppc64_altivec, implicit-def dead $lr8, implicit $rm, implicit-def $r1
+    ADJCALLSTACKUP 32, 0, implicit-def dead $r1, implicit $r1
+    %9:accrc = RESTORE_ACC 0, %stack.0 :: (load 64 from %stack.0, align 16)
+    %9:accrc = XXMFACC %9
+    STXV %9.sub_vsx0, 48, %6 :: (store 16 into %ir.2 + 48)
+    STXV %9.sub_vsx1, 32, %6 :: (store 16 into %ir.2 + 32, align 32)
+    STXV %9.sub_pair1_then_sub_vsx0, 16, %6 :: (store 16 into %ir.2 + 16)
+    undef %10.sub_pair1_then_sub_vsx1:accrc = COPY %9.sub_pair1_then_sub_vsx1
+    STXVX %10.sub_pair1_then_sub_vsx1, $zero8, %6 :: (store 16 into %ir.2, align 64)
+    BLR8 implicit $lr8, implicit $rm
+
+...