[GlobalISel] Check LLT size matches memory size for non-truncating stores.

[aemerson]

Summary

This was causing a bug where non-truncating stores would be selected instead of truncating ones.

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-store-flat.mir

106

This is breaking this, producing a non-truncating store instead of the correct trunc store

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-store-private.mir

55

This is breaking this, the correct trunc store now fails to select

llvm/utils/TableGen/GlobalISelEmitter.cpp

3314

Should also continue? Could make this a select on the match type to the one addPredicate call

[aemerson]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3314

Yeah, just seen this doesn't work. I'm not familiar with this code, but how do non-truncstore predicates end up on trunc stores such that it breaks?

[arsenm]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3314

I don't exactly understand what the expected relationship between predicate bits set on a PatFrag and the PatFrags which used it is.

It's possible something is wrong with the AMDGPU PatFrags. This system is pretty confusing, because each property is added in a different layer of PatFrags defined in TargetSelectionDAG.td. The AMDGPU PatFrags reproduce the same hierarchy, with the additional address space predicates.

There is also the extra IsTruncStore bit which needs to be manually set on the base truncstore PatFrag. For some reason the derivative PatFrags do not have it set, as it should be implied. However, they still need to set IsStore, although I would expect these to both behave the same way.

[arsenm]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3314

I'm also not sure the way this loop is structured makes sense. It seems to assume too much about what different predicates can be combined. The atomic predicates also don't work correctly

[aemerson]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3314

@dsanders any idea what to do?

[dsanders]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3314

That else is suspicious. isTruncStore() means that the predicate needs to be added but !isTruncStore() can mean either no predicate is needed or a non-trunc store predicate is needed. I believe it needs to be:

if (Predicate.isTruncStore())
  add relevant predicate for trunc store
if (Predicate.isNonTruncStore())
  add relevant predicate for non-trunc store

[dsanders]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3314

don't exactly understand what the expected relationship between predicate bits set on a PatFrag and the
PatFrags which used it is.

It's possible something is wrong with the AMDGPU PatFrags. This system is pretty confusing, because
each property is added in a different layer of PatFrags defined in TargetSelectionDAG.td. The AMDGPU
PatFrags reproduce the same hierarchy, with the additional address space predicates.

The input we start with is pretty weird too and it's mostly a straight import of that.

Suppose we start with the pattern:

(truncstorei8 node:$x)

when the PatFrag is expanded, we get:

(truncstore node:$x)<<code-to-check-memvt==i8>>

where the <<some C++ code>> is tablegen's way of representing predicates (this isn't quite true, it uses a short-hand name instead of the C++ when it's printing but the data in the pattern is the C++ code). Then after further expansion we get:

(unindexedstore node:$x)<<code-to-check-memvt==i8>><<code-to-check-for-truncating>>
(st node:$x)<<code-to-check-memvt==i8>><<code-to-check-for-truncating>><<code-to-check-for-unindexed>>

That last line is how it looks on input to the GlobalISelEmitter before runOnPattern(). The first thing we needed to do was abstract out the C++ as it needed to be different for DAGISel vs GISel. To do that, each predicate was given a field in PatFrag. For these three predicates it was:

ValueType MemoryVT = ?;
bit IsTruncStore = ?;
bit IsUnindexed = ?;

Only one of these is set for each PatFrag as each one directly corresponds to the original C++ code. For example: IsTruncStore = 0 emits this code for DAGISel:

!cast<StoreSDNode>(N)->isTruncatingStore();

I'm also not sure the way this loop is structured makes sense. It seems to assume too much about what
different predicates can be combined.

Each predicate can only be one C++ fragment. If you need multiple fragments (which happens when you have PatFrags inside PatFrag expansions) you have multiple Predicates, each with a different field set corresponding to a different fragment.

The atomic predicates also don't work correctly

Could you elaborate on this?

[arsenm]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3314

The atomic predicates also don't work correctly

Could you elaborate on this?

I didn't spend much time looking at it to focus on one broken thing at a time, but the AMDGPU patterns for atomic_load/atomic_store don't work. They should be easy to handle as they ignore the ordering type and just need to preserve the atomicness in the MemOperand. I think the combination of IsStore and IsAtomic wasn't behaving as expected

[aemerson]

Use isNonTruncStore() instead.

[dsanders]

That change LGTM. Do you have a test case for your target?

[aemerson]

In D64845#1613091, @dsanders wrote:

That change LGTM. Do you have a test case for your target?

Thanks, it's only exposed with another patch though so I can't add that test here.

llvm/test/TableGen/address-space-patfrags.td

123 ↗

(On Diff #213137)

Just seen this incorrect comment, will fix in the commit.

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-store-flat.mir

 body: |
   bb.0:
     liveins:  $vgpr0_vgpr1, $vgpr2
 
     ; GFX7-LABEL: name: store_flat_s32_to_2
     ; GFX7: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX7: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX7: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX7: FLAT_STORE_SHORT [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 2)
+    ; GFX7: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX8-LABEL: name: store_flat_s32_to_2
     ; GFX8: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX8: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX8: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX8: FLAT_STORE_SHORT [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 2)
+    ; GFX8: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX9-LABEL: name: store_flat_s32_to_2
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX9: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX9: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX9: FLAT_STORE_SHORT [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 2)
+    ; GFX9: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX10-LABEL: name: store_flat_s32_to_2
     ; GFX10: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX10: $vcc_hi = IMPLICIT_DEF
     ; GFX10: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX10: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX10: FLAT_STORE_SHORT [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 2)
+    ; GFX10: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     %0:vgpr(p1) = COPY $vgpr0_vgpr1
     %1:vgpr(s32) = COPY $vgpr2
     G_STORE %1, %0 :: (store 2, align 2, addrspace 0)
 
 ...
 
 ---
 name: store_flat_s32_to_1
 legalized:       true
 tracksRegLiveness: true
 regBankSelected: true
 
 body: |
   bb.0:
     liveins:  $vgpr0_vgpr1, $vgpr2
 
     ; GFX7-LABEL: name: store_flat_s32_to_1
     ; GFX7: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX7: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX7: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX7: FLAT_STORE_BYTE [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 1)
+    ; GFX7: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX8-LABEL: name: store_flat_s32_to_1
     ; GFX8: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX8: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX8: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX8: FLAT_STORE_BYTE [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 1)
+    ; GFX8: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX9-LABEL: name: store_flat_s32_to_1
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX9: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX9: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX9: FLAT_STORE_BYTE [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 1)
+    ; GFX9: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX10-LABEL: name: store_flat_s32_to_1
     ; GFX10: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX10: $vcc_hi = IMPLICIT_DEF
     ; GFX10: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX10: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX10: FLAT_STORE_BYTE [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 1)
+    ; GFX10: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     %0:vgpr(p1) = COPY $vgpr0_vgpr1
     %1:vgpr(s32) = COPY $vgpr2
     G_STORE %1, %0 :: (store 1, align 1, addrspace 0)
 
 ...
 
 ---
 

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-store-global.mir

 body: |
   bb.0:
     liveins:  $vgpr0_vgpr1, $vgpr2
 
     ; GFX7-LABEL: name: store_global_s32_to_2
     ; GFX7: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX7: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX7: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX7: FLAT_STORE_SHORT [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 2, addrspace 1)
+    ; GFX7: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX8-LABEL: name: store_global_s32_to_2
     ; GFX8: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX8: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX8: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX8: FLAT_STORE_SHORT [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 2, addrspace 1)
+    ; GFX8: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX9-LABEL: name: store_global_s32_to_2
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX9: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX9: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX9: GLOBAL_STORE_SHORT [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec :: (store 2, addrspace 1)
+    ; GFX9: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX10-LABEL: name: store_global_s32_to_2
     ; GFX10: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX10: $vcc_hi = IMPLICIT_DEF
     ; GFX10: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX10: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX10: GLOBAL_STORE_SHORT [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec :: (store 2, addrspace 1)
+    ; GFX10: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     %0:vgpr(p1) = COPY $vgpr0_vgpr1
     %1:vgpr(s32) = COPY $vgpr2
     G_STORE %1, %0 :: (store 2, align 2, addrspace 1)
 
 ...
 
 ---
 name: store_global_s32_to_1
 legalized:       true
 tracksRegLiveness: true
 regBankSelected: true
 
 body: |
   bb.0:
     liveins:  $vgpr0_vgpr1, $vgpr2
 
     ; GFX7-LABEL: name: store_global_s32_to_1
     ; GFX7: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX7: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX7: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX7: FLAT_STORE_BYTE [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 1, addrspace 1)
+    ; GFX7: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX8-LABEL: name: store_global_s32_to_1
     ; GFX8: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX8: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX8: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX8: FLAT_STORE_BYTE [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr :: (store 1, addrspace 1)
+    ; GFX8: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX9-LABEL: name: store_global_s32_to_1
     ; GFX9: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX9: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX9: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX9: GLOBAL_STORE_BYTE [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec :: (store 1, addrspace 1)
+    ; GFX9: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     ; GFX10-LABEL: name: store_global_s32_to_1
     ; GFX10: liveins: $vgpr0_vgpr1, $vgpr2
     ; GFX10: $vcc_hi = IMPLICIT_DEF
     ; GFX10: [[COPY:%[0-9]+]]:vreg_64 = COPY $vgpr0_vgpr1
     ; GFX10: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr2
-    ; GFX10: GLOBAL_STORE_BYTE [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec :: (store 1, addrspace 1)
+    ; GFX10: FLAT_STORE_DWORD [[COPY]], [[COPY1]], 0, 0, 0, 0, implicit $exec, implicit $flat_scr
     %0:vgpr(p1) = COPY $vgpr0_vgpr1
     %1:vgpr(s32) = COPY $vgpr2
     G_STORE %1, %0 :: (store 1, align 1, addrspace 1)
 
 ...
 
 ---
 

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-store-private.mir

machineFunctionInfo:
   stackPtrOffsetReg: $sgpr32
 
 body: |
   bb.0:
     liveins: $vgpr0, $vgpr1
 
     ; GFX6-LABEL: name: store_private_s32_to_2
     ; GFX6: liveins: $vgpr0, $vgpr1
-    ; GFX6: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX6: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; GFX6: BUFFER_STORE_SHORT_OFFEN [[COPY]], [[COPY1]], $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4, 0, 0, 0, 0, 0, implicit $exec :: (store 2, addrspace 5)
+    ; GFX6: [[COPY:%[0-9]+]]:vgpr(s32) = COPY $vgpr0
+    ; GFX6: [[COPY1:%[0-9]+]]:vgpr(p5) = COPY $vgpr1
+    ; GFX6: G_STORE [[COPY]](s32), [[COPY1]](p5) :: (store 2, addrspace 5)
     ; GFX9-LABEL: name: store_private_s32_to_2
     ; GFX9: liveins: $vgpr0, $vgpr1
-    ; GFX9: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX9: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; GFX9: BUFFER_STORE_SHORT_OFFEN [[COPY]], [[COPY1]], $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4, 0, 0, 0, 0, 0, implicit $exec :: (store 2, addrspace 5)
+    ; GFX9: [[COPY:%[0-9]+]]:vgpr(s32) = COPY $vgpr0
+    ; GFX9: [[COPY1:%[0-9]+]]:vgpr(p5) = COPY $vgpr1
+    ; GFX9: G_STORE [[COPY]](s32), [[COPY1]](p5) :: (store 2, addrspace 5)
     %0:vgpr(s32) = COPY $vgpr0
     %1:vgpr(p5) = COPY $vgpr1
     G_STORE %0, %1 :: (store 2, align 2, addrspace 5)
 
 ...
 
 ---
 
 name: store_private_s32_to_1
 legalized:       true
 regBankSelected: true
 tracksRegLiveness: true
 machineFunctionInfo:
   scratchRSrcReg: $sgpr0_sgpr1_sgpr2_sgpr3
   scratchWaveOffsetReg: $sgpr4
   stackPtrOffsetReg: $sgpr32
 
 body: |
   bb.0:
     liveins: $vgpr0, $vgpr1
 
     ; GFX6-LABEL: name: store_private_s32_to_1
     ; GFX6: liveins: $vgpr0, $vgpr1
-    ; GFX6: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX6: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; GFX6: BUFFER_STORE_BYTE_OFFEN [[COPY]], [[COPY1]], $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4, 0, 0, 0, 0, 0, implicit $exec :: (store 1, addrspace 5)
+    ; GFX6: [[COPY:%[0-9]+]]:vgpr(s32) = COPY $vgpr0
+    ; GFX6: [[COPY1:%[0-9]+]]:vgpr(p5) = COPY $vgpr1
+    ; GFX6: G_STORE [[COPY]](s32), [[COPY1]](p5) :: (store 1, addrspace 5)
     ; GFX9-LABEL: name: store_private_s32_to_1
     ; GFX9: liveins: $vgpr0, $vgpr1
-    ; GFX9: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX9: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; GFX9: BUFFER_STORE_BYTE_OFFEN [[COPY]], [[COPY1]], $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4, 0, 0, 0, 0, 0, implicit $exec :: (store 1, addrspace 5)
+    ; GFX9: [[COPY:%[0-9]+]]:vgpr(s32) = COPY $vgpr0
+    ; GFX9: [[COPY1:%[0-9]+]]:vgpr(p5) = COPY $vgpr1
+    ; GFX9: G_STORE [[COPY]](s32), [[COPY1]](p5) :: (store 1, addrspace 5)
     %0:vgpr(s32) = COPY $vgpr0
     %1:vgpr(p5) = COPY $vgpr1
     G_STORE %0, %1 :: (store 1, align 1, addrspace 5)
 
 ...
 
 ---
 
machineFunctionInfo:
   stackPtrOffsetReg: $sgpr32
 stack:
   - { id: 0, size: 4096, alignment: 4 }
 
 body: |
   bb.0:
 
     ; GFX6-LABEL: name: store_private_s32_to_1_fi_offset_4095
-    ; GFX6: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 %stack.0, implicit $exec
-    ; GFX6: [[V_MOV_B32_e32_1:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 4095, implicit $exec
-    ; GFX6: %2:vgpr_32, dead %4:sreg_64_xexec = V_ADD_I32_e64 [[V_MOV_B32_e32_]], [[V_MOV_B32_e32_1]], 0, implicit $exec
-    ; GFX6: [[V_MOV_B32_e32_2:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-    ; GFX6: BUFFER_STORE_BYTE_OFFEN [[V_MOV_B32_e32_2]], %2, $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4, 0, 0, 0, 0, 0, implicit $exec :: (store 1, addrspace 5)
+    ; GFX6: [[FRAME_INDEX:%[0-9]+]]:vgpr(p5) = G_FRAME_INDEX %stack.0
+    ; GFX6: [[C:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 4095
+    ; GFX6: [[GEP:%[0-9]+]]:vgpr(p5) = G_GEP [[FRAME_INDEX]], [[C]](s32)
+    ; GFX6: [[C1:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 0
+    ; GFX6: G_STORE [[C1]](s32), [[GEP]](p5) :: (store 1, addrspace 5)
     ; GFX9-LABEL: name: store_private_s32_to_1_fi_offset_4095
-    ; GFX9: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-    ; GFX9: BUFFER_STORE_BYTE_OFFEN [[V_MOV_B32_e32_]], %stack.0, $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr32, 4095, 0, 0, 0, 0, implicit $exec :: (store 1, addrspace 5)
+    ; GFX9: [[FRAME_INDEX:%[0-9]+]]:vgpr(p5) = G_FRAME_INDEX %stack.0
+    ; GFX9: [[C:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 4095
+    ; GFX9: [[GEP:%[0-9]+]]:vgpr(p5) = G_GEP [[FRAME_INDEX]], [[C]](s32)
+    ; GFX9: [[C1:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 0
+    ; GFX9: G_STORE [[C1]](s32), [[GEP]](p5) :: (store 1, addrspace 5)
     %0:vgpr(p5) = G_FRAME_INDEX %stack.0
     %1:vgpr(s32) = G_CONSTANT i32 4095
     %2:vgpr(p5) = G_GEP %0, %1
     %3:vgpr(s32) = G_CONSTANT i32 0
     G_STORE %3, %2 :: (store 1, align 1, addrspace 5)
 
 ...
 
machineFunctionInfo:
   stackPtrOffsetReg: $sgpr32
 stack:
   - { id: 0, size: 4096, alignment: 4 }
 
 body: |
   bb.0:
 
     ; GFX6-LABEL: name: store_private_s32_to_1_constant_4095
-    ; GFX6: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-    ; GFX6: BUFFER_STORE_BYTE_OFFSET [[V_MOV_B32_e32_]], $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4, 4095, 0, 0, 0, 0, implicit $exec :: (store 1, addrspace 5)
+    ; GFX6: [[C:%[0-9]+]]:vgpr(p5) = G_CONSTANT i32 4095
+    ; GFX6: [[C1:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 0
+    ; GFX6: G_STORE [[C1]](s32), [[C]](p5) :: (store 1, addrspace 5)
     ; GFX9-LABEL: name: store_private_s32_to_1_constant_4095
-    ; GFX9: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-    ; GFX9: BUFFER_STORE_BYTE_OFFSET [[V_MOV_B32_e32_]], $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4, 4095, 0, 0, 0, 0, implicit $exec :: (store 1, addrspace 5)
+    ; GFX9: [[C:%[0-9]+]]:vgpr(p5) = G_CONSTANT i32 4095
+    ; GFX9: [[C1:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 0
+    ; GFX9: G_STORE [[C1]](s32), [[C]](p5) :: (store 1, addrspace 5)
     %0:vgpr(p5) = G_CONSTANT i32 4095
     %1:vgpr(s32) = G_CONSTANT i32 0
     G_STORE %1, %0 :: (store 1, align 1, addrspace 5)
 
 ...
 
 ---
 
 name: store_private_s32_to_1_constant_4096
 legalized:       true
 regBankSelected: true
 tracksRegLiveness: true
 machineFunctionInfo:
   scratchRSrcReg: $sgpr0_sgpr1_sgpr2_sgpr3
   scratchWaveOffsetReg: $sgpr4
   stackPtrOffsetReg: $sgpr32
 stack:
   - { id: 0, size: 4096, alignment: 4 }
 
 body: |
   bb.0:
 
     ; GFX6-LABEL: name: store_private_s32_to_1_constant_4096
-    ; GFX6: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-    ; GFX6: [[V_MOV_B32_e32_1:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 4096, implicit $exec
-    ; GFX6: BUFFER_STORE_BYTE_OFFEN [[V_MOV_B32_e32_]], [[V_MOV_B32_e32_1]], $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4, 0, 0, 0, 0, 0, implicit $exec :: (store 1, addrspace 5)
+    ; GFX6: [[C:%[0-9]+]]:vgpr(p5) = G_CONSTANT i32 4096
+    ; GFX6: [[C1:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 0
+    ; GFX6: G_STORE [[C1]](s32), [[C]](p5) :: (store 1, addrspace 5)
     ; GFX9-LABEL: name: store_private_s32_to_1_constant_4096
-    ; GFX9: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-    ; GFX9: [[V_MOV_B32_e32_1:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 4096, implicit $exec
-    ; GFX9: BUFFER_STORE_BYTE_OFFEN [[V_MOV_B32_e32_]], [[V_MOV_B32_e32_1]], $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4, 0, 0, 0, 0, 0, implicit $exec :: (store 1, addrspace 5)
+    ; GFX9: [[C:%[0-9]+]]:vgpr(p5) = G_CONSTANT i32 4096
+    ; GFX9: [[C1:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 0
+    ; GFX9: G_STORE [[C1]](s32), [[C]](p5) :: (store 1, addrspace 5)
     %0:vgpr(p5) = G_CONSTANT i32 4096
     %1:vgpr(s32) = G_CONSTANT i32 0
     G_STORE %1, %0 :: (store 1, align 1, addrspace 5)
 
 ...

llvm/utils/TableGen/GlobalISelEmitter.cpp

if (Predicate.isLoad() && Predicate.isNonExtLoad()) {
       continue;
     }
     if (Predicate.isLoad() && Predicate.isAnyExtLoad()) {
       InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
           0, MemoryVsLLTSizePredicateMatcher::LessThan, 0);
       continue;
     }
 
-    if (Predicate.isStore() && Predicate.isTruncStore()) {
+    if (Predicate.isStore()) {
+      if (Predicate.isTruncStore()) {
         // FIXME: If MemoryVT is set, we end up with 2 checks for the MMO size.
         InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
             0, MemoryVsLLTSizePredicateMatcher::LessThan, 0);
         continue;
+      } else {
+        // We need to check the sizes match here otherwise we could incorrectly
+        // match truncating stores with non-truncating ones.
+        InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
+            0, MemoryVsLLTSizePredicateMatcher::EqualTo, 0);
+      }
     }
 
     // No check required. We already did it by swapping the opcode.
     if (!SrcGIEquivOrNull->isValueUnset("IfSignExtend") &&
         Predicate.isSignExtLoad())
       continue;
 
     // No check required. We already did it by swapping the opcode.