[GlobalISel] Check LLT size matches memory size for non-truncating stores.

[aemerson]

Summary

This was causing a bug where non-truncating stores would be selected instead of truncating ones.

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-store-flat.mir

106 ↗

(On Diff #210230)

This is breaking this, producing a non-truncating store instead of the correct trunc store

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-store-private.mir

55 ↗

(On Diff #210230)

This is breaking this, the correct trunc store now fails to select

llvm/utils/TableGen/GlobalISelEmitter.cpp

3360

Should also continue? Could make this a select on the match type to the one addPredicate call

[aemerson]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3360

Yeah, just seen this doesn't work. I'm not familiar with this code, but how do non-truncstore predicates end up on trunc stores such that it breaks?

[arsenm]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3360

I don't exactly understand what the expected relationship between predicate bits set on a PatFrag and the PatFrags which used it is.

It's possible something is wrong with the AMDGPU PatFrags. This system is pretty confusing, because each property is added in a different layer of PatFrags defined in TargetSelectionDAG.td. The AMDGPU PatFrags reproduce the same hierarchy, with the additional address space predicates.

There is also the extra IsTruncStore bit which needs to be manually set on the base truncstore PatFrag. For some reason the derivative PatFrags do not have it set, as it should be implied. However, they still need to set IsStore, although I would expect these to both behave the same way.

[arsenm]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3360

I'm also not sure the way this loop is structured makes sense. It seems to assume too much about what different predicates can be combined. The atomic predicates also don't work correctly

[aemerson]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3360

@dsanders any idea what to do?

[dsanders]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3360

That else is suspicious. isTruncStore() means that the predicate needs to be added but !isTruncStore() can mean either no predicate is needed or a non-trunc store predicate is needed. I believe it needs to be:

if (Predicate.isTruncStore())
  add relevant predicate for trunc store
if (Predicate.isNonTruncStore())
  add relevant predicate for non-trunc store

[dsanders]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3360

don't exactly understand what the expected relationship between predicate bits set on a PatFrag and the
PatFrags which used it is.

It's possible something is wrong with the AMDGPU PatFrags. This system is pretty confusing, because
each property is added in a different layer of PatFrags defined in TargetSelectionDAG.td. The AMDGPU
PatFrags reproduce the same hierarchy, with the additional address space predicates.

The input we start with is pretty weird too and it's mostly a straight import of that.

Suppose we start with the pattern:

(truncstorei8 node:$x)

when the PatFrag is expanded, we get:

(truncstore node:$x)<<code-to-check-memvt==i8>>

where the <<some C++ code>> is tablegen's way of representing predicates (this isn't quite true, it uses a short-hand name instead of the C++ when it's printing but the data in the pattern is the C++ code). Then after further expansion we get:

(unindexedstore node:$x)<<code-to-check-memvt==i8>><<code-to-check-for-truncating>>
(st node:$x)<<code-to-check-memvt==i8>><<code-to-check-for-truncating>><<code-to-check-for-unindexed>>

That last line is how it looks on input to the GlobalISelEmitter before runOnPattern(). The first thing we needed to do was abstract out the C++ as it needed to be different for DAGISel vs GISel. To do that, each predicate was given a field in PatFrag. For these three predicates it was:

ValueType MemoryVT = ?;
bit IsTruncStore = ?;
bit IsUnindexed = ?;

Only one of these is set for each PatFrag as each one directly corresponds to the original C++ code. For example: IsTruncStore = 0 emits this code for DAGISel:

!cast<StoreSDNode>(N)->isTruncatingStore();

I'm also not sure the way this loop is structured makes sense. It seems to assume too much about what
different predicates can be combined.

Each predicate can only be one C++ fragment. If you need multiple fragments (which happens when you have PatFrags inside PatFrag expansions) you have multiple Predicates, each with a different field set corresponding to a different fragment.

The atomic predicates also don't work correctly

Could you elaborate on this?

[arsenm]

llvm/utils/TableGen/GlobalISelEmitter.cpp

3360

The atomic predicates also don't work correctly

Could you elaborate on this?

I didn't spend much time looking at it to focus on one broken thing at a time, but the AMDGPU patterns for atomic_load/atomic_store don't work. They should be easy to handle as they ignore the ordering type and just need to preserve the atomicness in the MemOperand. I think the combination of IsStore and IsAtomic wasn't behaving as expected

[aemerson]

Use isNonTruncStore() instead.

[dsanders]

That change LGTM. Do you have a test case for your target?

[aemerson]

In D64845#1613091, @dsanders wrote:

That change LGTM. Do you have a test case for your target?

Thanks, it's only exposed with another patch though so I can't add that test here.

llvm/test/TableGen/address-space-patfrags.td

123

Just seen this incorrect comment, will fix in the commit.

llvm/test/TableGen/address-space-patfrags.td

def inst_b : Instruction {
   let InOperandList = (ins GPR32:$src);
 }
 
 def inst_c : Instruction {
   let OutOperandList = (outs);
   let InOperandList = (ins GPR32:$src0, GPR32:$src1);
 }
 
+def inst_d : Instruction {
+  let OutOperandList = (outs);
+  let InOperandList = (ins GPR32:$src0, GPR32:$src1);
+}
+
 // SDAG: case 2: {
 // SDAG-NEXT: // Predicate_pat_frag_b
 // SDAG-NEXT: SDNode *N = Node;
 // SDAG-NEXT: (void)N;
 // SDAG-NEXT: unsigned AddrSpace = cast<MemSDNode>(N)->getAddressSpace();
 // SDAG-NEXT: if (AddrSpace != 123 && AddrSpace != 455)
 // SDAG-NEXT: return false;
 // SDAG-NEXT: if (cast<MemSDNode>(N)->getMemoryVT() != MVT::i32) return false;

 // GISEL-NEXT: GIM_CheckAtomicOrdering, /*MI*/0, /*Order*/(int64_t)AtomicOrdering::NotAtomic,
 // GISEL-NEXT: // MIs[0] src0
 // GISEL-NEXT: GIM_CheckType, /*MI*/0, /*Op*/0, /*Type*/GILLT_s32,
 def : Pat <
   (truncstore GPR32:$src0, GPR32:$src1),
   (inst_c GPR32:$src0, GPR32:$src1)
 >;
 
 // Test truncstore with specific MemoryVT
 // GISEL: GIM_Try, /*On fail goto*//*Label 3*/ {{[0-9]+}}, // Rule ID 3 //
 // GISEL-NEXT: GIM_CheckNumOperands, /*MI*/0, /*Expected*/2,
 // GISEL-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_STORE,
+// GISEL-NEXT: GIM_CheckMemorySizeEqualToLLT, /*MI*/0, /*MMO*/0, /*OpIdx*/0,
+def : Pat <
+  (store GPR32:$src0, GPR32:$src1),
+  (inst_d GPR32:$src0, GPR32:$src1)
+>;
+
+// Test truncstore with specific MemoryVT
+// GISEL: GIM_Try, /*On fail goto*//*Label 4*/ {{[0-9]+}}, // Rule ID 4 //
+// GISEL-NEXT: GIM_CheckNumOperands, /*MI*/0, /*Expected*/2,
+// GISEL-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_STORE,
 // GISEL-NEXT: GIM_CheckMemorySizeLessThanLLT, /*MI*/0, /*MMO*/0, /*OpIdx*/0,
 // GISEL-NEXT: GIM_CheckMemoryAddressSpace, /*MI*/0, /*MMO*/0, /*NumAddrSpace*/2, /*AddrSpace*/123, /*AddrSpace*/455,
 // GISEL-NEXT: GIM_CheckMemorySizeEqualTo, /*MI*/0, /*MMO*/0, /*Size*/2,
 def : Pat <
   (truncstorei16_addrspace GPR32:$src0, GPR32:$src1),
   (inst_c GPR32:$src0, GPR32:$src1)
 >;

llvm/utils/TableGen/GlobalISelEmitter.cpp

if (Predicate.isLoad() && Predicate.isNonExtLoad()) {
       continue;
     }
     if (Predicate.isLoad() && Predicate.isAnyExtLoad()) {
       InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
           0, MemoryVsLLTSizePredicateMatcher::LessThan, 0);
       continue;
     }
 
-    if (Predicate.isStore() && Predicate.isTruncStore()) {
+    if (Predicate.isStore()) {
+      if (Predicate.isTruncStore()) {
         // FIXME: If MemoryVT is set, we end up with 2 checks for the MMO size.
         InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
             0, MemoryVsLLTSizePredicateMatcher::LessThan, 0);
         continue;
       }
+      if (Predicate.isNonTruncStore()) {
+        // We need to check the sizes match here otherwise we could incorrectly
+        // match truncating stores with non-truncating ones.
+        InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
+            0, MemoryVsLLTSizePredicateMatcher::EqualTo, 0);
+      }
+    }
 
     // No check required. We already did it by swapping the opcode.
     if (!SrcGIEquivOrNull->isValueUnset("IfSignExtend") &&
         Predicate.isSignExtLoad())
       continue;
 
     // No check required. We already did it by swapping the opcode.
     if (!SrcGIEquivOrNull->isValueUnset("IfZeroExtend") &&