[WIP][GlobalISel] Combiner Intrinsic Matching

[Pierre-vh]

Summary

Adds support for matching intrinsics in a match pattern in the combiner.
They can be matched using the same syntax as the DAG patterns, i.e.

(match (int_foo_intrinsic $d))

Current issues with the impl:

• Intrinsics cannot (yet) be used as leaf nodes if the previous node is a "fully-testable" node (e.g. a simple opcode match).

Solves #62628 (https://github.com/llvm/llvm-project/issues/62628)

[Pierre-vh]

It's not quite ready yet, but I need some help, so I'm leaving this as a WIP and just adding a few reviewers until the two issues highlighted are sorted out.
Once these are addressed I will add more reviewers + more testcases (including practical ones, e.g. add patterns using intrinsics in AMDGPU combiner) for a proper review.

llvm/test/TableGen/GICombinerEmitter/match-tree.td

373

Is it possible to de-duplicate this with the current MatchDag?

llvm/utils/TableGen/GlobalISel/GIMatchTree.cpp

462

This is the most annoying bit at the moment. It's quite restrictive and the example in the github issue doesn't work because of it: https://github.com/llvm/llvm-project/issues/62628

[arsenm]

I'm finding the lack of equivalent of ComplexPatterns and PatFrags to be frustrating. Should there be a new intermediate pattern source class to abstract over instructions?

llvm/test/TableGen/GICombinerEmitter/match-tree.td

35

Should have one that has side effects and one that doesn't to cover both G_INTRINSIC* cases.

Also, maybe try one with multiple return values. Should theoretically be matchable

llvm/utils/TableGen/GICombinerEmitter.cpp

64

These are supposed to use StringLiteral

[arsenm]

llvm/utils/TableGen/GlobalISel/GIMatchTree.cpp

462

Something is wrong with this "fully tested" code. It seems to just be wrong and mis-reports errors for cases which do match in the generated code (which is properly produced0

[Pierre-vh]

llvm/test/TableGen/GICombinerEmitter/match-tree.td

35

Will do, but about the G_INTRINSIC* case:
Currently I match both G_INTRINSIC and G_INSTRINC_W_SIDE_EFFECTS for any intrinsic. Is that the correct behaviour?

I did a quick look around in the tests, and I think they're interchangeable and we can't use whether the intrinsic has side effects or not to deduce the opcode to use, right? It's just that the "normal" opcode can be removed if it's unused, but the second one cannot because it's considered to have side effects?

llvm/utils/TableGen/GlobalISel/GIMatchTree.cpp

462

Indeed, I think it's not entirely right. I will take a deeper look at it but I also hope @dsanders can help here :)

[Pierre-vh]

In D150666#4345909, @arsenm wrote:

I'm finding the lack of equivalent of ComplexPatterns and PatFrags to be frustrating. Should there be a new intermediate pattern source class to abstract over instructions?

These intrinsic patterns basically work as (match opcode + run c++ match logic) so we could definitely piggyback on top of it when it lands to add complex patterns that match any opcode + some C++ logic

[Pierre-vh]

Address some comments, add multi ret test, add checking for number of arguments in instructions & intrinsics

[Pierre-vh]

llvm/utils/TableGen/GlobalISel/GIMatchTree.cpp

462

I just tried removing the error, but then I hit a crash in applyForPartition (Assertion Idx < Size && "Out-of-bounds Bit access."' failed.`) so I'm guessing there's indeed something wrong with the match tree

[arsenm]

llvm/test/TableGen/GICombinerEmitter/match-tree.td

35

I'd say more no than yes. There's a narrow set of cases where you could conceivably use G_INTRINSIC based on callsite attributes for a normally G_INTRINSIC_W_SIDE_EFFECTS. The DAG does not support this case. Off the top of my head the only cases where we could care about this are because of the limitations of buffer load intrinsics

TLDR not interchangeable

[Pierre-vh]

llvm/test/TableGen/GICombinerEmitter/match-tree.td

35

Are there any case where they could be interchangeable or not?

If not then I can just make the Intrinsic predicate a subclass of the "Opcode" predicate instead of "OneOfOpcode" predicate. It'll also fix one of my 2 issues which is duplicated code for both G_INTRINSIC instructions.

However if we think we may want to be able to match both G_INTRINSIC opcode for one pattern, then I can leave it in. I wouldn't be surprised if we see more G_INTRINSIC variants pop up so maybe there'll be a case down the line to handle 2+ opcodes for some kinds of intrinsics.

[arsenm]

llvm/test/TableGen/GICombinerEmitter/match-tree.td

35

The only case would be if we want to use callsite attributes to optimize opcode properties, but the only cases where we might want that are kind of workarounds. I wouldn't try to support that now

[Pierre-vh]

Address comments, handle side effect intrinsics
Now the last big issue is with reachability/leaf nodes.

[Pierre-vh]

Waiting to see if https://discourse.llvm.org/t/rfc-matchtable-based-globalisel-combiners/71457 makes it or not, if it does, then this will be abandoned.

[Pierre-vh]

Needs to be rewritten for new combiner

llvm/test/TableGen/GICombinerEmitter/match-invalid.td

 // RUN: not llvm-tblgen -I %p/../../../include -gen-global-isel-combiner \
 // RUN:     -combiners=MyCombiner %s 2>&1 | \
 // RUN:     FileCheck -implicit-check-not=error: %s
 
 include "llvm/Target/Target.td"
 include "llvm/Target/GlobalISel/Combine.td"
 
 def MyTargetISA : InstrInfo;
 def MyTarget : Target { let InstructionSet = MyTargetISA; }
 
 def dummy;
 
+def int_dummy : DefaultAttrsIntrinsic<[llvm_ptr_ty], [llvm_ptr_ty]>;
+
 def R0 : Register<"r0"> { let Namespace = "MyTarget"; }
 def GPR32 : RegisterClass<"MyTarget", [i32], 32, (add R0)>;
 class I<dag OOps, dag IOps, list<dag> Pat>
   : Instruction {
   let Namespace = "MyTarget";
   let OutOperandList = OOps;
   let InOperandList = IOps;
   let Pattern = Pat;

 
 def multidef_but_not_an_error: GICombineRule<
   (defs root:$a),
   (match (MOV $a, $b),
          (MOV $a, $b)),
   (dummy)>;
 // CHECK-NOT: :[[@LINE-5]]:{{[0-9]+}}: error:
 
+def not_enough_instr_operands : GICombineRule<
+  (defs root:$a),
+  (match (MOV $a)),
+  (dummy)>;
+// CHECK: :[[@LINE-4]]:{{[0-9]+}}: error: MOV expected 2 args, got 1
+// CHECK: :[[@LINE-5]]:{{[0-9]+}}: error: Expected a subclass of GIMatchKind or a sub-dag whose operator is either of a GIMatchKindWithArgs or Instruction
+// CHECK: :[[@LINE-6]]:{{[0-9]+}}: error: Failed to parse rule
+
+def too_many_instr_operands : GICombineRule<
+  (defs root:$a),
+  (match (MOV $a, $b, $c)),
+  (dummy)>;
+// CHECK: :[[@LINE-4]]:{{[0-9]+}}: error: MOV expected 2 args, got 3
+// CHECK: :[[@LINE-5]]:{{[0-9]+}}: error: Expected a subclass of GIMatchKind or a sub-dag whose operator is either of a GIMatchKindWithArgs or Instruction
+// CHECK: :[[@LINE-6]]:{{[0-9]+}}: error: Failed to parse rule
+
+def not_enough_intrin_operands : GICombineRule<
+  (defs root:$a),
+  (match (int_dummy $a)),
+  (dummy)>;
+// CHECK: :[[@LINE-4]]:{{[0-9]+}}: error: intrinsic 'llvm.dummy' expected 2 args, got 1
+// CHECK: :[[@LINE-5]]:{{[0-9]+}}: error: Expected a subclass of GIMatchKind or a sub-dag whose operator is either of a GIMatchKindWithArgs or Instruction
+// CHECK: :[[@LINE-6]]:{{[0-9]+}}: error: Failed to parse rule
+
+def too_many_intrin_operands : GICombineRule<
+  (defs root:$a),
+  (match (int_dummy $a, $b, $c)),
+  (dummy)>;
+// CHECK: :[[@LINE-4]]:{{[0-9]+}}: error: intrinsic 'llvm.dummy' expected 2 args, got 3
+// CHECK: :[[@LINE-5]]:{{[0-9]+}}: error: Expected a subclass of GIMatchKind or a sub-dag whose operator is either of a GIMatchKindWithArgs or Instruction
+// CHECK: :[[@LINE-6]]:{{[0-9]+}}: error: Failed to parse rule
+
 def MyCombiner: GICombinerHelper<"GenMyCombiner", [
 // CHECK: :[[@LINE-1]]:{{[0-9]+}}: error: Failed to parse one or more rules
   missing_match_node,
   null_matcher,
   unknown_kind1,
   unknown_kind2,
-  multidef
+  multidef,
+  not_enough_instr_operands,
+  too_many_instr_operands,
+  not_enough_intrin_operands,
+  too_many_intrin_operands
   // Rules omitted from a matcher can be as broken as you like. They will not be read.
   // multidef_but_not_an_error
 ]>;

llvm/test/TableGen/GICombinerEmitter/match-tree.td

 def ADD : I<(outs GPR32:$dst), (ins GPR32:$src1, GPR32:$src2), []>;
 def SUB : I<(outs GPR32:$dst), (ins GPR32:$src1, GPR32:$src2), []>;
 def MUL : I<(outs GPR32:$dst), (ins GPR32:$src1, GPR32:$src2), []>;
 def TRUNC : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
 def SEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
 def ZEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
 def ICMP : I<(outs GPR32:$dst), (ins GPR32:$tst, GPR32:$src1, GPR32:$src2), []>;
 
+def int_dummy : DefaultAttrsIntrinsic<[llvm_ptr_ty], [llvm_ptr_ty]>;
+def int_dummy_multi_ret : DefaultAttrsIntrinsic<[llvm_ptr_ty, llvm_ptr_ty], [llvm_ptr_ty]>;
+let TargetPrefix = "foo" in
+def int_foo_dummy_target : DefaultAttrsIntrinsic<[llvm_ptr_ty], []>;
+
 def HasFoo : Predicate<"Subtarget->hasFoo()">;
 def HasAnswerToEverything : Predicate<"Subtarget->getAnswerToUniverse() == 42 && Subtarget->getAnswerToLife() == 42">;
 
 def Rule0 : GICombineRule<
   (defs root:$d),
   (match (MUL $t, $s1, $s2),
          (SUB $d, $t, $s3)),
   (apply [{ APPLY }])>;
(match (ICMP $ic, $cc, $s2, $s3),
          [{ MATCH }]),
   (apply [{ APPLY }])>;
 
 def Rule9 : GICombineRule<
   (defs root:$d),
   (match (MUL $d, $t, $z)),
   (apply [{ APPLY }])>;
 
+def Rule10 : GICombineRule<
+  (defs root:$d),
+  (match (int_dummy $d, $s)),
+  (apply [{ APPLY (${d}), (${s}) }])>;
+
+def Rule11 : GICombineRule<
+  (defs root:$d),
+  (match (int_foo_dummy_target $d)),
+  (apply [{ APPLY (${d}) }])>;
+
+def Rule12 : GICombineRule<
+  (defs root:$d1),
+  (match (int_dummy_multi_ret $d1, $d2, $s1)),
+  (apply [{ APPLY (${d1}) (${d2}) (${s1}) }])>;
+
 def MyCombinerHelper: GICombinerHelper<"GenMyCombinerHelper", [
   Rule0,
   Rule1,
   Rule2,
   Rule3,
   Rule4,
   Rule5,
   Rule6,
   Rule7,
   Rule8,
-  Rule9
+  Rule9,
+  Rule10,
+  Rule11,
+  Rule12
 ]>;
 
 // CHECK-LABEL: digraph "matchtree" {
-// CHECK-DAG:   Node[[N0:(0x)?[0-9a-fA-F]+]] [shape=record,label="{MI[0].getOpcode()|5 partitions|Rule0,Rule1,Rule2,Rule3,Rule4,Rule5,Rule6,Rule7,Rule8,Rule9}"]
+// CHECK-DAG:   Node[[N0:(0x)?[0-9a-fA-F]+]] [shape=record,label="{MI[0].getOpcode()|7 partitions|Rule0,Rule1,Rule2,Rule3,Rule4,Rule5,Rule6,Rule7,Rule8,Rule9,Rule10,Rule11,Rule12}"]
 // CHECK-DAG:   Node[[N1:(0x)?[0-9a-fA-F]+]] [shape=record,label="{MI[1] = getVRegDef(MI[0].getOperand(1))|2 partitions|Rule0,Rule5}"]
 // CHECK-DAG:   Node[[N2:(0x)?[0-9a-fA-F]+]] [shape=record,label="{MI[1].getOpcode()|2 partitions|Rule0,Rule5}"]
 // CHECK-DAG:   Node[[N3:(0x)?[0-9a-fA-F]+]] [shape=record,label="{No partitioner|Rule0}"]
 // CHECK-DAG:   Node[[N4:(0x)?[0-9a-fA-F]+]] [shape=record,label="{No partitioner|Rule5}"]
 // CHECK-DAG:   Node[[N5:(0x)?[0-9a-fA-F]+]] [shape=record,label="{No partitioner|Rule5}"]
 // CHECK-DAG:   Node[[N6:(0x)?[0-9a-fA-F]+]] [shape=record,label="{MI[1] = getVRegDef(MI[0].getOperand(1))|2 partitions|Rule1,Rule2}"]
 // CHECK-DAG:   Node[[N7:(0x)?[0-9a-fA-F]+]] [shape=record,label="{MI[1].getOpcode()|2 partitions|Rule1,Rule2}"]
 // CHECK-DAG:   Node[[N8:(0x)?[0-9a-fA-F]+]] [shape=record,label="{No partitioner|Rule1}"]

 
 // Check the first partition. The numbers correspond to the labels above.
 // CODE:      switch (MIs[0]->getOpcode()) {
 // CODE-NEXT:   case MyTarget::SUB: Partition = 0; break;
 // CODE-NEXT:   case MyTarget::MOV: Partition = 1; break;
 // CODE-NEXT:   case MyTarget::ADD: Partition = 2; break;
 // CODE-NEXT:   case MyTarget::TRUNC: Partition = 3; break;
 // CODE-NEXT:   case MyTarget::MUL: Partition = 4; break;
+// CODE-NEXT:   case TargetOpcode::G_INTRINSIC: Partition = 5; break;
+// CODE-NEXT:   case TargetOpcode::G_INTRINSIC_W_SIDE_EFFECTS: Partition = 6; break;
 // CODE-NEXT: }
 
 // Check that the correct partition is choosen if operand 1 is a register.
 
 // CODE:       if (Partition == 0 /* MyTarget::SUB */) {
 // CODE-NEXT:    Partition = -1;
 // CODE-NEXT:    if (MIs.size() <= 1) MIs.resize(2);
 // CODE-NEXT:    MIs[1] = nullptr;

 // CODE-NEXT:            Subtarget->getAnswerToUniverse() == 42 && Subtarget->getAnswerToLife() == 42
 // CODE-NEXT:          )
 // CODE-NEXT:      ) {
 // CODE-NEXT:     LLVM_DEBUG(dbgs() << "Applying rule 'Rule6'\n");
 // CODE-NEXT:     APPLY
 // CODE-NEXT:     return true;
 // CODE-NEXT:   }
 // CODE-NEXT: }
+
+// Check intrinsic matching.
+// CODE:      if (Partition == 5 /* TargetOpcode::G_INTRINSIC */) {
+// CODE-NEXT:    // Leaf name: Rule10
+// CODE-NEXT:    // Rule: Rule10
+// CODE-NEXT:    if (!RuleConfig->isRuleDisabled(10)) {
+// CODE-NEXT:      if (1
+// CODE-NEXT:        && (
+// CODE-NEXT:             MIs[0]->getOperand(1).getIntrinsicID() == Intrinsic::dummy
+// CODE-NEXT:           )
+// CODE-NEXT:         ) {
+// CODE-NEXT:        LLVM_DEBUG(dbgs() << "Applying rule 'Rule10'\n");
+// CODE-NEXT:        APPLY (MIs[0]->getOperand(0)), (MIs[0]->getOperand(2))
+// CODE-NEXT:        return true;
+// CODE-NEXT:      }
+// CODE-NEXT:    }
+// CODE-NEXT:    // Leaf name: Rule11
+// CODE-NEXT:    // Rule: Rule11
+// CODE-NEXT:    if (!RuleConfig->isRuleDisabled(11)) {
+// CODE-NEXT:      if (1
+// CODE-NEXT:          && (
+// CODE-NEXT:              MIs[0]->getOperand(1).getIntrinsicID() == Intrinsic::foo_dummy_target
+// CODE-NEXT:             )
+// CODE-NEXT:         ) {
+// CODE-NEXT:        LLVM_DEBUG(dbgs() << "Applying rule 'Rule11'\n");
+// CODE-NEXT:        APPLY (MIs[0]->getOperand(0))
+// CODE-NEXT:        return true;
+// CODE-NEXT:      }
+// CODE-NEXT:    }
+// CODE-NEXT:    // Leaf name: Rule12
+// CODE-NEXT:    // Rule: Rule12
+// CODE-NEXT:    if (!RuleConfig->isRuleDisabled(12)) {
+// CODE-NEXT:      if (1
+// CODE-NEXT:          && (
+// CODE-NEXT:              MIs[0]->getOperand(2).getIntrinsicID() == Intrinsic::dummy_multi_ret
+// CODE-NEXT:             )
+// CODE-NEXT:         ) {
+// CODE-NEXT:          LLVM_DEBUG(dbgs() << "Applying rule 'Rule12'\n");
+// CODE-NEXT:          APPLY (MIs[0]->getOperand(0)) (MIs[0]->getOperand(1)) (MIs[0]->getOperand(3))
+// CODE-NEXT:          return true;
+// CODE-NEXT:      }
+// CODE-NEXT:    }
+// CODE-NEXT:    return false;
+// CODE-NEXT:}
+
+// TODO: This should be merged with the code above. The tree currently emits the same
+// code once for each G_INTRINSIC opcode variant which is useless bloat really.
+
+// CODE:      if (Partition == 6 /* TargetOpcode::G_INTRINSIC_W_SIDE_EFFECTS */) {
+// CODE-NEXT:    // Leaf name: Rule10
+// CODE-NEXT:    // Rule: Rule10
+// CODE-NEXT:    if (!RuleConfig->isRuleDisabled(10)) {
+// CODE-NEXT:      if (1
+// CODE-NEXT:        && (
+// CODE-NEXT:             MIs[0]->getOperand(1).getIntrinsicID() == Intrinsic::dummy
+// CODE-NEXT:           )
+// CODE-NEXT:         ) {
+// CODE-NEXT:        LLVM_DEBUG(dbgs() << "Applying rule 'Rule10'\n");
+// CODE-NEXT:        APPLY (MIs[0]->getOperand(0)), (MIs[0]->getOperand(2))
+// CODE-NEXT:        return true;
+// CODE-NEXT:      }
+// CODE-NEXT:    }
+// CODE-NEXT:    // Leaf name: Rule11
+// CODE-NEXT:    // Rule: Rule11
+// CODE-NEXT:    if (!RuleConfig->isRuleDisabled(11)) {
+// CODE-NEXT:      if (1
+// CODE-NEXT:          && (
+// CODE-NEXT:              MIs[0]->getOperand(1).getIntrinsicID() == Intrinsic::foo_dummy_target
+// CODE-NEXT:             )
+// CODE-NEXT:         ) {
+// CODE-NEXT:        LLVM_DEBUG(dbgs() << "Applying rule 'Rule11'\n");
+// CODE-NEXT:        APPLY (MIs[0]->getOperand(0))
+// CODE-NEXT:        return true;
+// CODE-NEXT:      }
+// CODE-NEXT:    }
+// CODE-NEXT:    // Leaf name: Rule12
+// CODE-NEXT:    // Rule: Rule12
+// CODE-NEXT:    if (!RuleConfig->isRuleDisabled(12)) {
+// CODE-NEXT:      if (1
+// CODE-NEXT:          && (
+// CODE-NEXT:              MIs[0]->getOperand(2).getIntrinsicID() == Intrinsic::dummy_multi_ret
+// CODE-NEXT:             )
+// CODE-NEXT:         ) {
+// CODE-NEXT:          LLVM_DEBUG(dbgs() << "Applying rule 'Rule12'\n");
+// CODE-NEXT:          APPLY (MIs[0]->getOperand(0)) (MIs[0]->getOperand(1)) (MIs[0]->getOperand(3))
+// CODE-NEXT:          return true;
+// CODE-NEXT:      }
+// CODE-NEXT:  }
+// CODE-NEXT:    return false;
+// CODE-NEXT:}

llvm/test/TableGen/GICombinerEmitter/parse-match-pattern.td

class I<dag OOps, dag IOps, list<dag> Pat>
   let Namespace = "MyTarget";
   let OutOperandList = OOps;
   let InOperandList = IOps;
   let Pattern = Pat;
 }
 def MOV : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
 def MOV2 : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
 
+def int_dummy : DefaultAttrsIntrinsic<[llvm_ptr_ty]>;
+
 def trivial : GICombineRule<
   (defs root:$d),
   (match (MOV $d, $s)),
   (apply [{ APPLY }])>;
 
 // CHECK-LABEL: Parsed rule defs/match for 'trivial'
 
 // The matchdag block is a fairly direct dump of the information that was read.

 // CHECK-NEXT:   Pred[[P2:(0x)?[0-9a-fA-F]+]] [shape=record,label="{{[{]{}}<s0>#0 $$|<s1>#1 $mi}|__anonpred4_3|$mi.getOpcode() == MOV|{{(0x)?[0-9a-fA-F]+}}|{<d0>#0 $$|<d1>#1 $mi}}",style=dotted]
 // CHECK-NEXT:   Pred[[P3:(0x)?[0-9a-fA-F]+]] [shape=record,label="{{[{]{}}<s0>#0 $$|<s1>#1 $mi0|<s2>#2 $mi1}|__anonpred4_4|$mi0 == $mi1|{{(0x)?[0-9a-fA-F]+}}|{<d0>#0 $$|<d1>#1 $mi0|<d2>#2 $mi1}}",style=dotted]
 // CHECK-NEXT:   Node[[N1]]:e -> Pred[[P1]]:d1:s [style=dotted]
 // CHECK-NEXT:   Node[[N2]]:e -> Pred[[P2]]:d1:s [style=dotted]
 // CHECK-NEXT:   Node[[N1]]:s1:n -> Pred[[P3]]:d1:s [style=dotted]
 // CHECK-NEXT:   Node[[N2]]:s1:n -> Pred[[P3]]:d2:s [style=dotted]
 // CHECK-NEXT: {{^}$}}
 
+def intrinsic_trivial : GICombineRule<
+  (defs root:$d),
+  (match (int_dummy $d)),
+  (apply [{ APPLY }])>;
+
+// CHECK-LABEL: Parsed rule defs/match for 'intrinsic_trivial'
+
+// CHECK-NEXT:  matchdag {
+// CHECK-NEXT:    (G_INTRINSIC 0:op0<def>, 1:$intrin_id, 2:op2):$__anon5_0 // $d=getOperand(0), $$intrin_id=getOperand(1)
+// CHECK-NEXT:    <<$mi.getOpcode() == oneof(G_INTRINSIC,G_INTRINSIC_W_SIDE_EFFECTS) && $intrin_id.getIntrinsicID() == Intrinsic::dummy>>:$__anonpred5_1
+// CHECK-NEXT:    __anon5_0 ==> __anonpred5_1[mi]
+// CHECK-NEXT:  }
+
+// CHECK-NEXT:  digraph "intrinsic_trivial" {
+// CHECK-NEXT:    rankdir="BT"
+// CHECK-NEXT:    Node[[N1:(0x)?[0-9a-fA-F]+]] [shape=record,label="{{[{]{}}<s0>#0 $op0|<s1>#1 $$intrin_id|<s2>#2 $op2}|__anon5_0|G_INTRINSIC|Match starts here|$d=getOperand(0), $$intrin_id=getOperand(1)|{{(0x)?[0-9a-fA-F]+}}|{<d0>#0 $op0|<d1>#1 $$intrin_id|<d2>#2 $op2}}",color=red]
+// CHECK-NEXT:    Pred[[P1:(0x)?[0-9a-fA-F]+]] [shape=record,label="{{[{]{}}<s0>#0 $$|<s1>#1 $mi}|__anonpred5_1|$mi.getOpcode() == oneof(G_INTRINSIC,G_INTRINSIC_W_SIDE_EFFECTS) && $intrin_id.getIntrinsicID() == Intrinsic::dummy|{{(0x)?[0-9a-fA-F]+}}|{<d0>#0 $$|<d1>#1 $mi}}",style=dotted]
+// CHECK-NEXT:    Node[[N1]]:e -> Pred[[P1]]:d1:s [style=dotted]
+// CHECK-NEXT:  }
+
 def MyCombiner: GICombinerHelper<"GenMyCombiner", [
   trivial,
   simple,
   multiroot,
   nonstandardroot,
-  multiref_use
+  multiref_use,
+  intrinsic_trivial
 ]>;
 
 // Verify we're sharing operand lists correctly
 // CHECK-LABEL: GIMatchDagOperandListContext {
 // CHECK-NEXT:    OperandLists {
 // CHECK-NEXT:      0:dst<def>, 1:src1{{$}}
 // CHECK-NEXT:      0:$<def>, 1:mi{{$}}
 // CHECK-NEXT:      0:$<def>, 1:mi0, 2:mi1{{$}}
+// CHECK-NEXT:      0:op0<def>, 1:$intrin_id, 2:op2
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }

llvm/utils/TableGen/CodeGenIntrinsics.h

struct CodeGenIntrinsic {
   /// false if the parameter is not a pointer, or \p ParamIdx is greater than
   /// the size of \p IS.ParamVTs.
   ///
   /// Note that this requires that \p IS.ParamVTs is available.
   bool isParamAPointer(unsigned ParamIdx) const;
 
   bool isParamImmArg(unsigned ParamIdx) const;
 
+  unsigned getNumDefs() const { return IS.RetTys.size(); }
+  unsigned getNumParams() const { return IS.ParamTys.size(); }
+
   CodeGenIntrinsic(Record *R, std::vector<Record *> DefaultProperties);
 };
 
 class CodeGenIntrinsicTable {
   std::vector<CodeGenIntrinsic> Intrinsics;
 
 public:
   struct TargetSet {

llvm/utils/TableGen/CodeGenTarget.h

public:
 
   CodeGenInstruction &getInstruction(const Record *InstRec) const {
     if (Instructions.empty()) ReadInstructions();
     auto I = Instructions.find(InstRec);
     assert(I != Instructions.end() && "Not an instruction");
     return *I->second;
   }
 
+  CodeGenInstruction &getInstructionByName(StringRef Name) const;
+
   /// Returns the number of predefined instructions.
   static unsigned getNumFixedInstructions();
 
   /// Returns the number of pseudo instructions.
   unsigned getNumPseudoInstructions() const {
     if (InstrsByEnum.empty())
       ComputeInstrsByEnum();
     return NumPseudoInstructions;

llvm/utils/TableGen/CodeGenTarget.cpp

GetInstByName(const char *Name,
   return I->second.get();
 }
 
 static const char *FixedInstrs[] = {
 #define HANDLE_TARGET_OPCODE(OPC) #OPC,
 #include "llvm/Support/TargetOpcodes.def"
     nullptr};
 
+CodeGenInstruction &CodeGenTarget::getInstructionByName(StringRef Name) const {
+  return getInstruction(Records.getDef(Name));
+}
+
 unsigned CodeGenTarget::getNumFixedInstructions() {
   return std::size(FixedInstrs) - 1;
 }
 
 /// Return all of the instructions defined by the target, ordered by
 /// their enum value.
 void CodeGenTarget::ComputeInstrsByEnum() const {
   const auto &Insts = getInstructions();

llvm/utils/TableGen/GICombinerEmitter.cpp

 //===- GlobalCombinerEmitter.cpp - Generate a combiner --------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 /// \file Generate a combiner implementation for GlobalISel from a declarative
 /// syntax
 ///
 //===----------------------------------------------------------------------===//
 
+#include "CodeGenInstruction.h"
+#include "CodeGenIntrinsics.h"
 #include "CodeGenTarget.h"
 #include "GlobalISel/CodeExpander.h"
 #include "GlobalISel/CodeExpansions.h"
 #include "GlobalISel/GIMatchDag.h"
 #include "GlobalISel/GIMatchDagEdge.h"
 #include "GlobalISel/GIMatchDagInstr.h"
 #include "GlobalISel/GIMatchDagOperands.h"
 #include "GlobalISel/GIMatchDagPredicate.h"
static cl::opt<bool> StopAfterParse(
     cl::desc("Stop processing after parsing rules and dump state"),
     cl::cat(GICombinerEmitterCat));
 static cl::opt<bool> StopAfterBuild(
     "gicombiner-stop-after-build",
     cl::desc("Stop processing after building the match tree"),
     cl::cat(GICombinerEmitterCat));
 
 namespace {
+static constexpr StringLiteral IntrinsicIDVarName = "$intrin_id";
+
 typedef uint64_t RuleID;
 
 // We're going to be referencing the same small strings quite a lot for operand
 // names and the like. Make their lifetime management simple with a global
 // string table.
 StringSet<> StrTab;
 
 StringRef insertStrTab(StringRef S) {

 }
 
 // Parse an (Instruction $a:Arg1, $b:Arg2, ...) matcher. Edges are formed
 // between matching operand names between different matchers.
 bool CombineRule::parseInstructionMatcher(
     const CodeGenTarget &Target, StringInit *ArgName, const Init &Arg,
     StringMap<std::vector<VarInfo>> &NamedEdgeDefs,
     StringMap<std::vector<VarInfo>> &NamedEdgeUses) {
-  if (const DagInit *Matcher =
-          getDagWithOperatorOfSubClass(Arg, "Instruction")) {
-    auto &Instr =
-        Target.getInstruction(Matcher->getOperatorAsDef(TheDef.getLoc()));
+
+  const DagInit *Matcher = getDagWithOperatorOfSubClass(Arg, "Instruction");
+  std::optional<CodeGenIntrinsic> Intrinsic;
+
+  if (!Matcher) {
+    Matcher = getDagWithOperatorOfSubClass(Arg, "Intrinsic");
+    if (!Matcher)
+      return false;
+
+    Intrinsic.emplace(Matcher->getOperatorAsDef(TheDef.getLoc()),
+                      std::vector<Record *>{});
+  }
+
+  // Collect all CodeGenInstructions. For non-intrinsics, this always only has
+  // one instruction. For intrinsics, it has all the variants of G_INTRINSICS
+  // supported.
+  SmallVector<CodeGenInstruction *, 2> Instrs;
+  if (Intrinsic) {
+    Instrs.push_back(&Target.getInstructionByName("G_INTRINSIC"));
+    Instrs.push_back(
+        &Target.getInstructionByName("G_INTRINSIC_W_SIDE_EFFECTS"));
+  } else {
+    Instrs.push_back(
+        &Target.getInstruction(Matcher->getOperatorAsDef(TheDef.getLoc())));
+  }
 
-    StringRef Name = ArgName ? ArgName->getValue() : "";
 
-    GIMatchDagInstr *N =
-        MatchDag.addInstrNode(makeDebugName(*this, Name), insertStrTab(Name),
-                              MatchDag.getContext().makeOperandList(Instr));
+  auto &MDCtx = MatchDag.getContext();
+  const auto &OpList = Intrinsic ? MDCtx.makeIntrinsicPredicateOperandList(
+                                       *Intrinsic, IntrinsicIDVarName)
+                                 : MDCtx.makeOperandList(*Instrs.front());
+  GIMatchDagInstr *N = MatchDag.addInstrNode(makeDebugName(*this, Name),
+                                             insertStrTab(Name), OpList);
+
+  N->setOpcodeAnnotation(Instrs.front());
+
+  StringRef PredName = makeNameForAnonPredicate(*this);
+  GIMatchDagPredicate *P = nullptr;
+
+  if (Intrinsic) {
+    // Add all variants of G_INTRINSIC.
+    auto IntrinPred = MatchDag.addPredicateNode<GIMatchDagIntrinsicPredicate>(
+        *Intrinsic, IntrinsicIDVarName, PredName);
+    for (const auto *Instr : Instrs)
+      IntrinPred->addOpcode(Instr);
+    P = IntrinPred;
+  } else {
+    P = MatchDag.addPredicateNode<GIMatchDagOpcodePredicate>(PredName,
+                                                             *Instrs.front());
+  }
 
-    N->setOpcodeAnnotation(&Instr);
-    const auto &P = MatchDag.addPredicateNode<GIMatchDagOpcodePredicate>(
-        makeNameForAnonPredicate(*this), Instr);
-    MatchDag.addPredicateDependency(N, nullptr, P, &P->getOperandInfo()["mi"]);
-    unsigned OpIdx = 0;
-    for (const auto &NameInit : Matcher->getArgNames()) {
-      StringRef Name = insertStrTab(NameInit->getAsUnquotedString());
-      if (Name.empty())
-        continue;
-      N->assignNameToOperand(OpIdx, Name);
+
+  // Check that the number of arguments is correct.
+  const unsigned NumMatcherArgs = Matcher->getNumArgs();
+  const unsigned ExpectedMatcherArgs =
+      Intrinsic ? (Intrinsic->getNumDefs() + Intrinsic->getNumParams())
+                : Instrs.front()->Operands.size();
+  if (NumMatcherArgs != ExpectedMatcherArgs) {
+    if (Intrinsic)
+      PrintError(TheDef.getLoc(), "intrinsic '" + Intrinsic->Name +
+                                      "' expected " +
+                                      Twine(ExpectedMatcherArgs) +
+                                      " args, got " + Twine(NumMatcherArgs));
+    else
+      PrintError(TheDef.getLoc(), Instrs.front()->TheDef->getName() +
+                                      " expected " +
+                                      Twine(ExpectedMatcherArgs) +
+                                      " args, got " + Twine(NumMatcherArgs));
+    return false;
+  }
+
+    // Insert IntrinsicID operand after the defs.
+    if (Intrinsic && Intrinsic->getNumDefs() == OpIdx)
+      N->assignNameToOperand(OpIdx++, IntrinsicIDVarName);
+
 
-      // Record the endpoints of any named edges. We'll add the cartesian
-      // product of edges later.
-      const auto &InstrOperand = N->getOperandInfo()[OpIdx];
-      if (InstrOperand.isDef()) {
-        NamedEdgeDefs.try_emplace(Name);
-        NamedEdgeDefs[Name].emplace_back(N, &InstrOperand, Matcher);
-      } else {
-        NamedEdgeUses.try_emplace(Name);
-        NamedEdgeUses[Name].emplace_back(N, &InstrOperand, Matcher);
-      }
 
-      if (InstrOperand.isDef()) {
-        if (any_of(Roots, [&](const RootInfo &X) {
-              return X.getPatternSymbol() == Name;
-            })) {
-          N->setMatchRoot();
-        }
-      }
 
-      OpIdx++;
-    }
 
-    return true;
-  }
-  return false;
-}
+  // Insert IntrinsicID operand after the defs (for intrinsics without any
+  // parameters).
+  if (Intrinsic && Intrinsic->getNumDefs() == OpIdx)
+    N->assignNameToOperand(OpIdx++, IntrinsicIDVarName);
+
 
 // Parse the wip_match_opcode placeholder that's temporarily present in lieu of
 // implementing macros or choices between two matchers.
 bool CombineRule::parseWipMatchOpcodeMatcher(const CodeGenTarget &Target,
                                              StringInit *ArgName,
                                              const Init &Arg) {
   if (const DagInit *Matcher =
           getDagWithSpecificOperator(Arg, "wip_match_opcode")) {
for (const auto &Leaf : Tree.possible_leaves()) {
     // Attempt to emit code for any untested predicates left over. Note that
     // isFullyTested() will remain false even if we succeed here and therefore
     // combine rule elision will not be performed. This is because we do not
     // know if there's any connection between the predicates for each leaf and
     // therefore can't tell if one makes another unreachable. Ideally, the
     // partitioner(s) would be sufficiently complete to prevent us from having
     // untested predicates left over.
     for (const GIMatchDagPredicate *Predicate : Leaf.untested_predicates()) {
-      if (Predicate->generateCheckCode(OS, (Indent + "      ").str(),
-                                       Expansions))
-        continue;
-      PrintError(RuleDef.getLoc(),
-                 "Unable to test predicate used in rule");
+      OS << Indent << "      && (\n";
+      if (!Predicate->generateCheckCode(OS, (Indent + "          ").str(),
+                                        Expansions)) {
+        PrintError(RuleDef.getLoc(), "Unable to test predicate used in rule");
         PrintNote(SMLoc(),
                   "This indicates an incomplete implementation in tablegen");
         Predicate->print(errs());
         errs() << "\n";
         OS << Indent
            << "llvm_unreachable(\"TableGen did not emit complete code for this "
               "path\");\n";
         break;
       }
+      OS << Indent << "         )\n";
+      continue;
+    }
 
     if (Rule->getMatchingFixupCode() &&
         !Rule->getMatchingFixupCode()->getValue().empty()) {
       // FIXME: Single-use lambda's like this are a serious compile-time
       // performance and memory issue. It's convenient for this early stage to
       // defer some work to successive patches but we need to eliminate this
       // before the ruleset grows to small-moderate size. Last time, it became
       // a big problem for low-mem systems around the 500 rule mark but by the

llvm/utils/TableGen/GlobalISel/GIMatchDag.h

public:
   const GIMatchDagOperandList &makeOperandList(const CodeGenInstruction &I) {
     return OperandListCtx.makeOperandList(I);
   }
 
   const GIMatchDagOperandList &makeMIPredicateOperandList() {
     return OperandListCtx.makeMIPredicateOperandList();
   }
 
+  const GIMatchDagOperandList &
+  makeIntrinsicPredicateOperandList(const CodeGenIntrinsic &I,
+                                    StringRef IntrinsicIDVarName) {
+    return OperandListCtx.makeIntrinsicPredicateOperandList(I,
+                                                            IntrinsicIDVarName);
+  }
 
   const GIMatchDagOperandList &makeTwoMOPredicateOperandList() {
     return OperandListCtx.makeTwoMOPredicateOperandList();
   }
 
   void print(raw_ostream &OS) const {
     OperandListCtx.print(OS);
   }

llvm/utils/TableGen/GlobalISel/GIMatchDagOperands.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_UTILS_TABLEGEN_GIMATCHDAGOPERANDS_H
 #define LLVM_UTILS_TABLEGEN_GIMATCHDAGOPERANDS_H
 
 #include "llvm/ADT/FoldingSet.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/StringSet.h"
 #include "llvm/Support/raw_ostream.h"
 
 #include <vector>
 
 namespace llvm {
 class CodeGenInstruction;
+struct CodeGenIntrinsic;
+
 /// Describes an operand of a MachineInstr w.r.t the DAG Matching. This
 /// information is derived from CodeGenInstruction::Operands but is more
 /// readily available for context-less access as we don't need to know which
 /// instruction it's used with or know how many defs that instruction had.
 ///
 /// There may be multiple GIMatchDagOperand's with the same contents. However,
 /// they are uniqued within the set of instructions that have the same overall
 /// operand list. For example, given:

 };
 
 /// This is the portion of GIMatchDagContext that directly relates to
 /// GIMatchDagOperandList and GIMatchDagOperandList.
 class GIMatchDagOperandListContext {
   FoldingSet<GIMatchDagOperandList> OperandLists;
   std::vector<std::unique_ptr<GIMatchDagOperandList>> OperandListsOwner;
 
+  StringSet<> StrTab;
+
+  StringRef createOperandName(unsigned Idx) {
+    return StrTab.insert("op" + std::to_string(Idx)).first->first();
+  }
+
 public:
   const GIMatchDagOperandList &makeEmptyOperandList();
   const GIMatchDagOperandList &makeOperandList(const CodeGenInstruction &I);
   const GIMatchDagOperandList &makeMIPredicateOperandList();
   const GIMatchDagOperandList &makeTwoMOPredicateOperandList();
+  const GIMatchDagOperandList &
+  makeIntrinsicPredicateOperandList(const CodeGenIntrinsic &I,
+                                    StringRef IntrinsicIDVarName);
 
   void print(raw_ostream &OS) const;
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
   LLVM_DUMP_METHOD void dump() const { print(errs()); }
 #endif // if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 };
 
 } // end namespace llvm
 #endif // ifndef LLVM_UTILS_TABLEGEN_GIMATCHDAGOPERANDS_H

llvm/utils/TableGen/GlobalISel/GIMatchDagOperands.cpp

 //===- GIMatchDagOperands.cpp - A shared operand list for nodes -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "GIMatchDagOperands.h"
 
 #include "../CodeGenInstruction.h"
+#include "../CodeGenIntrinsics.h"
 
 using namespace llvm;
 
 void GIMatchDagOperand::Profile(FoldingSetNodeID &ID) const {
   Profile(ID, Idx, Name, IsDef);
 }
 
 void GIMatchDagOperand::Profile(FoldingSetNodeID &ID, size_t Idx,
std::unique_ptr<GIMatchDagOperandList> NewValue =
       std::make_unique<GIMatchDagOperandList>();
   NewValue->add("$", 0, true);
   NewValue->add("mi", 1, false);
   OperandLists.InsertNode(NewValue.get(), InsertPoint);
   OperandListsOwner.push_back(std::move(NewValue));
   return *OperandListsOwner.back().get();
 }
 
+const GIMatchDagOperandList &
+GIMatchDagOperandListContext::makeIntrinsicPredicateOperandList(
+    const CodeGenIntrinsic &I, StringRef IntrinsicIDVarName) {
+  FoldingSetNodeID ID;
+  const unsigned NumOuts = I.getNumDefs();
+  const unsigned NumIns = I.getNumParams();
+
+  for (unsigned i = 0; i < (NumOuts + NumIns + 1); ++i) {
+    if (i == NumOuts)
+      GIMatchDagOperand::Profile(ID, i++, IntrinsicIDVarName, false);
+    GIMatchDagOperand::Profile(ID, i, createOperandName(i), i < NumOuts);
+  }
+
+  void *InsertPoint;
+  GIMatchDagOperandList *Value =
+      OperandLists.FindNodeOrInsertPos(ID, InsertPoint);
+  if (Value)
+    return *Value;
+
+  std::unique_ptr<GIMatchDagOperandList> NewValue =
+      std::make_unique<GIMatchDagOperandList>();
+  for (unsigned i = 0; i < (NumOuts + NumIns + 1); ++i) {
+    if (i == NumOuts)
+      NewValue->add(IntrinsicIDVarName, i++, false);
+    NewValue->add(createOperandName(i), i, i < NumOuts);
+  }
+  OperandLists.InsertNode(NewValue.get(), InsertPoint);
+  OperandListsOwner.push_back(std::move(NewValue));
+  return *OperandListsOwner.back().get();
+}
 
 const GIMatchDagOperandList &
 GIMatchDagOperandListContext::makeTwoMOPredicateOperandList() {
   FoldingSetNodeID ID;
   GIMatchDagOperand::Profile(ID, 0, "$", true);
   GIMatchDagOperand::Profile(ID, 1, "mi0", false);
   GIMatchDagOperand::Profile(ID, 2, "mi1", false);
 

llvm/utils/TableGen/GlobalISel/GIMatchDagPredicate.h

 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 #include "llvm/Support/raw_ostream.h"
 #endif
 
 namespace llvm {
 class CodeExpansions;
 class CodeGenInstruction;
+struct CodeGenIntrinsic;
 class GIMatchDagOperandList;
 class GIMatchDagContext;
 class raw_ostream;
 
 /// Represents a predicate on the match DAG. This records the details of the
 /// predicate. The dependencies are stored in the GIMatchDag as edges.
 ///
 /// Instances of this class objects are owned by the GIMatchDag and are not
 /// shareable between instances of GIMatchDag.
 class GIMatchDagPredicate {
 public:
   enum GIMatchDagPredicateKind {
     GIMatchDagPredicateKind_Opcode,
     GIMatchDagPredicateKind_OneOfOpcodes,
+    GIMatchDagPredicateKind_Intrinsic,
     GIMatchDagPredicateKind_SameMO,
   };
 
 protected:
   const GIMatchDagPredicateKind Kind;
 
   /// The name of the predicate. For example:
   ///     (FOO $a:s32, $b, $c)

 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
   LLVM_DUMP_METHOD void dump() const override { print(errs()); }
 #endif // if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 };
 
 class GIMatchDagOneOfOpcodesPredicate : public GIMatchDagPredicate {
   SmallVector<const CodeGenInstruction *, 4> Instrs;
 
+protected:
+  GIMatchDagOneOfOpcodesPredicate(GIMatchDagPredicateKind Kind,
+                                  GIMatchDagContext &Ctx, StringRef Name);
+
 public:
-  GIMatchDagOneOfOpcodesPredicate(GIMatchDagContext &Ctx, StringRef Name);
+  GIMatchDagOneOfOpcodesPredicate(GIMatchDagContext &Ctx, StringRef Name)
+      : GIMatchDagOneOfOpcodesPredicate(GIMatchDagPredicateKind_OneOfOpcodes,
+                                        Ctx, Name) {}
 
   void addOpcode(const CodeGenInstruction *Instr) { Instrs.push_back(Instr); }
 
   static bool classof(const GIMatchDagPredicate *P) {
-    return P->getKind() == GIMatchDagPredicateKind_OneOfOpcodes;
+    return P->getKind() == GIMatchDagPredicateKind_OneOfOpcodes ||
+           P->getKind() == GIMatchDagPredicateKind_Intrinsic;
   }
 
   const SmallVectorImpl<const CodeGenInstruction *> &getInstrs() const {
     return Instrs;
   }
 
   void printDescription(raw_ostream &OS) const override;
 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
   LLVM_DUMP_METHOD void dump() const override { print(errs()); }
 #endif // if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 };
 
+class GIMatchDagIntrinsicPredicate : public GIMatchDagOneOfOpcodesPredicate {
+  std::string IntrinEnumName;
+  StringRef IntrinsicIDVarName;
+
+public:
+  GIMatchDagIntrinsicPredicate(GIMatchDagContext &Ctx,
+                               const CodeGenIntrinsic &I,
+                               StringRef IntrinsicIDVarName, StringRef Name);
+
+  static bool classof(const GIMatchDagPredicate *P) {
+    return P->getKind() == GIMatchDagPredicateKind_Intrinsic;
+  }
+
+  std::string getIntrinsicEnumName() const { return IntrinEnumName; }
+  StringRef getIntrinsicIDVarName() const { return IntrinsicIDVarName; }
+
+  void printDescription(raw_ostream &OS) const override;
+
+  virtual bool
+  generateCheckCode(raw_ostream &OS, StringRef Indent,
+                    const CodeExpansions &Expansions) const override;
+
+#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
+  LLVM_DUMP_METHOD void dump() const override { print(errs()); }
+#endif // if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
+};
+
 class GIMatchDagSameMOPredicate : public GIMatchDagPredicate {
 public:
   GIMatchDagSameMOPredicate(GIMatchDagContext &Ctx, StringRef Name);
 
   static bool classof(const GIMatchDagPredicate *P) {
     return P->getKind() == GIMatchDagPredicateKind_SameMO;
   }
 

llvm/utils/TableGen/GlobalISel/GIMatchDagPredicate.cpp

 //===- GIMatchDagPredicate.cpp - Represent a predicate to check -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "GIMatchDagPredicate.h"
 
 #include "llvm/TableGen/Record.h"
 
 #include "../CodeGenInstruction.h"
+#include "../CodeGenIntrinsics.h"
+#include "../CodeGenTarget.h"
+#include "CodeExpansions.h"
 #include "GIMatchDag.h"
 
 using namespace llvm;
 
 void GIMatchDagPredicate::print(raw_ostream &OS) const {
   OS << "<<";
   printDescription(OS);
   OS << ">>:$" << Name;
 }
 
 void GIMatchDagPredicate::printDescription(raw_ostream &OS) const { OS << ""; }
 
 GIMatchDagOpcodePredicate::GIMatchDagOpcodePredicate(
     GIMatchDagContext &Ctx, StringRef Name, const CodeGenInstruction &Instr)
     : GIMatchDagPredicate(GIMatchDagPredicateKind_Opcode, Name,
                           Ctx.makeMIPredicateOperandList()),
       Instr(Instr) {}
 
 void GIMatchDagOpcodePredicate::printDescription(raw_ostream &OS) const {
   OS << "$mi.getOpcode() == " << Instr.TheDef->getName();
 }
 
 GIMatchDagOneOfOpcodesPredicate::GIMatchDagOneOfOpcodesPredicate(
-    GIMatchDagContext &Ctx, StringRef Name)
-    : GIMatchDagPredicate(GIMatchDagPredicateKind_OneOfOpcodes, Name,
-                          Ctx.makeMIPredicateOperandList()) {}
+    GIMatchDagPredicateKind Kind, GIMatchDagContext &Ctx, StringRef Name)
+    : GIMatchDagPredicate(Kind, Name, Ctx.makeMIPredicateOperandList()) {}
 
 void GIMatchDagOneOfOpcodesPredicate::printDescription(raw_ostream &OS) const {
   OS << "$mi.getOpcode() == oneof(";
   StringRef Separator = "";
   for (const CodeGenInstruction *Instr : Instrs) {
     OS << Separator << Instr->TheDef->getName();
     Separator = ",";
   }
   OS << ")";
 }
 
+GIMatchDagIntrinsicPredicate::GIMatchDagIntrinsicPredicate(
+    GIMatchDagContext &Ctx, const CodeGenIntrinsic &I,
+    StringRef IntrinsicIDVarName, StringRef Name)
+    : GIMatchDagOneOfOpcodesPredicate(GIMatchDagPredicateKind_Intrinsic, Ctx,
+                                      Name),
+      IntrinEnumName(I.EnumName), IntrinsicIDVarName(IntrinsicIDVarName) {}
+
+void GIMatchDagIntrinsicPredicate::printDescription(raw_ostream &OS) const {
+  GIMatchDagOneOfOpcodesPredicate::printDescription(OS);
+  OS << " && " << IntrinsicIDVarName
+     << ".getIntrinsicID() == Intrinsic::" << IntrinEnumName;
+}
+
+bool GIMatchDagIntrinsicPredicate::generateCheckCode(
+    raw_ostream &OS, StringRef Indent, const CodeExpansions &Expansions) const {
+  assert(!getInstrs().empty() && "No G_INTRINSIC opcode added!");
+
+  const auto IDVar = Expansions.lookup(IntrinsicIDVarName);
+  assert(!IDVar.empty() && "Expansion of intrinsic ID operand not defined!");
+  OS << Indent << IDVar << ".getIntrinsicID() == Intrinsic::" << IntrinEnumName
+     << "\n";
+  return true;
+}
+
 GIMatchDagSameMOPredicate::GIMatchDagSameMOPredicate(GIMatchDagContext &Ctx,
                                                      StringRef Name)
     : GIMatchDagPredicate(GIMatchDagPredicateKind_SameMO, Name,
                           Ctx.makeTwoMOPredicateOperandList()) {}
 
 void GIMatchDagSameMOPredicate::printDescription(raw_ostream &OS) const {
   OS << "$mi0 == $mi1";
 }

llvm/utils/TableGen/GlobalISel/GIMatchTree.cpp

for (unsigned PIdx : Leaf.value().TestablePredicates.set_bits()) {
         // If we get here twice then we've somehow ended up with two opcode
         // predicates for one instruction in the same DAG. That should be
         // impossible.
         assert(AllOpcodes && "Conflicting opcode predicates");
         append_range(OpcodesForThisPredicate, OpcodeP->getInstrs());
       }
 
       for (const CodeGenInstruction *Expected : OpcodesForThisPredicate) {
-        // Mark this predicate as one we're testing.
+        // Mark this predicate as tested, unless it needs additional match code.
+
+        // FIXME: This is needed, but prevents patterns such as
+        //   (match (G_FPTRUNC $fptrunc_dst, $fmed3_dst):$fptrunc,
+        //          (int_amdgcn_fmed3 $fmed3_dst, $src0, $src1, $src2))
+        // because G_FPTRUNC would be fully tested, and int_amdgcn_fmed3 becomes
+        // unreachble.
+        if (!isa<GIMatchDagIntrinsicPredicate>(P))
           TestedPredicatesForLeaf.set(PIdx);
 
         // Partitions must be numbered 0, 1, .., N but instructions don't meet
         // that requirement. Assign a partition number to each opcode if we
         // lack one ...
         auto Partition = InstrToPartition.find(Expected);
         if (Partition == InstrToPartition.end()) {
           BitVector Contents(Leaves.size());
           Partition = InstrToPartition