[RFC][GlobalISel] Overhauled MIR Patterns Support for Combiners

[Pierre-vh]

Summary

See https://discourse.llvm.org/t/rfc-overhauled-mir-patterns-for-globalisel-combiners/72264

This is a complete overrhaul of the recently-added GlobalISel Match Table backend which adds
support for MIR patterns for both match and apply patterns.

[Pierre-vh]

Improve findRoot + support variadics

[arsenm]

This is much clearer than what was documented before

llvm/docs/GlobalISel/MIRPatterns.rst

88–90

This is unusably broken? You can't produce a valid G_* instruction with a cimm operand

110

maybe mention somewhere that you need to copy.

How are patterns that discard their outputs handled? Does it error on those?

llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutorImpl.h

730–736

Maybe should invest in having a variable length encoded match table. And/or could have a lookup table of known used wide constants

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/patfrag-errors.td

222

Could this also state what it actually was instead?

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-errors.td

5

Lots of diagnostic tests is very good

173

too little/too few

llvm/utils/TableGen/GlobalISelMatchTable.h

1943–1945

Could this be APInt? Should there be a separate APInt renderer?

1962

Typo ConstantINt

[arsenm]

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-variadics.td

29

How would I go about enforcing element types here?

[Pierre-vh]

Address comments

llvm/docs/GlobalISel/MIRPatterns.rst

88–90

G_CONSTANT uses a CImm, AFAIK
See MachineIRBuilder::buildConstant with addCImm uses.

Do you have example of instructions that need a direct imm? I can test them and fix them.

I'm also wondering if I should just go ahead and make immediates always emit G_CONSTANT except in specific cases (currently only G_CONSTANT I think?). It'd be a usability improvement.

110

How are patterns that discard their outputs handled? Does it error on those?

Good question, I think it will indeed error out.
Do you mean that there's patterns that just delete instructions? I could add a special case for (apply) to just erase the root. Would that work for you?

llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutorImpl.h

730–736

We can do that in the future, yes, but I think it's too much for now. Aren't most constants small anyway? Do we have patterns with constants >64 bits? In the meantime I'll add it to the limitations.
Also note that for a vector splat, this only takes the scalar element size, so if you want to check that a 4 x i64 has a splat value of 0, that works.

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-variadics.td

29

This is not yet possible, I forgot to add that but it's not much effort (in theory), all of the infrastructure is there :)

I'll add it. To be sure do you mean something like (i32 $x) to add a check for register type?

llvm/utils/TableGen/GlobalISelMatchTable.h

1943–1945

We'd need a separate renderer APInt that can split the APInt into multiple 64-bit pieces & reassemble it after.
Do we even have a usecase for now for >64 bits here? I thought we didn't have any so I didn't think it was an issue.

[Pierre-vh]

Add typed machine operands

I opted for the syntax type:$name which is more consistent I believe.
It seems to work just fine, I even added some inference to avoid repeating types.
It's also smart enough to not reemit a type check everytime that operand is used again. It only emits it once.

[arsenm]

llvm/docs/GlobalISel/MIRPatterns.rst

88–90

addImm uses correspond to immarg arguments. Everything else uses G_CONSTANT with a isCImm argument as a materialized constant

110

We have some special case atomic optimizations that discard the result if use_empty, it's currently dealt with in selection but theoretically could be a combine

[Pierre-vh]

llvm/docs/GlobalISel/MIRPatterns.rst

88–90

Right, so I can just look at uses of addImm and special-case all of those instructions, then default to inserting a G_CONSTANT for the rest? Does that work?

Should I just special-case them in code, or add some field to GenericInstruction in TableGen and infer it from there?

110

If I understand correctly it'd look like this?

(match (G_ATOMIC_FOO $x, $y), "return MRI.use_empty(${x}.getReg())"))
(apply) // somehow delete it

Then yeah, that's not supported so far. I can add a special case to just delete the root if there are no apply patterns if you think that'd work?

An easy workaround would be to just set $x to IMPLICIT_DEF, but if G_ATOMIC_FOO has no defs then there's no workaround other than using C++ in the (apply).

[Pierre-vh]

Correctly pre-allocate all temporary registers

[gargaroff]

Awesome to finally see this, thanks for doing this! I checked the documentation and only had two minor nits.

llvm/docs/GlobalISel/MIRPatterns.rst

290

298

[Pierre-vh]

Fix docs

[Pierre-vh]

Minor changes + fix the name of anonymous patterns. Before it was very cryptic, now it's more structured.
e.g. __MatchFooPerms_alt0_pattern_0 for the first pattern of the first alternative for the PatFrag MatchFooPerms.

[arsenm]

llvm/docs/GlobalISel/MIRPatterns.rst

110

Yes the implicit_def thing should be good enough. It's not that important, but would be nice to document

[Pierre-vh]

• Simplify temp reg emission using insertAction
• Fix immediate handling in apply patterns (check updated docs for the full breakdown).

Something I also want to point out is that some of the things here haven't been tested
in practice (mostly the register type constraints & the immediates handling stuff in apply). I am confident that most things work just fine, but with this level of
complexity it's completely possible there's silly bugs here and there.
I'm anticipating that a lot of small fix patches will need to be made over time as
people start using MIR patterns and notice issues.

[Pierre-vh]

llvm/lib/CodeGen/GlobalISel/GIMatchTableExecutor.cpp

37

I'd like to point this out: Now, matching constants will also match Splats. This is a harmless change for ISel (DAGISel pattern import in GISel) I think because all immediates are typed there, so even if (i32 0) matches a splat for <4 x i32> the typecheck should fail. I didn't notice any test changes doing this so I assume we're good.

If we want to avoid ISel having the same behavior that's possible, I can just create a derived class and only implement this change there for Combiners.

[Pierre-vh]

llvm/utils/TableGen/GlobalISelCombinerMatchTableEmitter.cpp

1890

Note here too, I'm not sure if I should just do everything in terms for permutations.
I special-cased wip_match_opcode for now because it can only ever be the root pattern, but there might be value in making everything follow the same pattern.

[Pierre-vh]

Split up OperandTable + add named_operands() accessor for InstructionPatterns.
There is one TODO in there about using a pair of OperandTables instead of a complicated RuleOperandTable, I plan to look into that as soon as I can but review can continue in the meantime (I can always fix it later if needed anyway, it's not broken - just a code improvement)

[arsenm]

llvm/docs/GlobalISel/MIRPatterns.rst

50

Do these support vectors. i.e. can you use a constant and implicitly get a constant splat? vector constants are a recurring pain point

llvm/lib/CodeGen/GlobalISel/GIMatchTableExecutor.cpp

37

this is a plus?

[arsenm]

llvm/utils/TableGen/GlobalISelCombinerMatchTableEmitter.cpp

1942

don't need this string copy, can directly check and write out the twine

2142

continue on null and reduce indent?

2184

single quotes

2188

single quotes

2194

single quotes

2437

Move the insert before the move?

[Pierre-vh]

Comments

[Pierre-vh]

llvm/docs/GlobalISel/MIRPatterns.rst

50

The operand's type is checked, so 0 can match a splat, but i32 0 can only match a constant of type i32 (a register).
See match-table-imms.td InstTest0 vs InstTest1

llvm/utils/TableGen/GlobalISelCombinerMatchTableEmitter.cpp

2437

I still need to return the pointer after so I need to save it anyway, so it doesn't matter if I do SeenPatFrags.insert(NewPatFrag.get());

[Pierre-vh]

Use a singular, simple OperandTable datastructure for both PatFrags & Rules.
Works a lot better and eliminated about 70 lines of code.

[aemerson]

llvm/docs/GlobalISel/MIRPatterns.rst

190–193

It doesn't have to be in the initial patch, but can this be addressed later? If we migrate to using MIR patterns more often, this seems wasteful (generating a new inst, observer call, CSE, DCE)

Similar thing for the replacing a register case, it would be good to have a way to trigger a MRI.replaceRegWith().

349

s/in any all/in all

llvm/utils/TableGen/GlobalISelCombinerMatchTableEmitter.cpp

650–651

can move this above the definition of Def

1991

I'd prefer to not use single letter variable names, Alt and Op are fine for me.

2203

This method could use some documentation either here or in the definition,

3358–3367

Should we use a SmallVector for these?

[Pierre-vh]

Comments

llvm/docs/GlobalISel/MIRPatterns.rst

190–193

Sure, I'm leaning towards adding more apply magic operators so we can do things like (apply (erase $d), (replace $z, $x)) but I haven't looked into it yet. I'd do that in a follow up patch.

It's one of the things I want to look at alongside intrinsics matching support (+ migrating some combiner rules of course)

llvm/utils/TableGen/GlobalISelCombinerMatchTableEmitter.cpp

650–651

No, we always need to create an entry for any operand in the OperandTable. It's how we can differentiate "this is a live-in" from "this doesn't exist". I'll make it clearer

[arsenm]

At this point I think it's easier to just start using it and file bugs for issues I run into

[Pierre-vh]

@aemerson are you also fine with landing it?
(Note: I'll be away next week for about 10 days so earliest I can pick up bugs is after the 23rd/24th - I don't mind waiting until I'm back to land this if preferred)

[aemerson]

In D156315#4575583, @Pierre-vh wrote:

@aemerson are you also fine with landing it?
(Note: I'll be away next week for about 10 days so earliest I can pick up bugs is after the 23rd/24th - I don't mind waiting until I'm back to land this if preferred)

I’m fine with landing it.

llvm/docs/GlobalISel/MIRPatterns.rst

+
+.. _tblgen-mirpats:
+
+========================
+MIR Patterns in TableGen
+========================
+
+.. contents::
+   :local:
+
+
+User's Guide
+============
+
+This section is intended for developers who want to use MIR patterns in their
+TableGen files.
+
+``NOTE``:
+This feature is still in active development. This document may become outdated
+over time. If you see something that's incorrect, please update it.
+
+Use Cases
+---------
+
+MIR patterns are supported in the following places:
+
+* GlobalISel ``GICombineRule``
+* GlobalISel ``GICombinePatFrag``
+
+Syntax
+------
+
+MIR patterns use the DAG datatype in TableGen.
+
+.. code-block:: text
+
+  (inst operand0, operand1, ...)
+
+``inst`` must be a def which inherits from ``Instruction`` (e.g. ``G_FADD``)
+or ``GICombinePatFrag``.
+
+Operands essentially fall into one of two categories:
+
+* immediates
+
+  * untyped, unnamed: ``0``
+  * untyped, named: ``0:$y``
+  * typed, unnamed: ``(i32 0)``
+  * typed, named: ``(i32 0):$y``
+
+* machine operands
+
+  * untyped: ``$x``
+  * typed: ``i32:$x``
+
+Semantics:
+
+* A typed operand always adds an operand type check to the matcher.
+* There is a trivial type inference system to propagate types.
+
+  * e.g. You only need to use ``i32:$x`` once in any pattern, then all
+    other patterns can simply use ``$x`` (``i32:$x`` is redundant).
+
+* A nammed operand's behavior depends on whether the name has been seen before.
+
+  * For match patterns, reusing an operand name checks that the operands
+    are identical (see example 2 below).
+  * For apply patterns, reusing an operand name simply copies that operand into
+    the new instruction (see example 2 below).
+
+Operands are ordered just like they would be in a MachineInstr: the defs (outs)
+come first, then the uses (ins).
+
+Patterns are generally grouped into another DAG datatype with a dummy operator
+such as ``match``, ``apply`` or ``pattern``.
+
+Finally, any DAG datatype in TableGen can be named. This also holds for
+patterns. e.g. the following is valid: ``(G_FOO $root, (i32 0):$cst):$mypat``.
+This may also be helpful to debug issues. Patterns are *always* named, and if
+they don't have a name, an "anonymous" one is given to them. If you're trying
+to debug an error related to a MIR pattern, but the error mentions an anonymous
+pattern, you can try naming your patterns to see exactly where the issue is.
+
+.. code-block:: text
+  :caption: Pattern Example 1
+
+  // Match
+  //    %imp = G_IMPLICIT_DEF
+  //    %root = G_MUL %x, %imp
+  (match (G_IMPLICIT_DEF $imp),
+         (G_MUL $root, $x, $imp))
+
+.. code-block:: text
+  :caption: Pattern Example 2
+
+  // using $x twice here checks that the operand 1 and 2 of the G_AND are
+  // identical.
+  (match (G_AND $root, $x, $x))
+  // using $x again here copies operand 1 from G_AND into the new inst.
+  (apply (COPY $root, $x))
+
+
+Limitations
+-----------
+
+This a non-exhaustive list of known issues with MIR patterns at this time.
+
+* Matching intrinsics is not yet possible.
+* Using ``GICombinePatFrag`` within another ``GICombinePatFrag`` is not
+  supported.
+* Using ``GICombinePatFrag`` in the ``apply`` pattern of a ``GICombineRule``
+  is not supported.
+* Immediates must always be typed in the ``apply`` pattern of a
+  ``GICombineRule``.
+* Immediates always emit a CImm in the ``apply`` pattern of a
+  ``GICombineRule``. There is no way to implicitly emit a ``G_CONSTANT``,
+  or emit an simple immediate operand.
+* Matching a constant >64 bits is not supported (see comment in
+  ``GIM_CheckConstantInt```)
+
+GICombineRule
+----------------------
+
+MIR patterns can appear in the ``match`` or ``apply`` patterns of a
+``GICombineRule``.
+
+The ``root`` of the rule can either be a def of an instruction, or a
+named pattern. The latter is helpful when the instruction you want
+to match has no defs. The former is generally preferred because
+it's less verbose.
+
+.. code-block:: text
+  :caption: Combine Rule root is a def
+
+  // Fold x op 1 -> x
+  def right_identity_one: GICombineRule<
+    (defs root:$dst),
+    (match (G_MUL $dst, $x, 1)),
+    // Note: Patterns always need to create something, we can't just replace $dst with $x, so we need a COPY.
+    (apply (COPY $dst, $x))
+  >;
+
+.. code-block:: text
+  :caption: Combine Rule root is a named pattern
+
+  def Foo : GICombineRule<
+    (defs root:$root),
+    (match (G_ZEXT $tmp, (i32 0)),
+           (G_STORE $tmp, $ptr):$root),
+    (apply (G_STORE (i32 0), $ptr):$root)>;
+
+
+Combine Rules also allow mixing C++ code with MIR patterns, so that you
+may perform additional checks when matching, or run additional code after
+rewriting a pattern.
+
+The following expansions are available for MIR patterns:
+
+* operand names (``MachineOperand &``)
+* pattern names (``MachineInstr *`` for ``match``,
+  ``MachineInstrBuilder &`` for apply)
+
+GICombinePatFrag
+----------------
+
+``GICombinePatFrag`` is an equivalent of ``PatFrags`` for MIR patterns.
+They have two main usecases:
+
+* Reduce repetition by creating a ``GICombinePatFrag`` for common
+  patterns (see example 1).
+* Implicitly duplicate a CombineRule for multiple variants of a
+  pattern (see example 2).
+
+A ``GICombinePatFrag`` is composed of three elements:
+
+* zero or more ``in`` (def) parameter
+* zero or more ``out`` parameter
+* A list of MIR patterns that can match.
+
+  * When a ``GICombinePatFrag`` is used within a pattern, the pattern is
+    cloned once for each alternative that can match.
+
+Parameters can have the following types:
+
+* ``gi_mo``, which is the implicit default (no type = ``gi_mo``).
+
+  * Refers to any operand of an instruction (register, BB ref, imm, etc.).
+  * Can be used in both ``in`` and ``out`` parameters.
+  * Users of the PatFrag can only use an operand name for this
+    parameter (e.g. ``(my_pat_frag $foo)``).
+
+* ``root``
+
+  * This is identical to ``gi_mo``.
+  * Can only be used in ``out`` parameters to declare the root of the
+    pattern.
+  * Non-empty ``out`` parameter lists must always have exactly one ``root``.
+
+* ``gi_imm``
+
+  * Refers to an (potentially typed) immediate.
+  * Can only be used in ``in`` parameters.
+  * Users of the PatFrag can only use an immediate for this parameter
+    (e.g. ``(my_pat_frag 0)`` or ``(my_pat_frag (i32 0))``)
+
+``out`` operands can only be empty if the ``GICombinePatFrag`` only contains
+C++ code. If the fragment contains instruction patterns, it has to have at
+least one ``out`` operand of type ``root``.
+
+``in`` operands are less restricted, but there is one important concept to
+remember: you can pass "unbound" operand names, but only if the
+``GICombinePatFrag`` binds it. See example 3 below.
+
+``GICombinePatFrag`` are used just like any other instructions.
+Note that the ``out`` operands are defs, so they come first in the list
+of operands.
+
+.. code-block:: text
+  :caption: Example 1: Reduce Repetition
+
+  def zext_cst : GICombinePatFrag<(outs root:$dst, $cst), (ins gi_imm:$val),
+    [(pattern (G_CONSTANT $cst, $val),
+              (G_ZEXT $dst, $cst))]
+  >;
+
+  def foo_to_impdef : GICombineRule<
+   (defs root:$dst),
+   (match (zext_cst $y, $cst, (i32 0))
+          (G_FOO $dst, $y)),
+   (apply (G_IMPLICIT_DEF $dst))>;
+
+  def store_ext_zero : GICombineRule<
+   (defs root:$root),
+   (match (zext_cst $y, $cst, (i32 0))
+          (G_STORE $y, $ptr):$root),
+   (apply (G_STORE $cst, $ptr):$root)>;
+
+.. code-block:: text
+  :caption: Example 2: Generate Multiple Rules at Once
+
+  // Fold (freeze (freeze x)) -> (freeze x).
+  // Fold (fabs (fabs x)) -> (fabs x).
+  // Fold (fcanonicalize (fcanonicalize x)) -> (fcanonicalize x).
+  def idempotent_prop_frags : GICombinePatFrag<(outs root:$dst, $src), (ins),
+    [
+      (pattern (G_FREEZE $dst, $src), (G_FREEZE $src, $x)),
+      (pattern (G_FABS $dst, $src), (G_FABS $src, $x)),
+      (pattern (G_FCANONICALIZE $dst, $src), (G_FCANONICALIZE $src, $x))
+    ]
+  >;
+
+  def idempotent_prop : GICombineRule<
+    (defs root:$dst),
+    (match (idempotent_prop_frags $dst, $src)),
+    (apply (COPY $dst, $src))>;
+
+
+
+.. code-block:: text
+  :caption: Example 3: Unbound Operand Names
+
+  // This fragment binds $x to an operand in all of its
+  // alternative patterns.
+  def always_binds : GICombinePatFrag<
+    (outs root:$dst), (ins $x),
+    [
+      (pattern (G_FREEZE $dst, $x)),
+      (pattern (G_FABS $dst, $x)),
+    ]
+  >;
+
+  // This fragment does not bind $x to an operand in any
+  // all of its alternative patterns.
+  def does_not_bind : GICombinePatFrag<
+    (outs root:$dst), (ins $x),
+    [
+      (pattern (G_FREEZE $dst, $x)), // binds $x
+      (pattern (G_FOO $dst (i32 0))), // does not bind $x
+      (pattern "return myCheck(${x}.getReg())"), // does not bind $x
+    ]
+  >;
+
+  // Here we pass $x, which is unbound, to always_binds.
+  // This works because if $x is unbound, always_binds will bind it for us.
+  def test0 : GICombineRule<
+    (defs root:$dst),
+    (match (always_binds $dst, $x)),
+    (apply (COPY $dst, $x))>;
+
+  // Here we pass $src, which is unbound, to does_not_bind.
+  // This cannot work because $x may not have been initialized in 'apply'.
+  // error: operand 'x' (for parameter 'src' of 'does_not_bind') cannot be unbound
+  def test1 : GICombineRule<
+    (defs root:$dst),
+    (match (does_not_bind $dst, $x)),
+    (apply (COPY $dst, $x))>;
+
+  // Here we pass $src, which is bound, to does_not_bind.
+  // This is fine because $x will always be bound when emitting does_not_bind
+  def test2 : GICombineRule<
+    (defs root:$dst),
+    (match (does_not_bind $tmp, $x)
+           (G_MUL $dst, $x, $tmp)),
+    (apply (COPY $dst, $x))>;

llvm/docs/GlobalISel/index.rst

 More information on the design and implementation of GlobalISel can be found in
 the following sections.
 
 .. toctree::
   :maxdepth: 1
 
   GMIR
   GenericOpcode
+  MIRPatterns
   Pipeline
   Porting
   Resources
 
 More information on specific passes can be found in the following sections:
 
 .. toctree::
   :maxdepth: 1

llvm/docs/UserGuides.rst

.. toctree::
    CoverageMappingFormat
    CycleTerminology
    DebuggingJITedCode
    DirectXUsage
    Docker
    FatLTO
    ExtendingLLVM
    GoldPlugin
+   GlobalISel/MIRPatterns
    HowToBuildOnARM
    HowToBuildWithPGO
    HowToBuildWindowsItaniumPrograms
    HowToCrossCompileBuiltinsOnArm
    HowToCrossCompileLLVM
    HowToUpdateDebugInfo
    InstrRefDebugInfo
    LinkTimeOptimization
:doc:`CodeGenerator`
    The design and implementation of the LLVM code generator.  Useful if you are
    working on retargetting LLVM to a new architecture, designing a new codegen
    pass, or enhancing existing components.
 
 :doc:`TableGen <TableGen/index>`
    Describes the TableGen tool, which is used heavily by the LLVM code
    generator.
 
+==========
+GlobalISel
+==========
+
+:doc:`MIRPatterns <GlobalISel/MIRPatterns>`
+   Describes the design of MIR Patterns and how to use them.
+
 ===
 JIT
 ===
 
 :doc:`MCJITDesignAndImplementation`
    Describes the inner workings of MCJIT execution engine.
 
 :doc:`ORCv2`
:doc:`SPIRVUsage`
    This document describes using the SPIR-V target to compile GPU kernels.
 
 :doc:`DirectXUsage`
    This document describes using the DirectX target to compile GPU code for the
    DirectX runtime.
 
 :doc:`RISCVUsage`
    This document describes using the RISCV-V target.
-

llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h

enum {
   /// - SubRegIndex - The subregister index to set
   GIR_AddTempSubRegister,
 
   /// Add an immediate to the specified instruction
   /// - InsnID - Instruction ID to modify
   /// - Imm - The immediate to add
   GIR_AddImm,
 
+  /// Add an CImm to the specified instruction
+  /// - InsnID - Instruction ID to modify
+  /// - Ty - Type of the constant immediate.
+  /// - Imm - The immediate to add
+  GIR_AddCImm,
+
   /// Render complex operands to the specified instruction
   /// - InsnID - Instruction ID to modify
   /// - RendererID - The renderer to call
   GIR_ComplexRenderer,
   /// Render sub-operands of complex operands to the specified instruction
   /// - InsnID - Instruction ID to modify
   /// - RendererID - The renderer to call
   /// - RenderOpID - The suboperand to render.
enum {
   /// - InsnID - Instruction ID to modify
   /// - OldInsnID - Instruction ID to get the matched operand from
   /// - RendererFnID - Custom renderer function to call
   GIR_CustomRenderer,
 
   /// Calls a C++ function to perform an action when a match is complete.
   /// The MatcherState is passed to the function to allow it to modify
   /// instructions.
-  /// This is less constrained than a custom renderer and can update instructions
+  /// This is less constrained than a custom renderer and can update
+  /// instructions
   /// in the state.
   /// - FnID - The function to call.
   /// TODO: Remove this at some point when combiners aren't reliant on it. It's
   /// a bit of a hack.
   GIR_CustomAction,
 
   /// Render operands to the specified instruction using a custom function,
   /// reading from a specific operand.
template <class TgtExecutor, class PredicateBitset, class ComplexMatcherMemFn,
             class CustomRendererFn>
   bool executeMatchTable(
       TgtExecutor &Exec, NewMIVector &OutMIs, MatcherState &State,
       const ExecInfoTy<PredicateBitset, ComplexMatcherMemFn, CustomRendererFn>
           &ISelInfo,
       const int64_t *MatchTable, const TargetInstrInfo &TII,
       MachineRegisterInfo &MRI, const TargetRegisterInfo &TRI,
       const RegisterBankInfo &RBI, const PredicateBitset &AvailableFeatures,
-      CodeGenCoverage *CoverageInfo) const;
+      CodeGenCoverage *CoverageInfo,
+      GISelChangeObserver *Observer = nullptr) const;
 
   virtual const int64_t *getMatchTable() const {
     llvm_unreachable("Should have been overridden by tablegen if used");
   }
 
   virtual bool testImmPredicate_I64(unsigned, int64_t) const {
     llvm_unreachable(
         "Subclasses must override this with a tablegen-erated function");
virtual bool testMIPredicate_MI(unsigned, const MachineInstr &,
     llvm_unreachable(
         "Subclasses must override this with a tablegen-erated function");
   }
 
   virtual bool testSimplePredicate(unsigned) const {
     llvm_unreachable("Subclass does not implement testSimplePredicate!");
   }
 
-  virtual void runCustomAction(unsigned, const MatcherState &State) const {
+  virtual void runCustomAction(unsigned, const MatcherState &State,
+                               NewMIVector &OutMIs) const {
     llvm_unreachable("Subclass does not implement runCustomAction!");
   }
 
   bool isOperandImmEqual(const MachineOperand &MO, int64_t Value,
-                         const MachineRegisterInfo &MRI) const;
+                         const MachineRegisterInfo &MRI,
+                         bool Splat = false) const;
 
   /// Return true if the specified operand is a G_PTR_ADD with a G_CONSTANT on
   /// the right-hand side. GlobalISel's separation of pointer and integer types
   /// means that we don't need to worry about G_OR with equivalent semantics.
   bool isBaseWithConstantOffset(const MachineOperand &Root,
                                 const MachineRegisterInfo &MRI) const;
 
   /// Return true if MI can obviously be folded into IntoMI.
   /// MI and IntoMI do not need to be in the same basic blocks, but MI must
   /// preceed IntoMI.
   bool isObviouslySafeToFold(MachineInstr &MI, MachineInstr &IntoMI) const;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_GLOBALISEL_GIMATCHTABLEEXECUTOR_H

llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutorImpl.h

 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_GLOBALISEL_GIMATCHTABLEEXECUTORIMPL_H
 #define LLVM_CODEGEN_GLOBALISEL_GIMATCHTABLEEXECUTORIMPL_H
 
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h"
+#include "llvm/CodeGen/GlobalISel/GISelChangeObserver.h"
 #include "llvm/CodeGen/GlobalISel/Utils.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/RegisterBankInfo.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetOpcodes.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DataLayout.h"
+#include "llvm/IR/Type.h"
 #include "llvm/Support/CodeGenCoverage.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
 
 namespace llvm {
 
 template <class TgtExecutor, class PredicateBitset, class ComplexMatcherMemFn,
           class CustomRendererFn>
 bool GIMatchTableExecutor::executeMatchTable(
     TgtExecutor &Exec, NewMIVector &OutMIs, MatcherState &State,
     const ExecInfoTy<PredicateBitset, ComplexMatcherMemFn, CustomRendererFn>
         &ExecInfo,
     const int64_t *MatchTable, const TargetInstrInfo &TII,
     MachineRegisterInfo &MRI, const TargetRegisterInfo &TRI,
     const RegisterBankInfo &RBI, const PredicateBitset &AvailableFeatures,
-    CodeGenCoverage *CoverageInfo) const {
+    CodeGenCoverage *CoverageInfo, GISelChangeObserver *Observer) const {
 
   uint64_t CurrentIdx = 0;
   SmallVector<uint64_t, 4> OnFailResumeAt;
 
   // Bypass the flag check on the instruction, and only look at the MCInstrDesc.
   bool NoFPException = !State.MIs[0]->getDesc().mayRaiseFPException();
 
   const uint16_t Flags = State.MIs[0]->getFlags();
case GIM_CheckAPIntImmPredicate: {
       int64_t Predicate = MatchTable[CurrentIdx++];
       DEBUG_WITH_TYPE(TgtExecutor::getName(),
                       dbgs()
                           << CurrentIdx << ": GIM_CheckAPIntImmPredicate(MIs["
                           << InsnID << "], Predicate=" << Predicate << ")\n");
       assert(State.MIs[InsnID] != nullptr && "Used insn before defined");
       assert(State.MIs[InsnID]->getOpcode() == TargetOpcode::G_CONSTANT &&
              "Expected G_CONSTANT");
-      assert(Predicate > GICXXPred_Invalid &&
-             "Expected a valid predicate");
+      assert(Predicate > GICXXPred_Invalid && "Expected a valid predicate");
       if (!State.MIs[InsnID]->getOperand(1).isCImm())
         llvm_unreachable("Expected Imm or CImm operand");
 
       const APInt &Value =
           State.MIs[InsnID]->getOperand(1).getCImm()->getValue();
       if (!testImmPredicate_APInt(Predicate, Value))
         if (handleReject() == RejectAndGiveUp)
           return false;
       break;
     }
     case GIM_CheckAPFloatImmPredicate: {
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t Predicate = MatchTable[CurrentIdx++];
       DEBUG_WITH_TYPE(TgtExecutor::getName(),
                       dbgs()
                           << CurrentIdx << ": GIM_CheckAPFloatImmPredicate(MIs["
                           << InsnID << "], Predicate=" << Predicate << ")\n");
       assert(State.MIs[InsnID] != nullptr && "Used insn before defined");
       assert(State.MIs[InsnID]->getOpcode() == TargetOpcode::G_FCONSTANT &&
              "Expected G_FCONSTANT");
       assert(State.MIs[InsnID]->getOperand(1).isFPImm() &&
              "Expected FPImm operand");
-      assert(Predicate > GICXXPred_Invalid &&
-             "Expected a valid predicate");
+      assert(Predicate > GICXXPred_Invalid && "Expected a valid predicate");
       const APFloat &Value =
           State.MIs[InsnID]->getOperand(1).getFPImm()->getValueAPF();
 
       if (!testImmPredicate_APFloat(Predicate, Value))
         if (handleReject() == RejectAndGiveUp)
           return false;
       break;
     }
case GIM_CheckConstantInt: {
                       dbgs() << CurrentIdx << ": GIM_CheckConstantInt(MIs["
                              << InsnID << "]->getOperand(" << OpIdx
                              << "), Value=" << Value << ")\n");
       assert(State.MIs[InsnID] != nullptr && "Used insn before defined");
       MachineOperand &MO = State.MIs[InsnID]->getOperand(OpIdx);
       if (MO.isReg()) {
         // isOperandImmEqual() will sign-extend to 64-bits, so should we.
         LLT Ty = MRI.getType(MO.getReg());
-        Value = SignExtend64(Value, Ty.getSizeInBits());
+        // If the type is > 64 bits, it can't be a constant int, so we bail
+        // early because SignExtend64 will assert otherwise.
+        if (Ty.getScalarSizeInBits() > 64) {
+          if (handleReject() == RejectAndGiveUp)
+            return false;
+          break;
+        }
 
-        if (!isOperandImmEqual(MO, Value, MRI)) {
+        Value = SignExtend64(Value, Ty.getScalarSizeInBits());
+        if (!isOperandImmEqual(MO, Value, MRI, /*Splat=*/true)) {
           if (handleReject() == RejectAndGiveUp)
             return false;
         }
       } else if (handleReject() == RejectAndGiveUp)
         return false;
 
       break;
     }
case GIR_AddImm: {
       assert(OutMIs[InsnID] && "Attempted to add to undefined instruction");
       OutMIs[InsnID].addImm(Imm);
       DEBUG_WITH_TYPE(TgtExecutor::getName(),
                       dbgs() << CurrentIdx << ": GIR_AddImm(OutMIs[" << InsnID
                              << "], " << Imm << ")\n");
       break;
     }
 
+    case GIR_AddCImm: {
+      int64_t InsnID = MatchTable[CurrentIdx++];
+      int64_t TypeID = MatchTable[CurrentIdx++];
+      int64_t Imm = MatchTable[CurrentIdx++];
+      assert(OutMIs[InsnID] && "Attempted to add to undefined instruction");
+
+      unsigned Width = ExecInfo.TypeObjects[TypeID].getScalarSizeInBits();
+      LLVMContext &Ctx = MF->getFunction().getContext();
+      OutMIs[InsnID].addCImm(
+          ConstantInt::get(IntegerType::get(Ctx, Width), Imm, /*signed*/ true));
+      DEBUG_WITH_TYPE(TgtExecutor::getName(),
+                      dbgs() << CurrentIdx << ": GIR_AddCImm(OutMIs[" << InsnID
+                             << "], TypeID=" << TypeID << ", Imm=" << Imm
+                             << ")\n");
+      break;
+    }
+
     case GIR_ComplexRenderer: {
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t RendererID = MatchTable[CurrentIdx++];
       assert(OutMIs[InsnID] && "Attempted to add to undefined instruction");
       for (const auto &RenderOpFn : State.Renderers[RendererID])
         RenderOpFn(OutMIs[InsnID]);
       DEBUG_WITH_TYPE(TgtExecutor::getName(),
                       dbgs() << CurrentIdx << ": GIR_ComplexRenderer(OutMIs["
case GIR_CustomRenderer: {
       break;
     }
     case GIR_CustomAction: {
       int64_t FnID = MatchTable[CurrentIdx++];
       DEBUG_WITH_TYPE(TgtExecutor::getName(),
                       dbgs() << CurrentIdx << ": GIR_CustomAction(FnID=" << FnID
                              << ")\n");
       assert(FnID > GICXXCustomAction_Invalid && "Expected a valid FnID");
-      runCustomAction(FnID, State);
+      runCustomAction(FnID, State, OutMIs);
       break;
     }
     case GIR_CustomOperandRenderer: {
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t OldInsnID = MatchTable[CurrentIdx++];
       int64_t OpIdx = MatchTable[CurrentIdx++];
       int64_t RendererFnID = MatchTable[CurrentIdx++];
       assert(OutMIs[InsnID] && "Attempted to add to undefined instruction");
case GIR_MergeMemOperands: {
           OutMIs[InsnID].addMemOperand(MMO);
       }
       DEBUG_WITH_TYPE(TgtExecutor::getName(), dbgs() << ")\n");
       break;
     }
 
     case GIR_EraseFromParent: {
       int64_t InsnID = MatchTable[CurrentIdx++];
-      assert(State.MIs[InsnID] &&
-             "Attempted to erase an undefined instruction");
-      State.MIs[InsnID]->eraseFromParent();
+      MachineInstr *MI = State.MIs[InsnID];
+      assert(MI && "Attempted to erase an undefined instruction");
       DEBUG_WITH_TYPE(TgtExecutor::getName(),
                       dbgs() << CurrentIdx << ": GIR_EraseFromParent(MIs["
                              << InsnID << "])\n");
+      if (Observer)
+        Observer->erasingInstr(*MI);
+      MI->eraseFromParent();
       break;
     }
 
     case GIR_MakeTempReg: {
       int64_t TempRegID = MatchTable[CurrentIdx++];
       int64_t TypeID = MatchTable[CurrentIdx++];
 
       State.TempRegisters[TempRegID] =
case GIR_Coverage: {
                                                      << ": GIR_Coverage("
                                                      << RuleID << ")");
       break;
     }
 
     case GIR_Done:
       DEBUG_WITH_TYPE(TgtExecutor::getName(),
                       dbgs() << CurrentIdx << ": GIR_Done\n");
+      if (Observer) {
+        for (MachineInstr *MI : OutMIs)
+          Observer->createdInstr(*MI);
+      }
       propagateFlags(OutMIs);
       return true;
     default:
       llvm_unreachable("Unexpected command");
     }
   }
 }
 
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_GLOBALISEL_GIMATCHTABLEEXECUTORIMPL_H

llvm/include/llvm/Target/GlobalISel/Combine.td

class GICombineRule<dag defs, dag match, dag apply> : GICombine {
   /// combine rule.
   /// See GIApplyKind for details.
   dag Apply = apply;
 
   /// Defines the predicates that are checked before the match function
   /// is called. Targets can use this to, for instance, check Subtarget
   /// features.
   list<Predicate> Predicates = [];
-}
 
-/// The operator at the root of a GICombineRule.Defs dag.
-def defs;
+  // Maximum number of permutations of this rule that can be emitted.
+  // Set to -1 to disable the limit.
+  int MaxPermutations = 16;
+}
 
 /// All arguments of the defs operator must be subclasses of GIDefKind or
 /// sub-dags whose operator is GIDefKindWithArgs.
 class GIDefKind;
 class GIDefKindWithArgs;
 /// Declare a root node. There must be at least one of these in every combine
 /// rule.
 /// TODO: The plan is to elide `root` definitions and determine it from the DAG
 ///       itself with an overide for situations where the usual determination
 ///       is incorrect.
 def root : GIDefKind;
 
+def gi_mo;
+def gi_imm;
+def pattern;
+
+// This is an equivalent of PatFrags but for MIR Patterns.
+//
+// GICombinePatFrags can be used in place of instructions for 'match' patterns.
+// Much like normal instructions, the defs (outs) come first, and the ins second
+//
+// Out operands can only be of type "root" or "gi_mo", and they must be defined
+// by an instruction pattern in all alternatives.
+//
+// In operands can be gi_imm or gi_mo. They cannot be redefined in any alternative
+// pattern and may only appear in the C++ code, or in the output operand of an
+// instruction pattern.
+class GICombinePatFrag<dag outs, dag ins, list<dag> alts> {
+  dag InOperands = ins;
+  dag OutOperands = outs;
+  list<dag> Alternatives = alts;
+}
+
+/// The operator at the root of a GICombineRule.Defs dag.
+def defs;
+
 /// Declares data that is passed from the match stage to the apply stage.
 class GIDefMatchData<string type> : GIDefKind {
   /// A C++ type name indicating the storage type.
   string Type = type;
 }
 
 def extending_load_matchdata : GIDefMatchData<"PreferredTuple">;
 def indexed_load_store_matchdata : GIDefMatchData<"IndexedLoadStoreMatchInfo">;

llvm/lib/CodeGen/GlobalISel/GIMatchTableExecutor.cpp

 
 using namespace llvm;
 
 GIMatchTableExecutor::MatcherState::MatcherState(unsigned MaxRenderers)
     : Renderers(MaxRenderers) {}
 
 GIMatchTableExecutor::GIMatchTableExecutor() = default;
 
-bool GIMatchTableExecutor::isOperandImmEqual(
-    const MachineOperand &MO, int64_t Value,
-    const MachineRegisterInfo &MRI) const {
-  if (MO.isReg() && MO.getReg())
+bool GIMatchTableExecutor::isOperandImmEqual(const MachineOperand &MO,
+                                             int64_t Value,
+                                             const MachineRegisterInfo &MRI,
+                                             bool Splat) const {
+  if (MO.isReg() && MO.getReg()) {
     if (auto VRegVal = getIConstantVRegValWithLookThrough(MO.getReg(), MRI))
       return VRegVal->Value.getSExtValue() == Value;
+
+    if (Splat) {
+      if (auto VRegVal = getIConstantSplatVal(MO.getReg(), MRI))
+        return VRegVal->getSExtValue() == Value;
+    }
+  }
   return false;
 }
 
 bool GIMatchTableExecutor::isBaseWithConstantOffset(
     const MachineOperand &Root, const MachineRegisterInfo &MRI) const {
   if (!Root.isReg())
     return false;
 

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-imms.td

+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN:     -combiners=MyCombiner %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def InstTest0 : GICombineRule<
+  (defs root:$a),
+  (match (COPY $a, (i32 0))),
+  (apply [{ APPLY }])>;
+
+def InstTest1 : GICombineRule<
+  (defs root:$a),
+  (match (G_ZEXT $a, 0)),
+  (apply [{ APPLY }])>;
+
+def CImmInstTest1 : GICombineRule<
+  (defs root:$a),
+  (match (G_CONSTANT $a, (i32 0))),
+  (apply [{ APPLY }])>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+  InstTest0,
+  InstTest1,
+  CImmInstTest1
+]>;
+
+// CHECK:      const int64_t *GenMyCombiner::getMatchTable() const {
+// CHECK-NEXT:   constexpr static int64_t MatchTable0[] = {
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 0*/ 18, // Rule ID 0 //
+// CHECK-NEXT:       GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::COPY,
+// CHECK-NEXT:       GIM_CheckType, /*MI*/0, /*Op*/1, /*Type*/GILLT_s32,
+// CHECK-NEXT:       // MIs[0] a
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       GIM_CheckConstantInt, /*MI*/0, /*Op*/1, 0,
+// CHECK-NEXT:       // Combiner Rule #0: InstTest0
+// CHECK-NEXT:       GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:       GIR_Done,
+// CHECK-NEXT:     // Label 0: @18
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 1*/ 32, // Rule ID 1 //
+// CHECK-NEXT:       GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule1Enabled,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:       // MIs[0] a
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       // MIs[0] Operand 1
+// CHECK-NEXT:       GIM_CheckConstantInt, /*MI*/0, /*Op*/1, 0,
+// CHECK-NEXT:       // Combiner Rule #1: InstTest1
+// CHECK-NEXT:       GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:       GIR_Done,
+// CHECK-NEXT:     // Label 1: @32
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 2*/ 50, // Rule ID 2 //
+// CHECK-NEXT:       GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule2Enabled,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_CONSTANT,
+// CHECK-NEXT:       GIM_CheckType, /*MI*/0, /*Op*/1, /*Type*/GILLT_s32,
+// CHECK-NEXT:       // MIs[0] a
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       GIM_CheckLiteralInt, /*MI*/0, /*Op*/1, 0,
+// CHECK-NEXT:       // Combiner Rule #2: CImmInstTest1
+// CHECK-NEXT:       GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:       GIR_Done,
+// CHECK-NEXT:     // Label 2: @50
+// CHECK-NEXT:     GIM_Reject,
+// CHECK-NEXT:     };
+// CHECK-NEXT:   return MatchTable0;
+// CHECK-NEXT: }

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-operand-types.td

+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN:     -combiners=MyCombiner %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def InstTest0 : GICombineRule<
+  (defs root:$a),
+  (match  (G_MUL i32:$x, i32:$b, i32:$c),
+          (G_MUL $a, i32:$b, i32:$x)),
+  (apply  (G_ADD i64:$tmp, $b, i32:$c),
+          (G_ADD i8:$a, $b, i64:$tmp))>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+  InstTest0,
+]>;
+
+// CHECK:       const int64_t *GenMyCombiner::getMatchTable() const {
+// CHECK-NEXT:   constexpr static int64_t MatchTable0[] = {
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 0*/ 73, // Rule ID 0 //
+// CHECK-NEXT:       GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_MUL,
+// CHECK-NEXT:       GIM_CheckType, /*MI*/0, /*Op*/0, /*Type*/GILLT_s8,
+// CHECK-NEXT:       GIM_CheckType, /*MI*/0, /*Op*/1, /*Type*/GILLT_s32,
+// CHECK-NEXT:       GIM_CheckType, /*MI*/0, /*Op*/2, /*Type*/GILLT_s32,
+// CHECK-NEXT:       GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/2, // MIs[1]
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_MUL,
+// CHECK-NEXT:       GIM_CheckType, /*MI*/1, /*Op*/2, /*Type*/GILLT_s32,
+// CHECK-NEXT:       // MIs[1] b
+// CHECK-NEXT:       GIM_CheckIsSameOperandIgnoreCopies, /*MI*/1, /*OpIdx*/1, /*OtherMI*/0, /*OtherOpIdx*/1,
+// CHECK-NEXT:       GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:       // Combiner Rule #0: InstTest0
+// CHECK-NEXT:       GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s64,
+// CHECK-NEXT:       GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::G_ADD,
+// CHECK-NEXT:       GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/1, // b
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/1, /*OpIdx*/2, // c
+// CHECK-NEXT:       GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:       GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_ADD,
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/1, /*OldInsnID*/0, /*OpIdx*/0, // a
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/1, /*OldInsnID*/0, /*OpIdx*/1, // b
+// CHECK-NEXT:       GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT:       GIR_Done,
+// CHECK-NEXT:     // Label 0: @73
+// CHECK-NEXT:     GIM_Reject,
+// CHECK-NEXT:     };
+// CHECK-NEXT:   return MatchTable0;
+// CHECK-NEXT: }

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-patfrag-root.td

+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN:     -combiners=MyCombiner -gicombiner-matchtable-debug-cxxpreds %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def MatchFooPerms: GICombinePatFrag<
+    (outs root:$foo),
+    (ins gi_imm:$cst),
+    [
+      (pattern (G_ZEXT $foo, $b), (G_TRUNC $b, $x):$dbg0),
+      (pattern (G_TRUNC $foo, $z):$dbg1),
+      (pattern (G_FPEXT $foo, $z):$dbg1)
+    ]>;
+
+def Test0 : GICombineRule<
+  (defs root:$root),
+  (match (MatchFooPerms $root, (i32 10))),
+  (apply (COPY $root, (i32 0)))>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+  Test0
+]>;
+// CHECK:      const int64_t *GenMyCombiner::getMatchTable() const {
+// CHECK-NEXT:   constexpr static int64_t MatchTable0[] = {
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 0*/ 30, // Rule ID 0 //
+// CHECK-NEXT:       GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:       // MIs[0] root
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       // MIs[0] __anon_pat_match_0_0.b
+// CHECK-NEXT:       GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:       // MIs[1] __anon_pat_match_0_0.x
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:       // Combiner Rule #0: Test0 @ [__anon_pat_match_0_0[0]]
+// CHECK-NEXT:       GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // root
+// CHECK-NEXT:       GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:       GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:       GIR_Done,
+// CHECK-NEXT:     // Label 0: @30
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 1*/ 51, // Rule ID 1 //
+// CHECK-NEXT:       GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:       // MIs[0] root
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       // MIs[0] __anon_pat_match_0_0.z
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       // Combiner Rule #0: Test0 @ [__anon_pat_match_0_0[1]]
+// CHECK-NEXT:       GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // root
+// CHECK-NEXT:       GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:       GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:       GIR_Done,
+// CHECK-NEXT:     // Label 1: @51
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 2*/ 72, // Rule ID 2 //
+// CHECK-NEXT:       GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_FPEXT,
+// CHECK-NEXT:       // MIs[0] root
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       // MIs[0] __anon_pat_match_0_0.z
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       // Combiner Rule #0: Test0 @ [__anon_pat_match_0_0[2]]
+// CHECK-NEXT:       GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // root
+// CHECK-NEXT:       GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:       GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:       GIR_Done,
+// CHECK-NEXT:     // Label 2: @72
+// CHECK-NEXT:     GIM_Reject,
+// CHECK-NEXT:     };
+// CHECK-NEXT:   return MatchTable0;
+// CHECK-NEXT: }

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-permutations.td

+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN:     -combiners=MyCombiner -gicombiner-matchtable-debug-cxxpreds %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def MatchFooPerms: GICombinePatFrag<
+    (outs root:$foo),
+    (ins gi_imm:$cst),
+    [
+      (pattern (G_ZEXT $foo, $b), (G_TRUNC $b, $x):$dbg0, "return foo(${x}, ${cst})"),
+      (pattern (G_TRUNC $foo, $z):$dbg1, "return bar(${foo}, ${cst})")
+    ]>;
+
+def Test0 : GICombineRule<
+  (defs root:$dst),
+  (match (G_AND $dst, $cst0, $tmp),
+         (G_AND $tmp, $cst1, $cst2),
+         (MatchFooPerms $cst0, (i32 10)):$a,
+         (MatchFooPerms $cst1, (i32 20)):$b,
+         (MatchFooPerms $cst2, (i32 30)):$c
+  ),
+  (apply (COPY $dst, (i32 0)), "APPLY ${cst0}")>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+  Test0
+]>;
+
+// CHECK:      bool GenMyCombiner::testMIPredicate_MI(unsigned PredicateID, const MachineInstr & MI, const MatcherState &State) const {
+// CHECK-NEXT:   switch (PredicateID) {
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner0: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[0], c[0]]
+// CHECK-NEXT:     return foo(State.MIs[2]->getOperand(1), 10)
+// CHECK-NEXT:     llvm_unreachable("GICombiner0 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner1: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[0], c[0]]
+// CHECK-NEXT:     return foo(State.MIs[5]->getOperand(1), 20)
+// CHECK-NEXT:     llvm_unreachable("GICombiner1 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner2: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[0], c[0]]
+// CHECK-NEXT:     return foo(State.MIs[7]->getOperand(1), 30)
+// CHECK-NEXT:     llvm_unreachable("GICombiner2 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner3: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[0], c[1]]
+// CHECK-NEXT:     return foo(State.MIs[2]->getOperand(1), 10)
+// CHECK-NEXT:     llvm_unreachable("GICombiner3 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner4: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[0], c[1]]
+// CHECK-NEXT:     return foo(State.MIs[5]->getOperand(1), 20)
+// CHECK-NEXT:     llvm_unreachable("GICombiner4 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner5: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[0], c[1]]
+// CHECK-NEXT:     return bar(State.MIs[3]->getOperand(2), 30)
+// CHECK-NEXT:     llvm_unreachable("GICombiner5 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner6: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[1], c[0]]
+// CHECK-NEXT:     return foo(State.MIs[2]->getOperand(1), 10)
+// CHECK-NEXT:     llvm_unreachable("GICombiner6 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner7: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[1], c[0]]
+// CHECK-NEXT:     return bar(State.MIs[3]->getOperand(1), 20)
+// CHECK-NEXT:     llvm_unreachable("GICombiner7 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner8: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[1], c[0]]
+// CHECK-NEXT:     return foo(State.MIs[6]->getOperand(1), 30)
+// CHECK-NEXT:     llvm_unreachable("GICombiner8 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner9: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[1], c[1]]
+// CHECK-NEXT:     return foo(State.MIs[2]->getOperand(1), 10)
+// CHECK-NEXT:     llvm_unreachable("GICombiner9 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner10: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[1], c[1]]
+// CHECK-NEXT:     return bar(State.MIs[3]->getOperand(1), 20)
+// CHECK-NEXT:     llvm_unreachable("GICombiner10 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner11: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[0], b[1], c[1]]
+// CHECK-NEXT:     return bar(State.MIs[3]->getOperand(2), 30)
+// CHECK-NEXT:     llvm_unreachable("GICombiner11 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner12: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[0], c[0]]
+// CHECK-NEXT:     return bar(State.MIs[0]->getOperand(1), 10)
+// CHECK-NEXT:     llvm_unreachable("GICombiner12 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner13: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[0], c[0]]
+// CHECK-NEXT:     return foo(State.MIs[4]->getOperand(1), 20)
+// CHECK-NEXT:     llvm_unreachable("GICombiner13 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner14: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[0], c[0]]
+// CHECK-NEXT:     return foo(State.MIs[6]->getOperand(1), 30)
+// CHECK-NEXT:     llvm_unreachable("GICombiner14 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner15: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[0], c[1]]
+// CHECK-NEXT:     return bar(State.MIs[0]->getOperand(1), 10)
+// CHECK-NEXT:     llvm_unreachable("GICombiner15 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner16: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[0], c[1]]
+// CHECK-NEXT:     return foo(State.MIs[4]->getOperand(1), 20)
+// CHECK-NEXT:     llvm_unreachable("GICombiner16 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner17: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[0], c[1]]
+// CHECK-NEXT:     return bar(State.MIs[3]->getOperand(2), 30)
+// CHECK-NEXT:     llvm_unreachable("GICombiner17 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner18: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[1], c[0]]
+// CHECK-NEXT:     return bar(State.MIs[0]->getOperand(1), 10)
+// CHECK-NEXT:     llvm_unreachable("GICombiner18 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner19: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[1], c[0]]
+// CHECK-NEXT:     return bar(State.MIs[3]->getOperand(1), 20)
+// CHECK-NEXT:     llvm_unreachable("GICombiner19 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner20: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[1], c[0]]
+// CHECK-NEXT:     return foo(State.MIs[5]->getOperand(1), 30)
+// CHECK-NEXT:     llvm_unreachable("GICombiner20 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner21: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[1], c[1]]
+// CHECK-NEXT:     return bar(State.MIs[0]->getOperand(1), 10)
+// CHECK-NEXT:     llvm_unreachable("GICombiner21 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner22: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[1], c[1]]
+// CHECK-NEXT:     return bar(State.MIs[3]->getOperand(1), 20)
+// CHECK-NEXT:     llvm_unreachable("GICombiner22 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXPred_MI_Predicate_GICombiner23: {
+// CHECK-NEXT:     // Pattern Alternatives: [a[1], b[1], c[1]]
+// CHECK-NEXT:     return bar(State.MIs[3]->getOperand(2), 30)
+// CHECK-NEXT:     llvm_unreachable("GICombiner23 should have returned");
+// CHECK-NEXT:   }
+// CHECK-NEXT:   }
+// CHECK-NEXT:   llvm_unreachable("Unknown predicate");
+// CHECK-NEXT:   return false;
+// CHECK-NEXT: }
+
+// CHECK:      const int64_t *GenMyCombiner::getMatchTable() const {
+// CHECK-NEXT:   constexpr static int64_t MatchTable0[] = {
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 0*/ 634,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_AND,
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 1*/ 97, // Rule ID 0 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:         // MIs[0] dst
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] cst0
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[1] a.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/1, /*OpIdx*/1, // MIs[2]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[2] a.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] tmp
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/0, /*OpIdx*/2, // MIs[3]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_AND,
+// CHECK-NEXT:         // MIs[3] cst1
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[4] b.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/4, /*OpIdx*/1, // MIs[5]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[5] b.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[3] cst2
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/6, /*MI*/3, /*OpIdx*/2, // MIs[6]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/6, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[6] c.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/7, /*MI*/6, /*OpIdx*/1, // MIs[7]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/7, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[7] c.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner0,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner1,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner2,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/6,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/7,
+// CHECK-NEXT:         // Combiner Rule #0: Test0 @ [a[0], b[0], c[0]]
+// CHECK-NEXT:         GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:         GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:         GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:         GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 1: @97
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 2*/ 180, // Rule ID 1 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:         // MIs[0] dst
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] cst0
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[1] a.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/1, /*OpIdx*/1, // MIs[2]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[2] a.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] tmp
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/0, /*OpIdx*/2, // MIs[3]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_AND,
+// CHECK-NEXT:         // MIs[3] cst1
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[4] b.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/4, /*OpIdx*/1, // MIs[5]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[5] b.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[3] cst2
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/6, /*MI*/3, /*OpIdx*/2, // MIs[6]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/6, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[6] c.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner3,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner4,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner5,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/6,
+// CHECK-NEXT:         // Combiner Rule #0: Test0 @ [a[0], b[0], c[1]]
+// CHECK-NEXT:         GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:         GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:         GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:         GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 2: @180
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 3*/ 263, // Rule ID 2 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:         // MIs[0] dst
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] cst0
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[1] a.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/1, /*OpIdx*/1, // MIs[2]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[2] a.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] tmp
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/0, /*OpIdx*/2, // MIs[3]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_AND,
+// CHECK-NEXT:         // MIs[3] cst1
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[4] b.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[3] cst2
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/3, /*OpIdx*/2, // MIs[5]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[5] c.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/6, /*MI*/5, /*OpIdx*/1, // MIs[6]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/6, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[6] c.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner6,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner7,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner8,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/6,
+// CHECK-NEXT:         // Combiner Rule #0: Test0 @ [a[0], b[1], c[0]]
+// CHECK-NEXT:         GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:         GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:         GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:         GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 3: @263
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 4*/ 337, // Rule ID 3 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:         // MIs[0] dst
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] cst0
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[1] a.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/1, /*OpIdx*/1, // MIs[2]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[2] a.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] tmp
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/0, /*OpIdx*/2, // MIs[3]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_AND,
+// CHECK-NEXT:         // MIs[3] cst1
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[4] b.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[3] cst2
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/3, /*OpIdx*/2, // MIs[5]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[5] c.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner9,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner10,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner11,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT:         // Combiner Rule #0: Test0 @ [a[0], b[1], c[1]]
+// CHECK-NEXT:         GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:         GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:         GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:         GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 4: @337
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 5*/ 420, // Rule ID 4 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:         // MIs[0] dst
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] cst0
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[1] a.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] tmp
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/0, /*OpIdx*/2, // MIs[2]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_AND,
+// CHECK-NEXT:         // MIs[2] cst1
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/2, /*OpIdx*/1, // MIs[3]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[3] b.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[4] b.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[2] cst2
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/2, /*OpIdx*/2, // MIs[5]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[5] c.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/6, /*MI*/5, /*OpIdx*/1, // MIs[6]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/6, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[6] c.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner12,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner13,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner14,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/6,
+// CHECK-NEXT:         // Combiner Rule #0: Test0 @ [a[1], b[0], c[0]]
+// CHECK-NEXT:         GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:         GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:         GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:         GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 5: @420
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 6*/ 494, // Rule ID 5 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:         // MIs[0] dst
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] cst0
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[1] a.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] tmp
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/0, /*OpIdx*/2, // MIs[2]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_AND,
+// CHECK-NEXT:         // MIs[2] cst1
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/2, /*OpIdx*/1, // MIs[3]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[3] b.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[4] b.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[2] cst2
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/2, /*OpIdx*/2, // MIs[5]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[5] c.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner15,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner16,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner17,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT:         // Combiner Rule #0: Test0 @ [a[1], b[0], c[1]]
+// CHECK-NEXT:         GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:         GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:         GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:         GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 6: @494
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 7*/ 568, // Rule ID 6 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:         // MIs[0] dst
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] cst0
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[1] a.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] tmp
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/0, /*OpIdx*/2, // MIs[2]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_AND,
+// CHECK-NEXT:         // MIs[2] cst1
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/2, /*OpIdx*/1, // MIs[3]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[3] b.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[2] cst2
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/2, /*OpIdx*/2, // MIs[4]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:         // MIs[4] c.b
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/4, /*OpIdx*/1, // MIs[5]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[5] c.x
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner18,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner19,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner20,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT:         // Combiner Rule #0: Test0 @ [a[1], b[1], c[0]]
+// CHECK-NEXT:         GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:         GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:         GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:         GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 7: @568
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 8*/ 633, // Rule ID 7 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:         // MIs[0] dst
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] cst0
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[1] a.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] tmp
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/0, /*OpIdx*/2, // MIs[2]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_AND,
+// CHECK-NEXT:         // MIs[2] cst1
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/2, /*OpIdx*/1, // MIs[3]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[3] b.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[2] cst2
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/2, /*OpIdx*/2, // MIs[4]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_TRUNC,
+// CHECK-NEXT:         // MIs[4] c.z
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner21,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner22,
+// CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner23,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT:         // Combiner Rule #0: Test0 @ [a[1], b[1], c[1]]
+// CHECK-NEXT:         GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:         GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:         GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:         GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 8: @633
+// CHECK-NEXT:       GIM_Reject,
+// CHECK-NEXT:     // Label 0: @634
+// CHECK-NEXT:     GIM_Reject,
+// CHECK-NEXT:     };
+// CHECK-NEXT:   return MatchTable0;
+// CHECK-NEXT: }

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-variadics.td

+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN:     -combiners=MyCombiner %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def InstTest0 : GICombineRule<
+  (defs root:$a),
+  (match (G_BUILD_VECTOR $a, $b, $c, $d)),
+  (apply [{ APPLY }])>;
+
+def InstTest1 : GICombineRule<
+  (defs root:$a),
+  (match (G_BUILD_VECTOR $a, $b)),
+  (apply [{ APPLY }])>;
+
+def InstTest2 : GICombineRule<
+  (defs root:$a),
+  (match (G_UNMERGE_VALUES $a, $b)),
+  (apply [{ APPLY }])>;
+
+def InstTest3 : GICombineRule<
+  (defs root:$a),
+  (match (G_UNMERGE_VALUES $a, $b, $c, $d)),
+  (apply [{ APPLY }])>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+  InstTest0,
+  InstTest1,
+  InstTest2,
+  InstTest3
+]>;
+
+// CHECK:      const int64_t *GenMyCombiner::getMatchTable() const {
+// CHECK-NEXT:   constexpr static int64_t MatchTable0[] = {
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 0*/ 26,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_BUILD_VECTOR,
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 1*/ 15, // Rule ID 1 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule1Enabled,
+// CHECK-NEXT:         GIM_CheckNumOperands, /*MI*/0, /*Expected*/2,
+// CHECK-NEXT:         // MIs[0] a
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] b
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // Combiner Rule #1: InstTest1
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 1: @15
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 2*/ 25, // Rule ID 0 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:         GIM_CheckNumOperands, /*MI*/0, /*Expected*/4,
+// CHECK-NEXT:         // MIs[0] a
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] b
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] c
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] d
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // Combiner Rule #0: InstTest0
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 2: @25
+// CHECK-NEXT:       GIM_Reject,
+// CHECK-NEXT:     // Label 0: @26
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 3*/ 52,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_UNMERGE_VALUES,
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 4*/ 41, // Rule ID 2 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule2Enabled,
+// CHECK-NEXT:         GIM_CheckNumOperands, /*MI*/0, /*Expected*/2,
+// CHECK-NEXT:         // MIs[0] a
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] b
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // Combiner Rule #2: InstTest2
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 4: @41
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 5*/ 51, // Rule ID 3 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule3Enabled,
+// CHECK-NEXT:         GIM_CheckNumOperands, /*MI*/0, /*Expected*/4,
+// CHECK-NEXT:         // MIs[0] a
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] b
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] c
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] d
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // Combiner Rule #3: InstTest3
+// CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 5: @51
+// CHECK-NEXT:       GIM_Reject,
+// CHECK-NEXT:     // Label 3: @52
+// CHECK-NEXT:     GIM_Reject,
+// CHECK-NEXT:     };
+// CHECK-NEXT:   return MatchTable0;
+// CHECK-NEXT: }

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table.td

 // RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
 // RUN:     -combiners=MyCombiner %s | \
 // RUN: FileCheck %s
 
 include "llvm/Target/Target.td"
 include "llvm/Target/GlobalISel/Combine.td"
 
 def MyTargetISA : InstrInfo;
 def MyTarget : Target { let InstructionSet = MyTargetISA; }
 
 def dummy;
 
-def R0 : Register<"r0"> { let Namespace = "MyTarget"; }
-def GPR32 : RegisterClass<"MyTarget", [i32], 32, (add R0)>;
-class I<dag OOps, dag IOps, list<dag> Pat>
-  : Instruction {
-  let Namespace = "MyTarget";
-  let OutOperandList = OOps;
-  let InOperandList = IOps;
-  let Pattern = Pat;
-}
-def MOV : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def TRUNC : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def ZEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def SEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-
 def HasAnswerToEverything : Predicate<"Subtarget->getAnswerToUniverse() == 42 && Subtarget->getAnswerToLife() == 42">;
 
 def WipOpcodeTest0 : GICombineRule<
   (defs root:$d),
-  (match (wip_match_opcode TRUNC):$d),
+  (match (wip_match_opcode G_TRUNC):$d),
   (apply [{ APPLY }])>;
 
 def WipOpcodeTest1 : GICombineRule<
   (defs root:$d),
-  (match (wip_match_opcode TRUNC, SEXT):$d),
+  (match (wip_match_opcode G_TRUNC, G_SEXT):$d),
   (apply [{ APPLY }])>;
 
 // Note: also checks that spaces in the type name are removed.
 def reg_matchinfo : GIDefMatchData<"Register  ">;
 def InstTest0 : GICombineRule<
   (defs root:$d, reg_matchinfo:$r0, reg_matchinfo:$r1),
-  (match (MOV $a, $b):$d),
+  (match (COPY $a, $b):$d),
   (apply [{ APPLY ${r0}, ${r1} }])>;
 
 let Predicates = [HasAnswerToEverything] in
 def InstTest1 : GICombineRule<
   (defs root:$d, reg_matchinfo:$r0),
-  (match (MOV $a, $b):$d,
-         (ZEXT $b, $c),
+  (match (COPY $a, $b):$d,
+         (G_ZEXT $b, $c),
          [{ return CHECK ${a}, ${b}, ${c}, ${d} }]),
   (apply [{ APPLY }])>;
 
+def InOutInstTest0 : GICombineRule<
+  (defs root:$root),
+  (match (G_ZEXT $tmp, $ext),
+         (G_STORE $tmp, $ptr):$root),
+  (apply (G_STORE $ext, $ptr):$root, "APPLY ${ext} ${ptr} ${root}")>;
+
+// Imm operand of G_CONSTANT should match a literal int, while the second
+// should match a constant.
+def InOutInstTest1 : GICombineRule<
+  (defs root:$dst),
+  (match
+      (G_CONSTANT $x, -42:$z),
+      (G_AND $dst, $x, (i32 43))),
+  (apply (G_TRUNC $dst, $z))>;
+
+def MatchICst: GICombinePatFrag<
+    (outs),
+    (ins gi_mo:$foo, gi_imm:$cst),
+    [(pattern "return matchIConstant(${foo}, ${cst});")]>;
+
+def PatFragTest0 : GICombineRule<
+  (defs root:$dst),
+  (match (G_ZEXT $dst, $cst), (MatchICst $cst, (i32 0))),
+  (apply (COPY $dst, (i32 0)))>;
+
+// TODO: add test with temp reg use
+
 def MyCombiner: GICombinerHelper<"GenMyCombiner", [
   WipOpcodeTest0,
   WipOpcodeTest1,
   InstTest0,
-  InstTest1
+  InstTest1,
+  InOutInstTest0,
+  InOutInstTest1,
+  PatFragTest0
 ]>;
 
 // We have at most 2 registers used by one rule at a time, so we should only have 2 registers MDInfos.
 
 // CHECK:      struct MatchInfosTy {
 // CHECK-NEXT:   Register MDInfo0, MDInfo1;
 // CHECK-NEXT: };
 
 // Check predicates
 // CHECK:      switch (PredicateID) {
 // CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner0: {
 // CHECK-NEXT:   return CHECK State.MIs[0]->getOperand(0), State.MIs[0]->getOperand(1), State.MIs[1]->getOperand(1), State.MIs[0]
 // CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner1: {
+// CHECK-NEXT:   return matchIConstant(State.MIs[0]->getOperand(1), 0);
+// CHECK-NEXT: }
 
 // Verify we reset MatchData on each tryCombineAll
 // CHECK:      bool GenMyCombiner::tryCombineAll(MachineInstr &I) const {
 // CHECK-NEXT:   const TargetSubtargetInfo &ST = MF.getSubtarget();
 // CHECK-NEXT:   const PredicateBitset AvailableFeatures = getAvailableFeatures();
 // CHECK-NEXT:   NewMIVector OutMIs;
 // CHECK-NEXT:   State.MIs.clear();
 // CHECK-NEXT:   State.MIs.push_back(&I);
 // CHECK-NEXT:   MatchInfos = MatchInfosTy();
 // CHECK-EMPTY:
-// CHECK-NEXT:   if (executeMatchTable(*this, OutMIs, State, ExecInfo, getMatchTable(), *ST.getInstrInfo(), MRI, *MRI.getTargetRegisterInfo(), *ST.getRegBankInfo(), AvailableFeatures, /*CoverageInfo*/ nullptr))
+// CHECK-NEXT:   if (executeMatchTable(*this, OutMIs, State, ExecInfo, getMatchTable(), *ST.getInstrInfo(), MRI, *MRI.getTargetRegisterInfo(), *ST.getRegBankInfo(), AvailableFeatures, /*CoverageInfo*/ nullptr, &Observer))
 // CHECK-NEXT:     return true;
 // CHECK-NEXT:   }
 // CHECK-EMPTY:
 // CHECK-NEXT:   return false;
 // CHECK-NEXT: }
 
+// Check apply
+// CHECK:      enum {
+// CHECK-NEXT:   GICXXCustomAction_CombineApplyGICombiner0 = GICXXCustomAction_Invalid + 1,
+// CHECK-NEXT:   GICXXCustomAction_CombineApplyGICombiner1,
+// CHECK-NEXT:   GICXXCustomAction_CombineApplyGICombiner2,
+// CHECK-NEXT: };
+// CHECK-NEXT: void GenMyCombiner::runCustomAction(unsigned ApplyID, const MatcherState &State, NewMIVector &OutMIs) const {
+// CHECK-NEXT:   switch(ApplyID) {
+// CHECK-NEXT:   case GICXXCustomAction_CombineApplyGICombiner0:{
+// CHECK-NEXT:     APPLY
+// CHECK-NEXT:     return;
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXCustomAction_CombineApplyGICombiner1:{
+// CHECK-NEXT:     APPLY MatchInfos.MDInfo0, MatchInfos.MDInfo1
+// CHECK-NEXT:     return;
+// CHECK-NEXT:   }
+// CHECK-NEXT:   case GICXXCustomAction_CombineApplyGICombiner2:{
+// CHECK-NEXT:     APPLY State.MIs[1]->getOperand(1) State.MIs[0]->getOperand(1) OutMIs[0]
+// CHECK-NEXT:     return;
+// CHECK-NEXT:   }
+// CHECK-NEXT: }
+// CHECK-NEXT:   llvm_unreachable("Unknown Apply Action");
+// CHECK-NEXT: }
 
 // Verify match table.
 // CHECK:      const int64_t *GenMyCombiner::getMatchTable() const {
 // CHECK-NEXT:   constexpr static int64_t MatchTable0[] = {
 // CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 0*/ 20,
-// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, MyTarget::TRUNC,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_TRUNC,
 // CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 1*/ 12, // Rule ID 0 //
 // CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
-// CHECK-NEXT:         // Combiner Rule #0: WipOpcodeTest0; wip_match_opcode alternative 'TRUNC'
+// CHECK-NEXT:         // Combiner Rule #0: WipOpcodeTest0; wip_match_opcode 'G_TRUNC'
 // CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
 // CHECK-NEXT:         GIR_Done,
 // CHECK-NEXT:       // Label 1: @12
 // CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 2*/ 19, // Rule ID 1 //
 // CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule1Enabled,
-// CHECK-NEXT:         // Combiner Rule #1: WipOpcodeTest1; wip_match_opcode alternative 'TRUNC'
+// CHECK-NEXT:         // Combiner Rule #1: WipOpcodeTest1; wip_match_opcode 'G_TRUNC'
 // CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
 // CHECK-NEXT:         GIR_Done,
 // CHECK-NEXT:       // Label 2: @19
 // CHECK-NEXT:       GIM_Reject,
 // CHECK-NEXT:     // Label 0: @20
 // CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 3*/ 30, // Rule ID 2 //
 // CHECK-NEXT:       GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule1Enabled,
-// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, MyTarget::SEXT,
-// CHECK-NEXT:       // Combiner Rule #1: WipOpcodeTest1; wip_match_opcode alternative 'SEXT'
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_SEXT,
+// CHECK-NEXT:       // Combiner Rule #1: WipOpcodeTest1; wip_match_opcode 'G_SEXT'
 // CHECK-NEXT:       GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
 // CHECK-NEXT:       GIR_Done,
 // CHECK-NEXT:     // Label 3: @30
 // CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 4*/ 64,
-// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, MyTarget::MOV,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::COPY,
 // CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 5*/ 42, // Rule ID 3 //
 // CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule2Enabled,
 // CHECK-NEXT:         // MIs[0] a
 // CHECK-NEXT:         // No operand predicates
 // CHECK-NEXT:         // MIs[0] b
 // CHECK-NEXT:         // No operand predicates
 // CHECK-NEXT:         // Combiner Rule #2: InstTest0
 // CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner1,
 // CHECK-NEXT:         GIR_Done,
 // CHECK-NEXT:       // Label 5: @42
 // CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 6*/ 63, // Rule ID 4 //
 // CHECK-NEXT:         GIM_CheckFeatures, GIFBS_HasAnswerToEverything,
 // CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule3Enabled,
 // CHECK-NEXT:         // MIs[0] a
 // CHECK-NEXT:         // No operand predicates
 // CHECK-NEXT:         // MIs[0] b
-// CHECK-NEXT:         GIM_RecordInsn, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
-// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/1, MyTarget::ZEXT,
+// CHECK-NEXT:         GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT:         GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
 // CHECK-NEXT:         // MIs[1] c
 // CHECK-NEXT:         // No operand predicates
 // CHECK-NEXT:         GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner0,
 // CHECK-NEXT:         GIM_CheckIsSafeToFold, /*InsnID*/1,
 // CHECK-NEXT:         // Combiner Rule #3: InstTest1
 // CHECK-NEXT:         GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
 // CHECK-NEXT:         GIR_Done,
 // CHECK-NEXT:       // Label 6: @63
 // CHECK-NEXT:       GIM_Reject,
 // CHECK-NEXT:     // Label 4: @64
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 7*/ 101, // Rule ID 5 //
+// CHECK-NEXT:       GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule4Enabled,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_STORE,
+// CHECK-NEXT:       // MIs[0] tmp
+// CHECK-NEXT:       GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/0, // MIs[1]
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:       // MIs[1] ext
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       // MIs[0] ptr
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:       // Combiner Rule #4: InOutInstTest0
+// CHECK-NEXT:       GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::G_STORE,
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/1, /*OpIdx*/1, // ext
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/1, // ptr
+// CHECK-NEXT:       GIR_MergeMemOperands, /*InsnID*/0, /*MergeInsnID's*/0, 1, GIU_MergeMemOperands_EndOfList,
+// CHECK-NEXT:       GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:       GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner2,
+// CHECK-NEXT:       GIR_Done,
+// CHECK-NEXT:     // Label 7: @101
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 8*/ 143, // Rule ID 6 //
+// CHECK-NEXT:       GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule5Enabled,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_AND,
+// CHECK-NEXT:       GIM_CheckType, /*MI*/0, /*Op*/2, /*Type*/GILLT_s32,
+// CHECK-NEXT:       // MIs[0] dst
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       // MIs[0] x
+// CHECK-NEXT:       GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_CONSTANT,
+// CHECK-NEXT:       // MIs[1] z
+// CHECK-NEXT:       GIM_CheckLiteralInt, /*MI*/1, /*Op*/1, -42,
+// CHECK-NEXT:       GIM_CheckConstantInt, /*MI*/0, /*Op*/2, 43,
+// CHECK-NEXT:       GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT:       // Combiner Rule #5: InOutInstTest1
+// CHECK-NEXT:       GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::G_TRUNC,
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/1, /*OpIdx*/1, // z
+// CHECK-NEXT:       GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:       GIR_Done,
+// CHECK-NEXT:     // Label 8: @143
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 9*/ 167, // Rule ID 7 //
+// CHECK-NEXT:       GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule6Enabled,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:       // MIs[0] dst
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       // MIs[0] cst
+// CHECK-NEXT:       // No operand predicates
+// CHECK-NEXT:       GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner1,
+// CHECK-NEXT:       // Combiner Rule #6: PatFragTest0 @ [__anon_pat_match_6_1[0]]
+// CHECK-NEXT:       GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT:       GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:       GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT:       GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:       GIR_Done,
+// CHECK-NEXT:     // Label 9: @167
 // CHECK-NEXT:     GIM_Reject,
 // CHECK-NEXT:     };
 // CHECK-NEXT:   return MatchTable0;
 // CHECK-NEXT: }

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/operand-types.td

+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN:     -gicombiner-stop-after-parse -combiners=MyCombiner %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+// CHECK:      (CombineRule name:InstTest0 id:0 root:x
+// CHECK-NEXT:   (MatchPats
+// CHECK-NEXT:     __anon_pat_match_0_0:(CodeGenInstructionPattern G_BUILD_VECTOR operands:[<def>i64:$z, i32:$y, i16:$w])
+// CHECK-NEXT:     <root>__anon_pat_match_0_1:(CodeGenInstructionPattern G_BUILD_VECTOR operands:[<def>i64:$x, i32:$y, i64:$z])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (ApplyPats
+// CHECK-NEXT:     __anon_pat_apply_0_2:(CodeGenInstructionPattern G_MUL operands:[<def>i64:$x, i32:$y, i16:$w])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (OperandTable
+// CHECK-NEXT:     [z match_pat:__anon_pat_match_0_0]
+// CHECK-NEXT:     [y match-live-in]
+// CHECK-NEXT:     [w match-live-in]
+// CHECK-NEXT:     [x match_pat:__anon_pat_match_0_1 apply_pat:__anon_pat_apply_0_2]
+// CHECK-NEXT:   )
+// CHECK-NEXT: )
+def InstTest0 : GICombineRule<
+  (defs root:$x),
+  (match
+      (G_BUILD_VECTOR i64:$z, $y, $w),
+      (G_BUILD_VECTOR $x, i32:$y, $z)),
+  (apply (G_MUL i64:$x, $y, i16:$w))>;
+
+// CHECK:      (CombineRule name:PatFragTest0 id:1 root:dst
+// CHECK-NEXT:   (PatFrags
+// CHECK-NEXT:     (PatFrag name:FooPF
+// CHECK-NEXT:       (outs [cst:root])
+// CHECK-NEXT:       (alternatives [
+// CHECK-NEXT:         [
+// CHECK-NEXT:           (CodeGenInstructionPattern name:__anon_pat_pattern_1_1 G_BUILD_VECTOR operands:[<def>i64:$cst, i8:$y, $x]),
+// CHECK-NEXT:           (CodeGenInstructionPattern name:__anon_pat_pattern_1_2 G_BUILD_VECTOR operands:[<def>$x, i8:$y, $w]),
+// CHECK-NEXT:         ],
+// CHECK-NEXT:         [
+// CHECK-NEXT:           (CodeGenInstructionPattern name:__anon_pat_pattern_1_3 G_BUILD_VECTOR operands:[<def>i32:$cst, i32:$y, i16:$x]),
+// CHECK-NEXT:           (CodeGenInstructionPattern name:__anon_pat_pattern_1_4 G_BUILD_VECTOR operands:[<def>i16:$x, i32:$y, $w]),
+// CHECK-NEXT:         ],
+// CHECK-NEXT:       ])
+// CHECK-NEXT:     )
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (MatchPats
+// CHECK-NEXT:     <root>__anon_pat_match_1_0:(PatFragPattern FooPF operands:[<def>$dst])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (ApplyPats
+// CHECK-NEXT:     __anon_pat_apply_1_5:(CodeGenInstructionPattern COPY operands:[<def>$dst, (i32 0)])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (OperandTable
+// CHECK-NEXT:     [dst match_pat:__anon_pat_match_1_0 apply_pat:__anon_pat_apply_1_5]
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (PermutationsToEmit
+// CHECK-NEXT:     [__anon_pat_match_1_0[0]],
+// CHECK-NEXT:     [__anon_pat_match_1_0[1]],
+// CHECK-NEXT:   )
+// CHECK-NEXT: )
+def FooPF: GICombinePatFrag<
+    (outs root:$cst),(ins),
+    [
+      (pattern (G_BUILD_VECTOR i64:$cst, $y, $x), (G_BUILD_VECTOR $x, i8:$y, $w)),
+      (pattern (G_BUILD_VECTOR i32:$cst, $y, i16:$x), (G_BUILD_VECTOR $x, i32:$y, $w)),
+    ]>;
+def PatFragTest0 : GICombineRule<
+  (defs root:$dst),
+  (match (FooPF $dst)),
+  (apply (COPY $dst, (i32 0)))>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+  InstTest0,
+  PatFragTest0
+]>;

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/patfrag-errors.td

+// RUN: not llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN:     -combiners=MyCombiner %s 2>&1| \
+// RUN: FileCheck %s -implicit-check-not=error:
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def dummy;
+
+def MatchFooPerms: GICombinePatFrag<
+    (outs),
+    (ins gi_mo:$foo, gi_imm:$cst),
+    [
+      (pattern "return foo(${foo}, ${cst})"),
+      (pattern "return bar(${foo}, ${cst})"),
+      (pattern "return bux(${foo}, ${cst})"),
+    ]>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: cannot emit rule 'too_many_perms'; 27 permutations would be emitted, but the max is 16
+let MaxPermutations = 16 in
+def too_many_perms : GICombineRule<
+  (defs root:$dst),
+  (match (G_ZEXT $dst, $cst),
+         (MatchFooPerms $cst, (i32 0)):$a,
+         (MatchFooPerms $cst, (i32 0)):$b,
+         (MatchFooPerms $cst, (i32 0)):$c
+  ),
+  (apply (COPY $dst, (i32 0)), "APPLY ${src}")>;
+
+def CXXUsesLiveIn: GICombinePatFrag<
+    (outs),
+    (ins gi_mo:$in),
+    [
+      (pattern "return foo(${in})"),
+    ]>;
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: operand 'foo' (for parameter 'in' of 'CXXUsesLiveIn') cannot be unbound
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: one or more alternatives of 'CXXUsesLiveIn' do not bind 'in' to an instruction operand; either use a bound operand or ensure 'CXXUsesLiveIn' binds 'in' in all alternatives
+def undef_livein : GICombineRule<
+  (defs root:$dst),
+  (match (G_ZEXT $dst, $bar),
+         (CXXUsesLiveIn $foo)
+  ),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: GICombinePatFrag must have one root in its 'out' operands
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'OutMustBeRoot'
+def OutMustBeRoot: GICombinePatFrag<
+    (outs $foo, $bar),
+    (ins),
+    [
+      (pattern (G_ZEXT $foo, $bar), (G_FPEXT $bar, $y)),
+    ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(OutMustBeRoot ?:$bar, ?:$foo)'
+def out_must_be_root : GICombineRule<
+  (defs root:$dst),
+  (match (G_ZEXT $dst, $bar),
+         (OutMustBeRoot $bar, $foo)
+  ),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: output parameter 'bar' must be 'root' or 'gi_mo'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'BadOutType'
+def BadOutType: GICombinePatFrag<
+    (outs root:$foo, gi_imm:$bar),
+    (ins),
+    [
+      (pattern (G_ZEXT $foo, $bar), (G_FPEXT $bar, $y)),
+    ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(BadOutType ?:$bar, ?:$foo)'
+def bad_out_type : GICombineRule<
+  (defs root:$dst),
+  (match (G_ZEXT $dst, $bar),
+         (BadOutType $bar, $foo)
+  ),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: pattern 'dbg' ('G_FPEXT') is unreachable from the pattern root!
+def UnreachablePat: GICombinePatFrag<
+    (outs root:$foo, $bar),
+    (ins),
+    [
+      (pattern (G_ZEXT $foo, $x), (G_FPEXT $bar, $y):$dbg),
+    ]>;
+def unreachable_pat : GICombineRule<
+  (defs root:$dst),
+  (match (G_ZEXT $dst, $bar),
+         (UnreachablePat $bar, $foo)
+  ),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: wip_match_opcode cannot be used in GICombinePatFrag
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'WipMatchOpcodePatFrag'
+def WipMatchOpcodePatFrag: GICombinePatFrag<
+    (outs),
+    (ins),
+    [
+      (pattern (wip_match_opcode G_ZEXT)),
+    ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(WipMatchOpcodePatFrag)'
+def wip_match_opcode_patfrag : GICombineRule<
+  (defs root:$dst),
+  (match (WipMatchOpcodePatFrag)),
+  (apply (COPY $dst, (i32 0)))>;
+
+def DummyPF: GICombinePatFrag<(outs),(ins),[]>;
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: nested GICombinePatFrag are not supported
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'NestingPatFrag'
+def NestingPatFrag: GICombinePatFrag<
+    (outs),
+    (ins),
+    [
+      (pattern (DummyPF)),
+    ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(NestingPatFrag)'
+def nest_pat_frag : GICombineRule<
+  (defs root:$dst),
+  (match (NestingPatFrag)),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: input parameter 'k' cannot be redefined!
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'DupParamIn'
+def DupParamIn: GICombinePatFrag<
+    (outs),
+    (ins gi_mo:$k, gi_mo:$k),
+    [
+      (pattern (G_ZEXT $k, $x)),
+    ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(DupParamIn ?:$dst, ?:$bar)'
+def dup_params_in : GICombineRule<
+  (defs root:$dst),
+  (match (DupParamIn $dst, $bar)),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: duplicate parameter 'k'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'DupParamOut'
+def DupParamOut: GICombinePatFrag<
+  (outs root:$k, root:$k),
+  (ins),
+  [
+    (pattern (G_ZEXT $k, $x)),
+  ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(DupParamOut ?:$dst, ?:$bar)'
+def dup_params_out : GICombineRule<
+  (defs root:$dst),
+  (match (DupParamOut $dst, $bar)),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: input parameter 'k' cannot be redefined!
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'DupParamInOut'
+def DupParamInOut: GICombinePatFrag<
+  (outs root:$k),
+  (ins gi_mo:$k),
+  [
+    (pattern (G_ZEXT $k, $x)),
+  ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(DupParamInOut ?:$dst, ?:$bar)'
+def dup_params_inout : GICombineRule<
+  (defs root:$dst),
+  (match (DupParamInOut $dst, $bar)),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: output parameter 'k' must be defined by all alternative patterns in 'DefByAllAlts'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'DefByAllAlts'
+def DefByAllAlts: GICombinePatFrag<
+  (outs root:$k),
+  (ins),
+  [
+    (pattern (G_ZEXT $k, $x)),
+    (pattern (G_FPEXT $z, $k))
+  ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(DefByAllAlts ?:$dst)'
+def def_by_all_alts : GICombineRule<
+  (defs root:$dst),
+  (match (DefByAllAlts $dst)),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: Operand 'x' is defined multiple times in patterns of alternative #1
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'MultiDefInPat'
+def MultiDefInPat: GICombinePatFrag<
+  (outs root:$k),
+  (ins),
+  [
+    (pattern (G_ZEXT $k, $a)),
+    (pattern (G_ZEXT $k, $x), (G_ZEXT $x, $foo), (G_FPEXT $x, $foo)),
+  ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(MultiDefInPat ?:$dst)'
+def multi_def_in_pat : GICombineRule<
+  (defs root:$dst),
+  (match (MultiDefInPat $dst)),
+  (apply (COPY $dst, (i32 0)))>;
+
+def ExpectedImm: GICombinePatFrag<
+  (outs root:$k),
+  (ins gi_imm:$i),
+  [
+    (pattern (G_ZEXT $k, $i)),
+  ]>;
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: expected operand 1 of 'ExpectedImm' to be an immediate; got MachineOperand $z
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(ExpectedImm ?:$dst, ?:$z)'
+def expected_imm : GICombineRule<
+  (defs root:$dst),
+  (match (ExpectedImm $dst, $z)),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: operand 1 of 'ExpectedImm' cannot be a named immediate
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(ExpectedImm ?:$dst, (i32 0):$z)'
+def expected_imm_namedimm : GICombineRule<
+  (defs root:$dst),
+  (match (ExpectedImm $dst, (i32 0):$z)),
+  (apply (COPY $dst, (i32 0)))>;
+
+def ExpectedMO: GICombinePatFrag<
+  (outs root:$k),
+  (ins gi_mo:$i),
+  [
+    (pattern (G_ZEXT $k, $i)),
+  ]>;
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: expected operand 1 of 'ExpectedMO' to be a MachineOperand; got imm 0
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(ExpectedMO ?:$dst, (i32 0))'
+def expected_mo : GICombineRule<
+  (defs root:$dst),
+  (match (ExpectedMO $dst, (i32 0))),
+  (apply (COPY $dst, (i32 0)))>;
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: expected operand 1 of 'ExpectedMO' to be a MachineOperand; got imm 0:$z
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(ExpectedMO ?:$dst, (i32 0):$z)'
+def expected_mo_namedimm : GICombineRule<
+  (defs root:$dst),
+  (match (ExpectedMO $dst, (i32 0):$z)),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: 'dum': using GICombinePatFrag is not supported in apply patterns
+def patfrag_in_apply : GICombineRule<
+  (defs root:$dst),
+  (match (COPY $dst, (i32 0))),
+  (apply (DummyPF):$dum)>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: cannot use 'DummyPF as match root
+def patfrag_cannot_be_root : GICombineRule<
+  (defs root:$root),
+  (match (DummyPF):$root),
+  (apply (COPY $dst, (i32 0)):$root)>;
+
+def TypedParams: GICombinePatFrag<
+  (outs root:$k),
+  (ins gi_mo:$i),
+  [
+    (pattern (G_ZEXT $k, i32:$i)),
+  ]>;
+// CHECK: :[[@LINE+3]]:{{[0-9]+}}: warning: impossible type constraints: operand 1 of 'broken' has type 'i64', but 'TypedParams' constrains it to 'i32'
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: note: operand 1 of 'broken' is 'k'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: argument 1 of 'TypedParams' is 'i'
+def inconsistent_arg_type : GICombineRule<
+  (defs root:$dst),
+  (match (TypedParams $dst, i64:$k):$broken),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: error: Failed to parse one or more rules
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+  too_many_perms,
+  undef_livein,
+  out_must_be_root,
+  bad_out_type,
+  unreachable_pat,
+  wip_match_opcode_patfrag,
+  nest_pat_frag,
+  dup_params_in,
+  dup_params_out,
+  dup_params_inout,
+  def_by_all_alts,
+  multi_def_in_pat,
+  expected_imm,
+  expected_imm_namedimm,
+  expected_mo,
+  expected_mo_namedimm,
+  patfrag_in_apply,
+  patfrag_cannot_be_root,
+  inconsistent_arg_type
+]>;

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-errors.td

+// RUN: not llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN:     -combiners=MyCombiner %s 2>&1| \
+// RUN: FileCheck %s -implicit-check-not=error:
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Cannot find root 'missing' in match patterns!
+def root_not_found : GICombineRule<
+  (defs root:$missing),
+  (match (COPY $a, $b):$d),
+  (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: apply pattern 'd' is supposed to be a root but it does not redefine any of the defs of the match root
+def misleading_root : GICombineRule<
+  (defs root:$d),
+  (match (COPY $a, $b):$d),
+  (apply (COPY $tmp, $b):$d,
+         (COPY $a, $tmp))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: C++ code cannot be the root of a rule!
+def cxx_root : GICombineRule<
+  (defs root:$a),
+  (match "return MATCH":$a),
+  (apply [{ APPLY }]:$z)>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Cannot use live-in operand 'b' as match pattern root!
+def livein_root : GICombineRule<
+  (defs root:$b),
+  (match (COPY $a, $b)),
+  (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'COPY' expected 2 operands, got 1
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(COPY ?:$a)'
+def not_enough_operands : GICombineRule<
+  (defs root:$d),
+  (match (COPY $a):$d),
+  (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'COPY' expected 2 operands, got 3
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(COPY ?:$a, ?:$b, ?:$c)'
+def too_many_operands : GICombineRule<
+  (defs root:$d),
+  (match (COPY $a, $b, $c):$d),
+  (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Operand 'd' is defined multiple times in the 'match' patterns
+def multi_defs : GICombineRule<
+  (defs root:$d),
+  (match (COPY $d, $b), (COPY $d, $x)),
+  (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Operand 'd' is defined multiple times in the 'match' patterns
+def multi_defs_2 : GICombineRule<
+  (defs root:$d),
+  (match (G_UNMERGE_VALUES $d, $d, $b)),
+  (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: pattern 'foo' ('COPY') is unreachable from the pattern root!
+def unreachable_pat : GICombineRule<
+  (defs root:$d),
+  (match (COPY $a, $b):$d, (COPY $z, $k):$foo),
+  (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: 'applytest': wip_match_opcode is not supported in apply patterns
+def wip_match_opcode_in_apply : GICombineRule<
+  (defs root:$d),
+  (match (COPY $a, $b):$d, (wip_match_opcode G_ZEXT)),
+  (apply (wip_match_opcode G_ZEXT):$applytest)>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: wip_match_opcode can not be used with instruction patterns!
+def wip_match_opcode_with_inst_pat : GICombineRule<
+  (defs root:$d),
+  (match (COPY $a, $b):$d, (wip_match_opcode G_ZEXT)),
+  (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: wip_opcode_match can only be present once
+def multiple_wip_match_opcode : GICombineRule<
+  (defs root:$d),
+  (match (wip_match_opcode COPY):$d, (wip_match_opcode G_ZEXT)),
+  (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Operand 'a' is defined multiple times in the 'apply' patterns
+def multiple_def_in_apply : GICombineRule<
+  (defs root:$d),
+  (match (COPY $a, $b):$d),
+  (apply (COPY $a, $b), (COPY $a, $b))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: 'd' match pattern defined more than once!
+def redef_match : GICombineRule<
+  (defs root:$d),
+  (match (COPY $a, $b):$d, (COPY $b, $z):$d),
+  (apply (COPY $a, $b))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: 'o' apply pattern defined more than once!
+def redef_apply: GICombineRule<
+  (defs root:$d),
+  (match (COPY $a, $b):$d),
+  (apply (COPY $a, $x):$o, (COPY $x, $b):$o)>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: redefining an instruction other than the root is not supported (operand 'b'
+def redef_nonroot : GICombineRule<
+  (defs root:$a),
+  (match (COPY $a, $b), (COPY $b, $z)),
+  (apply (COPY $a, $b), (G_ZEXT $b, (i32 0)))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Cannot define live-in operand 'b' in the 'apply' pattern
+def cannot_def_match_livein : GICombineRule<
+  (defs root:$d),
+  (match (COPY $a, $b):$d),
+  (apply (COPY $a, $b), (COPY $b, $b))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: invalid output operand 'x': operand is not a live-in of the match pattern, and it has no definition
+def undef_in_apply : GICombineRule<
+  (defs root:$d),
+  (match (COPY $a, $b):$d),
+  (apply (COPY $a, $x))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: def of pattern root 'a' is not redefined in the apply pattern!
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: match pattern root is 'foo'
+def no_redef_in_apply : GICombineRule<
+  (defs root:$a),
+  (match (COPY $a, $b):$foo),
+  (apply (COPY $x, $b))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error:  def of pattern root 'b' is not redefined in the apply pattern!
+def no_redef_in_apply_multidefroot : GICombineRule<
+  (defs root:$a),
+  (match (G_UNMERGE_VALUES $a, $b, $c)),
+  (apply (COPY $a, $c))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: cannot use wip_match_opcode in combination with apply instruction patterns
+def instpat_with_wipmatch : GICombineRule<
+  (defs root:$a),
+  (match (wip_match_opcode COPY):$d),
+  (apply (COPY $x, $b):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: cannot parse operand '(i32)'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(COPY ?:$x, (i32))
+def bad_imm_noarg : GICombineRule<
+  (defs root:$a),
+  (match (COPY $x, (i32)):$d),
+  (apply (COPY $x, $b):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: cannot parse operand '(i32 0, 0)'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(COPY ?:$x, (i32 0, 0))
+def bad_imm_too_many_args : GICombineRule<
+  (defs root:$a),
+  (match (COPY $x, (i32 0, 0)):$d),
+  (apply (COPY $x, $b):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: cannot parse immediate '(COPY 0)', 'COPY' is not a ValueType
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(COPY ?:$x, (COPY 0))
+def bad_imm_not_a_valuetype : GICombineRule<
+  (defs root:$a),
+  (match (COPY $x, (COPY 0)):$d),
+  (apply (COPY $x, $b):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: invalid output operand 'imm': output immediates cannot be named
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: while emitting pattern 'd' (COPY)
+def output_imm_cannot_be_named : GICombineRule<
+  (defs root:$x),
+  (match (COPY $x, (i32 0)):$d),
+  (apply (COPY $x, (i32 0):$imm):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: output immediates must be typed!
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: while emitting pattern 'd' (COPY)
+def output_imm_must_be_typed : GICombineRule<
+  (defs root:$x),
+  (match (COPY $x, (i32 0)):$d),
+  (apply (COPY $x, 0):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'G_BUILD_VECTOR' expected at least 2 operands, got 1
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(G_BUILD_VECTOR ?:$x)'
+def too_few_ops_for_variadic : GICombineRule<
+  (defs root:$x),
+  (match (G_BUILD_VECTOR $x)),
+  (apply (COPY $x, 0))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: expected an operand name after 'i32'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(G_FNEG ?:$x, i32)'
+def expected_op_name : GICombineRule<
+  (defs root:$x),
+  (match (G_FNEG $x, i32)),
+  (apply (COPY $x, (i32 0)))>;
+
+// CHECK: :[[@LINE+3]]:{{[0-9]+}}: error: invalid operand type: 'not_a_type' is not a ValueType
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: Failed to parse pattern: '(G_FNEG ?:$x, not_a_type:$y)'
+def not_a_type;
+def bad_mo_type_not_a_valuetype : GICombineRule<
+  (defs root:$x),
+  (match (G_FNEG $x, not_a_type:$y)),
+  (apply (COPY $x, (i32 0)))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: def of a new register 'newreg' in the apply patterns must have a type
+def untyped_new_reg_in_apply : GICombineRule<
+  (defs root:$x),
+  (match (G_FNEG $x, $y)),
+  (apply (COPY $newreg, (i32 0)),
+         (COPY $x, $newreg))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'y' is a named immediate, it cannot be defined by another instruction
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: 'y' is defined by 'foo'
+def def_named_imm_match : GICombineRule<
+  (defs root:$x),
+  (match  (G_SEXT $y, $z):$foo,
+          (G_FNEG $x, (i32 0):$y)),
+  (apply (COPY $x, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: invalid output operand 'tmp': output immediates cannot be named
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: while emitting pattern 'foo' (COPY)
+def def_named_imm_apply : GICombineRule<
+  (defs root:$x),
+  (match (G_FNEG $x, $y)),
+  (apply (COPY i32:$tmp, $y),
+         (COPY $x, (i32 0):$tmp):$foo)>;
+
+// CHECK: error: Failed to parse one or more rules
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+  root_not_found,
+  misleading_root,
+  cxx_root,
+  livein_root,
+  not_enough_operands,
+  too_many_operands,
+  multi_defs,
+  multi_defs_2,
+  unreachable_pat,
+  wip_match_opcode_in_apply,
+  wip_match_opcode_with_inst_pat,
+  multiple_wip_match_opcode,
+  multiple_def_in_apply,
+  redef_match,
+  redef_apply,
+  redef_nonroot,
+  cannot_def_match_livein,
+  undef_in_apply,
+  no_redef_in_apply,
+  no_redef_in_apply_multidefroot,
+  instpat_with_wipmatch,
+  bad_imm_noarg,
+  bad_imm_too_many_args,
+  bad_imm_not_a_valuetype,
+  output_imm_cannot_be_named,
+  output_imm_must_be_typed,
+  too_few_ops_for_variadic,
+  expected_op_name,
+  bad_mo_type_not_a_valuetype,
+  untyped_new_reg_in_apply,
+  def_named_imm_match,
+  def_named_imm_apply
+]>;

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-parsing-errors.td

-// RUN: not llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
-// RUN:     -combiners=MyCombiner %s 2>&1| \
-// RUN: FileCheck %s -implicit-check-not=error:
-
-include "llvm/Target/Target.td"
-include "llvm/Target/GlobalISel/Combine.td"
-
-def MyTargetISA : InstrInfo;
-def MyTarget : Target { let InstructionSet = MyTargetISA; }
-
-def dummy;
-
-def R0 : Register<"r0"> { let Namespace = "MyTarget"; }
-def GPR32 : RegisterClass<"MyTarget", [i32], 32, (add R0)>;
-class I<dag OOps, dag IOps, list<dag> Pat>
-  : Instruction {
-  let Namespace = "MyTarget";
-  let OutOperandList = OOps;
-  let InOperandList = IOps;
-  let Pattern = Pat;
-}
-def MOV : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def ADD : I<(outs GPR32:$dst), (ins GPR32:$src1, GPR32:$src2), []>;
-def SUB : I<(outs GPR32:$dst), (ins GPR32:$src1, GPR32:$src2), []>;
-def MUL : I<(outs GPR32:$dst), (ins GPR32:$src1, GPR32:$src2), []>;
-def TRUNC : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def SEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def ZEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def ICMP : I<(outs GPR32:$dst), (ins GPR32:$tst, GPR32:$src1, GPR32:$src2), []>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Cannot find root 'missing' in match patterns!
-def root_not_found : GICombineRule<
-  (defs root:$missing),
-  (match (MOV $a, $b):$d),
-  (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Cannot use live-in operand 'b' as match pattern root!
-def livein_root : GICombineRule<
-  (defs root:$b),
-  (match (MOV $a, $b)),
-  (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'MOV' expected 2 operands, got 1
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Expected a subclass of GIMatchKind or a sub-dag whose operator is either of a GIMatchKindWithArgs or Instruction
-def not_enough_operands : GICombineRule<
-  (defs root:$d),
-  (match (MOV $a):$d),
-  (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'MOV' expected 2 operands, got 3
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Expected a subclass of GIMatchKind or a sub-dag whose operator is either of a GIMatchKindWithArgs or Instruction
-def too_many_operands : GICombineRule<
-  (defs root:$d),
-  (match (MOV $a, $b, $c):$d),
-  (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Operand 'd' is defined multiple times in the 'match' patterns
-def multi_defs : GICombineRule<
-  (defs root:$d),
-  (match (MOV $d, $b), (MOV $d, $x)),
-  (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Instruction pattern '__anon_pat_match_5_0' is unreachable from the pattern root!
-def unreachable_pat : GICombineRule<
-  (defs root:$d),
-  (match (MOV $a, $b):$d, (MOV $z, $k)),
-  (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error:  wip_match_opcode can not be used with instruction patterns!
-def wip_match_opcode_with_inst_pat : GICombineRule<
-  (defs root:$d),
-  (match (MOV $a, $b):$d, (wip_match_opcode SEXT)),
-  (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error:  wip_opcode_match can only be present once
-def multiple_wip_match_opcode : GICombineRule<
-  (defs root:$d),
-  (match (wip_match_opcode MOV):$d, (wip_match_opcode SEXT)),
-  (apply [{ APPLY }])>;
-
-// CHECK: error: Failed to parse one or more rules
-
-def MyCombiner: GICombinerHelper<"GenMyCombiner", [
-  root_not_found,
-  livein_root,
-  not_enough_operands,
-  too_many_operands,
-  multi_defs,
-  unreachable_pat,
-  wip_match_opcode_with_inst_pat,
-  multiple_wip_match_opcode
-]>;

llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-parsing.td

 // RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
 // RUN:     -gicombiner-stop-after-parse -combiners=MyCombiner %s | \
 // RUN: FileCheck %s
 
 include "llvm/Target/Target.td"
 include "llvm/Target/GlobalISel/Combine.td"
 
 def MyTargetISA : InstrInfo;
 def MyTarget : Target { let InstructionSet = MyTargetISA; }
 
 def dummy;
 
-def R0 : Register<"r0"> { let Namespace = "MyTarget"; }
-def GPR32 : RegisterClass<"MyTarget", [i32], 32, (add R0)>;
-class I<dag OOps, dag IOps, list<dag> Pat>
-  : Instruction {
-  let Namespace = "MyTarget";
-  let OutOperandList = OOps;
-  let InOperandList = IOps;
-  let Pattern = Pat;
-}
-def MOV : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def TRUNC : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def ZEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def SEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-
 def HasAnswerToEverything : Predicate<"Subtarget->getAnswerToUniverse() == 42 && Subtarget->getAnswerToLife() == 42">;
 def reg_matchinfo : GIDefMatchData<"Register">;
 
 // CHECK:      (CombineRule name:WipOpcodeTest0 id:0 root:d
-// CHECK-NEXT:   (MatchDatas <empty>)
 // CHECK-NEXT:   (MatchPats
-// CHECK-NEXT:     <root>d:(AnyOpcodePattern [TRUNC])
+// CHECK-NEXT:     <root>d:(AnyOpcodePattern [G_TRUNC])
 // CHECK-NEXT:   )
 // CHECK-NEXT:   (ApplyPats
 // CHECK-NEXT:     __anon_pat_apply_0_0:(CXXPattern apply code:"APPLY")
 // CHECK-NEXT:   )
 // CHECK-NEXT:   (OperandTable <empty>)
 // CHECK-NEXT: )
 def WipOpcodeTest0 : GICombineRule<
   (defs root:$d),
-  (match (wip_match_opcode TRUNC):$d),
+  (match (wip_match_opcode G_TRUNC):$d),
   (apply [{ APPLY }])>;
 
 // CHECK:     (CombineRule name:WipOpcodeTest1 id:1 root:d
-// CHECK-NEXT:   (MatchDatas <empty>)
 // CHECK-NEXT:   (MatchPats
-// CHECK-NEXT:     <root>d:(AnyOpcodePattern [TRUNC, SEXT])
+// CHECK-NEXT:     <root>d:(AnyOpcodePattern [G_TRUNC, G_SEXT])
 // CHECK-NEXT:   )
 // CHECK-NEXT:   (ApplyPats
 // CHECK-NEXT:     __anon_pat_apply_1_0:(CXXPattern apply code:"APPLY")
 // CHECK-NEXT:   )
 // CHECK-NEXT:   (OperandTable <empty>)
 // CHECK-NEXT: )
 def WipOpcodeTest1 : GICombineRule<
   (defs root:$d),
-  (match (wip_match_opcode TRUNC, SEXT):$d),
+  (match (wip_match_opcode G_TRUNC, G_SEXT):$d),
   (apply [{ APPLY }])>;
 
 // CHECK:     (CombineRule name:InstTest0 id:2 root:d
-// CHECK-NEXT:   (MatchDatas <empty>)
 // CHECK-NEXT:   (MatchPats
-// CHECK-NEXT:     <root>d:(InstructionPattern inst:MOV operands:[<def>a, b])
+// CHECK-NEXT:     <root>d:(CodeGenInstructionPattern COPY operands:[<def>$a, $b])
 // CHECK-NEXT:   )
 // CHECK-NEXT:   (ApplyPats
 // CHECK-NEXT:     __anon_pat_apply_2_0:(CXXPattern apply code:"APPLY")
 // CHECK-NEXT:   )
 // CHECK-NEXT:   (OperandTable
 // CHECK-NEXT:     [a match_pat:d]
-// CHECK-NEXT:     [b live-in]
+// CHECK-NEXT:     [b match-live-in]
 // CHECK-NEXT:   )
 // CHECK-NEXT: )
 def InstTest0 : GICombineRule<
   (defs root:$d),
-  (match (MOV $a, $b):$d),
+  (match (COPY $a, $b):$d),
   (apply [{ APPLY }])>;
 
 // CHECK:      (CombineRule name:InstTest1 id:3 root:d
 // CHECK-NEXT:   (MatchDatas
 // CHECK-NEXT:      (MatchDataInfo pattern_symbol:r0 type:'Register' var_name:MDInfo0)
 // CHECK-NEXT:   )
 // CHECK-NEXT:   (MatchPats
-// CHECK-NEXT:     <root>d:(InstructionPattern inst:MOV operands:[<def>a, b])
-// CHECK-NEXT:     __anon_pat_match_3_0:(InstructionPattern inst:ZEXT operands:[<def>x, a])
+// CHECK-NEXT:     <root>d:(CodeGenInstructionPattern COPY operands:[<def>$a, i32:$b])
+// CHECK-NEXT:     __anon_pat_match_3_0:(CodeGenInstructionPattern G_ZEXT operands:[<def>$x, 0])
 // CHECK-NEXT:   )
 // CHECK-NEXT:   (ApplyPats
 // CHECK-NEXT:     __anon_pat_apply_3_1:(CXXPattern apply code:"APPLY")
 // CHECK-NEXT:   )
 // CHECK-NEXT:   (OperandTable
 // CHECK-NEXT:     [a match_pat:d]
-// CHECK-NEXT:     [b live-in]
+// CHECK-NEXT:     [b match-live-in]
 // CHECK-NEXT:     [x match_pat:__anon_pat_match_3_0]
 // CHECK-NEXT:   )
 // CHECK-NEXT: )
 let Predicates = [HasAnswerToEverything] in
 def InstTest1 : GICombineRule<
   (defs root:$d, reg_matchinfo:$r0),
-  (match (MOV $a, $b):$d,
-         (ZEXT $x, $a)),
+  (match (COPY $a, i32:$b):$d,
+         (G_ZEXT $x, 0)),
   (apply [{ APPLY }])>;
 
+// CHECK:      (CombineRule name:InstTest2 id:4 root:d
+// CHECK-NEXT:   (MatchDatas
+// CHECK-NEXT:     (MatchDataInfo pattern_symbol:r0 type:'Register' var_name:MDInfo0)
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (MatchPats
+// CHECK-NEXT:     <root>__anon_pat_match_4_0:(CodeGenInstructionPattern COPY operands:[<def>$d, (i32 0):$x])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (ApplyPats
+// CHECK-NEXT:     __anon_pat_apply_4_1:(CXXPattern apply code:"APPLY")
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (OperandTable
+// CHECK-NEXT:     [d match_pat:__anon_pat_match_4_0]
+// CHECK-NEXT:     [x match-live-in]
+// CHECK-NEXT:   )
+// CHECK-NEXT: )
+def InstTest2 : GICombineRule<
+  (defs root:$d, reg_matchinfo:$r0),
+  (match (COPY $d, (i32 0):$x)),
+  (apply [{ APPLY }])>;
+
+// CHECK:      (CombineRule name:InOutInstTest0 id:5 root:dst
+// CHECK-NEXT:   (MatchPats
+// CHECK-NEXT:     <root>__anon_pat_match_5_0:(CodeGenInstructionPattern COPY operands:[<def>$dst, $tmp])
+// CHECK-NEXT:     __anon_pat_match_5_1:(CodeGenInstructionPattern G_ZEXT operands:[<def>$tmp, $src])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (ApplyPats
+// CHECK-NEXT:     __anon_pat_apply_5_2:(CodeGenInstructionPattern G_TRUNC operands:[<def>$dst, $src])
+// CHECK-NEXT:     __anon_pat_apply_5_3:(CXXPattern apply code:"APPLY ${src}")
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (OperandTable
+// CHECK-NEXT:     [dst match_pat:__anon_pat_match_5_0 apply_pat:__anon_pat_apply_5_2]
+// CHECK-NEXT:     [tmp match_pat:__anon_pat_match_5_1]
+// CHECK-NEXT:     [src match-live-in]
+// CHECK-NEXT:   )
+// CHECK-NEXT: )
+def InOutInstTest0 : GICombineRule<
+  (defs root:$dst),
+  (match (COPY $dst, $tmp),
+         (G_ZEXT $tmp, $src)),
+  (apply (G_TRUNC $dst, $src), "APPLY ${src}")>;
+
+def MatchICst: GICombinePatFrag<
+    (outs),
+    (ins gi_mo:$foo, gi_imm:$cst),
+    [(pattern "return matchIConstant(${foo}, ${cst})")]>;
+
+// CHECK:      (CombineRule name:PatFragTest0 id:6 root:dst
+// CHECK-NEXT:   (PatFrags
+// CHECK-NEXT:     (PatFrag name:MatchICst
+// CHECK-NEXT:       (ins [foo:machine_operand, cst:imm])
+// CHECK-NEXT:       (alternatives [
+// CHECK-NEXT:         [
+// CHECK-NEXT:           (CXXPattern name:__anon_pat_pattern_6_2 match code:"return matchIConstant(${foo}, ${cst})"),
+// CHECK-NEXT:         ],
+// CHECK-NEXT:       ])
+// CHECK-NEXT:     )
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (MatchPats
+// CHECK-NEXT:     <root>__anon_pat_match_6_0:(CodeGenInstructionPattern G_ZEXT operands:[<def>$dst, $cst])
+// CHECK-NEXT:     __anon_pat_match_6_1:(PatFragPattern MatchICst operands:[$cst, (i32 0)])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (ApplyPats
+// CHECK-NEXT:     __anon_pat_apply_6_3:(CodeGenInstructionPattern COPY operands:[<def>$dst, (i32 0)])
+// CHECK-NEXT:     __anon_pat_apply_6_4:(CXXPattern apply code:"APPLY ${src}")
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (OperandTable
+// CHECK-NEXT:     [dst match_pat:__anon_pat_match_6_0 apply_pat:__anon_pat_apply_6_3]
+// CHECK-NEXT:     [cst match-live-in]
+// CHECK-NEXT:   )
+// CHECK-NEXT: )
+def PatFragTest0 : GICombineRule<
+  (defs root:$dst),
+  (match (G_ZEXT $dst, $cst), (MatchICst $cst, (i32 0))),
+  (apply (COPY $dst, (i32 0)), "APPLY ${src}")>;
+
+def MatchFooPerms: GICombinePatFrag<
+    (outs),
+    (ins gi_mo:$foo, gi_imm:$cst),
+    [
+      (pattern "return foo(${foo}, ${cst})"),
+      (pattern "return bar(${foo}, ${cst})"),
+      (pattern "return bux(${foo}, ${cst})"),
+    ]>;
+
+// CHECK:      (CombineRule name:PatFragTest1 id:7 root:dst
+// CHECK-NEXT:   (PatFrags
+// CHECK-NEXT:     (PatFrag name:MatchFooPerms
+// CHECK-NEXT:       (ins [foo:machine_operand, cst:imm])
+// CHECK-NEXT:       (alternatives [
+// CHECK-NEXT:         [
+// CHECK-NEXT:           (CXXPattern name:__anon_pat_pattern_7_1 match code:"return foo(${foo}, ${cst})"),
+// CHECK-NEXT:         ],
+// CHECK-NEXT:         [
+// CHECK-NEXT:           (CXXPattern name:__anon_pat_pattern_7_2 match code:"return bar(${foo}, ${cst})"),
+// CHECK-NEXT:         ],
+// CHECK-NEXT:         [
+// CHECK-NEXT:           (CXXPattern name:__anon_pat_pattern_7_3 match code:"return bux(${foo}, ${cst})"),
+// CHECK-NEXT:         ],
+// CHECK-NEXT:       ])
+// CHECK-NEXT:     )
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (MatchPats
+// CHECK-NEXT:     <root>__anon_pat_match_7_0:(CodeGenInstructionPattern G_ZEXT operands:[<def>$dst, $cst])
+// CHECK-NEXT:     a:(PatFragPattern MatchFooPerms operands:[$cst, (i32 0)])
+// CHECK-NEXT:     b:(PatFragPattern MatchFooPerms operands:[$cst, (i32 0)])
+// CHECK-NEXT:     c:(PatFragPattern MatchFooPerms operands:[$cst, (i32 0)])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (ApplyPats
+// CHECK-NEXT:     __anon_pat_apply_7_4:(CodeGenInstructionPattern COPY operands:[<def>$dst, (i32 0)])
+// CHECK-NEXT:     __anon_pat_apply_7_5:(CXXPattern apply code:"APPLY ${src}")
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (OperandTable
+// CHECK-NEXT:     [dst match_pat:__anon_pat_match_7_0 apply_pat:__anon_pat_apply_7_4]
+// CHECK-NEXT:     [cst match-live-in]
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (PermutationsToEmit
+// CHECK-NEXT:     [a[0], b[0], c[0]],
+// CHECK-NEXT:     [a[0], b[0], c[1]],
+// CHECK-NEXT:     [a[0], b[0], c[2]],
+// CHECK-NEXT:     [a[0], b[1], c[0]],
+// CHECK-NEXT:     [a[0], b[1], c[1]],
+// CHECK-NEXT:     [a[0], b[1], c[2]],
+// CHECK-NEXT:     [a[0], b[2], c[0]],
+// CHECK-NEXT:     [a[0], b[2], c[1]],
+// CHECK-NEXT:     [a[0], b[2], c[2]],
+// CHECK-NEXT:     [a[1], b[0], c[0]],
+// CHECK-NEXT:     [a[1], b[0], c[1]],
+// CHECK-NEXT:     [a[1], b[0], c[2]],
+// CHECK-NEXT:     [a[1], b[1], c[0]],
+// CHECK-NEXT:     [a[1], b[1], c[1]],
+// CHECK-NEXT:     [a[1], b[1], c[2]],
+// CHECK-NEXT:     [a[1], b[2], c[0]],
+// CHECK-NEXT:     [a[1], b[2], c[1]],
+// CHECK-NEXT:     [a[1], b[2], c[2]],
+// CHECK-NEXT:     [a[2], b[0], c[0]],
+// CHECK-NEXT:     [a[2], b[0], c[1]],
+// CHECK-NEXT:     [a[2], b[0], c[2]],
+// CHECK-NEXT:     [a[2], b[1], c[0]],
+// CHECK-NEXT:     [a[2], b[1], c[1]],
+// CHECK-NEXT:     [a[2], b[1], c[2]],
+// CHECK-NEXT:     [a[2], b[2], c[0]],
+// CHECK-NEXT:     [a[2], b[2], c[1]],
+// CHECK-NEXT:     [a[2], b[2], c[2]],
+// CHECK-NEXT:   )
+// CHECK-NEXT: )
+let MaxPermutations = -1 in
+def PatFragTest1 : GICombineRule<
+  (defs root:$dst),
+  (match (G_ZEXT $dst, $cst),
+         (MatchFooPerms $cst, (i32 0)):$a,
+         (MatchFooPerms $cst, (i32 0)):$b,
+         (MatchFooPerms $cst, (i32 0)):$c
+  ),
+  (apply (COPY $dst, (i32 0)), "APPLY ${src}")>;
+
+// CHECK:     (CombineRule name:VariadicsInTest id:8 root:dst
+// CHECK-NEXT:   (MatchPats
+// CHECK-NEXT:     <root>__anon_pat_match_8_0:(CodeGenInstructionPattern G_BUILD_VECTOR operands:[<def>$dst, $a, $b])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (ApplyPats
+// CHECK-NEXT:     __anon_pat_apply_8_1:(CodeGenInstructionPattern COPY operands:[<def>$dst, (i32 0)])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (OperandTable
+// CHECK-NEXT:     [dst match_pat:__anon_pat_match_8_0 apply_pat:__anon_pat_apply_8_1]
+// CHECK-NEXT:     [a match-live-in]
+// CHECK-NEXT:     [b match-live-in]
+// CHECK-NEXT:   )
+// CHECK-NEXT: )
+def VariadicsInTest : GICombineRule<
+  (defs root:$dst),
+  (match (G_BUILD_VECTOR $dst, $a, $b)),
+  (apply (COPY $dst, (i32 0)))>;
+
+// CHECK:      (CombineRule name:VariadicsOutTest id:9 root:a
+// CHECK-NEXT:   (MatchPats
+// CHECK-NEXT:     <root>__anon_pat_match_9_0:(CodeGenInstructionPattern G_UNMERGE_VALUES operands:[<def>$a, <def>$b, $src])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (ApplyPats
+// CHECK-NEXT:     __anon_pat_apply_9_1:(CodeGenInstructionPattern COPY operands:[<def>$a, (i32 0)])
+// CHECK-NEXT:     __anon_pat_apply_9_2:(CodeGenInstructionPattern COPY operands:[<def>$b, (i32 0)])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (OperandTable
+// CHECK-NEXT:     [a match_pat:__anon_pat_match_9_0 apply_pat:__anon_pat_apply_9_1]
+// CHECK-NEXT:     [b match_pat:__anon_pat_match_9_0 apply_pat:__anon_pat_apply_9_2]
+// CHECK-NEXT:     [src match-live-in]
+// CHECK-NEXT:   )
+// CHECK-NEXT: )
+def VariadicsOutTest : GICombineRule<
+  (defs root:$a),
+  (match (G_UNMERGE_VALUES $a, $b, $src)),
+  (apply (COPY $a, (i32 0)),
+         (COPY $b, (i32 0)))>;
+
 def MyCombiner: GICombinerHelper<"GenMyCombiner", [
   WipOpcodeTest0,
   WipOpcodeTest1,
   InstTest0,
-  InstTest1
+  InstTest1,
+  InstTest2,
+  InOutInstTest0,
+  PatFragTest0,
+  PatFragTest1,
+  VariadicsInTest,
+  VariadicsOutTest
 ]>;

llvm/test/TableGen/GlobalISelEmitter.td

 // CHECK-NEXT:    static MyTargetInstructionSelector::ComplexMatcherMemFn ComplexPredicateFns[];
 // CHECK-NEXT:    static MyTargetInstructionSelector::CustomRendererFn CustomRenderers[];
 // CHECK-NEXT:    bool testImmPredicate_I64(unsigned PredicateID, int64_t Imm) const override;
 // CHECK-NEXT:    bool testImmPredicate_APInt(unsigned PredicateID, const APInt &Imm) const override;
 // CHECK-NEXT:    bool testImmPredicate_APFloat(unsigned PredicateID, const APFloat &Imm) const override;
 // CHECK-NEXT:    const int64_t *getMatchTable() const override;
 // CHECK-NEXT:    bool testMIPredicate_MI(unsigned PredicateID, const MachineInstr &MI, const MatcherState &State) const override;
 // CHECK-NEXT:    bool testSimplePredicate(unsigned PredicateID) const override;
-// CHECK-NEXT:    void runCustomAction(unsigned FnID, const MatcherState &State) const override;
+// CHECK-NEXT:    void runCustomAction(unsigned FnID, const MatcherState &State, NewMIVector &OutMIs) const override;
 // CHECK-NEXT:  #endif // ifdef GET_GLOBALISEL_TEMPORARIES_DECL
 
 // CHECK-LABEL: #ifdef GET_GLOBALISEL_TEMPORARIES_INIT
 // CHECK-NEXT:    , State(3),
 // CHECK-NEXT:    ExecInfo(TypeObjects, NumTypeObjects, FeatureBitsets, ComplexPredicateFns, CustomRenderers)
 // CHECK-NEXT:  #endif // ifdef GET_GLOBALISEL_TEMPORARIES_INIT
 
 // CHECK-LABEL: // LLT Objects.

llvm/utils/TableGen/CodeGenTarget.h

 class CodeGenRegBank;
 class CodeGenRegister;
 class CodeGenRegisterClass;
 class CodeGenSchedModels;
 class CodeGenSubRegIndex;
 
 /// getValueType - Return the MVT::SimpleValueType that the specified TableGen
 /// record corresponds to.
-MVT::SimpleValueType getValueType(Record *Rec);
+MVT::SimpleValueType getValueType(const Record *Rec);
 
 StringRef getName(MVT::SimpleValueType T);
 StringRef getEnumName(MVT::SimpleValueType T);
 
 /// getQualifiedName - Return the name of the specified record, with a
 /// namespace qualifier if the record contains one.
 std::string getQualifiedName(const Record *R);
 

llvm/utils/TableGen/CodeGenTarget.cpp

 
 static cl::opt<unsigned>
     AsmWriterNum("asmwriternum", cl::init(0),
                  cl::desc("Make -gen-asm-writer emit assembly writer #N"),
                  cl::cat(AsmWriterCat));
 
 /// getValueType - Return the MVT::SimpleValueType that the specified TableGen
 /// record corresponds to.
-MVT::SimpleValueType llvm::getValueType(Record *Rec) {
+MVT::SimpleValueType llvm::getValueType(const Record *Rec) {
   return (MVT::SimpleValueType)Rec->getValueAsInt("Value");
 }
 
 StringRef llvm::getName(MVT::SimpleValueType T) {
   switch (T) {
   case MVT::Other:   return "UNKNOWN";
   case MVT::iPTR:    return "TLI.getPointerTy()";
   case MVT::iPTRAny: return "TLI.getPointerTy()";

llvm/utils/TableGen/GlobalISel/CodeExpansions.h

 
 public:
   void declare(StringRef Name, StringRef Expansion) {
     // Duplicates are not inserted. The expansion refers to different
     // MachineOperands using the same virtual register.
     Expansions.try_emplace(Name, Expansion);
   }
 
+  void redeclare(StringRef Name, StringRef Expansion) {
+    Expansions[Name] = Expansion;
+  }
+
   std::string lookup(StringRef Variable) const {
     return Expansions.lookup(Variable);
   }
 
   const_iterator begin() const { return Expansions.begin(); }
   const_iterator end() const { return Expansions.end(); }
   const_iterator find(StringRef Variable) const {
     return Expansions.find(Variable);
   }
 };
 } // end namespace llvm
 #endif // ifndef LLVM_UTILS_TABLEGEN_CODEEXPANSIONS_H

llvm/utils/TableGen/GlobalISel/CombinerUtils.h

 
 /// A convenience function to check that an Init refers to a dag whose operator
 /// is a def that is a subclass of the given class and coerce it to a dag if it
 /// is. This is primarily useful for testing for subclasses of GIMatchKind and
 /// similar in DagInit's since DagInit's support any type inside them.
 inline const DagInit *getDagWithOperatorOfSubClass(const Init &N,
                                                    StringRef Cls) {
   if (const DagInit *I = dyn_cast<DagInit>(&N))
-    if (I->getNumArgs() > 0)
-      if (const DefInit *OpI = dyn_cast<DefInit>(I->getOperator()))
-        if (OpI->getDef()->isSubClassOf(Cls))
-          return I;
   return nullptr;
 }
 } // namespace llvm
 
 #endif

llvm/utils/TableGen/GlobalISelCombinerMatchTableEmitter.cpp

 //===- GlobalISelCombinerMatchTableEmitter.cpp - --------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 /// \file Generate a combiner implementation for GlobalISel from a declarative
 /// syntax using GlobalISelMatchTable.
 ///
+/// Usually, TableGen backends use "assert is an error" as a means to report
+/// invalid input. They try to diagnose common case but don't try very hard and
+/// crashes can be common. This backend aims to behave closer to how a language
+/// compiler frontend would behave: we try extra hard to diagnose invalid inputs
+/// early, and any crash should be considered a bug (= a feature or diagnostic
+/// is missing).
+///
+/// While this can make the backend a bit more complex than it needs to be, it
+/// pays off because MIR patterns can get complicated. Giving useful error
+/// messages to combine writers can help boost their productivity.
+///
+/// As with anything, a good balance has to be found. We also don't want to
+/// write hundreds of lines of code to detect edge cases. In practice, crashing
+/// very occasionally, or giving poor errors in some rare instances, is fine.
+///
 //===----------------------------------------------------------------------===//
 
 #include "CodeGenInstruction.h"
 #include "CodeGenTarget.h"
 #include "GlobalISel/CodeExpander.h"
 #include "GlobalISel/CodeExpansions.h"
 #include "GlobalISel/CombinerUtils.h"
 #include "GlobalISelMatchTable.h"
 #include "GlobalISelMatchTableExecutorEmitter.h"
 #include "SubtargetFeatureInfo.h"
+#include "llvm/ADT/APInt.h"
 #include "llvm/ADT/Hashing.h"
 #include "llvm/ADT/MapVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/StringSet.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
+#include "llvm/Support/PrettyStackTrace.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include "llvm/TableGen/Error.h"
 #include "llvm/TableGen/Record.h"
 #include "llvm/TableGen/StringMatcher.h"
 #include "llvm/TableGen/TableGenBackend.h"
 #include <cstdint>
 
 using namespace llvm;
 using namespace llvm::gi;
 
 #define DEBUG_TYPE "gicombiner-matchtable-emitter"
 
 extern cl::list<std::string> SelectedCombiners;
 extern cl::opt<bool> StopAfterParse;
+extern cl::OptionCategory GICombinerEmitterCat;
 
+static cl::opt<bool> DebugCXXPreds(
+    "gicombiner-matchtable-debug-cxxpreds",
+    cl::desc("Add Contextual/Debug comments to all C++ predicates"),
+    cl::cat(GICombinerEmitterCat));
 namespace {
 constexpr StringLiteral CXXApplyPrefix = "GICXXCustomAction_CombineApply";
 constexpr StringLiteral CXXPredPrefix = "GICXXPred_MI_Predicate_";
+constexpr StringLiteral PatFragClassName = "GICombinePatFrag";
 
 std::string getIsEnabledPredicateEnumName(unsigned CombinerRuleID) {
   return "GICXXPred_Simple_IsRule" + to_string(CombinerRuleID) + "Enabled";
 }
 
+/// Copies a StringRef into a static pool to make sure it has a static lifetime.
+StringRef insertStrRef(StringRef S) {
+  if (S.empty())
+    return {};
+
+  static StringSet<> Pool;
+  auto [It, Inserted] = Pool.insert(S);
+  return It->getKey();
+}
+
 void declareInstExpansion(CodeExpansions &CE, const InstructionMatcher &IM,
                           StringRef Name) {
   CE.declare(Name, "State.MIs[" + to_string(IM.getInsnVarID()) + "]");
 }
 
+void declareInstExpansion(CodeExpansions &CE, const BuildMIAction &A,
+                          StringRef Name) {
+  // Note: we use redeclare here because this may overwrite a matcher inst
+  // expansion.
+  CE.redeclare(Name, "OutMIs[" + to_string(A.getInsnID()) + "]");
+}
+
 void declareOperandExpansion(CodeExpansions &CE, const OperandMatcher &OM,
                              StringRef Name) {
   CE.declare(Name, "State.MIs[" + to_string(OM.getInsnVarID()) +
                        "]->getOperand(" + to_string(OM.getOpIdx()) + ")");
 }
 
+void declareTempRegExpansion(CodeExpansions &CE, unsigned TempRegID,
+                             StringRef Name) {
+  CE.declare(Name, "State.TempRegisters[" + to_string(TempRegID) + "]");
+}
+
 template <typename Container> auto keys(Container &&C) {
   return map_range(C, [](auto &Entry) -> auto & { return Entry.first; });
 }
 
 template <typename Container> auto values(Container &&C) {
   return map_range(C, [](auto &Entry) -> auto & { return Entry.second; });
 }
 
+LLTCodeGen getLLTCodeGenFromRecord(const Record *Ty) {
+  assert(Ty->isSubClassOf("ValueType"));
+  return LLTCodeGen(*MVTToLLT(getValueType(Ty)));
+}
+
 //===- MatchData Handling -------------------------------------------------===//
 
 /// Represents MatchData defined by the match stage and required by the apply
 /// stage.
 ///
 /// This allows the plumbing of arbitrary data from C++ predicates between the
 /// stages.
 ///
if (NumSeen == ExistingVars.size())
       ExistingVars.push_back("MDInfo" + to_string(NextVarID++));
 
     I.setVariableName(ExistingVars[NumSeen++]);
   }
 }
 
 //===- C++ Predicates Handling --------------------------------------------===//
 
-/// Entry into the static pool of all CXX Predicate code. This contains the
+/// Entry into the static pool of all CXX Predicate code. This contains
 /// fully expanded C++ code.
 ///
-/// Each CXXPattern creates a new entry in the pool to store its data, even
-/// after the pattern is destroyed.
+/// The static pool is hidden inside the object and can be accessed through
+/// getAllMatchCode/getAllApplyCode
 ///
 /// Note that CXXPattern trims C++ code, so the Code is already expected to be
 /// free of leading/trailing whitespace.
-struct CXXPredicateCode {
-  CXXPredicateCode(std::string Code, unsigned ID)
-      : Code(Code), ID(ID), BaseEnumName("GICombiner" + to_string(ID)) {
-    assert(StringRef(Code).trim() == Code &&
-           "Code was expected to be trimmed!");
-  }
-
-  const std::string Code;
-  const unsigned ID;
-  const std::string BaseEnumName;
-
-  bool needsUnreachable() const {
-    return !StringRef(Code).starts_with("return");
-  }
-
-  std::string getEnumNameWithPrefix(StringRef Prefix) const {
-    return Prefix.str() + BaseEnumName;
-  }
-};
-
+class CXXPredicateCode {
   using CXXPredicateCodePool =
       DenseMap<hash_code, std::unique_ptr<CXXPredicateCode>>;
-CXXPredicateCodePool AllCXXMatchCode;
-CXXPredicateCodePool AllCXXApplyCode;
+  static CXXPredicateCodePool AllCXXMatchCode;
+  static CXXPredicateCodePool AllCXXApplyCode;
+
+  /// Sorts a `CXXPredicateCodePool` by their IDs and returns it.
+  static std::vector<const CXXPredicateCode *>
+  getSorted(const CXXPredicateCodePool &Pool) {
+    std::vector<const CXXPredicateCode *> Out;
+    std::transform(Pool.begin(), Pool.end(), std::back_inserter(Out),
+                   [&](auto &Elt) { return Elt.second.get(); });
+    sort(Out, [](const auto *A, const auto *B) { return A->ID < B->ID; });
+    return Out;
+  }
 
   /// Gets an instance of `CXXPredicateCode` for \p Code, or returns an already
   /// existing one.
-const CXXPredicateCode &getOrInsert(CXXPredicateCodePool &Pool,
+  static const CXXPredicateCode &get(CXXPredicateCodePool &Pool,
                                      std::string Code) {
     // Check if we already have an identical piece of code, if not, create an
     // entry in the pool.
     const auto CodeHash = hash_value(Code);
     if (auto It = Pool.find(CodeHash); It != Pool.end())
       return *It->second;
 
     const auto ID = Pool.size();
-  auto OwnedData = std::make_unique<CXXPredicateCode>(std::move(Code), ID);
+    auto OwnedData = std::unique_ptr<CXXPredicateCode>(
+        new CXXPredicateCode(std::move(Code), ID));
     const auto &DataRef = *OwnedData;
     Pool[CodeHash] = std::move(OwnedData);
     return DataRef;
   }
 
-/// Sorts a `CXXPredicateCodePool` by their IDs and returns it.
-std::vector<const CXXPredicateCode *>
-getSorted(const CXXPredicateCodePool &Pool) {
-  std::vector<const CXXPredicateCode *> Out;
-  std::transform(Pool.begin(), Pool.end(), std::back_inserter(Out),
-                 [&](auto &Elt) { return Elt.second.get(); });
-  sort(Out, [](const auto *A, const auto *B) { return A->ID < B->ID; });
-  return Out;
+  CXXPredicateCode(std::string Code, unsigned ID)
+      : Code(Code), ID(ID), BaseEnumName("GICombiner" + to_string(ID)) {
+    // Don't assert if ErrorsPrinted is set. This may mean CodeExpander failed,
+    // and it may add spaces in such cases.
+    assert(ErrorsPrinted || StringRef(Code).trim() == Code &&
+                                "Code was expected to be trimmed!");
+  }
+
+public:
+  static const CXXPredicateCode &getMatchCode(std::string Code) {
+    return get(AllCXXMatchCode, std::move(Code));
+  }
+
+  static const CXXPredicateCode &getApplyCode(std::string Code) {
+    return get(AllCXXApplyCode, std::move(Code));
+  }
+
+  static std::vector<const CXXPredicateCode *> getAllMatchCode() {
+    return getSorted(AllCXXMatchCode);
+  }
+
+  static std::vector<const CXXPredicateCode *> getAllApplyCode() {
+    return getSorted(AllCXXApplyCode);
+  }
+
+  const std::string Code;
+  const unsigned ID;
+  const std::string BaseEnumName;
+
+  bool needsUnreachable() const {
+    return !StringRef(Code).starts_with("return");
   }
 
+  std::string getEnumNameWithPrefix(StringRef Prefix) const {
+    return Prefix.str() + BaseEnumName;
+  }
+};
+
+CXXPredicateCode::CXXPredicateCodePool CXXPredicateCode::AllCXXMatchCode;
+CXXPredicateCode::CXXPredicateCodePool CXXPredicateCode::AllCXXApplyCode;
+
 //===- Pattern Base Class -------------------------------------------------===//
 
-// An abstract pattern found in a combine rule. This can be an apply or match
-// pattern.
+/// Base class for all patterns that can be written in an `apply`, `match` or
+/// `pattern` DAG operator.
+///
+/// For example:
+///
+///     (apply (G_ZEXT $x, $y), (G_ZEXT $y, $z), "return isFoo(${z})")
+///
+/// Creates 3 Pattern objects:
+///   - Two CodeGenInstruction Patterns
+///   - A CXXPattern
 class Pattern {
 public:
   enum {
     K_AnyOpcode,
-    K_Inst,
     K_CXX,
+
+    K_CodeGenInstruction,
+    K_PatFrag,
   };
 
   virtual ~Pattern() = default;
 
   unsigned getKind() const { return Kind; }
   const char *getKindName() const;
 
   bool hasName() const { return !Name.empty(); }
   StringRef getName() const { return Name; }
 
   virtual void print(raw_ostream &OS, bool PrintName = true) const = 0;
   void dump() const { return print(dbgs()); }
 
 protected:
-  Pattern(unsigned Kind, StringRef Name) : Kind(Kind), Name(Name.str()) {
+  Pattern(unsigned Kind, StringRef Name)
+      : Kind(Kind), Name(insertStrRef(Name)) {
     assert(!Name.empty() && "unnamed pattern!");
   }
 
   void printImpl(raw_ostream &OS, bool PrintName,
                  function_ref<void()> ContentPrinter) const;
 
 private:
   unsigned Kind;
-
-  // Note: if this ever changes to a StringRef (e.g. allocated in a pool or
-  // something), CombineRuleBuilder::verify() needs to be updated as well.
-  // It currently checks that the StringRef in the PatternMap references this.
-  std::string Name;
+  StringRef Name;
 };
 
 const char *Pattern::getKindName() const {
   switch (Kind) {
   case K_AnyOpcode:
     return "AnyOpcodePattern";
-  case K_Inst:
-    return "InstructionPattern";
   case K_CXX:
     return "CXXPattern";
+  case K_CodeGenInstruction:
+    return "CodeGenInstructionPattern";
+  case K_PatFrag:
+    return "PatFragPattern";
   }
 
   llvm_unreachable("unknown pattern kind!");
 }
 
 void Pattern::printImpl(raw_ostream &OS, bool PrintName,
                         function_ref<void()> ContentPrinter) const {
   OS << "(" << getKindName() << " ";
   if (PrintName)
     OS << "name:" << getName() << " ";
   ContentPrinter();
   OS << ")";
 }
 
 //===- AnyOpcodePattern ---------------------------------------------------===//
 
 /// `wip_match_opcode` patterns.
 /// This matches one or more opcodes, and does not check any operands
 /// whatsoever.
+///
+/// TODO: Long-term, this needs to be removed. It's a hack around MIR
+///       pattern matching limitations.
 class AnyOpcodePattern : public Pattern {
 public:
   AnyOpcodePattern(StringRef Name) : Pattern(K_AnyOpcode, Name) {}
 
   static bool classof(const Pattern *P) { return P->getKind() == K_AnyOpcode; }
 
   void addOpcode(const CodeGenInstruction *I) { Insts.push_back(I); }
   const auto &insts() const { return Insts; }
printImpl(OS, PrintName, [&OS, this]() {
     OS << "["
        << join(map_range(Insts,
                          [](const auto *I) { return I->TheDef->getName(); }),
                ", ")
        << "]";
   });
 }
 
-//===- InstructionPattern -------------------------------------------------===//
-
-/// Matches an instruction, e.g. `G_ADD $x, $y, $z`.
-///
-/// This pattern is simply CodeGenInstruction + a list of operands.
-class InstructionPattern : public Pattern {
-public:
-  struct Operand {
-    std::string Name;
-    bool IsDef = false;
-  };
-
-  InstructionPattern(const CodeGenInstruction &I, StringRef Name)
-      : Pattern(K_Inst, Name), I(I) {}
-
-  static bool classof(const Pattern *P) { return P->getKind() == K_Inst; }
-
-  const auto &operands() const { return Operands; }
-  void addOperand(StringRef Name);
-  unsigned getNumDefs() const { return I.Operands.NumDefs; }
-
-  const CodeGenInstruction &getInst() const { return I; }
-  StringRef getInstName() const { return I.TheDef->getName(); }
-
-  void reportUnreachable(ArrayRef<SMLoc> Locs) const;
-  bool checkSemantics(ArrayRef<SMLoc> Loc) const;
-
-  void print(raw_ostream &OS, bool PrintName = true) const override;
-
-private:
-  const CodeGenInstruction &I;
-  SmallVector<Operand, 4> Operands;
-};
-
-void InstructionPattern::addOperand(StringRef Name) {
-  const bool IsDef = Operands.size() < getNumDefs();
-  Operands.emplace_back(Operand{Name.str(), IsDef});
-}
-
-void InstructionPattern::reportUnreachable(ArrayRef<SMLoc> Locs) const {
-  PrintError(Locs, "Instruction pattern '" + getName() +
-                       "' is unreachable from the pattern root!");
-}
-
-bool InstructionPattern::checkSemantics(ArrayRef<SMLoc> Loc) const {
-  unsigned NumExpectedOperands = I.Operands.size();
-  if (NumExpectedOperands != Operands.size()) {
-
-    PrintError(Loc, "'" + getInstName() + "' expected " +
-                        Twine(NumExpectedOperands) + " operands, got " +
-                        Twine(Operands.size()));
-    return false;
-  }
-  return true;
-}
-
-void InstructionPattern::print(raw_ostream &OS, bool PrintName) const {
-  printImpl(OS, PrintName, [&OS, this]() {
-    OS << "inst:" << I.TheDef->getName() << " operands:["
-       << join(map_range(Operands,
-                         [](const auto &O) {
-                           return (O.IsDef ? "<def>" : "") + O.Name;
-                         }),
-               ", ")
-       << "]";
-  });
-}
-
 //===- CXXPattern ---------------------------------------------------------===//
 
-/// Raw C++ code which may need some expansions.
+/// Represents raw C++ code which may need some expansions.
 ///
 ///   e.g. [{ return isFooBux(${src}.getReg()); }]
 ///
 /// For the expanded code, \see CXXPredicateCode. CXXPredicateCode objects are
 /// created through `expandCode`.
 ///
 /// \see CodeExpander and \see CodeExpansions for more information on code
 /// expansions.
 ///
 /// This object has two purposes:
 ///   - Represent C++ code as a pattern entry.
 ///   - Be a factory for expanded C++ code.
 ///     - It's immutable and only holds the raw code so we can expand the same
 ///       CXX pattern multiple times if we need to.
 ///
 /// Note that the code is always trimmed in the constructor, so leading and
 /// trailing whitespaces are removed. This removes bloat in the output, avoids
 /// formatting issues, but also allows us to check things like
 /// `.startswith("return")` trivially without worrying about spaces.
 class CXXPattern : public Pattern {
 public:
-  CXXPattern(const StringInit &Code, StringRef Name, bool IsApply)
-      : CXXPattern(Code.getAsUnquotedString(), Name, IsApply) {}
+  CXXPattern(const StringInit &Code, StringRef Name)
+      : CXXPattern(Code.getAsUnquotedString(), Name) {}
 
-  CXXPattern(StringRef Code, StringRef Name, bool IsApply)
-      : Pattern(K_CXX, Name), IsApply(IsApply), RawCode(Code.trim().str()) {}
+  CXXPattern(StringRef Code, StringRef Name)
+      : Pattern(K_CXX, Name), RawCode(Code.trim().str()) {}
 
   static bool classof(const Pattern *P) { return P->getKind() == K_CXX; }
 
-  bool isApply() const { return IsApply; }
+  void setIsApply(bool Value = true) { IsApply = Value; }
   StringRef getRawCode() const { return RawCode; }
 
   /// Expands raw code, replacing things such as `${foo}` with their
   /// substitution in \p CE.
   ///
   /// \param CE     Map of Code Expansions
   /// \param Locs   SMLocs for the Code Expander, in case it needs to emit
   ///               diagnostics.
+  /// \param AddComment If DebugCXXPreds is enabled, this is called to emit a
+  ///                   comment before the expanded code.
+  ///
   /// \return A CXXPredicateCode object that contains the expanded code. Note
   /// that this may or may not insert a new object. All CXXPredicateCode objects
   /// are held in a set to avoid emitting duplicate C++ code.
-  const CXXPredicateCode &expandCode(const CodeExpansions &CE,
-                                     ArrayRef<SMLoc> Locs) const;
+  const CXXPredicateCode &
+  expandCode(const CodeExpansions &CE, ArrayRef<SMLoc> Locs,
+             function_ref<void(raw_ostream &)> AddComment = {}) const;
 
   void print(raw_ostream &OS, bool PrintName = true) const override;
 
 private:
-  bool IsApply;
+  bool IsApply = false;
   std::string RawCode;
 };
 
-const CXXPredicateCode &CXXPattern::expandCode(const CodeExpansions &CE,
-                                               ArrayRef<SMLoc> Locs) const {
+const CXXPredicateCode &
+CXXPattern::expandCode(const CodeExpansions &CE, ArrayRef<SMLoc> Locs,
+                       function_ref<void(raw_ostream &)> AddComment) const {
   std::string Result;
   raw_string_ostream OS(Result);
+
+  if (DebugCXXPreds && AddComment)
+    AddComment(OS);
+
   CodeExpander Expander(RawCode, CE, Locs, /*ShowExpansions*/ false);
   Expander.emit(OS);
-  return getOrInsert(IsApply ? AllCXXApplyCode : AllCXXMatchCode,
-                     std::move(Result));
+  if (IsApply)
+    return CXXPredicateCode::getApplyCode(std::move(Result));
+  return CXXPredicateCode::getMatchCode(std::move(Result));
 }
 
 void CXXPattern::print(raw_ostream &OS, bool PrintName) const {
   printImpl(OS, PrintName, [&OS, this] {
     OS << (IsApply ? "apply" : "match") << " code:\"";
     printEscapedString(getRawCode(), OS);
     OS << "\"";
   });
 }
 
-//===- CombineRuleBuilder -------------------------------------------------===//
+//===- InstructionPattern ---------------------------------------------===//
+
+/// Base class for CodeGenInstructionPattern & PatFragPattern, which handles all
+/// the boilerplate for patterns that have a list of operands for some (pseudo)
+/// instruction.
+class InstructionPattern : public Pattern {
+public:
+  class Operand;
+
+  virtual ~InstructionPattern();
+
+  static bool classof(const Pattern *P) {
+    return P->getKind() == K_CodeGenInstruction || P->getKind() == K_PatFrag;
+  }
+
+  template <typename... Ty> void addOperand(Ty &&...Init) {
+    Operands.emplace_back(std::forward<Ty>(Init)...);
+  }
+
+  auto &operands() { return Operands; }
+  const auto &operands() const { return Operands; }
+  unsigned operands_size() const { return Operands.size(); }
+  Operand &getOperand(unsigned K) { return Operands[K]; }
+  const Operand &getOperand(unsigned K) const { return Operands[K]; }
+
+  virtual bool isVariadic() const { return false; }
+  virtual unsigned getNumInstOperands() const = 0;
+  virtual unsigned getNumInstDefs() const = 0;
+
+  bool hasAllDefs() const { return operands_size() >= getNumInstDefs(); }
+
+  virtual StringRef getInstName() const = 0;
+
+  void reportUnreachable(ArrayRef<SMLoc> Locs) const;
+  virtual bool checkSemantics(ArrayRef<SMLoc> Loc);
+
+  void print(raw_ostream &OS, bool PrintName = true) const override;
 
-/// Helper for CombineRuleBuilder.
+protected:
+  InstructionPattern(unsigned K, StringRef Name) : Pattern(K, Name) {}
+
+  // std::vector is used here because we can instantiate it using a forward
+  // declaration.
+  std::vector<Operand> Operands;
+};
+
+/// An operand for an InstructionPattern.
 ///
+/// Operands are composed of three elements:
+///   - (Optional) Value
+///   - (Optional) Name
+///   - (Optional) Type
+///
+/// Some examples:
+///   (i32 0):$x -> V=int(0), Name='x', Type=i32
+///   0:$x -> V=int(0), Name='x'
+///   $x -> Name='x'
+///   i32:$x -> Name='x', Type = i32
+class InstructionPattern::Operand {
+public:
+  using IntImmTy = int64_t;
+
+  Operand(IntImmTy Imm, StringRef Name, const Record *Type)
+      : Value(Imm), Name(insertStrRef(Name)), Type(Type) {
+    assert(!Type || Type->isSubClassOf("ValueType"));
+  }
+
+  Operand(StringRef Name, const Record *Type)
+      : Name(insertStrRef(Name)), Type(Type) {}
+
+  bool isNamedImmediate() const { return hasImmValue() && isNamedOperand(); }
+
+  bool hasImmValue() const { return Value.has_value(); }
+  IntImmTy getImmValue() const { return *Value; }
+
+  bool isNamedOperand() const { return !Name.empty(); }
+  StringRef getOperandName() const {
+    assert(isNamedOperand() && "Operand is unnamed");
+    return Name;
+  }
+
+  Operand withNewName(StringRef NewName) const {
+    Operand Result = *this;
+    Result.Name = insertStrRef(NewName);
+    return Result;
+  }
+
+  void setIsDef(bool Value = true) { Def = Value; }
+  bool isDef() const { return Def; }
+
+  void setType(const Record *R) {
+    assert((!Type || (Type == R)) && "Overwriting type!");
+    Type = R;
+  }
+  const Record *getType() const { return Type; }
+
+  std::string describe() const {
+    if (!hasImmValue())
+      return "MachineOperand $" + getOperandName().str() + "";
+    std::string Str = "imm " + to_string(getImmValue());
+    if (isNamedImmediate())
+      Str += ":$" + getOperandName().str() + "";
+    return Str;
+  }
+
+  void print(raw_ostream &OS) const {
+    if (isDef())
+      OS << "<def>";
+
+    bool NeedsColon = true;
+    if (const Record *Ty = getType()) {
+      if (hasImmValue())
+        OS << "(" << Ty->getName() << " " << getImmValue() << ")";
+      else
+        OS << Ty->getName();
+    } else if (hasImmValue())
+      OS << getImmValue();
+    else
+      NeedsColon = false;
+
+    if (isNamedOperand())
+      OS << (NeedsColon ? ":" : "") << "$" << getOperandName();
+  }
+
+  void dump() const { return print(dbgs()); }
+
+private:
+  std::optional<int64_t> Value;
+  StringRef Name;
+  const Record *Type = nullptr;
+  bool Def = false;
+};
+
+InstructionPattern::~InstructionPattern() = default;
+
+void InstructionPattern::reportUnreachable(ArrayRef<SMLoc> Locs) const {
+  PrintError(Locs, "pattern '" + getName() + "' ('" + getInstName() +
+                       "') is unreachable from the pattern root!");
+}
+
+bool InstructionPattern::checkSemantics(ArrayRef<SMLoc> Loc) {
+  unsigned NumExpectedOperands = getNumInstOperands();
+
+  if (isVariadic()) {
+    if (Operands.size() < NumExpectedOperands) {
+      PrintError(Loc, +"'" + getInstName() + "' expected at least " +
+                          Twine(NumExpectedOperands) + " operands, got " +
+                          Twine(Operands.size()));
+      return false;
+    }
+  } else if (NumExpectedOperands != Operands.size()) {
+    PrintError(Loc, +"'" + getInstName() + "' expected " +
+                        Twine(NumExpectedOperands) + " operands, got " +
+                        Twine(Operands.size()));
+    return false;
+  }
+
+  unsigned OpIdx = 0;
+  unsigned NumDefs = getNumInstDefs();
+  for (auto &O : Operands)
+    O.setIsDef(OpIdx++ < NumDefs);
+
+  return true;
+}
+
+void InstructionPattern::print(raw_ostream &OS, bool PrintName) const {
+  printImpl(OS, PrintName, [&OS, this] {
+    OS << getInstName() << " operands:[";
+    StringRef Sep = "";
+    for (const auto &O : Operands) {
+      OS << Sep;
+      O.print(OS);
+      Sep = ", ";
+    }
+    OS << "]";
+  });
+}
+
+//===- CodeGenInstructionPattern ------------------------------------------===//
+
+/// Matches an instruction, e.g. `G_ADD $x, $y, $z`.
+class CodeGenInstructionPattern : public InstructionPattern {
+public:
+  CodeGenInstructionPattern(const CodeGenInstruction &I, StringRef Name)
+      : InstructionPattern(K_CodeGenInstruction, Name), I(I) {}
+
+  static bool classof(const Pattern *P) {
+    return P->getKind() == K_CodeGenInstruction;
+  }
+
+  bool hasVariadicDefs() const;
+  bool isVariadic() const override { return I.Operands.isVariadic; }
+  unsigned getNumInstDefs() const override;
+  unsigned getNumInstOperands() const override;
+
+  const CodeGenInstruction &getInst() const { return I; }
+  StringRef getInstName() const override { return I.TheDef->getName(); }
+
+private:
+  const CodeGenInstruction &I;
+};
+
+bool CodeGenInstructionPattern::hasVariadicDefs() const {
+  // Note: we cannot use variadicOpsAreDefs, it's not set for
+  // GenericInstructions.
+  if (!isVariadic())
+    return false;
+
+  if (I.variadicOpsAreDefs)
+    return true;
+
+  DagInit *OutOps = I.TheDef->getValueAsDag("OutOperandList");
+  if (OutOps->arg_empty())
+    return false;
+
+  auto *LastArgTy = dyn_cast<DefInit>(OutOps->getArg(OutOps->arg_size() - 1));
+  return LastArgTy && LastArgTy->getDef()->getName() == "variable_ops";
+}
+
+unsigned CodeGenInstructionPattern::getNumInstDefs() const {
+  if (!isVariadic() || !hasVariadicDefs())
+    return I.Operands.NumDefs;
+  unsigned NumOuts = I.Operands.size() - I.Operands.NumDefs;
+  assert(Operands.size() > NumOuts);
+  return std::max<unsigned>(I.Operands.NumDefs, Operands.size() - NumOuts);
+}
+
+unsigned CodeGenInstructionPattern::getNumInstOperands() const {
+  unsigned NumCGIOps = I.Operands.size();
+  return isVariadic() ? std::max<unsigned>(NumCGIOps, Operands.size())
+                      : NumCGIOps;
+}
+
+//===- OperandTable -------------------------------------------------------===//
+
 /// Represents information about an operand.
 /// Operands with no MatchPat are considered live-in to the pattern.
+/// 'live-in' means that we're not sure what defines this operand/what it is.
+/// We just know its name to refer to it.
+///
+/// e.g. (G_SEXT $x, $y) -> $y is a live-in. $y could be MBB ref, an immediate,
+/// a flying squirrel or just a register, we don't know. All we know is that
+/// when we refer to $y, we refer to that G_SEXT's second operand.
 struct OperandTableEntry {
   // The matcher pattern that defines this operand.
   // null for live-ins.
   InstructionPattern *MatchPat = nullptr;
   // The apply pattern that (re)defines this operand.
   // This can only be non-null if MatchPat is.
-  InstructionPattern *ApplyPat = nullptr;
+  CodeGenInstructionPattern *ApplyPat = nullptr;
+  // If MatchPat is null, whether this is a live-in of the match pattern.
+  bool IsMatchLiveIn = false;
+};
+
+/// The Operand Table holds information about the operands in a set of Patterns.
+/// This is only useful for Instruction patterns.
+///
+/// This is a simple Key -> Value map where the Key is an operand's name, and
+/// the Value is an OperandTableEntry, a small struct holding information about
+/// the operand.
+///
+/// MapVector is used to have a deterministic iteration order for dump-parse.
+class OperandTable : public MapVector<StringRef, OperandTableEntry> {
+public:
+  bool addMatchPattern(ArrayRef<SMLoc> DiagLoc, InstructionPattern *P,
+                       StringRef ErrSuffix);
+  bool addApplyPattern(ArrayRef<SMLoc> DiagLoc, CodeGenInstructionPattern *P);
+
+  const InstructionPattern *getMatchDefPattern(StringRef OpName) const {
+    if (auto It = find(OpName); It != end())
+      return It->second.MatchPat;
+    return nullptr;
+  }
+
+  const CodeGenInstructionPattern *getApplyDefPattern(StringRef OpName) const {
+    if (auto It = find(OpName); It != end())
+      return It->second.ApplyPat;
+    return nullptr;
+  }
+
+  bool isMatchLiveInOrDef(StringRef OpName) const;
+
+  void print(raw_ostream &OS, StringRef Indent = "") const;
+};
+
+bool OperandTable::addMatchPattern(ArrayRef<SMLoc> DiagLoc,
+                                   InstructionPattern *P, StringRef ErrSuffix) {
+  for (const auto &Operand : P->operands()) {
+    if (!Operand.isNamedOperand())
+      continue;
+
+    StringRef OpName = Operand.getOperandName();
+
+    // Create an entry, no matter if it's a use or a def.
+    auto &Entry = (*this)[OpName];
+
+    if (Operand.isDef()) {
+      if (Entry.MatchPat) {
+        PrintError(DiagLoc, "Operand '" + OpName +
+                                "' is defined multiple times " + ErrSuffix);
+        return false;
+      }
+      Entry.MatchPat = P;
+      Entry.IsMatchLiveIn = false;
+    } else if (!Entry.MatchPat) {
+      Entry.IsMatchLiveIn = true;
+    }
+  }
+
+  return true;
+}
+
+bool OperandTable::addApplyPattern(ArrayRef<SMLoc> DiagLoc,
+                                   CodeGenInstructionPattern *P) {
+  for (const auto &Operand : P->operands()) {
+    if (!Operand.isNamedOperand())
+      continue;
+
+    StringRef OpName = Operand.getOperandName();
+    // Create an entry, no matter if it's a use or a def.
+    auto &Entry = (*this)[OpName];
+
+    if (Operand.isNamedImmediate())
+      continue;
+
+    if (Operand.isDef()) {
+      if (!Entry.MatchPat && Entry.IsMatchLiveIn) {
+        PrintError(DiagLoc, "Cannot define live-in operand '" + OpName +
+                                "' in the 'apply' pattern");
+        return false;
+      }
+      if (Entry.ApplyPat) {
+        PrintError(DiagLoc,
+                   "Operand '" + OpName +
+                       "' is defined multiple times in the 'apply' patterns");
+        return false;
+      }
+      Entry.ApplyPat = P;
+    }
+  }
+
+  return true;
+}
+
+bool OperandTable::isMatchLiveInOrDef(StringRef OpName) const {
+  const auto It = find(OpName);
+  if (It == end())
+    return false;
+  const auto &Entry = It->second;
+  return Entry.MatchPat || Entry.IsMatchLiveIn;
+}
+
+void OperandTable::print(raw_ostream &OS, StringRef Indent) const {
+  OS << Indent << "(OperandTable ";
+  if (empty()) {
+    OS << "<empty>)\n";
+    return;
+  }
+
+  OS << "\n";
+  for (const auto &[Name, Val] : *this) {
+    OS << "    [" << Name;
+    if (const auto *P = Val.MatchPat)
+      OS << " match_pat:" << P->getName();
+    if (const auto *P = Val.ApplyPat)
+      OS << " apply_pat:" << P->getName();
+    if (Val.IsMatchLiveIn)
+      OS << " match-live-in";
+    OS << "]\n";
+  }
+  OS << "  )\n";
+}
+
+//===- OperandTypeChecker -------------------------------------------------===//
+
+/// This is a trivial type checker for all operands in a set of
+/// InstructionPatterns.
+///
+/// It infers the type of each operand, check it's consistent with the known
+/// type of the operand, and then sets all of the types in all operands in
+/// setAllOperandTypes.
+class OperandTypeChecker {
+public:
+  OperandTypeChecker(ArrayRef<SMLoc> DiagLoc) : DiagLoc(DiagLoc) {}
+
+  bool check(InstructionPattern *P);
+
+  void setAllOperandTypes();
+
+private:
+  struct OpTypeInfo {
+    const Record *Type = nullptr;
+    InstructionPattern *TypeSrc = nullptr;
+  };
+
+  ArrayRef<SMLoc> DiagLoc;
+  StringMap<OpTypeInfo> Types;
+
+  SmallVector<InstructionPattern *, 16> Pats;
+};
+
+bool OperandTypeChecker::check(InstructionPattern *P) {
+  Pats.push_back(P);
+
+  for (auto &O : P->operands()) {
+    const Record *Ty = O.getType();
+    if (!O.isNamedOperand() || !Ty)
+      continue;
+
+    auto &Info = Types[O.getOperandName()];
+
+    if (!Info.Type) {
+      Info.Type = Ty;
+      Info.TypeSrc = P;
+      continue;
+    }
+
+    if (Info.Type != Ty) {
+      PrintError(DiagLoc, "conflicting types for operand '" +
+                              O.getOperandName() + "': first seen with '" +
+                              Info.Type->getName() + "' in '" +
+                              Info.TypeSrc->getName() + ", now seen with '" +
+                              Ty->getName() + "' in '" + P->getName() + "'");
+      return false;
+    }
+  }
+
+  return true;
+}
+
+void OperandTypeChecker::setAllOperandTypes() {
+  for (auto *P : Pats) {
+    for (auto &O : P->operands()) {
+      if (!O.isNamedOperand())
+        continue;
+
+      if (auto &Info = Types[O.getOperandName()]; Info.Type)
+        O.setType(Info.Type);
+    }
+  }
+}
+
+//===- PatFrag ------------------------------------------------------------===//
+
+/// Represents a parsed GICombinePatFrag. This can be thought of as the
+/// equivalent of a CodeGenInstruction, but for PatFragPatterns.
+///
+/// PatFrags are made of 3 things:
+///   - Out parameters (defs)
+///   - In parameters
+///   - A set of pattern lists (alternatives).
+///
+/// If the PatFrag uses instruction patterns, the root must be one of the defs.
+///
+/// Note that this DOES NOT represent the use of the PatFrag, only its
+/// definition. The use of the PatFrag in a Pattern is represented by
+/// PatFragPattern.
+///
+/// PatFrags use the term "parameter" instead of operand because they're
+/// essentially macros, and using that name avoids confusion. Other than that,
+/// they're structured similarly to a MachineInstruction  - all parameters
+/// (operands) are in the same list, with defs at the start. This helps mapping
+/// parameters to values, because, param N of a PatFrag is always operand N of a
+/// PatFragPattern.
+class PatFrag {
+public:
+  enum ParamKind {
+    PK_Root,
+    PK_MachineOperand,
+    PK_Imm,
+  };
+
+  struct Param {
+    StringRef Name;
+    ParamKind Kind;
+  };
+
+  using ParamVec = SmallVector<Param, 4>;
+  using ParamIt = ParamVec::const_iterator;
+
+  /// Represents an alternative of the PatFrag. When parsing a GICombinePatFrag,
+  /// this is created from its "Alternatives" list. Each alternative is a list
+  /// of patterns written wrapped in a  `(pattern ...)` dag init.
+  ///
+  /// Each argument to the `pattern` DAG operator is parsed into a Pattern
+  /// instance.
+  struct Alternative {
+    OperandTable OpTable;
+    SmallVector<std::unique_ptr<Pattern>, 4> Pats;
+  };
+
+  PatFrag(const Record &Def) : Def(Def) {}
+
+  static StringRef getParamKindStr(ParamKind OK);
+
+  StringRef getName() const { return Def.getName(); }
+
+  const Record &getDef() const { return Def; }
+  ArrayRef<SMLoc> getLoc() const { return Def.getLoc(); }
+
+  Alternative &addAlternative() { return Alts.emplace_back(); }
+  const Alternative &getAlternative(unsigned K) const { return Alts[K]; }
+  unsigned num_alternatives() const { return Alts.size(); }
+
+  void addInParam(StringRef Name, ParamKind Kind);
+  iterator_range<ParamIt> in_params() const;
+  unsigned num_in_params() const { return Params.size() - NumOutParams; }
+
+  void addOutParam(StringRef Name, ParamKind Kind);
+  iterator_range<ParamIt> out_params() const;
+  unsigned num_out_params() const { return NumOutParams; }
+
+  unsigned num_roots() const;
+  unsigned num_params() const { return num_in_params() + num_out_params(); }
+
+  /// Finds the operand \p Name and returns its index or -1 if not found.
+  /// Remember that all params are part of the same list, with out params at the
+  /// start. This means that the index returned can be used to access operands
+  /// of InstructionPatterns.
+  unsigned getParamIdx(StringRef Name) const;
+  const Param &getParam(unsigned K) const { return Params[K]; }
+
+  bool canBeMatchRoot() const { return num_roots() == 1; }
+
+  void print(raw_ostream &OS, StringRef Indent = "") const;
+  void dump() const { print(dbgs()); }
+
+  /// Checks if the in-param \p ParamName can be unbound or not.
+  /// \p ArgName is the name of the argument passed to the PatFrag.
+  ///
+  /// An argument can be unbound only if, for all alternatives:
+  ///   - There is no CXX pattern, OR:
+  ///   - There is an InstructionPattern that binds the parameter.
+  ///
+  /// e.g. in (MyPatFrag $foo), if $foo has never been seen before (= it's
+  /// unbound), this checks if MyPatFrag supports it or not.
+  bool handleUnboundInParam(StringRef ParamName, StringRef ArgName,
+                            ArrayRef<SMLoc> DiagLoc) const;
+
+  bool checkSemantics();
+  bool buildOperandsTables();
+
+private:
+  static void printParamsList(raw_ostream &OS, iterator_range<ParamIt> Params);
+
+  void PrintError(Twine Msg) const { ::PrintError(&Def, Msg); }
+
+  const Record &Def;
+  unsigned NumOutParams = 0;
+  ParamVec Params;
+  SmallVector<Alternative, 2> Alts;
+};
+
+StringRef PatFrag::getParamKindStr(ParamKind OK) {
+  switch (OK) {
+  case PK_Root:
+    return "root";
+  case PK_MachineOperand:
+    return "machine_operand";
+  case PK_Imm:
+    return "imm";
+  }
+
+  llvm_unreachable("Unknown operand kind!");
+}
+
+void PatFrag::addInParam(StringRef Name, ParamKind Kind) {
+  Params.emplace_back(Param{insertStrRef(Name), Kind});
+}
+
+iterator_range<PatFrag::ParamIt> PatFrag::in_params() const {
+  return {Params.begin() + NumOutParams, Params.end()};
+}
+
+void PatFrag::addOutParam(StringRef Name, ParamKind Kind) {
+  assert(NumOutParams == Params.size() &&
+         "Adding out-param after an in-param!");
+  Params.emplace_back(Param{insertStrRef(Name), Kind});
+  ++NumOutParams;
+}
+
+iterator_range<PatFrag::ParamIt> PatFrag::out_params() const {
+  return {Params.begin(), Params.begin() + NumOutParams};
+}
+
+unsigned PatFrag::num_roots() const {
+  return count_if(out_params(),
+                  [&](const auto &P) { return P.Kind == PK_Root; });
+}
+
+unsigned PatFrag::getParamIdx(StringRef Name) const {
+  for (const auto &[Idx, O] : enumerate(Params)) {
+    if (O.Name == Name)
+      return Idx;
+  }
+
+  return -1;
+}
+
+bool PatFrag::checkSemantics() {
+  for (const auto &A : Alts) {
+    for (const auto &Pat : A.Pats) {
+      switch (Pat->getKind()) {
+      case Pattern::K_AnyOpcode:
+        PrintError("wip_match_opcode cannot be used in " + PatFragClassName);
+        return false;
+      case Pattern::K_CXX:
+      case Pattern::K_CodeGenInstruction:
+        continue;
+      case Pattern::K_PatFrag:
+        // TODO: It's just that the emitter doesn't handle it but technically
+        // there is no reason why we can't. We just have to be careful with
+        // operand mappings, it could get complex.
+        PrintError("nested " + PatFragClassName + " are not supported");
+        return false;
+      }
+    }
+  }
+
+  StringSet<> SeenOps;
+  for (const auto &O : in_params()) {
+    if (SeenOps.count(O.Name)) {
+      PrintError("duplicate parameter '" + O.Name + "'");
+      return false;
+    }
+
+    // Check this operand is NOT defined in any alternative's patterns.
+    for (const auto &A : Alts) {
+      if (A.OpTable.getMatchDefPattern(O.Name)) {
+        PrintError("input parameter '" + O.Name + "' cannot be redefined!");
+        return false;
+      }
+    }
+
+    if (O.Kind == PK_Root) {
+      PrintError("input parameterr '" + O.Name + "' cannot be a root!");
+      return false;
+    }
+
+    SeenOps.insert(O.Name);
+  }
+
+  for (const auto &O : out_params()) {
+    if (O.Kind != PK_Root && O.Kind != PK_MachineOperand) {
+      PrintError("output parameter '" + O.Name + "' must be 'root' or 'gi_mo'");
+      return false;
+    }
+
+    if (SeenOps.count(O.Name)) {
+      PrintError("duplicate parameter '" + O.Name + "'");
+      return false;
+    }
+
+    // Check this operand is defined in all alternative's patterns.
+    for (const auto &A : Alts) {
+      if (!A.OpTable.getMatchDefPattern(O.Name)) {
+        PrintError("output parameter '" + O.Name +
+                   "' must be defined by all alternative patterns in '" +
+                   Def.getName() + "'");
+        return false;
+      }
+    }
+
+    SeenOps.insert(O.Name);
+  }
+
+  if (num_out_params() != 0 && num_roots() == 0) {
+    PrintError(PatFragClassName + " must have one root in its 'out' operands");
+    return false;
+  }
+
+  if (num_roots() > 1) {
+    PrintError(PatFragClassName + " can only have one root");
+    return false;
+  }
+
+  // TODO: find unused params
+
+  // Now, typecheck all alternatives.
+  for (auto &A : Alts) {
+    OperandTypeChecker OTC(Def.getLoc());
+    for (auto &Pat : A.Pats) {
+      if (auto *IP = dyn_cast<InstructionPattern>(Pat.get())) {
+        if (!OTC.check(IP))
+          return false;
+      }
+    }
+    OTC.setAllOperandTypes();
+  }
+
+  return true;
+}
+
+bool PatFrag::handleUnboundInParam(StringRef ParamName, StringRef ArgName,
+                                   ArrayRef<SMLoc> DiagLoc) const {
+  // The parameter must be a live-in of all alternatives for this to work.
+  // Otherwise, we risk having unbound parameters being used (= crashes).
+  //
+  // Examples:
+  //
+  // in (ins $y), (patterns (G_FNEG $dst, $y), "return matchFnegOp(${y})")
+  //    even if $y is unbound, we'll lazily bind it when emitting the G_FNEG.
+  //
+  // in (ins $y), (patterns "return matchFnegOp(${y})")
+  //    if $y is unbound when this fragment is emitted, C++ code expansion will
+  //    fail.
+  for (const auto &A : Alts) {
+    if (!A.OpTable.isMatchLiveInOrDef(ParamName) &&
+        any_of(A.Pats,
+               [](const auto &P) { return isa<CXXPattern>(P.get()); })) {
+      ::PrintError(DiagLoc, "operand '" + ArgName + "' (for parameter '" +
+                                ParamName + "' of '" + getName() +
+                                "') cannot be unbound");
+      PrintNote(
+          DiagLoc,
+          "one or more alternatives of '" + getName() + "' do not bind '" +
+              ParamName +
+              "' to an instruction operand; either use a bound operand or "
+              "ensure '" +
+              Def.getName() + "' binds '" + ParamName +
+              "' in all alternatives");
+      return false;
+    }
+  }
+
+  return true;
+}
+
+bool PatFrag::buildOperandsTables() {
+  // enumerate(...) doesn't seem to allow lvalues so we need to count the old
+  // way.
+  unsigned Idx = 0;
+  for (auto &A : Alts) {
+    for (auto &Pat : A.Pats) {
+      if (auto *IP = dyn_cast<InstructionPattern>(Pat.get())) {
+        if (!A.OpTable.addMatchPattern(Def.getLoc(), IP,
+                                       "in patterns of alternative #" +
+                                           to_string(Idx))) {
+          return false;
+        }
+      }
+    }
+
+    ++Idx;
+  }
+
+  return true;
+}
+
+void PatFrag::print(raw_ostream &OS, StringRef Indent) const {
+  OS << Indent << "(PatFrag name:" << getName() << "\n";
+  if (!in_params().empty()) {
+    OS << Indent << "  (ins ";
+    printParamsList(OS, in_params());
+    OS << ")\n";
+  }
+
+  if (!out_params().empty()) {
+    OS << Indent << "  (outs ";
+    printParamsList(OS, out_params());
+    OS << ")\n";
+  }
+
+  // TODO: Dump OperandTable as well.
+  OS << Indent << "  (alternatives [\n";
+  for (const auto &A : Alts) {
+    OS << Indent << "    [\n";
+    for (const auto &P : A.Pats) {
+      OS << Indent << "      ";
+      P->print(OS, /*PrintName=*/true);
+      OS << ",\n";
+    }
+    OS << Indent << "    ],\n";
+  }
+  OS << Indent << "  ])\n";
+
+  OS << Indent << ")";
+}
+
+void PatFrag::printParamsList(raw_ostream &OS, iterator_range<ParamIt> Params) {
+  OS << "["
+     << join(map_range(Params,
+                       [](auto &O) {
+                         return (O.Name + ":" + getParamKindStr(O.Kind)).str();
+                       }),
+             ", ")
+     << "]";
+}
+
+//===- PatFragPattern -----------------------------------------------------===//
+
+class PatFragPattern : public InstructionPattern {
+public:
+  PatFragPattern(const PatFrag &PF, StringRef Name)
+      : InstructionPattern(K_PatFrag, Name), PF(PF) {}
+
+  static bool classof(const Pattern *P) { return P->getKind() == K_PatFrag; }
+
+  const PatFrag &getPatFrag() const { return PF; }
+  StringRef getInstName() const override { return PF.getName(); }
+
+  unsigned getNumInstDefs() const override { return PF.num_out_params(); }
+  unsigned getNumInstOperands() const override { return PF.num_params(); }
+
+  bool checkSemantics(ArrayRef<SMLoc> DiagLoc) override;
+
+  bool mapInputCodeExpansions(const CodeExpansions &ParentCEs,
+                              CodeExpansions &PatFragCEs,
+                              ArrayRef<SMLoc> DiagLoc) const;
+
+private:
+  const PatFrag &PF;
+};
+
+bool PatFragPattern::checkSemantics(ArrayRef<SMLoc> DiagLoc) {
+  InstructionPattern::checkSemantics(DiagLoc);
+
+  for (const auto &[Idx, Op] : enumerate(Operands)) {
+    switch (PF.getParam(Idx).Kind) {
+    case PatFrag::PK_Imm:
+      if (!Op.hasImmValue()) {
+        PrintError(DiagLoc, "expected operand " + to_string(Idx) + " of '" +
+                                getInstName() + "' to be an immediate; got " +
+                                Op.describe());
+        return false;
+      }
+      if (Op.isNamedImmediate()) {
+        PrintError(DiagLoc, "operand " + to_string(Idx) + " of '" +
+                                getInstName() +
+                                "' cannot be a named immediate");
+        return false;
+      }
+      break;
+    case PatFrag::PK_Root:
+    case PatFrag::PK_MachineOperand:
+      if (!Op.isNamedOperand() || Op.isNamedImmediate()) {
+        PrintError(DiagLoc, "expected operand " + to_string(Idx) + " of '" +
+                                getInstName() +
+                                "' to be a MachineOperand; got " +
+                                Op.describe());
+        return false;
+      }
+      break;
+    }
+  }
+
+  return true;
+}
+
+bool PatFragPattern::mapInputCodeExpansions(const CodeExpansions &ParentCEs,
+                                            CodeExpansions &PatFragCEs,
+                                            ArrayRef<SMLoc> DiagLoc) const {
+  for (const auto &[Idx, O] : enumerate(operands())) {
+    StringRef ParamName = PF.getParam(Idx).Name;
+
+    // Operands to a PFP can only be named, or be an immediate, but not a named
+    // immediate.
+    assert(!O.isNamedImmediate());
+
+    if (O.isNamedOperand()) {
+      StringRef ArgName = O.getOperandName();
+      // Map it only if it's been defined.
+      auto It = ParentCEs.find(ArgName);
+      if (It == ParentCEs.end()) {
+        if (!PF.handleUnboundInParam(ParamName, ArgName, DiagLoc))
+          return false;
+      } else
+        PatFragCEs.declare(ParamName, It->second);
+      continue;
+    }
+
+    if (O.hasImmValue()) {
+      PatFragCEs.declare(ParamName, to_string(O.getImmValue()));
+      continue;
+    }
+
+    llvm_unreachable("Unknown Operand Type!");
+  }
+
+  return true;
+}
+
+//===- PrettyStackTrace Helpers  ------------------------------------------===//
+
+class PrettyStackTraceParse : public PrettyStackTraceEntry {
+  const Record &Def;
+
+public:
+  PrettyStackTraceParse(const Record &Def) : Def(Def) {}
+
+  void print(raw_ostream &OS) const override {
+    if (Def.isSubClassOf("GICombineRule"))
+      OS << "Parsing GICombineRule '" << Def.getName() << "'";
+    else if (Def.isSubClassOf(PatFragClassName))
+      OS << "Parsing " << PatFragClassName << " '" << Def.getName() << "'";
+    else
+      OS << "Parsing '" << Def.getName() << "'";
+    OS << "\n";
+  }
+};
 
-  bool isLiveIn() const { return !MatchPat; }
+class PrettyStackTraceEmit : public PrettyStackTraceEntry {
+  const Record &Def;
+  const Pattern *Pat = nullptr;
+
+public:
+  PrettyStackTraceEmit(const Record &Def, const Pattern *Pat = nullptr)
+      : Def(Def), Pat(Pat) {}
+
+  void print(raw_ostream &OS) const override {
+    if (Def.isSubClassOf("GICombineRule"))
+      OS << "Emitting GICombineRule '" << Def.getName() << "'";
+    else if (Def.isSubClassOf(PatFragClassName))
+      OS << "Emitting " << PatFragClassName << " '" << Def.getName() << "'";
+    else
+      OS << "Emitting '" << Def.getName() << "'";
+
+    if (Pat)
+      OS << " [" << Pat->getKindName() << " '" << Pat->getName() << "']";
+    OS << "\n";
+  }
 };
 
+//===- CombineRuleBuilder -------------------------------------------------===//
+
 /// Parses combine rule and builds a small intermediate representation to tie
 /// patterns together and emit RuleMatchers to match them. This may emit more
 /// than one RuleMatcher, e.g. for `wip_match_opcode`.
 ///
 /// Memory management for `Pattern` objects is done through `std::unique_ptr`.
 /// In most cases, there are two stages to a pattern's lifetime:
 ///   - Creation in a `parse` function
 ///     - The unique_ptr is stored in a variable, and may be destroyed if the
 ///       pattern is found to be semantically invalid.
 ///   - Ownership transfer into a `PatternMap`
 ///     - Once a pattern is moved into either the map of Match or Apply
 ///       patterns, it is known to be valid and it never moves back.
 class CombineRuleBuilder {
 public:
   using PatternMap = MapVector<StringRef, std::unique_ptr<Pattern>>;
+  using PatternAlternatives = DenseMap<const Pattern *, unsigned>;
 
   CombineRuleBuilder(const CodeGenTarget &CGT,
                      SubtargetFeatureInfoMap &SubtargetFeatures,
                      Record &RuleDef, unsigned ID,
                      std::vector<RuleMatcher> &OutRMs)
       : CGT(CGT), SubtargetFeatures(SubtargetFeatures), RuleDef(RuleDef),
         RuleID(ID), OutRMs(OutRMs) {}
 
   /// Parses all fields in the RuleDef record.
   bool parseAll();
 
   /// Emits all RuleMatchers into the vector of RuleMatchers passed in the
   /// constructor.
   bool emitRuleMatchers();
 
   void print(raw_ostream &OS) const;
   void dump() const { print(dbgs()); }
 
   /// Debug-only verification of invariants.
   void verify() const;
 
 private:
-  void PrintError(Twine Msg) const { ::PrintError(RuleDef.getLoc(), Msg); }
+  void PrintError(Twine Msg) const { ::PrintError(&RuleDef, Msg); }
+  void PrintWarning(Twine Msg) const { ::PrintWarning(RuleDef.getLoc(), Msg); }
+  void PrintNote(Twine Msg) const { ::PrintNote(RuleDef.getLoc(), Msg); }
+
+  void print(raw_ostream &OS, const PatternAlternatives &Alts) const;
+
+  bool addApplyPattern(std::unique_ptr<Pattern> Pat);
+  bool addMatchPattern(std::unique_ptr<Pattern> Pat);
 
   /// Adds the expansions from \see MatchDatas to \p CE.
   void declareAllMatchDatasExpansions(CodeExpansions &CE) const;
 
-  /// Adds \p P to \p IM, expanding its code using \p CE.
-  void addCXXPredicate(InstructionMatcher &IM, const CodeExpansions &CE,
-                       const CXXPattern &P);
+  /// Adds a matcher \p P to \p IM, expanding its code using \p CE.
+  /// Note that the predicate is added on the last InstructionMatcher.
+  ///
+  /// \p Alts is only used if DebugCXXPreds is enabled.
+  void addCXXPredicate(RuleMatcher &M, const CodeExpansions &CE,
+                       const CXXPattern &P, const PatternAlternatives &Alts);
+
+  /// Adds an apply \p P to \p IM, expanding its code using \p CE.
+  void addCXXAction(RuleMatcher &M, const CodeExpansions &CE,
 
+  bool hasOnlyCXXApplyPatterns() const;
+  bool hasAnyOpcodeMatchPattern() const {
+    return any_of(MatchPats, [&](const auto &E) {
+      return isa<AnyOpcodePattern>(E.second.get());
+    });
+  }
+
+  // Infer machine operand types and check their consistency.
+  bool typecheckPatterns();
+
+  /// For all PatFragPatterns, add a new entry in PatternAlternatives for each
+  /// PatternList it contains. This is multiplicative, so if we have 2
+  /// PatFrags with 3 alternatives each, we get 2*3 permutations added to
+  /// PermutationsToEmit. The "MaxPermutations" field controls how many
+  /// permutations are allowed before an error is emitted and this function
+  /// returns false. This is a simple safeguard to prevent combination of
+  /// PatFrags from generating enormous amounts of rules.
+  bool buildPermutationsToEmit();
+
   /// Generates a name for anonymous patterns.
   ///
   /// e.g. (G_ADD $x, $y, $z):$foo is a pattern named "foo", but if ":$foo" is
   /// absent, then the pattern is anonymous and this is used to assign it a
   /// name.
   std::string makeAnonPatName(StringRef Prefix) const;
   mutable unsigned AnonIDCnt = 0;
 
   /// Creates a new RuleMatcher with some boilerplate
   /// settings/actions/predicates, and and adds it to \p OutRMs.
   /// \see addFeaturePredicates too.
   ///
+  /// \param Alts Current set of alternatives, for debug comment.
   /// \param AdditionalComment Comment string to be added to the
   ///        `DebugCommentAction`.
-  RuleMatcher &addRuleMatcher(Twine AdditionalComment = "");
+  RuleMatcher &addRuleMatcher(const PatternAlternatives &Alts,
+                              Twine AdditionalComment = "");
   bool addFeaturePredicates(RuleMatcher &M);
 
   bool findRoots();
-  bool buildOperandsTable();
+  bool buildRuleOperandsTable();
 
-  bool parseDefs(DagInit &Def);
-  bool parseMatch(DagInit &Match);
-  bool parseApply(DagInit &Apply);
+  bool parseDefs(const DagInit &Def);
+  bool
+  parsePatternList(const DagInit &List,
+                   function_ref<bool(std::unique_ptr<Pattern>)> ParseAction,
+                   StringRef Operator, ArrayRef<SMLoc> DiagLoc) const;
 
-  std::unique_ptr<Pattern> parseInstructionMatcher(const Init &Arg,
-                                                   StringRef PatName);
+  std::unique_ptr<Pattern> parseInstructionPattern(const Init &Arg,
+                                                   StringRef PatName) const;
   std::unique_ptr<Pattern> parseWipMatchOpcodeMatcher(const Init &Arg,
-                                                      StringRef PatName);
-
-  bool emitMatchPattern(CodeExpansions &CE, const InstructionPattern &IP);
-  bool emitMatchPattern(CodeExpansions &CE, const AnyOpcodePattern &AOP);
+                                                      StringRef PatName) const;
+  bool parseInstructionPatternOperand(InstructionPattern &IP,
+                                      const Init *OpInit,
+                                      const StringInit *OpName) const;
+  std::unique_ptr<PatFrag> parsePatFragImpl(const Record *Def) const;
+  bool parsePatFragParamList(
+      ArrayRef<SMLoc> DiagLoc, const DagInit &OpsList,
+      function_ref<bool(StringRef, PatFrag::ParamKind)> ParseAction) const;
+  const PatFrag *parsePatFrag(const Record *Def) const;
+
+  bool emitMatchPattern(CodeExpansions &CE, const PatternAlternatives &Alts,
+                        const InstructionPattern &IP);
+  bool emitMatchPattern(CodeExpansions &CE, const PatternAlternatives &Alts,
+                        const AnyOpcodePattern &AOP);
+
+  bool emitPatFragMatchPattern(CodeExpansions &CE,
+                               const PatternAlternatives &Alts, RuleMatcher &RM,
+                               InstructionMatcher *IM,
+                               const PatFragPattern &PFP,
+                               DenseSet<const Pattern *> &SeenPats);
 
   bool emitApplyPatterns(CodeExpansions &CE, RuleMatcher &M);
+  bool emitTempRegisterActions(CodeExpansions &CE, RuleMatcher &M,
+                               StringMap<unsigned> &OperandToTempRegID) const;
+
+  // Recursively visits CodeGenInstructionPattern from P to build up the
+  // RuleMatcher actions.
+  bool emitCodeGenInstructionApplyPattern(
+      CodeExpansions &CE, RuleMatcher &M, const CodeGenInstructionPattern &P,
+      DenseSet<const Pattern *> &SeenPats,
+      const StringMap<unsigned> &OperandToTempRegID);
 
-  // Recursively visits InstructionPattern from P to build up the
+  // Recursively visits CodeGenInstructionPattern from P to build up the
   // RuleMatcher/InstructionMatcher. May create new InstructionMatchers as
   // needed.
-  bool emitInstructionMatchPattern(CodeExpansions &CE, RuleMatcher &M,
-                                   InstructionMatcher &IM,
-                                   const InstructionPattern &P,
-                                   DenseSet<const Pattern *> &SeenPats);
+  using OperandMapperFnRef = function_ref<InstructionPattern::Operand(
+      const InstructionPattern::Operand &)>;
+  bool emitCodeGenInstructionMatchPattern(
+      CodeExpansions &CE, const PatternAlternatives &Alts,
+      const OperandTable &OpTable, RuleMatcher &M, InstructionMatcher &IM,
+      const CodeGenInstructionPattern &P, DenseSet<const Pattern *> &SeenPats,
+      OperandMapperFnRef OperandMapper = [](const auto &O) { return O; });
 
   const CodeGenTarget &CGT;
   SubtargetFeatureInfoMap &SubtargetFeatures;
   Record &RuleDef;
   const unsigned RuleID;
   std::vector<RuleMatcher> &OutRMs;
 
   // For InstructionMatcher::addOperand
   unsigned AllocatedTemporariesBaseID = 0;
 
   /// The root of the pattern.
   StringRef RootName;
 
   /// These maps have ownership of the actual Pattern objects.
   /// They both map a Pattern's name to the Pattern instance.
   PatternMap MatchPats;
   PatternMap ApplyPats;
 
   /// Set by findRoots.
   Pattern *MatchRoot = nullptr;
+  SmallVector<InstructionPattern *, 4> ApplyRoots;
+
+  // Operand table for the whole rule.
+  OperandTable RuleOpTable;
 
-  MapVector<StringRef, OperandTableEntry> OperandTable;
   SmallVector<MatchDataInfo, 2> MatchDatas;
+  SmallVector<PatternAlternatives, 1> PermutationsToEmit;
+
+  // print()/debug-only members.
+  mutable SmallPtrSet<const PatFrag *, 2> SeenPatFrags;
 };
 
 bool CombineRuleBuilder::parseAll() {
+  auto StackTrace = PrettyStackTraceParse(RuleDef);
+
   if (!parseDefs(*RuleDef.getValueAsDag("Defs")))
     return false;
-  if (!parseMatch(*RuleDef.getValueAsDag("Match")))
-    return false;
-  if (!parseApply(*RuleDef.getValueAsDag("Apply")))
+
+  if (!parsePatternList(
+          *RuleDef.getValueAsDag("Match"),
+          [this](auto Pat) { return addMatchPattern(std::move(Pat)); }, "match",
+          RuleDef.getLoc()))
     return false;
-  if (!buildOperandsTable())
+
+  if (!parsePatternList(
+          *RuleDef.getValueAsDag("Apply"),
+          [this](auto Pat) { return addApplyPattern(std::move(Pat)); }, "apply",
+          RuleDef.getLoc()))
     return false;
-  if (!findRoots())
+
+  if (!buildRuleOperandsTable() || !typecheckPatterns() || !findRoots() ||
+      !buildPermutationsToEmit())
     return false;
   LLVM_DEBUG(verify());
   return true;
 }
 
 bool CombineRuleBuilder::emitRuleMatchers() {
+  auto StackTrace = PrettyStackTraceEmit(RuleDef);
+
   assert(MatchRoot);
   CodeExpansions CE;
   declareAllMatchDatasExpansions(CE);
 
+  assert(!PermutationsToEmit.empty());
+  for (const auto &Alts : PermutationsToEmit) {
     switch (MatchRoot->getKind()) {
     case Pattern::K_AnyOpcode: {
-    if (!emitMatchPattern(CE, *cast<AnyOpcodePattern>(MatchRoot)))
+      if (!emitMatchPattern(CE, Alts, *cast<AnyOpcodePattern>(MatchRoot)))
         return false;
       break;
     }
-  case Pattern::K_Inst:
-    if (!emitMatchPattern(CE, *cast<InstructionPattern>(MatchRoot)))
+    case Pattern::K_PatFrag:
+    case Pattern::K_CodeGenInstruction:
+      if (!emitMatchPattern(CE, Alts, *cast<InstructionPattern>(MatchRoot)))
         return false;
       break;
     case Pattern::K_CXX:
-    PrintError("C++ code cannot be the root of a pattern!");
+      PrintError("C++ code cannot be the root of a rule!");
       return false;
     default:
       llvm_unreachable("unknown pattern kind!");
     }
+  }
 
   return true;
 }
 
 void CombineRuleBuilder::print(raw_ostream &OS) const {
   OS << "(CombineRule name:" << RuleDef.getName() << " id:" << RuleID
      << " root:" << RootName << "\n";
 
-  OS << "  (MatchDatas ";
-  if (MatchDatas.empty())
-    OS << "<empty>)\n";
-  else {
-    OS << "\n";
+  if (!MatchDatas.empty()) {
+    OS << "  (MatchDatas\n";
     for (const auto &MD : MatchDatas) {
       OS << "    ";
       MD.print(OS);
       OS << "\n";
     }
     OS << "  )\n";
   }
 
+  if (!SeenPatFrags.empty()) {
+    OS << "  (PatFrags\n";
+    for (const auto *PF : SeenPatFrags) {
+      PF->print(OS, /*Indent=*/"    ");
+      OS << "\n";
+    }
+    OS << "  )\n";
+  }
+
   const auto DumpPats = [&](StringRef Name, const PatternMap &Pats) {
     OS << "  (" << Name << " ";
     if (Pats.empty()) {
       OS << "<empty>)\n";
       return;
     }
 
     OS << "\n";
     for (const auto &[Name, Pat] : Pats) {
       OS << "    ";
       if (Pat.get() == MatchRoot)
         OS << "<root>";
       OS << Name << ":";
       Pat->print(OS, /*PrintName=*/false);
       OS << "\n";
     }
     OS << "  )\n";
   };
 
   DumpPats("MatchPats", MatchPats);
   DumpPats("ApplyPats", ApplyPats);
 
-  OS << "  (OperandTable ";
-  if (OperandTable.empty())
-    OS << "<empty>)\n";
-  else {
-    OS << "\n";
-    for (const auto &[Key, Val] : OperandTable) {
-      OS << "    [" << Key;
-      if (const auto *P = Val.MatchPat)
-        OS << " match_pat:" << P->getName();
-      if (const auto *P = Val.ApplyPat)
-        OS << " apply_pat:" << P->getName();
-      if (Val.isLiveIn())
-        OS << " live-in";
-      OS << "]\n";
+  RuleOpTable.print(OS, /*Indent=*/"  ");
+
+  if (PermutationsToEmit.size() > 1) {
+    OS << "  (PermutationsToEmit\n";
+    for (const auto &P : PermutationsToEmit) {
+      OS << "    ";
+      print(OS, P);
+      OS << ",\n";
     }
     OS << "  )\n";
   }
 
   OS << ")\n";
 }
 
 void CombineRuleBuilder::verify() const {
   const auto VerifyPats = [&](const PatternMap &Pats) {
     for (const auto &[Name, Pat] : Pats) {
       if (!Pat)
         PrintFatalError("null pattern in pattern map!");
 
       if (Name != Pat->getName()) {
         Pat->dump();
         PrintFatalError("Pattern name mismatch! Map name: " + Name +
                         ", Pat name: " + Pat->getName());
       }
 
-      // As an optimization, the PatternMaps don't re-allocate the PatternName
-      // string. They simply reference the std::string inside Pattern. Ensure
-      // this is the case to avoid memory issues.
+      // Sanity check: the map should point to the same data as the Pattern.
+      // Both strings are allocated in the pool using insertStrRef.
       if (Name.data() != Pat->getName().data()) {
         dbgs() << "Map StringRef: '" << Name << "' @ "
                << (const void *)Name.data() << "\n";
         dbgs() << "Pat String: '" << Pat->getName() << "' @ "
                << (const void *)Pat->getName().data() << "\n";
         PrintFatalError("StringRef stored in the PatternMap is not referencing "
                         "the same string as its Pattern!");
       }
     }
   };
 
   VerifyPats(MatchPats);
   VerifyPats(ApplyPats);
 
-  for (const auto &[Name, Op] : OperandTable) {
+  // Check there are no wip_match_opcode patterns in the "apply" patterns.
+  if (any_of(ApplyPats,
+             [&](auto &E) { return isa<AnyOpcodePattern>(E.second.get()); })) {
+    dump();
+    PrintFatalError(
+        "illegal wip_match_opcode pattern in the 'apply' patterns!");
+  }
+
+  for (const auto &[Name, Op] : RuleOpTable) {
     if (Op.ApplyPat && !Op.MatchPat) {
       dump();
       PrintFatalError("Operand " + Name +
                       " has an apply pattern, but no match pattern!");
     }
+
+    if (Op.MatchPat && Op.IsMatchLiveIn) {
+      dump();
+      PrintFatalError(
+          "Operand " + Name +
+          " is defined in match pattern but marked as match-live-in");
+    }
   }
 }
 
-bool CombineRuleBuilder::addFeaturePredicates(RuleMatcher &M) {
-  if (!RuleDef.getValue("Predicates"))
-    return true;
+void CombineRuleBuilder::print(raw_ostream &OS,
+                               const PatternAlternatives &Alts) const {
+  SmallVector<std::string, 1> Strings(
+      map_range(Alts, [](const auto &PatAndPerm) {
+        return PatAndPerm.first->getName().str() + "[" +
+               to_string(PatAndPerm.second) + "]";
+      }));
+  // Sort so output is deterministic for tests. Otherwise it's sorted by pointer
+  // values.
+  sort(Strings);
+  OS << "[" << join(Strings, ", ") << "]";
+}
 
-  ListInit *Preds = RuleDef.getValueAsListInit("Predicates");
-  for (Init *I : Preds->getValues()) {
-    if (DefInit *Pred = dyn_cast<DefInit>(I)) {
-      Record *Def = Pred->getDef();
-      if (!Def->isSubClassOf("Predicate")) {
-        ::PrintError(Def->getLoc(), "Unknown 'Predicate' Type");
-        return false;
-      }
+bool CombineRuleBuilder::addApplyPattern(std::unique_ptr<Pattern> Pat) {
+  StringRef Name = Pat->getName();
+  if (ApplyPats.contains(Name)) {
+    PrintError("'" + Name + "' apply pattern defined more than once!");
 
-      if (Def->getValueAsString("CondString").empty())
-        continue;
+  if (isa<AnyOpcodePattern>(Pat.get())) {
+    PrintError("'" + Name +
+               "': wip_match_opcode is not supported in apply patterns");
+    return false;
+  }
 
-      if (SubtargetFeatures.count(Def) == 0) {
-        SubtargetFeatures.emplace(
-            Def, SubtargetFeatureInfo(Def, SubtargetFeatures.size()));
-      }
+  if (isa<InstructionPattern>(Pat.get())) {
+    if (hasAnyOpcodeMatchPattern()) {
+      PrintError("cannot use wip_match_opcode in combination with apply "
+                 "instruction patterns!");
+      return false;
 
-      M.addRequiredFeature(Def);
+    if (isa<PatFragPattern>(Pat.get())) {
+      PrintError("'" + Name + "': using " + PatFragClassName +
+                 " is not supported in apply patterns");
+      return false;
+    }
+  }
+
+  if (auto *CXXPat = dyn_cast<CXXPattern>(Pat.get()))
+    CXXPat->setIsApply();
+
+  ApplyPats[Name] = std::move(Pat);
+  return true;
+}
+
+bool CombineRuleBuilder::addMatchPattern(std::unique_ptr<Pattern> Pat) {
+  StringRef Name = Pat->getName();
+  if (MatchPats.contains(Name)) {
+    PrintError("'" + Name + "' match pattern defined more than once!");
+    return false;
+  }
+
+  // TODO: Could move this to some "checkSemantics" method?
+  if (isa<AnyOpcodePattern>(Pat.get())) {
+    if (hasAnyOpcodeMatchPattern()) {
+      PrintError("wip_opcode_match can only be present once");
+      return false;
+    }
+  }
+
+  if (const auto *CXXPat = dyn_cast<CXXPattern>(Pat.get())) {
+    if (!CXXPat->getRawCode().contains("return ")) {
+      PrintWarning("'match' C++ code does not seem to return!");
     }
   }
 
+  MatchPats[Name] = std::move(Pat);
   return true;
 }
 
 void CombineRuleBuilder::declareAllMatchDatasExpansions(
     CodeExpansions &CE) const {
   for (const auto &MD : MatchDatas)
     CE.declare(MD.getPatternSymbol(), MD.getQualifiedVariableName());
 }
 
-void CombineRuleBuilder::addCXXPredicate(InstructionMatcher &IM,
+void CombineRuleBuilder::addCXXPredicate(RuleMatcher &M,
                                          const CodeExpansions &CE,
-                                         const CXXPattern &P) {
+                                         const CXXPattern &P,
+                                         const PatternAlternatives &Alts) {
+  // FIXME: Hack so C++ code is executed last. May not work for more complex
+  // patterns.
+  auto &IM = *std::prev(M.insnmatchers().end());
+  const auto &ExpandedCode =
+      P.expandCode(CE, RuleDef.getLoc(), [&](raw_ostream &OS) {
+        OS << "// Pattern Alternatives: ";
+        print(OS, Alts);
+        OS << "\n";
+      });
+  IM->addPredicate<GenericInstructionPredicateMatcher>(
+      ExpandedCode.getEnumNameWithPrefix(CXXPredPrefix));
+}
+
+void CombineRuleBuilder::addCXXAction(RuleMatcher &M, const CodeExpansions &CE,
   const auto &ExpandedCode = P.expandCode(CE, RuleDef.getLoc());
-  IM.addPredicate<GenericInstructionPredicateMatcher>(
-      ExpandedCode.getEnumNameWithPrefix(CXXPredPrefix));
+  M.addAction<CustomCXXAction>(
+      ExpandedCode.getEnumNameWithPrefix(CXXApplyPrefix));
+}
+
+bool CombineRuleBuilder::hasOnlyCXXApplyPatterns() const {
+  return all_of(ApplyPats, [&](auto &Entry) {
+    return isa<CXXPattern>(Entry.second.get());
+  });
+}
+
+bool CombineRuleBuilder::typecheckPatterns() {
+  OperandTypeChecker OTC(RuleDef.getLoc());
+
+  for (auto &Pat : values(MatchPats)) {
+    if (auto *IP = dyn_cast<InstructionPattern>(Pat.get())) {
+      if (!OTC.check(IP))
+        return false;
+    }
+  }
+
+  for (auto &Pat : values(ApplyPats)) {
+    if (auto *IP = dyn_cast<InstructionPattern>(Pat.get())) {
+      if (!OTC.check(IP))
+        return false;
+    }
+  }
+
+  OTC.setAllOperandTypes();
+  return true;
+}
+
+bool CombineRuleBuilder::buildPermutationsToEmit() {
+  PermutationsToEmit.clear();
+
+  // Start with one empty set of alternatives.
+  PermutationsToEmit.emplace_back();
+  for (const auto &Pat : values(MatchPats)) {
+    unsigned NumAlts = 0;
+    // Note: technically, AnyOpcodePattern also needs permutations, but:
+    //    - We only allow a single one of them in the root.
+    //    - They cannot be mixed with any other pattern other than C++ code.
+    // So we don't really need to take them into account here. We could, but
+    // that pattern is a hack anyway and the less it's involved, the better.
+    if (const auto *PFP = dyn_cast<PatFragPattern>(Pat.get()))
+      NumAlts = PFP->getPatFrag().num_alternatives();
+    else
+      continue;
+
+    // For each pattern that needs permutations, multiply the current set of
+    // alternatives.
+    auto CurPerms = PermutationsToEmit;
+    PermutationsToEmit.clear();
+
+    for (const auto &Perm : CurPerms) {
+      assert(!Perm.count(Pat.get()) && "Pattern already emitted?");
+      for (unsigned K = 0; K < NumAlts; ++K) {
+        PatternAlternatives NewPerm = Perm;
+        NewPerm[Pat.get()] = K;
+        PermutationsToEmit.emplace_back(std::move(NewPerm));
+      }
+    }
+  }
+
+  if (int64_t MaxPerms = RuleDef.getValueAsInt("MaxPermutations");
+      MaxPerms > 0) {
+    if ((int64_t)PermutationsToEmit.size() > MaxPerms) {
+      PrintError("cannot emit rule '" + RuleDef.getName() + "'; " +
+                 Twine(PermutationsToEmit.size()) +
+                 " permutations would be emitted, but the max is " +
+                 Twine(MaxPerms));
+      return false;
+    }
+  }
+
+  // Ensure we always have a single empty entry, it simplifies the emission
+  // logic so it doesn't need to handle the case where there are no perms.
+  if (PermutationsToEmit.empty()) {
+    PermutationsToEmit.emplace_back();
+    return true;
+  }
+
+  return true;
 }
 
 std::string CombineRuleBuilder::makeAnonPatName(StringRef Prefix) const {
   return to_string("__anon_pat_" + Prefix + "_" + to_string(RuleID) + "_" +
                    to_string(AnonIDCnt++));
 }
 
-RuleMatcher &CombineRuleBuilder::addRuleMatcher(Twine AdditionalComment) {
+RuleMatcher &CombineRuleBuilder::addRuleMatcher(const PatternAlternatives &Alts,
+                                                Twine AdditionalComment) {
   auto &RM = OutRMs.emplace_back(RuleDef.getLoc());
   addFeaturePredicates(RM);
+  RM.setPermanentGISelFlags(GISF_IgnoreCopies);
   RM.addRequiredSimplePredicate(getIsEnabledPredicateEnumName(RuleID));
   const std::string AdditionalCommentStr = AdditionalComment.str();
-  RM.addAction<DebugCommentAction>(
-      "Combiner Rule #" + to_string(RuleID) + ": " + RuleDef.getName().str() +
-      (AdditionalCommentStr.empty() ? "" : "; " + AdditionalCommentStr));
+
+  std::string Comment;
+  raw_string_ostream CommentOS(Comment);
+  CommentOS << "Combiner Rule #" << RuleID << ": " << RuleDef.getName();
+  if (!Alts.empty()) {
+    CommentOS << " @ ";
+    print(CommentOS, Alts);
+  }
+  if (!AdditionalCommentStr.empty())
+    CommentOS << "; " << AdditionalCommentStr;
+  RM.addAction<DebugCommentAction>(Comment);
   return RM;
 }
 
+bool CombineRuleBuilder::addFeaturePredicates(RuleMatcher &M) {
+  if (!RuleDef.getValue("Predicates"))
+    return true;
+
+  ListInit *Preds = RuleDef.getValueAsListInit("Predicates");
+  for (Init *I : Preds->getValues()) {
+    if (DefInit *Pred = dyn_cast<DefInit>(I)) {
+      Record *Def = Pred->getDef();
+      if (!Def->isSubClassOf("Predicate")) {
+        ::PrintError(Def, "Unknown 'Predicate' Type");
+        return false;
+      }
+
+      if (Def->getValueAsString("CondString").empty())
+        continue;
+
+      if (SubtargetFeatures.count(Def) == 0) {
+        SubtargetFeatures.emplace(
+            Def, SubtargetFeatureInfo(Def, SubtargetFeatures.size()));
+      }
+
+      M.addRequiredFeature(Def);
+    }
+  }
+
+  return true;
+}
+
 bool CombineRuleBuilder::findRoots() {
+  const auto Finish = [&]() {
+    assert(MatchRoot);
+
+    if (hasOnlyCXXApplyPatterns())
+      return true;
+
+    auto *IPRoot = dyn_cast<InstructionPattern>(MatchRoot);
+    if (!IPRoot)
+      return true;
+
+    if (IPRoot->getNumInstDefs() == 0) {
+      // No defs to work with -> find the root using the pattern name.
+      auto It = ApplyPats.find(RootName);
+      if (It == ApplyPats.end()) {
+        PrintError("Cannot find root '" + RootName + "' in apply patterns!");
+        return false;
+      }
+
+      auto *ApplyRoot = dyn_cast<InstructionPattern>(It->second.get());
+      if (!ApplyRoot) {
+        PrintError("apply pattern root '" + RootName +
+                   "' must be an instruction pattern");
+        return false;
+      }
+      dbgs() << "ApplyRoot is " << ApplyRoot->getName() << "\n";
+      ApplyRoots.push_back(ApplyRoot);
+      return true;
+    }
+
+    // Collect all redefinitions of the MatchRoot's defs and put them in
+    // ApplyRoots.
+    for (unsigned K = 0; K < IPRoot->getNumInstDefs(); ++K) {
+      auto &O = IPRoot->getOperand(K);
+      assert(O.isDef() && O.isNamedOperand());
+      StringRef Name = O.getOperandName();
+
+      auto *ApplyRedef = RuleOpTable.lookup(Name).ApplyPat;
+      if (!ApplyRedef) {
+        PrintError("def of pattern root '" + Name +
+                   "' is not redefined in the apply pattern!");
+        PrintNote("match pattern root is '" + MatchRoot->getName() + "'");
+        return false;
+      }
+
+      ApplyRoots.push_back(ApplyRedef);
+    }
+
+    if (auto It = ApplyPats.find(RootName); It != ApplyPats.end()) {
+      if (find(ApplyRoots, It->second.get()) == ApplyRoots.end()) {
+        PrintError("apply pattern '" + RootName +
+                   "' is supposed to be a root but it does not redefine any of "
+                   "the defs of the match root");
+        return false;
+      }
+    }
+
+    return true;
+  };
+
   // Look by pattern name, e.g.
   //    (G_FNEG $x, $y):$root
-  if (auto It = MatchPats.find(RootName); It != MatchPats.end()) {
-    MatchRoot = It->second.get();
-    return true;
+  if (auto MatchPatIt = MatchPats.find(RootName);
+      MatchPatIt != MatchPats.end()) {
+    MatchRoot = MatchPatIt->second.get();
+    return Finish();
   }
 
   // Look by def:
   //    (G_FNEG $root, $y)
-  auto It = OperandTable.find(RootName);
-  if (It == OperandTable.end()) {
+  auto OpIt = RuleOpTable.find(RootName);
+  if (OpIt == RuleOpTable.end()) {
     PrintError("Cannot find root '" + RootName + "' in match patterns!");
     return false;
   }
 
-  if (!It->second.MatchPat) {
+  if (!OpIt->second.MatchPat) {
     PrintError("Cannot use live-in operand '" + RootName +
                "' as match pattern root!");
     return false;
   }
 
-  MatchRoot = It->second.MatchPat;
-  return true;
+  MatchRoot = OpIt->second.MatchPat;
+  return Finish();
 }
 
-bool CombineRuleBuilder::buildOperandsTable() {
+bool CombineRuleBuilder::buildRuleOperandsTable() {
   // Walk each instruction pattern
   for (auto &P : values(MatchPats)) {
     auto *IP = dyn_cast<InstructionPattern>(P.get());
-    if (!IP)
-      continue;
-    for (const auto &Operand : IP->operands()) {
-      // Create an entry, no matter if it's a use or a def.
-      auto &Entry = OperandTable[Operand.Name];
-
-      // We only need to do additional checking on defs, though.
-      if (!Operand.IsDef)
-        continue;
-
-      if (Entry.MatchPat) {
-        PrintError("Operand '" + Operand.Name +
-                   "' is defined multiple times in the 'match' patterns");
-        return false;
-      }
-      Entry.MatchPat = IP;
-    }
-  }
+    if (IP && !RuleOpTable.addMatchPattern(RuleDef.getLoc(), IP,
+                                           "in the 'match' patterns"))
 
   for (auto &P : values(ApplyPats)) {
-    auto *IP = dyn_cast<InstructionPattern>(P.get());
-    if (!IP)
-      continue;
-    for (const auto &Operand : IP->operands()) {
-      // Create an entry, no matter if it's a use or a def.
-      auto &Entry = OperandTable[Operand.Name];
-
-      // We only need to do additional checking on defs, though.
-      if (!Operand.IsDef)
-        continue;
-
-      if (!Entry.MatchPat) {
-        PrintError("Cannot define live-in operand '" + Operand.Name +
-                   "' in the 'apply' pattern");
-        return false;
-      }
-      if (Entry.ApplyPat) {
-        PrintError("Operand '" + Operand.Name +
-                   "' is defined multiple times in the 'apply' patterns");
-        return false;
-      }
-      Entry.ApplyPat = IP;
-    }
-  }
+    // Only CodeGenInstructionPatterns should be in the "apply" pattern.
+    auto *IP = dyn_cast<CodeGenInstructionPattern>(P.get());
+    if (IP && !RuleOpTable.addApplyPattern(RuleDef.getLoc(), IP))
 
   return true;
 }
 
-bool CombineRuleBuilder::parseDefs(DagInit &Def) {
+bool CombineRuleBuilder::parseDefs(const DagInit &Def) {
   if (Def.getOperatorAsDef(RuleDef.getLoc())->getName() != "defs") {
     PrintError("Expected defs operator");
     return false;
   }
 
   SmallVector<StringRef> Roots;
   for (unsigned I = 0, E = Def.getNumArgs(); I < E; ++I) {
     if (isSpecificDef(*Def.getArg(I), "root")) {
bool CombineRuleBuilder::parseDefs(const DagInit &Def) {
 
   RootName = Roots.front();
 
   // Assign variables to all MatchDatas.
   AssignMatchDataVariables(MatchDatas);
   return true;
 }
 
-bool CombineRuleBuilder::parseMatch(DagInit &Match) {
-  if (Match.getOperatorAsDef(RuleDef.getLoc())->getName() != "match") {
-    PrintError("Expected match operator");
+bool CombineRuleBuilder::parsePatternList(
+    const DagInit &List,
+    function_ref<bool(std::unique_ptr<Pattern>)> ParseAction,
+    StringRef Operator, ArrayRef<SMLoc> DiagLoc) const {
+  if (List.getOperatorAsDef(RuleDef.getLoc())->getName() != Operator) {
+    ::PrintError(DiagLoc, "Expected " + Operator + " operator");
     return false;
   }
 
-  if (Match.getNumArgs() == 0) {
-    PrintError("Matcher is empty");
+  if (List.getNumArgs() == 0) {
+    ::PrintError(DiagLoc, Operator + " pattern list is empty");
     return false;
   }
 
   // The match section consists of a list of matchers and predicates. Parse each
   // one and add the equivalent GIMatchDag nodes, predicates, and edges.
-  bool HasOpcodeMatcher = false;
-  for (unsigned I = 0; I < Match.getNumArgs(); ++I) {
-    Init *Arg = Match.getArg(I);
-    std::string Name = Match.getArgName(I)
-                           ? Match.getArgName(I)->getValue().str()
-                           : makeAnonPatName("match");
+  for (unsigned I = 0; I < List.getNumArgs(); ++I) {
+    Init *Arg = List.getArg(I);
+    std::string Name = List.getArgName(I) ? List.getArgName(I)->getValue().str()
+                                          : makeAnonPatName(Operator);
 
-    if (MatchPats.contains(Name)) {
-      PrintError("'" + Name + "' match pattern defined more than once!");
+    if (auto Pat = parseInstructionPattern(*Arg, Name)) {
+      if (!ParseAction(std::move(Pat)))
         return false;
-    }
-
-    if (auto Pat = parseInstructionMatcher(*Arg, Name)) {
-      MatchPats[Pat->getName()] = std::move(Pat);
       continue;
     }
 
     if (auto Pat = parseWipMatchOpcodeMatcher(*Arg, Name)) {
-      if (HasOpcodeMatcher) {
-        PrintError("wip_opcode_match can only be present once");
+      if (!ParseAction(std::move(Pat)))
         return false;
-      }
-      HasOpcodeMatcher = true;
-      MatchPats[Pat->getName()] = std::move(Pat);
       continue;
     }
 
     // Parse arbitrary C++ code
     if (const auto *StringI = dyn_cast<StringInit>(Arg)) {
-      auto CXXPat =
-          std::make_unique<CXXPattern>(*StringI, Name, /*IsApply*/ false);
-      if (!CXXPat->getRawCode().contains("return ")) {
-        PrintWarning(RuleDef.getLoc(),
-                     "'match' C++ code does not seem to return!");
-      }
-      MatchPats[CXXPat->getName()] = std::move(CXXPat);
-      continue;
-    }
-
-    // TODO: don't print this on, e.g. bad operand count in inst pat
-    PrintError("Expected a subclass of GIMatchKind or a sub-dag whose "
-               "operator is either of a GIMatchKindWithArgs or Instruction");
-    PrintNote("Pattern was `" + Arg->getAsString() + "'");
-    return false;
-  }
-
-  return true;
-}
-
-bool CombineRuleBuilder::parseApply(DagInit &Apply) {
-  // Currently we only support C++ :(
-  if (Apply.getOperatorAsDef(RuleDef.getLoc())->getName() != "apply") {
-    PrintError("Expected 'apply' operator in Apply DAG");
+      auto CXXPat = std::make_unique<CXXPattern>(*StringI, Name);
+      if (!ParseAction(std::move(CXXPat)))
         return false;
+      continue;
     }
 
-  if (Apply.getNumArgs() != 1) {
-    PrintError("Expected exactly 1 argument in 'apply'");
+    ::PrintError(DiagLoc,
+                 "Failed to parse pattern: '" + Arg->getAsString() + "'");
     return false;
   }
 
-  const StringInit *Code = dyn_cast<StringInit>(Apply.getArg(0));
-  auto Pat = std::make_unique<CXXPattern>(*Code, makeAnonPatName("apply"),
-                                          /*IsApply*/ true);
-  ApplyPats[Pat->getName()] = std::move(Pat);
   return true;
 }
 
 std::unique_ptr<Pattern>
-CombineRuleBuilder::parseInstructionMatcher(const Init &Arg, StringRef Name) {
-  const DagInit *Matcher = getDagWithOperatorOfSubClass(Arg, "Instruction");
-  if (!Matcher)
+CombineRuleBuilder::parseInstructionPattern(const Init &Arg,
+                                            StringRef Name) const {
+  const DagInit *DagPat = dyn_cast<DagInit>(&Arg);
+  if (!DagPat)
     return nullptr;
 
-  auto &Instr = CGT.getInstruction(Matcher->getOperatorAsDef(RuleDef.getLoc()));
-  auto Pat = std::make_unique<InstructionPattern>(Instr, Name);
+  std::unique_ptr<InstructionPattern> Pat;
+  if (const DagInit *IP = getDagWithOperatorOfSubClass(Arg, "Instruction")) {
+    auto &Instr = CGT.getInstruction(IP->getOperatorAsDef(RuleDef.getLoc()));
+    Pat = std::make_unique<CodeGenInstructionPattern>(Instr, Name);
+  } else if (const DagInit *PFP =
+                 getDagWithOperatorOfSubClass(Arg, PatFragClassName)) {
+    const Record *Def = PFP->getOperatorAsDef(RuleDef.getLoc());
+    const PatFrag *PF = parsePatFrag(Def);
+    if (!PF)
+      return nullptr; // Already diagnosed by parsePatFrag
+    Pat = std::make_unique<PatFragPattern>(*PF, Name);
+  } else {
+    return nullptr;
+  }
 
-  for (const auto &NameInit : Matcher->getArgNames())
-    Pat->addOperand(NameInit->getAsUnquotedString());
+  for (unsigned K = 0; K < DagPat->getNumArgs(); ++K) {
+    if (!parseInstructionPatternOperand(*Pat, DagPat->getArg(K),
+                                        DagPat->getArgName(K)))
+      return nullptr;
+  }
 
   if (!Pat->checkSemantics(RuleDef.getLoc()))
     return nullptr;
 
   return std::move(Pat);
 }
 
 std::unique_ptr<Pattern>
 CombineRuleBuilder::parseWipMatchOpcodeMatcher(const Init &Arg,
-                                               StringRef Name) {
+                                               StringRef Name) const {
   const DagInit *Matcher = getDagWithSpecificOperator(Arg, "wip_match_opcode");
   if (!Matcher)
     return nullptr;
 
   if (Matcher->getNumArgs() == 0) {
     PrintError("Empty wip_match_opcode");
     return nullptr;
   }
for (const auto &Arg : Matcher->getArgs()) {
 
     PrintError("Arguments to wip_match_opcode must be instructions");
     return nullptr;
   }
 
   return std::move(Result);
 }
 
+bool CombineRuleBuilder::parseInstructionPatternOperand(
+    InstructionPattern &IP, const Init *OpInit,
+    const StringInit *OpName) const {
+  const auto ParseErr = [&]() {
+    PrintError("cannot parse operand '" + OpInit->getAsUnquotedString() + "' ");
+    if (OpName)
+      PrintNote("operand name is '" + OpName->getAsUnquotedString() + "'");
+    return false;
+  };
+
+  // untyped immediate, e.g. 0
+  if (const auto *IntImm = dyn_cast<IntInit>(OpInit)) {
+    std::string Name = OpName ? OpName->getAsUnquotedString() : "";
+    IP.addOperand(IntImm->getValue(), Name, /*Type=*/nullptr);
+    return true;
+  }
+
+  // typed immediate, e.g. (i32 0)
+  if (const auto *DagOp = dyn_cast<DagInit>(OpInit)) {
+    if (DagOp->getNumArgs() != 1)
+      return ParseErr();
+
+    Record *ImmTy = DagOp->getOperatorAsDef(RuleDef.getLoc());
+    if (!ImmTy->isSubClassOf("ValueType")) {
+      PrintError("cannot parse immediate '" + OpInit->getAsUnquotedString() +
+                 "', '" + ImmTy->getName() + "' is not a ValueType!");
+      return false;
+    }
+
+    if (!IP.hasAllDefs()) {
+      PrintError("out operand of '" + IP.getInstName() +
+                 "' cannot be an immediate");
+      return false;
+    }
+
+    const auto *Val = dyn_cast<IntInit>(DagOp->getArg(0));
+    if (!Val)
+      return ParseErr();
+
+    std::string Name = OpName ? OpName->getAsUnquotedString() : "";
+    IP.addOperand(Val->getValue(), Name, ImmTy);
+    return true;
+  }
+
+  // Typed operand e.g. $x/$z in (G_FNEG $x, $z)
+  if (auto *DefI = dyn_cast<DefInit>(OpInit)) {
+    if (!OpName) {
+      PrintError("expected an operand name after '" + OpInit->getAsString() +
+                 "'");
+      return false;
+    }
+    const Record *Def = DefI->getDef();
+    if (!Def->isSubClassOf("ValueType")) {
+      PrintError("invalid operand type: '" + Def->getName() +
+                 "' is not a ValueType");
+      return false;
+    }
+    IP.addOperand(OpName->getAsUnquotedString(), Def);
+    return true;
+  }
+
+  // Untyped operand e.g. $x/$z in (G_FNEG $x, $z)
+  if (isa<UnsetInit>(OpInit)) {
+    assert(OpName && "Unset w/ no OpName?");
+    IP.addOperand(OpName->getAsUnquotedString(), /*Type=*/nullptr);
+    return true;
+  }
+
+  return ParseErr();
+}
+
+std::unique_ptr<PatFrag>
+CombineRuleBuilder::parsePatFragImpl(const Record *Def) const {
+  auto StackTrace = PrettyStackTraceParse(*Def);
+  if (!Def->isSubClassOf(PatFragClassName))
+    return nullptr;
+
+  const DagInit *Ins = Def->getValueAsDag("InOperands");
+  if (Ins->getOperatorAsDef(Def->getLoc())->getName() != "ins") {
+    ::PrintError(Def, "expected 'ins' operator for " + PatFragClassName +
+                          " in operands list");
+    return nullptr;
+  }
+
+  const DagInit *Outs = Def->getValueAsDag("OutOperands");
+  if (Outs->getOperatorAsDef(Def->getLoc())->getName() != "outs") {
+    ::PrintError(Def, "expected 'outs' operator for " + PatFragClassName +
+                          " out operands list");
+    return nullptr;
+  }
+
+  auto Result = std::make_unique<PatFrag>(*Def);
+  if (!parsePatFragParamList(Def->getLoc(), *Outs,
+                             [&](StringRef Name, PatFrag::ParamKind Kind) {
+                               Result->addOutParam(Name, Kind);
+                               return true;
+                             }))
+    return nullptr;
+
+  if (!parsePatFragParamList(Def->getLoc(), *Ins,
+                             [&](StringRef Name, PatFrag::ParamKind Kind) {
+                               Result->addInParam(Name, Kind);
+                               return true;
+                             }))
+    return nullptr;
+
+  const ListInit *Alts = Def->getValueAsListInit("Alternatives");
+  for (const Init *Alt : *Alts) {
+    const auto *PatDag = dyn_cast<DagInit>(Alt);
+    if (!PatDag) {
+      ::PrintError(Def, "expected dag init for PatFrag pattern alternative");
+      return nullptr;
+    }
+
+    PatFrag::Alternative &A = Result->addAlternative();
+    const auto AddPat = [&](std::unique_ptr<Pattern> Pat) {
+      A.Pats.push_back(std::move(Pat));
+      return true;
+    };
+
+    if (!parsePatternList(*PatDag, AddPat, "pattern", Def->getLoc()))
+      return nullptr;
+  }
+
+  if (!Result->buildOperandsTables() || !Result->checkSemantics())
+    return nullptr;
+
+  return Result;
+}
+
+bool CombineRuleBuilder::parsePatFragParamList(
+    ArrayRef<SMLoc> DiagLoc, const DagInit &OpsList,
+    function_ref<bool(StringRef, PatFrag::ParamKind)> ParseAction) const {
+  for (unsigned K = 0; K < OpsList.getNumArgs(); ++K) {
+    const StringInit *Name = OpsList.getArgName(K);
+    const Init *Ty = OpsList.getArg(K);
+
+    if (!Name) {
+      ::PrintError(DiagLoc, "all operands must be named'");
+      return false;
+    }
+    const std::string NameStr = Name->getAsUnquotedString();
+
+    PatFrag::ParamKind OpKind;
+    if (isSpecificDef(*Ty, "gi_imm"))
+      OpKind = PatFrag::PK_Imm;
+    else if (isSpecificDef(*Ty, "root"))
+      OpKind = PatFrag::PK_Root;
+    else if (isa<UnsetInit>(Ty) ||
+             isSpecificDef(*Ty, "gi_mo")) // no type = gi_mo.
+      OpKind = PatFrag::PK_MachineOperand;
+    else {
+      ::PrintError(
+          DiagLoc,
+          "'" + NameStr +
+              "' operand type was expected to be 'root', 'gi_imm' or 'gi_mo'");
+      return false;
+    }
+
+    if (!ParseAction(NameStr, OpKind))
+      return false;
+  }
+
+  return true;
+}
+
+const PatFrag *CombineRuleBuilder::parsePatFrag(const Record *Def) const {
+  // Cache already parsed PatFrags to avoid doing extra work.
+  static DenseMap<const Record *, std::unique_ptr<PatFrag>> ParsedPatFrags;
+
+  auto It = ParsedPatFrags.find(Def);
+  if (It != ParsedPatFrags.end()) {
+    SeenPatFrags.insert(It->second.get());
+    return It->second.get();
+  }
+
+  std::unique_ptr<PatFrag> NewPatFrag = parsePatFragImpl(Def);
+  if (!NewPatFrag) {
+    ::PrintError(Def, "Could not parse " + PatFragClassName + " '" +
+                          Def->getName() + "'");
+    // Put a nullptr in the map so we don't attempt parsing this again.
+    ParsedPatFrags[Def] = nullptr;
+    return nullptr;
+  }
+
+  const auto *Res = NewPatFrag.get();
+  ParsedPatFrags[Def] = std::move(NewPatFrag);
+  SeenPatFrags.insert(Res);
+  return Res;
+}
+
 bool CombineRuleBuilder::emitMatchPattern(CodeExpansions &CE,
+                                          const PatternAlternatives &Alts,
                                           const InstructionPattern &IP) {
-  auto &M = addRuleMatcher();
+  auto StackTrace = PrettyStackTraceEmit(RuleDef, &IP);
+
+  auto &M = addRuleMatcher(Alts);
   InstructionMatcher &IM = M.addInstructionMatcher("root");
   declareInstExpansion(CE, IM, IP.getName());
 
   DenseSet<const Pattern *> SeenPats;
-  if (!emitInstructionMatchPattern(CE, M, IM, IP, SeenPats))
+  if (const auto *CGP = dyn_cast<CodeGenInstructionPattern>(&IP)) {
+    if (!emitCodeGenInstructionMatchPattern(CE, Alts, RuleOpTable, M, IM, *CGP,
+                                            SeenPats))
+      return false;
+  } else if (const auto *PFP = dyn_cast<PatFragPattern>(&IP)) {
+    if (!PFP->getPatFrag().canBeMatchRoot()) {
+      PrintError("cannot use '" + PFP->getInstName() + " as match root");
+      return false;
+    }
+
+    if (!emitPatFragMatchPattern(CE, Alts, M, &IM, *PFP, SeenPats))
       return false;
+  } else
+    llvm_unreachable("Unknown kind of InstructionPattern!");
 
   // Emit remaining patterns
   for (auto &Pat : values(MatchPats)) {
     if (SeenPats.contains(Pat.get()))
       continue;
 
     switch (Pat->getKind()) {
     case Pattern::K_AnyOpcode:
       PrintError("wip_match_opcode can not be used with instruction patterns!");
       return false;
-    case Pattern::K_Inst:
+    case Pattern::K_PatFrag: {
+      if (!emitPatFragMatchPattern(CE, Alts, M, /*IM*/ nullptr,
+                                   *cast<PatFragPattern>(Pat.get()), SeenPats))
+        return false;
+      continue;
+    }
+    case Pattern::K_CodeGenInstruction:
       cast<InstructionPattern>(Pat.get())->reportUnreachable(RuleDef.getLoc());
       return false;
     case Pattern::K_CXX: {
-      addCXXPredicate(IM, CE, *cast<CXXPattern>(Pat.get()));
+      addCXXPredicate(M, CE, *cast<CXXPattern>(Pat.get()), Alts);
       continue;
     }
     default:
       llvm_unreachable("unknown pattern kind!");
     }
   }
 
   return emitApplyPatterns(CE, M);
 }
 
 bool CombineRuleBuilder::emitMatchPattern(CodeExpansions &CE,
+                                          const PatternAlternatives &Alts,
                                           const AnyOpcodePattern &AOP) {
+  auto StackTrace = PrettyStackTraceEmit(RuleDef, &AOP);
 
   for (const CodeGenInstruction *CGI : AOP.insts()) {
-    auto &M = addRuleMatcher("wip_match_opcode alternative '" +
+    auto &M = addRuleMatcher(Alts, "wip_match_opcode '" +
                                        CGI->TheDef->getName() + "'");
 
     InstructionMatcher &IM = M.addInstructionMatcher(AOP.getName());
     declareInstExpansion(CE, IM, AOP.getName());
     // declareInstExpansion needs to be identical, otherwise we need to create a
     // CodeExpansions object here instead.
     assert(IM.getInsnVarID() == 0);
 
     IM.addPredicate<InstructionOpcodeMatcher>(CGI);
 
     // Emit remaining patterns.
     for (auto &Pat : values(MatchPats)) {
       if (Pat.get() == &AOP)
         continue;
 
       switch (Pat->getKind()) {
       case Pattern::K_AnyOpcode:
         PrintError("wip_match_opcode can only be present once!");
         return false;
-      case Pattern::K_Inst:
+      case Pattern::K_PatFrag: {
+        DenseSet<const Pattern *> SeenPats;
+        if (!emitPatFragMatchPattern(CE, Alts, M, /*IM*/ nullptr,
+                                     *cast<PatFragPattern>(Pat.get()),
+                                     SeenPats))
+          return false;
+        continue;
+      }
+      case Pattern::K_CodeGenInstruction:
         cast<InstructionPattern>(Pat.get())->reportUnreachable(
             RuleDef.getLoc());
         return false;
       case Pattern::K_CXX: {
-        addCXXPredicate(IM, CE, *cast<CXXPattern>(Pat.get()));
+        addCXXPredicate(M, CE, *cast<CXXPattern>(Pat.get()), Alts);
         break;
       }
       default:
         llvm_unreachable("unknown pattern kind!");
       }
     }
 
     if (!emitApplyPatterns(CE, M))
       return false;
   }
 
   return true;
 }
 
+bool CombineRuleBuilder::emitPatFragMatchPattern(
+    CodeExpansions &CE, const PatternAlternatives &Alts, RuleMatcher &RM,
+    InstructionMatcher *IM, const PatFragPattern &PFP,
+    DenseSet<const Pattern *> &SeenPats) {
+  auto StackTrace = PrettyStackTraceEmit(RuleDef, &PFP);
+
+  if (SeenPats.contains(&PFP))
+    return true;
+  SeenPats.insert(&PFP);
+
+  const auto &PF = PFP.getPatFrag();
+
+  if (!IM) {
+    // When we don't have an IM, this means this PatFrag isn't reachable from
+    // the root. This is only acceptable if it doesn't define anything (e.g. a
+    // pure C++ PatFrag).
+    if (PF.num_out_params() != 0) {
+      PFP.reportUnreachable(RuleDef.getLoc());
+      return false;
+    }
+  } else {
+    // When an IM is provided, this is reachable from the root, and we're
+    // expecting to have output operands.
+    // TODO: If we want to allow for multiple roots we'll need a map of IMs
+    // then, and emission becomes a bit more complicated.
+    assert(PF.num_roots() == 1);
+  }
+
+  CodeExpansions PatFragCEs;
+  if (!PFP.mapInputCodeExpansions(CE, PatFragCEs, RuleDef.getLoc()))
+    return false;
+
+  // List of {ParamName, ArgName}.
+  // When all patterns have been emitted, find expansions in PatFragCEs named
+  // ArgName and add their expansion to CE using ParamName as the key.
+  SmallVector<std::pair<std::string, std::string>, 4> CEsToImport;
+
+  // Map parameter names to the actual argument.
+  const auto OperandMapper =
+      [&](const InstructionPattern::Operand &O) -> InstructionPattern::Operand {
+    if (!O.isNamedOperand())
+      return O;
+
+    StringRef ParamName = O.getOperandName();
+
+    // Not sure what to do with those tbh. They should probably never be here.
+    assert(!O.isNamedImmediate() && "TODO: handle named imms");
+    unsigned PIdx = PF.getParamIdx(ParamName);
+
+    // Map parameters to the argument values.
+    if (PIdx == (unsigned)-1) {
+      // This is a temp of the PatFragPattern, prefix the name to avoid
+      // conflicts.
+      return O.withNewName((PFP.getName() + "." + ParamName).str());
+    }
+
+    // The operand will be added to PatFragCEs's code expansions using the
+    // parameter's name. If it's bound to some operand during emission of the
+    // patterns, we'll want to add it to CE.
+    auto ArgOp = PFP.getOperand(PIdx);
+    if (ArgOp.isNamedOperand())
+      CEsToImport.emplace_back(ArgOp.getOperandName().str(), ParamName);
+
+    if (ArgOp.getType() && O.getType() && ArgOp.getType() != O.getType()) {
+      StringRef PFName = PF.getName();
+      PrintWarning("impossible type constraints: operand " + Twine(PIdx) +
+                   " of '" + PFP.getName() + "' has type '" +
+                   ArgOp.getType()->getName() + "', but '" + PFName +
+                   "' constrains it to '" + O.getType()->getName() + "'");
+      if (ArgOp.isNamedOperand())
+        PrintNote("operand " + Twine(PIdx) + " of '" + PFP.getName() +
+                  "' is '" + ArgOp.getOperandName() + "'");
+      if (O.isNamedOperand())
+        PrintNote("argument " + Twine(PIdx) + " of '" + PFName + "' is '" +
+                  ParamName + "'");
+    }
+
+    return ArgOp;
+  };
+
+  // PatFragPatterns are only made of InstructionPatterns or CXXPatterns.
+  // Emit instructions from the root.
+  const auto &FragAlt = PF.getAlternative(Alts.lookup(&PFP));
+  DenseSet<const Pattern *> PatFragSeenPats;
+  for (const auto &[Idx, InOp] : enumerate(PF.out_params())) {
+    if (InOp.Kind != PatFrag::PK_Root)
+      continue;
+
+    StringRef ParamName = InOp.Name;
+    const auto *Def = FragAlt.OpTable.getMatchDefPattern(ParamName);
+    assert(Def && "PatFrag::checkSemantics should have emitted an error if "
+                  "an out operand isn't defined!");
+    assert(isa<CodeGenInstructionPattern>(Def) &&
+           "Nested PatFrags not supported yet");
+
+    if (!emitCodeGenInstructionMatchPattern(
+            PatFragCEs, Alts, FragAlt.OpTable, RM, *IM,
+            *cast<CodeGenInstructionPattern>(Def), PatFragSeenPats,
+            OperandMapper))
+      return false;
+  }
+
+  // Emit leftovers.
+  for (const auto &Pat : FragAlt.Pats) {
+    if (PatFragSeenPats.contains(Pat.get()))
+      continue;
+
+    if (const auto *CXXPat = dyn_cast<CXXPattern>(Pat.get())) {
+      addCXXPredicate(RM, PatFragCEs, *CXXPat, Alts);
+      continue;
+    }
+
+    if (const auto *IP = dyn_cast<InstructionPattern>(Pat.get())) {
+      IP->reportUnreachable(PF.getLoc());
+      return false;
+    }
+
+    llvm_unreachable("Unexpected pattern kind in PatFrag");
+  }
+
+  for (const auto &[ParamName, ArgName] : CEsToImport) {
+    // Note: we're find if ParamName already exists. It just means it's been
+    // bound before, so we prefer to keep the first binding.
+    CE.declare(ParamName, PatFragCEs.lookup(ArgName));
+  }
+
+  return true;
+}
+
 bool CombineRuleBuilder::emitApplyPatterns(CodeExpansions &CE, RuleMatcher &M) {
+  if (hasOnlyCXXApplyPatterns()) {
+    for (auto &Pat : values(ApplyPats))
+      addCXXAction(M, CE, *cast<CXXPattern>(Pat.get()));
+    return true;
+  }
+
+  DenseSet<const Pattern *> SeenPats;
+
+  // Preallocate all of the temporary registers we need.
+  // This needs to be done first so all of the MakeTempRegisterActions are
+  // emitted first.
+  StringMap<unsigned> OperandToTempRegID;
+  if (!emitTempRegisterActions(CE, M, OperandToTempRegID))
+    return false;
+
+  for (auto *ApplyRoot : ApplyRoots) {
+    assert(isa<CodeGenInstructionPattern>(ApplyRoot) &&
+           "PatFragPatterns are not supported in apply patterns yet!");
+    if (!emitCodeGenInstructionApplyPattern(
+            CE, M, *cast<CodeGenInstructionPattern>(ApplyRoot), SeenPats,
+            OperandToTempRegID))
+      return false;
+  }
+
   for (auto &Pat : values(ApplyPats)) {
+    if (SeenPats.contains(Pat.get()))
+      continue;
+
     switch (Pat->getKind()) {
     case Pattern::K_AnyOpcode:
-    case Pattern::K_Inst:
-      llvm_unreachable("Unsupported pattern kind in output pattern!");
+      llvm_unreachable("Unexpected pattern in apply!");
+    case Pattern::K_PatFrag:
+      // TODO: We could support pure C++ PatFrags as a temporary thing.
+      llvm_unreachable("Unexpected pattern in apply!");
+    case Pattern::K_CodeGenInstruction:
+      cast<CodeGenInstructionPattern>(Pat.get())->reportUnreachable(
+          RuleDef.getLoc());
+      return false;
     case Pattern::K_CXX: {
-      CXXPattern *CXXPat = cast<CXXPattern>(Pat.get());
-      const auto &ExpandedCode = CXXPat->expandCode(CE, RuleDef.getLoc());
-      M.addAction<CustomCXXAction>(
-          ExpandedCode.getEnumNameWithPrefix(CXXApplyPrefix));
+      addCXXAction(M, CE, *cast<CXXPattern>(Pat.get()));
       continue;
     }
     default:
-      llvm_unreachable("Unknown pattern kind!");
+      llvm_unreachable("unknown pattern kind!");
+    }
+  }
+
+  return true;
+}
+
+bool CombineRuleBuilder::emitTempRegisterActions(
+    CodeExpansions &CE, RuleMatcher &M,
+    StringMap<unsigned> &OperandToTempRegID) const {
+  for (const auto &Pat : values(ApplyPats)) {
+    const auto *IP = dyn_cast<InstructionPattern>(Pat.get());
+    if (!IP)
+      continue;
+
+    for (const auto &O : IP->operands()) {
+      // Skip non-defs & rewrites of matched instructions.
+      if (!O.isDef() || RuleOpTable.getMatchDefPattern(O.getOperandName()))
+        continue;
+
+      // This is a brand new register.
+      StringRef OpName = O.getOperandName();
+      assert(OperandToTempRegID.count(OpName) == 0 &&
+             "Register has multiple defs?!");
+      const unsigned TempRegID = M.allocateTempRegID();
+      OperandToTempRegID[OpName] = TempRegID;
+      const Record *Ty = O.getType();
+      if (!Ty) {
+        PrintError("def of a new register '" + OpName +
+                   "' in the apply patterns must have a type");
+        return false;
+      }
+      declareTempRegExpansion(CE, TempRegID, OpName);
+      M.addAction<MakeTempRegisterAction>(getLLTCodeGenFromRecord(Ty),
+                                          TempRegID);
     }
   }
 
   return true;
 }
 
-bool CombineRuleBuilder::emitInstructionMatchPattern(
-    CodeExpansions &CE, RuleMatcher &M, InstructionMatcher &IM,
-    const InstructionPattern &P, DenseSet<const Pattern *> &SeenPats) {
+bool CombineRuleBuilder::emitCodeGenInstructionApplyPattern(
+    CodeExpansions &CE, RuleMatcher &M, const CodeGenInstructionPattern &P,
+    DenseSet<const Pattern *> &SeenPats,
+    const StringMap<unsigned> &OperandToTempRegID) {
+  auto StackTrace = PrettyStackTraceEmit(RuleDef, &P);
+
+  if (SeenPats.contains(&P))
+    return true;
+
+  SeenPats.insert(&P);
+
+  // First, render the uses.
+  for (auto &O : P.operands()) {
+    if (O.isDef() || !O.isNamedOperand())
+      continue;
+
+    StringRef OpName = O.getOperandName();
+    if (const auto *DefPat = RuleOpTable.getApplyDefPattern(OpName)) {
+      if (!emitCodeGenInstructionApplyPattern(CE, M, *DefPat, SeenPats,
+                                              OperandToTempRegID))
+        return false;
+    } else {
+      // If we have no def, check this exists in the MatchRoot.
+      if (!O.isNamedImmediate() && !RuleOpTable.isMatchLiveInOrDef(OpName)) {
+        PrintError("invalid output operand '" + OpName +
+                   "': operand is not a live-in of the match pattern, and it "
+                   "has no definition");
+        return false;
+      }
+    }
+  }
+
+  // Now render this inst.
+  auto &DstMI =
+      M.addAction<BuildMIAction>(M.allocateOutputInsnID(), &P.getInst());
+
+  for (auto &O : P.operands()) {
+    if (O.isNamedImmediate()) {
+      PrintError("invalid output operand '" + O.getOperandName() +
+                 "': output immediates cannot be named");
+      PrintNote("while emitting pattern '" + P.getName() + "' (" +
+                P.getInstName() + ")");
+      return false;
+    }
+
+    if (O.hasImmValue()) {
+      if (!O.getType()) {
+        PrintError("output immediates must be typed!");
+        PrintNote("while emitting pattern '" + P.getName() + "' (" +
+                  P.getInstName() + ")");
+        return false;
+      }
+
+      DstMI.addRenderer<ImmRenderer>(O.getImmValue(),
+                                     getLLTCodeGenFromRecord(O.getType()));
+      continue;
+    }
+
+    StringRef OpName = O.getOperandName();
+    if (!O.isDef()) {
+      if (auto It = OperandToTempRegID.find(OpName);
+          It != OperandToTempRegID.end())
+        DstMI.addRenderer<TempRegRenderer>(It->second);
+      else {
+        // This should be a match live in or a redef of a matched instr.
+        // If not, it means the register hasn't been allocated.
+        assert(RuleOpTable.isMatchLiveInOrDef(OpName) &&
+               !RuleOpTable.getApplyDefPattern(OpName) &&
+               "Temp reg not emitted yet!");
+        DstMI.addRenderer<CopyRenderer>(OpName);
+      }
+      continue;
+    }
+
+    if (const auto *MatchDef = RuleOpTable.getMatchDefPattern(OpName)) {
+      // TODO: Handle this. We need to mutate the instr, or delete the old
+      // one.
+      //       Likewise, we also need to ensure we redef everything, if the
+      //       instr has more than one def, we need to redef all or nothing.
+      if (MatchDef != MatchRoot) {
+        PrintError("redefining an instruction other than the root is not "
+                   "supported (operand '" +
+                   OpName + "')");
+        return false;
+      }
+      // redef of a match
+      DstMI.addRenderer<CopyRenderer>(OpName);
+    } else
+      DstMI.addRenderer<TempRegRenderer>(OperandToTempRegID.lookup(OpName));
+  }
+
+  // TODO: works?
+  DstMI.chooseInsnToMutate(M);
+  declareInstExpansion(CE, DstMI, P.getName());
+
+  return true;
+}
+
+bool isLiteralImm(const InstructionPattern &P, unsigned OpIdx) {
+  if (const auto *CGP = dyn_cast<CodeGenInstructionPattern>(&P)) {
+    StringRef InstName = CGP->getInst().TheDef->getName();
+    return (InstName == "G_CONSTANT" || InstName == "G_FCONSTANT") &&
+           OpIdx == 1;
+  }
+
+  llvm_unreachable("TODO");
+}
+
+bool CombineRuleBuilder::emitCodeGenInstructionMatchPattern(
+    CodeExpansions &CE, const PatternAlternatives &Alts,
+    const OperandTable &OpTable, RuleMatcher &M, InstructionMatcher &IM,
+    const CodeGenInstructionPattern &P, DenseSet<const Pattern *> &SeenPats,
+    OperandMapperFnRef OperandMapper) {
+  auto StackTrace = PrettyStackTraceEmit(RuleDef, &P);
+
   if (SeenPats.contains(&P))
     return true;
 
   SeenPats.insert(&P);
 
   IM.addPredicate<InstructionOpcodeMatcher>(&P.getInst());
   declareInstExpansion(CE, IM, P.getName());
 
-  unsigned OpIdx = 0;
-  for (auto &O : P.operands()) {
-    auto &OpTableEntry = OperandTable.find(O.Name)->second;
+  for (const auto &[Idx, OriginalO] : enumerate(P.operands())) {
+    // Remap the operand. This is used when emitting InstructionPatterns inside
+    // PatFrags, so it can remap them to the arguments passed to the pattern.
+    //
+    // We use the remapped operand to emit immediates, and for the symbolic
+    // operand names (in IM.addOperand). CodeExpansions and OperandTable lookups
+    // still use the original name.
+    //
+    // The "def" flag on the remapped operand is always ignored.
+    auto RemappedO = OperandMapper(OriginalO);
+    assert(RemappedO.isNamedOperand() == OriginalO.isNamedOperand() &&
+           "Cannot remap an unnamed operand to a named one!");
 
+    const auto OpName =
+        RemappedO.isNamedOperand() ? RemappedO.getOperandName().str() : "";
     OperandMatcher &OM =
-        IM.addOperand(OpIdx++, O.Name, AllocatedTemporariesBaseID++);
-    declareOperandExpansion(CE, OM, O.Name);
+        IM.addOperand(Idx, OpName, AllocatedTemporariesBaseID++);
+    if (!OpName.empty())
+      declareOperandExpansion(CE, OM, OriginalO.getOperandName());
+
+    // Handle immediates.
+    if (RemappedO.hasImmValue()) {
+      if (isLiteralImm(P, Idx))
+        OM.addPredicate<LiteralIntOperandMatcher>(RemappedO.getImmValue());
+      else
+        OM.addPredicate<ConstantIntOperandMatcher>(RemappedO.getImmValue());
+    }
 
-    if (O.IsDef)
+    // Handle typed operands, but only bother to check if it hasn't been done
+    // before.
+    //
+    // getOperandMatcher will always return the first OM to have been created
+    // for that Operand. "OM" here is always a new OperandMatcher.
+    //
+    // Always emit a check for unnamed operands.
+    if (OpName.empty() ||
+        !M.getOperandMatcher(OpName).contains<LLTOperandMatcher>()) {
+      if (const Record *Ty = RemappedO.getType())
+        OM.addPredicate<LLTOperandMatcher>(getLLTCodeGenFromRecord(Ty));
+    }
+
+    // Stop here if the operand is a def, or if it had no name.
+    if (OriginalO.isDef() || !OriginalO.isNamedOperand())
       continue;
 
-    if (InstructionPattern *DefPat = OpTableEntry.MatchPat) {
-      auto InstOpM = OM.addPredicate<InstructionOperandMatcher>(M, O.Name);
-      if (!InstOpM) {
-        // TODO: copy-pasted from GlobalISelEmitter.cpp. Is it still relevant
-        // here?
-        PrintError("Nested instruction '" + DefPat->getName() +
-                   "' cannot be the same as another operand '" + O.Name + "'");
+    const auto *DefPat = OpTable.getMatchDefPattern(OriginalO.getOperandName());
+    if (!DefPat)
+      continue;
+
+    if (OriginalO.hasImmValue()) {
+      assert(!OpName.empty());
+      // This is a named immediate that also has a def, that's not okay.
+      // e.g.
+      //    (G_SEXT $y, (i32 0))
+      //    (COPY $x, 42:$y)
+      PrintError("'" + OpName +
+                 "' is a named immediate, it cannot be defined by another "
+                 "instruction");
+      PrintNote("'" + OpName + "' is defined by '" + DefPat->getName() + "'");
+      return false;
+    }
+
+    // From here we know that the operand defines an instruction, and we need to
+    // emit it.
+    auto InstOpM =
+        OM.addPredicate<InstructionOperandMatcher>(M, DefPat->getName());
+                 "' cannot be the same as another operand '" +
+                 OriginalO.getOperandName() + "'");
+      return false;
+    }
+
+    auto &IM = (*InstOpM)->getInsnMatcher();
+    if (const auto *CGIDef = dyn_cast<CodeGenInstructionPattern>(DefPat)) {
+      if (!emitCodeGenInstructionMatchPattern(CE, Alts, OpTable, M, IM, *CGIDef,
+                                              SeenPats, OperandMapper))
         return false;
-      }
+      continue;
 
-      if (!emitInstructionMatchPattern(CE, M, (*InstOpM)->getInsnMatcher(),
-                                       *DefPat, SeenPats))
+    if (const auto *PFPDef = dyn_cast<PatFragPattern>(DefPat)) {
+      if (!emitPatFragMatchPattern(CE, Alts, M, &IM, *PFPDef, SeenPats))
         return false;
+      continue;
     }
+
+    llvm_unreachable("unknown type of InstructionPattern");
   }
 
   return true;
 }
 
 //===- GICombinerEmitter --------------------------------------------------===//
 
-/// This class is essentially the driver. It fetches all TableGen records, calls
-/// CombineRuleBuilder to build the MatchTable's RuleMatchers, then creates the
-/// MatchTable & emits it. It also handles emitting all the supporting code such
-/// as the list of LLTs, the CXXPredicates, etc.
+/// Main implementation class. This emits the tablegenerated output.
+///
+/// It collects rules, uses `CombineRuleBuilder` to parse them and accumulate
+/// RuleMatchers, then takes all the necessary state/data from the various
+/// static storage pools and wires them together to emit the match table &
+/// associated function/data structures.
 class GICombinerEmitter final : public GlobalISelMatchTableExecutorEmitter {
   RecordKeeper &Records;
   StringRef Name;
   const CodeGenTarget &Target;
   Record *Combiner;
   unsigned NextRuleID = 0;
 
   // List all combine rules (ID, name) imported.
OS << "bool " << getClassName()
      << "  NewMIVector OutMIs;\n"
      << "  State.MIs.clear();\n"
      << "  State.MIs.push_back(&I);\n"
      << "  " << MatchDataInfo::StructName << " = "
      << MatchDataInfo::StructTypeName << "();\n\n"
      << "  if (executeMatchTable(*this, OutMIs, State, ExecInfo"
      << ", getMatchTable(), *ST.getInstrInfo(), MRI, "
         "*MRI.getTargetRegisterInfo(), *ST.getRegBankInfo(), AvailableFeatures"
-     << ", /*CoverageInfo*/ nullptr)) {\n"
+     << ", /*CoverageInfo*/ nullptr, &Observer)) {\n"
      << "    return true;\n"
      << "  }\n\n"
      << "  return false;\n"
      << "}\n\n";
 }
 
 void GICombinerEmitter::emitMIPredicateFns(raw_ostream &OS) {
-  auto MatchCode = getSorted(AllCXXMatchCode);
+  auto MatchCode = CXXPredicateCode::getAllMatchCode();
   emitMIPredicateFnsImpl<const CXXPredicateCode *>(
       OS, "", ArrayRef<const CXXPredicateCode *>(MatchCode),
       [](const CXXPredicateCode *C) -> StringRef { return C->BaseEnumName; },
       [](const CXXPredicateCode *C) -> StringRef { return C->Code; });
 }
 
 void GICombinerEmitter::emitI64ImmPredicateFns(raw_ostream &OS) {
   // Unused, but still needs to be called.
OS << "bool " << getClassName()
      << "::testSimplePredicate(unsigned Predicate) const {\n"
      << "    return RuleConfig.isRuleEnabled(Predicate - "
         "GICXXPred_Invalid - "
         "1);\n"
      << "}\n";
 }
 
 void GICombinerEmitter::emitRunCustomAction(raw_ostream &OS) {
-  const auto ApplyCode = getSorted(AllCXXApplyCode);
+  const auto ApplyCode = CXXPredicateCode::getAllApplyCode();
 
   if (!ApplyCode.empty()) {
     OS << "enum {\n";
     std::string EnumeratorSeparator = " = GICXXCustomAction_Invalid + 1,\n";
     for (const auto &Apply : ApplyCode) {
       OS << "  " << Apply->getEnumNameWithPrefix(CXXApplyPrefix)
          << EnumeratorSeparator;
       EnumeratorSeparator = ",\n";
     }
     OS << "};\n";
   }
 
   OS << "void " << getClassName()
-     << "::runCustomAction(unsigned ApplyID, const MatcherState &State) const "
+     << "::runCustomAction(unsigned ApplyID, const MatcherState &State, "
+        "NewMIVector &OutMIs) const "
         "{\n";
   if (!ApplyCode.empty()) {
     OS << "  switch(ApplyID) {\n";
     for (const auto &Apply : ApplyCode) {
       OS << "  case " << Apply->getEnumNameWithPrefix(CXXApplyPrefix) << ":{\n"
-         << "    " << Apply->Code << "\n"
+         << "    " << join(split(Apply->Code, "\n"), "\n    ") << "\n"
          << "    return;\n";
       OS << "  }\n";
     }
     OS << "}\n";
   }
   OS << "  llvm_unreachable(\"Unknown Apply Action\");\n"
      << "}\n";
 }
void GICombinerEmitter::gatherRules(
     std::vector<RuleMatcher> &ActiveRules,
     const std::vector<Record *> &&RulesAndGroups) {
   for (Record *R : RulesAndGroups) {
     if (R->isValueUnset("Rules")) {
       AllCombineRules.emplace_back(NextRuleID, R->getName().str());
       CombineRuleBuilder CRB(Target, SubtargetFeatures, *R, NextRuleID++,
                              ActiveRules);
 
-      if (!CRB.parseAll())
+      if (!CRB.parseAll()) {
+        assert(ErrorsPrinted && "Parsing failed without errors!");
         continue;
+      }
 
       if (StopAfterParse) {
         CRB.print(outs());
         continue;
       }
 
-      if (!CRB.emitRuleMatchers())
+      if (!CRB.emitRuleMatchers()) {
+        assert(ErrorsPrinted && "Emission failed without errors!");
         continue;
+      }
     } else
       gatherRules(ActiveRules, R->getValueAsListOfDefs("Rules"));
   }
 }
 
 void GICombinerEmitter::run(raw_ostream &OS) {
   Records.startTimer("Gather rules");
   std::vector<RuleMatcher> Rules;
void GICombinerEmitter::run(raw_ostream &OS) {
   const MatchTable Table = buildMatchTable(Rules);
 
   Records.startTimer("Emit combiner");
 
   emitSourceFileHeader(getClassName().str() + " Combiner Match Table", OS);
 
   // Unused
   std::vector<StringRef> CustomRendererFns;
-  // Unused, but hack to avoid empty declarator
-  std::vector<LLTCodeGen> TypeObjects = {LLTCodeGen(LLT::scalar(1))};
   // Unused
   std::vector<Record *> ComplexPredicates;
 
+  std::vector<LLTCodeGen> TypeObjects;
+  append_range(TypeObjects, KnownTypes);
+  llvm::sort(TypeObjects);
+
+  // Hack: Avoid empty declarator.
+  if (TypeObjects.empty())
+    TypeObjects.push_back(LLT::scalar(1));
+
   // GET_GICOMBINER_DEPS, which pulls in extra dependencies.
   OS << "#ifdef GET_GICOMBINER_DEPS\n"
      << "#include \"llvm/ADT/SparseBitVector.h\"\n"
      << "namespace llvm {\n"
      << "extern cl::OptionCategory GICombinerOptionCategory;\n"
      << "} // end namespace llvm\n"
      << "#endif // ifdef GET_GICOMBINER_DEPS\n\n";
 
   // GET_GICOMBINER_TYPES, which needs to be included before the declaration of
void GICombinerEmitter::run(raw_ostream &OS) {
   emitTemporariesInit(OS, MaxTemporaries, "GET_GICOMBINER_CONSTRUCTOR_INITS");
 }
 
 } // end anonymous namespace
 
 //===----------------------------------------------------------------------===//
 
 static void EmitGICombiner(RecordKeeper &RK, raw_ostream &OS) {
+  EnablePrettyStackTrace();
   CodeGenTarget Target(RK);
 
   if (SelectedCombiners.empty())
     PrintFatalError("No combiners selected with -combiners");
   for (const auto &Combiner : SelectedCombiners) {
     Record *CombinerDef = RK.getDef(Combiner);
     if (!CombinerDef)
       PrintFatalError("Could not find " + Combiner);
     GICombinerEmitter(RK, Target, Combiner, CombinerDef).run(OS);
   }
 }
 
 static TableGen::Emitter::Opt X("gen-global-isel-combiner-matchtable",
                                 EmitGICombiner,
                                 "Generate GlobalISel combiner Match Table");

llvm/utils/TableGen/GlobalISelEmitter.cpp

OS << "bool " << getClassName() << "::testSimplePredicate(unsigned) const {\n"
      << "    llvm_unreachable(\"" + getClassName() +
             " does not support simple predicates!\");\n"
      << "  return false;\n"
      << "}\n";
 }
 
 void GlobalISelEmitter::emitRunCustomAction(raw_ostream &OS) {
   OS << "void " << getClassName()
-     << "::runCustomAction(unsigned, const MatcherState&) const {\n"
+     << "::runCustomAction(unsigned, const MatcherState&, NewMIVector &) const "
+        "{\n"
      << "    llvm_unreachable(\"" + getClassName() +
             " does not support custom C++ actions!\");\n"
      << "}\n";
 }
 
 void GlobalISelEmitter::run(raw_ostream &OS) {
   if (!UseCoverageFile.empty()) {
     RuleCoverage = CodeGenCoverage();

llvm/utils/TableGen/GlobalISelMatchTable.h

public:
   // exceeds the capacity. Otherwise, only invalidates the iterators from the
   // insertion point onwards.
   template <class Kind, class... Args>
   action_iterator insertAction(action_iterator InsertPt, Args &&...args) {
     return Actions.emplace(InsertPt,
                            std::make_unique<Kind>(std::forward<Args>(args)...));
   }
 
+  void setPermanentGISelFlags(GISelFlags V) { Flags = V; }
+
   // Update the active GISelFlags based on the GISelFlags Record R.
   // A SaveAndRestore object is returned so the old GISelFlags are restored
   // at the end of the scope.
   SaveAndRestore<GISelFlags> setGISelFlags(const Record *R);
   GISelFlags getGISelFlags() const { return Flags; }
 
   /// Define an instruction without emitting any code to do so.
   unsigned implicitlyDefineInsnVar(InstructionMatcher &Matcher);
public:
   iterator_range<typename PredicatesTy::iterator> predicates() {
     return make_range(predicates_begin(), predicates_end());
   }
   typename PredicatesTy::size_type predicates_size() const {
     return Predicates.size();
   }
   bool predicates_empty() const { return Predicates.empty(); }
 
+  template <typename Ty> bool contains() const {
+    return any_of(Predicates, [&](auto &P) { return isa<Ty>(P.get()); });
+  }
+
   std::unique_ptr<PredicateTy> predicates_pop_front() {
     std::unique_ptr<PredicateTy> Front = std::move(Predicates.front());
     Predicates.pop_front();
     Optimized = true;
     return Front;
   }
 
   void prependPredicate(std::unique_ptr<PredicateTy> &&Predicate) {
public:
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_TempRegister;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override;
 };
 
 /// Adds a specific immediate to the instruction being built.
+/// If a LLT is passed, a ConstantInt immediate is created instead.
 class ImmRenderer : public OperandRenderer {
 protected:
   unsigned InsnID;
   int64_t Imm;
+  std::optional<LLTCodeGen> CImmLLT;
 
 public:
   ImmRenderer(unsigned InsnID, int64_t Imm)
       : OperandRenderer(OR_Imm), InsnID(InsnID), Imm(Imm) {}
 
+  ImmRenderer(unsigned InsnID, int64_t Imm, const LLTCodeGen &CImmLLT)
+      : OperandRenderer(OR_Imm), InsnID(InsnID), Imm(Imm), CImmLLT(CImmLLT) {
+    KnownTypes.insert(CImmLLT);
+  }
+
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_Imm;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
+    if (CImmLLT) {
+      assert(Table.isCombiner() &&
+             "ConstantInt immediate are only for combiners!");
+      Table << MatchTable::Opcode("GIR_AddCImm")
+            << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
+            << MatchTable::Comment("Type")
+            << MatchTable::NamedValue(CImmLLT->getCxxEnumValue())
+            << MatchTable::Comment("Imm") << MatchTable::IntValue(Imm)
+            << MatchTable::LineBreak;
+    } else {
       Table << MatchTable::Opcode("GIR_AddImm") << MatchTable::Comment("InsnID")
             << MatchTable::IntValue(InsnID) << MatchTable::Comment("Imm")
             << MatchTable::IntValue(Imm) << MatchTable::LineBreak;
     }
+  }
 };
 
 /// Adds an enum value for a subreg index to the instruction being built.
 class SubRegIndexRenderer : public OperandRenderer {
 protected:
   unsigned InsnID;
   const CodeGenSubRegIndex *SubRegIdx;
 

llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.h

OS << "bool " << getClassName() << "::test" << ArgName << "Predicate_"
        << TypeIdentifier << "(unsigned PredicateID, " << ArgType << " "
        << ArgName << AdditionalArgs << ") const {\n"
        << AdditionalDeclarations;
     if (!AdditionalDeclarations.empty())
       OS << "\n";
     if (!Predicates.empty()) {
       OS << "  switch (PredicateID) {\n";
       for (const auto &Pred : Predicates) {
-        const auto Code = GetPredCode(Pred);
+        // Ensure all code is indented.
+        const auto Code = join(split(GetPredCode(Pred).str(), "\n"), "\n    ");
         OS << "  case GICXXPred_" << TypeIdentifier << "_Predicate_"
            << GetPredEnumName(Pred) << ": {\n"
            << "    " << Code << "\n";
         if (!StringRef(Code).ltrim().startswith("return")) {
           OS << "    llvm_unreachable(\"" << GetPredEnumName(Pred)
              << " should have returned\");\n";
         }
         OS << "  }\n";

llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.cpp

<< "  bool testImmPredicate_APInt(unsigned PredicateID, const APInt &Imm) "
         "const override;\n"
      << "  bool testImmPredicate_APFloat(unsigned PredicateID, const APFloat "
         "&Imm) const override;\n"
      << "  const int64_t *getMatchTable() const override;\n"
      << "  bool testMIPredicate_MI(unsigned PredicateID, const MachineInstr &MI"
         ", const MatcherState &State) "
         "const override;\n"
      << "  bool testSimplePredicate(unsigned PredicateID) const override;\n"
-     << "  void runCustomAction(unsigned FnID, const MatcherState &State) "
+     << "  void runCustomAction(unsigned FnID, const MatcherState &State, "
+        "NewMIVector &OutMIs) "
         "const override;\n";
   emitAdditionalTemporariesDecl(OS, "  ");
   OS << "#endif // ifdef " << IfDefName << "\n\n";
 }
 
 void GlobalISelMatchTableExecutorEmitter::emitTemporariesInit(
     raw_ostream &OS, unsigned MaxTemporaries, StringRef IfDefName) {
   OS << "#ifdef " << IfDefName << "\n"