[TableGen][GlobalISel] Add support for matching/writing MIFlags in MIR Patterns

[Pierre-vh]

Summary

Allow matching one or more MIFlags, and writing them out as well.

Depends on D158975

[Pierre-vh]

Docs

[arsenm]

llvm/docs/GlobalISel/MIRPatterns.rst

170

173

Should mention the default behavior. Flags should be dropped unless explicitly preserved. Can you and/or the incoming flags?

llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h

566–568

Don't see why this needs to be configurable

[Pierre-vh]

llvm/docs/GlobalISel/MIRPatterns.rst

173

The current behavior is that the output instruction is a "blank canvas" so no flags are preserved, and no there's no way to fetch the flags from an instruction & preserve them (hence why I added the PropagateFlags bool).

Is there any use case where we need to match only one flag and optionally preserve the other flags?

If the set of possibilities is relatively small, a PatFrag can do it - just explicitly match & rewrite them. Not ideal but it works. I'm sure it could all be written with some loop in TableGen as well.

Otherwise, maybe we could have a builtin for that, something like GIMIFlagsAnd<$a, $b> but we also need to be careful about GIR_MutateOpcode (which may erase the flags from the original inst), so it's not 100% trivial.

llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h

566–568

I was wondering what happens if a root has some MI flags, and the pattern rewrites the root & other instructions. Do we really have to propagate flags from the root inst in the combiner?

I have no strong opinion, it just seemed wrong to me so I disabled it for the combiner.

[Pierre-vh]

Update docs

[Pierre-vh]

Rebase

[arsenm]

llvm/docs/GlobalISel/MIRPatterns.rst

173

Most cases involving flags are match one or two and preserve all. There are a fair number of match one or two and drop specific flag

[Pierre-vh]

llvm/docs/GlobalISel/MIRPatterns.rst

173

So what we need is:

• A way to copy flags from a matched instruction to a new instruction
  • Is the old PropagateFlags system good enough? It only copies the flags from the root AFAIK. I'm also wondering if it's not better to make flags not preserved by default, and add something like GICopyMIFlags<$mi> to explicitly copy them)
• A way to remove a flag (e.g. GIRemoveMIFlag)

[Pierre-vh]

llvm/docs/GlobalISel/MIRPatterns.rst

173

For the PropagateFlag system, my worry is something like this: Imagine a combine where you want to transform (fneg (fmul)) into (fmul (fneg)).

With the current system, the G_FNEG would be the root and its flags would be propagated on the G_FMUL, but if what we actually want is to preserve the matched G_FMUL flag on the new G_FMUL, it doesn't work as expected.

It's why I assume it's better to have to explicitly import/set flags in combine rules. I think it's less confusing.

Though it can get verbose quickly if you want to add a few flags and unset one, e.g.

(G_FMUL $a, $b, $c, MIFlagsCopy<"$matchedfmul">, MIFlagsSet<[a, b]>, MIFlagsUnset<c>)

Maybe we enforce a single MIFlags operand but give it more parameters, so we can write MIFlags<"$matchedfmul", [a, b], [c]> ?

[arsenm]

llvm/docs/GlobalISel/MIRPatterns.rst

173

I have no real idea what the "PropagateFlag" was, but it certainly can't be a global mode. It has to be explicit in the pattern. I think we need and/or and unset

[Pierre-vh]

https://github.com/llvm/llvm-project/pull/71179

llvm/docs/GlobalISel/MIRPatterns.rst

.. code-block:: text
   (apply (GIEraseRoot))
 
 Semantics:
 
 * Can only appear as the only pattern of an 'apply' pattern list.
 * The root cannot have any output operands.
 * The root must be a CodeGenInstruction
 
+Instruction Flags
+-----------------
+
+MIR Patterns support both matching & writing ``MIFlags``.
+``MIFlags`` are never preserved; output instructions have never have
+any flags unless explicitly set.
+
+.. code-block:: text
+  :caption: Example
+
+  # MIFlags<list> supports multiple flags.
+  # MIFlag<x> is a shorthand version for a single flag.
+  def Test : GICombineRule<
+    (defs root:$dst),
+    (match (G_ZEXT $dst, $src, MIFlags<[FmNoNans, FmNoInfs]>)),
+    (apply (G_SEXT $dst, $src, MIFlag<FmReassoc>))>;
+
+``MIFlags`` operands are not actual instruction operands - they just tell
+the backend that the pattern being matched/written has some flags.
+
+``MIFlags`` may appear multiple times, and anywhere in a
+CodeGenInstruction pattern, but for readability, please try to keep the style
+consistent. e.g. If they're always at the end of the pattern in the current
+file, keep it that way. Similarly, if the current file prefers using one
+``MIFlags`` instead of multiple ``MIFlag`` to describe multiple flags,
+keep it that way as well.
+
+Note that flags are uniqued for the whole pattern, so even if a given flag is
+mentioned multiple times, it's only checked/added once.
 
 Limitations
 -----------
 
 This a non-exhaustive list of known issues with MIR patterns at this time.
 
 * Matching intrinsics is not yet possible.
 * Using ``GICombinePatFrag`` within another ``GICombinePatFrag`` is not
   supported.

llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h

enum {
 
   /// Check we can replace all uses of a register with another.
   /// - OldInsnID
   /// - OldOpIdx
   /// - NewInsnID
   /// - NewOpIdx
   GIM_CheckCanReplaceReg,
 
+  /// Checks for the presence of a MIFlag.
+  /// Supports checking for multiple OR'd flags at once
+  /// (then all of them must be present)
+  ///  - InsnID - Instruction ID to check.
+  ///  - Flag - Flag(s) to check.
+  GIM_CheckHasMIFlag,
+
   /// Predicates with 'let PredicateCodeUsesOperands = 1' need to examine some
   /// named operands that will be recorded in RecordedOperands. Names of these
   /// operands are referenced in predicate argument list. Emitter determines
   /// StoreIdx(corresponds to the order in which names appear in argument list).
   /// - InsnID - Instruction ID
   /// - OpIdx - Operand index
   /// - StoreIdx - Store location in RecordedOperands.
   GIM_RecordNamedOperand,
enum {
 
   /// Marks the implicit def of a register as dead.
   /// - InsnID - Instruction ID to modify
   /// - OpIdx - The implicit def operand index
   ///
   /// OpIdx starts at 0 for the first implicit def.
   GIR_SetImplicitDefDead,
 
+  /// Sets MIFlags of a given instruction.
+  /// Supports add multiple OR'd flags at once.
+  /// Note: Overwrites all previously set flags on the instruction!
+  ///  - InsnID - Instruction ID to modify.
+  ///  - Flag - Flag(s) to add.
+  GIR_SetMIFlags,
+
   /// Add a temporary register to the specified instruction
   /// - InsnID - Instruction ID to modify
   /// - TempRegID - The temporary register ID to add
   /// - TempRegFlags - The register flags to set
   GIR_AddTempRegister,
 
   /// Add a temporary register to the specified instruction
   /// - InsnID - Instruction ID to modify
struct ExecInfoTy {
     const PredicateBitset *FeatureBitsets;
     const ComplexMatcherMemFn *ComplexPredicates;
     const CustomRendererFn *CustomRenderers;
 
     SmallDenseMap<LLT, unsigned, 64> TypeIDMap;
   };
 
 protected:
-  GIMatchTableExecutor();
+  const bool PropagateMIFlags;
+
+  GIMatchTableExecutor(bool PropagateMIFlags);
 
   /// Execute a given matcher table and return true if the match was successful
   /// and false otherwise.
   template <class TgtExecutor, class PredicateBitset, class ComplexMatcherMemFn,
             class CustomRendererFn>
   bool executeMatchTable(
       TgtExecutor &Exec, NewMIVector &OutMIs, MatcherState &State,
       const ExecInfoTy<PredicateBitset, ComplexMatcherMemFn, CustomRendererFn>

llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutorImpl.h

auto propagateFlags = [=](NewMIVector &OutMIs) {
     for (auto MIB : OutMIs) {
       // Set the NoFPExcept flag when no original matched instruction could
       // raise an FP exception, but the new instruction potentially might.
       uint16_t MIBFlags = Flags;
       if (NoFPException && MIB->mayRaiseFPException())
         MIBFlags |= MachineInstr::NoFPExcept;
       MIB.setMIFlags(MIBFlags);
     }
-
-    return true;
   };
 
   while (true) {
     assert(CurrentIdx != ~0u && "Invalid MatchTable index");
     int64_t MatcherOpcode = MatchTable[CurrentIdx++];
     switch (MatcherOpcode) {
     case GIM_Try: {
       DEBUG_WITH_TYPE(TgtExecutor::getName(),
case GIM_CheckCanReplaceReg: {
       Register Old = State.MIs[OldInsnID]->getOperand(OldOpIdx).getReg();
       Register New = State.MIs[NewInsnID]->getOperand(NewOpIdx).getReg();
       if (!canReplaceReg(Old, New, MRI)) {
         if (handleReject() == RejectAndGiveUp)
           return false;
       }
       break;
     }
+    case GIM_CheckHasMIFlag: {
+      int64_t InsnID = MatchTable[CurrentIdx++];
+      int64_t Flags = MatchTable[CurrentIdx++];
+
+      DEBUG_WITH_TYPE(TgtExecutor::getName(),
+                      dbgs()
+                          << CurrentIdx << ": GIM_CheckHasMIFlag(MIs[" << InsnID
+                          << "], Flags=" << (uint64_t)Flags << ")\n");
+
+      // Need to support multiple OR'd flags, so we can't just do a simple AND.
+      const int64_t MIFlags = State.MIs[InsnID]->getFlags();
+      if ((MIFlags & Flags) != Flags) {
+        if (handleReject() == RejectAndGiveUp)
+          return false;
+      }
+      break;
+    }
     case GIM_Reject:
       DEBUG_WITH_TYPE(TgtExecutor::getName(),
                       dbgs() << CurrentIdx << ": GIM_Reject\n");
       if (handleReject() == RejectAndGiveUp)
         return false;
       break;
     case GIR_MutateOpcode: {
       int64_t OldInsnID = MatchTable[CurrentIdx++];
case GIR_SetImplicitDefDead: {
       DEBUG_WITH_TYPE(TgtExecutor::getName(),
                       dbgs() << CurrentIdx << ": GIR_SetImplicitDefDead(OutMIs["
                              << InsnID << "], OpIdx=" << OpIdx << ")\n");
       MachineInstr *MI = OutMIs[InsnID];
       assert(MI && "Modifying undefined instruction");
       MI->getOperand(MI->getNumExplicitOperands() + OpIdx).setIsDead();
       break;
     }
+    case GIR_SetMIFlags: {
+      int64_t InsnID = MatchTable[CurrentIdx++];
+      int64_t Flags = MatchTable[CurrentIdx++];
+      DEBUG_WITH_TYPE(TgtExecutor::getName(),
+                      dbgs() << CurrentIdx << ": GIR_SetMIFlags(OutMIs["
+                             << InsnID << "], Flags=" << Flags << ")\n");
+      auto &MI = OutMIs[InsnID];
+      // TODO: Should we really clear MIFlags here? It's to avoid MutateOpcode
+      // keeping old flags around.
+      MI->clearFlag(MachineInstr::MIFlag(~uint32_t(0)));
+      MI.setMIFlags(Flags);
+      break;
+    }
     case GIR_AddTempRegister:
     case GIR_AddTempSubRegister: {
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t TempRegID = MatchTable[CurrentIdx++];
       uint64_t TempRegFlags = MatchTable[CurrentIdx++];
       unsigned SubReg = 0;
       if (MatcherOpcode == GIR_AddTempSubRegister)
         SubReg = MatchTable[CurrentIdx++];

 
     case GIR_Done:
       DEBUG_WITH_TYPE(TgtExecutor::getName(),
                       dbgs() << CurrentIdx << ": GIR_Done\n");
       if (Observer) {
         for (MachineInstr *MI : OutMIs)
           Observer->createdInstr(*MI);
       }
+      if (PropagateMIFlags)
         propagateFlags(OutMIs);
       return true;
     default:
       llvm_unreachable("Unexpected command");
     }
   }
 }
 
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_GLOBALISEL_GIMATCHTABLEEXECUTORIMPL_H

llvm/include/llvm/CodeGen/GlobalISel/InstructionSelector.h

 #ifndef LLVM_CODEGEN_GLOBALISEL_INSTRUCTIONSELECTOR_H
 #define LLVM_CODEGEN_GLOBALISEL_INSTRUCTIONSELECTOR_H
 
 #include "llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h"
 
 namespace llvm {
 class InstructionSelector : public GIMatchTableExecutor {
 public:
+  InstructionSelector() : GIMatchTableExecutor(/*PropagateMIFlags*/ true) {}
   virtual ~InstructionSelector();
 
   /// Select the (possibly generic) instruction \p I to only use target-specific
   /// opcodes. It is OK to insert multiple instructions, but they cannot be
   /// generic pre-isel instructions.
   ///
   /// \returns whether selection succeeded.
   /// \pre  I.getParent() && I.getParent()->getParent()

llvm/include/llvm/Target/GlobalISel/Combine.td

 // instruction pattern.
 class GICombinePatFrag<dag outs, dag ins, list<dag> alts> {
   dag InOperands = ins;
   dag OutOperands = outs;
   list<dag> Alternatives = alts;
 }
 
 //===----------------------------------------------------------------------===//
+// Pattern MIFlags
+//===----------------------------------------------------------------------===//
+
+class MIFlagEnum<string enumName> {
+  string EnumName = "MachineInstr::" # enumName;
+}
+
+def FmNoNans : MIFlagEnum<"FmNoNans">;
+def FmNoInfs : MIFlagEnum<"FmNoInfs">;
+def FmNsz : MIFlagEnum<"FmNsz">;
+def FmArcp : MIFlagEnum<"FmArcp">;
+def FmContract : MIFlagEnum<"FmContract">;
+def FmAfn : MIFlagEnum<"FmAfn">;
+def FmReassoc : MIFlagEnum<"FmReassoc">;
+
+class MIFlags<list<MIFlagEnum> flags> {
+  list<MIFlagEnum> Flags = flags;
+}
+
+class MIFlag<MIFlagEnum flag> : MIFlags<[flag]>;
+
+//===----------------------------------------------------------------------===//
 // Pattern Builtins
 //===----------------------------------------------------------------------===//
 
 // "Magic" Builtin instructions for MIR patterns.
 // The definitions that implement
 class GIBuiltinInst;
 
 // Replace all references to a register with another one.

llvm/lib/CodeGen/GlobalISel/Combiner.cpp

void reportFullyCreatedInstrs() {
     });
     LLVM_DEBUG(CreatedInstrs.clear());
   }
 };
 
 Combiner::Combiner(MachineFunction &MF, CombinerInfo &CInfo,
                    const TargetPassConfig *TPC, GISelKnownBits *KB,
                    GISelCSEInfo *CSEInfo)
-    : Builder(CSEInfo ? std::make_unique<CSEMIRBuilder>()
+    : GIMatchTableExecutor(/*PropagateMIFlags=*/false),
+      Builder(CSEInfo ? std::make_unique<CSEMIRBuilder>()
                       : std::make_unique<MachineIRBuilder>()),
       WLObserver(std::make_unique<WorkListMaintainer>(WorkList)),
       ObserverWrapper(std::make_unique<GISelObserverWrapper>()), CInfo(CInfo),
       Observer(*ObserverWrapper), B(*Builder), MF(MF), MRI(MF.getRegInfo()),
       KB(KB), TPC(TPC), CSEInfo(CSEInfo) {
   (void)this->TPC; // FIXME: Remove when used.
 
   // Setup builder.

llvm/lib/CodeGen/GlobalISel/GIMatchTableExecutor.cpp

 
 #define DEBUG_TYPE "gi-match-table-executor"
 
 using namespace llvm;
 
 GIMatchTableExecutor::MatcherState::MatcherState(unsigned MaxRenderers)
     : Renderers(MaxRenderers) {}
 
-GIMatchTableExecutor::GIMatchTableExecutor() = default;
+GIMatchTableExecutor::GIMatchTableExecutor(bool PropagateMIFlags)
+    : PropagateMIFlags(PropagateMIFlags) {}
 
 bool GIMatchTableExecutor::isOperandImmEqual(const MachineOperand &MO,
                                              int64_t Value,
                                              const MachineRegisterInfo &MRI,
                                              bool Splat) const {
   if (MO.isReg() && MO.getReg()) {
     if (auto VRegVal = getIConstantVRegValWithLookThrough(MO.getReg(), MRI))
       return VRegVal->Value.getSExtValue() == Value;

llvm/test/TableGen/GlobalISelCombinerEmitter/builtins/builtin-pattern-errors.td

def replacereg_ops : GICombineRule<
   (apply (GIReplaceReg $dst))>;
 
 // CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: GIReplaceReg cannot replace 'tmp': this builtin can only replace a register defined by the match root
 def replacereg_nonroot : GICombineRule<
   (defs root:$dst),
   (match (COPY $dst, $tmp), (COPY $tmp, $src)),
   (apply (GIReplaceReg $dst, $src), (GIReplaceReg $tmp, $src))>;
 
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: MIFlags operands are only allowed on CodeGenInstruction patterns
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(GIEraseRoot anonymous_
+def miflags_notinst : GICombineRule<
+  (defs root:$mi),
+  (match (G_STORE $a, $b):$mi),
+  (apply (GIEraseRoot MIFlag<FmNoInfs>))>;
+
 // CHECK: error: Failed to parse one or more rules
 
 def MyCombiner: GICombiner<"GenMyCombiner", [
   builtinpat_immop,
   builtinpat_namedimmop,
   eraseroot_in_match,
   eraseroot_ops,
   eraseroot_multiapply,
   eraseroot_root_has_def,
   eraseroot_notinstmatch,
   replacereg_in_match,
   replacereg_ops,
-  replacereg_nonroot
+  replacereg_nonroot,
+  miflags_notinst
 ]>;

llvm/test/TableGen/GlobalISelCombinerEmitter/matchtable-miflags.td

+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner \
+// RUN:     -combiners=MyCombiner %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def TestMutate : GICombineRule<
+  (defs root:$dst),
+  (match (G_ZEXT $dst, $src, MIFlag<FmNoInfs>, MIFlags<[FmNoNans, FmNoInfs]>)),
+  (apply (G_SEXT $dst, $src, MIFlag<FmReassoc>, MIFlags<[FmReassoc, FmAfn]>))>;
+
+def TestNewInst : GICombineRule<
+  (defs root:$dst),
+  (match (G_ZEXT $dst, $src, MIFlag<FmReassoc>, MIFlags<[FmAfn, FmReassoc]>)),
+  (apply (G_FADD $dst, $src, $src, MIFlag<FmNoNans>, MIFlags<[FmNoNans, FmAfn]>))>;
+
+// CHECK:      const int64_t *GenMyCombiner::getMatchTable() const {
+// CHECK-NEXT:   constexpr static int64_t MatchTable0[] = {
+// CHECK-NEXT:     GIM_Try, /*On fail goto*//*Label 0*/ 49,
+// CHECK-NEXT:       GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_ZEXT,
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 1*/ 20, // Rule ID 0 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT:         GIM_CheckHasMIFlag, /*MI*/0, MachineInstr::FmNoInfs | MachineInstr::FmNoNans,
+// CHECK-NEXT:         // MIs[0] dst
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] src
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // Combiner Rule #0: TestMutate
+// CHECK-NEXT:         GIR_MutateOpcode, /*InsnID*/0, /*RecycleInsnID*/0, /*Opcode*/TargetOpcode::G_SEXT,
+// CHECK-NEXT:         GIR_SetMIFlags, /*InsnID*/0, MachineInstr::FmReassoc | MachineInstr::FmAfn,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 1: @20
+// CHECK-NEXT:       GIM_Try, /*On fail goto*//*Label 2*/ 48, // Rule ID 1 //
+// CHECK-NEXT:         GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule1Enabled,
+// CHECK-NEXT:         GIM_CheckHasMIFlag, /*MI*/0, MachineInstr::FmAfn | MachineInstr::FmReassoc,
+// CHECK-NEXT:         // MIs[0] dst
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // MIs[0] src
+// CHECK-NEXT:         // No operand predicates
+// CHECK-NEXT:         // Combiner Rule #1: TestNewInst
+// CHECK-NEXT:         GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::G_FADD,
+// CHECK-NEXT:         GIR_SetMIFlags, /*InsnID*/0, MachineInstr::FmNoNans | MachineInstr::FmAfn,
+// CHECK-NEXT:         GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:         GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/1, // src
+// CHECK-NEXT:         GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/1, // src
+// CHECK-NEXT:         GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:         GIR_Done,
+// CHECK-NEXT:       // Label 2: @48
+// CHECK-NEXT:       GIM_Reject,
+// CHECK-NEXT:     // Label 0: @49
+// CHECK-NEXT:     GIM_Reject,
+// CHECK-NEXT:     };
+// CHECK-NEXT:   return MatchTable0;
+// CHECK-NEXT: }
+
+def MyCombiner: GICombiner<"GenMyCombiner", [
+  TestMutate,
+  TestNewInst
+]>;

llvm/test/TableGen/GlobalISelCombinerEmitter/patfrag-errors.td

[
       (pattern (G_UNMERGE_VALUES $foo, $z, $y))
     ]>;
 // CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(RootDefHasMultiDefs ?:$root, (i32 10))'
 def root_def_has_multi_defs : GICombineRule<
   (defs root:$root),
   (match (RootDefHasMultiDefs $root, (i32 10))),
   (apply (COPY $root, (i32 0)))>;
 
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: MIFlags operands are only allowed on CodeGenInstruction patterns
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(DummyPF anonymous_
+def miflags_notinst : GICombineRule<
+  (defs root:$root),
+  (match (COPY $root, $src), (DummyPF MIFlag<FmNoInfs>)),
+  (apply (COPY $root, (i32 0)))>;
+
 // CHECK: error: Failed to parse one or more rules
 
 def MyCombiner: GICombiner<"GenMyCombiner", [
   too_many_perms,
   undef_livein,
   out_must_be_root,
   bad_out_type,
   unreachable_pat,
   wip_match_opcode_patfrag,
   nest_pat_frag,
   dup_params_in,
   dup_params_out,
   dup_params_inout,
   def_by_all_alts,
   multi_def_in_pat,
   expected_imm,
   expected_imm_namedimm,
   expected_mo,
   expected_mo_namedimm,
   patfrag_in_apply,
   patfrag_cannot_be_root,
   inconsistent_arg_type,
-  root_def_has_multi_defs
+  root_def_has_multi_defs,
+  miflags_notinst
 ]>;

llvm/test/TableGen/GlobalISelCombinerEmitter/pattern-errors.td

 // CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: invalid output operand 'tmp': output immediates cannot be named
 // CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: while emitting pattern 'foo' (COPY)
 def def_named_imm_apply : GICombineRule<
   (defs root:$x),
   (match (G_FNEG $x, $y)),
   (apply (COPY i32:$tmp, $y),
          (COPY $x, (i32 0):$tmp):$foo)>;
 
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: MIFlags operands cannot be named
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(COPY ?:$x, ?:$y, anonymous_6988:$tmp)'
+def miflags_named : GICombineRule<
+  (defs root:$x),
+  (match (G_FNEG $x, $y)),
+  (apply (COPY $x, $y, MIFlag<FmNoInfs>:$tmp))>;
+
 // CHECK: error: Failed to parse one or more rules
 
 def MyCombiner: GICombiner<"GenMyCombiner", [
   root_not_found,
   misleading_root,
   cxx_root,
   livein_root,
   not_enough_operands,
def MyCombiner: GICombiner<"GenMyCombiner", [
   bad_imm_not_a_valuetype,
   output_imm_cannot_be_named,
   output_imm_must_be_typed,
   too_few_ops_for_variadic,
   expected_op_name,
   bad_mo_type_not_a_valuetype,
   untyped_new_reg_in_apply,
   def_named_imm_match,
-  def_named_imm_apply
+  def_named_imm_apply,
+  miflags_named
 ]>;

llvm/test/TableGen/GlobalISelCombinerEmitter/pattern-parsing.td

 // CHECK-NEXT:   )
 // CHECK-NEXT: )
 def VariadicsOutTest : GICombineRule<
   (defs root:$a),
   (match (G_UNMERGE_VALUES $a, $b, $src)),
   (apply (COPY $a, (i32 0)),
          (COPY $b, (i32 0)))>;
 
+// CHECK:      (CombineRule name:MIFlagsTest id:10 root:dst
+// CHECK-NEXT:   (MatchPats
+// CHECK-NEXT:     <match_root>__MIFlagsTest_match_0:(CodeGenInstructionPattern G_ZEXT operands:[<def>$dst, $src] flags:[MachineInstr::FmNoInfs, MachineInstr::FmNoNans])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (ApplyPats
+// CHECK-NEXT:     <apply_root>__MIFlagsTest_apply_0:(CodeGenInstructionPattern G_ZEXT operands:[<def>$dst, $src] flags:[MachineInstr::FmReassoc, MachineInstr::FmAfn])
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (OperandTable MatchPats
+// CHECK-NEXT:     dst -> __MIFlagsTest_match_0
+// CHECK-NEXT:     src -> <live-in>
+// CHECK-NEXT:   )
+// CHECK-NEXT:   (OperandTable ApplyPats
+// CHECK-NEXT:     dst -> __MIFlagsTest_apply_0
+// CHECK-NEXT:     src -> <live-in>
+// CHECK-NEXT:   )
+// CHECK-NEXT: )
+def MIFlagsTest : GICombineRule<
+  (defs root:$dst),
+  (match (G_ZEXT $dst, $src, MIFlag<FmNoInfs>, MIFlags<[FmNoNans, FmNoInfs]>)),
+  (apply (G_ZEXT $dst, $src, MIFlag<FmReassoc>, MIFlags<[FmReassoc, FmAfn]>))>;
 
 def MyCombiner: GICombiner<"GenMyCombiner", [
   WipOpcodeTest0,
   WipOpcodeTest1,
   InstTest0,
   InstTest1,
   InstTest2,
   InOutInstTest0,
   PatFragTest0,
   PatFragTest1,
   VariadicsInTest,
-  VariadicsOutTest
+  VariadicsOutTest,
+  MIFlagsTest
 ]>;

llvm/utils/TableGen/GlobalISelCombinerEmitter.cpp

cl::opt<bool> DebugCXXPreds(
     "gicombiner-debug-cxxpreds",
     cl::desc("Add Contextual/Debug comments to all C++ predicates"),
     cl::cat(GICombinerEmitterCat));
 
 constexpr StringLiteral CXXApplyPrefix = "GICXXCustomAction_CombineApply";
 constexpr StringLiteral CXXPredPrefix = "GICXXPred_MI_Predicate_";
 constexpr StringLiteral PatFragClassName = "GICombinePatFrag";
 constexpr StringLiteral BuiltinInstClassName = "GIBuiltinInst";
+constexpr StringLiteral MIFlagsEnumClassName = "MIFlagEnum";
+constexpr StringLiteral MIFlagsClassName = "MIFlags";
 
 std::string getIsEnabledPredicateEnumName(unsigned CombinerRuleID) {
   return "GICXXPred_Simple_IsRule" + to_string(CombinerRuleID) + "Enabled";
 }
 
 /// Copies a StringRef into a static pool to make sure it has a static lifetime.
 StringRef insertStrRef(StringRef S) {
   if (S.empty())
public:
   void reportUnreachable(ArrayRef<SMLoc> Locs) const;
   virtual bool checkSemantics(ArrayRef<SMLoc> Loc);
 
   void print(raw_ostream &OS, bool PrintName = true) const override;
 
 protected:
   InstructionPattern(unsigned K, StringRef Name) : Pattern(K, Name) {}
 
+  virtual void printExtras(raw_ostream &OS) const {}
+
   SmallVector<InstructionOperand, 4> Operands;
 };
 
 void InstructionPattern::reportUnreachable(ArrayRef<SMLoc> Locs) const {
   PrintError(Locs, "pattern '" + getName() + "' ('" + getInstName() +
                        "') is unreachable from the pattern root!");
 }
 
printImpl(OS, PrintName, [&OS, this] {
     OS << getInstName() << " operands:[";
     StringRef Sep = "";
     for (const auto &Op : Operands) {
       OS << Sep;
       Op.print(OS);
       Sep = ", ";
     }
     OS << "]";
+
+    printExtras(OS);
   });
 }
 
 //===- OperandTable -------------------------------------------------------===//
 
 /// Maps InstructionPattern operands to their definitions. This allows us to tie
 /// different patterns of a (apply), (match) or (patterns) set of patterns
 /// together.
bool is(StringRef OpcodeName) const {
     return I.TheDef->getName() == OpcodeName;
   }
 
   bool hasVariadicDefs() const;
   bool isVariadic() const override { return I.Operands.isVariadic; }
   unsigned getNumInstDefs() const override;
   unsigned getNumInstOperands() const override;
 
+  bool hasMIFlags() const { return !MIFlags.empty(); }
+  void addMIFlag(StringRef Flag) {
+    if (find(MIFlags, Flag) == MIFlags.end())
+      MIFlags.push_back(Flag);
+  }
+  const auto &mi_flags() const { return MIFlags; }
+
   const CodeGenInstruction &getInst() const { return I; }
   StringRef getInstName() const override { return I.TheDef->getName(); }
 
 private:
+  void printExtras(raw_ostream &OS) const override;
+
   const CodeGenInstruction &I;
+  std::vector<StringRef> MIFlags;
 };
 
 bool CodeGenInstructionPattern::hasVariadicDefs() const {
   // Note: we cannot use variadicOpsAreDefs, it's not set for
   // GenericInstructions.
   if (!isVariadic())
     return false;
 

 }
 
 unsigned CodeGenInstructionPattern::getNumInstOperands() const {
   unsigned NumCGIOps = I.Operands.size();
   return isVariadic() ? std::max<unsigned>(NumCGIOps, Operands.size())
                       : NumCGIOps;
 }
 
+void CodeGenInstructionPattern::printExtras(raw_ostream &OS) const {
+  if (MIFlags.empty())
+    return;
+
+  OS << " flags:[" << join(MIFlags, ", ") << ']';
+}
+
 //===- OperandTypeChecker -------------------------------------------------===//
 
 /// This is a trivial type checker for all operands in a set of
 /// InstructionPatterns.
 ///
 /// It infers the type of each operand, check it's consistent with the known
 /// type of the operand, and then sets all of the types in all operands in
 /// setAllOperandTypes.
private:
 
   std::unique_ptr<Pattern> parseInstructionPattern(const Init &Arg,
                                                    StringRef PatName) const;
   std::unique_ptr<Pattern> parseWipMatchOpcodeMatcher(const Init &Arg,
                                                       StringRef PatName) const;
   bool parseInstructionPatternOperand(InstructionPattern &IP,
                                       const Init *OpInit,
                                       const StringInit *OpName) const;
+  bool parseInstructionPatternMIFlagsOperand(InstructionPattern &IP,
+                                             const Record *Flag) const;
   std::unique_ptr<PatFrag> parsePatFragImpl(const Record *Def) const;
   bool parsePatFragParamList(
       ArrayRef<SMLoc> DiagLoc, const DagInit &OpsList,
       function_ref<bool(StringRef, PatFrag::ParamKind)> ParseAction) const;
   const PatFrag *parsePatFrag(const Record *Def) const;
 
   bool emitMatchPattern(CodeExpansions &CE, const PatternAlternatives &Alts,
                         const InstructionPattern &IP);
Pat = std::make_unique<PatFragPattern>(*PF, Name);
                  getDagWithOperatorOfSubClass(Arg, BuiltinInstClassName)) {
     Pat = std::make_unique<BuiltinPattern>(
         *BP->getOperatorAsDef(RuleDef.getLoc()), Name);
   } else {
     return nullptr;
   }
 
   for (unsigned K = 0; K < DagPat->getNumArgs(); ++K) {
+    Init *Arg = DagPat->getArg(K);
+    StringInit *Name = DagPat->getArgName(K);
+
+    if (auto *Def = dyn_cast<DefInit>(Arg);
+        Def && Def->getDef()->isSubClassOf(MIFlagsClassName)) {
+      if (!parseInstructionPatternMIFlagsOperand(*Pat, Def->getDef()))
+        return nullptr;
+
+      if (Name) {
+        PrintError("" + MIFlagsClassName + " operands cannot be named");
+        return nullptr;
+      }
+
+      continue;
+    }
+
     if (!parseInstructionPatternOperand(*Pat, DagPat->getArg(K),
                                         DagPat->getArgName(K)))
       return nullptr;
   }
 
   if (!Pat->checkSemantics(RuleDef.getLoc()))
     return nullptr;
 
if (isa<UnsetInit>(OpInit)) {
     assert(OpName && "Unset w/ no OpName?");
     IP.addOperand(OpName->getAsUnquotedString(), /*Type=*/nullptr);
     return true;
   }
 
   return ParseErr();
 }
 
+bool CombineRuleBuilder::parseInstructionPatternMIFlagsOperand(
+    InstructionPattern &IP, const Record *OpDef) const {
+  assert(OpDef->isSubClassOf(MIFlagsClassName));
+
+  auto *CGIP = dyn_cast<CodeGenInstructionPattern>(&IP);
+  if (!CGIP) {
+    PrintError(MIFlagsClassName +
+               " operands are only allowed on CodeGenInstruction patterns");
+    return false;
+  }
+
+  const auto Flags = OpDef->getValueAsListOfDefs("Flags");
+  for (auto *Flag : Flags) {
+    // TableGen file should ensure this is correct.
+    assert(Flag->isSubClassOf(MIFlagsEnumClassName));
+    CGIP->addMIFlag(Flag->getValueAsString("EnumName"));
+  }
+
+  return true;
+}
+
 std::unique_ptr<PatFrag>
 CombineRuleBuilder::parsePatFragImpl(const Record *Def) const {
   auto StackTrace = PrettyStackTraceParse(*Def);
   if (!Def->isSubClassOf(PatFragClassName))
     return nullptr;
 
   const DagInit *Ins = Def->getValueAsDag("InOperands");
   if (Ins->getOperatorAsDef(Def->getLoc())->getName() != "ins") {
bool CombineRuleBuilder::emitInstructionApplyPattern(
 
   if (isa<PatFragPattern>(&P))
     llvm_unreachable("PatFragPatterns is not supported in 'apply'!");
 
   auto &CGIP = cast<CodeGenInstructionPattern>(P);
 
   // Now render this inst.
   auto &DstMI =
-      M.addAction<BuildMIAction>(M.allocateOutputInsnID(), &CGIP.getInst());
+      M.addAction<BuildMIAction>(M.allocateOutputInsnID(), &CGIP.getInst(),
+                                 std::vector<StringRef>(CGIP.mi_flags()));
 
   for (auto &Op : P.operands()) {
     if (Op.isNamedImmediate()) {
       PrintError("invalid output operand '" + Op.getOperandName() +
                  "': output immediates cannot be named");
       PrintNote("while emitting pattern '" + P.getName() + "' (" +
                 P.getInstName() + ")");
       return false;
bool CombineRuleBuilder::emitCodeGenInstructionMatchPattern(
   if (SeenPats.contains(&P))
     return true;
 
   SeenPats.insert(&P);
 
   IM.addPredicate<InstructionOpcodeMatcher>(&P.getInst());
   declareInstExpansion(CE, IM, P.getName());
 
+  if (P.hasMIFlags())
+    IM.addPredicate<MIFlagsInstructionPredicateMatcher>(P.mi_flags());
+
   for (const auto &[Idx, OriginalO] : enumerate(P.operands())) {
     // Remap the operand. This is used when emitting InstructionPatterns inside
     // PatFrags, so it can remap them to the arguments passed to the pattern.
     //
     // We use the remapped operand to emit immediates, and for the symbolic
     // operand names (in IM.addOperand). CodeExpansions and OperandTable lookups
     // still use the original name.
     //

llvm/utils/TableGen/GlobalISelMatchTable.h

enum PredicateKind {
     IPM_AtomicOrderingMMO,
     IPM_MemoryLLTSize,
     IPM_MemoryVsLLTSize,
     IPM_MemoryAddressSpace,
     IPM_MemoryAlignment,
     IPM_VectorSplatImm,
     IPM_NoUse,
     IPM_GenericPredicate,
+    IPM_MIFlags,
     OPM_SameOperand,
     OPM_ComplexPattern,
     OPM_IntrinsicID,
     OPM_CmpPredicate,
     OPM_Instruction,
     OPM_Int,
     OPM_LiteralInt,
     OPM_LLT,
public:
   static bool classof(const InstructionPredicateMatcher *P) {
     return P->getKind() == IPM_GenericPredicate;
   }
   bool isIdentical(const PredicateMatcher &B) const override;
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override;
 };
 
+class MIFlagsInstructionPredicateMatcher : public InstructionPredicateMatcher {
+  std::vector<StringRef> Flags;
+
+public:
+  MIFlagsInstructionPredicateMatcher(unsigned InsnVarID,
+                                     std::vector<StringRef> FlagsToCheck)
+      : InstructionPredicateMatcher(IPM_MIFlags, InsnVarID),
+        Flags(FlagsToCheck) {
+    sort(Flags);
+  }
+
+  static bool classof(const InstructionPredicateMatcher *P) {
+    return P->getKind() == IPM_MIFlags;
+  }
+
+  bool isIdentical(const PredicateMatcher &B) const override;
+  void emitPredicateOpcodes(MatchTable &Table,
+                            RuleMatcher &Rule) const override;
+};
+
 /// Generates code to check for the absence of use of the result.
 // TODO? Generalize this to support checking for one use.
 class NoUsePredicateMatcher : public InstructionPredicateMatcher {
 public:
   NoUsePredicateMatcher(unsigned InsnVarID)
       : InstructionPredicateMatcher(IPM_NoUse, InsnVarID) {}
 
   static bool classof(const PredicateMatcher *P) {

 /// into the desired instruction when this is possible.
 class BuildMIAction : public MatchAction {
 private:
   unsigned InsnID;
   const CodeGenInstruction *I;
   InstructionMatcher *Matched;
   std::vector<std::unique_ptr<OperandRenderer>> OperandRenderers;
   SmallPtrSet<Record *, 4> DeadImplicitDefs;
+  std::vector<StringRef> Flags;
 
   /// True if the instruction can be built solely by mutating the opcode.
   bool canMutate(RuleMatcher &Rule, const InstructionMatcher *Insn) const;
 
 public:
-  BuildMIAction(unsigned InsnID, const CodeGenInstruction *I)
-      : MatchAction(AK_BuildMI), InsnID(InsnID), I(I), Matched(nullptr) {}
+  BuildMIAction(unsigned InsnID, const CodeGenInstruction *I,
+                const std::vector<StringRef> &MIFlags = {})
+      : MatchAction(AK_BuildMI), InsnID(InsnID), I(I), Matched(nullptr),
+        Flags(MIFlags) {}
 
   static bool classof(const MatchAction *A) {
     return A->getKind() == AK_BuildMI;
   }
 
   unsigned getInsnID() const { return InsnID; }
   const CodeGenInstruction *getCGI() const { return I; }
 

llvm/utils/TableGen/GlobalISelMatchTable.cpp

 void GenericInstructionPredicateMatcher::emitPredicateOpcodes(
     MatchTable &Table, RuleMatcher &Rule) const {
   Table << MatchTable::Opcode("GIM_CheckCxxInsnPredicate")
         << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
         << MatchTable::Comment("FnId") << MatchTable::NamedValue(EnumVal)
         << MatchTable::LineBreak;
 }
 
+//===- MIFlagsInstructionPredicateMatcher ---------------------------------===//
+
+bool MIFlagsInstructionPredicateMatcher::isIdentical(
+    const PredicateMatcher &B) const {
+  return InstructionPredicateMatcher::isIdentical(B) &&
+         Flags ==
+             static_cast<const MIFlagsInstructionPredicateMatcher &>(B).Flags;
+}
+void MIFlagsInstructionPredicateMatcher::emitPredicateOpcodes(
+    MatchTable &Table, RuleMatcher &Rule) const {
+  Table << MatchTable::Opcode("GIM_CheckHasMIFlag") << MatchTable::Comment("MI")
+        << MatchTable::IntValue(InsnVarID)
+        << MatchTable::NamedValue(join(Flags, " | ")) << MatchTable::LineBreak;
+}
+
 //===- InstructionMatcher -------------------------------------------------===//
 
 OperandMatcher &
 InstructionMatcher::addOperand(unsigned OpIdx, const std::string &SymbolicName,
                                unsigned AllocatedTemporariesBaseID) {
   Operands.emplace_back(new OperandMatcher(*this, OpIdx, SymbolicName,
                                            AllocatedTemporariesBaseID));
   if (!SymbolicName.empty())
if (canMutate(Rule, MutateCandidate)) {
       Matched = MutateCandidate;
       return;
     }
   }
 }
 
 void BuildMIAction::emitActionOpcodes(MatchTable &Table,
                                       RuleMatcher &Rule) const {
+
+  const auto AddMIFlags = [&]() {
+    if (Flags.empty())
+      return;
+
+    Table << MatchTable::Opcode("GIR_SetMIFlags")
+          << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
+          << MatchTable::NamedValue(join(Flags, " | "))
+          << MatchTable::LineBreak;
+  };
+
   if (Matched) {
     assert(canMutate(Rule, Matched) &&
            "Arranged to mutate an insn that isn't mutatable");
 
     unsigned RecycleInsnID = Rule.getInsnVarID(*Matched);
     Table << MatchTable::Opcode("GIR_MutateOpcode")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::Comment("RecycleInsnID")
           << MatchTable::IntValue(RecycleInsnID)
           << MatchTable::Comment("Opcode")
           << MatchTable::NamedValue(I->Namespace, I->TheDef->getName())
           << MatchTable::LineBreak;
 
+    // GIR_SetMIFlags clears flags so we don't need to worry about pre-existing
+    // ones.
+    AddMIFlags();
+
     if (!I->ImplicitDefs.empty() || !I->ImplicitUses.empty()) {
       for (auto *Def : I->ImplicitDefs) {
         auto Namespace = Def->getValue("Namespace")
                              ? Def->getValueAsString("Namespace")
                              : "";
         const bool IsDead = DeadImplicitDefs.contains(Def);
         Table << MatchTable::Opcode("GIR_AddImplicitDef")
               << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
void BuildMIAction::emitActionOpcodes(MatchTable &Table,
 
   // TODO: Simple permutation looks like it could be almost as common as
   //       mutation due to commutative operations.
 
   Table << MatchTable::Opcode("GIR_BuildMI") << MatchTable::Comment("InsnID")
         << MatchTable::IntValue(InsnID) << MatchTable::Comment("Opcode")
         << MatchTable::NamedValue(I->Namespace, I->TheDef->getName())
         << MatchTable::LineBreak;
+
+  AddMIFlags();
+
   for (const auto &Renderer : OperandRenderers)
     Renderer->emitRenderOpcodes(Table, Rule);
 
   for (auto [OpIdx, Def] : enumerate(I->ImplicitDefs)) {
     auto Namespace =
         Def->getValue("Namespace") ? Def->getValueAsString("Namespace") : "";
     if (DeadImplicitDefs.contains(Def)) {
       Table