[globalisel][tablegen] Add support for ImmLeaf without SDNodeXForm

[dsanders]

Summary

This patch adds support for predicates on imm nodes but only for ImmLeaf and not for PatLeaf or PatFrag and only where the value does not need to be transformed before being rendered into the instruction.

The limitation on PatLeaf/PatFrag/SDNodeXForm is due to differences in the necessary target-supplied C++ for GlobalISel.

Depends on D36085

[dsanders]

Rebase

[dsanders]

ping

[rovka]

This should have a summary.

I'm probably just bikeshedding a bit, but why is this an instruction predicate and not an operand predicate? I understand that you know it always refers to the second operand of the G_CONSTANT, but I think it looks a bit strange e.g. in your test you have a "No predicates" comment on the MIs[0] Operand 1 which looks a bit surprising (I had a bit of an "ooooh, of course, because it's an instruction predicate" moment reading that test). It's especially weird considering that the Matcher has this comment: "Generates code to check that an operand meets an immediate predicate." - I'd expect it to be an operand predicate then.

include/llvm/CodeGen/GlobalISel/InstructionSelector.h

96

typo (truction)

include/llvm/CodeGen/GlobalISel/InstructionSelectorImpl.h

137

Maybe you should also assert that this is different from GIPFP_Invalid.

141

You forgot == G_CONSTANT :)

[dsanders]

Thanks

In D36086#844494, @rovka wrote:

This should have a summary.

I'm probably just bikeshedding a bit, but why is this an instruction predicate and not an operand predicate? I understand that you know it always refers to the second operand of the G_CONSTANT, but I think it looks a bit strange e.g. in your test you have a "No predicates" comment on the MIs[0] Operand 1 which looks a bit surprising (I had a bit of an "ooooh, of course, because it's an instruction predicate" moment reading that test).

I was surprised to find out it was an operator in the TreePatternNode. For predicated immediates like simm8:$src, TreePatternNode translates it to (imm:i32)<<P:Predicate_simm8>>:$imm with both the predicate and the name attached to the imm operator rather than the value inside it. I'd always thought imm and similar were leaves. The tablegen classes are also called ImmLeaf/PatLeaf which reinforced that. It only makes sense when you see that ImmLeaf/PatLeaf are subclasses of PatFrag and that ConstantSDNode holds the immediate internally instead of referencing another SDNode.

I've handled it the same way because some predicates belong on the operator (particularly load/store predicates). As a result, sinking it to the operand would require special casing the immediate case and handling it separately from other PatFrags. Also, at the point the predicates are handled, the relevant OperandMatcher hasn't been created yet so it's just easier to apply the predicate where it was found.

For the 'No predicates comment', making it specify which kind of predicates might help a little but it's doesn't fully remove the confusion.

It's especially weird considering that the Matcher has this comment: "Generates code to check that an operand meets an immediate predicate." - I'd expect it to be an operand predicate then.

I agree. What it says is correct but it fails to mention important details. I'll update this comment.

include/llvm/CodeGen/GlobalISel/InstructionSelectorImpl.h

137

Ok

141

Again? :-) I must have copy/pasted from the last one and forgot to fix this one too

utils/TableGen/GlobalISelEmitter.cpp

1020

This should mention that it's an instruction predicate and why.

[dsanders]

Fixed the nits and added some additional tests I noticed were missing from test/TableGen/GlobalISelEmitter.td

[rovka]

Thanks for the updates, this looks really good now!

include/llvm/CodeGen/GlobalISel/InstructionSelectorImpl.h

143

Supernit: Predicate != GIPFP_Invalid seems more direct and readable...

[dsanders]

include/llvm/CodeGen/GlobalISel/InstructionSelectorImpl.h

143

Thanks for spotting this. The 0 /* GIPFP_Invalid */ bit was an intermediate step caused by a circular dependency problem. I fixed that in this update but it looks like I didn't correct this bit afterwards. I'll correct this in the commit

The reason for Predicate > GIPFP_Invalid is that the table consists of signed integers and valid predicates are all positive non-zeros.

include/llvm/CodeGen/GlobalISel/InstructionSelector.h

enum {
   /// Check the opcode on the specified instruction
   /// - InsnID - Instruction ID
   /// - Expected opcode
   GIM_CheckOpcode,
   /// Check the instruction has the right number of operands
   /// - InsnID - Instruction ID
   /// - Expected number of operands
   GIM_CheckNumOperands,
+  /// Check an immediate predicate on the specified instruction
+  /// - InsnID - Instruction ID
+  /// - The predicate to test
+  GIM_CheckImmPredicate,
 
   /// Check the type for the specified operand
   /// - InsnID - Instruction ID
   /// - OpIdx - Operand index
   /// - Expected type
   GIM_CheckType,
   /// Check the register bank for the specified operand
   /// - InsnID - Instruction ID
enum {
   /// Indicates the end of the variable-length MergeInsnID list in a
   /// GIR_MergeMemOperands opcode.
   GIU_MergeMemOperands_EndOfList = -1,
 };
 
 /// Provides the logic to select generic machine instructions.
 class InstructionSelector {
 public:
+  typedef bool(*ImmediatePredicateFn)(int64_t);
+
   virtual ~InstructionSelector() = default;
 
   /// Select the (possibly generic) instruction \p I to only use target-specific
   /// opcodes. It is OK to insert multiple instructions, but they cannot be
   /// generic pre-isel instructions.
   ///
   /// \returns whether selection succeeded.
   /// \pre  I.getParent() && I.getParent()->getParent()
struct MatcherState {
     MatcherState(unsigned MaxRenderers);
   };
 
 public:
   template <class PredicateBitset, class ComplexMatcherMemFn>
   struct MatcherInfoTy {
     const LLT *TypeObjects;
     const PredicateBitset *FeatureBitsets;
+    const ImmediatePredicateFn *ImmPredicateFns;
     const std::vector<ComplexMatcherMemFn> ComplexPredicates;
   };
 
 protected:
   InstructionSelector();
 
   /// Execute a given matcher table and return true if the match was successful
   /// and false otherwise.

include/llvm/CodeGen/GlobalISel/InstructionSelectorImpl.h

 /// It's implemented by the target. It's used by the InstructionSelect pass.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_GLOBALISEL_INSTRUCTIONSELECTORIMPL_H
 #define LLVM_CODEGEN_GLOBALISEL_INSTRUCTIONSELECTORIMPL_H
 
 namespace llvm {
+
+/// GlobalISel PatFrag Predicates
+enum {
+  GIPFP_Invalid,
+};
+
 template <class TgtInstructionSelector, class PredicateBitset,
           class ComplexMatcherMemFn>
 bool InstructionSelector::executeMatchTable(
     TgtInstructionSelector &ISel, NewMIVector &OutMIs, MatcherState &State,
     const MatcherInfoTy<PredicateBitset, ComplexMatcherMemFn> &MatcherInfo,
     const int64_t *MatchTable, const TargetInstrInfo &TII,
     MachineRegisterInfo &MRI, const TargetRegisterInfo &TRI,
     const RegisterBankInfo &RBI,
case GIM_CheckNumOperands: {
                    << "], Expected=" << Expected << ")\n");
       assert(State.MIs[InsnID] != nullptr && "Used insn before defined");
       if (State.MIs[InsnID]->getNumOperands() != Expected) {
         if (handleReject() == RejectAndGiveUp)
           return false;
       }
       break;
     }
+    case GIM_CheckImmPredicate: {
+      int64_t InsnID = MatchTable[CurrentIdx++];
+      int64_t Predicate = MatchTable[CurrentIdx++];
+      DEBUG(dbgs() << CurrentIdx << ": GIM_CheckImmPredicate(MIs[" << InsnID
+                   << "], Predicate=" << Predicate << ")\n");
+      assert(State.MIs[InsnID] != nullptr && "Used insn before defined");
+      assert(State.MIs[InsnID]->getOpcode() == TargetOpcode::G_CONSTANT &&
+             "Expected G_CONSTANT");
+      assert(Predicate > 0 /*GIPFP_Invalid*/ && "Expected a valid predicate");
+      int64_t Value = 0;
+      if (State.MIs[InsnID]->getOperand(1).isCImm())
+        Value = State.MIs[InsnID]->getOperand(1).getCImm()->getSExtValue();
+      else if (State.MIs[InsnID]->getOperand(1).isImm())
+        Value = State.MIs[InsnID]->getOperand(1).getImm();
+      else
+        llvm_unreachable("Expected Imm or CImm operand");
+
+      if (!MatcherInfo.ImmPredicateFns[Predicate](Value))
+        if (handleReject() == RejectAndGiveUp)
+          return false;
+      break;
+    }
 
     case GIM_CheckType: {
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t OpIdx = MatchTable[CurrentIdx++];
       int64_t TypeID = MatchTable[CurrentIdx++];
       DEBUG(dbgs() << CurrentIdx << ": GIM_CheckType(MIs[" << InsnID
                    << "]->getOperand(" << OpIdx << "), TypeID=" << TypeID
                    << ")\n");

test/CodeGen/AArch64/GlobalISel/select-neon-vcvtfxu2fp.mir

+# RUN: llc -mtriple=aarch64-- -mattr=+neon,+fullfp16 -run-pass=instruction-select -verify-machineinstrs -global-isel %s -o - | FileCheck %s
+
+--- |
+  target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
+
+  define void @vcvtfxu2fp_s64_fpr() { ret void }
+...
+
+---
+# Check that we select a 64-bit FPR vcvtfxu2fp intrinsic into UCVTFd for FPR64.
+# CHECK-LABEL: name: vcvtfxu2fp_s64_fpr
+name:            vcvtfxu2fp_s64_fpr
+legalized:       true
+regBankSelected: true
+
+# CHECK:      registers:
+# CHECK-NEXT:  - { id: 0, class: fpr64, preferred-register: '' }
+# CHECK-NEXT:  - { id: 1, class: gpr, preferred-register: '' }
+# CHECK-NEXT:  - { id: 2, class: fpr64, preferred-register: '' }
+registers:
+  - { id: 0, class: fpr }
+  - { id: 1, class: gpr }
+  - { id: 2, class: fpr }
+
+# CHECK:  body:
+# CHECK:    %0 = COPY %d0
+# CHECK:    %2 = UCVTFd %0, 12
+# CHECK:    %d1 = COPY %2
+body:             |
+  bb.0:
+    liveins: %d0
+
+    %0(s64) = COPY %d0
+    %1(s32) = G_CONSTANT i32 12
+    %2(s64) = G_INTRINSIC intrinsic(@llvm.aarch64.neon.vcvtfxu2fp.f64), %0, %1
+    %d1 = COPY %2(s64)
+...

test/TableGen/GlobalISelEmitter.td

 // CHECK-LABEL: #ifdef GET_GLOBALISEL_TEMPORARIES_DECL
 // CHECK-NEXT:    mutable MatcherState State;
 // CHECK-NEXT:    typedef ComplexRendererFn(MyTargetInstructionSelector::*ComplexMatcherMemFn)(MachineOperand &) const;
 // CHECK-NEXT:    const MatcherInfoTy<PredicateBitset, ComplexMatcherMemFn> MatcherInfo;
 // CHECK-NEXT:  #endif // ifdef GET_GLOBALISEL_TEMPORARIES_DECL
 
 // CHECK-LABEL: #ifdef GET_GLOBALISEL_TEMPORARIES_INIT
 // CHECK-NEXT:    , State(2),
-// CHECK-NEXT:    MatcherInfo({TypeObjects, FeatureBitsets, {
+// CHECK-NEXT:    MatcherInfo({TypeObjects, FeatureBitsets, ImmPredicateFns, {
 // CHECK-NEXT:      nullptr, // GICP_Invalid
 // CHECK-NEXT:      &MyTargetInstructionSelector::selectComplexPattern, // gi_complex
 // CHECK-NEXT:    }})
 // CHECK-NEXT:  #endif // ifdef GET_GLOBALISEL_TEMPORARIES_INIT
 
 // CHECK-LABEL: enum SubtargetFeatureBits : uint8_t {
 // CHECK-NEXT:    Feature_HasABit = 0,
 // CHECK-NEXT:    Feature_HasBBit = 1,

 
 // CHECK-LABEL: enum {
 // CHECK-NEXT:    GILLT_s32,
 // CHECK-NEXT:  }
 // CHECK-NEXT:  const static LLT TypeObjects[] = {
 // CHECK-NEXT:    LLT::scalar(32),
 // CHECK-NEXT:  };
 
+// CHECK-LABEL: // PatFrag predicates.
+// CHECK-NEXT:  enum {
+// CHECK-NEXT:    GIPFP_Predicate_simm8 = GIPFP_Invalid,
+// CHECK-NEXT:  };
+// CHECK-NEXT:  static bool Predicate_simm8(int64_t Imm) { return isInt<8>(Imm);   }
+// CHECK-NEXT:  static InstructionSelector::ImmediatePredicateFn ImmPredicateFns[] = {
+// CHECK-NEXT:    Predicate_simm8,
+// CHECK-NEXT:  };
+
 // CHECK: bool MyTargetInstructionSelector::selectImpl(MachineInstr &I) const {
 // CHECK-NEXT: MachineFunction &MF = *I.getParent()->getParent();
 // CHECK-NEXT: MachineRegisterInfo &MRI = MF.getRegInfo();
 // CHECK:      AvailableFunctionFeatures = computeAvailableFunctionFeatures(&STI, &MF);
 // CHECK-NEXT: const PredicateBitset AvailableFeatures = getAvailableFeatures();
 // CHECK-NEXT: NewMIVector OutMIs;
 // CHECK-NEXT: State.MIs.clear();
 // CHECK-NEXT: State.MIs.push_back(&I);

 // CHECK-NEXT:    GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
 // CHECK-NEXT:    GIR_EraseFromParent, /*InsnID*/0,
 // CHECK-NEXT:    GIR_ConstrainSelectedInstOperands, /*InsnID*/0,
 // CHECK-NEXT:    GIR_Done,
 // CHECK-NEXT:  // Label 15: @[[LABEL]]
 
 def MOV1 : I<(outs GPR32:$dst), (ins), [(set GPR32:$dst, 1)]>;
 
-//===- Test a simple pattern with just a leaf immediate. ------------------===//
+//===- Test a simple pattern with a leaf immediate and a predicate. -------===//
 
 // CHECK-NEXT:  GIM_Try, /*On fail goto*//*Label 16*/ [[LABEL:[0-9]+]],
 // CHECK-NEXT:    GIM_CheckNumOperands, /*MI*/0, /*Expected*/2,
 // CHECK-NEXT:    GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_CONSTANT,
+// CHECK-NEXT:    GIM_CheckImmPredicate, /*MI*/0, /*Predicate*/GIPFP_Predicate_simm8,
 // CHECK-NEXT:    // MIs[0] dst
 // CHECK-NEXT:    GIM_CheckType, /*MI*/0, /*Op*/0, /*Type*/GILLT_s32,
 // CHECK-NEXT:    GIM_CheckRegBankForClass, /*MI*/0, /*Op*/0, /*RC*/MyTarget::GPR32RegClassID,
 // CHECK-NEXT:    // MIs[0] Operand 1
-// CHECK-NEXT:    // No predicates
+// CHECK-NEXT:    // No operand predicates
+// CHECK-NEXT:    // (imm:i32)<<P:Predicate_simm8>>:$imm =>  (MOVimm8:i32 (imm:i32):$imm)
+// CHECK-NEXT:    GIR_BuildMI, /*InsnID*/0, /*Opcode*/MyTarget::MOVimm8,
+// CHECK-NEXT:    GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT:    GIR_CopyConstantAsSImm, /*NewInsnID*/0, /*OldInsnID*/0, // imm
+// CHECK-NEXT:    GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT:    GIR_ConstrainSelectedInstOperands, /*InsnID*/0,
+// CHECK-NEXT:    GIR_Done,
+// CHECK-NEXT:  // Label 16: @[[LABEL]]
+
+def simm8 : ImmLeaf<i32, [{ return isInt<8>(Imm); }]>;
+def MOVimm8 : I<(outs GPR32:$dst), (ins i32imm:$imm), [(set GPR32:$dst, simm8:$imm)]>;
+
+//===- Test a simple pattern with just a leaf immediate. ------------------===//
+
+// CHECK-NEXT:  GIM_Try, /*On fail goto*//*Label 17*/ [[LABEL:[0-9]+]],
+// CHECK-NEXT:    GIM_CheckNumOperands, /*MI*/0, /*Expected*/2,
+// CHECK-NEXT:    GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_CONSTANT,
+// CHECK-NEXT:    // MIs[0] dst
+// CHECK-NEXT:    GIM_CheckType, /*MI*/0, /*Op*/0, /*Type*/GILLT_s32,
+// CHECK-NEXT:    GIM_CheckRegBankForClass, /*MI*/0, /*Op*/0, /*RC*/MyTarget::GPR32RegClassID,
+// CHECK-NEXT:    // MIs[0] Operand 1
+// CHECK-NEXT:    // No operand predicates
 // CHECK-NEXT:    // (imm:i32):$imm =>  (MOVimm:i32 (imm:i32):$imm)
 // CHECK-NEXT:    GIR_BuildMI, /*InsnID*/0, /*Opcode*/MyTarget::MOVimm,
 // CHECK-NEXT:    GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
 // CHECK-NEXT:    GIR_CopyConstantAsSImm, /*NewInsnID*/0, /*OldInsnID*/0, // imm
 // CHECK-NEXT:    GIR_EraseFromParent, /*InsnID*/0,
 // CHECK-NEXT:    GIR_ConstrainSelectedInstOperands, /*InsnID*/0,
 // CHECK-NEXT:    GIR_Done,
-// CHECK-NEXT:  // Label 16: @[[LABEL]]
+// CHECK-NEXT:  // Label 17: @[[LABEL]]
 
 def MOVimm : I<(outs GPR32:$dst), (ins i32imm:$imm), [(set GPR32:$dst, imm:$imm)]>;
 
 //===- Test a pattern with an MBB operand. --------------------------------===//
 
-// CHECK-NEXT:  GIM_Try, /*On fail goto*//*Label 17*/ [[LABEL:[0-9]+]],
+// CHECK-NEXT:  GIM_Try, /*On fail goto*//*Label 18*/ [[LABEL:[0-9]+]],
 // CHECK-NEXT:    GIM_CheckNumOperands, /*MI*/0, /*Expected*/1,
 // CHECK-NEXT:    GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_BR,
 // CHECK-NEXT:    // MIs[0] target
 // CHECK-NEXT:    GIM_CheckIsMBB, /*MI*/0, /*Op*/0,
 // CHECK-NEXT:    // (br (bb:Other):$target) => (BR (bb:Other):$target)
 // CHECK-NEXT:    GIR_MutateOpcode, /*InsnID*/0, /*RecycleInsnID*/0, /*Opcode*/MyTarget::BR,
 // CHECK-NEXT:    GIR_ConstrainSelectedInstOperands, /*InsnID*/0,
 // CHECK-NEXT:    GIR_Done,
-// CHECK-NEXT:  // Label 17: @[[LABEL]]
+// CHECK-NEXT:  // Label 18: @[[LABEL]]
 
 def BR : I<(outs), (ins unknown:$target),
             [(br bb:$target)]>;
 
 // CHECK-NEXT:    GIM_Reject,
 // CHECK-NEXT:  };
 // CHECK-NEXT:  if (executeMatchTable(*this, OutMIs, State, MatcherInfo, MatchTable0, TII, MRI, TRI, RBI, AvailableFeatures)) {
 // CHECK-NEXT:    return true;
 // CHECK-NEXT:  }

utils/TableGen/GlobalISelEmitter.cpp

 /// Helper function to let the emitter report skip reason error messages.
 static Error failedImport(const Twine &Reason) {
   return make_error<StringError>(Reason, inconvertibleErrorCode());
 }
 
 static Error isTrivialOperatorNode(const TreePatternNode *N) {
   std::string Explanation = "";
   std::string Separator = "";
-  if (N->hasAnyPredicate()) {
+
+  bool HasUnsupportedPredicate = false;
+  for (const auto &Predicate : N->getPredicateFns()) {
+    if (Predicate.isAlwaysTrue())
+      continue;
+
+    if (Predicate.isImmediatePattern())
+      continue;
+
+    HasUnsupportedPredicate = true;
     Explanation = Separator + "Has a predicate (" + explainPredicates(N) + ")";
     Separator = ", ";
+    break;
   }
 
   if (N->getTransformFn()) {
     Explanation += Separator + "Has a transform function";
     Separator = ", ";
   }
 
-  if (!N->hasAnyPredicate() && !N->getTransformFn())
+  if (!HasUnsupportedPredicate && !N->getTransformFn())
     return Error::success();
 
   return failedImport(Explanation);
 }
 
 static Record *getInitValueAsRegClass(Init *V) {
   if (DefInit *VDefInit = dyn_cast<DefInit>(V)) {
     if (VDefInit->getDef()->isSubClassOf("RegisterOperand"))
public:
   InstructionMatcher &insnmatcher_front() const { return *Matchers.front(); }
 };
 
 template <class PredicateTy> class PredicateListMatcher {
 private:
   typedef std::vector<std::unique_ptr<PredicateTy>> PredicateVec;
   PredicateVec Predicates;
 
+  /// Template instantiations should specialize this to return a string to use
+  /// for the comment emitted when there are no predicates.
+  std::string getNoPredicateComment() const;
+
 public:
   /// Construct a new operand predicate and add it to the matcher.
   template <class Kind, class... Args>
   Kind &addPredicate(Args&&... args) {
     Predicates.emplace_back(
         llvm::make_unique<Kind>(std::forward<Args>(args)...));
     return *static_cast<Kind *>(Predicates.back().get());
   }
public:
   typename PredicateVec::size_type predicates_size() const {
     return Predicates.size();
   }
 
   /// Emit MatchTable opcodes that tests whether all the predicates are met.
   template <class... Args>
   void emitPredicateListOpcodes(MatchTable &Table, Args &&... args) const {
     if (Predicates.empty()) {
-      Table << MatchTable::Comment("No predicates") << MatchTable::LineBreak;
+      Table << MatchTable::Comment(getNoPredicateComment())
+            << MatchTable::LineBreak;
       return;
     }
 
     for (const auto &Predicate : predicates())
       Predicate->emitPredicateOpcodes(Table, std::forward<Args>(args)...);
   }
 };
 
public:
   /// Returns true if this object is more important than B.
   virtual bool isHigherPriorityThan(const OperandPredicateMatcher &B) const;
 
   /// Report the maximum number of temporary operands needed by the predicate
   /// matcher.
   virtual unsigned countRendererFns() const { return 0; }
 };
 
+template <>
+std::string
+PredicateListMatcher<OperandPredicateMatcher>::getNoPredicateComment() const {
+  return "No operand predicates";
+}
+
 /// Generates code to check that an operand is a particular LLT.
 class LLTOperandMatcher : public OperandPredicateMatcher {
 protected:
   LLTCodeGen Ty;
 
 public:
   static std::set<LLTCodeGen> KnownTypes;
 

 /// * The nsw/nuw flag is/isn't set.
 class InstructionPredicateMatcher {
 protected:
   /// This enum is used for RTTI and also defines the priority that is given to
   /// the predicate when generating the matcher code. Kinds with higher priority
   /// must be tested first.
   enum PredicateKind {
     IPM_Opcode,
+    IPM_ImmPredicate,
   };
 
   PredicateKind Kind;
 
 public:
   InstructionPredicateMatcher(PredicateKind Kind) : Kind(Kind) {}
   virtual ~InstructionPredicateMatcher() {}
 
isHigherPriorityThan(const InstructionPredicateMatcher &B) const {
     return Kind < B.Kind;
   };
 
   /// Report the maximum number of temporary operands needed by the predicate
   /// matcher.
   virtual unsigned countRendererFns() const { return 0; }
 };
 
+template <>
+std::string
+PredicateListMatcher<InstructionPredicateMatcher>::getNoPredicateComment() const {
+  return "No instruction predicates";
+}
+
 /// Generates code to check the opcode of an instruction.
 class InstructionOpcodeMatcher : public InstructionPredicateMatcher {
 protected:
   const CodeGenInstruction *I;
 
 public:
   InstructionOpcodeMatcher(const CodeGenInstruction *I)
       : InstructionPredicateMatcher(IPM_Opcode), I(I) {}
isHigherPriorityThan(const InstructionPredicateMatcher &B) const override {
     return false;
   };
 
   bool isConstantInstruction() const {
     return I->TheDef->getName() == "G_CONSTANT";
   }
 };
 
+/// Generates code to check that this instruction is a constant whose value
+/// meets an immediate predicate.
+///
+/// Immediates are slightly odd since they are typically used like an operand
+/// but are represented as an operator internally. We typically write simm8:$src
+/// in a tablegen pattern, but this is just syntactic sugar for
+/// (imm:i32)<<P:Predicate_simm8>>:$imm which more directly describes the nodes
+/// that will be matched and the predicate (which is attached to the imm
+/// operator) that will be tested. In SelectionDAG this describes a
+/// ConstantSDNode whose internal value will be tested using the simm8 predicate.
+///
+/// The corresponding GlobalISel representation is %1 = G_CONSTANT iN Value. In
+/// this representation, the immediate could be tested with an
+/// InstructionMatcher, InstructionOpcodeMatcher, OperandMatcher, and a
+/// OperandPredicateMatcher-subclass to check the Value meets the predicate but
+/// there are two implementation issues with producing that matcher
+/// configuration from the SelectionDAG pattern:
+/// * ImmLeaf is a PatFrag whose root is an InstructionMatcher. This means that
+///   were we to sink the immediate predicate to the operand we would have to
+///   have two partial implementations of PatFrag support, one for immediates
+///   and one for non-immediates.
+/// * At the point we handle the predicate, the OperandMatcher hasn't been
+///   created yet. If we were to sink the predicate to the OperandMatcher we
+///   would also have to complicate (or duplicate) the code that descends and
+///   creates matchers for the subtree.
+/// Overall, it's simpler to handle it in the place it was found.
+class InstructionImmPredicateMatcher : public InstructionPredicateMatcher {
+protected:
+  TreePredicateFn Predicate;
+
+public:
+  InstructionImmPredicateMatcher(const TreePredicateFn &Predicate)
+      : InstructionPredicateMatcher(IPM_ImmPredicate), Predicate(Predicate) {}
+
+  static bool classof(const InstructionPredicateMatcher *P) {
+    return P->getKind() == IPM_ImmPredicate;
+  }
+
+  void emitPredicateOpcodes(MatchTable &Table, RuleMatcher &Rule,
+                            unsigned InsnVarID) const override {
+    Table << MatchTable::Opcode("GIM_CheckImmPredicate")
+          << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
+          << MatchTable::Comment("Predicate")
+          << MatchTable::NamedValue("GIPFP_" + Predicate.getFnName())
+          << MatchTable::LineBreak;
+  }
+};
+
 /// Generates code to check that a set of predicates and operands match for a
 /// particular instruction.
 ///
 /// Typical predicates include:
 /// * Has a specific opcode.
 /// * Has an nsw/nuw flag or doesn't.
 class InstructionMatcher
     : public PredicateListMatcher<InstructionPredicateMatcher> {
if (!OpTyOrNone)
           "Result of Src pattern operator has an unsupported type");
 
     // Results don't have a name unless they are the root node. The caller will
     // set the name if appropriate.
     OperandMatcher &OM = InsnMatcher.addOperand(OpIdx++, "", TempOpIdx);
     OM.addPredicate<LLTOperandMatcher>(*OpTyOrNone);
   }
 
+  for (const auto &Predicate : Src->getPredicateFns()) {
+    if (Predicate.isAlwaysTrue())
+      continue;
+
+    if (Predicate.isImmediatePattern()) {
+      InsnMatcher.addPredicate<InstructionImmPredicateMatcher>(Predicate);
+      continue;
+    }
+
+    return failedImport("Src pattern child has predicate (" +
+                        explainPredicates(Src) + ")");
+  }
+
   if (Src->isLeaf()) {
     Init *SrcInit = Src->getLeafValue();
     if (IntInit *SrcIntInit = dyn_cast<IntInit>(SrcInit)) {
       OperandMatcher &OM = InsnMatcher.addOperand(OpIdx++, "", TempOpIdx);
       OM.addPredicate<LiteralIntOperandMatcher>(SrcIntInit->getValue());
     } else
       return failedImport(
           "Unable to deduce gMIR opcode to handle Src (which is a leaf)");

 
 Error GlobalISelEmitter::importChildMatcher(InstructionMatcher &InsnMatcher,
                                             const TreePatternNode *SrcChild,
                                             unsigned OpIdx,
                                             unsigned &TempOpIdx) const {
   OperandMatcher &OM =
       InsnMatcher.addOperand(OpIdx, SrcChild->getName(), TempOpIdx);
 
-  if (SrcChild->hasAnyPredicate())
-    return failedImport("Src pattern child has predicate (" +
-                        explainPredicates(SrcChild) + ")");
-
   ArrayRef<EEVT::TypeSet> ChildTypes = SrcChild->getExtTypes();
   if (ChildTypes.size() != 1)
     return failedImport("Src pattern child has multiple results");
 
   // Check MBB's before the type check since they are not a known type.
   if (!SrcChild->isLeaf()) {
     if (SrcChild->getOperator()->isSubClassOf("SDNode")) {
       auto &ChildSDNI = CGP.getSDNodeInfo(SrcChild->getOperator());
Error GlobalISelEmitter::importChildMatcher(InstructionMatcher &InsnMatcher,
   }
 
   return failedImport("Src pattern child is an unsupported kind");
 }
 
 Error GlobalISelEmitter::importExplicitUseRenderer(
     BuildMIAction &DstMIBuilder, TreePatternNode *DstChild,
     const InstructionMatcher &InsnMatcher) const {
+  if (DstChild->getTransformFn() != nullptr) {
+    return failedImport("Dst pattern child has transform fn " +
+                        DstChild->getTransformFn()->getName());
+  }
+
   if (!DstChild->isLeaf()) {
     // We accept 'bb' here. It's an operator because BasicBlockSDNode isn't
     // inline, but in MI it's just another operand.
     if (DstChild->getOperator()->isSubClassOf("SDNode")) {
       auto &ChildSDNI = CGP.getSDNodeInfo(DstChild->getOperator());
       if (ChildSDNI.getSDClassName() == "BasicBlockSDNode") {
         DstMIBuilder.addRenderer<CopyRenderer>(0, InsnMatcher,
                                                DstChild->getName());
         return Error::success();
       }
     }
 
     // Similarly, imm is an operator in TreePatternNode's view but must be
     // rendered as operands.
     // FIXME: The target should be able to choose sign-extended when appropriate
     //        (e.g. on Mips).
     if (DstChild->getOperator()->getName() == "imm") {
       DstMIBuilder.addRenderer<CopyConstantAsImmRenderer>(0,
                                                           DstChild->getName());
       return Error::success();
     }
 
-    return failedImport("Dst pattern child isn't a leaf node or an MBB");
+    return failedImport("Dst pattern child isn't a leaf node or an MBB" + llvm::to_string(*DstChild));
   }
 
   // Otherwise, we're looking for a bog-standard RegisterClass operand.
-  if (DstChild->hasAnyPredicate())
-    return failedImport("Dst pattern child has predicate (" +
-                        explainPredicates(DstChild) + ")");
-
   if (auto *ChildDefInit = dyn_cast<DefInit>(DstChild->getLeafValue())) {
     auto *ChildRec = ChildDefInit->getDef();
 
     ArrayRef<EEVT::TypeSet> ChildTypes = DstChild->getExtTypes();
     if (ChildTypes.size() != 1)
       return failedImport("Dst pattern child has multiple results");
 
     auto OpTyOrNone = MVTToLLT(ChildTypes.front().getConcrete());
OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n"
      << Target.getName()
      << "InstructionSelector::*ComplexMatcherMemFn)(MachineOperand &) const;\n"
      << "const MatcherInfoTy<PredicateBitset, ComplexMatcherMemFn> "
         "MatcherInfo;\n"
      << "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n\n";
 
   OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n"
      << ", State(" << MaxTemporaries << "),\n"
-     << "MatcherInfo({TypeObjects, FeatureBitsets, {\n"
+     << "MatcherInfo({TypeObjects, FeatureBitsets, ImmPredicateFns, {\n"
      << "  nullptr, // GICP_Invalid\n";
   for (const auto &Record : ComplexPredicates)
     OS << "  &" << Target.getName()
        << "InstructionSelector::" << Record->getValueAsString("MatcherFn")
        << ", // " << Record->getName() << "\n";
   OS << "}})\n"
      << "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n\n";
 
void GlobalISelEmitter::run(raw_ostream &OS) {
   // Emit complex predicate table and an enum to reference them with.
   OS << "enum {\n"
      << "  GICP_Invalid,\n";
   for (const auto &Record : ComplexPredicates)
     OS << "  GICP_" << Record->getName() << ",\n";
   OS << "};\n"
      << "// See constructor for table contents\n\n";
 
+  // Emit imm predicate table and an enum to reference them with.
+  // The 'Predicate_' part of the name is redundant but eliminating it is more
+  // trouble than it's worth.
+  {
+    OS << "// PatFrag predicates.\n"
+       << "enum {\n";
+    StringRef EnumeratorSeparator = " = GIPFP_Invalid,\n";
+    for (const auto *Record : RK.getAllDerivedDefinitions("PatFrag")) {
+      if (!Record->getValueAsString("ImmediateCode").empty()) {
+        OS << "  GIPFP_Predicate_" << Record->getName() << EnumeratorSeparator;
+        EnumeratorSeparator = ",\n";
+      }
+    }
+    OS << "};\n";
+  }
+  for (const auto *Record : RK.getAllDerivedDefinitions("PatFrag"))
+    if (!Record->getValueAsString("ImmediateCode").empty())
+      OS << "  static bool Predicate_" << Record->getName() << "(int64_t Imm) {"
+         << Record->getValueAsString("ImmediateCode") << "  }\n";
+  OS << "static InstructionSelector::ImmediatePredicateFn ImmPredicateFns[] = {\n";
+  for (const auto *Record : RK.getAllDerivedDefinitions("PatFrag"))
+    if (!Record->getValueAsString("ImmediateCode").empty())
+      OS << "  Predicate_" << Record->getName() << ",\n";
+  OS << "};\n";
+
   OS << "bool " << Target.getName()
      << "InstructionSelector::selectImpl(MachineInstr &I) const {\n"
      << "  MachineFunction &MF = *I.getParent()->getParent();\n"
      << "  MachineRegisterInfo &MRI = MF.getRegInfo();\n"
      << "  // FIXME: This should be computed on a per-function basis rather "
         "than per-insn.\n"
      << "  AvailableFunctionFeatures = computeAvailableFunctionFeatures(&STI, "
         "&MF);\n"