[globalisel][tblgen] Add support for ComplexPatterns

[dsanders]

Summary

Adds a new kind of MachineOperand: MO_Placeholder.
This operand must not appear in the MIR and only exists as a way of
creating an 'uninitialized' operand until a matcher function overwrites it.

Depends on D30046, D29712

[qcolombet]

lib/Target/AArch64/AArch64InstructionSelector.cpp

1264

Can't we modify the existing placeholder instead of creating a temporary object and then use the copy constructor?

[dsanders]

lib/Target/AArch64/AArch64InstructionSelector.cpp

1264

I think we should be able to use ChangeToImmediate(), but I haven't tried it yet. The bit that worried me about doing that was having stale values in members that aren't affected by the ChangeTo*() functions. Most of them look ok (since they only matter for registers and ChangeToRegister() does reset them) but I'm not sure about MachineOperand::ParentMI.

[qcolombet]

One remark: I can see how a placeholder operand can help detecting errors (like we forgot to update one of the operand), however, I think it gets in the way of efficiency.
Indeed, I imagine that each time a complex pattern will fail half way through, we will have to reset all the operands to placeholder instead of just keeping them in the state they are.

What I am saying is that we may not introduce a placeholder operand at all. We should have a placeholder opcode though that reset the operands array to 0 (without freeing memory).

lib/Target/AArch64/AArch64InstructionSelector.cpp

1264

That should be easy to check/fix :).

[dsanders]

Sorry for the slow reply on this one.

In D30089#680485, @qcolombet wrote:

One remark: I can see how a placeholder operand can help detecting errors (like we forgot to update one of the operand), however, I think it gets in the way of efficiency.
Indeed, I imagine that each time a complex pattern will fail half way through, we will have to reset all the operands to placeholder instead of just keeping them in the state they are.

It's not really aimed at detecting errors but rather at constructing the MachineOperand without creating fake data to put in it. Each successful ComplexPattern predicate will fully overwrite the placeholder (using the default copy constructor) so I don't think we need to reset it between rules. I can see how it would be useful to reset it to the placeholder data to ensure the predicate is behaving correctly though since if a predicate returns true but the placeholder data is still there then there's a bug in the predicate.

One other thing to mention is that I'd like to skip the constructor call completely and leave it uninitialized until the first successful ComplexPattern predicate overwrites it. Initializing the operands to a placeholder is just unnecessary overhead.

What I am saying is that we may not introduce a placeholder operand at all. We should have a placeholder opcode though that reset the operands array to 0 (without freeing memory).

What would be the benefit of a placeholder instruction over placeholder operands? I think it might be slightly higher cost since if we reset the operand list between rules then it increases the number of MachineOperand constructor calls (but not allocations since it uses the placement new operator). A placeholder instruction also means the instruction and operands are heap allocated instead of stack allocated and will therefore cause a memory leak unless additional overhead to free the instruction is added.

lib/Target/AArch64/AArch64InstructionSelector.cpp

1264

When we create new operands, the ChangeTo*() family are ok so long as you also call clearParent() afterwards to clear and parents left over from previous predicates.

[dsanders]

Use the ChangeTo*() family.

[qcolombet]

What would be the benefit of a placeholder instruction over placeholder operands?

I expect the placeholder reset to be something just a single assignment of the style size of array operand = 0. The operands remain as they are, they are just not accessed. In other words, no constructor call.

[dsanders]

Further reduce constructor call overhead by moving the temporaries to the class.

[dsanders]

In D30089#695620, @qcolombet wrote:

What would be the benefit of a placeholder instruction over placeholder operands?

I expect the placeholder reset to be something just a single assignment of the style size of array operand = 0. The operands remain as they are, they are just not accessed. In other words, no constructor call.

The constructor call comes from the need to call addOperand(). If we reset the operand list for the placeholder instruction on each call to selectImpl() then we would need something like:

PlaceholderInsn.addOperand(MachineOperand::CreatePlaceholder());
PlaceholderInsn.addOperand(MachineOperand::CreatePlaceholder());
if (matches && selectComplexPattern(PlaceholderInsn.getOperand(0), PlaceholderInsn.getOperand(1))) {
  BuildMI(...).add(PlaceholderInsn.getOperand(0)).add(PlaceholderInsn.getOperand(1));
  ...
}
PlaceholderInsn.resetOperands();

Each call to addOperand() involves a placement new operator which uses the copy-constructor and therefore pays the same overhead as the current local variable approach.

I think your main concern is that the overhead is unnecessarily paid on each call to selectImpl(). I've updated the patch with a version that only pays the constructor overhead once (during <Target>InstructionSelection's constructor). Does that resolve your concerns?

[qcolombet]

The constructor call comes from the need to call addOperand(). If we reset the operand list for the placeholder instruction on each call to selectImpl() then we would need something like:

PlaceholderInsn.addOperand(MachineOperand::CreatePlaceholder());
PlaceholderInsn.addOperand(MachineOperand::CreatePlaceholder());
if (matches && selectComplexPattern(PlaceholderInsn.getOperand(0), PlaceholderInsn.getOperand(1))) {
BuildMI(...).add(PlaceholderInsn.getOperand(0)).add(PlaceholderInsn.getOperand(1));
...
}
PlaceholderInsn.resetOperands();
Each call to addOperand() involves a placement new operator which uses the copy-constructor and therefore pays the same overhead as the current local variable approach.

I was actually expecting more something like:
PlaceHolderInst.setNumOperand(NbOp);
With that call just allocating/resizing the operand array accordingly. The operands created could be left uninitialized as they are supposed to be all properly set by the matching.

I think your main concern is that the overhead is unnecessarily paid on each call to selectImpl(). I've updated the patch with a version that only pays the constructor overhead once (during <Target>InstructionSelection's constructor). Does that resolve your concerns?

Yes, partly. I believe we don't need the OperandHolder all together, but this is an optimization that we definitely don't need to do now.

All in all, the patch looks sensible to me.

lib/Target/AArch64/AArch64InstructionSelector.cpp

1264

I would have expected this to be handled by the ChangeTo* APIs.
There might be a good reason why this is not the case. Do you know?

[dsanders]

In D30089#698324, @qcolombet wrote:

The constructor call comes from the need to call addOperand(). If we reset the operand list for the placeholder instruction on each call to selectImpl() then we would need something like:

PlaceholderInsn.addOperand(MachineOperand::CreatePlaceholder());
PlaceholderInsn.addOperand(MachineOperand::CreatePlaceholder());
if (matches && selectComplexPattern(PlaceholderInsn.getOperand(0), PlaceholderInsn.getOperand(1))) {
BuildMI(...).add(PlaceholderInsn.getOperand(0)).add(PlaceholderInsn.getOperand(1));
...
}
PlaceholderInsn.resetOperands();
Each call to addOperand() involves a placement new operator which uses the copy-constructor and therefore pays the same overhead as the current local variable approach.

I was actually expecting more something like:
PlaceHolderInst.setNumOperand(NbOp);
With that call just allocating/resizing the operand array accordingly. The operands created could be left uninitialized as they are supposed to be all properly set by the matching.

Ah ok, I see how that would avoid the redundant initializations. Wouldn't it be more efficient to call setNumOperands(MaxNumOperands) in advance though to save on re-allocating and copying the array?

I think your main concern is that the overhead is unnecessarily paid on each call to selectImpl(). I've updated the patch with a version that only pays the constructor overhead once (during <Target>InstructionSelection's constructor). Does that resolve your concerns?

Yes, partly. I believe we don't need the OperandHolder all together, but this is an optimization that we definitely don't need to do now.

I'm not sure we can eliminate the placeholders since storing directly to the output instruction requires us to build that instruction before we've finished matching but I'll have a think on how we can do it.

All in all, the patch looks sensible to me.

Thanks.

include/llvm/CodeGen/MachineOperand.h

enum MachineOperandType : unsigned char {
     MO_BlockAddress,      ///< Address of a basic block
     MO_RegisterMask,      ///< Mask of preserved registers.
     MO_RegisterLiveOut,   ///< Mask of live-out registers.
     MO_Metadata,          ///< Metadata reference (for debug info)
     MO_MCSymbol,          ///< MCSymbol reference (for debug/eh info)
     MO_CFIIndex,          ///< MCCFIInstruction index.
     MO_IntrinsicID,       ///< Intrinsic ID for ISel
     MO_Predicate,         ///< Generic predicate for ISel
+    MO_Placeholder,       ///< Placeholder for GlobalISel ComplexPattern result.
   };
 
 private:
   /// OpKind - Specify what kind of operand this is.  This discriminates the
   /// union.
   MachineOperandType OpKind : 8;
 
   /// Subregister number for MO_Register.  A value of 0 indicates the
public:
   }
 
   static MachineOperand CreatePredicate(unsigned Pred) {
     MachineOperand Op(MachineOperand::MO_Predicate);
     Op.Contents.Pred = Pred;
     return Op;
   }
 
+  static MachineOperand CreatePlaceholder() {
+    MachineOperand Op(MachineOperand::MO_Placeholder);
+    return Op;
+  }
+
   friend class MachineInstr;
   friend class MachineRegisterInfo;
 private:
   void removeRegFromUses();
 
   //===--------------------------------------------------------------------===//
   // Methods for handling register use/def lists.
   //===--------------------------------------------------------------------===//

include/llvm/Target/GlobalISel/SelectionDAGCompat.td

 def : GINodeEquiv<G_ASHR, sra>;
 
 def : GINodeEquiv<G_SDIV, sdiv>;
 def : GINodeEquiv<G_UDIV, udiv>;
 def : GINodeEquiv<G_SREM, srem>;
 def : GINodeEquiv<G_UREM, urem>;
 
 def : GINodeEquiv<G_BR, br>;
+
+// Specifies the GlobalISel equivalents for SelectionDAG's ComplexPattern.
+// Should be used on defs that subclass GIComplexOperandMatcher<>.
+class GIComplexPatternEquiv<ComplexPattern seldag> {
+  ComplexPattern SelDAGEquivalent = seldag;
+}

include/llvm/Target/GlobalISel/Target.td

+//===- Target.td - Define GlobalISel rules -----------------*- tablegen -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the target-independent interfaces used to support
+// SelectionDAG instruction selection patterns (specified in
+// TargetSelectionDAG.td) when generating GlobalISel instruction selectors.
+//
+// This is intended as a compatibility layer, to enable reuse of target
+// descriptions written for SelectionDAG without requiring explicit GlobalISel
+// support.  It will eventually supersede SelectionDAG patterns.
+//
+//===----------------------------------------------------------------------===//
+
+// Definitions that inherit from LLT define types that will be used in the
+// GlobalISel matcher.
+class LLT;
+
+def s32 : LLT;
+
+// Defines a matcher for complex operands. This is analogous to ComplexPattern
+// from SelectionDAG.
+//
+// Definitions that inherit from this may also inherit from
+// GIComplexPatternEquiv to enable the import of SelectionDAG patterns involving
+// those ComplexPatterns.
+class GIComplexOperandMatcher<LLT type, dag operands, string matcherfn> {
+  // The expected type of the root of the match.
+  //
+  // TODO: We should probably support, any-type, any-scalar, and multiple types
+  //       in the future.
+  LLT Type = type;
+
+  // The operands that result from a successful match
+  // Should be of the form '(ops ty1, ty2, ...)' where ty1/ty2 are definitions
+  // that inherit from Operand.
+  //
+  // FIXME: Which definition is used for ty1/ty2 doesn't actually matter at the
+  //        moment. Only the number of operands is used.
+  dag Operands = operands;
+
+  // The function that determines whether the operand matches. It should be of
+  // the form:
+  //   bool select(const MatchOperand &Root, MatchOperand &Result1)
+  // and should have the same number of ResultX arguments as the number of
+  // result operands. It must return true on successful match and false
+  // otherwise. If it returns true, then all the ResultX arguments must be
+  // overwritten.
+  string MatcherFn = matcherfn;
+}

include/llvm/Target/Target.td

 include "llvm/Target/TargetSelectionDAG.td"
 
 //===----------------------------------------------------------------------===//
 // Pull in the common support for Global ISel register bank info generation.
 //
 include "llvm/Target/GlobalISel/RegisterBank.td"
 
 //===----------------------------------------------------------------------===//
+// Pull in the common support for DAG isel generation.
+//
+include "llvm/Target/GlobalISel/Target.td"
+
+//===----------------------------------------------------------------------===//
 // Pull in the common support for the Global ISel DAG-based selector generation.
 //
 include "llvm/Target/GlobalISel/SelectionDAGCompat.td"

lib/CodeGen/MIRPrinter.cpp

case MachineOperand::MO_IntrinsicID: {
     break;
   }
   case MachineOperand::MO_Predicate: {
     auto Pred = static_cast<CmpInst::Predicate>(Op.getPredicate());
     OS << (CmpInst::isIntPredicate(Pred) ? "int" : "float") << "pred("
        << CmpInst::getPredicateName(Pred) << ')';
     break;
   }
+  case MachineOperand::MO_Placeholder:
+    OS << "<placeholder>";
+    break;
   }
 }
 
 void MIPrinter::print(const MachineMemOperand &Op) {
   OS << '(';
   // TODO: Print operand's target specific flags.
   if (Op.isVolatile())
     OS << "volatile ";

lib/CodeGen/MachineInstr.cpp

bool MachineOperand::isIdenticalTo(const MachineOperand &Other) const {
   case MachineOperand::MO_CFIIndex:
     return getCFIIndex() == Other.getCFIIndex();
   case MachineOperand::MO_Metadata:
     return getMetadata() == Other.getMetadata();
   case MachineOperand::MO_IntrinsicID:
     return getIntrinsicID() == Other.getIntrinsicID();
   case MachineOperand::MO_Predicate:
     return getPredicate() == Other.getPredicate();
+  case MachineOperand::MO_Placeholder:
+    return true;
   }
   llvm_unreachable("Invalid machine operand type");
 }
 
 // Note: this must stay exactly in sync with isIdenticalTo above.
 hash_code llvm::hash_value(const MachineOperand &MO) {
   switch (MO.getType()) {
   case MachineOperand::MO_Register:
hash_code llvm::hash_value(const MachineOperand &MO) {
   case MachineOperand::MO_MCSymbol:
     return hash_combine(MO.getType(), MO.getTargetFlags(), MO.getMCSymbol());
   case MachineOperand::MO_CFIIndex:
     return hash_combine(MO.getType(), MO.getTargetFlags(), MO.getCFIIndex());
   case MachineOperand::MO_IntrinsicID:
     return hash_combine(MO.getType(), MO.getTargetFlags(), MO.getIntrinsicID());
   case MachineOperand::MO_Predicate:
     return hash_combine(MO.getType(), MO.getTargetFlags(), MO.getPredicate());
+  case MachineOperand::MO_Placeholder:
+    return hash_combine();
   }
   llvm_unreachable("Invalid machine operand type");
 }
 
 void MachineOperand::print(raw_ostream &OS, const TargetRegisterInfo *TRI,
                            const TargetIntrinsicInfo *IntrinsicInfo) const {
   ModuleSlotTracker DummyMST(nullptr);
   print(OS, DummyMST, TRI, IntrinsicInfo);
case MachineOperand::MO_IntrinsicID: {
     else
       OS << "<intrinsic:" << ID << '>';
     break;
   }
   case MachineOperand::MO_Predicate: {
     auto Pred = static_cast<CmpInst::Predicate>(getPredicate());
     OS << '<' << (CmpInst::isIntPredicate(Pred) ? "intpred" : "floatpred")
        << CmpInst::getPredicateName(Pred) << '>';
+    break;
   }
+  case MachineOperand::MO_Placeholder:
+    OS << "<placeholder>";
+    break;
   }
   if (unsigned TF = getTargetFlags())
     OS << "[TF=" << TF << ']';
 }
 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 LLVM_DUMP_METHOD void MachineOperand::dump() const {
   dbgs() << *this << '\n';

lib/Target/AArch64/AArch64InstrFormats.td

class addsub_shifted_imm_neg<ValueType Ty>
   let MIOperandInfo = (ops i32imm, i32imm);
 }
 
 def addsub_shifted_imm32 : addsub_shifted_imm<i32>;
 def addsub_shifted_imm64 : addsub_shifted_imm<i64>;
 def addsub_shifted_imm32_neg : addsub_shifted_imm_neg<i32>;
 def addsub_shifted_imm64_neg : addsub_shifted_imm_neg<i64>;
 
+def gi_addsub_shifted_imm32 :
+    GIComplexOperandMatcher<s32, (ops i32imm, i32imm), "selectArithImmed">,
+    GIComplexPatternEquiv<addsub_shifted_imm32>;
+
 class neg_addsub_shifted_imm<ValueType Ty>
     : Operand<Ty>, ComplexPattern<Ty, 2, "SelectNegArithImmed", [imm]> {
   let PrintMethod = "printAddSubImm";
   let EncoderMethod = "getAddSubImmOpValue";
   let ParserMatchClass = AddSubImmOperand;
   let MIOperandInfo = (ops i32imm, i32imm);
 }
 

lib/Target/AArch64/AArch64InstructionSelector.h

 /// This file declares the targeting of the InstructionSelector class for
 /// AArch64.
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIB_TARGET_AARCH64_AARCH64INSTRUCTIONSELECTOR_H
 #define LLVM_LIB_TARGET_AARCH64_AARCH64INSTRUCTIONSELECTOR_H
 
 #include "llvm/CodeGen/GlobalISel/InstructionSelector.h"
+#include "llvm/CodeGen/MachineOperand.h"
 
 namespace llvm {
 
 class AArch64InstrInfo;
 class AArch64RegisterBankInfo;
 class AArch64RegisterInfo;
 class AArch64Subtarget;
 class AArch64TargetMachine;
+class MachineOperand;
 
 class MachineFunction;
 class MachineRegisterInfo;
 
 class AArch64InstructionSelector : public InstructionSelector {
 public:
   AArch64InstructionSelector(const AArch64TargetMachine &TM,
                              const AArch64Subtarget &STI,
                              const AArch64RegisterBankInfo &RBI);
 
   bool select(MachineInstr &I) const override;
 
 private:
   bool selectVaStartAAPCS(MachineInstr &I, MachineFunction &MF,
                           MachineRegisterInfo &MRI) const;
   bool selectVaStartDarwin(MachineInstr &I, MachineFunction &MF,
                            MachineRegisterInfo &MRI) const;
 
   /// tblgen-erated 'select' implementation, used as the initial selector for
   /// the patterns that don't require complex C++.
   bool selectImpl(MachineInstr &I) const;
 
+  bool selectArithImmed(MachineOperand &Root, MachineOperand &Result1,
+                        MachineOperand &Result2) const;
+
   const AArch64TargetMachine &TM;
   const AArch64Subtarget &STI;
   const AArch64InstrInfo &TII;
   const AArch64RegisterInfo &TRI;
   const AArch64RegisterBankInfo &RBI;
+
+// We declare the temporaries used by selectImpl() in the class to minimize the
+// cost of constructing placeholder values.
+#define GET_GLOBALISEL_TEMPORARIES_DECL
+#include "AArch64GenGlobalISel.inc"
+#undef GET_GLOBALISEL_TEMPORARIES_DECL
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_AARCH64_AARCH64INSTRUCTIONSELECTOR_H

lib/Target/AArch64/AArch64InstructionSelector.cpp

 #define DEBUG_TYPE "aarch64-isel"
 
 using namespace llvm;
 
 #ifndef LLVM_BUILD_GLOBAL_ISEL
 #error "You shouldn't build this"
 #endif
 
+#define GET_GLOBALISEL_IMPL
 #include "AArch64GenGlobalISel.inc"
+#undef GET_GLOBALISEL_IMPL
 
 AArch64InstructionSelector::AArch64InstructionSelector(
     const AArch64TargetMachine &TM, const AArch64Subtarget &STI,
     const AArch64RegisterBankInfo &RBI)
     : InstructionSelector(), TM(TM), STI(STI), TII(*STI.getInstrInfo()),
-      TRI(*STI.getRegisterInfo()), RBI(RBI) {}
+      TRI(*STI.getRegisterInfo()), RBI(RBI)
+#define GET_GLOBALISEL_TEMPORARIES_INIT
+#include "AArch64GenGlobalISel.inc"
+#undef GET_GLOBALISEL_TEMPORARIES_INIT
+{
+}
 
 // FIXME: This should be target-independent, inferred from the types declared
 // for each class in the bank.
 static const TargetRegisterClass *
 getRegClassForTypeOnBank(LLT Ty, const RegisterBank &RB,
                          const RegisterBankInfo &RBI) {
   if (RB.getID() == AArch64::GPRRegBankID) {
     if (Ty.getSizeInBits() <= 32)
#endif
   }
   case TargetOpcode::G_VASTART:
     return STI.isTargetDarwin() ? selectVaStartDarwin(I, MF, MRI)
                                 : selectVaStartAAPCS(I, MF, MRI);
   }
 
   return false;
 }
+
+/// SelectArithImmed - Select an immediate value that can be represented as
+/// a 12-bit value shifted left by either 0 or 12.  If so, return true with
+/// Val set to the 12-bit value and Shift set to the shifter operand.
+bool AArch64InstructionSelector::selectArithImmed(
+    MachineOperand &Root, MachineOperand &Result1,
+    MachineOperand &Result2) const {
+  MachineInstr &MI = *Root.getParent();
+  MachineBasicBlock &MBB = *MI.getParent();
+  MachineFunction &MF = *MBB.getParent();
+  MachineRegisterInfo &MRI = MF.getRegInfo();
+
+  // This function is called from the addsub_shifted_imm ComplexPattern,
+  // which lists [imm] as the list of opcode it's interested in, however
+  // we still need to check whether the operand is actually an immediate
+  // here because the ComplexPattern opcode list is only used in
+  // root-level opcode matching.
+  uint64_t Immed;
+  if (Root.isImm())
+    Immed = Root.getImm();
+  else if (Root.isCImm())
+    Immed = Root.getCImm()->getZExtValue();
+  else if (Root.isReg()) {
+    MachineInstr *Def = MRI.getVRegDef(Root.getReg());
+    if (Def->getOpcode() != TargetOpcode::G_CONSTANT)
+      return false;
+    Immed = Def->getOperand(1).getImm();
+  } else
+    return false;
+
+  unsigned ShiftAmt;
+
+  if (Immed >> 12 == 0) {
+    ShiftAmt = 0;
+  } else if ((Immed & 0xfff) == 0 && Immed >> 24 == 0) {
+    ShiftAmt = 12;
+    Immed = Immed >> 12;
+  } else
+    return false;
+
+  unsigned ShVal = AArch64_AM::getShifterImm(AArch64_AM::LSL, ShiftAmt);
+  Result1.ChangeToImmediate(Immed);
+  Result1.clearParent();
+  Result2.ChangeToImmediate(ShVal);
+  Result2.clearParent();
+  return true;
+}

lib/Target/ARM/ARMExpandPseudoInsts.cpp

case MachineOperand::MO_RegisterLiveOut:
     return false;
   case MachineOperand::MO_Metadata:
   case MachineOperand::MO_MCSymbol:
     return true;
   case MachineOperand::MO_CFIIndex:
     return false;
   case MachineOperand::MO_IntrinsicID:
   case MachineOperand::MO_Predicate:
+  case MachineOperand::MO_Placeholder:
     llvm_unreachable("should not exist post-isel");
   }
   llvm_unreachable("unhandled machine operand type");
 }
 
 void ARMExpandPseudo::ExpandMOV32BitImm(MachineBasicBlock &MBB,
                                         MachineBasicBlock::iterator &MBBI) {
   MachineInstr &MI = *MBBI;

lib/Target/X86/X86InstructionSelector.h

 //===- X86InstructionSelector --------------------------------*- C++ -*-==//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 /// \file
 /// This file declares the targeting of the InstructionSelector class for X86.
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIB_TARGET_X86_X86INSTRUCTIONSELECTOR_H
 #define LLVM_LIB_TARGET_X86_X86INSTRUCTIONSELECTOR_H
 
 #include "llvm/CodeGen/GlobalISel/InstructionSelector.h"
+#include "llvm/CodeGen/MachineOperand.h"
 
 namespace llvm {
 
 class X86InstrInfo;
 class X86RegisterBankInfo;
 class X86RegisterInfo;
 class X86Subtarget;
 class X86TargetMachine;
private:
   unsigned getAddOp(LLT &Ty, const RegisterBank &RB) const;
   unsigned getSubOp(LLT &Ty, const RegisterBank &RB) const;
   bool selectBinaryOp(MachineInstr &I, MachineRegisterInfo &MRI) const;
 
   const X86Subtarget &STI;
   const X86InstrInfo &TII;
   const X86RegisterInfo &TRI;
   const X86RegisterBankInfo &RBI;
+
+#define GET_GLOBALISEL_TEMPORARIES_DECL
+#include "X86GenGlobalISel.inc"
+#undef GET_GLOBALISEL_TEMPORARIES_DECL
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_X86_X86INSTRUCTIONSELECTOR_H

lib/Target/X86/X86InstructionSelector.cpp

 #define DEBUG_TYPE "X86-isel"
 
 using namespace llvm;
 
 #ifndef LLVM_BUILD_GLOBAL_ISEL
 #error "You shouldn't build this"
 #endif
 
+#define GET_GLOBALISEL_IMPL
 #include "X86GenGlobalISel.inc"
+#undef GET_GLOBALISEL_IMPL
 
 X86InstructionSelector::X86InstructionSelector(const X86Subtarget &STI,
                                                const X86RegisterBankInfo &RBI)
     : InstructionSelector(), STI(STI), TII(*STI.getInstrInfo()),
-      TRI(*STI.getRegisterInfo()), RBI(RBI) {}
+      TRI(*STI.getRegisterInfo()), RBI(RBI)
+#define GET_GLOBALISEL_TEMPORARIES_INIT
+#include "X86GenGlobalISel.inc"
+#undef GET_GLOBALISEL_TEMPORARIES_INIT
+{
+}
 
 // FIXME: This should be target-independent, inferred from the types declared
 // for each class in the bank.
 static const TargetRegisterClass *
 getRegClassForTypeOnBank(LLT Ty, const RegisterBank &RB) {
   if (RB.getID() == X86::GPRRegBankID) {
     if (Ty.getSizeInBits() == 32)
       return &X86::GR32RegClass;

test/CodeGen/AArch64/GlobalISel/arm64-instructionselect.mir

 # RUN: llc -O0 -mtriple=aarch64-apple-ios -run-pass=instruction-select -verify-machineinstrs -global-isel %s -o - | FileCheck %s -check-prefix=CHECK -check-prefix=IOS
 # RUN: llc -O0 -mtriple=aarch64-linux-gnu -run-pass=instruction-select -verify-machineinstrs -global-isel %s -o - | FileCheck %s -check-prefix=CHECK -check-prefix=LINUX-DEFAULT
 # RUN: llc -O0 -mtriple=aarch64-linux-gnu -relocation-model=pic -run-pass=instruction-select -verify-machineinstrs -global-isel %s -o - | FileCheck %s -check-prefix=CHECK -check-prefix=LINUX-PIC
 
 # Test the instruction selector.
 # As we support more instructions, we need to split this up.
 
 --- |
   target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
 
   define void @add_s32_gpr() { ret void }
   define void @add_s64_gpr() { ret void }
 
+  define void @add_imm_s32_gpr() { ret void }
+  define void @add_imm_s32_gpr_bb() { ret void }
+
   define void @sub_s32_gpr() { ret void }
   define void @sub_s64_gpr() { ret void }
 
   define void @or_s32_gpr() { ret void }
   define void @or_s64_gpr() { ret void }
   define void @or_v2s32_fpr() { ret void }
 
   define void @and_s32_gpr() { ret void }
bb.0:
     liveins: %x0, %x1
 
     %0(s64) = COPY %x0
     %1(s64) = COPY %x1
     %2(s64) = G_ADD %0, %1
 ...
 
 ---
+# Check that we select a 32-bit GPR G_ADD into ADDWrr on GPR32.
+# Also check that we constrain the register class of the COPY to GPR32.
+# CHECK-LABEL: name: add_imm_s32_gpr
+name:            add_imm_s32_gpr
+legalized:       true
+regBankSelected: true
+
+# CHECK:      registers:
+# CHECK-NEXT:  - { id: 0, class: gpr32sp }
+# CHECK-NEXT:  - { id: 1, class: gpr32 }
+# CHECK-NEXT:  - { id: 2, class: gpr32sp }
+registers:
+  - { id: 0, class: gpr }
+  - { id: 1, class: gpr }
+  - { id: 2, class: gpr }
+
+# CHECK:  body:
+# CHECK:    %0 = COPY %w0
+# CHECK:    %2 = ADDWri %0, 1, 0
+body:             |
+  bb.0:
+    liveins: %w0, %w1
+
+    %0(s32) = COPY %w0
+    %1(s32) = G_CONSTANT 1
+    %2(s32) = G_ADD %0, %1
+...
+
+---
+# Check that we select a 32-bit GPR G_ADD into ADDWrr on GPR32.
+# Also check that we constrain the register class of the COPY to GPR32.
+# CHECK-LABEL: name: add_imm_s32_gpr_bb
+name:            add_imm_s32_gpr_bb
+legalized:       true
+regBankSelected: true
+
+# CHECK:      registers:
+# CHECK-NEXT:  - { id: 0, class: gpr32sp }
+# CHECK-NEXT:  - { id: 1, class: gpr32 }
+# CHECK-NEXT:  - { id: 2, class: gpr32sp }
+registers:
+  - { id: 0, class: gpr }
+  - { id: 1, class: gpr }
+  - { id: 2, class: gpr }
+
+# CHECK:  body:
+# CHECK:    %0 = COPY %w0
+# CHECK:  bb.1:
+# CHECK:    %2 = ADDWri %0, 1, 0
+body:             |
+  bb.0:
+    liveins: %w0, %w1
+    successors: %bb.1
+
+    %0(s32) = COPY %w0
+    %1(s32) = G_CONSTANT 1
+    G_BR %bb.1
+
+  bb.1:
+    %2(s32) = G_ADD %0, %1
+...
+
+---
 # Same as add_s32_gpr, for G_SUB operations.
 # CHECK-LABEL: name: sub_s32_gpr
 name:            sub_s32_gpr
 legalized:       true
 regBankSelected: true
 
 # CHECK:      registers:
 # CHECK-NEXT:  - { id: 0, class: gpr32 }

test/TableGen/GlobalISelEmitter.td

class I<dag OOps, dag IOps, list<dag> Pat>
   let OutOperandList = OOps;
   let InOperandList = IOps;
   let Pattern = Pat;
 }
 
 //===- Test the function definition boilerplate. --------------------------===//
 
 // CHECK: bool MyTargetInstructionSelector::selectImpl(MachineInstr &I) const {
-// CHECK: const MachineRegisterInfo &MRI = I.getParent()->getParent()->getRegInfo();
+// CHECK: MachineFunction &MF = *I.getParent()->getParent();
+// CHECK: const MachineRegisterInfo &MRI = MF.getRegInfo();
 
 //===- Test a simple pattern with regclass operands. ----------------------===//
 
 // CHECK-LABEL: if ((I.getOpcode() == TargetOpcode::G_ADD) &&
 // CHECK-NEXT: ((/* dst */ (MRI.getType(I.getOperand(0).getReg()) == (LLT::scalar(32))) &&
 // CHECK-NEXT:  ((&RBI.getRegBankFromRegClass(MyTarget::GPR32RegClass) == RBI.getRegBank(I.getOperand(0).getReg(), MRI, TRI))))) &&
 // CHECK-NEXT: ((/* src1 */ (MRI.getType(I.getOperand(1).getReg()) == (LLT::scalar(32))) &&
 // CHECK-NEXT:  ((&RBI.getRegBankFromRegClass(MyTarget::GPR32RegClass) == RBI.getRegBank(I.getOperand(1).getReg(), MRI, TRI))))) &&

utils/TableGen/GlobalISelEmitter.cpp

 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/LowLevelTypeImpl.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include "llvm/TableGen/Error.h"
 #include "llvm/TableGen/Record.h"
 #include "llvm/TableGen/TableGenBackend.h"
 #include <string>
+#include <numeric>
 using namespace llvm;
 
 #define DEBUG_TYPE "gisel-emitter"
 
 STATISTIC(NumPatternTotal, "Total number of patterns");
 STATISTIC(NumPatternImported, "Number of patterns imported from SelectionDAG");
 STATISTIC(NumPatternImportsSkipped, "Number of SelectionDAG imports skipped");
 STATISTIC(NumPatternEmitted, "Number of patterns emitted");
void emitCxxConstructorCall(raw_ostream &OS) const {
     }
     if (Ty.isVector()) {
       OS << "LLT::vector(" << Ty.getNumElements() << ", " << Ty.getSizeInBits()
          << ")";
       return;
     }
     llvm_unreachable("Unhandled LLT");
   }
+
+  const LLT &get() const { return Ty; }
+};
+
+class InstructionMatcher;
+class OperandPlaceholder {
+private:
+  enum PlaceholderKind {
+    OP_MatchReference,
+    OP_Temporary,
+  } Kind;
+
+  struct MatchReferenceData {
+    InstructionMatcher *InsnMatcher;
+    StringRef InsnVarName;
+    StringRef SymbolicName;
+  };
+
+  struct TemporaryData {
+    unsigned OpIdx;
+  };
+
+  union {
+    struct MatchReferenceData MatchReference;
+    struct TemporaryData Temporary;
+  };
+
+  OperandPlaceholder(PlaceholderKind Kind) : Kind(Kind) {}
+
+public:
+  ~OperandPlaceholder() {}
+
+  static OperandPlaceholder
+  CreateMatchReference(InstructionMatcher *InsnMatcher,
+                       const StringRef InsnVarName, const StringRef SymbolicName) {
+    OperandPlaceholder Result(OP_MatchReference);
+    Result.MatchReference.InsnMatcher = InsnMatcher;
+    Result.MatchReference.InsnVarName = InsnVarName;
+    Result.MatchReference.SymbolicName = SymbolicName;
+    return Result;
+  }
+
+  static OperandPlaceholder CreateTemporary(unsigned OpIdx) {
+    OperandPlaceholder Result(OP_Temporary);
+    Result.Temporary.OpIdx = OpIdx;
+    return Result;
+  }
+
+  void emitCxxValueExpr(raw_ostream &OS) const;
 };
 
 /// Convert an MVT to an equivalent LLT if possible, or the invalid LLT() for
 /// MVTs that don't map cleanly to an LLT (e.g., iPTR, *any, ...).
 static Optional<LLTCodeGen> MVTToLLT(MVT::SimpleValueType SVT) {
   MVT VT(SVT);
   if (VT.isVector() && VT.getVectorNumElements() != 1)
     return LLTCodeGen(LLT::vector(VT.getVectorNumElements(), VT.getScalarSizeInBits()));
public:
   /// This enum is used for RTTI and also defines the priority that is given to
   /// the predicate when generating the matcher code. Kinds with higher priority
   /// must be tested first.
   ///
   /// The relative priority of OPM_LLT, OPM_RegBank, and OPM_MBB do not matter
   /// but OPM_Int must have priority over OPM_RegBank since constant integers
   /// are represented by a virtual register defined by a G_CONSTANT instruction.
   enum PredicateKind {
+    OPM_ComplexPattern,
     OPM_Int,
     OPM_LLT,
     OPM_RegBank,
     OPM_MBB,
   };
 
 protected:
   PredicateKind Kind;
virtual void emitCxxPredicateExpr(raw_ostream &OS,
                                     StringRef OperandExpr) const = 0;
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   virtual bool isHigherPriorityThan(const OperandPredicateMatcher &B) const {
     return Kind < B.Kind;
   };
+
+  /// Report the maximum number of temporary operands needed by the predicate
+  /// matcher.
+  virtual unsigned countTemporaryOperands() const { return 0; }
 };
 
 /// Generates code to check that an operand is a particular LLT.
 class LLTOperandMatcher : public OperandPredicateMatcher {
 protected:
   LLTCodeGen Ty;
 
 public:
   LLTOperandMatcher(const LLTCodeGen &Ty)
       : OperandPredicateMatcher(OPM_LLT), Ty(Ty) {}
 
   static bool classof(const OperandPredicateMatcher *P) {
     return P->getKind() == OPM_LLT;
   }
 
   void emitCxxPredicateExpr(raw_ostream &OS,
                             StringRef OperandExpr) const override {
     OS << "MRI.getType(" << OperandExpr << ".getReg()) == (";
     Ty.emitCxxConstructorCall(OS);
     OS << ")";
   }
 };
 
+/// Generates code to check that an operand is a particular target constant.
+class ComplexPatternOperandMatcher : public OperandPredicateMatcher {
+protected:
+  const Record &TheDef;
+  /// The index of the first temporary operand to allocate to this
+  /// ComplexPattern.
+  unsigned BaseTemporaryID;
+
+  unsigned getNumOperands() const {
+    return TheDef.getValueAsDag("Operands")->getNumArgs();
+  }
+
+public:
+  ComplexPatternOperandMatcher(const Record &TheDef, unsigned BaseTemporaryID)
+      : OperandPredicateMatcher(OPM_ComplexPattern), TheDef(TheDef),
+        BaseTemporaryID(BaseTemporaryID) {}
+
+  void emitCxxPredicateExpr(raw_ostream &OS,
+                            StringRef OperandExpr) const override {
+    OS << TheDef.getValueAsString("MatcherFn") << "(" << OperandExpr;
+    for (unsigned I = 0; I < getNumOperands(); ++I) {
+      OS << ", ";
+      OperandPlaceholder::CreateTemporary(BaseTemporaryID + I)
+          .emitCxxValueExpr(OS);
+    }
+    OS << ")";
+  }
+
+  unsigned countTemporaryOperands() const override {
+    return getNumOperands();
+  }
+};
+
 /// Generates code to check that an operand is in a particular register bank.
 class RegisterBankOperandMatcher : public OperandPredicateMatcher {
 protected:
   const CodeGenRegisterClass &RC;
 
 public:
   RegisterBankOperandMatcher(const CodeGenRegisterClass &RC)
       : OperandPredicateMatcher(OPM_RegBank), RC(RC) {}
for (const auto &Predicate : zip(predicates(), B.predicates())) {
       if (std::get<0>(Predicate)->isHigherPriorityThan(*std::get<1>(Predicate)))
         return true;
       if (std::get<1>(Predicate)->isHigherPriorityThan(*std::get<0>(Predicate)))
         return false;
     }
 
     return false;
   };
+
+  /// Report the maximum number of temporary operands needed by the operand
+  /// matcher.
+  unsigned countTemporaryOperands() const {
+    return std::accumulate(
+        predicates().begin(), predicates().end(), 0,
+        [](unsigned A,
+           const std::unique_ptr<OperandPredicateMatcher> &Predicate) {
+          return A + Predicate->countTemporaryOperands();
+        });
+  }
 };
 
 /// Generates code to check a predicate on an instruction.
 ///
 /// Typical predicates include:
 /// * The opcode of the instruction is a particular value.
 /// * The nsw/nuw flag is/isn't set.
 class InstructionPredicateMatcher {
virtual void emitCxxPredicateExpr(raw_ostream &OS,
                                     StringRef InsnVarName) const = 0;
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   virtual bool isHigherPriorityThan(const InstructionPredicateMatcher &B) const {
     return Kind < B.Kind;
   };
+
+  /// Report the maximum number of temporary operands needed by the predicate
+  /// matcher.
+  virtual unsigned countTemporaryOperands() const { return 0; }
 };
 
 /// Generates code to check the opcode of an instruction.
 class InstructionOpcodeMatcher : public InstructionPredicateMatcher {
 protected:
   const CodeGenInstruction *I;
 
 public:
for (const auto &Operand : zip(Operands, B.Operands)) {
       if (std::get<0>(Operand).isHigherPriorityThan(std::get<1>(Operand)))
         return true;
       if (std::get<1>(Operand).isHigherPriorityThan(std::get<0>(Operand)))
         return false;
     }
 
     return false;
   };
+
+  /// Report the maximum number of temporary operands needed by the instruction
+  /// matcher.
+  unsigned countTemporaryOperands() const {
+    return std::accumulate(predicates().begin(), predicates().end(), 0,
+                           [](unsigned A,
+                              const std::unique_ptr<InstructionPredicateMatcher>
+                                  &Predicate) {
+                             return A + Predicate->countTemporaryOperands();
+                           }) +
+           std::accumulate(Operands.begin(), Operands.end(), 0,
+                           [](unsigned A, const OperandMatcher &Operand) {
+                             return A + Operand.countTemporaryOperands();
+                           });
+  }
 };
 
 //===- Actions ------------------------------------------------------------===//
+void OperandPlaceholder::emitCxxValueExpr(raw_ostream &OS) const {
+  switch (Kind) {
+  case OP_MatchReference:
+    OS << MatchReference.InsnMatcher->getOperand(MatchReference.SymbolicName)
+              .getOperandExpr(MatchReference.InsnVarName);
+    break;
+  case OP_Temporary:
+    OS << "TempOp" << Temporary.OpIdx;
+    break;
+  }
+}
 
 namespace {
 class OperandRenderer {
 public:
-  enum RendererKind { OR_Copy, OR_Register };
+  enum RendererKind { OR_Copy, OR_Register, OR_ComplexPattern };
 
 protected:
   RendererKind Kind;
 
 public:
   OperandRenderer(RendererKind Kind) : Kind(Kind) {}
   virtual ~OperandRenderer() {}
 
public:
   }
 
   void emitCxxRenderStmts(raw_ostream &OS) const override {
     OS << "    MIB.addReg(" << RegisterDef->getValueAsString("Namespace")
        << "::" << RegisterDef->getName() << ");\n";
   }
 };
 
+class RenderComplexPatternOperand : public OperandRenderer {
+private:
+  const Record &TheDef;
+  std::vector<OperandPlaceholder> Sources;
+
+  unsigned getNumOperands() const {
+    return TheDef.getValueAsDag("Operands")->getNumArgs();
+  }
+
+public:
+  RenderComplexPatternOperand(const Record &TheDef,
+                              const ArrayRef<OperandPlaceholder> Sources)
+      : OperandRenderer(OR_ComplexPattern), TheDef(TheDef), Sources(Sources) {}
+
+  static bool classof(const OperandRenderer *R) {
+    return R->getKind() == OR_ComplexPattern;
+  }
+
+  void emitCxxRenderStmts(raw_ostream &OS) const override {
+    assert(Sources.size() == getNumOperands() && "Inconsistent number of operands");
+    for (const auto &Source : Sources) {
+      OS << "MIB.add(";
+      Source.emitCxxValueExpr(OS);
+      OS << ");\n";
+    }
+  }
+};
+
 /// An action taken when all Matcher predicates succeeded for a parent rule.
 ///
 /// Typical actions include:
 /// * Changing the opcode of an instruction.
 /// * Adding an operand to an instruction.
 class MatchAction {
 public:
   virtual ~MatchAction() {}
for (const auto &Matcher : zip(Matchers, B.Matchers)) {
       if (std::get<0>(Matcher)->isHigherPriorityThan(*std::get<1>(Matcher)))
         return true;
       if (std::get<1>(Matcher)->isHigherPriorityThan(*std::get<0>(Matcher)))
         return false;
     }
 
     return false;
   };
+
+  /// Report the maximum number of temporary operands needed by the rule
+  /// matcher.
+  unsigned countTemporaryOperands() const {
+    return std::accumulate(Matchers.begin(), Matchers.end(), 0,
+                           [](unsigned A, const std::unique_ptr<InstructionMatcher> &Matcher) {
+                             return A + Matcher->countTemporaryOperands();
+                           });
+  }
 };
 
 //===- GlobalISelEmitter class --------------------------------------------===//
 
 class GlobalISelEmitter {
 public:
   explicit GlobalISelEmitter(RecordKeeper &RK);
   void run(raw_ostream &OS);
 
 private:
   const RecordKeeper &RK;
   const CodeGenDAGPatterns CGP;
   const CodeGenTarget &Target;
 
   /// Keep track of the equivalence between SDNodes and Instruction.
   /// This is defined using 'GINodeEquiv' in the target description.
   DenseMap<Record *, const CodeGenInstruction *> NodeEquivs;
 
+  /// Keep track of the equivalence between ComplexPattern's and
+  /// GIComplexOperandMatcher. Map entries are specified by subclassing
+  /// GIComplexPatternEquiv.
+  DenseMap<const Record *, const Record *> ComplexPatternEquivs;
+
   void gatherNodeEquivs();
   const CodeGenInstruction *findNodeEquiv(Record *N);
 
   /// Analyze pattern \p P, returning a matcher for it if possible.
   /// Otherwise, return an Error explaining why we don't support it.
   Expected<RuleMatcher> runOnPattern(const PatternToMatch &P);
 };
 
 void GlobalISelEmitter::gatherNodeEquivs() {
   assert(NodeEquivs.empty());
   for (Record *Equiv : RK.getAllDerivedDefinitions("GINodeEquiv"))
     NodeEquivs[Equiv->getValueAsDef("Node")] =
         &Target.getInstruction(Equiv->getValueAsDef("I"));
+
+  assert(ComplexPatternEquivs.empty());
+  for (Record *Equiv : RK.getAllDerivedDefinitions("GIComplexPatternEquiv")) {
+    Record *SelDAGEquiv = Equiv->getValueAsDef("SelDAGEquivalent");
+    if (!SelDAGEquiv)
+      continue;
+    ComplexPatternEquivs[SelDAGEquiv] = Equiv;
+ }
 }
 
 const CodeGenInstruction *GlobalISelEmitter::findNodeEquiv(Record *N) {
   return NodeEquivs.lookup(N);
 }
 
 GlobalISelEmitter::GlobalISelEmitter(RecordKeeper &RK)
     : RK(RK), CGP(RK), Target(CGP.getTargetInfo()) {}
 
 //===- Emitter ------------------------------------------------------------===//
 
 /// Helper function to let the emitter report skip reason error messages.
 static Error failedImport(const Twine &Reason) {
   return make_error<StringError>(Reason, inconvertibleErrorCode());
 }
 
 Expected<RuleMatcher> GlobalISelEmitter::runOnPattern(const PatternToMatch &P) {
+  unsigned TempOpIdx = 0;
+
   // Keep track of the matchers and actions to emit.
   RuleMatcher M;
   M.addAction<DebugCommentAction>(P);
 
   // First, analyze the whole pattern.
   // If the entire pattern has a predicate (e.g., target features), ignore it.
   if (!P.getPredicates()->getValues().empty())
     return failedImport("Pattern has a predicate");
for (unsigned i = 0, e = Src->getNumChildren(); i != e; ++i) {
     if (auto *ChildInt = dyn_cast<IntInit>(SrcChild->getLeafValue())) {
       OM.addPredicate<IntOperandMatcher>(ChildInt->getValue());
       continue;
     }
 
     if (auto *ChildDefInit = dyn_cast<DefInit>(SrcChild->getLeafValue())) {
       auto *ChildRec = ChildDefInit->getDef();
 
-      // Otherwise, we're looking for a bog-standard RegisterClass operand.
-      if (!ChildRec->isSubClassOf("RegisterClass"))
-        return failedImport("Src pattern child isn't a RegisterClass");
-
+      if (ChildRec->isSubClassOf("RegisterClass")) {
         OM.addPredicate<RegisterBankOperandMatcher>(
             Target.getRegisterClass(ChildRec));
         continue;
       }
 
+      if (ChildRec->isSubClassOf("ComplexPattern")) {
+        const auto &ComplexPattern = ComplexPatternEquivs.find(ChildRec);
+        if (ComplexPattern == ComplexPatternEquivs.end())
+          return failedImport(
+              "SelectionDAG ComplexPattern not mapped to GlobalISel");
+
+        const auto &Predicate = OM.addPredicate<ComplexPatternOperandMatcher>(
+            *ComplexPattern->second, TempOpIdx);
+        TempOpIdx += Predicate.countTemporaryOperands();
+        continue;
+      }
+
+      return failedImport(
+          "Src pattern child def is an unsupported tablegen class");
+    }
+
     return failedImport("Src pattern child is an unsupported kind");
   }
 
+  TempOpIdx = 0;
   // Finally render the used operands (i.e., the children of the root operator).
   for (unsigned i = 0, e = Dst->getNumChildren(); i != e; ++i) {
     auto *DstChild = Dst->getChild(i);
 
     // The only non-leaf child we accept is 'bb': it's an operator because
     // BasicBlockSDNode isn't inline, but in MI it's just another operand.
     if (!DstChild->isLeaf()) {
       if (DstChild->getOperator()->isSubClassOf("SDNode")) {
if (auto *ChildDefInit = dyn_cast<DefInit>(DstChild->getLeafValue())) {
       }
 
       if (ChildRec->isSubClassOf("RegisterClass")) {
         DstMIBuilder.addRenderer<CopyRenderer>(InsnMatcher, "I",
                                                DstChild->getName());
         continue;
       }
 
+      if (ChildRec->isSubClassOf("ComplexPattern")) {
+        const auto &ComplexPattern = ComplexPatternEquivs.find(ChildRec);
+        if (ComplexPattern == ComplexPatternEquivs.end())
+          return failedImport(
+              "SelectionDAG ComplexPattern not mapped to GlobalISel");
+
+        SmallVector<OperandPlaceholder, 2> RenderedOperands;
+        for (unsigned I = 0; I < InsnMatcher.getOperand(DstChild->getName())
+                                     .countTemporaryOperands();
+             ++I) {
+          RenderedOperands.push_back(OperandPlaceholder::CreateTemporary(I));
+          TempOpIdx++;
+        }
+        DstMIBuilder.addRenderer<RenderComplexPatternOperand>(
+            *ComplexPattern->second,
+            RenderedOperands);
+        continue;
+      }
+
       return failedImport(
           "Dst pattern child def is an unsupported tablegen class");
     }
 
-    return failedImport("Src pattern child is an unsupported kind");
+    return failedImport("Dst pattern child is an unsupported kind");
   }
 
   // We're done with this pattern!  It's eligible for GISel emission; return it.
   ++NumPatternImported;
   return std::move(M);
 }
 
 void GlobalISelEmitter::run(raw_ostream &OS) {
   // Track the GINodeEquiv definitions.
   gatherNodeEquivs();
 
   emitSourceFileHeader(("Global Instruction Selector for the " +
                        Target.getName() + " target").str(), OS);
-  OS << "bool " << Target.getName()
-     << "InstructionSelector::selectImpl"
-        "(MachineInstr &I) const {\n  const MachineRegisterInfo &MRI = "
-        "I.getParent()->getParent()->getRegInfo();\n\n";
-
   std::vector<RuleMatcher> Rules;
   // Look through the SelectionDAG patterns we found, possibly emitting some.
   for (const PatternToMatch &Pat : CGP.ptms()) {
     ++NumPatternTotal;
     auto MatcherOrErr = runOnPattern(Pat);
 
     // The pattern analysis can fail, indicating an unsupported pattern.
     // Report that if we've been asked to do so.
std::stable_sort(Rules.begin(), Rules.end(),
                 assert(!B.isHigherPriorityThan(A) && "Cannot be more important "
                                                      "and less important at "
                                                      "the same time");
                 return true;
               }
               return false;
             });
 
+  unsigned MaxTemporaries = 0;
+  for (const auto &Rule : Rules)
+    MaxTemporaries = std::max(MaxTemporaries, Rule.countTemporaryOperands());
+
+  OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n";
+  for (unsigned I = 0; I < MaxTemporaries; ++I)
+    OS << "  mutable MachineOperand TempOp" << I << ";\n";
+  OS << "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n\n";
+
+  OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n";
+  for (unsigned I = 0; I < MaxTemporaries; ++I)
+    OS << ", TempOp" << I << "(MachineOperand::CreatePlaceholder())\n";
+  OS << "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n\n";
+
+  OS << "#ifdef GET_GLOBALISEL_IMPL\n"
+     << "bool " << Target.getName()
+     << "InstructionSelector::selectImpl(MachineInstr &I) const {\n"
+     << "  MachineFunction &MF = *I.getParent()->getParent();\n"
+     << "  const MachineRegisterInfo &MRI = MF.getRegInfo();\n";
+
   for (const auto &Rule : Rules) {
     Rule.emit(OS);
     ++NumPatternEmitted;
   }
 
-  OS << "  return false;\n}\n";
+  OS << "  return false;\n"
+     << "}\n"
+     << "#endif // ifdef GET_GLOBALISEL_IMPL\n";
 }
 
 } // end anonymous namespace
 
 //===----------------------------------------------------------------------===//
 
 namespace llvm {
 void EmitGlobalISel(RecordKeeper &RK, raw_ostream &OS) {
   GlobalISelEmitter(RK).run(OS);
 }
 } // End llvm namespace