[RFC][GISel] Add a way to ignore COPY instructions in InstructionSelector

[Pierre-vh]

Summary

RFC to add a way to ignore COPY instructions when pattern-matching MIR in GISel.

• Add a new "GISelFlags" class to TableGen. Both Pattern and PatFrags defs can use it to alter matching behaviour.
• Flags start at zero and are scoped: the setter returns a SaveAndRestore object so that when the current scope ends, the flags are restored to their previous values. This allows child patterns to modify the flags without affecting the parent pattern.
• Child patterns always reuse the parent's pattern, but they can override its values. For more examples, see GlobalISelEmitterFlags.td tests.
• [AMDGPU] Use the IgnoreCopies flag in BFI patterns, which are known to be bothered by cross-regbank copies.

[Pierre-vh]

Rebase, rework

[arsenm]

llvm/utils/TableGen/GlobalISelEmitter.cpp

5220

Would it be be possible to set this on a PatFrag level instead of a top-level pattern?

[Pierre-vh]

• Add test

@arsenm for the PatFrags it looks like they don't (always) generate a Try block so the system as-is cannot handle them.
I was thinking of making GIM_SetFlag work only per-instruction and be used as a prefix, so instead of having

GIM_Try ...
  GIM_SetFlag GIMSF_IgnoreCopies

to set flags for a whole block we'd have

GIM_SetFlag, GIMSF_IgnoreCopies, GIM_Try ... # Sets the flag on the next instruction -> Try saves them in the whole block

Then we could have flags set for a single instruction which would allow us to support PatFrags and virtually anything else
after a bit more backend work:

GIM_SetFlag, GIMSF_IgnoreCopies, GIM_RecordInsn ...

[Pierre-vh]

Current design doesn't work on PatFrags as they are inlined. I'm looking for solutions.

[Pierre-vh]

Rework so the system works on both Pattern and PatFrags

[Pierre-vh]

Rebase

This is ready for review, I don't have anymore changes to make for now.

[Petar.Avramovic]

I like the GISelFlags, looks pretty lightweight.
Are you planning to add more flags?
About IgnoreCopies, it looks correct to me. Considering the possibility to break constant bus limit, patch synergies well with GISelPredicateCode that checks for it.
Can you add look through copies to Three Op Pats from VOP3 td file?
Upcoming regbankselect changes should be able to deal with copies but nevertheless I would like to see this one merged.
LGTM but I would wait for feedback from other reviewers.

[arsenm]

llvm/lib/Target/AMDGPU/SIInstructions.td

1911

I think it's cleaner to hide these in an AMDGPUPat subclass, like AMDGPUPatIgnoreCopies

[Pierre-vh]

Address comment

Can this land (once the parent diff does) or does this need to wait for more people to approve?

llvm/include/llvm/CodeGen/GlobalISel/InstructionSelector.h

enum {
   GIM_SwitchType,
 
   /// Record the specified instruction
   /// - NewInsnID - Instruction ID to define
   /// - InsnID - Instruction ID
   /// - OpIdx - Operand index
   GIM_RecordInsn,
 
+  /// Variant of GIM_RecordInsn that ignores COPY instructions when
+  /// possible.
+  /// - NewInsnID - Instruction ID to define
+  /// - InsnID - Instruction ID
+  /// - OpIdx - Operand index
+  GIM_RecordInsnIgnoreCopies,
+
   /// Check the feature bits
   /// - Expected features
   GIM_CheckFeatures,
 
   /// Check the opcode on the specified instruction
   /// - InsnID - Instruction ID
   /// - Expected opcode
   GIM_CheckOpcode,
enum {
   /// - OldInsnID - Instruction ID to get the matched operand from
   /// - RendererFnID - Custom renderer function to call
   GIR_CustomRenderer,
 
   /// Render operands to the specified instruction using a custom function,
   /// reading from a specific operand.
   /// - InsnID - Instruction ID to modify
   /// - OldInsnID - Instruction ID to get the matched operand from
-  /// - OpIdx - Operand index in OldInsnID the render function should read from..
+  /// - OpIdx - Operand index in OldInsnID the render function should read
+  /// from..
   /// - RendererFnID - Custom renderer function to call
   GIR_CustomOperandRenderer,
 
   /// Render a G_CONSTANT operator as a sign-extended immediate.
   /// - NewInsnID - Instruction ID to modify
   /// - OldInsnID - Instruction ID to copy from
   /// The operand index is implicitly 1.
   GIR_CopyConstantAsSImm,

llvm/include/llvm/CodeGen/GlobalISel/InstructionSelectorImpl.h

while (true) {
     switch (MatcherOpcode) {
     case GIM_Try: {
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                       dbgs() << CurrentIdx << ": Begin try-block\n");
       OnFailResumeAt.push_back(MatchTable[CurrentIdx++]);
       break;
     }
 
+    case GIM_RecordInsnIgnoreCopies:
     case GIM_RecordInsn: {
+      const bool IgnoreCopies = (MatcherOpcode == GIM_RecordInsnIgnoreCopies);
       int64_t NewInsnID = MatchTable[CurrentIdx++];
       int64_t InsnID = MatchTable[CurrentIdx++];
       int64_t OpIdx = MatchTable[CurrentIdx++];
 
       // As an optimisation we require that MIs[0] is always the root. Refuse
       // any attempt to modify it.
       assert(NewInsnID != 0 && "Refusing to modify MIs[0]");
 
       MachineOperand &MO = State.MIs[InsnID]->getOperand(OpIdx);
       if (!MO.isReg()) {
         DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                         dbgs() << CurrentIdx << ": Not a register\n");
         if (handleReject() == RejectAndGiveUp)
           return false;
         break;
       }
       if (Register::isPhysicalRegister(MO.getReg())) {
         DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                         dbgs() << CurrentIdx << ": Is a physical register\n");
         if (handleReject() == RejectAndGiveUp)
           return false;
         break;
       }
 
-      MachineInstr *NewMI = MRI.getVRegDef(MO.getReg());
+      MachineInstr *NewMI;
+      if (IgnoreCopies)
+        NewMI = getDefIgnoringCopies(MO.getReg(), MRI);
+      else
+        NewMI = MRI.getVRegDef(MO.getReg());
+
       if ((size_t)NewInsnID < State.MIs.size())
         State.MIs[NewInsnID] = NewMI;
       else {
         assert((size_t)NewInsnID == State.MIs.size() &&
                "Expected to store MIs in order");
         State.MIs.push_back(NewMI);
       }
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
-                      dbgs() << CurrentIdx << ": MIs[" << NewInsnID
-                             << "] = GIM_RecordInsn(" << InsnID << ", " << OpIdx
-                             << ")\n");
+                      dbgs() << CurrentIdx << ": MIs[" << NewInsnID << "] = "
+                             << (IgnoreCopies ? "GIM_RecordInsnIgnoreCopies"
+                                              : "GIM_RecordInsn")
+                             << "(" << InsnID << ", " << OpIdx << ")\n");
       break;
     }
 
     case GIM_CheckFeatures: {
       int64_t ExpectedBitsetID = MatchTable[CurrentIdx++];
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                       dbgs() << CurrentIdx
                              << ": GIM_CheckFeatures(ExpectedBitsetID="
case GIM_CheckIsSameOperand: {
       int64_t OtherInsnID = MatchTable[CurrentIdx++];
       int64_t OtherOpIdx = MatchTable[CurrentIdx++];
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                       dbgs() << CurrentIdx << ": GIM_CheckIsSameOperand(MIs["
                              << InsnID << "][" << OpIdx << "], MIs["
                              << OtherInsnID << "][" << OtherOpIdx << "])\n");
       assert(State.MIs[InsnID] != nullptr && "Used insn before defined");
       assert(State.MIs[OtherInsnID] != nullptr && "Used insn before defined");
-      if (!State.MIs[InsnID]->getOperand(OpIdx).isIdenticalTo(
-              State.MIs[OtherInsnID]->getOperand(OtherOpIdx))) {
+
+      // FIXME: Gross hack, needs a better solution here.
+      MachineOperand &Op = State.MIs[InsnID]->getOperand(OpIdx);
+      MachineOperand &OtherOp = State.MIs[OtherInsnID]->getOperand(OtherOpIdx);
+      if (Op.isReg() && OtherOp.isReg()) {
+        MachineInstr *MI = getDefIgnoringCopies(Op.getReg(), MRI);
+        MachineInstr *OtherMI = getDefIgnoringCopies(OtherOp.getReg(), MRI);
+        if (MI && MI == OtherMI)
+          break;
+      }
+
+      if (!Op.isIdenticalTo(OtherOp)) {
         if (handleReject() == RejectAndGiveUp)
           return false;
       }
       break;
     }
     case GIM_Reject:
       DEBUG_WITH_TYPE(TgtInstructionSelector::getName(),
                       dbgs() << CurrentIdx << ": GIM_Reject\n");

llvm/include/llvm/Target/TargetSelectionDAG.td

 // their GPRs).
 //
 
 class Pattern<dag patternToMatch, list<dag> resultInstrs> {
   dag             PatternToMatch  = patternToMatch;
   list<dag>       ResultInstrs    = resultInstrs;
   list<Predicate> Predicates      = [];  // See class Instruction in Target.td.
   int             AddedComplexity = 0;   // See class Instruction in Target.td.
+
+  // When this is set, the GISel matcher will look through COPY instructions
+  // before checking opcodes for instruction operands.
+  bit GISelIgnoreCopies = ?;
 }
 
 // Pat - A simple (but common) form of a pattern, which produces a simple result
 // not needing a full list.
 class Pat<dag pattern, dag result> : Pattern<pattern, [result]>;
 
 //===----------------------------------------------------------------------===//
 // Complex pattern definitions.

llvm/lib/Target/AMDGPU/AMDGPUInstructions.td

class PredicateControl {
   list<Predicate> Predicates = PredConcat<
                                  PredConcat<PredConcat<OtherPredicates,
                                                        SubtargetPredicate>.ret,
                                             AssemblerPredicate>.ret,
                                  WaveSizePredicate>.ret;
 }
 
 class AMDGPUPat<dag pattern, dag result> : Pat<pattern, result>,
-      PredicateControl;
+      PredicateControl {
+  let GISelIgnoreCopies = 1;
+}
 
 let RecomputePerFunction = 1 in {
 def FP16Denormals : Predicate<"MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP64FP16Denormals()">;
 def FP32Denormals : Predicate<"MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP32Denormals()">;
 def FP64Denormals : Predicate<"MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP64FP16Denormals()">;
 def NoFP16Denormals : Predicate<"!MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP64FP16Denormals()">;
 def NoFP32Denormals : Predicate<"!MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP32Denormals()">;
 def NoFP64Denormals : Predicate<"!MF->getInfo<SIMachineFunctionInfo>()->getMode().allFP64FP16Denormals()">;

llvm/lib/Target/AMDGPU/SIInstructions.td

 //===-- SIInstructions.td - SI Instruction Definitions --------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // This file was originally auto-generated from a GPU register header file and
 // all the instruction definitions were originally commented out.  Instructions
 // that are not yet supported remain commented out.
 //===----------------------------------------------------------------------===//
 
 class GCNPat<dag pattern, dag result> : Pat<pattern, result>, GCNPredicateControl {
-
+  let GISelIgnoreCopies = 1;
 }
 
 class UniformSextInreg<ValueType VT> : PatFrag<
   (ops node:$src),
   (sext_inreg $src, VT),
   [{ return !N->isDivergent(); }]>;
 
 class DivergentSextInreg<ValueType VT> : PatFrag<
def BFIImm32 : PatFrag<
   }]
 >;
 
 // Definition from ISA doc:
 // (y & x) | (z & ~x)
 def : AMDGPUPat <
   (DivergentBinFrag<or> (and i32:$y, i32:$x), (and i32:$z, (not i32:$x))),
   (V_BFI_B32_e64 VSrc_b32:$x, VSrc_b32:$y, VSrc_b32:$z)
 >;
 
 // (y & C) | (z & ~C)
 def : AMDGPUPat <
   (BFIImm32 i32:$x, i32:$y, i32:$z),
   (V_BFI_B32_e64 VSrc_b32:$x, VSrc_b32:$y, VSrc_b32:$z)
 >;
 
 // 64-bit version

llvm/utils/TableGen/GlobalISelEmitter.cpp

 ///   (set $dst, (G_MUL (G_ADD $src1, $src2), $src3))
 /// would use an InstructionOperandMatcher for operand 1 of the G_MUL to match
 /// the:
 ///   (G_ADD $src1, $src2)
 /// subpattern.
 class InstructionOperandMatcher : public OperandPredicateMatcher {
 protected:
   std::unique_ptr<InstructionMatcher> InsnMatcher;
+  bool IgnoreCopies;
 
 public:
   InstructionOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                             RuleMatcher &Rule, StringRef SymbolicName,
-                            bool NumOpsCheck = true)
+                            bool IgnoreCopies, bool NumOpsCheck = true)
       : OperandPredicateMatcher(OPM_Instruction, InsnVarID, OpIdx),
-        InsnMatcher(new InstructionMatcher(Rule, SymbolicName, NumOpsCheck)) {}
+        InsnMatcher(new InstructionMatcher(Rule, SymbolicName, NumOpsCheck)),
+        IgnoreCopies(IgnoreCopies) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_Instruction;
   }
 
   InstructionMatcher &getInsnMatcher() const { return *InsnMatcher; }
 
   void emitCaptureOpcodes(MatchTable &Table, RuleMatcher &Rule) const {
     const unsigned NewInsnVarID = InsnMatcher->getInsnVarID();
-    Table << MatchTable::Opcode("GIM_RecordInsn")
-          << MatchTable::Comment("DefineMI")
+    const StringRef Opcode =
+        IgnoreCopies ? "GIM_RecordInsnIgnoreCopies" : "GIM_RecordInsn";
+    Table << MatchTable::Opcode(Opcode) << MatchTable::Comment("DefineMI")
           << MatchTable::IntValue(NewInsnVarID) << MatchTable::Comment("MI")
           << MatchTable::IntValue(getInsnVarID())
           << MatchTable::Comment("OpIdx") << MatchTable::IntValue(getOpIdx())
           << MatchTable::Comment("MIs[" + llvm::to_string(NewInsnVarID) + "]")
           << MatchTable::LineBreak;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
private:
   void gatherTypeIDValues();
   void gatherNodeEquivs();
 
   Record *findNodeEquiv(Record *N) const;
   const CodeGenInstruction *getEquivNode(Record &Equiv,
                                          const TreePatternNode *N) const;
 
   Error importRulePredicates(RuleMatcher &M, ArrayRef<Record *> Predicates);
-  Expected<InstructionMatcher &>
-  createAndImportSelDAGMatcher(RuleMatcher &Rule,
-                               InstructionMatcher &InsnMatcher,
-                               const TreePatternNode *Src, unsigned &TempOpIdx);
+  Expected<InstructionMatcher &> createAndImportSelDAGMatcher(
+      RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
+      const TreePatternNode *Src, unsigned &TempOpIdx, bool IgnoreCopies);
   Error importComplexPatternOperandMatcher(OperandMatcher &OM, Record *R,
                                            unsigned &TempOpIdx) const;
   Error importChildMatcher(RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
                            const TreePatternNode *SrcChild,
                            bool OperandIsAPointer, bool OperandIsImmArg,
-                           unsigned OpIdx, unsigned &TempOpIdx);
+                           bool IgnoreCopies, unsigned OpIdx,
+                           unsigned &TempOpIdx);
 
   Expected<BuildMIAction &> createAndImportInstructionRenderer(
       RuleMatcher &M, InstructionMatcher &InsnMatcher,
       const TreePatternNode *Src, const TreePatternNode *Dst);
   Expected<action_iterator> createAndImportSubInstructionRenderer(
       action_iterator InsertPt, RuleMatcher &M, const TreePatternNode *Dst,
       unsigned TempReg);
   Expected<action_iterator>
if (Predicate.isAtomicOrderingWeakerThanRelease()) {
     return InsnMatcher;
   }
   HasAddedMatcher = false;
   return InsnMatcher;
 }
 
 Expected<InstructionMatcher &> GlobalISelEmitter::createAndImportSelDAGMatcher(
     RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
-    const TreePatternNode *Src, unsigned &TempOpIdx) {
+    const TreePatternNode *Src, unsigned &TempOpIdx, bool IgnoreCopies) {
   Record *SrcGIEquivOrNull = nullptr;
   const CodeGenInstruction *SrcGIOrNull = nullptr;
 
   // Start with the defined operands (i.e., the results of the root operator).
   if (Src->getExtTypes().size() > 1)
     return failedImport("Src pattern has multiple results");
 
   if (Src->isLeaf()) {
if (Src->isLeaf()) {
     // opcode, so we need to swap the operands here to get the right type check.
     if (IsAtomic && SrcGIOrNull->TheDef->getName() == "G_STORE") {
       assert(NumChildren == 2 && "wrong operands for atomic store");
 
       TreePatternNode *PtrChild = Src->getChild(0);
       TreePatternNode *ValueChild = Src->getChild(1);
 
       if (auto Error = importChildMatcher(Rule, InsnMatcher, PtrChild, true,
-                                          false, 1, TempOpIdx))
+                                          false, IgnoreCopies, 1, TempOpIdx))
         return std::move(Error);
 
       if (auto Error = importChildMatcher(Rule, InsnMatcher, ValueChild, false,
-                                          false, 0, TempOpIdx))
+                                          false, IgnoreCopies, 0, TempOpIdx))
         return std::move(Error);
       return InsnMatcher;
     }
 
     // Match the used operands (i.e. the children of the operator).
     bool IsIntrinsic =
         SrcGIOrNull->TheDef->getName() == "G_INTRINSIC" ||
         SrcGIOrNull->TheDef->getName() == "G_INTRINSIC_W_SIDE_EFFECTS";
for (unsigned i = 0; i != NumChildren; ++i) {
         // We have to check intrinsics for llvm_anyptr_ty and immarg parameters.
         //
         // Note that we have to look at the i-1th parameter, because we don't
         // have the intrinsic ID in the intrinsic's parameter list.
         OperandIsAPointer |= II->isParamAPointer(i - 1);
         OperandIsImmArg |= II->isParamImmArg(i - 1);
       }
 
-      if (auto Error =
-              importChildMatcher(Rule, InsnMatcher, SrcChild, OperandIsAPointer,
-                                 OperandIsImmArg, OpIdx++, TempOpIdx))
+      if (auto Error = importChildMatcher(Rule, InsnMatcher, SrcChild,
+                                          OperandIsAPointer, OperandIsImmArg,
+                                          IgnoreCopies, OpIdx++, TempOpIdx))
         return std::move(Error);
     }
   }
 
   return InsnMatcher;
 }
 
 Error GlobalISelEmitter::importComplexPatternOperandMatcher(
if (auto *ChildDefInit = dyn_cast<DefInit>(SrcChild->getLeafValue())) {
         PhysReg = ChildRec;
       }
     }
   }
 
   return SrcChildName;
 }
 
-Error GlobalISelEmitter::importChildMatcher(
-    RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
-    const TreePatternNode *SrcChild, bool OperandIsAPointer,
-    bool OperandIsImmArg, unsigned OpIdx, unsigned &TempOpIdx) {
+Error GlobalISelEmitter::importChildMatcher(RuleMatcher &Rule,
+                                            InstructionMatcher &InsnMatcher,
+                                            const TreePatternNode *SrcChild,
+                                            bool OperandIsAPointer,
+                                            bool OperandIsImmArg,
+                                            bool IgnoreCopies, unsigned OpIdx,
+                                            unsigned &TempOpIdx) {
 
   Record *PhysReg = nullptr;
   std::string SrcChildName = std::string(getSrcChildName(SrcChild, PhysReg));
   if (!SrcChild->isLeaf() &&
       SrcChild->getOperator()->isSubClassOf("ComplexPattern")) {
     // The "name" of a non-leaf complex pattern (MY_PAT $op1, $op2) is
     // "MY_PAT:op1:op2" and the ones with same "name" represent same operand.
     std::string PatternName = std::string(SrcChild->getOperator()->getName());
if (SrcChild->getOperator()->isSubClassOf("ComplexPattern")) {
             return Error;
         }
       }
 
       return Error::success();
     }
 
     auto MaybeInsnOperand = OM.addPredicate<InstructionOperandMatcher>(
-        InsnMatcher.getRuleMatcher(), SrcChild->getName());
+        InsnMatcher.getRuleMatcher(), SrcChild->getName(), IgnoreCopies);
     if (!MaybeInsnOperand) {
       // This isn't strictly true. If the user were to provide exactly the same
       // matchers as the original operand then we could allow it. However, it's
       // simpler to not permit the redundant specification.
       return failedImport("Nested instruction cannot be the same as another operand");
     }
 
     // Map the node to a gMIR instruction.
     InstructionOperandMatcher &InsnOperand = **MaybeInsnOperand;
     auto InsnMatcherOrError = createAndImportSelDAGMatcher(
-        Rule, InsnOperand.getInsnMatcher(), SrcChild, TempOpIdx);
+        Rule, InsnOperand.getInsnMatcher(), SrcChild, TempOpIdx, IgnoreCopies);
     if (auto Error = InsnMatcherOrError.takeError())
       return Error;
 
     return Error::success();
   }
 
   if (SrcChild->hasAnyPredicate())
     return failedImport("Src pattern child has unsupported predicate");
if (auto *ChildDefInit = dyn_cast<DefInit>(SrcChild->getLeafValue())) {
 
     // Place holder for SRCVALUE nodes. Nothing to do here.
     if (ChildRec->getName() == "srcvalue")
       return Error::success();
 
     const bool ImmAllOnesV = ChildRec->getName() == "immAllOnesV";
     if (ImmAllOnesV || ChildRec->getName() == "immAllZerosV") {
       auto MaybeInsnOperand = OM.addPredicate<InstructionOperandMatcher>(
-          InsnMatcher.getRuleMatcher(), SrcChild->getName(), false);
+          InsnMatcher.getRuleMatcher(), SrcChild->getName(), IgnoreCopies,
+          false);
       InstructionOperandMatcher &InsnOperand = **MaybeInsnOperand;
 
       ValueTypeByHwMode VTy = ChildTypes.front().getValueTypeByHwMode();
 
       const CodeGenInstruction &BuildVector
         = Target.getInstruction(RK.getDef("G_BUILD_VECTOR"));
       const CodeGenInstruction &BuildVectorTrunc
         = Target.getInstruction(RK.getDef("G_BUILD_VECTOR_TRUNC"));
Expected<RuleMatcher> GlobalISelEmitter::runOnPattern(const PatternToMatch &P) {
   // naming being the same. One possible solution would be to have
   // explicit operator for operation capture and reference those.
   // The plus side is that it would expose opportunities to share
   // the capture accross rules. The downside is that it would
   // introduce a dependency between predicates (captures must happen
   // before their first use.)
   InstructionMatcher &InsnMatcherTemp = M.addInstructionMatcher(Src->getName());
   unsigned TempOpIdx = 0;
-  auto InsnMatcherOrError =
-      createAndImportSelDAGMatcher(M, InsnMatcherTemp, Src, TempOpIdx);
+
+  bool IgnoreCopies = false;
+  if (P.getSrcRecord()->isSubClassOf("Pattern")) {
+    bool Unset = false;
+    IgnoreCopies =
+        P.getSrcRecord()->getValueAsBitOrUnset("GISelIgnoreCopies", Unset);
+  }
+
+  auto InsnMatcherOrError = createAndImportSelDAGMatcher(
+      M, InsnMatcherTemp, Src, TempOpIdx, IgnoreCopies);
   if (auto Error = InsnMatcherOrError.takeError())
     return std::move(Error);
   InstructionMatcher &InsnMatcher = InsnMatcherOrError.get();
 
   if (Dst->isLeaf()) {
     Record *RCDef = getInitValueAsRegClass(Dst->getLeafValue());
     if (RCDef) {
       const CodeGenRegisterClass &RC = Target.getRegisterClass(RCDef);