[GlobalISel] Making VRegs w/ LLTs and RegClasses both illegal

[rtereshin]

Summary

anywhere but mid-InstructionSelect pass.

The rationale behind this is the consideration that dealing w/
instructions that def/use VRegs each of which could be in one of the 3
configurations:

a) LLT (optional RegBank)
b) RegClass
c) LLT and RegClass both

throughout the GlobalISel is getting tedious and error-prone very
quickly as the number of VRegs def/used increases, and it's better to
forbid the config (c) altogether to simplify the implementation and
decrease the number of hidden bugs. Apparently, the GlobalISel
implementation already assumes that in a few places, this decision was
made a while ago, but it was never actually enforced.

This commit explicitly enforces the invariant at passes' boundaries
with MachineVerifier, and implicitly intra-pass except
InstructionSelect, which is expected to introduce such VRegs for
performance reasons, as the pass eliminates LLTs by the end of it
anyway.

This patch also tries its best to fix all the failures that came out
of it. The most affected parts are:

1. CallLowering: lowering ABI naturally introduces a few selected MIR / non-selected MIR boundaries, which many targets shaped with LLT + RegClass VRegs, now illegal.

2. selectCopy: as GlobalISel doesn't supply tools for assigning RegClasses during selection to def/uses of COPYs, this is done by every target independently in a very similar manner with common drawbacks, e.g. assuming that SRC-vreg of a COPY could not have (meaningful) RegClass attached and whatnot. As the number of COPYs between typed and regclassed VRegs increases with this commit, some of these issues got exposed and had to be fixed to the best of my understanding of the target-specific implementation.

3. constrainOperandRegClass set of GlobalISel utils, apparently promising but not delivering auto-COPY-insertion for defs, not just uses was fixed as well and updated in a way that should make it easier to migrate legalizer-implementations and other parts of non-upstream targets.

This is related to, but not dependent on https://reviews.llvm.org/D45644 and https://reviews.llvm.org/D45640

[rtereshin]

Rebased against thunk and removed a test that is no longer valid due to https://reviews.llvm.org/rL330660 (https://reviews.llvm.org/D45644)

[rtereshin]

ping

[aditya_nandakumar]

Also, I wonder if there would be passes (non ISEL), that would break the invariant intra pass, but cleans up at the end of the pass..

lib/CodeGen/GlobalISel/IRTranslator.cpp

598

This can be simplified into a more compact form (if you care enough).

auto MIB = MIRBuilder.buildCopy(DstReg,
                          MIRBuilder.buildInstr(TRI->getPointerRegClass(*MF), LOAD_STACK_GUARD))

[rtereshin]

In D45732#1086494, @aditya_nandakumar wrote:

Also, I wonder if there would be passes (non ISEL), that would break the invariant intra pass, but cleans up at the end of the pass..

I don't know, maybe? It's probably not going too much against this patch, I just decided to maintain invariant intra pass to make this work for existing passes. New passes are free to make a different decision and clean up at the end of the pass, they will have the opportunity to call the constrain reg classes helper functions and methods with ISel = true to stop them from maintaining the invariant. If that happens, we will rename the function argument so it doesn't suggest instruction select pass specifically anymore.

Sounds good?

Also, thanks for looking into this.

[rtereshin]

Rebasing against master (there was a patch committed containing some of the changes from this one), addressing comments, and tidying up the patch a little.

[rtereshin]

ping

include/llvm/CodeGen/GlobalISel/Utils.h

 class TargetInstrInfo;
 class TargetPassConfig;
 class TargetRegisterInfo;
 class TargetRegisterClass;
 class Twine;
 class ConstantFP;
 class APFloat;
 
-/// Try to constrain Reg to the specified register class. If this fails,
-/// create a new virtual register in the correct class and insert a COPY before
-/// \p InsertPt. The debug location of \p InsertPt is used for the new copy.
+/// Try to constrain Reg of the \p RegMO to the specified register class. If
+/// this fails, create a new virtual register in the correct class and insert a
+/// COPY before \p InsertPt for uses, or after it for defs. The debug location
+/// of \p InsertPt is used for the new copy.
+//
+/// If used from within an Instruction Selector (ISel = true) it can leave some
+/// vregs having a type and a regclass both to limit the number of copies
+/// created, which is safe as the types will be eliminated at the end of the
+/// selection. Otherwise it will avoid introducing such registers by inserting
+/// necessary copies.
 ///
 /// \return The virtual register constrained to the right register class.
-unsigned constrainRegToClass(MachineRegisterInfo &MRI,
-                             const TargetInstrInfo &TII,
-                             const RegisterBankInfo &RBI,
-                             MachineInstr &InsertPt, unsigned Reg,
-                             const TargetRegisterClass &RegClass);
-
-/// Try to constrain Reg so that it is usable by argument OpIdx of the
-/// provided MCInstrDesc \p II. If this fails, create a new virtual
-/// register in the correct class and insert a COPY before \p InsertPt.
-/// This is equivalent to constrainRegToClass() with RegClass obtained from the
+unsigned
+constrainOperandRegClass(MachineRegisterInfo &MRI, const TargetInstrInfo &TII,
+                         const RegisterBankInfo &RBI, MachineInstr &InsertPt,
+                         const MachineOperand &RegMO,
+                         const TargetRegisterClass &RegClass, bool ISel = true);
+
+/// Try to constrain Reg so that it is usable by argument OpIdx of the provided
+/// MCInstrDesc \p II. If this fails, create a new virtual register in the
+/// correct class and insert a COPY before \p InsertPt for uses, or after it for
+/// defs. This is equivalent to constrainOperandRegClass(..., const
+/// TargetRegisterClass &RegClass, ...) with the RegClass obtained from the
 /// MCInstrDesc. The debug location of \p InsertPt is used for the new copy.
 ///
 /// \return The virtual register constrained to the right register class.
-unsigned constrainOperandRegClass(const MachineFunction &MF,
-                                  const TargetRegisterInfo &TRI,
-                                  MachineRegisterInfo &MRI,
-                                  const TargetInstrInfo &TII,
-                                  const RegisterBankInfo &RBI,
-                                  MachineInstr &InsertPt, const MCInstrDesc &II,
-                                  const MachineOperand &RegMO, unsigned OpIdx);
+unsigned constrainOperandRegClass(
+    const MachineFunction &MF, const TargetRegisterInfo &TRI,
+    MachineRegisterInfo &MRI, const TargetInstrInfo &TII,
+    const RegisterBankInfo &RBI, MachineInstr &InsertPt, const MCInstrDesc &II,
+    const MachineOperand &RegMO, unsigned OpIdx, bool ISel = true);
 
 /// Mutate the newly-selected instruction \p I to constrain its (possibly
 /// generic) virtual register operands to the instruction's register class.
 /// This could involve inserting COPYs before (for uses) or after (for defs).
 /// This requires the number of operands to match the instruction description.
 /// \returns whether operand regclass constraining succeeded.
 ///
 // FIXME: Not all instructions have the same number of operands. We should
 // probably expose a constrain helper per operand and let the target selector
 // constrain individual registers, like fast-isel.
 bool constrainSelectedInstRegOperands(MachineInstr &I,
                                       const TargetInstrInfo &TII,
                                       const TargetRegisterInfo &TRI,
-                                      const RegisterBankInfo &RBI);
+                                      const RegisterBankInfo &RBI,
+                                      bool ISel = true);
+
 /// Check whether an instruction \p MI is dead: it only defines dead virtual
 /// registers, and doesn't have other side effects.
 bool isTriviallyDead(const MachineInstr &MI, const MachineRegisterInfo &MRI);
 
 /// Report an ISel error as a missed optimization remark to the LLVMContext's
 /// diagnostic stream.  Set the FailedISel MachineFunction property.
 void reportGISelFailure(MachineFunction &MF, const TargetPassConfig &TPC,
                         MachineOptimizationRemarkEmitter &MORE,

include/llvm/CodeGen/MachineInstrBuilder.h

public:
 
   /// Copy all the implicit operands from OtherMI onto this one.
   const MachineInstrBuilder &
   copyImplicitOps(const MachineInstr &OtherMI) const {
     MI->copyImplicitOps(*MF, OtherMI);
     return *this;
   }
 
-  bool constrainAllUses(const TargetInstrInfo &TII,
+  bool constrainAllRegOperands(const TargetInstrInfo &TII,
                                const TargetRegisterInfo &TRI,
-                        const RegisterBankInfo &RBI) const {
-    return constrainSelectedInstRegOperands(*MI, TII, TRI, RBI);
+                               const RegisterBankInfo &RBI,
+                               bool ISel = true) const {
+    constrainSelectedInstRegOperands(*MI, TII, TRI, RBI, ISel);
+    return *this;
   }
 };
 
 /// Builder interface. Specify how to create the initial instruction itself.
 inline MachineInstrBuilder BuildMI(MachineFunction &MF, const DebugLoc &DL,
                                    const MCInstrDesc &MCID) {
   return MachineInstrBuilder(MF, MF.CreateMachineInstr(MCID, DL));
 }

include/llvm/CodeGen/MachineOperand.h

public:
   /// Unlike the previous function, this one will not try and get the
   /// information from it's parent.
   /// \param IntrinsicInfo - same as \p TRI.
   void print(raw_ostream &os, ModuleSlotTracker &MST, LLT TypeToPrint,
              bool PrintDef, bool IsStandalone, bool ShouldPrintRegisterTies,
              unsigned TiedOperandIdx, const TargetRegisterInfo *TRI,
              const TargetIntrinsicInfo *IntrinsicInfo) const;
 
+  /// Same as print(os, TRI, IntrinsicInfo), but allows to specify the low-level
+  /// type to be printed the same way the full version of print(...) does it.
+  void print(raw_ostream &os, LLT TypeToPrint,
+             const TargetRegisterInfo *TRI = nullptr,
+             const TargetIntrinsicInfo *IntrinsicInfo = nullptr) const;
+
   void dump() const;
 
   //===--------------------------------------------------------------------===//
   // Accessors that tell you what kind of MachineOperand you're looking at.
   //===--------------------------------------------------------------------===//
 
   /// isReg - Tests if this is a MO_Register operand.
   bool isReg() const { return OpKind == MO_Register; }

lib/CodeGen/GlobalISel/IRTranslator.cpp

bool IRTranslator::translateMemfunc(const CallInst &CI,
   return CLI->lowerCall(MIRBuilder, CI.getCallingConv(),
                         MachineOperand::CreateES(Callee),
                         CallLowering::ArgInfo(0, CI.getType()), Args);
 }
 
 void IRTranslator::getStackGuard(unsigned DstReg,
                                  MachineIRBuilder &MIRBuilder) {
   const TargetRegisterInfo *TRI = MF->getSubtarget().getRegisterInfo();
-  MRI->setRegClass(DstReg, TRI->getPointerRegClass(*MF));
+  unsigned TmpReg = MRI->createVirtualRegister(TRI->getPointerRegClass(*MF));
   auto MIB = MIRBuilder.buildInstr(TargetOpcode::LOAD_STACK_GUARD);
-  MIB.addDef(DstReg);
+  MIB.addDef(TmpReg);
+  MIRBuilder.buildInstr(TargetOpcode::COPY).addDef(DstReg).addUse(TmpReg);
 
   auto &TLI = *MF->getSubtarget().getTargetLowering();
   Value *Global = TLI.getSDagStackGuard(*MF->getFunction().getParent());
   if (!Global)
     return;
 
   MachinePointerInfo MPInfo(Global);
   MachineInstr::mmo_iterator MemRefs = MF->allocateMemRefsArray(1);
   auto Flags = MachineMemOperand::MOLoad | MachineMemOperand::MOInvariant |

lib/CodeGen/GlobalISel/InstructionSelector.cpp

 bool InstructionSelector::constrainOperandRegToRegClass(
     MachineInstr &I, unsigned OpIdx, const TargetRegisterClass &RC,
     const TargetInstrInfo &TII, const TargetRegisterInfo &TRI,
     const RegisterBankInfo &RBI) const {
   MachineBasicBlock &MBB = *I.getParent();
   MachineFunction &MF = *MBB.getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();
 
-  return
-      constrainRegToClass(MRI, TII, RBI, I, I.getOperand(OpIdx).getReg(), RC);
+  return constrainOperandRegClass(MRI, TII, RBI, I, I.getOperand(OpIdx), RC);
 }
 
 bool InstructionSelector::isOperandImmEqual(
     const MachineOperand &MO, int64_t Value,
     const MachineRegisterInfo &MRI) const {
   if (MO.isReg() && MO.getReg())
     if (auto VRegVal = getConstantVRegVal(MO.getReg(), MRI))
       return *VRegVal == Value;

lib/CodeGen/GlobalISel/Utils.cpp

 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/IR/Constants.h"
 
 #define DEBUG_TYPE "globalisel-utils"
 
 using namespace llvm;
 
-unsigned llvm::constrainRegToClass(MachineRegisterInfo &MRI,
+unsigned llvm::constrainOperandRegClass(MachineRegisterInfo &MRI,
                                         const TargetInstrInfo &TII,
                                         const RegisterBankInfo &RBI,
-                                   MachineInstr &InsertPt, unsigned Reg,
-                                   const TargetRegisterClass &RegClass) {
-  if (!RBI.constrainGenericRegister(Reg, RegClass, MRI)) {
-    unsigned NewReg = MRI.createVirtualRegister(&RegClass);
-    BuildMI(*InsertPt.getParent(), InsertPt, InsertPt.getDebugLoc(),
-            TII.get(TargetOpcode::COPY), NewReg)
-        .addReg(Reg);
+                                        MachineInstr &InsertPt,
+                                        const MachineOperand &RegMO,
+                                        const TargetRegisterClass &RegClass,
+                                        bool ISel) {
+  assert(InsertPt.getParent() && "The MachineInstr being constrained is "
+                                 "expected to belong to a MachineBasicBlock");
+  const unsigned Reg = RegMO.getReg();
+  if ((!ISel && MRI.getType(Reg).isValid()) ||
+      !RBI.constrainGenericRegister(Reg, RegClass, MRI)) {
+
+    MachineBasicBlock &MBB = *InsertPt.getParent();
+    DebugLoc const &DL = InsertPt.getDebugLoc();
+    MCInstrDesc const &MCID = TII.get(TargetOpcode::COPY);
+    const unsigned NewReg = MRI.createVirtualRegister(&RegClass);
+
+    if (RegMO.isUse())
+      BuildMI(MBB, InsertPt, DL, MCID, NewReg).addReg(Reg);
+    else {
+      if (InsertPt.isInsideBundle())
+        BuildMI(MBB, ++MachineBasicBlock::instr_iterator(InsertPt), DL, MCID,
+                Reg)
+            .addReg(NewReg);
+      else
+        BuildMI(MBB, ++MachineBasicBlock::iterator(InsertPt), DL, MCID, Reg)
+            .addReg(NewReg);
+    }
     return NewReg;
   }
 
   return Reg;
 }
 
 unsigned llvm::constrainOperandRegClass(
     const MachineFunction &MF, const TargetRegisterInfo &TRI,
     MachineRegisterInfo &MRI, const TargetInstrInfo &TII,
     const RegisterBankInfo &RBI, MachineInstr &InsertPt, const MCInstrDesc &II,
-    const MachineOperand &RegMO, unsigned OpIdx) {
+    const MachineOperand &RegMO, unsigned OpIdx, bool ISel) {
   unsigned Reg = RegMO.getReg();
   // Assume physical registers are properly constrained.
   assert(TargetRegisterInfo::isVirtualRegister(Reg) &&
          "PhysReg not implemented");
 
   const TargetRegisterClass *RegClass = TII.getRegClass(II, OpIdx, &TRI, MF);
   // Some of the target independent instructions, like COPY, may not impose any
   // register class constraints on some of their operands: If it's a use, we can
if (!RegClass) {
     //   v2 = COPY v1
     // v1 here may end up not being constrained at all. Please notice that to
     // reproduce the issue we likely need a destination pattern of a selection
     // rule producing such extra copies, not just an input GMIR with them as
     // every existing target using selectImpl handles copies before calling it
     // and they never reach this function.
     return Reg;
   }
-  return constrainRegToClass(MRI, TII, RBI, InsertPt, Reg, *RegClass);
+  return constrainOperandRegClass(MRI, TII, RBI, InsertPt, RegMO, *RegClass,
+                                  ISel);
 }
 
 bool llvm::constrainSelectedInstRegOperands(MachineInstr &I,
                                             const TargetInstrInfo &TII,
                                             const TargetRegisterInfo &TRI,
-                                            const RegisterBankInfo &RBI) {
+                                            const RegisterBankInfo &RBI,
+                                            bool ISel) {
   assert(!isPreISelGenericOpcode(I.getOpcode()) &&
          "A selected instruction is expected");
   MachineBasicBlock &MBB = *I.getParent();
   MachineFunction &MF = *MBB.getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();
 
   for (unsigned OpI = 0, OpE = I.getNumExplicitOperands(); OpI != OpE; ++OpI) {
     MachineOperand &MO = I.getOperand(OpI);
if (TRI.isPhysicalRegister(Reg))
       continue;
 
     // Register operands with a value of 0 (e.g. predicate operands) don't need
     // to be constrained.
     if (Reg == 0)
       continue;
 
     // If the operand is a vreg, we should constrain its regclass, and only
-    // insert COPYs if that's impossible.
-    // constrainOperandRegClass does that for us.
+    // insert COPYs if that's impossible. constrainOperandRegClass does that for
+    // us.
     MO.setReg(constrainOperandRegClass(MF, TRI, MRI, TII, RBI, I, I.getDesc(),
-                                       MO, OpI));
+                                       MO, OpI, ISel));
 
     // Tie uses to defs as indicated in MCInstrDesc if this hasn't already been
     // done.
     if (MO.isUse()) {
       int DefIdx = I.getDesc().getOperandConstraint(OpI, MCOI::TIED_TO);
       if (DefIdx != -1 && !I.isRegTiedToUseOperand(DefIdx))
         I.tieOperands(DefIdx, OpI);
     }

lib/CodeGen/MachineOperand.cpp

default:
     // TODO: Print the other CFI Operations.
     OS << "<unserializable cfi directive>";
     break;
   }
 }
 
 void MachineOperand::print(raw_ostream &OS, const TargetRegisterInfo *TRI,
                            const TargetIntrinsicInfo *IntrinsicInfo) const {
+  print(OS, LLT{}, TRI, IntrinsicInfo);
+}
+
+void MachineOperand::print(raw_ostream &OS, LLT TypeToPrint,
+                           const TargetRegisterInfo *TRI,
+                           const TargetIntrinsicInfo *IntrinsicInfo) const {
   tryToGetTargetInfo(*this, TRI, IntrinsicInfo);
   ModuleSlotTracker DummyMST(nullptr);
-  print(OS, DummyMST, LLT{}, /*PrintDef=*/false, /*IsStandalone=*/true,
+  print(OS, DummyMST, TypeToPrint, /*PrintDef=*/false, /*IsStandalone=*/true,
         /*ShouldPrintRegisterTies=*/true,
         /*TiedOperandIdx=*/0, TRI, IntrinsicInfo);
 }
 
 void MachineOperand::print(raw_ostream &OS, ModuleSlotTracker &MST,
                            LLT TypeToPrint, bool PrintDef, bool IsStandalone,
                            bool ShouldPrintRegisterTies,
                            unsigned TiedOperandIdx,

lib/CodeGen/MachineVerifier.cpp

struct MachineVerifier {
     void visitMachineInstrAfter(const MachineInstr *MI);
     void visitMachineBundleAfter(const MachineInstr *MI);
     void visitMachineBasicBlockAfter(const MachineBasicBlock *MBB);
     void visitMachineFunctionAfter();
 
     void report(const char *msg, const MachineFunction *MF);
     void report(const char *msg, const MachineBasicBlock *MBB);
     void report(const char *msg, const MachineInstr *MI);
-    void report(const char *msg, const MachineOperand *MO, unsigned MONum);
+    void report(const char *msg, const MachineOperand *MO, unsigned MONum,
+                LLT MOVRegType = LLT{});
 
     void report_context(const LiveInterval &LI) const;
     void report_context(const LiveRange &LR, unsigned VRegUnit,
                         LaneBitmask LaneMask) const;
     void report_context(const LiveRange::Segment &S) const;
     void report_context(const VNInfo &VNI) const;
     void report_context(SlotIndex Pos) const;
     void report_context_liverange(const LiveRange &LR) const;
void MachineVerifier::report(const char *msg, const MachineInstr *MI) {
   report(msg, MI->getParent());
   errs() << "- instruction: ";
   if (Indexes && Indexes->hasIndex(*MI))
     errs() << Indexes->getInstructionIndex(*MI) << '\t';
   MI->print(errs(), /*SkipOpers=*/true);
   errs() << '\n';
 }
 
-void MachineVerifier::report(const char *msg,
-                             const MachineOperand *MO, unsigned MONum) {
+void MachineVerifier::report(const char *msg, const MachineOperand *MO,
+                             unsigned MONum, LLT MOVRegType) {
   assert(MO);
   report(msg, MO->getParent());
   errs() << "- operand " << MONum << ":   ";
-  MO->print(errs(), TRI);
+  MO->print(errs(), MOVRegType, TRI);
   errs() << "\n";
 }
 
 void MachineVerifier::report_context(SlotIndex Pos) const {
   errs() << "- at:          " << Pos << '\n';
 }
 
 void MachineVerifier::report_context(const LiveInterval &LI) const {
case TargetOpcode::STATEPOINT:
     VerifyStackMapConstant(VarStart + StatepointOpers::CCOffset);
     VerifyStackMapConstant(VarStart + StatepointOpers::FlagsOffset);
     VerifyStackMapConstant(VarStart + StatepointOpers::NumDeoptOperandsOffset);
 
     // TODO: verify we have properly encoded deopt arguments
   };
 }
 
-void
-MachineVerifier::visitMachineOperand(const MachineOperand *MO, unsigned MONum) {
+void MachineVerifier::visitMachineOperand(const MachineOperand *MO,
+                                          unsigned MONum) {
+  const auto &Props = MF->getProperties();
+  const bool CouldFailMidInstructionSelect =
+      Props.hasProperty(MachineFunctionProperties::Property::RegBankSelected) &&
+      Props.hasProperty(MachineFunctionProperties::Property::FailedISel);
+
   const MachineInstr *MI = MO->getParent();
   const MCInstrDesc &MCID = MI->getDesc();
   unsigned NumDefs = MCID.getNumDefs();
   if (MCID.getOpcode() == TargetOpcode::PATCHPOINT)
     NumDefs = (MONum == 0 && MO->isReg()) ? NumDefs : 0;
 
   // The first MCID.NumDefs operands must be explicit register defines
   if (MONum < NumDefs) {
if (TargetRegisterInfo::isPhysicalRegister(Reg)) {
         if (MRI->isReserved(Reg)) {
           report("isRenamable set on reserved register", MO, MONum);
           return;
         }
       }
     } else {
       // Virtual register.
       const TargetRegisterClass *RC = MRI->getRegClassOrNull(Reg);
+      const LLT Ty = MRI->getType(Reg);
       if (!RC) {
         // This is a generic virtual register.
 
         // If we're post-Select, we can't have gvregs anymore.
         if (isFunctionSelected) {
-          report("Generic virtual register invalid in a Selected function",
-                 MO, MONum);
+          report("Generic virtual register invalid in a Selected function", MO,
+                 MONum, Ty);
           return;
         }
 
         // The gvreg must have a type and it must not have a SubIdx.
-        LLT Ty = MRI->getType(Reg);
         if (!Ty.isValid()) {
           report("Generic virtual register must have a valid type", MO,
                  MONum);
           return;
         }
 
         const RegisterBank *RegBank = MRI->getRegBankOrNull(Reg);
 
         // If we're post-RegBankSelect, the gvreg must have a bank.
         if (!RegBank && isFunctionRegBankSelected) {
           report("Generic virtual register must have a bank in a "
                  "RegBankSelected function",
-                 MO, MONum);
+                 MO, MONum, Ty);
           return;
         }
 
         // Make sure the register fits into its register bank if any.
         if (RegBank && Ty.isValid() &&
             RegBank->getSize() < Ty.getSizeInBits()) {
-          report("Register bank is too small for virtual register", MO,
-                 MONum);
+          report("Register bank is too small for virtual register", MO, MONum,
+                 Ty);
           errs() << "Register bank " << RegBank->getName() << " too small("
                  << RegBank->getSize() << ") to fit " << Ty.getSizeInBits()
                  << "-bits\n";
           return;
         }
-        if (SubIdx)  {
+        if (SubIdx) {
           report("Generic virtual register does not subregister index", MO,
-                 MONum);
+                 MONum, Ty);
           return;
         }
 
         // If this is a target specific instruction and this operand
         // has register class constraint, the virtual register must
         // comply to it.
         if (!isPreISelGenericOpcode(MCID.getOpcode()) &&
             MONum < MCID.getNumOperands() &&
             TII->getRegClass(MCID, MONum, TRI, *MF)) {
           report("Virtual register does not match instruction constraint", MO,
-                 MONum);
+                 MONum, Ty);
           errs() << "Expect register class "
                  << TRI->getRegClassName(
                         TII->getRegClass(MCID, MONum, TRI, *MF))
                  << " but got nothing\n";
           return;
         }
 
         break;
       }
+      if (!CouldFailMidInstructionSelect && Ty.isValid()) {
+        report("Register class-constrained virtual registers are allowed to "
+               "have types mid-instruction select pass only",
+               MO, MONum, Ty);
+        return;
+      }
       if (SubIdx) {
         const TargetRegisterClass *SRC =
           TRI->getSubClassWithSubReg(RC, SubIdx);
         if (!SRC) {
           report("Invalid subregister index for virtual register", MO, MONum);
           errs() << "Register class " << TRI->getRegClassName(RC)
               << " does not support subreg index " << SubIdx << "\n";
           return;

lib/Target/AArch64/AArch64CallLowering.cpp

bool AArch64CallLowering::lowerCall(MachineIRBuilder &MIRBuilder,
   MIRBuilder.insertInstr(MIB);
 
   // If Callee is a reg, since it is used by a target specific
   // instruction, it must have a register class matching the
   // constraint of that instruction.
   if (Callee.isReg())
     MIB->getOperand(0).setReg(constrainOperandRegClass(
         MF, *TRI, MRI, *MF.getSubtarget().getInstrInfo(),
-        *MF.getSubtarget().getRegBankInfo(), *MIB, MIB->getDesc(), Callee, 0));
+        *MF.getSubtarget().getRegBankInfo(), *MIB, MIB->getDesc(), Callee, 0,
+        /*ISel*/ false));
 
   // Finally we can copy the returned value back into its virtual-register. In
   // symmetry with the arugments, the physical register must be an
   // implicit-define of the call instruction.
   CCAssignFn *RetAssignFn = TLI.CCAssignFnForReturn(F.getCallingConv());
   if (OrigRet.Reg) {
     SplitArgs.clear();
 

lib/Target/AArch64/AArch64InstructionSelector.cpp

 }
 
 // FIXME: This should be target-independent, inferred from the types declared
 // for each class in the bank.
 static const TargetRegisterClass *
 getRegClassForTypeOnBank(LLT Ty, const RegisterBank &RB,
                          const RegisterBankInfo &RBI,
                          bool GetAllRegSet = false) {
+  assert(Ty.isValid() && "Expected a valid type");
   if (RB.getID() == AArch64::GPRRegBankID) {
     if (Ty.getSizeInBits() <= 32)
       return GetAllRegSet ? &AArch64::GPR32allRegClass
                           : &AArch64::GPR32RegClass;
     if (Ty.getSizeInBits() == 64)
       return GetAllRegSet ? &AArch64::GPR64allRegClass
                           : &AArch64::GPR64RegClass;
     return nullptr;
if (RB.getID() == AArch64::FPRRegBankID) {
     if (Ty.getSizeInBits() == 128)
       return &AArch64::FPR128RegClass;
     return nullptr;
   }
 
   return nullptr;
 }
 
+static const TargetRegisterClass *getRegClass(unsigned Reg,
+                                              const MachineRegisterInfo &MRI,
+                                              const RegisterBankInfo &RBI) {
+  const RegClassOrRegBank &RegClassOrBank = MRI.getRegClassOrRegBank(Reg);
+  const auto *RC = RegClassOrBank.dyn_cast<const TargetRegisterClass *>();
+  if (RC)
+    return RC;
+  const auto *RB = RegClassOrBank.get<const RegisterBank *>();
+  return getRegClassForTypeOnBank(MRI.getType(Reg), *RB, RBI,
+                                  /*GetAllRegSet*/ true);
+}
+
 /// Check whether \p I is a currently unsupported binary operation:
 /// - it has an unsized type
 /// - an operand is not a vreg
 /// - all operands are not in the same bank
 /// These are checks that should someday live in the verifier, but right now,
 /// these are mostly limitations of the aarch64 selector.
 static bool unsupportedBinOp(const MachineInstr &I,
                              const AArch64RegisterBankInfo &RBI,
static bool selectCopy(MachineInstr &I, const TargetInstrInfo &TII,
                        const RegisterBankInfo &RBI) {
 
   unsigned DstReg = I.getOperand(0).getReg();
   unsigned SrcReg = I.getOperand(1).getReg();
 
   if (TargetRegisterInfo::isPhysicalRegister(DstReg)) {
     if (TRI.getRegClass(AArch64::FPR16RegClassID)->contains(DstReg) &&
         !TargetRegisterInfo::isPhysicalRegister(SrcReg)) {
-      const RegisterBank &RegBank = *RBI.getRegBank(SrcReg, MRI, TRI);
-      const TargetRegisterClass *SrcRC = getRegClassForTypeOnBank(
-          MRI.getType(SrcReg), RegBank, RBI, /* GetAllRegSet */ true);
-      if (SrcRC == &AArch64::GPR32allRegClass)
+      const TargetRegisterClass *SrcRC = getRegClass(SrcReg, MRI, RBI);
+      if (SrcRC->hasSuperClassEq(&AArch64::GPR32allRegClass))
         return selectFP16CopyFromGPR32(I, TII, MRI, SrcReg);
     }
     assert(I.isCopy() && "Generic operators do not allow physical registers");
     return true;
   }
 
   const RegisterBank &RegBank = *RBI.getRegBank(DstReg, MRI, TRI);
-  const unsigned DstSize = MRI.getType(DstReg).getSizeInBits();
+  (void)RegBank;
+  const unsigned DstSize = RBI.getSizeInBits(DstReg, MRI, TRI);
   (void)DstSize;
   const unsigned SrcSize = RBI.getSizeInBits(SrcReg, MRI, TRI);
   (void)SrcSize;
   assert((!TargetRegisterInfo::isPhysicalRegister(SrcReg) || I.isCopy()) &&
          "No phys reg on generic operators");
   assert(
       (DstSize == SrcSize ||
        // Copies are a mean to setup initial types, the number of
        // bits may not exactly match.
-       (TargetRegisterInfo::isPhysicalRegister(SrcReg) &&
-        DstSize <= RBI.getSizeInBits(SrcReg, MRI, TRI)) ||
+       (TargetRegisterInfo::isPhysicalRegister(SrcReg) && DstSize <= SrcSize) ||
        // Copies are a mean to copy bits around, as long as we are
        // on the same register class, that's fine. Otherwise, that
        // means we need some SUBREG_TO_REG or AND & co.
        (((DstSize + 31) / 32 == (SrcSize + 31) / 32) && DstSize > SrcSize)) &&
       "Copy with different width?!");
   assert((DstSize <= 64 || RegBank.getID() == AArch64::FPRRegBankID) &&
          "GPRs cannot get more than 64-bit width values");
 
-  const TargetRegisterClass *RC = getRegClassForTypeOnBank(
-      MRI.getType(DstReg), RegBank, RBI, /* GetAllRegSet */ true);
+  const TargetRegisterClass *RC = getRegClass(DstReg, MRI, RBI);
   if (!RC) {
     DEBUG(dbgs() << "Unexpected bitcast size " << DstSize << '\n');
     return false;
   }
 
   if (!TargetRegisterInfo::isPhysicalRegister(SrcReg)) {
-    const RegClassOrRegBank &RegClassOrBank = MRI.getRegClassOrRegBank(SrcReg);
-    const TargetRegisterClass *SrcRC =
-        RegClassOrBank.dyn_cast<const TargetRegisterClass *>();
-    const RegisterBank *RB = nullptr;
-    if (!SrcRC) {
-      RB = RegClassOrBank.get<const RegisterBank *>();
-      SrcRC = getRegClassForTypeOnBank(MRI.getType(SrcReg), *RB, RBI, true);
-    }
+    const TargetRegisterClass *SrcRC = getRegClass(SrcReg, MRI, RBI);
     // Copies from fpr16 to gpr32 need to use SUBREG_TO_REG.
-    if (RC == &AArch64::GPR32allRegClass && SrcRC == &AArch64::FPR16RegClass) {
+    if (RC->hasSuperClassEq(&AArch64::GPR32allRegClass) &&
+        SrcRC->hasSuperClassEq(&AArch64::FPR16RegClass)) {
       unsigned PromoteReg = MRI.createVirtualRegister(&AArch64::FPR32RegClass);
       BuildMI(*I.getParent(), I, I.getDebugLoc(),
               TII.get(AArch64::SUBREG_TO_REG))
           .addDef(PromoteReg)
           .addImm(0)
           .addUse(SrcReg)
           .addImm(AArch64::hsub);
       MachineOperand &RegOp = I.getOperand(1);
       RegOp.setReg(PromoteReg);
-    } else if (RC == &AArch64::FPR16RegClass &&
-               SrcRC == &AArch64::GPR32allRegClass) {
+    } else if (RC->hasSuperClassEq(&AArch64::FPR16RegClass) &&
+               SrcRC->hasSuperClassEq(&AArch64::GPR32allRegClass)) {
       selectFP16CopyFromGPR32(I, TII, MRI, SrcReg);
     }
   }
 
   // No need to constrain SrcReg. It will get constrained when
   // we hit another of its use or its defs.
   // Copies do not have constraints.
   if (!RBI.constrainGenericRegister(DstReg, *RC, MRI)) {
bool AArch64InstructionSelector::selectCompareBranch(
   else if (CmpWidth == 64)
     CBOpc = (Pred == CmpInst::ICMP_EQ ? AArch64::CBZX : AArch64::CBNZX);
   else
     return false;
 
   BuildMI(*I.getParent(), I, I.getDebugLoc(), TII.get(CBOpc))
       .addUse(LHS)
       .addMBB(DestMBB)
-      .constrainAllUses(TII, TRI, RBI);
+      .constrainAllRegOperands(TII, TRI, RBI);
 
   I.eraseFromParent();
   return true;
 }
 
 bool AArch64InstructionSelector::selectVaStartAAPCS(
     MachineInstr &I, MachineFunction &MF, MachineRegisterInfo &MRI) const {
   return false;

lib/Target/ARM/ARMCallLowering.cpp

bool ARMCallLowering::lowerCall(MachineIRBuilder &MIRBuilder,
   auto CallOpcode =
       isDirect ? ARM::BL
                : STI.hasV5TOps()
                      ? ARM::BLX
                      : STI.hasV4TOps() ? ARM::BX_CALL : ARM::BMOVPCRX_CALL;
   auto MIB = MIRBuilder.buildInstrNoInsert(CallOpcode)
                  .add(Callee)
                  .addRegMask(TRI->getCallPreservedMask(MF, CallConv));
-  if (Callee.isReg()) {
-    auto CalleeReg = Callee.getReg();
-    if (CalleeReg && !TRI->isPhysicalRegister(CalleeReg))
-      MIB->getOperand(0).setReg(constrainOperandRegClass(
-          MF, *TRI, MRI, *STI.getInstrInfo(), *STI.getRegBankInfo(),
-          *MIB.getInstr(), MIB->getDesc(), Callee, 0));
-  }
 
   SmallVector<ArgInfo, 8> ArgInfos;
   for (auto Arg : OrigArgs) {
     if (!isSupportedType(DL, TLI, Arg.Ty))
       return false;
 
     if (!Arg.IsFixed)
       return false;
bool ARMCallLowering::lowerCall(MachineIRBuilder &MIRBuilder,
   auto ArgAssignFn = TLI.CCAssignFnForCall(CallConv, /*IsVarArg=*/false);
   OutgoingValueHandler ArgHandler(MIRBuilder, MRI, MIB, ArgAssignFn);
   if (!handleAssignments(MIRBuilder, ArgInfos, ArgHandler))
     return false;
 
   // Now we can add the actual call instruction to the correct basic block.
   MIRBuilder.insertInstr(MIB);
 
+  if (Callee.isReg()) {
+    auto CalleeReg = Callee.getReg();
+    if (CalleeReg && !TRI->isPhysicalRegister(CalleeReg))
+      MIB->getOperand(0).setReg(constrainOperandRegClass(
+          MF, *TRI, MRI, *STI.getInstrInfo(), *STI.getRegBankInfo(),
+          *MIB.getInstr(), MIB->getDesc(), Callee, 0, /*ISel*/ false));
+  }
+
   if (!OrigRet.Ty->isVoidTy()) {
     if (!isSupportedType(DL, TLI, OrigRet.Ty))
       return false;
 
     ArgInfos.clear();
     SmallVector<unsigned, 8> SplitRegs;
     splitToValueTypes(OrigRet, ArgInfos, MF,
                       [&](unsigned Reg, uint64_t Offset) {

lib/Target/ARM/ARMInstructionSelector.cpp

 #undef GET_GLOBALISEL_TEMPORARIES_INIT
 {
 }
 
 static const TargetRegisterClass *guessRegClass(unsigned Reg,
                                                 MachineRegisterInfo &MRI,
                                                 const TargetRegisterInfo &TRI,
                                                 const RegisterBankInfo &RBI) {
+  const TargetRegisterClass *RC = MRI.getRegClassOrNull(Reg);
+  if (RC)
+    return RC;
+
   const RegisterBank *RegBank = RBI.getRegBank(Reg, MRI, TRI);
   assert(RegBank && "Can't get reg bank for virtual register");
 
   const unsigned Size = MRI.getType(Reg).getSizeInBits();
   assert((RegBank->getID() == ARM::GPRRegBankID ||
           RegBank->getID() == ARM::FPRRegBankID) &&
          "Unsupported reg bank");
 

lib/Target/X86/X86CallLowering.cpp

bool X86CallLowering::lowerCall(MachineIRBuilder &MIRBuilder,
   MIRBuilder.insertInstr(MIB);
 
   // If Callee is a reg, since it is used by a target specific
   // instruction, it must have a register class matching the
   // constraint of that instruction.
   if (Callee.isReg())
     MIB->getOperand(0).setReg(constrainOperandRegClass(
         MF, *TRI, MRI, *MF.getSubtarget().getInstrInfo(),
-        *MF.getSubtarget().getRegBankInfo(), *MIB, MIB->getDesc(), Callee, 0));
+        *MF.getSubtarget().getRegBankInfo(), *MIB, MIB->getDesc(), Callee, 0,
+        /*ISel*/ false));
 
   // Finally we can copy the returned value back into its virtual-register. In
   // symmetry with the arguments, the physical register must be an
   // implicit-define of the call instruction.
 
   if (OrigRet.Reg) {
     SplitArgs.clear();
     SmallVector<unsigned, 8> NewRegs;

lib/Target/X86/X86InstructionSelector.cpp

X86InstructionSelector::getRegClass(LLT Ty, const RegisterBank &RB) const {
   }
 
   llvm_unreachable("Unknown RegBank!");
 }
 
 const TargetRegisterClass *
 X86InstructionSelector::getRegClass(LLT Ty, unsigned Reg,
                                     MachineRegisterInfo &MRI) const {
+  assert(Ty.isValid() && "Expected a valid type");
   const RegisterBank &RegBank = *RBI.getRegBank(Reg, MRI, TRI);
   return getRegClass(Ty, RegBank);
 }
 
 static unsigned getSubRegIndex(const TargetRegisterClass *RC) {
   unsigned SubIdx = X86::NoSubRegister;
   if (RC == &X86::GR32RegClass) {
     SubIdx = X86::sub_32bit;
bool X86InstructionSelector::selectCopy(MachineInstr &I,
   const RegisterBank &SrcRegBank = *RBI.getRegBank(SrcReg, MRI, TRI);
 
   if (TargetRegisterInfo::isPhysicalRegister(DstReg)) {
     assert(I.isCopy() && "Generic operators do not allow physical registers");
 
     if (DstSize > SrcSize && SrcRegBank.getID() == X86::GPRRegBankID &&
         DstRegBank.getID() == X86::GPRRegBankID) {
 
+      const TargetRegisterClass *OldRC = MRI.getRegClassOrNull(SrcReg);
       const TargetRegisterClass *SrcRC =
-          getRegClass(MRI.getType(SrcReg), SrcRegBank);
+          OldRC ? OldRC : getRegClass(MRI.getType(SrcReg), SrcRegBank);
       const TargetRegisterClass *DstRC = getRegClassFromGRPhysReg(DstReg);
 
       if (SrcRC != DstRC) {
         // This case can be generated by ABI lowering, performe anyext
         unsigned ExtSrc = MRI.createVirtualRegister(DstRC);
         BuildMI(*I.getParent(), I, I.getDebugLoc(),
                 TII.get(TargetOpcode::SUBREG_TO_REG))
             .addDef(ExtSrc)
assert((!TargetRegisterInfo::isPhysicalRegister(SrcReg) || I.isCopy()) &&
          "No phys reg on generic operators");
   assert((DstSize == SrcSize ||
           // Copies are a mean to setup initial types, the number of
           // bits may not exactly match.
           (TargetRegisterInfo::isPhysicalRegister(SrcReg) &&
            DstSize <= RBI.getSizeInBits(SrcReg, MRI, TRI))) &&
          "Copy with different width?!");
 
+  const TargetRegisterClass *OldRC = MRI.getRegClassOrNull(DstReg);
   const TargetRegisterClass *DstRC =
-      getRegClass(MRI.getType(DstReg), DstRegBank);
+      OldRC ? OldRC : getRegClass(MRI.getType(DstReg), DstRegBank);
 
   if (SrcRegBank.getID() == X86::GPRRegBankID &&
       DstRegBank.getID() == X86::GPRRegBankID && SrcSize > DstSize &&
       TargetRegisterInfo::isPhysicalRegister(SrcReg)) {
     // Change the physical register to performe truncate.
 
     const TargetRegisterClass *SrcRC = getRegClassFromGRPhysReg(SrcReg);
 
     if (DstRC != SrcRC) {
       I.getOperand(1).setSubReg(getSubRegIndex(DstRC));
       I.getOperand(1).substPhysReg(SrcReg, TRI);
     }
   }
 
   // No need to constrain SrcReg. It will get constrained when
   // we hit another of its use or its defs.
   // Copies do not have constraints.
-  const TargetRegisterClass *OldRC = MRI.getRegClassOrNull(DstReg);
   if (!OldRC || !DstRC->hasSubClassEq(OldRC)) {
     if (!RBI.constrainGenericRegister(DstReg, *DstRC, MRI)) {
       DEBUG(dbgs() << "Failed to constrain " << TII.getName(I.getOpcode())
                    << " operand\n");
       return false;
     }
   }
   I.setDesc(TII.get(X86::COPY));

test/CodeGen/AArch64/GlobalISel/arm64-irtranslator-stackprotect.ll

 ; RUN: llc -verify-machineinstrs -mtriple=aarch64-apple-ios %s -stop-after=irtranslator -o - -global-isel | FileCheck %s
 
 
 ; CHECK: name: test_stack_guard
 
 ; CHECK: stack:
 ; CHECK:  - { id: 0, name: StackGuardSlot,  type: default, offset: 0, size: 8, alignment: 8,
 ; CHECK-NOT: id: 1
 
 ; CHECK: [[GUARD_SLOT:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.0.StackGuardSlot
-; CHECK: [[GUARD:%[0-9]+]]:gpr64sp(p0) = LOAD_STACK_GUARD :: (dereferenceable invariant load 8 from @__stack_chk_guard)
-; CHECK: G_STORE [[GUARD]](p0), [[GUARD_SLOT]](p0) :: (volatile store 8 into %stack.0.StackGuardSlot)
+; CHECK: [[GUARD_RC:%[0-9]+]]:gpr64sp = LOAD_STACK_GUARD :: (dereferenceable invariant load 8 from @__stack_chk_guard)
+; CHECK: [[GUARD_TY:%[0-9]+]]:_(p0) = COPY [[GUARD_RC]]
+; CHECK: G_STORE [[GUARD_TY]](p0), [[GUARD_SLOT]](p0) :: (volatile store 8 into %stack.0.StackGuardSlot)
 declare void @llvm.stackprotector(i8*, i8**)
 define void @test_stack_guard_remat2() {
   %StackGuardSlot = alloca i8*
   call void @llvm.stackprotector(i8* undef, i8** %StackGuardSlot)
   ret void
 }
 
 @__stack_chk_guard = external global i64*

test/CodeGen/AArch64/GlobalISel/arm64-regbankselect.mir

 
 ---
 # Check that we are able to propagate register banks from phis.
 name:            phiPropagation
 legalized:       true
 tracksRegLiveness:   true
 # CHECK:      registers:
 # CHECK-NEXT:   - { id: 0, class: gpr32, preferred-register: '' }
-# CHECK-NEXT:   - { id: 1, class: gpr64sp, preferred-register: '' }
+# CHECK-NEXT:   - { id: 1, class: gpr, preferred-register: '' }
 # CHECK-NEXT:   - { id: 2, class: gpr32, preferred-register: '' }
 # CHECK-NEXT:   - { id: 3, class: gpr, preferred-register: '' }
 # CHECK-NEXT:   - { id: 4, class: gpr, preferred-register: '' }
 registers:
   - { id: 0, class: gpr32 }
-  - { id: 1, class: gpr64sp }
+  - { id: 1, class: gpr }
   - { id: 2, class: gpr32 }
   - { id: 3, class: _ }
   - { id: 4, class: _ }
   - { id: 5, class: _ }
+  - { id: 6, class: _ }
 body: |
   bb.0.entry:
     successors: %bb.2.end, %bb.1.then
     liveins: $x0, $x1, $w2
 
     %0 = LDRWui killed $x0, 0 :: (load 4 from %ir.src)
     %5(s32) = COPY %0
     %1(p0) = COPY $x1
     %2 = COPY $w2
     TBNZW killed %2, 0, %bb.2.end
 
   bb.1.then:
     successors: %bb.2.end
     %3(s32) = G_ADD %5, %5
 
   bb.2.end:
-    %4(s32) = PHI %0, %bb.0.entry, %3, %bb.1.then
+    %4(s32) = PHI %5, %bb.0.entry, %3, %bb.1.then
     G_STORE killed %4, killed %1 :: (store 4 into %ir.dst)
     RET_ReallyLR
 ...
 
 ---
 # Make sure we can repair physical register uses as well.
 name:            defaultMappingUseRepairPhysReg
 legalized:       true

test/CodeGen/AArch64/GlobalISel/call-translator.ll

 define void @test_simple_arg(i32 %in) {
   call void @simple_arg_callee(i32 %in)
   ret void
 }
 
 ; CHECK-LABEL: name: test_indirect_call
 ; CHECK: registers:
 ; Make sure the register feeding the indirect call is properly constrained.
-; CHECK: - { id: [[FUNC:[0-9]+]], class: gpr64, preferred-register: '' }
-; CHECK: %[[FUNC]]:gpr64(p0) = COPY $x0
-; CHECK: BLR %[[FUNC]](p0), csr_aarch64_aapcs, implicit-def $lr, implicit $sp
+; CHECK: - { id: [[FUNC_TY:[0-9]+]], class: _, preferred-register: '' }
+; CHECK: - { id: [[FUNC_RC:[0-9]+]], class: gpr64, preferred-register: '' }
+; CHECK: %[[FUNC_TY]]:_(p0) = COPY $x0
+; CHECK: %[[FUNC_RC]]:gpr64 = COPY %[[FUNC_TY]](p0)
+; CHECK: BLR %[[FUNC_RC]], csr_aarch64_aapcs, implicit-def $lr, implicit $sp
 ; CHECK: RET_ReallyLR
 define void @test_indirect_call(void()* %func) {
   call void %func()
   ret void
 }
 
 ; CHECK-LABEL: name: test_multiple_args
 ; CHECK: [[IN:%[0-9]+]]:_(s64) = COPY $x0

test/CodeGen/AArch64/GlobalISel/irtranslator-exceptions.ll

 
 continue:
   %sel.int = tail call i32 @llvm.eh.typeid.for(i8* bitcast(i8** @_ZTIi to i8*))
   %res.good = insertvalue { i8*, i32 } undef, i32 %sel.int, 1
   ret { i8*, i32 } %res.good
 }
 
 ; CHECK-LABEL: name: test_invoke_indirect
-; CHECK: [[CALLEE:%[0-9]+]]:gpr64(p0) = COPY $x0
-; CHECK: BLR [[CALLEE]]
+; CHECK: [[CALLEE_TY:%[0-9]+]]:_(p0) = COPY $x0
+; CHECK: [[CALLEE_RC:%[0-9]+]]:gpr64 = COPY [[CALLEE_TY]](p0)
+; CHECK: BLR [[CALLEE_RC]]
 define void @test_invoke_indirect(void()* %callee) personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
   invoke void %callee() to label %continue unwind label %broken
 
 broken:
   landingpad { i8*, i32 } catch i8* bitcast(i8** @_ZTIi to i8*)
   ret void
 
 continue:

test/CodeGen/AArch64/GlobalISel/select-copy.ll

+; RUN: llc -mtriple=aarch64-apple-ios -global-isel -O0 -start-before=irtranslator -stop-after=irtranslator       -verify-machineinstrs -simplify-mir %s -o - | FileCheck %s --check-prefix=TRANSLD
+; RUN: llc -mtriple=aarch64-apple-ios -global-isel -O0 -start-before=irtranslator -stop-after=instruction-select -verify-machineinstrs -simplify-mir %s -o - | FileCheck %s --check-prefix=SELECTD
+; RUN: llc -mtriple=aarch64-apple-ios -global-isel -O0 -start-before=irtranslator -stop-after=regallocfast       -verify-machineinstrs -simplify-mir %s -o - | FileCheck %s --check-prefix=RALLOCD
+
+@_ZTIi = external global i8*
+
+define void @test_indirect_call(void()* %func) {
+  ; TRANSLD-LABEL: name: test_indirect_call
+  ; TRANSLD: bb.1 (%ir-block.0):
+  ; TRANSLD:   liveins: $x0
+  ; TRANSLD:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; TRANSLD:   ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
+
+  ; The copy of interest, inserted here to avoid having vregs with both types and classes
+  ; assigned crossing GlobalISel passes' boundaries:
+  ; TRANSLD:   [[COPY1:%[0-9]+]]:gpr64 = COPY [[COPY]](p0)
+  ; TRANSLD:   BLR [[COPY1]], csr_aarch64_aapcs, implicit-def $lr, implicit $sp
+  ; TRANSLD:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+  ; TRANSLD:   RET_ReallyLR
+
+  ; SELECTD-LABEL: name: test_indirect_call
+  ; SELECTD-NOT:   failedISel: true
+  ; SELECTD:      bb.1 (%ir-block.0):
+  ; SELECTD-NEXT:   liveins: $x0
+  ; SELECTD:        [[COPY:%[0-9]+]]:gpr64all = COPY $x0
+  ; SELECTD-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
+
+  ; Check that the copy from a typed (generic) vreg to a reg-classed (non-generic) vreg
+  ; is correctly selected:
+  ; SELECTD-NEXT:   [[COPY1:%[0-9]+]]:gpr64 = COPY [[COPY]]
+  ; SELECTD-NEXT:   BLR [[COPY1]], csr_aarch64_aapcs, implicit-def $lr, implicit $sp
+  ; SELECTD-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+  ; SELECTD-NEXT:   RET_ReallyLR
+
+  ; RALLOCD-LABEL: name: test_indirect_call
+  ; RALLOCD:      bb.1 (%ir-block.0):
+  ; RALLOCD:        liveins: $x0
+  ; RALLOCD:        ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
+
+  ; Check that the extra copy gets eliminated:
+  ; RALLOCD-NEXT:   BLR killed renamable $x0, csr_aarch64_aapcs, implicit-def $lr, implicit $sp
+  ; RALLOCD-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+  ; RALLOCD-NEXT:   RET_ReallyLR
+  call void %func()
+  ret void
+}
+
+; Same as test_indirect_call, just for invoke
+define void @test_invoke_indirect(void()* %callee) personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
+  ; TRANSLD-LABEL: name: test_invoke_indirect
+  ; TRANSLD: bb.1 (%ir-block.0):
+  ; TRANSLD:   successors: %bb.3, %bb.2
+  ; TRANSLD:   liveins: $x0
+  ; TRANSLD:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; TRANSLD:   EH_LABEL <mcsymbol Ltmp0>
+  ; TRANSLD:   ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
+  ; TRANSLD:   [[COPY1:%[0-9]+]]:gpr64 = COPY [[COPY]](p0)
+  ; TRANSLD:   BLR [[COPY1]], csr_aarch64_aapcs, implicit-def $lr, implicit $sp
+  ; TRANSLD:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+  ; TRANSLD:   EH_LABEL <mcsymbol Ltmp1>
+  ; TRANSLD:   G_BR %bb.3
+
+  ; SELECTD-LABEL: name: test_invoke_indirect
+  ; SELECTD-NOT:   failedISel: true
+  ; SELECTD:      bb.1 (%ir-block.0):
+  ; SELECTD-NEXT:   successors: %bb.3, %bb.2
+  ; SELECTD-NEXT:   liveins: $x0
+  ; SELECTD:        [[COPY:%[0-9]+]]:gpr64all = COPY $x0
+  ; SELECTD-NEXT:   EH_LABEL <mcsymbol Ltmp0>
+  ; SELECTD-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
+  ; SELECTD-NEXT:   [[COPY1:%[0-9]+]]:gpr64 = COPY [[COPY]]
+  ; SELECTD-NEXT:   BLR [[COPY1]], csr_aarch64_aapcs, implicit-def $lr, implicit $sp
+  ; SELECTD-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+  ; SELECTD-NEXT:   EH_LABEL <mcsymbol Ltmp1>
+  ; SELECTD-NEXT:   B %bb.3
+
+  ; RALLOCD-LABEL: name: test_invoke_indirect
+  ; RALLOCD:      bb.1 (%ir-block.0):
+  ; RALLOCD:        liveins: $x0
+  ; RALLOCD:        ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
+  ; RALLOCD-NEXT:   BLR killed renamable $x0, csr_aarch64_aapcs, implicit-def $lr, implicit $sp
+  ; RALLOCD-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+  ; RALLOCD:        B %bb.3
+  invoke void %callee() to label %continue unwind label %broken
+
+broken:
+  landingpad { i8*, i32 } catch i8* bitcast(i8** @_ZTIi to i8*)
+  ret void
+
+continue:
+  ret void
+}
+
+; Check that if GlobalISel fails mid-Instruction Select pass, it still leaves MIR in a consistent state,
+; as well as that MachineVerifier is able to recognize that having vregs with types and classes both
+; assigned is allowed mid-Instruction Select pass.
+define void @test_struct_formal({double, i64, i8} %in, {double, i64, i8}* %addr) {
+  ; TRANSLD-LABEL: name: test_struct_formal
+  ; TRANSLD: bb.1 (%ir-block.0):
+  ; TRANSLD:   liveins: $d0, $w1, $x0, $x2
+  ; TRANSLD:   [[COPY:%[0-9]+]]:_(s64) = COPY $d0
+  ; TRANSLD:   [[COPY1:%[0-9]+]]:_(s64) = COPY $x0
+  ; TRANSLD:   [[COPY2:%[0-9]+]]:_(s32) = COPY $w1
+  ; TRANSLD:   [[TRUNC:%[0-9]+]]:_(s8) = G_TRUNC [[COPY2]](s32)
+  ; TRANSLD:   [[COPY3:%[0-9]+]]:_(p0) = COPY $x2
+  ; TRANSLD:   [[DEF:%[0-9]+]]:_(s192) = G_IMPLICIT_DEF
+  ; TRANSLD:   [[INSERT:%[0-9]+]]:_(s192) = G_INSERT [[DEF]], [[COPY]](s64), 0
+  ; TRANSLD:   [[INSERT1:%[0-9]+]]:_(s192) = G_INSERT [[INSERT]], [[COPY1]](s64), 64
+  ; TRANSLD:   [[INSERT2:%[0-9]+]]:_(s192) = G_INSERT [[INSERT1]], [[TRUNC]](s8), 128
+  ; TRANSLD:   [[COPY4:%[0-9]+]]:_(s192) = COPY [[INSERT2]](s192)
+  ; TRANSLD:   G_STORE [[COPY4]](s192), [[COPY3]](p0) :: (store 24 into %ir.addr, align 8)
+  ; TRANSLD:   RET_ReallyLR
+
+  ; SELECTED-LABEL: name: test_struct_formal
+  ; SELECTED:       failedISel: true
+
+  ; SELECTED:      bb.1 (%ir-block.0):
+  ; SELECTED-NEXT:   liveins: $d0, $w1, $x0, $x2
+
+  ; Not selected, all vregs having either bank and type (the majority), or class, but not both:
+  ; SELECTD:        [[COPY:%[0-9]+]]:fpr(s64) = COPY $d0
+  ; SELECTD-NEXT:   [[COPY1:%[0-9]+]]:gpr(s64) = COPY $x0
+  ; SELECTD-NEXT:   [[COPY2:%[0-9]+]]:gpr(s32) = COPY $w1
+  ; SELECTD-NEXT:   [[TRUNC:%[0-9]+]]:gpr(s8) = G_TRUNC [[COPY2]](s32)
+  ; SELECTD-NEXT:   [[COPY3:%[0-9]+]]:gpr64sp(p0) = COPY $x2
+  ; SELECTD-NEXT:   [[DEF:%[0-9]+]]:gpr(s64) = G_IMPLICIT_DEF
+  ; SELECTD-NEXT:   [[DEF1:%[0-9]+]]:gpr(s64) = G_IMPLICIT_DEF
+  ; SELECTD-NEXT:   [[DEF2:%[0-9]+]]:gpr(s64) = G_IMPLICIT_DEF
+  ; SELECTD-NEXT:   [[INSERT:%[0-9]+]]:gpr(s64) = G_INSERT [[DEF2]], [[TRUNC]](s8), 0
+  ; SELECTD-NEXT:   [[COPY4:%[0-9]+]]:gpr(s64) = COPY [[COPY]](s64)
+  ; SELECTD-NEXT:   [[MV:%[0-9]+]]:fpr(s192) = G_MERGE_VALUES [[COPY4]](s64), [[COPY1]](s64), [[INSERT]](s64)
+  ; SELECTD-NEXT:   [[COPY5:%[0-9]+]]:fpr(s192) = COPY [[MV]](s192)
+
+  ; Not selected, but defining (or using) some of the vregs used in selected code, therefore
+  ; having reglasses (from selected code) and types (to maintain the instruction selectable) assigned both,
+  ; as well as using vregs from non-selected code, therefore having banks and types assigned
+  ; (to maintain the instruction selectable), but not classes (as non-selected code doesn't define or use vregs
+  ; with types and classes both):
+  ; SELECTD-NEXT:   [[UV:%[0-9]+]]:gpr64(s64), [[UV1:%[0-9]+]]:gpr64(s64), [[UV2:%[0-9]+]]:gpr64(s64) = G_UNMERGE_VALUES [[COPY5]](s192)
+
+  ; Selected, all vregs having a reg-class assigned, and also possibly types not cleaned up just yet:
+  ; SELECTD-NEXT:   STRXui [[UV]](s64), [[COPY3]](p0), 0 :: (store 24 into %ir.addr, align 8)
+  ; SELECTD-NEXT:   STRXui [[UV1]](s64), [[COPY3]](p0), 1 :: (store 24 into %ir.addr, align 8)
+  ; SELECTD-NEXT:   STRXui [[UV2]](s64), [[COPY3]](p0), 2 :: (store 24 into %ir.addr, align 8)
+  ; SELECTD-NEXT:   RET_ReallyLR
+  store {double, i64, i8} %in, {double, i64, i8}* %addr
+  ret void
+}
+
+declare i32 @__gxx_personality_v0(...)
+declare i32 @llvm.eh.typeid.for(i8*)

test/CodeGen/ARM/GlobalISel/arm-call-lowering.ll

-; RUN: llc -mtriple arm-unknown -mattr=-v4t -global-isel -stop-after=irtranslator -verify-machineinstrs %s -o - | FileCheck %s -check-prefixes=CHECK,NOV4T
-; RUN: llc -mtriple arm-unknown -mattr=+v4t -global-isel -stop-after=irtranslator -verify-machineinstrs %s -o - | FileCheck %s -check-prefixes=CHECK,V4T
-; RUN: llc -mtriple arm-unknown -mattr=+v5t -global-isel -stop-after=irtranslator -verify-machineinstrs %s -o - | FileCheck %s -check-prefixes=CHECK,V5T
+; RUN: llc -mtriple arm-unknown -mattr=-v4t -global-isel -stop-after=irtranslator       -verify-machineinstrs %s -o - | FileCheck %s -check-prefixes=COMMON,LOWERED,NOV4T,LOWERED-NOV4T
+; RUN: llc -mtriple arm-unknown -mattr=+v4t -global-isel -stop-after=irtranslator       -verify-machineinstrs %s -o - | FileCheck %s -check-prefixes=COMMON,LOWERED,V4T,LOWERED-V4T
+; RUN: llc -mtriple arm-unknown -mattr=+v5t -global-isel -stop-after=irtranslator       -verify-machineinstrs %s -o - | FileCheck %s -check-prefixes=COMMON,LOWERED,V5T,LOWERED-V5T
+
+; RUN: llc -mtriple arm-unknown -mattr=-v4t -global-isel -stop-after=instruction-select -verify-machineinstrs %s -o - | FileCheck %s -check-prefixes=COMMON,SELECTED,NOV4T,SELECTED-NOV4T
+; RUN: llc -mtriple arm-unknown -mattr=+v4t -global-isel -stop-after=instruction-select -verify-machineinstrs %s -o - | FileCheck %s -check-prefixes=COMMON,SELECTED,V4T,SELECTED-V4T
+; RUN: llc -mtriple arm-unknown -mattr=+v5t -global-isel -stop-after=instruction-select -verify-machineinstrs %s -o - | FileCheck %s -check-prefixes=COMMON,SELECTED,V5T,SELECTED-V5T
 
 define arm_aapcscc void @test_indirect_call(void() *%fptr) {
-; CHECK-LABEL: name: test_indirect_call
-; V5T: %[[FPTR:[0-9]+]]:gpr(p0) = COPY $r0
-; V4T: %[[FPTR:[0-9]+]]:tgpr(p0) = COPY $r0
-; NOV4T: %[[FPTR:[0-9]+]]:tgpr(p0) = COPY $r0
-; CHECK: ADJCALLSTACKDOWN 0, 0, 14, $noreg, implicit-def $sp, implicit $sp
-; V5T: BLX %[[FPTR]](p0), csr_aapcs, implicit-def $lr, implicit $sp
-; V4T: BX_CALL %[[FPTR]](p0), csr_aapcs, implicit-def $lr, implicit $sp
-; NOV4T: BMOVPCRX_CALL %[[FPTR]](p0), csr_aapcs, implicit-def $lr, implicit $sp
-; CHECK: ADJCALLSTACKUP 0, 0, 14, $noreg, implicit-def $sp, implicit $sp
+; COMMON-LABEL: name: test_indirect_call
+; LOWERED:        %[[FPTR_TY:[0-9]+]]:_(p0) = COPY $r0
+; SELECTED-V5T:   %[[FPTR:[0-9]+]]:gpr = COPY $r0
+; SELECTED-V4T:   %[[FPTR_RC:[0-9]+]]:gpr = COPY $r0
+; SELECTED-NOV4T: %[[FPTR_RC:[0-9]+]]:gpr = COPY $r0
+; COMMON:         ADJCALLSTACKDOWN 0, 0, 14, $noreg, implicit-def $sp, implicit $sp
+; LOWERED-V5T:    %[[FPTR:[0-9]+]]:gpr = COPY %[[FPTR_TY]](p0)
+; LOWERED-V4T:    %[[FPTR:[0-9]+]]:tgpr = COPY %[[FPTR_TY]](p0)
+; LOWERED-NOV4T:  %[[FPTR:[0-9]+]]:tgpr = COPY %[[FPTR_TY]](p0)
+; SELECTED-V4T:   %[[FPTR:[0-9]+]]:tgpr = COPY %[[FPTR_RC]]
+; SELECTED-NOV4T: %[[FPTR:[0-9]+]]:tgpr = COPY %[[FPTR_RC]]
+; V5T:            BLX %[[FPTR]], csr_aapcs, implicit-def $lr, implicit $sp
+; V4T:            BX_CALL %[[FPTR]], csr_aapcs, implicit-def $lr, implicit $sp
+; NOV4T:          BMOVPCRX_CALL %[[FPTR]], csr_aapcs, implicit-def $lr, implicit $sp
+; COMMON:         ADJCALLSTACKUP 0, 0, 14, $noreg, implicit-def $sp, implicit $sp
 entry:
   notail call arm_aapcscc void %fptr()
   ret void
 }
 
 declare arm_aapcscc void @call_target()
 
 define arm_aapcscc void @test_direct_call() {
-; CHECK-LABEL: name: test_direct_call
-; CHECK: ADJCALLSTACKDOWN 0, 0, 14, $noreg, implicit-def $sp, implicit $sp
-; CHECK: BL @call_target, csr_aapcs, implicit-def $lr, implicit $sp
-; CHECK: ADJCALLSTACKUP 0, 0, 14, $noreg, implicit-def $sp, implicit $sp
+; COMMON-LABEL: name: test_direct_call
+; COMMON: ADJCALLSTACKDOWN 0, 0, 14, $noreg, implicit-def $sp, implicit $sp
+; COMMON: BL @call_target, csr_aapcs, implicit-def $lr, implicit $sp
+; COMMON: ADJCALLSTACKUP 0, 0, 14, $noreg, implicit-def $sp, implicit $sp
 entry:
   notail call arm_aapcscc void @call_target()
   ret void
 }

test/CodeGen/X86/GlobalISel/irtranslator-callingconv.ll

 ; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
-; RUN: llc -mtriple=i386-linux-gnu   -mattr=+sse2 -global-isel -stop-after=irtranslator < %s -o - | FileCheck %s --check-prefix=ALL --check-prefix=X32
-; RUN: llc -mtriple=x86_64-linux-gnu              -global-isel -stop-after=irtranslator < %s -o - | FileCheck %s --check-prefix=ALL --check-prefix=X64
+; RUN: llc -mtriple=i386-linux-gnu   -mattr=+sse2 -global-isel -stop-after=irtranslator -verify-machineinstrs < %s -o - | FileCheck %s --check-prefix=ALL --check-prefix=X32
+; RUN: llc -mtriple=x86_64-linux-gnu              -global-isel -stop-after=irtranslator -verify-machineinstrs < %s -o - | FileCheck %s --check-prefix=ALL --check-prefix=X64
 
 @a1_8bit = external global i8
 @a7_8bit = external global i8
 @a8_8bit = external global i8
 
 define i8 @test_i8_args_8(i8 %arg1, i8 %arg2, i8 %arg3, i8 %arg4,
   ; X32-LABEL: name: test_i8_args_8
   ; X32: bb.1.entry:
define <8 x i32> @test_split_return_callee(<8 x i32> %arg1, <8 x i32> %arg2) {
   %r = add <8 x i32> %arg1, %call
   ret  <8 x i32> %r
 }
 
 define void @test_indirect_call(void()* %func) {
   ; X32-LABEL: name: test_indirect_call
   ; X32: bb.1 (%ir-block.0):
   ; X32:   [[FRAME_INDEX:%[0-9]+]]:_(p0) = G_FRAME_INDEX %fixed-stack.0
-  ; X32:   [[LOAD:%[0-9]+]]:gr32(p0) = G_LOAD [[FRAME_INDEX]](p0) :: (invariant load 4 from %fixed-stack.0, align 0)
+  ; X32:   [[LOAD_TY:%[0-9]+]]:_(p0) = G_LOAD [[FRAME_INDEX]](p0) :: (invariant load 4 from %fixed-stack.0, align 0)
   ; X32:   ADJCALLSTACKDOWN32 0, 0, 0, implicit-def $esp, implicit-def $eflags, implicit-def $ssp, implicit $esp, implicit $ssp
-  ; X32:   CALL32r [[LOAD]](p0), csr_32, implicit $esp, implicit $ssp
+  ; X32:   [[LOAD_RC:%[0-9]+]]:gr32 = COPY [[LOAD_TY]]
+  ; X32:   CALL32r [[LOAD_RC]], csr_32, implicit $esp, implicit $ssp
   ; X32:   ADJCALLSTACKUP32 0, 0, implicit-def $esp, implicit-def $eflags, implicit-def $ssp, implicit $esp, implicit $ssp
   ; X32:   RET 0
   ; X64-LABEL: name: test_indirect_call
   ; X64: bb.1 (%ir-block.0):
   ; X64:   liveins: $rdi
-  ; X64:   [[COPY:%[0-9]+]]:gr64(p0) = COPY $rdi
+  ; X64:   [[COPY_TY:%[0-9]+]]:_(p0) = COPY $rdi
   ; X64:   ADJCALLSTACKDOWN64 0, 0, 0, implicit-def $rsp, implicit-def $eflags, implicit-def $ssp, implicit $rsp, implicit $ssp
-  ; X64:   CALL64r [[COPY]](p0), csr_64, implicit $rsp, implicit $ssp
+  ; X64:   [[COPY_RC:%[0-9]+]]:gr64 = COPY [[COPY_TY]]
+  ; X64:   CALL64r [[COPY_RC]], csr_64, implicit $rsp, implicit $ssp
   ; X64:   ADJCALLSTACKUP64 0, 0, implicit-def $rsp, implicit-def $eflags, implicit-def $ssp, implicit $rsp, implicit $ssp
   ; X64:   RET 0
   call void %func()
   ret void
 }
 
 
 declare void @take_char(i8)