CodeGen, AArch64, ARM, X86: Simplify SplitCSR

[MatzeB]

Summary

Simplify SplitCSR code:

• Move generic legality checking code from SelectionDAG::runOnMachineFunction() to TargetLoweringBase::mayUseSplitCSR().
• Avoid TargetLoweringInfo::supportSplitCSR() and initializeSplitCSR() callbacks simply by performing the lowering as part of TargetLowering::LowerFormals()
• Avoid TargetLoweringInfo::insertCopiesSplitCSR() callback by using MachingFunction::addLiveIn() to get copies to vregs at the entry block and by creating CopyFromReg/CopyToReg nodes when lowering the return in LowerReturn. Provide a convenience function addCalleeSaveRegOps() for this.

This will be used to handle parameters passed in callee saves with the SplitCSR logic.

[manmanren]

Some general comments:
Is this NFC (no functionality change)? I noticed a small testing case change.

I think the original code tries to make it easy for a target to start supporting splitCSR, or for a calling convention to start using splitCSR.
With this updated approach, is it still easy to do so? I didn't really look through the patch to figure out how :]

Cheers,
Manman

include/llvm/Target/TargetLowering.h

2670

typo: kepy

lib/Target/AArch64/AArch64ISelLowering.cpp

2678

I don't quite like this getting duplicated in every target that supports SplitCSR and in the already heavy LowerFormalArguments. Can we wrap this up somehow?

If we want to support another Calling convention, we will have more conditions here.

[MatzeB]

In D19005#398802, @manmanren wrote:

Some general comments:
Is this NFC (no functionality change)? I noticed a small testing case change.

It changes the order in which the physregs are copied to vregs in the entry block. We are unlucky in the testcase and the register allocator chooses registers in a way that fewer stmia instructions are formed, the code gets slightly bigger and we need a wide jump.
I can make this true NFC but that will mean changing the code to add some registers in reverse order without any good explanation of why...

I think the original code tries to make it easy for a target to start supporting splitCSR, or for a calling convention to start using splitCSR.
With this updated approach, is it still easy to do so? I didn't really look through the patch to figure out how :]

The main motivation of the change is that the decision on whether to use SplitCSR had to be done before any other SelectionDAG code, this meant that there was no CCState object available and you cannot create one because ISD::InputArgs is not available. So you had to make that decisions purely on the IR function. The commit depending on this wants to use SplitCSR for all cases when a parameter is passed in a callee save register. Without CCState available you can only match on calling convention IDs and you would need to hardcode knowledge about how registers will be chosen later based on that, doing it later you can just take the information that is already computed.

To use SplitCSR with this change you basically: Decide whether you need it in LowerFormals() and if so call addLiveIn() for every register handled this way, you also alter LowerReturn to copy the register back. I found this easier to understand as that closely follows the logic that you copy the register to a vreg in the entry block and copy it back in the return block. LowerFormals() and LowerReturn() should already be familiar to a person implementing the target. The problem I had with a series of target callback is that to understand the program flow you need to read the generic SelectionDAG code to find out when those callbacks are called. The fact that the new code is 169 lines less is also
an indication that it is simpler.

[MatzeB]

ping

[manmanren]

In D19005#398881, @MatzeB wrote:

In D19005#398802, @manmanren wrote:

Some general comments:

It changes the order in which the physregs are copied to vregs in the entry block.

Can you explain a little more on how the order is changed?

We are unlucky in the testcase and the register allocator chooses registers in a way that fewer stmia instructions are formed, the code gets slightly bigger and we need a wide jump.
I can make this true NFC but that will mean changing the code to add some registers in reverse order without any good explanation of why...

There is no need to match exactly ... But I would like to understand why, since this change may break TLS.

I think the original code tries to make it easy for a target to start supporting splitCSR, or for a calling convention to start using splitCSR.
With this updated approach, is it still easy to do so? I didn't really look through the patch to figure out how :]

The main motivation of the change is that the decision on whether to use SplitCSR had to be done before any other SelectionDAG code, this meant that there was no CCState object available and you cannot create one because ISD::InputArgs is not available. So you had to make that decisions purely on the IR function. The commit depending on this wants to use SplitCSR for all cases when a parameter is passed in a callee save register.

Makes sense, I was thinking about just handling the swiftself attribute, but I guess we can generalize it to support using CSR to pass parameters. But please make it clear that using a CSR to pass parameter means that you are opting into SplitCSR and also note that it is possible for PEI to use a CSR so CSRs that are explicitly handled should not be used by PEI.

Without CCState available you can only match on calling convention IDs and you would need to hardcode knowledge about how registers will be chosen later based on that, doing it later you can just take the information that is already computed.

To use SplitCSR with this change you basically: Decide whether you need it in LowerFormals() and if so call addLiveIn() for every register handled this way, you also alter LowerReturn to copy the register back. I found this easier to understand as that closely follows the logic that you copy the register to a vreg in the entry block and copy it back in the return block. LowerFormals() and LowerReturn() should already be familiar to a person implementing the target. The problem I had with a series of target callback is that to understand the program flow you need to read the generic SelectionDAG code to find out when those callbacks are called. The fact that the new code is 169 lines less is also
an indication that it is simpler.

Fewer lines do not necessarily mean simpler logic :] There are other ways to de-duplicate the implementations of insertCopiesSplitCSR etc.

After this patch, the only thing that mentions SplitCSR is the function mayUseSplitCSR in TargetLowering. Please add more comments there about SplitCSR.

Cheers,
Manman

include/llvm/Target/TargetLowering.h

1223

This is the only place mentioning SplitCSR, please add more comments here.

2670

I found this comment hard to read. Can you add more context on when this will be called? Instead of addCalleeSaveRegOps, how about copyCSRSFromVregsToPhys or something more concrete?

include/llvm/CodeGen/FunctionLoweringInfo.h

public:
   MachineFunction *MF;
   const TargetLowering *TLI;
   MachineRegisterInfo *RegInfo;
   BranchProbabilityInfo *BPI;
   /// CanLowerReturn - true iff the function's return value can be lowered to
   /// registers.
   bool CanLowerReturn;
 
-  /// True if part of the CSRs will be handled via explicit copies.
-  bool SplitCSR;
-
   /// DemoteRegister - if CanLowerReturn is false, DemoteRegister is a vreg
   /// allocated to hold a pointer to the hidden sret parameter.
   unsigned DemoteRegister;
 
   /// MBBMap - A mapping from LLVM basic blocks to their machine code entry.
   DenseMap<const BasicBlock*, MachineBasicBlock *> MBBMap;
 
   typedef SmallVector<unsigned, 1> SwiftErrorVRegs;

include/llvm/Target/TargetLowering.h

if (hasMultipleConditionRegisters())
       return false;
     // Only do the transform if the value won't be split into multiple
     // registers.
     LegalizeTypeAction Action = getTypeAction(Context, VT);
     return Action != TypeExpandInteger && Action != TypeExpandFloat &&
       Action != TypeSplitVector;
   }
 
+  /// Returns true if the given function only ends in return (or unreachable)
+  /// instructions.
+  static bool mayUseSplitCSR(const MachineFunction &MF);
+
   //===--------------------------------------------------------------------===//
   // TargetLowering Configuration Methods - These methods should be invoked by
   // the derived class constructor to configure this object for the target.
   //
 protected:
   /// Specify how the target extends the result of integer and floating point
   /// boolean values from i1 to a wider type.  See getBooleanContents.
   void setBooleanContents(BooleanContent Ty) {
public:
   }
 
   /// Return true if the target supports swifterror attribute. It optimizes
   /// loads and stores to reading and writing a specific register.
   virtual bool supportSwiftError() const {
     return false;
   }
 
-  /// Return true if the target supports that a subset of CSRs for the given
-  /// machine function is handled explicitly via copies.
-  virtual bool supportSplitCSR(MachineFunction *MF) const {
-    return false;
-  }
-
   /// Return true if the MachineFunction contains a COPY which would imply
   /// HasCopyImplyingStackAdjustment.
   virtual bool hasCopyImplyingStackAdjustment(MachineFunction *MF) const {
     return false;
   }
 
-  /// Perform necessary initialization to handle a subset of CSRs explicitly
-  /// via copies. This function is called at the beginning of instruction
-  /// selection.
-  virtual void initializeSplitCSR(MachineBasicBlock *Entry) const {
-    llvm_unreachable("Not Implemented");
-  }
-
-  /// Insert explicit copies in entry and exit blocks. We copy a subset of
-  /// CSRs to virtual registers in the entry block, and copy them back to
-  /// physical registers in the exit blocks. This function is called at the end
-  /// of instruction selection.
-  virtual void insertCopiesSplitCSR(
-      MachineBasicBlock *Entry,
-      const SmallVectorImpl<MachineBasicBlock *> &Exits) const {
-    llvm_unreachable("Not Implemented");
-  }
-
   //===--------------------------------------------------------------------===//
   // Lowering methods - These methods must be implemented by targets so that
   // the SelectionDAGBuilder code knows how to lower these.
   //
 
   /// This hook must be implemented to lower the incoming (formal) arguments,
   /// described by the Ins array, into the specified DAG. The implementation
   /// should fill in the InVals array with legal-type argument values, and
virtual FastISel *createFastISel(FunctionLoweringInfo &,
                                    const TargetLibraryInfo *) const {
     return nullptr;
   }
 
 
   bool verifyReturnAddressArgumentIsConstant(SDValue Op,
                                              SelectionDAG &DAG) const;
 
+  /// Copy callee saved registers that are saved in vregs back to physregs.
+  /// Takes a zero-terminated list of registers that have been added with
+  /// MachineFunction::addLiveIn().  Creates a sequence of
+  /// CopyFromReg/CopyToReg nodes and appends physreg operands to the given
+  /// operand list.  Returns the glue value of the last CopyToReg.
+  /// Note: Typically used to add operands to the target return instruction when
+  /// some callee saved registers are kepy in vregs because they are also used
+  /// for function parameters.
+  SDValue addCalleeSaveRegOps(const MCPhysReg *Regs, SDLoc DL, SDValue Chain,
+                              SDValue Glue, SmallVectorImpl<SDValue> &Ops,
+                              SelectionDAG &DAG) const;
+
   //===--------------------------------------------------------------------===//
   // Inline Asm Support hooks
   //
 
   /// This hook allows the target to expand an inline asm call to be explicit
   /// llvm code if it wants to.  This is useful for turning simple inline asms
   /// into LLVM intrinsics, which gives the compiler more information about the
   /// behavior of the code.

include/llvm/Target/TargetRegisterInfo.h

public:
   /// Return a null-terminated list of all of the callee-saved registers on
   /// this target. The register should be in the order of desired callee-save
   /// stack frame offset. The first register is closest to the incoming stack
   /// pointer if stack grows down, and vice versa.
   ///
   virtual const MCPhysReg*
   getCalleeSavedRegs(const MachineFunction *MF) const = 0;
 
-  virtual const MCPhysReg*
-  getCalleeSavedRegsViaCopy(const MachineFunction *MF) const {
-    return nullptr;
-  }
-
   /// Return a mask of call-preserved registers for the given calling convention
   /// on the current function. The mask should include all call-preserved
   /// aliases. This is used by the register allocator to determine which
   /// registers can be live across a call.
   ///
   /// The mask is an array containing (TRI::getNumRegs()+31)/32 entries.
   /// A set bit indicates that all bits of the corresponding register are
   /// preserved across the function call.  The bit mask is expected to be

lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp

if (UseMBPI && OptLevel != CodeGenOpt::None)
     FuncInfo->BPI = &getAnalysis<BranchProbabilityInfoWrapperPass>().getBPI();
   else
     FuncInfo->BPI = nullptr;
 
   SDB->init(GFI, *AA, LibInfo);
 
   MF->setHasInlineAsm(false);
 
-  FuncInfo->SplitCSR = false;
-
-  // We split CSR if the target supports it for the given function
-  // and the function has only return exits.
-  if (OptLevel != CodeGenOpt::None && TLI->supportSplitCSR(MF)) {
-    FuncInfo->SplitCSR = true;
-
-    // Collect all the return blocks.
-    for (const BasicBlock &BB : Fn) {
-      if (!succ_empty(&BB))
-        continue;
-
-      const TerminatorInst *Term = BB.getTerminator();
-      if (isa<UnreachableInst>(Term) || isa<ReturnInst>(Term))
-        continue;
-
-      // Bail out if the exit block is not Return nor Unreachable.
-      FuncInfo->SplitCSR = false;
-      break;
-    }
-  }
-
-  MachineBasicBlock *EntryMBB = &MF->front();
-  if (FuncInfo->SplitCSR)
-    // This performs initialization so lowering for SplitCSR will be correct.
-    TLI->initializeSplitCSR(EntryMBB);
-
   SelectAllBasicBlocks(Fn);
 
   // If the first basic block in the function has live ins that need to be
   // copied into vregs, emit the copies into the top of the block before
   // emitting the code for the block.
   const TargetRegisterInfo &TRI = *MF->getSubtarget().getRegisterInfo();
+  MachineBasicBlock *EntryMBB = &MF->front();
   RegInfo->EmitLiveInCopies(EntryMBB, TRI, *TII);
 
-  // Insert copies in the entry block and the return blocks.
-  if (FuncInfo->SplitCSR) {
-    SmallVector<MachineBasicBlock*, 4> Returns;
-    // Collect all the return blocks.
-    for (MachineBasicBlock &MBB : mf) {
-      if (!MBB.succ_empty())
-        continue;
-
-      MachineBasicBlock::iterator Term = MBB.getFirstTerminator();
-      if (Term != MBB.end() && Term->isReturn()) {
-        Returns.push_back(&MBB);
-        continue;
-      }
-    }
-    TLI->insertCopiesSplitCSR(EntryMBB, Returns);
-  }
-
   DenseMap<unsigned, unsigned> LiveInMap;
   if (!FuncInfo->ArgDbgValues.empty())
     for (MachineRegisterInfo::livein_iterator LI = RegInfo->livein_begin(),
            E = RegInfo->livein_end(); LI != E; ++LI)
       if (LI->second)
         LiveInMap.insert(std::make_pair(LI->first, LI->second));
 
   // Insert DBG_VALUE instructions for function arguments to the entry block.

lib/CodeGen/SelectionDAG/TargetLowering.cpp

 
 #include "llvm/Target/TargetLowering.h"
 #include "llvm/ADT/BitVector.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/CodeGen/Analysis.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineJumpTableInfo.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/SelectionDAG.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCExpr.h"
 #include "llvm/Support/CommandLine.h"
if (!isa<ConstantSDNode>(Op.getOperand(0))) {
     DAG.getContext()->emitError("argument to '__builtin_return_address' must "
                                 "be a constant integer");
     return true;
   }
 
   return false;
 }
 
+SDValue TargetLowering::addCalleeSaveRegOps(const MCPhysReg *Regs, SDLoc DL,
+    SDValue Chain, SDValue Glue, SmallVectorImpl<SDValue> &Ops,
+    SelectionDAG &DAG) const {
+  const MachineFunction &MF = DAG.getMachineFunction();
+  const MachineRegisterInfo &MRI = MF.getRegInfo();
+  for (const MCPhysReg *I = Regs; *I; ++I) {
+    const MCPhysReg PReg = *I;
+    unsigned VReg = MRI.getLiveInVirtReg(PReg);
+
+    const TargetRegisterClass &RC = *MRI.getRegClass(VReg);
+    MVT RegVT = *RC.vt_begin();
+    SDValue Val = DAG.getCopyFromReg(Chain, DL, VReg, RegVT);
+    SDValue CopyToReg = DAG.getCopyToReg(Chain, DL, PReg, Val, Glue);
+    Glue = CopyToReg.getValue(1);
+
+    Ops.push_back(DAG.getRegister(PReg, RegVT));
+  }
+  return Glue;
+}
+
 //===----------------------------------------------------------------------===//
 // Legalization Utilities
 //===----------------------------------------------------------------------===//
 
 bool TargetLowering::expandMUL(SDNode *N, SDValue &Lo, SDValue &Hi, EVT HiLoVT,
                                SelectionDAG &DAG, SDValue LL, SDValue LH,
                                SDValue RL, SDValue RH) const {
   EVT VT = N->getValueType(0);

lib/CodeGen/TargetLoweringBase.cpp

 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/CodeGen/Analysis.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineJumpTableInfo.h"
 #include "llvm/CodeGen/StackMaps.h"
+#include "llvm/IR/CFG.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/Mangler.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCContext.h"
 #include "llvm/MC/MCExpr.h"
 #include "llvm/Support/CommandLine.h"
case 2:
     break;
   default: // Don't allow n * r
     return false;
   }
 
   return true;
 }
 
+bool TargetLoweringBase::mayUseSplitCSR(const MachineFunction &MF) {
+  for (const BasicBlock &BB : *MF.getFunction()) {
+    if (!succ_empty(&BB))
+      continue;
+
+    const TerminatorInst *Term = BB.getTerminator();
+    if (!isa<UnreachableInst>(Term) && !isa<ReturnInst>(Term))
+      return false;
+  }
+  return true;
+}
+
 //===----------------------------------------------------------------------===//
 //  Stack Protector
 //===----------------------------------------------------------------------===//
 
 // For OpenBSD return its special guard variable. Otherwise return nullptr,
 // so that SelectionDAG handle SSP.
 Value *TargetLoweringBase::getIRStackGuard(IRBuilder<> &IRB) const {
   if (getTargetMachine().getTargetTriple().isOSOpenBSD()) {

lib/Target/AArch64/AArch64FastISel.cpp

bool AArch64FastISel::selectRet(const Instruction *I) {
 
   if (F.isVarArg())
     return false;
 
   if (TLI.supportSwiftError() &&
       F.getAttributes().hasAttrSomewhere(Attribute::SwiftError))
     return false;
 
-  if (TLI.supportSplitCSR(FuncInfo.MF))
-    return false;
-
   // Build a list of return value registers.
   SmallVector<unsigned, 4> RetRegs;
 
   if (Ret->getNumOperands() > 0) {
     CallingConv::ID CC = F.getCallingConv();
     SmallVector<ISD::OutputArg, 4> Outs;
     GetReturnInfo(F.getReturnType(), F.getAttributes(), Outs, TLI, DL);
 

lib/Target/AArch64/AArch64ISelLowering.h

public:
 
   bool isCheapToSpeculateCttz() const override {
     return true;
   }
 
   bool isCheapToSpeculateCtlz() const override {
     return true;
   }
-  bool supportSplitCSR(MachineFunction *MF) const override {
-    return MF->getFunction()->getCallingConv() == CallingConv::CXX_FAST_TLS &&
-           MF->getFunction()->hasFnAttribute(Attribute::NoUnwind);
-  }
-  void initializeSplitCSR(MachineBasicBlock *Entry) const override;
-  void insertCopiesSplitCSR(
-      MachineBasicBlock *Entry,
-      const SmallVectorImpl<MachineBasicBlock *> &Exits) const override;
 
   bool supportSwiftError() const override {
     return true;
   }
 
 private:
   bool isExtFreeImpl(const Instruction *Ext) const override;
 

lib/Target/AArch64/AArch64ISelLowering.cpp

if (DoesCalleeRestoreStack(CallConv, TailCallOpt)) {
     // This realignment carries over to the available bytes below. Our own
     // callers will guarantee the space is free by giving an aligned value to
     // CALLSEQ_START.
   }
   // Even if we're not expected to free up the space, it's useful to know how
   // much is there while considering tail calls (because we can reuse it).
   FuncInfo->setBytesInStackArgArea(StackArgSize);
 
+  // Should we save callee save registers via vregs?
+  if (CallConv == CallingConv::CXX_FAST_TLS &&
+      MF.getFunction()->hasFnAttribute(Attribute::NoUnwind) &&
+      !MF.getTarget().Options.EnableFastISel && mayUseSplitCSR(MF)) {
+    const MCPhysReg *SavedViaCopy =
+      AArch64RegisterInfo::getFastTLSSavedViaCopy();
+    FuncInfo->setCSRSavedViaCopy(SavedViaCopy);
+
+    // Add live-ins.
+    for (const MCPhysReg *I = SavedViaCopy; *I; ++I) {
+      const MCPhysReg Reg = *I;
+      const TargetRegisterClass *RC;
+      if (AArch64::GPR64RegClass.contains(Reg))
+        RC = &AArch64::GPR64RegClass;
+      else if (AArch64::FPR64RegClass.contains(Reg))
+        RC = &AArch64::FPR64RegClass;
+      else
+        llvm_unreachable("Unexpected register class in CSRsViaCopy!");
+      MF.addLiveIn(Reg, RC);
+    }
+  }
+
   return Chain;
 }
 
 void AArch64TargetLowering::saveVarArgRegisters(CCState &CCInfo,
                                                 SelectionDAG &DAG, SDLoc DL,
                                                 SDValue &Chain) const {
   MachineFunction &MF = DAG.getMachineFunction();
   MachineFrameInfo *MFI = MF.getFrameInfo();
AArch64TargetLowering::LowerReturn(SDValue Chain, CallingConv::ID CallConv,
                                    bool isVarArg,
                                    const SmallVectorImpl<ISD::OutputArg> &Outs,
                                    const SmallVectorImpl<SDValue> &OutVals,
                                    SDLoc DL, SelectionDAG &DAG) const {
   CCAssignFn *RetCC = CallConv == CallingConv::WebKit_JS
                           ? RetCC_AArch64_WebKit_JS
                           : RetCC_AArch64_AAPCS;
   SmallVector<CCValAssign, 16> RVLocs;
-  CCState CCInfo(CallConv, isVarArg, DAG.getMachineFunction(), RVLocs,
-                 *DAG.getContext());
+  MachineFunction &MF = DAG.getMachineFunction();
+  CCState CCInfo(CallConv, isVarArg, MF, RVLocs, *DAG.getContext());
   CCInfo.AnalyzeReturn(Outs, RetCC);
 
   // Copy the result values into the output registers.
+  SDValue IncomingChain = Chain;
   SDValue Flag;
   SmallVector<SDValue, 4> RetOps(1, Chain);
   for (unsigned i = 0, realRVLocIdx = 0; i != RVLocs.size();
        ++i, ++realRVLocIdx) {
     CCValAssign &VA = RVLocs[i];
     assert(VA.isRegLoc() && "Can only return in registers!");
     SDValue Arg = OutVals[realRVLocIdx];
 
case CCValAssign::BCvt:
       Arg = DAG.getNode(ISD::BITCAST, DL, VA.getLocVT(), Arg);
       break;
     }
 
     Chain = DAG.getCopyToReg(Chain, DL, VA.getLocReg(), Arg, Flag);
     Flag = Chain.getValue(1);
     RetOps.push_back(DAG.getRegister(VA.getLocReg(), VA.getLocVT()));
   }
-  const AArch64RegisterInfo *TRI = Subtarget->getRegisterInfo();
-  const MCPhysReg *I =
-      TRI->getCalleeSavedRegsViaCopy(&DAG.getMachineFunction());
-  if (I) {
-    for (; *I; ++I) {
-      if (AArch64::GPR64RegClass.contains(*I))
-        RetOps.push_back(DAG.getRegister(*I, MVT::i64));
-      else if (AArch64::FPR64RegClass.contains(*I))
-        RetOps.push_back(DAG.getRegister(*I, MVT::getFloatingPointVT(64)));
-      else
-        llvm_unreachable("Unexpected register class in CSRsViaCopy!");
-    }
-  }
+  const AArch64FunctionInfo &FuncInfo = *MF.getInfo<AArch64FunctionInfo>();
+  const MCPhysReg *SavedViaCopy = FuncInfo.getCSRSavedViaCopy();
+  if (SavedViaCopy != nullptr)
+    Flag = addCalleeSaveRegOps(SavedViaCopy, DL, IncomingChain, Flag, RetOps,
+                               DAG);
 
   RetOps[0] = Chain; // Update chain.
 
   // Add the flag if we have it.
   if (Flag.getNode())
     RetOps.push_back(Flag);
 
   return DAG.getNode(AArch64ISD::RET_FLAG, DL, MVT::Other, RetOps);
Value *AArch64TargetLowering::getSafeStackPointerLocation(IRBuilder<> &IRB) const {
   Module *M = IRB.GetInsertBlock()->getParent()->getParent();
   Function *ThreadPointerFunc =
       Intrinsic::getDeclaration(M, Intrinsic::aarch64_thread_pointer);
   return IRB.CreatePointerCast(
       IRB.CreateConstGEP1_32(IRB.CreateCall(ThreadPointerFunc), TlsOffset),
       Type::getInt8PtrTy(IRB.getContext())->getPointerTo(0));
 }
 
-void AArch64TargetLowering::initializeSplitCSR(MachineBasicBlock *Entry) const {
-  // Update IsSplitCSR in AArch64unctionInfo.
-  AArch64FunctionInfo *AFI = Entry->getParent()->getInfo<AArch64FunctionInfo>();
-  AFI->setIsSplitCSR(true);
-}
-
-void AArch64TargetLowering::insertCopiesSplitCSR(
-    MachineBasicBlock *Entry,
-    const SmallVectorImpl<MachineBasicBlock *> &Exits) const {
-  const AArch64RegisterInfo *TRI = Subtarget->getRegisterInfo();
-  const MCPhysReg *IStart = TRI->getCalleeSavedRegsViaCopy(Entry->getParent());
-  if (!IStart)
-    return;
-
-  const TargetInstrInfo *TII = Subtarget->getInstrInfo();
-  MachineRegisterInfo *MRI = &Entry->getParent()->getRegInfo();
-  MachineBasicBlock::iterator MBBI = Entry->begin();
-  for (const MCPhysReg *I = IStart; *I; ++I) {
-    const TargetRegisterClass *RC = nullptr;
-    if (AArch64::GPR64RegClass.contains(*I))
-      RC = &AArch64::GPR64RegClass;
-    else if (AArch64::FPR64RegClass.contains(*I))
-      RC = &AArch64::FPR64RegClass;
-    else
-      llvm_unreachable("Unexpected register class in CSRsViaCopy!");
-
-    unsigned NewVR = MRI->createVirtualRegister(RC);
-    // Create copy from CSR to a virtual register.
-    // FIXME: this currently does not emit CFI pseudo-instructions, it works
-    // fine for CXX_FAST_TLS since the C++-style TLS access functions should be
-    // nounwind. If we want to generalize this later, we may need to emit
-    // CFI pseudo-instructions.
-    assert(Entry->getParent()->getFunction()->hasFnAttribute(
-               Attribute::NoUnwind) &&
-           "Function should be nounwind in insertCopiesSplitCSR!");
-    Entry->addLiveIn(*I);
-    BuildMI(*Entry, MBBI, DebugLoc(), TII->get(TargetOpcode::COPY), NewVR)
-        .addReg(*I);
-
-    // Insert the copy-back instructions right before the terminator.
-    for (auto *Exit : Exits)
-      BuildMI(*Exit, Exit->getFirstTerminator(), DebugLoc(),
-              TII->get(TargetOpcode::COPY), *I)
-          .addReg(NewVR);
-  }
-}
-
 bool AArch64TargetLowering::isIntDivCheap(EVT VT, AttributeSet Attr) const {
   // Integer division on AArch64 is expensive. However, when aggressively
   // optimizing for code size, we prefer to use a div instruction, as it is
   // usually smaller than the alternative sequence.
   // The exception to this is vector division. Since AArch64 doesn't have vector
   // integer division, leaving the division as-is is a loss even in terms of
   // size, because it will have to be scalarized, while the alternative code
   // sequence can be performed in vector form.
   bool OptSize =
       Attr.hasAttribute(AttributeSet::FunctionIndex, Attribute::MinSize);
   return OptSize && !VT.isVector();
 }

lib/Target/AArch64/AArch64MachineFunctionInfo.h

class AArch64FunctionInfo : public MachineFunctionInfo {
   /// \brief FrameIndex for start of varargs area for arguments passed in
   /// floating-point registers.
   int VarArgsFPRIndex;
 
   /// \brief Size of the varargs area for arguments passed in floating-point
   /// registers.
   unsigned VarArgsFPRSize;
 
-  /// True if this function has a subset of CSRs that is handled explicitly via
-  /// copies.
-  bool IsSplitCSR;
+  /// zero-terminated list of callee saved registers saved by copying to a vreg.
+  const MCPhysReg *CSRSavedViaCopy = nullptr;
 
   /// True when the stack gets realigned dynamically because the size of stack
   /// frame is unknown at compile time. e.g., in case of VLAs.
   bool StackRealigned;
 
 public:
   AArch64FunctionInfo()
       : BytesInStackArgArea(0), ArgumentStackToRestore(0), HasStackFrame(false),
         NumLocalDynamicTLSAccesses(0), VarArgsStackIndex(0), VarArgsGPRIndex(0),
         VarArgsGPRSize(0), VarArgsFPRIndex(0), VarArgsFPRSize(0),
-        IsSplitCSR(false), StackRealigned(false) {}
+        StackRealigned(false) {}
 
   explicit AArch64FunctionInfo(MachineFunction &MF)
       : BytesInStackArgArea(0), ArgumentStackToRestore(0), HasStackFrame(false),
         NumLocalDynamicTLSAccesses(0), VarArgsStackIndex(0), VarArgsGPRIndex(0),
         VarArgsGPRSize(0), VarArgsFPRIndex(0), VarArgsFPRSize(0),
-        IsSplitCSR(false), StackRealigned(false) {
+        StackRealigned(false) {
     (void)MF;
   }
 
   unsigned getBytesInStackArgArea() const { return BytesInStackArgArea; }
   void setBytesInStackArgArea(unsigned bytes) { BytesInStackArgArea = bytes; }
 
   unsigned getArgumentStackToRestore() const { return ArgumentStackToRestore; }
   void setArgumentStackToRestore(unsigned bytes) {
     ArgumentStackToRestore = bytes;
   }
 
   bool hasStackFrame() const { return HasStackFrame; }
   void setHasStackFrame(bool s) { HasStackFrame = s; }
 
   bool isStackRealigned() const { return StackRealigned; }
   void setStackRealigned(bool s) { StackRealigned = s; }
 
-  bool isSplitCSR() const { return IsSplitCSR; }
-  void setIsSplitCSR(bool s) { IsSplitCSR = s; }
+  const MCPhysReg *getCSRSavedViaCopy() const { return CSRSavedViaCopy; }
+  void setCSRSavedViaCopy(const MCPhysReg *CSRSavedViaCopy) {
+    this->CSRSavedViaCopy = CSRSavedViaCopy;
+  }
 
   void setLocalStackSize(unsigned Size) { LocalStackSize = Size; }
   unsigned getLocalStackSize() const { return LocalStackSize; }
 
   void setCalleeSavedStackSize(unsigned Size) { CalleeSavedStackSize = Size; }
   unsigned getCalleeSavedStackSize() const { return CalleeSavedStackSize; }
 
   void incNumLocalDynamicTLSAccesses() { ++NumLocalDynamicTLSAccesses; }

lib/Target/AArch64/AArch64RegisterInfo.h

 
 public:
   AArch64RegisterInfo(const Triple &TT);
 
   bool isReservedReg(const MachineFunction &MF, unsigned Reg) const;
 
   /// Code Generation virtual methods...
   const MCPhysReg *getCalleeSavedRegs(const MachineFunction *MF) const override;
-  const MCPhysReg *
-  getCalleeSavedRegsViaCopy(const MachineFunction *MF) const override;
   const uint32_t *getCallPreservedMask(const MachineFunction &MF,
                                        CallingConv::ID) const override;
 
   unsigned getCSRFirstUseCost() const override {
     // The cost will be compared against BlockFrequency where entry has the
     // value of 1 << 14. A value of 5 will choose to spill or split really
     // cold path instead of using a callee-saved register.
     return 5;
public:
   bool hasBasePointer(const MachineFunction &MF) const;
   unsigned getBaseRegister() const;
 
   // Debug information queries.
   unsigned getFrameRegister(const MachineFunction &MF) const override;
 
   unsigned getRegPressureLimit(const TargetRegisterClass *RC,
                                MachineFunction &MF) const override;
+
+  /// Returns CSR_AArch64_CXX_TLS_Darwin_ViaCopy_SaveList.
+  static const MCPhysReg *getFastTLSSavedViaCopy();
 };
 
 } // end namespace llvm
 
 #endif

lib/Target/AArch64/AArch64RegisterInfo.cpp

AArch64RegisterInfo::getCalleeSavedRegs(const MachineFunction *MF) const {
   assert(MF && "Invalid MachineFunction pointer.");
   if (MF->getFunction()->getCallingConv() == CallingConv::GHC)
     // GHC set of callee saved regs is empty as all those regs are
     // used for passing STG regs around
     return CSR_AArch64_NoRegs_SaveList;
   if (MF->getFunction()->getCallingConv() == CallingConv::AnyReg)
     return CSR_AArch64_AllRegs_SaveList;
   if (MF->getFunction()->getCallingConv() == CallingConv::CXX_FAST_TLS)
-    return MF->getInfo<AArch64FunctionInfo>()->isSplitCSR() ?
+    return MF->getInfo<AArch64FunctionInfo>()->getCSRSavedViaCopy() ?
            CSR_AArch64_CXX_TLS_Darwin_PE_SaveList :
            CSR_AArch64_CXX_TLS_Darwin_SaveList;
   if (MF->getSubtarget<AArch64Subtarget>().getTargetLowering()
           ->supportSwiftError() &&
       MF->getFunction()->getAttributes().hasAttrSomewhere(
           Attribute::SwiftError))
     return CSR_AArch64_AAPCS_SwiftError_SaveList;
   if (MF->getFunction()->getCallingConv() == CallingConv::PreserveMost)
     return CSR_AArch64_RT_MostRegs_SaveList;
   else
     return CSR_AArch64_AAPCS_SaveList;
 }
 
-const MCPhysReg *AArch64RegisterInfo::getCalleeSavedRegsViaCopy(
-    const MachineFunction *MF) const {
-  assert(MF && "Invalid MachineFunction pointer.");
-  if (MF->getFunction()->getCallingConv() == CallingConv::CXX_FAST_TLS &&
-      MF->getInfo<AArch64FunctionInfo>()->isSplitCSR())
-    return CSR_AArch64_CXX_TLS_Darwin_ViaCopy_SaveList;
-  return nullptr;
+const MCPhysReg *AArch64RegisterInfo::getFastTLSSavedViaCopy() {
 }
 
 const uint32_t *
 AArch64RegisterInfo::getCallPreservedMask(const MachineFunction &MF,
                                           CallingConv::ID CC) const {
   if (CC == CallingConv::GHC)
     // This is academic becase all GHC calls are (supposed to be) tail calls
     return CSR_AArch64_NoRegs_RegMask;

lib/Target/ARM/ARMBaseRegisterInfo.h

protected:
   explicit ARMBaseRegisterInfo();
 
   // Return the opcode that implements 'Op', or 0 if no opcode
   unsigned getOpcode(int Op) const;
 
 public:
   /// Code Generation virtual methods...
   const MCPhysReg *getCalleeSavedRegs(const MachineFunction *MF) const override;
-  const MCPhysReg *
-  getCalleeSavedRegsViaCopy(const MachineFunction *MF) const override;
   const uint32_t *getCallPreservedMask(const MachineFunction &MF,
                                        CallingConv::ID) const override;
   const uint32_t *getNoPreservedMask() const override;
   const uint32_t *getTLSCallPreservedMask(const MachineFunction &MF) const;
 
   /// getThisReturnPreservedMask - Returns a call preserved mask specific to the
   /// case that 'returned' is on an i32 first argument if the calling convention
   /// is one that can (partially) model this attribute with a preserved mask
public:
 
   /// \brief SrcRC and DstRC will be morphed into NewRC if this returns true
   bool shouldCoalesce(MachineInstr *MI,
                       const TargetRegisterClass *SrcRC,
                       unsigned SubReg,
                       const TargetRegisterClass *DstRC,
                       unsigned DstSubReg,
                       const TargetRegisterClass *NewRC) const override;
+
+  /// Returns CSR_iOS_CXX_TLS_ViaCopy_SaveList.
+  static const MCPhysReg *getFastTLSSavedViaCopy();
 };
 
 } // end namespace llvm
 
 #endif

lib/Target/ARM/ARMBaseRegisterInfo.cpp

if (F->getCallingConv() == CallingConv::GHC) {
     }
   }
 
   if (STI.isTargetDarwin() && STI.getTargetLowering()->supportSwiftError() &&
       F->getAttributes().hasAttrSomewhere(Attribute::SwiftError))
     return CSR_iOS_SwiftError_SaveList;
 
   if (STI.isTargetDarwin() && F->getCallingConv() == CallingConv::CXX_FAST_TLS)
-    return MF->getInfo<ARMFunctionInfo>()->isSplitCSR()
+    return MF->getInfo<ARMFunctionInfo>()->getCSRSavedViaCopy()
                ? CSR_iOS_CXX_TLS_PE_SaveList
                : CSR_iOS_CXX_TLS_SaveList;
   return RegList;
 }
 
-const MCPhysReg *ARMBaseRegisterInfo::getCalleeSavedRegsViaCopy(
-    const MachineFunction *MF) const {
-  assert(MF && "Invalid MachineFunction pointer.");
-  if (MF->getFunction()->getCallingConv() == CallingConv::CXX_FAST_TLS &&
-      MF->getInfo<ARMFunctionInfo>()->isSplitCSR())
-    return CSR_iOS_CXX_TLS_ViaCopy_SaveList;
-  return nullptr;
-}
-
 const uint32_t *
 ARMBaseRegisterInfo::getCallPreservedMask(const MachineFunction &MF,
                                           CallingConv::ID CC) const {
   const ARMSubtarget &STI = MF.getSubtarget<ARMSubtarget>();
   if (CC == CallingConv::GHC)
     // This is academic becase all GHC calls are (supposed to be) tail calls
     return CSR_NoRegs_RegMask;
 
bool ARMBaseRegisterInfo::shouldCoalesce(MachineInstr *MI,
   unsigned SizeMultiplier = MBB->size()/100;
   SizeMultiplier = SizeMultiplier ? SizeMultiplier : 1;
   if (It->second < NewRCWeight.WeightLimit * SizeMultiplier) {
     It->second += NewRCWeight.RegWeight;
     return true;
   }
   return false;
 }
+
+const MCPhysReg *ARMBaseRegisterInfo::getFastTLSSavedViaCopy() {
+  return CSR_iOS_CXX_TLS_ViaCopy_SaveList;
+}

lib/Target/ARM/ARMFastISel.cpp

bool ARMFastISel::SelectRet(const Instruction *I) {
 
   if (!FuncInfo.CanLowerReturn)
     return false;
 
   if (TLI.supportSwiftError() &&
       F.getAttributes().hasAttrSomewhere(Attribute::SwiftError))
     return false;
 
-  if (TLI.supportSplitCSR(FuncInfo.MF))
-    return false;
-
   // Build a list of return value registers.
   SmallVector<unsigned, 4> RetRegs;
 
   CallingConv::ID CC = F.getCallingConv();
   if (Ret->getNumOperands() > 0) {
     SmallVector<ISD::OutputArg, 4> Outs;
     GetReturnInfo(F.getReturnType(), F.getAttributes(), Outs, TLI, DL);
 

lib/Target/ARM/ARMISelLowering.h

private:
 
     SDValue LowerCallResult(SDValue Chain, SDValue InFlag,
                             CallingConv::ID CallConv, bool isVarArg,
                             const SmallVectorImpl<ISD::InputArg> &Ins,
                             SDLoc dl, SelectionDAG &DAG,
                             SmallVectorImpl<SDValue> &InVals,
                             bool isThisReturn, SDValue ThisVal) const;
 
-    bool supportSplitCSR(MachineFunction *MF) const override {
-      return MF->getFunction()->getCallingConv() == CallingConv::CXX_FAST_TLS &&
-          MF->getFunction()->hasFnAttribute(Attribute::NoUnwind);
-    }
-    void initializeSplitCSR(MachineBasicBlock *Entry) const override;
-    void insertCopiesSplitCSR(
-      MachineBasicBlock *Entry,
-      const SmallVectorImpl<MachineBasicBlock *> &Exits) const override;
-
     SDValue
       LowerFormalArguments(SDValue Chain,
                            CallingConv::ID CallConv, bool isVarArg,
                            const SmallVectorImpl<ISD::InputArg> &Ins,
                            SDLoc dl, SelectionDAG &DAG,
                            SmallVectorImpl<SDValue> &InVals) const override;
 
     int StoreByValRegs(CCState &CCInfo, SelectionDAG &DAG,

lib/Target/ARM/ARMISelLowering.cpp

ARMTargetLowering::LowerReturn(SDValue Chain,
   // CCState - Info about the registers and stack slots.
   ARMCCState CCInfo(CallConv, isVarArg, DAG.getMachineFunction(), RVLocs,
                     *DAG.getContext(), Call);
 
   // Analyze outgoing return values.
   CCInfo.AnalyzeReturn(Outs, CCAssignFnForNode(CallConv, /* Return */ true,
                                                isVarArg));
 
+  SDValue IncomingChain = Chain;
   SDValue Flag;
   SmallVector<SDValue, 4> RetOps;
   RetOps.push_back(Chain); // Operand #0 = Chain (updated below)
   bool isLittleEndian = Subtarget->isLittle();
 
   MachineFunction &MF = DAG.getMachineFunction();
   ARMFunctionInfo *AFI = MF.getInfo<ARMFunctionInfo>();
   AFI->setReturnRegsCount(RVLocs.size());
for (unsigned i = 0, realRVLocIdx = 0;
     } else
       Chain = DAG.getCopyToReg(Chain, dl, VA.getLocReg(), Arg, Flag);
 
     // Guarantee that all emitted copies are
     // stuck together, avoiding something bad.
     Flag = Chain.getValue(1);
     RetOps.push_back(DAG.getRegister(VA.getLocReg(), VA.getLocVT()));
   }
-  const ARMBaseRegisterInfo *TRI = Subtarget->getRegisterInfo();
-  const MCPhysReg *I =
-      TRI->getCalleeSavedRegsViaCopy(&DAG.getMachineFunction());
-  if (I) {
-    for (; *I; ++I) {
-      if (ARM::GPRRegClass.contains(*I))
-        RetOps.push_back(DAG.getRegister(*I, MVT::i32));
-      else if (ARM::DPRRegClass.contains(*I))
-        RetOps.push_back(DAG.getRegister(*I, MVT::getFloatingPointVT(64)));
-      else
-        llvm_unreachable("Unexpected register class in CSRsViaCopy!");
-    }
-  }
+  const MCPhysReg *SavedViaCopy = AFI->getCSRSavedViaCopy();
+  if (SavedViaCopy != nullptr)
+    Flag = addCalleeSaveRegOps(SavedViaCopy, dl, IncomingChain, Flag, RetOps,
+                               DAG);
 
   // Update chain and glue.
   RetOps[0] = Chain;
   if (Flag.getNode())
     RetOps.push_back(Flag);
 
   // CPUs which aren't M-class use a special sequence to return from
   // exceptions (roughly, any instruction setting pc and cpsr simultaneously,
ARMTargetLowering::LowerFormalArguments(SDValue Chain,
   // varargs
   if (isVarArg && MFI->hasVAStart())
     VarArgStyleRegisters(CCInfo, DAG, dl, Chain,
                          CCInfo.getNextStackOffset(),
                          TotalArgRegsSaveSize);
 
   AFI->setArgumentStackSize(CCInfo.getNextStackOffset());
 
+  // Should we save callee save registers via vregs?
+  if (CallConv == CallingConv::CXX_FAST_TLS &&
+      MF.getFunction()->hasFnAttribute(Attribute::NoUnwind) &&
+      !MF.getTarget().Options.EnableFastISel && mayUseSplitCSR(MF)) {
+    const MCPhysReg *SavedViaCopy =
+      ARMBaseRegisterInfo::getFastTLSSavedViaCopy();
+    AFI->setCSRSavedViaCopy(SavedViaCopy);
+
+    for (const MCPhysReg *I = SavedViaCopy; *I; ++I) {
+      const MCPhysReg Reg = *I;
+      const TargetRegisterClass *RC;
+      if (ARM::GPRRegClass.contains(Reg))
+        RC = &ARM::GPRRegClass;
+      else if (ARM::DPRRegClass.contains(Reg))
+        RC = &ARM::DPRRegClass;
+      else
+        llvm_unreachable("Unexpected register class in CSRsViaCopy!");
+      MF.addLiveIn(Reg, RC);
+    }
+  }
+
   return Chain;
 }
 
 /// isFloatingPointZero - Return true if this is +0.0.
 static bool isFloatingPointZero(SDValue Op) {
   if (ConstantFPSDNode *CFP = dyn_cast<ConstantFPSDNode>(Op))
     return CFP->getValueAPF().isPosZero();
   else if (ISD::isEXTLoad(Op.getNode()) || ISD::isNON_EXTLoad(Op.getNode())) {

 }
 
 unsigned ARMTargetLowering::getExceptionSelectorRegister(
     const Constant *PersonalityFn) const {
   // Platforms which do not use SjLj EH may return values in these registers
   // via the personality function.
   return Subtarget->useSjLjEH() ? ARM::NoRegister : ARM::R1;
 }
-
-void ARMTargetLowering::initializeSplitCSR(MachineBasicBlock *Entry) const {
-  // Update IsSplitCSR in ARMFunctionInfo.
-  ARMFunctionInfo *AFI = Entry->getParent()->getInfo<ARMFunctionInfo>();
-  AFI->setIsSplitCSR(true);
-}
-
-void ARMTargetLowering::insertCopiesSplitCSR(
-    MachineBasicBlock *Entry,
-    const SmallVectorImpl<MachineBasicBlock *> &Exits) const {
-  const ARMBaseRegisterInfo *TRI = Subtarget->getRegisterInfo();
-  const MCPhysReg *IStart = TRI->getCalleeSavedRegsViaCopy(Entry->getParent());
-  if (!IStart)
-    return;
-
-  const TargetInstrInfo *TII = Subtarget->getInstrInfo();
-  MachineRegisterInfo *MRI = &Entry->getParent()->getRegInfo();
-  MachineBasicBlock::iterator MBBI = Entry->begin();
-  for (const MCPhysReg *I = IStart; *I; ++I) {
-    const TargetRegisterClass *RC = nullptr;
-    if (ARM::GPRRegClass.contains(*I))
-      RC = &ARM::GPRRegClass;
-    else if (ARM::DPRRegClass.contains(*I))
-      RC = &ARM::DPRRegClass;
-    else
-      llvm_unreachable("Unexpected register class in CSRsViaCopy!");
-
-    unsigned NewVR = MRI->createVirtualRegister(RC);
-    // Create copy from CSR to a virtual register.
-    // FIXME: this currently does not emit CFI pseudo-instructions, it works
-    // fine for CXX_FAST_TLS since the C++-style TLS access functions should be
-    // nounwind. If we want to generalize this later, we may need to emit
-    // CFI pseudo-instructions.
-    assert(Entry->getParent()->getFunction()->hasFnAttribute(
-               Attribute::NoUnwind) &&
-           "Function should be nounwind in insertCopiesSplitCSR!");
-    Entry->addLiveIn(*I);
-    BuildMI(*Entry, MBBI, DebugLoc(), TII->get(TargetOpcode::COPY), NewVR)
-        .addReg(*I);
-
-    // Insert the copy-back instructions right before the terminator.
-    for (auto *Exit : Exits)
-      BuildMI(*Exit, Exit->getFirstTerminator(), DebugLoc(),
-              TII->get(TargetOpcode::COPY), *I)
-          .addReg(NewVR);
-  }
-}

lib/Target/ARM/ARMMachineFunctionInfo.h

class ARMFunctionInfo : public MachineFunctionInfo {
   /// ArgumentStackSize - amount of bytes on stack consumed by the arguments
   /// being passed on the stack
   unsigned ArgumentStackSize;
 
   /// CoalescedWeights - mapping of basic blocks to the rolling counter of
   /// coalesced weights.
   DenseMap<const MachineBasicBlock*, unsigned> CoalescedWeights;
 
-  /// True if this function has a subset of CSRs that is handled explicitly via
-  /// copies.
-  bool IsSplitCSR;
+  /// zero-terminated list of callee saved registers saved by copying to a vreg.
+  const MCPhysReg *CSRSavedViaCopy = nullptr;
 
 public:
   ARMFunctionInfo() :
     isThumb(false),
     hasThumb2(false),
     ArgRegsSaveSize(0), ReturnRegsCount(0), HasStackFrame(false),
     RestoreSPFromFP(false),
     LRSpilledForFarJump(false),
     FramePtrSpillOffset(0), GPRCS1Offset(0), GPRCS2Offset(0), DPRCSOffset(0),
     GPRCS1Size(0), GPRCS2Size(0), DPRCSAlignGapSize(0), DPRCSSize(0),
     NumAlignedDPRCS2Regs(0), PICLabelUId(0),
-    VarArgsFrameIndex(0), HasITBlocks(false), IsSplitCSR(false) {}
+    VarArgsFrameIndex(0), HasITBlocks(false) {}
 
   explicit ARMFunctionInfo(MachineFunction &MF);
 
   bool isThumbFunction() const { return isThumb; }
   bool isThumb1OnlyFunction() const { return isThumb && !hasThumb2; }
   bool isThumb2Function() const { return isThumb && hasThumb2; }
 
   unsigned getStoredByValParamsPadding() const { return StByValParamsPadding; }
public:
   }
 
   int getVarArgsFrameIndex() const { return VarArgsFrameIndex; }
   void setVarArgsFrameIndex(int Index) { VarArgsFrameIndex = Index; }
 
   bool hasITBlocks() const { return HasITBlocks; }
   void setHasITBlocks(bool h) { HasITBlocks = h; }
 
-  bool isSplitCSR() const { return IsSplitCSR; }
-  void setIsSplitCSR(bool s) { IsSplitCSR = s; }
+  const MCPhysReg *getCSRSavedViaCopy() const { return CSRSavedViaCopy; }
+  void setCSRSavedViaCopy(const MCPhysReg *CSRSavedViaCopy) {
+    this->CSRSavedViaCopy = CSRSavedViaCopy;
+  }
 
   void recordCPEClone(unsigned CPIdx, unsigned CPCloneIdx) {
     if (!CPEClones.insert(std::make_pair(CPCloneIdx, CPIdx)).second)
       llvm_unreachable("Duplicate entries!");
   }
 
   unsigned getOriginalCPIdx(unsigned CloneIdx) const {
     DenseMap<unsigned, unsigned>::const_iterator I = CPEClones.find(CloneIdx);

lib/Target/ARM/ARMMachineFunctionInfo.cpp

 
 ARMFunctionInfo::ARMFunctionInfo(MachineFunction &MF)
     : isThumb(MF.getSubtarget<ARMSubtarget>().isThumb()),
       hasThumb2(MF.getSubtarget<ARMSubtarget>().hasThumb2()),
       StByValParamsPadding(0), ArgRegsSaveSize(0), HasStackFrame(false),
       RestoreSPFromFP(false), LRSpilledForFarJump(false),
       FramePtrSpillOffset(0), GPRCS1Offset(0), GPRCS2Offset(0), DPRCSOffset(0),
       GPRCS1Size(0), GPRCS2Size(0), DPRCSSize(0),
-      PICLabelUId(0), VarArgsFrameIndex(0), HasITBlocks(false),
-      IsSplitCSR(false) {}
+      PICLabelUId(0), VarArgsFrameIndex(0), HasITBlocks(false) {}

lib/Target/X86/X86FastISel.cpp

bool X86FastISel::X86SelectRet(const Instruction *I) {
 
   if (!FuncInfo.CanLowerReturn)
     return false;
 
   if (TLI.supportSwiftError() &&
       F.getAttributes().hasAttrSomewhere(Attribute::SwiftError))
     return false;
 
-  if (TLI.supportSplitCSR(FuncInfo.MF))
-    return false;
-
   CallingConv::ID CC = F.getCallingConv();
   if (CC != CallingConv::C &&
       CC != CallingConv::Fast &&
       CC != CallingConv::X86_FastCall &&
       CC != CallingConv::X86_64_SysV)
     return false;
 
   if (Subtarget->isCallingConvWin64(CC))

lib/Target/X86/X86ISelLowering.h

SDValue LowerCall(CallLoweringInfo &CLI,
                       SmallVectorImpl<SDValue> &InVals) const override;
 
     SDValue LowerReturn(SDValue Chain,
                         CallingConv::ID CallConv, bool isVarArg,
                         const SmallVectorImpl<ISD::OutputArg> &Outs,
                         const SmallVectorImpl<SDValue> &OutVals,
                         SDLoc dl, SelectionDAG &DAG) const override;
 
-    bool supportSplitCSR(MachineFunction *MF) const override {
-      return MF->getFunction()->getCallingConv() == CallingConv::CXX_FAST_TLS &&
-          MF->getFunction()->hasFnAttribute(Attribute::NoUnwind);
-    }
-    void initializeSplitCSR(MachineBasicBlock *Entry) const override;
-    void insertCopiesSplitCSR(
-      MachineBasicBlock *Entry,
-      const SmallVectorImpl<MachineBasicBlock *> &Exits) const override;
-
     bool isUsedByReturnOnly(SDNode *N, SDValue &Chain) const override;
 
     bool mayBeEmittedAsTailCall(CallInst *CI) const override;
 
     EVT getTypeForExtReturn(LLVMContext &Context, EVT VT,
                             ISD::NodeType ExtendKind) const override;
 
     bool CanLowerReturn(CallingConv::ID CallConv, MachineFunction &MF,

lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::LowerReturn(SDValue Chain,
 
   if (CallConv == CallingConv::X86_INTR && !Outs.empty())
     report_fatal_error("X86 interrupts may not return any value");
 
   SmallVector<CCValAssign, 16> RVLocs;
   CCState CCInfo(CallConv, isVarArg, MF, RVLocs, *DAG.getContext());
   CCInfo.AnalyzeReturn(Outs, RetCC_X86);
 
+  SDValue IncomingChain = Chain;
   SDValue Flag;
   SmallVector<SDValue, 6> RetOps;
   RetOps.push_back(Chain); // Operand #0 = Chain (updated below)
   // Operand #1 = Bytes To Pop
   RetOps.push_back(DAG.getTargetConstant(FuncInfo->getBytesToPopOnReturn(), dl,
                    MVT::i32));
 
   // Copy the result values into the output registers.
X86TargetLowering::LowerReturn(SDValue Chain,
   // so now we copy the value out and into %rax/%eax.
   //
   // Checking Function.hasStructRetAttr() here is insufficient because the IR
   // may not have an explicit sret argument. If FuncInfo.CanLowerReturn is
   // false, then an sret argument may be implicitly inserted in the SelDAG. In
   // either case FuncInfo->setSRetReturnReg() will have been called.
   if (unsigned SRetReg = FuncInfo->getSRetReturnReg()) {
     // When we have both sret and another return value, we should use the
-    // original Chain stored in RetOps[0], instead of the current Chain updated
-    // in the above loop. If we only have sret, RetOps[0] equals to Chain.
+    // original Chain, instead of the current Chain updated in the above loop.
+    // If we only have sret, RetOps[0] equals to Chain.
 
     // For the case of sret and another return value, we have
     //   Chain_0 at the function entry
     //   Chain_1 = getCopyToReg(Chain_0) in the above loop
     // If we use Chain_1 in getCopyFromReg, we will have
     //   Val = getCopyFromReg(Chain_1)
     //   Chain_2 = getCopyToReg(Chain_1, Val) from below
 
     // getCopyToReg(Chain_0) will be glued together with
     // getCopyToReg(Chain_1, Val) into Unit A, getCopyFromReg(Chain_1) will be
     // in Unit B, and we will have cyclic dependency between Unit A and Unit B:
     //   Data dependency from Unit B to Unit A due to usage of Val in
     //     getCopyToReg(Chain_1, Val)
     //   Chain dependency from Unit A to Unit B
 
     // So here, we use RetOps[0] (i.e Chain_0) for getCopyFromReg.
-    SDValue Val = DAG.getCopyFromReg(RetOps[0], dl, SRetReg,
+    SDValue Val = DAG.getCopyFromReg(IncomingChain, dl, SRetReg,
                                      getPointerTy(MF.getDataLayout()));
 
     unsigned RetValReg
         = (Subtarget.is64Bit() && !Subtarget.isTarget64BitILP32()) ?
           X86::RAX : X86::EAX;
     Chain = DAG.getCopyToReg(Chain, dl, RetValReg, Val, Flag);
     Flag = Chain.getValue(1);
 
     // RAX/EAX now acts like a return value.
     RetOps.push_back(
         DAG.getRegister(RetValReg, getPointerTy(DAG.getDataLayout())));
   }
 
-  const X86RegisterInfo *TRI = Subtarget.getRegisterInfo();
-  const MCPhysReg *I =
-      TRI->getCalleeSavedRegsViaCopy(&DAG.getMachineFunction());
-  if (I) {
-    for (; *I; ++I) {
-      if (X86::GR64RegClass.contains(*I))
-        RetOps.push_back(DAG.getRegister(*I, MVT::i64));
-      else
-        llvm_unreachable("Unexpected register class in CSRsViaCopy!");
-    }
-  }
+  // Ensure callee saved registers that are saved via copies are back in their
+  // registers.
+  const MCPhysReg *SavedViaCopy = FuncInfo->getCSRSavedViaCopy();
+  if (SavedViaCopy != nullptr)
+    Flag = addCalleeSaveRegOps(SavedViaCopy, dl, IncomingChain, Flag, RetOps,
+                               DAG);
 
   RetOps[0] = Chain;  // Update chain.
 
   // Add the flag if we have it.
   if (Flag.getNode())
     RetOps.push_back(Flag);
 
   X86ISD::NodeType opcode = X86ISD::RET_FLAG;
if (Personality == EHPersonality::CoreCLR) {
       // same, so the size of funclets' (mostly empty) frames is dictated by
       // how far this slot is from the bottom (since they allocate just enough
       // space to accommodate holding this slot at the correct offset).
       int PSPSymFI = MFI->CreateStackObject(8, 8, /*isSS=*/false);
       EHInfo->PSPSymFrameIdx = PSPSymFI;
     }
   }
 
+  // Should we save callee save registers via vregs?
+  if (CallConv == CallingConv::CXX_FAST_TLS && Is64Bit &&
+      MF.getFunction()->hasFnAttribute(Attribute::NoUnwind) &&
+      !MF.getTarget().Options.EnableFastISel && mayUseSplitCSR(MF)) {
+    const MCPhysReg *SavedViaCopy = X86RegisterInfo::getFastTLSSavedViaCopy();
+    FuncInfo->setCSRSavedViaCopy(SavedViaCopy);
+
+    // Add live-ins.
+    for (const MCPhysReg *I = SavedViaCopy; *I; ++I) {
+      const MCPhysReg Reg = *I;
+      const TargetRegisterClass *RC;
+      if (X86::GR64RegClass.contains(Reg))
+        RC = &X86::GR64RegClass;
+      else
+        llvm_unreachable("Unexpected register class in CSRsViaCopy!");
+      MF.addLiveIn(Reg, RC);
+    }
+  }
+
   return Chain;
 }
 
 SDValue
 X86TargetLowering::LowerMemOpCallTo(SDValue Chain,
                                     SDValue StackPtr, SDValue Arg,
                                     SDLoc dl, SelectionDAG &DAG,
                                     const CCValAssign &VA,
bool X86TargetLowering::isIntDivCheap(EVT VT, AttributeSet Attr) const {
   // The exception to this is vector division. Since x86 doesn't have vector
   // integer division, leaving the division as-is is a loss even in terms of
   // size, because it will have to be scalarized, while the alternative code
   // sequence can be performed in vector form.
   bool OptSize = Attr.hasAttribute(AttributeSet::FunctionIndex,
                                    Attribute::MinSize);
   return OptSize && !VT.isVector();
 }
-
-void X86TargetLowering::initializeSplitCSR(MachineBasicBlock *Entry) const {
-  if (!Subtarget.is64Bit())
-    return;
-
-  // Update IsSplitCSR in X86MachineFunctionInfo.
-  X86MachineFunctionInfo *AFI =
-    Entry->getParent()->getInfo<X86MachineFunctionInfo>();
-  AFI->setIsSplitCSR(true);
-}
-
-void X86TargetLowering::insertCopiesSplitCSR(
-    MachineBasicBlock *Entry,
-    const SmallVectorImpl<MachineBasicBlock *> &Exits) const {
-  const X86RegisterInfo *TRI = Subtarget.getRegisterInfo();
-  const MCPhysReg *IStart = TRI->getCalleeSavedRegsViaCopy(Entry->getParent());
-  if (!IStart)
-    return;
-
-  const TargetInstrInfo *TII = Subtarget.getInstrInfo();
-  MachineRegisterInfo *MRI = &Entry->getParent()->getRegInfo();
-  MachineBasicBlock::iterator MBBI = Entry->begin();
-  for (const MCPhysReg *I = IStart; *I; ++I) {
-    const TargetRegisterClass *RC = nullptr;
-    if (X86::GR64RegClass.contains(*I))
-      RC = &X86::GR64RegClass;
-    else
-      llvm_unreachable("Unexpected register class in CSRsViaCopy!");
-
-    unsigned NewVR = MRI->createVirtualRegister(RC);
-    // Create copy from CSR to a virtual register.
-    // FIXME: this currently does not emit CFI pseudo-instructions, it works
-    // fine for CXX_FAST_TLS since the C++-style TLS access functions should be
-    // nounwind. If we want to generalize this later, we may need to emit
-    // CFI pseudo-instructions.
-    assert(Entry->getParent()->getFunction()->hasFnAttribute(
-               Attribute::NoUnwind) &&
-           "Function should be nounwind in insertCopiesSplitCSR!");
-    Entry->addLiveIn(*I);
-    BuildMI(*Entry, MBBI, DebugLoc(), TII->get(TargetOpcode::COPY), NewVR)
-        .addReg(*I);
-
-    // Insert the copy-back instructions right before the terminator.
-    for (auto *Exit : Exits)
-      BuildMI(*Exit, Exit->getFirstTerminator(), DebugLoc(),
-              TII->get(TargetOpcode::COPY), *I)
-          .addReg(NewVR);
-  }
-}

lib/Target/X86/X86MachineFunctionInfo.h

class X86MachineFunctionInfo : public MachineFunctionInfo {
   /// True if the function recovers from an SEH exception, and therefore needs
   /// to spill and restore the frame pointer.
   bool HasSEHFramePtrSave = false;
 
   /// The frame index of a stack object containing the original frame pointer
   /// used to address arguments in a function using a base pointer.
   int SEHFramePtrSaveIndex = 0;
 
-  /// True if this function has a subset of CSRs that is handled explicitly via
-  /// copies.
-  bool IsSplitCSR = false;
+  /// zero-terminated list of callee saved registers saved by copying to a vreg.
+  const MCPhysReg *CSRSavedViaCopy = nullptr;
 
   /// True if this function uses the red zone.
   bool UsesRedZone = false;
 
 private:
   /// ForwardedMustTailRegParms - A list of virtual and physical registers
   /// that must be forwarded to every musttail call.
   SmallVector<ForwardedRegister, 1> ForwardedMustTailRegParms;
public:
 
   int getSEHFramePtrSaveIndex() const { return SEHFramePtrSaveIndex; }
   void setSEHFramePtrSaveIndex(int Index) { SEHFramePtrSaveIndex = Index; }
 
   SmallVectorImpl<ForwardedRegister> &getForwardedMustTailRegParms() {
     return ForwardedMustTailRegParms;
   }
 
-  bool isSplitCSR() const { return IsSplitCSR; }
-  void setIsSplitCSR(bool s) { IsSplitCSR = s; }
+  const MCPhysReg *getCSRSavedViaCopy() const { return CSRSavedViaCopy; }
+  void setCSRSavedViaCopy(const MCPhysReg *CSRSavedViaCopy) {
+    this->CSRSavedViaCopy = CSRSavedViaCopy;
+  }
 
   bool getUsesRedZone() const { return UsesRedZone; }
   void setUsesRedZone(bool V) { UsesRedZone = V; }
 };
 
 } // End llvm namespace
 
 #endif

lib/Target/X86/X86RegisterInfo.h

public:
 
   unsigned getRegPressureLimit(const TargetRegisterClass *RC,
                                MachineFunction &MF) const override;
 
   /// getCalleeSavedRegs - Return a null-terminated list of all of the
   /// callee-save registers on this target.
   const MCPhysReg *
   getCalleeSavedRegs(const MachineFunction* MF) const override;
-  const MCPhysReg *
-  getCalleeSavedRegsViaCopy(const MachineFunction *MF) const override;
   const uint32_t *getCallPreservedMask(const MachineFunction &MF,
                                        CallingConv::ID) const override;
   const uint32_t *getNoPreservedMask() const override;
 
   // Calls involved in thread-local variable lookup save more registers than
   // normal calls, so they need a different mask to represent this.
   const uint32_t *getDarwinTLSCallPreservedMask() const;
 
public:
 
   // Debug information queries.
   unsigned getFrameRegister(const MachineFunction &MF) const override;
   unsigned getPtrSizedFrameRegister(const MachineFunction &MF) const;
   unsigned getStackRegister() const { return StackPtr; }
   unsigned getBaseRegister() const { return BasePtr; }
   // FIXME: Move to FrameInfok
   unsigned getSlotSize() const { return SlotSize; }
+
+  /// Returns CSR_64_CXX_TLS_Darwin_ViaCopy_SaveList.
+  static const MCPhysReg *getFastTLSSavedViaCopy();
 };
 
 //get512BitRegister - X86 utility - returns 512-bit super register
 unsigned get512BitSuperRegister(unsigned Reg);
 
 } // End llvm namespace
 
 #endif

lib/Target/X86/X86RegisterInfo.cpp

X86RegisterInfo::getCalleeSavedRegs(const MachineFunction *MF) const {
   case CallingConv::PreserveMost:
     return CSR_64_RT_MostRegs_SaveList;
   case CallingConv::PreserveAll:
     if (HasAVX)
       return CSR_64_RT_AllRegs_AVX_SaveList;
     return CSR_64_RT_AllRegs_SaveList;
   case CallingConv::CXX_FAST_TLS:
     if (Is64Bit)
-      return MF->getInfo<X86MachineFunctionInfo>()->isSplitCSR() ?
+      return MF->getInfo<X86MachineFunctionInfo>()->getCSRSavedViaCopy() ?
              CSR_64_CXX_TLS_Darwin_PE_SaveList : CSR_64_TLS_Darwin_SaveList;
     break;
   case CallingConv::Intel_OCL_BI: {
     if (HasAVX512 && IsWin64)
       return CSR_Win64_Intel_OCL_BI_AVX512_SaveList;
     if (HasAVX512 && Is64Bit)
       return CSR_64_Intel_OCL_BI_AVX512_SaveList;
     if (HasAVX && IsWin64)
if (Subtarget.getTargetLowering()->supportSwiftError() &&
       return CSR_64_SwiftError_SaveList;
     return CSR_64_SaveList;
   }
   if (CallsEHReturn)
     return CSR_32EHRet_SaveList;
   return CSR_32_SaveList;
 }
 
-const MCPhysReg *X86RegisterInfo::getCalleeSavedRegsViaCopy(
-    const MachineFunction *MF) const {
-  assert(MF && "Invalid MachineFunction pointer.");
-  if (MF->getFunction()->getCallingConv() == CallingConv::CXX_FAST_TLS &&
-      MF->getInfo<X86MachineFunctionInfo>()->isSplitCSR())
-    return CSR_64_CXX_TLS_Darwin_ViaCopy_SaveList;
-  return nullptr;
-}
-
 const uint32_t *
 X86RegisterInfo::getCallPreservedMask(const MachineFunction &MF,
                                       CallingConv::ID CC) const {
   const X86Subtarget &Subtarget = MF.getSubtarget<X86Subtarget>();
   bool HasSSE = Subtarget.hasSSE1();
   bool HasAVX = Subtarget.hasAVX();
   bool HasAVX512 = Subtarget.hasAVX512();
 
unsigned llvm::get512BitSuperRegister(unsigned Reg) {
   if (Reg >= X86::XMM0 && Reg <= X86::XMM31)
     return X86::ZMM0 + (Reg - X86::XMM0);
   if (Reg >= X86::YMM0 && Reg <= X86::YMM31)
     return X86::ZMM0 + (Reg - X86::YMM0);
   if (Reg >= X86::ZMM0 && Reg <= X86::ZMM31)
     return Reg;
   llvm_unreachable("Unexpected SIMD register");
 }
+
+const MCPhysReg *X86RegisterInfo::getFastTLSSavedViaCopy() {
+  return CSR_64_CXX_TLS_Darwin_ViaCopy_SaveList;
+}

test/CodeGen/ARM/cxx-tlscc.ll

 declare %struct.S* @_ZN1SC1Ev(%struct.S* returned)
 declare %struct.S* @_ZN1SD1Ev(%struct.S* returned)
 declare i32 @_tlv_atexit(void (i8*)*, i8*, i8*)
 
 ; Make sure Epilog does not overwrite an explicitly-handled CSR in CXX_FAST_TLS.
 ; THUMB-LABEL: _ZTW2sg
 ; THUMB: push {{.*}}lr
 ; THUMB: blx
-; THUMB: bne [[TH_end:.?LBB0_[0-9]+]]
+; THUMB: bne{{.w}} [[TH_end:.?LBB0_[0-9]+]]
 ; THUMB: blx
 ; THUMB: tlv_atexit
 ; THUMB: [[TH_end]]:
 ; THUMB: blx
 ; THUMB: r4
 ; THUMB: pop {{.*}}r4
 define cxx_fast_tlscc nonnull %struct.S* @_ZTW2sg() nounwind {
   %.b.i = load i1, i1* @__tls_guard, align 1