Disable Callee Saved Registers

[oren_ben_simhon]

Summary

Each Calling convention (CC) defines a static list of registers that should be preserved by a callee function. All other registers should be saved by the caller.
Some CCs use additional condition: If the register is used for passing/returning arguments – the caller needs to save it - even if it is part of the Callee Saved Registers (CSR) list.

The current LLVM implementation doesn’t support it. It will save a register if it is part of the static CSR list and will not care if the register is passed/returned by the callee.

The solution is to dynamically allocate the CSR lists (Only for these CCs). The lists will be updated with actual registers that should be saved by the callee.

Since we need the allocated lists to live as long as the function exists, the list should reside inside the Machine Register Info (MRI) which is a property of the Machine Function and managed by it (and has the same life span).

The lists should be saved in the MRI and populated upon LowerCall and LowerFormalArguments.

The patch will also assist to implement future no_caller_saved_regsiters attribute intended for interrupt handler CC.

[oren_ben_simhon]

Using the MF Allocator to allocate the register mask.

[kparzysz]

Can't you just change X86FrameLowering::determineCalleeSaves? The registers used to pass arguments to a function are already listed as live-in to the function in MRI.

[oren_ben_simhon]

Populated CalleeSaveDisableRegs list from LiveIns list (if in regcall calling convention).

[oren_ben_simhon]

Thanks Krzysztof for pointing it out. I updated the patch to use LiveIns to populate CalleeSaveDisableRegs List.
Notice that they are not the same because:

1. Each target may populate different Callee Saved Registers that should be disabled (not necessarily arguments).
2. For x86, it includes LiveIns and also return values - only for Regcall Calling Convention.

[oren_ben_simhon]

Updated lit test.

[oren_ben_simhon]

A kindly reminder. I will appreciate your inputs on the review.

[oren_ben_simhon]

If there are no additional comments please finish the review.

[MatzeB]

This is going to make the code confusing. We keep TRI::getCalleeSavedRegs() just that now it doesn't always give you the right answer and you have to check that other thing as well. We really should find a way so that getCalleeSavedRegs() (or some other new function that replaces the former) gives you a single correct answer.

include/llvm/CodeGen/RegisterClassInfo.h

57

There are several targets out there with more than 256 registers! AMDGPU already has > 1000, and I know of even crazier out-of-tree targets.

[oren_ben_simhon]

I will create a wrapper function to be used instead of getCalleeSavedRegs().

include/llvm/CodeGen/RegisterClassInfo.h

57

The size here is just the initial size. The size is updated according to the number of registers.

Notice that following code exmaple that actually changes the size (using push_back):

SmallVector<uint8_t, 16> ActualCalleeSaved;
...

for (unsigned I = 0; unsigned Reg = CSR[I]; ++I)
  ActualCalleeSaved.push_back(Reg);

...

CalleeSavedRegs = ActualCalleeSaved;

[oren_ben_simhon]

Implemented comments posted until 01/31

[MatzeB]

include/llvm/CodeGen/RegisterClassInfo.h

57

You hold a register number in an uint8_t with UINT8_MAX==256, don't you?
You should use MCPhysReg (or unsigned if you want to store phys+virtual regs)

[oren_ben_simhon]

Implemented comments submitted until 02/01 (Thank you Matthias).

[oren_ben_simhon]

Please assist in reviewing the patch.

[qcolombet]

Hi,

High level comment: if we are going to use getUpdatedCalleeSavedReg everywhere in place of getCalleeSavedReg, shouldn't we change directly the implementation of getCalleeSavedReg?
I am afraid that having the distinction is going to be confusing and error-prone.

Cheers,
-Quentin

include/llvm/CodeGen/MachineRegisterInfo.h

65

Don't repeat the name of the field in the comment.

72

*contrary

> As opposed to?

[qcolombet]

Erratum: I missed that we switched from TRI to MRI with the new API. I still believe the names are confusing and I would at least expect some comment update on TRI::getCalleeSavedRegs to point out the MRI variant and explain when we should use one or the other.

[MatzeB]

I completely agree with Quentin that we have to change the comment (or even better the name) of TargetRegisterInfo::getUpdatedCalleeSavedRegs() to discourage users to use the "wrong" function.

include/llvm/CodeGen/MachineRegisterInfo.h

211

Maybe disableCalleeSavedRegister() and improve the comment to explain the effects this has for a user (rather than explaining that a bit is set in a private datastructure).

213

Typo: invalif

220–224

Should be a doxygen comment (///)

This seems to be THE way to get a list of callee saved registers now. So at least the brief description should reflect that and avoid words like "updated" or explanations on what is happening internally. I'd expect something like this (the lazy allocation bits seem like an implementation detail that can be left out):

/// Return list of callee saved registers.
///
/// This returns targets default callee saved register list without the registers explicitely disabled by disableCalleeSavedRegister().

include/llvm/CodeGen/RegisterClassInfo.h

57

I didn't follow the code completely, but do we still need an extra SmallVector if there is MachineRegisterInfo::getUpdatedCalleeSavedRegs() that allocates storage itself?

lib/CodeGen/AggressiveAntiDepBreaker.cpp

161–162

The remark in braces is probably not necessary? My understanding is that registers that pass/return arguments simply are not callee saved registers so there shouldn't be a need to mention this here.

lib/CodeGen/CriticalAntiDepBreaker.cpp

69–70

dito.

lib/CodeGen/MachineRegisterInfo.cpp

548–549

Could you move the lazy memory allocation to the addDisableCalleeSavedReg() function so this can function can become const?

There are a bunch of not so nice const_casts now because of this.

lib/Target/X86/X86ISelLowering.cpp

3198–3200

const MachineRegisterInfo &MRI = MF.getRegInfo();
for (std::pair<unsigned,unsigned> P : make_range(MRI.livein_begin(), MRI.livein_end()) { ... }
```?

[oren_ben_simhon]

Implemented comments posted until 2/14 (Thank You Matthias & Quentin).

The main difference from last revision is that i moved the allocation and calculation of the updates CSR list.
The calculation will be performed every time a register is disabled and not when the list is requested.
This way we don't need a const cast when retrieving the CSR list.

[oren_ben_simhon]

Hi Guys, I would kindly like to remind of the review. Thanks in advance.

[aaboud]

include/llvm/CodeGen/MachineRegisterInfo.h

211

Typos:

CSR - Callee Saved Registers.
So, do not write CSR register mask, either say "CSR mask", or "callee saved register mask"

"allocating a the updated lisy" -> "allocating the updated list"

include/llvm/CodeGen/RegisterClassInfo.h

60

should be:

SmallVector<MCPhysReg, 4> CalleeSavedAliases;

In the previous implementation CSRNum[i] was an index into CalleeSaved, and the assumption was that CalleeSaved will not have more than 2^8 = 256 entries.
This is not the case with your new implementation where CalleeSavedAliases[i] is not an index into CalleeSavedRegs, but the actual MCPhysReg.

lib/CodeGen/LivePhysRegs.cpp

180

You may move this down to line 185, just before the line it is used at.

lib/CodeGen/MachineInstr.cpp

267

It is still worth to check if both have the same pointer, in such case we can return true.
Only if we have different pointers we need to check each element.

if(getRegMask() == Other.getRegMask())
  return true;

lib/CodeGen/MachineRegisterInfo.cpp

559

UpdatedCalleeSavedRegs can be of type SmallVector<MCPhysReg, 16>.
And at getUpdatedCalleeSavedRegs just return UpdatedCalleeSavedRegs.data();

In this case you can get rid of all these:

allocateRegisterList
CSRVector
CalleeSaveDisableRegs

Just make sure that last entry in UpdatedCalleeSavedRegs is a zero.
And you can remove registers from this buffer using

UpdatedCalleeSavedRegs.erase(std::remove(UpdatedCalleeSavedRegs.begin(), UpdatedCalleeSavedRegs.end(), Reg)

There will not be many registers to remove, and we only add registers once, at first time we entry this function.

lib/Target/X86/X86ISelLowering.cpp

3721

Initialize this to nullptr.

3746

sink this into the above if-else.
This case, there will be no case where we need to do const_cast, and for sure will not be sending the static "Mask" as "RegMask" to the LowerCallResult function.

[oren_ben_simhon]

Implemented comments posted until 03/11 (Thank You, Amjad).

[aaboud]

LGTM.
Just few minor comments below.

include/llvm/CodeGen/MachineRegisterInfo.h

70

These comment lines are not aligned to 80 characters.
Please, run clang-format on this?

209

Why did you remove the dot at the end of this line?

test/CodeGen/X86/DynamicCalleeSavedRegisters.ll

9

"retuning"-> "returning"
"used" -> "modified"

10

"calle" -> "callee"

13

Can you mentioned how this "hipe CC" function get's its arguments: a0, b0, c0, d0, and e0?

35

Please explain how the parameters a0, b0, c0, d0, and e0 are expected to be passed to this function, i.e., what register is used for each one?

test/CodeGen/X86/sse-regcall.ll

40

I believe these tests should be improved, the regular expression is misused in this test.
However, I think it can be done in separate patch.

[oren_ben_simhon]

Implemented comments posted until 03/13 (Thanks Amjad).

[oren_ben_simhon]

Updated the dynamic CSR test.

[aaboud]

The code looks much better now.
Craig, do you have any more comments?
If not, can you approve this patch?

[oren_ben_simhon]

Fixed a typo in dynamic CSR test.

[MatzeB]

include/llvm/CodeGen/MachineRegisterInfo.h

217

Please call this getCalleeSavedRegs() to make it clear that this should be the default function to call now to get the list of callee saved registers!
At the same time please add a sentence in the documentation of TargetRegisterInfo::getCalleeSavedRegs() to state that in most cases people should rather call the version in MachineRegisterInfo.

lib/CodeGen/MachineInstr.cpp

274–278

This can be simplified to
return std::equal(RegMask, RegMask+RegMaskSize, OtherRegMask);

[oren_ben_simhon]

Implemented comments posted by Matthias (Thank you).

[MatzeB]

LGTM.

include/llvm/Target/TargetRegisterInfo.h

429–430 ↗

(On Diff #91603)

How about "This function does not consider calling conventions adapting to parameters; consider using MachineRegisterInfo::getCalleeSavedRegs() instead."?

include/llvm/CodeGen/MachineRegisterInfo.h

private:
   /// VRegInfo - Information we keep for each virtual register.
   ///
   /// Each element in this list contains the register class of the vreg and the
   /// start of the use/def list for the register.
   IndexedMap<std::pair<RegClassOrRegBank, MachineOperand *>,
              VirtReg2IndexFunctor>
       VRegInfo;
 
+  /// The flag is true upon \p UpdatedCSRs initialization
+  /// and false otherwise.
+  bool IsUpdatedCSRsInitizialied;
+
+  /// Contains the updated callee saved register list.
+  /// As opposed to the static list defined in register info,
+  /// all registers that were disabled (in CalleeSaveDisableRegs)
+  /// are removed from the list.
+  SmallVector<MCPhysReg, 16> UpdatedCSRs;
+
   /// RegAllocHints - This vector records register allocation hints for virtual
   /// registers. For each virtual register, it keeps a register and hint type
   /// pair making up the allocation hint. Hint type is target specific except
   /// for the value 0 which means the second value of the pair is the preferred
   /// register for allocation. For example, if the hint is <0, 1024>, it means
   /// the allocator should prefer the physical register allocated to the virtual
   /// register of the hint.
   IndexedMap<std::pair<unsigned, unsigned>, VirtReg2IndexFunctor> RegAllocHints;
public:
   bool subRegLivenessEnabled() const {
     return TracksSubRegLiveness;
   }
 
   //===--------------------------------------------------------------------===//
   // Register Info
   //===--------------------------------------------------------------------===//
 
+  /// Disables the register from the list of CSRs
+  /// I.e. the register will not appear as part of the CSR mask.
+  /// \see UpdatedCalleeSavedRegs.
+  void disableCalleeSavedRegister(unsigned Reg);
+
+  /// Returns list of callee saved registers.
+  /// The function returns the updated CSR list (after taking into account
+  /// registers that are disabled from the CSR list).
+  const MCPhysReg *getUpdatedCalleeSavedRegs() const;
+
   // Strictly for use by MachineInstr.cpp.
   void addRegOperandToUseList(MachineOperand *MO);
 
   // Strictly for use by MachineInstr.cpp.
   void removeRegOperandFromUseList(MachineOperand *MO);
 
   // Strictly for use by MachineInstr.cpp.
   void moveOperands(MachineOperand *Dst, MachineOperand *Src, unsigned NumOps);
 
   /// Verify the sanity of the use list for Reg.
   void verifyUseList(unsigned Reg) const;
 
   /// Verify the use list of all registers.
   void verifyUseLists() const;

include/llvm/CodeGen/RegisterClassInfo.h

class RegisterClassInfo {
   // entry is valid when its tag matches.
   unsigned Tag;
 
   const MachineFunction *MF;
   const TargetRegisterInfo *TRI;
 
   // Callee saved registers of last MF. Assumed to be valid until the next
   // runOnFunction() call.
-  const MCPhysReg *CalleeSaved;
+  // Used only to determine if an update was made to CalleeSavedAliases.
+  const MCPhysReg *CalleeSavedRegs;
 
-  // Map register number to CalleeSaved index + 1;
-  SmallVector<uint8_t, 4> CSRNum;
+  // Map register alias to the callee saved Register.
+  SmallVector<MCPhysReg, 4> CalleeSavedAliases;
 
   // Reserved registers in the current MF.
   BitVector Reserved;
 
   std::unique_ptr<unsigned[]> PSetLimits;
 
   // Compute all information about RC.
   void compute(const TargetRegisterClass *RC) const;
public:
   /// Register classes like GR32_NOSP are not proper sub-classes because %esp
   /// is not allocatable.  Similarly, tGPR is not a proper sub-class in Thumb
   /// mode because the GPR super-class is not legal.
   bool isProperSubClass(const TargetRegisterClass *RC) const {
     return get(RC).ProperSubClass;
   }
 
   /// getLastCalleeSavedAlias - Returns the last callee saved register that
-  /// overlaps PhysReg, or 0 if Reg doesn't overlap a CSR.
+  /// overlaps PhysReg, or 0 if Reg doesn't overlap a CalleeSavedAliases.
   unsigned getLastCalleeSavedAlias(unsigned PhysReg) const {
     assert(TargetRegisterInfo::isPhysicalRegister(PhysReg));
-    if (unsigned N = CSRNum[PhysReg])
-      return CalleeSaved[N-1];
+    if (PhysReg < CalleeSavedAliases.size())
+      return CalleeSavedAliases[PhysReg];
     return 0;
   }
 
   /// Get the minimum register cost in RC's allocation order.
   /// This is the smallest value returned by TRI->getCostPerUse(Reg) for all
   /// the registers in getOrder(RC).
   unsigned getMinCost(const TargetRegisterClass *RC) {
     return get(RC).MinCost;

lib/CodeGen/AggressiveAntiDepBreaker.cpp

for (const auto &LI : (*SI)->liveins()) {
       for (MCRegAliasIterator AI(LI.PhysReg, TRI, true); AI.isValid(); ++AI) {
         unsigned Reg = *AI;
         State->UnionGroups(Reg, 0);
         KillIndices[Reg] = BB->size();
         DefIndices[Reg] = ~0u;
       }
     }
 
   // Mark live-out callee-saved registers. In a return block this is
   // all callee-saved registers. In non-return this is any
   // callee-saved register that is not saved in the prolog.
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   BitVector Pristine = MFI.getPristineRegs(MF);
-  for (const MCPhysReg *I = TRI->getCalleeSavedRegs(&MF); *I; ++I) {
+  for (const MCPhysReg *I = MF.getRegInfo().getUpdatedCalleeSavedRegs(); *I;
+       ++I) {
     unsigned Reg = *I;
     if (!IsReturnBlock && !Pristine.test(Reg)) continue;
     for (MCRegAliasIterator AI(Reg, TRI, true); AI.isValid(); ++AI) {
       unsigned AliasReg = *AI;
       State->UnionGroups(AliasReg, 0);
       KillIndices[AliasReg] = BB->size();
       DefIndices[AliasReg] = ~0u;
     }

lib/CodeGen/CriticalAntiDepBreaker.cpp

for (const auto &LI : (*SI)->liveins()) {
       for (MCRegAliasIterator AI(LI.PhysReg, TRI, true); AI.isValid(); ++AI) {
         unsigned Reg = *AI;
         Classes[Reg] = reinterpret_cast<TargetRegisterClass *>(-1);
         KillIndices[Reg] = BBSize;
         DefIndices[Reg] = ~0u;
       }
     }
 
   // Mark live-out callee-saved registers. In a return block this is
   // all callee-saved registers. In non-return this is any
   // callee-saved register that is not saved in the prolog.
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   BitVector Pristine = MFI.getPristineRegs(MF);
-  for (const MCPhysReg *I = TRI->getCalleeSavedRegs(&MF); *I; ++I) {
+  for (const MCPhysReg *I = MF.getRegInfo().getUpdatedCalleeSavedRegs(); *I;
+       ++I) {
     if (!IsReturnBlock && !Pristine.test(*I)) continue;
     for (MCRegAliasIterator AI(*I, TRI, true); AI.isValid(); ++AI) {
       unsigned Reg = *AI;
       Classes[Reg] = reinterpret_cast<TargetRegisterClass *>(-1);
       KillIndices[Reg] = BBSize;
       DefIndices[Reg] = ~0u;
     }
   }

lib/CodeGen/LivePhysRegs.cpp

void LivePhysRegs::addBlockLiveIns(const MachineBasicBlock &MBB) {
   }
 }
 
 /// Add pristine registers to the given \p LiveRegs. This function removes
 /// actually saved callee save registers when \p InPrologueEpilogue is false.
 static void addPristines(LivePhysRegs &LiveRegs, const MachineFunction &MF,
                          const MachineFrameInfo &MFI,
                          const TargetRegisterInfo &TRI) {
-  for (const MCPhysReg *CSR = TRI.getCalleeSavedRegs(&MF); CSR && *CSR; ++CSR)
+  const MachineRegisterInfo &MRI = MF.getRegInfo();
+  for (const MCPhysReg *CSR = MRI.getUpdatedCalleeSavedRegs(); CSR && *CSR;
+       ++CSR)
     LiveRegs.addReg(*CSR);
   for (const CalleeSavedInfo &Info : MFI.getCalleeSavedInfo())
     LiveRegs.removeReg(Info.getReg());
 }
 
 void LivePhysRegs::addLiveOutsNoPristines(const MachineBasicBlock &MBB) {
   // To get the live-outs we simply merge the live-ins of all successors.
   for (const MachineBasicBlock *Succ : MBB.successors())
     addBlockLiveIns(*Succ);
 }
 
 void LivePhysRegs::addLiveOuts(const MachineBasicBlock &MBB) {
   const MachineFunction &MF = *MBB.getParent();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   if (MFI.isCalleeSavedInfoValid()) {
     if (MBB.isReturnBlock()) {
       // The return block has no successors whose live-ins we could merge
       // below. So instead we add the callee saved registers manually.
-      for (const MCPhysReg *I = TRI->getCalleeSavedRegs(&MF); *I; ++I)
+      const MachineRegisterInfo &MRI = MF.getRegInfo();
+      for (const MCPhysReg *I = MRI.getUpdatedCalleeSavedRegs(); *I; ++I)
         addReg(*I);
     } else {
       addPristines(*this, MF, MFI, *TRI);
     }
   }
 
   addLiveOutsNoPristines(MBB);
 }

lib/CodeGen/MachineFunction.cpp

BitVector MachineFrameInfo::getPristineRegs(const MachineFunction &MF) const {
   const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
   BitVector BV(TRI->getNumRegs());
 
   // Before CSI is calculated, no registers are considered pristine. They can be
   // freely used and PEI will make sure they are saved.
   if (!isCalleeSavedInfoValid())
     return BV;
 
-  for (const MCPhysReg *CSR = TRI->getCalleeSavedRegs(&MF); CSR && *CSR; ++CSR)
+  const MachineRegisterInfo &MRI = MF.getRegInfo();
+  for (const MCPhysReg *CSR = MRI.getUpdatedCalleeSavedRegs(); CSR && *CSR;
+       ++CSR)
     BV.set(*CSR);
 
   // Saved CSRs are not pristine.
   for (auto &I : getCalleeSavedInfo())
     for (MCSubRegIterator S(I.getReg(), TRI, true); S.isValid(); ++S)
       BV.reset(*S);
 
   return BV;

lib/CodeGen/MachineInstr.cpp

case MachineOperand::MO_GlobalAddress:
     return getGlobal() == Other.getGlobal() && getOffset() == Other.getOffset();
   case MachineOperand::MO_ExternalSymbol:
     return !strcmp(getSymbolName(), Other.getSymbolName()) &&
            getOffset() == Other.getOffset();
   case MachineOperand::MO_BlockAddress:
     return getBlockAddress() == Other.getBlockAddress() &&
            getOffset() == Other.getOffset();
   case MachineOperand::MO_RegisterMask:
-  case MachineOperand::MO_RegisterLiveOut:
-    return getRegMask() == Other.getRegMask();
+  case MachineOperand::MO_RegisterLiveOut: {
+    // Shallow compare of the two RegMasks
+    const uint32_t *RegMask = getRegMask();
+    const uint32_t *OtherRegMask = Other.getRegMask();
+    if (RegMask == OtherRegMask)
+      return true;
+
+    // Calculate the size of the RegMask
+    const MachineFunction *MF = getParent()->getParent()->getParent();
+    const TargetRegisterInfo *TRI = MF->getSubtarget().getRegisterInfo();
+    unsigned RegMaskSize = (TRI->getNumRegs() + 31) / 32;
+
+    // Deep compare of the two RegMasks
+    for (unsigned I = 0, E = RegMaskSize; I < E; ++I)
+      if (RegMask[I] != OtherRegMask[I])
+        return false;
+
+    return true;
+  }
   case MachineOperand::MO_MCSymbol:
     return getMCSymbol() == Other.getMCSymbol();
   case MachineOperand::MO_CFIIndex:
     return getCFIIndex() == Other.getCFIIndex();
   case MachineOperand::MO_Metadata:
     return getMetadata() == Other.getMetadata();
   case MachineOperand::MO_IntrinsicID:
     return getIntrinsicID() == Other.getIntrinsicID();

lib/CodeGen/MachineRegisterInfo.cpp

 
 static cl::opt<bool> EnableSubRegLiveness("enable-subreg-liveness", cl::Hidden,
   cl::init(true), cl::desc("Enable subregister liveness tracking."));
 
 // Pin the vtable to this file.
 void MachineRegisterInfo::Delegate::anchor() {}
 
 MachineRegisterInfo::MachineRegisterInfo(MachineFunction *MF)
-    : MF(MF), TheDelegate(nullptr),
+    : MF(MF), TheDelegate(nullptr), IsUpdatedCSRsInitizialied(false),
       TracksSubRegLiveness(MF->getSubtarget().enableSubRegLiveness() &&
                            EnableSubRegLiveness) {
   unsigned NumRegs = getTargetRegisterInfo()->getNumRegs();
   VRegInfo.reserve(256);
   RegAllocHints.reserve(256);
   UsedPhysRegMask.resize(NumRegs);
   PhysRegUseDefLists.reset(new MachineOperand*[NumRegs]());
 }
bool MachineRegisterInfo::isPhysRegUsed(unsigned PhysReg) const {
   const TargetRegisterInfo *TRI = getTargetRegisterInfo();
   for (MCRegAliasIterator AliasReg(PhysReg, TRI, true); AliasReg.isValid();
        ++AliasReg) {
     if (!reg_nodbg_empty(*AliasReg))
       return true;
   }
   return false;
 }
+
+void MachineRegisterInfo::disableCalleeSavedRegister(unsigned Reg) {
+
+  const TargetRegisterInfo *TRI = getTargetRegisterInfo();
+  assert(Reg && (Reg < TRI->getNumRegs()) &&
+         "Trying to disable an invalid register");
+
+  if (!IsUpdatedCSRsInitizialied) {
+    const MCPhysReg *CSR = TRI->getCalleeSavedRegs(MF);
+    for (const MCPhysReg *I = CSR; *I; ++I)
+      UpdatedCSRs.push_back(*I);
+
+    // Zero value represents the end of the register list
+    // (no more registers should be pushed).
+    UpdatedCSRs.push_back(0);
+
+    IsUpdatedCSRsInitizialied = true;
+  }
+
+  // Remove the register (and its aliases from the list).
+  for (MCRegAliasIterator AI(Reg, TRI, true); AI.isValid(); ++AI)
+    UpdatedCSRs.erase(std::remove(UpdatedCSRs.begin(), UpdatedCSRs.end(), *AI),
+                      UpdatedCSRs.end());
+}
+
+const MCPhysReg *MachineRegisterInfo::getUpdatedCalleeSavedRegs() const {
+  if (IsUpdatedCSRsInitizialied)
+    return UpdatedCSRs.data();
+
+  return getTargetRegisterInfo()->getCalleeSavedRegs(MF);
+}

lib/CodeGen/PrologEpilogInserter.cpp

 static void assignCalleeSavedSpillSlots(MachineFunction &F,
                                         const BitVector &SavedRegs,
                                         unsigned &MinCSFrameIndex,
                                         unsigned &MaxCSFrameIndex) {
   if (SavedRegs.empty())
     return;
 
   const TargetRegisterInfo *RegInfo = F.getSubtarget().getRegisterInfo();
-  const MCPhysReg *CSRegs = RegInfo->getCalleeSavedRegs(&F);
+  const MCPhysReg *CSRegs = F.getRegInfo().getUpdatedCalleeSavedRegs();
 
   std::vector<CalleeSavedInfo> CSI;
   for (unsigned i = 0; CSRegs[i]; ++i) {
     unsigned Reg = CSRegs[i];
     if (SavedRegs.test(Reg))
       CSI.push_back(CalleeSavedInfo(Reg));
   }
 

lib/CodeGen/RegAllocPBQP.cpp

for (unsigned I = 0, E = MRI.getNumVirtRegs(); I != E; ++I) {
     } else {
       EmptyIntervalVRegs.insert(LI.reg);
     }
   }
 }
 
 static bool isACalleeSavedRegister(unsigned reg, const TargetRegisterInfo &TRI,
                                    const MachineFunction &MF) {
-  const MCPhysReg *CSR = TRI.getCalleeSavedRegs(&MF);
+  const MCPhysReg *CSR = MF.getRegInfo().getUpdatedCalleeSavedRegs();
   for (unsigned i = 0; CSR[i] != 0; ++i)
     if (TRI.regsOverlap(reg, CSR[i]))
       return true;
   return false;
 }
 
 void RegAllocPBQP::initializeGraph(PBQPRAGraph &G, VirtRegMap &VRM,
                                    Spiller &VRegSpiller) {

lib/CodeGen/RegisterClassInfo.cpp

 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/RegisterClassInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
+#include "llvm/Target/TargetFrameLowering.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "regalloc"
 
 static cl::opt<unsigned>
 StressRA("stress-regalloc", cl::Hidden, cl::init(0), cl::value_desc("N"),
          cl::desc("Limit all regclasses to N registers"));
 
-RegisterClassInfo::RegisterClassInfo()
-  : Tag(0), MF(nullptr), TRI(nullptr), CalleeSaved(nullptr) {}
+RegisterClassInfo::RegisterClassInfo() : Tag(0), MF(nullptr), TRI(nullptr) {}
 
 void RegisterClassInfo::runOnMachineFunction(const MachineFunction &mf) {
   bool Update = false;
   MF = &mf;
 
   // Allocate new array the first time we see a new target.
   if (MF->getSubtarget().getRegisterInfo() != TRI) {
     TRI = MF->getSubtarget().getRegisterInfo();
     RegClass.reset(new RCInfo[TRI->getNumRegClasses()]);
     unsigned NumPSets = TRI->getNumRegPressureSets();
     PSetLimits.reset(new unsigned[NumPSets]);
     std::fill(&PSetLimits[0], &PSetLimits[NumPSets], 0);
     Update = true;
   }
 
   // Does this MF have different CSRs?
   assert(TRI && "no register info set");
-  const MCPhysReg *CSR = TRI->getCalleeSavedRegs(MF);
-  if (Update || CSR != CalleeSaved) {
-    // Build a CSRNum map. Every CSR alias gets an entry pointing to the last
+
+  // Get the callee saved registers.
+  const MCPhysReg *CSR = MF->getRegInfo().getUpdatedCalleeSavedRegs();
+  if (Update || CSR != CalleeSavedRegs) {
+    // Build a CSRAlias map. Every CSR alias saves the last
     // overlapping CSR.
-    CSRNum.clear();
-    CSRNum.resize(TRI->getNumRegs(), 0);
-    for (unsigned N = 0; unsigned Reg = CSR[N]; ++N)
-      for (MCRegAliasIterator AI(Reg, TRI, true); AI.isValid(); ++AI)
-        CSRNum[*AI] = N + 1; // 0 means no CSR, 1 means CalleeSaved[0], ...
+    CalleeSavedAliases.resize(TRI->getNumRegs(), 0);
+    for (const MCPhysReg *I = CSR; *I; ++I)
+      for (MCRegAliasIterator AI(*I, TRI, true); AI.isValid(); ++AI)
+        CalleeSavedAliases[*AI] = *I;
+
     Update = true;
   }
-  CalleeSaved = CSR;
+  CalleeSavedRegs = CSR;
 
   // Different reserved registers?
   const BitVector &RR = MF->getRegInfo().getReservedRegs();
   if (Reserved.size() != RR.size() || RR != Reserved) {
     Update = true;
     Reserved = RR;
   }
 
void RegisterClassInfo::compute(const TargetRegisterClass *RC) const {
   for (unsigned i = 0; i != RawOrder.size(); ++i) {
     unsigned PhysReg = RawOrder[i];
     // Remove reserved registers from the allocation order.
     if (Reserved.test(PhysReg))
       continue;
     unsigned Cost = TRI->getCostPerUse(PhysReg);
     MinCost = std::min(MinCost, Cost);
 
-    if (CSRNum[PhysReg])
+    if (CalleeSavedAliases[PhysReg])
       // PhysReg aliases a CSR, save it for later.
       CSRAlias.push_back(PhysReg);
     else {
       if (Cost != LastCost)
         LastCostChange = N;
       RCI.Order[N++] = PhysReg;
       LastCost = Cost;
     }

lib/CodeGen/TargetFrameLoweringImpl.cpp

void TargetFrameLowering::determineCalleeSaves(MachineFunction &MF,
   SavedRegs.resize(TRI.getNumRegs());
 
   // When interprocedural register allocation is enabled caller saved registers
   // are preferred over callee saved registers.
   if (MF.getTarget().Options.EnableIPRA && isSafeForNoCSROpt(MF.getFunction()))
     return;
 
   // Get the callee saved register list...
-  const MCPhysReg *CSRegs = TRI.getCalleeSavedRegs(&MF);
+  const MCPhysReg *CSRegs = MF.getRegInfo().getUpdatedCalleeSavedRegs();
 
   // Early exit if there are no callee saved registers.
   if (!CSRegs || CSRegs[0] == 0)
     return;
 
   // In Naked functions we aren't going to save any registers.
   if (MF.getFunction()->hasFnAttribute(Attribute::Naked))
     return;

lib/Target/X86/X86ISelLowering.h

private:
     void addLegalFPImmediate(const APFloat& Imm) {
       LegalFPImmediates.push_back(Imm);
     }
 
     SDValue LowerCallResult(SDValue Chain, SDValue InFlag,
                             CallingConv::ID CallConv, bool isVarArg,
                             const SmallVectorImpl<ISD::InputArg> &Ins,
                             const SDLoc &dl, SelectionDAG &DAG,
-                            SmallVectorImpl<SDValue> &InVals) const;
+                            SmallVectorImpl<SDValue> &InVals,
+                            uint32_t *RegMask) const;
     SDValue LowerMemArgument(SDValue Chain, CallingConv::ID CallConv,
                              const SmallVectorImpl<ISD::InputArg> &ArgInfo,
                              const SDLoc &dl, SelectionDAG &DAG,
                              const CCValAssign &VA, MachineFrameInfo &MFI,
                              unsigned i) const;
     SDValue LowerMemOpCallTo(SDValue Chain, SDValue StackPtr, SDValue Arg,
                              const SDLoc &dl, SelectionDAG &DAG,
                              const CCValAssign &VA,

lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::LowerReturn(SDValue Chain, CallingConv::ID CallConv,
   RetOps.push_back(DAG.getTargetConstant(FuncInfo->getBytesToPopOnReturn(), dl,
                    MVT::i32));
 
   // Copy the result values into the output registers.
   for (unsigned I = 0, OutsIndex = 0, E = RVLocs.size(); I != E;
        ++I, ++OutsIndex) {
     CCValAssign &VA = RVLocs[I];
     assert(VA.isRegLoc() && "Can only return in registers!");
+
+    // Add the register to the CalleeSaveDisableRegs list.
+    if (CallConv == CallingConv::X86_RegCall)
+      MF.getRegInfo().disableCalleeSavedRegister(VA.getLocReg());
+
     SDValue ValToCopy = OutVals[OutsIndex];
     EVT ValVT = ValToCopy.getValueType();
 
     // Promote values to the appropriate types.
     if (VA.getLocInfo() == CCValAssign::SExt)
       ValToCopy = DAG.getNode(ISD::SIGN_EXTEND, dl, VA.getLocVT(), ValToCopy);
     else if (VA.getLocInfo() == CCValAssign::ZExt)
       ValToCopy = DAG.getNode(ISD::ZERO_EXTEND, dl, VA.getLocVT(), ValToCopy);
if (VA.needsCustom()) {
       assert(VA.getValVT() == MVT::v64i1 &&
              "Currently the only custom case is when we split v64i1 to 2 regs");
 
       Passv64i1ArgInRegs(dl, DAG, Chain, ValToCopy, RegsToPass, VA, RVLocs[++I],
                          Subtarget);
 
       assert(2 == RegsToPass.size() &&
              "Expecting two registers after Pass64BitArgInRegs");
+
+      // Add the second register to the CalleeSaveDisableRegs list.
+      if (CallConv == CallingConv::X86_RegCall)
+        MF.getRegInfo().disableCalleeSavedRegister(RVLocs[I].getLocReg());
     } else {
       RegsToPass.push_back(std::make_pair(VA.getLocReg(), ValToCopy));
     }
 
     // Add nodes to the DAG and add the values into the RetOps list
     for (auto &Reg : RegsToPass) {
       Chain = DAG.getCopyToReg(Chain, dl, Reg.first, Reg.second, Flag);
       Flag = Chain.getValue(1);
unsigned RetValReg
         = (Subtarget.is64Bit() && !Subtarget.isTarget64BitILP32()) ?
           X86::RAX : X86::EAX;
     Chain = DAG.getCopyToReg(Chain, dl, RetValReg, Val, Flag);
     Flag = Chain.getValue(1);
 
     // RAX/EAX now acts like a return value.
     RetOps.push_back(
         DAG.getRegister(RetValReg, getPointerTy(DAG.getDataLayout())));
+
+    // Add the returned register to the CalleeSaveDisableRegs list.
+    if (CallConv == CallingConv::X86_RegCall)
+      MF.getRegInfo().disableCalleeSavedRegister(RetValReg);
   }
 
   const X86RegisterInfo *TRI = Subtarget.getRegisterInfo();
   const MCPhysReg *I =
       TRI->getCalleeSavedRegsViaCopy(&DAG.getMachineFunction());
   if (I) {
     for (; *I; ++I) {
       if (X86::GR64RegClass.contains(*I))

 }
 
 /// Lower the result values of a call into the
 /// appropriate copies out of appropriate physical registers.
 ///
 SDValue X86TargetLowering::LowerCallResult(
     SDValue Chain, SDValue InFlag, CallingConv::ID CallConv, bool isVarArg,
     const SmallVectorImpl<ISD::InputArg> &Ins, const SDLoc &dl,
-    SelectionDAG &DAG, SmallVectorImpl<SDValue> &InVals) const {
+    SelectionDAG &DAG, SmallVectorImpl<SDValue> &InVals,
+    uint32_t *RegMask) const {
 
+  const TargetRegisterInfo *TRI = Subtarget.getRegisterInfo();
   // Assign locations to each value returned by this call.
   SmallVector<CCValAssign, 16> RVLocs;
   bool Is64Bit = Subtarget.is64Bit();
   CCState CCInfo(CallConv, isVarArg, DAG.getMachineFunction(), RVLocs,
                  *DAG.getContext());
   CCInfo.AnalyzeCallResult(Ins, RetCC_X86);
 
   // Copy all of the result registers out of their specified physreg.
   for (unsigned I = 0, InsIndex = 0, E = RVLocs.size(); I != E;
        ++I, ++InsIndex) {
     CCValAssign &VA = RVLocs[I];
     EVT CopyVT = VA.getLocVT();
 
+    // In some calling conventions we need to remove the used registers
+    // from the register mask.
+    if (RegMask && CallConv == CallingConv::X86_RegCall) {
+      for (MCSubRegIterator SubRegs(VA.getLocReg(), TRI, /*IncludeSelf=*/true);
+           SubRegs.isValid(); ++SubRegs)
+        RegMask[*SubRegs / 32] &= ~(1u << (*SubRegs % 32));
+    }
+
     // If this is x86-64, and we disabled SSE, we can't return FP values
     if ((CopyVT == MVT::f32 || CopyVT == MVT::f64 || CopyVT == MVT::f128) &&
         ((Is64Bit || Ins[InsIndex].Flags.isInReg()) && !Subtarget.hasSSE1())) {
       report_fatal_error("SSE register return with SSE disabled");
     }
 
     // If we prefer to use the value in xmm registers, copy it out as f80 and
     // use a truncate to move it from fp stack reg to xmm reg.
if (Personality == EHPersonality::CoreCLR) {
       // same, so the size of funclets' (mostly empty) frames is dictated by
       // how far this slot is from the bottom (since they allocate just enough
       // space to accommodate holding this slot at the correct offset).
       int PSPSymFI = MFI.CreateStackObject(8, 8, /*isSS=*/false);
       EHInfo->PSPSymFrameIdx = PSPSymFI;
     }
   }
 
+  if (CallConv == CallingConv::X86_RegCall) {
+    const MachineRegisterInfo &MRI = MF.getRegInfo();
+    for (const auto &Pair : make_range(MRI.livein_begin(), MRI.livein_end()))
+      MF.getRegInfo().disableCalleeSavedRegister(Pair.first);
+  }
+
   return Chain;
 }
 
 SDValue X86TargetLowering::LowerMemOpCallTo(SDValue Chain, SDValue StackPtr,
                                             SDValue Arg, const SDLoc &dl,
                                             SelectionDAG &DAG,
                                             const CCValAssign &VA,
                                             ISD::ArgFlagsTy Flags) const {
if (!Is64Bit && CLI.CS && CLI.CS->isInvoke()) {
     EHPersonality Pers =
         CallerFn->hasPersonalityFn()
             ? classifyEHPersonality(CallerFn->getPersonalityFn())
             : EHPersonality::Unknown;
     if (isFuncletEHPersonality(Pers))
       Mask = RegInfo->getNoPreservedMask();
   }
 
+  // Define a new register mask from the existing mask.
+  uint32_t *RegMask = nullptr;
+
+  // In some calling conventions we need to remove the used physical registers
+  // from the reg mask.
+  if (CallConv == CallingConv::X86_RegCall) {
+    /// @todo Need to add support in MIPrinter and MIParser to represent
+    ///       the customed RegMask.
+    const TargetRegisterInfo *TRI = Subtarget.getRegisterInfo();
+
+    // Allocate a new Reg Mask and copy Mask.
+    RegMask = MF.allocateRegisterMask(TRI->getNumRegs());
+    unsigned RegMaskSize = (TRI->getNumRegs() + 31) / 32;
+    memcpy(RegMask, Mask, sizeof(uint32_t) * RegMaskSize);
+
+    // Make sure all sub registers of the argument registers are reset
+    // in the RegMask.
+    for (auto const &RegPair : RegsToPass)
+      for (MCSubRegIterator SubRegs(RegPair.first, TRI, /*IncludeSelf=*/true);
+           SubRegs.isValid(); ++SubRegs)
+        RegMask[*SubRegs / 32] &= ~(1u << (*SubRegs % 32));
+
+    // Create the RegMask Operand according to our updated mask.
+    Ops.push_back(DAG.getRegisterMask(RegMask));
+  } else {
+    // Create the RegMask Operand according to the static mask.
     Ops.push_back(DAG.getRegisterMask(Mask));
+  }
 
   if (InFlag.getNode())
     Ops.push_back(InFlag);
 
   if (isTailCall) {
     // We used to do:
     //// If this is the first return lowered for this function, add the regs
     //// to the liveout set for the function.
Chain = DAG.getCALLSEQ_END(Chain,
                                DAG.getIntPtrConstant(NumBytesForCalleeToPop, dl,
                                                      true),
                                InFlag, dl);
     InFlag = Chain.getValue(1);
   }
 
   // Handle result values, copying them out of physregs into vregs that we
   // return.
-  return LowerCallResult(Chain, InFlag, CallConv, isVarArg,
-                         Ins, dl, DAG, InVals);
+  return LowerCallResult(Chain, InFlag, CallConv, isVarArg, Ins, dl, DAG,
+                         InVals, RegMask);
 }
 
 //===----------------------------------------------------------------------===//
 //                Fast Calling Convention (tail call) implementation
 //===----------------------------------------------------------------------===//
 
 //  Like std call, callee cleans arguments, convention except that ECX is
 //  reserved for storing the tail called function address. Only 2 registers are
X86TargetLowering::EmitSjLjDispatchBlock(MachineInstr &MI,
   // Add the jump table entries as successors to the MBB.
   SmallPtrSet<MachineBasicBlock *, 8> SeenMBBs;
   for (auto &LP : LPadList)
     if (SeenMBBs.insert(LP).second)
       DispContBB->addSuccessor(LP);
 
   // N.B. the order the invoke BBs are processed in doesn't matter here.
   SmallVector<MachineBasicBlock *, 64> MBBLPads;
-  const MCPhysReg *SavedRegs =
-      Subtarget.getRegisterInfo()->getCalleeSavedRegs(MF);
+  const MCPhysReg *SavedRegs = MF->getRegInfo().getUpdatedCalleeSavedRegs();
   for (MachineBasicBlock *MBB : InvokeBBs) {
     // Remove the landing pad successor from the invoke block and replace it
     // with the new dispatch block.
     // Keep a copy of Successors since it's modified inside the loop.
     SmallVector<MachineBasicBlock *, 8> Successors(MBB->succ_rbegin(),
                                                    MBB->succ_rend());
     // FIXME: Avoid quadratic complexity.
     for (auto MBBS : Successors) {

lib/Target/X86/X86MachineFunctionInfo.cpp

 //===-- X86MachineFunctionInfo.cpp - X86 machine function info ------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "X86MachineFunctionInfo.h"
 #include "X86RegisterInfo.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/Target/TargetSubtargetInfo.h"
 
 using namespace llvm;
 
 void X86MachineFunctionInfo::anchor() { }
 
 void X86MachineFunctionInfo::setRestoreBasePointer(const MachineFunction *MF) {
   if (!RestoreBasePointerOffset) {
     const X86RegisterInfo *RegInfo = static_cast<const X86RegisterInfo *>(
       MF->getSubtarget().getRegisterInfo());
     unsigned SlotSize = RegInfo->getSlotSize();
-    for (const MCPhysReg *CSR =
-      RegInfo->X86RegisterInfo::getCalleeSavedRegs(MF);
-      unsigned Reg = *CSR;
-       ++CSR)
-    {
+    for (const MCPhysReg *CSR = MF->getRegInfo().getUpdatedCalleeSavedRegs();
+         unsigned Reg = *CSR; ++CSR) {
       if (X86::GR64RegClass.contains(Reg) || X86::GR32RegClass.contains(Reg))
         RestoreBasePointerOffset -= SlotSize;
     }
   }
 }
 

test/CodeGen/X86/DynamicCalleeSavedRegisters.ll

+; RUN: llc < %s -mtriple=i386-linux-gnu  | FileCheck --check-prefix=CHECK %s
+
+declare x86_regcallcc i32 @callee(i32 %a0, i32 %b0, i32 %c0, i32 %d0, i32 %e0);
+
+; In RegCall calling convention, %esi and %edi are callee saved registers.
+; One might think that the caller could assume that %esi value is the same before
+; and after calling the callee.
+; However, RegCall also says that a register that was used for 
+; passing/retuning argumnets, can be assumed to be used by the callee.
+; In other words, it is no longer a calle saved register.
+; In this case we want to see that %edi and %esi are saved and %esi is assumed
+; to be used by the caller.
+; This is a hipe CC function that doesn't save any register for the caller 
+; of the callee. So we can be sure that there is no other reason to save 
+; %esi, %edi and other registers (it returns the result in %esi).
+define cc 11 i32 @caller(i32 %a0, i32 %b0, i32 %c0, i32 %d0, i32 %e0) nounwind {
+  %b1 = call x86_regcallcc i32 @callee(i32 %a0, i32 %b0, i32 %c0, i32 %d0, i32 %e0)
+  %b2 = add i32 %b1, %d0
+  %b3 = add i32 %b2, %e0
+  ret i32 %b3
+}
+; CHECK-LABEL: caller
+; CHECK:       movl    %edx, %ebx
+; CHECK:       movl    %ebp, %esi
+; CHECK:       calll   callee
+; CHECK:       leal    (%eax,%ebx), %esi
+; CHECK:       addl    %ebp, %esi
+; CHECK:       retl
+
+!hipe.literals = !{ !0, !1, !2 }
+!0 = !{ !"P_NSP_LIMIT", i32 120 }
+!1 = !{ !"X86_LEAF_WORDS", i32 24 }
+!2 = !{ !"AMD64_LEAF_WORDS", i32 18 }
+
+; Make sure that the callee doesn't save paramteres that were passed as arguments.
+define x86_regcallcc {i32, i32, i32} @test_callee(i32 %a0, i32 %b0, i32 %c0, i32 %d0, i32 %e0) nounwind {
+  %b1 = mul i32 7, %e0
+  %b2 = udiv i32 5, %e0
+  %b3 = mul i32 7, %d0
+  %b4 = insertvalue {i32, i32, i32} undef, i32 %b1, 0
+  %b5 = insertvalue {i32, i32, i32} %b4, i32 %b2, 1
+  %b6 = insertvalue {i32, i32, i32} %b5, i32 %b3, 2
+  ret {i32, i32, i32} %b6
+}
+; CHECK-LABEL: test_callee
+; CHECK-NOT:   pushl %esi
+; CHECK-NOT:   pushl %edi
+; CHECK:       retl

test/CodeGen/X86/avx512-regcall-NoMask.ll

 define x86_regcallcc <32 x float> @testf32_inp(<32 x float> %a, <32 x float> %b, <32 x float> %c) nounwind {
   %x1 = fadd <32 x float> %a, %b
   %x2 = fmul <32 x float> %a, %b
   %x3 = fsub <32 x float> %x1, %x2
   %x4 = fadd <32 x float> %x3, %c
   ret <32 x float> %x4
 }
 
-; X32-LABEL: pushl {{%e(si|di|bx|bp)}}
-; X32: pushl {{%e(si|di|bx|bp)}}
-; X32: pushl {{%e(si|di|bx|bp)}}
-; X32: pushl {{%e(si|di|bx|bp)}}
-; X32: popl {{%e(si|di|bx|bp)}}
-; X32: popl {{%e(si|di|bx|bp)}}
-; X32: popl {{%e(si|di|bx|bp)}}
-; X32: popl {{%e(si|di|bx|bp)}}
+; X32-LABEL: testi32_inp
+; X32: pushl {{%e(bx|bp)}}
+; X32: pushl {{%e(bx|bp)}}
+; X32: popl {{%e(bx|bp)}}
+; X32: popl {{%e(bx|bp)}}
 ; X32: retl
 
-; WIN64-LABEL: pushq  {{%r(bp|bx|1[0-5])}}
+; WIN64-LABEL: testi32_inp
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
 ; WIN64: popq {{%r(bp|bx|1[0-5])}}
 ; WIN64: popq {{%r(bp|bx|1[0-5])}}
 ; WIN64: popq {{%r(bp|bx|1[0-5])}}
-; WIN64: popq {{%r(bp|bx|1[0-5])}}
 ; WIN64: retq
 
-; LINUXOSX64-LABEL: pushq {{%r(bp|bx|1[2-5])}}
+; LINUXOSX64-LABEL: testi32_inp
 ; LINUXOSX64: pushq {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX64: pushq {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX64: popq  {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX64: popq  {{%r(bp|bx|1[2-5])}}
-; LINUXOSX64: popq  {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX64: retq
 
 ; Test regcall when running multiple input parameters - callee saved GPRs
 define x86_regcallcc i32 @testi32_inp(i32 %a1, i32 %a2, i32 %a3, i32 %a4, i32 %a5, i32 %a6,
                                       i32 %b1, i32 %b2, i32 %b3, i32 %b4, i32 %b5, i32 %b6) nounwind {
   %x1 = sub i32 %a1, %a2
   %x2 = sub i32 %a3, %a4
   %x3 = sub i32 %a5, %a6

test/CodeGen/X86/sse-regcall.ll

 define x86_regcallcc i1 @test_CallargReti1(i1 %a)  {
   %b = add i1 %a, 1
   %c = call x86_regcallcc i1 @test_argReti1(i1 %b)
   %d = add i1 %c, 1
   ret i1 %d
 }
 
 ; WIN64-LABEL: testf32_inp
-; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
+; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
 ; WIN64: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; WIN64: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; WIN64: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; WIN64: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
-; WIN64: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; WIN64: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; WIN64: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; WIN64: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; WIN64: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; WIN64: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; WIN64: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; WIN64: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
 ; WIN64: retq
 
 ; WIN32-LABEL: testf32_inp
-; WIN32: movaps {{%xmm([4-7])}}, {{.*(%ebp).*}}  {{#+}} 16-byte Spill
-; WIN32: movaps {{%xmm([4-7])}}, {{.*(%ebp).*}}  {{#+}} 16-byte Spill
-; WIN32: movaps {{%xmm([4-7])}}, {{.*(%ebp).*}}  {{#+}} 16-byte Spill
-; WIN32: movaps {{%xmm([4-7])}}, {{.*(%ebp).*}}  {{#+}} 16-byte Spill
+; WIN32: movaps {{%xmm([0-7])}}, {{.*(%e(b|s)p).*}}  {{#+}} 16-byte Spill
 ; WIN32: {{.*}} {{%xmm[0-7]}}, {{%xmm[4-7]}}
 ; WIN32: {{.*}} {{%xmm[0-7]}}, {{%xmm[4-7]}}
 ; WIN32: {{.*}} {{%xmm[0-7]}}, {{%xmm[4-7]}}
 ; WIN32: {{.*}} {{%xmm[0-7]}}, {{%xmm[4-7]}}
-; WIN32: movaps {{.*(%ebp).*}}, {{%xmm([4-7])}}  {{#+}} 16-byte Reload
-; WIN32: movaps {{.*(%ebp).*}}, {{%xmm([4-7])}}  {{#+}} 16-byte Reload
-; WIN32: movaps {{.*(%ebp).*}}, {{%xmm([4-7])}}  {{#+}} 16-byte Reload
-; WIN32: movaps {{.*(%ebp).*}}, {{%xmm([4-7])}}  {{#+}} 16-byte Reload
+; WIN32: movaps {{.*(%e(b|s)p).*}}, {{%xmm([0-7])}}  {{#+}} 16-byte Reload
 ; WIN32: retl
 
 ; LINUXOSX-LABEL: testf32_inp
-; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
+; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
 ; LINUXOSX: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; LINUXOSX: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; LINUXOSX: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; LINUXOSX: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
-; LINUXOSX: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; LINUXOSX: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; LINUXOSX: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; LINUXOSX: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; LINUXOSX: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; LINUXOSX: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; LINUXOSX: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; LINUXOSX: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
 ; LINUXOSX: retq
 
 ;test calling conventions - input parameters, callee saved XMMs
 define x86_regcallcc <16 x float> @testf32_inp(<16 x float> %a, <16 x float> %b, <16 x float> %c) nounwind {
   %x1 = fadd <16 x float> %a, %b
   %x2 = fmul <16 x float> %a, %b
   %x3 = fsub <16 x float> %x1, %x2
   %x4 = fadd <16 x float> %x3, %c
   ret <16 x float> %x4
 }
 
 ; WIN32-LABEL: testi32_inp
 ; WIN32: pushl {{%e(si|di|bx|bp)}}
 ; WIN32: pushl {{%e(si|di|bx|bp)}}
-; WIN32: pushl {{%e(si|di|bx|bp)}}
-; WIN32: pushl {{%e(si|di|bx|bp)}}
-; WIN32: popl {{%e(si|di|bx|bp)}}
-; WIN32: popl {{%e(si|di|bx|bp)}}
 ; WIN32: popl {{%e(si|di|bx|bp)}}
 ; WIN32: popl {{%e(si|di|bx|bp)}}
 ; WIN32: retl
 
 ; WIN64-LABEL: testi32_inp
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
-; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
-; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
-; WIN64: popq {{%r(bp|bx|1[0-5])}}
-; WIN64: popq {{%r(bp|bx|1[0-5])}}
 ; WIN64: popq {{%r(bp|bx|1[0-5])}}
 ; WIN64: popq {{%r(bp|bx|1[0-5])}}
 ; WIN64: popq {{%r(bp|bx|1[0-5])}}
 ; WIN64: retq
 
 ; LINUXOSX-LABEL: testi32_inp
 ; LINUXOSX: pushq {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX: pushq {{%r(bp|bx|1[2-5])}}
-; LINUXOSX: pushq {{%r(bp|bx|1[2-5])}}
-; LINUXOSX: pushq {{%r(bp|bx|1[2-5])}}
-; LINUXOSX: popq  {{%r(bp|bx|1[2-5])}}
-; LINUXOSX: popq  {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX: popq  {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX: popq  {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX: retq
 
 ;test calling conventions - input parameters, callee saved GPRs
 define x86_regcallcc i32 @testi32_inp(i32 %a1, i32 %a2, i32 %a3, i32 %a4, i32 %a5, i32 %a6,
                                       i32 %b1, i32 %b2, i32 %b3, i32 %b4, i32 %b5, i32 %b6) nounwind {
   %x1 = sub i32 %a1, %a2