Disable Callee Saved Registers

[oren_ben_simhon]

Summary

Each Calling convention (CC) defines a static list of registers that should be preserved by a callee function. All other registers should be saved by the caller.
Some CCs use additional condition: If the register is used for passing/returning arguments – the caller needs to save it - even if it is part of the Callee Saved Registers (CSR) list.

The current LLVM implementation doesn’t support it. It will save a register if it is part of the static CSR list and will not care if the register is passed/returned by the callee.

The solution is to dynamically allocate the CSR lists (Only for these CCs). The lists will be updated with actual registers that should be saved by the callee.

Since we need the allocated lists to live as long as the function exists, the list should reside inside the Machine Register Info (MRI) which is a property of the Machine Function and managed by it (and has the same life span).

The lists should be saved in the MRI and populated upon LowerCall and LowerFormalArguments.

The patch will also assist to implement future no_caller_saved_regsiters attribute intended for interrupt handler CC.

[oren_ben_simhon]

Using the MF Allocator to allocate the register mask.

[kparzysz]

Can't you just change X86FrameLowering::determineCalleeSaves? The registers used to pass arguments to a function are already listed as live-in to the function in MRI.

[oren_ben_simhon]

Populated CalleeSaveDisableRegs list from LiveIns list (if in regcall calling convention).

[oren_ben_simhon]

Thanks Krzysztof for pointing it out. I updated the patch to use LiveIns to populate CalleeSaveDisableRegs List.
Notice that they are not the same because:

1. Each target may populate different Callee Saved Registers that should be disabled (not necessarily arguments).
2. For x86, it includes LiveIns and also return values - only for Regcall Calling Convention.

[oren_ben_simhon]

Updated lit test.

[oren_ben_simhon]

A kindly reminder. I will appreciate your inputs on the review.

[oren_ben_simhon]

If there are no additional comments please finish the review.

[MatzeB]

This is going to make the code confusing. We keep TRI::getCalleeSavedRegs() just that now it doesn't always give you the right answer and you have to check that other thing as well. We really should find a way so that getCalleeSavedRegs() (or some other new function that replaces the former) gives you a single correct answer.

include/llvm/CodeGen/RegisterClassInfo.h

56

There are several targets out there with more than 256 registers! AMDGPU already has > 1000, and I know of even crazier out-of-tree targets.

[oren_ben_simhon]

I will create a wrapper function to be used instead of getCalleeSavedRegs().

include/llvm/CodeGen/RegisterClassInfo.h

56

The size here is just the initial size. The size is updated according to the number of registers.

Notice that following code exmaple that actually changes the size (using push_back):

SmallVector<uint8_t, 16> ActualCalleeSaved;
...

for (unsigned I = 0; unsigned Reg = CSR[I]; ++I)
  ActualCalleeSaved.push_back(Reg);

...

CalleeSavedRegs = ActualCalleeSaved;

[oren_ben_simhon]

Implemented comments posted until 01/31

[MatzeB]

include/llvm/CodeGen/RegisterClassInfo.h

56

You hold a register number in an uint8_t with UINT8_MAX==256, don't you?
You should use MCPhysReg (or unsigned if you want to store phys+virtual regs)

[oren_ben_simhon]

Implemented comments submitted until 02/01 (Thank you Matthias).

[oren_ben_simhon]

Please assist in reviewing the patch.

[qcolombet]

Hi,

High level comment: if we are going to use getUpdatedCalleeSavedReg everywhere in place of getCalleeSavedReg, shouldn't we change directly the implementation of getCalleeSavedReg?
I am afraid that having the distinction is going to be confusing and error-prone.

Cheers,
-Quentin

include/llvm/CodeGen/MachineRegisterInfo.h

65

Don't repeat the name of the field in the comment.

72

*contrary

> As opposed to?

[qcolombet]

Erratum: I missed that we switched from TRI to MRI with the new API. I still believe the names are confusing and I would at least expect some comment update on TRI::getCalleeSavedRegs to point out the MRI variant and explain when we should use one or the other.

[MatzeB]

I completely agree with Quentin that we have to change the comment (or even better the name) of TargetRegisterInfo::getUpdatedCalleeSavedRegs() to discourage users to use the "wrong" function.

include/llvm/CodeGen/MachineRegisterInfo.h

220

Maybe disableCalleeSavedRegister() and improve the comment to explain the effects this has for a user (rather than explaining that a bit is set in a private datastructure).

222

Typo: invalif

229–233

Should be a doxygen comment (///)

This seems to be THE way to get a list of callee saved registers now. So at least the brief description should reflect that and avoid words like "updated" or explanations on what is happening internally. I'd expect something like this (the lazy allocation bits seem like an implementation detail that can be left out):

/// Return list of callee saved registers.
///
/// This returns targets default callee saved register list without the registers explicitely disabled by disableCalleeSavedRegister().

include/llvm/CodeGen/RegisterClassInfo.h

56

I didn't follow the code completely, but do we still need an extra SmallVector if there is MachineRegisterInfo::getUpdatedCalleeSavedRegs() that allocates storage itself?

lib/CodeGen/AggressiveAntiDepBreaker.cpp

161–163

The remark in braces is probably not necessary? My understanding is that registers that pass/return arguments simply are not callee saved registers so there shouldn't be a need to mention this here.

lib/CodeGen/CriticalAntiDepBreaker.cpp

69–71

dito.

lib/CodeGen/MachineRegisterInfo.cpp

549–550

Could you move the lazy memory allocation to the addDisableCalleeSavedReg() function so this can function can become const?

There are a bunch of not so nice const_casts now because of this.

lib/Target/X86/X86ISelLowering.cpp

3206–3208

const MachineRegisterInfo &MRI = MF.getRegInfo();
for (std::pair<unsigned,unsigned> P : make_range(MRI.livein_begin(), MRI.livein_end()) { ... }
```?

[oren_ben_simhon]

Implemented comments posted until 2/14 (Thank You Matthias & Quentin).

The main difference from last revision is that i moved the allocation and calculation of the updates CSR list.
The calculation will be performed every time a register is disabled and not when the list is requested.
This way we don't need a const cast when retrieving the CSR list.

[oren_ben_simhon]

Hi Guys, I would kindly like to remind of the review. Thanks in advance.

[aaboud]

include/llvm/CodeGen/MachineRegisterInfo.h

220

Typos:

CSR - Callee Saved Registers.
So, do not write CSR register mask, either say "CSR mask", or "callee saved register mask"

"allocating a the updated lisy" -> "allocating the updated list"

include/llvm/CodeGen/RegisterClassInfo.h

59

should be:

SmallVector<MCPhysReg, 4> CalleeSavedAliases;

In the previous implementation CSRNum[i] was an index into CalleeSaved, and the assumption was that CalleeSaved will not have more than 2^8 = 256 entries.
This is not the case with your new implementation where CalleeSavedAliases[i] is not an index into CalleeSavedRegs, but the actual MCPhysReg.

lib/CodeGen/LivePhysRegs.cpp

181

You may move this down to line 185, just before the line it is used at.

lib/CodeGen/MachineInstr.cpp

267

It is still worth to check if both have the same pointer, in such case we can return true.
Only if we have different pointers we need to check each element.

if(getRegMask() == Other.getRegMask())
  return true;

lib/CodeGen/MachineRegisterInfo.cpp

560

UpdatedCalleeSavedRegs can be of type SmallVector<MCPhysReg, 16>.
And at getUpdatedCalleeSavedRegs just return UpdatedCalleeSavedRegs.data();

In this case you can get rid of all these:

allocateRegisterList
CSRVector
CalleeSaveDisableRegs

Just make sure that last entry in UpdatedCalleeSavedRegs is a zero.
And you can remove registers from this buffer using

UpdatedCalleeSavedRegs.erase(std::remove(UpdatedCalleeSavedRegs.begin(), UpdatedCalleeSavedRegs.end(), Reg)

There will not be many registers to remove, and we only add registers once, at first time we entry this function.

lib/Target/X86/X86ISelLowering.cpp

3723

Initialize this to nullptr.

3748

sink this into the above if-else.
This case, there will be no case where we need to do const_cast, and for sure will not be sending the static "Mask" as "RegMask" to the LowerCallResult function.

[oren_ben_simhon]

Implemented comments posted until 03/11 (Thank You, Amjad).

[aaboud]

LGTM.
Just few minor comments below.

include/llvm/CodeGen/MachineRegisterInfo.h

70

These comment lines are not aligned to 80 characters.
Please, run clang-format on this?

218

Why did you remove the dot at the end of this line?

test/CodeGen/X86/DynamicCalleeSavedRegisters.ll

10

"retuning"-> "returning"
"used" -> "modified"

11

"calle" -> "callee"

14

Can you mentioned how this "hipe CC" function get's its arguments: a0, b0, c0, d0, and e0?

36

Please explain how the parameters a0, b0, c0, d0, and e0 are expected to be passed to this function, i.e., what register is used for each one?

test/CodeGen/X86/sse-regcall.ll

40–43

I believe these tests should be improved, the regular expression is misused in this test.
However, I think it can be done in separate patch.

[oren_ben_simhon]

Implemented comments posted until 03/13 (Thanks Amjad).

[oren_ben_simhon]

Updated the dynamic CSR test.

[aaboud]

The code looks much better now.
Craig, do you have any more comments?
If not, can you approve this patch?

[oren_ben_simhon]

Fixed a typo in dynamic CSR test.

[MatzeB]

include/llvm/CodeGen/MachineRegisterInfo.h

226

Please call this getCalleeSavedRegs() to make it clear that this should be the default function to call now to get the list of callee saved registers!
At the same time please add a sentence in the documentation of TargetRegisterInfo::getCalleeSavedRegs() to state that in most cases people should rather call the version in MachineRegisterInfo.

lib/CodeGen/MachineInstr.cpp

274–278

This can be simplified to
return std::equal(RegMask, RegMask+RegMaskSize, OtherRegMask);

[oren_ben_simhon]

Implemented comments posted by Matthias (Thank you).

[MatzeB]

LGTM.

include/llvm/Target/TargetRegisterInfo.h

429–430 ↗

(On Diff #91603)

How about "This function does not consider calling conventions adapting to parameters; consider using MachineRegisterInfo::getCalleeSavedRegs() instead."?

include/llvm/CodeGen/MachineRegisterInfo.h

private:
   /// VRegInfo - Information we keep for each virtual register.
   ///
   /// Each element in this list contains the register class of the vreg and the
   /// start of the use/def list for the register.
   IndexedMap<std::pair<RegClassOrRegBank, MachineOperand *>,
              VirtReg2IndexFunctor>
       VRegInfo;
 
+  /// CalleeSaveDisableRegs - Lists physical registers that should not be
+  /// preserved.
+  /// Some calling conventions disable CSRs that are passed to the
+  /// function as arguments or are returned by the function.
+  BitVector CalleeSaveDisableRegs;
+
+  /// AllocatedRegMasks - Lists reg masks that were allocated by the machine
+  /// register info. It includes all the register masks from the call
+  /// instructions in the current function.
+  /// It is the responsibilty of the machine register info to allocate and
+  /// free the memory.
+  std::vector<uint32_t *> AllocatedRegMasks;
+
   /// RegAllocHints - This vector records register allocation hints for virtual
   /// registers. For each virtual register, it keeps a register and hint type
   /// pair making up the allocation hint. Hint type is target specific except
   /// for the value 0 which means the second value of the pair is the preferred
   /// register for allocation. For example, if the hint is <0, 1024>, it means
   /// the allocator should prefer the physical register allocated to the virtual
   /// register of the hint.
   IndexedMap<std::pair<unsigned, unsigned>, VirtReg2IndexFunctor> RegAllocHints;
private:
   /// second element.
   std::vector<std::pair<unsigned, unsigned> > LiveIns;
 
   MachineRegisterInfo(const MachineRegisterInfo&) = delete;
   void operator=(const MachineRegisterInfo&) = delete;
 public:
   explicit MachineRegisterInfo(MachineFunction *MF);
 
+  /// Destructor - deallocate the register masks.
+  ~MachineRegisterInfo() {
+    for (auto RegMask : AllocatedRegMasks)
+      delete[] RegMask;
+  }
+
   const TargetRegisterInfo *getTargetRegisterInfo() const {
     return MF->getSubtarget().getRegisterInfo();
   }
 
   void resetDelegate(Delegate *delegate) {
     // Ensure another delegate does not take over unless the current
     // delegate first unattaches itself. If we ever need to multicast
     // notifications, we will need to change to using a list.
public:
   bool subRegLivenessEnabled() const {
     return TracksSubRegLiveness;
   }
 
   //===--------------------------------------------------------------------===//
   // Register Info
   //===--------------------------------------------------------------------===//
 
+  /// Sets a new register (and its aliases) in the CalleeSaveDisableRegs bit vector.
+  void addCalleeSaveDisableRegister(unsigned Reg) {
+    assert(Reg < CalleeSaveDisableRegs.size() &&
+           "Trying to disable an invalif register");
+    const TargetRegisterInfo * TRI = MF->getSubtarget().getRegisterInfo();
+    for (MCSubRegIterator SubRegs(Reg, TRI, /*IncludeSelf=*/true);
+         SubRegs.isValid(); ++SubRegs)
+      CalleeSaveDisableRegs.set(*SubRegs);
+  }
+
+  /// \Returns a bit vector of registers that should not be
+  /// callee saved.
+  const BitVector &getCalleeSaveDisabledRegs() const {
+    return CalleeSaveDisableRegs;
+  }
+
+  /// Flips the CalleeSaveDisableRegs and \returnes a copy of
+  /// the flipped bit vector.
+  BitVector getCalleeSaveEnabledRegs() const {
+    BitVector CalleeSaveEnabledRegs = CalleeSaveDisableRegs;
+    return CalleeSaveEnabledRegs.flip();
+  }
+
+  /// Create a new Reg Mask by copying the \p Mask.
+  /// \returns the alloctaed Reg Mask.
+  uint32_t *AllocateNewRegMask(const uint32_t *Mask) {
+    assert(Mask && "An invalid Mask pointer was passed");
+    const TargetRegisterInfo *TRI = MF->getSubtarget().getRegisterInfo();
+
+    // Calculate the size of the RegMask.
+    unsigned RegMaskSize = (TRI->getNumRegs() + 31) / 32;
+
+    // Allocate a new Reg Mask and copy Mask.
+    if (uint32_t *RegMask = new uint32_t[RegMaskSize]) {
+      memcpy(RegMask, Mask, sizeof(uint32_t) * RegMaskSize);
+      AllocatedRegMasks.push_back(RegMask);
+      return RegMask;
+    }
+
+    // Failed allocating a register mask.
+    return nullptr;
+  }
+
   // Strictly for use by MachineInstr.cpp.
   void addRegOperandToUseList(MachineOperand *MO);
 
   // Strictly for use by MachineInstr.cpp.
   void removeRegOperandFromUseList(MachineOperand *MO);
 
   // Strictly for use by MachineInstr.cpp.
   void moveOperands(MachineOperand *Dst, MachineOperand *Src, unsigned NumOps);

include/llvm/CodeGen/RegisterClassInfo.h

class RegisterClassInfo {
   // Tag changes whenever cached information needs to be recomputed. An RCInfo
   // entry is valid when its tag matches.
   unsigned Tag;
 
   const MachineFunction *MF;
   const TargetRegisterInfo *TRI;
 
   // Callee saved registers of last MF. Assumed to be valid until the next
-  // runOnFunction() call.
-  const MCPhysReg *CalleeSaved;
+  // runOnFunction() call. Used only to determine if an update was made.
+  SmallVector<uint8_t, 16> CalleeSavedRegs;
 
-  // Map register number to CalleeSaved index + 1;
-  SmallVector<uint8_t, 4> CSRNum;
+  // Map register alias to the callee saved Register.
+  SmallVector<uint8_t, 4> CalleeSavedAliases;
 
   // Reserved registers in the current MF.
   BitVector Reserved;
 
   std::unique_ptr<unsigned[]> PSetLimits;
 
   // Compute all information about RC.
   void compute(const TargetRegisterClass *RC) const;
public:
   /// Register classes like GR32_NOSP are not proper sub-classes because %esp
   /// is not allocatable.  Similarly, tGPR is not a proper sub-class in Thumb
   /// mode because the GPR super-class is not legal.
   bool isProperSubClass(const TargetRegisterClass *RC) const {
     return get(RC).ProperSubClass;
   }
 
   /// getLastCalleeSavedAlias - Returns the last callee saved register that
-  /// overlaps PhysReg, or 0 if Reg doesn't overlap a CSR.
+  /// overlaps PhysReg, or 0 if Reg doesn't overlap a CalleeSavedAliases.
   unsigned getLastCalleeSavedAlias(unsigned PhysReg) const {
     assert(TargetRegisterInfo::isPhysicalRegister(PhysReg));
-    if (unsigned N = CSRNum[PhysReg])
-      return CalleeSaved[N-1];
+    if (PhysReg < CalleeSavedAliases.size())
+      return CalleeSavedAliases[PhysReg];
     return 0;
   }
 
   /// Get the minimum register cost in RC's allocation order.
   /// This is the smallest value returned by TRI->getCostPerUse(Reg) for all
   /// the registers in getOrder(RC).
   unsigned getMinCost(const TargetRegisterClass *RC) {
     return get(RC).MinCost;

lib/CodeGen/AggressiveAntiDepBreaker.cpp

for (const auto &LI : (*SI)->liveins()) {
       for (MCRegAliasIterator AI(LI.PhysReg, TRI, true); AI.isValid(); ++AI) {
         unsigned Reg = *AI;
         State->UnionGroups(Reg, 0);
         KillIndices[Reg] = BB->size();
         DefIndices[Reg] = ~0u;
       }
     }
 
-  // Mark live-out callee-saved registers. In a return block this is
+  // Mark live-out callee-saved registers (that are not
+  // passed/returned as arguments). In a return block this is
   // all callee-saved registers. In non-return this is any
   // callee-saved register that is not saved in the prolog.
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   BitVector Pristine = MFI.getPristineRegs(MF);
+  BitVector CSRDisabled = MF.getRegInfo().getCalleeSaveDisabledRegs();
   for (const MCPhysReg *I = TRI->getCalleeSavedRegs(&MF); *I; ++I) {
     unsigned Reg = *I;
+    if (CSRDisabled.test(Reg)) continue;
     if (!IsReturnBlock && !Pristine.test(Reg)) continue;
     for (MCRegAliasIterator AI(Reg, TRI, true); AI.isValid(); ++AI) {
       unsigned AliasReg = *AI;
       State->UnionGroups(AliasReg, 0);
       KillIndices[AliasReg] = BB->size();
       DefIndices[AliasReg] = ~0u;
     }
   }

lib/CodeGen/CriticalAntiDepBreaker.cpp

for (const auto &LI : (*SI)->liveins()) {
       for (MCRegAliasIterator AI(LI.PhysReg, TRI, true); AI.isValid(); ++AI) {
         unsigned Reg = *AI;
         Classes[Reg] = reinterpret_cast<TargetRegisterClass *>(-1);
         KillIndices[Reg] = BBSize;
         DefIndices[Reg] = ~0u;
       }
     }
 
-  // Mark live-out callee-saved registers. In a return block this is
+  // Mark live-out callee-saved registers (that are not
+  // passed/returned as arguments). In a return block this is
   // all callee-saved registers. In non-return this is any
   // callee-saved register that is not saved in the prolog.
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   BitVector Pristine = MFI.getPristineRegs(MF);
+  BitVector CSRDisabled = MF.getRegInfo().getCalleeSaveDisabledRegs();
   for (const MCPhysReg *I = TRI->getCalleeSavedRegs(&MF); *I; ++I) {
+    if (CSRDisabled.test(*I)) continue;
     if (!IsReturnBlock && !Pristine.test(*I)) continue;
     for (MCRegAliasIterator AI(*I, TRI, true); AI.isValid(); ++AI) {
       unsigned Reg = *AI;
       Classes[Reg] = reinterpret_cast<TargetRegisterClass *>(-1);
       KillIndices[Reg] = BBSize;
       DefIndices[Reg] = ~0u;
     }
   }

lib/CodeGen/LivePhysRegs.cpp

void LivePhysRegs::addBlockLiveIns(const MachineBasicBlock &MBB) {
   }
 }
 
 /// Add pristine registers to the given \p LiveRegs. This function removes
 /// actually saved callee save registers when \p InPrologueEpilogue is false.
 static void addPristines(LivePhysRegs &LiveRegs, const MachineFunction &MF,
                          const MachineFrameInfo &MFI,
                          const TargetRegisterInfo &TRI) {
+  BitVector CSRDisabled = MF.getRegInfo().getCalleeSaveDisabledRegs();
   for (const MCPhysReg *CSR = TRI.getCalleeSavedRegs(&MF); CSR && *CSR; ++CSR)
+    if (!CSRDisabled.test(*CSR))
       LiveRegs.addReg(*CSR);
+
   for (const CalleeSavedInfo &Info : MFI.getCalleeSavedInfo())
     LiveRegs.removeReg(Info.getReg());
 }
 
 void LivePhysRegs::addLiveOutsNoPristines(const MachineBasicBlock &MBB) {
   // To get the live-outs we simply merge the live-ins of all successors.
   for (const MachineBasicBlock *Succ : MBB.successors())
     addBlockLiveIns(*Succ);
 }
 
 void LivePhysRegs::addLiveOuts(const MachineBasicBlock &MBB) {
   const MachineFunction &MF = *MBB.getParent();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   if (MFI.isCalleeSavedInfoValid()) {
     if (MBB.isReturnBlock()) {
       // The return block has no successors whose live-ins we could merge
       // below. So instead we add the callee saved registers manually.
+      BitVector CSRDisabled = MF.getRegInfo().getCalleeSaveDisabledRegs();
       for (const MCPhysReg *I = TRI->getCalleeSavedRegs(&MF); *I; ++I)
+        if (!CSRDisabled.test(*I))
           addReg(*I);
     } else {
       addPristines(*this, MF, MFI, *TRI);
     }
   }
 
   addLiveOutsNoPristines(MBB);
 }
 

lib/CodeGen/MachineFunction.cpp

BitVector MachineFrameInfo::getPristineRegs(const MachineFunction &MF) const {
   // Before CSI is calculated, no registers are considered pristine. They can be
   // freely used and PEI will make sure they are saved.
   if (!isCalleeSavedInfoValid())
     return BV;
 
   for (const MCPhysReg *CSR = TRI->getCalleeSavedRegs(&MF); CSR && *CSR; ++CSR)
     BV.set(*CSR);
 
+  // Reset all callee save disabled registers
+  BV &= MF.getRegInfo().getCalleeSaveEnabledRegs();
+
   // Saved CSRs are not pristine.
   for (auto &I : getCalleeSavedInfo())
     for (MCSubRegIterator S(I.getReg(), TRI, true); S.isValid(); ++S)
       BV.reset(*S);
 
   return BV;
 }
 

lib/CodeGen/MachineInstr.cpp

case MachineOperand::MO_GlobalAddress:
     return getGlobal() == Other.getGlobal() && getOffset() == Other.getOffset();
   case MachineOperand::MO_ExternalSymbol:
     return !strcmp(getSymbolName(), Other.getSymbolName()) &&
            getOffset() == Other.getOffset();
   case MachineOperand::MO_BlockAddress:
     return getBlockAddress() == Other.getBlockAddress() &&
            getOffset() == Other.getOffset();
   case MachineOperand::MO_RegisterMask:
-  case MachineOperand::MO_RegisterLiveOut:
-    return getRegMask() == Other.getRegMask();
+  case MachineOperand::MO_RegisterLiveOut: {
+    // Calculate the size of the RegMask
+    const MachineFunction *MF = getParent()->getParent()->getParent();
+    const TargetRegisterInfo *TRI = MF->getSubtarget().getRegisterInfo();
+    unsigned RegMaskSize = (TRI->getNumRegs() + 31) / 32;
+
+    // Compare the two RegMasks
+    const uint32_t *RegMask = getRegMask();
+    const uint32_t *OtherRegMask = Other.getRegMask();
+    for (unsigned I = 0, E = RegMaskSize; I < E; ++I)
+      if (RegMask[I] != OtherRegMask[I])
+        return false;
+
+    return true;
+  }
   case MachineOperand::MO_MCSymbol:
     return getMCSymbol() == Other.getMCSymbol();
   case MachineOperand::MO_CFIIndex:
     return getCFIIndex() == Other.getCFIIndex();
   case MachineOperand::MO_Metadata:
     return getMetadata() == Other.getMetadata();
   case MachineOperand::MO_IntrinsicID:
     return getIntrinsicID() == Other.getIntrinsicID();

lib/CodeGen/MachineRegisterInfo.cpp

 MachineRegisterInfo::MachineRegisterInfo(MachineFunction *MF)
     : MF(MF), TheDelegate(nullptr),
       TracksSubRegLiveness(MF->getSubtarget().enableSubRegLiveness() &&
                            EnableSubRegLiveness) {
   unsigned NumRegs = getTargetRegisterInfo()->getNumRegs();
   VRegInfo.reserve(256);
   RegAllocHints.reserve(256);
   UsedPhysRegMask.resize(NumRegs);
+  CalleeSaveDisableRegs.resize(NumRegs);
   PhysRegUseDefLists.reset(new MachineOperand*[NumRegs]());
 }
 
 /// setRegClass - Set the register class of the specified virtual register.
 ///
 void
 MachineRegisterInfo::setRegClass(unsigned Reg, const TargetRegisterClass *RC) {
   assert(RC && RC->isAllocatable() && "Invalid RC for virtual register");
if (UsedPhysRegMask.test(PhysReg))
     return true;
   const TargetRegisterInfo *TRI = getTargetRegisterInfo();
   for (MCRegAliasIterator AliasReg(PhysReg, TRI, true); AliasReg.isValid();
        ++AliasReg) {
     if (!reg_nodbg_empty(*AliasReg))
       return true;
   }
   return false;
 }

lib/CodeGen/RegAllocPBQP.cpp

if (!LI.empty()) {
       EmptyIntervalVRegs.insert(LI.reg);
     }
   }
 }
 
 static bool isACalleeSavedRegister(unsigned reg, const TargetRegisterInfo &TRI,
                                    const MachineFunction &MF) {
   const MCPhysReg *CSR = TRI.getCalleeSavedRegs(&MF);
+  BitVector CSRDisabled = MF.getRegInfo().getCalleeSaveDisabledRegs();
   for (unsigned i = 0; CSR[i] != 0; ++i)
-    if (TRI.regsOverlap(reg, CSR[i]))
+    if (!CSRDisabled.test(CSR[i]) && TRI.regsOverlap(reg, CSR[i]))
       return true;
   return false;
 }
 
 void RegAllocPBQP::initializeGraph(PBQPRAGraph &G, VirtRegMap &VRM,
                                    Spiller &VRegSpiller) {
   MachineFunction &MF = G.getMetadata().MF;
 

lib/CodeGen/RegisterClassInfo.cpp

 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/RegisterClassInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
+#include "llvm/Target/TargetFrameLowering.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "regalloc"
 
 static cl::opt<unsigned>
 StressRA("stress-regalloc", cl::Hidden, cl::init(0), cl::value_desc("N"),
          cl::desc("Limit all regclasses to N registers"));
 
-RegisterClassInfo::RegisterClassInfo()
-  : Tag(0), MF(nullptr), TRI(nullptr), CalleeSaved(nullptr) {}
+RegisterClassInfo::RegisterClassInfo() : Tag(0), MF(nullptr), TRI(nullptr) {}
 
 void RegisterClassInfo::runOnMachineFunction(const MachineFunction &mf) {
   bool Update = false;
   MF = &mf;
 
   // Allocate new array the first time we see a new target.
   if (MF->getSubtarget().getRegisterInfo() != TRI) {
     TRI = MF->getSubtarget().getRegisterInfo();
     RegClass.reset(new RCInfo[TRI->getNumRegClasses()]);
     unsigned NumPSets = TRI->getNumRegPressureSets();
     PSetLimits.reset(new unsigned[NumPSets]);
     std::fill(&PSetLimits[0], &PSetLimits[NumPSets], 0);
     Update = true;
   }
 
   // Does this MF have different CSRs?
   assert(TRI && "no register info set");
+
+  // Calculate the callee saved registers.
   const MCPhysReg *CSR = TRI->getCalleeSavedRegs(MF);
-  if (Update || CSR != CalleeSaved) {
-    // Build a CSRNum map. Every CSR alias gets an entry pointing to the last
+  SmallVector<uint8_t, 16> ActualCalleeSaved;
+  for (unsigned I = 0; unsigned Reg = CSR[I]; ++I) {
+    if (MF->getRegInfo().getCalleeSaveDisabledRegs().test(Reg))
+      continue;
+    ActualCalleeSaved.push_back(Reg);
+  }
+
+  if (Update || ActualCalleeSaved != CalleeSavedRegs) {
+    // Build a CSRAlias map. Every CSR alias saves the last
     // overlapping CSR.
-    CSRNum.clear();
-    CSRNum.resize(TRI->getNumRegs(), 0);
-    for (unsigned N = 0; unsigned Reg = CSR[N]; ++N)
+    CalleeSavedAliases.resize(TRI->getNumRegs(), 0);
+    for (auto Reg : ActualCalleeSaved)
       for (MCRegAliasIterator AI(Reg, TRI, true); AI.isValid(); ++AI)
-        CSRNum[*AI] = N + 1; // 0 means no CSR, 1 means CalleeSaved[0], ...
+        CalleeSavedAliases[*AI] = Reg;
     Update = true;
+    CalleeSavedRegs = ActualCalleeSaved;
   }
-  CalleeSaved = CSR;
 
   // Different reserved registers?
   const BitVector &RR = MF->getRegInfo().getReservedRegs();
   if (Reserved.size() != RR.size() || RR != Reserved) {
     Update = true;
     Reserved = RR;
   }
 
void RegisterClassInfo::compute(const TargetRegisterClass *RC) const {
   for (unsigned i = 0; i != RawOrder.size(); ++i) {
     unsigned PhysReg = RawOrder[i];
     // Remove reserved registers from the allocation order.
     if (Reserved.test(PhysReg))
       continue;
     unsigned Cost = TRI->getCostPerUse(PhysReg);
     MinCost = std::min(MinCost, Cost);
 
-    if (CSRNum[PhysReg])
+    if (CalleeSavedAliases[PhysReg])
       // PhysReg aliases a CSR, save it for later.
       CSRAlias.push_back(PhysReg);
     else {
       if (Cost != LastCost)
         LastCostChange = N;
       RCI.Order[N++] = PhysReg;
       LastCost = Cost;
     }

lib/CodeGen/TargetFrameLoweringImpl.cpp

void TargetFrameLowering::determineCalleeSaves(MachineFunction &MF,
 
   // In Naked functions we aren't going to save any registers.
   if (MF.getFunction()->hasFnAttribute(Attribute::Naked))
     return;
 
   // Functions which call __builtin_unwind_init get all their registers saved.
   bool CallsUnwindInit = MF.callsUnwindInit();
   const MachineRegisterInfo &MRI = MF.getRegInfo();
+  BitVector CSRDisabled = MRI.getCalleeSaveDisabledRegs();
   for (unsigned i = 0; CSRegs[i]; ++i) {
     unsigned Reg = CSRegs[i];
+    if (CSRDisabled.test(Reg)) continue;
     if (CallsUnwindInit || MRI.isPhysRegModified(Reg))
       SavedRegs.set(Reg);
   }
 }
 
 unsigned TargetFrameLowering::getStackAlignmentSkew(
     const MachineFunction &MF) const {
   // When HHVM function is called, the stack is skewed as the return address
   // is removed from the stack before we enter the function.
   if (LLVM_UNLIKELY(MF.getFunction()->getCallingConv() == CallingConv::HHVM))
     return MF.getTarget().getPointerSize();
 
   return 0;
 }

lib/Target/X86/X86ISelLowering.h

private:
     void addLegalFPImmediate(const APFloat& Imm) {
       LegalFPImmediates.push_back(Imm);
     }
 
     SDValue LowerCallResult(SDValue Chain, SDValue InFlag,
                             CallingConv::ID CallConv, bool isVarArg,
                             const SmallVectorImpl<ISD::InputArg> &Ins,
                             const SDLoc &dl, SelectionDAG &DAG,
-                            SmallVectorImpl<SDValue> &InVals) const;
+                            SmallVectorImpl<SDValue> &InVals,
+                            uint32_t *RegMask) const;
     SDValue LowerMemArgument(SDValue Chain, CallingConv::ID CallConv,
                              const SmallVectorImpl<ISD::InputArg> &ArgInfo,
                              const SDLoc &dl, SelectionDAG &DAG,
                              const CCValAssign &VA, MachineFrameInfo &MFI,
                              unsigned i) const;
     SDValue LowerMemOpCallTo(SDValue Chain, SDValue StackPtr, SDValue Arg,
                              const SDLoc &dl, SelectionDAG &DAG,
                              const CCValAssign &VA,

lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::LowerReturn(SDValue Chain, CallingConv::ID CallConv,
   RetOps.push_back(DAG.getTargetConstant(FuncInfo->getBytesToPopOnReturn(), dl,
                    MVT::i32));
 
   // Copy the result values into the output registers.
   for (unsigned I = 0, OutsIndex = 0, E = RVLocs.size(); I != E;
        ++I, ++OutsIndex) {
     CCValAssign &VA = RVLocs[I];
     assert(VA.isRegLoc() && "Can only return in registers!");
+
+    // Add the register to the CalleeSaveDisableRegs list.
+    if (CallConv == CallingConv::X86_RegCall)
+      MF.getRegInfo().addCalleeSaveDisableRegister(VA.getLocReg());
+
     SDValue ValToCopy = OutVals[OutsIndex];
     EVT ValVT = ValToCopy.getValueType();
 
     // Promote values to the appropriate types.
     if (VA.getLocInfo() == CCValAssign::SExt)
       ValToCopy = DAG.getNode(ISD::SIGN_EXTEND, dl, VA.getLocVT(), ValToCopy);
     else if (VA.getLocInfo() == CCValAssign::ZExt)
       ValToCopy = DAG.getNode(ISD::ZERO_EXTEND, dl, VA.getLocVT(), ValToCopy);
if (VA.needsCustom()) {
       assert(VA.getValVT() == MVT::v64i1 &&
              "Currently the only custom case is when we split v64i1 to 2 regs");
 
       Passv64i1ArgInRegs(dl, DAG, Chain, ValToCopy, RegsToPass, VA, RVLocs[++I],
                          Subtarget);
 
       assert(2 == RegsToPass.size() &&
              "Expecting two registers after Pass64BitArgInRegs");
+
+      // Add the second register to the CalleeSaveDisableRegs list.
+      if (CallConv == CallingConv::X86_RegCall)
+        MF.getRegInfo().addCalleeSaveDisableRegister(RVLocs[I].getLocReg());
     } else {
       RegsToPass.push_back(std::make_pair(VA.getLocReg(), ValToCopy));
     }
 
     // Add nodes to the DAG and add the values into the RetOps list
     for (auto &Reg : RegsToPass) {
       Chain = DAG.getCopyToReg(Chain, dl, Reg.first, Reg.second, Flag);
       Flag = Chain.getValue(1);
unsigned RetValReg
         = (Subtarget.is64Bit() && !Subtarget.isTarget64BitILP32()) ?
           X86::RAX : X86::EAX;
     Chain = DAG.getCopyToReg(Chain, dl, RetValReg, Val, Flag);
     Flag = Chain.getValue(1);
 
     // RAX/EAX now acts like a return value.
     RetOps.push_back(
         DAG.getRegister(RetValReg, getPointerTy(DAG.getDataLayout())));
+
+    // Add the returned register to the CalleeSaveDisableRegs list.
+    if (CallConv == CallingConv::X86_RegCall)
+      MF.getRegInfo().addCalleeSaveDisableRegister(RetValReg);
   }
 
   const X86RegisterInfo *TRI = Subtarget.getRegisterInfo();
   const MCPhysReg *I =
       TRI->getCalleeSavedRegsViaCopy(&DAG.getMachineFunction());
   if (I) {
     for (; *I; ++I) {
       if (X86::GR64RegClass.contains(*I))

 }
 
 /// Lower the result values of a call into the
 /// appropriate copies out of appropriate physical registers.
 ///
 SDValue X86TargetLowering::LowerCallResult(
     SDValue Chain, SDValue InFlag, CallingConv::ID CallConv, bool isVarArg,
     const SmallVectorImpl<ISD::InputArg> &Ins, const SDLoc &dl,
-    SelectionDAG &DAG, SmallVectorImpl<SDValue> &InVals) const {
+    SelectionDAG &DAG, SmallVectorImpl<SDValue> &InVals,
+    uint32_t *RegMask) const {
 
+  const TargetRegisterInfo *TRI = Subtarget.getRegisterInfo();
   // Assign locations to each value returned by this call.
   SmallVector<CCValAssign, 16> RVLocs;
   bool Is64Bit = Subtarget.is64Bit();
   CCState CCInfo(CallConv, isVarArg, DAG.getMachineFunction(), RVLocs,
                  *DAG.getContext());
   CCInfo.AnalyzeCallResult(Ins, RetCC_X86);
 
   // Copy all of the result registers out of their specified physreg.
   for (unsigned I = 0, InsIndex = 0, E = RVLocs.size(); I != E;
        ++I, ++InsIndex) {
     CCValAssign &VA = RVLocs[I];
     EVT CopyVT = VA.getLocVT();
 
+    // In some calling conventions we need to remove the used registers
+    // from the register mask.
+    if (RegMask && CallConv == CallingConv::X86_RegCall) {
+      for (MCSubRegIterator SubRegs(VA.getLocReg(), TRI, /*IncludeSelf=*/true);
+           SubRegs.isValid(); ++SubRegs)
+        RegMask[*SubRegs / 32] &= ~(1u << (*SubRegs % 32));
+    }
+
     // If this is x86-64, and we disabled SSE, we can't return FP values
     if ((CopyVT == MVT::f32 || CopyVT == MVT::f64 || CopyVT == MVT::f128) &&
         ((Is64Bit || Ins[InsIndex].Flags.isInReg()) && !Subtarget.hasSSE1())) {
       report_fatal_error("SSE register return with SSE disabled");
     }
 
     // If we prefer to use the value in xmm registers, copy it out as f80 and
     // use a truncate to move it from fp stack reg to xmm reg.
SDValue X86TargetLowering::LowerFormalArguments(
 
   SDValue ArgValue;
   for (unsigned I = 0, InsIndex = 0, E = ArgLocs.size(); I != E;
        ++I, ++InsIndex) {
     assert(InsIndex < Ins.size() && "Invalid Ins index");
     CCValAssign &VA = ArgLocs[I];
 
     if (VA.isRegLoc()) {
+      // Add the register to the CalleeSaveDisableRegs list.
+      if (CallConv == CallingConv::X86_RegCall)
+        MF.getRegInfo().addCalleeSaveDisableRegister(VA.getLocReg());
+
       EVT RegVT = VA.getLocVT();
       if (VA.needsCustom()) {
         assert(
             VA.getValVT() == MVT::v64i1 &&
             "Currently the only custom case is when we split v64i1 to 2 regs");
 
         // v64i1 values, in regcall calling convention, that are
         // compiled to 32 bit arch, are splited up into two registers.
         ArgValue =
             getv64i1Argument(VA, ArgLocs[++I], Chain, DAG, dl, Subtarget);
+
+        // Add the second register to the CalleeSaveDisableRegs list.
+        if (CallConv == CallingConv::X86_RegCall)
+          MF.getRegInfo().addCalleeSaveDisableRegister(ArgLocs[I].getLocReg());
       } else {
         const TargetRegisterClass *RC;
         if (RegVT == MVT::i32)
           RC = &X86::GR32RegClass;
         else if (Is64Bit && RegVT == MVT::i64)
           RC = &X86::GR64RegClass;
         else if (RegVT == MVT::f32)
           RC = Subtarget.hasAVX512() ? &X86::FR32XRegClass : &X86::FR32RegClass;
if (Personality == EHPersonality::CoreCLR) {
       // same, so the size of funclets' (mostly empty) frames is dictated by
       // how far this slot is from the bottom (since they allocate just enough
       // space to accommodate holding this slot at the correct offset).
       int PSPSymFI = MFI.CreateStackObject(8, 8, /*isSS=*/false);
       EHInfo->PSPSymFrameIdx = PSPSymFI;
     }
   }
 
   return Chain;
 }
 
 SDValue X86TargetLowering::LowerMemOpCallTo(SDValue Chain, SDValue StackPtr,
                                             SDValue Arg, const SDLoc &dl,
                                             SelectionDAG &DAG,
                                             const CCValAssign &VA,
                                             ISD::ArgFlagsTy Flags) const {
   unsigned LocMemOffset = VA.getLocMemOffset();
   SDValue PtrOff = DAG.getIntPtrConstant(LocMemOffset, dl);
   PtrOff = DAG.getNode(ISD::ADD, dl, getPointerTy(DAG.getDataLayout()),
if (!Is64Bit && CLI.CS && CLI.CS->isInvoke()) {
     EHPersonality Pers =
         CallerFn->hasPersonalityFn()
             ? classifyEHPersonality(CallerFn->getPersonalityFn())
             : EHPersonality::Unknown;
     if (isFuncletEHPersonality(Pers))
       Mask = RegInfo->getNoPreservedMask();
   }
 
-  Ops.push_back(DAG.getRegisterMask(Mask));
+  // Define a new register mask from the existing mask.
+  uint32_t *RegMask;
+
+  // In some calling conventions we need to remove the used physical registers
+  // from the reg mask.
+  if (CallConv == CallingConv::X86_RegCall) {
+    /// @todo Need to add support in MIPrinter and MIParser to represent
+    ///       the customed RegMask.
+
+    // Allocating a new RegMask and save it in the machine function.
+    RegMask = MF.getRegInfo().AllocateNewRegMask(Mask);
+    const TargetRegisterInfo *TRI = Subtarget.getRegisterInfo();
+
+    // Make sure all sub registers of the argument registers are reset
+    // in the RegMask.
+    for (auto const &RegPair : RegsToPass)
+      for (MCSubRegIterator SubRegs(RegPair.first, TRI, /*IncludeSelf=*/true);
+           SubRegs.isValid(); ++SubRegs)
+        RegMask[*SubRegs / 32] &= ~(1u << (*SubRegs % 32));
+  } else
+    // Simply use the same mask instead of allocatign a new one.
+    RegMask = const_cast<uint32_t *>(Mask);
+
+  // Create the RegMask Operand according to our updated mask.
+  Ops.push_back(DAG.getRegisterMask(RegMask));
 
   if (InFlag.getNode())
     Ops.push_back(InFlag);
 
   if (isTailCall) {
     // We used to do:
     //// If this is the first return lowered for this function, add the regs
     //// to the liveout set for the function.
     // This isn't right, although it's probably harmless on x86; liveouts
     // should be computed from returns not tail calls.  Consider a void
Chain = DAG.getCALLSEQ_END(Chain,
                                DAG.getIntPtrConstant(NumBytesForCalleeToPop, dl,
                                                      true),
                                InFlag, dl);
     InFlag = Chain.getValue(1);
   }
 
   // Handle result values, copying them out of physregs into vregs that we
   // return.
-  return LowerCallResult(Chain, InFlag, CallConv, isVarArg,
-                         Ins, dl, DAG, InVals);
+  return LowerCallResult(Chain, InFlag, CallConv, isVarArg, Ins, dl, DAG,
+                         InVals, RegMask);
 }
 
 //===----------------------------------------------------------------------===//
 //                Fast Calling Convention (tail call) implementation
 //===----------------------------------------------------------------------===//
 
 //  Like std call, callee cleans arguments, convention except that ECX is
 //  reserved for storing the tail called function address. Only 2 registers are
for (auto &II : reverse(*MBB)) {
       DenseMap<unsigned, bool> DefRegs;
       for (auto &MOp : II.operands())
         if (MOp.isReg())
           DefRegs[MOp.getReg()] = true;
 
       MachineInstrBuilder MIB(*MF, &II);
       for (unsigned RI = 0; SavedRegs[RI]; ++RI) {
         unsigned Reg = SavedRegs[RI];
+        if (MRI->getCalleeSaveDisabledRegs().test(Reg)) continue;
         if (!DefRegs[Reg])
           MIB.addReg(Reg, RegState::ImplicitDefine | RegState::Dead);
       }
 
       break;
     }
   }
 

lib/Target/X86/X86MachineFunctionInfo.cpp

 //===-- X86MachineFunctionInfo.cpp - X86 machine function info ------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "X86MachineFunctionInfo.h"
 #include "X86RegisterInfo.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/Target/TargetSubtargetInfo.h"
 
 using namespace llvm;
 
 void X86MachineFunctionInfo::anchor() { }
 
 void X86MachineFunctionInfo::setRestoreBasePointer(const MachineFunction *MF) {
   if (!RestoreBasePointerOffset) {
     const X86RegisterInfo *RegInfo = static_cast<const X86RegisterInfo *>(
       MF->getSubtarget().getRegisterInfo());
     unsigned SlotSize = RegInfo->getSlotSize();
     for (const MCPhysReg *CSR =
       RegInfo->X86RegisterInfo::getCalleeSavedRegs(MF);
       unsigned Reg = *CSR;
        ++CSR)
     {
+      if (MF->getRegInfo().getCalleeSaveDisabledRegs().test(Reg))
+        continue;
       if (X86::GR64RegClass.contains(Reg) || X86::GR32RegClass.contains(Reg))
         RestoreBasePointerOffset -= SlotSize;
     }
   }
 }
 

test/CodeGen/X86/DynamicCalleeSavedRegisters.ll

+; RUN: llc < %s -mtriple=i386-linux-gnu  | FileCheck --check-prefix=CHECK %s
+
+define x86_regcallcc {i32, i32, i32} @test_callee(i32 %a0, i32 %b0, i32 %c0, i32 %d0, i32 %e0) nounwind {
+  %b1 = mul i32 7, %e0
+  %b2 = udiv i32 5, %e0
+  %b3 = mul i32 7, %d0
+  %b4 = insertvalue {i32, i32, i32} undef, i32 %b1, 0
+  %b5 = insertvalue {i32, i32, i32} %b4, i32 %b2, 1
+  %b6 = insertvalue {i32, i32, i32} %b5, i32 %b3, 2
+  ret {i32, i32, i32} %b6
+}
+
+declare x86_regcallcc i32 @test_caller(i32 %a0, i32 %b0, i32 %c0, i32 %d0, i32 %e0);
+
+; This is a hipe CC function that doesn't save any register for the caller of the caller
+; In this case we see that %edi and %esi are not saved and %esi is assumed to be saved by test function
+define cc 11 i32  @caller(i32 %a0) nounwind {
+  %b1 = call x86_regcallcc i32 @test_caller(i32 %a0, i32 %a0, i32 %a0, i32 %a0, i32 %a0)
+  %b2 = add i32 %b1, %a0
+  ret i32 %b2
+}
+; CHECK-LABEL: caller
+; CHECK:       movl  %esi, %ebx
+; CHECK:       calll  test
+; CHECK:       movl  %ebx, %esi
+; CHECK:       ret{{l|q}}
+
+!hipe.literals = !{ !0, !1, !2 }
+!0 = !{ !"P_NSP_LIMIT", i32 120 }
+!1 = !{ !"X86_LEAF_WORDS", i32 24 }
+!2 = !{ !"AMD64_LEAF_WORDS", i32 18 }

test/CodeGen/X86/avx512-regcall-NoMask.ll

 define x86_regcallcc <32 x float> @testf32_inp(<32 x float> %a, <32 x float> %b, <32 x float> %c) nounwind {
   %x1 = fadd <32 x float> %a, %b
   %x2 = fmul <32 x float> %a, %b
   %x3 = fsub <32 x float> %x1, %x2
   %x4 = fadd <32 x float> %x3, %c
   ret <32 x float> %x4
 }
 
-; X32-LABEL: pushl {{%e(si|di|bx|bp)}}
-; X32: pushl {{%e(si|di|bx|bp)}}
-; X32: pushl {{%e(si|di|bx|bp)}}
-; X32: pushl {{%e(si|di|bx|bp)}}
-; X32: popl {{%e(si|di|bx|bp)}}
-; X32: popl {{%e(si|di|bx|bp)}}
-; X32: popl {{%e(si|di|bx|bp)}}
-; X32: popl {{%e(si|di|bx|bp)}}
+; X32-LABEL: testi32_inp
+; X32: pushl {{%e(bx|bp)}}
+; X32: pushl {{%e(bx|bp)}}
+; X32: popl {{%e(bx|bp)}}
+; X32: popl {{%e(bx|bp)}}
 ; X32: retl
 
-; WIN64-LABEL: pushq  {{%r(bp|bx|1[0-5])}}
+; WIN64-LABEL: testi32_inp
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
 ; WIN64: popq {{%r(bp|bx|1[0-5])}}
 ; WIN64: popq {{%r(bp|bx|1[0-5])}}
 ; WIN64: popq {{%r(bp|bx|1[0-5])}}
 ; WIN64: popq {{%r(bp|bx|1[0-5])}}
 ; WIN64: retq
 
-; LINUXOSX64-LABEL: pushq {{%r(bp|bx|1[2-5])}}
+; LINUXOSX64-LABEL: testi32_inp
 ; LINUXOSX64: pushq {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX64: pushq {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX64: popq  {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX64: popq  {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX64: popq  {{%r(bp|bx|1[2-5])}}
 ; LINUXOSX64: retq
 
 ; Test regcall when running multiple input parameters - callee saved GPRs

test/CodeGen/X86/sse-regcall.ll

 define x86_regcallcc i1 @test_CallargReti1(i1 %a)  {
   %b = add i1 %a, 1
   %c = call x86_regcallcc i1 @test_argReti1(i1 %b)
   %d = add i1 %c, 1
   ret i1 %d
 }
 
 ; WIN64-LABEL: testf32_inp
-; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
+; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; WIN64: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
 ; WIN64: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; WIN64: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; WIN64: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; WIN64: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
-; WIN64: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; WIN64: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; WIN64: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; WIN64: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; WIN64: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; WIN64: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; WIN64: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; WIN64: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
 ; WIN64: retq
 
 ; WIN32-LABEL: testf32_inp
-; WIN32: movaps {{%xmm([4-7])}}, {{.*(%ebp).*}}  {{#+}} 16-byte Spill
-; WIN32: movaps {{%xmm([4-7])}}, {{.*(%ebp).*}}  {{#+}} 16-byte Spill
-; WIN32: movaps {{%xmm([4-7])}}, {{.*(%ebp).*}}  {{#+}} 16-byte Spill
-; WIN32: movaps {{%xmm([4-7])}}, {{.*(%ebp).*}}  {{#+}} 16-byte Spill
+; WIN32: movaps {{%xmm([0-7])}}, {{.*(%e(b|s)p).*}}  {{#+}} 16-byte Spill
 ; WIN32: {{.*}} {{%xmm[0-7]}}, {{%xmm[4-7]}}
 ; WIN32: {{.*}} {{%xmm[0-7]}}, {{%xmm[4-7]}}
 ; WIN32: {{.*}} {{%xmm[0-7]}}, {{%xmm[4-7]}}
 ; WIN32: {{.*}} {{%xmm[0-7]}}, {{%xmm[4-7]}}
-; WIN32: movaps {{.*(%ebp).*}}, {{%xmm([4-7])}}  {{#+}} 16-byte Reload
-; WIN32: movaps {{.*(%ebp).*}}, {{%xmm([4-7])}}  {{#+}} 16-byte Reload
-; WIN32: movaps {{.*(%ebp).*}}, {{%xmm([4-7])}}  {{#+}} 16-byte Reload
-; WIN32: movaps {{.*(%ebp).*}}, {{%xmm([4-7])}}  {{#+}} 16-byte Reload
+; WIN32: movaps {{.*(%e(b|s)p).*}}, {{%xmm([0-7])}}  {{#+}} 16-byte Reload
 ; WIN32: retl
 
 ; LINUXOSX-LABEL: testf32_inp
-; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
-; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%rsp).*}}  {{#+}} 16-byte Spill
+; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
+; LINUXOSX: movaps {{%xmm(1[2-5])}}, {{.*(%r(b|s)p).*}}  {{#+}} 16-byte Spill
 ; LINUXOSX: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; LINUXOSX: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; LINUXOSX: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
 ; LINUXOSX: {{.*}} {{%xmm([0-9]|1[0-1])}}, {{%xmm(1[2-5])}}
-; LINUXOSX: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; LINUXOSX: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; LINUXOSX: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
-; LINUXOSX: movaps {{.*(%rsp).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; LINUXOSX: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; LINUXOSX: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; LINUXOSX: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
+; LINUXOSX: movaps {{.*(%r(b|s)p).*}}, {{%xmm(1[2-5])}}  {{#+}} 16-byte Reload
 ; LINUXOSX: retq
 
 ;test calling conventions - input parameters, callee saved XMMs
 define x86_regcallcc <16 x float> @testf32_inp(<16 x float> %a, <16 x float> %b, <16 x float> %c) nounwind {
   %x1 = fadd <16 x float> %a, %b
   %x2 = fmul <16 x float> %a, %b
   %x3 = fsub <16 x float> %x1, %x2
   %x4 = fadd <16 x float> %x3, %c
   ret <16 x float> %x4
 }
 
 ; WIN32-LABEL: testi32_inp
 ; WIN32: pushl {{%e(si|di|bx|bp)}}
 ; WIN32: pushl {{%e(si|di|bx|bp)}}
-; WIN32: pushl {{%e(si|di|bx|bp)}}
-; WIN32: pushl {{%e(si|di|bx|bp)}}
-; WIN32: popl {{%e(si|di|bx|bp)}}
-; WIN32: popl {{%e(si|di|bx|bp)}}
 ; WIN32: popl {{%e(si|di|bx|bp)}}
 ; WIN32: popl {{%e(si|di|bx|bp)}}
 ; WIN32: retl
 
 ; WIN64-LABEL: testi32_inp
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}
 ; WIN64: pushq  {{%r(bp|bx|1[0-5])}}