[X86] Codegen for preallocated

[aeubanks]

Summary

See https://reviews.llvm.org/D74651 for the preallocated IR constructs
and LangRef changes.

In X86TargetLowering::LowerCall(), if a call is preallocated, record
each argument's offset from the stack pointer and the total stack
adjustment. Associate the call Value with an integer index. Store the
info in X86MachineFunctionInfo with the integer index as the key.

This adds two new target independent ISDOpcodes and two new target
dependent Opcodes corresponding to @llvm.call.preallocated.{setup,arg}.

The setup ISelDAG node takes in a chain and outputs a chain and a
SrcValue of the preallocated call Value. It is lowered to a target
dependent node with the SrcValue replaced with the integer index key by
looking in X86MachineFunctionInfo. In
X86TargetLowering::EmitInstrWithCustomInserter() this is lowered to an
%esp adjustment, the exact amount determined by looking in
X86MachineFunctionInfo with the integer index key.

The arg ISelDAG node takes in a chain, a SrcValue of the preallocated
call Value, and the arg index int constant. It produces a chain and the
pointer fo the arg. It is lowered to a target dependent node with the
SrcValue replaced with the integer index key by looking in
X86MachineFunctionInfo. In
X86TargetLowering::EmitInstrWithCustomInserter() this is lowered to a
lea of the stack pointer plus an offset determined by looking in
X86MachineFunctionInfo with the integer index key.

Force any function containing a preallocated call to use the frame
pointer.

Does not yet handle a setup without a call, or a conditional call.

Tried to look at all references to inalloca and see if they apply to
preallocated. I've made preallocated versions of tests testing inalloca
whenever possible and when they make sense (e.g. not alloca related,
inalloca edge cases).

Aside from the tests added here, I checked that this codegen produces
correct code for something like

struct A {
        A();
        A(A&&);
        ~A();
};

void bar() {
        foo(foo(foo(foo(foo(A(), 4), 5), 6), 7), 8);
}

by replacing the inalloca version of the .ll file with the appropriate
preallocated code. Running the executable produces the same results as
using the current inalloca implementation.

[aeubanks]

Rebase

[aeubanks]

Add tests

[aeubanks]

Add tests
Rename callsetup -> preallocated

[aeubanks]

Look around for inalloca and add preallocated when relevant

Included lots of tests

Update commit message to be much more descriptive

[aeubanks]

ready for review

[jdoerfert]

Nit: The commit name is still call setup.
Did the lang ref patch land already? We should link it in the commit message.

The llvm/lib/Transforms/IPO look straight forward except the one in llvm/lib/Transforms/IPO/GlobalOpt.cpp. I would suggest to split that one off and add a comment to the new code explaining what is happening and why. It also seems that not all IPO passes have tests with the new attribute. Don't wait for me to review the rest or to accept this.

[aeubanks]

More tests
Remove some handling of preallocated in places we don't support yet
Rebase

[aeubanks]

In D77689#2016748, @jdoerfert wrote:

Nit: The commit name is still call setup.
Did the lang ref patch land already? We should link it in the commit message.

Done.
Is there a way to automatically update the Phab change with the commit message if you're only uploading one commit?

The llvm/lib/Transforms/IPO look straight forward except the one in llvm/lib/Transforms/IPO/GlobalOpt.cpp. I would suggest to split that one off and add a comment to the new code explaining what is happening and why. It also seems that not all IPO passes have tests with the new attribute. Don't wait for me to review the rest or to accept this.

Specifically for GlobalOpt.cpp, I realized that I probably shouldn't have done that since we haven't implemented the part where you strip off the operand bundle and replace the calls to the intrinsics with something like alloca. So I removed that and added a TODO.
In general I went through all the IPO transforms and did some archaeology to see what tests they had. I did end up finding more tests to copy/add to, but some also never had tests (not a great excuse)... Anyway thanks for the quick review!

[rnk]

I think the approach looks good, sorry for the delay.

In D77689#2020636, @aeubanks wrote:

Is there a way to automatically update the Phab change with the commit message if you're only uploading one commit?

I don't think so, I think the only way to update the commit message is through the web interface.

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

5803

The common code here seems to be this search of the use list to find the call that consumes the setup. I'd suggest factoring that out into a helper. Once that is done, it seems like this code might be simpler if you separate the setup and arg cases.

llvm/lib/Target/X86/X86ISelLowering.cpp

3898

This seems like it could be an assert, but I could go either way. If the condition occurs, it's an LLVM bug: it means the x86 calling conv code missed the case. We generally use report_fatal_error when user input could trigger the condition, and we don't want UB to occur.

llvm/lib/Target/X86/X86MachineFunctionInfo.h

109

This could be tracked as PreallocatedStackSizes.size(), perhaps.

111

I tend to micro-optimize data structures, but they keys to this are densely packed integers, so this could be a vector instead of a hash map.

112

DenseMap is open-addressed, so this takes up more memory than you might imagine. If you change it to SmallVector, the common case is that there are no call sites using preallocated, so I'd use 0 builtin elements.

203

Can this function use the get-or-insert pattern:

auto Insert = PreallocatedIds.insert({CS, PreallocatedNextId});
if (Insert.second)
  ++PreallocatedNextId;
return Insert.first->second;

215

You can shorten this with .count(Id)

225

You can shorten this with .count(Id)

llvm/lib/Target/X86/X86RegisterInfo.cpp

629–630

Ooh, base pointers are expensive. Is this necessary?

[aeubanks]

Address code review comments

[aeubanks]

llvm/lib/Target/X86/X86ISelLowering.cpp

3898

I copied this from inalloca right above, but yeah assert seems fine.

llvm/lib/Target/X86/X86RegisterInfo.cpp

629–630

Yup, I tried removing it and it makes my test program crash.
I spent quite a while trying to figure out how to force a frame pointer with preallocated calls (is the distinction between frame/base pointer relevant here?).

[rnk]

I have some surface level code style comments, please address those and land this if you agree.

The test case raises an interesting issue: how to handle musttail thunks. However, that will require verifier and maybe langref changes, so let's address that separately.

llvm/lib/Target/X86/X86MachineFunctionInfo.h

199

LLVM is inconsistent about the casing of names, but this class seems to consisitently use leadingLowerCamelCase, so let's do the same here and below, unless you see a reason not to.

217

Generally, we try not to pass SmallVector by value. It is only small in the sense that we believe they contain few elements. Embedding storage in the vector makes it large, and expensive to copy. Generally to pass in a list of things, we tend to prefer ArrayRef<size_t>, so the caller can be flexible about the collection they are using, as long as its a flat array.

221

Similarly, to give the caller a readonly view of the offsets, I would recommend ArrayRef.

llvm/test/CodeGen/X86/musttail-indirect.ll

58 ↗

(On Diff #262784)

This is interesting, we forgot about this when writing the LangRef.

If you look at the previous inalloca test, it forwards its inalloca parameter directly to the musttail callee. It doesn't allocate new memory. That's the real use case for this musttail thing, and we probably need to bend the verifier rules to allow direct forwarding of preallocated arguments to a musttail call site.

I think for now, committing this as is with a FIXME and coming back to it later would be fine. I assume that the verifier rejects it if you remove the setup and arg calls.

[efriedma]

llvm/test/CodeGen/X86/musttail-indirect.ll

58 ↗

(On Diff #262784)

I don't think it makes sense to commit this test as-is. The verifier should reject the combination of musttail and a preallocated bundle. The llvm.call.preallocated.setup would have to allocate memory on top of the argument, and there's no reasonable way to prove that's safe.

And yes, we probably need to bend the rules for the "preallocated" attribute to allow forwarding arguments in musttail calls.

[aeubanks]

Remove musttail tests and add TODOs
Address code review comments
Rebase

[aeubanks]

llvm/test/CodeGen/X86/musttail-indirect.ll

58 ↗

(On Diff #262784)

Added verifier check in https://reviews.llvm.org/D80132.
Is it ok to proceed with this and do the LangRef/codegen changes to support musttail and preallocated together in a later change?

llvm/include/llvm/CodeGen/ISDOpcodes.h

namespace ISD {
   /// instruction sets as much as possible, and only use target-dependent
   /// operators when they have special requirements.
   ///
   /// Finally, during and after selection proper, SNodes may use special
   /// operator codes that correspond directly with MachineInstr opcodes. These
   /// are used to represent selected instructions. See the isMachineOpcode()
   /// and getMachineOpcode() member functions of SDNode.
   ///
   enum NodeType {
     /// DELETED_NODE - This is an illegal value that is used to catch
     /// errors.  This opcode is not a legal opcode for any node.
     DELETED_NODE,
 
     /// EntryToken - This is the marker used to indicate the start of a region.
     EntryToken,
 
     /// TokenFactor - This node takes multiple tokens as input and produces a
enum NodeType {
     /// a source pointer, a SRCVALUE for the destination, and a SRCVALUE for the
     /// source.
     VACOPY,
 
     /// VAEND, VASTART - VAEND and VASTART have three operands: an input chain,
     /// pointer, and a SRCVALUE.
     VAEND, VASTART,
 
+    CALL_SETUP, CALL_ALLOC,
+
     /// SRCVALUE - This is a node type that holds a Value* that is used to
     /// make reference to a value in the LLVM IR.
     SRCVALUE,
 
     /// MDNODE_SDNODE - This is a node that holdes an MDNode*, which is used to
     /// reference metadata in the IR.
     MDNODE_SDNODE,
 
namespace ISD {
   /// this value. Those that do must not be less than this value, and can
   /// be used with SelectionDAG::getMemIntrinsicNode.
   static const int FIRST_TARGET_MEMORY_OPCODE = BUILTIN_OP_END+500;
 
   //===--------------------------------------------------------------------===//
   /// MemIndexedMode enum - This enum defines the load / store indexed
   /// addressing modes.
   ///
   /// UNINDEXED    "Normal" load / store. The effective address is already
   ///              computed and is available in the base pointer. The offset
   ///              operand is always undefined. In addition to producing a
   ///              chain, an unindexed load produces one value (result of the
   ///              load); an unindexed store does not produce a value.
   ///
   /// PRE_INC      Similar to the unindexed mode where the effective address is
   /// PRE_DEC      the value of the base pointer add / subtract the offset.
   ///              It considers the computation as being folded into the load /
   ///              store operation (i.e. the load / store does the address
   ///              computation as well as performing the memory transaction).
   ///              The base operand is always undefined. In addition to
   ///              producing a chain, pre-indexed load produces two values
   ///              (result of the load and the result of the address
   ///              computation); a pre-indexed store produces one value (result
   ///              of the address computation).
   ///
   /// POST_INC     The effective address is the value of the base pointer. The
   /// POST_DEC     value of the offset operand is then added to / subtracted
   ///              from the base after memory transaction. In addition to
   ///              producing a chain, post-indexed load produces two values
   ///              (the result of the load and the result of the base +/- offset
   ///              computation); a post-indexed store produces one value (the
   ///              the result of the base +/- offset computation).
   enum MemIndexedMode {
     UNINDEXED = 0,
     PRE_INC,
     PRE_DEC,
     POST_INC,
     POST_DEC
   };
 
   static const int LAST_INDEXED_MODE = POST_DEC + 1;
 
   //===--------------------------------------------------------------------===//
   /// MemIndexType enum - This enum defines how to interpret MGATHER/SCATTER's
   /// index parameter when calculating addresses.
   ///
   /// SIGNED_SCALED     Addr = Base + ((signed)Index * sizeof(element))
   /// SIGNED_UNSCALED   Addr = Base + (signed)Index
   /// UNSIGNED_SCALED   Addr = Base + ((unsigned)Index * sizeof(element))
   /// UNSIGNED_UNSCALED Addr = Base + (unsigned)Index
   enum MemIndexType {
     SIGNED_SCALED = 0,
     SIGNED_UNSCALED,
     UNSIGNED_SCALED,
     UNSIGNED_UNSCALED

   //===--------------------------------------------------------------------===//
   /// ISD::CondCode enum - These are ordered carefully to make the bitfields
   /// below work out, when considering SETFALSE (something that never exists
   /// dynamically) as 0.  "U" -> Unsigned (for integer operands) or Unordered
   /// (for floating point), "L" -> Less than, "G" -> Greater than, "E" -> Equal
   /// to.  If the "N" column is 1, the result of the comparison is undefined if
   /// the input is a NAN.
   ///
   /// All of these (except for the 'always folded ops') should be handled for
   /// floating point.  For integer, only the SETEQ,SETNE,SETLT,SETLE,SETGT,
   /// SETGE,SETULT,SETULE,SETUGT, and SETUGE opcodes are used.
   ///
   /// Note that these are laid out in a specific order to allow bit-twiddling
   /// to transform conditions.
   enum CondCode {
     // Opcode          N U L G E       Intuitive operation
     SETFALSE,      //    0 0 0 0       Always false (always folded)
     SETOEQ,        //    0 0 0 1       True if ordered and equal
     SETOGT,        //    0 0 1 0       True if ordered and greater than
     SETOGE,        //    0 0 1 1       True if ordered and greater than or equal
     SETOLT,        //    0 1 0 0       True if ordered and less than

llvm/include/llvm/CodeGen/TargetCallingConv.h

private:
     unsigned IsSExt : 1;     ///< Sign extended
     unsigned IsInReg : 1;    ///< Passed in register
     unsigned IsSRet : 1;     ///< Hidden struct-ret ptr
     unsigned IsByVal : 1;    ///< Struct passed by value
     unsigned IsNest : 1;     ///< Nested fn static chain
     unsigned IsReturned : 1; ///< Always returned
     unsigned IsSplit : 1;
     unsigned IsInAlloca : 1;   ///< Passed with inalloca
+    unsigned IsPreallocated : 1; ///< ByVal without the copy
     unsigned IsSplitEnd : 1;   ///< Last part of a split
     unsigned IsSwiftSelf : 1;  ///< Swift self parameter
     unsigned IsSwiftError : 1; ///< Swift error parameter
     unsigned IsCFGuardTarget : 1; ///< Control Flow Guard target
     unsigned IsHva : 1;        ///< HVA field for
     unsigned IsHvaStart : 1;   ///< HVA structure start
     unsigned IsSecArgPass : 1; ///< Second argument
     unsigned ByValAlign : 4;   ///< Log 2 of byval alignment
     unsigned OrigAlign : 5;    ///< Log 2 of original alignment
     unsigned IsInConsecutiveRegsLast : 1;
     unsigned IsInConsecutiveRegs : 1;
     unsigned IsCopyElisionCandidate : 1; ///< Argument copy elision candidate
     unsigned IsPointer : 1;
 
     unsigned ByValSize; ///< Byval struct size
 
     unsigned PointerAddrSpace; ///< Address space of pointer argument
 
   public:
     ArgFlagsTy()
         : IsZExt(0), IsSExt(0), IsInReg(0), IsSRet(0), IsByVal(0), IsNest(0),
-          IsReturned(0), IsSplit(0), IsInAlloca(0), IsSplitEnd(0),
-          IsSwiftSelf(0), IsSwiftError(0), IsCFGuardTarget(0), IsHva(0),
-          IsHvaStart(0), IsSecArgPass(0), ByValAlign(0), OrigAlign(0),
+          IsReturned(0), IsSplit(0), IsInAlloca(0), IsPreallocated(0),
+          IsSplitEnd(0), IsSwiftSelf(0), IsSwiftError(0), IsCFGuardTarget(0),
+          IsHva(0), IsHvaStart(0), IsSecArgPass(0), ByValAlign(0), OrigAlign(0),
           IsInConsecutiveRegsLast(0), IsInConsecutiveRegs(0),
           IsCopyElisionCandidate(0), IsPointer(0), ByValSize(0),
           PointerAddrSpace(0) {
       static_assert(sizeof(*this) == 3 * sizeof(unsigned), "flags are too big");
     }
 
     bool isZExt() const { return IsZExt; }
     void setZExt() { IsZExt = 1; }
 
     bool isSExt() const { return IsSExt; }
     void setSExt() { IsSExt = 1; }
 
     bool isInReg() const { return IsInReg; }
     void setInReg() { IsInReg = 1; }
 
     bool isSRet() const { return IsSRet; }
     void setSRet() { IsSRet = 1; }
 
     bool isByVal() const { return IsByVal; }
     void setByVal() { IsByVal = 1; }
 
     bool isInAlloca() const { return IsInAlloca; }
     void setInAlloca() { IsInAlloca = 1; }
 
+    bool isPreallocated() const { return IsPreallocated; }
+    void setPreallocated() { IsPreallocated = 1; }
+
     bool isSwiftSelf() const { return IsSwiftSelf; }
     void setSwiftSelf() { IsSwiftSelf = 1; }
 
     bool isSwiftError() const { return IsSwiftError; }
     void setSwiftError() { IsSwiftError = 1; }
 
     bool isCFGuardTarget() const { return IsCFGuardTarget; }
     void setCFGuardTarget() { IsCFGuardTarget = 1; }

llvm/include/llvm/CodeGen/TargetLowering.h

public:
     Type *Ty = nullptr;
     bool IsSExt : 1;
     bool IsZExt : 1;
     bool IsInReg : 1;
     bool IsSRet : 1;
     bool IsNest : 1;
     bool IsByVal : 1;
     bool IsInAlloca : 1;
+    bool IsPreallocated : 1;
     bool IsReturned : 1;
     bool IsSwiftSelf : 1;
     bool IsSwiftError : 1;
     bool IsCFGuardTarget : 1;
     MaybeAlign Alignment = None;
     Type *ByValType = nullptr;
+    Type *PreallocatedType = nullptr;
 
     ArgListEntry()
         : IsSExt(false), IsZExt(false), IsInReg(false), IsSRet(false),
-          IsNest(false), IsByVal(false), IsInAlloca(false), IsReturned(false),
-          IsSwiftSelf(false), IsSwiftError(false), IsCFGuardTarget(false) {}
+          IsNest(false), IsByVal(false), IsInAlloca(false),
+          IsPreallocated(false), IsReturned(false), IsSwiftSelf(false),
+          IsSwiftError(false), IsCFGuardTarget(false) {}
 
     void setAttributes(const CallBase *Call, unsigned ArgIdx);
   };
   using ArgListTy = std::vector<ArgListEntry>;
 
   virtual void markLibCallAttributes(MachineFunction *MF, unsigned CC,
                                      ArgListTy &Args) const {};
 
struct CallLoweringInfo {
     bool RetSExt           : 1;
     bool RetZExt           : 1;
     bool IsVarArg          : 1;
     bool IsInReg           : 1;
     bool DoesNotReturn     : 1;
     bool IsReturnValueUsed : 1;
     bool IsConvergent      : 1;
     bool IsPatchPoint      : 1;
+    bool IsPreallocated : 1;
 
     // IsTailCall should be modified by implementations of
     // TargetLowering::LowerCall that perform tail call conversions.
     bool IsTailCall = false;
 
     // Is Call lowering done post SelectionDAG type legalization.
     bool IsPostTypeLegalization = false;
 
     unsigned NumFixedArgs = -1;
     CallingConv::ID CallConv = CallingConv::C;
     SDValue Callee;
     ArgListTy Args;
     SelectionDAG &DAG;
     SDLoc DL;
     const CallBase *CB = nullptr;
     SmallVector<ISD::OutputArg, 32> Outs;
     SmallVector<SDValue, 32> OutVals;
     SmallVector<ISD::InputArg, 32> Ins;
     SmallVector<SDValue, 4> InVals;
 
     CallLoweringInfo(SelectionDAG &DAG)
         : RetSExt(false), RetZExt(false), IsVarArg(false), IsInReg(false),
           DoesNotReturn(false), IsReturnValueUsed(true), IsConvergent(false),
-          IsPatchPoint(false), DAG(DAG) {}
+          IsPatchPoint(false), IsPreallocated(false), DAG(DAG) {}
 
     CallLoweringInfo &setDebugLoc(const SDLoc &dl) {
       DL = dl;
       return *this;
     }
 
     CallLoweringInfo &setChain(SDValue InChain) {
       Chain = InChain;
CallLoweringInfo &setZExtResult(bool Value = true) {
       return *this;
     }
 
     CallLoweringInfo &setIsPatchPoint(bool Value = true) {
       IsPatchPoint = Value;
       return *this;
     }
 
+    CallLoweringInfo &setIsPreallocated(bool Value = true) {
+      IsPreallocated = Value;
+      return *this;
+    }
+
     CallLoweringInfo &setIsPostTypeLegalization(bool Value=true) {
       IsPostTypeLegalization = Value;
       return *this;
     }
 
     ArgListTy &getArgs() {
       return Args;
     }

llvm/include/llvm/IR/Attributes.h

public:
   MaybeAlign getRetAlignment() const;
 
   /// Return the alignment for the specified function parameter.
   MaybeAlign getParamAlignment(unsigned ArgNo) const;
 
   /// Return the byval type for the specified function parameter.
   Type *getParamByValType(unsigned ArgNo) const;
 
+  /// Return the preallocated type for the specified function parameter.
+  Type *getParamPreallocatedType(unsigned ArgNo) const;
+
   /// Get the stack alignment.
   MaybeAlign getStackAlignment(unsigned Index) const;
 
   /// Get the number of dereferenceable bytes (or zero if unknown).
   uint64_t getDereferenceableBytes(unsigned Index) const;
 
   /// Get the number of dereferenceable bytes (or zero if unknown) of an
   /// arg.

llvm/include/llvm/IR/InstrTypes.h

public:
   }
 
   /// Extract the byval type for a call or parameter.
   Type *getParamByValType(unsigned ArgNo) const {
     Type *Ty = Attrs.getParamByValType(ArgNo);
     return Ty ? Ty : getArgOperand(ArgNo)->getType()->getPointerElementType();
   }
 
+  /// Extract the preallocated type for a call or parameter.
+  Type *getParamPreallocatedType(unsigned ArgNo) const {
+    Type *Ty = Attrs.getParamPreallocatedType(ArgNo);
+    return Ty ? Ty : getArgOperand(ArgNo)->getType()->getPointerElementType();
+  }
+
   /// Extract the number of dereferenceable bytes for a call or
   /// parameter (0=unknown).
   uint64_t getDereferenceableBytes(unsigned i) const {
     return Attrs.getDereferenceableBytes(i);
   }
 
   /// Extract the number of dereferenceable_or_null bytes for a call or
   /// parameter (0=unknown).

llvm/include/llvm/Support/TargetOpcodes.def

 HANDLE_TARGET_OPCODE(PATCHPOINT)
 
 /// This pseudo-instruction loads the stack guard value. Targets which need
 /// to prevent the stack guard value or address from being spilled to the
 /// stack should override TargetLowering::emitLoadStackGuardNode and
 /// additionally expand this pseudo after register allocation.
 HANDLE_TARGET_OPCODE(LOAD_STACK_GUARD)
 
+HANDLE_TARGET_OPCODE(CALL_SETUP)
+HANDLE_TARGET_OPCODE(CALL_ALLOC)
+
 /// Call instruction with associated vm state for deoptimization and list
 /// of live pointers for relocation by the garbage collector.  It is
 /// intended to support garbage collection with fully precise relocating
 /// collectors and deoptimizations in either the callee or caller.
 HANDLE_TARGET_OPCODE(STATEPOINT)
 
 /// Instruction that records the offset of a local stack allocation passed to
 /// llvm.localescape. It has two arguments: the symbol for the label and the

llvm/include/llvm/Target/Target.td

 def LOAD_STACK_GUARD : StandardPseudoInstruction {
   let OutOperandList = (outs ptr_rc:$dst);
   let InOperandList = (ins);
   let mayLoad = 1;
   bit isReMaterializable = 1;
   let hasSideEffects = 0;
   bit isPseudo = 1;
 }
+def CALL_SETUP : StandardPseudoInstruction {
+  let OutOperandList = (outs);
+  let InOperandList = (ins i32imm:$a);
+  let usesCustomInserter = 1;
+  let hasSideEffects = 1;
+}
+def CALL_ALLOC : StandardPseudoInstruction {
+  let OutOperandList = (outs ptr_rc:$loc);
+  let InOperandList = (ins i32imm:$a, i32imm:$b);
+  let usesCustomInserter = 1;
+  let hasSideEffects = 1;
+}
 def LOCAL_ESCAPE : StandardPseudoInstruction {
   // This instruction is really just a label. It has to be part of the chain so
   // that it doesn't get dropped from the DAG, but it produces nothing and has
   // no side effects.
   let OutOperandList = (outs);
   let InOperandList = (ins ptr_rc:$symbol, i32imm:$id);
   let hasSideEffects = 0;
   let hasCtrlDep = 1;

llvm/include/llvm/Target/TargetCallingConv.td

class CCIf<string predicate, CCAction A> : CCPredicateAction<A> {
   string Predicate = predicate;
 }
 
 /// CCIfByVal - If the current argument has ByVal parameter attribute, apply
 /// Action A.
 class CCIfByVal<CCAction A> : CCIf<"ArgFlags.isByVal()", A> {
 }
 
+/// CCIfPreallocated - If the current argument has Preallocated parameter attribute,
+/// apply Action A.
+class CCIfPreallocated<CCAction A> : CCIf<"ArgFlags.isPreallocated()", A> {
+}
+
 /// CCIfSwiftSelf - If the current argument has swiftself parameter attribute,
 /// apply Action A.
 class CCIfSwiftSelf<CCAction A> : CCIf<"ArgFlags.isSwiftSelf()", A> {
 }
 
 /// CCIfSwiftError - If the current argument has swifterror parameter attribute,
 /// apply Action A.
 class CCIfSwiftError<CCAction A> : CCIf<"ArgFlags.isSwiftError()", A> {

llvm/lib/CodeGen/GlobalISel/CallLowering.cpp

void CallLowering::setArgFlags(CallLowering::ArgInfo &Arg, unsigned OpIdx,
   if (Attrs.hasAttribute(OpIdx, Attribute::StructRet))
     Flags.setSRet();
   if (Attrs.hasAttribute(OpIdx, Attribute::SwiftSelf))
     Flags.setSwiftSelf();
   if (Attrs.hasAttribute(OpIdx, Attribute::SwiftError))
     Flags.setSwiftError();
   if (Attrs.hasAttribute(OpIdx, Attribute::ByVal))
     Flags.setByVal();
+  if (Attrs.hasAttribute(OpIdx, Attribute::Preallocated))
+    Flags.setPreallocated();
   if (Attrs.hasAttribute(OpIdx, Attribute::InAlloca))
     Flags.setInAlloca();
 
-  if (Flags.isByVal() || Flags.isInAlloca()) {
+  if (Flags.isByVal() || Flags.isInAlloca() || Flags.isPreallocated()) {
     Type *ElementTy = cast<PointerType>(Arg.Ty)->getElementType();
 
     auto Ty = Attrs.getAttribute(OpIdx, Attribute::ByVal).getValueAsType();
     Flags.setByValSize(DL.getTypeAllocSize(Ty ? Ty : ElementTy));
 
     // For ByVal, alignment should be passed from FE.  BE will guess if
     // this info is not there but there are cases it cannot get right.
     Align FrameAlign;

llvm/lib/CodeGen/SelectionDAG/FastISel.cpp

for (auto &Arg : CLI.getArgs()) {
     if (Arg.IsSwiftSelf)
       Flags.setSwiftSelf();
     if (Arg.IsSwiftError)
       Flags.setSwiftError();
     if (Arg.IsCFGuardTarget)
       Flags.setCFGuardTarget();
     if (Arg.IsByVal)
       Flags.setByVal();
+    if (Arg.IsPreallocated)
+      Flags.setPreallocated();
     if (Arg.IsInAlloca) {
       Flags.setInAlloca();
       // Set the byval flag for CCAssignFn callbacks that don't know about
       // inalloca. This way we can know how many bytes we should've allocated
       // and how many bytes a callee cleanup function will pop.  If we port
       // inalloca to more targets, we'll have to add custom inalloca handling in
       // the various CC lowering callbacks.
       Flags.setByVal();
     }
-    if (Arg.IsByVal || Arg.IsInAlloca) {
+    if (Arg.IsByVal || Arg.IsInAlloca || Arg.IsPreallocated) {
       PointerType *Ty = cast<PointerType>(Arg.Ty);
       Type *ElementTy = Ty->getElementType();
       unsigned FrameSize =
           DL.getTypeAllocSize(Arg.ByValType ? Arg.ByValType : ElementTy);
 
       // For ByVal, alignment should come from FE. BE will guess if this info
       // is not there, but there are cases it cannot get right.
       MaybeAlign FrameAlign = Arg.Alignment;

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

SDValue SelectionDAG::getBlockAddress(const BlockAddress *BA, EVT VT,
 
   auto *N = newSDNode<BlockAddressSDNode>(Opc, VT, BA, Offset, TargetFlags);
   CSEMap.InsertNode(N, IP);
   InsertNode(N);
   return SDValue(N, 0);
 }
 
 SDValue SelectionDAG::getSrcValue(const Value *V) {
-  assert((!V || V->getType()->isPointerTy()) &&
-         "SrcValue is not a pointer?");
-
   FoldingSetNodeID ID;
   AddNodeIDNode(ID, ISD::SRCVALUE, getVTList(MVT::Other), None);
   ID.AddPointer(V);
 
   void *IP = nullptr;
   if (SDNode *E = FindNodeOrInsertPos(ID, IP))
     return SDValue(E, 0);
 

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

case Intrinsic::memset_element_unordered_atomic: {
     unsigned ElemSz = MI.getElementSizeInBytes();
     bool isTC = I.isTailCall() && isInTailCallPosition(I, DAG.getTarget());
     SDValue MC = DAG.getAtomicMemset(getRoot(), sdl, Dst, DstAlign, Val, Length,
                                      LengthTy, ElemSz, isTC,
                                      MachinePointerInfo(MI.getRawDest()));
     updateDAGForMaybeTailCall(MC);
     return;
   }
+  case Intrinsic::call_preallocated_setup:
+  case Intrinsic::call_preallocated_arg: {
+    const Value *CallSetup =
+        Intrinsic == Intrinsic::call_preallocated_setup ? &I : I.getOperand(0);
+    assert(cast<CallBase>(CallSetup)->getCalledFunction()->getIntrinsicID() ==
+               Intrinsic::call_preallocated_setup &&
+           "expected call_preallocated_setup Value");
+    const CallBase *Call = nullptr;
+    for (auto U : CallSetup->users()) {
+      auto *UseCall = cast<CallBase>(U);
+      const Function *Fn = UseCall->getCalledFunction();
+      if (Fn->getIntrinsicID() != Intrinsic::call_preallocated_arg) {
+        Call = UseCall;
+        break;
+      }
+    }
+    if (Call) {
+      SDValue SrcValue = DAG.getSrcValue(Call);
+      if (Intrinsic == Intrinsic::call_preallocated_setup) {
+        SDValue Res =
+            DAG.getNode(ISD::CALL_SETUP, sdl, MVT::Other, getRoot(), SrcValue);
+        setValue(&I, Res);
+        DAG.setRoot(Res);
+      } else {
+        SDValue Ops[3];
+        Ops[0] = getRoot();
+        Ops[1] = SrcValue;
+        Ops[2] = DAG.getTargetConstant(*cast<ConstantInt>(I.getArgOperand(1)),
+                                       sdl, MVT::i32); // arg index
+        SDValue Res = DAG.getNode(
+            ISD::CALL_ALLOC, sdl,
+            DAG.getVTList(TLI.getPointerTy(DAG.getDataLayout()), MVT::Other),
+            Ops);
+        setValue(&I, Res);
+        DAG.setRoot(Res.getValue(1));
+      }
+    }
+    return;
+  }
   case Intrinsic::dbg_addr:
   case Intrinsic::dbg_declare: {
     const auto &DI = cast<DbgVariableIntrinsic>(I);
     DILocalVariable *Variable = DI.getVariable();
     DIExpression *Expression = DI.getExpression();
     dropDanglingDebugInfo(Variable, Expression);
     assert(Variable && "Missing variable");
     LLVM_DEBUG(dbgs() << "SelectionDAG visiting debug intrinsic: " << DI
void SelectionDAGBuilder::LowerCallTo(const CallBase &CB, SDValue Callee,
   if (TLI.supportSwiftError() && SwiftErrorVal)
     isTailCall = false;
 
   TargetLowering::CallLoweringInfo CLI(DAG);
   CLI.setDebugLoc(getCurSDLoc())
       .setChain(getRoot())
       .setCallee(RetTy, FTy, Callee, std::move(Args), CB)
       .setTailCall(isTailCall)
-      .setConvergent(CB.isConvergent());
+      .setConvergent(CB.isConvergent())
+      .setIsPreallocated(
+          CB.countOperandBundlesOfType(LLVMContext::OB_preallocated) != 0);
   std::pair<SDValue, SDValue> Result = lowerInvokable(CLI, EHPadBB);
 
   if (Result.first.getNode()) {
     Result.first = lowerRangeToAssertZExt(DAG, CB, Result.first);
     setValue(&CB, Result.first);
   }
 
   // The last element of CLI.InVals has the SDValue for swifterror return.
if (!I.isNoBuiltin() && !I.isStrictFP() && !F->hasLocalLinkage() &&
         break;
       }
     }
   }
 
   // Deopt bundles are lowered in LowerCallSiteWithDeoptBundle, and we don't
   // have to do anything here to lower funclet bundles.
   // CFGuardTarget bundles are lowered in LowerCallTo.
-  assert(!I.hasOperandBundlesOtherThan({LLVMContext::OB_deopt,
-                                        LLVMContext::OB_funclet,
-                                        LLVMContext::OB_cfguardtarget}) &&
+  assert(!I.hasOperandBundlesOtherThan(
+             {LLVMContext::OB_deopt, LLVMContext::OB_funclet,
+              LLVMContext::OB_cfguardtarget, LLVMContext::OB_preallocated}) &&
          "Cannot lower calls with arbitrary operand bundles!");
 
   SDValue Callee = getValue(I.getCalledValue());
 
   if (I.countOperandBundlesOfType(LLVMContext::OB_deopt))
     LowerCallSiteWithDeoptBundle(&I, Callee, nullptr);
   else
     // Check if we can potentially perform a tail call. More detailed checking
for (unsigned ArgI = ArgIdx, ArgE = ArgIdx + NumArgs;
     Entry.setAttributes(Call, ArgI);
     Args.push_back(Entry);
   }
 
   CLI.setDebugLoc(getCurSDLoc())
       .setChain(getRoot())
       .setCallee(Call->getCallingConv(), ReturnTy, Callee, std::move(Args))
       .setDiscardResult(Call->use_empty())
-      .setIsPatchPoint(IsPatchPoint);
+      .setIsPatchPoint(IsPatchPoint)
+      .setIsPreallocated(
+          Call->countOperandBundlesOfType(LLVMContext::OB_preallocated) != 0);
 }
 
 /// Add a stack map intrinsic call's live variable operands to a stackmap
 /// or patchpoint target node's operand list.
 ///
 /// Constants are converted to TargetConstants purely as an optimization to
 /// avoid constant materialization and register allocation.
 ///
for (unsigned Value = 0, NumValues = ValueVTs.size(); Value != NumValues;
       if (Args[i].IsSwiftSelf)
         Flags.setSwiftSelf();
       if (Args[i].IsSwiftError)
         Flags.setSwiftError();
       if (Args[i].IsCFGuardTarget)
         Flags.setCFGuardTarget();
       if (Args[i].IsByVal)
         Flags.setByVal();
+      if (Args[i].IsPreallocated)
+        Flags.setPreallocated();
       if (Args[i].IsInAlloca) {
         Flags.setInAlloca();
         // Set the byval flag for CCAssignFn callbacks that don't know about
         // inalloca.  This way we can know how many bytes we should've allocated
         // and how many bytes a callee cleanup function will pop.  If we port
         // inalloca to more targets, we'll have to add custom inalloca handling
         // in the various CC lowering callbacks.
         Flags.setByVal();
       }
-      if (Args[i].IsByVal || Args[i].IsInAlloca) {
+      if (Args[i].IsByVal || Args[i].IsInAlloca || Args[i].IsPreallocated) {
         PointerType *Ty = cast<PointerType>(Args[i].Ty);
         Type *ElementTy = Ty->getElementType();
 
         unsigned FrameSize = DL.getTypeAllocSize(
             Args[i].ByValType ? Args[i].ByValType : ElementTy);
         Flags.setByValSize(FrameSize);
 
         // info is not there but there are cases it cannot get right.

 }
 
 SDValue TargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   llvm_unreachable("LowerOperation not implemented for this target!");
 }
 
 void
 SelectionDAGBuilder::CopyValueToVirtualRegister(const Value *V, unsigned Reg) {
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+
   SDValue Op = getNonRegisterValue(V);
   assert((Op.getOpcode() != ISD::CopyFromReg ||
           cast<RegisterSDNode>(Op.getOperand(1))->getReg() != Reg) &&
          "Copy from a reg to the same reg!");
   assert(!Register::isPhysicalRegister(Reg) && "Is a physreg");
 
-  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   // If this is an InlineAsm we have to match the registers required, not the
   // notional registers required by the type.
 
   RegsForValue RFV(V->getContext(), TLI, DAG.getDataLayout(), Reg, V->getType(),
                    None); // This is not an ABI copy.
   SDValue Chain = DAG.getEntryNode();
 
   ISD::NodeType ExtendType = (FuncInfo.PreferredExtendType.find(V) ==
for (unsigned Value = 0, NumValues = ValueVTs.size();
         Flags.setInAlloca();
         // Set the byval flag for CCAssignFn callbacks that don't know about
         // inalloca.  This way we can know how many bytes we should've allocated
         // and how many bytes a callee cleanup function will pop.  If we port
         // inalloca to more targets, we'll have to add custom inalloca handling
         // in the various CC lowering callbacks.
         Flags.setByVal();
       }
+      if (Arg.hasAttribute(Attribute::Preallocated)) {
+        Flags.setPreallocated();
+        // Set the byval flag for CCAssignFn callbacks that don't know about
+        // inalloca.  This way we can know how many bytes we should've allocated
+        // and how many bytes a callee cleanup function will pop.  If we port
+        // inalloca to more targets, we'll have to add custom inalloca handling
+        // in the various CC lowering callbacks.
+        Flags.setByVal();
+      }
       if (F.getCallingConv() == CallingConv::X86_INTR) {
         // IA Interrupt passes frame (1st parameter) by value in the stack.
         if (ArgNo == 0)
           Flags.setByVal();
       }
-      if (Flags.isByVal() || Flags.isInAlloca()) {
+      if (Flags.isByVal() || Flags.isInAlloca() || Flags.isPreallocated()) {
         Type *ElementTy = Arg.getParamByValType();
 
         // For ByVal, size and alignment should be passed from FE.  BE will
         // guess if this info is not there but there are cases it cannot get
         // right.
         unsigned FrameSize = DL.getTypeAllocSize(Arg.getParamByValType());
         Flags.setByValSize(FrameSize);
 

llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp

#endif
   case ISD::TRAP:                       return "trap";
   case ISD::DEBUGTRAP:                  return "debugtrap";
   case ISD::LIFETIME_START:             return "lifetime.start";
   case ISD::LIFETIME_END:               return "lifetime.end";
   case ISD::GC_TRANSITION_START:        return "gc_transition.start";
   case ISD::GC_TRANSITION_END:          return "gc_transition.end";
   case ISD::GET_DYNAMIC_AREA_OFFSET:    return "get.dynamic.area.offset";
   case ISD::FREEZE:                     return "freeze";
+  case ISD::CALL_SETUP:
+    return "call_setup";
+  case ISD::CALL_ALLOC:
+    return "call_alloc";
 
   // Bit manipulation
   case ISD::ABS:                        return "abs";
   case ISD::BITREVERSE:                 return "bitreverse";
   case ISD::BSWAP:                      return "bswap";
   case ISD::CTPOP:                      return "ctpop";
   case ISD::CTTZ:                       return "cttz";
   case ISD::CTTZ_ZERO_UNDEF:            return "cttz_zero_undef";

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

 void TargetLoweringBase::ArgListEntry::setAttributes(const CallBase *Call,
                                                      unsigned ArgIdx) {
   IsSExt = Call->paramHasAttr(ArgIdx, Attribute::SExt);
   IsZExt = Call->paramHasAttr(ArgIdx, Attribute::ZExt);
   IsInReg = Call->paramHasAttr(ArgIdx, Attribute::InReg);
   IsSRet = Call->paramHasAttr(ArgIdx, Attribute::StructRet);
   IsNest = Call->paramHasAttr(ArgIdx, Attribute::Nest);
   IsByVal = Call->paramHasAttr(ArgIdx, Attribute::ByVal);
+  IsPreallocated = Call->paramHasAttr(ArgIdx, Attribute::Preallocated);
   IsInAlloca = Call->paramHasAttr(ArgIdx, Attribute::InAlloca);
   IsReturned = Call->paramHasAttr(ArgIdx, Attribute::Returned);
   IsSwiftSelf = Call->paramHasAttr(ArgIdx, Attribute::SwiftSelf);
   IsSwiftError = Call->paramHasAttr(ArgIdx, Attribute::SwiftError);
   Alignment = Call->getParamAlign(ArgIdx);
   ByValType = nullptr;
-  if (Call->paramHasAttr(ArgIdx, Attribute::ByVal))
+  if (IsByVal)
     ByValType = Call->getParamByValType(ArgIdx);
+  PreallocatedType = nullptr;
+  if (IsPreallocated)
+    PreallocatedType = Call->getParamPreallocatedType(ArgIdx);
 }
 
 /// Generate a libcall taking the given operands as arguments and returning a
 /// result of type RetVT.
 std::pair<SDValue, SDValue>
 TargetLowering::makeLibCall(SelectionDAG &DAG, RTLIB::Libcall LC, EVT RetVT,
                             ArrayRef<SDValue> Ops,
                             MakeLibCallOptions CallOptions,

llvm/lib/IR/Attributes.cpp

 MaybeAlign AttributeList::getParamAlignment(unsigned ArgNo) const {
   return getAttributes(ArgNo + FirstArgIndex).getAlignment();
 }
 
 Type *AttributeList::getParamByValType(unsigned Index) const {
   return getAttributes(Index+FirstArgIndex).getByValType();
 }
 
+Type *AttributeList::getParamPreallocatedType(unsigned Index) const {
+  return getAttributes(Index + FirstArgIndex).getPreallocatedType();
+}
+
 MaybeAlign AttributeList::getStackAlignment(unsigned Index) const {
   return getAttributes(Index).getStackAlignment();
 }
 
 uint64_t AttributeList::getDereferenceableBytes(unsigned Index) const {
   return getAttributes(Index).getDereferenceableBytes();
 }
 

llvm/lib/Target/X86/X86CallingConv.td

CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
                 CCAssignToReg<[ZMM0, ZMM1, ZMM2, ZMM3]>>>,
 
   CCDelegateTo<CC_X86_32_Vector_Common>
 ]>;
 
 /// CC_X86_32_Common - In all X86-32 calling conventions, extra integers and FP
 /// values are spilled on the stack.
 def CC_X86_32_Common : CallingConv<[
-  // Handles byval parameters.
+  // Handles byval/preallocated parameters.
   CCIfByVal<CCPassByVal<4, 4>>,
+  CCIfPreallocated<CCPassByVal<4, 4>>,
 
   // The first 3 float or double arguments, if marked 'inreg' and if the call
   // is not a vararg call and if SSE2 is available, are passed in SSE registers.
   CCIfNotVarArg<CCIfInReg<CCIfType<[f32,f64],
                 CCIfSubtarget<"hasSSE2()",
                 CCAssignToReg<[XMM0,XMM1,XMM2]>>>>>,
 
   // The first 3 __m64 vector arguments are passed in mmx registers if the

llvm/lib/Target/X86/X86FrameLowering.cpp

X86FrameLowering::X86FrameLowering(const X86Subtarget &STI,
   IsLP64 = STI.isTarget64BitLP64();
   // standard x86_64 and NaCl use 64-bit frame/stack pointers, x32 - 32-bit.
   Uses64BitFramePtr = STI.isTarget64BitLP64() || STI.isTargetNaCl64();
   StackPtr = TRI->getStackRegister();
 }
 
 bool X86FrameLowering::hasReservedCallFrame(const MachineFunction &MF) const {
   return !MF.getFrameInfo().hasVarSizedObjects() &&
-         !MF.getInfo<X86MachineFunctionInfo>()->getHasPushSequences();
+         !MF.getInfo<X86MachineFunctionInfo>()->getHasPushSequences() &&
+         !MF.getInfo<X86MachineFunctionInfo>()->hasCallSetup();
 }
 
 /// canSimplifyCallFramePseudos - If there is a reserved call frame, the
 /// call frame pseudos can be simplified.  Having a FP, as in the default
 /// implementation, is not sufficient here since we can't always use it.
 /// Use a more nuanced condition.
 bool
 X86FrameLowering::canSimplifyCallFramePseudos(const MachineFunction &MF) const {
   return hasReservedCallFrame(MF) ||
+         MF.getInfo<X86MachineFunctionInfo>()->hasCallSetup() ||
          (hasFP(MF) && !TRI->needsStackRealignment(MF)) ||
          TRI->hasBasePointer(MF);
 }
 
 // needsFrameIndexResolution - Do we need to perform FI resolution for
 // this function. Normally, this is required only when the function
 // has any stack objects. However, FI resolution actually has another job,
 // not apparent from the title - it resolves callframesetup/destroy

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

case X86ISD::MSCATTER: {
     SDValue Ops[] = {Base, Scale, Index, Disp, Segment, Mask, Value, Chain};
 
     MachineSDNode *NewNode = CurDAG->getMachineNode(Opc, SDLoc(dl), VTs, Ops);
     CurDAG->setNodeMemRefs(NewNode, {Sc->getMemOperand()});
     ReplaceUses(SDValue(Node, 0), SDValue(NewNode, 1));
     CurDAG->RemoveDeadNode(Node);
     return;
   }
+  case ISD::CALL_SETUP: {
+    auto MFI = CurDAG->getMachineFunction().getInfo<X86MachineFunctionInfo>();
+    auto CallId = MFI->CallSetupIdForCallSite(
+        cast<SrcValueSDNode>(Node->getOperand(1))->getValue());
+    SDValue Chain = Node->getOperand(0);
+    SDValue CallIdValue = CurDAG->getTargetConstant(CallId, dl, MVT::i32);
+    MachineSDNode *New = CurDAG->getMachineNode(TargetOpcode::CALL_SETUP, dl,
+                                                MVT::Other, CallIdValue, Chain);
+    ReplaceUses(SDValue(Node, 0), SDValue(New, 0)); // Chain
+    CurDAG->RemoveDeadNode(Node);
+    return;
+  }
+  case ISD::CALL_ALLOC: {
+    auto MFI = CurDAG->getMachineFunction().getInfo<X86MachineFunctionInfo>();
+    auto CallId = MFI->CallSetupIdForCallSite(
+        cast<SrcValueSDNode>(Node->getOperand(1))->getValue());
+    SDValue Chain = Node->getOperand(0);
+    SDValue CallIdValue = CurDAG->getTargetConstant(CallId, dl, MVT::i32);
+    SDValue ArgIndex = Node->getOperand(2);
+    SDValue Ops[3];
+    Ops[0] = CallIdValue;
+    Ops[1] = ArgIndex;
+    Ops[2] = Chain;
+    MachineSDNode *New = CurDAG->getMachineNode(
+        TargetOpcode::CALL_ALLOC, dl,
+        CurDAG->getVTList(TLI->getPointerTy(CurDAG->getDataLayout()),
+                          MVT::Other),
+        Ops);
+    ReplaceUses(SDValue(Node, 0), SDValue(New, 0)); // Arg pointer
+    ReplaceUses(SDValue(Node, 1), SDValue(New, 1)); // Chain
+    CurDAG->RemoveDeadNode(Node);
+    return;
+  }
   }
 
   SelectCode(Node);
 }
 
 bool X86DAGToDAGISel::
 SelectInlineAsmMemoryOperand(const SDValue &Op, unsigned ConstraintID,
                              std::vector<SDValue> &OutOps) {

llvm/lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI,
   if (!Outs.empty() && Outs.back().Flags.isInAlloca()) {
     NumBytesToPush = 0;
     if (!ArgLocs.back().isMemLoc())
       report_fatal_error("cannot use inalloca attribute on a register "
                          "parameter");
     if (ArgLocs.back().getLocMemOffset() != 0)
       report_fatal_error("any parameter with the inalloca attribute must be "
                          "the only memory argument");
+  } else if (CLI.IsPreallocated) {
+    if (!ArgLocs.back().isMemLoc()) {
+      report_fatal_error("cannot use preallocated attribute on a register "
+                         "parameter");
+    }
+    SmallVector<size_t, 4> PreallocatedOffsets;
+    for (size_t i = 0; i < CLI.OutVals.size(); ++i) {
+      if (CLI.CB->paramHasAttr(i, Attribute::Preallocated)) {
+        PreallocatedOffsets.push_back(ArgLocs[i].getLocMemOffset());
+      }
+    }
+    auto MFI = DAG.getMachineFunction().getInfo<X86MachineFunctionInfo>();
+    size_t CallSetupId = MFI->CallSetupIdForCallSite(CLI.CB);
+    MFI->SetCallSetupStackSize(CallSetupId, NumBytes);
+    MFI->SetCallSetupArgOffsets(CallSetupId, PreallocatedOffsets);
+    NumBytesToPush = 0;
   }
 
-  if (!IsSibcall && !IsMustTail)
+  if (!IsSibcall && !IsMustTail) {
     Chain = DAG.getCALLSEQ_START(Chain, NumBytesToPush,
                                  NumBytes - NumBytesToPush, dl);
+  }
 
   SDValue RetAddrFrIdx;
   // Load return address for tail calls.
   if (isTailCall && FPDiff)
     Chain = EmitTailCallLoadRetAddr(DAG, RetAddrFrIdx, Chain, isTailCall,
                                     Is64Bit, FPDiff, dl);
 
   SmallVector<std::pair<unsigned, SDValue>, 8> RegsToPass;
   SmallVector<SDValue, 8> MemOpChains;
   SDValue StackPtr;
 
   // The next loop assumes that the locations are in the same order of the
   // input arguments.
   assert(isSortedByValueNo(ArgLocs) &&
          "Argument Location list must be sorted before lowering");
 
   // Walk the register/memloc assignments, inserting copies/loads.  In the case
   // of tail call optimization arguments are handle later.
   const X86RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
   for (unsigned I = 0, OutIndex = 0, E = ArgLocs.size(); I != E;
        ++I, ++OutIndex) {
     assert(OutIndex < Outs.size() && "Invalid Out index");
-    // Skip inalloca arguments, they have already been written.
+    // Skip inalloca/preallocated arguments, they have already been written.
     ISD::ArgFlagsTy Flags = Outs[OutIndex].Flags;
-    if (Flags.isInAlloca())
+    if (Flags.isInAlloca() || Flags.isPreallocated())
       continue;
 
     CCValAssign &VA = ArgLocs[I];
     EVT RegVT = VA.getLocVT();
     SDValue Arg = OutVals[OutIndex];
     bool isByVal = Flags.isByVal();
 
     // Promote the value if needed.
for (unsigned I = 0, OutsIndex = 0, E = ArgLocs.size(); I != E;
         }
 
         continue;
       }
 
       assert(VA.isMemLoc());
       SDValue Arg = OutVals[OutsIndex];
       ISD::ArgFlagsTy Flags = Outs[OutsIndex].Flags;
-      // Skip inalloca arguments.  They don't require any work.
-      if (Flags.isInAlloca())
+      // Skip inalloca/preallocated arguments.  They don't require any work.
+      if (Flags.isInAlloca() || Flags.isPreallocated())
         continue;
       // Create frame index.
       int32_t Offset = VA.getLocMemOffset()+FPDiff;
       uint32_t OpSize = (VA.getLocVT().getSizeInBits()+7)/8;
       FI = MF.getFrameInfo().CreateFixedObject(OpSize, Offset, true);
       FIN = DAG.getFrameIndex(FI, getPointerTy(DAG.getDataLayout()));
 
       if (Flags.isByVal()) {
X86TargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
   case X86::LCMPXCHG8B_SAVE_EBX:
   case X86::LCMPXCHG16B_SAVE_RBX: {
     unsigned BasePtr =
         MI.getOpcode() == X86::LCMPXCHG8B_SAVE_EBX ? X86::EBX : X86::RBX;
     if (!BB->isLiveIn(BasePtr))
       BB->addLiveIn(BasePtr);
     return BB;
   }
+  case TargetOpcode::CALL_SETUP: {
+    assert(Subtarget.is32Bit() && "callsetup only used in 32-bit");
+    auto MFI = MF->getInfo<X86MachineFunctionInfo>();
+    MFI->setHasCallSetup(true);
+    // Offsets from the stack pointer may be wrong because we have adjusted it
+    MFI->setForceFramePointer(true);
+    int64_t CallSetupId = MI.getOperand(0).getImm();
+    size_t StackAdjustment = MFI->GetCallSetupStackSize(CallSetupId);
+    assert(StackAdjustment != 0 && "0 stack adjustment");
+    LLVM_DEBUG(dbgs() << "CALL_SETUP stack adjustment " << StackAdjustment
+                      << "\n");
+    BuildMI(*BB, MI, DL, TII->get(X86::SUB32ri), X86::ESP)
+        .addReg(X86::ESP)
+        .addImm(StackAdjustment);
+    MI.eraseFromParent();
+    return BB;
+  }
+  case TargetOpcode::CALL_ALLOC: {
+    assert(Subtarget.is32Bit() && "callsetup only used in 32-bit");
+    int64_t CallSetupId = MI.getOperand(1).getImm();
+    int64_t ArgIdx = MI.getOperand(2).getImm();
+    auto MFI = MF->getInfo<X86MachineFunctionInfo>();
+    size_t ArgOffset = MFI->GetCallSetupArgOffsets(CallSetupId)[ArgIdx];
+    LLVM_DEBUG(dbgs() << "CALL_ALLOC arg index " << ArgIdx << ", arg offset "
+                      << ArgOffset << "\n");
+    // copy stack pointer + offset into reg
+    addRegOffset(
+        BuildMI(*BB, MI, DL, TII->get(X86::LEA32r), MI.getOperand(0).getReg()),
+        X86::ESP, false, ArgOffset);
+    MI.eraseFromParent();
+    return BB;
+  }
   }
 }
 
 //===----------------------------------------------------------------------===//
 //                           X86 Optimization Hooks
 //===----------------------------------------------------------------------===//
 
 bool

llvm/lib/Target/X86/X86MachineFunctionInfo.h

class X86MachineFunctionInfo : public MachineFunctionInfo {
   bool IsSplitCSR = false;
 
   /// True if this function uses the red zone.
   bool UsesRedZone = false;
 
   /// True if this function has WIN_ALLOCA instructions.
   bool HasWinAlloca = false;
 
+  ///
+  bool HasCallSetup = false;
+
+  size_t CallSetupNextId = 0;
+  ValueMap<const Value *, size_t> CallSetupIds;
+  DenseMap<size_t, size_t> CallSetupStackSizes;
+  DenseMap<size_t, SmallVector<size_t, 4>> CallSetupArgOffsets;
+
 private:
   /// ForwardedMustTailRegParms - A list of virtual and physical registers
   /// that must be forwarded to every musttail call.
   SmallVector<ForwardedRegister, 1> ForwardedMustTailRegParms;
 
 public:
   X86MachineFunctionInfo() = default;
 
public:
   bool isSplitCSR() const { return IsSplitCSR; }
   void setIsSplitCSR(bool s) { IsSplitCSR = s; }
 
   bool getUsesRedZone() const { return UsesRedZone; }
   void setUsesRedZone(bool V) { UsesRedZone = V; }
 
   bool hasWinAlloca() const { return HasWinAlloca; }
   void setHasWinAlloca(bool v) { HasWinAlloca = v; }
+
+  bool hasCallSetup() const { return HasCallSetup; }
+  void setHasCallSetup(bool v) { HasCallSetup = v; }
+
+  size_t CallSetupIdForCallSite(const Value *CS) {
+    auto Id = CallSetupIds.find(CS);
+    if (Id == CallSetupIds.end()) {
+      size_t NewId = CallSetupNextId++;
+      CallSetupIds.insert({CS, NewId});
+      return NewId;
+    } else {
+      return Id->second;
+    }
+  }
+
+  void SetCallSetupStackSize(size_t Id, size_t StackSize) {
+    CallSetupStackSizes[Id] = StackSize;
+  }
+
+  size_t GetCallSetupStackSize(const size_t Id) {
+    assert(CallSetupStackSizes.find(Id) != CallSetupStackSizes.end() &&
+           "stack size not set");
+    return CallSetupStackSizes[Id];
+  }
+
+  void SetCallSetupArgOffsets(size_t Id, SmallVector<size_t, 4> AO) {
+    CallSetupArgOffsets[Id] = AO;
+  }
+
+  const SmallVector<size_t, 4> &GetCallSetupArgOffsets(const size_t Id) {
+    assert(CallSetupArgOffsets.find(Id) != CallSetupArgOffsets.end() &&
+           "arg offsets not set");
+    return CallSetupArgOffsets[Id];
+  }
 };
 
 } // End llvm namespace
 
 #endif

llvm/lib/Target/X86/X86RegisterInfo.cpp

 //===----------------------------------------------------------------------===//
 // Stack Frame Processing methods
 //===----------------------------------------------------------------------===//
 
 static bool CantUseSP(const MachineFrameInfo &MFI) {
   return MFI.hasVarSizedObjects() || MFI.hasOpaqueSPAdjustment();
 }
 
 bool X86RegisterInfo::hasBasePointer(const MachineFunction &MF) const {
-   const MachineFrameInfo &MFI = MF.getFrameInfo();
+  const X86MachineFunctionInfo *X86FI = MF.getInfo<X86MachineFunctionInfo>();
+  if (X86FI->hasCallSetup())
+    return true;
+
 
-   if (!EnableBasePointer)
-     return false;
 
-   // When we need stack realignment, we can't address the stack from the frame
-   // pointer.  When we have dynamic allocas or stack-adjusting inline asm, we
-   // can't address variables from the stack pointer.  MS inline asm can
-   // reference locals while also adjusting the stack pointer.  When we can't
-   // use both the SP and the FP, we need a separate base pointer register.
-   bool CantUseFP = needsStackRealignment(MF);
-   return CantUseFP && CantUseSP(MFI);
 }
 
 bool X86RegisterInfo::canRealignStack(const MachineFunction &MF) const {
   if (!TargetRegisterInfo::canRealignStack(MF))
     return false;
 
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const MachineRegisterInfo *MRI = &MF.getRegInfo();

llvm/test/CodeGen/X86/preallocated.ll

+; RUN: llc < %s -mtriple=i686-pc-win32 | FileCheck %s
+
+declare token @llvm.call.preallocated.setup(i32)
+declare i8* @llvm.call.preallocated.arg(token, i32)
+
+%Foo = type { i32, i32 }
+
+declare void @f(%Foo* preallocated(%Foo))
+declare void @g(%Foo* preallocated(%Foo), i32)
+declare void @h(%Foo* sret, %Foo* preallocated(%Foo))
+declare void @init(%Foo*)
+
+define void @preallocated() {
+; CHECK-LABEL: _preallocated:
+entry:
+; CHECK: subl $8, %esp
+  %t = call token @llvm.call.preallocated.setup(i32 1)
+; CHECK: leal (%esp), [[REGISTER:%[a-z]+]]
+  %a = call i8* @llvm.call.preallocated.arg(token %t, i32 0)
+  %b = bitcast i8* %a to %Foo*
+; CHECK: pushl [[REGISTER]]
+; CHECK: calll _init
+  call void @init(%Foo* %b)
+; CHECK-NOT: subl {{\$[0-9]+}}, %esp
+; CHECK-NOT: pushl
+; CHECK: calll _f
+  call void @f(%Foo* preallocated(%Foo) %b) ["preallocated"(token %t)]
+  ret void
+}
+
+define void @one_preallocated_one_normal() {
+; CHECK-LABEL: _one_preallocated_one_normal:
+entry:
+; CHECK: subl $12, %esp
+  %t = call token @llvm.call.preallocated.setup(i32 1)
+; CHECK: leal (%esp), [[REGISTER:%[a-z]+]]
+  %a = call i8* @llvm.call.preallocated.arg(token %t, i32 0)
+  %b = bitcast i8* %a to %Foo*
+; CHECK: pushl [[REGISTER]]
+; CHECK: calll _init
+  call void @init(%Foo* %b)
+; CHECK-NOT: subl {{\$[0-9]+}}, %esp
+; CHECK-NOT: pushl
+; CHECK: movl $2, 8(%esp)
+; CHECK: calll _g
+  call void @g(%Foo* preallocated(%Foo) %b, i32 2) ["preallocated"(token %t)]
+  ret void
+}
+
+define void @nested() {
+; CHECK-LABEL: _nested:
+entry:
+  %tmp = alloca %Foo
+
+  %t1 = call token @llvm.call.preallocated.setup(i32 1)
+; CHECK: subl $12, %esp
+  %a1 = call i8* @llvm.call.preallocated.arg(token %t1, i32 0)
+  %b1 = bitcast i8* %a1 to %Foo*
+; CHECK: leal 4(%esp), [[REGISTER1:%[a-z]+]]
+
+  %t2 = call token @llvm.call.preallocated.setup(i32 1)
+; CHECK: subl $12, %esp
+  %a2 = call i8* @llvm.call.preallocated.arg(token %t2, i32 0)
+; CHECK: leal 4(%esp), [[REGISTER2:%[a-z]+]]
+  %b2 = bitcast i8* %a2 to %Foo*
+
+  call void @init(%Foo* %b2)
+; CHECK: pushl [[REGISTER2]]
+; CHECK: calll _init
+
+  call void @h(%Foo* %b1, %Foo* preallocated(%Foo) %b2) ["preallocated"(token %t2)]
+; CHECK-NOT: subl {{\$[0-9]+}}, %esp
+; CHECK-NOT: pushl
+; CHECK: calll _h
+  call void @h(%Foo* %tmp, %Foo* preallocated(%Foo) %b1) ["preallocated"(token %t1)]
+; CHECK-NOT: subl {{\$[0-9]+}}, %esp
+; CHECK-NOT: pushl
+; CHECK: calll _h
+  ret void
+}