Elide argument copies during instruction selection

[rnk]

Summary

Avoids tons of prologue boilerplate when arguments are passed in memory
and left in memory. This can happen in a debug build or in a release
build when an argument alloca is escaped. This will dramatically affect
the code size of x86 debug builds, because X86 fast isel doesn't handle
arguments passed in memory at all. It only handles the x86_64 case of up
to 6 basic register parameters.

This is implemented by analyzing the entry block before to ISel to
identify copy elision candidates. If an argument is a copy elision
candidate, we set a flag on the InputArg that we pass to
TargetLowering::LowerFormalArguments, similar to how we mark byval
arguments. The target does normal calling convention processing to
compute argument locations, and if the argument lives in memory and the
target recognizes the elision candidate flag, it will return a
FrameIndexSDNode instead of the usual load or physical register copy
SDValue. The generic ISel code recognizes this and replaces the old
frame index with the new fixed frame index in the StaticAllocaMap and
the debug declare table.

Supersedes D28388

Fixes PR26328

[hans]

Looks good as far as I can tell.

It adds more complexity to SelectionDAG, but I suppose it's the best place to do it.

include/llvm/CodeGen/SelectionDAGISel.h

59

These are pretty generalist names in a broad scope. Maybe name them around "ArgCopy" rather than just Copy or Elided instructions?

I suppose for the consumer of this set it doesn't care what kind of instruction it is really.. bot for ElisionFrameIndexMap at least maybe ArgCpyElisionFrameIndexMap would be better? (Pretty long though..)

lib/CodeGen/AsmPrinter/DwarfDebug.h

88

Unrelated, just commit?

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8057

I might have gone with just return &StaticAllocas[AI]

8461

You cleared it above too before starting to fill it, so maybe one of these is redundant?

test/CodeGen/X86/arg-copy-elide.ll

20

Is there a pop or something restoring the stack after the call? Should we check for that too?

[rnk]

• Rename things & tweak code

[chandlerc]

include/llvm/CodeGen/SelectionDAGISel.h

318

no doxygen?

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8047

SmallDenseMap? Seems like we can predict common upper bounds and this is a leaf-y function, so can avoid most memory allocation here.

8077

For a sequence of N casts of an alloca, doesn't this do O(N^2) work?

Maybe make GetInfoIfStaticAlloca fully memoize each Value?

8168–8171

Both findArgumentCopyElisionCandidates and elideArgumentCopy are only ever called from this function. Maybe make them local static helpers instead of putting them into the API?

Also, the only data structure used outside this function is the ElidedArgCopyInstrs. Maybe sink all the other data structtures down to be local? That seems like it would also help address Hans' comments about name.

I also don't see why you need flags for this when you have a data structure that tracks which things are candidates...

[rnk]

lib/CodeGen/AsmPrinter/DwarfDebug.h

88

It's related because this change triggers the assertion. As written, the assert is equivalent to:

assert(FI != -1);

However, -1 is a valid fixed frame index, and it is typically the one used for the first argument in memory.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8057

This is C++! We can't do two hash lookups. :)

8168–8171

I had it that way originally, but the set of elided instructions needs to make it into SelectBasicBlock so we can skip over them. At that point I made these things instance methods and fields. I'll flip it around and pass the data explicitly and see if we like that better.

test/CodeGen/X86/arg-copy-elide.ll

20

There is, but I don't want to overfit our choice for how to adjust the stack. It could be "pop" or "add $4, %esp". Matching the push doesn't feel like overfitting because it's a profitable transform that we're likely to keep.

[rnk]

• Sink elision back to static helpers
• Try to optimize away heap allocation in elision analysis

[rnk]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8047

Done, but I don't think the signals are that good.

8057

Elaborating on this, I think that it would be less clear that on first use, a static alloca starts in the "unknown" state.

8077

Do you think the memory use of memoization is worth optimizing for long chains of casts that mid-level optimizations usually remove?

[MatzeB]

include/llvm/CodeGen/MachineFrameInfo.h

573

Do not repeat the name of the documented function.

include/llvm/Target/TargetCallingConv.h

71–72

This header makes me cry: Define everything twice as bit and as the number of the bit. Just to use the number of the bit in a One << XXX expression below... Anyway unrelated to this patch which stays consistent with the existing madness.

128–129

Wrong Copy&Paste!

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8051

FuncInfo could be a reference as it mustn't be nullptr.

8074

Maybe document that the address of those allocas must not escape until a store is found (because that explains some of the complexity in the loop).

8077–8082

You could move those two checks into the if (!SI) block.

8131–8135

Most arguments could be reference as they mustn't be nullptr.

8382

Intuitively I would assume this should get a MO_Invariant flag for pretty much all ABIs here. I think however there is other code deducing this from the frame index, so it is probably fine to use the default pointer info here. If you haven't yet, please test that when you build an optimized function with/without your patch you get the same memory operands.

8442

extra space after =

lib/Target/X86/X86ISelLowering.cpp

2718

I don't know right now what the ABIs say, but I would expect the argument memory to be immutable. So maybe something like !AlwayseUseMutable would work here?

[rnk]

Thanks for the review!

include/llvm/Target/TargetCallingConv.h

71–72

Given my copy-pasto, I fixed this in rL294989. I can't believe how long we've lived with this manual masking.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8131–8135

Many of these are used in pointer-y ways like being inserted into a pointer set. IMO they're better as pointers.

8382

Actually, we need to ensure that the load is not invariant. Once we elide the copy, we can end up mutating the argument in the caller allocated memory. I have a test for this.

lib/Target/X86/X86ISelLowering.cpp

2718

x86 at least says the opposite. If the ABI required that we not mutate the argument in memory, we wouldn't be able to do this transformation. So, maybe this is a good place to comment that the non-immutability is important.

Also, it looks like during the evolution of this patch, I ended up not needing setIsImmutableObjectIndex. I'll remove that. It's redundant with this object creation.

[rnk]

• Implement suggestions from Mathias

[MatzeB]

LGTM

include/llvm/Target/TargetCallingConv.h

71–72

Thanks, this new version is so much simpler!

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8131–8135

It's certainly a subjective thing given that much of the existing code is written with pointers. I don't have a strong opinion on it, so keep it.

8382

Interesting, I didn't know that is legal. Fine with me then.

[chandlerc]

More comments...

include/llvm/Target/TargetCallingConv.h

30

This isn't really that the argument is an elision candidate, it means the argument's copy is *elided* and you transform based on this.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8095

What if the value being stored is the address of a static alloca? We don't mark it as clobbered above because we're in the store instruction case.

8121–8123

Would it be worth stopping the scan when you're out of arguments?

8250–8251

You erase things from the set below but don't clear the flags.... I assume this can't manifest because of how the flag is currently used, but it seems surprising at the least. Maybe sink this to the below code where you're going to do the elision?

[majnemer]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8097–8098

Why not just pass Dst into GetInfoIfStaticAlloca?

8111

Would !Arg->getType()->isSized() be more precise?

[rnk]

include/llvm/Target/TargetCallingConv.h

30

No, we don't. This flag tells the target that if the argument lives in memory, then it should return a FrameIndexSDNode, similar to the way that we would if the IsByVal flag was set. If the argument doesn't live in memory, it should ignore this flag. We only do the transform if a frame index comes back from TLI->LowerFormalArguments.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8111

That ignores element counts on arrays and vectors, right? Do you think that's right? I think what I really wanted to say was "I don't know what the heck would happen if we elided the copy of a zero sized argument, so let's not try".

8121–8123

Yeah, probably for -O0 builds with tons of allocas.

8250–8251

The flags are just temporary data, and they're only supposed to indicate candidates, not successfully transformed arguments. It doesn't seem worth clearing them.

I did notice that I can check the flag instead of doing a hash lookup below, though.

[rnk]

• David and Chandler's comments

[rnk]

• Fix bug when splitting i64 between registers and memory
• Add support for simple copy elision on non-x86 platforms

[rnk]

• Fix bug when splitting i64 between registers and memory
• Add support for simple copy elision on non-x86 platforms

[rnk]

This changed enough that it's worth another look.

[chandlerc]

FWIW, this looks fine to me with minor tweaks below. Would be great to have @MatzeB be happy as well though, this is a part of the backend I'm somewhat less familiar with.

include/llvm/Target/TargetCallingConv.h

30

Ah, OK. The term "candidate" is confusing me I think, but I don't reall yhave a better idea for a name...

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

8077

I just don't want a quadratic path. You could add a recursion limit instead if you never expect the tall chains in practice?

lib/Target/X86/X86ISelLowering.cpp

2746

Dy ou need the else? It's after a return...

2760–2768

You could just put this inside the if above in the for loop... Dunno, the loop above just feels awkward, but not sure there is really better way to do this.

[MatzeB]

In D29668#680683, @chandlerc wrote:

FWIW, this looks fine to me with minor tweaks below. Would be great to have @MatzeB be happy as well though, this is a part of the backend I'm somewhat less familiar with.

My LGTM still stands (though I agree with all the comments made in between). I'm not super familiar with SelectionDAG either though.

test/CodeGen/X86/pr30430.ll

96–103

Why are those classified as spills?

[MatzeB]

This seems to break swift. We probably have an i1 parameter ending in

int FI = MFI.CreateFixedObject(ArgVT.getSizeInBits() / 8,
                               VA.getLocMemOffset(), /*Immutable=*/false);

resulting in Assertion failed: (Size != 0 && "Cannot allocate zero size fixed stack objects!"), function CreateFixedObject, file /Volumes/Data/swift-next/lib/CodeGen/MachineFunction.cpp, line 824.

I am still working on a reproducer.

[MatzeB]

Reproducer in http://llvm.org/PR32136

[MatzeB]

I disabled the argument copy elision for illegal types in r296950 to get the swift bots back to green.

include/llvm/CodeGen/MachineFrameInfo.h

bool isImmutableObjectIndex(int ObjectIdx) const {
     // Tail calling functions can clobber their function arguments.
     if (HasTailCall)
       return false;
     assert(unsigned(ObjectIdx+NumFixedObjects) < Objects.size() &&
            "Invalid Object Idx!");
     return Objects[ObjectIdx+NumFixedObjects].isImmutable;
   }
 
+  /// setIsImmutableObjectIndex - Marks the immutability of an object.
+  void setIsImmutableObjectIndex(int ObjectIdx, bool Immutable) {
+    assert(unsigned(ObjectIdx+NumFixedObjects) < Objects.size() &&
+           "Invalid Object Idx!");
+    Objects[ObjectIdx+NumFixedObjects].isImmutable = Immutable;
+  }
+
   /// Returns true if the specified index corresponds to a spill slot.
   bool isSpillSlotObjectIndex(int ObjectIdx) const {
     assert(unsigned(ObjectIdx+NumFixedObjects) < Objects.size() &&
            "Invalid Object Idx!");
     return Objects[ObjectIdx+NumFixedObjects].isSpillSlot;
   }
 
   bool isStatepointSpillSlotObjectIndex(int ObjectIdx) const {

include/llvm/CodeGen/SelectionDAGISel.h

public:
   SelectionDAG *CurDAG;
   SelectionDAGBuilder *SDB;
   AliasAnalysis *AA;
   GCFunctionInfo *GFI;
   CodeGenOpt::Level OptLevel;
   const TargetInstrInfo *TII;
   const TargetLowering *TLI;
 
+  DenseMap<const Argument *, std::pair<const AllocaInst *, const StoreInst *>>
+      ArgCopyElisionCandidates;
+  SmallPtrSet<const Instruction *, 4> ElidedArgCopyInstrs;
+  DenseMap<int, int> ArgCopyElisionFrameIndexMap;
+
   static char ID;
 
   explicit SelectionDAGISel(TargetMachine &tm,
                             CodeGenOpt::Level OL = CodeGenOpt::Default);
   ~SelectionDAGISel() override;
 
   const TargetLowering *getTargetLowering() const { return TLI; }
 
private:
 
   /// OpcodeOffset - This is a cache used to dispatch efficiently into isel
   /// state machines that start with a OPC_SwitchOpcode node.
   std::vector<unsigned> OpcodeOffset;
 
   void UpdateChains(SDNode *NodeToMatch, SDValue InputChain,
                     SmallVectorImpl<SDNode *> &ChainNodesMatched,
                     bool isMorphNodeTo);
+
+  void findArgumentCopyElisionCandidates();
+
+  /// Elides the copy of Arg into the static alloca AI. Replaces all uses of the
+  /// original stack object created for AI with FixedIndex. Suppresses the code
+  /// generation of the store SI. Returns true if there are any remaining uses
+  /// Arg.
+  bool elideArgumentCopy(const Argument *Arg, const AllocaInst *AI,
+                         int FixedIndex, const StoreInst *SI);
 };
 
 }
 
 #endif /* LLVM_CODEGEN_SELECTIONDAGISEL_H */

include/llvm/Target/TargetCallingConv.h

 #include <cstdint>
 
 namespace llvm {
 namespace ISD {
 
   struct ArgFlagsTy {
   private:
     static const uint64_t NoFlagSet      = 0ULL;
     static const uint64_t ZExt           = 1ULL<<0;  ///< Zero extended
     static const uint64_t ZExtOffs       = 0;
     static const uint64_t SExt           = 1ULL<<1;  ///< Sign extended
     static const uint64_t SExtOffs       = 1;
     static const uint64_t InReg          = 1ULL<<2;  ///< Passed in register
     static const uint64_t InRegOffs      = 2;
     static const uint64_t SRet           = 1ULL<<3;  ///< Hidden struct-ret ptr
     static const uint64_t SRetOffs       = 3;
     static const uint64_t ByVal          = 1ULL<<4;  ///< Struct passed by value
private:
     static const uint64_t HvaStart       = 1ULL << 17; ///< HVA structure start
                                                        ///< for vectorcall
     static const uint64_t HvaStartOffs   = 17;
     static const uint64_t SecArgPass     = 1ULL << 18; ///< Second argument
                                                        ///< pass for vectorcall
     static const uint64_t SecArgPassOffs = 18;
     static const uint64_t OrigAlign      = 0x1FULL<<27;
     static const uint64_t OrigAlignOffs  = 27;
-    static const uint64_t ByValSize      = 0x3fffffffULL<<32; ///< Struct size
+    static const uint64_t ByValSize      = 0x1fffffffULL<<32; ///< Struct size
     static const uint64_t ByValSizeOffs  = 32;
-    static const uint64_t InConsecutiveRegsLast      = 0x1ULL<<62; ///< Struct size
+    static const uint64_t ElideCopy      = 0x1ULL<<61;
+    static const uint64_t ElideCopyOffs  = 61;
+    static const uint64_t InConsecutiveRegsLast      = 0x1ULL<<62;
     static const uint64_t InConsecutiveRegsLastOffs  = 62;
-    static const uint64_t InConsecutiveRegs      = 0x1ULL<<63; ///< Struct size
+    static const uint64_t InConsecutiveRegs      = 0x1ULL<<63;
     static const uint64_t InConsecutiveRegsOffs  = 63;
 
     static const uint64_t One            = 1ULL; ///< 1 of this type, for shifts
 
     uint64_t Flags = 0;
 
   public:
     ArgFlagsTy() = default;
public:
     void setReturned() { Flags |= One << ReturnedOffs; }
 
     bool isInConsecutiveRegs()  const { return Flags & InConsecutiveRegs; }
     void setInConsecutiveRegs() { Flags |= One << InConsecutiveRegsOffs; }
 
     bool isInConsecutiveRegsLast()  const { return Flags & InConsecutiveRegsLast; }
     void setInConsecutiveRegsLast() { Flags |= One << InConsecutiveRegsLastOffs; }
 
+    bool isCopyElisionCandidate()  const { return Flags & InConsecutiveRegsLast; }
+    void setCopyElisionCandidate() { Flags |= One << InConsecutiveRegsLastOffs; }
+
     unsigned getByValAlign() const {
       return (unsigned)
         ((One << ((Flags & ByValAlign) >> ByValAlignOffs)) / 2);
     }
     void setByValAlign(unsigned A) {
       Flags = (Flags & ~ByValAlign) |
         (uint64_t(Log2_32(A) + 1) << ByValAlignOffs);
     }

lib/CodeGen/AsmPrinter/DwarfDebug.h

public:
 
   /// Initialize from the MMI table.
   void initializeMMI(const DIExpression *E, int FI) {
     assert(Expr.empty() && "Already initialized?");
     assert(FrameIndex.empty() && "Already initialized?");
     assert(!MInsn && "Already initialized?");
 
     assert((!E || E->isValid()) && "Expected valid expression");
-    assert(~FI && "Expected valid index");
+    assert(FI != INT_MAX && "Expected valid index");
 
     Expr.push_back(E);
     FrameIndex.push_back(FI);
   }
 
   /// Initialize from a DBG_VALUE instruction.
   void initializeDbgValue(const MachineInstr *DbgValue) {
     assert(Expr.empty() && "Already initialized?");

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

static bool isOnlyUsedInEntryBlock(const Argument *A, bool FastISel) {
   const BasicBlock &Entry = A->getParent()->front();
   for (const User *U : A->users())
     if (cast<Instruction>(U)->getParent() != &Entry || isa<SwitchInst>(U))
       return false;  // Use not in entry block.
 
   return true;
 }
 
+void SelectionDAGISel::findArgumentCopyElisionCandidates() {
+  const Function &F = *MF->getFunction();
+  const DataLayout &DL = SDB->DAG.getDataLayout();
+
+  enum StaticAllocaInfo { Unknown, Clobbered, Elidable };
+  DenseMap<const AllocaInst *, StaticAllocaInfo> StaticAllocas;
+
+  auto GetInfoIfStaticAlloca = [&](const Value *V) -> StaticAllocaInfo * {
+    if (!V)
+      return nullptr;
+    V = V->stripPointerCasts();
+    const auto *AI = dyn_cast<AllocaInst>(V);
+    if (!AI || !AI->isStaticAlloca() || !FuncInfo->StaticAllocaMap.count(AI))
+      return nullptr;
+    auto Iter = StaticAllocas.insert({AI, Unknown});
+    return &Iter.first->second;
+  };
+
+  // Look for stores of arguments to static allocas. Look through bitcasts and
+  // GEPs to handle type coercions, as long as the alloca is fully initialized
+  // by the store.
+  // FIXME: Handle split stores.
+  for (const Instruction &I : F.getEntryBlock()) {
+    // We will look through cast uses, so ignore them completely.
+    if (I.isCast())
+      continue;
+    // Ignore debug info intrinsics, they don't escape or store to allocas.
+    if (isa<DbgInfoIntrinsic>(I))
+      continue;
+
+    // Look for stores, and handle non-store uses conservatively. Assume that
+    // this instruction clobbers all allocas it uses.
+    const auto *SI = dyn_cast<StoreInst>(&I);
+    if (!SI) {
+      for (const Use &U : I.operands()) {
+        if (StaticAllocaInfo *Info = GetInfoIfStaticAlloca(U))
+          *Info = StaticAllocaInfo::Clobbered;
+      }
+      continue;
+    }
+
+    // Check if the destination is a static alloca.
+    const Value *Dst = SI->getPointerOperand()->stripPointerCasts();
+    const auto *AI = dyn_cast<AllocaInst>(Dst);
+    StaticAllocaInfo *Info = GetInfoIfStaticAlloca(AI);
+    if (!Info)
+      continue;
+
+    // Skip allocas that have been initialized or clobbered.
+    if (*Info != StaticAllocaInfo::Unknown)
+      continue;
+
+    // Check if the stored value is an argument, and that this store fully
+    // initializes the alloca. Don't elide copies from the same argument twice.
+    const Value *Val = SI->getValueOperand()->stripPointerCasts();
+    const auto *Arg = dyn_cast<Argument>(Val);
+    if (!Arg || Arg->hasInAllocaAttr() || Arg->hasByValAttr() ||
+        Arg->getType()->isEmptyTy() ||
+        DL.getTypeStoreSize(Arg->getType()) !=
+            DL.getTypeAllocSize(AI->getAllocatedType()) ||
+        ArgCopyElisionCandidates.count(Arg)) {
+      *Info = StaticAllocaInfo::Clobbered;
+      continue;
+    }
+
+    DEBUG(dbgs() << "Found argument copy elision candidate: " << *AI << '\n');
+
+    // Mark this alloca and store for argument copy elision.
+    *Info = StaticAllocaInfo::Elidable;
+    ArgCopyElisionCandidates.insert({Arg, {AI, SI}});
+  }
+}
+
+bool SelectionDAGISel::elideArgumentCopy(const Argument *Arg,
+                                         const AllocaInst *AI,
+                                         int FixedIndex,
+                                         const StoreInst *SI) {
+  // Delete the old stack object and replace it with the fixed stack object.
+  // Mark the fixed stack object as being mutable. We may no longer move loads
+  // from these objects across calls or stores.
+  int &AllocaIndex = FuncInfo->StaticAllocaMap[AI];
+  int OldIndex = AllocaIndex;
+  MF->getFrameInfo().RemoveStackObject(OldIndex);
+  MF->getFrameInfo().setIsImmutableObjectIndex(FixedIndex, false);
+  AllocaIndex = FixedIndex;
+
+  DEBUG({
+    dbgs() << "Eliding argument copy from " << *Arg << " to " << *AI << '\n'
+           << "  Replacing frame index " << OldIndex << " with " << FixedIndex
+           << '\n';
+  });
+
+  ArgCopyElisionFrameIndexMap.insert({OldIndex, FixedIndex});
+
+  // Avoid emitting code for the store implementing the copy.
+  ElidedArgCopyInstrs.insert(SI);
+
+  for (const Value *U : Arg->users()) {
+    if (U != SI)
+      return true;
+  }
+  return false;
+}
+
 void SelectionDAGISel::LowerArguments(const Function &F) {
   SelectionDAG &DAG = SDB->DAG;
   SDLoc dl = SDB->getCurSDLoc();
   const DataLayout &DL = DAG.getDataLayout();
   SmallVector<ISD::InputArg, 16> Ins;
 
   if (!FuncInfo->CanLowerReturn) {
     // Put in an sret pointer parameter before all the other parameters.
     SmallVector<EVT, 1> ValueVTs;
     ComputeValueVTs(*TLI, DAG.getDataLayout(),
                     PointerType::getUnqual(F.getReturnType()), ValueVTs);
 
     // NOTE: Assuming that a pointer will never break down to more than one VT
     // or one register.
     ISD::ArgFlagsTy Flags;
     Flags.setSRet();
     MVT RegisterVT = TLI->getRegisterType(*DAG.getContext(), ValueVTs[0]);
     ISD::InputArg RetArg(Flags, RegisterVT, ValueVTs[0], true,
                          ISD::InputArg::NoArgIndex, 0);
     Ins.push_back(RetArg);
   }
 
+  // Look for stores of arguments to static allocas. Mark such arguments with a
+  // flag to ask the target to give us the memory location of that argument if
+  // available.
+  findArgumentCopyElisionCandidates();
+
   // Set up the incoming argument description vector.
   unsigned Idx = 0;
   for (const Argument &Arg : F.args()) {
     ++Idx;
     SmallVector<EVT, 4> ValueVTs;
     ComputeValueVTs(*TLI, DAG.getDataLayout(), Arg.getType(), ValueVTs);
     bool isArgValueUsed = !Arg.use_empty();
     unsigned PartBase = 0;
for (unsigned Value = 0, NumValues = ValueVTs.size();
           FrameAlign = TLI->getByValTypeAlignment(ElementTy, DL);
         Flags.setByValAlign(FrameAlign);
       }
       if (F.getAttributes().hasAttribute(Idx, Attribute::Nest))
         Flags.setNest();
       if (NeedsRegBlock)
         Flags.setInConsecutiveRegs();
       Flags.setOrigAlign(OriginalAlignment);
+      if (ArgCopyElisionCandidates.count(&Arg))
+        Flags.setCopyElisionCandidate();
 
       MVT RegisterVT = TLI->getRegisterType(*CurDAG->getContext(), VT);
       unsigned NumRegs = TLI->getNumRegisters(*CurDAG->getContext(), VT);
       for (unsigned i = 0; i != NumRegs; ++i) {
         ISD::InputArg MyFlags(Flags, RegisterVT, VT, isArgValueUsed,
                               Idx-1, PartBase+i*RegisterVT.getStoreSize());
         if (NumRegs > 1 && i == 0)
           MyFlags.Flags.setSplit();
NewRoot =
         SDB->DAG.getCopyToReg(NewRoot, SDB->getCurSDLoc(), SRetReg, ArgValue);
     DAG.setRoot(NewRoot);
 
     // i indexes lowered arguments.  Bump it past the hidden sret argument.
     // Idx indexes LLVM arguments.  Don't touch it.
     ++i;
   }
 
+  SmallVector<SDValue, 4> Chains;
+
   for (const Argument &Arg : F.args()) {
     ++Idx;
     SmallVector<SDValue, 4> ArgValues;
     SmallVector<EVT, 4> ValueVTs;
     ComputeValueVTs(*TLI, DAG.getDataLayout(), Arg.getType(), ValueVTs);
     unsigned NumValues = ValueVTs.size();
+    bool ArgHasUses = !Arg.use_empty();
+
+    // Elide the copying store if the target was able to give us a memory
+    // location for this argument.
+    bool ElidedArgCopy = false;
+    int ArgFrameIndex = INT_MAX;
+    auto ArgCopyIter = ArgCopyElisionCandidates.find(&Arg);
+    if (ArgCopyIter != ArgCopyElisionCandidates.end()) {
+      FrameIndexSDNode *FI = dyn_cast<FrameIndexSDNode>(InVals[i].getNode());
+      if (FI) {
+        ElidedArgCopy = true;
+        ArgFrameIndex = FI->getIndex();
+        ArgHasUses = elideArgumentCopy(&Arg, ArgCopyIter->second.first, ArgFrameIndex,
+                          ArgCopyIter->second.second);
+      } else {
+        ArgCopyElisionCandidates.erase(ArgCopyIter);
+      }
+    }
 
     // If this argument is unused then remember its value. It is used to generate
     // debugging information.
     bool isSwiftErrorArg =
         TLI->supportSwiftError() &&
         F.getAttributes().hasAttribute(Idx, Attribute::SwiftError);
-    if (Arg.use_empty() && NumValues && !isSwiftErrorArg) {
+    if (!ArgHasUses && NumValues && !isSwiftErrorArg) {
       SDB->setUnusedArgValue(&Arg, InVals[i]);
 
       // Also remember any frame index for use in FastISel.
       if (FrameIndexSDNode *FI =
           dyn_cast<FrameIndexSDNode>(InVals[i].getNode()))
         FuncInfo->setArgumentFrameIndex(&Arg, FI->getIndex());
     }
 
     for (unsigned Val = 0; Val != NumValues; ++Val) {
       EVT VT = ValueVTs[Val];
       MVT PartVT = TLI->getRegisterType(*CurDAG->getContext(), VT);
       unsigned NumParts = TLI->getNumRegisters(*CurDAG->getContext(), VT);
 
       // Even an apparant 'unused' swifterror argument needs to be returned. So
       // we do generate a copy for it that can be used on return from the
       // function.
-      if (!Arg.use_empty() || isSwiftErrorArg) {
+      if (ArgHasUses || isSwiftErrorArg) {
         Optional<ISD::NodeType> AssertOp;
         if (F.getAttributes().hasAttribute(Idx, Attribute::SExt))
           AssertOp = ISD::AssertSext;
         else if (F.getAttributes().hasAttribute(Idx, Attribute::ZExt))
           AssertOp = ISD::AssertZext;
 
-        ArgValues.push_back(getCopyFromParts(DAG, dl, &InVals[i],
-                                             NumParts, PartVT, VT,
-                                             nullptr, AssertOp));
+        if (ElidedArgCopy) {
+          ArgValues.push_back(
+              DAG.getLoad(VT, dl, NewRoot, InVals[i], MachinePointerInfo()));
+          Chains.push_back(ArgValues.back().getValue(1));
+        } else {
+          ArgValues.push_back(getCopyFromParts(DAG, dl, &InVals[i], NumParts,
+                                               PartVT, VT, nullptr, AssertOp));
+        }
       }
 
       i += NumParts;
     }
 
     // We don't need to do anything else for unused arguments.
     if (ArgValues.empty())
       continue;
if (!TM.Options.EnableFastISel && Res.getOpcode() == ISD::CopyFromReg) {
       }
     }
     if (!isOnlyUsedInEntryBlock(&Arg, TM.Options.EnableFastISel)) {
       FuncInfo->InitializeRegForValue(&Arg);
       SDB->CopyToExportRegsIfNeeded(&Arg);
     }
   }
 
+  if (!Chains.empty())
+    NewRoot =  DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Chains);
+
+  DAG.setRoot(NewRoot);
+
   assert(i == InVals.size() && "Argument register count mismatch!");
 
+  // If any argument copy elisions occurred and we have debug info, replace the
+  // stale frame indices.
+  MachineFunction::VariableDbgInfoMapTy &DbgDeclareInfo = MF->getVariableDbgInfo();
+  if (!DbgDeclareInfo.empty() && !ArgCopyElisionFrameIndexMap.empty()) {
+    for (MachineFunction::VariableDbgInfo &VI : DbgDeclareInfo) {
+      auto I = ArgCopyElisionFrameIndexMap.find(VI.Slot);
+      if (I != ArgCopyElisionFrameIndexMap.end())
+        VI.Slot = I->second;
+    }
+  }
+
   // Finally, if the target has anything special to do, allow it to do so.
   EmitFunctionEntryCode();
+  ArgCopyElisionCandidates.clear();
 }
 
 /// Handle PHI nodes in successor blocks.  Emit code into the SelectionDAG to
 /// ensure constants are generated when needed.  Remember the virtual registers
 /// that need to be added to the Machine PHI nodes as input.  We cannot just
 /// directly add them, because expansion might result in multiple MBB's for one
 /// BB.  As such, the start of the BB might correspond to a different MBB than
 /// the end.

lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp

bool SelectionDAGISel::runOnMachineFunction(MachineFunction &mf) {
   return true;
 }
 
 void SelectionDAGISel::SelectBasicBlock(BasicBlock::const_iterator Begin,
                                         BasicBlock::const_iterator End,
                                         bool &HadTailCall) {
   // Lower the instructions. If a call is emitted as a tail call, cease emitting
   // nodes for this block.
-  for (BasicBlock::const_iterator I = Begin; I != End && !SDB->HasTailCall; ++I)
+  for (BasicBlock::const_iterator I = Begin; I != End && !SDB->HasTailCall; ++I) {
+    if (!ElidedArgCopyInstrs.count(&*I))
       SDB->visit(*I);
+  }
 
   // Make sure the root of the DAG is up-to-date.
   CurDAG->setRoot(SDB->getControlRoot());
   HadTailCall = SDB->HasTailCall;
   SDB->clear();
 
   // Final step, emit the lowered DAG as machine code.
   CodeGenAndEmitDAG();
if (FastIS) {
         FastIS->startNewBlock();
 
       unsigned NumFastIselRemaining = std::distance(Begin, End);
       // Do FastISel on as many instructions as possible.
       for (; BI != Begin; --BI) {
         const Instruction *Inst = &*std::prev(BI);
 
         // If we no longer require this instruction, skip it.
-        if (isFoldedOrDeadInstruction(Inst, FuncInfo)) {
+        if (isFoldedOrDeadInstruction(Inst, FuncInfo) ||
+            ElidedArgCopyInstrs.count(Inst)) {
           --NumFastIselRemaining;
           continue;
         }
 
         // Bottom-up: reset the insert pos at the top, after any local-value
         // instructions.
         FastIS->recomputeInsertPt();
 
if (Begin != BI) {
       // not handled by FastISel. If FastISel is not run, this is the entire
       // block.
       bool HadTailCall;
       SelectBasicBlock(Begin, BI, HadTailCall);
     }
 
     FinishBasicBlock();
     FuncInfo->PHINodesToUpdate.clear();
+    ElidedArgCopyInstrs.clear();
   }
 
   propagateSwiftErrorVRegs(FuncInfo);
 
   delete FastIS;
   SDB->clearDanglingDebugInfo();
   SDB->SPDescriptor.resetPerFunctionState();
 }

lib/Target/X86/X86ISelLowering.cpp

if (Flags.isByVal()) {
     unsigned Bytes = Flags.getByValSize();
     if (Bytes == 0) Bytes = 1; // Don't create zero-sized stack objects.
     int FI = MFI.CreateFixedObject(Bytes, VA.getLocMemOffset(), isImmutable);
     // Adjust SP offset of interrupt parameter.
     if (CallConv == CallingConv::X86_INTR) {
       MFI.setObjectOffset(FI, Offset);
     }
     return DAG.getFrameIndex(FI, getPointerTy(DAG.getDataLayout()));
+  } else if (Flags.isCopyElisionCandidate()) {
+    // This is a copy elision candidate that happened to be in memory. Return
+    // the frame index without forming the load to indicate that copy elision
+    // succeeded.
+    int FI = MFI.CreateFixedObject(ValVT.getSizeInBits() / 8,
+                                   VA.getLocMemOffset(), false);
+    return DAG.getFrameIndex(FI, getPointerTy(DAG.getDataLayout()));
   } else {
     int FI = MFI.CreateFixedObject(ValVT.getSizeInBits()/8,
                                    VA.getLocMemOffset(), isImmutable);
 
     // Set SExt or ZExt flag.
     if (VA.getLocInfo() == CCValAssign::ZExt) {
       MFI.setObjectZExt(FI, true);
     } else if (VA.getLocInfo() == CCValAssign::SExt) {
SDValue Val = DAG.getLoad(
         ValVT, dl, Chain, FIN,
         MachinePointerInfo::getFixedStack(DAG.getMachineFunction(), FI));
     return ExtendedInMem ?
       DAG.getNode(ISD::TRUNCATE, dl, VA.getValVT(), Val) : Val;
   }
 }
 
 // FIXME: Get this from tablegen.
 static ArrayRef<MCPhysReg> get64BitArgumentGPRs(CallingConv::ID CallConv,
                                                 const X86Subtarget &Subtarget) {
   assert(Subtarget.is64Bit());
 
   if (Subtarget.isCallingConvWin64(CallConv)) {
     static const MCPhysReg GPR64ArgRegsWin64[] = {
       X86::RCX, X86::RDX, X86::R8,  X86::R9
     };
     return makeArrayRef(std::begin(GPR64ArgRegsWin64), std::end(GPR64ArgRegsWin64));
   }
 
   static const MCPhysReg GPR64ArgRegs64Bit[] = {
     X86::RDI, X86::RSI, X86::RDX, X86::RCX, X86::R8, X86::R9
   };
   return makeArrayRef(std::begin(GPR64ArgRegs64Bit), std::end(GPR64ArgRegs64Bit));
 }
 
 // FIXME: Get this from tablegen.
 static ArrayRef<MCPhysReg> get64BitArgumentXMMs(MachineFunction &MF,
                                                 CallingConv::ID CallConv,
                                                 const X86Subtarget &Subtarget) {
   assert(Subtarget.is64Bit());
   if (Subtarget.isCallingConvWin64(CallConv)) {
     // The XMM registers which might contain var arg parameters are shadowed
     // in their paired GPR.  So we only need to save the GPR to their home
     // slots.
     // TODO: __vectorcall will change this.
     return None;
   }
 
   const Function *Fn = MF.getFunction();

test/CodeGen/X86/2010-04-30-LocalAlloc-LandingPad.ll

 ; RUN: llc < %s -O0 -regalloc=fast -relocation-model=pic -disable-fp-elim | FileCheck %s
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:32:64-v64:64:64-v128:128:128-a0:0:64-f80:128:128-n8:16:32"
 target triple = "i386-apple-darwin10.0.0"
 
 %struct.S = type { [2 x i8*] }
 
 @_ZTIi = external constant i8*                    ; <i8**> [#uses=1]
 @.str = internal constant [4 x i8] c"%p\0A\00"    ; <[4 x i8]*> [#uses=1]
 @llvm.used = appending global [1 x i8*] [i8* bitcast (i8* (%struct.S*, i32, %struct.S*)* @_Z4test1SiS_ to i8*)], section "llvm.metadata" ; <[1 x i8*]*> [#uses=0]
 
-; Verify that %esi gets spilled before the call.
+; Verify that %s1 gets spilled before the call.
 ; CHECK: Z4test1SiS
-; CHECK: movl %esi,{{.*}}(%ebp) 
+; CHECK: leal 8(%ebp), %[[reg:[^ ]*]]
+; CHECK: movl %[[reg]],{{.*}}(%ebp) ## 4-byte Spill
 ; CHECK: calll __Z6throwsv
 
 define i8* @_Z4test1SiS_(%struct.S* byval %s1, i32 %n, %struct.S* byval %s2) ssp personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
 entry:
   %retval = alloca i8*, align 4                   ; <i8**> [#uses=2]
   %n.addr = alloca i32, align 4                   ; <i32*> [#uses=1]
   %_rethrow = alloca i8*                          ; <i8**> [#uses=4]
   %0 = alloca i32, align 4                        ; <i32*> [#uses=1]

test/CodeGen/X86/arg-copy-elide.ll

+; RUN: llc -mtriple=i686-windows < %s | FileCheck %s
+
+declare void @addrof_i32(i32*)
+declare void @addrof_i64(i64*)
+declare void @addrof_i128(i128*)
+declare void @addrof_i32_x3(i32*, i32*, i32*)
+
+define void @simple(i32 %x) {
+entry:
+  %x.addr = alloca i32
+  store i32 %x, i32* %x.addr
+  call void @addrof_i32(i32* %x.addr)
+  ret void
+}
+
+; CHECK-LABEL: _simple:
+; CHECK: leal 4(%esp), %[[reg:[^ ]*]]
+; CHECK: pushl %[[reg]]
+; CHECK: calll _addrof_i32
+; CHECK: retl
+
+
+; We need to load %x before calling addrof_i32 now because it could mutate %x in
+; place.
+
+define i32 @use_arg(i32 %x) {
+entry:
+  %x.addr = alloca i32
+  store i32 %x, i32* %x.addr
+  call void @addrof_i32(i32* %x.addr)
+  ret i32 %x
+}
+
+; CHECK-LABEL: _use_arg:
+; CHECK: pushl %[[csr:[^ ]*]]
+; CHECK-DAG: movl 8(%esp), %[[csr]]
+; CHECK-DAG: leal 8(%esp), %[[reg:[^ ]*]]
+; CHECK: pushl %[[reg]]
+; CHECK: calll _addrof_i32
+; CHECK: movl %[[csr]], %eax
+; CHECK: popl %[[csr]]
+; CHECK: retl
+
+
+define i64 @split_i64(i64 %x) {
+entry:
+  %x.addr = alloca i64
+  store i64 %x, i64* %x.addr
+  call void @addrof_i64(i64* %x.addr)
+  ret i64 %x
+}
+
+; CHECK-LABEL: _split_i64:
+; CHECK: pushl %ebp
+; CHECK: movl %esp, %ebp
+; CHECK: pushl %[[csr2:[^ ]*]]
+; CHECK: pushl %[[csr1:[^ ]*]]
+; CHECK: andl $-8, %esp
+; CHECK-DAG: movl 8(%ebp), %[[csr1]]
+; CHECK-DAG: movl 12(%ebp), %[[csr2]]
+; CHECK-DAG: leal 8(%ebp), %[[reg:[^ ]*]]
+; CHECK: pushl %[[reg]]
+; CHECK: calll _addrof_i64
+; CHECK-DAG: movl %[[csr1]], %eax
+; CHECK-DAG: movl %[[csr2]], %edx
+; CHECK: leal -8(%ebp), %esp
+; CHECK: popl %[[csr1]]
+; CHECK: popl %[[csr2]]
+; CHECK: popl %ebp
+; CHECK: retl
+
+
+; The code we generate for this is unimportant. This is mostly a crash test.
+
+define void @split_i128(i128* %sret, i128 %x) {
+entry:
+  %x.addr = alloca i128
+  store i128 %x, i128* %x.addr
+  call void @addrof_i128(i128* %x.addr)
+  store i128 %x, i128* %sret
+  ret void
+}
+
+; CHECK-LABEL: _split_i128:
+; CHECK: pushl %ebp
+; CHECK: calll _addrof_i128
+; CHECK: retl
+
+
+; Check that we load all of x, y, and z before the call.
+
+define i32 @three_args(i32 %x, i32 %y, i32 %z) {
+entry:
+  %z.addr = alloca i32, align 4
+  %y.addr = alloca i32, align 4
+  %x.addr = alloca i32, align 4
+  store i32 %z, i32* %z.addr, align 4
+  store i32 %y, i32* %y.addr, align 4
+  store i32 %x, i32* %x.addr, align 4
+  call void @addrof_i32_x3(i32* %x.addr, i32* %y.addr, i32* %z.addr)
+  %s1 = add i32 %x, %y
+  %sum = add i32 %s1, %z
+  ret i32 %sum
+}
+
+; CHECK-LABEL: _three_args:
+; CHECK: pushl %[[csr:[^ ]*]]
+; CHECK-DAG: movl {{[0-9]+}}(%esp), %[[csr]]
+; CHECK-DAG: addl {{[0-9]+}}(%esp), %[[csr]]
+; CHECK-DAG: addl {{[0-9]+}}(%esp), %[[csr]]
+; CHECK-DAG: leal 8(%esp), %[[x:[^ ]*]]
+; CHECK-DAG: leal 12(%esp), %[[y:[^ ]*]]
+; CHECK-DAG: leal 16(%esp), %[[z:[^ ]*]]
+; CHECK: pushl %[[z]]
+; CHECK: pushl %[[y]]
+; CHECK: pushl %[[x]]
+; CHECK: calll _addrof_i32_x3
+; CHECK: movl %[[csr]], %eax
+; CHECK: popl %[[csr]]
+; CHECK: retl
+
+
+define void @two_args_same_alloca(i32 %x, i32 %y) {
+entry:
+  %x.addr = alloca i32
+  store i32 %x, i32* %x.addr
+  store i32 %y, i32* %x.addr
+  call void @addrof_i32(i32* %x.addr)
+  ret void
+}
+
+; CHECK-LABEL: _two_args_same_alloca:
+; CHECK: movl 8(%esp), {{.*}}
+; CHECK: movl {{.*}}, 4(%esp)
+; CHECK: leal 4(%esp), %[[reg:[^ ]*]]
+; CHECK: pushl %[[reg]]
+; CHECK: calll _addrof_i32
+; CHECK: retl
+
+
+define void @avoid_byval(i32* byval %x) {
+entry:
+  %x.p.p = alloca i32*
+  store i32* %x, i32** %x.p.p
+  call void @addrof_i32(i32* %x)
+  ret void
+}
+
+; CHECK-LABEL: _avoid_byval:
+; CHECK: leal {{[0-9]+}}(%esp), %[[reg:[^ ]*]]
+; CHECK: pushl %[[reg]]
+; CHECK: calll _addrof_i32
+; CHECK: retl
+
+
+define void @avoid_inalloca(i32* inalloca %x) {
+entry:
+  %x.p.p = alloca i32*
+  store i32* %x, i32** %x.p.p
+  call void @addrof_i32(i32* %x)
+  ret void
+}
+
+; CHECK-LABEL: _avoid_inalloca:
+; CHECK: leal {{[0-9]+}}(%esp), %[[reg:[^ ]*]]
+; CHECK: pushl %[[reg]]
+; CHECK: calll _addrof_i32
+; CHECK: retl

test/CodeGen/X86/inline-asm-tied.ll

 ; RUN: llc < %s -mtriple=i386-apple-darwin9 -O0 -optimize-regalloc -regalloc=basic -no-integrated-as | FileCheck %s
 ; rdar://6992609
 
-; CHECK: movl %ecx, 4([[ESP:%e..]])
-; CHECK: movl 4([[ESP]]), [[EDX:%e..]]
-; CHECK: movl [[EDX]], 4([[ESP]])
 target triple = "i386-apple-darwin9.0"
[e-mail] = appending global [1 x i8*] [i8* bitcast (i64 (i64)* @_OSSwapInt64 to i8*)], section "llvm.metadata"    ; <[1 x i8*]*> [#uses=0]
 
 define i64 @_OSSwapInt64(i64 %_data) nounwind {
 entry:
-  %retval = alloca i64    ; <i64*> [#uses=2]
-  %_data.addr = alloca i64    ; <i64*> [#uses=4]
-  store i64 %_data, i64* %_data.addr
-  %tmp = load i64, i64* %_data.addr    ; <i64> [#uses=1]
-  %0 = call i64 asm "bswap   %eax\0A\09bswap   %edx\0A\09xchgl   %eax, %edx", "=A,0,~{dirflag},~{fpsr},~{flags}"(i64 %tmp) nounwind    ; <i64> [#uses=1]
-  store i64 %0, i64* %_data.addr
-  %tmp1 = load i64, i64* %_data.addr    ; <i64> [#uses=1]
-  store i64 %tmp1, i64* %retval
-  %1 = load i64, i64* %retval    ; <i64> [#uses=1]
-  ret i64 %1
+  %0 = call i64 asm "bswap   %eax\0A\09bswap   %edx\0A\09xchgl   %eax, %%edx", "=A,0,~{dirflag},~{fpsr},~{flags}"(i64 %_data) nounwind
+  ret i64 %0
 }
 
+; CHECK-LABEL: __OSSwapInt64:
+; CHECK-DAG: movl 8(%esp), %edx
+; CHECK-DAG: movl 4(%esp), %eax
+; CHECK: ## InlineAsm Start
+; CHECK: ## InlineAsm End
+;       Everything is set up in EAX:EDX, return immediately.
+; CHECK-NEXT: retl
+
 ; The tied operands are not necessarily in the same order as the defs.
 ; PR13742
 define i64 @swapped(i64 %x, i64 %y) nounwind {
 entry:
-  %x0 = call { i64, i64 } asm "foo", "=r,=r,1,0,~{dirflag},~{fpsr},~{flags}"(i64 %x, i64 %y) nounwind
-        %x1 = extractvalue { i64, i64 } %x0, 0
-        ret i64 %x1
+  %x0 = call { i64, i64 } asm "foo", "=r,=r,1,0,~{dirflag},~{fpsr},~{flags}"(i64 %x, i64 %y) nounwind
 }

test/CodeGen/X86/pr30430.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown -mattr=+avx512f -O0 | FileCheck %s
 
 define <16 x float> @makefloat(float %f1, float %f2, float %f3, float %f4, float %f5, float %f6, float %f7, float %f8, float %f9, float %f10, float %f11, float %f12, float %f13, float %f14, float %f15, float %f16) #0 {
 ; CHECK-LABEL: makefloat:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    pushq %rbp
 ; CHECK-NEXT:  .Lcfi0:
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:  .Lcfi1:
 ; CHECK-NEXT:    .cfi_offset %rbp, -16
 ; CHECK-NEXT:    movq %rsp, %rbp
 ; CHECK-NEXT:  .Lcfi2:
 ; CHECK-NEXT:    .cfi_def_cfa_register %rbp
 ; CHECK-NEXT:    andq $-64, %rsp
 ; CHECK-NEXT:    subq $256, %rsp # imm = 0x100
-; CHECK-NEXT:    vmovss {{.*#+}} xmm8 = mem[0],zero,zero,zero
-; CHECK-NEXT:    vmovss {{.*#+}} xmm9 = mem[0],zero,zero,zero
-; CHECK-NEXT:    vmovss {{.*#+}} xmm10 = mem[0],zero,zero,zero
-; CHECK-NEXT:    vmovss {{.*#+}} xmm11 = mem[0],zero,zero,zero
-; CHECK-NEXT:    vmovss {{.*#+}} xmm12 = mem[0],zero,zero,zero
-; CHECK-NEXT:    vmovss {{.*#+}} xmm13 = mem[0],zero,zero,zero
-; CHECK-NEXT:    vmovss {{.*#+}} xmm14 = mem[0],zero,zero,zero
-; CHECK-NEXT:    vmovss {{.*#+}} xmm15 = mem[0],zero,zero,zero
 ; CHECK-NEXT:    vmovss %xmm0, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    vmovss %xmm1, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    vmovss %xmm2, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    vmovss %xmm3, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    vmovss %xmm4, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    vmovss %xmm5, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    vmovss %xmm6, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    vmovss %xmm7, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    vmovss %xmm15, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    vmovss %xmm14, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    vmovss %xmm13, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    vmovss %xmm12, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    vmovss %xmm11, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    vmovss %xmm10, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    vmovss %xmm9, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    vmovss %xmm8, (%rsp)
 ; CHECK-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; CHECK-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
 ; CHECK-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
 ; CHECK-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
 ; CHECK-NEXT:    vmovss {{.*#+}} xmm4 = mem[0],zero,zero,zero
 ; CHECK-NEXT:    vmovss {{.*#+}} xmm5 = mem[0],zero,zero,zero
 ; CHECK-NEXT:    vmovss {{.*#+}} xmm6 = mem[0],zero,zero,zero
 ; CHECK-NEXT:    vmovss {{.*#+}} xmm7 = mem[0],zero,zero,zero

 ; CHECK-NEXT:    # implicit-def: %YMM3
 ; CHECK-NEXT:    vmovaps %xmm1, %xmm3
 ; CHECK-NEXT:    vinsertf128 $1, %xmm0, %ymm3, %ymm3
 ; CHECK-NEXT:    # implicit-def: %ZMM16
 ; CHECK-NEXT:    vmovaps %zmm3, %zmm16
 ; CHECK-NEXT:    vinsertf64x4 $1, %ymm2, %zmm16, %zmm16
 ; CHECK-NEXT:    vmovaps %zmm16, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    vmovaps {{[0-9]+}}(%rsp), %zmm0
 ; CHECK-NEXT:    movq %rbp, %rsp
 ; CHECK-NEXT:    popq %rbp
 ; CHECK-NEXT:    retq
 entry:
   %__A.addr.i = alloca float, align 4
   %__B.addr.i = alloca float, align 4
   %__C.addr.i = alloca float, align 4
   %__D.addr.i = alloca float, align 4
   %__E.addr.i = alloca float, align 4
   %__F.addr.i = alloca float, align 4
   %__G.addr.i = alloca float, align 4
   %__H.addr.i = alloca float, align 4
   %__I.addr.i = alloca float, align 4
   %__J.addr.i = alloca float, align 4
   %__K.addr.i = alloca float, align 4
   %__L.addr.i = alloca float, align 4

test/CodeGen/X86/sse-intrinsics-fast-isel.ll

; X64-NEXT:    retq
   %res2  = insertelement <4 x float> %res1, float %a0, i32 2
   %res3  = insertelement <4 x float> %res2, float %a0, i32 3
   ret <4 x float> %res3
 }
 
 define void @test_mm_setcsr(i32 %a0) nounwind {
 ; X32-LABEL: test_mm_setcsr:
 ; X32:       # BB#0:
-; X32-NEXT:    pushl %eax
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    movl %esp, %ecx
-; X32-NEXT:    movl %eax, (%esp)
-; X32-NEXT:    ldmxcsr (%ecx)
-; X32-NEXT:    popl %eax
+; X32-NEXT:    leal 4(%esp), %eax
+; X32-NEXT:    ldmxcsr (%eax)
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_setcsr:
 ; X64:       # BB#0:
 ; X64-NEXT:    leaq -{{[0-9]+}}(%rsp), %rax
 ; X64-NEXT:    movl %edi, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    ldmxcsr (%rax)
 ; X64-NEXT:    retq

test/CodeGen/X86/win64_eh.ll

entry:
   %tmp13 = add i32 %tmp10, %tmp12
   %tmp14 = load i32, i32* %f
   %tmp15 = mul i32 %tmp14, 13
   %tmp16 = add i32 %tmp13, %tmp15
   ret i32 %tmp16
 }
 ; WIN64-LABEL: foo3:
 ; WIN64: .seh_proc foo3
-; NORM:  subq $24, %rsp
-; ATOM:  leaq -24(%rsp), %rsp
-; WIN64: .seh_stackalloc 24
+; NORM:  subq $16, %rsp
+; ATOM:  leaq -16(%rsp), %rsp
+; WIN64: .seh_stackalloc 16
 ; WIN64: .seh_endprologue
-; WIN64: addq $24, %rsp
+; WIN64: addq $16, %rsp
 ; WIN64: ret
 ; WIN64: .seh_endproc
 
 
 ; Check emission of eh handler and handler data
 declare i32 @_d_eh_personality(i32, i32, i64, i8*, i8*)
 declare void @_d_eh_resume_unwind(i8*)
 

test/DebugInfo/X86/discriminator.ll

 !11 = distinct !DILexicalBlock(line: 2, column: 0, file: !1, scope: !4)
 !12 = !DILocation(line: 3, scope: !4)
 !13 = !DILocation(line: 4, scope: !4)
 !14 = !DILocation(line: 2, scope: !15)
 !15 = !DILexicalBlockFile(discriminator: 42, file: !1, scope: !4)
 
 ; CHECK: Address            Line   Column File   ISA Discriminator Flags
 ; CHECK: ------------------ ------ ------ ------ --- ------------- -------------
-; CHECK: 0x0000000000000011      2      0      1   0            42 {{$}}
+; CHECK: 0x000000000000000a      2      0      1   0            42 {{$}}