[DAGCombiner] Eliminate dead stores to stack.

[courbet]

Summary

A store to an object whose lifetime is about to end can be removed.

See PR40550 for motivation.

[courbet]

Sorry, something wrong happened with the diff

[courbet]

rebase

[courbet]

test/DebugInfo/COFF/inlining.ll

32 ↗

(On Diff #184601)

Before:

addl    $6, x(%rip)
 addl    $4, x(%rip)
 movl    $1, -4(%rsp)
 leaq    -4(%rsp), %rax
 addl    %eax, x(%rip)
 addl    $2, x(%rip)
 addl    $3, x(%rip)
 addl    $5, x(%rip)
 addl    $7, x(%rip)
 retq

After:

addl    $6, x(%rip)
 addl    $4, x(%rip)
 leaq    -4(%rsp), %rax
 addl    %eax, x(%rip)
 addl    $2, x(%rip)
 addl    $3, x(%rip)
 addl    $5, x(%rip)
 addl    $7, x(%rip)

[courbet]

I just realized that there is nothing that guarantees that an alloca's lifetime is ended in one go, so I'll also have to check that the store is within the lifetime.end base and size.

[niravd]

In D57541#1380701, @courbet wrote:

I just realized that there is nothing that guarantees that an alloca's lifetime is ended in one go, so I'll also have to check that the store is within the lifetime.end base and size.

Yes. You should use the BaseIndexOffset alias framework here.

Also, you'll need to deal with the additional output from indexed stores.

We should probably also add this some LIFETIME nodes casing to FindBetterChains and improve LIFETIME_END's chain so we can deal with overlapping lifetimes.

[courbet]

Only remove stores to dead stack space when they are fully contained inside the
object whose lifetime gets terminated.

[courbet]

Also, you'll need to deal with the additional output from indexed stores.

I think I'm fine because of the hasOneUse() check. I'm not sure how I should test this though, do you have a suggestion for some test IR for an indexed store ?

[niravd]

In D57541#1384961, @courbet wrote:

Also, you'll need to deal with the additional output from indexed stores.

I think I'm fine because of the hasOneUse() check. I'm not sure how I should test this though, do you have a suggestion for some test IR for an indexed store ?

I believe the CombineTo will fail from too few values on indexed stores, even if the value isn't used. Better to check isIndexed() or do the unfolding back to an ADD/SUB.

I suspect the only way this would trigger currently is if an intrinsic lowers into an indexed store in ARM or AArch64.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

15386 ↗

(On Diff #185280)

I think we should merge this with FindBetterChains. At the least we should have load/store aliasing be able to bypass unrelated lifetime nodes and vice versa.

[courbet]

• Add an explicit isIndexed() check to ignore indexed stores.
• Teach FindBetterChain to bypass unrelated lifetime nodes.

[courbet]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

15386 ↗

(On Diff #185280)

I'm not exactly sure how to merge it, but teaching it to bypass unrelated lifetime nodes sounds like a good idea. Done.

[courbet]

clang-format

[niravd]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

15386 ↗

(On Diff #185280)

I was thinking of this as two combines: One sharing logic with FindBetterChains to improve LIFETIME_END's chain to simplify it's chain to (presumably just the store chain) and the other to check the chain of a LIFETIME_END like we do with dead stores via overlapping stores. You already have the second.

15397 ↗

(On Diff #185577)

LIFETIME_START as well?

15402 ↗

(On Diff #185577)

You can remove "!ST->hasOneUse()" now that you have the indexed check.

15412 ↗

(On Diff #185577)

Nit: return CombineTo(ST, ST->getChain);

lib/CodeGen/SelectionDAG/SelectionDAGAddressAnalysis.cpp

90 ↗

(On Diff #185577)

This is "contains".

[courbet]

• search dead stores through non-aliasing LIFETIME_STARTs
• Better comments.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

15386 ↗

(On Diff #185280)

I see. There is an issue with making GatherAllAliases work with LifetimeSDNode: LifetimeSDNode cannot be a MemSDNode (or LSBaseSDNode) because MemSDNodes really represent memory accesses, and lifetime nodes really are just references to memory, they never access it. Code manipulating the MemSDNode member MachineMemOperand assumes either load or store, and changing this would require significant changes.

On the other hand now that I've made FindBetterChain able to go past lifetime start/end nodes as you've suggested in previous comments, it will be able to move stores out of unrelated LIFETIME_{START,END}.

15397 ↗

(On Diff #185577)

Indeed, but only for lifetime_starts that do not overlap. I'm not checking the overlap for lifetime_end because you cannot have:

lifetime_end(a);
lifetime_end(a);

but you might have:

lifetime_start(a);
lifetime_end(a);
lifetime_start(a); // 3
lifetime_end(a);   // 4

and you don't want 4 to go past 3.

15402 ↗

(On Diff #185577)

I still need to check the !hasOneUse() check because the store might be depended on by several users.

15412 ↗

(On Diff #185577)

Not sure I understand why ?

lib/CodeGen/SelectionDAG/SelectionDAGAddressAnalysis.cpp

90 ↗

(On Diff #185577)

Not exactly. contains means "one is within the other" (and is not symmetrical), here it's "one might overlap the other" (and is symmetrical). I've added graphical comments + the symmetric formula for reference.

[niravd]

LGTM, modulo potentially excising the hasOneUse as I've suggested provided you agree with my line of reasoning there.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

15397 ↗

(On Diff #185577)

Yes. Maybe add the check for LIFETIME_END as well, just as a sanity check. I worry that we may have a case where a and a' overlap.

lifetime_start(a);
lifetime_end(a);
lifetime_start(a'); 3
lifetime_end(a'); 4

But it's fine as is.

15402 ↗

(On Diff #185577)

Shoudn't the fact that that the store is an immediate (or at least only through TF) predecessor to the LIFETIME_END be sufficient proof that the store can be removed? In which case we should be able to remove it completely.

The only way it'll fail the hasOneUse case is if we have some redundancy somewhere (presumably because we gave up partway in our alias analysis).

I don't expect to see this very often, so either way is probably fine.

15412 ↗

(On Diff #185577)

Oh right. You're returning LIFETIME_END. Nevermind.

lib/CodeGen/SelectionDAG/SelectionDAGAddressAnalysis.cpp

90 ↗

(On Diff #185577)

Right!

[courbet]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

15402 ↗

(On Diff #185577)

Shoudn't the fact that that the store is an immediate (or at least only through TF) predecessor to the LIFETIME_END be sufficient proof that > the store can be removed? In which case we should be able to remove it completely.

Consider the example in test onealloc_readback_1. During the combine, just before we visit t23 in the dag looks like this:

SelectionDAG has 26 nodes:
  t0: ch = EntryToken
  t2: i64,ch = CopyFromReg t0, Register:i64 %0
    t9: ch = lifetime.start<16 to 32> t0, TargetFrameIndex:i64<0>
  t10: ch = lifetime.start<0 to 16> t9, TargetFrameIndex:i64<0>
  t11: i64 = Constant<0>
  t13: v16i8,ch = load<(load 16 from %ir.a, align 1)> t10, t2, undef:i64
      t7: i64 = add nuw FrameIndex:i64<0>, Constant:i64<16>
    t14: ch = store<(store 16 into %ir.part1, align 1)> t10, t13, t7, undef:i64
  t15: ch = TokenFactor t13:1, t14
    t4: i64,ch = CopyFromReg t0, Register:i64 %1
  t16: v16i8,ch = load<(load 16 from %ir.b, align 1)> t15, t4, undef:i64
  t17: ch = store<(store 16 into %ir.part21)> t15, t16, FrameIndex:i64<0>, undef:i64
  t28: v16i8,ch = load<(load 16 from %ir.part21)> t17, FrameIndex:i64<0>, undef:i64
          t18: ch = TokenFactor t16:1, t17
        t19: ch = lifetime.end<16 to 32> t18, TargetFrameIndex:i64<0>
        t26: ch = store<(store 16 into %ir.a, align 1)> t0, t28, t2, undef:i64
      t30: ch = TokenFactor t19, t28:1, t26
    t23: ch = lifetime.end<0 to 16> t30, TargetFrameIndex:i64<0>
  t25: ch = X86ISD::RET_FLAG t23, TargetConstant:i32<0>

Note how the store t17 is in immediate predecessor (modulo TF & non-aliasing lifetime end) of lifetime end t23. However, load t28 also depends on t17. But the lifetime end t23 does not care about the load, so it makes sense for t23 not to have a chain on t28.

[courbet]

• Add the check for overlap when traversing LIFETIME_ENDs.
• print the extents of the lifetime nodes when printing the DAG for debug.

[niravd]

llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

15524

Ahha! We never add the lifetime to the Alias list like we should if we dont' skip past it. I suspect this is why we had to revert this patch.

[courbet]

I finally had time to track this down.

The issue was that we were missing a hasOneUse() check on the chain (we were only checking it on the store itself), so there could be other nodes dependind on the store (typically through tokenfactors), e.g.

store ----> TF ---> LIFETIME_END
             \-----> some_user

We were deleting the store...

llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

15524

I think you meant to put this comment on the lifetime skipping in GatherAllAliases. But you're right, thanks.

llvm/trunk/include/llvm/CodeGen/SelectionDAG.h

SDValue getMemIntrinsicNode(
     MachineMemOperand::Flags Flags
     = MachineMemOperand::MOLoad | MachineMemOperand::MOStore,
     unsigned Size = 0);
 
   SDValue getMemIntrinsicNode(unsigned Opcode, const SDLoc &dl, SDVTList VTList,
                               ArrayRef<SDValue> Ops, EVT MemVT,
                               MachineMemOperand *MMO);
 
+  /// Creates a LifetimeSDNode that starts (`IsStart==true`) or ends
+  /// (`IsStart==false`) the lifetime of the portion of `FrameIndex` between
+  /// offsets `Offset` and `Offset + Size`.
+  SDValue getLifetimeNode(bool IsStart, const SDLoc &dl, SDValue Chain,
+                          int FrameIndex, int64_t Size, int64_t Offset = -1);
+
   /// Create a MERGE_VALUES node from the given operands.
   SDValue getMergeValues(ArrayRef<SDValue> Ops, const SDLoc &dl);
 
   /// Loads are not normal binary operators: their result type is not
   /// determined by their operands, and they produce a value AND a token chain.
   ///
   /// This function will set the MOLoad flag on MMOFlags, but you can set it if
   /// you want.  The MOStore flag must not be set.

llvm/trunk/include/llvm/CodeGen/SelectionDAGAddressAnalysis.h

BaseIndexOffset(SDValue Base, SDValue Index, int64_t Offset,
       : Base(Base), Index(Index), Offset(Offset),
         IsIndexSignExt(IsIndexSignExt) {}
 
   SDValue getBase() { return Base; }
   SDValue getBase() const { return Base; }
   SDValue getIndex() { return Index; }
   SDValue getIndex() const { return Index; }
 
+  // Returns true if `Other` and `*this` are both some offset from the same base
+  // pointer. In that case, `Off` is set to the offset between `*this` and
+  // `Other` (negative if `Other` is before `*this`).
+  bool equalBaseIndex(const BaseIndexOffset &Other, const SelectionDAG &DAG,
+                      int64_t &Off) const;
+
   bool equalBaseIndex(const BaseIndexOffset &Other,
                       const SelectionDAG &DAG) const {
     int64_t Off;
     return equalBaseIndex(Other, DAG, Off);
   }
 
-  bool equalBaseIndex(const BaseIndexOffset &Other, const SelectionDAG &DAG,
-                      int64_t &Off) const;
+  // Returns true if `Other` (with size `OtherSize`) can be proven to be fully
+  // contained in `*this` (with size `Size`).
+  bool contains(int64_t Size, const BaseIndexOffset &Other, int64_t OtherSize,
+                const SelectionDAG &DAG) const;
+
+  // Returns true `BasePtr0` and `BasePtr1` can be proven to alias/not alias, in
+  // which case `IsAlias` is set to true/false.
+  static bool computeAliasing(const BaseIndexOffset &BasePtr0,
+                              const int64_t NumBytes0,
+                              const BaseIndexOffset &BasePtr1,
+                              const int64_t NumBytes1, const SelectionDAG &DAG,
+                              bool &IsAlias);
 
   /// Parses tree in Ptr for base, index, offset addresses.
   static BaseIndexOffset match(const LSBaseSDNode *N, const SelectionDAG &DAG);
 
   void print(raw_ostream& OS) const;
   void dump() const;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_SELECTIONDAGADDRESSANALYSIS_H

llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h

public:
   int getIndex() const { return FI; }
 
   static bool classof(const SDNode *N) {
     return N->getOpcode() == ISD::FrameIndex ||
            N->getOpcode() == ISD::TargetFrameIndex;
   }
 };
 
+/// This SDNode is used for LIFETIME_START/LIFETIME_END values, which indicate
+/// the offet and size that are started/ended in the underlying FrameIndex.
+class LifetimeSDNode : public SDNode {
+  int64_t Size;
+  int64_t Offset; // -1 if offset is unknown.
+public:
+  LifetimeSDNode(unsigned Opcode, unsigned Order, const DebugLoc &dl,
+                 SDVTList VTs, int64_t Size, int64_t Offset)
+      : SDNode(Opcode, Order, dl, VTs), Size(Size), Offset(Offset) {}
+
+  int64_t getFrameIndex() const {
+    return cast<FrameIndexSDNode>(getOperand(1))->getIndex();
+  }
+
+  bool hasOffset() const { return Offset >= 0; }
+  int64_t getOffset() const {
+    assert(hasOffset() && "offset is unknown");
+    return Offset;
+  }
+  int64_t getSize() const {
+    assert(hasOffset() && "offset is unknown");
+    return Size;
+  }
+
+  // Methods to support isa and dyn_cast
+  static bool classof(const SDNode *N) {
+    return N->getOpcode() == ISD::LIFETIME_START ||
+           N->getOpcode() == ISD::LIFETIME_END;
+  }
+};
+
 class JumpTableSDNode : public SDNode {
   friend class SelectionDAG;
 
   int JTI;
   unsigned char TargetFlags;
 
   JumpTableSDNode(int jti, EVT VT, bool isTarg, unsigned char TF)
     : SDNode(isTarg ? ISD::TargetJumpTable : ISD::JumpTable,

llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

private:
     SDValue visitBRCOND(SDNode *N);
     SDValue visitBR_CC(SDNode *N);
     SDValue visitLOAD(SDNode *N);
 
     SDValue replaceStoreChain(StoreSDNode *ST, SDValue BetterChain);
     SDValue replaceStoreOfFPConstant(StoreSDNode *ST);
 
     SDValue visitSTORE(SDNode *N);
+    SDValue visitLIFETIME_END(SDNode *N);
     SDValue visitINSERT_VECTOR_ELT(SDNode *N);
     SDValue visitEXTRACT_VECTOR_ELT(SDNode *N);
     SDValue visitBUILD_VECTOR(SDNode *N);
     SDValue visitCONCAT_VECTORS(SDNode *N);
     SDValue visitEXTRACT_SUBVECTOR(SDNode *N);
     SDValue visitVECTOR_SHUFFLE(SDNode *N);
     SDValue visitSCALAR_TO_VECTOR(SDNode *N);
     SDValue visitINSERT_SUBVECTOR(SDNode *N);
SDValue DAGCombiner::visit(SDNode *N) {
   case ISD::EXTRACT_SUBVECTOR:  return visitEXTRACT_SUBVECTOR(N);
   case ISD::VECTOR_SHUFFLE:     return visitVECTOR_SHUFFLE(N);
   case ISD::SCALAR_TO_VECTOR:   return visitSCALAR_TO_VECTOR(N);
   case ISD::INSERT_SUBVECTOR:   return visitINSERT_SUBVECTOR(N);
   case ISD::MGATHER:            return visitMGATHER(N);
   case ISD::MLOAD:              return visitMLOAD(N);
   case ISD::MSCATTER:           return visitMSCATTER(N);
   case ISD::MSTORE:             return visitMSTORE(N);
+  case ISD::LIFETIME_END:       return visitLIFETIME_END(N);
   case ISD::FP_TO_FP16:         return visitFP_TO_FP16(N);
   case ISD::FP16_TO_FP:         return visitFP16_TO_FP(N);
   }
   return SDValue();
 }
 
 SDValue DAGCombiner::combine(SDNode *N) {
   SDValue RV = visit(N);
SDValue DAGCombiner::visitSTORE(SDNode *N) {
   }
 
   if (SDValue NewSt = splitMergedValStore(ST))
     return NewSt;
 
   return ReduceLoadOpStoreWidth(N);
 }
 
+SDValue DAGCombiner::visitLIFETIME_END(SDNode *N) {
+  const auto *LifetimeEnd = cast<LifetimeSDNode>(N);
+  if (!LifetimeEnd->hasOffset())
+    return SDValue();
+
+  const BaseIndexOffset LifetimeEndBase(N->getOperand(1), SDValue(),
+                                        LifetimeEnd->getOffset(), false);
+
+  // We walk up the chains to find stores.
+  SmallVector<SDValue, 8> Chains = {N->getOperand(0)};
+  while (!Chains.empty()) {
+    SDValue Chain = Chains.back();
+    Chains.pop_back();
+    switch (Chain.getOpcode()) {
+    case ISD::TokenFactor:
+      for (unsigned Nops = Chain.getNumOperands(); Nops;)
+        Chains.push_back(Chain.getOperand(--Nops));
+      break;
+    case ISD::LIFETIME_START:
+    case ISD::LIFETIME_END: {
+      // We can forward past any lifetime start/end that can be proven not to
+      // alias the node.
+      const auto *LifetimeStart = cast<LifetimeSDNode>(Chain);
+      if (!LifetimeStart->hasOffset())
+        break; // Be conservative if we don't know the extents of the object.
+
+      const BaseIndexOffset LifetimeStartBase(
+          LifetimeStart->getOperand(1), SDValue(), LifetimeStart->getOffset(),
+          false);
+      bool IsAlias;
+      if (BaseIndexOffset::computeAliasing(
+              LifetimeEndBase, LifetimeEnd->getSize(), LifetimeStartBase,
+              LifetimeStart->getSize(), DAG, IsAlias) &&
+          !IsAlias) {
+        Chains.push_back(Chain.getOperand(0));
+      }
+      break;
+    }
+
+    case ISD::STORE: {
+      StoreSDNode *ST = dyn_cast<StoreSDNode>(Chain);
+      if (ST->isVolatile() || !ST->hasOneUse() || ST->isIndexed())
+        continue;
+      const BaseIndexOffset StoreBase = BaseIndexOffset::match(ST, DAG);
+      // If we store purely within object bounds just before its lifetime ends,
+      // we can remove the store.
+      if (LifetimeEndBase.contains(LifetimeEnd->getSize(), StoreBase,
+                                   ST->getMemoryVT().getStoreSize(), DAG)) {
+        LLVM_DEBUG(dbgs() << "\nRemoving store:"; StoreBase.dump();
+                   dbgs() << "\nwithin LIFETIME_END of : ";
+                   LifetimeEndBase.dump(); dbgs() << "\n");
+        CombineTo(ST, ST->getChain());
+        return SDValue(N, 0);
+      }
+    }
+    }
+  }
+  return SDValue();
+}
+
 /// For the instruction sequence of store below, F and I values
 /// are bundled together as an i64 value before being stored into memory.
 /// Sometimes it is more efficent to generate separate stores for F and I,
 /// which can remove the bitwise instructions or sink them to colder places.
 ///
 ///   (store (or (zext (bitcast F to i32) to i64),
 ///              (shl (zext I to i64), 32)), addr)  -->
 ///   (store F, addr) and (store I, addr+4)
bool DAGCombiner::isAlias(LSBaseSDNode *Op0, LSBaseSDNode *Op1) const {
 
   if (Op1->isInvariant() && Op0->writeMem())
     return false;
 
   unsigned NumBytes0 = Op0->getMemoryVT().getStoreSize();
   unsigned NumBytes1 = Op1->getMemoryVT().getStoreSize();
 
   // Check for BaseIndexOffset matching.
-  BaseIndexOffset BasePtr0 = BaseIndexOffset::match(Op0, DAG);
-  BaseIndexOffset BasePtr1 = BaseIndexOffset::match(Op1, DAG);
-  int64_t PtrDiff;
-  if (BasePtr0.getBase().getNode() && BasePtr1.getBase().getNode()) {
-    if (BasePtr0.equalBaseIndex(BasePtr1, DAG, PtrDiff))
-      return !((NumBytes0 <= PtrDiff) || (PtrDiff + NumBytes1 <= 0));
-
-    // If both BasePtr0 and BasePtr1 are FrameIndexes, we will not be
-    // able to calculate their relative offset if at least one arises
-    // from an alloca. However, these allocas cannot overlap and we
-    // can infer there is no alias.
-    if (auto *A = dyn_cast<FrameIndexSDNode>(BasePtr0.getBase()))
-      if (auto *B = dyn_cast<FrameIndexSDNode>(BasePtr1.getBase())) {
-        MachineFrameInfo &MFI = DAG.getMachineFunction().getFrameInfo();
-        // If the base are the same frame index but the we couldn't find a
-        // constant offset, (indices are different) be conservative.
-        if (A != B && (!MFI.isFixedObjectIndex(A->getIndex()) ||
-                       !MFI.isFixedObjectIndex(B->getIndex())))
-          return false;
-      }
-
-    bool IsFI0 = isa<FrameIndexSDNode>(BasePtr0.getBase());
-    bool IsFI1 = isa<FrameIndexSDNode>(BasePtr1.getBase());
-    bool IsGV0 = isa<GlobalAddressSDNode>(BasePtr0.getBase());
-    bool IsGV1 = isa<GlobalAddressSDNode>(BasePtr1.getBase());
-    bool IsCV0 = isa<ConstantPoolSDNode>(BasePtr0.getBase());
-    bool IsCV1 = isa<ConstantPoolSDNode>(BasePtr1.getBase());
-
-    // If of mismatched base types or checkable indices we can check
-    // they do not alias.
-    if ((BasePtr0.getIndex() == BasePtr1.getIndex() || (IsFI0 != IsFI1) ||
-         (IsGV0 != IsGV1) || (IsCV0 != IsCV1)) &&
-        (IsFI0 || IsGV0 || IsCV0) && (IsFI1 || IsGV1 || IsCV1))
-      return false;
-  }
+  bool IsAlias;
+  if (BaseIndexOffset::computeAliasing(
+          BaseIndexOffset::match(Op0, DAG), NumBytes0,
+          BaseIndexOffset::match(Op1, DAG), NumBytes1, DAG, IsAlias))
+    return IsAlias;
 
   // If we know required SrcValue1 and SrcValue2 have relatively large
   // alignment compared to the size and offset of the access, we may be able
   // to prove they do not alias. This check is conservative for now to catch
   // cases created by splitting vector types.
   int64_t SrcValOffset0 = Op0->getSrcValueOffset();
   int64_t SrcValOffset1 = Op1->getSrcValueOffset();
   unsigned OrigAlignment0 = Op0->getOriginalAlignment();

 /// looking for aliasing nodes and adding them to the Aliases vector.
 void DAGCombiner::GatherAllAliases(LSBaseSDNode *N, SDValue OriginalChain,
                                    SmallVectorImpl<SDValue> &Aliases) {
   SmallVector<SDValue, 8> Chains;     // List of chains to visit.
   SmallPtrSet<SDNode *, 16> Visited;  // Visited node set.
 
   // Get alias information for node.
   bool IsLoad = isa<LoadSDNode>(N) && !N->isVolatile();
+  const BaseIndexOffset LSBasePtr = BaseIndexOffset::match(N, DAG);
+  const unsigned LSNumBytes = N->getMemoryVT().getStoreSize();
 
   // Starting off.
   Chains.push_back(OriginalChain);
   unsigned Depth = 0;
 
   // Look at each chain and determine if it is an alias.  If so, add it to the
   // aliases list.  If not, then continue up the chain looking for the next
   // candidate.
case ISD::TokenFactor:
       break;
 
     case ISD::CopyFromReg:
       // Forward past CopyFromReg.
       Chains.push_back(Chain.getOperand(0));
       ++Depth;
       break;
 
+    case ISD::LIFETIME_START:
+    case ISD::LIFETIME_END: {
+      // We can forward past any lifetime start/end that can be proven not to
+      // alias the memory access.
+      const auto *Lifetime = cast<LifetimeSDNode>(Chain);
+      if (!Lifetime->hasOffset())
+        break; // Be conservative if we don't know the extents of the object.
+
+      const BaseIndexOffset LifetimePtr(Lifetime->getOperand(1), SDValue(),
+                                        Lifetime->getOffset(), false);
+      bool IsAlias;
+      if (BaseIndexOffset::computeAliasing(LifetimePtr, Lifetime->getSize(),
+                                           LSBasePtr, LSNumBytes, DAG,
+                                           IsAlias) &&
+          !IsAlias) {
+        Chains.push_back(Chain.getOperand(0));
+        ++Depth;
+      }
+      break;
+    }
+
     default:
       // For all other instructions we will just have to take what we can get.
       Aliases.push_back(Chain);
       break;
     }
   }
 }
 

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

CSEMap.InsertNode(N, IP);
     N = newSDNode<MemIntrinsicSDNode>(Opcode, dl.getIROrder(), dl.getDebugLoc(),
                                       VTList, MemVT, MMO);
     createOperands(N, Ops);
   }
   InsertNode(N);
   return SDValue(N, 0);
 }
 
+SDValue SelectionDAG::getLifetimeNode(bool IsStart, const SDLoc &dl,
+                                      SDValue Chain, int FrameIndex,
+                                      int64_t Size, int64_t Offset) {
+  const unsigned Opcode = IsStart ? ISD::LIFETIME_START : ISD::LIFETIME_END;
+  const auto VTs = getVTList(MVT::Other);
+  SDValue Ops[2] = {
+      Chain,
+      getFrameIndex(FrameIndex,
+                    getTargetLoweringInfo().getFrameIndexTy(getDataLayout()),
+                    true)};
+
+  FoldingSetNodeID ID;
+  AddNodeIDNode(ID, Opcode, VTs, Ops);
+  ID.AddInteger(FrameIndex);
+  ID.AddInteger(Size);
+  ID.AddInteger(Offset);
+  void *IP = nullptr;
+  if (SDNode *E = FindNodeOrInsertPos(ID, dl, IP))
+    return SDValue(E, 0);
+
+  LifetimeSDNode *N = newSDNode<LifetimeSDNode>(
+      Opcode, dl.getIROrder(), dl.getDebugLoc(), VTs, Size, Offset);
+  createOperands(N, Ops);
+  CSEMap.InsertNode(N, IP);
+  InsertNode(N);
+  SDValue V(N, 0);
+  NewSDValueDbgMsg(V, "Creating new node: ", this);
+  return V;
+}
+
 /// InferPointerInfo - If the specified ptr/offset is a frame index, infer a
 /// MachinePointerInfo record from it.  This is particularly useful because the
 /// code generator has many cases where it doesn't bother passing in a
 /// MachinePointerInfo to getLoad or getStore when it has "FI+Cst".
 static MachinePointerInfo InferPointerInfo(const MachinePointerInfo &Info,
                                            SelectionDAG &DAG, SDValue Ptr,
                                            int64_t Offset = 0) {
   // If this is FI+Offset, we can model it.

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGAddressAnalysis.cpp

if (auto *A = dyn_cast<FrameIndexSDNode>(Base))
                  MFI.getObjectOffset(A->getIndex());
           return true;
         }
       }
   }
   return false;
 }
 
+bool BaseIndexOffset::computeAliasing(const BaseIndexOffset &BasePtr0,
+                                      const int64_t NumBytes0,
+                                      const BaseIndexOffset &BasePtr1,
+                                      const int64_t NumBytes1,
+                                      const SelectionDAG &DAG, bool &IsAlias) {
+  if (!(BasePtr0.getBase().getNode() && BasePtr1.getBase().getNode()))
+    return false;
+  int64_t PtrDiff;
+  if (BasePtr0.equalBaseIndex(BasePtr1, DAG, PtrDiff)) {
+    // BasePtr1 is PtrDiff away from BasePtr0. They alias if none of the
+    // following situations arise:
+    IsAlias = !(
+        // [----BasePtr0----]
+        //                         [---BasePtr1--]
+        // ========PtrDiff========>
+        (NumBytes0 <= PtrDiff) ||
+        //                     [----BasePtr0----]
+        // [---BasePtr1--]
+        // =====(-PtrDiff)====>
+        (PtrDiff + NumBytes1 <= 0)); // i.e. NumBytes1 < -PtrDiff.
+    return true;
+  }
+  // If both BasePtr0 and BasePtr1 are FrameIndexes, we will not be
+  // able to calculate their relative offset if at least one arises
+  // from an alloca. However, these allocas cannot overlap and we
+  // can infer there is no alias.
+  if (auto *A = dyn_cast<FrameIndexSDNode>(BasePtr0.getBase()))
+    if (auto *B = dyn_cast<FrameIndexSDNode>(BasePtr1.getBase())) {
+      MachineFrameInfo &MFI = DAG.getMachineFunction().getFrameInfo();
+      // If the base are the same frame index but the we couldn't find a
+      // constant offset, (indices are different) be conservative.
+      if (A != B && (!MFI.isFixedObjectIndex(A->getIndex()) ||
+                     !MFI.isFixedObjectIndex(B->getIndex()))) {
+        IsAlias = false;
+        return true;
+      }
+    }
+
+  bool IsFI0 = isa<FrameIndexSDNode>(BasePtr0.getBase());
+  bool IsFI1 = isa<FrameIndexSDNode>(BasePtr1.getBase());
+  bool IsGV0 = isa<GlobalAddressSDNode>(BasePtr0.getBase());
+  bool IsGV1 = isa<GlobalAddressSDNode>(BasePtr1.getBase());
+  bool IsCV0 = isa<ConstantPoolSDNode>(BasePtr0.getBase());
+  bool IsCV1 = isa<ConstantPoolSDNode>(BasePtr1.getBase());
+
+  // If of mismatched base types or checkable indices we can check
+  // they do not alias.
+  if ((BasePtr0.getIndex() == BasePtr1.getIndex() || (IsFI0 != IsFI1) ||
+       (IsGV0 != IsGV1) || (IsCV0 != IsCV1)) &&
+      (IsFI0 || IsGV0 || IsCV0) && (IsFI1 || IsGV1 || IsCV1)) {
+    IsAlias = false;
+    return true;
+  }
+  return false; // Cannot determine whether the pointers alias.
+}
+
+bool BaseIndexOffset::contains(int64_t Size, const BaseIndexOffset &Other,
+                               int64_t OtherSize,
+                               const SelectionDAG &DAG) const {
+  int64_t Offset;
+  if (!equalBaseIndex(Other, DAG, Offset))
+    return false;
+  if (Offset >= 0) {
+    // Other is after *this:
+    // [-------*this---------]
+    //            [---Other--]
+    // ==Offset==>
+    return Offset + OtherSize <= Size;
+  }
+  // Other starts strictly before *this, it cannot be fully contained.
+  //    [-------*this---------]
+  // [--Other--]
+  return false;
+}
+
 /// Parses tree in Ptr for base, index, offset addresses.
 BaseIndexOffset BaseIndexOffset::match(const LSBaseSDNode *N,
                                        const SelectionDAG &DAG) {
   SDValue Ptr = N->getBasePtr();
 
   // (((B + I*M) + c)) + c ...
   SDValue Base = DAG.getTargetLoweringInfo().unwrapAddress(Ptr);
   SDValue Index = SDValue();

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I, unsigned Intrinsic) {
   }
   case Intrinsic::lifetime_start:
   case Intrinsic::lifetime_end: {
     bool IsStart = (Intrinsic == Intrinsic::lifetime_start);
     // Stack coloring is not enabled in O0, discard region information.
     if (TM.getOptLevel() == CodeGenOpt::None)
       return nullptr;
 
+    const int64_t ObjectSize =
+        cast<ConstantInt>(I.getArgOperand(0))->getSExtValue();
+    Value *const ObjectPtr = I.getArgOperand(1);
     SmallVector<Value *, 4> Allocas;
-    GetUnderlyingObjects(I.getArgOperand(1), Allocas, *DL);
+    GetUnderlyingObjects(ObjectPtr, Allocas, *DL);
 
     for (SmallVectorImpl<Value*>::iterator Object = Allocas.begin(),
            E = Allocas.end(); Object != E; ++Object) {
       AllocaInst *LifetimeObject = dyn_cast_or_null<AllocaInst>(*Object);
 
       // Could not find an Alloca.
       if (!LifetimeObject)
         continue;
 
       // First check that the Alloca is static, otherwise it won't have a
       // valid frame index.
       auto SI = FuncInfo.StaticAllocaMap.find(LifetimeObject);
       if (SI == FuncInfo.StaticAllocaMap.end())
         return nullptr;
 
-      int FI = SI->second;
-
-      SDValue Ops[2];
-      Ops[0] = getRoot();
-      Ops[1] =
-          DAG.getFrameIndex(FI, TLI.getFrameIndexTy(DAG.getDataLayout()), true);
-      unsigned Opcode = (IsStart ? ISD::LIFETIME_START : ISD::LIFETIME_END);
-
-      Res = DAG.getNode(Opcode, sdl, MVT::Other, Ops);
+      const int FrameIndex = SI->second;
+      int64_t Offset;
+      if (GetPointerBaseWithConstantOffset(
+              ObjectPtr, Offset, DAG.getDataLayout()) != LifetimeObject)
+        Offset = -1; // Cannot determine offset from alloca to lifetime object.
+      Res = DAG.getLifetimeNode(IsStart, sdl, getRoot(), FrameIndex, ObjectSize,
+                                Offset);
       DAG.setRoot(Res);
     }
     return nullptr;
   }
   case Intrinsic::invariant_start:
     // Discard region information.
     setValue(&I, DAG.getUNDEF(TLI.getPointerTy(DAG.getDataLayout())));
     return nullptr;

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp

if (unsigned int TF = BA->getTargetFlags())
       OS << " [TF=" << TF << ']';
   } else if (const AddrSpaceCastSDNode *ASC =
                dyn_cast<AddrSpaceCastSDNode>(this)) {
     OS << '['
        << ASC->getSrcAddressSpace()
        << " -> "
        << ASC->getDestAddressSpace()
        << ']';
+  } else if (const LifetimeSDNode *LN = dyn_cast<LifetimeSDNode>(this)) {
+    if (LN->hasOffset())
+      OS << "<" << LN->getOffset() << " to " << LN->getOffset() + LN->getSize() << ">";
   }
 
   if (VerboseDAGDumping) {
     if (unsigned Order = getIROrder())
         OS << " [ORD=" << Order << ']';
 
     if (getNodeId() != -1)
       OS << " [ID=" << getNodeId() << ']';

llvm/trunk/test/CodeGen/BPF/remove_truncate_5.ll

 ; Function Attrs: nounwind
 define dso_local void @test() local_unnamed_addr #0 {
 ; CHECK-LABEL: test:
   %1 = alloca %struct.test_t, align 4
   %2 = bitcast %struct.test_t* %1 to i8*
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %2) #3
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 4 %2, i8* align 4 bitcast (%struct.test_t* @test.t to i8*), i64 16, i1 false)
 ; CHECK: r1 = 0
-; CHECK: r1 <<= 32
-; CHECK: r2 = r1
-; CHECK: r2 |= 0
-; CHECK: *(u64 *)(r10 - 8) = r2
-; CHECK: r1 |= 5
-; CHECK: *(u64 *)(r10 - 16) = r1
+; CHECK-NEXT: r1 <<= 32
+; CHECK-NEXT: r2 = r1
+; CHECK: [[SECOND:r[12]]] |= 5
+; CHECK-NEXT: *(u64 *)(r10 - 16) = [[SECOND]]
+; CHECK: [[FIRST:r[12]]] |= 0
+; CHECK-NEXT: *(u64 *)(r10 - 8) = [[FIRST]]
   call void @foo(i8* nonnull %2) #3
 ; CHECK: call foo
   call void @llvm.lifetime.end.p0i8(i64 16, i8* nonnull %2) #3
   ret void
 }
 
 ; Function Attrs: argmemonly nounwind
 declare void @llvm.lifetime.start.p0i8(i64, i8* nocapture) #1

llvm/trunk/test/CodeGen/X86/swap.ll

 ; NOAA-NEXT:    vmovups %xmm0, (%rdi)
 ; NOAA-NEXT:    vmovaps -{{[0-9]+}}(%rsp), %xmm0
 ; NOAA-NEXT:    vmovups %xmm0, (%rsi)
 ; NOAA-NEXT:    retq
 ;
 ; AA-LABEL: _Z4SwapP1SS0_:
 ; AA:       # %bb.0: # %entry
 ; AA-NEXT:    vmovups (%rdi), %xmm0
-; AA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
 ; AA-NEXT:    vmovups (%rsi), %xmm1
 ; AA-NEXT:    vmovups %xmm1, (%rdi)
 ; AA-NEXT:    vmovups %xmm0, (%rsi)
 ; AA-NEXT:    retq
 entry:
   %tmp.sroa.0 = alloca [16 x i8], align 1
   %tmp.sroa.0.0..sroa_idx6 = getelementptr inbounds [16 x i8], [16 x i8]* %tmp.sroa.0, i64 0, i64 0
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %tmp.sroa.0.0..sroa_idx6)
   %tmp.sroa.0.0..sroa_idx1 = getelementptr inbounds %struct.S, %struct.S* %a, i64 0, i32 0, i64 0
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 %tmp.sroa.0.0..sroa_idx6, i8* align 1 %tmp.sroa.0.0..sroa_idx1, i64 16, i1 false)
   %0 = getelementptr inbounds %struct.S, %struct.S* %b, i64 0, i32 0, i64 0
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %tmp.sroa.0.0..sroa_idx1, i8* align 1 %0, i64 16, i1 false), !tbaa.struct !2
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %0, i8* nonnull align 1 %tmp.sroa.0.0..sroa_idx6, i64 16, i1 false)
   call void @llvm.lifetime.end.p0i8(i64 16, i8* nonnull %tmp.sroa.0.0..sroa_idx6)
   ret void
 }
 
 define dso_local void @onealloc_noreadback(i8* nocapture %a, i8* nocapture %b) local_unnamed_addr {
 ; NOAA-LABEL: onealloc_noreadback:
 ; NOAA:       # %bb.0: # %entry
-; NOAA-NEXT:    vmovups (%rdi), %xmm0
-; NOAA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; NOAA-NEXT:    vmovups (%rsi), %xmm0
-; NOAA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
 ; NOAA-NEXT:    retq
 ;
 ; AA-LABEL: onealloc_noreadback:
 ; AA:       # %bb.0: # %entry
-; AA-NEXT:    vmovups (%rdi), %xmm0
-; AA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; AA-NEXT:    vmovups (%rsi), %xmm0
-; AA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
 ; AA-NEXT:    retq
 entry:
   %alloc = alloca [16 x i8], i8 2, align 1
   %part1 = getelementptr inbounds [16 x i8], [16 x i8]* %alloc, i64 0, i64 0
   %part2 = getelementptr inbounds [16 x i8], [16 x i8]* %alloc, i64 1, i64 0
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %part1)
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %part2)
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 %part1, i8* align 1 %a, i64 16, i1 false)
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 %part2, i8* align 1 %b, i64 16, i1 false)
   call void @llvm.lifetime.end.p0i8(i64 16, i8* nonnull %part1)
   call void @llvm.lifetime.end.p0i8(i64 16, i8* nonnull %part2)
   ret void
 }
 
 
 define dso_local void @twoallocs_trivial(i8* nocapture %a, i8* nocapture %b) local_unnamed_addr {
 ; NOAA-LABEL: twoallocs_trivial:
 ; NOAA:       # %bb.0: # %entry
-; NOAA-NEXT:    vmovups (%rdi), %xmm0
-; NOAA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; NOAA-NEXT:    vmovups (%rsi), %xmm0
-; NOAA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
 ; NOAA-NEXT:    retq
 ;
 ; AA-LABEL: twoallocs_trivial:
 ; AA:       # %bb.0: # %entry
-; AA-NEXT:    vmovups (%rdi), %xmm0
-; AA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; AA-NEXT:    vmovups (%rsi), %xmm0
-; AA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
 ; AA-NEXT:    retq
 entry:
   %alloc1 = alloca [16 x i8], align 1
   %alloc2 = alloca [16 x i8], align 1
   %part1 = getelementptr inbounds [16 x i8], [16 x i8]* %alloc1, i64 0, i64 0
   %part2 = getelementptr inbounds [16 x i8], [16 x i8]* %alloc2, i64 0, i64 0
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %part1)
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %part2)
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 %part1, i8* align 1 %a, i64 16, i1 false)
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 %part2, i8* align 1 %b, i64 16, i1 false)
   call void @llvm.lifetime.end.p0i8(i64 16, i8* nonnull %part1)
   call void @llvm.lifetime.end.p0i8(i64 16, i8* nonnull %part2)
   ret void
 }
 
 define dso_local void @twoallocs(i8* nocapture %a, i8* nocapture %b) local_unnamed_addr {
 ; NOAA-LABEL: twoallocs:
 ; NOAA:       # %bb.0: # %entry
 ; NOAA-NEXT:    vmovups (%rdi), %xmm0
-; NOAA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; NOAA-NEXT:    vmovaps -{{[0-9]+}}(%rsp), %xmm0
 ; NOAA-NEXT:    vmovups %xmm0, (%rsi)
 ; NOAA-NEXT:    retq
 ;
 ; AA-LABEL: twoallocs:
 ; AA:       # %bb.0: # %entry
 ; AA-NEXT:    vmovups (%rdi), %xmm0
-; AA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; AA-NEXT:    vmovaps -{{[0-9]+}}(%rsp), %xmm0
 ; AA-NEXT:    vmovups %xmm0, (%rsi)
 ; AA-NEXT:    retq
 entry:
   %alloc1 = alloca [16 x i8], align 1
   %alloc2 = alloca [16 x i8], align 1
   %part1 = getelementptr inbounds [16 x i8], [16 x i8]* %alloc1, i64 0, i64 0
   %part2 = getelementptr inbounds [16 x i8], [16 x i8]* %alloc2, i64 0, i64 0
   %part2_alias = getelementptr inbounds [16 x i8], [16 x i8]* %alloc2, i64 0, i64 0
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %part1)
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %part2)
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 %part2, i8* align 1 %a, i64 16, i1 false)
   call void @llvm.lifetime.end.p0i8(i64 16, i8* nonnull %part1)
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 %b, i8* align 1 %part2_alias, i64 16, i1 false)
   call void @llvm.lifetime.end.p0i8(i64 16, i8* nonnull %part2)
   ret void
 }
 
 define dso_local void @onealloc_readback_1(i8* nocapture %a, i8* nocapture %b) local_unnamed_addr {
 ; NOAA-LABEL: onealloc_readback_1:
 ; NOAA:       # %bb.0: # %entry
 ; NOAA-NEXT:    vmovups (%rdi), %xmm0
 ; NOAA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
 ; NOAA-NEXT:    vmovups (%rsi), %xmm0
-; NOAA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; NOAA-NEXT:    vmovaps -{{[0-9]+}}(%rsp), %xmm0
 ; NOAA-NEXT:    vmovups %xmm0, (%rdi)
 ; NOAA-NEXT:    retq
 ;
 ; AA-LABEL: onealloc_readback_1:
 ; AA:       # %bb.0: # %entry
-; AA-NEXT:    vmovups (%rdi), %xmm0
-; AA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
 ; AA-NEXT:    vmovups (%rsi), %xmm0
-; AA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; AA-NEXT:    vmovaps -{{[0-9]+}}(%rsp), %xmm0
 ; AA-NEXT:    vmovups %xmm0, (%rdi)
 ; AA-NEXT:    retq
 entry:
   %alloc = alloca [16 x i8], i8 2, align 1
   %part2 = getelementptr inbounds [16 x i8], [16 x i8]* %alloc, i64 0, i64 0
   %part1 = getelementptr inbounds [16 x i8], [16 x i8]* %alloc, i64 1, i64 0
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %part1)
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %part2)
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 %part1, i8* align 1 %a, i64 16, i1 false)
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 %part2, i8* align 1 %b, i64 16, i1 false)
   call void @llvm.lifetime.end.p0i8(i64 16, i8* nonnull %part1)
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 %a, i8* align 1 %part2, i64 16, i1 false)
   call void @llvm.lifetime.end.p0i8(i64 16, i8* nonnull %part2)
   ret void
 }
 
 define dso_local void @onealloc_readback_2(i8* nocapture %a, i8* nocapture %b) local_unnamed_addr {
 ; NOAA-LABEL: onealloc_readback_2:
 ; NOAA:       # %bb.0: # %entry
 ; NOAA-NEXT:    vmovups (%rdi), %xmm0
 ; NOAA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
 ; NOAA-NEXT:    vmovups (%rsi), %xmm0
-; NOAA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; NOAA-NEXT:    vmovaps -{{[0-9]+}}(%rsp), %xmm0
 ; NOAA-NEXT:    vmovups %xmm0, (%rdi)
 ; NOAA-NEXT:    retq
 ;
 ; AA-LABEL: onealloc_readback_2:
 ; AA:       # %bb.0: # %entry
-; AA-NEXT:    vmovups (%rdi), %xmm0
-; AA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
 ; AA-NEXT:    vmovups (%rsi), %xmm0
-; AA-NEXT:    vmovaps %xmm0, -{{[0-9]+}}(%rsp)
-; AA-NEXT:    vmovaps -{{[0-9]+}}(%rsp), %xmm0
 ; AA-NEXT:    vmovups %xmm0, (%rdi)
 ; AA-NEXT:    retq
 entry:
   %alloc = alloca [16 x i8], i8 2, align 1
   %part1 = getelementptr inbounds [16 x i8], [16 x i8]* %alloc, i64 0, i64 0
   %part2 = getelementptr inbounds [16 x i8], [16 x i8]* %alloc, i64 1, i64 0
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %part1)
   call void @llvm.lifetime.start.p0i8(i64 16, i8* nonnull %part2)

llvm/trunk/test/DebugInfo/COFF/inlining.ll

 ; ASM: .cv_func_id 0
 ; ASM: .cv_loc 0 1 13 0      # t.cpp:13:0
 ; ASM: .cv_loc 0 1 14 5                # t.cpp:14:5
 ; ASM: addl    $6, "?x@@3HC"
 ; ASM: .cv_inline_site_id 1 within 0 inlined_at 1 15 3
 ; ASM: .cv_loc 1 1 9 5                 # t.cpp:9:5
 ; ASM: addl    $4, "?x@@3HC"
 ; ASM: .cv_inline_site_id 2 within 1 inlined_at 1 10 3
-; ASM: .cv_loc 2 1 3 7                 # t.cpp:3:7
 ; ASM: .cv_loc 2 1 4 5                 # t.cpp:4:5
 ; ASM: addl    {{.*}}, "?x@@3HC"
 ; ASM: .cv_loc 2 1 5 5                 # t.cpp:5:5
 ; ASM: addl    $2, "?x@@3HC"
 ; ASM: .cv_loc 2 1 6 5                 # t.cpp:6:5
 ; ASM: addl    $3, "?x@@3HC"
 ; ASM: .cv_loc 1 1 11 5                # t.cpp:11:5
 ; ASM: addl    $5, "?x@@3HC"

 ; OBJ:   }
 ; OBJ: ]
 ; OBJ: Subsection [
 ; OBJ:   SubSectionType: Symbols (0xF1)
 ; OBJ:   {{.*}}Proc{{.*}}Sym {
 ; OBJ:     PtrParent: 0x0
 ; OBJ:     PtrEnd: 0x0
 ; OBJ:     PtrNext: 0x0
-; OBJ:     CodeSize: 0x3C
+; OBJ:     CodeSize: 0x35
 ; OBJ:     DbgStart: 0x0
 ; OBJ:     DbgEnd: 0x0
 ; OBJ:     FunctionType: baz (0x1004)
 ; OBJ:     CodeOffset: ?baz@@YAXXZ+0x0
 ; OBJ:     Segment: 0x0
 ; OBJ:     Flags [ (0x0)
 ; OBJ:     ]
 ; OBJ:     DisplayName: baz
 ; OBJ:     LinkageName: ?baz@@YAXXZ
 ; OBJ:   }
 ; OBJ:   InlineSiteSym {
 ; OBJ:     PtrParent: 0x0
 ; OBJ:     PtrEnd: 0x0
 ; OBJ:     Inlinee: bar (0x1002)
 ; OBJ:      BinaryAnnotations [
 ; OBJ-NEXT:   ChangeCodeOffsetAndLineOffset: {CodeOffset: 0x8, LineOffset: 1}
-; OBJ-NEXT:   ChangeCodeOffsetAndLineOffset: {CodeOffset: 0x7, LineOffset: 1}
+; OBJ-NEXT:   ChangeCodeOffsetAndLineOffset: {CodeOffset: 0x9, LineOffset: 1}
 ; OBJ-NEXT:   ChangeLineOffset: 1
-; OBJ-NEXT:   ChangeCodeOffset: 0x1D
+; OBJ-NEXT:   ChangeCodeOffset: 0x14
 ; OBJ-NEXT:   ChangeCodeLength: 0x7
 ; OBJ:      ]
 ; OBJ:   }
 ; OBJ:   InlineSiteSym {
 ; OBJ:     PtrParent: 0x0
 ; OBJ:     PtrEnd: 0x0
 ; OBJ:     Inlinee: foo (0x1003)
 ; OBJ:     BinaryAnnotations [
-; OBJ-NEXT:  ChangeCodeOffsetAndLineOffset: {CodeOffset: 0xF, LineOffset: 1}
-; OBJ-NEXT:  ChangeCodeOffsetAndLineOffset: {CodeOffset: 0x9, LineOffset: 1}
+; OBJ-NEXT:  ChangeLineOffset: 2
+; OBJ-NEXT:  ChangeCodeOffset: 0x11
 ; OBJ-NEXT:  ChangeCodeOffsetAndLineOffset: {CodeOffset: 0x6, LineOffset: 1}
 ; OBJ-NEXT:  ChangeCodeOffsetAndLineOffset: {CodeOffset: 0x7, LineOffset: 1}
 ; OBJ-NEXT:  ChangeCodeLength: 0x7
 ; OBJ:     ]
 ; OBJ:   }
 ; OBJ:   InlineSiteEnd {
 ; OBJ:   }
 ; OBJ:   InlineSiteEnd {

llvm/trunk/test/DebugInfo/COFF/lexicalblock.ll

 ; CHECK: }
 ; CHECK: LocalSym {
 ; CHECK:   VarName: localB
 ; CHECK: }
 ; CHECK: BlockSym {
 ; CHECK:   Kind: S_BLOCK32 {{.*}}
 ; CHECK:   BlockName: 
 ; CHECK: }
-; CHECK: LocalSym {
-; CHECK:   VarName: localC
-; CHECK: }
 ; CHECK: ScopeEndSym {
 ; CHECK:   Kind: S_END {{.*}}
 ; CHECK: }
 ; CHECK: BlockSym {
 ; CHECK:   Kind: S_BLOCK32 {{.*}}
 ; CHECK:   BlockName: 
 ; CHECK: }
-; CHECK: LocalSym {
-; CHECK:   VarName: localD
-; CHECK: }
 ; CHECK: ScopeEndSym {
 ; CHECK:   Kind: S_END {{.*}}
 ; CHECK: }
 ; CHECK: BlockSym {
 ; CHECK:   Kind: S_BLOCK32 {{.*}}
 ; CHECK:   BlockName: 
 ; CHECK: }
-; CHECK: LocalSym {
-; CHECK:   VarName: localE
-; CHECK: }
-; CHECK: ScopeEndSym {
-; CHECK: }
-; CHECK: BlockSym {
-; CHECK:   Kind: S_BLOCK32 {{.*}}
-; CHECK:   BlockName: 
-; CHECK: }
-; CHECK: LocalSym {
-; CHECK:   VarName: localF
-; CHECK: }
-; CHECK: BlockSym {
-; CHECK:   Kind: S_BLOCK32 {{.*}}
-; CHECK:   BlockName: 
-; CHECK: }
-; CHECK: LocalSym {
-; CHECK:   VarName: localG
-; CHECK: }
 ; CHECK: ScopeEndSym {
-; CHECK:   Kind: S_END {{.*}}
-; CHECK: }
-; CHECK: ScopeEndSym {
-; CHECK:   Kind: S_END {{.*}}
 ; CHECK: }
 ; CHECK: BlockSym {
 ; CHECK:   Kind: S_BLOCK32 {{.*}}
 ; CHECK:   BlockName: 
 ; CHECK: }
-; CHECK: LocalSym {
-; CHECK:   VarName: localH
-; CHECK: }
 ; CHECK: ScopeEndSym {
 ; CHECK:   Kind: S_END {{.*}}
 ; CHECK: }
 ; CHECK: ProcEnd {
 ; CHECK: }
 ;
 ; ModuleID = 'lexicalblock.cxx'
 source_filename = "lexicalblock.cxx"