This patch introduces MemorySSA, a virtual SSA form for memory. Details on what it looks like are in MemorySSA.h

[george.burgess.iv]

Summary

There is no grand scheme to use this across LLVM (at least, that i have).
The main purposees for it are to clean and speed up things like MemoryDependenceAnalysis
(which spends a lot of time walking to find use/defs of memory, etc), and to
be able to provide load value numbering to GVN (and PRE).
Secondarily, it may be possible to speed up some of the other load/store
motion passes using it.

Note that creation of MemorySSA is linear time and space relative to the IR
size. It uses the same linear time phi placement algorithm PromoteMemoryToRegisters does
(though sadly, not the same code because abstracting PromoteMemoryToRegisters
is not easy since we don't derive the memory access classes from things
like Value, etc)

It's lived both as a utility that passes could create and delete, and as a pass.
In the current patch, it's a functionpass that things can require
as an analysis.
Happy to put it back to being a utility, not sure which makes more sense.
I only have GVN using it right now.

As a small note: The immediate-use lists are ugly right now. Suggestions for how to do
that better welcome.

Anyway, happy to answer any questions, and not looking to commit this tomorrow,
more it got to the point where i think it would be good to get some other eyes
on it.

[reames]

include/llvm/Transforms/Utils/MemorySSA.h

12 ↗

(On Diff #20684)

This comment will need to be expanded at some point. That can wait until the code is closer to ready though.

Correction, the comment I wanted was below. Might make sense to move some of it into the file comment or provide a forward reference.

30 ↗

(On Diff #20684)

I would phrase this as: links together memory access instructions such as loads, stores, ...

91 ↗

(On Diff #20684)

I might avoid the term "Type" here since that has a well defined meaning in LLVM. Using AccessType consistently would be clearer.

107 ↗

(On Diff #20684)

A bounds assertion here would be good.

129 ↗

(On Diff #20684)

The term "UseOperand" isn't clear to me here. Is this the link back to a def or phi?

155 ↗

(On Diff #20684)

I'm not sure we need an explicit def version here. Doesn't the address of the underlying instruction serve that purpose as well? (And when that instruction disappears, we need to renumber anyways.)

[• dberlin]

Thanks so much for taking a look at what i've got so far!

[jmolloy]

Hi Daniel,

One inline comment about the design. Sorry for the naiveity.

Cheers,

James

include/llvm/Transforms/Utils/MemorySSA.h

150 ↗

(On Diff #20684)

I don't understand this: Firstly, why must all defs have a use? We can store without loading the resulting value back.

Secondly, you've used inheritance which is an "is-a" relationship. Is there something fundamental I'm missing here?

[• dberlin]

Also, as a natural followup, i'll pointed out that GCC experimented with
must/may defs (IE we have MemoryMustDef to represent strong updates, and
MemoryMayDef to everything else), as well as a separate MemoryKill (for
inline assembly we didn't understand or instructions that were "clobbers of
all memory").

It also allowed for multiple "heap versions" to be live at once (To try to
reduce the length of the chains).
This required multiple memorydefs/uses on a single instruction, as well as
multiple phis

In practice, nothing used any of these distinctions to any good effect.
In the end, they all just wanted to walk the chains, and having multiple
heap versions means walking more chains (even if some are shorter), so in
the end it did not end up making the compiler faster, but it did make it
more complicated :)

After roughly 10-12 years (i forget how long it's been) of playing with
different options, GCC is also now back to just MemoryDef and MemoryUse,
and a single heap version.

[• dberlin]

Update for commments received so far.

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

include/llvm/Transforms/Utils/MemorySSA.h

12 ↗

(On Diff #20684)

fixed

30 ↗

(On Diff #20684)

fixed

91 ↗

(On Diff #20684)

Fixed

155 ↗

(On Diff #20684)

fixed in version i'm about to upload

This does make debugging a bit harder (IMHO), but such is life :)

[• dberlin]

Reserve space for args

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

Initialize Use lists
Fix bug in getClobberingMemoryAccess

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[hfinkel]

lib/Transforms/Utils/MemorySSA.cpp

86 ↗

(On Diff #20784)

What exactly does "updated to handle call instructions properly" mean? Also, this looks like it should be a member function of AA. Can you please add it there (then we can update MDA to use it too?).

100 ↗

(On Diff #20784)

Part of my confusion is this: What does the call's return value have to do with the memory location(s) accessed by the call?

We do have these in AA:

static Location getLocationForSource(const MemTransferInst *MTI);
static Location getLocationForDest(const MemIntrinsic *MI);

and I imagine those would be good to use here.

But for generic calls, we don't have any facility to get a location, or list of locations, touched by the call. We can do call-site vs. call-site queries, and call-site vs. Location queries, however.

136 ↗

(On Diff #20784)

Can you please define here what a "false" version is? (non-aliasing?)

241 ↗

(On Diff #20784)

I'm likely missing something here, but why don't you just call AA->getModRefInfo for all types of instructions? The overload for Instruction* handles all of the various types:

/// getModRefInfo - Return information about whether or not an instruction may
/// read or write the specified memory location.  An instruction
/// that doesn't read or write memory may be trivially LICM'd for example.
ModRefResult getModRefInfo(const Instruction *I,
                           const Location &Loc) {
  switch (I->getOpcode()) {
  case Instruction::VAArg:  return getModRefInfo((const VAArgInst*)I, Loc);
  case Instruction::Load:   return getModRefInfo((const LoadInst*)I,  Loc);
  case Instruction::Store:  return getModRefInfo((const StoreInst*)I, Loc);
  case Instruction::Fence:  return getModRefInfo((const FenceInst*)I, Loc);
  case Instruction::AtomicCmpXchg:
    return getModRefInfo((const AtomicCmpXchgInst*)I, Loc);
  case Instruction::AtomicRMW:
    return getModRefInfo((const AtomicRMWInst*)I, Loc);
  case Instruction::Call:   return getModRefInfo((const CallInst*)I,  Loc);
  case Instruction::Invoke: return getModRefInfo((const InvokeInst*)I,Loc);
  default:                  return NoModRef;
  }
}

269 ↗

(On Diff #20784)

Why are you printing the instruction addresses here? Would it be more useful to print *I?

357 ↗

(On Diff #20784)

Range-based for?

411 ↗

(On Diff #20784)

Range-based for using succ_range?

435 ↗

(On Diff #20784)

Range-based for here?

490 ↗

(On Diff #20784)

Range-based for?

539 ↗

(On Diff #20784)

Range-based for?

555 ↗

(On Diff #20784)

Maybe add here?

assert(!DT->isReachableFromEntry(BB) &&
            "Reachable block found while handling unreachable blocks");

613 ↗

(On Diff #20784)

Range-based for?

635 ↗

(On Diff #20784)

Range-based for?

637 ↗

(On Diff #20784)

Range-based for?

651 ↗

(On Diff #20784)

You're testing against an empty location? I don't think this is what you want, how about using this:

/// getModRefBehavior - Return the behavior when calling the given call site.
virtual ModRefBehavior getModRefBehavior(ImmutableCallSite CS);

687 ↗

(On Diff #20784)

in to -> into

688 ↗

(On Diff #20784)

But you don't really insert anything into the IR proper. I'd rephrase this.

717 ↗

(On Diff #20784)

No need for { } here.

729 ↗

(On Diff #20784)

Range-based for?

737 ↗

(On Diff #20784)

Range-based for? And you don't need the { } here.

770 ↗

(On Diff #20784)

Range-based for?

780 ↗

(On Diff #20784)

Range-based for?

[• dberlin]

Comment at: lib/Transforms/Utils/MemorySSA.cpp:357
@@ +356,3 @@
+ unsigned ID = 0;
+ for (auto I = F.begin(), E = F.end(); I != E; ++I)

+ BBNumbers[I] = ID++;

Range-based for?

BTW, at least the obvious construction of that for( auto I : F) gives:
../lib/Transforms/Utils/MemorySSA.cpp:350:13: error: call to deleted
constructor of 'llvm::BasicBlock'

for (auto I : F)
          ^ ~

../include/llvm/IR/BasicBlock.h:85:3: note: 'BasicBlock' has been
explicitly marked deleted here

BasicBlock(const BasicBlock &) = delete;
^

Is there some way to iterate over basic block instructions using range
based for that we've added?

[• dberlin]

Update for review comments (though not quite ready for re-review, need to document a few changes better)

Also handle call caching and clobbering better

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[majnemer]

include/llvm/Analysis/AliasAnalysis.h

155–156 ↗

(On Diff #20966)

No need for else here.

include/llvm/Transforms/Utils/MemorySSA.h

1–2 ↗

(On Diff #20966)

This looks strange, can you make it look like the other files?

lib/Transforms/Utils/MemorySSA.cpp

1 ↗

(On Diff #20966)

No need for this whitespace.

[• dberlin]

Move state into a structure, clean up a bit, and use structure to stop abusing Loc.Ptr as storage

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[hfinkel]

lib/Transforms/Utils/MemorySSA.cpp

237 ↗

(On Diff #21154)

I had also been thinking that the logic of this else block could be moved into a utility function of AliasAnalysis. It seems to be that this is exactly what you'd want

AA->getModRefInfo(Inst1, Inst2)

to mean. Do you agree?

[• dberlin]

Add a dominance verifier, make a new alias analysis API and use it, fix a small bug in incomplete phi handling

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

Bug fix i had forgotten to commit locally

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

Add dump and verify options so we can use them for tests
Write basic tests
Remove compute live in, it's useless for memory ssa (everything is live-in since defs are uses)

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

Make MemorySSA able to be lazy, and add a non-lazy pass for tests to use
Separate out walking interface, move caching walker to use it

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[reames]

This is looking reasonable complete. Any chance we could land it in it's current form and evolve it in tree? Tracking changes is becoming quite hard.

Using this to implement a simple cross block DSE might be an interesting proof of concept. I'm not set on that idea, but it would be nice to have something in tree to show this actually working. Once we have that, we can migrate other passes to it one by one.

[• dberlin]

Update API, fix all documentation, fix use list bugs
Should be ready enough now.

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

Trigger an update so phabricator resends this review email.
This code should now be ready to go in.

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

Update MemorySSA API to allow specifying where insertion occurs

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

Update linear time phi placement algorithm for bug fixes
Add an API for adding memory uses

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

Just a friendly ping. I think i've addressed all previous comments on
MemorySSA, and have patches ready to make it used in-tree (see the
MergedLoadStoreMotion rewrite. I also have a MemCpyOptimizer rewrite
in progress).

[reames]

Ok, this review excludes MemorySSA.cpp. I'll come back to that in a separate pass shortly.

I see a bunch of small issues, but nothing so far that absolutely needs to be addressed before submission. Given that a) this is an off by default change, and b) the author is a long established trusted committer, I have no problem with review comments being addresses as this evolves further in tree.

When you do submit, please separate the AA changes into their own submission. It should go in a few days before the memorySSA so that we can identify breakage in isolation.

I would also strongly prefer to see more test cases included. If nothing else, having a set of small examples for the semi obvious cases would make the usage of memory ssa much easier to understand.

include/llvm/Analysis/AliasAnalysis.h

374 ↗

(On Diff #23347)

I might make this a switch so that all the cases are obviously covered and checked by the compiler.

508 ↗

(On Diff #23347)

This seems like a subset of the getModRefInfo interface. Is there a strong reason it needs to be phrased differently?

include/llvm/IR/Function.h

457 ↗

(On Diff #23347)

Again, submit separately please.

include/llvm/Transforms/Utils/MemorySSA.h

15 ↗

(On Diff #23347)

Given clobbers aren't instructions, please phrase this as:
such as loads, stores, and calls, and atomics.

Also, you *really* need to update this comment to include the bits about only mayalias accesses being on the immediate use list. This is badly needed to prevent confusion.

61 ↗

(On Diff #23347)

This comment is now explicitly wrong.

100 ↗

(On Diff #23347)

Should this be part of the public interface?

117 ↗

(On Diff #23347)

Aren't all of these actually users not uses? Not sure there's a real distinction for memory SSA, but this might be slightly confusing by analogy to normal def-use. If there is no difference, maybe add a comment that clarifies that?

One possible interesting case, what happens if you have a partially overlapping memcpy? Is that a single user or two uses?

132 ↗

(On Diff #23347)

Why do these need to be friends? Shouldn't protected access be enough? I *really* dislike having friend declarations here unless they're absolutely necessary.

148 ↗

(On Diff #23347)

"find" seems like the wrong name here. Possible hasUse?

153 ↗

(On Diff #23347)

Should this possibly be a pure virtual function? 0 doesn't seem overly meaningful here.

195 ↗

(On Diff #23347)

Again, does this need to be public?

227 ↗

(On Diff #23347)

Define clobber. Preferrably using mayalias terminology. References to AA docs are fine.

260 ↗

(On Diff #23347)

Is this still true with the mayalias changes? If so, how does that work? Respond as a comment in code please.

269 ↗

(On Diff #23347)

Should this be called on a phi? Maybe llvm_unreachable?

292 ↗

(On Diff #23347)

Move definition to cpp please.

326 ↗

(On Diff #23347)

Args seems like a confusing name here. What does the PHINode class call these? Use that for consistency please.

407 ↗

(On Diff #23347)

When should this (and the next few routines) be used?

429 ↗

(On Diff #23347)

Hopefully, this is checked in an assertion somewhere?

539 ↗

(On Diff #23347)

Can you expand this comment? From the naming, it's not clear when I'd use one of these and why.

550 ↗

(On Diff #23347)

I'd suggest clarifying this as "by skipping any def which AA can prove does not alias the location(s) accessed by the instruction given"

553 ↗

(On Diff #23347)

This comment is unclear?

562 ↗

(On Diff #23347)

Dead code, please remove!

571 ↗

(On Diff #23347)

I'm confused, am I allowed to call getClobberingMemoryAccess on a DoNothingMemorySSAWalker? What are the semantics here?

576 ↗

(On Diff #23347)

"real"? As opposed to?

lib/Analysis/AliasAnalysis.cpp

351 ↗

(On Diff #23347)

These *need* to submitted separately with test cases if possible.

[reames]

include/llvm/Analysis/AliasAnalysis.h

148 ↗

(On Diff #23347)

This can and should go in separately.

371 ↗

(On Diff #23347)

Same as above.

[reames]

Ok, here's the MemorySSA.cpp parts of the review. Lots of small style comments, a couple of larger concerns. At least some of the code duplication would be nice to address before submission, though I wouldn't say it's mandatory as long as it was done quickly thereafter.

I'm uncomfortable with some of the bits inside the CachingMemorySSAWalker routines. Would you mind submitting the first version of the memory SSA code without them and then reviewing those separately? There's some very complex logic there that duplicates parts of MDA. Given how tricky MDA has been, I'm really not comfortable with the code in the walkers as it stands today without a much more careful review

lib/Transforms/Utils/MemorySSA.cpp

49 ↗

(On Diff #23347)

Isn't this line redundant with the above?

79 ↗

(On Diff #23347)

I'm skipping all the code copied from PMtR. I'm going to assume it's correct.

p.s. Any chance this could be expressed in a common template used by both places?

281 ↗

(On Diff #23347)

Can we use smart pointers by chance? The manual memory management seems unnecessary.

303 ↗

(On Diff #23347)

We no longer throw this out do we?

321 ↗

(On Diff #23347)

This should be pulled out in a helper lambda or otherwise commoned.

327 ↗

(On Diff #23347)

else if - once you do that, can't this just become:
if (def) {
} else if (use) {
} else {

llvm_unreachable()

}

415 ↗

(On Diff #23347)

Don't you need to add these to the per-BB list? Also, can this code be commoned with the memory SSA construction path?

427 ↗

(On Diff #23347)

It looks like you're not really using the instruction at all, maybe just pass the basicblock to begin with?

462 ↗

(On Diff #23347)

Again, can we common this somewhere? You've got this lazy creation scatter all over the place.

Accesses = ensureBBList(BB)?

504 ↗

(On Diff #23347)

This seems odd. Shouldn't we be putting Replacer in the spot Replacee was?

547 ↗

(On Diff #23347)

Did you possible mean a != here?

560 ↗

(On Diff #23347)

This condition really doesn't feel like a "replacement". Is it time to reconsider the interface and turn this into an assert?

571 ↗

(On Diff #23347)

It might be cleaner to unconditionally insert the new value, then fold single use memory phis. Having a fold single use phi utility might also be useful on the public interface.

Actually, are you expecting an invariant where such phis are always folded? If so, please update comments to say so.

611 ↗

(On Diff #23347)

If there are no uses, why is it required? Did you mean:
use_empty || onlySingleValue?

620 ↗

(On Diff #23347)

You've got this code repeated a bit. Utility function on memoryPhi?

821 ↗

(On Diff #23347)

Can't you just use cl opt for this?

852 ↗

(On Diff #23347)

smart pointer?

942 ↗

(On Diff #23347)

Er, huh? How can this happen? And should it?

1005 ↗

(On Diff #23347)

Er, what? This just seems flat out wrong. Two loads on the same thread to the same location are always ordered regardless of flags. Ordering doesn't bypass a mustalias...

Unless this is absolutely required, I would request you separate this and review it separately.

1025 ↗

(On Diff #23347)

I might be missing something here, but how is this not an infinite recursion on itself?

1042 ↗

(On Diff #23347)

This seems suspicious to me as said above.

More generally, the smarts of this routine deserve to be reviewed independently.

1083 ↗

(On Diff #23347)

Mild preference, change this to:
else if (CallSite CS = ...) {
} else {
}

1110 ↗

(On Diff #23347)

Does this optimization actually help? Just curious.

[jmolloy]

Hi Daniel,

I'm looking at making LLVM recognize an idiom that I think MemorySSA+PRE would handle. What's the status of MemorySSA?

I know you haven't had much time to work on it recently; is it in such a state that someone (me) could help progress it?

Cheers,

James

[• dberlin]

Yes, it was looked at and approved, it just needs more tests written
and the initial update infrastructure ripped out and then slowly
rebuilt.
I can send you the latest patch, which fixes a bunch of bugs/etc.

[hfinkel]

In D7864#216848, @dberlin wrote:

Yes, it was looked at and approved, it just needs more tests written
and the initial update infrastructure ripped out and then slowly
rebuilt.
I can send you the latest patch, which fixes a bunch of bugs/etc.

Can you please post it here too, so we can continue the review?

[• dberlin]

Refactor and rewrite a lot of stuff. This is somewhat of an intermediate state, so it's a bit messier than i'd like.
This version builds but i haven't played with it in a few months, so i expect it is bitrotted and may be a bit broken.
I am updating it mainly to give someone else a chance to takeover since i'm going to be swamped for a while.

Main algorithmic changes:
We now use BFS to find nearest dominating access in the caching walker.
I've performed extensive performance testing of hybrid DFS and BFS schemes, and BFS wins every time.
A bunch of things have started being iteratorized.
The caching walker has been modified a bit. The current caching scheme is fast, but can't easily be invalidated on a per-access basis.
The overhead of simple schemes that supported that level of invalidation was *quite* large.
(the ideal caching storage scheme here is actually quadtrees).

The caching walker now uses the same phi translation mechanism that memorydepedenceanalysis does.
I'm not sure this is a good thing, as that phi translation mechanism is ... strange and weird and has weird optimizations embedded in it.
It may be better to get worse optimization to start and clean that stuff up.

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

Also note: BFS or a hybrid DFS scheme is required to actually fulfill the
API guarantee we make - that we will give you the nearest dominating access.
DFS may find an access that is dominating but not the nearest dominating
one, depending on what paths it follows.

[hfinkel]

Thanks for the update!

Quick comment (just so I don't forget), there are a few places that have:

assert(N < 10000 && ..)

that need to become actual conservative bailouts instead of asserts.

lib/Transforms/Utils/MemorySSA.cpp

1284 ↗

(On Diff #43896)

You might also want to check AA->pointsToConstantMemory here.

[• dberlin]

Yeah, these were just debugging asserts. They are there so i got asserts
on infinite loops and could bugpoint the testcases.
There shouldn't be any bugs left (most of the infinite loops were due to
inconsistency between the iterator and phitransaddr in what the result of
phi translation was. Now that they use the same thing, there is no
inconsistency) :)

They should definitely be removed, and i've run memoryssa across google's
codebase without them triggering.

I'm pretty sure i won't get to this patch in the next 6 months, though i'm
trying to find someone on my side to take it over. If someone else wants
to, that'd be awesome as well.

[• dberlin]

Fix comments, remove DFS, update tests to work again (aliasing got too smart)

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[hfinkel]

lib/Transforms/Utils/MemorySSA.cpp

1124 ↗

(On Diff #43896)

This algorithm in getClobberingMemoryAccess does not seem correct. If we have this:

   [ entry ]
       |
      ...
       |
   (clobbering access - a)
      ...
  /          \
...         (clobbering access - b)
 |              |
...           ....
 \              /
        ...
 (starting access)

UpwardsBFSWalkAccess seems to walk upward until it finds its first clobbering access, and returns that when found. In the case above, if we find clobbering access (b) first, then we're okay, because we'll pick some dominating def farther down the path which is on all paths to both dominating accesses.

But, if we hit clobbering access (a) first, then we'll accept that (incorrectly) as the new final access, because it dominates the original. However, not all paths to all aliasing defs terminate (or pass through) that access.

I think that you can continue the walk so that you search the entire subgraph to make sure that all paths terminate at the same aliasing def, and only in that case, use it (and, in that case, you know it must dominate).

My attempts to think up a good solution to the more-general problem have thus-far been stymied by this situation:

[entry]
  |
.....
(clobbering access - b)
 |
....   ________________________________
  \   /                                                               |
   (x)                                                               |
 ......                                                               |
    |                                                                 |
    |    ______________________                 |
     \   /                                        |                   |
(starting access)                        |                   |
     ...                                           |                   |
 (clobbering access - a)              |                   |
    ...                                            |                   |
    | |                                            |                   |
    | |______________________|                   |
    |                                                                  |
    |_________________________________|

The idea is that the block with the starting access also later has a clobbering access (a). This block is one of its own predecessors, but that's not the only path from clobbering access (a) back to the starting access. And furthermore, the other path flows through other blocks that do properly dominate the starting access. Thus, even if you were to pick a def (or phi) on the path from the starting access through its dominating predecessor, say at (x), and even though there are paths to all aliasing defs along that route, you'd still be wrong because there exists a path to clobbering access (a) that does not contain (x) (the path following the inner-loop backedge).

Thoughts? (am I off base here?)

[• dberlin]

In-progress update

This brings back the DFS code and starts to update the BFS code to do the right thing with phi nodes.
Given the structure of the SSA graph and our requirements, i'm pretty positive both DFS and BFS will give correct results (and testing them against each other bears this out, though is not exhaustive).

Given that, and how ugly the BFS code is now starting to become to support the phi node rules, i'm tempted to get rid of it and just stick with DFS.

Thoughts welcome :)

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[• dberlin]

Remove BFS code, fix a bug in DFS code.

(Updating so gbiv can take over)

Updating D7864: This patch introduces MemorySSA, a virtual SSA form for memory.

Details on what it looks like are in MemorySSA.h

[george.burgess.iv]

It turns out that *can* steal revisions -- yay.

Taking ownership, killing D16743

[george.burgess.iv]

Added Danny's patch that makes everything a User, as well as all of the updates published originally in the now-abandoned D16743.

Addressed all feedback in D16743, as well.

[• dberlin]

You need

static inline bool classof(const Value *V) {
<right stuff>
}

For each of the classes. I only pasted it into memorydef, but without it, walking over operands and casting uses will not work.

[• dberlin]

But other than that, LGTM, i don't really have anything to add.
Thanks for taking this on!

[george.burgess.iv]

• Marked dead feedback as done, responded to others, will respond to remaining non-done + unanswered by tomorrow
• Fixed Danny's comment about having classof(Value*)
• Moved initialize[...]Pass methods to fix a bug with how we were trying to link things
• Ripped out some optimizations (e.g. checks for invariant loads, etc.) because those seem mildly controversial, and can be added in a later patch

[george.burgess.iv]

Thanks for the LGTM, Danny.

First wave of responses...

include/llvm/Transforms/Utils/MemorySSA.h

133 ↗

(On Diff #46563)

Because it breaks code like (ripped from MemoryDef::print(raw_ostream &)):

MemoryAccess *UO = getDefiningAccess();
if (UO && UO->getID()) // Breaks
  // ...

...Because protected doesn't let you call protected methods on other MemoryAccesses. You can call them on yourself, or you can call them on other MemoryDefs. That's all.

Do you think it would be better to make getID() public, or to keep the friend declarations? I'm happy with either.

151 ↗

(On Diff #46563)

It wasn't entirely an isa relationship. It's now refactored to be more clear.

293 ↗

(On Diff #46563)

Definition of MemoryPhi::setIncomingValue is now substantially smaller. Leaving in header, unless you'd still like to see it moved.

408 ↗

(On Diff #46563)

Update API has been removed for now.

lib/Transforms/Utils/MemorySSA.cpp

80 ↗

(On Diff #46563)

Comment is on code that seems to have been removed.

87 ↗

(On Diff #46563)

Comment is on code that seems to have been removed.

152 ↗

(On Diff #46563)

Comment is on code that seems to have been removed.

621 ↗

(On Diff #46563)

Now only exists in one place AFAICT.

963 ↗

(On Diff #46563)

Nuked possiblyAffectedBy from orbit; it's an optimization that can be readded later.

963 ↗

(On Diff #46563)

Do you still hold this opinion? (Potentially dubious is now in CachingMemorySSAWalker::instructionClobbersQuery)

963 ↗

(On Diff #46563)

Marked with a TODO so we can look into that when we have users of MemorySSA/a more complete API/...

963 ↗

(On Diff #46563)

Seems old; we now use DFS. Marking as done, but added a test case of your second example.

963 ↗

(On Diff #46563)

I think it's better to get MemorySSA in and work on little optimizations like this after that. Removed the invariant check (but kept the test as an XFAIL), to be readded in the near future as a set of enhancements.

[george.burgess.iv]

After further investigation, the rest of the comments seem to be dead/already addressed, too; marked them as done.

Also, wow. Responding to comments at the bottom *really* didn't go well. I thought it would reflow them, but apparently not. Sorry about that. :/

Either way, I'll leave this open for a bit, in case anyone has any other remarks. Assuming Danny's LGTM is sufficient to let me commit, I'll do so lateish Wednesday if there are no objections by then. :)

Trying to make the comment/response chain at the bottom look less confusing...

reames

Er, what? This just seems flat out wrong. Two loads on the same thread to the same location are always ordered regardless of flags. Ordering doesn't bypass a mustalias...
Unless this is absolutely required, I would request you separate this and review it separately.

Nuked possiblyAffectedBy from orbit; it's an optimization that can be readded later.

Does this optimization actually help? Just curious.

Marked with a TODO so we can look into that when we have users of MemorySSA/a more complete API/...

This seems suspicious to me as said above.
More generally, the smarts of this routine deserve to be reviewed independently.

possiblyAffectedBy was removed, so I think this comment doesn't apply anymore (nor does my initial response to it). :)

hfinkel

You might also want to check AA->pointsToConstantMemory here.

I think it's better to get MemorySSA in and work on little optimizations like this after that. Removed the invariant check (but kept the test as an XFAIL), to be readded in the near future as a set of enhancements. (Good point, though!)

This algorithm in getClobberingMemoryAccess does not seem correct [...]

Seems old; we now use DFS. Marking as done, but added a test case of your second example.

[reames]

Explicitly LGTM as well. I'm really looking forward to having this in tree so that we can start experimenting and iterating on top of it.

I don't think we're going to be ready to switch the default for a while yet, but being able to iterate and work through the bugs in tree is going to be very empowering and will likely speedup progress.

p.s. What's your planned next step? I'd suggest implementing a very simple pass which just runs obvious peepholes over the MemorySSA representation. (e.g. a MemCombine pass). My suspicion is that will be a good way to stress the implementation and might even be profitable enough to just turn on by default. In particular, a pass that got the easy forms of cross block DSE would be very useful..

[• dberlin]

FWIW: My plan was going to start to replace some of the existing
memorydependence users which are O(N^2) and ugly with simpler O(N)
algorithms using MemorySSA

[george.burgess.iv]

Explicitly LGTM as well. I'm really looking forward to having this in tree so that we can start experimenting and iterating on top of it.

Thanks! (Also, naturally, thanks to everyone who reviewed/commented/... :) )

I don't think we're going to be ready to switch the default for a while yet, but being able to iterate and work through the bugs in tree is going to be very empowering and will likely speedup progress.

It seems that there are a number of people who are concerned about subtle bugs (both in MemorySSA, and in user code -- especially if this lets us optimize code in new and exciting ways). So, I agree that making this opt-in for a time is a good idea.

What's your planned next step? I'd suggest implementing a very simple pass which just runs obvious peepholes over the MemorySSA representation. (e.g. a MemCombine pass). My suspicion is that will be a good way to stress the implementation and might even be profitable enough to just turn on by default. In particular, a pass that got the easy forms of cross block DSE would be very useful.

I don't have any plans. If we want to port existing passes soon, we'll either need an update API, or we'll need to restrict ourselves to passes that don't use the update API. Also, we'll need a wrapper that either queries MemDep or MemorySSA if we want to have a global "use MemorySSA?" flag. If we think making a simple MemorySSA-based DSE pass would be a better first step, I'm fine doing that, as well. Same thing goes for readding the optimizations that were stripped out today.

So long as progress is made, I'm happy. :)

[davidxl]

Excellent progress.

As Danny mentioned, the O(N^2) behavior of some MemoryDependence users
are biting us badly in some cases. For instance, some of the C++
source takes forever to compile with profile instrumentation turned on
and the root cause is due to that. Replacing those with memSSA based
algorithms will have very large (good) impact.

thanks,

David

[• dberlin]

I don't have any plans. If we want to port existing passes soon, we'll
either need an update API, or we'll need to restrict ourselves to passes
that don't use the update API.

FWIW: For the passes i played with converting, timing compute memssa + use
it vs memdep, for basically all testcases i could find, the time difference
was in the noise.
For larger cases, memssa is a straight win.
This is because most of the memdep passes are walking a ton of blocks/insts
repeatedly anyway.

Also, we'll need a wrapper that either queries MemDep or MemorySSA if we
want to have a global "use MemorySSA?" flag.

I'm not sure such a flag makes sense, given the above. I'm not sure you
*want* to try to alternate between them, because a lot of your order of
magnitude speedups will come from doing things fundamentally different
than random querying (IE move to more "SSA-like" algorithms).

(Though i expect using memssa as a straight replacement will still likely
be faster).

If we think making a simple MemorySSA-based DSE pass would be a better

first step, I'm fine doing that, as well. Same thing goes for readding the
optimizations that were stripped out today.

If you do the optimizations, i'm happy to start to convert passes. I was
going to just do them in close to the order necessary to preserve memssa
from beginning to end of opts that use it, but happy to do whatever.

(mergedloadstoremotion and memcpyoptimizer are the easiest things to
convert and will give speedups).

Converting most passes requires building an update API, and converting a
few of them and seeing what they want out of updating is the best way i can
think of to design that API.

(You can see where i played with it in the mergedloadstoremotion mpass).

So long as progress is made, I'm happy. :)

http://reviews.llvm.org/D7864

[jlebar]

I'm seeing an asan failure in one of the tests added here. I'm building from tip, clean tree, linux x86-64.

-- Testing: 1 of 15992 tests, 1 threads --
FAIL: LLVM-Unit :: Transforms/Utils/UtilsTests/MemorySSA.RemoveMemoryAccess (1 of 1)
******************** TEST 'LLVM-Unit :: Transforms/Utils/UtilsTests/MemorySSA.RemoveMemoryAccess' FAILED ********************
Note: Google Test filter = MemorySSA.RemoveMemoryAccess
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from MemorySSA
[ RUN      ] MemorySSA.RemoveMemoryAccess
=================================================================
==5162==ERROR: AddressSanitizer: heap-use-after-free on address 0x60800000bb68 at pc 0x000000c96299 bp 0x7ffea8a84a90 sp 0x7ffea8a84a88
READ of size 8 at 0x60800000bb68 thread T0
  #0 0xc96298 in llvm::MemoryAccess::getBlock() const /usr/local/google/home/jlebar/llvm/src/include/llvm/Transforms/Utils/MemorySSA.h:116:34
  #1 0xc96298 in llvm::MemorySSA::removeFromLookups(llvm::MemoryAccess*) /usr/local/google/home/jlebar/llvm/src/lib/Transforms/Utils/MemorySSA.cpp:469
  #2 0x58ffee in MemorySSA_RemoveMemoryAccess_Test::TestBody() /usr/local/google/home/jlebar/llvm/src/unittests/Transforms/Utils/MemorySSA.cpp:65:3
  #3 0xd66ec3 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
  #4 0xd66ec3 in testing::Test::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2161
  #5 0xd6b854 in testing::TestInfo::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2309:5
  #6 0xd6cb16 in testing::TestCase::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2416:5
  #7 0xd85c7a in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:4207:11
  #8 0xd84eaf in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
  #9 0xd84eaf in testing::UnitTest::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:3841
  #10 0xd3cc67 in main /usr/local/google/home/jlebar/llvm/src/utils/unittest/UnitTestMain/TestMain.cpp:47:10
  #11 0x7fc886a0dec4 in __libc_start_main /build/eglibc-3GlaMS/eglibc-2.19/csu/libc-start.c:287
  #12 0x54bd09 in _start (/usr/local/google/home/jlebar/code/llvm/asan/unittests/Transforms/Utils/UtilsTests+0x54bd09)

0x60800000bb68 is located 72 bytes inside of 96-byte region [0x60800000bb20,0x60800000bb80)
freed by thread T0 here:
  #0 0x4c734b in operator delete(void*) (/usr/local/google/home/jlebar/code/llvm/asan/unittests/Transforms/Utils/UtilsTests+0x4c734b)
  #1 0xc9602e in llvm::ilist_node_traits<llvm::MemoryAccess>::deleteNode(llvm::MemoryAccess*) /usr/local/google/home/jlebar/llvm/src/include/llvm/ADT/ilist.h:160:39
  #2 0xc9602e in llvm::iplist<llvm::MemoryAccess, llvm::ilist_traits<llvm::MemoryAccess> >::erase(llvm::ilist_iterator<llvm::MemoryAccess>) /usr/local/google/home/jlebar/llvm/src/include/llvm/ADT/ilist.h:466
  #3 0xc9602e in llvm::iplist<llvm::MemoryAccess, llvm::ilist_traits<llvm::MemoryAccess> >::erase(llvm::MemoryAccess*) /usr/local/google/home/jlebar/llvm/src/include/llvm/ADT/ilist.h:470
  #4 0xc9602e in llvm::MemorySSA::removeFromLookups(llvm::MemoryAccess*) /usr/local/google/home/jlebar/llvm/src/lib/Transforms/Utils/MemorySSA.cpp:467
  #5 0x58ffee in MemorySSA_RemoveMemoryAccess_Test::TestBody() /usr/local/google/home/jlebar/llvm/src/unittests/Transforms/Utils/MemorySSA.cpp:65:3
  #6 0xd66ec3 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
  #7 0xd66ec3 in testing::Test::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2161
  #8 0xd6b854 in testing::TestInfo::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2309:5
  #9 0xd6cb16 in testing::TestCase::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2416:5
  #10 0xd85c7a in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:4207:11
  #11 0xd84eaf in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
  #12 0xd84eaf in testing::UnitTest::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:3841
  #13 0xd3cc67 in main /usr/local/google/home/jlebar/llvm/src/utils/unittest/UnitTestMain/TestMain.cpp:47:10
  #14 0x7fc886a0dec4 in __libc_start_main /build/eglibc-3GlaMS/eglibc-2.19/csu/libc-start.c:287

previously allocated by thread T0 here:
  #0 0x4c6e0b in operator new(unsigned long) (/usr/local/google/home/jlebar/code/llvm/asan/unittests/Transforms/Utils/UtilsTests+0x4c6e0b)
  #1 0xa7ec05 in llvm::User::allocateFixedOperandUser(unsigned long, unsigned int, unsigned int) /usr/local/google/home/jlebar/llvm/src/lib/IR/User.cpp:129:7
  #2 0xa7ec05 in llvm::User::operator new(unsigned long, unsigned int) /usr/local/google/home/jlebar/llvm/src/lib/IR/User.cpp:147
  #3 0xc92784 in llvm::MemoryDef::operator new(unsigned long) /usr/local/google/home/jlebar/llvm/src/include/llvm/Transforms/Utils/MemorySSA.h:271:41
  #4 0xc92784 in llvm::MemorySSA::createNewAccess(llvm::Instruction*, bool) /usr/local/google/home/jlebar/llvm/src/lib/Transforms/Utils/MemorySSA.cpp:378
  #5 0xc9098f in llvm::MemorySSA::buildMemorySSA(llvm::AAResults*, llvm::DominatorTree*) /usr/local/google/home/jlebar/llvm/src/lib/Transforms/Utils/MemorySSA.cpp:260:26
  #6 0x58f91b in MemorySSA_RemoveMemoryAccess_Test::TestBody() /usr/local/google/home/jlebar/llvm/src/unittests/Transforms/Utils/MemorySSA.cpp:55:29
  #7 0xd66ec3 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
  #8 0xd66ec3 in testing::Test::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2161
  #9 0xd6b854 in testing::TestInfo::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2309:5
  #10 0xd6cb16 in testing::TestCase::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2416:5
  #11 0xd85c7a in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:4207:11
  #12 0xd84eaf in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
  #13 0xd84eaf in testing::UnitTest::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:3841
  #14 0xd3cc67 in main /usr/local/google/home/jlebar/llvm/src/utils/unittest/UnitTestMain/TestMain.cpp:47:10
  #15 0x7fc886a0dec4 in __libc_start_main /build/eglibc-3GlaMS/eglibc-2.19/csu/libc-start.c:287

SUMMARY: AddressSanitizer: heap-use-after-free /usr/local/google/home/jlebar/llvm/src/include/llvm/Transforms/Utils/MemorySSA.h:116 llvm::MemoryAccess::getBlock() const
Shadow bytes around the buggy address:
0x0c107fff9710: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c107fff9720: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c107fff9730: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c107fff9740: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c107fff9750: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
=>0x0c107fff9760: fa fa fa fa fd fd fd fd fd fd fd fd fd[fd]fd fd
0x0c107fff9770: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
0x0c107fff9780: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c107fff9790: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c107fff97a0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c107fff97b0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 03 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable:           00
Partially addressable: 01 02 03 04 05 06 07 
Heap left redzone:       fa
Heap right redzone:      fb
Freed heap region:       fd
Stack left redzone:      f1
Stack mid redzone:       f2
Stack right redzone:     f3
Stack partial redzone:   f4
Stack after return:      f5
Stack use after scope:   f8
Global redzone:          f9
Global init order:       f6
Poisoned by user:        f7
Container overflow:      fc
ASan internal:           fe
==5162==ABORTING

********************
Testing Time: 0.56s
********************
Failing Tests (1):
  LLVM-Unit :: Transforms/Utils/UtilsTests/MemorySSA.RemoveMemoryAccess

Unexpected Failures: 1

[george.burgess.iv]

Thanks for the report -- looking into it now :)

[george.burgess.iv]

It seems that the test in question wasn't introduced by this patch, but the fix seems somewhat trivial, so my attempt was checked in as r262452. Please let me know if that didn't end up fixing it (I lack an ASAN-instrumented build of LLVM :) ).

[jlebar]

Please let me know if that didn't end up fixing it (I lack an ASAN-instrumented build of LLVM :) ).

Works now, although now that it's fixed asan is showing me memory leaks. :) One seems to be in the test itself, but another appears to be in the actual code.

Clearly not as urgent, but here's the report:

********************
FAIL: LLVM-Unit :: Transforms/Utils/UtilsTests/MemorySSA.RemoveMemoryAccess (15985 of 15994)
******************** TEST 'LLVM-Unit :: Transforms/Utils/UtilsTests/MemorySSA.RemoveMemoryAccess' FAILED ********************
Note: Google Test filter = MemorySSA.RemoveMemoryAccess
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from MemorySSA
[ RUN      ] MemorySSA.RemoveMemoryAccess
[       OK ] MemorySSA.RemoveMemoryAccess (25 ms)
[----------] 1 test from MemorySSA (25 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (26 ms total)
[  PASSED  ] 1 test.

=================================================================
==147403==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 304 byte(s) in 1 object(s) allocated from:
    #0 0x4c66ab in operator new(unsigned long) (/usr/local/google/home/jlebar/code/llvm/asan/unittests/Transforms/Utils/UtilsTests+0x4c66ab)
    #1 0xc89da5 in llvm::MemorySSA::buildMemorySSA(llvm::AAResults*, llvm::DominatorTree*) /usr/local/google/home/jlebar/llvm/src/lib/Transforms/Utils/MemorySSA.cpp:234:3
    #2 0x58f1eb in MemorySSA_RemoveMemoryAccess_Test::TestBody() /usr/local/google/home/jlebar/llvm/src/unittests/Transforms/Utils/MemorySSA.cpp:55:29
    #3 0xd60843 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
    #4 0xd60843 in testing::Test::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2161
    #5 0xd651d4 in testing::TestInfo::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2309:5
    #6 0xd66496 in testing::TestCase::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2416:5
    #7 0xd7f5fa in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:4207:11
    #8 0xd7e82f in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
    #9 0xd7e82f in testing::UnitTest::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:3841
    #10 0xd365e7 in main /usr/local/google/home/jlebar/llvm/src/utils/unittest/UnitTestMain/TestMain.cpp:47:10
    #11 0x7fb091525ec4 in __libc_start_main /build/eglibc-3GlaMS/eglibc-2.19/csu/libc-start.c:287

Indirect leak of 1024 byte(s) in 1 object(s) allocated from:
    #0 0x4c66ab in operator new(unsigned long) (/usr/local/google/home/jlebar/code/llvm/asan/unittests/Transforms/Utils/UtilsTests+0x4c66ab)
    #1 0x58e7b8 in MemorySSA_RemoveMemoryAccess_Test::TestBody() /usr/local/google/home/jlebar/llvm/src/unittests/Transforms/Utils/MemorySSA.cpp:53:3
    #2 0xd60843 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
    #3 0xd60843 in testing::Test::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2161
    #4 0xd651d4 in testing::TestInfo::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2309:5
    #5 0xd66496 in testing::TestCase::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2416:5
    #6 0xd7f5fa in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:4207:11
    #7 0xd7e82f in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
    #8 0xd7e82f in testing::UnitTest::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:3841
    #9 0xd365e7 in main /usr/local/google/home/jlebar/llvm/src/utils/unittest/UnitTestMain/TestMain.cpp:47:10
    #10 0x7fb091525ec4 in __libc_start_main /build/eglibc-3GlaMS/eglibc-2.19/csu/libc-start.c:287

ndirect leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x4c66ab in operator new(unsigned long) (/usr/local/google/home/jlebar/code/llvm/asan/unittests/Transforms/Utils/UtilsTests+0x4c66ab)
    #1 0x58e777 in MemorySSA_RemoveMemoryAccess_Test::TestBody() /usr/local/google/home/jlebar/llvm/src/unittests/Transforms/Utils/MemorySSA.cpp:52:3
    #2 0xd60843 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
    #3 0xd60843 in testing::Test::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2161
    #4 0xd651d4 in testing::TestInfo::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2309:5
    #5 0xd66496 in testing::TestCase::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2416:5
    #6 0xd7f5fa in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:4207:11
    #7 0xd7e82f in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
    #8 0xd7e82f in testing::UnitTest::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:3841
    #9 0xd365e7 in main /usr/local/google/home/jlebar/llvm/src/utils/unittest/UnitTestMain/TestMain.cpp:47:10
    #10 0x7fb091525ec4 in __libc_start_main /build/eglibc-3GlaMS/eglibc-2.19/csu/libc-start.c:287

Indirect leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x4c66ab in operator new(unsigned long) (/usr/local/google/home/jlebar/code/llvm/asan/unittests/Transforms/Utils/UtilsTests+0x4c66ab)
    #1 0x58ee7f in void llvm::AAResults::addAAResult<llvm::BasicAAResult>(llvm::BasicAAResult&) /usr/local/google/home/jlebar/llvm/src/include/llvm/Analysis/AliasAnalysis.h:173:5
    #2 0x58ee7f in MemorySSA_RemoveMemoryAccess_Test::TestBody() /usr/local/google/home/jlebar/llvm/src/unittests/Transforms/Utils/MemorySSA.cpp:54
    #3 0xd60843 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
    #4 0xd60843 in testing::Test::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2161
    #5 0xd651d4 in testing::TestInfo::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2309:5
    #6 0xd66496 in testing::TestCase::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2416:5
    #7 0xd7f5fa in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:4207:11
    #8 0xd7e82f in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
    #9 0xd7e82f in testing::UnitTest::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:3841
    #10 0xd365e7 in main /usr/local/google/home/jlebar/llvm/src/utils/unittest/UnitTestMain/TestMain.cpp:47:10
    #11 0x7fb091525ec4 in __libc_start_main /build/eglibc-3GlaMS/eglibc-2.19/csu/libc-start.c:287

Indirect leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x4c66ab in operator new(unsigned long) (/usr/local/google/home/jlebar/code/llvm/asan/unittests/Transforms/Utils/UtilsTests+0x4c66ab)
    #1 0x592596 in __gnu_cxx::new_allocator<std::unique_ptr<llvm::AAResults::Concept, std::default_delete<llvm::AAResults::Concept> > >::allocate(unsigned long, void const*) /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/ext/new_allocator.h:104:27
    #2 0x592596 in std::_Vector_base<std::unique_ptr<llvm::AAResults::Concept, std::default_delete<llvm::AAResults::Concept> >, std::allocator<std::unique_ptr<llvm::AAResults::Concept, std::default_delete<llvm::AAResults::Concept> > > >::_M_allocate(unsigned long) /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/stl_vector.h:168
    #3 0x592596 in void std::vector<std::unique_ptr<llvm::AAResults::Concept, std::default_delete<llvm::AAResults::Concept> >, std::allocator<std::unique_ptr<llvm::AAResults::Concept, std::default_delete<
llvm::AAResults::Concept> > > >::_M_emplace_back_aux<llvm::AAResults::Model<llvm::BasicAAResult>*>(llvm::AAResults::Model<llvm::BasicAAResult>*&&) /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/vector.tcc:404
    #4 0x58f1cd in void std::vector<std::unique_ptr<llvm::AAResults::Concept, std::default_delete<llvm::AAResults::Concept> >, std::allocator<std::unique_ptr<llvm::AAResults::Concept, std::default_delete<
llvm::AAResults::Concept> > > >::emplace_back<llvm::AAResults::Model<llvm::BasicAAResult>*>(llvm::AAResults::Model<llvm::BasicAAResult>*&&) /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/vector.tcc:101:4
    #5 0x58f1cd in void llvm::AAResults::addAAResult<llvm::BasicAAResult>(llvm::BasicAAResult&) /usr/local/google/home/jlebar/llvm/src/include/llvm/Analysis/AliasAnalysis.h:173
    #6 0x58f1cd in MemorySSA_RemoveMemoryAccess_Test::TestBody() /usr/local/google/home/jlebar/llvm/src/unittests/Transforms/Utils/MemorySSA.cpp:54
    #7 0xd60843 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
    #8 0xd60843 in testing::Test::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2161
    #9 0xd651d4 in testing::TestInfo::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2309:5
    #10 0xd66496 in testing::TestCase::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2416:5
    #11 0xd7f5fa in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:4207:11
    #12 0xd7e82f in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:2145:12
    #13 0xd7e82f in testing::UnitTest::Run() /usr/local/google/home/jlebar/llvm/src/utils/unittest/googletest/src/gtest.cc:3841
    #14 0xd365e7 in main /usr/local/google/home/jlebar/llvm/src/utils/unittest/UnitTestMain/TestMain.cpp:47:10
    #15 0x7fb091525ec4 in __libc_start_main /build/eglibc-3GlaMS/eglibc-2.19/csu/libc-start.c:287

SUMMARY: AddressSanitizer: 1384 byte(s) leaked in 5 allocation(s).

[• dberlin]

Actually, sigh, we didn't do that already because we hand them back to the
user as well.
Anyway, i'm working up a patch to fix these.

[• dberlin]

This should hopefully fix all the leaks and asan errors (there is one real
bug here).
It is asan clean on OSX now, i am building on linux to do the leak check.

• fixasanissues.diff3 KBDownload

llvm/trunk/include/llvm/IR/Function.h

/// @}
   bool hasPrologueData() const {
     return getSubclassDataFromValue() & (1<<2);
   }
 
   /// \brief Get the prologue data associated with this function.
   Constant *getPrologueData() const;
   void setPrologueData(Constant *PrologueData);
 
+  /// Print the function to an output stream with an optional
+  /// AssemblyAnnotationWriter.
+  void print(raw_ostream &OS, AssemblyAnnotationWriter *AAW = nullptr,
+             bool ShouldPreserveUseListOrder = false,
+             bool IsForDebug = false) const;
+
   /// viewCFG - This function is meant for use from the debugger.  You can just
   /// say 'call F->viewCFG()' and a ghostview window should pop up from the
   /// program, displaying the CFG of the current function with the code for each
   /// basic block inside.  This depends on there being a 'dot' and 'gv' program
   /// in your path.
   ///
   void viewCFG() const;
 

llvm/trunk/include/llvm/IR/Value.def

 #endif
 
 #ifndef HANDLE_CONSTANT_MARKER
 #define HANDLE_CONSTANT_MARKER(MarkerName, ValueName)
 #endif
 
 HANDLE_VALUE(Argument)
 HANDLE_VALUE(BasicBlock)
+HANDLE_VALUE(MemoryUse)
+HANDLE_VALUE(MemoryDef)
+HANDLE_VALUE(MemoryPhi)
 
 HANDLE_GLOBAL_VALUE(Function)
 HANDLE_GLOBAL_VALUE(GlobalAlias)
 HANDLE_GLOBAL_VALUE(GlobalVariable)
 HANDLE_CONSTANT(UndefValue)
 HANDLE_CONSTANT(BlockAddress)
 HANDLE_CONSTANT(ConstantExpr)
 HANDLE_CONSTANT(ConstantAggregateZero)

llvm/trunk/include/llvm/InitializePasses.h

 void initializeMachineSchedulerPass(PassRegistry&);
 void initializeMachineSinkingPass(PassRegistry&);
 void initializeMachineTraceMetricsPass(PassRegistry&);
 void initializeMachineVerifierPassPass(PassRegistry&);
 void initializeMemCpyOptPass(PassRegistry&);
 void initializeMemDepPrinterPass(PassRegistry&);
 void initializeMemDerefPrinterPass(PassRegistry&);
 void initializeMemoryDependenceAnalysisPass(PassRegistry&);
+void initializeMemorySSALazyPass(PassRegistry&);
+void initializeMemorySSAPrinterPassPass(PassRegistry&);
 void initializeMergedLoadStoreMotionPass(PassRegistry &);
 void initializeMetaRenamerPass(PassRegistry&);
 void initializeMergeFunctionsPass(PassRegistry&);
 void initializeModuleDebugInfoPrinterPass(PassRegistry&);
 void initializeNaryReassociatePass(PassRegistry&);
 void initializeNoAAPass(PassRegistry&);
 void initializeObjCARCAAWrapperPassPass(PassRegistry&);
 void initializeObjCARCAPElimPass(PassRegistry&);

llvm/trunk/include/llvm/Transforms/Utils/MemorySSA.h

+//===- MemorySSA.h - Build Memory SSA ---------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// \file
+// \brief This file exposes an interface to building/using memory SSA to
+// walk memory instructions using a use/def graph.
+//
+// Memory SSA class builds an SSA form that links together memory access
+// instructions such loads, stores, atomics, and calls. Additionally, it does a
+// trivial form of "heap versioning" Every time the memory state changes in the
+// program, we generate a new heap version. It generates MemoryDef/Uses/Phis
+// that are overlayed on top of the existing instructions.
+//
+// As a trivial example,
+// define i32 @main() #0 {
+// entry:
+//   %call = call noalias i8* @_Znwm(i64 4) #2
+//   %0 = bitcast i8* %call to i32*
+//   %call1 = call noalias i8* @_Znwm(i64 4) #2
+//   %1 = bitcast i8* %call1 to i32*
+//   store i32 5, i32* %0, align 4
+//   store i32 7, i32* %1, align 4
+//   %2 = load i32* %0, align 4
+//   %3 = load i32* %1, align 4
+//   %add = add nsw i32 %2, %3
+//   ret i32 %add
+// }
+//
+// Will become
+// define i32 @main() #0 {
+// entry:
+//   ; 1 = MemoryDef(0)
+//   %call = call noalias i8* @_Znwm(i64 4) #3
+//   %2 = bitcast i8* %call to i32*
+//   ; 2 = MemoryDef(1)
+//   %call1 = call noalias i8* @_Znwm(i64 4) #3
+//   %4 = bitcast i8* %call1 to i32*
+//   ; 3 = MemoryDef(2)
+//   store i32 5, i32* %2, align 4
+//   ; 4 = MemoryDef(3)
+//   store i32 7, i32* %4, align 4
+//   ; MemoryUse(4)
+//   %7 = load i32* %2, align 4
+//   ; MemoryUse(3)
+//   %8 = load i32* %4, align 4
+//   %add = add nsw i32 %7, %8
+//   ret i32 %add
+// }
+//
+// Given this form, all the stores that could ever effect the load at %8 can be
+// gotten by using the MemoryUse associated with it, and walking from use to def
+// until you hit the top of the function.
+//
+// Each def also has a list of users associated with it, so you can walk from
+// both def to users, and users to defs. Note that we disambiguate MemoryUses,
+// but not the RHS of MemoryDefs. You can see this above at %8, which would
+// otherwise be a MemoryUse(4). Being disambiguated means that for a given
+// store, all the MemoryUses on its use lists are may-aliases of that store (but
+// the MemoryDefs on its use list may not be).
+//
+// MemoryDefs are not disambiguated because it would require multiple reaching
+// definitions, which would require multiple phis, and multiple memoryaccesses
+// per instruction.
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_TRANSFORMS_UTILS_MEMORYSSA_H
+#define LLVM_TRANSFORMS_UTILS_MEMORYSSA_H
+
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/GraphTraits.h"
+#include "llvm/ADT/SmallPtrSet.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/ilist_node.h"
+#include "llvm/ADT/iterator.h"
+#include "llvm/Analysis/AliasAnalysis.h"
+#include "llvm/Analysis/PHITransAddr.h"
+#include "llvm/IR/Dominators.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/OperandTraits.h"
+#include "llvm/IR/Type.h"
+#include "llvm/IR/User.h"
+#include "llvm/IR/Value.h"
+#include "llvm/Pass.h"
+
+namespace llvm {
+class BasicBlock;
+class DominatorTree;
+class Function;
+class MemoryAccess;
+template <class T> class memoryaccess_def_iterator_base;
+using memoryaccess_def_iterator = memoryaccess_def_iterator_base<MemoryAccess>;
+using const_memoryaccess_def_iterator =
+    memoryaccess_def_iterator_base<const MemoryAccess>;
+
+// \brief The base for all memory accesses. All memory accesses in a block are
+// linked together using an intrusive list.
+class MemoryAccess : public User, public ilist_node<MemoryAccess> {
+  void *operator new(size_t, unsigned) = delete;
+  void *operator new(size_t) = delete;
+
+public:
+  // Methods for support type inquiry through isa, cast, and
+  // dyn_cast
+  static inline bool classof(const MemoryAccess *) { return true; }
+  static inline bool classof(const Value *V) {
+    unsigned ID = V->getValueID();
+    return ID == MemoryUseVal || ID == MemoryPhiVal || ID == MemoryDefVal;
+  }
+
+  virtual ~MemoryAccess();
+  BasicBlock *getBlock() const { return Block; }
+
+  virtual void print(raw_ostream &OS) const = 0;
+  virtual void dump() const;
+
+  /// \brief The user iterators for a memory access
+  typedef user_iterator iterator;
+  typedef const_user_iterator const_iterator;
+
+  /// \brief This iterator walks over all of the defs in a given
+  /// MemoryAccess. For MemoryPhi nodes, this walks arguments. For
+  /// MemoryUse/MemoryDef, this walks the defining access.
+  memoryaccess_def_iterator defs_begin();
+  const_memoryaccess_def_iterator defs_begin() const;
+  memoryaccess_def_iterator defs_end();
+  const_memoryaccess_def_iterator defs_end() const;
+
+protected:
+  friend class MemorySSA;
+  friend class MemoryUseOrDef;
+  friend class MemoryUse;
+  friend class MemoryDef;
+  friend class MemoryPhi;
+
+  /// \brief Used internally to give IDs to MemoryAccesses for printing
+  virtual unsigned getID() const = 0;
+
+  MemoryAccess(LLVMContext &C, unsigned Vty, BasicBlock *BB,
+               unsigned NumOperands)
+      : User(Type::getVoidTy(C), Vty, nullptr, NumOperands), Block(BB) {}
+
+private:
+  MemoryAccess(const MemoryAccess &);
+  void operator=(const MemoryAccess &);
+  BasicBlock *Block;
+};
+
+template <>
+struct ilist_traits<MemoryAccess> : public ilist_default_traits<MemoryAccess> {
+  /// See details of the instruction class for why this trick works
+  /// FIXME: The downcast is UB.
+  MemoryAccess *createSentinel() const {
+    return static_cast<MemoryAccess *>(&Sentinel);
+  }
+
+  static void destroySentinel(MemoryAccess *) {}
+
+  MemoryAccess *provideInitialHead() const { return createSentinel(); }
+  MemoryAccess *ensureHead(MemoryAccess *) const { return createSentinel(); }
+  static void noteHead(MemoryAccess *, MemoryAccess *) {}
+
+private:
+  mutable ilist_half_node<MemoryAccess> Sentinel;
+};
+
+inline raw_ostream &operator<<(raw_ostream &OS, const MemoryAccess &MA) {
+  MA.print(OS);
+  return OS;
+}
+
+/// \brief Class that has the common methods + fields of memory uses/defs. It's
+/// a little awkward to have, but there are many cases where we want either a
+/// use or def, and there are many cases where uses are needed (defs aren't
+/// acceptable), and vice-versa.
+///
+/// This class should never be instantiated directly; make a MemoryUse or
+/// MemoryDef instead.
+class MemoryUseOrDef : public MemoryAccess {
+  void *operator new(size_t, unsigned) = delete;
+  void *operator new(size_t) = delete;
+
+public:
+  DECLARE_TRANSPARENT_OPERAND_ACCESSORS(MemoryAccess);
+
+  /// \brief Get the instruction that this MemoryUse represents.
+  Instruction *getMemoryInst() const { return MemoryInst; }
+
+  /// \brief Get the access that produces the memory state used by this Use.
+  MemoryAccess *getDefiningAccess() const { return getOperand(0); }
+
+  static inline bool classof(const MemoryUseOrDef *) { return true; }
+  static inline bool classof(const Value *MA) {
+    return MA->getValueID() == MemoryUseVal || MA->getValueID() == MemoryDefVal;
+  }
+
+protected:
+  friend class MemorySSA;
+
+  MemoryUseOrDef(LLVMContext &C, MemoryAccess *DMA, unsigned Vty,
+                 Instruction *MI, BasicBlock *BB)
+      : MemoryAccess(C, Vty, BB, 1), MemoryInst(MI) {
+    setDefiningAccess(DMA);
+  }
+
+  void setDefiningAccess(MemoryAccess *DMA) { setOperand(0, DMA); }
+
+private:
+  Instruction *MemoryInst;
+};
+template <>
+struct OperandTraits<MemoryUseOrDef>
+    : public FixedNumOperandTraits<MemoryUseOrDef, 1> {};
+DEFINE_TRANSPARENT_OPERAND_ACCESSORS(MemoryUseOrDef, MemoryAccess)
+
+/// \brief Represents read-only accesses to memory
+///
+/// In particular, the set of Instructions that will be represented by
+/// MemoryUse's is exactly the set of Instructions for which
+/// AliasAnalysis::getModRefInfo returns "Ref".
+class MemoryUse final : public MemoryUseOrDef {
+  void *operator new(size_t, unsigned) = delete;
+
+public:
+  DECLARE_TRANSPARENT_OPERAND_ACCESSORS(MemoryAccess);
+
+  // allocate space for exactly one operand
+  void *operator new(size_t s) { return User::operator new(s, 1); }
+
+  MemoryUse(LLVMContext &C, MemoryAccess *DMA, Instruction *MI, BasicBlock *BB)
+      : MemoryUseOrDef(C, DMA, MemoryUseVal, MI, BB) {}
+
+  static inline bool classof(const MemoryUse *) { return true; }
+  static inline bool classof(const Value *MA) {
+    return MA->getValueID() == MemoryUseVal;
+  }
+
+  void print(raw_ostream &OS) const override;
+
+protected:
+  friend class MemorySSA;
+
+  unsigned getID() const override {
+    llvm_unreachable("MemoryUses do not have IDs");
+  }
+};
+template <>
+struct OperandTraits<MemoryUse> : public FixedNumOperandTraits<MemoryUse, 1> {};
+DEFINE_TRANSPARENT_OPERAND_ACCESSORS(MemoryUse, MemoryAccess)
+
+/// \brief Represents a read-write access to memory, whether it is a must-alias,
+/// or a may-alias.
+///
+/// In particular, the set of Instructions that will be represented by
+/// MemoryDef's is exactly the set of Instructions for which
+/// AliasAnalysis::getModRefInfo returns "Mod" or "ModRef".
+/// Note that, in order to provide def-def chains, all defs also have a use
+/// associated with them. This use points to the nearest reaching
+/// MemoryDef/MemoryPhi.
+class MemoryDef final : public MemoryUseOrDef {
+  void *operator new(size_t, unsigned) = delete;
+
+public:
+  DECLARE_TRANSPARENT_OPERAND_ACCESSORS(MemoryAccess);
+
+  // allocate space for exactly one operand
+  void *operator new(size_t s) { return User::operator new(s, 1); }
+
+  MemoryDef(LLVMContext &C, MemoryAccess *DMA, Instruction *MI, BasicBlock *BB,
+            unsigned Ver)
+      : MemoryUseOrDef(C, DMA, MemoryDefVal, MI, BB), ID(Ver) {}
+
+  static inline bool classof(const MemoryDef *) { return true; }
+  static inline bool classof(const Value *MA) {
+    return MA->getValueID() == MemoryDefVal;
+  }
+
+  void print(raw_ostream &OS) const override;
+
+protected:
+  friend class MemorySSA;
+
+  // For debugging only. This gets used to give memory accesses pretty numbers
+  // when printing them out
+  unsigned getID() const override { return ID; }
+
+private:
+  const unsigned ID;
+};
+template <>
+struct OperandTraits<MemoryDef> : public FixedNumOperandTraits<MemoryDef, 1> {};
+DEFINE_TRANSPARENT_OPERAND_ACCESSORS(MemoryDef, MemoryAccess)
+
+/// \brief Represents phi nodes for memory accesses.
+///
+/// These have the same semantic as regular phi nodes, with the exception that
+/// only one phi will ever exist in a given basic block.
+/// Guaranteeing one phi per block means guaranteeing there is only ever one
+/// valid reaching MemoryDef/MemoryPHI along each path to the phi node.
+/// This is ensured by not allowing disambiguation of the RHS of a MemoryDef or
+/// a MemoryPhi's operands.
+/// That is, given
+/// if (a) {
+///   store %a
+///   store %b
+/// }
+/// it *must* be transformed into
+/// if (a) {
+///    1 = MemoryDef(liveOnEntry)
+///    store %a
+///    2 = MemoryDef(1)
+///    store %b
+/// }
+/// and *not*
+/// if (a) {
+///    1 = MemoryDef(liveOnEntry)
+///    store %a
+///    2 = MemoryDef(liveOnEntry)
+///    store %b
+/// }
+/// even if the two stores do not conflict. Otherwise, both 1 and 2 reach the
+/// end of the branch, and if there are not two phi nodes, one will be
+/// disconnected completely from the SSA graph below that point.
+/// Because MemoryUse's do not generate new definitions, they do not have this
+/// issue.
+class MemoryPhi final : public MemoryAccess {
+  void *operator new(size_t, unsigned) = delete;
+  // allocate space for exactly zero operands
+  void *operator new(size_t s) { return User::operator new(s); }
+
+public:
+  /// Provide fast operand accessors
+  DECLARE_TRANSPARENT_OPERAND_ACCESSORS(MemoryAccess);
+
+  MemoryPhi(LLVMContext &C, BasicBlock *BB, unsigned Ver, unsigned NumPreds = 0)
+      : MemoryAccess(C, MemoryPhiVal, BB, 0), ID(Ver), ReservedSpace(NumPreds) {
+    allocHungoffUses(ReservedSpace);
+  }
+
+  // Block iterator interface. This provides access to the list of incoming
+  // basic blocks, which parallels the list of incoming values.
+  typedef BasicBlock **block_iterator;
+  typedef BasicBlock *const *const_block_iterator;
+
+  block_iterator block_begin() {
+    auto *Ref = reinterpret_cast<Use::UserRef *>(op_begin() + ReservedSpace);
+    return reinterpret_cast<block_iterator>(Ref + 1);
+  }
+
+  const_block_iterator block_begin() const {
+    const auto *Ref =
+        reinterpret_cast<const Use::UserRef *>(op_begin() + ReservedSpace);
+    return reinterpret_cast<const_block_iterator>(Ref + 1);
+  }
+
+  block_iterator block_end() { return block_begin() + getNumOperands(); }
+
+  const_block_iterator block_end() const {
+    return block_begin() + getNumOperands();
+  }
+
+  op_range incoming_values() { return operands(); }
+
+  const_op_range incoming_values() const { return operands(); }
+
+  /// \brief Return the number of incoming edges
+  unsigned getNumIncomingValues() const { return getNumOperands(); }
+
+  /// \brief Return incoming value number x
+  MemoryAccess *getIncomingValue(unsigned I) const { return getOperand(I); }
+  void setIncomingValue(unsigned I, MemoryAccess *V) {
+    assert(V && "PHI node got a null value!");
+    assert(getType() == V->getType() &&
+           "All operands to PHI node must be the same type as the PHI node!");
+    setOperand(I, V);
+  }
+  static unsigned getOperandNumForIncomingValue(unsigned I) { return I; }
+  static unsigned getIncomingValueNumForOperand(unsigned I) { return I; }
+
+  /// \brief Return incoming basic block number @p i.
+  BasicBlock *getIncomingBlock(unsigned I) const { return block_begin()[I]; }
+
+  /// \brief Return incoming basic block corresponding
+  /// to an operand of the PHI.
+  BasicBlock *getIncomingBlock(const Use &U) const {
+    assert(this == U.getUser() && "Iterator doesn't point to PHI's Uses?");
+    return getIncomingBlock(unsigned(&U - op_begin()));
+  }
+
+  /// \brief Return incoming basic block corresponding
+  /// to value use iterator.
+  BasicBlock *getIncomingBlock(MemoryAccess::const_user_iterator I) const {
+    return getIncomingBlock(I.getUse());
+  }
+
+  void setIncomingBlock(unsigned I, BasicBlock *BB) {
+    assert(BB && "PHI node got a null basic block!");
+    block_begin()[I] = BB;
+  }
+
+  /// \brief Add an incoming value to the end of the PHI list
+  void addIncoming(MemoryAccess *V, BasicBlock *BB) {
+    if (getNumOperands() == ReservedSpace)
+      growOperands(); // Get more space!
+    // Initialize some new operands.
+    setNumHungOffUseOperands(getNumOperands() + 1);
+    setIncomingValue(getNumOperands() - 1, V);
+    setIncomingBlock(getNumOperands() - 1, BB);
+  }
+
+  /// \brief Return the first index of the specified basic
+  /// block in the value list for this PHI.  Returns -1 if no instance.
+  int getBasicBlockIndex(const BasicBlock *BB) const {
+    for (unsigned I = 0, E = getNumOperands(); I != E; ++I)
+      if (block_begin()[I] == BB)
+        return I;
+    return -1;
+  }
+
+  Value *getIncomingValueForBlock(const BasicBlock *BB) const {
+    int Idx = getBasicBlockIndex(BB);
+    assert(Idx >= 0 && "Invalid basic block argument!");
+    return getIncomingValue(Idx);
+  }
+
+  static inline bool classof(const MemoryPhi *) { return true; }
+  static inline bool classof(const Value *V) {
+    return V->getValueID() == MemoryPhiVal;
+  }
+
+  void print(raw_ostream &OS) const override;
+
+protected:
+  friend class MemorySSA;
+  /// \brief this is more complicated than the generic
+  /// User::allocHungoffUses, because we have to allocate Uses for the incoming
+  /// values and pointers to the incoming blocks, all in one allocation.
+  void allocHungoffUses(unsigned N) {
+    User::allocHungoffUses(N, /* IsPhi */ true);
+  }
+
+  /// For debugging only. This gets used to give memory accesses pretty numbers
+  /// when printing them out
+  virtual unsigned getID() const final { return ID; }
+
+private:
+  // For debugging only
+  const unsigned ID;
+  unsigned ReservedSpace;
+
+  /// \brief This grows the operand list in response to a push_back style of
+  /// operation.  This grows the number of ops by 1.5 times.
+  void growOperands() {
+    unsigned E = getNumOperands();
+    // 2 op PHI nodes are VERY common, so reserve at least enough for that.
+    ReservedSpace = std::max(E + E / 2, 2u);
+    growHungoffUses(ReservedSpace, /* IsPhi */ true);
+  }
+};
+template <> struct OperandTraits<MemoryPhi> : public HungoffOperandTraits<2> {};
+
+DEFINE_TRANSPARENT_OPERAND_ACCESSORS(MemoryPhi, MemoryAccess)
+
+class MemorySSAWalker;
+
+/// \brief Encapsulates MemorySSA, including all data associated with memory
+/// accesses.
+class MemorySSA {
+public:
+  MemorySSA(Function &);
+  ~MemorySSA();
+
+  /// \brief Build Memory SSA, and return the walker we used during building,
+  /// for later reuse. If MemorySSA is already built, just return the walker.
+  MemorySSAWalker *buildMemorySSA(AliasAnalysis *, DominatorTree *);
+
+  /// \brief Returns false if you need to call buildMemorySSA.
+  bool isFinishedBuilding() const { return Walker; }
+
+  /// \brief Given a memory Mod/Ref'ing instruction, get the MemorySSA
+  /// access associated with it. If passed a basic block gets the memory phi
+  /// node that exists for that block, if there is one. Otherwise, this will get
+  /// a MemoryUseOrDef.
+  MemoryAccess *getMemoryAccess(const Value *) const;
+  MemoryPhi *getMemoryAccess(const BasicBlock *BB) const;
+
+  void dump() const;
+  void print(raw_ostream &) const;
+
+  /// \brief Return true if \p MA represents the live on entry value
+  ///
+  /// Loads and stores from pointer arguments and other global values may be
+  /// defined by memory operations that do not occur in the current function, so
+  /// they may be live on entry to the function. MemorySSA represents such
+  /// memory state by the live on entry definition, which is guaranteed to occur
+  /// before any other memory access in the function.
+  inline bool isLiveOnEntryDef(const MemoryAccess *MA) const {
+    return MA == LiveOnEntryDef.get();
+  }
+
+  inline MemoryAccess *getLiveOnEntryDef() const {
+    return LiveOnEntryDef.get();
+  }
+
+  using AccessListType = iplist<MemoryAccess>;
+
+  /// \brief Return the list of MemoryAccess's for a given basic block.
+  ///
+  /// This list is not modifiable by the user.
+  const AccessListType *getBlockAccesses(const BasicBlock *BB) const {
+    auto It = PerBlockAccesses.find(BB);
+    return It == PerBlockAccesses.end() ? nullptr : It->second.get();
+  }
+
+  enum InsertionPlace { Beginning, End };
+
+  /// \brief Given two memory accesses in the same basic block, determine
+  /// whether MemoryAccess \p A dominates MemoryAccess \p B.
+  bool locallyDominates(const MemoryAccess *A, const MemoryAccess *B) const;
+
+protected:
+  // Used by Memory SSA annotater, dumpers, and wrapper pass
+  friend class MemorySSAAnnotatedWriter;
+  friend class MemorySSAPrinterPass;
+  void verifyDefUses(Function &F);
+  void verifyDomination(Function &F);
+
+private:
+  void verifyUseInDefs(MemoryAccess *, MemoryAccess *);
+  using AccessMap =
+      DenseMap<const BasicBlock *, std::unique_ptr<AccessListType>>;
+
+  void
+  determineInsertionPoint(const SmallPtrSetImpl<BasicBlock *> &DefiningBlocks);
+  void computeDomLevels(DenseMap<DomTreeNode *, unsigned> &DomLevels);
+  void markUnreachableAsLiveOnEntry(BasicBlock *BB);
+  bool dominatesUse(const MemoryAccess *, const MemoryAccess *) const;
+  MemoryAccess *createNewAccess(Instruction *, bool ignoreNonMemory = false);
+  MemoryAccess *findDominatingDef(BasicBlock *, enum InsertionPlace);
+
+  MemoryAccess *renameBlock(BasicBlock *, MemoryAccess *);
+  void renamePass(DomTreeNode *, MemoryAccess *IncomingVal,
+                  SmallPtrSet<BasicBlock *, 16> &Visited);
+  AccessListType *getOrCreateAccessList(BasicBlock *);
+  AliasAnalysis *AA;
+  DominatorTree *DT;
+  Function &F;
+
+  // Memory SSA mappings
+  DenseMap<const Value *, MemoryAccess *> InstructionToMemoryAccess;
+  AccessMap PerBlockAccesses;
+  std::unique_ptr<MemoryAccess> LiveOnEntryDef;
+
+  // Memory SSA building info
+  MemorySSAWalker *Walker;
+  unsigned NextID;
+};
+
+// This pass does eager building and then printing of MemorySSA. It is used by
+// the tests to be able to build, dump, and verify Memory SSA.
+class MemorySSAPrinterPass : public FunctionPass {
+public:
+  MemorySSAPrinterPass();
+
+  static char ID;
+  bool doInitialization(Module &M) override;
+  bool runOnFunction(Function &) override;
+  void releaseMemory() override;
+  void getAnalysisUsage(AnalysisUsage &AU) const override;
+  void print(raw_ostream &OS, const Module *M) const override;
+  static void registerOptions();
+  MemorySSA &getMSSA() { return *MSSA; }
+
+private:
+  bool VerifyMemorySSA;
+
+  std::unique_ptr<MemorySSA> MSSA;
+  // FIXME(gbiv): It seems that MemorySSA doesn't own the walker it returns?
+  std::unique_ptr<MemorySSAWalker> Walker;
+  Function *F;
+};
+
+class MemorySSALazy : public FunctionPass {
+public:
+  MemorySSALazy();
+
+  static char ID;
+  bool runOnFunction(Function &) override;
+  void releaseMemory() override;
+  MemorySSA &getMSSA() {
+    assert(MSSA);
+    return *MSSA;
+  }
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesAll();
+  }
+
+private:
+  std::unique_ptr<MemorySSA> MSSA;
+};
+
+/// \brief This is the generic walker interface for walkers of MemorySSA.
+/// Walkers are used to be able to further disambiguate the def-use chains
+/// MemorySSA gives you, or otherwise produce better info than MemorySSA gives
+/// you.
+/// In particular, while the def-use chains provide basic information, and are
+/// guaranteed to give, for example, the nearest may-aliasing MemoryDef for a
+/// MemoryUse as AliasAnalysis considers it, a user mant want better or other
+/// information. In particular, they may want to use SCEV info to further
+/// disambiguate memory accesses, or they may want the nearest dominating
+/// may-aliasing MemoryDef for a call or a store. This API enables a
+/// standardized interface to getting and using that info.
+class MemorySSAWalker {
+public:
+  MemorySSAWalker(MemorySSA *);
+  virtual ~MemorySSAWalker() {}
+
+  using MemoryAccessSet = SmallVector<MemoryAccess *, 8>;
+
+  /// \brief Given a memory Mod/Ref/ModRef'ing instruction, calling this
+  /// will give you the nearest dominating MemoryAccess that Mod's the location
+  /// the instruction accesses (by skipping any def which AA can prove does not
+  /// alias the location(s) accessed by the instruction given).
+  ///
+  /// Note that this will return a single access, and it must dominate the
+  /// Instruction, so if an operand of a MemoryPhi node Mod's the instruction,
+  /// this will return the MemoryPhi, not the operand. This means that
+  /// given:
+  /// if (a) {
+  ///   1 = MemoryDef(liveOnEntry)
+  ///   store %a
+  /// } else {
+  ///   2 = MemoryDef(liveOnEntry)
+  ///    store %b
+  /// }
+  /// 3 = MemoryPhi(2, 1)
+  /// MemoryUse(3)
+  /// load %a
+  ///
+  /// calling this API on load(%a) will return the MemoryPhi, not the MemoryDef
+  /// in the if (a) branch.
+  virtual MemoryAccess *getClobberingMemoryAccess(const Instruction *) = 0;
+
+  /// \brief Given a potentially clobbering memory access and a new location,
+  /// calling this will give you the nearest dominating clobbering MemoryAccess
+  /// (by skipping non-aliasing def links).
+  ///
+  /// This version of the function is mainly used to disambiguate phi translated
+  /// pointers, where the value of a pointer may have changed from the initial
+  /// memory access. Note that this expects to be handed either a MemoryUse,
+  /// or an already potentially clobbering access. Unlike the above API, if
+  /// given a MemoryDef that clobbers the pointer as the starting access, it
+  /// will return that MemoryDef, whereas the above would return the clobber
+  /// starting from the use side of  the memory def.
+  virtual MemoryAccess *getClobberingMemoryAccess(MemoryAccess *,
+                                                  MemoryLocation &) = 0;
+
+protected:
+  MemorySSA *MSSA;
+};
+
+/// \brief A MemorySSAWalker that does no alias queries, or anything else. It
+/// simply returns the links as they were constructed by the builder.
+class DoNothingMemorySSAWalker final : public MemorySSAWalker {
+public:
+  MemoryAccess *getClobberingMemoryAccess(const Instruction *) override;
+  MemoryAccess *getClobberingMemoryAccess(MemoryAccess *,
+                                          MemoryLocation &) override;
+};
+
+using MemoryAccessPair = std::pair<MemoryAccess *, MemoryLocation>;
+using ConstMemoryAccessPair = std::pair<const MemoryAccess *, MemoryLocation>;
+
+/// \brief A MemorySSAWalker that does AA walks and caching of lookups to
+/// disambiguate accesses.
+class CachingMemorySSAWalker final : public MemorySSAWalker {
+public:
+  CachingMemorySSAWalker(MemorySSA *, AliasAnalysis *, DominatorTree *);
+  virtual ~CachingMemorySSAWalker();
+  MemoryAccess *getClobberingMemoryAccess(const Instruction *) override;
+  MemoryAccess *getClobberingMemoryAccess(MemoryAccess *,
+                                          MemoryLocation &) override;
+
+protected:
+  struct UpwardsMemoryQuery;
+  MemoryAccess *doCacheLookup(const MemoryAccess *, const UpwardsMemoryQuery &,
+                              const MemoryLocation &);
+
+  void doCacheInsert(const MemoryAccess *, MemoryAccess *,
+                     const UpwardsMemoryQuery &, const MemoryLocation &);
+
+  void doCacheRemove(const MemoryAccess *, const UpwardsMemoryQuery &,
+                     const MemoryLocation &);
+
+private:
+  MemoryAccessPair UpwardsDFSWalk(MemoryAccess *, const MemoryLocation &,
+                                  UpwardsMemoryQuery &, bool);
+  MemoryAccess *getClobberingMemoryAccess(MemoryAccess *, UpwardsMemoryQuery &);
+  bool instructionClobbersQuery(const MemoryDef *, UpwardsMemoryQuery &,
+                                const MemoryLocation &Loc) const;
+  SmallDenseMap<ConstMemoryAccessPair, MemoryAccess *>
+      CachedUpwardsClobberingAccess;
+  DenseMap<const MemoryAccess *, MemoryAccess *> CachedUpwardsClobberingCall;
+  AliasAnalysis *AA;
+  DominatorTree *DT;
+};
+
+/// \brief Iterator base class used to implement const and non-const iterators
+/// over the defining accesses of a MemoryAccess.
+template <class T>
+class memoryaccess_def_iterator_base
+    : public iterator_facade_base<memoryaccess_def_iterator_base<T>,
+                                  std::forward_iterator_tag, T, ptrdiff_t, T *,
+                                  T *> {
+  using BaseT = typename memoryaccess_def_iterator_base::iterator_facade_base;
+
+public:
+  memoryaccess_def_iterator_base(T *Start) : Access(Start), ArgNo(0) {}
+  memoryaccess_def_iterator_base() : Access(nullptr), ArgNo(0) {}
+  bool operator==(const memoryaccess_def_iterator_base &Other) const {
+    return Access == Other.Access && (!Access || ArgNo == Other.ArgNo);
+  }
+
+  // This is a bit ugly, but for MemoryPHI's, unlike PHINodes, you can't get the
+  // block from the operand in constant time (In a PHINode, the uselist has
+  // both, so it's just subtraction). We provide it as part of the
+  // iterator to avoid callers having to linear walk to get the block.
+  // If the operation becomes constant time on MemoryPHI's, this bit of
+  // abstraction breaking should be removed.
+  BasicBlock *getPhiArgBlock() const {
+    MemoryPhi *MP = dyn_cast<MemoryPhi>(Access);
+    assert(MP && "Tried to get phi arg block when not iterating over a PHI");
+    return MP->getIncomingBlock(ArgNo);
+  }
+  typename BaseT::iterator::pointer operator*() const {
+    assert(Access && "Tried to access past the end of our iterator");
+    // Go to the first argument for phis, and the defining access for everything
+    // else.
+    if (MemoryPhi *MP = dyn_cast<MemoryPhi>(Access))
+      return MP->getIncomingValue(ArgNo);
+    return cast<MemoryUseOrDef>(Access)->getDefiningAccess();
+  }
+  using BaseT::operator++;
+  memoryaccess_def_iterator &operator++() {
+    assert(Access && "Hit end of iterator");
+    if (MemoryPhi *MP = dyn_cast<MemoryPhi>(Access)) {
+      if (++ArgNo >= MP->getNumIncomingValues()) {
+        ArgNo = 0;
+        Access = nullptr;
+      }
+    } else {
+      Access = nullptr;
+    }
+    return *this;
+  }
+
+private:
+  T *Access;
+  unsigned ArgNo;
+};
+
+inline memoryaccess_def_iterator MemoryAccess::defs_begin() {
+  return memoryaccess_def_iterator(this);
+}
+inline const_memoryaccess_def_iterator MemoryAccess::defs_begin() const {
+  return const_memoryaccess_def_iterator(this);
+}
+
+inline memoryaccess_def_iterator MemoryAccess::defs_end() {
+  return memoryaccess_def_iterator();
+}
+
+inline const_memoryaccess_def_iterator MemoryAccess::defs_end() const {
+  return const_memoryaccess_def_iterator();
+}
+
+/// \brief GraphTraits for a MemoryAccess, which walks defs in the normal case,
+/// and uses in the inverse case.
+template <> struct GraphTraits<MemoryAccess *> {
+  using NodeType = MemoryAccess;
+  using ChildIteratorType = memoryaccess_def_iterator;
+
+  static NodeType *getEntryNode(NodeType *N) { return N; }
+  static inline ChildIteratorType child_begin(NodeType *N) {
+    return N->defs_begin();
+  }
+  static inline ChildIteratorType child_end(NodeType *N) {
+    return N->defs_end();
+  }
+};
+
+template <> struct GraphTraits<Inverse<MemoryAccess *>> {
+  using NodeType = MemoryAccess;
+  using ChildIteratorType = MemoryAccess::iterator;
+
+  static NodeType *getEntryNode(NodeType *N) { return N; }
+  static inline ChildIteratorType child_begin(NodeType *N) {
+    return N->user_begin();
+  }
+  static inline ChildIteratorType child_end(NodeType *N) {
+    return N->user_end();
+  }
+};
+
+/// \brief Provide an iterator that walks defs, giving both the memory access,
+/// and the current pointer location, updating the pointer location as it
+/// changes due to phi node translation.
+///
+/// This iterator, while somewhat specialized, is what most clients actually
+/// want when walking upwards through MemorySSA def chains. It takes a pair of
+/// <MemoryAccess,MemoryLocation>, and walks defs, properly translating the
+/// memory location through phi nodes for the user.
+class upward_defs_iterator
+    : public iterator_facade_base<upward_defs_iterator,
+                                  std::forward_iterator_tag,
+                                  const MemoryAccessPair> {
+  using BaseT = upward_defs_iterator::iterator_facade_base;
+
+public:
+  upward_defs_iterator(const MemoryAccessPair &Info)
+      : DefIterator(Info.first), Location(Info.second),
+        OriginalAccess(Info.first) {
+    CurrentPair.first = nullptr;
+
+    WalkingPhi = Info.first && isa<MemoryPhi>(Info.first);
+    fillInCurrentPair();
+  }
+
+  upward_defs_iterator()
+      : DefIterator(), Location(), OriginalAccess(), WalkingPhi(false) {
+    CurrentPair.first = nullptr;
+  }
+
+  bool operator==(const upward_defs_iterator &Other) const {
+    return DefIterator == Other.DefIterator;
+  }
+
+  typename BaseT::iterator::reference operator*() const {
+    assert(DefIterator != OriginalAccess->defs_end() &&
+           "Tried to access past the end of our iterator");
+    return CurrentPair;
+  }
+
+  using BaseT::operator++;
+  upward_defs_iterator &operator++() {
+    assert(DefIterator != OriginalAccess->defs_end() &&
+           "Tried to access past the end of the iterator");
+    ++DefIterator;
+    if (DefIterator != OriginalAccess->defs_end())
+      fillInCurrentPair();
+    return *this;
+  }
+
+  BasicBlock *getPhiArgBlock() const { return DefIterator.getPhiArgBlock(); }
+
+private:
+  void fillInCurrentPair() {
+    CurrentPair.first = *DefIterator;
+    if (WalkingPhi && Location.Ptr) {
+      PHITransAddr Translator(
+          const_cast<Value *>(Location.Ptr),
+          OriginalAccess->getBlock()->getModule()->getDataLayout(), nullptr);
+      if (!Translator.PHITranslateValue(OriginalAccess->getBlock(),
+                                        DefIterator.getPhiArgBlock(), nullptr,
+                                        false))
+        if (Translator.getAddr() != Location.Ptr) {
+          CurrentPair.second = Location.getWithNewPtr(Translator.getAddr());
+          return;
+        }
+    }
+    CurrentPair.second = Location;
+  }
+
+  MemoryAccessPair CurrentPair;
+  memoryaccess_def_iterator DefIterator;
+  MemoryLocation Location;
+  MemoryAccess *OriginalAccess;
+  bool WalkingPhi;
+};
+
+inline upward_defs_iterator upward_defs_begin(const MemoryAccessPair &Pair) {
+  return upward_defs_iterator(Pair);
+}
+inline upward_defs_iterator upward_defs_end() { return upward_defs_iterator(); }
+}
+
+#endif

llvm/trunk/lib/IR/AsmWriter.cpp

while (hasMore()) {
     UseListOrders.pop_back();
   }
 }
 
 //===----------------------------------------------------------------------===//
 //                       External Interface declarations
 //===----------------------------------------------------------------------===//
 
+void Function::print(raw_ostream &ROS, AssemblyAnnotationWriter *AAW,
+                     bool ShouldPreserveUseListOrder,
+                     bool IsForDebug) const {
+  SlotTracker SlotTable(this->getParent());
+  formatted_raw_ostream OS(ROS);
+  AssemblyWriter W(OS, SlotTable, this->getParent(), AAW,
+                   IsForDebug,
+                   ShouldPreserveUseListOrder);
+  W.printFunction(this);
+}
+
 void Module::print(raw_ostream &ROS, AssemblyAnnotationWriter *AAW,
                    bool ShouldPreserveUseListOrder, bool IsForDebug) const {
   SlotTracker SlotTable(this);
   formatted_raw_ostream OS(ROS);
   AssemblyWriter W(OS, SlotTable, this, AAW, IsForDebug,
                    ShouldPreserveUseListOrder);
   W.printModule(this);
 }

llvm/trunk/lib/Transforms/Utils/CMakeLists.txt

add_llvm_library(LLVMTransformUtils
   LoopSimplify.cpp
   LoopUnroll.cpp
   LoopUnrollRuntime.cpp
   LoopUtils.cpp
   LoopVersioning.cpp
   LowerInvoke.cpp
   LowerSwitch.cpp
   Mem2Reg.cpp
+  MemorySSA.cpp
   MetaRenamer.cpp
   ModuleUtils.cpp
   PromoteMemoryToRegister.cpp
   SSAUpdater.cpp
   SanitizerStats.cpp
   SimplifyCFG.cpp
   SimplifyIndVar.cpp
   SimplifyInstructions.cpp

llvm/trunk/lib/Transforms/Utils/MemorySSA.cpp

+//===-- MemorySSA.cpp - Memory SSA Builder---------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------===//
+//
+// This file implements the MemorySSA class.
+//
+//===----------------------------------------------------------------===//
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/DepthFirstIterator.h"
+#include "llvm/ADT/GraphTraits.h"
+#include "llvm/ADT/PostOrderIterator.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallPtrSet.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/Analysis/AliasAnalysis.h"
+#include "llvm/Analysis/CFG.h"
+#include "llvm/Analysis/GlobalsModRef.h"
+#include "llvm/Analysis/IteratedDominanceFrontier.h"
+#include "llvm/Analysis/MemoryLocation.h"
+#include "llvm/Analysis/PHITransAddr.h"
+#include "llvm/IR/AssemblyAnnotationWriter.h"
+#include "llvm/IR/DataLayout.h"
+#include "llvm/IR/Dominators.h"
+#include "llvm/IR/GlobalVariable.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/LLVMContext.h"
+#include "llvm/IR/Metadata.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PatternMatch.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/FormattedStream.h"
+#include "llvm/Transforms/Scalar.h"
+#include "llvm/Transforms/Utils/MemorySSA.h"
+#include <algorithm>
+
+#define DEBUG_TYPE "memoryssa"
+using namespace llvm;
+STATISTIC(NumClobberCacheLookups, "Number of Memory SSA version cache lookups");
+STATISTIC(NumClobberCacheHits, "Number of Memory SSA version cache hits");
+STATISTIC(NumClobberCacheInserts, "Number of MemorySSA version cache inserts");
+INITIALIZE_PASS_WITH_OPTIONS_BEGIN(MemorySSAPrinterPass, "print-memoryssa",
+                                   "Memory SSA", true, true)
+INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(GlobalsAAWrapperPass)
+INITIALIZE_PASS_END(MemorySSAPrinterPass, "print-memoryssa", "Memory SSA", true,
+                    true)
+INITIALIZE_PASS(MemorySSALazy, "memoryssalazy", "Memory SSA", true, true)
+
+namespace llvm {
+
+/// \brief An assembly annotator class to print Memory SSA information in
+/// comments.
+class MemorySSAAnnotatedWriter : public AssemblyAnnotationWriter {
+  friend class MemorySSA;
+  const MemorySSA *MSSA;
+
+public:
+  MemorySSAAnnotatedWriter(const MemorySSA *M) : MSSA(M) {}
+
+  virtual void emitBasicBlockStartAnnot(const BasicBlock *BB,
+                                        formatted_raw_ostream &OS) {
+    if (MemoryAccess *MA = MSSA->getMemoryAccess(BB))
+      OS << "; " << *MA << "\n";
+  }
+
+  virtual void emitInstructionAnnot(const Instruction *I,
+                                    formatted_raw_ostream &OS) {
+    if (MemoryAccess *MA = MSSA->getMemoryAccess(I))
+      OS << "; " << *MA << "\n";
+  }
+};
+}
+
+namespace {
+struct RenamePassData {
+  DomTreeNode *DTN;
+  DomTreeNode::const_iterator ChildIt;
+  MemoryAccess *IncomingVal;
+
+  RenamePassData(DomTreeNode *D, DomTreeNode::const_iterator It,
+                 MemoryAccess *M)
+      : DTN(D), ChildIt(It), IncomingVal(M) {}
+  void swap(RenamePassData &RHS) {
+    std::swap(DTN, RHS.DTN);
+    std::swap(ChildIt, RHS.ChildIt);
+    std::swap(IncomingVal, RHS.IncomingVal);
+  }
+};
+}
+
+namespace llvm {
+/// \brief Rename a single basic block into MemorySSA form.
+/// Uses the standard SSA renaming algorithm.
+/// \returns The new incoming value.
+MemoryAccess *MemorySSA::renameBlock(BasicBlock *BB,
+                                     MemoryAccess *IncomingVal) {
+  auto It = PerBlockAccesses.find(BB);
+  // Skip most processing if the list is empty.
+  if (It != PerBlockAccesses.end()) {
+    AccessListType *Accesses = It->second.get();
+    for (MemoryAccess &L : *Accesses) {
+      switch (L.getValueID()) {
+      case Value::MemoryUseVal:
+        cast<MemoryUse>(&L)->setDefiningAccess(IncomingVal);
+        break;
+      case Value::MemoryDefVal:
+        // We can't legally optimize defs, because we only allow single
+        // memory phis/uses on operations, and if we optimize these, we can
+        // end up with multiple reaching defs. Uses do not have this
+        // problem, since they do not produce a value
+        cast<MemoryDef>(&L)->setDefiningAccess(IncomingVal);
+        IncomingVal = &L;
+        break;
+      case Value::MemoryPhiVal:
+        IncomingVal = &L;
+        break;
+      }
+    }
+  }
+
+  // Pass through values to our successors
+  for (const BasicBlock *S : successors(BB)) {
+    auto It = PerBlockAccesses.find(S);
+    // Rename the phi nodes in our successor block
+    if (It == PerBlockAccesses.end() || !isa<MemoryPhi>(It->second->front()))
+      continue;
+    AccessListType *Accesses = It->second.get();
+    auto *Phi = cast<MemoryPhi>(&Accesses->front());
+    assert(std::find(succ_begin(BB), succ_end(BB), S) != succ_end(BB) &&
+           "Must be at least one edge from Succ to BB!");
+    Phi->addIncoming(IncomingVal, BB);
+  }
+
+  return IncomingVal;
+}
+
+/// \brief This is the standard SSA renaming algorithm.
+///
+/// We walk the dominator tree in preorder, renaming accesses, and then filling
+/// in phi nodes in our successors.
+void MemorySSA::renamePass(DomTreeNode *Root, MemoryAccess *IncomingVal,
+                           SmallPtrSet<BasicBlock *, 16> &Visited) {
+  SmallVector<RenamePassData, 32> WorkStack;
+  IncomingVal = renameBlock(Root->getBlock(), IncomingVal);
+  WorkStack.push_back({Root, Root->begin(), IncomingVal});
+  Visited.insert(Root->getBlock());
+
+  while (!WorkStack.empty()) {
+    DomTreeNode *Node = WorkStack.back().DTN;
+    DomTreeNode::const_iterator ChildIt = WorkStack.back().ChildIt;
+    IncomingVal = WorkStack.back().IncomingVal;
+
+    if (ChildIt == Node->end()) {
+      WorkStack.pop_back();
+    } else {
+      DomTreeNode *Child = *ChildIt;
+      ++WorkStack.back().ChildIt;
+      BasicBlock *BB = Child->getBlock();
+      Visited.insert(BB);
+      IncomingVal = renameBlock(BB, IncomingVal);
+      WorkStack.push_back({Child, Child->begin(), IncomingVal});
+    }
+  }
+}
+
+/// \brief Compute dominator levels, used by the phi insertion algorithm above.
+void MemorySSA::computeDomLevels(DenseMap<DomTreeNode *, unsigned> &DomLevels) {
+  for (auto DFI = df_begin(DT->getRootNode()), DFE = df_end(DT->getRootNode());
+       DFI != DFE; ++DFI)
+    DomLevels[*DFI] = DFI.getPathLength() - 1;
+}
+
+/// \brief This handles unreachable block acccesses by deleting phi nodes in
+/// unreachable blocks, and marking all other unreachable MemoryAccess's as
+/// being uses of the live on entry definition.
+void MemorySSA::markUnreachableAsLiveOnEntry(BasicBlock *BB) {
+  assert(!DT->isReachableFromEntry(BB) &&
+         "Reachable block found while handling unreachable blocks");
+
+  auto It = PerBlockAccesses.find(BB);
+  if (It == PerBlockAccesses.end())
+    return;
+
+  auto &Accesses = It->second;
+  for (auto AI = Accesses->begin(), AE = Accesses->end(); AI != AE;) {
+    auto Next = std::next(AI);
+    // If we have a phi, just remove it. We are going to replace all
+    // users with live on entry.
+    if (auto *UseOrDef = dyn_cast<MemoryUseOrDef>(AI))
+      UseOrDef->setDefiningAccess(LiveOnEntryDef.get());
+    else
+      Accesses->erase(AI);
+    AI = Next;
+  }
+}
+
+MemorySSA::MemorySSA(Function &Func)
+    : AA(nullptr), DT(nullptr), F(Func), LiveOnEntryDef(nullptr),
+      Walker(nullptr), NextID(0) {}
+
+MemorySSA::~MemorySSA() {
+  // Drop all our references
+  for (const auto &Pair : PerBlockAccesses)
+    for (MemoryAccess &MA : *Pair.second)
+      MA.dropAllReferences();
+}
+
+MemorySSA::AccessListType *MemorySSA::getOrCreateAccessList(BasicBlock *BB) {
+  auto Res = PerBlockAccesses.insert(std::make_pair(BB, nullptr));
+
+  if (Res.second)
+    Res.first->second = make_unique<AccessListType>();
+  return Res.first->second.get();
+}
+
+MemorySSAWalker *MemorySSA::buildMemorySSA(AliasAnalysis *AA,
+                                           DominatorTree *DT) {
+  if (Walker)
+    return Walker;
+
+  assert(!this->AA && !this->DT &&
+         "MemorySSA without a walker already has AA or DT?");
+
+  auto *Result = new CachingMemorySSAWalker(this, AA, DT);
+  this->AA = AA;
+  this->DT = DT;
+
+  // We create an access to represent "live on entry", for things like
+  // arguments or users of globals, where the memory they use is defined before
+  // the beginning of the function. We do not actually insert it into the IR.
+  // We do not define a live on exit for the immediate uses, and thus our
+  // semantics do *not* imply that something with no immediate uses can simply
+  // be removed.
+  BasicBlock &StartingPoint = F.getEntryBlock();
+  LiveOnEntryDef = make_unique<MemoryDef>(F.getContext(), nullptr, nullptr,
+                                          &StartingPoint, NextID++);
+
+  // We maintain lists of memory accesses per-block, trading memory for time. We
+  // could just look up the memory access for every possible instruction in the
+  // stream.
+  SmallPtrSet<BasicBlock *, 32> DefiningBlocks;
+
+  // Go through each block, figure out where defs occur, and chain together all
+  // the accesses.
+  for (BasicBlock &B : F) {
+    AccessListType *Accesses = nullptr;
+    for (Instruction &I : B) {
+      MemoryAccess *MA = createNewAccess(&I, true);
+      if (!MA)
+        continue;
+      if (isa<MemoryDef>(MA))
+        DefiningBlocks.insert(&B);
+      if (!Accesses)
+        Accesses = getOrCreateAccessList(&B);
+      Accesses->push_back(MA);
+    }
+  }
+
+  // Determine where our MemoryPhi's should go
+  IDFCalculator IDFs(*DT);
+  IDFs.setDefiningBlocks(DefiningBlocks);
+  SmallVector<BasicBlock *, 32> IDFBlocks;
+  IDFs.calculate(IDFBlocks);
+
+  // Now place MemoryPhi nodes.
+  for (auto &BB : IDFBlocks) {
+    // Insert phi node
+    AccessListType *Accesses = getOrCreateAccessList(BB);
+    MemoryPhi *Phi = new MemoryPhi(F.getContext(), BB, NextID++);
+    InstructionToMemoryAccess.insert(std::make_pair(BB, Phi));
+    // Phi's always are placed at the front of the block.
+    Accesses->push_front(Phi);
+  }
+
+  // Now do regular SSA renaming on the MemoryDef/MemoryUse. Visited will get
+  // filled in with all blocks.
+  SmallPtrSet<BasicBlock *, 16> Visited;
+  renamePass(DT->getRootNode(), LiveOnEntryDef.get(), Visited);
+
+  // Now optimize the MemoryUse's defining access to point to the nearest
+  // dominating clobbering def.
+  // This ensures that MemoryUse's that are killed by the same store are
+  // immediate users of that store, one of the invariants we guarantee.
+  for (auto DomNode : depth_first(DT)) {
+    BasicBlock *BB = DomNode->getBlock();
+    auto AI = PerBlockAccesses.find(BB);
+    if (AI == PerBlockAccesses.end())
+      continue;
+    AccessListType *Accesses = AI->second.get();
+    for (auto &MA : *Accesses) {
+      if (auto *MU = dyn_cast<MemoryUse>(&MA)) {
+        Instruction *Inst = MU->getMemoryInst();
+        MU->setDefiningAccess(Result->getClobberingMemoryAccess(Inst));
+      }
+    }
+  }
+
+  // Mark the uses in unreachable blocks as live on entry, so that they go
+  // somewhere.
+  for (auto &BB : F)
+    if (!Visited.count(&BB))
+      markUnreachableAsLiveOnEntry(&BB);
+
+  Walker = Result;
+  return Walker;
+}
+
+/// \brief Helper function to create new memory accesses
+MemoryAccess *MemorySSA::createNewAccess(Instruction *I, bool IgnoreNonMemory) {
+  // Find out what affect this instruction has on memory.
+  ModRefInfo ModRef = AA->getModRefInfo(I);
+  bool Def = bool(ModRef & MRI_Mod);
+  bool Use = bool(ModRef & MRI_Ref);
+
+  // It's possible for an instruction to not modify memory at all. During
+  // construction, we ignore them.
+  if (IgnoreNonMemory && !Def && !Use)
+    return nullptr;
+
+  assert((Def || Use) &&
+         "Trying to create a memory access with a non-memory instruction");
+
+  MemoryUseOrDef *MA;
+  if (Def)
+    MA = new MemoryDef(I->getModule()->getContext(), nullptr, I, I->getParent(),
+                       NextID++);
+  else
+    MA =
+        new MemoryUse(I->getModule()->getContext(), nullptr, I, I->getParent());
+  InstructionToMemoryAccess.insert(std::make_pair(I, MA));
+  return MA;
+}
+
+MemoryAccess *MemorySSA::findDominatingDef(BasicBlock *UseBlock,
+                                           enum InsertionPlace Where) {
+  // Handle the initial case
+  if (Where == Beginning)
+    // The only thing that could define us at the beginning is a phi node
+    if (MemoryPhi *Phi = getMemoryAccess(UseBlock))
+      return Phi;
+
+  DomTreeNode *CurrNode = DT->getNode(UseBlock);
+  // Need to be defined by our dominator
+  if (Where == Beginning)
+    CurrNode = CurrNode->getIDom();
+  Where = End;
+  while (CurrNode) {
+    auto It = PerBlockAccesses.find(CurrNode->getBlock());
+    if (It != PerBlockAccesses.end()) {
+      auto &Accesses = It->second;
+      for (auto RAI = Accesses->rbegin(), RAE = Accesses->rend(); RAI != RAE;
+           ++RAI) {
+        if (isa<MemoryDef>(*RAI) || isa<MemoryPhi>(*RAI))
+          return &*RAI;
+      }
+    }
+    CurrNode = CurrNode->getIDom();
+  }
+  return LiveOnEntryDef.get();
+}
+
+/// \brief Returns true if \p Replacer dominates \p Replacee .
+bool MemorySSA::dominatesUse(const MemoryAccess *Replacer,
+                             const MemoryAccess *Replacee) const {
+  if (isa<MemoryUseOrDef>(Replacee))
+    return DT->dominates(Replacer->getBlock(), Replacee->getBlock());
+  const auto *MP = cast<MemoryPhi>(Replacee);
+  // For a phi node, the use occurs in the predecessor block of the phi node.
+  // Since we may occur multiple times in the phi node, we have to check each
+  // operand to ensure Replacer dominates each operand where Replacee occurs.
+  for (const Use &Arg : MP->operands()) {
+    if (Arg != Replacee &&
+        !DT->dominates(Replacer->getBlock(), MP->getIncomingBlock(Arg)))
+      return false;
+  }
+  return true;
+}
+
+void MemorySSA::print(raw_ostream &OS) const {
+  MemorySSAAnnotatedWriter Writer(this);
+  F.print(OS, &Writer);
+}
+
+void MemorySSA::dump() const {
+  MemorySSAAnnotatedWriter Writer(this);
+  F.print(dbgs(), &Writer);
+}
+
+/// \brief Verify the domination properties of MemorySSA by checking that each
+/// definition dominates all of its uses.
+void MemorySSA::verifyDomination(Function &F) {
+  for (BasicBlock &B : F) {
+    // Phi nodes are attached to basic blocks
+    if (MemoryPhi *MP = getMemoryAccess(&B)) {
+      for (User *U : MP->users()) {
+        BasicBlock *UseBlock;
+        // Phi operands are used on edges, we simulate the right domination by
+        // acting as if the use occurred at the end of the predecessor block.
+        if (MemoryPhi *P = dyn_cast<MemoryPhi>(U)) {
+          for (const auto &Arg : P->operands()) {
+            if (Arg == MP) {
+              UseBlock = P->getIncomingBlock(Arg);
+              break;
+            }
+          }
+        } else {
+          UseBlock = cast<MemoryAccess>(U)->getBlock();
+        }
+        assert(DT->dominates(MP->getBlock(), UseBlock) &&
+               "Memory PHI does not dominate it's uses");
+      }
+    }
+
+    for (Instruction &I : B) {
+      MemoryAccess *MD = dyn_cast_or_null<MemoryDef>(getMemoryAccess(&I));
+      if (!MD)
+        continue;
+
+      for (const auto &U : MD->users()) {
+        BasicBlock *UseBlock;
+        // Things are allowed to flow to phi nodes over their predecessor edge.
+        if (auto *P = dyn_cast<MemoryPhi>(U)) {
+          for (const auto &Arg : P->operands()) {
+            if (Arg == MD) {
+              UseBlock = P->getIncomingBlock(Arg);
+              break;
+            }
+          }
+        } else {
+          UseBlock = cast<MemoryAccess>(U)->getBlock();
+        }
+        assert(DT->dominates(MD->getBlock(), UseBlock) &&
+               "Memory Def does not dominate it's uses");
+      }
+    }
+  }
+}
+
+/// \brief Verify the def-use lists in MemorySSA, by verifying that \p Use
+/// appears in the use list of \p Def.
+///
+/// llvm_unreachable is used instead of asserts because this may be called in
+/// a build without asserts. In that case, we don't want this to turn into a
+/// nop.
+void MemorySSA::verifyUseInDefs(MemoryAccess *Def, MemoryAccess *Use) {
+  // The live on entry use may cause us to get a NULL def here
+  if (!Def) {
+    if (!isLiveOnEntryDef(Use))
+      llvm_unreachable("Null def but use not point to live on entry def");
+  } else if (std::find(Def->user_begin(), Def->user_end(), Use) ==
+             Def->user_end()) {
+    llvm_unreachable("Did not find use in def's use list");
+  }
+}
+
+/// \brief Verify the immediate use information, by walking all the memory
+/// accesses and verifying that, for each use, it appears in the
+/// appropriate def's use list
+void MemorySSA::verifyDefUses(Function &F) {
+  for (BasicBlock &B : F) {
+    // Phi nodes are attached to basic blocks
+    if (MemoryPhi *Phi = getMemoryAccess(&B))
+      for (unsigned I = 0, E = Phi->getNumIncomingValues(); I != E; ++I)
+        verifyUseInDefs(Phi->getIncomingValue(I), Phi);
+
+    for (Instruction &I : B) {
+      if (MemoryAccess *MA = getMemoryAccess(&I)) {
+        assert(isa<MemoryUseOrDef>(MA) &&
+               "Found a phi node not attached to a bb");
+        verifyUseInDefs(cast<MemoryUseOrDef>(MA)->getDefiningAccess(), MA);
+      }
+    }
+  }
+}
+
+MemoryAccess *MemorySSA::getMemoryAccess(const Value *I) const {
+  return InstructionToMemoryAccess.lookup(I);
+}
+
+MemoryPhi *MemorySSA::getMemoryAccess(const BasicBlock *BB) const {
+  return cast_or_null<MemoryPhi>(getMemoryAccess((const Value *)BB));
+}
+
+/// \brief Determine, for two memory accesses in the same block,
+/// whether \p Dominator dominates \p Dominatee.
+/// \returns True if \p Dominator dominates \p Dominatee.
+bool MemorySSA::locallyDominates(const MemoryAccess *Dominator,
+                                 const MemoryAccess *Dominatee) const {
+
+  assert((Dominator->getBlock() == Dominatee->getBlock()) &&
+         "Asking for local domination when accesses are in different blocks!");
+  // Get the access list for the block
+  const AccessListType *AccessList = getBlockAccesses(Dominator->getBlock());
+  AccessListType::const_reverse_iterator It(Dominator->getIterator());
+
+  // If we hit the beginning of the access list before we hit dominatee, we must
+  // dominate it
+  return std::none_of(It, AccessList->rend(),
+                      [&](const MemoryAccess &MA) { return &MA == Dominatee; });
+}
+
+const static char LiveOnEntryStr[] = "liveOnEntry";
+
+void MemoryDef::print(raw_ostream &OS) const {
+  MemoryAccess *UO = getDefiningAccess();
+
+  OS << getID() << " = MemoryDef(";
+  if (UO && UO->getID())
+    OS << UO->getID();
+  else
+    OS << LiveOnEntryStr;
+  OS << ')';
+}
+
+void MemoryPhi::print(raw_ostream &OS) const {
+  bool First = true;
+  OS << getID() << " = MemoryPhi(";
+  for (const auto &Op : operands()) {
+    BasicBlock *BB = getIncomingBlock(Op);
+    MemoryAccess *MA = cast<MemoryAccess>(Op);
+    if (!First)
+      OS << ',';
+    else
+      First = false;
+
+    OS << '{';
+    if (BB->hasName())
+      OS << BB->getName();
+    else
+      BB->printAsOperand(OS, false);
+    OS << ',';
+    if (unsigned ID = MA->getID())
+      OS << ID;
+    else
+      OS << LiveOnEntryStr;
+    OS << '}';
+  }
+  OS << ')';
+}
+
+MemoryAccess::~MemoryAccess() {}
+
+void MemoryUse::print(raw_ostream &OS) const {
+  MemoryAccess *UO = getDefiningAccess();
+  OS << "MemoryUse(";
+  if (UO && UO->getID())
+    OS << UO->getID();
+  else
+    OS << LiveOnEntryStr;
+  OS << ')';
+}
+
+void MemoryAccess::dump() const {
+  print(dbgs());
+  dbgs() << "\n";
+}
+
+char MemorySSAPrinterPass::ID = 0;
+
+MemorySSAPrinterPass::MemorySSAPrinterPass() : FunctionPass(ID) {
+  initializeMemorySSAPrinterPassPass(*PassRegistry::getPassRegistry());
+}
+
+void MemorySSAPrinterPass::releaseMemory() {
+  // Subtlety: Be sure to delete the walker before MSSA, because the walker's
+  // dtor may try to access MemorySSA.
+  Walker.reset();
+  MSSA.reset();
+}
+
+void MemorySSAPrinterPass::getAnalysisUsage(AnalysisUsage &AU) const {
+  AU.setPreservesAll();
+  AU.addRequired<AAResultsWrapperPass>();
+  AU.addRequired<DominatorTreeWrapperPass>();
+  AU.addPreserved<DominatorTreeWrapperPass>();
+  AU.addPreserved<GlobalsAAWrapperPass>();
+}
+
+bool MemorySSAPrinterPass::doInitialization(Module &M) {
+  VerifyMemorySSA =
+      M.getContext()
+          .template getOption<bool, MemorySSAPrinterPass,
+                              &MemorySSAPrinterPass::VerifyMemorySSA>();
+  return false;
+}
+
+void MemorySSAPrinterPass::registerOptions() {
+  OptionRegistry::registerOption<bool, MemorySSAPrinterPass,
+                                 &MemorySSAPrinterPass::VerifyMemorySSA>(
+      "verify-memoryssa", "Run the Memory SSA verifier", false);
+}
+
+void MemorySSAPrinterPass::print(raw_ostream &OS, const Module *M) const {
+  MSSA->print(OS);
+}
+
+bool MemorySSAPrinterPass::runOnFunction(Function &F) {
+  this->F = &F;
+  MSSA.reset(new MemorySSA(F));
+  AliasAnalysis *AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
+  DominatorTree *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
+  Walker.reset(MSSA->buildMemorySSA(AA, DT));
+
+  if (VerifyMemorySSA) {
+    MSSA->verifyDefUses(F);
+    MSSA->verifyDomination(F);
+  }
+
+  return false;
+}
+
+char MemorySSALazy::ID = 0;
+
+MemorySSALazy::MemorySSALazy() : FunctionPass(ID) {
+  initializeMemorySSALazyPass(*PassRegistry::getPassRegistry());
+}
+
+void MemorySSALazy::releaseMemory() { MSSA.reset(); }
+
+bool MemorySSALazy::runOnFunction(Function &F) {
+  MSSA.reset(new MemorySSA(F));
+  return false;
+}
+
+MemorySSAWalker::MemorySSAWalker(MemorySSA *M) : MSSA(M) {}
+
+CachingMemorySSAWalker::CachingMemorySSAWalker(MemorySSA *M, AliasAnalysis *A,
+                                               DominatorTree *D)
+    : MemorySSAWalker(M), AA(A), DT(D) {}
+
+CachingMemorySSAWalker::~CachingMemorySSAWalker() {}
+
+struct CachingMemorySSAWalker::UpwardsMemoryQuery {
+  // True if we saw a phi whose predecessor was a backedge
+  bool SawBackedgePhi;
+  // True if our original query started off as a call
+  bool IsCall;
+  // The pointer location we started the query with. This will be empty if
+  // IsCall is true.
+  MemoryLocation StartingLoc;
+  // This is the instruction we were querying about.
+  const Instruction *Inst;
+  // Set of visited Instructions for this query.
+  DenseSet<MemoryAccessPair> Visited;
+  // Set of visited call accesses for this query. This is separated out because
+  // you can always cache and lookup the result of call queries (IE when IsCall
+  // == true) for every call in the chain. The calls have no AA location
+  // associated with them with them, and thus, no context dependence.
+  SmallPtrSet<const MemoryAccess *, 32> VisitedCalls;
+  // The MemoryAccess we actually got called with, used to test local domination
+  const MemoryAccess *OriginalAccess;
+  // The Datalayout for the module we started in
+  const DataLayout *DL;
+
+  UpwardsMemoryQuery()
+      : SawBackedgePhi(false), IsCall(false), Inst(nullptr),
+        OriginalAccess(nullptr), DL(nullptr) {}
+};
+
+void CachingMemorySSAWalker::doCacheRemove(const MemoryAccess *M,
+                                           const UpwardsMemoryQuery &Q,
+                                           const MemoryLocation &Loc) {
+  if (Q.IsCall)
+    CachedUpwardsClobberingCall.erase(M);
+  else
+    CachedUpwardsClobberingAccess.erase({M, Loc});
+}
+
+void CachingMemorySSAWalker::doCacheInsert(const MemoryAccess *M,
+                                           MemoryAccess *Result,
+                                           const UpwardsMemoryQuery &Q,
+                                           const MemoryLocation &Loc) {
+  ++NumClobberCacheInserts;
+  if (Q.IsCall)
+    CachedUpwardsClobberingCall[M] = Result;
+  else
+    CachedUpwardsClobberingAccess[{M, Loc}] = Result;
+}
+
+MemoryAccess *CachingMemorySSAWalker::doCacheLookup(const MemoryAccess *M,
+                                                    const UpwardsMemoryQuery &Q,
+                                                    const MemoryLocation &Loc) {
+  ++NumClobberCacheLookups;
+  MemoryAccess *Result = nullptr;
+
+  if (Q.IsCall)
+    Result = CachedUpwardsClobberingCall.lookup(M);
+  else
+    Result = CachedUpwardsClobberingAccess.lookup({M, Loc});
+
+  if (Result)
+    ++NumClobberCacheHits;
+  return Result;
+}
+
+bool CachingMemorySSAWalker::instructionClobbersQuery(
+    const MemoryDef *MD, UpwardsMemoryQuery &Q,
+    const MemoryLocation &Loc) const {
+  Instruction *DefMemoryInst = MD->getMemoryInst();
+  assert(DefMemoryInst && "Defining instruction not actually an instruction");
+
+  if (!Q.IsCall)
+    return AA->getModRefInfo(DefMemoryInst, Loc) & MRI_Mod;
+
+  // If this is a call, mark it for caching
+  if (ImmutableCallSite(DefMemoryInst))
+    Q.VisitedCalls.insert(MD);
+  ModRefInfo I = AA->getModRefInfo(DefMemoryInst, ImmutableCallSite(Q.Inst));
+  return I != MRI_NoModRef;
+}
+
+MemoryAccessPair CachingMemorySSAWalker::UpwardsDFSWalk(
+    MemoryAccess *StartingAccess, const MemoryLocation &Loc,
+    UpwardsMemoryQuery &Q, bool FollowingBackedge) {
+  MemoryAccess *ModifyingAccess = nullptr;
+
+  auto DFI = df_begin(StartingAccess);
+  for (auto DFE = df_end(StartingAccess); DFI != DFE;) {
+    MemoryAccess *CurrAccess = *DFI;
+    if (MSSA->isLiveOnEntryDef(CurrAccess))
+      return {CurrAccess, Loc};
+    if (auto CacheResult = doCacheLookup(CurrAccess, Q, Loc))
+      return {CacheResult, Loc};
+    // If this is a MemoryDef, check whether it clobbers our current query.
+    if (auto *MD = dyn_cast<MemoryDef>(CurrAccess)) {
+      // If we hit the top, stop following this path.
+      // While we can do lookups, we can't sanely do inserts here unless we were
+      // to track everything we saw along the way, since we don't know where we
+      // will stop.
+      if (instructionClobbersQuery(MD, Q, Loc)) {
+        ModifyingAccess = CurrAccess;
+        break;
+      }
+    }
+
+    // We need to know whether it is a phi so we can track backedges.
+    // Otherwise, walk all upward defs.
+    if (!isa<MemoryPhi>(CurrAccess)) {
+      ++DFI;
+      continue;
+    }
+
+    // Recurse on PHI nodes, since we need to change locations.
+    // TODO: Allow graphtraits on pairs, which would turn this whole function
+    // into a normal single depth first walk.
+    MemoryAccess *FirstDef = nullptr;
+    DFI = DFI.skipChildren();
+    const MemoryAccessPair PHIPair(CurrAccess, Loc);
+    bool VisitedOnlyOne = true;
+    for (auto MPI = upward_defs_begin(PHIPair), MPE = upward_defs_end();
+         MPI != MPE; ++MPI) {
+      // Don't follow this path again if we've followed it once
+      if (!Q.Visited.insert(*MPI).second)
+        continue;
+
+      bool Backedge =
+          !FollowingBackedge &&
+          DT->dominates(CurrAccess->getBlock(), MPI.getPhiArgBlock());
+
+      MemoryAccessPair CurrentPair =
+          UpwardsDFSWalk(MPI->first, MPI->second, Q, Backedge);
+      // All the phi arguments should reach the same point if we can bypass
+      // this phi. The alternative is that they hit this phi node, which
+      // means we can skip this argument.
+      if (FirstDef && CurrentPair.first != PHIPair.first &&
+          CurrentPair.first != FirstDef) {
+        ModifyingAccess = CurrAccess;
+        break;
+      }
+
+      if (!FirstDef)
+        FirstDef = CurrentPair.first;
+      else
+        VisitedOnlyOne = false;
+    }
+
+    // The above loop determines if all arguments of the phi node reach the
+    // same place. However we skip arguments that are cyclically dependent
+    // only on the value of this phi node. This means in some cases, we may
+    // only visit one argument of the phi node, and the above loop will
+    // happily say that all the arguments are the same. However, in that case,
+    // we still can't walk past the phi node, because that argument still
+    // kills the access unless we hit the top of the function when walking
+    // that argument.
+    if (VisitedOnlyOne && FirstDef && !MSSA->isLiveOnEntryDef(FirstDef))
+      ModifyingAccess = CurrAccess;
+  }
+
+  if (!ModifyingAccess)
+    return {MSSA->getLiveOnEntryDef(), Q.StartingLoc};
+
+  const BasicBlock *OriginalBlock = Q.OriginalAccess->getBlock();
+  unsigned N = DFI.getPathLength();
+  MemoryAccess *FinalAccess = ModifyingAccess;
+  for (; N != 0; --N) {
+    ModifyingAccess = DFI.getPath(N - 1);
+    BasicBlock *CurrBlock = ModifyingAccess->getBlock();
+    if (!FollowingBackedge)
+      doCacheInsert(ModifyingAccess, FinalAccess, Q, Loc);
+    if (DT->dominates(CurrBlock, OriginalBlock) &&
+        (CurrBlock != OriginalBlock || !FollowingBackedge ||
+         MSSA->locallyDominates(ModifyingAccess, Q.OriginalAccess)))
+      break;
+  }
+
+  // Cache everything else on the way back. The caller should cache
+  // Q.OriginalAccess for us.
+  for (; N != 0; --N) {
+    MemoryAccess *CacheAccess = DFI.getPath(N - 1);
+    doCacheInsert(CacheAccess, ModifyingAccess, Q, Loc);
+  }
+  assert(Q.Visited.size() < 1000 && "Visited too much");
+
+  return {ModifyingAccess, Loc};
+}
+
+/// \brief Walk the use-def chains starting at \p MA and find
+/// the MemoryAccess that actually clobbers Loc.
+///
+/// \returns our clobbering memory access
+MemoryAccess *
+CachingMemorySSAWalker::getClobberingMemoryAccess(MemoryAccess *StartingAccess,
+                                                  UpwardsMemoryQuery &Q) {
+  return UpwardsDFSWalk(StartingAccess, Q.StartingLoc, Q, false).first;
+}
+
+MemoryAccess *
+CachingMemorySSAWalker::getClobberingMemoryAccess(MemoryAccess *StartingAccess,
+                                                  MemoryLocation &Loc) {
+  if (isa<MemoryPhi>(StartingAccess))
+    return StartingAccess;
+
+  auto *StartingUseOrDef = cast<MemoryUseOrDef>(StartingAccess);
+  if (MSSA->isLiveOnEntryDef(StartingUseOrDef))
+    return StartingUseOrDef;
+
+  Instruction *I = StartingUseOrDef->getMemoryInst();
+
+  // Conservatively, fences are always clobbers, so don't perform the walk if we
+  // hit a fence.
+  if (isa<FenceInst>(I))
+    return StartingUseOrDef;
+
+  UpwardsMemoryQuery Q;
+  Q.OriginalAccess = StartingUseOrDef;
+  Q.StartingLoc = Loc;
+  Q.Inst = StartingUseOrDef->getMemoryInst();
+  Q.IsCall = false;
+  Q.DL = &Q.Inst->getModule()->getDataLayout();
+
+  if (auto CacheResult = doCacheLookup(StartingUseOrDef, Q, Q.StartingLoc))
+    return CacheResult;
+
+  // Unlike the other function, do not walk to the def of a def, because we are
+  // handed something we already believe is the clobbering access.
+  MemoryAccess *DefiningAccess = isa<MemoryUse>(StartingUseOrDef)
+                                     ? StartingUseOrDef->getDefiningAccess()
+                                     : StartingUseOrDef;
+
+  MemoryAccess *Clobber = getClobberingMemoryAccess(DefiningAccess, Q);
+  doCacheInsert(Q.OriginalAccess, Clobber, Q, Q.StartingLoc);
+  DEBUG(dbgs() << "Starting Memory SSA clobber for " << *I << " is ");
+  DEBUG(dbgs() << *StartingUseOrDef << "\n");
+  DEBUG(dbgs() << "Final Memory SSA clobber for " << *I << " is ");
+  DEBUG(dbgs() << *Clobber << "\n");
+  return Clobber;
+}
+
+MemoryAccess *
+CachingMemorySSAWalker::getClobberingMemoryAccess(const Instruction *I) {
+  // There should be no way to lookup an instruction and get a phi as the
+  // access, since we only map BB's to PHI's. So, this must be a use or def.
+  auto *StartingAccess = cast<MemoryUseOrDef>(MSSA->getMemoryAccess(I));
+
+  // We can't sanely do anything with a FenceInst, they conservatively
+  // clobber all memory, and have no locations to get pointers from to
+  // try to disambiguate
+  if (isa<FenceInst>(I))
+    return StartingAccess;
+
+  UpwardsMemoryQuery Q;
+  Q.OriginalAccess = StartingAccess;
+  Q.IsCall = bool(ImmutableCallSite(I));
+  if (!Q.IsCall)
+    Q.StartingLoc = MemoryLocation::get(I);
+  Q.Inst = I;
+  Q.DL = &Q.Inst->getModule()->getDataLayout();
+  if (auto CacheResult = doCacheLookup(StartingAccess, Q, Q.StartingLoc))
+    return CacheResult;
+
+  // Start with the thing we already think clobbers this location
+  MemoryAccess *DefiningAccess = StartingAccess->getDefiningAccess();
+
+  // At this point, DefiningAccess may be the live on entry def.
+  // If it is, we will not get a better result.
+  if (MSSA->isLiveOnEntryDef(DefiningAccess))
+    return DefiningAccess;
+
+  MemoryAccess *Result = getClobberingMemoryAccess(DefiningAccess, Q);
+  doCacheInsert(Q.OriginalAccess, Result, Q, Q.StartingLoc);
+  // TODO: When this implementation is more mature, we may want to figure out
+  // what this additional caching buys us. It's most likely A Good Thing.
+  if (Q.IsCall)
+    for (const MemoryAccess *MA : Q.VisitedCalls)
+      doCacheInsert(MA, Result, Q, Q.StartingLoc);
+
+  DEBUG(dbgs() << "Starting Memory SSA clobber for " << *I << " is ");
+  DEBUG(dbgs() << *DefiningAccess << "\n");
+  DEBUG(dbgs() << "Final Memory SSA clobber for " << *I << " is ");
+  DEBUG(dbgs() << *Result << "\n");
+
+  return Result;
+}
+
+MemoryAccess *
+DoNothingMemorySSAWalker::getClobberingMemoryAccess(const Instruction *I) {
+  MemoryAccess *MA = MSSA->getMemoryAccess(I);
+  if (auto *Use = dyn_cast<MemoryUseOrDef>(MA))
+    return Use->getDefiningAccess();
+  return MA;
+}
+
+MemoryAccess *DoNothingMemorySSAWalker::getClobberingMemoryAccess(
+    MemoryAccess *StartingAccess, MemoryLocation &) {
+  if (auto *Use = dyn_cast<MemoryUseOrDef>(StartingAccess))
+    return Use->getDefiningAccess();
+  return StartingAccess;
+}
+}

llvm/trunk/lib/Transforms/Utils/Utils.cpp

void llvm::initializeTransformUtils(PassRegistry &Registry) {
   initializeLCSSAPass(Registry);
   initializeLoopSimplifyPass(Registry);
   initializeLowerInvokePass(Registry);
   initializeLowerSwitchPass(Registry);
   initializePromotePassPass(Registry);
   initializeUnifyFunctionExitNodesPass(Registry);
   initializeInstSimplifierPass(Registry);
   initializeMetaRenamerPass(Registry);
+  initializeMemorySSALazyPass(Registry);
+  initializeMemorySSAPrinterPassPass(Registry);
 }
 
 /// LLVMInitializeTransformUtils - C binding for initializeTransformUtilsPasses.
 void LLVMInitializeTransformUtils(LLVMPassRegistryRef R) {
   initializeTransformUtils(*unwrap(R));
 }

llvm/trunk/test/Transforms/Util/MemorySSA/atomic-clobber.ll

+; RUN: opt -basicaa -print-memoryssa -verify-memoryssa -analyze < %s 2>&1 | FileCheck %s
+;
+; Ensures that atomic loads count as MemoryDefs
+
+define i32 @foo(i32* %a, i32* %b) {
+; CHECK: 1 = MemoryDef(liveOnEntry)
+; CHECK-NEXT: store i32 4
+  store i32 4, i32* %a, align 4
+; CHECK: 2 = MemoryDef(1)
+; CHECK-NEXT: %1 = load atomic i32
+  %1 = load atomic i32, i32* %b acquire, align 4
+; CHECK: MemoryUse(2)
+; CHECK-NEXT: %2 = load i32
+  %2 = load i32, i32* %a, align 4
+  %3 = add i32 %1, %2
+  ret i32 %3
+}

llvm/trunk/test/Transforms/Util/MemorySSA/cyclicphi.ll

+; RUN: opt -basicaa -print-memoryssa -verify-memoryssa -analyze < %s 2>&1 | FileCheck %s
+
+%struct.hoge = type { i32, %struct.widget }
+%struct.widget = type { i64 }
+
+define hidden void @quux(%struct.hoge *%f) align 2 {
+  %tmp = getelementptr inbounds %struct.hoge, %struct.hoge* %f, i64 0, i32 1, i32 0
+  %tmp24 = getelementptr inbounds %struct.hoge, %struct.hoge* %f, i64 0, i32 1
+  %tmp25 = bitcast %struct.widget* %tmp24 to i64**
+  br label %bb26
+
+bb26:                                             ; preds = %bb77, %0
+; CHECK:  3 = MemoryPhi({%0,liveOnEntry},{bb77,2})
+; CHECK-NEXT:   br i1 undef, label %bb68, label %bb77
+  br i1 undef, label %bb68, label %bb77
+
+bb68:                                             ; preds = %bb26
+; CHECK:  MemoryUse(liveOnEntry)
+; CHECK-NEXT:   %tmp69 = load i64, i64* null, align 8
+  %tmp69 = load i64, i64* null, align 8
+; CHECK:  1 = MemoryDef(3)
+; CHECK-NEXT:   store i64 %tmp69, i64* %tmp, align 8
+  store i64 %tmp69, i64* %tmp, align 8
+  br label %bb77
+
+bb77:                                             ; preds = %bb68, %bb26
+; CHECK:  2 = MemoryPhi({bb26,3},{bb68,1})
+; CHECK:  MemoryUse(2)
+; CHECK-NEXT:   %tmp78 = load i64*, i64** %tmp25, align 8
+  %tmp78 = load i64*, i64** %tmp25, align 8
+  %tmp79 = getelementptr inbounds i64, i64* %tmp78, i64 undef
+  br label %bb26
+}

llvm/trunk/test/Transforms/Util/MemorySSA/function-clobber.ll

+; RUN: opt -basicaa -print-memoryssa -verify-memoryssa -analyze < %s 2>&1 | FileCheck %s
+;
+; Ensuring that external functions without attributes are MemoryDefs
+
+@g = external global i32
+declare void @modifyG()
+
+define i32 @foo() {
+; CHECK: MemoryUse(liveOnEntry)
+; CHECK-NEXT: %1 = load i32
+  %1 = load i32, i32* @g
+
+; CHECK: 1 = MemoryDef(liveOnEntry)
+; CHECK-NEXT: store i32 4
+  store i32 4, i32* @g, align 4
+
+; CHECK: 2 = MemoryDef(1)
+; CHECK-NEXT: call void @modifyG()
+  call void @modifyG()
+
+; CHECK: MemoryUse(2)
+; CHECK-NEXT: %2 = load i32
+  %2 = load i32, i32* @g
+  %3 = add i32 %2, %1
+  ret i32 %3
+}

llvm/trunk/test/Transforms/Util/MemorySSA/function-mem-attrs.ll

+; RUN: opt -basicaa -print-memoryssa -verify-memoryssa -analyze < %s 2>&1 | FileCheck %s
+;
+; Test that various function attributes give us sane results.
+
+@g = external global i32
+
+declare void @readonlyFunction() readonly
+declare void @noattrsFunction()
+
+define void @readonlyAttr() {
+; CHECK: 1 = MemoryDef(liveOnEntry)
+; CHECK-NEXT: store i32 0
+  store i32 0, i32* @g, align 4
+
+  %1 = alloca i32, align 4
+; CHECK: 2 = MemoryDef(1)
+; CHECK-NEXT: store i32 0
+  store i32 0, i32* %1, align 4
+
+; CHECK: MemoryUse(1)
+; CHECK-NEXT: call void @readonlyFunction()
+  call void @readonlyFunction()
+
+; CHECK: MemoryUse(1)
+; CHECK-NEXT: call void @noattrsFunction() #
+; Assume that #N is readonly
+  call void @noattrsFunction() readonly
+
+  ; Sanity check that noattrsFunction is otherwise a MemoryDef
+; CHECK: 3 = MemoryDef(2)
+; CHECK-NEXT: call void @noattrsFunction()
+  call void @noattrsFunction()
+  ret void
+}
+
+declare void @argMemOnly(i32*) argmemonly
+
+define void @inaccessableOnlyAttr() {
+  %1 = alloca i32, align 4
+; CHECK: 1 = MemoryDef(liveOnEntry)
+; CHECK-NEXT: store i32 0
+  store i32 0, i32* %1, align 4
+
+; CHECK: 2 = MemoryDef(1)
+; CHECK-NEXT: store i32 0
+  store i32 0, i32* @g, align 4
+
+; CHECK: MemoryUse(1)
+; CHECK-NEXT: call void @argMemOnly(i32* %1) #
+; Assume that #N is readonly
+  call void @argMemOnly(i32* %1) readonly
+
+; CHECK: 3 = MemoryDef(2)
+; CHECK-NEXT: call void @argMemOnly(i32* %1)
+  call void @argMemOnly(i32* %1)
+
+  ret void
+}

llvm/trunk/test/Transforms/Util/MemorySSA/load-invariant.ll

+; XFAIL:
+; RUN: opt -basicaa -print-memoryssa -verify-memoryssa -analyze < %s 2>&1 | FileCheck %s
+;
+; Invariant loads should be considered live on entry, because, once the
+; location is known to be dereferenceable, the value can never change.
+;
+; Currently XFAILed because this optimization was held back from the initial
+; commit.
+
+@g = external global i32
+
+declare void @clobberAllTheThings()
+
+define i32 @foo() {
+; CHECK: 1 = MemoryDef(liveOnEntry)
+; CHECK-NEXT: call void @clobberAllTheThings()
+  call void @clobberAllTheThings()
+; CHECK: MemoryUse(liveOnEntry)
+; CHECK-NEXT: %1 = load i32
+  %1 = load i32, i32* @g, align 4, !invariant.load !0
+  ret i32 %1
+}
+
+!0 = !{}

llvm/trunk/test/Transforms/Util/MemorySSA/many-dom-backedge.ll

+; RUN: opt -basicaa -print-memoryssa -verify-memoryssa -analyze < %s 2>&1 | FileCheck %s
+;
+; many-dom.ll, with an added back-edge back into the switch.
+; Because people love their gotos.
+
+declare i1 @getBool() readnone
+
+define i32 @foo(i32* %p) {
+entry:
+  br label %loopbegin
+
+loopbegin:
+; CHECK: 9 = MemoryPhi({entry,liveOnEntry},{sw.epilog,6})
+; CHECK-NEXT: %n =
+  %n = phi i32 [ 0, %entry ], [ %1, %sw.epilog ]
+  %m = alloca i32, align 4
+  switch i32 %n, label %sw.default [
+    i32 0, label %sw.bb
+    i32 1, label %sw.bb1
+    i32 2, label %sw.bb2
+    i32 3, label %sw.bb3
+  ]
+
+sw.bb:
+; CHECK: 1 = MemoryDef(9)
+; CHECK-NEXT: store i32 1
+  store i32 1, i32* %m, align 4
+  br label %sw.epilog
+
+sw.bb1:
+; CHECK: 2 = MemoryDef(9)
+; CHECK-NEXT: store i32 2
+  store i32 2, i32* %m, align 4
+  br label %sw.epilog
+
+sw.bb2:
+; CHECK: 3 = MemoryDef(9)
+; CHECK-NEXT: store i32 3
+  store i32 3, i32* %m, align 4
+  br label %sw.epilog
+
+sw.bb3:
+; CHECK: 10 = MemoryPhi({loopbegin,9},{sw.almostexit,6})
+; CHECK: 4 = MemoryDef(10)
+; CHECK-NEXT: store i32 4
+  store i32 4, i32* %m, align 4
+  br label %sw.epilog
+
+sw.default:
+; CHECK: 5 = MemoryDef(9)
+; CHECK-NEXT: store i32 5
+  store i32 5, i32* %m, align 4
+  br label %sw.epilog
+
+sw.epilog:
+; CHECK: 8 = MemoryPhi({sw.default,5},{sw.bb3,4},{sw.bb,1},{sw.bb1,2},{sw.bb2,3})
+; CHECK-NEXT: MemoryUse(8)
+; CHECK-NEXT: %0 =
+  %0 = load i32, i32* %m, align 4
+; CHECK: 6 = MemoryDef(8)
+; CHECK-NEXT: %1 =
+  %1 = load volatile i32, i32* %p, align 4
+  %2 = icmp eq i32 %0, %1
+  br i1 %2, label %sw.almostexit, label %loopbegin
+
+sw.almostexit:
+  %3 = icmp eq i32 0, %1
+  br i1 %3, label %exit, label %sw.bb3
+
+exit:
+; CHECK: 7 = MemoryDef(6)
+; CHECK-NEXT: %4 = load volatile i32
+  %4 = load volatile i32, i32* %p, align 4
+  %5 = add i32 %4, %1
+  ret i32 %5
+}

llvm/trunk/test/Transforms/Util/MemorySSA/many-doms.ll

+; RUN: opt -basicaa -print-memoryssa -verify-memoryssa -analyze < %s 2>&1 | FileCheck %s
+;
+; Testing many dominators, specifically from a switch statement in C.
+
+declare i1 @getBool() readnone
+
+define i32 @foo(i32* %p) {
+entry:
+  br label %loopbegin
+
+loopbegin:
+; CHECK: 8 = MemoryPhi({entry,liveOnEntry},{sw.epilog,6})
+; CHECK-NEXT: %n =
+  %n = phi i32 [ 0, %entry ], [ %1, %sw.epilog ]
+  %m = alloca i32, align 4
+  switch i32 %n, label %sw.default [
+    i32 0, label %sw.bb
+    i32 1, label %sw.bb1
+    i32 2, label %sw.bb2
+    i32 3, label %sw.bb3
+  ]
+
+sw.bb:
+; CHECK: 1 = MemoryDef(8)
+; CHECK-NEXT: store i32 1
+  store i32 1, i32* %m, align 4
+  br label %sw.epilog
+
+sw.bb1:
+; CHECK: 2 = MemoryDef(8)
+; CHECK-NEXT: store i32 2
+  store i32 2, i32* %m, align 4
+  br label %sw.epilog
+
+sw.bb2:
+; CHECK: 3 = MemoryDef(8)
+; CHECK-NEXT: store i32 3
+  store i32 3, i32* %m, align 4
+  br label %sw.epilog
+
+sw.bb3:
+; CHECK: 4 = MemoryDef(8)
+; CHECK-NEXT: store i32 4
+  store i32 4, i32* %m, align 4
+  br label %sw.epilog
+
+sw.default:
+; CHECK: 5 = MemoryDef(8)
+; CHECK-NEXT: store i32 5
+  store i32 5, i32* %m, align 4
+  br label %sw.epilog
+
+sw.epilog:
+; CHECK: 7 = MemoryPhi({sw.default,5},{sw.bb,1},{sw.bb1,2},{sw.bb2,3},{sw.bb3,4})
+; CHECK-NEXT: MemoryUse(7)
+; CHECK-NEXT: %0 =
+  %0 = load i32, i32* %m, align 4
+; CHECK: 6 = MemoryDef(7)
+; CHECK-NEXT: %1 =
+  %1 = load volatile i32, i32* %p, align 4
+  %2 = icmp eq i32 %0, %1
+  br i1 %2, label %exit, label %loopbegin
+
+exit:
+  ret i32 %1
+}

llvm/trunk/test/Transforms/Util/MemorySSA/multi-edges.ll

+; RUN: opt -basicaa -print-memoryssa -verify-memoryssa -analyze < %s 2>&1 | FileCheck %s
+;
+; Makes sure we have a sane model if both successors of some block is the same
+; block.
+
+define i32 @foo(i1 %a) {
+entry:
+  %0 = alloca i32, align 4
+; CHECK: 1 = MemoryDef(liveOnEntry)
+; CHECK-NEXT: store i32 4
+  store i32 4, i32* %0
+  br i1 %a, label %Loop.Body, label %Loop.End
+
+Loop.Body:
+; CHECK: 4 = MemoryPhi({entry,1},{Loop.End,3})
+; CHECK-NEXT: 2 = MemoryDef(4)
+; CHECK-NEXT: store i32 5
+  store i32 5, i32* %0, align 4
+  br i1 %a, label %Loop.End, label %Loop.End ; WhyDoWeEvenHaveThatLever.gif
+
+Loop.End:
+; CHECK: 3 = MemoryPhi({entry,1},{Loop.Body,2},{Loop.Body,2})
+; CHECK-NEXT: MemoryUse(3)
+; CHECK-NEXT: %1 = load
+  %1 = load i32, i32* %0, align 4
+  %2 = icmp eq i32 5, %1
+  br i1 %2, label %Ret, label %Loop.Body
+
+Ret:
+  ret i32 %1
+}

llvm/trunk/test/Transforms/Util/MemorySSA/multiple-backedges-hal.ll

+; RUN: opt -basicaa -print-memoryssa -verify-memoryssa -analyze < %s 2>&1 | FileCheck %s
+
+; hfinkel's case
+; [entry]
+;  |
+; .....
+; (clobbering access - b)
+;  |
+; ....  ________________________________
+;  \   /                               |
+;   (x)                                |
+;  ......                              |
+;    |                                 |
+;    |    ______________________       |
+;     \   /                    |       |
+; (starting access)            |       |
+;     ...                      |       |
+; (clobbering access - a)      |       |
+;    ...                       |       |
+;    | |                       |       |
+;    | |_______________________|       |
+;    |                                 |
+;    |_________________________________|
+;
+; More specifically, one access, with multiple clobbering accesses. One of
+; which strictly dominates the access, the other of which has a backedge
+
+; readnone so we don't have a 1:1 mapping of MemorySSA edges to Instructions.
+declare void @doThingWithoutReading() readnone
+declare i8 @getValue() readnone
+declare i1 @getBool() readnone
+
+define hidden void @testcase(i8* %Arg) {
+Entry:
+  call void @doThingWithoutReading()
+  %Val.Entry = call i8 @getValue()
+; CHECK: 1 = MemoryDef(liveOnEntry)
+; CHECK-NEXT: store i8 %Val.Entry
+  store i8 %Val.Entry, i8* %Arg
+  call void @doThingWithoutReading()
+  br label %OuterLoop
+
+OuterLoop:
+; CHECK: 5 = MemoryPhi({Entry,1},{InnerLoop.Tail,3})
+; CHECK-NEXT: %Val.Outer =
+  %Val.Outer = call i8 @getValue()
+; CHECK: 2 = MemoryDef(5)
+; CHECK-NEXT: store i8 %Val.Outer
+  store i8 %Val.Outer, i8* %Arg
+  call void @doThingWithoutReading()
+  br label %InnerLoop
+
+InnerLoop:
+; CHECK: 4 = MemoryPhi({OuterLoop,2},{InnerLoop,3})
+; CHECK-NEXT: ; MemoryUse(4)
+; CHECK-NEXT: %StartingAccess = load
+  %StartingAccess = load i8, i8* %Arg, align 4
+  %Val.Inner = call i8 @getValue()
+; CHECK: 3 = MemoryDef(4)
+; CHECK-NEXT: store i8 %Val.Inner
+  store i8 %Val.Inner, i8* %Arg
+  call void @doThingWithoutReading()
+  %KeepGoing = call i1 @getBool()
+  br i1 %KeepGoing, label %InnerLoop.Tail, label %InnerLoop
+
+InnerLoop.Tail:
+  %KeepGoing.Tail = call i1 @getBool()
+  br i1 %KeepGoing.Tail, label %End, label %OuterLoop
+
+End:
+  ret void
+}

llvm/trunk/test/Transforms/Util/MemorySSA/no-disconnected.ll

+; RUN: opt -basicaa -print-memoryssa -analyze -verify-memoryssa < %s 2>&1 | FileCheck %s
+;
+; This test ensures we don't end up with multiple reaching defs for a single
+; use/phi edge If we were to optimize defs, we would end up with 2=
+; MemoryDef(liveOnEntry) and 4 = MemoryDef(liveOnEntry) Both would mean both
+; 1,2, and 3,4 would reach the phi node.  Because the phi node can only have one
+; entry on each edge, it would choose 2, 4 and disconnect 1 and 3 completely
+; from the SSA graph, even though they are not dead
+
+define void @sink_store(i32 %index, i32* %foo, i32* %bar) {
+entry:
+  %cmp = trunc i32 %index to i1
+  br i1 %cmp, label %if.then, label %if.else
+
+if.then:                                          ; preds = %entry
+; CHECK: 1 = MemoryDef(liveOnEntry)
+; CHECK-NEXT:   store i32 %index, i32* %foo, align 4
+  store i32 %index, i32* %foo, align 4
+; CHECK: 2 = MemoryDef(1)
+; CHECK-NEXT:   store i32 %index, i32* %bar, align 4
+  store i32 %index, i32* %bar, align 4
+  br label %if.end
+
+if.else:                                          ; preds = %entry
+; CHECK: 3 = MemoryDef(liveOnEntry)
+; CHECK-NEXT:   store i32 %index, i32* %foo, align 4
+  store i32 %index, i32* %foo, align 4
+; CHECK: 4 = MemoryDef(3)
+; CHECK-NEXT:   store i32 %index, i32* %bar, align 4
+  store i32 %index, i32* %bar, align 4
+  br label %if.end
+
+if.end:                                           ; preds = %if.else, %if.then
+; CHECK: 5 = MemoryPhi({if.then,2},{if.else,4})
+; CHECK: MemoryUse(5)
+; CHECK-NEXT:   %c = load i32, i32* %foo
+  %c = load i32, i32* %foo
+; CHECK: MemoryUse(5)
+; CHECK-NEXT:   %d = load i32, i32* %bar
+  %d = load i32, i32* %bar
+  ret void
+}

llvm/trunk/test/Transforms/Util/MemorySSA/optimize-use.ll

+; RUN: opt -basicaa -print-memoryssa -analyze -verify-memoryssa < %s 2>&1 | FileCheck %s
+
+; Function Attrs: ssp uwtable
+define i32 @main() {
+entry:
+; CHECK: 1 = MemoryDef(liveOnEntry)
+; CHECK-NEXT:   %call = call noalias i8* @_Znwm(i64 4)
+  %call = call noalias i8* @_Znwm(i64 4)
+  %0 = bitcast i8* %call to i32*
+; CHECK: 2 = MemoryDef(1)
+; CHECK-NEXT:   %call1 = call noalias i8* @_Znwm(i64 4)
+  %call1 = call noalias i8* @_Znwm(i64 4)
+  %1 = bitcast i8* %call1 to i32*
+; CHECK: 3 = MemoryDef(2)
+; CHECK-NEXT:   store i32 5, i32* %0, align 4
+  store i32 5, i32* %0, align 4
+; CHECK: 4 = MemoryDef(3)
+; CHECK-NEXT:   store i32 7, i32* %1, align 4
+  store i32 7, i32* %1, align 4
+; CHECK: MemoryUse(3)
+; CHECK-NEXT:   %2 = load i32, i32* %0, align 4
+  %2 = load i32, i32* %0, align 4
+; CHECK: MemoryUse(4)
+; CHECK-NEXT:   %3 = load i32, i32* %1, align 4
+  %3 = load i32, i32* %1, align 4
+; CHECK: MemoryUse(3)
+; CHECK-NEXT:   %4 = load i32, i32* %0, align 4
+  %4 = load i32, i32* %0, align 4
+; CHECK: MemoryUse(4)
+; CHECK-NEXT:   %5 = load i32, i32* %1, align 4
+  %5 = load i32, i32* %1, align 4
+  %add = add nsw i32 %3, %5
+  ret i32 %add
+}
+
+declare noalias i8* @_Znwm(i64)

llvm/trunk/test/Transforms/Util/MemorySSA/volatile-clobber.ll

+; RUN: opt -basicaa -print-memoryssa -verify-memoryssa -analyze < %s 2>&1 | FileCheck %s
+;
+; Ensures that volatile stores/loads count as MemoryDefs
+
+define i32 @foo() {
+  %1 = alloca i32, align 4
+; CHECK: 1 = MemoryDef(liveOnEntry)
+; CHECK-NEXT: store volatile i32 4
+  store volatile i32 4, i32* %1, align 4
+; CHECK: 2 = MemoryDef(1)
+; CHECK-NEXT: store volatile i32 8
+  store volatile i32 8, i32* %1, align 4
+; CHECK: 3 = MemoryDef(2)
+; CHECK-NEXT: %2 = load volatile i32
+  %2 = load volatile i32, i32* %1, align 4
+; CHECK: 4 = MemoryDef(3)
+; CHECK-NEXT: %3 = load volatile i32
+  %3 = load volatile i32, i32* %1, align 4
+  %4 = add i32 %3, %2
+  ret i32 %4
+}