Codegen: Tail-duplicate during placement.

[iteratee]

Summary

The tail duplication pass uses an assumed layout when making duplication
decisions. This is fine, but passes up duplication opportunities that
may arise when blocks are outlined. This change allows duplication after
layout decisions. Because we want the updated CFG to affect subsequent
placement decisions, duplication must occur during placement.

In order to achieve this goal, TailDuplicationPass is split into a
utility class, TailDuplicator, and the pass itself. The pass delegates
nearly everything to the TailDuplicator object, except for looping over
the blocks in a function. This allows the same code to be used for tail
duplication in both places.

This change, in concert with outlining optional branches, allows
triangle shaped code to perform much better, esepecially when the
taken/untaken branches are correlated, as it creates a second spine when
the tests are small enough.

[iteratee]

Add a test and a fix for duplicating blocks that have un-analyzable fallthrough.
Note that the test doesn't currently fail, but D20505 will fail without the fix.

[iteratee]

Are there any other concerns with this patch other than performance and size numbers from Quentin?

[iteratee]

This comment was removed by iteratee.

[qcolombet]

Hi Kyle,

The performance results pretty look good so far! (Up to 4% improvement on some SPEC!)
Go ahead with the commit, we will adjust if things turn out to be bad for other workload, but I do not think it will happen.

The only actionable regression on your end is SingleSource/Benchmarks/Misc/ReedSolomon, which regressed by ~4% on x86_64 -O3. Other than that, this is a net win or the regression are within the noise!

Cheers,
-Quentin

[iteratee]

Modified to work with tail-merging during layout.
Had to make tail-merging during layout less agressive so that it wouldn't undo tail-duplication.

[iteratee]

clean up some spurious debugging.

[echristo]

Couple of inline comments. I think some of the overall design can be separated out and be made a bit simpler - especially as relates to ForcedLayoutPred etc, but we can worry about that as a cleanup that'll happen next.

Thanks!

-eric

lib/CodeGen/BranchFolding.cpp

659

Can you elaborate here? Perhaps with a FIXME or a TODO.

lib/CodeGen/MachineBlockPlacement.cpp

1949

The MachineFunction has an MMI so you don't need this argument. :)

[davidxl]

test/CodeGen/PowerPC/tail-dup-analyzable-fallthrough.ll

5 ↗

(On Diff #62154)

Please simplify this test case and document what is the expected layout with tailDup. The test as it stands now is hard to read.

test/CodeGen/PowerPC/tail-dup-layout.ll

3

Same here : simplify the test case and document the expected output with description.

[iteratee]

Added comments to tests to try and make them more readable

[iteratee]

Added comment explaining why tail-merging can't use the unconditional branch heuristic when it is run during layout.

[iteratee]

lib/CodeGen/BranchFolding.cpp

659

I don't think that there is a way to fix it. The heuristic overcounts fractional tails where one block would end up with a jump, but another would not. It's probably fine to use it early, but during layout we have to be stricter about the thresholds.

test/CodeGen/PowerPC/tail-dup-analyzable-fallthrough.ll

5 ↗

(On Diff #62154)

This test was reduced from an actual regression that I introduced and then fixed.

I've added comments to explain what went wrong.

test/CodeGen/PowerPC/tail-dup-layout.ll

3

I don't see that this test is that complex at all, except for the optional bodies. Unfortunately, they have to a certain size or they won't get outlined. I've added an easier to read expectation of layout, and a description of what the CHECK lines are looking for.

[iteratee]

Fixed the last 2 tests.
One was minor,
For the WebAssembly test, I just disabled tail-duplication during placement, as previously instructed.

[davidxl]

test/CodeGen/PowerPC/tail-dup-analyzable-fallthrough.ll

6 ↗

(On Diff #62531)

Kyle, thanks.

I think it is better get a minimal reproducible and use that as test case. for instance, I am not sure if the struct HashBucket etc and some other things are relevant here -- they are all distractions. I have pointed to an example of the minimalist test case.

test/CodeGen/PowerPC/tail-dup-layout.ll

4

The test case should be as simple as possible and remove all irrelevant pieces.

Regarding the size limit, you can use "outline-optional-threshold", option to specify a lower limit so that you can reduce the size of the optional body.

53

An example of a simple clean test case:

CodeGen/X86/code_placement_outline_optional_branches.ll

[iteratee]

Simplify test cases.

[iteratee]

Please take a look at the tests again.

test/CodeGen/PowerPC/tail-dup-analyzable-fallthrough.ll

7 ↗

(On Diff #62551)

I simplified the code quite a bit. The essence of what was happening was a linked-list walk was getting miscompiled, so I shrunk it to that.

test/CodeGen/PowerPC/tail-dup-layout.ll

5

OK, I made it look more like the other test that you pointed too, with repeated calls for bodies.

[davidxl]

Very nice. I like the patch a lot! I will go though it one more time shortly.

thanks,

David

[davidxl]

lib/CodeGen/BranchFolding.cpp

610

Since you are here, please document arguments to this method: MBB1, MBB2, PredBB, SuccBB, AfterPlacement.

643

Expand the comment a little more: TailMerging invoked after block placement. Do not undo ...

672

Is there a better way to identify Pred BBs which are tailduplicated into during block placement? Looks to me we should avoid undoing any taildups done in layout stage.

lib/CodeGen/TailDuplicator.cpp

509

Document new parameters. While you are here, document IsSimple as well.

777

Same here -- document parameter.

859

Are there other places in taildup where blocks may get removed? It seems to me that tailDupllicator needs to have member to remember it is in 'layout' state, and the block deletor code should check that flag instead.

[iteratee]

Added comments, simplified flags to tail-duplicator

lib/CodeGen/BranchFolding.cpp

672

While we could pass a set of blocks around, the thresholds don't overlap by default. e.g. We tail-duplicate blocks of 2 instructions or fewer, and tail-merge blocks of 3 instructions or more. The problem is with fractional overlap. e.g. a block of 2 instructions was duplicated and in one of the resulting block an additional branch had to be added.

Would you prefer that I create a set of blocks and pass them into the branch optimizer? It would make it faster to skip those blocks altogether.

lib/CodeGen/TailDuplicator.cpp

509

New parameter removed.

859

No, nowhere else that blocks may get removed, and ultimately I want to not have different modes.

I decided that for duplication during layout, it doesn't make sense to try to delete the block. I've updated the code accordingly, and will fix D20505 to match.

[davidxl]

lib/CodeGen/BranchFolding.cpp

672

Depending on the threshold difference can be fragile.

How about changing AfterPlacement into a callback/lambda function using function_ref<..>. It tells tailMerge if a block should be skipped.

[iteratee]

Took a more principled stand on when to tail merge and when not to, so that tail-duplicating and tail merging now don't overlap.

[davidxl]

lib/CodeGen/BranchFolding.cpp

616

I think it is cleaner to check at the beginning if anyone of MBB1 or MBB2 need to be skipped (either be the taildupped block or a block tail-dupped into), and return false if so. The set of skipped blocks can be passed into the BF by block placement.

638

Is this change related?

lib/CodeGen/TailDuplicator.cpp

519

Expand the comments a little more.

860

nit: block placement does not expect any blocks to be deleted.

[iteratee]

lib/CodeGen/BranchFolding.cpp

643

I changed this heuristic to avoid undoing valuable duplication. In fact, some of the cases where it doesn't merge are cases where we wanted to tail duplicate, but couldn't. See Hexagon/rdf-copy.ll

I'll split the changes here out into a separate patch.

672

Discussed this with chandlerc. He thinks that passing around the list of blocks that were tail-duplicated too tightly couples the passes, and that we should strive to have non-overlap. He suggested that a list of blocks that were duplicated might be a good debug measure to make sure they aren't being re-merged.

Overall I agree with him, and I made a change so there shouldn't be any overlap.
Do you want to see the set as a debugging measure?

[iteratee]

I'll handle the 2 comment changes you asked for before I upload a new patch, but please take a look at the comments I added. I wrote them before I saw your comments, because I was thinking through some cases.

lib/CodeGen/BranchFolding.cpp

616

See below for the discussion about this, but I don't think passing in the set of blocks is the right choice.

638

Yes, but I plan to split it out into a separate patch.

[davidxl]

lib/CodeGen/BranchFolding.cpp

672

I don't agree. Without making it explicit, the the pass coupling does not disappear -- it is still there but in a more implicit but hard to maintain way.

For instance, can we guarantee this still work when TailMerger and TailDuplicator uses non-default size thresholds?

[chandlerc]

lib/CodeGen/BranchFolding.cpp

672

My thought was specifically that we should enforce an invariant that these are defined in a non-overlapping way, *and* check it in asserts builds precisely so that things like non-default size thresholds work in a sane way.

As long as there is the potential for overlap, we have the problem (IMO) that the innards of tail dup or tail merge can do things that user programs cannot. I would much prefer that we have a set of rules that are sufficiently precise and enforced that one pass doesn't have to have a back-channel to communicate to the other pass.

But I do agree that it is important that this is actually a guaranteed property and one we check that we don't violate.

[iteratee]

Use single explicit threshold between tail-dup and tail-merge during layout.
Tail-duplicate indirect branches with a larger threshold during layout (Mirrors pre reg-alloc behavior).

[iteratee]

Fix WebAssembly cfg stackify test.
Broke because we don't tail-duplicate loop latches early any more.

[iteratee]

Fixed additional tests.
Early tail-duplication now respects loops.
Explicit threshold for merging/duplication during layout.

[iteratee]

Factored out explicit threshold code.

[iteratee]

lib/CodeGen/BranchFolding.cpp

672

This should all be resolved now. See D23383 and D23390 for the explicit threshold. The changes to make merging not overlap during layout have already gone in.

lib/CodeGen/TailDuplicator.cpp

860–862

This is resolved by allowing the deletion to occur.

[iteratee]

Updated tests. No other changes.

[iteratee]

Quentin, can you re-run the benchmark suite? This patch has changed a fair bit since you last ran them.
If you could do D20505 as well, which depends on this, that would be awesome.

Thanks,
Kyle.

[davidxl]

include/llvm/CodeGen/TailDuplicator.h

58

Document the new parameters and behavior (zero default etc)

lib/CodeGen/BranchFolding.cpp

699

CurMPIter = HighestMPIter;

Maybe a separate cleanup patch

707

MBB1 can be computed in common path

709

On what platforms? Is there a better way to query it?

713

where does 21 magic number come from? Why not just define a special value to signal merging is disabled/skipped for such blocks?

lib/CodeGen/MachineBlockPlacement.cpp

139

This name is very confusing. Just name it as TailDupThresholdInLayout. Perhaps document that this is also used to 'force' the tail merge at the high threshold to avoid it undoing tailDup decisions.

1009

This makes the primary loop body too large. Outline it into its own method.

lib/CodeGen/TailDuplicator.cpp

545

Irrelevant change?

572

Irrelevant change?

575

Is this change needed (handled by previous taildup invocation)?

[iteratee]

Fixes from review.

[iteratee]

I have some questions, but most of the suggestions are done.

lib/CodeGen/BranchFolding.cpp

709

I believe ARM marks the actual return as an indirect branch.
I don't see a better way. Indirect tail-calls will likely get caught here as well.
Since what we're looking for is jump tables within the function, checking the successor size seems reasonable.
Would you like me to re-write the comment so that it seems more intentional?

713

It's one more than 20, from tail-duplication of the same pattern. I'm open to other suggestions as to how to make these two cooperate.

lib/CodeGen/TailDuplicator.cpp

545

No. With tail-duplication occurring only once, we had to duplicate those blocks when we got the chance. Now with layout-duplication we can wait and duplicate them later.

572

No. Because this check is only necessary if you want to expand these blocks later in layout mode.
This change is required by the previous one above, because otherwise it breaks a test.

575

Yes, there are tests where the loop-latch is a switch. I presume it occurs in the real world as well.

[iteratee]

lib/CodeGen/TailDuplicator.cpp

545

This was related to a performance regression I found. I can probably pull this out into a separate patch that gets committed at the same time. Would that be better?

[davidxl]

lib/CodeGen/TailDuplicator.cpp

545

Yes this should be in a different patch with different review (and test cases etc).

[iteratee]

Changes missed from previous upload

[davidxl]

lib/CodeGen/BranchFolding.cpp

709

How abut Block1.back().isIndirectBranch() && !Block1.back().isReturn() ?

713

I think this value should be passed in either via constructor or a separate API:

setMergeThresholdForBBwithIndirectBr(...).

In tailDup, the value 20 should also be controlled by an option.

lib/CodeGen/MachineBlockPlacement.cpp

1015

Why can't this part be folded into maybeTailDuplicateBlock? If not, the big chunk of code between if (Removed) { ..} belongs to its own method.

lib/CodeGen/TailDuplicator.cpp

572

I don't get it. This one looks like a heuristic change to not give block with return instruction (as indirect branch) a larger threshold for duplication, which is independent of this patch.

[iteratee]

lib/CodeGen/TailDuplicator.cpp

572

I'll split these out, but they're all intertwined.

1. Don't duplicate latches/pre-headers early.
2. There are now tests that fail, so duplicate switches during layout.
3. Because the indirect branch test is now post Reg-Alloc, it has to handle the actual return instructions.

[davidxl]

lib/CodeGen/TailDuplicator.cpp

572

thx for the expalnation. Getting Part-1 right is the key here.

[iteratee]

Removed changes as requested. They'll show up in another patch shortly.

[davidxl]

lib/CodeGen/MachineBlockPlacement.cpp

1016

BB = *std::prev(Chain.end())

1853

early return if not removed to reduce code nesting level

1854

remove 'into'

1859

std::prev

1863

This does not look right the place to update. The successors of DupBB may also get more unscheduled predecessors. The update should be done in one place where the total number of Preds (BB is dupped into) is known.

1868

Can you give more explanation on the iterative tailDup?

1904

Should an early return be done here?
if (!IsSimple && BB->succ_size() == 1) return false;
if (!TailDup.shouldTailDuplicate(...) return false;

1909

should also skip if Pred is in the same chain as BB (e.g, when BB is a loop head of another loop, and Pred is that loop's latch)

1912

Need to count the number of other Pred that BB can tail duplicate into.

If that number is zero, should return false.

1924

To early to do the update. Wait until the tailDup actually happens. Actually, do we really need to update? When tailDup happens, Pred will become NewSucc's new predecessor, and Pred is still not scheduled.

1929

Early return false when '!CanDupToChain'

1953

It is cleaner to use RemoveList.erase(remove_if(...)..) pattern

Also bypass if RemBB's unscheduled pred number is not zero.

[iteratee]

Simplifications from review.

[iteratee]

Applied some suggestions, answered others.

lib/CodeGen/MachineBlockPlacement.cpp

1859

Can't. I need to actually update the iterator so that I can check to see if there is an additional block in the chain.

1863

You're correct, It only needs to be done once because only one block is actually going from unscheduled to scheduled: BB.
That makes the function simpler overall and easier to understand.

1909

No, even if they're in the same chain, blocks that end up with both as predecessors still have an additional predecessor.

I think you're thinking of the check that I do below of NewChain vs PredChain.

1912

No. because although we don't have predecessors to worry about (Loop filter), there may be unfiltered predecessors that we can duplicate into, and now is the only time to do so.

1924

We can't do it any later. BB may disappear after tail duplication.

I could make tail duplication return a vector of blocks that were duplicated into, and save the successor list before duplication.
That may be easier to follow, but wouldn't be any more correct.

Yes, we need to update. For each duplication the successors now have an additional unscheduled predecessor

1929

That would be incorrect. There are plenty of cases where we can't duplicate into the layout predecessor where we still want to perform tail duplication.

[iteratee]

Remove unused variable.

[davidxl]

lib/CodeGen/MachineBlockPlacement.cpp

1912

It is not correct to do. BB and Pred are in the same nested loop that already been laid out. You can not duplicate BB into Pred here.

LPred

|
|

BB <-----

|               \
..                 \
Pred              |
 \__________|

1924

I think a list of Preds that BB can be duplicated into need to be passed in.

The reason doing update here is not bullet proof is that if later if there are other heuristic to decide not to tailDup, then the early update will be wrong. It needs to be tied to the actual transformation

1929

There are two problems here

1. many Pred (other than LPred) has been skipped as illegal candidates, but tailDupiicator does not know about will happily do tailDup -- as the tailDuplicator does not know about the new constraints from MBP
2. if the total number of candidate predecessors that BB can tailDuped into is fewer than 2, there is no point do the tail duplication

[iteratee]

lib/CodeGen/MachineBlockPlacement.cpp

1912

I don't see why not. There's not a correctness issue, and if BB was the loop top of the inner loop, it was never considered to be duplicated. Now is the chance that we have to do it.

1924

The goal of this loop is not to find constraints.

I agree it would be cleaner to get the list out of tail-duplication. I'll do that.

1929

I think you misunderstand the point of this loop. The point is not to find "legal candidates", but to find the candidates for which duplicating will require update.

Because of that, we don't have an accurate count, and don't want one. We'll gladly let tailDuplicator do its job.

[davidxl]

lib/CodeGen/MachineBlockPlacement.cpp

1868

This one needs a test case to cover.

1929

Ok.

Do you see the possibility that some Pred simply can not be selected as dup target bb (for legality or performance reasons)? Overall, the code should be organizied in the way that the list of BBs that MBP thinks it can/should be tailDup should match the actual transformations.

[iteratee]

lib/CodeGen/MachineBlockPlacement.cpp

1929

I can see that we might want to limit the set of predecessors duplicate to. A good case would be to filter out predecessors that are too cold.

The performance results have been reasonable so far without that change, and It would certainly make this more complicated.
For now, I think it would be best to simply go with the cleaner version of getting a list of blocks back from tailDuplicateAndUpdate.

This will make it easy to add filters later, and to add them inside of tailDuplicator as well without worrying about breaking this loop.

[davidxl]

lib/CodeGen/MachineBlockPlacement.cpp

1929

I am fine with unfiltered solution for now -- but the code structure should make future tuning easy to do :)

[iteratee]

Update unscheduled predecessors after tail duplication.

[iteratee]

Changes missed in previous patch. Get list of duplicated predecessors from TailDuplicator and use that list to update unscheduled predecessors.

[iteratee]

Remove extra debugging.

[iteratee]

lib/CodeGen/BranchFolding.cpp

709

I checked. There are platforms where the "return" instruction does not return true for isReturn after epilog lowering.
Mips, Hexagon, Thumb.

I think I'll re-write the comment so that it sounds less like a workaround.

[iteratee]

Add test case for repeated tail-duplication.

[davidxl]

test/CodeGen/X86/tail-dup-repeat.ll

7 ↗

(On Diff #69762)

Can this function's declaration be simplified? If the parameters are not necessary, remove them

11 ↗

(On Diff #69762)

Clean up the parameters (shorten names etc)). Remove unnecessary attribute

15 ↗

(On Diff #69762)

Please also clean up variable names and label names.

For label names, make them more meaningful. For instance, if a block is duplicated, the label for the block can be 'dup:'. For iterative tail dup, also name them in sequence numbers dup1, dup2 etc. Others can be cleaned up suchas header, loopexit etc.

Some simple comments in the test case (where tail dup happens) is also helpful.

[iteratee]

Found new, shorter test case for repeated tail duplication.

[davidxl]

lib/CodeGen/MachineBlockPlacement.cpp

1892

Is this comment correct? The TailBB may be dup'ed in to Lpred, but still not removed as there are some other predecessors it can not be dup'ed into ?

1981

Should DuplicatedToLPred == CanDupToChain?

[iteratee]

Simplified computation of whether a block was duplicated to the layout pred.

lib/CodeGen/MachineBlockPlacement.cpp

1892

It is currently correct.
The tail duplication code only copies the block into the layout predecessor if it can be copied into all other predecessors as well.

1981

Even better, since we now return the list of predecessors that received a copy of BB, we should check to see if LPred is in that list. Basically free since we loop over the list anyway.

[davidxl]

lib/CodeGen/MachineBlockPlacement.cpp

1016

The update of BB should be wrapped into 'repeatedlyTailDuplicateBlock'.

1856

Missing coma. ".. it was removed, markChainSuccessors .."

1866

Add a comment here that Chain End is updated when DupBB is removed by the removal call back.

There also seem to be a bug here: if TailDup happens, but DupBB is not removed, the loop will check the same BB again. Should it just break the loop when removed is not true?

1893

It is cleaner to make this interface return bool (either Removed or DuplicatedToLPred)

[iteratee]

Added comments.

Made changes from review:
Chain end is now updated in repeatedlyTailDuplicateBlock
MaybeTailDuplicate now returns Removed instead of using out parameter.

[iteratee]

Please take a look. I cleaned up the issues you pointed out and added comments.

lib/CodeGen/MachineBlockPlacement.cpp

1856

I split the sentence into 2 and then added a comma. It's easier to read now.

1866

I added some comments.

Currently, duplication to layout predecessor only occurs if the block is removed. I added a comment about that as well.

[davidxl]

lgtm.

This is large change so it bounds to run into some corner case issues. Please test thoroughly and watch for any fallouts.

lib/CodeGen/MachineBlockPlacement.cpp

1882

Add an assert that Removed is true when DuplicatedToLPred is true

include/llvm/CodeGen/TailDuplicator.h

 // This file defines the TailDuplicator class. Used by the
 // TailDuplication pass, and MachineBlockPlacement.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_TAILDUPLICATOR_H
 #define LLVM_CODEGEN_TAILDUPLICATOR_H
 
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/CodeGen/MachineBranchProbabilityInfo.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/MachineSSAUpdater.h"
 #include "llvm/CodeGen/RegisterScavenging.h"
 #include "llvm/Target/TargetInstrInfo.h"
 #include "llvm/Target/TargetRegisterInfo.h"
 #include "llvm/Target/TargetSubtargetInfo.h"
 
 namespace llvm {
 
 /// Utility class to perform tail duplication.
 class TailDuplicator {
   const TargetInstrInfo *TII;
   const TargetRegisterInfo *TRI;
   const MachineBranchProbabilityInfo *MBPI;
+  MachineLoopInfo *MLI;
   const MachineModuleInfo *MMI;
   MachineRegisterInfo *MRI;
   bool PreRegAlloc;
+  bool LayoutMode;
   unsigned TailDupSize;
 
   // A list of virtual registers for which to update SSA form.
   SmallVector<unsigned, 16> SSAUpdateVRs;
 
   // For each virtual register in SSAUpdateVals keep a list of source virtual
   // registers.
   typedef std::vector<std::pair<MachineBasicBlock *, unsigned>> AvailableValsTy;
 
   DenseMap<unsigned, AvailableValsTy> SSAUpdateVals;
 
 public:
   /// Prepare to run on a specific machine function.
+  /// @param LayoutMode - When true, don't use the existing layout to make
+  ///                     decisions.
   /// @param TailDupSize - Maxmimum size of blocks to tail-duplicate.
   void initMF(MachineFunction &MF, const MachineModuleInfo *MMI,
               const MachineBranchProbabilityInfo *MBPI,
-              unsigned TailDupSize = 0);
+              MachineLoopInfo *MLI, bool LayoutMode, unsigned TailDupSize = 0);
   bool tailDuplicateBlocks(MachineFunction &MF);
   static bool isSimpleBB(MachineBasicBlock *TailBB);
   bool shouldTailDuplicate(const MachineFunction &MF, bool IsSimple,
                            MachineBasicBlock &TailBB);
   /// Returns true if TailBB can successfully be duplicated into PredBB
   bool canTailDuplicate(MachineBasicBlock *TailBB, MachineBasicBlock *PredBB);
   bool tailDuplicateAndUpdate(MachineFunction &MF, bool IsSimple,
-                              MachineBasicBlock *MBB);
+                              MachineBasicBlock *MBB,
+                              MachineBasicBlock *ForcedLayoutPred,
+                              llvm::function_ref<void(MachineBasicBlock *)> *RemovalCallback = nullptr);
 
 private:
   typedef TargetInstrInfo::RegSubRegPair RegSubRegPair;
 
   void addSSAUpdateEntry(unsigned OrigReg, unsigned NewReg,
                          MachineBasicBlock *BB);
   void processPHI(MachineInstr *MI, MachineBasicBlock *TailBB,
                   MachineBasicBlock *PredBB,
void updateSuccessorsPHIs(MachineBasicBlock *FromBB, bool isDead,
                             SmallSetVector<MachineBasicBlock *, 8> &Succs);
   bool canCompletelyDuplicateBB(MachineBasicBlock &BB);
   bool duplicateSimpleBB(MachineBasicBlock *TailBB,
                          SmallVectorImpl<MachineBasicBlock *> &TDBBs,
                          const DenseSet<unsigned> &RegsUsedByPhi,
                          SmallVectorImpl<MachineInstr *> &Copies);
   bool tailDuplicate(MachineFunction &MF, bool IsSimple,
                      MachineBasicBlock *TailBB,
+                     MachineBasicBlock *ForcedLayoutPred,
                      SmallVectorImpl<MachineBasicBlock *> &TDBBs,
                      SmallVectorImpl<MachineInstr *> &Copies);
   void appendCopies(MachineBasicBlock *MBB,
                  SmallVectorImpl<std::pair<unsigned,RegSubRegPair>> &CopyInfos,
                  SmallVectorImpl<MachineInstr *> &Copies);
 
-  void removeDeadBlock(MachineBasicBlock *MBB);
+  void removeDeadBlock(MachineBasicBlock *MBB,
+                       llvm::function_ref<void(MachineBasicBlock *)> *RemovalCallback = nullptr);
 };
 
 } // End llvm namespace
 
 #endif

lib/CodeGen/BranchFolding.cpp

 ///                 instruction in the common tail shared by MBB1,MBB2
 /// SuccBB          A common successor of MBB1, MBB2 which are in a canonical form
 ///                 relative to SuccBB
 /// PredBB          The layout predecessor of SuccBB, if any.
 /// FuncletMembership  map from block to funclet #.
 /// AfterPlacement  True if we are merging blocks after layout. Stricter
 ///                 thresholds apply to prevent undoing tail-duplication.
 static bool
 ProfitableToMerge(MachineBasicBlock *MBB1, MachineBasicBlock *MBB2,
                   unsigned MinCommonTailLength, unsigned &CommonTailLen,
                   MachineBasicBlock::iterator &I1,
                   MachineBasicBlock::iterator &I2, MachineBasicBlock *SuccBB,
                   MachineBasicBlock *PredBB,
                   DenseMap<const MachineBasicBlock *, int> &FuncletMembership,
                   bool AfterPlacement) {
   // It is never profitable to tail-merge blocks from two different funclets.
   if (!FuncletMembership.empty()) {
     auto Funclet1 = FuncletMembership.find(MBB1);
     assert(Funclet1 != FuncletMembership.end());
     auto Funclet2 = FuncletMembership.find(MBB2);
     assert(Funclet2 != FuncletMembership.end());
     if (Funclet1->second != Funclet2->second)
       return false;
   }
 
   CommonTailLen = ComputeCommonTailLength(MBB1, MBB2, I1, I2);
   if (CommonTailLen == 0)
     return false;
   DEBUG(dbgs() << "Common tail length of BB#" << MBB1->getNumber()
                << " and BB#" << MBB2->getNumber() << " is " << CommonTailLen
                << '\n');
 
   // It's almost always profitable to merge any number of non-terminator
   // instructions with the block that falls through into the common successor.
   // This is true only for a single successor. For multiple successors, we are
   // trading a conditional branch for an unconditional one.
   // TODO: Re-visit successor size for non-layout tail merging.
   if ((MBB1 == PredBB || MBB2 == PredBB) &&
       (!AfterPlacement || MBB1->succ_size() == 1)) {
     MachineBasicBlock::iterator I;
     unsigned NumTerms = CountTerminators(MBB1 == PredBB ? MBB2 : MBB1, I);
     if (CommonTailLen > NumTerms)
       return true;
   }
 
   // If one of the blocks can be completely merged and happens to be in
   // a position where the other could fall through into it, merge any number
   // of instructions, because it can be done without a branch.
   // TODO: If the blocks are not adjacent, move one of them so that they are?
   if (MBB1->isLayoutSuccessor(MBB2) && I2 == MBB2->begin())
     return true;
   if (MBB2->isLayoutSuccessor(MBB1) && I1 == MBB1->begin())
     return true;
 
   // If both blocks have an unconditional branch temporarily stripped out,
   // count that as an additional common instruction for the following
   // heuristics. This heuristic is only accurate for single-succ blocks, so to
   // make sure that during layout merging and duplicating don't crash, we check
   // for that when merging during layout.
   unsigned EffectiveTailLen = CommonTailLen;
   if (SuccBB && MBB1 != PredBB && MBB2 != PredBB &&
       (MBB1->succ_size() == 1 || !AfterPlacement) &&
       !MBB1->back().isBarrier() &&
       !MBB2->back().isBarrier())
     ++EffectiveTailLen;
 
   // Check if the common tail is long enough to be worthwhile.
   if (EffectiveTailLen >= MinCommonTailLength)
     return true;
 
   // If we are optimizing for code size, 2 instructions in common is enough if
   // we don't have to split a block.  At worst we will be introducing 1 new
   // branch instruction, which is likely to be smaller than the 2
   // instructions that would be deleted in the merge.
   MachineFunction *MF = MBB1->getParent();
   return EffectiveTailLen >= 2 && MF->getFunction()->optForSize() &&
          (I1 == MBB1->begin() || I2 == MBB2->begin());
 }
 

 unsigned BranchFolder::ComputeSameTails(unsigned CurHash,
                                         unsigned MinCommonTailLength,
                                         MachineBasicBlock *SuccBB,
                                         MachineBasicBlock *PredBB) {
   unsigned maxCommonTailLength = 0U;
   SameTails.clear();
   MachineBasicBlock::iterator TrialBBI1, TrialBBI2;
   MPIterator HighestMPIter = std::prev(MergePotentials.end());
   for (MPIterator CurMPIter = std::prev(MergePotentials.end()),
                   B = MergePotentials.begin();
        CurMPIter != B && CurMPIter->getHash() == CurHash; --CurMPIter) {
     for (MPIterator I = std::prev(CurMPIter); I->getHash() == CurHash; --I) {
       unsigned CommonTailLen;
+      unsigned MinCommonTailLengthInside = MinCommonTailLength;
+      // Prefer to leave indirect branches duplicated after placement.
+      if (AfterBlockPlacement) {
+        MachineBasicBlock &Block1 = *CurMPIter->getBlock();
+        if (Block1.empty())
+          // On some platforms return instructions are marked as indirect
+          // branches and not as return instructions. Work around this by
+          // checking for return via the CFG.
+          if (Block1.back().isIndirectBranch() && Block1.succ_size() != 0)
+            MinCommonTailLengthInside = 21;
+      }
       if (ProfitableToMerge(CurMPIter->getBlock(), I->getBlock(),
-                            MinCommonTailLength,
+                            MinCommonTailLengthInside,
                             CommonTailLen, TrialBBI1, TrialBBI2,
                             SuccBB, PredBB,
                             FuncletMembership,
                             AfterBlockPlacement)) {
         if (CommonTailLen > maxCommonTailLength) {
           SameTails.clear();
           maxCommonTailLength = CommonTailLen;
           HighestMPIter = CurMPIter;

lib/CodeGen/MachineBlockPlacement.cpp

 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineBlockFrequencyInfo.h"
 #include "llvm/CodeGen/MachineBranchProbabilityInfo.h"
 #include "llvm/CodeGen/MachineDominators.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
+#include "llvm/CodeGen/TailDuplicator.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Target/TargetInstrInfo.h"
 #include "llvm/Target/TargetLowering.h"
 #include "llvm/Target/TargetSubtargetInfo.h"
 #include <algorithm>
static cl::opt<unsigned> MisfetchCost(
     cl::desc("Cost that models the probabilistic risk of an instruction "
              "misfetch due to a jump comparing to falling through, whose cost "
              "is zero."),
     cl::init(1), cl::Hidden);
 
 static cl::opt<unsigned> JumpInstCost("jump-inst-cost",
                                       cl::desc("Cost of jump instructions."),
                                       cl::init(1), cl::Hidden);
+static cl::opt<bool>
+TailDupPlacement("tail-dup-placement",
+              cl::desc("Perform tail duplication during placement. "
+                       "Creates more fallthrough opportunites in "
+                       "outline branches."),
+              cl::init(true), cl::Hidden);
 
 static cl::opt<bool>
 BranchFoldPlacement("branch-fold-placement",
               cl::desc("Perform branch folding during placement. "
                        "Reduces code size."),
               cl::init(true), cl::Hidden);
 
+// Heuristic for tail duplication.
+static cl::opt<unsigned> TailDuplicateMergeThresholdPlacement(
+    "tail-dup-merge-threshold-placement",
+    cl::desc("Instruction cutoff to consider tail duplicating vs merging"), cl::init(2),
+    cl::Hidden);
+
 extern cl::opt<unsigned> StaticLikelyProb;
 extern cl::opt<unsigned> ProfileLikelyProb;
 
 namespace {
 class BlockChain;
 /// \brief Type for our function-wide basic block -> block chain mapping.
 typedef DenseMap<MachineBasicBlock *, BlockChain *> BlockToChainMapType;
 }
public:
   typedef SmallVectorImpl<MachineBasicBlock *>::iterator iterator;
 
   /// \brief Beginning of blocks within the chain.
   iterator begin() { return Blocks.begin(); }
 
   /// \brief End of blocks within the chain.
   iterator end() { return Blocks.end(); }
 
+  bool remove(MachineBasicBlock* BB) {
+    for(iterator i = begin(); i != end(); ++i) {
+      if (*i == BB) {
+        Blocks.erase(i);
+        return true;
+      }
+    }
+    return false;
+  }
+
   /// \brief Merge a block chain into this one.
   ///
   /// This routine merges a block chain into this one. It takes care of forming
   /// a contiguous sequence of basic blocks, updating the edge list, and
   /// updating the block -> chain mapping. It does not free or tear down the
   /// old chain, but the old chain's block list is no longer valid.
   void merge(MachineBasicBlock *BB, BlockChain *Chain) {
     assert(BB);
public:
   }
 
 #ifndef NDEBUG
   /// \brief Dump the blocks in this chain.
   LLVM_DUMP_METHOD void dump() {
     for (MachineBasicBlock *MBB : *this)
       MBB->dump();
   }
+
+  void dump_brief() {
+    iterator i = begin();
+    dbgs() << "[";
+    if (i != end())
+      dbgs() << "#" << (*i)->getNumber();
+    for (++i; i != end(); ++i)
+      dbgs() << ", #" << (*i)->getNumber();
+    dbgs() << "]\n";
+  }
 #endif // NDEBUG
 
   /// \brief Count of predecessors of any block within the chain which have not
   /// yet been scheduled.  In general, we will delay scheduling this chain
   /// until those predecessors are scheduled (or we find a sufficiently good
   /// reason to override this heuristic.)  Note that when forming loop chains,
   /// blocks outside the loop are ignored and treated as if they were already
   /// scheduled.
class MachineBlockPlacement : public MachineFunctionPass {
   const TargetInstrInfo *TII;
 
   /// \brief A handle to the target's lowering info.
   const TargetLoweringBase *TLI;
 
   /// \brief A handle to the post dominator tree.
   MachineDominatorTree *MDT;
 
+  /// \brief Duplicator used to duplicate tails during placement.
+  ///
+  /// Placement decisions can open up new tail duplication opportunities, but
+  /// since tail duplication affects placement decisions of later blocks, it
+  /// must be done inline.
+  TailDuplicator TailDup;
+
   /// \brief A set of blocks that are unavoidably execute, i.e. they dominate
   /// all terminators of the MachineFunction.
   SmallPtrSet<MachineBasicBlock *, 4> UnavoidableBlocks;
 
   /// \brief Allocator and owner of BlockChain structures.
   ///
   /// We build BlockChains lazily while processing the loop structure of
   /// a function. To reduce malloc traffic, we allocate them using this
   /// slab-like allocator, and destroy them after the pass completes. An
   /// important guarantee is that this allocator produces stable pointers to
   /// the chains.
   SpecificBumpPtrAllocator<BlockChain> ChainAllocator;
 
   /// \brief Function wide BasicBlock to BlockChain mapping.
   ///
   /// This mapping allows efficiently moving from any given basic block to the
   /// BlockChain it participates in, if any. We use it to, among other things,
   /// allow implicitly defining edges between chains as the existing edges
   /// between basic blocks.
   DenseMap<MachineBasicBlock *, BlockChain *> BlockToChain;
 
+  /// Decrease the UnscheduledPredecessors count for all blocks in chain, and
+  /// if the count goes to 0, add them to the appropriate work list.
   void markChainSuccessors(BlockChain &Chain, MachineBasicBlock *LoopHeaderBB,
                            const BlockFilterSet *BlockFilter = nullptr);
+
+  /// Decrease the UnscheduledPredecessors count for a single block, and
+  /// if the count goes to 0, add them to the appropriate work list.
+  void markBlockSuccessors(
+      BlockChain &Chain, MachineBasicBlock *BB, MachineBasicBlock *LoopHeaderBB,
+      const BlockFilterSet *BlockFilter = nullptr);
+
+
   BranchProbability
   collectViableSuccessors(MachineBasicBlock *BB, BlockChain &Chain,
                           const BlockFilterSet *BlockFilter,
                           SmallVector<MachineBasicBlock *, 4> &Successors);
   bool shouldPredBlockBeOutlined(MachineBasicBlock *BB, MachineBasicBlock *Succ,
                                  BlockChain &Chain,
                                  const BlockFilterSet *BlockFilter,
                                  BranchProbability SuccProb,
                                  BranchProbability HotProb);
+  void maybeTailDuplicateBlock(MachineBasicBlock *BB, MachineBasicBlock *LPred,
+                               const BlockChain &Chain,
+                               BlockFilterSet *BlockFilter,
+                               MachineFunction::iterator &PrevUnplacedBlockIt,
+                               bool &Removed, bool &DuplicatedToPred);
   bool
   hasBetterLayoutPredecessor(MachineBasicBlock *BB, MachineBasicBlock *Succ,
                              BlockChain &SuccChain, BranchProbability SuccProb,
                              BranchProbability RealSuccProb, BlockChain &Chain,
                              const BlockFilterSet *BlockFilter);
   MachineBasicBlock *selectBestSuccessor(MachineBasicBlock *BB,
                                          BlockChain &Chain,
                                          const BlockFilterSet *BlockFilter);
class MachineBlockPlacement : public MachineFunctionPass {
   ///
   /// If the optional parameter BlockFilter is provided, only MBB
   /// present in the set will be added to the worklist. If nullptr
   /// is provided, no filtering occurs.
   void fillWorkLists(MachineBasicBlock *MBB,
                      SmallPtrSetImpl<BlockChain *> &UpdatedPreds,
                      const BlockFilterSet *BlockFilter);
   void buildChain(MachineBasicBlock *BB, BlockChain &Chain,
-                  const BlockFilterSet *BlockFilter = nullptr);
+                  BlockFilterSet *BlockFilter = nullptr);
   MachineBasicBlock *findBestLoopTop(MachineLoop &L,
                                      const BlockFilterSet &LoopBlockSet);
   MachineBasicBlock *findBestLoopExit(MachineLoop &L,
                                       const BlockFilterSet &LoopBlockSet);
   BlockFilterSet collectLoopBlockSet(MachineLoop &L);
   void buildLoopChains(MachineLoop &L);
   void rotateLoop(BlockChain &LoopChain, MachineBasicBlock *ExitingBB,
                   const BlockFilterSet &LoopBlockSet);

 }
 #endif
 
 /// \brief Mark a chain's successors as having one fewer preds.
 ///
 /// When a chain is being merged into the "placed" chain, this routine will
 /// quickly walk the successors of each block in the chain and mark them as
 /// having one fewer active predecessor. It also adds any successors of this
-/// chain which reach the zero-predecessor state to the worklist passed in.
+/// chain which reach the zero-predecessor state to the appropriate worklist.
 void MachineBlockPlacement::markChainSuccessors(
     BlockChain &Chain, MachineBasicBlock *LoopHeaderBB,
     const BlockFilterSet *BlockFilter) {
   // Walk all the blocks in this chain, marking their successors as having
   // a predecessor placed.
   for (MachineBasicBlock *MBB : Chain) {
-    // Add any successors for which this is the only un-placed in-loop
-    // predecessor to the worklist as a viable candidate for CFG-neutral
-    // placement. No subsequent placement of this block will violate the CFG
-    // shape, so we get to use heuristics to choose a favorable placement.
-    for (MachineBasicBlock *Succ : MBB->successors()) {
-      if (BlockFilter && !BlockFilter->count(Succ))
-        continue;
-      BlockChain &SuccChain = *BlockToChain[Succ];
-      // Disregard edges within a fixed chain, or edges to the loop header.
-      if (&Chain == &SuccChain || Succ == LoopHeaderBB)
-        continue;
+    markBlockSuccessors(Chain, MBB, LoopHeaderBB, BlockFilter);
+  }
+}
+
+/// \brief Mark a single block's successors as having one fewer preds.
+///
+/// Under normal circumstances, this is only called by markChainSuccessors,
+/// but if a block that was to be placed is completely tail-duplicated away,
+/// and was duplicated into the chain end, we need to redo markBlockSuccessors
+/// for just that block.
+void MachineBlockPlacement::markBlockSuccessors(
+    BlockChain &Chain, MachineBasicBlock *MBB, MachineBasicBlock *LoopHeaderBB,
+    const BlockFilterSet *BlockFilter) {
 
-      // This is a cross-chain edge that is within the loop, so decrement the
-      // loop predecessor count of the destination chain.
-      if (SuccChain.UnscheduledPredecessors == 0 ||
-          --SuccChain.UnscheduledPredecessors > 0)
-        continue;
 
-      auto *MBB = *SuccChain.begin();
-      if (MBB->isEHPad())
-        EHPadWorkList.push_back(MBB);
-      else
-        BlockWorkList.push_back(MBB);
-    }
-  }
-}
 
 /// This helper function collects the set of successors of block
 /// \p BB that are allowed to be its layout successors, and return
 /// the total branch probability of edges from \p BB to those
 /// blocks.
 BranchProbability MachineBlockPlacement::collectViableSuccessors(
     MachineBasicBlock *BB, BlockChain &Chain, const BlockFilterSet *BlockFilter,
     SmallVector<MachineBasicBlock *, 4> &Successors) {
void MachineBlockPlacement::fillWorkLists(
   if (MBB->isEHPad())
     EHPadWorkList.push_back(MBB);
   else
     BlockWorkList.push_back(MBB);
 }
 
 void MachineBlockPlacement::buildChain(
     MachineBasicBlock *BB, BlockChain &Chain,
-    const BlockFilterSet *BlockFilter) {
+    BlockFilterSet *BlockFilter) {
   assert(BB && "BB must not be null.\n");
   assert(BlockToChain[BB] == &Chain && "BlockToChainMap mis-match.\n");
   MachineFunction::iterator PrevUnplacedBlockIt = F->begin();
 
   MachineBasicBlock *LoopHeaderBB = BB;
   markChainSuccessors(Chain, LoopHeaderBB, BlockFilter);
   BB = *std::prev(Chain.end());
   for (;;) {
if (!BestSucc) {
       BestSucc = getFirstUnplacedBlock(Chain, PrevUnplacedBlockIt, BlockFilter);
       if (!BestSucc)
         break;
 
       DEBUG(dbgs() << "Unnatural loop CFG detected, forcibly merging the "
                       "layout successor until the CFG reduces\n");
     }
 
+    // Placement may have changed tail duplication opportunities.
+    // Check for that now.
+    if (TailDupPlacement && BestSucc) {
+
+      bool Removed, DuplicatedToBB;
+      maybeTailDuplicateBlock(BestSucc, BB, Chain, BlockFilter,
+                              PrevUnplacedBlockIt,
+                              Removed, DuplicatedToBB);
+      if (Removed) {
+        if (DuplicatedToBB) {
+          // Do 2 things: If we duplicated into BB, we need to update
+          // unscheduled predecessors. Then try to duplicate again.
+          while (DuplicatedToBB) {
+            MachineBasicBlock *DupBB, *DupPred;
+            BlockChain::iterator ChainEnd = Chain.end();
+            DupBB = *(--ChainEnd);
+            // If we duplicated into DupPred, we need to update unscheduled
+            // predecessors. The way to do that is to update just DupPred, and
+            // not the whole chain.
+            markBlockSuccessors(Chain, DupBB, LoopHeaderBB, BlockFilter);
+            // Now try to duplicate again.
+            if (ChainEnd == Chain.begin())
+              break;
+            DupPred = *(--ChainEnd);
+            maybeTailDuplicateBlock(DupBB, DupPred, Chain, BlockFilter,
+                                    PrevUnplacedBlockIt,
+                                    Removed, DuplicatedToBB);
+            // If tail maybeTailDuplicateBlock removes DupBB, it will remove it
+            // from the chain also, leaving DupPred as the end of the chain.
+          }
+          BlockChain::iterator ChainEnd = Chain.end();
+          BB = *(--ChainEnd);
+        }
+        // If the chosen successor was duplicated into all its predecessors,
+        // don't bother laying it out, just go round the loop again with BB as
+        // the chain end.
+        continue;
+      }
+    }
+
     // Place this block, updating the datastructures to reflect its placement.
     BlockChain &SuccChain = *BlockToChain[BestSucc];
     // Zero out UnscheduledPredecessors for the successor we're about to merge in case
     // we selected a successor that didn't fit naturally into the CFG.
     SuccChain.UnscheduledPredecessors = 0;
     DEBUG(dbgs() << "Merging from " << getBlockName(BB) << " to "
                  << getBlockName(BestSucc) << "\n");
     markChainSuccessors(SuccChain, LoopHeaderBB, BlockFilter);
+
     Chain.merge(BestSucc, &SuccChain);
+    DEBUG(Chain.dump_brief());
     BB = *std::prev(Chain.end());
   }
 
   DEBUG(dbgs() << "Finished forming chain for header block "
                << getBlockName(*Chain.begin()) << "\n");
 }
 
 /// \brief Find the best loop top block for layout.
for (MachineBasicBlock *ChainBB : FunctionChain) {
     BranchProbability LayoutProb =
         MBPI->getEdgeProbability(LayoutPred, ChainBB);
     BlockFrequency LayoutEdgeFreq = MBFI->getBlockFreq(LayoutPred) * LayoutProb;
     if (LayoutEdgeFreq <= (Freq * ColdProb))
       ChainBB->setAlignment(Align);
   }
 }
 
+/// maybeTailDuplicateBlock - Tail duplicate Succ into (some) predecessors if
+/// profitable.
+/// @param BB    - Basic block that may be duplicated
+/// @param LPred - Chosen layout predecessor of Succ
+/// @param Chain - Chain to which LPred belongs, and BB will belong.
+/// @param BlockFilter - Set of blocks that belong to the loop being laid out.
+///                      Used to identify which blocks to update predecessor
+///                      counts.
+/// @param PrevUnplacedBlockIt - Iterator pointing to the last block that was
+///                              chosen in the given order due to unnatural CFG
+///                              only needed if BB is removed and
+///                              PrevUnplacedBlockIt pointed to BB.
+void MachineBlockPlacement::maybeTailDuplicateBlock(
+    MachineBasicBlock *BB, MachineBasicBlock *LPred,
+    const BlockChain &Chain, BlockFilterSet *BlockFilter,
+    MachineFunction::iterator &PrevUnplacedBlockIt,
+    bool &Removed, bool &DuplicatedToBB) {
+
+  Removed = false;
+  DuplicatedToBB = false;
+  bool CanDupToChain = false;
+  DEBUG(dbgs() << "Redoing tail duplication for Succ#"
+        << BB->getNumber() << "\n");
+  bool IsSimple = TailDup.isSimpleBB(BB);
+  // Blocks with single successors don't create additional fallthrough
+  // opportunities. Don't duplicate them. TODO: When conditional exits are
+  // analyzable, allow them to be duplicated.
+  if ((IsSimple || BB->succ_size() != 1) &&
+      TailDup.shouldTailDuplicate(*F, IsSimple, *BB)) {
+    // Update UnscheduledPredecessors to reflect tail-duplication.
+    for (MachineBasicBlock *Pred : BB->predecessors()) {
+      // We're only looking for unscheduled predecessors that match the filter.
+      BlockChain* PredChain = BlockToChain[Pred];
+      if (Pred == LPred || (BlockFilter && !BlockFilter->count(Pred))
+          || PredChain == &Chain)
+        continue;
+      if (TailDup.canTailDuplicate(BB, Pred)) {
+        for (MachineBasicBlock *NewSucc : BB->successors()) {
+          if (BlockFilter && !BlockFilter->count(NewSucc))
+            continue;
+          BlockChain *NewChain = BlockToChain[NewSucc];
+          if (NewChain != &Chain && NewChain != PredChain) {
+            DEBUG(dbgs() << "Bumping unscheduled pred count for chain:\n");
+            DEBUG(BlockToChain[NewSucc]->dump_brief());
+            DEBUG(dbgs() << "Because of edges: "
+                  << getBlockName(Pred) << " -> "
+                  << getBlockName(BB) << " -> "
+                  << getBlockName(NewSucc) << "\n");
+            BlockToChain[NewSucc]->UnscheduledPredecessors++;
+          }
+        }
+      }
+    }
+    CanDupToChain = TailDup.canTailDuplicate(BB, LPred);
+    // This has to be a callback because none of it can be done after
+    // BB is deleted.
+    auto RemovalCallback =
+        [&](MachineBasicBlock *RemBB) {
+          // Signal to outer function
+          Removed = true;
+
+          // Remove from the Chain and Chain Map
+          if (BlockToChain.count(RemBB)) {
+            BlockChain *Chain = BlockToChain[RemBB];
+            Chain->remove(RemBB);
+            BlockToChain.erase(RemBB);
+          }
+
+          // Handle the unplaced block iterator
+          if (&(*PrevUnplacedBlockIt) == RemBB) {
+            PrevUnplacedBlockIt++;
+          }
+
+          // Handle the Work Lists
+          SmallVectorImpl<MachineBasicBlock *> &RemoveList = BlockWorkList;
+          if (RemBB->isEHPad())
+            RemoveList = EHPadWorkList;
+          for (auto it = RemoveList.begin(); it != RemoveList.end(); ++it) {
+            if (*it == RemBB) {
+              RemoveList.erase(it);
+              break;
+            }
+          }
+
+          // Handle the filter set
+          if (BlockFilter) {
+            BlockFilter->erase(RemBB);
+          }
+
+          // TailDuplicator handles removing it from loops.
+          DEBUG(dbgs() << "TailDuplicator deleted block: "
+                << getBlockName(RemBB) << "\n");
+        };
+    auto RemovalCallbackRef =
+        llvm::function_ref<void(MachineBasicBlock*)>(RemovalCallback);
+    TailDup.tailDuplicateAndUpdate(*F, IsSimple, BB, LPred,
+                                   &RemovalCallbackRef);
+  }
+  DuplicatedToBB = Removed && CanDupToChain;
+}
+
 bool MachineBlockPlacement::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(*MF.getFunction()))
     return false;
 
   // Check for single-block functions and skip them.
   if (std::next(MF.begin()) == MF.end())
     return false;
 
   F = &MF;
   MBPI = &getAnalysis<MachineBranchProbabilityInfo>();
   MBFI = llvm::make_unique<BranchFolder::MBFIWrapper>(
       getAnalysis<MachineBlockFrequencyInfo>());
   MLI = &getAnalysis<MachineLoopInfo>();
   TII = MF.getSubtarget().getInstrInfo();
   TLI = MF.getSubtarget().getTargetLowering();
   MDT = &getAnalysis<MachineDominatorTree>();
+  auto MMI = getAnalysisIfAvailable<MachineModuleInfo>();
+  if (TailDupPlacement) {
+    unsigned TailDupSize = TailDuplicateMergeThresholdPlacement;
+    if (MF.getFunction()->optForSize())
+      TailDupSize = 1;
+    TailDup.initMF(MF, MMI, MBPI, MLI, /* LayoutMode */ true, TailDupSize);
+  }
+
   assert(BlockToChain.empty());
 
   buildCFGChains();
 
   // Changing the layout can create new tail merging opportunities.
   TargetPassConfig *PassConfig = &getAnalysis<TargetPassConfig>();
   // TailMerge can create jump into if branches that make CFG irreducible for
   // HW that requires structured CFG.
   bool EnableTailMerge = !MF.getTarget().requiresStructuredCFG() &&
                          PassConfig->getEnableTailMerge() &&
                          BranchFoldPlacement;
   // No tail merging opportunities if the block number is less than four.
   if (MF.size() > 3 && EnableTailMerge) {
-    // Default to the standard tail-merge-size option.
-    unsigned TailMergeSize = 0;
+    unsigned TailMergeSize = TailDuplicateMergeThresholdPlacement + 1;
     BranchFolder BF(/*EnableTailMerge=*/true, /*CommonHoist=*/false, *MBFI,
                     *MBPI, TailMergeSize);
 
+    DEBUG(MF.dump());
     if (BF.OptimizeFunction(MF, TII, MF.getSubtarget().getRegisterInfo(),
                             getAnalysisIfAvailable<MachineModuleInfo>(), MLI,
                             /*AfterBlockPlacement=*/true)) {
       // Redo the layout if tail merging creates/removes/moves blocks.
+      DEBUG(MF.dump());
       BlockToChain.clear();
+      // Must redo the dominator tree if blocks were changed.
+      MDT->runOnMachineFunction(MF);
       ChainAllocator.DestroyAll();
       buildCFGChains();
     }
   }
 
   optimizeBranches();
   alignBlocks();
 
   BlockToChain.clear();

lib/CodeGen/TailDuplication.cpp

 //===-- TailDuplication.cpp - Duplicate blocks into predecessors' tails ---===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // This pass duplicates basic blocks ending in unconditional branches into
 // the tails of their predecessors, using the TailDuplicator utility class.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/CodeGen/Passes.h"
 #include "llvm/CodeGen/TailDuplicator.h"
 #include "llvm/IR/Function.h"
 #include "llvm/Support/Debug.h"
 using namespace llvm;
 
 #define DEBUG_TYPE "tailduplication"
 
INITIALIZE_PASS(TailDuplicatePass, "tailduplication", "Tail Duplication", false,
                 false)
 
 bool TailDuplicatePass::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(*MF.getFunction()))
     return false;
 
   auto MMI = getAnalysisIfAvailable<MachineModuleInfo>();
   auto MBPI = &getAnalysis<MachineBranchProbabilityInfo>();
+  auto MLI = &getAnalysis<MachineLoopInfo>();
 
-  Duplicator.initMF(MF, MMI, MBPI);
+  Duplicator.initMF(MF, MMI, MBPI, MLI, /* LayoutMode */ false);
 
   bool MadeChange = false;
   while (Duplicator.tailDuplicateBlocks(MF))
     MadeChange = true;
 
   return MadeChange;
 }
 
 void TailDuplicatePass::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.addRequired<MachineBranchProbabilityInfo>();
+  AU.addRequired<MachineLoopInfo>();
   MachineFunctionPass::getAnalysisUsage(AU);
 }

lib/CodeGen/TailDuplicator.cpp

 #include "llvm/CodeGen/TailDuplicator.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/CodeGen/MachineBranchProbabilityInfo.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/Passes.h"
 #include "llvm/IR/Function.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/raw_ostream.h"
 using namespace llvm;

 
 static cl::opt<unsigned> TailDupLimit("tail-dup-limit", cl::init(~0U),
                                       cl::Hidden);
 
 namespace llvm {
 
 void TailDuplicator::initMF(MachineFunction &MF, const MachineModuleInfo *MMIin,
                             const MachineBranchProbabilityInfo *MBPIin,
-                            unsigned TailDupSizeIn) {
+                            MachineLoopInfo *MLIin,
+                            bool LayoutModeIn, unsigned TailDupSizeIn) {
   TII = MF.getSubtarget().getInstrInfo();
   TRI = MF.getSubtarget().getRegisterInfo();
   MRI = &MF.getRegInfo();
   MMI = MMIin;
   MBPI = MBPIin;
+  MLI = MLIin;
   TailDupSize = TailDupSizeIn;
 
   assert(MBPI != nullptr && "Machine Branch Probability Info required");
+  assert((LayoutModeIn || MLI != nullptr) &&
+         "Machine Loop Info required in non-layout mode.");
 
+  LayoutMode = LayoutModeIn;
   PreRegAlloc = MRI->isSSA();
 }
 
 static void VerifyPHIs(MachineFunction &MF, bool CheckExtra) {
   for (MachineFunction::iterator I = ++MF.begin(), E = MF.end(); I != E; ++I) {
     MachineBasicBlock *MBB = &*I;
     SmallSetVector<MachineBasicBlock *, 8> Preds(MBB->pred_begin(),
                                                  MBB->pred_end());
while (MI != MBB->end()) {
         }
       }
       ++MI;
     }
   }
 }
 
 /// Tail duplicate the block and cleanup.
-bool TailDuplicator::tailDuplicateAndUpdate(MachineFunction &MF, bool IsSimple,
-                                            MachineBasicBlock *MBB) {
+bool TailDuplicator::tailDuplicateAndUpdate(
+    MachineFunction &MF, bool IsSimple, MachineBasicBlock *MBB,
+    MachineBasicBlock *ForcedLayoutPred,
+    llvm::function_ref<void(MachineBasicBlock *)> *RemovalCallback) {
   // Save the successors list.
   SmallSetVector<MachineBasicBlock *, 8> Succs(MBB->succ_begin(),
                                                MBB->succ_end());
 
   SmallVector<MachineBasicBlock *, 8> TDBBs;
   SmallVector<MachineInstr *, 16> Copies;
-  if (!tailDuplicate(MF, IsSimple, MBB, TDBBs, Copies))
+  if (!tailDuplicate(MF, IsSimple, MBB, ForcedLayoutPred, TDBBs, Copies))
     return false;
 
   ++NumTails;
 
   SmallVector<MachineInstr *, 8> NewPHIs;
   MachineSSAUpdater SSAUpdate(MF, &NewPHIs);
 
   // TailBB's immediate successors are now successors of those predecessors
   // which duplicated TailBB. Add the predecessors as sources to the PHI
   // instructions.
   bool isDead = MBB->pred_empty() && !MBB->hasAddressTaken();
   if (PreRegAlloc)
     updateSuccessorsPHIs(MBB, isDead, TDBBs, Succs);
 
   // If it is dead, remove it.
   if (isDead) {
     NumTailDupRemoved += MBB->size();
-    removeDeadBlock(MBB);
+    removeDeadBlock(MBB, RemovalCallback);
     ++NumDeadBlocks;
   }
 
   // Update SSA form.
   if (!SSAUpdateVRs.empty()) {
     for (unsigned i = 0, e = SSAUpdateVRs.size(); i != e; ++i) {
       unsigned VReg = SSAUpdateVRs[i];
       SSAUpdate.Initialize(VReg);
for (MachineFunction::iterator I = ++MF.begin(), E = MF.end(); I != E;) {
     if (NumTails == TailDupLimit)
       break;
 
     bool IsSimple = isSimpleBB(MBB);
 
     if (!shouldTailDuplicate(MF, IsSimple, *MBB))
       continue;
 
-    MadeChange |= tailDuplicateAndUpdate(MF, IsSimple, MBB);
+    MadeChange |= tailDuplicateAndUpdate(MF, IsSimple, MBB, nullptr);
   }
 
   if (PreRegAlloc && TailDupVerify)
     VerifyPHIs(MF, false);
 
   return MadeChange;
 }
 
for (MachineInstr &MI : *SuccBB) {
         MI.RemoveOperand(Idx + 1);
         MI.RemoveOperand(Idx);
       }
     }
   }
 }
 
 /// Determine if it is profitable to duplicate this block.
 bool TailDuplicator::shouldTailDuplicate(const MachineFunction &MF,
                                          bool IsSimple,
                                          MachineBasicBlock &TailBB) {
-  // Only duplicate blocks that end with unconditional branches.
-  if (TailBB.canFallThrough())
+  // When doing tail-duplication during layout, the block ordering is in flux,
+  // so canFallThrough returns a result based on incorrect information and
+  // should just be ignored.
+  if (!LayoutMode && TailBB.canFallThrough())
     return false;
 
   // Don't try to tail-duplicate single-block loops.
   if (TailBB.isSuccessor(&TailBB))
     return false;
 
   // Set the limit on the cost to duplicate. When optimizing for size,
   // duplicate only one, because one branch instruction can be eliminated to
   // compensate for the duplication.
   unsigned MaxDuplicateCount;
   if (TailDupSize == 0 &&
       TailDuplicateSize.getNumOccurrences() == 0 &&
bool TailDuplicator::shouldTailDuplicate(const MachineFunction &MF,
   // A similar check is necessary in MachineBlockPlacement to make sure pairs of
   // blocks with unanalyzable fallthrough get layed out contiguously.
   MachineBasicBlock *PredTBB = nullptr, *PredFBB = nullptr;
   SmallVector<MachineOperand, 4> PredCond;
   if (TII->analyzeBranch(TailBB, PredTBB, PredFBB, PredCond, true)
       && TailBB.canFallThrough())
     return false;
 
+  // When tail-duplicating relatively early, we shouldnt' de-normalize loops by
+  // duplicating latches, headers, or loop-predecessors. This interferes with
+  // later optimizations. During layout this is fine.
+  if (!LayoutMode) {
+    MachineLoop *ML = MLI->getLoopFor(&TailBB);
+    if (ML && (&TailBB == ML->getHeader() ||
+               &TailBB == ML->getLoopLatch()))
+      return false;
+    for (auto SB : TailBB.successors()) {
+      MachineLoop *SML = MLI->getLoopFor(SB);
+      if (SML && (&TailBB == SML->getLoopPredecessor()))
+        return false;
+    }
+  }
+
   // If the target has hardware branch prediction that can handle indirect
   // branches, duplicating them can often make them predictable when there
   // are common paths through the code.  The limit needs to be high enough
   // to allow undoing the effects of tail merging and other optimizations
   // that rearrange the predecessors of the indirect branch.
 
   bool HasIndirectbr = false;
-  if (!TailBB.empty())
-    HasIndirectbr = TailBB.back().isIndirectBranch();
+  if (!TailBB.empty()) {
+    MachineInstr &back = TailBB.back();
+    // On some platforms return instructions are marked as indirect branches and
+    // not as return instructions. Work around this by checking for return via
+    // the CFG.
+    HasIndirectbr = back.isIndirectBranch() && TailBB.succ_size() != 0;
+  }
 
-  if (HasIndirectbr && PreRegAlloc)
+  if (HasIndirectbr && (PreRegAlloc || LayoutMode))
     MaxDuplicateCount = 20;
 
   // Check the instructions in the block to determine whether tail-duplication
   // is invalid or unlikely to be profitable.
   unsigned InstrCount = 0;
   for (MachineInstr &MI : TailBB) {
     // Non-duplicable things shouldn't be tail-duplicated.
     if (MI.isNotDuplicable())
for (MachineBasicBlock *PredBB : Preds) {
 
     TDBBs.push_back(PredBB);
   }
   return Changed;
 }
 
 bool TailDuplicator::canTailDuplicate(MachineBasicBlock *TailBB,
                                       MachineBasicBlock *PredBB) {
-  // EH edges are ignored by AnalyzeBranch.
+  // EH edges are ignored by analyzeBranch.
   if (PredBB->succ_size() > 1)
     return false;
 
   MachineBasicBlock *PredTBB, *PredFBB;
   SmallVector<MachineOperand, 4> PredCond;
   if (TII->analyzeBranch(*PredBB, PredTBB, PredFBB, PredCond, true))
     return false;
   if (!PredCond.empty())
     return false;
   return true;
 }
 
 /// If it is profitable, duplicate TailBB's contents in each
 /// of its predecessors.
+/// ForcedLayoutPred  When non-null, use this block as the layout predecessor
+///                   instead of the previous block in MF's order.
+/// TDBBs             A vector to keep track of all blocks tail-duplicated into.
+/// Copies            A vector of copy instructions inserted. Used later to walk
+///                   all the inserted copies and remove redundant ones.
 bool TailDuplicator::tailDuplicate(MachineFunction &MF, bool IsSimple,
                                    MachineBasicBlock *TailBB,
+                                   MachineBasicBlock *ForcedLayoutPred,
                                    SmallVectorImpl<MachineBasicBlock *> &TDBBs,
                                    SmallVectorImpl<MachineInstr *> &Copies) {
   DEBUG(dbgs() << "\n*** Tail-duplicating BB#" << TailBB->getNumber() << '\n');
 
   DenseSet<unsigned> UsedByPhi;
   getRegsUsedByPHIs(*TailBB, &UsedByPhi);
 
   if (IsSimple)
     return duplicateSimpleBB(TailBB, TDBBs, UsedByPhi, Copies);
 
   // Iterate through all the unique predecessors and tail-duplicate this
   // block into them, if possible. Copying the list ahead of time also
   // avoids trouble with the predecessor list reallocating.
   bool Changed = false;
   SmallSetVector<MachineBasicBlock *, 8> Preds(TailBB->pred_begin(),
                                                TailBB->pred_end());
   for (MachineBasicBlock *PredBB : Preds) {
     assert(TailBB != PredBB &&
            "Single-block loop should have been rejected earlier!");
 
     if (!canTailDuplicate(TailBB, PredBB))
       continue;
 
     // Don't duplicate into a fall-through predecessor (at least for now).
-    if (PredBB->isLayoutSuccessor(TailBB) && PredBB->canFallThrough())
+    bool IsLayoutSuccessor = false;
+    if (ForcedLayoutPred)
+      IsLayoutSuccessor = (ForcedLayoutPred == PredBB);
+    else if (PredBB->isLayoutSuccessor(TailBB) && PredBB->canFallThrough())
+      IsLayoutSuccessor = true;
+    if (IsLayoutSuccessor)
       continue;
 
     DEBUG(dbgs() << "\nTail-duplicating into PredBB: " << *PredBB
                  << "From Succ: " << *TailBB);
 
     TDBBs.push_back(PredBB);
 
     // Remove PredBB's unconditional branch.
for (MachineBasicBlock *Succ : TailBB->successors())
       PredBB->addSuccessor(Succ, MBPI->getEdgeProbability(TailBB, Succ));
 
     Changed = true;
     ++NumTailDups;
   }
 
   // If TailBB was duplicated into all its predecessors except for the prior
   // block, which falls through unconditionally, move the contents of this
   // block into the prior block.
-  MachineBasicBlock *PrevBB = &*std::prev(TailBB->getIterator());
+  MachineBasicBlock *PrevBB = ForcedLayoutPred;
+  if (!PrevBB)
+    PrevBB = &*std::prev(TailBB->getIterator());
   MachineBasicBlock *PriorTBB = nullptr, *PriorFBB = nullptr;
   SmallVector<MachineOperand, 4> PriorCond;
   // This has to check PrevBB->succ_size() because EH edges are ignored by
-  // AnalyzeBranch.
+  // analyzeBranch.
   if (PrevBB->succ_size() == 1 &&
       // Layout preds are not always CFG preds. Check.
       *PrevBB->succ_begin() == TailBB &&
       !TII->analyzeBranch(*PrevBB, PriorTBB, PriorFBB, PriorCond, true) &&
-      PriorCond.empty() && !PriorTBB && TailBB->pred_size() == 1 &&
+      PriorCond.empty() &&
+      (!PriorTBB || PriorTBB == TailBB) &&
+      TailBB->pred_size() == 1 &&
       !TailBB->hasAddressTaken()) {
     DEBUG(dbgs() << "\nMerging into block: " << *PrevBB
                  << "From MBB: " << *TailBB);
     if (PreRegAlloc) {
       DenseMap<unsigned, RegSubRegPair> LocalVRMap;
       SmallVector<std::pair<unsigned, RegSubRegPair>, 4> CopyInfos;
       MachineBasicBlock::iterator I = TailBB->begin();
       // Process PHI instructions first.
if (PreRegAlloc) {
         // uses with PHI source register or the new registers.
         MachineInstr *MI = &*I++;
         assert(!MI->isBundle() && "Not expecting bundles before regalloc!");
         duplicateInstruction(MI, TailBB, PrevBB, MF, LocalVRMap, UsedByPhi);
         MI->eraseFromParent();
       }
       appendCopies(PrevBB, CopyInfos, Copies);
     } else {
+      TII->RemoveBranch(*PrevBB);
       // No PHIs to worry about, just splice the instructions over.
       PrevBB->splice(PrevBB->end(), TailBB, TailBB->begin(), TailBB->end());
     }
     PrevBB->removeSuccessor(PrevBB->succ_begin());
     assert(PrevBB->succ_empty());
     PrevBB->transferSuccessors(TailBB);
     TDBBs.push_back(PrevBB);
     Changed = true;
for (auto &CI : CopyInfos) {
     auto C = BuildMI(*MBB, Loc, DebugLoc(), CopyD, CI.first)
                 .addReg(CI.second.Reg, 0, CI.second.SubReg);
     Copies.push_back(C);
   }
 }
 
 /// Remove the specified dead machine basic block from the function, updating
 /// the CFG.
-void TailDuplicator::removeDeadBlock(MachineBasicBlock *MBB) {
+void TailDuplicator::removeDeadBlock(
+    MachineBasicBlock *MBB,
+    llvm::function_ref<void(MachineBasicBlock *)> *RemovalCallback) {
   assert(MBB->pred_empty() && "MBB must be dead!");
   DEBUG(dbgs() << "\nRemoving MBB: " << *MBB);
 
+  if (RemovalCallback)
+    (*RemovalCallback)(MBB);
+  if (MLI)
+    MLI->removeBlock(MBB);
+
   // Remove all successors.
   while (!MBB->succ_empty())
     MBB->removeSuccessor(MBB->succ_end() - 1);
 
   // Remove the block.
   MBB->eraseFromParent();
 }
 
 } // End llvm namespace

test/CodeGen/AArch64/arm64-extload-knownzero.ll

 ; RUN: llc < %s -mtriple=arm64-eabi | FileCheck %s
 ; rdar://12771555
 
 define void @foo(i16* %ptr, i32 %a) nounwind {
 entry:
 ; CHECK-LABEL: foo:
   %tmp1 = icmp ult i32 %a, 100
   br i1 %tmp1, label %bb1, label %bb2
 bb1:
 ; CHECK: %bb1
 ; CHECK: ldrh [[REG:w[0-9]+]]
   %tmp2 = load i16, i16* %ptr, align 2
   br label %bb2
 bb2:
-; CHECK: %bb2
 ; CHECK-NOT: and {{w[0-9]+}}, [[REG]], #0xffff
 ; CHECK: cmp [[REG]], #23
   %tmp3 = phi i16 [ 0, %entry ], [ %tmp2, %bb1 ]
   %cmp = icmp ult i16 %tmp3, 24
   br i1 %cmp, label %bb3, label %exit
 bb3:
   call void @bar() nounwind
   br label %exit
 exit:
   ret void
 }
 
 declare void @bar ()

test/CodeGen/AArch64/machine_cse.ll

-; RUN: llc < %s -mtriple=aarch64-linux-gnuabi -O2 | FileCheck %s
+; RUN: llc < %s -mtriple=aarch64-linux-gnuabi -O2 -tail-dup-placement=0 | FileCheck %s
+; -tail-dup-placement causes tail duplication during layout. This breaks the
+; assumptions of the test case as written (specifically, it creates an
+; additional cmp instruction, creating a false positive), so we pass
+; -tail-dup-placement=0 to restore the original behavior
 
 ; marked as external to prevent possible optimizations
 @a = external global i32
 @b = external global i32
 @c = external global i32
 @d = external global i32
 @e = external global i32
 

test/CodeGen/ARM/2011-03-23-PeepholeBug.ll

 bb1:                                              ; preds = %bb
 ; CHECK: bb1
 ; CHECK: subs [[REG:r[0-9]+]], #1
   %tmp = tail call i32 @puts() nounwind
   %indvar.next = add i32 %indvar, 1
   br label %bb2
 
 bb2:                                              ; preds = %bb1, %entry
-; CHECK: bb2
 ; CHECK: cmp [[REG]], #0
 ; CHECK: ble
   %indvar = phi i32 [ %indvar.next, %bb1 ], [ 0, %entry ]
   %tries.0 = sub i32 2147483647, %indvar
   %tmp1 = icmp sgt i32 %tries.0, 0
   br i1 %tmp1, label %bb, label %bb3
 
 bb3:                                              ; preds = %bb2, %bb
   ret i32 0
 }
 
 declare i32 @rand()
 
 declare i32 @puts() nounwind

test/CodeGen/PowerPC/branch-opt.ll

-; RUN: llc -verify-machineinstrs < %s -march=ppc32 | \
-; RUN:   grep "b LBB.*" | count 4
+; RUN: llc -verify-machineinstrs < %s -march=ppc32 | FileCheck %s
 
 target datalayout = "E-p:32:32"
 target triple = "powerpc-apple-darwin8.7.0"
 
+;CHECK-LABEL: foo:
+; There are 3 inner loops (%bb, %bb12, %bb25) that all exit to %cond_next48
+; The last (whichever it is) should have a fallthrough exit, and the other two
+; need an unconditional branch. No other block should have an unconditional
+; branch to cond_next48
+;CHECK: b LBB0_13
+;CHECK: b LBB0_13
+;CHECK-NOT: b LBB0_13
+;CHECK: LBB0_13: ; %cond_next48
+
 define void @foo(i32 %W, i32 %X, i32 %Y, i32 %Z) {
 entry:
   %tmp1 = and i32 %W, 1   ; <i32> [#uses=1]
   %tmp1.upgrd.1 = icmp eq i32 %tmp1, 0    ; <i1> [#uses=1]
   br i1 %tmp1.upgrd.1, label %cond_false, label %bb5
 bb:   ; preds = %bb5, %bb
   %indvar77 = phi i32 [ %indvar.next78, %bb ], [ 0, %bb5 ]    ; <i32> [#uses=1]
   %tmp2 = tail call i32 (...) @bar( )    ; <i32> [#uses=0]

test/CodeGen/PowerPC/sjlj.ll

 ; CHECK-DAG: stvx
 
 ; CHECK-DAG: addis [[REG:[0-9]+]], 2, env_sigill@toc@ha
 ; CHECK-DAG: std 31, env_sigill@toc@l([[REG]])
 ; CHECK-DAG: addi [[REGA:[0-9]+]], [[REG]], env_sigill@toc@l
 ; CHECK-DAG: std [[REGA]], [[OFF:[0-9]+]](31)                  # 8-byte Folded Spill
 ; CHECK-DAG: std 1, 16([[REGA]])
 ; CHECK-DAG: std 2, 24([[REGA]])
-; CHECK: bcl 20, 31, .LBB1_5
+; CHECK: bcl 20, 31, .LBB1_3
 ; CHECK: li 3, 1
-; CHECK: #EH_SjLj_Setup .LBB1_5
+; CHECK: #EH_SjLj_Setup .LBB1_3
 ; CHECK: b .LBB1_1
 
-; CHECK: .LBB1_4:
+; CHECK: .LBB1_3:
+; CHECK: mflr [[REGL:[0-9]+]]
+; CHECK: ld [[REG2:[0-9]+]], [[OFF]](31)                   # 8-byte Folded Reload
+; CHECK: std [[REGL]], 8([[REG2]])
+; CHECK: li 3, 0
+
+; CHECK: .LBB1_5:
 
 ; CHECK: lfd
 ; CHECK: lvx
 ; CHECK: ld
 ; CHECK: blr
 
-; CHECK: .LBB1_5:
-; CHECK: mflr [[REGL:[0-9]+]]
-; CHECK: ld [[REG2:[0-9]+]], [[OFF]](31)                   # 8-byte Folded Reload
-; CHECK: std [[REGL]], 8([[REG2]])
-; CHECK: li 3, 0
-
 ; CHECK-NOAV: @main
 ; CHECK-NOAV-NOT: stvx
 ; CHECK-NOAV: bcl
 ; CHECK-NOAV: mflr
 ; CHECK-NOAV: bl foo
 ; CHECK-NOAV-NOT: lvx
 ; CHECK-NOAV: blr
 }

test/CodeGen/PowerPC/tail-dup-layout.ll

+; RUN: llc -outline-optional-branches -O2 < %s | FileCheck %s
+target datalayout = "e-m:e-i64:64-n32:64"
+target triple = "powerpc64le-grtev4-linux-gnu"
+
+; Intended layout:
+; The outlining flag produces the layout
+; test1
+; test2
+; test3
+; test4
+; exit
+; optional1
+; optional2
+; optional3
+; optional4
+; Tail duplication puts test n+1 at the end of optional n
+; so optional1 includes a copy of test2 at the end, and branches
+; to test3 (at the top) or falls through to optional 2.
+; The CHECK statements check for the whole string of tests and exit block,
+; and then check that the correct test has been duplicated into the end of
+; the optional blocks and that the optional blocks are in the correct order.
+;CHECK-LABEL: f:
+; test1 may have been merged with entry
+;CHECK: mr [[TAGREG:[0-9]+]], 3
+;CHECK: andi. {{[0-9]+}}, [[TAGREG]], 1
+;CHECK-NEXT: bc 12, 1, [[OPT1LABEL:[._0-9A-Za-z]+]]
+;CHECK-NEXT: [[TEST2LABEL:[._0-9A-Za-z]+]]: # %test2
+;CHECK-NEXT: rlwinm. {{[0-9]+}}, [[TAGREG]], 0, 30, 30
+;CHECK-NEXT: bne 0, [[OPT2LABEL:[._0-9A-Za-z]+]]
+;CHECK-NEXT: [[TEST3LABEL:[._0-9A-Za-z]+]]: # %test3
+;CHECK-NEXT: rlwinm. {{[0-9]+}}, [[TAGREG]], 0, 29, 29
+;CHECK-NEXT: bne 0, .[[OPT3LABEL:[._0-9A-Za-z]+]]
+;CHECK-NEXT: [[TEST4LABEL:[._0-9A-Za-z]+]]: # %test4
+;CHECK-NEXT: rlwinm. {{[0-9]+}}, [[TAGREG]], 0, 28, 28
+;CHECK-NEXT: bne 0, .[[OPT4LABEL:[._0-9A-Za-z]+]]
+;CHECK-NEXT: [[EXITLABEL:[._0-9A-Za-z]+]]: # %exit
+;CHECK: blr
+;CHECK-NEXT: [[OPT1LABEL]]
+;CHECK: rlwinm. {{[0-9]+}}, [[TAGREG]], 0, 30, 30
+;CHECK-NEXT: beq 0, [[TEST3LABEL]]
+;CHECK-NEXT: [[OPT2LABEL]]
+;CHECK: rlwinm. {{[0-9]+}}, [[TAGREG]], 0, 29, 29
+;CHECK-NEXT: beq 0, [[TEST4LABEL]]
+;CHECK-NEXT: [[OPT3LABEL]]
+;CHECK: rlwinm. {{[0-9]+}}, [[TAGREG]], 0, 28, 28
+;CHECK-NEXT: beq 0, [[EXITLABEL]]
+;CHECK-NEXT: [[OPT4LABEL]]
+;CHECK: b [[EXITLABEL]]
+
+define void @f(i32 %tag) {
+entry:
+  br label %test1
+test1:
+  %tagbit1 = and i32 %tag, 1
+  %tagbit1eq0 = icmp eq i32 %tagbit1, 0
+  br i1 %tagbit1eq0, label %test2, label %optional1
+optional1:
+  call void @a()
+  call void @a()
+  call void @a()
+  call void @a()
+  br label %test2
+test2:
+  %tagbit2 = and i32 %tag, 2
+  %tagbit2eq0 = icmp eq i32 %tagbit2, 0
+  br i1 %tagbit2eq0, label %test3, label %optional2
+optional2:
+  call void @b()
+  call void @b()
+  call void @b()
+  call void @b()
+  br label %test3
+test3:
+  %tagbit3 = and i32 %tag, 4
+  %tagbit3eq0 = icmp eq i32 %tagbit3, 0
+  br i1 %tagbit3eq0, label %test4, label %optional3
+optional3:
+  call void @c()
+  call void @c()
+  call void @c()
+  call void @c()
+  br label %test4
+test4:
+  %tagbit4 = and i32 %tag, 8
+  %tagbit4eq0 = icmp eq i32 %tagbit4, 0
+  br i1 %tagbit4eq0, label %exit, label %optional4
+optional4:
+  call void @d()
+  call void @d()
+  call void @d()
+  call void @d()
+  br label %exit
+exit:
+  ret void
+}
+
+declare void @a()
+declare void @b()
+declare void @c()
+declare void @d()

test/CodeGen/WebAssembly/cfg-stackify.ll

 ; RUN: llc < %s -asm-verbose=false -disable-wasm-fallthrough-return-opt -disable-block-placement -verify-machineinstrs -fast-isel=false | FileCheck %s
-; RUN: llc < %s -asm-verbose=false -disable-wasm-fallthrough-return-opt -verify-machineinstrs -fast-isel=false | FileCheck -check-prefix=OPT %s
+; RUN: llc < %s -asm-verbose=false -disable-wasm-fallthrough-return-opt -tail-dup-placement=0 -verify-machineinstrs -fast-isel=false | FileCheck -check-prefix=OPT %s
 
 ; Test the CFG stackifier pass.
 
 ; Explicitly disable fast-isel, since it gets implicitly enabled in the
 ; optnone test.
 
 target datalayout = "e-m:e-p:32:32-i64:64-n32:64-S128"
 target triple = "wasm32-unknown-unknown"
exit:
   store volatile i32 4, i32* %p
   ret i32 0
 }
 
 ; CHECK-LABEL: doublediamond_in_a_loop:
 ; CHECK: .LBB11_1:
 ; CHECK: loop{{$}}
 ; CHECK: block{{$}}
+; CHECK: block{{$}}
 ; CHECK: br_if           0, $0{{$}}
 ; CHECK: br              1{{$}}
 ; CHECK: .LBB11_3:
-; CHECK: end_block{{$}}
+; CHECK-NEXT: end_block{{$}}
 ; CHECK: block{{$}}
 ; CHECK: br_if           0, $1{{$}}
 ; CHECK: br              1{{$}}
 ; CHECK: .LBB11_5:
-; CHECK: br              0{{$}}
+; CHECK-NEXT: end_block{{$}}
 ; CHECK: .LBB11_6:
+; CHECK-NEXT: end_block{{$}}
+; CHECK: br              0{{$}}
+; CHECK: .LBB11_7:
 ; CHECK-NEXT: end_loop{{$}}
 ; OPT-LABEL: doublediamond_in_a_loop:
 ; OPT:      .LBB11_1:
 ; OPT:      loop{{$}}
 ; OPT:      block{{$}}
-; OPT:      br_if           0, {{[^,]+}}{{$}}
+; OPT:      block{{$}}
 ; OPT:      block{{$}}
 ; OPT:      br_if           0, {{[^,]+}}{{$}}
+; OPT:      br_if           1, {{[^,]+}}{{$}}
 ; OPT:      br              2{{$}}
 ; OPT-NEXT: .LBB11_4:
 ; OPT-NEXT: end_block{{$}}
 ; OPT:      br              1{{$}}
 ; OPT:      .LBB11_5:
 ; OPT-NEXT: end_block{{$}}
-; OPT:      br              0{{$}}
 ; OPT:      .LBB11_6:
+; OPT-NEXT: end_block{{$}}
+; OPT:      br              0{{$}}
+; OPT:      .LBB11_7:
 ; OPT-NEXT: end_loop{{$}}
 define i32 @doublediamond_in_a_loop(i32 %a, i32 %b, i32* %p) {
 entry:
   br label %header
 header:
   %c = icmp eq i32 %a, 0
   %d = icmp eq i32 %b, 0
   store volatile i32 0, i32* %p

test/CodeGen/WebAssembly/mem-intrinsics.ll

-; RUN: llc < %s -asm-verbose=false -disable-wasm-fallthrough-return-opt | FileCheck %s
+; RUN: llc < %s -asm-verbose=false -disable-wasm-fallthrough-return-opt -tail-dup-placement=0| FileCheck %s
 
 ; Test memcpy, memmove, and memset intrinsics.
 
 target datalayout = "e-m:e-p:32:32-i64:64-n32:64-S128"
 target triple = "wasm32-unknown-unknown"
 
 declare void @llvm.memcpy.p0i8.p0i8.i32(i8* nocapture, i8* nocapture readonly, i32, i32, i1)
 declare void @llvm.memmove.p0i8.p0i8.i32(i8* nocapture, i8* nocapture readonly, i32, i32, i1)

test/CodeGen/X86/avx512-i1test.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=x86_64-apple-darwin -mcpu=knl | FileCheck %s
+; RUN: llc -O2 < %s -mtriple=x86_64-apple-darwin -mcpu=knl | FileCheck %s
 
 ; ModuleID = 'bugpoint-reduced-simplified.bc'
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define void @func() {
 ; CHECK-LABEL: func:
 ; CHECK:       ## BB#0: ## %L_10
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    je LBB0_1
 ; CHECK-NEXT:  ## BB#4: ## %L_30
 ; CHECK-NEXT:    retq
 ; CHECK-NEXT:  LBB0_1: ## %bb56
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    jmp LBB0_2
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  LBB0_3: ## %bb35
 ; CHECK-NEXT:    ## in Loop: Header=BB0_2 Depth=1
 ; CHECK-NEXT:    testb %al, %al
+; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:  LBB0_2: ## %bb33
 ; CHECK-NEXT:    ## =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    jne LBB0_2
 ; CHECK-NEXT:    jmp LBB0_3
 bb1:
   br i1 undef, label %L_10, label %L_10
 

test/CodeGen/X86/block-placement.ll

body4:
   %next = add i32 %iv, 1
   %exitcond = icmp eq i32 %next, %i
   br i1 %exitcond, label %exit, label %body1
 
 exit:
   ret i32 %sum
 }
 
+; Tail duplication during layout can entirely remove body0 by duplicating it
+; into the entry block and into body1. This is a good thing but it isn't what
+; this test is looking for. So to make the blocks longer so they don't get
+; duplicated, we add some calls to dummy.
+declare void @dummy()
+
 define i32 @test_loop_rotate(i32 %i, i32* %a) {
 ; Check that we rotate conditional exits from the loop to the bottom of the
 ; loop, eliminating unconditional branches to the top.
 ; CHECK-LABEL: test_loop_rotate:
 ; CHECK: %entry
 ; CHECK: %body1
 ; CHECK: %body0
 ; CHECK: %exit
 
 entry:
   br label %body0
 
 body0:
   %iv = phi i32 [ 0, %entry ], [ %next, %body1 ]
   %base = phi i32 [ 0, %entry ], [ %sum, %body1 ]
   %next = add i32 %iv, 1
   %exitcond = icmp eq i32 %next, %i
+  call void @dummy()
+  call void @dummy()
   br i1 %exitcond, label %exit, label %body1
 
 body1:
   %arrayidx = getelementptr inbounds i32, i32* %a, i32 %iv
   %0 = load i32, i32* %arrayidx
   %sum = add nsw i32 %0, %base
   %bailcond1 = icmp eq i32 %sum, 42
   br label %body0

 ;    strange layouts that are siginificantly less efficient, often times maing
 ;    it discontiguous.
 ;
 ; CHECK: @benchmark_heapsort
 ; CHECK: %entry
 ; First rotated loop top.
 ; CHECK: .p2align
 ; CHECK: %while.end
-; CHECK: %for.cond
+; %for.cond gets completely tail-duplicated away.
 ; CHECK: %if.then
 ; CHECK: %if.else
 ; CHECK: %if.end10
 ; Second rotated loop top
 ; CHECK: .p2align
 ; CHECK: %if.then24
 ; CHECK: %while.cond.outer
 ; Third rotated loop top

test/CodeGen/X86/fma-intrinsics-phi-213-to-231.ll

 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2,+fma | FileCheck %s
 
 ; CHECK-LABEL: fmaddsubpd_loop_128:
 ; CHECK:   vfmaddsub231pd %xmm1, %xmm0, %xmm2
-; CHECK:   vmovaps %xmm2, %xmm0
+; CHECK:   vmovapd %xmm2, %xmm0
 ; CHECK-NEXT: retq
 define <2 x double> @fmaddsubpd_loop_128(i32 %iter, <2 x double> %a, <2 x double> %b, <2 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <2 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
for.inc:
   br label %for.cond
 
 for.end:
   ret <2 x double> %c.addr.0
 }
 
 ; CHECK-LABEL: fmsubaddpd_loop_128:
 ; CHECK:   vfmsubadd231pd %xmm1, %xmm0, %xmm2
-; CHECK:   vmovaps %xmm2, %xmm0
+; CHECK:   vmovapd %xmm2, %xmm0
 ; CHECK-NEXT: retq
 define <2 x double> @fmsubaddpd_loop_128(i32 %iter, <2 x double> %a, <2 x double> %b, <2 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <2 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
for.inc:
   br label %for.cond
 
 for.end:
   ret <2 x double> %c.addr.0
 }
 
 ; CHECK-LABEL: fmaddpd_loop_128:
 ; CHECK:   vfmadd231pd %xmm1, %xmm0, %xmm2
-; CHECK:   vmovaps %xmm2, %xmm0
+; CHECK:   vmovapd %xmm2, %xmm0
 ; CHECK-NEXT: retq
 define <2 x double> @fmaddpd_loop_128(i32 %iter, <2 x double> %a, <2 x double> %b, <2 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <2 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
for.inc:
   br label %for.cond
 
 for.end:
   ret <2 x double> %c.addr.0
 }
 
 ; CHECK-LABEL: fmsubpd_loop_128:
 ; CHECK:   vfmsub231pd %xmm1, %xmm0, %xmm2
-; CHECK:   vmovaps %xmm2, %xmm0
+; CHECK:   vmovapd %xmm2, %xmm0
 ; CHECK-NEXT: retq
 define <2 x double> @fmsubpd_loop_128(i32 %iter, <2 x double> %a, <2 x double> %b, <2 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <2 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
for.inc:
   br label %for.cond
 
 for.end:
   ret <2 x double> %c.addr.0
 }
 
 ; CHECK-LABEL: fnmaddpd_loop_128:
 ; CHECK:   vfnmadd231pd %xmm1, %xmm0, %xmm2
-; CHECK:   vmovaps %xmm2, %xmm0
+; CHECK:   vmovapd %xmm2, %xmm0
 ; CHECK-NEXT: retq
 define <2 x double> @fnmaddpd_loop_128(i32 %iter, <2 x double> %a, <2 x double> %b, <2 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <2 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
for.inc:
   br label %for.cond
 
 for.end:
   ret <2 x double> %c.addr.0
 }
 
 ; CHECK-LABEL: fnmsubpd_loop_128:
 ; CHECK:   vfnmsub231pd %xmm1, %xmm0, %xmm2
-; CHECK:   vmovaps %xmm2, %xmm0
+; CHECK:   vmovapd %xmm2, %xmm0
 ; CHECK-NEXT: retq
 define <2 x double> @fnmsubpd_loop_128(i32 %iter, <2 x double> %a, <2 x double> %b, <2 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <2 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]

 declare <4 x float> @llvm.x86.fma.vfmsubadd.ps(<4 x float>, <4 x float>, <4 x float>)
 declare <4 x float> @llvm.x86.fma.vfmadd.ps(<4 x float>, <4 x float>, <4 x float>)
 declare <4 x float> @llvm.x86.fma.vfmsub.ps(<4 x float>, <4 x float>, <4 x float>)
 declare <4 x float> @llvm.x86.fma.vfnmadd.ps(<4 x float>, <4 x float>, <4 x float>)
 declare <4 x float> @llvm.x86.fma.vfnmsub.ps(<4 x float>, <4 x float>, <4 x float>)
 
 ; CHECK-LABEL: fmaddsubpd_loop_256:
 ; CHECK:   vfmaddsub231pd %ymm1, %ymm0, %ymm2
-; CHECK:   vmovaps %ymm2, %ymm0
+; CHECK:   vmovapd %ymm2, %ymm0
 ; CHECK-NEXT: retq
 define <4 x double> @fmaddsubpd_loop_256(i32 %iter, <4 x double> %a, <4 x double> %b, <4 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <4 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
for.inc:
   br label %for.cond
 
 for.end:
   ret <4 x double> %c.addr.0
 }
 
 ; CHECK-LABEL: fmsubaddpd_loop_256:
 ; CHECK:   vfmsubadd231pd %ymm1, %ymm0, %ymm2
-; CHECK:   vmovaps %ymm2, %ymm0
+; CHECK:   vmovapd %ymm2, %ymm0
 ; CHECK-NEXT: retq
 define <4 x double> @fmsubaddpd_loop_256(i32 %iter, <4 x double> %a, <4 x double> %b, <4 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <4 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
for.inc:
   br label %for.cond
 
 for.end:
   ret <4 x double> %c.addr.0
 }
 
 ; CHECK-LABEL: fmaddpd_loop_256:
 ; CHECK:   vfmadd231pd %ymm1, %ymm0, %ymm2
-; CHECK:   vmovaps %ymm2, %ymm0
+; CHECK:   vmovapd %ymm2, %ymm0
 ; CHECK-NEXT: retq
 define <4 x double> @fmaddpd_loop_256(i32 %iter, <4 x double> %a, <4 x double> %b, <4 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <4 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
for.inc:
   br label %for.cond
 
 for.end:
   ret <4 x double> %c.addr.0
 }
 
 ; CHECK-LABEL: fmsubpd_loop_256:
 ; CHECK:   vfmsub231pd %ymm1, %ymm0, %ymm2
-; CHECK:   vmovaps %ymm2, %ymm0
+; CHECK:   vmovapd %ymm2, %ymm0
 ; CHECK-NEXT: retq
 define <4 x double> @fmsubpd_loop_256(i32 %iter, <4 x double> %a, <4 x double> %b, <4 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <4 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
for.inc:
   br label %for.cond
 
 for.end:
   ret <4 x double> %c.addr.0
 }
 
 ; CHECK-LABEL: fnmaddpd_loop_256:
 ; CHECK:   vfnmadd231pd %ymm1, %ymm0, %ymm2
-; CHECK:   vmovaps %ymm2, %ymm0
+; CHECK:   vmovapd %ymm2, %ymm0
 ; CHECK-NEXT: retq
 define <4 x double> @fnmaddpd_loop_256(i32 %iter, <4 x double> %a, <4 x double> %b, <4 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <4 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]
for.inc:
   br label %for.cond
 
 for.end:
   ret <4 x double> %c.addr.0
 }
 
 ; CHECK-LABEL: fnmsubpd_loop_256:
 ; CHECK:   vfnmsub231pd %ymm1, %ymm0, %ymm2
-; CHECK:   vmovaps %ymm2, %ymm0
+; CHECK:   vmovapd %ymm2, %ymm0
 ; CHECK-NEXT: retq
 define <4 x double> @fnmsubpd_loop_256(i32 %iter, <4 x double> %a, <4 x double> %b, <4 x double> %c) {
 entry:
   br label %for.cond
 
 for.cond:
   %c.addr.0 = phi <4 x double> [ %c, %entry ], [ %0, %for.inc ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ]

test/CodeGen/X86/fp-une-cmp.ll

 define double @profile_metadata(double %x, double %y) {
 ; CHECK-LABEL: profile_metadata:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    mulsd %xmm1, %xmm0
 ; CHECK-NEXT:    xorpd %xmm1, %xmm1
 ; CHECK-NEXT:    ucomisd %xmm1, %xmm0
 ; CHECK-NEXT:    jne .LBB1_1
 ; CHECK-NEXT:    jp .LBB1_1
-; CHECK-NEXT:  .LBB1_2: # %bb2
+; CHECK-NEXT:  # %bb2
 ; CHECK-NEXT:    retq
 ; CHECK-NEXT:  .LBB1_1: # %bb1
 ; CHECK-NEXT:    addsd {{.*}}(%rip), %xmm0
-; CHECK-NEXT:    jmp .LBB1_2
+; CHECK-NEXT:    retq
 
 entry:
   %mul = fmul double %x, %y
   %cmp = fcmp une double %mul, 0.000000e+00
   br i1 %cmp, label %bb1, label %bb2, !prof !1
 
 bb1:
   %add = fadd double %mul, -1.000000e+00

test/CodeGen/X86/pr11202.ll

l1:                                               ; preds = %l2, %entry
   %a = zext i1 false to i32
   br label %l2
 
 l2:                                               ; preds = %l1
   %b = zext i1 false to i32
   br label %l1
 }
 
-; CHECK: .Ltmp0:                                 # Address of block that was removed by CodeGen
+; It is correct for either l1 or l2 to be removed.
+; If l2 is removed, the message should be "Address of block that was removed by CodeGen"
+; If l1 is removed, it should be "Block address taken."
+; CHECK: .Ltmp0:                                 # {{Address of block that was removed by CodeGen|Block address taken}}
 ; CHECK: .quad  .Ltmp0

test/CodeGen/X86/ragreedy-bug.ll

 ; RUN: llc < %s -mtriple=x86_64-apple-macosx -regalloc=greedy | FileCheck %s
 
 ; This testing case is reduced from 197.parser prune_match function.
 ; We make sure register copies are not generated on isupper.exit blocks.
 
-; CHECK: isupper.exit
+; isupper.exit and isupper.exit223 get tail-duplicated into all their
+; predecessors.
+; CHECK: cond.true.i.i
 ; CHECK-NEXT: in Loop
+; Mem-move
+; CHECK-NEXT: movl
+; CHECK-NEXT: andl
 ; CHECK-NEXT: testl
 ; CHECK-NEXT: jne
-; CHECK: isupper.exit
+; CHECK: cond.true.i.i217
 ; CHECK-NEXT: in Loop
+; Mem-move
+; CHECK-NEXT: movl
+; CHECK-NEXT: andl
 ; CHECK-NEXT: testl
 ; CHECK-NEXT: je
+; CHECK: cond.false.i.i
 ; CHECK: maskrune
+; CHECK-NEXT: movzbl
+; CHECK-NEXT: movzbl
+; CHECK-NEXT: testl
+; CHECK-NEXT: je
+; CHECK: cond.false.i.i219
 ; CHECK: maskrune
+; CHECK-NEXT: movzbl
+; CHECK-NEXT: movzbl
+; CHECK-NEXT: testl
+; CHECK-NEXT: jne
 
 %struct.List_o_links_struct = type { i32, i32, i32, %struct.List_o_links_struct* }
 %struct.Connector_struct = type { i16, i16, i8, i8, %struct.Connector_struct*, i8* }
 %struct._RuneLocale = type { [8 x i8], [32 x i8], i32 (i8*, i64, i8**)*, i32 (i32, i8*, i64, i8**)*, i32, [256 x i32], [256 x i32], [256 x i32], %struct._RuneRange, %struct._RuneRange, %struct._RuneRange, i8*, i32, i32, %struct._RuneCharClass* }
 %struct._RuneRange = type { i32, %struct._RuneEntry* }
 %struct._RuneEntry = type { i32, i32, i32, i32* }
 %struct._RuneCharClass = type { [14 x i8], i32 }
 %struct.Exp_struct = type { i8, i8, i8, i8, %union.anon }

test/CodeGen/X86/sse1.ll

 define <4 x float> @vselect(<4 x float>*%p, <4 x i32> %q) {
 ; X32-LABEL: vselect:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
 ; X32-NEXT:    xorps %xmm0, %xmm0
 ; X32-NEXT:    je .LBB1_1
 ; X32-NEXT:  # BB#2: # %entry
 ; X32-NEXT:    xorps %xmm1, %xmm1
-; X32-NEXT:    jmp .LBB1_3
+; X32-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
+; X32-NEXT:    jne .LBB1_5
+; X32-NEXT:    jmp .LBB1_4
 ; X32-NEXT:  .LBB1_1:
 ; X32-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
-; X32-NEXT:  .LBB1_3: # %entry
 ; X32-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
 ; X32-NEXT:    je .LBB1_4
-; X32-NEXT:  # BB#5: # %entry
+; X32-NEXT:  .LBB1_5: # %entry
 ; X32-NEXT:    xorps %xmm2, %xmm2
-; X32-NEXT:    jmp .LBB1_6
+; X32-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
+; X32-NEXT:    jne .LBB1_8
+; X32-NEXT:    jmp .LBB1_7
 ; X32-NEXT:  .LBB1_4:
 ; X32-NEXT:    movss {{.*#+}} xmm2 = mem[0],zero,zero,zero
-; X32-NEXT:  .LBB1_6: # %entry
 ; X32-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
 ; X32-NEXT:    je .LBB1_7
-; X32-NEXT:  # BB#8: # %entry
+; X32-NEXT:  .LBB1_8: # %entry
 ; X32-NEXT:    xorps %xmm3, %xmm3
 ; X32-NEXT:    jmp .LBB1_9
 ; X32-NEXT:  .LBB1_7:
 ; X32-NEXT:    movss {{.*#+}} xmm3 = mem[0],zero,zero,zero
 ; X32-NEXT:  .LBB1_9: # %entry
 ; X32-NEXT:    cmpl $0, {{[0-9]+}}(%esp)
 ; X32-NEXT:    unpcklps {{.*#+}} xmm2 = xmm2[0],xmm3[0],xmm2[1],xmm3[1]
 ; X32-NEXT:    jne .LBB1_11
 ; X32-NEXT:  # BB#10:
 ; X32-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; X32-NEXT:  .LBB1_11: # %entry
 ; X32-NEXT:    unpcklps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
 ; X32-NEXT:    unpcklps {{.*#+}} xmm0 = xmm0[0],xmm2[0],xmm0[1],xmm2[1]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: vselect:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    testl %ecx, %ecx
 ; X64-NEXT:    xorps %xmm0, %xmm0
 ; X64-NEXT:    je .LBB1_1
 ; X64-NEXT:  # BB#2: # %entry
 ; X64-NEXT:    xorps %xmm1, %xmm1
-; X64-NEXT:    jmp .LBB1_3
+; X64-NEXT:    testl %edx, %edx
+; X64-NEXT:    jne .LBB1_5
+; X64-NEXT:    jmp .LBB1_4
 ; X64-NEXT:  .LBB1_1:
 ; X64-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
-; X64-NEXT:  .LBB1_3: # %entry
 ; X64-NEXT:    testl %edx, %edx
 ; X64-NEXT:    je .LBB1_4
-; X64-NEXT:  # BB#5: # %entry
+; X64-NEXT:  .LBB1_5: # %entry
 ; X64-NEXT:    xorps %xmm2, %xmm2
-; X64-NEXT:    jmp .LBB1_6
+; X64-NEXT:    testl %r8d, %r8d
+; X64-NEXT:    jne .LBB1_8
+; X64-NEXT:    jmp .LBB1_7
 ; X64-NEXT:  .LBB1_4:
 ; X64-NEXT:    movss {{.*#+}} xmm2 = mem[0],zero,zero,zero
-; X64-NEXT:  .LBB1_6: # %entry
 ; X64-NEXT:    testl %r8d, %r8d
 ; X64-NEXT:    je .LBB1_7
-; X64-NEXT:  # BB#8: # %entry
+; X64-NEXT:  .LBB1_8: # %entry
 ; X64-NEXT:    xorps %xmm3, %xmm3
 ; X64-NEXT:    jmp .LBB1_9
 ; X64-NEXT:  .LBB1_7:
 ; X64-NEXT:    movss {{.*#+}} xmm3 = mem[0],zero,zero,zero
 ; X64-NEXT:  .LBB1_9: # %entry
 ; X64-NEXT:    testl %esi, %esi
 ; X64-NEXT:    unpcklps {{.*#+}} xmm2 = xmm2[0],xmm3[0],xmm2[1],xmm3[1]
 ; X64-NEXT:    jne .LBB1_11

test/CodeGen/X86/update-terminator.mir

 # RUN: llc -march=x86-64 -verify-machineinstrs -run-pass block-placement -o - %s | FileCheck %s
 # Check the conditional jump in bb.1 is changed to unconditional after block placement swaps bb.2 and bb.3.
 
 --- |
   @a = external global i16
   @b = external global i32
 
+  declare void @dummy1()
+  declare void @dummy2()
+  declare void @dummy3()
+
   ; Function Attrs: nounwind
   define void @f2() {
     br i1 undef, label %bb1, label %bb3
 
   bb1:
+    call void @dummy1()
+    call void @dummy1()
+    call void @dummy1()
     br i1 undef, label %bb2, label %bb2
 
   bb2:
+    call void @dummy2()
+    call void @dummy2()
+    call void @dummy2()
     br label %bb4
 
   bb3:
+    call void @dummy3()
+    call void @dummy3()
+    call void @dummy3()
     br label %bb2
 
   bb4:
     ret void
   }
 
 
 ...
 ---
 # CHECK-LABEL: name: f2
 # CHECK: bb.1:
 # CHECK: JMP_1 %bb.2
 # CHECK: bb.3:
 # CHECK: bb.2:
 name:            f2
 body:             |
   bb.0 (%ir-block.0):
     successors: %bb.1(50), %bb.3(50)
 
     JNE_1 %bb.1, implicit %eflags
     JMP_1 %bb.3
   bb.1:
     successors: %bb.2(100)
 
+    CALL64pcrel32 @dummy1, csr_64, implicit %rsp, implicit-def %rsp
+    CALL64pcrel32 @dummy1, csr_64, implicit %rsp, implicit-def %rsp
+    CALL64pcrel32 @dummy1, csr_64, implicit %rsp, implicit-def %rsp
     JNE_1 %bb.2, implicit %eflags
 
   bb.2:
     successors: %bb.4(100)
 
+    CALL64pcrel32 @dummy2, csr_64, implicit %rsp, implicit-def %rsp
+    CALL64pcrel32 @dummy2, csr_64, implicit %rsp, implicit-def %rsp
+    CALL64pcrel32 @dummy2, csr_64, implicit %rsp, implicit-def %rsp
     JMP_1 %bb.4
 
   bb.3:
     successors: %bb.2(100)
+    CALL64pcrel32 @dummy3, csr_64, implicit %rsp, implicit-def %rsp
+    CALL64pcrel32 @dummy3, csr_64, implicit %rsp, implicit-def %rsp
+    CALL64pcrel32 @dummy3, csr_64, implicit %rsp, implicit-def %rsp
     JMP_1 %bb.2
 
   bb.4:
     RETQ
 
 ...