CodeGen: Allow small copyable blocks to "break" the CFG.

[iteratee]

Summary

When choosing the best successor for a block, ordinarily we would have preferred
a block that preserves the CFG unless there is a strong probability the other
direction. For small blocks that can be duplicated we now skip that requirement
as well, because the copy means that the CFG won't actually be violated.

[iteratee]

davidxl - This should be quite a bit shorter and easier to review. Thanks for the direction. There is a 2nd part, but it's also really short.

t.p.northover - Can you look at the arm cmpxchg tests? I mentioned it on IRC, but here it is concrete. This code causes the layout, and therefore fallthrough of the generated cmpxchg code to no longer to favor the success case. I think we should change the generating code to either supply a small hint, or to provide the inverted test and the branches in the other order. Both will put the fallthrough back the way it was.

[davidxl]

Please add one or two test cases for this feature.

lib/CodeGen/MachineBlockPlacement.cpp

568

&& --> || ?

In other words, do we care about block dup of simple blocks in layout mode?

581

Perhaps change the name to "canTailDuplicateUnplacedPreds" ?

593

We have have a situation where some of the predecessors can share a common successor (newly created), and the block can be tail-Duped into that block. It might be worth considering supporting.

[iteratee]

Fixes from review comments, and new test case.

[iteratee]

lib/CodeGen/MachineBlockPlacement.cpp

568

Simple blocks contain only an unconditional jump. I don't see a good reason to exclude them during layout.

581

Done. Thanks.

593

While I'll keep that in mind, that would basically be block copying. We should support that as well, but I think it's out of scope for what I'm trying to do here.

[davidxl]

do you forget to include the new test in the patch, I still can not find it.

lib/CodeGen/MachineBlockPlacement.cpp

568

Basically you are trying to tail duplicate such simple blocks in layout -- what is the benefit? It does not create more fall-through opportunities.

[iteratee]

Put test into correct patch.

[davidxl]

lib/CodeGen/MachineBlockPlacement.cpp

568

Should the condition be just

if (BB->succ_size() == 1)

return false;

If not, can you come up with a test case showing tail-dup simple BB helps the layout?

test/CodeGen/PowerPC/tail-dup-layout.ll

97

The intention of this test case is not clear. I expect a simpler and straightforward test case for instance just a control flow with two concatenated triangles. The first triangle branch is annotated with profile data such that without this patch it will layout the blocks in topo order.

116

The term 'avoidable'/'unavoidable' is not well defined and can be confusing. I also suggest add this test case in a different patch.

[iteratee]

test/CodeGen/PowerPC/tail-dup-layout.ll

97

The straightline test that you're asking for will be redundant as soon as D20505 lands. (It removes the outlining flags from this file, and verifies the single triangle case by just doing multiple triangles)

This test is basically as small as possible. It needs a block that:
a) has predecessors that can accept a tail-duplicated copy
b) wouldn't be chosen based on some outlining algorithm
c) has at least 2 successors

The block then2 has 2 predecessors: else1, and test2

It has 2 successors: end1 and end2
and test2 has an alternate path around then2: else2

116

The goal of the test is a block that wouldn't be chosen due to some outlining strategy. Do you have a name you prefer?

[davidxl]

test/CodeGen/PowerPC/tail-dup-layout.ll

97

It may become redundant after D20505 lands, but this patch still needs a simplest test case possible in addition to this one.

116

The name ties to much to the implementation details of outlining heuristics. Just change the function name to something like tail_dup_test or something and add a comment that test2 has an alternate path around then2 to disable outllining. Perhaps add a check to make sure outlining does not happen?

[iteratee]

Added new test.

[iteratee]

I added a simple test in a new file that doesn't pass the outlining flag to llc.

[davidxl]

test/CodeGen/PowerPC/tail-dup-break-cfg.ll

7 ↗

(On Diff #83334)

This test demonstrates that this patch helps when test2 is more likely (but the branch probability is not biased enough to make the successor selection to pick test2 as successor without the patch).

What happens if body1 a more likely successor such that even with the patch, body1 will be selected as test1's successor. Later when block test2 is checked, it is still beneficial to tail duplicate it. Does it happen with the patch?

[iteratee]

Things that I had marked as done, but had missed, perhaps by putting them in D20505

[iteratee]

test/CodeGen/PowerPC/tail-dup-break-cfg.ll

7 ↗

(On Diff #83334)

This patch only prefers blocks that can be tail-duplicated if they are more probable.

selectBestSuccessor looks at the blocks in order and picks the most probable block that doesn't have a CFG conflict, or is hot enough to ignore the CFG conflict. Most of that logic is in hasBetterLayoutPredecessor. This patch just adds "can tail-duplicate to unplaced preds" as another condition to allow CFG-breaking.

If body1 is laid out after test1, then test2 will not be copied into the end of test1, as test1 has more than one successor. Since test2 would be laid out after body1, it would not be tail-duplicated anywhere.

In an if-else case where both bodies go to a small test afterward:

   t1
  / \
b1   e1
  \ /
   t2

The existing code will duplicate the test without this patch, producing either

t1 b1 t2 ... e1(t2) or t1 e1 t2 ... b2(t2)

I can create that test if you'd like, but since it's not changed by this patch, it should go in as a separate change.

[davidxl]

lgtm

[iteratee]

Simple tidying of whitespace.

[echristo]

A few comments if you wouldn't mind?

llvm/trunk/lib/CodeGen/MachineBlockPlacement.cpp

570 ↗

(On Diff #83881)

const arguments?

589 ↗

(On Diff #83881)

Ditto.

595 ↗

(On Diff #83881)

Ditto.

598 ↗

(On Diff #83881)

Can you comment/explain the BlockFilter part here?

599 ↗

(On Diff #83881)

Same with BlockToChain.

1961 ↗

(On Diff #83881)

Maybe this should be above the previous line?

lib/CodeGen/MachineBlockPlacement.cpp

class MachineBlockPlacement : public MachineFunctionPass {
   void rotateLoop(BlockChain &LoopChain, MachineBasicBlock *ExitingBB,
                   const BlockFilterSet &LoopBlockSet);
   void rotateLoopWithProfile(BlockChain &LoopChain, MachineLoop &L,
                              const BlockFilterSet &LoopBlockSet);
   void collectMustExecuteBBs();
   void buildCFGChains();
   void optimizeBranches();
   void alignBlocks();
+  bool shouldTailDuplicate(MachineBasicBlock *BB);
+  bool canTailDuplicateAllPreds(MachineBasicBlock *BB, MachineBasicBlock *Succ,
+                                BlockChain &Chain,
+                                const BlockFilterSet *BlockFilter);
 
 public:
   static char ID; // Pass identification, replacement for typeid
   MachineBlockPlacement() : MachineFunctionPass(ID) {
     initializeMachineBlockPlacementPass(*PassRegistry::getPassRegistry());
   }
 
   bool runOnMachineFunction(MachineFunction &F) override;
getAdjustedProbability(BranchProbability OrigProb,
   if (SuccProbN >= SuccProbD)
     SuccProb = BranchProbability::getOne();
   else
     SuccProb = BranchProbability(SuccProbN, SuccProbD);
 
   return SuccProb;
 }
 
+/// Check if a block should be tail duplicated.
+/// \p BB Block to check.
+bool MachineBlockPlacement::shouldTailDuplicate(MachineBasicBlock *BB) {
+  // Blocks with single successors don't create additional fallthrough
+  // opportunities. Don't duplicate them. TODO: When conditional exits are
+  // analyzable, allow them to be duplicated.
+  bool IsSimple = TailDup.isSimpleBB(BB);
+
+  if (!IsSimple && BB->succ_size() == 1)
+    return false;
+  return TailDup.shouldTailDuplicate(IsSimple, *BB);
+}
+
+/// When the option TailDupPlacement is on, this method checks if the
+/// fallthrough candidate block \p Succ (of block \p BB) can be tail-duplicated
+/// into all of its unplaced, unfiltered predecessors, that are not BB. In
+/// addition we keep a set of blocks that have been tail-duplicated into and
+/// allow those blocks to be unplaced as well. This allows the creation of a
+/// second (larger) spine and a short fallthrough spine.
+/// We also identify blocks with the CFG that would have been produced by
+/// tail-duplication and lay them out in the same manner.
+bool MachineBlockPlacement::canTailDuplicateAllPreds(
+    MachineBasicBlock *BB, MachineBasicBlock *Succ, BlockChain &Chain,
+    const BlockFilterSet *BlockFilter) {
+  if (!shouldTailDuplicate(Succ))
+    return false;
+
+  for (MachineBasicBlock *Pred : Succ->predecessors()) {
+    // Make sure all unplaced and unfiltered predecessors can be
+    // tail-duplicated into.
+    if (Pred == BB || (BlockFilter && !BlockFilter->count(Pred))
+        || BlockToChain[Pred] == &Chain)
+      continue;
+    if (!TailDup.canTailDuplicate(Succ, Pred))
+      return false;
+  }
+  return true;
+}
+
 /// When the option OutlineOptionalBranches is on, this method
 /// checks if the fallthrough candidate block \p Succ (of block
 /// \p BB) also has other unscheduled predecessor blocks which
 /// are also successors of \p BB (forming triangular shape CFG).
 /// If none of such predecessors are small, it returns true.
 /// The caller can choose to select \p Succ as the layout successors
 /// so that \p Succ's predecessors (optional branches) can be
 /// outlined.
bool MachineBlockPlacement::hasBetterLayoutPredecessor(
     MachineBasicBlock *BB, MachineBasicBlock *Succ, BlockChain &SuccChain,
     BranchProbability SuccProb, BranchProbability RealSuccProb,
     BlockChain &Chain, const BlockFilterSet *BlockFilter) {
 
   // There isn't a better layout when there are no unscheduled predecessors.
   if (SuccChain.UnscheduledPredecessors == 0)
     return false;
 
+  // As a heuristic, if we can duplicate the block into all its unscheduled
+  // predecessors, we return false.
+  if (TailDupPlacement
+      && canTailDuplicateAllPreds(BB, Succ, Chain, BlockFilter))
+    return false;
+
   // There are two basic scenarios here:
   // -------------------------------------
   // Case 1: triangular shape CFG (if-then):
   //     BB
   //     | \
   //     |  \
   //     |   Pred
   //     |   /
bool MachineBlockPlacement::maybeTailDuplicateBlock(
     MachineBasicBlock *BB, MachineBasicBlock *LPred,
     const BlockChain &Chain, BlockFilterSet *BlockFilter,
     MachineFunction::iterator &PrevUnplacedBlockIt,
     bool &DuplicatedToLPred) {
 
   DuplicatedToLPred = false;
   DEBUG(dbgs() << "Redoing tail duplication for Succ#"
         << BB->getNumber() << "\n");
-  bool IsSimple = TailDup.isSimpleBB(BB);
-  // Blocks with single successors don't create additional fallthrough
-  // opportunities. Don't duplicate them. TODO: When conditional exits are
-  // analyzable, allow them to be duplicated.
-  if (!IsSimple && BB->succ_size() == 1)
-    return false;
-  if (!TailDup.shouldTailDuplicate(IsSimple, *BB))
+
+  if (!shouldTailDuplicate(BB))
     return false;
   // This has to be a callback because none of it can be done after
   // BB is deleted.
   bool Removed = false;
   auto RemovalCallback =
       [&](MachineBasicBlock *RemBB) {
         // Signal to outer function
         Removed = true;
auto RemovalCallback =
 
         DEBUG(dbgs() << "TailDuplicator deleted block: "
               << getBlockName(RemBB) << "\n");
       };
   auto RemovalCallbackRef =
       llvm::function_ref<void(MachineBasicBlock*)>(RemovalCallback);
 
   SmallVector<MachineBasicBlock *, 8> DuplicatedPreds;
+  bool IsSimple = TailDup.isSimpleBB(BB);
   TailDup.tailDuplicateAndUpdate(IsSimple, BB, LPred,
                                  &DuplicatedPreds, &RemovalCallbackRef);
 
   // Update UnscheduledPredecessors to reflect tail-duplication.
   DuplicatedToLPred = false;
   for (MachineBasicBlock *Pred : DuplicatedPreds) {
     // We're only looking for unscheduled predecessors that match the filter.
     BlockChain* PredChain = BlockToChain[Pred];

test/CodeGen/AArch64/addsub.ll

 ; CHECK: b.gt [[RET]]
   %newval4 = add i32 %val, 3
   store i32 %newval4, i32* @var_i32
   %cmp_pos_sgt = icmp sgt i32 %val2, 321
   br i1 %cmp_pos_sgt, label %ret, label %test5
 
 test5:
 ; CHECK: cmn {{w[0-9]+}}, #444
-; CHECK: b.gt [[RET]]
+; CHECK: b.le [[TEST6:.?LBB[0-9]+_[0-9]+]]
   %newval5 = add i32 %val, 4
   store i32 %newval5, i32* @var_i32
   %cmp_neg_uge = icmp sgt i32 %val2, -444
   br i1 %cmp_neg_uge, label %ret, label %test6
 
+; CHECK: {{^}}[[RET]]:
+; CHECK: ret
+; CHECK: {{^}}[[TEST6]]:
+; CHECK: ret
+
 test6:
   %newval6 = add i32 %val, 5
   store i32 %newval6, i32* @var_i32
   ret void
 
 ret:
   ret void
 }
 ; TODO: adds/subs

test/CodeGen/AArch64/arm64-atomic.ll

 ; RUN: llc < %s -mtriple=arm64-eabi -asm-verbose=false -verify-machineinstrs -mcpu=cyclone | FileCheck %s
 
 define i32 @val_compare_and_swap(i32* %p, i32 %cmp, i32 %new) #0 {
 ; CHECK-LABEL: val_compare_and_swap:
 ; CHECK-NEXT: mov    x[[ADDR:[0-9]+]], x0
 ; CHECK-NEXT: [[TRYBB:.?LBB[0-9_]+]]:
 ; CHECK-NEXT: ldaxr  [[RESULT:w[0-9]+]], [x[[ADDR]]]
 ; CHECK-NEXT: cmp    [[RESULT]], w1
 ; CHECK-NEXT: b.ne   [[FAILBB:.?LBB[0-9_]+]]
 ; CHECK-NEXT: stxr   [[SCRATCH_REG:w[0-9]+]], w2, [x[[ADDR]]]
 ; CHECK-NEXT: cbnz   [[SCRATCH_REG]], [[TRYBB]]
-; CHECK-NEXT: b      [[EXITBB:.?LBB[0-9_]+]]
+; CHECK-NEXT: ret
 ; CHECK-NEXT: [[FAILBB]]:
 ; CHECK-NEXT: clrex
-; CHECK-NEXT: [[EXITBB]]:
+; CHECK-NEXT: ret
   %pair = cmpxchg i32* %p, i32 %cmp, i32 %new acquire acquire
   %val = extractvalue { i32, i1 } %pair, 0
   ret i32 %val
 }
 
 define i32 @val_compare_and_swap_from_load(i32* %p, i32 %cmp, i32* %pnew) #0 {
 ; CHECK-LABEL: val_compare_and_swap_from_load:
 ; CHECK-NEXT: ldr    [[NEW:w[0-9]+]], [x2]
 ; CHECK-NEXT: [[TRYBB:.?LBB[0-9_]+]]:
 ; CHECK-NEXT: ldaxr  [[RESULT:w[0-9]+]], [x0]
 ; CHECK-NEXT: cmp    [[RESULT]], w1
 ; CHECK-NEXT: b.ne   [[FAILBB:.?LBB[0-9_]+]]
 ; CHECK-NEXT: stxr   [[SCRATCH_REG:w[0-9]+]], [[NEW]], [x0]
 ; CHECK-NEXT: cbnz   [[SCRATCH_REG]], [[TRYBB]]
-; CHECK-NEXT: b      [[EXITBB:.?LBB[0-9_]+]]
+; CHECK-NEXT: mov    x0, x[[ADDR]]
+; CHECK-NEXT: ret
 ; CHECK-NEXT: [[FAILBB]]:
 ; CHECK-NEXT: clrex
-; CHECK-NEXT: [[EXITBB]]:
+; CHECK-NEXT: mov    x0, x[[ADDR]]
+; CHECK-NEXT: ret
   %new = load i32, i32* %pnew
   %pair = cmpxchg i32* %p, i32 %cmp, i32 %new acquire acquire
   %val = extractvalue { i32, i1 } %pair, 0
   ret i32 %val
 }
 
 define i32 @val_compare_and_swap_rel(i32* %p, i32 %cmp, i32 %new) #0 {
 ; CHECK-LABEL: val_compare_and_swap_rel:
 ; CHECK-NEXT: mov    x[[ADDR:[0-9]+]], x0
 ; CHECK-NEXT: [[TRYBB:.?LBB[0-9_]+]]:
-; CHECK-NEXT: ldaxr  [[RESULT:w[0-9]+]], [x[[ADDR]]
+; CHECK-NEXT: ldaxr  [[RESULT:w[0-9]+]], [x[[ADDR]]]
 ; CHECK-NEXT: cmp    [[RESULT]], w1
 ; CHECK-NEXT: b.ne   [[FAILBB:.?LBB[0-9_]+]]
-; CHECK-NEXT: stlxr  [[SCRATCH_REG:w[0-9]+]], w2, [x[[ADDR]]
+; CHECK-NEXT: stlxr  [[SCRATCH_REG:w[0-9]+]], w2, [x[[ADDR]]]
 ; CHECK-NEXT: cbnz   [[SCRATCH_REG]], [[TRYBB]]
-; CHECK-NEXT: b      [[EXITBB:.?LBB[0-9_]+]]
+; CHECK-NEXT: ret
 ; CHECK-NEXT: [[FAILBB]]:
 ; CHECK-NEXT: clrex
-; CHECK-NEXT: [[EXITBB]]:
+; CHECK-NEXT: ret
   %pair = cmpxchg i32* %p, i32 %cmp, i32 %new acq_rel monotonic
   %val = extractvalue { i32, i1 } %pair, 0
   ret i32 %val
 }
 
 define i64 @val_compare_and_swap_64(i64* %p, i64 %cmp, i64 %new) #0 {
 ; CHECK-LABEL: val_compare_and_swap_64:
 ; CHECK-NEXT: mov    x[[ADDR:[0-9]+]], x0
 ; CHECK-NEXT: [[TRYBB:.?LBB[0-9_]+]]:
 ; CHECK-NEXT: ldxr   [[RESULT:x[0-9]+]], [x[[ADDR]]]
 ; CHECK-NEXT: cmp    [[RESULT]], x1
 ; CHECK-NEXT: b.ne   [[FAILBB:.?LBB[0-9_]+]]
 ; CHECK-NEXT: stxr   [[SCRATCH_REG:w[0-9]+]], x2, [x[[ADDR]]]
 ; CHECK-NEXT: cbnz   [[SCRATCH_REG]], [[TRYBB]]
-; CHECK-NEXT: b      [[EXITBB:.?LBB[0-9_]+]]
+; CHECK-NEXT: ret
 ; CHECK-NEXT: [[FAILBB]]:
 ; CHECK-NEXT: clrex
-; CHECK-NEXT: [[EXITBB]]:
+; CHECK-NEXT: ret
   %pair = cmpxchg i64* %p, i64 %cmp, i64 %new monotonic monotonic
   %val = extractvalue { i64, i1 } %pair, 0
   ret i64 %val
 }
 
 define i32 @fetch_and_nand(i32* %p) #0 {
 ; CHECK-LABEL: fetch_and_nand:
 ; CHECK: [[TRYBB:.?LBB[0-9_]+]]:

test/CodeGen/AArch64/arm64-ccmp.ll

 
 ; Speculatively execute division by zero.
 ; The sdiv/udiv instructions do not trap when the divisor is zero, so they are
 ; safe to speculate.
 ; CHECK-LABEL: speculate_division:
 ; CHECK: cmp w0, #1
 ; CHECK: sdiv [[DIVRES:w[0-9]+]], w1, w0
 ; CHECK: ccmp [[DIVRES]], #16, #0, ge
-; CHECK: b.gt [[BLOCK:LBB[0-9_]+]]
-; CHECK: bl _foo
-; CHECK: [[BLOCK]]:
+; CHECK: b.le [[BLOCK:LBB[0-9_]+]]
 ; CHECK: orr w0, wzr, #0x7
+; CHECK: [[BLOCK]]:
+; CHECK: bl _foo
 define i32 @speculate_division(i32 %a, i32 %b) nounwind ssp {
 entry:
   %cmp = icmp sgt i32 %a, 0
   br i1 %cmp, label %land.lhs.true, label %if.end
 
 land.lhs.true:
   %div = sdiv i32 %b, %a
   %cmp1 = icmp slt i32 %div, 17
   br i1 %cmp1, label %if.then, label %if.end
 
 if.then:
   %call = tail call i32 @foo() nounwind
   br label %if.end
 
 if.end:
   ret i32 7
 }
 
 ; Floating point compare.
 ; CHECK: single_fcmp
 ; CHECK: cmp
 ; CHECK-NOT: b.
 ; CHECK: fccmp {{.*}}, #8, ge
-; CHECK: b.lt
+; CHECK: b.ge
 define i32 @single_fcmp(i32 %a, float %b) nounwind ssp {
 entry:
   %cmp = icmp sgt i32 %a, 0
   br i1 %cmp, label %land.lhs.true, label %if.end
 
 land.lhs.true:
   %conv = sitofp i32 %a to float
   %div = fdiv float %b, %conv

test/CodeGen/AArch64/arm64-shrink-wrapping.ll

 ; CHECK: [[LOOP_LABEL:LBB[0-9_]+]]: ; %for.body
 ; CHECK: ldr [[VA_ADDR:x[0-9]+]], [sp, #8]
 ; CHECK-NEXT: add [[NEXT_VA_ADDR:x[0-9]+]], [[VA_ADDR]], #8
 ; CHECK-NEXT: str [[NEXT_VA_ADDR]], [sp, #8]
 ; CHECK-NEXT: ldr [[VA_VAL:w[0-9]+]], {{\[}}[[VA_ADDR]]]
 ; CHECK-NEXT: sub w1, w1, #1
 ; CHECK-NEXT: add [[SUM]], [[SUM]], [[VA_VAL]]
 ; CHECK-NEXT: cbnz w1, [[LOOP_LABEL]]
-; DISABLE-NEXT: b [[IFEND_LABEL]]
-;
-; DISABLE: [[ELSE_LABEL]]: ; %if.else
-; DISABLE: lsl w0, w1, #1
-;
-; CHECK: [[IFEND_LABEL]]:
+; CHECK-NEXT: [[IFEND_LABEL]]:
 ; Epilogue code.
 ; CHECK: add sp, sp, #16
 ; CHECK-NEXT: ret
 ;
-; ENABLE: [[ELSE_LABEL]]: ; %if.else
-; ENABLE-NEXT: lsl w0, w1, #1
-; ENABLE_NEXT: ret
+; CHECK: [[ELSE_LABEL]]: ; %if.else
+; CHECK-NEXT: lsl w0, w1, #1
+; DISABLE-NEXT: add sp, sp, #16
+; CHECK-NEXT: ret
 define i32 @variadicFunc(i32 %cond, i32 %count, ...) #0 {
 entry:
   %ap = alloca i8*, align 8
   %tobool = icmp eq i32 %cond, 0
   br i1 %tobool, label %if.else, label %if.then
 
 if.then:                                          ; preds = %entry
   %ap1 = bitcast i8** %ap to i8*

test/CodeGen/AArch64/compare-branch.ll

test3:
   %tst3 = icmp eq i64 %val3, 0
   br i1 %tst3, label %end, label %test4, !prof !1
 ; CHECK: cbz {{x[0-9]+}}, .LBB
 
 test4:
   %val4 = load volatile i64, i64* @var64
   %tst4 = icmp ne i64 %val4, 0
   br i1 %tst4, label %end, label %test5, !prof !1
-; CHECK: cbnz {{x[0-9]+}}, .LBB
+; CHECK: cbz {{x[0-9]+}}, .LBB
 
 test5:
   store volatile i64 %val4, i64* @var64
   ret void
 
 end:
   ret void
 }
 
 
 !1 = !{!"branch_weights", i32 1, i32 1}

test/CodeGen/AArch64/logical_shifted_reg.ll

 ; CHECK: b.lt .L
   %shifted_op = shl i64 %val2, 63
   %shifted_and = and i64 %val1, %shifted_op
   %tst2 = icmp slt i64 %shifted_and, 0
   br i1 %tst2, label %ret, label %test3, !prof !1
 
 test3:
 ; CHECK: tst {{x[0-9]+}}, {{x[0-9]+}}, asr #12
-; CHECK: b.gt .L
+; CHECK: b.le .L
   %asr_op = ashr i64 %val2, 12
   %asr_and = and i64 %asr_op, %val1
   %tst3 = icmp sgt i64 %asr_and, 0
   br i1 %tst3, label %ret, label %other_exit, !prof !1
 
 other_exit:
   store volatile i64 %val1, i64* @var1_64
   ret void
 ret:
   ret void
 }
 
 !1 = !{!"branch_weights", i32 1, i32 1}

test/CodeGen/AArch64/tbz-tbnz.ll

 ; RUN: llc < %s -O1 -mtriple=aarch64-eabi | FileCheck %s
 
 declare void @t()
 
 define void @test1(i32 %a) {
 ; CHECK-LABEL: @test1
 entry:
   %sub = add nsw i32 %a, -12
   %cmp = icmp slt i32 %sub, 0
   br i1 %cmp, label %if.then, label %if.end
 
 ; CHECK: sub [[CMP:w[0-9]+]], w0, #12
-; CHECK: tbz [[CMP]], #31
+; CHECK: tbnz [[CMP]], #31
 
 if.then:
   call void @t()
   br label %if.end
 
 if.end:
   ret void
 }
 
 define void @test2(i64 %a) {
 ; CHECK-LABEL: @test2
 entry:
   %sub = add nsw i64 %a, -12
   %cmp = icmp slt i64 %sub, 0
   br i1 %cmp, label %if.then, label %if.end
 
 ; CHECK: sub [[CMP:x[0-9]+]], x0, #12
-; CHECK: tbz [[CMP]], #63
+; CHECK: tbnz [[CMP]], #63
 
 if.then:
   call void @t()
   br label %if.end
 
 if.end:
   ret void
 }

 define void @test7(i32 %a) {
 ; CHECK-LABEL: @test7
 entry:
   %sub = sub nsw i32 %a, 12
   %cmp = icmp slt i32 %sub, 0
   br i1 %cmp, label %if.then, label %if.end
 
 ; CHECK: sub [[CMP:w[0-9]+]], w0, #12
-; CHECK: tbz [[CMP]], #31
+; CHECK: tbnz [[CMP]], #31
 
 if.then:
   call void @t()
   br label %if.end
 
 if.end:
   ret void
 }

 }
 
 define void @test9(i64 %val1) {
 ; CHECK-LABEL: @test9
   %tst = icmp slt i64 %val1, 0
   br i1 %tst, label %if.then, label %if.end
 
 ; CHECK-NOT: cmp
-; CHECK: tbz x0, #63
+; CHECK: tbnz x0, #63
 
 if.then:
   call void @t()
   br label %if.end
 
 if.end:
   ret void
 }
 
 define void @test10(i64 %val1) {
 ; CHECK-LABEL: @test10
   %tst = icmp slt i64 %val1, 0
   br i1 %tst, label %if.then, label %if.end
 
 ; CHECK-NOT: cmp
-; CHECK: tbz x0, #63
+; CHECK: tbnz x0, #63
 
 if.then:
   call void @t()
   br label %if.end
 
 if.end:
   ret void
 }
 
 define void @test11(i64 %val1, i64* %ptr) {
 ; CHECK-LABEL: @test11
 
 ; CHECK: ldr [[CMP:x[0-9]+]], [x1]
 ; CHECK-NOT: cmp
-; CHECK: tbz [[CMP]], #63
+; CHECK: tbnz [[CMP]], #63
 
   %val = load i64, i64* %ptr
   %tst = icmp slt i64 %val, 0
   br i1 %tst, label %if.then, label %if.end
 
 if.then:
   call void @t()
   br label %if.end
 
 if.end:
   ret void
 }
 
 define void @test12(i64 %val1) {
 ; CHECK-LABEL: @test12
   %tst = icmp slt i64 %val1, 0
   br i1 %tst, label %if.then, label %if.end
 
 ; CHECK-NOT: cmp
-; CHECK: tbz x0, #63
+; CHECK: tbnz x0, #63
 
 if.then:
   call void @t()
   br label %if.end
 
 if.end:
   ret void
 }
 
 define void @test13(i64 %val1, i64 %val2) {
 ; CHECK-LABEL: @test13
   %or = or i64 %val1, %val2
   %tst = icmp slt i64 %or, 0
   br i1 %tst, label %if.then, label %if.end
 
 ; CHECK: orr [[CMP:x[0-9]+]], x0, x1
 ; CHECK-NOT: cmp
-; CHECK: tbz [[CMP]], #63
+; CHECK: tbnz [[CMP]], #63
 
 if.then:
   call void @t()
   br label %if.end
 
 if.end:
   ret void
 }
 
 define void @test14(i1 %cond) {
 ; CHECK-LABEL: @test14
   br i1 %cond, label %if.end, label %if.then
 
 ; CHECK-NOT: and
-; CHECK: tbnz w0, #0
+; CHECK: tbz w0, #0
 
 if.then:
   call void @t()
   br label %if.end
 
 if.end:
   ret void
 }

test/CodeGen/AMDGPU/branch-relaxation.ll

 ; GCN-NEXT: ;;#ASMSTART
 ; GCN-NEXT: v_nop_e64
 ; GCN-NEXT: v_nop_e64
 ; GCN-NEXT: v_nop_e64
 ; GCN-NEXT: v_nop_e64
 ; GCN-NEXT: ;;#ASMEND
 
 ; GCN-NEXT: [[BB3]]: ; %bb3
+; GCN-NEXT: ;;#ASMSTART
+; GCN-NEXT: v_nop_e64
+; GCN-NEXT: ;;#ASMEND
+; GCN-NEXT: ;;#ASMSTART
+; GCN-NEXT: v_nop_e64
+; GCN-NEXT: ;;#ASMEND
 ; GCN-NEXT: s_endpgm
 define void @expand_requires_expand(i32 %cond0) #0 {
 bb0:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #0
   %cmp0 = icmp slt i32 %cond0, 0
   br i1 %cmp0, label %bb2, label %bb1
 
 bb1:
   %val = load volatile i32, i32 addrspace(2)* undef
   %cmp1 = icmp eq i32 %val, 3
   br i1 %cmp1, label %bb3, label %bb2
 
 bb2:
   call void asm sideeffect
    "v_nop_e64
     v_nop_e64
     v_nop_e64
     v_nop_e64", ""() #0
   br label %bb3
 
 bb3:
+; These NOPs prevent tail-duplication-based outlining
+; from firing, which defeats the need to expand the branches and this test.
+  call void asm sideeffect
+   "v_nop_e64", ""() #0
+  call void asm sideeffect
+   "v_nop_e64", ""() #0
   ret void
 }
 
 ; Requires expanding of required skip branch.
 
 ; GCN-LABEL: {{^}}uniform_inside_divergent:
 ; GCN: v_cmp_gt_u32_e32 vcc, 16, v{{[0-9]+}}
 ; GCN-NEXT: s_and_saveexec_b64 [[MASK:s\[[0-9]+:[0-9]+\]]], vcc

 ; GCN: s_cbranch_scc1 [[ENDIF]]
 
 ; GCN-NEXT: ; BB#2: ; %if_uniform
 ; GCN: buffer_store_dword
 ; GCN: s_waitcnt vmcnt(0)
 
 ; GCN-NEXT: [[ENDIF]]: ; %endif
 ; GCN-NEXT: s_or_b64 exec, exec, [[MASK]]
+; GCN-NEXT: s_sleep 5
 ; GCN-NEXT: s_endpgm
 define void @uniform_inside_divergent(i32 addrspace(1)* %out, i32 %cond) #0 {
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %d_cmp = icmp ult i32 %tid, 16
   br i1 %d_cmp, label %if, label %endif
 
 if:
   store i32 0, i32 addrspace(1)* %out
   %u_cmp = icmp eq i32 %cond, 0
   br i1 %u_cmp, label %if_uniform, label %endif
 
 if_uniform:
   store i32 1, i32 addrspace(1)* %out
   br label %endif
 
 endif:
+  ; layout can remove the split branch if it can copy the return block.
+  ; This call makes the return block long enough that it doesn't get copied.
+  call void @llvm.amdgcn.s.sleep(i32 5);
   ret void
 }
 
 ; si_mask_branch
 ; s_cbranch_execz
 ; s_branch
 
 ; GCN-LABEL: {{^}}analyze_mask_branch:

test/CodeGen/AMDGPU/si-annotate-cf-noloop.ll

bb5:                                              ; preds = %bb3, %bb1
   unreachable
 }
 
 
 ; OPT-LABEL: @annotate_ret_noloop(
 ; OPT-NOT: call i1 @llvm.amdgcn.loop
 
 ; GCN-LABEL: {{^}}annotate_ret_noloop:
-; GCN: s_cbranch_vccnz
+; GCN: s_cbranch_vccz [[BODY:BB[0-9]+_[0-9]+]]
+; GCN: s_endpgm
+
+; GCN: {{^}}[[BODY]]:
 ; GCN: s_endpgm
 ; GCN: .Lfunc_end1
 define void @annotate_ret_noloop(<4 x float> addrspace(1)* noalias nocapture readonly %arg) #0 {
 bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
   br label %bb1
 
 bb1:                                              ; preds = %bb

test/CodeGen/AMDGPU/skip-if-dead.ll

 ; CHECK: v_cndmask_b32_e64 [[PHIREG:v[0-9]+]], 0, -1.0,
 ; CHECK: v_cmpx_le_f32_e32 vcc, 0,
 ; CHECK-NEXT: s_cbranch_execnz [[BB4:BB[0-9]+_[0-9]+]]
 
 ; CHECK: exp
 ; CHECK-NEXT: s_endpgm
 
 ; CHECK: [[KILLBB:BB[0-9]+_[0-9]+]]:
-; CHECK-NEXT: s_cbranch_vccz [[PHIBB:BB[0-9]+_[0-9]+]]
+; CHECK-NEXT: s_cbranch_vccnz [[BB8:BB[0-9]+_[0-9]+]]
 
-; CHECK: [[PHIBB]]:
 ; CHECK: v_cmp_eq_f32_e32 vcc, 0, [[PHIREG]]
-; CHECK-NEXT: s_cbranch_vccz [[ENDBB:BB[0-9]+_[0-9]+]]
+; CHECK-NEXT: s_cbranch_vccnz [[BB10:BB[0-9]+_[0-9]+]]
+; CHECK-NEXT: s_branch [[END:BB[0-9]+_[0-9]+]]
 
-; CHECK: ; %bb10
+; CHECK [[BB8]]: ; %BB8
+; CHECK: v_mov_b32_e32 v{{[0-9]+}}, 8
+; CHECK: buffer_store_dword
+; CHECK: v_cmp_eq_f32_e32 vcc, 0, [[PHIREG]]
+; CHECK-NEXT: s_cbranch_vccz [[END]]
+
+; CHECK: [[BB10]]: ; %bb10
 ; CHECK: v_mov_b32_e32 v{{[0-9]+}}, 9
 ; CHECK: buffer_store_dword
 
-; CHECK: [[ENDBB]]:
+; CHECK: [[END:BB[0-9]+_[0-9]+]]: ; %end
 ; CHECK-NEXT: s_endpgm
+
 define amdgpu_ps void @phi_use_def_before_kill() #0 {
 bb:
   %tmp = fadd float undef, 1.000000e+00
   %tmp1 = fcmp olt float 0.000000e+00, %tmp
   %tmp2 = select i1 %tmp1, float -1.000000e+00, float 0.000000e+00
   call void @llvm.AMDGPU.kill(float %tmp2)
   br i1 undef, label %phibb, label %bb8
 

test/CodeGen/AMDGPU/uniform-cfg.ll

ENDIF:                                            ; preds = %IF, %main_body
   ret void
 }
 
 ; GCN-LABEL: {{^}}icmp_users_different_blocks:
 ; GCN: s_load_dword [[COND:s[0-9]+]]
 ; GCN: s_cmp_lt_i32 [[COND]], 1
 ; GCN: s_cbranch_scc1 [[EXIT:[A-Za-z0-9_]+]]
 ; GCN: v_cmp_gt_i32_e64 vcc, [[COND]], 0{{$}}
-; GCN: s_cbranch_vccnz [[EXIT]]
-; GCN: buffer_store
+; GCN: s_cbranch_vccz [[BODY:[A-Za-z0-9_]+]]
 ; GCN: {{^}}[[EXIT]]:
 ; GCN: s_endpgm
+; GCN: {{^}}[[BODY]]:
+; GCN: buffer_store
+; GCN: s_endpgm
 define void @icmp_users_different_blocks(i32 %cond0, i32 %cond1, i32 addrspace(1)* %out) {
 bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #0
   %cmp0 = icmp sgt i32 %cond0, 0
   %cmp1 = icmp sgt i32 %cond1, 0
   br i1 %cmp0, label %bb2, label %bb9
 
 bb2:                                              ; preds = %bb

 }
 
 ; Test uniform and divergent.
 
 ; GCN-LABEL: {{^}}uniform_inside_divergent:
 ; GCN: v_cmp_gt_u32_e32 vcc, 16, v{{[0-9]+}}
 ; GCN: s_and_saveexec_b64 [[MASK:s\[[0-9]+:[0-9]+\]]], vcc
 ; GCN: s_xor_b64  [[MASK1:s\[[0-9]+:[0-9]+\]]], exec, [[MASK]]
-; GCN: s_cbranch_execz [[ENDIF_LABEL:[0-9_A-Za-z]+]]
 ; GCN: s_cmp_lg_u32 {{s[0-9]+}}, 0
-; GCN: s_cbranch_scc1 [[ENDIF_LABEL]]
+; GCN: s_cbranch_scc0 [[IF_UNIFORM_LABEL:[A-Z0-9_a-z]+]]
+; GCN: s_endpgm
+; GCN: {{^}}[[IF_UNIFORM_LABEL]]:
 ; GCN: v_mov_b32_e32 [[ONE:v[0-9]+]], 1
 ; GCN: buffer_store_dword [[ONE]]
 define void @uniform_inside_divergent(i32 addrspace(1)* %out, i32 %cond) {
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x() #0
   %d_cmp = icmp ult i32 %tid, 16
   br i1 %d_cmp, label %if, label %endif
 
 if:
   store i32 0, i32 addrspace(1)* %out
   %u_cmp = icmp eq i32 %cond, 0
   br i1 %u_cmp, label %if_uniform, label %endif
 
 if_uniform:
   store i32 1, i32 addrspace(1)* %out
   br label %endif
 
 endif:
   ret void
 }
 
 ; GCN-LABEL: {{^}}divergent_inside_uniform:
 ; GCN: s_cmp_lg_u32 s{{[0-9]+}}, 0
-; GCN: s_cbranch_scc1 [[ENDIF_LABEL:[0-9_A-Za-z]+]]
+; GCN: s_cbranch_scc0 [[IF_LABEL:[0-9_A-Za-z]+]]
+; GCN: [[IF_LABEL]]:
 ; GCN: v_cmp_gt_u32_e32 vcc, 16, v{{[0-9]+}}
 ; GCN: s_and_saveexec_b64 [[MASK:s\[[0-9]+:[0-9]+\]]], vcc
 ; GCN: s_xor_b64  [[MASK1:s\[[0-9]+:[0-9]+\]]], exec, [[MASK]]
 ; GCN: v_mov_b32_e32 [[ONE:v[0-9]+]], 1
 ; GCN: buffer_store_dword [[ONE]]
-; GCN: [[ENDIF_LABEL]]:
-; GCN: s_endpgm
 define void @divergent_inside_uniform(i32 addrspace(1)* %out, i32 %cond) {
 entry:
   %u_cmp = icmp eq i32 %cond, 0
   br i1 %u_cmp, label %if, label %endif
 
 if:
   store i32 0, i32 addrspace(1)* %out
   %tid = call i32 @llvm.amdgcn.workitem.id.x() #0

 ; GCN-LABEL: {{^}}divergent_if_uniform_if:
 ; GCN: v_cmp_eq_u32_e32 vcc, 0, v0
 ; GCN: s_and_saveexec_b64 [[MASK:s\[[0-9]+:[0-9]+\]]], vcc
 ; GCN: s_xor_b64 [[MASK:s\[[0-9]+:[0-9]+\]]], exec, [[MASK]]
 ; GCN: v_mov_b32_e32 [[ONE:v[0-9]+]], 1
 ; GCN: buffer_store_dword [[ONE]]
 ; GCN: s_or_b64 exec, exec, [[MASK]]
 ; GCN: s_cmp_lg_u32 s{{[0-9]+}}, 0
-; GCN: s_cbranch_scc1 [[EXIT:[A-Z0-9_]+]]
+; GCN: s_cbranch_scc0 [[IF_UNIFORM:[A-Z0-9_]+]]
+; GCN: s_endpgm
+; GCN: [[IF_UNIFORM]]:
 ; GCN: v_mov_b32_e32 [[TWO:v[0-9]+]], 2
 ; GCN: buffer_store_dword [[TWO]]
-; GCN: [[EXIT]]:
-; GCN: s_endpgm
 define void @divergent_if_uniform_if(i32 addrspace(1)* %out, i32 %cond) {
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x() #0
   %d_cmp = icmp eq i32 %tid, 0
   br i1 %d_cmp, label %if, label %endif
 
 if:
   store i32 1, i32 addrspace(1)* %out

 ; The condition of the branches in the two blocks are
 ; uniform. MachineCSE replaces the 2nd condition with the inverse of
 ; the first, leaving an scc use in a different block than it was
 ; defed.
 
 ; GCN-LABEL: {{^}}cse_uniform_condition_different_blocks:
 ; GCN: s_load_dword [[COND:s[0-9]+]]
 ; GCN: s_cmp_lt_i32 [[COND]], 1
-; GCN: s_cbranch_scc1 BB[[FNNUM:[0-9]+]]_3
+; GCN: s_cbranch_scc1 [[FN:BB[0-9_]+]]
 
 ; GCN: BB#1:
 ; GCN-NOT: cmp
 ; GCN: buffer_load_dword
 ; GCN: buffer_store_dword
-; GCN: s_cbranch_scc1 BB[[FNNUM]]_3
+; GCN: s_cbranch_scc0 [[BB7:BB[0-9_]+]]
 
-; GCN: BB[[FNNUM]]_3:
+; GCN: [[FN]]:
 ; GCN: s_endpgm
+
+; GCN: [[BB7]]:
+; GCN: s_endpgm
+
 define void @cse_uniform_condition_different_blocks(i32 %cond, i32 addrspace(1)* %out) {
 bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x() #0
   %tmp1 = icmp sgt i32 %cond, 0
   br i1 %tmp1, label %bb2, label %bb9
 
 bb2:                                              ; preds = %bb
   %tmp3 = load volatile i32, i32 addrspace(1)* undef

test/CodeGen/ARM/arm-and-tst-peephole.ll

 ; V8-LABEL: %tailrecurse.switch
 ; V8: cmp
 ; V8-NEXT: beq
 ; V8-NEXT: %tailrecurse.switch
 ; V8: cmp
 ; V8-NEXT: beq
 ; V8-NEXT: %tailrecurse.switch
 ; V8: cmp
-; V8-NEXT: bne
-; V8-NEXT: b  
-; The trailing space in the last line checks that the branch is unconditional
+; V8-NEXT: beq
+; V8-NEXT: %sw.epilog
+; V8-NEXT: bx lr
   switch i32 %and, label %sw.epilog [
     i32 1, label %sw.bb
     i32 3, label %sw.bb6
     i32 2, label %sw.bb8
   ], !prof !1
 
 sw.bb:                                            ; preds = %tailrecurse.switch, %tailrecurse
   %shl = shl i32 %acc.tr, 1

test/CodeGen/ARM/atomic-op.ll

 ; CHECK-NOT:     dmb ish
 ; CHECK: [[LOOP_BB:\.?LBB[0-9]+_1]]:
 ; CHECK:     ldrex   [[OLDVAL:r[0-9]+]], [r[[ADDR:[0-9]+]]]
 ; CHECK:     cmp     [[OLDVAL]], r1
 ; CHECK:     bne     [[FAIL_BB:\.?LBB[0-9]+_[0-9]+]]
 ; CHECK:     strex   [[SUCCESS:r[0-9]+]], r2, [r[[ADDR]]]
 ; CHECK:     cmp     [[SUCCESS]], #0
 ; CHECK:     bne     [[LOOP_BB]]
-; CHECK:     b       [[END_BB:\.?LBB[0-9]+_[0-9]+]]
+; CHECK:     dmb     ish
+; CHECK:     bx      lr
 ; CHECK: [[FAIL_BB]]:
 ; CHECK-NEXT: clrex
-; CHECK-NEXT: [[END_BB]]:
 ; CHECK:     dmb     ish
 ; CHECK:     bx      lr
 
   ret i32 %oldval
 }
 
 define i32 @load_load_add_acquire(i32* %mem1, i32* %mem2) nounwind {
 ; CHECK-LABEL: load_load_add_acquire

test/CodeGen/ARM/atomic-ops-v8.ll

 ; CHECK-THUMB-DAG: mov r[[WANTED:[0-9]+]], r0
 
 ; CHECK: .LBB{{[0-9]+}}_1:
 ; CHECK: ldaexb r[[OLD:[0-9]+]], [r[[ADDR]]]
   ; r0 below is a reasonable guess but could change: it certainly comes into the
   ;  function there.
 ; CHECK-ARM-NEXT:   cmp r[[OLD]], r0
 ; CHECK-THUMB-NEXT: cmp r[[OLD]], r[[WANTED]]
-; CHECK-NEXT: bne .LBB{{[0-9]+}}_3
+; CHECK-NEXT: bne .LBB{{[0-9]+}}_4
 ; CHECK-NEXT: BB#2:
   ; As above, r1 is a reasonable guess.
 ; CHECK: strexb [[STATUS:r[0-9]+]], r1, [r[[ADDR]]]
 ; CHECK-NEXT: cmp [[STATUS]], #0
 ; CHECK-NEXT: bne .LBB{{[0-9]+}}_1
-; CHECK-NEXT: b .LBB{{[0-9]+}}_4
-; CHECK-NEXT: .LBB{{[0-9]+}}_3:
-; CHECK-NEXT: clrex
+; CHECK-ARM: mov r0, r[[OLD]]
+; CHECK: bx lr
 ; CHECK-NEXT: .LBB{{[0-9]+}}_4:
+; CHECK-NEXT: clrex
 ; CHECK-NOT: dmb
 ; CHECK-NOT: mcr
 
 ; CHECK-ARM: mov r0, r[[OLD]]
+; CHECK-ARM-NEXT: bx lr
    ret i8 %old
 }
 
 define i16 @test_atomic_cmpxchg_i16(i16 zeroext %wanted, i16 zeroext %new) nounwind {
 ; CHECK-LABEL: test_atomic_cmpxchg_i16:
    %pair = cmpxchg i16* @var16, i16 %wanted, i16 %new seq_cst seq_cst
    %old = extractvalue { i16, i1 } %pair, 0
 ; CHECK-NOT: dmb
 ; CHECK-NOT: mcr
 ; CHECK-DAG: movw r[[ADDR:[0-9]+]], :lower16:var16
 ; CHECK-DAG: movt r[[ADDR]], :upper16:var16
 ; CHECK-THUMB-DAG: mov r[[WANTED:[0-9]+]], r0
 
 ; CHECK: .LBB{{[0-9]+}}_1:
 ; CHECK: ldaexh r[[OLD:[0-9]+]], [r[[ADDR]]]
   ; r0 below is a reasonable guess but could change: it certainly comes into the
   ;  function there.
 ; CHECK-ARM-NEXT:   cmp r[[OLD]], r0
 ; CHECK-THUMB-NEXT: cmp r[[OLD]], r[[WANTED]]
-; CHECK-NEXT: bne .LBB{{[0-9]+}}_3
+; CHECK-NEXT: bne .LBB{{[0-9]+}}_4
 ; CHECK-NEXT: BB#2:
   ; As above, r1 is a reasonable guess.
 ; CHECK: stlexh [[STATUS:r[0-9]+]], r1, [r[[ADDR]]]
 ; CHECK-NEXT: cmp [[STATUS]], #0
 ; CHECK-NEXT: bne .LBB{{[0-9]+}}_1
-; CHECK-NEXT: b .LBB{{[0-9]+}}_4
-; CHECK-NEXT: .LBB{{[0-9]+}}_3:
-; CHECK-NEXT: clrex
+; CHECK-ARM: mov r0, r[[OLD]]
+; CHECK: bx lr
 ; CHECK-NEXT: .LBB{{[0-9]+}}_4:
+; CHECK-NEXT: clrex
 ; CHECK-NOT: dmb
 ; CHECK-NOT: mcr
 
 ; CHECK-ARM: mov r0, r[[OLD]]
+; CHECK-ARM-NEXT: bx lr
    ret i16 %old
 }
 
 define void @test_atomic_cmpxchg_i32(i32 %wanted, i32 %new) nounwind {
 ; CHECK-LABEL: test_atomic_cmpxchg_i32:
    %pair = cmpxchg i32* @var32, i32 %wanted, i32 %new release monotonic
    %old = extractvalue { i32, i1 } %pair, 0
    store i32 %old, i32* @var32
 ; CHECK-NOT: dmb
 ; CHECK-NOT: mcr
 ; CHECK: movw r[[ADDR:[0-9]+]], :lower16:var32
 ; CHECK: movt r[[ADDR]], :upper16:var32
 
 ; CHECK: .LBB{{[0-9]+}}_1:
 ; CHECK: ldrex r[[OLD:[0-9]+]], [r[[ADDR]]]
   ; r0 below is a reasonable guess but could change: it certainly comes into the
   ;  function there.
 ; CHECK-NEXT: cmp r[[OLD]], r0
-; CHECK-NEXT: bne .LBB{{[0-9]+}}_3
+; CHECK-NEXT: bne .LBB{{[0-9]+}}_4
 ; CHECK-NEXT: BB#2:
   ; As above, r1 is a reasonable guess.
 ; CHECK: stlex [[STATUS:r[0-9]+]], r1, [r[[ADDR]]]
 ; CHECK-NEXT: cmp [[STATUS]], #0
 ; CHECK-NEXT: bne .LBB{{[0-9]+}}_1
-; CHECK-NEXT: b .LBB{{[0-9]+}}_4
-; CHECK-NEXT: .LBB{{[0-9]+}}_3:
-; CHECK-NEXT: clrex
+; CHECK: str{{(.w)?}} r[[OLD]],
+; CHECK-NEXT: bx lr
 ; CHECK-NEXT: .LBB{{[0-9]+}}_4:
+; CHECK-NEXT: clrex
 ; CHECK-NOT: dmb
 ; CHECK-NOT: mcr
 
 ; CHECK: str{{(.w)?}} r[[OLD]],
+; CHECK-ARM-NEXT: bx lr
    ret void
 }
 
 define void @test_atomic_cmpxchg_i64(i64 %wanted, i64 %new) nounwind {
 ; CHECK-LABEL: test_atomic_cmpxchg_i64:
    %pair = cmpxchg i64* @var64, i64 %wanted, i64 %new monotonic monotonic
    %old = extractvalue { i64, i1 } %pair, 0
 ; CHECK-NOT: dmb
 ; CHECK-NOT: mcr
 ; CHECK: movw r[[ADDR:[0-9]+]], :lower16:var64
 ; CHECK: movt r[[ADDR]], :upper16:var64
 
 ; CHECK: .LBB{{[0-9]+}}_1:
 ; CHECK: ldrexd [[OLD1:r[0-9]+|lr]], [[OLD2:r[0-9]+|lr]], [r[[ADDR]]]
   ; r0, r1 below is a reasonable guess but could change: it certainly comes into the
   ; function there.
 ; CHECK-LE-DAG: eor{{(\.w)?}} [[MISMATCH_LO:r[0-9]+|lr]], [[OLD1]], r0
 ; CHECK-LE-DAG: eor{{(\.w)?}} [[MISMATCH_HI:r[0-9]+|lr]], [[OLD2]], r1
 ; CHECK-ARM-LE: orrs{{(\.w)?}} {{r[0-9]+}}, [[MISMATCH_LO]], [[MISMATCH_HI]]
 ; CHECK-THUMB-LE: orrs{{(\.w)?}} {{(r[0-9]+, )?}}[[MISMATCH_HI]], [[MISMATCH_LO]]
 ; CHECK-BE-DAG: eor{{(\.w)?}} [[MISMATCH_HI:r[0-9]+|lr]], [[OLD2]], r1
 ; CHECK-BE-DAG: eor{{(\.w)?}} [[MISMATCH_LO:r[0-9]+|lr]], [[OLD1]], r0
 ; CHECK-ARM-BE: orrs{{(\.w)?}} {{r[0-9]+}}, [[MISMATCH_HI]], [[MISMATCH_LO]]
 ; CHECK-THUMB-BE: orrs{{(\.w)?}} {{(r[0-9]+, )?}}[[MISMATCH_LO]], [[MISMATCH_HI]]
-; CHECK-NEXT: bne .LBB{{[0-9]+}}_3
+; CHECK-NEXT: bne .LBB{{[0-9]+}}_4
 ; CHECK-NEXT: BB#2:
   ; As above, r2, r3 is a reasonable guess.
 ; CHECK: strexd [[STATUS:r[0-9]+]], r2, r3, [r[[ADDR]]]
 ; CHECK-NEXT: cmp [[STATUS]], #0
 ; CHECK-NEXT: bne .LBB{{[0-9]+}}_1
-; CHECK-NEXT: b .LBB{{[0-9]+}}_4
-; CHECK-NEXT: .LBB{{[0-9]+}}_3:
-; CHECK-NEXT: clrex
+; CHECK: strd [[OLD1]], [[OLD2]], [r[[ADDR]]]
+; CHECK-NEXT: pop
 ; CHECK-NEXT: .LBB{{[0-9]+}}_4:
+; CHECK-NEXT: clrex
 ; CHECK-NOT: dmb
 ; CHECK-NOT: mcr
 
 ; CHECK-ARM: strd [[OLD1]], [[OLD2]], [r[[ADDR]]]
    store i64 %old, i64* @var64
    ret void
 }
 

test/CodeGen/ARM/cmpxchg-weak.ll

 ; RUN: llc < %s -mtriple=armv7-apple-ios -verify-machineinstrs | FileCheck %s
 
 define void @test_cmpxchg_weak(i32 *%addr, i32 %desired, i32 %new) {
 ; CHECK-LABEL: test_cmpxchg_weak:
 
   %pair = cmpxchg weak i32* %addr, i32 %desired, i32 %new seq_cst monotonic
   %oldval = extractvalue { i32, i1 } %pair, 0
 ; CHECK-NEXT: BB#0:
 ; CHECK-NEXT:     ldrex   [[LOADED:r[0-9]+]], [r0]
 ; CHECK-NEXT:     cmp     [[LOADED]], r1
 ; CHECK-NEXT:     bne     [[LDFAILBB:LBB[0-9]+_[0-9]+]]
 ; CHECK-NEXT: BB#1:
 ; CHECK-NEXT:     dmb ish
 ; CHECK-NEXT:     strex   [[SUCCESS:r[0-9]+]], r2, [r0]
 ; CHECK-NEXT:     cmp     [[SUCCESS]], #0
-; CHECK-NEXT:     bne     [[FAILBB:LBB[0-9]+_[0-9]+]]
+; CHECK-NEXT:     beq     [[SUCCESSBB:LBB[0-9]+_[0-9]+]]
 ; CHECK-NEXT: BB#2:
-; CHECK-NEXT:     dmb     ish
 ; CHECK-NEXT:     str     r3, [r0]
 ; CHECK-NEXT:     bx      lr
 ; CHECK-NEXT: [[LDFAILBB]]:
 ; CHECK-NEXT:     clrex
-; CHECK-NEXT: [[FAILBB]]:
+; CHECK-NEXT:     str     r3, [r0]
+; CHECK-NEXT:     bx      lr
+; CHECK-NEXT: [[SUCCESSBB]]:
+; CHECK-NEXT:     dmb     ish
 ; CHECK-NEXT:     str     r3, [r0]
 ; CHECK-NEXT:     bx      lr
 
   store i32 %oldval, i32* %addr
   ret void
 }
 
 

test/CodeGen/ARM/machine-cse-cmp.ll

 declare void @llvm.memset.p0i8.i32(i8* nocapture, i8, i32, i32, i1) nounwind
 
 ; rdar://12462006
 define i8* @f3(i8* %base, i32* nocapture %offset, i32 %size) nounwind {
 entry:
 ; CHECK-LABEL: f3:
 ; CHECK-NOT: sub
 ; CHECK: cmp
-; CHECK: blt
+; CHECK: bge
 %0 = load i32, i32* %offset, align 4
 %cmp = icmp slt i32 %0, %size
 %s = sub nsw i32 %0, %size
 %size2 = sub nsw i32 %size, 0
 br i1 %cmp, label %return, label %if.end
 
 if.end:
 ; We are checking cse between %sub here and %s in entry block.

test/CodeGen/Mips/brconeq.ll

 ; RUN: llc  -march=mipsel -mattr=mips16 -relocation-model=pic -O3 < %s | FileCheck %s -check-prefix=16
 
 @i = global i32 5, align 4
 @j = global i32 10, align 4
 @result = global i32 0, align 4
 
 define void @test() nounwind {
 entry:
   %0 = load i32, i32* @i, align 4
   %1 = load i32, i32* @j, align 4
-  %cmp = icmp eq i32 %0, %1
+  %cmp = icmp ne i32 %0, %1
 ; 16: cmp ${{[0-9]+}}, ${{[0-9]+}}
 ; 16: bteqz $[[LABEL:[0-9A-Ba-b_]+]]
 ; 16: $[[LABEL]]:
-  br i1 %cmp, label %if.end, label %if.then
+  br i1 %cmp, label %if.then, label %if.end
 
 if.then:                                          ; preds = %entry
   store i32 1, i32* @result, align 4
   br label %if.end
 
 if.end:                                           ; preds = %entry, %if.then
   ret void
 }

test/CodeGen/Mips/brconeqk.ll

 ; RUN: llc  -march=mipsel -mattr=mips16 -relocation-model=pic -O3 < %s | FileCheck %s -check-prefix=16
 
 @i = global i32 5, align 4
 @result = global i32 0, align 4
 
 define void @test() nounwind {
 entry:
   %0 = load i32, i32* @i, align 4
-  %cmp = icmp eq i32 %0, 10
-  br i1 %cmp, label %if.end, label %if.then
+  %cmp = icmp ne i32 %0, 10
+  br i1 %cmp, label %if.then, label %if.end
 ; 16: cmpi  ${{[0-9]+}}, {{[0-9]+}}
 ; 16: bteqz $[[LABEL:[0-9A-Ba-b_]+]]
 ; 16: $[[LABEL]]:
 if.then:                                          ; preds = %entry
   store i32 1, i32* @result, align 4
   br label %if.end
 
 if.end:                                           ; preds = %entry, %if.then
   ret void
 }
 
 

test/CodeGen/Mips/brcongt.ll

 ; RUN: llc  -march=mipsel -mattr=mips16 -relocation-model=pic -O3 < %s | FileCheck %s -check-prefix=16
 
 @i = global i32 5, align 4
 @j = global i32 10, align 4
 @k = global i32 5, align 4
 @result = global i32 0, align 4
 
 define void @test() nounwind {
 entry:
   %0 = load i32, i32* @i, align 4
   %1 = load i32, i32* @j, align 4
-  %cmp = icmp sgt i32 %0, %1
-  br i1 %cmp, label %if.end, label %if.then
+  %cmp = icmp sle i32 %0, %1
+  br i1 %cmp, label %if.then, label %if.end
 ; 16: slt ${{[0-9]+}}, ${{[0-9]+}}
 ; 16: btnez $[[LABEL:[0-9A-Ba-b_]+]]
 ; 16: $[[LABEL]]:
 if.then:                                          ; preds = %entry
   store i32 1, i32* @result, align 4
   br label %if.end
 
 if.end:                                           ; preds = %entry, %if.then
   ret void
 }
 
 

test/CodeGen/Mips/brconlt.ll

 ; RUN: llc  -march=mipsel -mattr=mips16 -relocation-model=pic -O3 < %s | FileCheck %s -check-prefix=16
 ; RUN: llc  -march=mips -mattr=micromips -mcpu=mips32r6 -relocation-model=pic -O3 < %s | FileCheck %s -check-prefix=MM32R6
 
 @i = global i32 5, align 4
 @j = global i32 10, align 4
 @k = global i32 5, align 4
 @result = global i32 0, align 4
 
 define void @test() nounwind {
 entry:
   %0 = load i32, i32* @j, align 4
   %1 = load i32, i32* @i, align 4
-  %cmp = icmp slt i32 %0, %1
-  br i1 %cmp, label %if.end, label %if.then
+  %cmp = icmp sge i32 %0, %1
+  br i1 %cmp, label %if.then, label %if.end
 
 ; 16:     slt   ${{[0-9]+}}, ${{[0-9]+}}
 ; MM32R6: slt   ${{[0-9]+}}, ${{[0-9]+}}
 ; 16:     btnez $[[LABEL:[0-9A-Ba-b_]+]]
 ; 16:     $[[LABEL]]:
 
 if.then:                                          ; preds = %entry
   store i32 1, i32* @result, align 4
   br label %if.end
 
 if.end:                                           ; preds = %entry, %if.then
   ret void
 }
 
 

test/CodeGen/Mips/brconnez.ll

 ; RUN: llc  -march=mipsel -mattr=mips16 -relocation-model=pic -O3 < %s | FileCheck %s -check-prefix=16
 
 @j = global i32 0, align 4
 @result = global i32 0, align 4
 
 define void @test() nounwind {
 entry:
   %0 = load i32, i32* @j, align 4
   %cmp = icmp eq i32 %0, 0
-  br i1 %cmp, label %if.then, label %if.end
+  br i1 %cmp, label %if.then, label %if.end, !prof !1
 
 ; 16: bnez  ${{[0-9]+}}, $[[LABEL:[0-9A-Ba-b_]+]]
 ; 16:   lw ${{[0-9]+}}, %got(result)(${{[0-9]+}})
 ; 16: $[[LABEL]]:
 
 if.then:                                          ; preds = %entry
   store i32 1, i32* @result, align 4
   br label %if.end
 
 if.end:                                           ; preds = %if.then, %entry
   ret void
 }
 
-
+!1 = !{!"branch_weights", i32 2, i32 1}

test/CodeGen/Mips/llvm-ir/ashr.ll

; ALL-LABEL: ashr_i64:
   ; M2:         bnez      $[[T1]], $[[BB0:BB[0-9_]+]]
   ; M2:         move      $3, $[[T0]]
   ; M2:         srlv      $[[T2:[0-9]+]], $5, $7
   ; M2:         not       $[[T3:[0-9]+]], $7
   ; M2:         sll       $[[T4:[0-9]+]], $4, 1
   ; M2:         sllv      $[[T5:[0-9]+]], $[[T4]], $[[T3]]
   ; M2:         or        $3, $[[T3]], $[[T2]]
   ; M2:         $[[BB0]]:
-  ; M2:         beqz      $[[T1]], $[[BB1:BB[0-9_]+]]
+  ; M2:         bnez      $[[T1]], $[[BB1:BB[0-9_]+]]
   ; M2:         nop
-  ; M2:         sra       $2, $4, 31
-  ; M2:         $[[BB1]]:
   ; M2:         jr        $ra
   ; M2:         nop
+  ; M2:         $[[BB1]]:
+  ; M2:         jr        $ra
+  ; M2:         sra       $2, $4, 31
 
   ; 32R1-R5:    srlv      $[[T0:[0-9]+]], $5, $7
   ; 32R1-R5:    not       $[[T1:[0-9]+]], $7
   ; 32R1-R5:    sll       $[[T2:[0-9]+]], $4, 1
   ; 32R1-R5:    sllv      $[[T3:[0-9]+]], $[[T2]], $[[T1]]
   ; 32R1-R5:    or        $3, $[[T3]], $[[T0]]
   ; 32R1-R5:    srav      $[[T4:[0-9]+]], $4, $7
   ; 32R1-R5:    andi      $[[T5:[0-9]+]], $7, 32
; ALL-LABEL: ashr_i128:
   ; M3:             bnez      $[[T3:[0-9]+]], [[BB0:.LBB[0-9_]+]]
   ; M3:             move      $3, $[[T1]]
   ; M3:             dsrlv     $[[T4:[0-9]+]], $5, $7
   ; M3:             dsll      $[[T5:[0-9]+]], $4, 1
   ; M3:             not       $[[T6:[0-9]+]], $[[T0]]
   ; M3:             dsllv     $[[T7:[0-9]+]], $[[T5]], $[[T6]]
   ; M3:             or        $3, $[[T7]], $[[T4]]
   ; M3:             [[BB0]]:
-  ; M3:             beqz      $[[T3]], [[BB1:.LBB[0-9_]+]]
+  ; M3:             bnez      $[[T3]], [[BB1:.LBB[0-9_]+]]
   ; M3:             nop
-  ; M3:             dsra      $2, $4, 63
-  ; M3:             [[BB1]]:
   ; M3:             jr        $ra
   ; M3:             nop
+  ; M3:             [[BB1]]:
+  ; M3:             jr        $ra
+  ; M3:             dsra      $2, $4, 63
 
   ; GP64-NOT-R6:    dsrlv     $[[T0:[0-9]+]], $5, $7
   ; GP64-NOT-R6:    dsll      $[[T1:[0-9]+]], $4, 1
   ; GP64-NOT-R6:    sll       $[[T2:[0-9]+]], $7, 0
   ; GP64-NOT-R6:    not       $[[T3:[0-9]+]], $[[T2]]
   ; GP64-NOT-R6:    dsllv     $[[T4:[0-9]+]], $[[T1]], $[[T3]]
   ; GP64-NOT-R6:    or        $3, $[[T4]], $[[T0]]
   ; GP64-NOT-R6:    dsrav     $2, $4, $7

test/CodeGen/Mips/micromips-compact-branches.ll

 ; RUN: llc %s -march=mipsel -mattr=micromips -filetype=asm -O3 \
 ; RUN: -disable-mips-delay-filler -relocation-model=pic -o - | FileCheck %s
 
 define void @main() nounwind uwtable {
 entry:
   %x = alloca i32, align 4
   %0 = load i32, i32* %x, align 4
   %cmp = icmp eq i32 %0, 0
-  br i1 %cmp, label %if.then, label %if.end
+  br i1 %cmp, label %if.then, label %if.end, !prof !1
 
 if.then:
   store i32 10, i32* %x, align 4
   br label %if.end
 
 if.end:
   ret void
 }
 
 ; CHECK: bnezc
+!1 = !{!"branch_weights", i32 2, i32 1}

test/CodeGen/PowerPC/misched-inorder-latency.ll

 ; CHECK: addi
 ; CHECK: bne
 ; CHECK: %true
 define i32 @testload(i32 *%ptr, i32 %sumin) {
 entry:
   %sum1 = add i32 %sumin, 1
   %val1 = load i32, i32* %ptr
   %p = icmp eq i32 %sumin, 0
-  br i1 %p, label %true, label %end
+  br i1 %p, label %true, label %end, !prof !1
 true:
   %sum2 = add i32 %sum1, 1
   %ptr2 = getelementptr i32, i32* %ptr, i32 1
   %val = load i32, i32* %ptr2
   %val2 = add i32 %val1, %val
   br label %end
 end:
   %valmerge = phi i32 [ %val1, %entry], [ %val2, %true ]

 true:
   %val2 = add i32 %val1, 1
   br label %end
 end:
   %valmerge = phi i32 [ %val1, %entry], [ %val2, %true ]
   ret i32 %valmerge
 }
 declare void @llvm.prefetch(i8*, i32, i32, i32) nounwind
+
+!1 = !{!"branch_weights", i32 2, i32 1}

test/CodeGen/PowerPC/tail-dup-layout.ll

 ; optional3
 ; optional4
 ; Tail duplication puts test n+1 at the end of optional n
 ; so optional1 includes a copy of test2 at the end, and branches
 ; to test3 (at the top) or falls through to optional 2.
 ; The CHECK statements check for the whole string of tests and exit block,
 ; and then check that the correct test has been duplicated into the end of
 ; the optional blocks and that the optional blocks are in the correct order.
-;CHECK-LABEL: f:
+;CHECK-LABEL: straight_test:
 ; test1 may have been merged with entry
 ;CHECK: mr [[TAGREG:[0-9]+]], 3
 ;CHECK: andi. {{[0-9]+}}, [[TAGREG]], 1
 ;CHECK-NEXT: bc 12, 1, [[OPT1LABEL:[._0-9A-Za-z]+]]
 ;CHECK-NEXT: [[TEST2LABEL:[._0-9A-Za-z]+]]: # %test2
 ;CHECK-NEXT: rlwinm. {{[0-9]+}}, [[TAGREG]], 0, 30, 30
 ;CHECK-NEXT: bne 0, [[OPT2LABEL:[._0-9A-Za-z]+]]
 ;CHECK-NEXT: [[TEST3LABEL:[._0-9A-Za-z]+]]: # %test3

 ;CHECK: rlwinm. {{[0-9]+}}, [[TAGREG]], 0, 29, 29
 ;CHECK-NEXT: beq 0, [[TEST4LABEL]]
 ;CHECK-NEXT: [[OPT3LABEL]]
 ;CHECK: rlwinm. {{[0-9]+}}, [[TAGREG]], 0, 28, 28
 ;CHECK-NEXT: beq 0, [[EXITLABEL]]
 ;CHECK-NEXT: [[OPT4LABEL]]
 ;CHECK: b [[EXITLABEL]]
 
-define void @f(i32 %tag) {
+define void @straight_test(i32 %tag) {
 entry:
   br label %test1
 test1:
   %tagbit1 = and i32 %tag, 1
   %tagbit1eq0 = icmp eq i32 %tagbit1, 0
   br i1 %tagbit1eq0, label %test2, label %optional1
 optional1:
   call void @a()
optional4:
   call void @d()
   call void @d()
   call void @d()
   br label %exit
 exit:
   ret void
 }
 
+; The block then2 is not unavoidable, but since it can be tail-duplicated, it
+; should be placed as a fallthrough from test2 and copied.
+; CHECK-LABEL: avoidable_test:
+; CHECK: # %entry
+; CHECK: andi.
+; CHECK: # %test2
+; Make sure then2 falls through from test2
+; CHECK-NOT: # %{{[-_a-zA-Z0-9]+}}
+; CHECK: # %then2
+; CHECK: rlwinm. {{[0-9]+}}, {{[0-9]+}}, 0, 29, 29
+; CHECK-NOT: # %{{[-_a-zA-Z0-9]+}}
+; CHECK: # %end2
+; CHECK: # %else1
+; CHECK: bl a
+; CHECK: bl a
+; Make sure then2 was copied into else1
+; CHECK: rlwinm. {{[0-9]+}}, {{[0-9]+}}, 0, 29, 29
+; CHECK: # %else2
+; CHECK: bl c
+define void @avoidable_test(i32 %tag) {
+entry:
+  br label %test1
+test1:
+  %tagbit1 = and i32 %tag, 1
+  %tagbit1eq0 = icmp eq i32 %tagbit1, 0
+  br i1 %tagbit1eq0, label %test2, label %else1, !prof !1 ; %test2 more likely
+else1:
+  call void @a()
+  call void @a()
+  br label %then2
+test2:
+  %tagbit2 = and i32 %tag, 2
+  %tagbit2eq0 = icmp eq i32 %tagbit2, 0
+  br i1 %tagbit2eq0, label %then2, label %else2, !prof !1 ; %then2 more likely
+then2:
+  %tagbit3 = and i32 %tag, 4
+  %tagbit3eq0 = icmp eq i32 %tagbit3, 0
+  br i1 %tagbit3eq0, label %end2, label %end1, !prof !1 ; %end2 more likely
+else2:
+  call void @c()
+  br label %end2
+end2:
+  ret void
+end1:
+  call void @d()
+  ret void
+}
+
 declare void @a()
 declare void @b()
 declare void @c()
 declare void @d()
+
+!1 = !{!"branch_weights", i32 2, i32 1}

test/CodeGen/SPARC/sjlj.ll

 ; CHECK:  or %i1, %lo(.LBB1_2), %i1
 ; CHECK:  st %i1, [%i0+4]
 ; CHECK:  st %sp, [%i0+8]
 ; CHECK:  bn   .LBB1_2
 ; CHECK:  st %i7, [%i0+12]
 ; CHECK:  ba   .LBB1_1
 ; CHECK:  nop
 ; CHECK:.LBB1_1:                                ! %entry
-; CHECK:  ba   .LBB1_3
 ; CHECK:  mov  %g0, %i0
+; CHECK:  cmp %i0, 0
+; CHECK:  bne  .LBB1_4
+; CHECK:  ba   .LBB1_5
 ; CHECK:.LBB1_2:                                ! Block address taken
 ; CHECK:  mov  1, %i0
-; CHECK:.LBB1_3:                                ! %entry
-; CHECK:  cmp %i0, 0
 ; CHECK:  be   .LBB1_5
-; CHECK:  nop
+; CHECK:.LBB1_4:
+; CHECK:  ba   .LBB1_6
 }
 declare i8* @llvm.frameaddress(i32) #2
 
 declare i8* @llvm.stacksave() #3
 
 declare i32 @llvm.eh.sjlj.setjmp(i8*) #3
 
 attributes #0 = { nounwind "less-precise-fpmad"="false" "no-frame-pointer-elim"="false" "no-frame-pointer-elim-non-leaf" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "unsafe-fp-math"="false" "use-soft-float"="false" }
 attributes #1 = { noreturn nounwind }
 attributes #2 = { nounwind readnone }
 attributes #3 = { nounwind }
 

test/CodeGen/SystemZ/asm-18.ll

 ; Test selects involving high registers.
 define void @f13(i32 %x, i32 %y) {
 ; CHECK-LABEL: f13:
 ; CHECK: llihl [[REG:%r[0-5]]], 0
 ; CHECK: cije %r2, 0
 ; CHECK: iihf [[REG]], 2102030405
 ; CHECK: blah [[REG]]
 ; CHECK: br %r14
-  %cmp = icmp eq i32 %x, 0
+  %cmp = icmp ne i32 %x, 0
   %val = select i1 %cmp, i32 0, i32 2102030405
   call void asm sideeffect "blah $0", "h"(i32 %val)
   ret void
 }
 
 ; Test selects involving low registers.
 define void @f14(i32 %x, i32 %y) {
 ; CHECK-LABEL: f14:
 ; CHECK: lhi [[REG:%r[0-5]]], 0
 ; CHECK: cije %r2, 0
 ; CHECK: iilf [[REG]], 2102030405
 ; CHECK: blah [[REG]]
 ; CHECK: br %r14
-  %cmp = icmp eq i32 %x, 0
+  %cmp = icmp ne i32 %x, 0
   %val = select i1 %cmp, i32 0, i32 2102030405
   call void asm sideeffect "blah $0", "r"(i32 %val)
   ret void
 }
 
 ; Test immediate insertion involving high registers.
 define void @f15() {
 ; CHECK-LABEL: f15:

test/CodeGen/SystemZ/cond-store-01.ll

; CHECK: br %r14
   store i8 %res, i8 *%ptr
   ret void
 }
 
 ; Check that volatile loads are not matched.
 define void @f18(i8 *%ptr, i8 %alt, i32 %limit) {
 ; CHECK-LABEL: f18:
 ; CHECK: lb {{%r[0-5]}}, 0(%r2)
-; CHECK: {{jl|jnl}} [[LABEL:[^ ]*]]
+; CHECK: {{jhe|jnhe}} [[LABEL:[^ ]*]]
+; CHECK: stc {{%r[0-5]}}, 0(%r2)
+; CHECK: br %r14
 ; CHECK: [[LABEL]]:
+; CHECK: lr {{%r[0-5]}}, {{%r[0-5]}}
 ; CHECK: stc {{%r[0-5]}}, 0(%r2)
 ; CHECK: br %r14
   %cond = icmp ult i32 %limit, 420
   %orig = load volatile i8 , i8 *%ptr
   %res = select i1 %cond, i8 %orig, i8 %alt
   store i8 %res, i8 *%ptr
   ret void
 }

 ; Check that atomic loads are not matched.  The transformation is OK for
 ; the "unordered" case tested here, but since we don't try to handle atomic
 ; operations at all in this context, it seems better to assert that than
 ; to restrict the test to a stronger ordering.
 define void @f20(i8 *%ptr, i8 %alt, i32 %limit) {
 ; FIXME: should use a normal load instead of CS.
 ; CHECK-LABEL: f20:
 ; CHECK: lb {{%r[0-9]+}}, 0(%r2)
-; CHECK: {{jl|jnl}} [[LABEL:[^ ]*]]
+; CHECK: {{jhe|jnhe}} [[LABEL:[^ ]*]]
+; CHECK: stc {{%r[0-9]+}}, 0(%r2)
+; CHECK: br %r14
 ; CHECK: [[LABEL]]:
+; CHECK: lr {{%r[0-5]}}, {{%r[0-5]}}
 ; CHECK: stc {{%r[0-9]+}}, 0(%r2)
 ; CHECK: br %r14
   %cond = icmp ult i32 %limit, 420
   %orig = load atomic i8 , i8 *%ptr unordered, align 1
   %res = select i1 %cond, i8 %orig, i8 %alt
   store i8 %res, i8 *%ptr
   ret void
 }

test/CodeGen/SystemZ/cond-store-02.ll

; CHECK: br %r14
   store i16 %res, i16 *%ptr
   ret void
 }
 
 ; Check that volatile loads are not matched.
 define void @f18(i16 *%ptr, i16 %alt, i32 %limit) {
 ; CHECK-LABEL: f18:
 ; CHECK: lh {{%r[0-5]}}, 0(%r2)
-; CHECK: {{jl|jnl}} [[LABEL:[^ ]*]]
+; CHECK: {{jhe|jnhe}} [[LABEL:[^ ]*]]
+; CHECK: sth {{%r[0-5]}}, 0(%r2)
+; CHECK: br %r14
 ; CHECK: [[LABEL]]:
+; CHECK: lr {{%r[0-5]}}, {{%r[0-5]}}
 ; CHECK: sth {{%r[0-5]}}, 0(%r2)
 ; CHECK: br %r14
   %cond = icmp ult i32 %limit, 420
   %orig = load volatile i16 , i16 *%ptr
   %res = select i1 %cond, i16 %orig, i16 %alt
   store i16 %res, i16 *%ptr
   ret void
 }

 ; Check that atomic loads are not matched.  The transformation is OK for
 ; the "unordered" case tested here, but since we don't try to handle atomic
 ; operations at all in this context, it seems better to assert that than
 ; to restrict the test to a stronger ordering.
 define void @f20(i16 *%ptr, i16 %alt, i32 %limit) {
 ; FIXME: should use a normal load instead of CS.
 ; CHECK-LABEL: f20:
 ; CHECK: lh {{%r[0-9]+}}, 0(%r2)
-; CHECK: {{jl|jnl}} [[LABEL:[^ ]*]]
+; CHECK: {{jhe|jnhe}} [[LABEL:[^ ]*]]
+; CHECK: sth {{%r[0-9]+}}, 0(%r2)
+; CHECK: br %r14
 ; CHECK: [[LABEL]]:
+; CHECK: lr {{%r[0-9]+}}, {{%r[0-9]+}}
 ; CHECK: sth {{%r[0-9]+}}, 0(%r2)
 ; CHECK: br %r14
   %cond = icmp ult i32 %limit, 420
   %orig = load atomic i16 , i16 *%ptr unordered, align 2
   %res = select i1 %cond, i16 %orig, i16 %alt
   store i16 %res, i16 *%ptr
   ret void
 }

test/CodeGen/SystemZ/cond-store-03.ll

; CHECK: br %r14
   store i32 %res, i32 *%ptr
   ret void
 }
 
 ; Check that volatile loads are not matched.
 define void @f14(i32 *%ptr, i32 %alt, i32 %limit) {
 ; CHECK-LABEL: f14:
 ; CHECK: l {{%r[0-5]}}, 0(%r2)
-; CHECK: {{jl|jnl}} [[LABEL:[^ ]*]]
+; CHECK: {{jhe|jnhe}} [[LABEL:[^ ]*]]
+; CHECK: st {{%r[0-5]}}, 0(%r2)
+; CHECK: br %r14
 ; CHECK: [[LABEL]]:
+; CHECK: lr {{%r[0-5]}}, {{%r[0-5]}}
 ; CHECK: st {{%r[0-5]}}, 0(%r2)
 ; CHECK: br %r14
   %cond = icmp ult i32 %limit, 420
   %orig = load volatile i32 , i32 *%ptr
   %res = select i1 %cond, i32 %orig, i32 %alt
   store i32 %res, i32 *%ptr
   ret void
 }

 ; Check that atomic loads are not matched.  The transformation is OK for
 ; the "unordered" case tested here, but since we don't try to handle atomic
 ; operations at all in this context, it seems better to assert that than
 ; to restrict the test to a stronger ordering.
 define void @f16(i32 *%ptr, i32 %alt, i32 %limit) {
 ; FIXME: should use a normal load instead of CS.
 ; CHECK-LABEL: f16:
 ; CHECK: l {{%r[0-5]}}, 0(%r2)
-; CHECK: {{jl|jnl}} [[LABEL:[^ ]*]]
+; CHECK: {{jhe|jnhe}} [[LABEL:[^ ]*]]
+; CHECK: st {{%r[0-5]}}, 0(%r2)
+; CHECK: br %r14
 ; CHECK: [[LABEL]]:
+; CHECK: lr {{%r[0-5]}}, {{%r[0-5]}}
 ; CHECK: st {{%r[0-5]}}, 0(%r2)
 ; CHECK: br %r14
   %cond = icmp ult i32 %limit, 420
   %orig = load atomic i32 , i32 *%ptr unordered, align 4
   %res = select i1 %cond, i32 %orig, i32 %alt
   store i32 %res, i32 *%ptr
   ret void
 }

test/CodeGen/SystemZ/cond-store-04.ll

; CHECK: br %r14
   store i64 %res, i64 *%ptr
   ret void
 }
 
 ; Check that volatile loads are not matched.
 define void @f8(i64 *%ptr, i64 %alt, i32 %limit) {
 ; CHECK-LABEL: f8:
 ; CHECK: lg {{%r[0-5]}}, 0(%r2)
-; CHECK: {{jl|jnl}} [[LABEL:[^ ]*]]
+; CHECK: {{jhe|jnhe}} [[LABEL:[^ ]*]]
+; CHECK: stg {{%r[0-5]}}, 0(%r2)
+; CHECK: br %r14
 ; CHECK: [[LABEL]]:
+; CHECK: lgr {{%r[0-5]}}, {{%r[0-5]}}
 ; CHECK: stg {{%r[0-5]}}, 0(%r2)
 ; CHECK: br %r14
   %cond = icmp ult i32 %limit, 420
   %orig = load volatile i64 , i64 *%ptr
   %res = select i1 %cond, i64 %orig, i64 %alt
   store i64 %res, i64 *%ptr
   ret void
 }

 ; Check that atomic loads are not matched.  The transformation is OK for
 ; the "unordered" case tested here, but since we don't try to handle atomic
 ; operations at all in this context, it seems better to assert that than
 ; to restrict the test to a stronger ordering.
 define void @f10(i64 *%ptr, i64 %alt, i32 %limit) {
 ; FIXME: should use a normal load instead of CSG.
 ; CHECK-LABEL: f10:
 ; CHECK: lg {{%r[0-5]}}, 0(%r2)
-; CHECK: {{jl|jnl}} [[LABEL:[^ ]*]]
+; CHECK: {{jhe|jnhe}} [[LABEL:[^ ]*]]
+; CHECK: stg {{%r[0-5]}}, 0(%r2)
+; CHECK: br %r14
 ; CHECK: [[LABEL]]:
+; CHECK: lgr {{%r[0-5]}}, {{%r[0-5]}}
 ; CHECK: stg {{%r[0-5]}}, 0(%r2)
 ; CHECK: br %r14
   %cond = icmp ult i32 %limit, 420
   %orig = load atomic i64 , i64 *%ptr unordered, align 8
   %res = select i1 %cond, i64 %orig, i64 %alt
   store i64 %res, i64 *%ptr
   ret void
 }

test/CodeGen/SystemZ/cond-store-05.ll

; CHECK: br %r14
   store float %res, float *%ptr
   ret void
 }
 
 ; Check that volatile loads are not matched.
 define void @f10(float *%ptr, float %alt, i32 %limit) {
 ; CHECK-LABEL: f10:
 ; CHECK: le {{%f[0-5]}}, 0(%r2)
-; CHECK: {{jl|jnl}} [[LABEL:[^ ]*]]
+; CHECK: {{jhe|jnhe}} [[LABEL:[^ ]*]]
+; CHECK: ste {{%f[0-5]}}, 0(%r2)
+; CHECK: br %r14
 ; CHECK: [[LABEL]]:
+; CHECK: ler {{%f[0-5]}}, {{%f[0-5]}}
 ; CHECK: ste {{%f[0-5]}}, 0(%r2)
 ; CHECK: br %r14
   %cond = icmp ult i32 %limit, 420
   %orig = load volatile float , float *%ptr
   %res = select i1 %cond, float %orig, float %alt
   store float %res, float *%ptr
   ret void
 }

test/CodeGen/SystemZ/cond-store-06.ll

; CHECK: br %r14
   store double %res, double *%ptr
   ret void
 }
 
 ; Check that volatile loads are not matched.
 define void @f10(double *%ptr, double %alt, i32 %limit) {
 ; CHECK-LABEL: f10:
 ; CHECK: ld {{%f[0-5]}}, 0(%r2)
-; CHECK: {{jl|jnl}} [[LABEL:[^ ]*]]
+; CHECK: {{jhe|jnhe}} [[LABEL:[^ ]*]]
+; CHECK: std {{%f[0-5]}}, 0(%r2)
+; CHECK: br %r14
 ; CHECK: [[LABEL]]:
+; CHECK: ldr {{%f[0-5]}}, {{%f[0-5]}}
 ; CHECK: std {{%f[0-5]}}, 0(%r2)
 ; CHECK: br %r14
   %cond = icmp ult i32 %limit, 420
   %orig = load volatile double , double *%ptr
   %res = select i1 %cond, double %orig, double %alt
   store double %res, double *%ptr
   ret void
 }

test/CodeGen/SystemZ/int-cmp-37.ll

 define i32 @f1(i32 %src1) {
 ; CHECK-LABEL: f1:
 ; CHECK: clhrl %r2, g
 ; CHECK-NEXT: jl
 ; CHECK: br %r14
 entry:
   %val = load i16 , i16 *@g
   %src2 = zext i16 %val to i32
-  %cond = icmp ult i32 %src1, %src2
-  br i1 %cond, label %exit, label %mulb
+  %cond = icmp uge i32 %src1, %src2
+  br i1 %cond, label %mulb, label %exit
 mulb:
   %mul = mul i32 %src1, %src1
   br label %exit
 exit:
   %tmp = phi i32 [ %src1, %entry ], [ %mul, %mulb ]
   %res = add i32 %tmp, 1
   ret i32 %res
 }
 
 ; Check signed comparison.
 define i32 @f2(i32 %src1) {
 ; CHECK-LABEL: f2:
 ; CHECK-NOT: clhrl
 ; CHECK: br %r14
 entry:
   %val = load i16 , i16 *@g
   %src2 = zext i16 %val to i32
-  %cond = icmp slt i32 %src1, %src2
-  br i1 %cond, label %exit, label %mulb
+  %cond = icmp sge i32 %src1, %src2
+  br i1 %cond, label %mulb, label %exit
 mulb:
   %mul = mul i32 %src1, %src1
   br label %exit
 exit:
   %tmp = phi i32 [ %src1, %entry ], [ %mul, %mulb ]
   %res = add i32 %tmp, 1
   ret i32 %res
 }
 
 ; Check equality.
 define i32 @f3(i32 %src1) {
 ; CHECK-LABEL: f3:
 ; CHECK: clhrl %r2, g
 ; CHECK-NEXT: je
 ; CHECK: br %r14
 entry:
   %val = load i16 , i16 *@g
   %src2 = zext i16 %val to i32
-  %cond = icmp eq i32 %src1, %src2
-  br i1 %cond, label %exit, label %mulb
+  %cond = icmp ne i32 %src1, %src2
+  br i1 %cond, label %mulb, label %exit
 mulb:
   %mul = mul i32 %src1, %src1
   br label %exit
 exit:
   %tmp = phi i32 [ %src1, %entry ], [ %mul, %mulb ]
   %res = add i32 %tmp, 1
   ret i32 %res
 }
 
 ; Check inequality.
 define i32 @f4(i32 %src1) {
 ; CHECK-LABEL: f4:
 ; CHECK: clhrl %r2, g
 ; CHECK-NEXT: jlh
 ; CHECK: br %r14
 entry:
   %val = load i16 , i16 *@g
   %src2 = zext i16 %val to i32
-  %cond = icmp ne i32 %src1, %src2
-  br i1 %cond, label %exit, label %mulb
+  %cond = icmp eq i32 %src1, %src2
+  br i1 %cond, label %mulb, label %exit
 mulb:
   %mul = mul i32 %src1, %src1
   br label %exit
 exit:
   %tmp = phi i32 [ %src1, %entry ], [ %mul, %mulb ]
   %res = add i32 %tmp, 1
   ret i32 %res
 }
 
 ; Repeat f1 with an unaligned address.
 define i32 @f5(i32 %src1) {
 ; CHECK-LABEL: f5:
 ; CHECK: lgrl [[REG:%r[0-5]]], h@GOT
 ; CHECK: llh [[VAL:%r[0-5]]], 0([[REG]])
 ; CHECK: clrjl %r2, [[VAL]],
 ; CHECK: br %r14
 entry:
   %val = load i16 , i16 *@h, align 1
   %src2 = zext i16 %val to i32
-  %cond = icmp ult i32 %src1, %src2
-  br i1 %cond, label %exit, label %mulb
+  %cond = icmp uge i32 %src1, %src2
+  br i1 %cond, label %mulb, label %exit
 mulb:
   %mul = mul i32 %src1, %src1
   br label %exit
 exit:
   %tmp = phi i32 [ %src1, %entry ], [ %mul, %mulb ]
   %res = add i32 %tmp, 1
   ret i32 %res
 }
 
 ; Check the comparison can be reversed if that allows CLHRL to be used.
 define i32 @f6(i32 %src2) {
 ; CHECK-LABEL: f6:
 ; CHECK: clhrl %r2, g
 ; CHECK-NEXT: jh {{\.L.*}}
 ; CHECK: br %r14
 entry:
   %val = load i16 , i16 *@g
   %src1 = zext i16 %val to i32
-  %cond = icmp ult i32 %src1, %src2
-  br i1 %cond, label %exit, label %mulb
+  %cond = icmp uge i32 %src1, %src2
+  br i1 %cond, label %mulb, label %exit
 mulb:
   %mul = mul i32 %src2, %src2
   br label %exit
 exit:
   %tmp = phi i32 [ %src2, %entry ], [ %mul, %mulb ]
   %res = add i32 %tmp, 1
   ret i32 %res
 }

test/CodeGen/SystemZ/int-cmp-40.ll

 define i64 @f1(i64 %src1) {
 ; CHECK-LABEL: f1:
 ; CHECK: clghrl %r2, g
 ; CHECK-NEXT: jl
 ; CHECK: br %r14
 entry:
   %val = load i16 , i16 *@g
   %src2 = zext i16 %val to i64
-  %cond = icmp ult i64 %src1, %src2
-  br i1 %cond, label %exit, label %mulb
+  %cond = icmp uge i64 %src1, %src2
+  br i1 %cond, label %mulb, label %exit
 mulb:
   %mul = mul i64 %src1, %src1
   br label %exit
 exit:
   %tmp = phi i64 [ %src1, %entry ], [ %mul, %mulb ]
   %res = add i64 %tmp, 1
   ret i64 %res
 }

 define i64 @f3(i64 %src1) {
 ; CHECK-LABEL: f3:
 ; CHECK: clghrl %r2, g
 ; CHECK-NEXT: je
 ; CHECK: br %r14
 entry:
   %val = load i16 , i16 *@g
   %src2 = zext i16 %val to i64
-  %cond = icmp eq i64 %src1, %src2
-  br i1 %cond, label %exit, label %mulb
+  %cond = icmp ne i64 %src1, %src2
+  br i1 %cond, label %mulb, label %exit
 mulb:
   %mul = mul i64 %src1, %src1
   br label %exit
 exit:
   %tmp = phi i64 [ %src1, %entry ], [ %mul, %mulb ]
   %res = add i64 %tmp, 1
   ret i64 %res
 }
 
 ; Check inequality.
 define i64 @f4(i64 %src1) {
 ; CHECK-LABEL: f4:
 ; CHECK: clghrl %r2, g
 ; CHECK-NEXT: jlh
 ; CHECK: br %r14
 entry:
   %val = load i16 , i16 *@g
   %src2 = zext i16 %val to i64
-  %cond = icmp ne i64 %src1, %src2
-  br i1 %cond, label %exit, label %mulb
+  %cond = icmp eq i64 %src1, %src2
+  br i1 %cond, label %mulb, label %exit
 mulb:
   %mul = mul i64 %src1, %src1
   br label %exit
 exit:
   %tmp = phi i64 [ %src1, %entry ], [ %mul, %mulb ]
   %res = add i64 %tmp, 1
   ret i64 %res
 }
 
 ; Repeat f1 with an unaligned address.
 define i64 @f5(i64 %src1) {
 ; CHECK-LABEL: f5:
 ; CHECK: lgrl [[REG:%r[0-5]]], h@GOT
 ; CHECK: llgh [[VAL:%r[0-5]]], 0([[REG]])
 ; CHECK: clgrjl %r2, [[VAL]],
 ; CHECK: br %r14
 entry:
   %val = load i16 , i16 *@h, align 1
   %src2 = zext i16 %val to i64
-  %cond = icmp ult i64 %src1, %src2
-  br i1 %cond, label %exit, label %mulb
+  %cond = icmp uge i64 %src1, %src2
+  br i1 %cond, label %mulb, label %exit
 mulb:
   %mul = mul i64 %src1, %src1
   br label %exit
 exit:
   %tmp = phi i64 [ %src1, %entry ], [ %mul, %mulb ]
   %res = add i64 %tmp, 1
   ret i64 %res
 }
 
 ; Check the comparison can be reversed if that allows CLGHRL to be used.
 define i64 @f6(i64 %src2) {
 ; CHECK-LABEL: f6:
 ; CHECK: clghrl %r2, g
 ; CHECK-NEXT: jh {{\.L.*}}
 ; CHECK: br %r14
 entry:
   %val = load i16 , i16 *@g
   %src1 = zext i16 %val to i64
-  %cond = icmp ult i64 %src1, %src2
-  br i1 %cond, label %exit, label %mulb
+  %cond = icmp uge i64 %src1, %src2
+  br i1 %cond, label %mulb, label %exit
 mulb:
   %mul = mul i64 %src2, %src2
   br label %exit
 exit:
   %tmp = phi i64 [ %src2, %entry ], [ %mul, %mulb ]
   %res = add i64 %tmp, 1
   ret i64 %res
 }

test/CodeGen/SystemZ/int-cmp-44.ll

 ; CHECK-NEXT: brasl %r14, foo@PLT
 ; CHECK-NEXT: cijlh [[REG]], 0, .L{{.*}}
 ; CHECK: br %r14
 entry:
   %val = load i32 , i32 *%ptr
   %xor = xor i32 %val, 1
   %add = add i32 %xor, 1000000
   call void @foo()
-  %cmp = icmp ne i32 %add, 0
-  br i1 %cmp, label %exit, label %store
+  %cmp = icmp eq i32 %add, 0
+  br i1 %cmp, label %store, label %exit, !prof !1
 
 store:
   store i32 %add, i32 *%ptr
   br label %exit
 
 exit:
   ret void
 }

 
 store:
   store i64 %res, i64 *%dest
   br label %exit
 
 exit:
   ret i64 %res
 }
+
+!1 = !{!"branch_weights", i32 2, i32 1}

test/CodeGen/SystemZ/int-cmp-48.ll

 exit:
   ret void
 }
 
 ; Check a simple select-based use of TM.
 define double @f3(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f3:
 ; CHECK: tm 0(%r2), 1
-; CHECK: je {{\.L.*}}
+; CHECK: jne {{\.L.*}}
 ; CHECK: br %r14
   %byte = load i8 , i8 *%src
   %and = and i8 %byte, 1
   %cmp = icmp eq i8 %and, 0
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
; CHECK: br %r14
   store i8 0, i8 *%src
   ret double %res
 }
 
 ; Check an inequality check.
 define double @f5(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f5:
 ; CHECK: tm 0(%r2), 1
-; CHECK: jne {{\.L.*}}
+; CHECK: je {{\.L.*}}
 ; CHECK: br %r14
   %byte = load i8 , i8 *%src
   %and = and i8 %byte, 1
   %cmp = icmp ne i8 %and, 0
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
 ; Check that we can also use TM for equality comparisons with the mask.
 define double @f6(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f6:
 ; CHECK: tm 0(%r2), 254
-; CHECK: jo {{\.L.*}}
+; CHECK: jno {{\.L.*}}
 ; CHECK: br %r14
   %byte = load i8 , i8 *%src
   %and = and i8 %byte, 254
   %cmp = icmp eq i8 %and, 254
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
 ; Check inequality comparisons with the mask.
 define double @f7(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f7:
 ; CHECK: tm 0(%r2), 254
-; CHECK: jno {{\.L.*}}
+; CHECK: jo {{\.L.*}}
 ; CHECK: br %r14
   %byte = load i8 , i8 *%src
   %and = and i8 %byte, 254
   %cmp = icmp ne i8 %and, 254
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
 ; Check that we do not use the memory TM instruction when CC is being tested
 ; for 2.
 define double @f8(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f8:
 ; CHECK: llc [[REG:%r[0-5]]], 0(%r2)
 ; CHECK: tmll [[REG]], 3
-; CHECK: jh {{\.L.*}}
+; CHECK: jnh {{\.L.*}}
 ; CHECK: br %r14
   %byte = load i8 , i8 *%src
   %and = and i8 %byte, 3
   %cmp = icmp eq i8 %and, 2
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
 ; ...likewise 1.
 define double @f9(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f9:
 ; CHECK: llc [[REG:%r[0-5]]], 0(%r2)
 ; CHECK: tmll [[REG]], 3
-; CHECK: jl {{\.L.*}}
+; CHECK: jnl {{\.L.*}}
 ; CHECK: br %r14
   %byte = load i8 , i8 *%src
   %and = and i8 %byte, 3
   %cmp = icmp eq i8 %and, 1
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
 ; Check the high end of the TM range.
 define double @f10(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f10:
 ; CHECK: tm 4095(%r2), 1
-; CHECK: je {{\.L.*}}
+; CHECK: jne {{\.L.*}}
 ; CHECK: br %r14
   %ptr = getelementptr i8, i8 *%src, i64 4095
   %byte = load i8 , i8 *%ptr
   %and = and i8 %byte, 1
   %cmp = icmp eq i8 %and, 0
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
 ; Check the low end of the positive TMY range.
 define double @f11(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f11:
 ; CHECK: tmy 4096(%r2), 1
-; CHECK: je {{\.L.*}}
+; CHECK: jne {{\.L.*}}
 ; CHECK: br %r14
   %ptr = getelementptr i8, i8 *%src, i64 4096
   %byte = load i8 , i8 *%ptr
   %and = and i8 %byte, 1
   %cmp = icmp eq i8 %and, 0
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
 ; Check the high end of the TMY range.
 define double @f12(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f12:
 ; CHECK: tmy 524287(%r2), 1
-; CHECK: je {{\.L.*}}
+; CHECK: jne {{\.L.*}}
 ; CHECK: br %r14
   %ptr = getelementptr i8, i8 *%src, i64 524287
   %byte = load i8 , i8 *%ptr
   %and = and i8 %byte, 1
   %cmp = icmp eq i8 %and, 0
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
 ; Check the next byte up, which needs separate address logic.
 define double @f13(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f13:
 ; CHECK: agfi %r2, 524288
 ; CHECK: tm 0(%r2), 1
-; CHECK: je {{\.L.*}}
+; CHECK: jne {{\.L.*}}
 ; CHECK: br %r14
   %ptr = getelementptr i8, i8 *%src, i64 524288
   %byte = load i8 , i8 *%ptr
   %and = and i8 %byte, 1
   %cmp = icmp eq i8 %and, 0
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
 ; Check the low end of the TMY range.
 define double @f14(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f14:
 ; CHECK: tmy -524288(%r2), 1
-; CHECK: je {{\.L.*}}
+; CHECK: jne {{\.L.*}}
 ; CHECK: br %r14
   %ptr = getelementptr i8, i8 *%src, i64 -524288
   %byte = load i8 , i8 *%ptr
   %and = and i8 %byte, 1
   %cmp = icmp eq i8 %and, 0
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
 ; Check the next byte down, which needs separate address logic.
 define double @f15(i8 *%src, double %a, double %b) {
 ; CHECK-LABEL: f15:
 ; CHECK: agfi %r2, -524289
 ; CHECK: tm 0(%r2), 1
-; CHECK: je {{\.L.*}}
+; CHECK: jne {{\.L.*}}
 ; CHECK: br %r14
   %ptr = getelementptr i8, i8 *%src, i64 -524289
   %byte = load i8 , i8 *%ptr
   %and = and i8 %byte, 1
   %cmp = icmp eq i8 %and, 0
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }
 
 ; Check that TM(Y) does not allow an index
 define double @f16(i8 *%src, i64 %index, double %a, double %b) {
 ; CHECK-LABEL: f16:
 ; CHECK: tm 0({{%r[1-5]}}), 1
-; CHECK: je {{\.L.*}}
+; CHECK: jne {{\.L.*}}
 ; CHECK: br %r14
   %ptr = getelementptr i8, i8 *%src, i64 %index
   %byte = load i8 , i8 *%ptr
   %and = and i8 %byte, 1
   %cmp = icmp eq i8 %and, 0
   %res = select i1 %cmp, double %b, double %a
   ret double %res
 }

test/CodeGen/SystemZ/tdc-06.ll

 ; CHECK: cdbr %f0, %f0
 ; CHECK: jo [[RET]]
   %testnan = fcmp uno double %x, 0.000000e+00
   br i1 %testnan, label %ret, label %nonzeroord, !prof !1
 
 nonzeroord:
 ; CHECK: lhi %r2, 2
 ; CHECK: tcdb %f0, 48
-; CHECK: jl [[RET]]
+; CHECK: je [[FINITE:.]]
   %abs = tail call double @llvm.fabs.f64(double %x)
   %testinf = fcmp oeq double %abs, 0x7FF0000000000000
   br i1 %testinf, label %ret, label %finite, !prof !1
 
+ret:
+; CHECK: [[RET]]:
+; CHECK: br %r14
+  %res = phi i32 [ 5, %entry ], [ 1, %nonzero ], [ 2, %nonzeroord ], [ %finres, %finite ]
+  ret i32 %res
+
 finite:
 ; CHECK: lhi %r2, 3
 ; CHECK: tcdb %f0, 831
 ; CHECK: blr %r14
 ; CHECK: lhi %r2, 4
+; CHECK: br %r14
   %testnormal = fcmp uge double %abs, 0x10000000000000
   %finres = select i1 %testnormal, i32 3, i32 4
   br label %ret
 
-ret:
-; CHECK: [[RET]]:
-; CHECK: br %r14
-  %res = phi i32 [ 5, %entry ], [ 1, %nonzero ], [ 2, %nonzeroord ], [ %finres, %finite ]
-  ret i32 %res
 }
 
 !1 = !{!"branch_weights", i32 1, i32 1}

test/CodeGen/Thumb/thumb-shrink-wrapping.ll

-; RUN: llc %s -o - -enable-shrink-wrap=true -ifcvt-fn-start=1 -ifcvt-fn-stop=0 -mtriple=thumb-macho \
+; RUN: llc %s -o - -enable-shrink-wrap=true -ifcvt-fn-start=1 -ifcvt-fn-stop=0 -tail-dup-placement=0 -mtriple=thumb-macho \
 ; RUN:      | FileCheck %s --check-prefix=CHECK --check-prefix=ENABLE --check-prefix=ENABLE-V4T
-; RUN: llc %s -o - -enable-shrink-wrap=true -ifcvt-fn-start=1 -ifcvt-fn-stop=0 -mtriple=thumbv5-macho \
+; RUN: llc %s -o - -enable-shrink-wrap=true -ifcvt-fn-start=1 -ifcvt-fn-stop=0 -tail-dup-placement=0 -mtriple=thumbv5-macho \
 ; RUN:      | FileCheck %s --check-prefix=CHECK --check-prefix=ENABLE --check-prefix=ENABLE-V5T
-; RUN: llc %s -o - -enable-shrink-wrap=false -ifcvt-fn-start=1 -ifcvt-fn-stop=0 -mtriple=thumb-macho \
+; RUN: llc %s -o - -enable-shrink-wrap=false -ifcvt-fn-start=1 -ifcvt-fn-stop=0 -tail-dup-placement=0 -mtriple=thumb-macho \
 ; RUN:      | FileCheck %s --check-prefix=CHECK --check-prefix=DISABLE --check-prefix=DISABLE-V4T
-; RUN: llc %s -o - -enable-shrink-wrap=false -ifcvt-fn-start=1 -ifcvt-fn-stop=0 -mtriple=thumbv5-macho \
+; RUN: llc %s -o - -enable-shrink-wrap=false -ifcvt-fn-start=1 -ifcvt-fn-stop=0 -tail-dup-placement=0 -mtriple=thumbv5-macho \
 ; RUN:      | FileCheck %s --check-prefix=CHECK --check-prefix=DISABLE --check-prefix=DISABLE-V5T
+
 ;
 ; Note: Lots of tests use inline asm instead of regular calls.
 ; This allows to have a better control on what the allocation will do.
 ; Otherwise, we may have spill right in the entry block, defeating
 ; shrink-wrapping. Moreover, some of the inline asm statements (nop)
 ; are here to ensure that the related paths do not end up as critical
 ; edges.
 ; Also disable the late if-converter as it makes harder to reason on
 ; the diffs.
+; Disable tail-duplication during placement, as v4t vs v5t get different
+; results due to branches not being analyzable under v5
 
 ; Initial motivating example: Simple diamond with a call just on one side.
 ; CHECK-LABEL: foo:
 ;
 ; Compare the arguments and jump to exit.
 ; No prologue needed.
 ; ENABLE: cmp r0, r1
 ; ENABLE-NEXT: bge [[EXIT_LABEL:LBB[0-9_]+]]

test/CodeGen/Thumb2/cbnz.ll

t:
   call void @x()
   call void @x()
   call void @x()
   call void @x()
   call void @x()
   call void @x()
   call void @x()
   call void @x()
-  ; CHECK: cbnz
+  ; CHECK: cbz
   %q = icmp eq i32 %y, 0
   br i1 %q, label %t2, label %f
 
 t2:
   call void @x()
   call void @x()
   call void @x()
   call void @x()

test/CodeGen/Thumb2/ifcvt-compare.ll

 ; RUN: llc -mtriple=thumbv7-unknown-linux %s -o - | FileCheck %s
 
 declare void @x()
 
 define void @f0(i32 %x) optsize {
   ; CHECK-LABEL: f0:
-  ; CHECK: cbnz
+  ; CHECK: cbz
   %p = icmp eq i32 %x, 0
   br i1 %p, label %t, label %f
 
 t:
   call void @x()
   br label %f
 
 f:

test/CodeGen/Thumb2/v8_IT_4.ll

 ; RUN: llc < %s -mtriple=thumbv8-eabi -float-abi=hard | FileCheck %s
 ; RUN: llc < %s -mtriple=thumbv7-eabi -float-abi=hard -arm-restrict-it | FileCheck %s
 ; RUN: llc < %s -mtriple=thumbv8-eabi -float-abi=hard -regalloc=basic | FileCheck %s
 ; RUN: llc < %s -mtriple=thumbv7-eabi -float-abi=hard -regalloc=basic -arm-restrict-it | FileCheck %s
 
 %"struct.__gnu_cxx::__normal_iterator<char*,std::basic_string<char, std::char_traits<char>, std::allocator<char> > >" = type { i8* }
 %"struct.__gnu_cxx::new_allocator<char>" = type <{ i8 }>
 %"struct.std::basic_string<char,std::char_traits<char>,std::allocator<char> >" = type { %"struct.__gnu_cxx::__normal_iterator<char*,std::basic_string<char, std::char_traits<char>, std::allocator<char> > >" }
 %"struct.std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Rep" = type { %"struct.std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Rep_base" }
 %"struct.std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Rep_base" = type { i32, i32, i32 }
 
 
 define weak arm_aapcs_vfpcc i32 @_ZNKSs7compareERKSs(%"struct.std::basic_string<char,std::char_traits<char>,std::allocator<char> >"* %this, %"struct.std::basic_string<char,std::char_traits<char>,std::allocator<char> >"* %__str) {
 ; CHECK-LABEL: _ZNKSs7compareERKSs:
-; CHECK:      cbnz  r0,
+; CHECK:      cbz r0,
+; CHECK-NEXT: %bb1
+; CHECK-NEXT: pop.w
 ; CHECK-NEXT: %bb
 ; CHECK-NEXT: sub{{(.w)?}} r0, r{{[0-9]+}}, r{{[0-9]+}}
-; CHECK-NEXT: %bb1
 ; CHECK-NEXT: pop.w
 entry:
   %0 = tail call arm_aapcs_vfpcc  i32 @_ZNKSs4sizeEv(%"struct.std::basic_string<char,std::char_traits<char>,std::allocator<char> >"* %this) ; <i32> [#uses=3]
   %1 = tail call arm_aapcs_vfpcc  i32 @_ZNKSs4sizeEv(%"struct.std::basic_string<char,std::char_traits<char>,std::allocator<char> >"* %__str) ; <i32> [#uses=3]
   %2 = icmp ult i32 %1, %0                        ; <i1> [#uses=1]
   %3 = select i1 %2, i32 %1, i32 %0               ; <i32> [#uses=1]
   %4 = tail call arm_aapcs_vfpcc  i8* @_ZNKSs7_M_dataEv(%"struct.std::basic_string<char,std::char_traits<char>,std::allocator<char> >"* %this) ; <i8*> [#uses=1]
   %5 = tail call arm_aapcs_vfpcc  i8* @_ZNKSs4dataEv(%"struct.std::basic_string<char,std::char_traits<char>,std::allocator<char> >"* %__str) ; <i8*> [#uses=1]

test/CodeGen/WebAssembly/phi.ll

 ; RUN: llc < %s -asm-verbose=false -disable-wasm-fallthrough-return-opt -verify-machineinstrs | FileCheck %s
 
 ; Test that phis are lowered.
 
 target datalayout = "e-m:e-p:32:32-i64:64-n32:64-S128"
 target triple = "wasm32-unknown-unknown"
 
 ; Basic phi triangle.
 
 ; CHECK-LABEL: test0:
-; CHECK: div_s $[[NUM0:[0-9]+]]=, $0, $pop[[NUM1:[0-9]+]]{{$}}
-; CHECK: return $[[NUM0]]{{$}}
+; CHECK: return $0
+; CHECK: div_s $push[[NUM0:[0-9]+]]=, $0, $pop[[NUM1:[0-9]+]]{{$}}
+; CHECK: return $pop[[NUM0]]{{$}}
 define i32 @test0(i32 %p) {
 entry:
   %t = icmp slt i32 %p, 0
   br i1 %t, label %true, label %done
 true:
   %a = sdiv i32 %p, 3
   br label %done
 done:

test/CodeGen/X86/2008-11-29-ULT-Sign.ll

 ; RUN:  llc < %s -mtriple=i686-pc-linux-gnu | grep "jns" | count 1
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:32:64-v64:64:64-v128:128:128-a0:0:64-f80:32:32"
 target triple = "i686-pc-linux-gnu"
 
 define i32 @a(i32 %x) nounwind {
 entry:
-  %cmp = icmp ult i32 %x, -2147483648    ; <i1> [#uses=1]
-  br i1 %cmp, label %if.end, label %if.then
+  %cmp = icmp uge i32 %x, -2147483648    ; <i1> [#uses=1]
+  br i1 %cmp, label %if.then, label %if.end
 
 if.then:    ; preds = %entry
   %call = call i32 (...) @b()   ; <i32> [#uses=0]
   br label %if.end
 
 if.end:   ; preds = %if.then, %entry
   br label %return
 
 return:   ; preds = %if.end
   ret i32 undef
 }
 
 declare i32 @b(...)
 

test/CodeGen/X86/add.ll

 ; X64: subq $-128,
 }
 
 define i1 @test4(i32 %v1, i32 %v2, i32* %X) nounwind {
 entry:
   %t = call {i32, i1} @llvm.sadd.with.overflow.i32(i32 %v1, i32 %v2)
   %sum = extractvalue {i32, i1} %t, 0
   %obit = extractvalue {i32, i1} %t, 1
-  br i1 %obit, label %overflow, label %normal
+  %notobit = xor i1 1, %obit
+  br i1 %notobit, label %normal, label %overflow
 
 normal:
   store i32 0, i32* %X
   br label %overflow
 
 overflow:
   ret i1 false
 
 ; X32-LABEL: test4:
 ; X32: addl
 ; X32-NEXT: jo
 
 ; X64-LABEL:        test4:
 ; X64:          addl  %e[[A1:si|dx]], %e[[A0:di|cx]]
 ; X64-NEXT: jo
 }
 
 define i1 @test5(i32 %v1, i32 %v2, i32* %X) nounwind {
 entry:
   %t = call {i32, i1} @llvm.uadd.with.overflow.i32(i32 %v1, i32 %v2)
   %sum = extractvalue {i32, i1} %t, 0
   %obit = extractvalue {i32, i1} %t, 1
-  br i1 %obit, label %carry, label %normal
+  %notobit = xor i1 1, %obit
+  br i1 %notobit, label %normal, label %carry
 
 normal:
   store i32 0, i32* %X
   br label %carry
 
 carry:
   ret i1 false
 

test/CodeGen/X86/avx-splat.ll

 ;   shuffle (scalar_to_vector (load (ptr + 4))), undef, <0, 0, 0, 0>
 ;
 define <8 x float> @funcE() nounwind {
 ; CHECK-LABEL: funcE:
 ; CHECK:       ## BB#0: ## %for_exit499
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    ## implicit-def: %YMM0
 ; CHECK-NEXT:    testb %al, %al
-; CHECK-NEXT:    jne LBB4_2
-; CHECK-NEXT:  ## BB#1: ## %load.i1247
+; CHECK-NEXT:    je LBB4_1
+; CHECK-NEXT:  ## BB#2: ## %__load_and_broadcast_32.exit1249
+; CHECK-NEXT:    retq
+; CHECK-NEXT:  LBB4_1: ## %load.i1247
 ; CHECK-NEXT:    pushq %rbp
 ; CHECK-NEXT:    movq %rsp, %rbp
 ; CHECK-NEXT:    andq $-32, %rsp
 ; CHECK-NEXT:    subq $1312, %rsp ## imm = 0x520
 ; CHECK-NEXT:    vbroadcastss {{[0-9]+}}(%rsp), %ymm0
 ; CHECK-NEXT:    movq %rbp, %rsp
 ; CHECK-NEXT:    popq %rbp
-; CHECK-NEXT:  LBB4_2: ## %__load_and_broadcast_32.exit1249
 ; CHECK-NEXT:    retq
 allocas:
   %udx495 = alloca [18 x [18 x float]], align 32
   br label %for_test505.preheader
 
 for_test505.preheader:                            ; preds = %for_test505.preheader, %allocas
   br i1 undef, label %for_exit499, label %for_test505.preheader
 

test/CodeGen/X86/avx512-cmp.ll

 }
 
 define float @test5(float %p) #0 {
 ; ALL-LABEL: test5:
 ; ALL:       ## BB#0: ## %entry
 ; ALL-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; ALL-NEXT:    vucomiss %xmm1, %xmm0
 ; ALL-NEXT:    jne LBB3_1
-; ALL-NEXT:    jnp LBB3_2
+; ALL-NEXT:    jp  LBB3_1
+; ALL-NEXT:  ## BB#2: ## %return
+; ALL-NEXT:    retq
 ; ALL-NEXT:  LBB3_1: ## %if.end
 ; ALL-NEXT:    seta %al
 ; ALL-NEXT:    movzbl %al, %eax
 ; ALL-NEXT:    leaq {{.*}}(%rip), %rcx
 ; ALL-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; ALL-NEXT:  LBB3_2: ## %return
 ; ALL-NEXT:    retq
 entry:
   %cmp = fcmp oeq float %p, 0.000000e+00
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = fcmp ogt float %p, 0.000000e+00
   %cond = select i1 %cmp1, float 1.000000e+00, float -1.000000e+00

test/CodeGen/X86/bt.ll

 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    btl %esi, %edi
 ; CHECK-NEXT:    jb .LBB1_2
 ;
 entry:
   %tmp29 = lshr i32 %x, %n
   %tmp3 = and i32 1, %tmp29
   %tmp4 = icmp eq i32 %tmp3, 0
-  br i1 %tmp4, label %bb, label %UnifiedReturnBlock
+  br i1 %tmp4, label %bb, label %UnifiedReturnBlock, !prof !1
 
 bb:
   call void @foo()
   ret void
 
 UnifiedReturnBlock:
   ret void
 }

 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    btl %esi, %edi
 ; CHECK-NEXT:    jb .LBB3_2
 ;
 entry:
   %tmp29 = ashr i32 %x, %n
   %tmp3 = and i32 1, %tmp29
   %tmp4 = icmp eq i32 %tmp3, 0
-  br i1 %tmp4, label %bb, label %UnifiedReturnBlock
+  br i1 %tmp4, label %bb, label %UnifiedReturnBlock, !prof !1
 
 bb:
   call void @foo()
   ret void
 
 UnifiedReturnBlock:
   ret void
 }
 
 define void @test3(i32 %x, i32 %n) nounwind {
 ; CHECK-LABEL: test3:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    btl %esi, %edi
 ; CHECK-NEXT:    jb .LBB4_2
 ;
 entry:
   %tmp29 = shl i32 1, %n
   %tmp3 = and i32 %tmp29, %x
   %tmp4 = icmp eq i32 %tmp3, 0
-  br i1 %tmp4, label %bb, label %UnifiedReturnBlock
+  br i1 %tmp4, label %bb, label %UnifiedReturnBlock, !prof !1
 
 bb:
   call void @foo()
   ret void
 
 UnifiedReturnBlock:
   ret void
 }
 
 define void @test3b(i32 %x, i32 %n) nounwind {
 ; CHECK-LABEL: test3b:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    btl %esi, %edi
 ; CHECK-NEXT:    jb .LBB5_2
 ;
 entry:
   %tmp29 = shl i32 1, %n
   %tmp3 = and i32 %x, %tmp29
   %tmp4 = icmp eq i32 %tmp3, 0
-  br i1 %tmp4, label %bb, label %UnifiedReturnBlock
+  br i1 %tmp4, label %bb, label %UnifiedReturnBlock, !prof !1
 
 bb:
   call void @foo()
   ret void
 
 UnifiedReturnBlock:
   ret void
 }

 entry:
   %and = and i32 %bit, 31
   %sh_prom = zext i32 %and to i64
   %shl = shl i64 1, %sh_prom
   %and1 = and i64 %shl, %bits
   %tobool = icmp ne i64 %and1, 0
   ret i1 %tobool
 }
+
+!1 = !{!"branch_weights", i32 2, i32 1}

test/CodeGen/X86/critical-edge-split-2.ll

 
 cond.end.i:                                       ; preds = %entry
   %call1 = phi i16 [ trunc (i32 srem (i32 1, i32 zext (i1 icmp eq (%1* bitcast (i8* getelementptr inbounds (%0, %0* @g_2, i64 0, i32 1, i32 0) to %1*), %1* @g_4) to i32)) to i16), %cond.false.i ], [ 1, %entry ]
   ret i16 %call1
 }
 
 ; CHECK-LABEL: test1:
 ; CHECK: testb %dil, %dil
-; CHECK: jne LBB0_2
+; CHECK: je LBB0_1
+; CHECK: retq
+; CHECK: LBB0_1:
 ; CHECK: divl
-; CHECK: LBB0_2:

test/CodeGen/X86/fp-une-cmp.ll

 ; CHECK-NEXT:    jp .LBB0_2
 ; CHECK-NEXT:  # BB#1: # %bb1
 ; CHECK-NEXT:    addsd {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:  .LBB0_2: # %bb2
 ; CHECK-NEXT:    retq
 
 entry:
   %mul = fmul double %x, %y
-  %cmp = fcmp une double %mul, 0.000000e+00
-  br i1 %cmp, label %bb2, label %bb1
+  %cmp = fcmp oeq double %mul, 0.000000e+00
+  br i1 %cmp, label %bb1, label %bb2
 
 bb1:
   %add = fadd double %mul, -1.000000e+00
   br label %bb2
 
 bb2:
   %phi = phi double [ %add, %bb1 ], [ %mul, %entry ]
   ret double %phi

test/CodeGen/X86/jump_sign.ll

 ; RUN: llc < %s -march=x86 -mcpu=pentiumpro -verify-machineinstrs | FileCheck %s
 
 define i32 @func_f(i32 %X) {
 entry:
 ; CHECK-LABEL: func_f:
 ; CHECK: jns
   %tmp1 = add i32 %X, 1   ; <i32> [#uses=1]
   %tmp = icmp slt i32 %tmp1, 0    ; <i1> [#uses=1]
-  br i1 %tmp, label %cond_true, label %cond_next
+  br i1 %tmp, label %cond_true, label %cond_next, !prof !1
 
 cond_true:    ; preds = %entry
   %tmp2 = tail call i32 (...) @bar( )    ; <i32> [#uses=0]
   br label %cond_next
 
 cond_next:    ; preds = %cond_true, %entry
   %tmp3 = tail call i32 (...) @baz( )    ; <i32> [#uses=0]
   ret i32 undef

 if.then:
   %dec = add nsw i32 %1, -1
   store i32 %dec, i32* @a, align 4
   br label %if.end
 
 if.end:
   ret i32 undef
 }
+
+!1 = !{!"branch_weights", i32 2, i32 1}

test/CodeGen/X86/machine-cse.ll

 
 ; CSE physical register defining instruction across MBB boundary.
 ; rdar://10660865
 define i32 @cross_mbb_phys_cse(i32 %a, i32 %b) nounwind ssp {
 entry:
 ; CHECK-LABEL: cross_mbb_phys_cse:
 ; CHECK: cmpl
 ; CHECK: ja
-  %cmp = icmp ugt i32 %a, %b
-  br i1 %cmp, label %return, label %if.end
+  %cmp = icmp ule i32 %a, %b
+  br i1 %cmp, label %if.end, label %return
 
 if.end:                                           ; preds = %entry
 ; CHECK-NOT: cmpl
 ; CHECK: sbbl
   %cmp1 = icmp ult i32 %a, %b
   %. = sext i1 %cmp1 to i32
   br label %return
 

test/CodeGen/X86/shift-double.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-unknown | FileCheck %s
 
 ; Shift i64 integers on 32-bit target
 
 define i64 @test1(i64 %X, i8 %C) nounwind {
 ; CHECK-LABEL: test1:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    pushl %esi
 ; CHECK-NEXT:    movb {{[0-9]+}}(%esp), %cl
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; CHECK-NEXT:    movl %esi, %eax
 ; CHECK-NEXT:    shll %cl, %eax
 ; CHECK-NEXT:    shldl %cl, %esi, %edx
 ; CHECK-NEXT:    testb $32, %cl
-; CHECK-NEXT:    je .LBB0_2
-; CHECK-NEXT:  # BB#1:
+; CHECK-NEXT:    jne .LBB0_1
+; CHECK-NEXT:  # BB#2:
+; CHECK-NEXT:    popl %esi
+; CHECK-NEXT:    retl
+; CHECK-NEXT:  .LBB0_1:
 ; CHECK-NEXT:    movl %eax, %edx
 ; CHECK-NEXT:    xorl %eax, %eax
-; CHECK-NEXT:  .LBB0_2:
 ; CHECK-NEXT:    popl %esi
 ; CHECK-NEXT:    retl
         %shift.upgrd.1 = zext i8 %C to i64              ; <i64> [#uses=1]
         %Y = shl i64 %X, %shift.upgrd.1         ; <i64> [#uses=1]
         ret i64 %Y
 }
 
 define i64 @test2(i64 %X, i8 %C) nounwind {
 ; CHECK-LABEL: test2:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    pushl %esi
 ; CHECK-NEXT:    movb {{[0-9]+}}(%esp), %cl
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; CHECK-NEXT:    movl %esi, %edx
 ; CHECK-NEXT:    sarl %cl, %edx
 ; CHECK-NEXT:    shrdl %cl, %esi, %eax
 ; CHECK-NEXT:    testb $32, %cl
-; CHECK-NEXT:    je .LBB1_2
-; CHECK-NEXT:  # BB#1:
+; CHECK-NEXT:    jne .LBB1_1
+; CHECK-NEXT:  # BB#2:
+; CHECK-NEXT:    popl %esi
+; CHECK-NEXT:    retl
+; CHECK-NEXT:  .LBB1_1:
 ; CHECK-NEXT:    sarl $31, %esi
 ; CHECK-NEXT:    movl %edx, %eax
 ; CHECK-NEXT:    movl %esi, %edx
-; CHECK-NEXT:  .LBB1_2:
 ; CHECK-NEXT:    popl %esi
 ; CHECK-NEXT:    retl
         %shift.upgrd.2 = zext i8 %C to i64              ; <i64> [#uses=1]
         %Y = ashr i64 %X, %shift.upgrd.2                ; <i64> [#uses=1]
         ret i64 %Y
 }
 
 define i64 @test3(i64 %X, i8 %C) nounwind {
 ; CHECK-LABEL: test3:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    pushl %esi
 ; CHECK-NEXT:    movb {{[0-9]+}}(%esp), %cl
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; CHECK-NEXT:    movl %esi, %edx
 ; CHECK-NEXT:    shrl %cl, %edx
 ; CHECK-NEXT:    shrdl %cl, %esi, %eax
 ; CHECK-NEXT:    testb $32, %cl
-; CHECK-NEXT:    je .LBB2_2
-; CHECK-NEXT:  # BB#1:
+; CHECK-NEXT:    jne .LBB2_1
+; CHECK-NEXT:  # BB#2:
+; CHECK-NEXT:    popl %esi
+; CHECK-NEXT:    retl
+; CHECK-NEXT:  .LBB2_1:
 ; CHECK-NEXT:    movl %edx, %eax
 ; CHECK-NEXT:    xorl %edx, %edx
-; CHECK-NEXT:  .LBB2_2:
 ; CHECK-NEXT:    popl %esi
 ; CHECK-NEXT:    retl
         %shift.upgrd.3 = zext i8 %C to i64              ; <i64> [#uses=1]
         %Y = lshr i64 %X, %shift.upgrd.3                ; <i64> [#uses=1]
         ret i64 %Y
 }
 
 ; Combine 2xi32/2xi16 shifts into SHLD

test/CodeGen/X86/sink-hoist.ll

 }
 
 ; Make sure the critical edge is broken so the divsd is sunken below
 ; the conditional branch.
 ; rdar://8454886
 
 ; CHECK-LABEL: split:
 ; CHECK-NEXT: testb $1, %dil
-; CHECK-NEXT: je
+; CHECK-NEXT: jne
+; CHECK:      ret
 ; CHECK:      divsd
 ; CHECK:      movapd
 ; CHECK:      ret
 define double @split(double %x, double %y, i1 %c) nounwind {
   %a = fdiv double %x, 3.2
   %z = select i1 %c, double %a, double %y
   ret double %z
 }

test/CodeGen/X86/testb-je-fusion.ll

 ; RUN: llc < %s -march=x86-64 -mcpu=corei7-avx | FileCheck %s
 
 ; testb should be scheduled right before je to enable macro-fusion.
 
 ; CHECK: testb $2, %{{[abcd]}}h
 ; CHECK-NEXT: je
 
 define i32 @check_flag(i32 %flags, ...) nounwind {
 entry:
   %and = and i32 %flags, 512
   %tobool = icmp eq i32 %and, 0
-  br i1 %tobool, label %if.end, label %if.then
+  br i1 %tobool, label %if.end, label %if.then, !prof !1
 
 if.then:
   br label %if.end
 
 if.end:
   %hasflag = phi i32 [ 1, %if.then ], [ 0, %entry ]
   ret i32 %hasflag
 }
+!1 = !{!"branch_weights", i32 1, i32 2}