[CGP] Make ICMP_EQ use CR result of ICMP_S(L|G)T dominators

[Yi-Hong.Lyu]

Summary

For example:

long long test(long long a, long long b) {
  if (a << b > 0)
    return b;
  if (a << b < 0)
    return a;
  return a*b;
}

Produces:

# %bb.0:                                # %entry
        sld. 5, 3, 4
        ble 0, .LBB0_2
# %bb.1:                                # %return
        mr 3, 4
        blr
.LBB0_2:                                # %if.end
        cmpldi  5, 0
        li 5, 1
        isel 4, 4, 5, 2
        mulld 3, 4, 3
        blr

But the compare (cmpldi 5, 0) is redundant and can be removed (CR0 already contains the result of that comparison).

The root cause of this is that LLVM converts signed comparisons into equality comparison based on dominance. Equality comparisons are unsigned by default, so we get either a record-form or cmp (without the l for logical) feeding a cmpl. That is the situation we want to avoid here.

[nemanjai]

Please reduce the test case only to cases that are changing behaviour. Also, it is probably useful to commit the test case initially as an NFC patch so we can see the actual difference in codegen.

llvm/lib/CodeGen/CodeGenPrepare.cpp

1409

I think it would be very helpful to comment the code in this function. Specifically, it's good to state the pattern(s) you're looking for. Something like:

Block A:
icmp slt i32 %1, 100
...
Block B (with A being the single predecessor):
icmp eq i32 %1, 100 ; Equality is exactly opposite to SGT.
...

1414

Why do we restrict this to EQ rather than also supporting NE? Is this just the canonical form and NE never shows up?

1417

More descriptive names would probably aid readability - perhaps CmpOp1/CmpOp2.

1419

Why does this need to be restricted to constant comparisons? Could we not equivalently handle comparisons between two virtual registers?

1427

We could probably handle logical operations just as easily, couldn't we?

1446

Nit: naming convention - if you have other violations of CamelCase variable naming, please fix those as well.

llvm/test/CodeGen/PowerPC/use-cr-result-of-dom-icmp-st.ll

36

This is already in the form you want it to be in, so it doesn't really test anything this pass does.

[Yi-Hong.Lyu]

llvm/lib/CodeGen/CodeGenPrepare.cpp

1414

The pattern we observed that could be optimized is something like:

Block A:
icmp slt i32 %1, 100
...
Block B (with A being the single predecessor):
icmp eq i32 %1, 100 ; Equality is exactly opposite to SGT.
...

icmp ne i32 %1, 100 is not supposed to be there, otherwise it would be a redundant LLVM IR.

1419

Well. I think we might be able to handle comparisons between two virtual registers equivalently. However I cannot generate test cases for it. I have tried:

long long test1(long long a, long long b, long long c) {
  if (a << b > c)
    return b;
  if (a << b < c)
    return a;
  return a * b;
}

Here is the generated LLVM IR:

; Function Attrs: norecurse nounwind readnone
define dso_local i64 @_Z5test1xxx(i64 %a, i64 %b, i64 %c) local_unnamed_addr #0 {
entry:
  %shl = shl i64 %a, %b
  %cmp = icmp sgt i64 %shl, %c
  br i1 %cmp, label %return, label %if.end

if.end:                                           ; preds = %entry
  %cmp2 = icmp slt i64 %shl, %c
  %mul = select i1 %cmp2, i64 1, i64 %b
  %spec.select = mul nsw i64 %mul, %a
  ret i64 %spec.select

return:                                           ; preds = %entry
  ret i64 %b
}

Apparently, it is not the test case we want. Do you have any test cases in mind?

1427

For other logical operations, we need to introduce a not instruction. It would add more LLVM IR and that might produce more instructions, which is we would like to avoid.

[Yi-Hong.Lyu]

Change according to comments

[Yi-Hong.Lyu]

Rebase the master (NFC testcases merged). Since it is a rebased one, please see the latest commit for review

[nemanjai]

llvm/lib/CodeGen/CodeGenPrepare.cpp

1414

I don't follow why this would be redundant IR.

BlockA:
%cmp1 = icmp slt i32 %1, 100
br i1 %cmp1, BlockC, BlockB

BlockB: ; %1 is signed-greater-than-or-equal-to 100
%cmp2 = icmp ne i32 %1, 100
br i1 %cmp2, BlockD, BlockE
...
BlockC: ; %1 is signed-less-than 100
...
BlockD: ; %1 is signed-greater-than 100
...
BlockE: ; %1 is equal to 100

Of course, if the EQ case is the canonical form, that's a different statement, but I don't think NE is automatically redundant.

[Yi-Hong.Lyu]

llvm/lib/CodeGen/CodeGenPrepare.cpp

1414

After some experiments, I think EQ is canonical form and NE never shows up. I change your pseudo LLVM IR a little bit to make it work:

; ModuleID = '<stdin>'
source_filename = "<stdin>"
target datalayout = "e-m:e-i64:64-n32:64"
target triple = "powerpc64le-unknown-linux-gnu"

define i64 @icmp_ne(i64 %a, i64 %b) {
BlockA:
  %shl = shl i64 %a, %b
  %cmp1 = icmp slt i64 %shl, 1
  br i1 %cmp1, label %BlockC, label %BlockB

BlockB: ; %1 is signed-greater-than-or-equal-to 1
  %cmp2 = icmp ne i64 %shl, 1
  br i1 %cmp2, label %BlockD, label %BlockE

BlockC: ; %1 is signed-less-than 1
  ret i64 %a

BlockD: ; %1 is signed-greater-than 1
  ret i64 %b

BlockE: ; %1 is equal to 1
  %mul = mul nsw i64 %a, %b
  ret i64 %mul
}

However, the NE would be transformed into EQ as:

$ opt -instcombine -S < icmp_ne.ll -o -
; ModuleID = '<stdin>'
source_filename = "<stdin>"
target datalayout = "e-m:e-i64:64-n32:64"
target triple = "powerpc64le-unknown-linux-gnu"

define i64 @icmp_ne(i64 %a, i64 %b) {
BlockA:
  %shl = shl i64 %a, %b
  %cmp1 = icmp slt i64 %shl, 1
  br i1 %cmp1, label %BlockC, label %BlockB

BlockB:                                           ; preds = %BlockA
  %cmp2 = icmp eq i64 %shl, 1
  br i1 %cmp2, label %BlockE, label %BlockD

BlockC:                                           ; preds = %BlockA
  ret i64 %a

BlockD:                                           ; preds = %BlockB
  ret i64 %b

BlockE:                                           ; preds = %BlockB
  %mul = mul nsw i64 %a, %b
  ret i64 %mul
}

[Yi-Hong.Lyu]

Relax the restriction so we can handle comparisons between two virtual registers equivalently. I rebase the patch to TOT so history would not work. Please review this commit directly.

[lei]

llvm/include/llvm/CodeGen/TargetLowering.h

464

nit: missing . at the end of the sentence.

llvm/lib/CodeGen/CodeGenPrepare.cpp

1409

Please add a brief description for this function and some inline documentation so we can easily follow the logic.

llvm/lib/Target/PowerPC/PPCISelLowering.h

625

follow previous conventions and put return false; in the next line.

[lebedev.ri]

llvm/include/llvm/CodeGen/TargetLowering.h

465

Please add more tests for other arches - x86, aarch64 at least

[Yi-Hong.Lyu]

1. Address Lei's comments
2. Add tests for other arches - X86, AArch64 by following Roman 's comments

[Yi-Hong.Lyu]

Add the option -cgp-icmp-eq2icmp-st to enable transformation on AArch64/X86 tests (complete addressing Roman's comments)

[Yi-Hong.Lyu]

To see how my transformation affect AArch64 and X86 tests. Please use History to compare Diff 5 (221979) and Diff 6 (221983)

[Yi-Hong.Lyu]

ping ...

[lebedev.ri]

Please precommit all the tests and rebase the patch.

[nemanjai]

And yes, please pre-commit all the tests as NFC so we can see just the code-gen differences.

llvm/include/llvm/CodeGen/TargetLowering.h

465

I think a more descriptive name would be:
isEqualityCmpFoldedWithSignedCmp()

[Yi-Hong.Lyu]

Test cases are posted here https://reviews.llvm.org/D68534. I would need approval to commit the NFC patch.

[Yi-Hong.Lyu]

Pushed https://reviews.llvm.org/D68534. Rebase the patch to TOT and address Roman and Nemanja' comments

[lebedev.ri]

Some thoughts

llvm/lib/CodeGen/CodeGenPrepare.cpp

225

What's the plan with defaulting to false?

1434

What about ICmpInst::ICMP_NE?

1437

s/not only used by/has users other than/ s/or/and/

1437–1446

You want to use canFreelyInvertAllUsersOf() i think.

1458–1459

Is this guaranteed, or are some commutative cases being missed?

1468–1469

ICmpInst::Predicate NewPred = CmpInst::getSwappedPredicate(DomPred);

1479–1483

SI->swapValues();

[Yi-Hong.Lyu]

Address Roman's comments

[Yi-Hong.Lyu]

llvm/lib/CodeGen/CodeGenPrepare.cpp

225

We indeed know it remove comparison on PowerPC but we don't know whether it has negative impacts for other targets. We disable it by default and let other target to decide whether to turn it on.

1434

Please see discussion in https://reviews.llvm.org/D60506?id=194492#inline-551643. In short, EQ is canonical form and NE never shows up.

1437–1446

canFreelyInvertAllUsersOf is internal to llvm/lib/Transforms/InstCombine/InstCombineInternal.h that I cannot include and use. Should I submit a NFC patch to move it to some header so I can invoke it?

1458–1459

It is guaranteed and I am not aware of any commutative cases being missed. Without this check, some pattern like

///   DomCond = icmp sgt/slt CmpOp0, CmpOp1 (might not be in DomBB)
///   ...
/// DomBB:
///   ...
///   br DomCond, CmpBB, FalseBB
/// CmpBB: (with DomBB being the single predecessor)
///   ...
///   Cmp = icmp eq CmpOp0, CmpOp1

would continue execution. Apparently the Cmp = icmp eq CmpOp0, CmpOp1 would be always false (i.e., redundant) and it is not the pattern we are looking for.

[Yi-Hong.Lyu]

ping ...

[nemanjai]

LGTM with some minor nits related to comments/usage addressed. I personally don't have an issue with you addressing this on the commit, but give it until early next week in case someone else speaks up regarding wanting another review.

llvm/lib/CodeGen/CodeGenPrepare.cpp

1440

Nit: the sequence isa<>; cast<> is discouraged. The preferred way is to dyn_cast and the subsequent checks are done if the return value is non-null.
Also, is the check for isConditional() actually needed? Can an unconditional branch use the result of a compare?

1454

I think a comment before this section along the following lines would help readability:

// We want to ensure that the only way control gets to the comparison
// of interest is that a less/greater than comparison on the same operands
// is false.

1468

Also a comment here would be nice. Something like:

// Convert the equality comparison to the opposite of the dominating
// comparison and swap the direction for all branch/select users.
// We have conceptually converted:
// Res = (a < b) ? <LT_RES> : (a == b) ? <EQ_RES> : <GT_RES>;
// to
// Res = (a < b) ? <LT_RES> : (a > b)  ? <GT_RES> : <EQ_RES>;
// And similarly for branches.

llvm/include/llvm/CodeGen/TargetLowering.h

virtual bool isCheapToSpeculateCtlz() const {
     return false;
   }
 
   /// Return true if ctlz instruction is fast.
   virtual bool isCtlzFast() const {
     return false;
   }
 
+  /// Return true if instruction generated for ICMP_EQ could be folded with
+  /// instruction generated for ICMP_S(G|L)T
+  virtual bool isICMP_EQFoldedWithICMP_ST() const { return true; }
+
   /// Return true if it is safe to transform an integer-domain bitwise operation
   /// into the equivalent floating-point operation. This should be set to true
   /// if the target has IEEE-754-compliant fabs/fneg operations for the input
   /// type.
   virtual bool hasBitPreservingFPLogic(EVT VT) const {
     return false;
   }
 

llvm/lib/CodeGen/CodeGenPrepare.cpp

static cl::opt<bool> AddrSinkCombineScaledReg(
     cl::desc("Allow combining of ScaledReg field in Address sinking."));
 
 static cl::opt<bool>
     EnableGEPOffsetSplit("cgp-split-large-offset-gep", cl::Hidden,
                          cl::init(true),
                          cl::desc("Enable splitting large offset of GEP."));
 
 namespace {
 
 enum ExtType {
   ZeroExtension,   // Zero extension has been seen.
   SignExtension,   // Sign extension has been seen.
   BothExtension    // This extension type is used if we saw sext after
                    // ZeroExtension had been set, or if we saw zext after
                    // SignExtension had been set. It makes the type
                    // information of a promoted instruction invalid.
 };
static bool sinkCmpExpression(CmpInst *Cmp, const TargetLowering &TLI) {
   if (Cmp->use_empty()) {
     Cmp->eraseFromParent();
     MadeChange = true;
   }
 
   return MadeChange;
 }
 
+static bool foldICmpWithDominatingICmp(CmpInst *Cmp,
+                                       const TargetLowering &TLI) {
+  //   DomCond = icmp sgt/slt CmpOp0, CmpOp1 (might not be in DomBB)
+  //   ...
+  // DomBB:
+  //   ...
+  //   br DomCond, TrueBB, CmpBB
+  // CmpBB: (with DomBB being the single predecessor)
+  //   ...
+  //   Cmp = icmp eq CmpOp0, CmpOp1
+  //   ...
+  if (TLI.isICMP_EQFoldedWithICMP_ST())
+    return false;
+
+  ICmpInst::Predicate Pred = Cmp->getPredicate();
+  if (Pred != ICmpInst::ICMP_EQ)
+    return false;
+
+  for (User *U : Cmp->users()) {
+    if (isa<BranchInst>(U) && cast<BranchInst>(U)->isConditional())
+      continue;
+    if (isa<SelectInst>(U) && cast<SelectInst>(U)->getCondition() == Cmp)
+      continue;
+    return false;
+  }
+
+  BasicBlock *CmpBB = Cmp->getParent();
+  BasicBlock *DomBB = CmpBB->getSinglePredecessor();
+  if (!DomBB)
+    return false;
+
+  Value *DomCond;
+  BasicBlock *TrueBB, *FalseBB;
+  if (!match(DomBB->getTerminator(), m_Br(m_Value(DomCond), TrueBB, FalseBB)))
+    return false;
+  if (CmpBB != FalseBB)
+    return false;
+
+  Value *CmpOp0 = Cmp->getOperand(0), *CmpOp1 = Cmp->getOperand(1);
+  ICmpInst::Predicate DomPred;
+  if (!match(DomCond, m_ICmp(DomPred, m_Specific(CmpOp0), m_Specific(CmpOp1))))
+    return false;
+  if (DomPred != ICmpInst::ICMP_SGT && DomPred != ICmpInst::ICMP_SLT)
+    return false;
+
+  ICmpInst::Predicate NewPred =
+      DomPred == ICmpInst::ICMP_SGT ? ICmpInst::ICMP_SLT : ICmpInst::ICMP_SGT;
+
+  for (User *U : Cmp->users()) {
+    if (auto *BI = dyn_cast<BranchInst>(U)) {
+      assert(BI->isConditional() && "Must be conditional");
+      BI->swapSuccessors();
+      continue;
+    }
+    if (auto *SI = dyn_cast<SelectInst>(U)) {
+      // Swap operands
+      Value *TrueValue = SI->getTrueValue();
+      Value *FalseValue = SI->getFalseValue();
+      SI->setTrueValue(FalseValue);
+      SI->setFalseValue(TrueValue);
+      SI->swapProfMetadata();
+      continue;
+    }
+    llvm_unreachable("Must be a branch or a select");
+  }
+  Cmp->setPredicate(NewPred);
+  return true;
+}
+
 bool CodeGenPrepare::optimizeCmp(CmpInst *Cmp, bool &ModifiedDT) {
   if (sinkCmpExpression(Cmp, *TLI))
     return true;
 
   if (combineToUAddWithOverflow(Cmp, ModifiedDT))
     return true;
 
   if (combineToUSubWithOverflow(Cmp, ModifiedDT))
     return true;
 
+  if (foldICmpWithDominatingICmp(Cmp, *TLI))
+    return true;
+
   return false;
 }
 
 /// Duplicate and sink the given 'and' instruction into user blocks where it is
 /// used in a compare to allow isel to generate better code for targets where
 /// this operation can be combined.
 ///
 /// Return true if any changes are made.

llvm/lib/Target/PowerPC/PPCISelLowering.h

public:
     bool isCheapToSpeculateCtlz() const override {
       return true;
     }
 
     bool isCtlzFast() const override {
       return true;
     }
 
+    bool isICMP_EQFoldedWithICMP_ST() const override { return false; }
+
     bool hasAndNotCompare(SDValue) const override {
       return true;
     }
 
     bool convertSetCCLogicToBitwiseLogic(EVT VT) const override {
       return VT.isScalarInteger();
     }
 

llvm/test/CodeGen/PowerPC/use-cr-result-of-dom-icmp-st.ll

 ; PPC64LE-NEXT:    sld 5, 3, 4
 ; PPC64LE-NEXT:    cmpdi 5, -2
 ; PPC64LE-NEXT:    ble 0, .LBB0_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB0_2: # %if.end
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i64 %a, %b
   %cmp = icmp sgt i64 %shl, -2
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp2 = icmp eq i64 %shl, -2
   %mul = select i1 %cmp2, i64 %b, i64 1
   %spec.select = mul nsw i64 %mul, %a
   ret i64 %spec.select
 
 return:                                           ; preds = %entry
   ret i64 %b
 }
 
 define i64 @ll_a_op_b__1(i64 %a, i64 %b) {
 ; PPC64LE-LABEL: ll_a_op_b__1:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    sld 5, 3, 4
 ; PPC64LE-NEXT:    cmpdi 5, -1
 ; PPC64LE-NEXT:    ble 0, .LBB1_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB1_2: # %if.end
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i64 %a, %b
   %cmp = icmp sgt i64 %shl, -1
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry

 ; PPC64LE-LABEL: ll_a_op_b_0:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    sld. 5, 3, 4
 ; PPC64LE-NEXT:    ble 0, .LBB2_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB2_2: # %if.end
-; PPC64LE-NEXT:    cmpldi 5, 0
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i64 %a, %b
   %cmp = icmp sgt i64 %shl, 0
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry

 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    sld 5, 3, 4
 ; PPC64LE-NEXT:    cmpdi 5, 1
 ; PPC64LE-NEXT:    ble 0, .LBB3_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB3_2: # %if.end
-; PPC64LE-NEXT:    cmpldi 5, 1
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i64 %a, %b
   %cmp = icmp sgt i64 %shl, 1
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry

 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    sld 5, 3, 4
 ; PPC64LE-NEXT:    cmpdi 5, 2
 ; PPC64LE-NEXT:    ble 0, .LBB4_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB4_2: # %if.end
-; PPC64LE-NEXT:    cmpldi 5, 2
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i64 %a, %b
   %cmp = icmp sgt i64 %shl, 2
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry

 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    cmpdi 3, -2
 ; PPC64LE-NEXT:    ble 0, .LBB5_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB5_2: # %if.end
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i64 %a, -2
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = icmp eq i64 %a, -2

 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    cmpdi 3, -1
 ; PPC64LE-NEXT:    ble 0, .LBB6_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB6_2: # %if.end
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i64 %a, -1
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = icmp eq i64 %a, -1

 ; PPC64LE-LABEL: ll_a_0:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    cmpdi 3, 0
 ; PPC64LE-NEXT:    ble 0, .LBB7_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB7_2: # %if.end
-; PPC64LE-NEXT:    cmpldi 3, 0
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i64 %a, 0
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = icmp eq i64 %a, 0

 ; PPC64LE-LABEL: ll_a_1:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    cmpdi 3, 1
 ; PPC64LE-NEXT:    ble 0, .LBB8_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB8_2: # %if.end
-; PPC64LE-NEXT:    cmpldi 3, 1
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i64 %a, 1
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = icmp eq i64 %a, 1

 ; PPC64LE-LABEL: ll_a_2:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    cmpdi 3, 2
 ; PPC64LE-NEXT:    ble 0, .LBB9_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB9_2: # %if.end
-; PPC64LE-NEXT:    cmpldi 3, 2
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i64 %a, 2
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = icmp eq i64 %a, 2
   %mul = select i1 %cmp1, i64 %b, i64 1
   %spec.select = mul nsw i64 %mul, %a
   ret i64 %spec.select
 
 return:                                           ; preds = %entry
   ret i64 %b
 }
 
 define i64 @i_a_op_b__2(i32 signext %a, i32 signext %b) {
 ; PPC64LE-LABEL: i_a_op_b__2:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    slw 6, 3, 4
 ; PPC64LE-NEXT:    li 5, 1
 ; PPC64LE-NEXT:    cmpwi 6, -2
-; PPC64LE-NEXT:    isel 5, 4, 5, 2
+; PPC64LE-NEXT:    isel 5, 5, 4, 0
 ; PPC64LE-NEXT:    mullw 3, 5, 3
 ; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i32 %a, %b
   %cmp = icmp sgt i32 %shl, -2
   br i1 %cmp, label %return, label %if.end

 }
 
 define i64 @i_a_op_b__1(i32 signext %a, i32 signext %b) {
 ; PPC64LE-LABEL: i_a_op_b__1:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    slw 6, 3, 4
 ; PPC64LE-NEXT:    li 5, 1
 ; PPC64LE-NEXT:    cmpwi 6, -1
-; PPC64LE-NEXT:    isel 5, 4, 5, 2
+; PPC64LE-NEXT:    isel 5, 5, 4, 0
 ; PPC64LE-NEXT:    mullw 3, 5, 3
 ; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i32 %a, %b
   %cmp = icmp sgt i32 %shl, -1
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp2 = icmp eq i32 %shl, -1
   %mul = select i1 %cmp2, i32 %b, i32 1
   %spec.select = mul nsw i32 %mul, %a
   br label %return
 
 return:                                           ; preds = %if.end, %entry
   %retval.0.in = phi i32 [ %b, %entry ], [ %spec.select, %if.end ]
   %retval.0 = sext i32 %retval.0.in to i64
   ret i64 %retval.0
 }
 
 define i64 @i_a_op_b_0(i32 signext %a, i32 signext %b) {
 ; PPC64LE-LABEL: i_a_op_b_0:
 ; PPC64LE:       # %bb.0: # %entry
-; PPC64LE-NEXT:    slw. 5, 3, 4
-; PPC64LE-NEXT:    li 6, 1
-; PPC64LE-NEXT:    isel 6, 4, 6, 2
-; PPC64LE-NEXT:    cmpwi 5, 0
-; PPC64LE-NEXT:    mullw 3, 6, 3
+; PPC64LE-NEXT:    slw 6, 3, 4
+; PPC64LE-NEXT:    li 5, 1
+; PPC64LE-NEXT:    cmpwi 6, 0
+; PPC64LE-NEXT:    isel 5, 5, 4, 0
+; PPC64LE-NEXT:    mullw 3, 5, 3
 ; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i32 %a, %b
   %cmp = icmp sgt i32 %shl, 0
   br i1 %cmp, label %return, label %if.end
 
return:                                           ; preds = %if.end, %entry
   ret i64 %retval.0
 }
 
 define i64 @i_a_op_b_1(i32 signext %a, i32 signext %b) {
 ; PPC64LE-LABEL: i_a_op_b_1:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    slw 6, 3, 4
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    cmplwi 6, 1
-; PPC64LE-NEXT:    isel 5, 4, 5, 2
 ; PPC64LE-NEXT:    cmpwi 6, 1
+; PPC64LE-NEXT:    isel 5, 5, 4, 0
 ; PPC64LE-NEXT:    mullw 3, 5, 3
 ; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i32 %a, %b
   %cmp = icmp sgt i32 %shl, 1
   br i1 %cmp, label %return, label %if.end
return:                                           ; preds = %if.end, %entry
   ret i64 %retval.0
 }
 
 define i64 @i_a_op_b_2(i32 signext %a, i32 signext %b) {
 ; PPC64LE-LABEL: i_a_op_b_2:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    slw 6, 3, 4
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    cmplwi 6, 2
-; PPC64LE-NEXT:    isel 5, 4, 5, 2
 ; PPC64LE-NEXT:    cmpwi 6, 2
+; PPC64LE-NEXT:    isel 5, 5, 4, 0
 ; PPC64LE-NEXT:    mullw 3, 5, 3
 ; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i32 %a, %b
   %cmp = icmp sgt i32 %shl, 2
   br i1 %cmp, label %return, label %if.end
return:                                           ; preds = %if.end, %entry
   ret i64 %retval.0
 }
 
 define i64 @i_a__2(i32 signext %a, i32 signext %b) {
 ; PPC64LE-LABEL: i_a__2:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    li 5, 1
 ; PPC64LE-NEXT:    cmpwi 3, -2
-; PPC64LE-NEXT:    isel 5, 4, 5, 2
+; PPC64LE-NEXT:    isel 5, 5, 4, 0
 ; PPC64LE-NEXT:    mullw 3, 5, 3
 ; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i32 %a, -2
   br i1 %cmp, label %return, label %if.end
 
return:                                           ; preds = %if.end, %entry
   ret i64 %retval.0
 }
 
 define i64 @i_a__1(i32 signext %a, i32 signext %b) {
 ; PPC64LE-LABEL: i_a__1:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    li 5, 1
 ; PPC64LE-NEXT:    cmpwi 3, -1
-; PPC64LE-NEXT:    isel 5, 4, 5, 2
+; PPC64LE-NEXT:    isel 5, 5, 4, 0
 ; PPC64LE-NEXT:    mullw 3, 5, 3
 ; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i32 %a, -1
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = icmp eq i32 %a, -1
   %mul = select i1 %cmp1, i32 %b, i32 1
   %spec.select = mul nsw i32 %mul, %a
   br label %return
 
 return:                                           ; preds = %if.end, %entry
   %retval.0.in = phi i32 [ %b, %entry ], [ %spec.select, %if.end ]
   %retval.0 = sext i32 %retval.0.in to i64
   ret i64 %retval.0
 }
 
 define i64 @i_a_0(i32 signext %a, i32 signext %b) {
 ; PPC64LE-LABEL: i_a_0:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    cmplwi 3, 0
-; PPC64LE-NEXT:    isel 5, 4, 5, 2
-; PPC64LE-NEXT:    cmpwi 0, 3, 0
-; PPC64LE-NEXT:    mullw 5, 5, 3
-; PPC64LE-NEXT:    isel 3, 4, 5, 1
+; PPC64LE-NEXT:    cmpwi 3, 0
+; PPC64LE-NEXT:    isel 5, 5, 4, 0
+; PPC64LE-NEXT:    mullw 3, 5, 3
+; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i32 %a, 0
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = icmp eq i32 %a, 0
   %mul = select i1 %cmp1, i32 %b, i32 1
   %spec.select = mul nsw i32 %mul, %a
   br label %return
 
 return:                                           ; preds = %if.end, %entry
   %retval.0.in = phi i32 [ %b, %entry ], [ %spec.select, %if.end ]
   %retval.0 = sext i32 %retval.0.in to i64
   ret i64 %retval.0
 }
 
 define i64 @i_a_1(i32 signext %a, i32 signext %b) {
 ; PPC64LE-LABEL: i_a_1:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    cmplwi 3, 1
-; PPC64LE-NEXT:    isel 5, 4, 5, 2
-; PPC64LE-NEXT:    cmpwi 0, 3, 1
-; PPC64LE-NEXT:    mullw 5, 5, 3
-; PPC64LE-NEXT:    isel 3, 4, 5, 1
+; PPC64LE-NEXT:    cmpwi 3, 1
+; PPC64LE-NEXT:    isel 5, 5, 4, 0
+; PPC64LE-NEXT:    mullw 3, 5, 3
+; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i32 %a, 1
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = icmp eq i32 %a, 1
   %mul = select i1 %cmp1, i32 %b, i32 1
   %spec.select = mul nsw i32 %mul, %a
   br label %return
 
 return:                                           ; preds = %if.end, %entry
   %retval.0.in = phi i32 [ %b, %entry ], [ %spec.select, %if.end ]
   %retval.0 = sext i32 %retval.0.in to i64
   ret i64 %retval.0
 }
 
 define i64 @i_a_2(i32 signext %a, i32 signext %b) {
 ; PPC64LE-LABEL: i_a_2:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    cmplwi 3, 2
-; PPC64LE-NEXT:    isel 5, 4, 5, 2
-; PPC64LE-NEXT:    cmpwi 0, 3, 2
-; PPC64LE-NEXT:    mullw 5, 5, 3
-; PPC64LE-NEXT:    isel 3, 4, 5, 1
+; PPC64LE-NEXT:    cmpwi 3, 2
+; PPC64LE-NEXT:    isel 5, 5, 4, 0
+; PPC64LE-NEXT:    mullw 3, 5, 3
+; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i32 %a, 2
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = icmp eq i32 %a, 2

 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    sld 6, 3, 4
 ; PPC64LE-NEXT:    cmpd 6, 5
 ; PPC64LE-NEXT:    ble 0, .LBB20_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB20_2: # %if.end
-; PPC64LE-NEXT:    cmpld 6, 5
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i64 %a, %b
   %cmp = icmp sgt i64 %shl, %c
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry

 ; PPC64LE-LABEL: ll_a_c:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    cmpd 3, 5
 ; PPC64LE-NEXT:    ble 0, .LBB21_2
 ; PPC64LE-NEXT:  # %bb.1: # %return
 ; PPC64LE-NEXT:    mr 3, 4
 ; PPC64LE-NEXT:    blr
 ; PPC64LE-NEXT:  .LBB21_2: # %if.end
-; PPC64LE-NEXT:    cmpld 3, 5
 ; PPC64LE-NEXT:    li 5, 1
-; PPC64LE-NEXT:    isel 4, 4, 5, 2
+; PPC64LE-NEXT:    isel 4, 5, 4, 0
 ; PPC64LE-NEXT:    mulld 3, 4, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i64 %a, %c
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = icmp eq i64 %a, %c
   %mul = select i1 %cmp1, i64 %b, i64 1
   %spec.select = mul nsw i64 %mul, %a
   ret i64 %spec.select
 
 return:                                           ; preds = %entry
   ret i64 %b
 }
 
 define i64 @i_a_op_b_c(i32 signext %a, i32 signext %b, i32 signext %c) {
 ; PPC64LE-LABEL: i_a_op_b_c:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    slw 7, 3, 4
 ; PPC64LE-NEXT:    li 6, 1
-; PPC64LE-NEXT:    cmplw 7, 5
-; PPC64LE-NEXT:    isel 6, 4, 6, 2
 ; PPC64LE-NEXT:    cmpw 0, 7, 5
-; PPC64LE-NEXT:    mullw 3, 6, 3
+; PPC64LE-NEXT:    isel 5, 6, 4, 0
+; PPC64LE-NEXT:    mullw 3, 5, 3
 ; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %shl = shl i32 %a, %b
   %cmp = icmp sgt i32 %shl, %c
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp2 = icmp eq i32 %shl, %c
   %mul = select i1 %cmp2, i32 %b, i32 1
   %spec.select = mul nsw i32 %mul, %a
   br label %return
 
 return:                                           ; preds = %if.end, %entry
   %retval.0.in = phi i32 [ %b, %entry ], [ %spec.select, %if.end ]
   %retval.0 = sext i32 %retval.0.in to i64
   ret i64 %retval.0
 }
 
 define i64 @i_a_c(i32 signext %a, i32 signext %b, i32 signext %c) {
 ; PPC64LE-LABEL: i_a_c:
 ; PPC64LE:       # %bb.0: # %entry
 ; PPC64LE-NEXT:    li 6, 1
-; PPC64LE-NEXT:    cmplw 3, 5
-; PPC64LE-NEXT:    isel 6, 4, 6, 2
 ; PPC64LE-NEXT:    cmpw 0, 3, 5
-; PPC64LE-NEXT:    mullw 6, 6, 3
-; PPC64LE-NEXT:    isel 3, 4, 6, 1
+; PPC64LE-NEXT:    isel 5, 6, 4, 0
+; PPC64LE-NEXT:    mullw 3, 5, 3
+; PPC64LE-NEXT:    isel 3, 4, 3, 1
 ; PPC64LE-NEXT:    extsw 3, 3
 ; PPC64LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i32 %a, %c
   br i1 %cmp, label %return, label %if.end
 
 if.end:                                           ; preds = %entry
   %cmp1 = icmp eq i32 %a, %c