[ARM] Materialise some boolean values to avoid a branch

[rogfer01]

Summary

This patch combines some cases of ARMISD::CMOV for integers that arise in comparisons of the form

a != b ? x : 0
a == b ? 0 : x

and that currently (e.g. in Thumb1) are emitted as branches.

[efriedma]

A lot of these combines look very similar to the work done for ISD::ADDCARRY/SUBCARRY; see https://reviews.llvm.org/D29872 . Can we reuse that work rather than add a bunch of ARM-specific code?

[rogfer01]

Hi @efriedma,

thanks for pointing me to the ADDCARRY/SUBCARRY nodes. I tried to use them but I ran into serious problems (maybe due to some inexperience on my side working with SelectionDAG). First if I want to use them I assume I have to make them legal for the ARM back end, this means that I have to lower them somehow. After the short discussion in the list about the semantics of SUBCARRY I made ISD::ADDCARRY to lower to ARMISD::ADDC and ISD::SUBCARRY to lower to ARMISD::SUBC plus doing an extra (ISD::ADD 1, c) to the carry result, to change from borrow semantics (ISD::SUBCARRY semantics) to ARM carry semantics.

One side effect of doing this is that many operations that were previously legalized using ISD::ADD and ISD::ADDC like i64 add/sub are now legalized using ISD::ADDCARRY. And one generic combiner for ISD::ADDCARRY when it has a zero carry transforms it into ISD::UADDO (similarly ISD::USUBO for ISD::SUBCARRY without borrow). This, unfortunately, leads to very poor (and probably wrong) code involving msr / mrs instructions.

I tried to fix some of these issues but still it is very easy that sometimes the carry value is moved to some general purpose register and the backend decides to use mrs. For instance a case like this

define i64 @f3(i32 %al, i32 %bl) {
entry:
    ; unsigned wide add
    %aw = zext i32 %al to i64
    %bw = zext i32 %bl to i64
    %cw = add i64 %aw, %bw
    ; ch == carry bit
    %ch = lshr i64 %cw, 32
	%dw = add i64 %ch, %bw
	ret i64 %dw
}

leads to something like this before instruction selection

SelectionDAG has 13 nodes:
  t0: ch = EntryToken
  t4: i32,ch = CopyFromReg t0, Register:i32 %vreg1
  t17: ch,glue = CopyToReg t0, Register:i32 %R0, t28
  t19: ch,glue = CopyToReg t17, Register:i32 %R1, t28:1, t17:1
      t2: i32,ch = CopyFromReg t0, Register:i32 %vreg0
    t29: i32,i32 = ARMISD::ADDC t2, t4
  t28: i32,i32 = ARMISD::ADDE t4, Constant:i32<0>, t29:1
  t20: ch = ARMISD::RET_FLAG t19, Register:i32 %R0, Register:i32 %R1, t19:1

but the t19: ch,glue = CopyToReg t17, Register:i32 %R1, t28:1, t17:1 ends being emitted as mrs r1, apsr, below is the final code generated by llc -mtriple=armv6t2-eabi which I think is really wrong as the apsr register has more bits than the carry.

f3:
        adds    r0, r0, r1
        adcs    r0, r1, #0
        mrs     r1, apsr
        bx      lr

So my understanding is that using these nodes, while something that has to be done in the future, is not an easy addition to this change.

The upside is that it is possible to express the original change of this patch using the ADDCARRY and SUBCARRY nodes, so this may still be worth the effort.

Do you agree with my analysis or I am missing something very obvious here. I can upload a phab with the current change as it is if it helps.

Thank you very much,

[efriedma]

If you're seeing references to apsr, you aren't lowering ADDCARRY correctly.

ARMISD::ADDE has three operands; two integers, and apsr. It has two results: an integer, and the resulting apsr. If you're lowering ADDCARRY to ARMISD::ADDE, you actually need three operations: one to convert the boolean carry input to an apsr, one ARMISD::ADDE to perform the actual operation, and one operation to convert the result apsr to a boolean. See LowerADDSUBCARRY for how the x86 backend does this.

(Afterwards, you use a target dagcombine to eliminate the redundant operations.)

[rogfer01]

Hi Eli,

to make this more manageable I opened D35192 in which I only use ADDCARRY and SUBCARRY. Once we're happy with that one I will update this change to use the new nodes.

Regards,

[rogfer01]

ChangeLog:

• Updated the patch to use ADDCARRY / SUBCARRY nodes.

[rogfer01]

lib/Target/ARM/ARMISelLowering.cpp

9873–9883

I think this may make the ADDC combiner above redundant as ADDC x, -1 will usually become SUBC x, 1. I'll try to see if I can remove the ADDC one.

[rogfer01]

Friendly ping :)

[samparker]

lib/Target/ARM/ARMISelLowering.cpp

12067

early exit when not integer instead?

12097

I don't follow how this sequence better? What makes this more efficient than a cmp and conditional move? On a Thumb1Only target, even a cmp and branch will only be 3 cycles, I think.

12146

Same as above, I guess I'm missing something... could you explain why this is better please?

[rogfer01]

lib/Target/ARM/ARMISelLowering.cpp

12067

Sadly there is some code after this block that does not require integer (if I'm reading it correctly).

12097

Consider a case like

int test(int a, int b) {
  return a != b;
}

ToT is currently generating (--target=arm -mcpu=cortex-m0 -O2)

test:
	.fnstart
@ BB#0:                                 @ %entry
	mov	r2, r0
	movs	r0, #1
	movs	r3, #0
	cmp	r2, r1
	bne	.LBB0_2
@ BB#1:                                 @ %entry
	mov	r0, r3
.LBB0_2:                                @ %entry
	bx	lr

with this combiner we can generate

test:
	.fnstart
@ BB#0:                                 @ %entry
	subs	r0, r0, r1
	subs	r1, r0, #1
	sbcs	r0, r1
	bx	lr

So, not counting bx, we go from a sequence of 5/6 to 3.

12146

Consider a case like

int test(int a, int b) {
  return a == b;
}

ToT is currently generating (--target=arm -mcpu=cortex-m0 -O2)

test:
	.fnstart
@ BB#0:                                 @ %entry
	mov	r2, r0
	movs	r0, #1
	movs	r3, #0
	cmp	r2, r1
	beq	.LBB0_2
@ BB#1:                                 @ %entry
	mov	r0, r3
.LBB0_2:                                @ %entry
	bx	lr

But the code above is in practice like

int test(int a, int b) {
  return a == b ? 1 : 0;
}

where k = 0 (2^k = 1).

So with this change we can generate

test:
	.fnstart
@ BB#0:                                 @ %entry
	subs	r1, r0, r1
	movs	r0, #0
	subs	r0, r0, r1
	adcs	r0, r1
	bx	lr

We go from 5/6 instructions to always 4 (not counting the bx).

When k != 0 (e.g. return a == b ? 0 : k) the improvement is more modest from 5/6 to always 5, due to the LSL

12172–12175

Ignore these comments, this ended here accidentally. I'll remove them in the next update.

[samparker]

lib/Target/ARM/ARMISelLowering.cpp

12067

Couldn't you move the original combiner to live above this? Doesn't look like the known bits will be operating on floats.

12097

aaah, awesome, thanks!

12146

thanks for the great explanation!

test/CodeGen/ARM/atomic-cmpxchg.ll

29

Sorry, I really should of checked the tested before I asked you to explain... so thanks again for taking the time.

test/CodeGen/ARM/cmn.ll

10–11

Why change the input?

test/CodeGen/ARM/cmpxchg-O0.ll

24

Is this still beneficial when conditional moves are available? This test makes it look like an extra instruction is used.

test/CodeGen/ARM/select-imm.ll

155

I'd say that it's also worth checking that branches aren't generated for these tests.

test/CodeGen/Thumb/branchless-cmp.ll

4

Same here, it would be nice to ensure that there isn't a cmp and a br generated.

test/CodeGen/Thumb2/thumb2-cmn.ll

9–10

Why the input change for these tests?

[rogfer01]

ChangeLog:

• Reordered combiner and added early exit for non-integers.
• Constrained the CLZ combiner to T32 because it is not clear whether it is profitable in A32 (where conditional moves are more accessible)
• Added negative tests to make sure no branches are emitted where we don't want them.

test/CodeGen/ARM/cmn.ll

10–11

By doing an artificial select we avoid triggering the new combiner while forcing the previous cmn to appear.

test/CodeGen/ARM/cmpxchg-O0.ll

24

I've just reduced the scope of this change to only Thumb1 for this case as it is less obvious for Arm that this is an improvement.

[efriedma]

lib/Target/ARM/ARMISelLowering.cpp

12157

What are you trying to do here? With Thumb2, the relevant instructions are basically identical in ARM mode vs. Thumb mode, so doing this transform exclusively in Thumb mode doesn't really make sense.

[rogfer01]

lib/Target/ARM/ARMISelLowering.cpp

12157

Mmmh, I was trying to emit this in ARM because it is not always beneficial in cmpxcg-O0 below. We're going from 2 instructions to 3.

For Thumb 2 I understand there is an IT block (which the test does not check) so we stay in 3 instructions.

(I'm aware that the metric "number of instructions" is a bit poor)

I may be missing something here.

[rogfer01]

lib/Target/ARM/ARMISelLowering.cpp

12157

Also I noticed that if I'm not doing this in ARM any more checking for V5TOps is unnecessary.

[efriedma]

"it" is more like an instruction prefix than an actual instruction from the perspective of the CPU; it gets decoded into the same thing as an ARM mode conditional instruction. It's not an improvement to replace "it" with an actual instruction.

[rogfer01]

@efriedma Ah I see. If I get you right, the initial change was more sensible. Also Sam's concerns were on a file that is explicitly marked as -O0.

I will go back the original change.

[rogfer01]

ChangeLog:

• Reinstate the transformation for ARM as well.

[efriedma]

This patch would be easier to review if you would commit the changes to add new tests and RUN lines separately.

test/CodeGen/Thumb/long-setcc.ll

18

Could we fix this test to include some higher-quality CHECK lines, so it's clear what we're actually generating?

[rogfer01]

ChangeLog

• Split tests in D41122
• Improve test long-setcc.ll

[rogfer01]

ChangeLog:

• I forgot to update the Thumb counterpart of long-setcc.ll in the last update

[rogfer01]

ChangeLog

• New tests in D41122 are now updated in this change to show the change in codegen

[rogfer01]

Ping. Modulo the bugs we may find with ADDCARRY / SUBCARRY further thoughts on this?

[efriedma]

The use of clz is very similar to what the PowerPC backend does; I wonder if we can pick up any additional tricks from there.

lib/Target/ARM/ARMISelLowering.cpp

7386

When do we hit this case? I would expect that normally DAGCombiner::visitADDCARRY would combine this away.

9873–9883

Did you end up checking this?

12150

Do we generate ARMISD::CMOV for values which aren't integers?

[rogfer01]

lib/Target/ARM/ARMISelLowering.cpp

7386

In a testcase like

define i32 @test1a(i32 %a, i32 %b) {
entry:
  %cmp = icmp ne i32 %a, %b
  %cond = zext i1 %cmp to i32
  ret i32 %cond
}

without this we generate

	subs	r0, r0, r1
	movs	r1, #1
	subs	r2, r1, #1
	mov	r2, r0
	sbcs	r2, r1
	sbcs	r0, r2
	bx	lr

with it we can generate

	subs	r0, r0, r1
	subs	r1, r0, #1
	sbcs	r0, r1
	bx	lr

Alternatively I could queue a combine to the newly created ISD::SUBCARRY in line 12270 below. I think DCI.CombineTo should be able to do that. Does this sound right?

9873–9883

Yep, neither is redundant apparently, but I will check with more detail why it happens.

12150

In cases like this

float a;
int b;
a = (b & 0x1) != 0;

there is a moment where the DAG contains this node

t19: f32 = ARMISD::CMOV ConstantFP:f32<0.000000e+00>, ConstantFP:f32<1.000000e+00>, Constant:i32<1>, Register:i32 %cpsr, t18

[efriedma]

lib/Target/ARM/ARMISelLowering.cpp

7386

If we're not correctly adding new nodes to the DAGCombiner queue in a custom DAGCombine, we should fix that.

12150

Huh.

I guess we're getting lucky that the code below to create an AssertZext doesn't crash.

[rogfer01]

ChangeLog:

• Change ConvertBooleanCarryToCarryFlag to use ARMISD::SUBC x, 1 instead of ARMISD::ADDC x, ~0, this way a single combiner in PerformAddcSubcCombine is enough.
• Do not use ISD::SUBCARRY if the third operand is a zero. While the generic combiner can lower this to ISD::USUBO it may not have a chance to run before we lower that ISD::SUBCARRY.

[rogfer01]

ChangeLog:

• Remove whitespace and unused variables in LowerADDSUBCARRY

[rogfer01]

@efriedma @samparker any further comments on this change? We can add more clz-like tricks in later changes if needed.

[rogfer01]

Ping :-)

[samparker]

test/CodeGen/ARM/select-imm.ll

154

Why do we have a sxtb for T1 and not T2?

[rogfer01]

test/CodeGen/ARM/select-imm.ll

154

This comes from the load byte + sext in that testcase.

In Thumb2 we can coalesce the load byte + sext in a single ldrsb.w Rt, [Rn, #0] while in Thumb1 we can't do that (at least directly) because there ldsrb is of the form ldsrb Rt, [Rn, Rm] so a ldrb Rt, [Rn, #0] and then a sxtb Rt, Rn are used instead.

[samparker]

No other points from me, lets get this monster in. (fingers crossed)

test/CodeGen/ARM/select-imm.ll

154

Ok cheers, I didn't realise we didn't have ldrsb in thumb-1. But still odd this doesn't get combined away, guess it must be because the load has two users.

[rogfer01]

ChangeLog:

• Rebase with ToT

[rogfer01]

Let's see how this one goes :)

Thanks @samparker and @efriedma

lib/Target/ARM/ARMISelLowering.cpp

return DAG.getNode(Opc, SDLoc(Op), VTs, Op.getOperand(0),
                        Op.getOperand(1));
   return DAG.getNode(Opc, SDLoc(Op), VTs, Op.getOperand(0),
                      Op.getOperand(1), Op.getOperand(2));
 }
 
 SDValue ARMTargetLowering::LowerFSINCOS(SDValue Op, SelectionDAG &DAG) const {
   assert(Subtarget->isTargetDarwin());
 
   // For iOS, we want to call an alternative entry point: __sincos_stret,
   // return values are passed via sret.
   SDLoc dl(Op);
   SDValue Arg = Op.getOperand(0);
   EVT ArgVT = Arg.getValueType();
   Type *ArgTy = ArgVT.getTypeForEVT(*DAG.getContext());
   auto PtrVT = getPointerTy(DAG.getDataLayout());
 
   MachineFrameInfo &MFI = DAG.getMachineFunction().getFrameInfo();
return DAG.getNode(ARMISD::UMAAL, SDLoc(N),
                        {N->getOperand(0), N->getOperand(1),
                         AddcNode->getOperand(0), AddcNode->getOperand(1)});
   else
     return SDValue();
 }
 
 static SDValue PerformAddcSubcCombine(SDNode *N, SelectionDAG &DAG,
                                       const ARMSubtarget *Subtarget) {
   if (Subtarget->isThumb1Only()) {
     SDValue RHS = N->getOperand(1);
     if (ConstantSDNode *C = dyn_cast<ConstantSDNode>(RHS)) {
       int32_t imm = C->getSExtValue();
       if (imm < 0 && imm > INT_MIN) {
         SDLoc DL(N);
         RHS = DAG.getConstant(-imm, DL, MVT::i32);
         unsigned Opcode = (N->getOpcode() == ARMISD::ADDC) ? ARMISD::SUBC
                                                            : ARMISD::ADDC;
         return DAG.getNode(Opcode, DL, N->getVTList(), N->getOperand(0), RHS);
       }
     }
   }
   return SDValue();
 }
 
+static SDValue PerformADDCCombine(SDNode *N,
+                                  TargetLowering::DAGCombinerInfo &DCI,
+                                  const ARMSubtarget *Subtarget) {
+  SelectionDAG &DAG = DCI.DAG;
+
+  SDValue LHS = N->getOperand(0);
+  SDValue RHS = N->getOperand(1);
+  EVT VT = LHS.getValueType();
+  SDLoc DL(N);
+
+  // If the flag result is dead, turn this into an ADD.
+  if (!N->hasAnyUseOfValue(1))
+    return DCI.CombineTo(N, DAG.getNode(ISD::ADD, DL, VT, LHS, RHS),
+                         DAG.getConstant(0, DL, MVT::i32));
+
+  // Canonicalize constant to RHS.
+  ConstantSDNode *N0C = dyn_cast<ConstantSDNode>(LHS);
+  ConstantSDNode *N1C = dyn_cast<ConstantSDNode>(RHS);
+  if (N0C && !N1C)
+    return DAG.getNode(ARMISD::ADDC, DL, N->getVTList(), RHS, LHS);
+
+  // fold (addc x, 0) -> x + no carry out
+  if (isNullConstant(RHS))
+    return DCI.CombineTo(N, LHS, DAG.getConstant(0, DL, MVT::i32));
+
+  // If it cannot overflow, transform into an add.
+  if (DAG.computeOverflowKind(LHS, RHS) == SelectionDAG::OFK_Never)
+    return DCI.CombineTo(N, DAG.getNode(ISD::ADD, DL, VT, LHS, RHS),
+                         DAG.getConstant(0, DL, MVT::i32));
+
+  // Handle negative addend.
+  return PerformAddcSubcCombine(N, DAG, Subtarget);
+}
+
 static SDValue PerformAddeSubeCombine(SDNode *N, SelectionDAG &DAG,
                                       const ARMSubtarget *Subtarget) {
   if (Subtarget->isThumb1Only()) {
     SDValue RHS = N->getOperand(1);
     if (ConstantSDNode *C = dyn_cast<ConstantSDNode>(RHS)) {
       int64_t imm = C->getSExtValue();
       if (imm < 0) {
         SDLoc DL(N);

 }
 
 /// PerformADDECombine - Target-specific dag combine transform from
 /// ARMISD::ADDC, ARMISD::ADDE, and ISD::MUL_LOHI to MLAL or
 /// ARMISD::ADDC, ARMISD::ADDE and ARMISD::UMLAL to ARMISD::UMAAL
 static SDValue PerformADDECombine(SDNode *N,
                                   TargetLowering::DAGCombinerInfo &DCI,
                                   const ARMSubtarget *Subtarget) {
+  SelectionDAG &DAG = DCI.DAG;
+
+  SDValue LHS = N->getOperand(0);
+  SDValue RHS = N->getOperand(1);
+  SDValue Carry = N->getOperand(2);
+
+  // canonicalize constant to RHS
+  ConstantSDNode *N0C = dyn_cast<ConstantSDNode>(LHS);
+  ConstantSDNode *N1C = dyn_cast<ConstantSDNode>(RHS);
+  if (N0C && !N1C)
+    return DAG.getNode(ARMISD::ADDE, SDLoc(N), N->getVTList(),
+                       RHS, LHS, Carry);
+
+  // fold (adde x, y, false) -> (addc x, y)
+  if (isNullConstant(Carry))
+    return DAG.getNode(ARMISD::ADDC, SDLoc(N), N->getVTList(), LHS, RHS);
+
+  // fold (adde x, const, 1) -> (addc x, const+1)
+  if (isOneConstant(Carry) && N1C) {
+    SDLoc DL(N);
+    EVT VT = RHS.getValueType();
+    uint64_t plus1 = N1C->getZExtValue() + 1;
+    uint64_t max_uint = (1LL<<VT.getSizeInBits()) - 1;
+    RHS = DAG.getConstant(plus1 & max_uint, DL, VT);
+    if (plus1 <= max_uint)
+      return DAG.getNode(ARMISD::ADDC, DL, N->getVTList(), LHS, RHS);
+    else
+      return DCI.CombineTo(N, DAG.getNode(ISD::ADD, DL, VT, LHS, RHS),
+                           DAG.getConstant(1, DL, MVT::i32));
+  }
+
   // Only ARM and Thumb2 support UMLAL/SMLAL.
   if (Subtarget->isThumb1Only())
     return PerformAddeSubeCombine(N, DCI.DAG, Subtarget);
 
   // Only perform the checks after legalize when the pattern is available.
   if (DCI.isBeforeLegalize()) return SDValue();
 
   return AddCombineTo64bitUMAAL(N, DCI, Subtarget);
 }
 
+static SDValue PerformSUBCCombine(SDNode *N,
+                                  TargetLowering::DAGCombinerInfo &DCI,
+                                  const ARMSubtarget *Subtarget) {
+  SelectionDAG &DAG = DCI.DAG;
+
+  SDValue LHS = N->getOperand(0);
+  SDValue RHS = N->getOperand(1);
+  EVT VT = LHS.getValueType();
+  SDLoc DL(N);
+
+  // If the flag result is dead, turn this into an SUB.
+  if (!N->hasAnyUseOfValue(1))
+    return DCI.CombineTo(N, DAG.getNode(ISD::SUB, DL, VT, LHS, RHS),
+                         DAG.getConstant(1, DL, MVT::i32));
+
+  // fold (subc x, x) -> 0 + no borrow
+  if (LHS == RHS)
+    return DCI.CombineTo(N, DAG.getConstant(0, DL, VT),
+                         DAG.getConstant(1, DL, MVT::i32));
+
+  // fold (subc x, 0) -> x + no borrow
+  if (isNullConstant(RHS))
+    return DCI.CombineTo(N, LHS, DAG.getConstant(1, DL, MVT::i32));
+
+  // Canonicalize (sub -1, x) -> ~x, i.e. (xor x, -1) + no borrow
+  if (isAllOnesConstant(LHS))
+    return DCI.CombineTo(N, DAG.getNode(ISD::XOR, DL, VT, RHS, LHS),
+                         DAG.getConstant(1, DL, MVT::i32));
+
+  // Handle negative subtrahend.
+  return PerformAddcSubcCombine(N, DAG, Subtarget);
+}
+
+static SDValue PerformSUBECombine(SDNode *N, SelectionDAG &DAG,
+                                  const ARMSubtarget *Subtarget) {
+
+  SDValue LHS = N->getOperand(0);
+  SDValue RHS = N->getOperand(1);
+  SDValue Carry = N->getOperand(2);
+
+  // fold (sube x, y, false) -> (subc x, y)
+  if (isOneConstant(Carry))
+    return DAG.getNode(ARMISD::SUBC, SDLoc(N), N->getVTList(), LHS, RHS);
+
+  return PerformAddeSubeCombine(N, DAG, Subtarget);
+}
+
+
 /// PerformADDCombineWithOperands - Try DAG combinations for an ADD with
 /// operands N0 and N1.  This is a helper for PerformADDCombine that is
 /// called with the default operands, and if that fails, with commuted
 /// operands.
 static SDValue PerformADDCombineWithOperands(SDNode *N, SDValue N0, SDValue N1,
                                           TargetLowering::DAGCombinerInfo &DCI,
                                           const ARMSubtarget *Subtarget){
   // Attempt to create vpadd for this add.
if (VT == MVT::i32 &&
       }
       return DAG.getNode(Opc, SDLoc(N), VT, Vec, Lane);
     }
   }
 
   return SDValue();
 }
 
+static const APInt *isPowerOf2Constant(SDValue V) {
+  ConstantSDNode *C = dyn_cast<ConstantSDNode>(V);
+  if (!C)
+    return nullptr;
+  const APInt *CV = &C->getAPIntValue();
+  return CV->isPowerOf2() ? CV : nullptr;
+}
+
 SDValue ARMTargetLowering::PerformCMOVToBFICombine(SDNode *CMOV, SelectionDAG &DAG) const {
   // If we have a CMOV, OR and AND combination such as:
   //   if (x & CN)
   //     y |= CM;
   //
   // And:
   //   * CN is a single bit;
   //   * All bits covered by CM are known zero in y
SDValue ARMTargetLowering::PerformCMOVToBFICombine(SDNode *CMOV, SelectionDAG &DAG) const {
   // The compare must be against zero.
   if (!isNullConstant(CmpZ->getOperand(1)))
     return SDValue();
 
   assert(CmpZ->getOpcode() == ARMISD::CMPZ);
   SDValue And = CmpZ->getOperand(0);
   if (And->getOpcode() != ISD::AND)
     return SDValue();
-  ConstantSDNode *AndC = dyn_cast<ConstantSDNode>(And->getOperand(1));
-  if (!AndC || !AndC->getAPIntValue().isPowerOf2())
+  const APInt *AndC = isPowerOf2Constant(And->getOperand(1));
+  if (!AndC)
     return SDValue();
   SDValue X = And->getOperand(0);
 
   if (CC == ARMCC::EQ) {
     // We're performing an "equal to zero" compare. Swap the operands so we
     // canonicalize on a "not equal to zero" compare.
     std::swap(Op0, Op1);
   } else {
SDValue ARMTargetLowering::PerformCMOVToBFICombine(SDNode *CMOV, SelectionDAG &DAG) const {
   DAG.computeKnownBits(Y, Known);
   if ((OrCI & Known.Zero) != OrCI)
     return SDValue();
 
   // OK, we can do the combine.
   SDValue V = Y;
   SDLoc dl(X);
   EVT VT = X.getValueType();
-  unsigned BitInX = AndC->getAPIntValue().logBase2();
+  unsigned BitInX = AndC->logBase2();
 
   if (BitInX != 0) {
     // We must shift X first.
     X = DAG.getNode(ISD::SRL, dl, VT, X,
                     DAG.getConstant(BitInX, dl, VT));
   }
 
   for (unsigned BitInY = 0, NumActiveBits = OrCI.getActiveBits();
Res = DAG.getNode(ARMISD::CMOV, dl, VT, LHS, TrueVal, ARMcc,
                       N->getOperand(3), Cmp);
   } else if (CC == ARMCC::EQ && TrueVal == RHS) {
     SDValue ARMcc;
     SDValue NewCmp = getARMCmp(LHS, RHS, ISD::SETNE, ARMcc, DAG, dl);
     Res = DAG.getNode(ARMISD::CMOV, dl, VT, LHS, FalseVal, ARMcc,
                       N->getOperand(3), NewCmp);
   }
 
+  // Materialize a boolean comparison for integers so we can avoid branching.
+  if (VT.isInteger()) {
+    if (isNullConstant(FalseVal)) {
+      if (CC == ARMCC::EQ && isOneConstant(TrueVal)) {
+        if (!Subtarget->isThumb1Only() && Subtarget->hasV5TOps()) {
+          // If x == y then x - y == 0 and ARM's CLZ will return 32, shifting it
+          // right 5 bits will make that 32 be 1, otherwise it will be 0.
+          // CMOV 0, 1, ==, (CMPZ x, y) -> SRL (CTLZ (SUB x, y)), 5
+          SDValue Sub = DAG.getNode(ISD::SUB, dl, VT, LHS, RHS);
+          Res =
+              DAG.getNode(ISD::SRL, dl, VT, DAG.getNode(ISD::CTLZ, dl, VT, Sub),
+                          DAG.getConstant(5, dl, MVT::i32));
+        } else {
+          // ARMISD::SUBC will set the carry only when x == y which means that
+          // the final ARMISD::ADDE (which uses the carry generated by
+          // ARMISD::SUBC) will be 1. If x != y then the rightmost SUB will
+          // give a nonzero unsigned number which will always be higher than 0
+          // so the carry won't be set in that case and the two operands of
+          // ARMISD::ADDE will cancel each other.
+          // CMOV 0, 1, ==, (CMPZ x, y) ->
+          //      ARMISD::ADDE ((SUB x, y), (ARMISD::SUBC 0, (SUB x, y)))
+          SDValue Sub = DAG.getNode(ISD::SUB, dl, VT, LHS, RHS);
+          SDVTList VTs = DAG.getVTList(VT, MVT::i32);
+          SDValue Neg = DAG.getNode(ARMISD::SUBC, dl, VTs, FalseVal, Sub);
+          Res = DAG.getNode(ARMISD::ADDE, dl, VTs, Sub, Neg, Neg.getValue(1));
+        }
+      } else if (CC == ARMCC::NE && LHS != RHS &&
+                 (!Subtarget->isThumb1Only() || isPowerOf2Constant(TrueVal))) {
+        // This seems pointless but will allow us to combine it further below.
+        // CMOV 0, z, !=, (CMPZ x, y) -> CMOV (SUB x, y), z, !=, (CMPZ x, y)
+        SDValue Sub = DAG.getNode(ISD::SUB, dl, VT, LHS, RHS);
+        Res = DAG.getNode(ARMISD::CMOV, dl, VT, Sub, TrueVal, ARMcc,
+                          N->getOperand(3), Cmp);
+      }
+    } else if (isNullConstant(TrueVal)) {
+      if (CC == ARMCC::EQ && LHS != RHS &&
+          (!Subtarget->isThumb1Only() || isPowerOf2Constant(FalseVal))) {
+        // This seems pointless but will allow us to combine it further below
+        // Note that we change == for != as this is the dual for the case above.
+        // CMOV z, 0, ==, (CMPZ x, y) -> CMOV (SUB x, y), z, !=, (CMPZ x, y)
+        SDValue Sub = DAG.getNode(ISD::SUB, dl, VT, LHS, RHS);
+        Res = DAG.getNode(ARMISD::CMOV, dl, VT, Sub, FalseVal,
+                          DAG.getConstant(ARMCC::NE, dl, MVT::i32),
+                          N->getOperand(3), Cmp);
+      }
+    }
+
+    // On Thumb1, the DAG above may be further combined.
+    //  CMOV (SUB x, y), z, !=, (CMPZ x, y) ->
+    //      ARMISD::SUBE ((SUB x, y), (ARMISD::SUBC (SUB x, y), z))
+    // to avoid branching.
+    const APInt *TrueConst;
+    if (Subtarget->isThumb1Only() && CC == ARMCC::NE &&
+        (FalseVal.getOpcode() == ISD::SUB) && (FalseVal.getOperand(0) == LHS) &&
+        (FalseVal.getOperand(1) == RHS) &&
+        (TrueConst = isPowerOf2Constant(TrueVal))) {
+      SDVTList VTs = DAG.getVTList(VT, MVT::i32);
+      unsigned ShAmt = TrueConst->logBase2();
+      if (ShAmt)
+        TrueVal = DAG.getConstant(1, dl, VT);
+      SDValue Subc = DAG.getNode(ARMISD::SUBC, dl, VTs, FalseVal, TrueVal);
+      Res =
+          DAG.getNode(ARMISD::SUBE, dl, VTs, FalseVal, Subc, Subc.getValue(1));
+      if (ShAmt)
+        Res = DAG.getNode(ISD::SHL, dl, VT, Res,
+                          DAG.getConstant(ShAmt, dl, MVT::i32));
+    }
+  }
+
   // (cmov F T ne CPSR (cmpz (cmov 0 1 CC CPSR Cmp) 0))
   // -> (cmov F T CC CPSR Cmp)
   if (CC == ARMCC::NE && LHS.getOpcode() == ARMISD::CMOV && LHS->hasOneUse()) {
     auto *LHS0C = dyn_cast<ConstantSDNode>(LHS->getOperand(0));
     auto *LHS1C = dyn_cast<ConstantSDNode>(LHS->getOperand(1));
     auto *RHSC = dyn_cast<ConstantSDNode>(RHS);
     if ((LHS0C && LHS0C->getZExtValue() == 0) &&
         (LHS1C && LHS1C->getZExtValue() == 1) &&
         (RHSC && RHSC->getZExtValue() == 0)) {
       return DAG.getNode(ARMISD::CMOV, dl, VT, FalseVal, TrueVal,
                          LHS->getOperand(2), LHS->getOperand(3),
                          LHS->getOperand(4));
     }
   }
 
   if (Res.getNode()) {
     KnownBits Known;
     DAG.computeKnownBits(SDValue(N,0), Known);
     // Capture demanded bits information that would be otherwise lost.
     if (Known.Zero == 0xfffffffe)
       Res = DAG.getNode(ISD::AssertZext, dl, MVT::i32, Res,
                         DAG.getValueType(MVT::i1));
     else if (Known.Zero == 0xffffff00)
       Res = DAG.getNode(ISD::AssertZext, dl, MVT::i32, Res,
                         DAG.getValueType(MVT::i8));
     else if (Known.Zero == 0xffff0000)
       Res = DAG.getNode(ISD::AssertZext, dl, MVT::i32, Res,
                         DAG.getValueType(MVT::i16));
   }
 
   return Res;
 }
 
 SDValue ARMTargetLowering::PerformDAGCombine(SDNode *N,
                                              DAGCombinerInfo &DCI) const {
   switch (N->getOpcode()) {
   default: break;
+  case ARMISD::ADDC:    return PerformADDCCombine(N, DCI, Subtarget);
   case ARMISD::ADDE:    return PerformADDECombine(N, DCI, Subtarget);
+  case ARMISD::SUBC:    return PerformSUBCCombine(N, DCI, Subtarget);
+  case ARMISD::SUBE:    return PerformSUBECombine(N, DCI.DAG, Subtarget);
   case ARMISD::UMLAL:   return PerformUMLALCombine(N, DCI.DAG, Subtarget);
   case ISD::ADD:        return PerformADDCombine(N, DCI, Subtarget);
   case ISD::SUB:        return PerformSUBCombine(N, DCI);
   case ISD::MUL:        return PerformMULCombine(N, DCI, Subtarget);
   case ISD::OR:         return PerformORCombine(N, DCI, Subtarget);
   case ISD::XOR:        return PerformXORCombine(N, DCI, Subtarget);
   case ISD::AND:        return PerformANDCombine(N, DCI, Subtarget);
-  case ARMISD::ADDC:
-  case ARMISD::SUBC:    return PerformAddcSubcCombine(N, DCI.DAG, Subtarget);
-  case ARMISD::SUBE:    return PerformAddeSubeCombine(N, DCI.DAG, Subtarget);
   case ARMISD::BFI:     return PerformBFICombine(N, DCI);
   case ARMISD::VMOVRRD: return PerformVMOVRRDCombine(N, DCI, Subtarget);
   case ARMISD::VMOVDRR: return PerformVMOVDRRCombine(N, DCI.DAG);
   case ISD::STORE:      return PerformSTORECombine(N, DCI);
   case ISD::BUILD_VECTOR: return PerformBUILD_VECTORCombine(N, DCI, Subtarget);
   case ISD::INSERT_VECTOR_ELT: return PerformInsertEltCombine(N, DCI);
   case ISD::VECTOR_SHUFFLE: return PerformVECTOR_SHUFFLECombine(N, DCI.DAG);
   case ARMISD::VDUPLANE: return PerformVDUPLANECombine(N, DCI);

test/CodeGen/ARM/atomic-cmpxchg.ll

 entry:
   %0 = cmpxchg i8* %addr, i8 %desired, i8 %new monotonic monotonic
   %1 = extractvalue { i8, i1 } %0, 1
   ret i1 %1
 }
 
 ; CHECK-ARM-LABEL: test_cmpxchg_res_i8
 ; CHECK-ARM: bl __sync_val_compare_and_swap_1
-; CHECK-ARM: mov [[REG:r[0-9]+]], #0
-; CHECK-ARM: cmp r0, {{r[0-9]+}}
-; CHECK-ARM: moveq [[REG]], #1
-; CHECK-ARM: mov r0, [[REG]]
+; CHECK-ARM: sub r0, r0, {{r[0-9]+}}
+; CHECK-ARM: rsbs [[REG:r[0-9]+]], r0, #0
+; CHECK-ARM: adc r0, r0, [[REG]]
 
 ; CHECK-THUMB-LABEL: test_cmpxchg_res_i8
 ; CHECK-THUMB: bl __sync_val_compare_and_swap_1
 ; CHECK-THUMB-NOT: mov [[R1:r[0-7]]], r0
-; CHECK-THUMB: movs [[R1:r[0-7]]], r0
-; CHECK-THUMB: movs r0, #1
-; CHECK-THUMB: movs [[R2:r[0-9]+]], #0
-; CHECK-THUMB: cmp [[R1]], {{r[0-9]+}}
-; CHECK-THUMB: beq
-; CHECK-THUMB: movs r0, [[R2]]
+; CHECK-THUMB: subs [[R1:r[0-7]]], r0, {{r[0-9]+}}
+; CHECK-THUMB: movs r0, #0
+; CHECK-THUMB: subs r0, r0, [[R1]]
+; CHECK-THUMB: adcs r0, [[R1]]
 
 ; CHECK-ARMV6-LABEL: test_cmpxchg_res_i8:
 ; CHECK-ARMV6-NEXT:  .fnstart
 ; CHECK-ARMV6-NEXT: uxtb [[DESIRED:r[0-9]+]], r1
 ; CHECK-ARMV6-NEXT: [[TRY:.LBB[0-9_]+]]:
 ; CHECK-ARMV6-NEXT: ldrexb [[LD:r[0-9]+]], [r0]
 ; CHECK-ARMV6-NEXT: cmp [[LD]], [[DESIRED]]
 ; CHECK-ARMV6-NEXT: movne [[RES:r[0-9]+]], #0
 ; CHECK-ARMV6-NEXT: bxne lr
 ; CHECK-ARMV6-NEXT: strexb [[SUCCESS:r[0-9]+]], r2, [r0]
 ; CHECK-ARMV6-NEXT: cmp [[SUCCESS]], #0
 ; CHECK-ARMV6-NEXT: moveq [[RES]], #1
 ; CHECK-ARMV6-NEXT: bxeq lr
 ; CHECK-ARMV6-NEXT: b [[TRY]]
 
 ; CHECK-THUMBV6-LABEL: test_cmpxchg_res_i8:
 ; CHECK-THUMBV6:       mov [[EXPECTED:r[0-9]+]], r1
 ; CHECK-THUMBV6-NEXT:  bl __sync_val_compare_and_swap_1
-; CHECK-THUMBV6-NEXT:  mov [[RES:r[0-9]+]], r0
-; CHECK-THUMBV6-NEXT:  movs r0, #1
-; CHECK-THUMBV6-NEXT:  movs [[ZERO:r[0-9]+]], #0
-; CHECK-THUMBV6-NEXT:  cmp [[RES]], [[EXPECTED]]
-; CHECK-THUMBV6-NEXT:  beq [[END:.LBB[0-9_]+]]
-; CHECK-THUMBV6-NEXT:  mov r0, [[ZERO]]
-; CHECK-THUMBV6-NEXT: [[END]]:
-; CHECK-THUMBV6-NEXT:  pop {{.*}}pc}
+; CHECK-THUMBV6-NEXT:  subs [[R1:r[0-7]]], r0, {{r[0-9]+}}
+; CHECK-THUMBV6-NEXT:  movs r0, #0
+; CHECK-THUMBV6-NEXT:  subs r0, r0, [[R1]]
+; CHECK-THUMBV6-NEXT:  adcs r0, [[R1]]
 
 ; CHECK-ARMV7-LABEL: test_cmpxchg_res_i8:
 ; CHECK-ARMV7-NEXT: .fnstart
 ; CHECK-ARMV7-NEXT: uxtb [[DESIRED:r[0-9]+]], r1
 ; CHECK-ARMV7-NEXT: b [[TRY:.LBB[0-9_]+]]
 ; CHECK-ARMV7-NEXT: [[HEAD:.LBB[0-9_]+]]:
 ; CHECK-ARMV7-NEXT: strexb [[SUCCESS:r[0-9]+]], r2, [r0]
 ; CHECK-ARMV7-NEXT: cmp [[SUCCESS]], #0

test/CodeGen/ARM/cmn.ll

 ; RUN: llc < %s -mtriple thumbv7-apple-ios | FileCheck %s
 ; <rdar://problem/7569620>
 
 define i32 @compare_i_gt(i32 %a) {
 entry:
 ; CHECK:     compare_i_gt
 ; CHECK-NOT: mvn
 ; CHECK:     cmn
   %cmp = icmp sgt i32 %a, -78
-  %. = zext i1 %cmp to i32
-  ret i32 %.
+  %ret = select i1 %cmp, i32 42, i32 24
+  ret i32 %ret
 }
 
 define i32 @compare_r_eq(i32 %a, i32 %b) {
 entry:
 ; CHECK: compare_r_eq
 ; CHECK: cmn
   %sub = sub nsw i32 0, %b
   %cmp = icmp eq i32 %a, %sub
-  %. = zext i1 %cmp to i32
-  ret i32 %.
+  %ret = select i1 %cmp, i32 42, i32 24
+  ret i32 %ret
 }

test/CodeGen/ARM/cmpxchg-O0.ll

 ; CHECK:     mov{{s?}} [[STATUS:r[0-9]+]], #0
 ; CHECK:     ldrexb [[OLD:r[0-9]+]], [r0]
 ; CHECK:     cmp [[OLD]], [[DESIRED]]
 ; CHECK:     bne [[DONE:.LBB[0-9]+_[0-9]+]]
 ; CHECK:     strexb [[STATUS]], r2, [r0]
 ; CHECK:     cmp{{(\.w)?}} [[STATUS]], #0
 ; CHECK:     bne [[RETRY]]
 ; CHECK: [[DONE]]:
-; CHECK:     cmp{{(\.w)?}} [[OLD]], [[DESIRED]]
-; CHECK:     {{moveq|movweq}} {{r[0-9]+}}, #1
+; Materialisation of a boolean is done with sub/clz/lsr
+; CHECK:     sub{{(s)?}} [[CMP1:r[0-9]+]], [[OLD]], [[DESIRED]]
+; CHECK:     clz [[CMP2:r[0-9]+]], [[CMP1]]
+; CHECK:     lsr{{(s)?}} {{r[0-9]+}}, [[CMP2]], #5
 ; CHECK:     dmb ish
   %res = cmpxchg i8* %addr, i8 %desired, i8 %new seq_cst monotonic
   ret { i8, i1 } %res
 }
 
 define { i16, i1 } @test_cmpxchg_16(i16* %addr, i16 %desired, i16 %new) nounwind {
 ; CHECK-LABEL: test_cmpxchg_16:
 ; CHECK:     dmb ish
 ; CHECK:     uxth [[DESIRED:r[0-9]+]], [[DESIRED]]
 ; CHECK: [[RETRY:.LBB[0-9]+_[0-9]+]]:
 ; CHECK:     mov{{s?}} [[STATUS:r[0-9]+]], #0
 ; CHECK:     ldrexh [[OLD:r[0-9]+]], [r0]
 ; CHECK:     cmp [[OLD]], [[DESIRED]]
 ; CHECK:     bne [[DONE:.LBB[0-9]+_[0-9]+]]
 ; CHECK:     strexh [[STATUS]], r2, [r0]
 ; CHECK:     cmp{{(\.w)?}} [[STATUS]], #0
 ; CHECK:     bne [[RETRY]]
 ; CHECK: [[DONE]]:
-; CHECK:     cmp{{(\.w)?}} [[OLD]], [[DESIRED]]
-; CHECK:     {{moveq|movweq}} {{r[0-9]+}}, #1
+; Materialisation of a boolean is done with sub/clz/lsr
+; CHECK:     sub{{(s)?}} [[CMP1:r[0-9]+]], [[OLD]], [[DESIRED]]
+; CHECK:     clz [[CMP2:r[0-9]+]], [[CMP1]]
+; CHECK:     lsr{{(s)?}} {{r[0-9]+}}, [[CMP2]], #5
 ; CHECK:     dmb ish
   %res = cmpxchg i16* %addr, i16 %desired, i16 %new seq_cst monotonic
   ret { i16, i1 } %res
 }
 
 define { i32, i1 } @test_cmpxchg_32(i32* %addr, i32 %desired, i32 %new) nounwind {
 ; CHECK-LABEL: test_cmpxchg_32:
 ; CHECK:     dmb ish
 ; CHECK-NOT:     uxt
 ; CHECK: [[RETRY:.LBB[0-9]+_[0-9]+]]:
 ; CHECK:     mov{{s?}} [[STATUS:r[0-9]+]], #0
 ; CHECK:     ldrex [[OLD:r[0-9]+]], [r0]
 ; CHECK:     cmp [[OLD]], [[DESIRED]]
 ; CHECK:     bne [[DONE:.LBB[0-9]+_[0-9]+]]
 ; CHECK:     strex [[STATUS]], r2, [r0]
 ; CHECK:     cmp{{(\.w)?}} [[STATUS]], #0
 ; CHECK:     bne [[RETRY]]
 ; CHECK: [[DONE]]:
-; CHECK:     cmp{{(\.w)?}} [[OLD]], [[DESIRED]]
-; CHECK:     {{moveq|movweq}} {{r[0-9]+}}, #1
+; Materialisation of a boolean is done with sub/clz/lsr
+; CHECK:     sub{{(s)?}} [[CMP1:r[0-9]+]], [[OLD]], [[DESIRED]]
+; CHECK:     clz [[CMP2:r[0-9]+]], [[CMP1]]
+; CHECK:     lsr{{(s)?}} {{r[0-9]+}}, [[CMP2]], #5
 ; CHECK:     dmb ish
   %res = cmpxchg i32* %addr, i32 %desired, i32 %new seq_cst monotonic
   ret { i32, i1 } %res
 }
 
 define { i64, i1 } @test_cmpxchg_64(i64* %addr, i64 %desired, i64 %new) nounwind {
 ; CHECK-LABEL: test_cmpxchg_64:
 ; CHECK:     dmb ish

test/CodeGen/ARM/fp16-promote.ll

 ; instructions anyway.
 ; CHECK-ALL-LABEL: test_fcmp_une:
 ; CHECK-FP16: vcvtb.f32.f16
 ; CHECK-FP16: vcvtb.f32.f16
 ; CHECK-LIBCALL: bl __aeabi_h2f
 ; CHECK-LIBCALL: bl __aeabi_h2f
 ; CHECK-VFP: vcmp.f32
 ; CHECK-NOVFP: bl __aeabi_fcmpeq
-; CHECK-FP16: vmrs APSR_nzcv, fpscr
-; CHECK-ALL: movw{{ne|eq}}
+; CHECK-VFP-NEXT: vmrs APSR_nzcv, fpscr
+; CHECK-VFP-NEXT: movwne
+; CHECK-NOVFP-NEXT: clz r0, r0
+; CHECK-NOVFP-NEXT: lsr r0, r0, #5
 define i1 @test_fcmp_une(half* %p, half* %q) #0 {
   %a = load half, half* %p, align 2
   %b = load half, half* %q, align 2
   %r = fcmp une half %a, %b
   ret i1 %r
 }
 
 ; CHECK-ALL-LABEL: test_fcmp_ueq:

test/CodeGen/ARM/long-setcc.ll

define i1 @t2(i64 %x) {
   ret i1 %tmp
 }
 
 define i1 @t3(i32 %x) {
   %tmp = icmp ugt i32 %x, -1
   ret i1 %tmp
 }
 
-; CHECK: cmp
 ; CHECK-NOT: cmp
 

test/CodeGen/ARM/select-imm.ll

 ; RUN: llc -mtriple=arm-eabi %s -o - | FileCheck %s --check-prefix=ARM
 
 ; RUN: llc -mtriple=arm-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - \
 ; RUN:  | FileCheck %s --check-prefix=ARMT2
 
+; RUN: llc -mtriple=thumb-eabi -mcpu=cortex-m0 %s -o - \
+; RUN:  | FileCheck %s --check-prefix=THUMB1
+
 ; RUN: llc -mtriple=thumb-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - \
 ; RUN:  | FileCheck %s --check-prefix=THUMB2
 
+; RUN: llc -mtriple=thumbv8m.base-eabi %s -o - \
+; RUN:  | FileCheck %s --check-prefix=V8MBASE
+
 define i32 @t1(i32 %c) nounwind readnone {
 entry:
 ; ARM-LABEL: t1:
 ; ARM: mov [[R1:r[0-9]+]], #101
 ; ARM: orr [[R1b:r[0-9]+]], [[R1]], #256
 ; ARM: movgt {{r[0-1]}}, #123
 
 ; ARMT2-LABEL: t1:
 ; ARMT2: movw [[R:r[0-1]]], #357
 ; ARMT2: movwgt [[R]], #123
 
+; THUMB1-LABEL: t1:
+; THUMB1: cmp r{{[0-9]+}}, #1
+; THUMB1: bgt
+
 ; THUMB2-LABEL: t1:
 ; THUMB2: movw [[R:r[0-1]]], #357
 ; THUMB2: movgt [[R]], #123
 
   %0 = icmp sgt i32 %c, 1
   %1 = select i1 %0, i32 123, i32 357
   ret i32 %1
 }
 
 define i32 @t2(i32 %c) nounwind readnone {
 entry:
 ; ARM-LABEL: t2:
 ; ARM: mov [[R:r[0-9]+]], #101
 ; ARM: orr [[R]], [[R]], #256
 ; ARM: movle [[R]], #123
 
 ; ARMT2-LABEL: t2:
 ; ARMT2: mov [[R:r[0-1]]], #123
 ; ARMT2: movwgt [[R]], #357
 
+; THUMB1-LABEL: t2:
+; THUMB1: cmp r{{[0-9]+}}, #1
+; THUMB1: bgt
+
 ; THUMB2-LABEL: t2:
 ; THUMB2: mov{{(s|\.w)}} [[R:r[0-1]]], #123
 ; THUMB2: movwgt [[R]], #357
 
   %0 = icmp sgt i32 %c, 1
   %1 = select i1 %0, i32 357, i32 123
   ret i32 %1
 }
 
 define i32 @t3(i32 %a) nounwind readnone {
 entry:
 ; ARM-LABEL: t3:
-; ARM: mov [[R:r[0-1]]], #0
-; ARM: moveq [[R]], #1
+; ARM: rsbs r1, r0, #0
+; ARM: adc  r0, r0, r1
 
 ; ARMT2-LABEL: t3:
-; ARMT2: mov [[R:r[0-1]]], #0
-; ARMT2: movweq [[R]], #1
+; ARMT2: clz r0, r0
+; ARMT2: lsr r0, r0, #5
+
+; THUMB1-LABEL: t3:
+; THUMB1: movs r1, #0
+; THUMB1: subs r1, r1, r0
+; THUMB1: adcs r0, r1
 
 ; THUMB2-LABEL: t3:
-; THUMB2: mov{{(s|\.w)}} [[R:r[0-1]]], #0
-; THUMB2: moveq [[R]], #1
+; THUMB2: clz r0, r0
+; THUMB2: lsrs r0, r0, #5
   %0 = icmp eq i32 %a, 160
   %1 = zext i1 %0 to i32
   ret i32 %1
 }
 
 define i32 @t4(i32 %a, i32 %b, i32 %x) nounwind {
 entry:
 ; ARM-LABEL: t4:
 ; ARM: ldr
 ; ARM: mov{{lt|ge}}
 
 ; ARMT2-LABEL: t4:
 ; ARMT2: movwlt [[R0:r[0-9]+]], #65365
 ; ARMT2: movtlt [[R0]], #65365
 
+; THUMB1-LABEL: t4:
+; THUMB1: cmp r{{[0-9]+}}, r{{[0-9]+}}
+; THUMB1: blt
+
 ; THUMB2-LABEL: t4:
 ; THUMB2: mvnlt [[R0:r[0-9]+]], #11141290
   %0 = icmp slt i32 %a, %b
   %1 = select i1 %0, i32 4283826005, i32 %x
   ret i32 %1
 }
 
 ; rdar://9758317
 define i32 @t5(i32 %a) nounwind {
 entry:
 ; ARM-LABEL: t5:
 ; ARM-NOT: mov
-; ARM: cmp r0, #1
+; ARM: sub  r0, r0, #1
 ; ARM-NOT: mov
-; ARM: movne r0, #0
+; ARM: rsbs r1, r0, #0
+; ARM: adc  r0, r0, r1
+
+; THUMB1-LABEL: t5:
+; THUMB1-NOT: bne
+; THUMB1: movs r0, #0
+; THUMB1: subs r0, r0, r1
+; THUMB1: adcs r0, r1
 
 ; THUMB2-LABEL: t5:
 ; THUMB2-NOT: mov
-; THUMB2: cmp r0, #1
-; THUMB2: it ne
-; THUMB2: movne r0, #0
+; THUMB2: subs r0, #1
+; THUMB2: clz  r0, r0
+; THUMB2: lsrs r0, r0, #5
+
   %cmp = icmp eq i32 %a, 1
   %conv = zext i1 %cmp to i32
   ret i32 %conv
 }
 
 define i32 @t6(i32 %a) nounwind {
 entry:
 ; ARM-LABEL: t6:
 ; ARM-NOT: mov
 ; ARM: cmp r0, #0
 ; ARM: movne r0, #1
 
+; THUMB1-LABEL: t6:
+; THUMB1: cmp r{{[0-9]+}}, #0
+; THUMB1: bne
+
 ; THUMB2-LABEL: t6:
 ; THUMB2-NOT: mov
 ; THUMB2: cmp r0, #0
 ; THUMB2: it ne
 ; THUMB2: movne r0, #1
   %tobool = icmp ne i32 %a, 0
   %lnot.ext = zext i1 %tobool to i32
   ret i32 %lnot.ext
 }
+
+define i32 @t7(i32 %a, i32 %b) nounwind readnone {
+entry:
+; ARM-LABEL: t7:
+; ARM: subs r0, r0, r1
+; ARM: movne r0, #4
+
+; ARMT2-LABEL: t7:
+; ARMT2: subs r0, r0, r1
+; ARMT2: movwne r0, #4
+
+; THUMB1-LABEL: t7:
+; THUMB1: subs r0, r0, r1
+; THUMB1: subs r1, r0, #1
+; THUMB1: sbcs r0, r1
+; THUMB1: lsls r0, r0, #2
+
+; THUMB2-LABEL: t7:
+; THUMB2: subs r0, r0, r1
+; THUMB2: it ne
+; THUMB2: movne r0, #4
+  %0 = icmp ne i32 %a, %b
+  %1 = select i1 %0, i32 4, i32 0
+  ret i32 %1
+}
+
+define void @t8(i32 %a) {
+entry:
+
+; ARM scheduler emits icmp/zext before both calls, so isn't relevant
+
+; ARMT2-LABEL: t8:
+; ARMT2: bl t7
+; ARMT2: mov r1, r0
+; ARMT2: sub r0, r4, #5
+; ARMT2: clz r0, r0
+; ARMT2: lsr r0, r0, #5
+
+; THUMB1-LABEL: t8:
+; THUMB1: bl t7
+; THUMB1: mov r1, r0
+; THUMB1: subs r2, r4, #5
+; THUMB1: movs r0, #0
+; THUMB1: subs r0, r0, r2
+; THUMB1: adcs r0, r2
+
+; THUMB2-LABEL: t8:
+; THUMB2: bl t7
+; THUMB2: mov r1, r0
+; THUMB2: subs r0, r4, #5
+; THUMB2: clz r0, r0
+; THUMB2: lsrs r0, r0, #5
+
+  %cmp = icmp eq i32 %a, 5
+  %conv = zext i1 %cmp to i32
+  %call = tail call i32 @t7(i32 9, i32 %a)
+  tail call i32 @t7(i32 %conv, i32 %call)
+  ret void
+}
+
+define void @t9(i8* %a, i8 %b) {
+entry:
+
+; ARM scheduler emits icmp/zext before both calls, so isn't relevant
+
+; ARMT2-LABEL: t9:
+; ARMT2: bl f
+; ARMT2: uxtb r0, r4
+; ARMT2: cmp  r0, r0
+; ARMT2: add  r1, r4, #1
+; ARMT2: mov  r2, r0
+; ARMT2: add  r2, r2, #1
+; ARMT2: add  r1, r1, #1
+; ARMT2: uxtb r3, r2
+; ARMT2: cmp  r3, r0
+
+; THUMB1-LABEL: t9:
+; THUMB1: bl f
+; THUMB1: sxtb r1, r4
+; THUMB1: uxtb r0, r1
+; THUMB1: cmp  r0, r0
+; THUMB1: adds r1, r1, #1
+; THUMB1: mov  r2, r0
+; THUMB1: adds r1, r1, #1
+; THUMB1: adds r2, r2, #1
+; THUMB1: uxtb r3, r2
+; THUMB1: cmp  r3, r0
+
+; THUMB2-LABEL: t9:
+; THUMB2: bl f
+; THUMB2: uxtb r0, r4
+; THUMB2: cmp  r0, r0
+; THUMB2: adds r1, r4, #1
+; THUMB2: mov  r2, r0
+; THUMB2: adds r2, #1
+; THUMB2: adds r1, #1
+; THUMB2: uxtb r3, r2
+; THUMB2: cmp  r3, r0
+
+  %0 = load i8, i8* %a
+  %conv = sext i8 %0 to i32
+  %conv119 = zext i8 %0 to i32
+  %conv522 = and i32 %conv, 255
+  %cmp723 = icmp eq i32 %conv522, %conv119
+  tail call void @f(i1 zeroext %cmp723)
+  br i1 %cmp723, label %while.body, label %while.end
+
+while.body:                                       ; preds = %entry, %while.body
+  %ref.025 = phi i8 [ %inc9, %while.body ], [ %0, %entry ]
+  %in.024 = phi i32 [ %inc, %while.body ], [ %conv, %entry ]
+  %inc = add i32 %in.024, 1
+  %inc9 = add i8 %ref.025, 1
+  %conv1 = zext i8 %inc9 to i32
+  %cmp = icmp slt i32 %conv1, %conv119
+  %conv5 = and i32 %inc, 255
+  br i1 %cmp, label %while.body, label %while.end
+
+while.end:
+  ret void
+}
+
+declare void @f(i1 zeroext)
+
+
+define i1 @t10() {
+entry:
+  %q = alloca i32
+  %p = alloca i32
+  store i32 -3, i32* %q
+  store i32 -8, i32* %p
+  %0 = load i32, i32* %q
+  %1 = load i32, i32* %p
+  %div = sdiv i32 %0, %1
+  %mul = mul nsw i32 %div, %1
+  %rem = srem i32 %0, %1
+  %add = add nsw i32 %mul, %rem
+  %cmp = icmp eq i32 %add, %0
+  ret i1 %cmp
+
+; ARM-LABEL: t10:
+; ARM: rsbs r1, r0, #0
+; ARM: adc  r0, r0, r1
+
+; ARMT2-LABEL: t10:
+; ARMT2: clz r0, r0
+; ARMT2: lsr r0, r0, #5
+
+; THUMB1-LABEL: t10:
+; THUMB1: movs r0, #0
+; THUMB1: subs r0, r0, r1
+; THUMB1: adcs r0, r1
+
+; THUMB2-LABEL: t10:
+; THUMB2: clz r0, r0
+; THUMB2: lsrs r0, r0, #5
+
+; V8MBASE-LABEL: t10:
+; V8MBASE-NOT: movs r0, #0
+; V8MBASE: movs r0, #1
+}
+
+define i1 @t11() {
+entry:
+  %bit = alloca i32
+  %load = load i32, i32* %bit
+  %clear = and i32 %load, -4096
+  %set = or i32 %clear, 33
+  store i32 %set, i32* %bit
+  %load1 = load i32, i32* %bit
+  %clear2 = and i32 %load1, -33550337
+  %set3 = or i32 %clear2, 40960
+  %clear5 = and i32 %set3, 4095
+  %rem = srem i32 %clear5, 10
+  %clear9 = and i32 %set3, -4096
+  %set10 = or i32 %clear9, %rem
+  store i32 %set10, i32* %bit
+  %clear12 = and i32 %set10, 4095
+  %cmp = icmp eq i32 %clear12, 3
+  ret i1 %cmp
+
+; ARM-LABEL: t11:
+; ARM: rsbs r1, r0, #0
+; ARM: adc  r0, r0, r1
+
+; ARMT2-LABEL: t11:
+; ARMT2: clz r0, r0
+; ARMT2: lsr r0, r0, #5
+
+; THUMB1-LABEL: t11:
+; THUMB1-NOT: movs r0, #0
+; THUMB1: movs r0, #1
+
+; THUMB2-LABEL: t11:
+; THUMB2: clz r0, r0
+; THUMB2: lsrs r0, r0, #5
+
+; V8MBASE-LABEL: t11:
+; V8MBASE-NOT: movs r0, #0
+; V8MBASE: movs r0, #1
+}
+

test/CodeGen/ARM/setcc-logic.ll

; CHECK-NEXT:    bx lr
   ret i1 %and
 }
 
 ; PR32401 - https://bugs.llvm.org/show_bug.cgi?id=32401
 
 define zeroext i1 @and_eq(i32 %a, i32 %b, i32 %c, i32 %d) nounwind {
 ; CHECK-LABEL: and_eq:
 ; CHECK:       @ BB#0:
-; CHECK-NEXT:    eor r2, r2, r3
-; CHECK-NEXT:    eor r0, r0, r1
-; CHECK-NEXT:    orrs r0, r0, r2
-; CHECK-NEXT:    mov r0, #0
-; CHECK-NEXT:    movweq r0, #1
-; CHECK-NEXT:    bx lr
+; CHECK: eor     r2, r2, r3
+; CHECK: eor     r0, r0, r1
+; CHECK: orr     r0, r0, r2
+; CHECK: clz     r0, r0
+; CHECK: lsr     r0, r0, #5
+; CHECK: bx      lr
   %cmp1 = icmp eq i32 %a, %b
   %cmp2 = icmp eq i32 %c, %d
   %and = and i1 %cmp1, %cmp2
   ret i1 %and
 }
 
 define zeroext i1 @or_ne(i32 %a, i32 %b, i32 %c, i32 %d) nounwind {
 ; CHECK-LABEL: or_ne:

test/CodeGen/Thumb/branchless-cmp.ll

+; RUN: llc -mtriple=thumb-eabi -mcpu=cortex-m0 %s -verify-machineinstrs -o - | FileCheck %s
+
+define i32 @test1a(i32 %a, i32 %b) {
+entry:
+  %cmp = icmp ne i32 %a, %b
+  %cond = zext i1 %cmp to i32
+  ret i32 %cond
+; CHECK-LABEL: test1a:
+; CHECK:       subs r0, r0, r1
+; CHECK-NEXT:  subs r1, r0, #1
+; CHECK-NEXT:  sbcs r0, r1
+}
+
+define i32 @test1b(i32 %a, i32 %b) {
+entry:
+  %cmp = icmp eq i32 %a, %b
+  %cond = zext i1 %cmp to i32
+  ret i32 %cond
+; CHECK-LABEL: test1b:
+; CHECK:       subs    r1, r0, r1
+; CHECK-NEXT:  movs    r0, #0
+; CHECK-NEXT:  subs    r0, r0, r1
+; CHECK-NEXT:  adcs    r0, r1
+}
+
+define i32 @test2a(i32 %a, i32 %b) {
+entry:
+  %cmp = icmp eq i32 %a, %b
+  %cond = zext i1 %cmp to i32
+  ret i32 %cond
+; CHECK-LABEL: test2a:
+; CHECK:       subs    r1, r0, r1
+; CHECK-NEXT:  movs    r0, #0
+; CHECK-NEXT:  subs    r0, r0, r1
+; CHECK-NEXT:  adcs    r0, r1
+}
+
+define i32 @test2b(i32 %a, i32 %b) {
+entry:
+  %cmp = icmp ne i32 %a, %b
+  %cond = zext i1 %cmp to i32
+  ret i32 %cond
+; CHECK-LABEL: test2b:
+; CHECK:       subs    r0, r0, r1
+; CHECK-NEXT:  subs    r1, r0, #1
+; CHECK-NEXT:  sbcs    r0, r1
+}
+
+define i32 @test3a(i32 %a, i32 %b) {
+entry:
+  %cmp = icmp eq i32 %a, %b
+  %cond = select i1 %cmp, i32 0, i32 4
+  ret i32 %cond
+; CHECK-LABEL: test3a:
+; CHECK:       subs    r0, r0, r1
+; CHECK-NEXT:  subs    r1, r0, #1
+; CHECK-NEXT:  sbcs    r0, r1
+; CHECK-NEXT:  lsls    r0, r0, #2
+}
+
+; This one hasn't changed actually.
+define i32 @test3b(i32 %a, i32 %b) {
+entry:
+  %cmp = icmp eq i32 %a, %b
+  %cond = select i1 %cmp, i32 4, i32 0
+  ret i32 %cond
+; CHECK-LABEL: test3b:
+; CHECK:       mov     r2, r0
+; CHECK-NEXT:  movs    r0, #4
+; CHECK-NEXT:  movs    r3, #0
+; CHECK-NEXT:  cmp     r2, r1
+; CHECK-NEXT:  beq     .[[BRANCH:[A-Z0-9_]+]]
+; CHECK:       mov     r0, r3
+; CHECK:       .[[BRANCH]]:
+}
+
+; This one hasn't changed actually.
+define i32 @test4a(i32 %a, i32 %b) {
+entry:
+  %cmp = icmp ne i32 %a, %b
+  %cond = select i1 %cmp, i32 0, i32 4
+  ret i32 %cond
+; CHECK-LABEL: test4a:
+; CHECK:       mov     r2, r0
+; CHECK-NEXT:  movs    r0, #0
+; CHECK-NEXT:  movs    r3, #4
+; CHECK-NEXT:  cmp     r2, r1
+; CHECK-NEXT:  bne     .[[BRANCH:[A-Z0-9_]+]]
+; CHECK:       mov     r0, r3
+; CHECK:       .[[BRANCH]]:
+}
+
+define i32 @test4b(i32 %a, i32 %b) {
+entry:
+  %cmp = icmp ne i32 %a, %b
+  %cond = select i1 %cmp, i32 4, i32 0
+  ret i32 %cond
+; CHECK-LABEL: test4b:
+; CHECK:       subs  r0, r0, r1
+; CHECK-NEXT:  subs  r1, r0, #1
+; CHECK-NEXT:  sbcs  r0, r1
+; CHECK-NEXT:  lsls  r0, r0, #2
+}
+

test/CodeGen/Thumb/constants.ll

 
 ; CHECK-T1-LABEL: @mov_and_add2
 ; CHECK-T2-LABEL: @mov_and_add2
 ; CHECK-T1: ldr r0,
 ; CHECK-T2: movw r0, #511
 define i32 @mov_and_add2() {
   ret i32 511
 }
+
+; CHECK-T1-LABEL: @test64
+; CHECK-T2-LABEL: @test64
+; CHECK-T1: movs r0, #15
+; CHECK-T1-NEXT:  mvns r0, r0
+; CHECK-T2: subs.w r0, r{{[0-9]+}}, #15
+; CHECK-T2-NEXT: sbc r1, r{{[0-9]+}}, #0
+define i32 @test64() {
+entry:
+  tail call void @fn1(i64 -1)
+  tail call void @fn1(i64 -1)
+  tail call void @fn1(i64 -16)
+  ret i32 0
+}
+declare void @fn1(i64) ;

test/CodeGen/Thumb/long-setcc.ll

define i1 @t2(i64 %x) {
   ret i1 %tmp
 }
 
 define i1 @t3(i32 %x) {
   %tmp = icmp ugt i32 %x, -1
   ret i1 %tmp
 }
 
-; CHECK: cmp
 ; CHECK-NOT: cmp
 
-
-

test/CodeGen/Thumb2/float-cmp.ll

 }
 define i1 @cmp_f_ord(float %a, float %b) {
 ; CHECK-LABEL: cmp_f_ord:
 ; NONE: bl __aeabi_fcmpun
 ; HARD: vcmpe.f32
 ; HARD: movvc r0, #1
   %1 = fcmp ord float %a, %b
   ret i1 %1
-}define i1 @cmp_f_ueq(float %a, float %b) {
+}
+define i1 @cmp_f_ueq(float %a, float %b) {
 ; CHECK-LABEL: cmp_f_ueq:
 ; NONE: bl __aeabi_fcmpeq
 ; NONE: bl __aeabi_fcmpun
 ; HARD: vcmp.f32
 ; HARD: moveq r0, #1
 ; HARD: movvs r0, #1
   %1 = fcmp ueq float %a, %b
   ret i1 %1
 }
 define i1 @cmp_f_ugt(float %a, float %b) {
 ; CHECK-LABEL: cmp_f_ugt:
 ; NONE: bl __aeabi_fcmple
-; NONE: cmp r0, #0
-; NONE-NEXT: it eq
+; NONE-NEXT: clz r0, r0
+; NONE-NEXT: lsrs r0, r0, #5
 ; HARD: vcmpe.f32
 ; HARD: movhi r0, #1
   %1 = fcmp ugt float %a, %b
   ret i1 %1
 }
 define i1 @cmp_f_uge(float %a, float %b) {
 ; CHECK-LABEL: cmp_f_uge:
 ; NONE: bl __aeabi_fcmplt
-; NONE: cmp r0, #0
-; NONE-NEXT: it eq
+; NONE-NEXT: clz r0, r0
+; NONE-NEXT: lsrs r0, r0, #5
 ; HARD: vcmpe.f32
 ; HARD: movpl r0, #1
   %1 = fcmp uge float %a, %b
   ret i1 %1
 }
 define i1 @cmp_f_ult(float %a, float %b) {
 ; CHECK-LABEL: cmp_f_ult:
 ; NONE: bl __aeabi_fcmpge
-; NONE: cmp r0, #0
-; NONE-NEXT: it eq
+; NONE-NEXT: clz r0, r0
+; NONE-NEXT: lsrs r0, r0, #5
 ; HARD: vcmpe.f32
 ; HARD: movlt r0, #1
   %1 = fcmp ult float %a, %b
   ret i1 %1
 }
 define i1 @cmp_f_ule(float %a, float %b) {
 ; CHECK-LABEL: cmp_f_ule:
 ; NONE: bl __aeabi_fcmpgt
-; NONE: cmp r0, #0
-; NONE-NEXT: it eq
+; NONE-NEXT: clz r0, r0
+; NONE-NEXT: lsrs r0, r0, #5
 ; HARD: vcmpe.f32
 ; HARD: movle r0, #1
   %1 = fcmp ule float %a, %b
   ret i1 %1
 }
 define i1 @cmp_f_une(float %a, float %b) {
 ; CHECK-LABEL: cmp_f_une:
 ; NONE: bl __aeabi_fcmpeq

test/CodeGen/Thumb2/thumb2-cmn.ll

 ; RUN: llc -mtriple=thumb-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - | FileCheck %s
 
 ; These tests could be improved by 'movs r0, #0' being rematerialized below the
 ; test as 'mov.w r0, #0'.
 
-define i1 @f1(i32 %a, i32 %b) {
+define i32 @f1(i32 %a, i32 %b) {
     %nb = sub i32 0, %b
     %tmp = icmp ne i32 %a, %nb
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f1:
 ; CHECK:  cmn {{.*}}, r1
 
-define i1 @f2(i32 %a, i32 %b) {
+define i32 @f2(i32 %a, i32 %b) {
     %nb = sub i32 0, %b
     %tmp = icmp ne i32 %nb, %a
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f2:
 ; CHECK:  cmn {{.*}}, r1
 
-define i1 @f3(i32 %a, i32 %b) {
+define i32 @f3(i32 %a, i32 %b) {
     %nb = sub i32 0, %b
     %tmp = icmp eq i32 %a, %nb
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f3:
 ; CHECK:  cmn {{.*}}, r1
 
-define i1 @f4(i32 %a, i32 %b) {
+define i32 @f4(i32 %a, i32 %b) {
     %nb = sub i32 0, %b
     %tmp = icmp eq i32 %nb, %a
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f4:
 ; CHECK:  cmn {{.*}}, r1
 
-define i1 @f5(i32 %a, i32 %b) {
+define i32 @f5(i32 %a, i32 %b) {
     %tmp = shl i32 %b, 5
     %nb = sub i32 0, %tmp
     %tmp1 = icmp eq i32 %nb, %a
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f5:
 ; CHECK:  cmn.w {{.*}}, r1, lsl #5
 
-define i1 @f6(i32 %a, i32 %b) {
+define i32 @f6(i32 %a, i32 %b) {
     %tmp = lshr i32 %b, 6
     %nb = sub i32 0, %tmp
     %tmp1 = icmp ne i32 %nb, %a
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f6:
 ; CHECK:  cmn.w {{.*}}, r1, lsr #6
 
-define i1 @f7(i32 %a, i32 %b) {
+define i32 @f7(i32 %a, i32 %b) {
     %tmp = ashr i32 %b, 7
     %nb = sub i32 0, %tmp
     %tmp1 = icmp eq i32 %a, %nb
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f7:
 ; CHECK:  cmn.w {{.*}}, r1, asr #7
 
-define i1 @f8(i32 %a, i32 %b) {
+define i32 @f8(i32 %a, i32 %b) {
     %l8 = shl i32 %a, 24
     %r8 = lshr i32 %a, 8
     %tmp = or i32 %l8, %r8
     %nb = sub i32 0, %tmp
     %tmp1 = icmp ne i32 %a, %nb
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f8:
 ; CHECK:  cmn.w {{.*}}, {{.*}}, ror #8
 
 
 define void @f9(i32 %a, i32 %b) nounwind optsize {
   tail call void asm sideeffect "cmn.w     r0, r1", ""() nounwind, !srcloc !0
   ret void
 }
 
 !0 = !{i32 81}
 
 ; CHECK-LABEL: f9:
 ; CHECK:  cmn.w r0, r1

test/CodeGen/Thumb2/thumb2-cmn2.ll

 ; RUN: llc -mtriple=thumb-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - | FileCheck %s
 
 ; -0x000000bb = 4294967109
-define i1 @f1(i32 %a) {
+define i32 @f1(i32 %a) {
 ; CHECK-LABEL: f1:
 ; CHECK: adds {{r.*}}, #187
     %tmp = icmp ne i32 %a, 4294967109
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 
 ; -0x00aa00aa = 4283826006
-define i1 @f2(i32 %a) {
+define i32 @f2(i32 %a) {
 ; CHECK-LABEL: f2:
 ; CHECK: cmn.w {{r.*}}, #11141290
     %tmp = icmp eq i32 %a, 4283826006
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 
 ; -0xcc00cc00 = 872363008
-define i1 @f3(i32 %a) {
+define i32 @f3(i32 %a) {
 ; CHECK-LABEL: f3:
 ; CHECK: cmn.w {{r.*}}, #-872363008
     %tmp = icmp ne i32 %a, 872363008
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 
 ; -0x00110000 = 4293853184
-define i1 @f4(i32 %a) {
+define i32 @f4(i32 %a) {
 ; CHECK-LABEL: f4:
 ; CHECK: cmn.w {{r.*}}, #1114112
     %tmp = icmp eq i32 %a, 4293853184
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }

test/CodeGen/Thumb2/thumb2-cmp.ll

 ; RUN: llc -mtriple=thumb-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - | FileCheck %s
 
 ; These tests would be improved by 'movs r0, #0' being rematerialized below the
 ; test as 'mov.w r0, #0'.
 
 ; 0x000000bb = 187
-define i1 @f1(i32 %a) {
+define i32 @f1(i32 %a) {
 ; CHECK-LABEL: f1:
 ; CHECK: cmp {{.*}}, #187
     %tmp = icmp ne i32 %a, 187
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 
 ; 0x00aa00aa = 11141290
-define i1 @f2(i32 %a) {
+define i32 @f2(i32 %a) {
 ; CHECK-LABEL: f2:
 ; CHECK: cmp.w {{.*}}, #11141290
     %tmp = icmp eq i32 %a, 11141290 
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 
 ; 0xcc00cc00 = 3422604288
-define i1 @f3(i32 %a) {
+define i32 @f3(i32 %a) {
 ; CHECK-LABEL: f3:
 ; CHECK: cmp.w {{.*}}, #-872363008
     %tmp = icmp ne i32 %a, 3422604288
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 
 ; 0xdddddddd = 3722304989
-define i1 @f4(i32 %a) {
+define i32 @f4(i32 %a) {
 ; CHECK-LABEL: f4:
 ; CHECK: cmp.w {{.*}}, #-572662307
     %tmp = icmp ne i32 %a, 3722304989
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 
 ; 0x00110000 = 1114112
-define i1 @f5(i32 %a) {
+define i32 @f5(i32 %a) {
 ; CHECK-LABEL: f5:
 ; CHECK: cmp.w {{.*}}, #1114112
     %tmp = icmp eq i32 %a, 1114112
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 
 ; Check that we don't do an invalid (a > b) --> !(a < b + 1) transform.
 ;
 ; CHECK-LABEL: f6:
 ; CHECK-NOT: cmp.w {{.*}}, #-2147483648
 ; CHECK: bx lr
 define i32 @f6(i32 %a) {
     %tmp = icmp sgt i32 %a, 2147483647
     br i1 %tmp, label %true, label %false
 true:
     ret i32 2
 false:
     ret i32 0
 }

test/CodeGen/Thumb2/thumb2-cmp2.ll

 ; RUN: llc -mtriple=thumb-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - | FileCheck %s
 
 ; These tests would be improved by 'movs r0, #0' being rematerialized below the
 ; test as 'mov.w r0, #0'.
 
-define i1 @f1(i32 %a, i32 %b) {
+define i32 @f1(i32 %a, i32 %b) {
 ; CHECK-LABEL: f1:
 ; CHECK: cmp {{.*}}, r1
     %tmp = icmp ne i32 %a, %b
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f2(i32 %a, i32 %b) {
+define i32 @f2(i32 %a, i32 %b) {
 ; CHECK-LABEL: f2:
 ; CHECK: cmp {{.*}}, r1
     %tmp = icmp eq i32 %a, %b
-    ret i1 %tmp
+    %ret = select i1 %tmp, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f6(i32 %a, i32 %b) {
+define i32 @f6(i32 %a, i32 %b) {
 ; CHECK-LABEL: f6:
 ; CHECK: cmp.w {{.*}}, r1, lsl #5
     %tmp = shl i32 %b, 5
     %tmp1 = icmp eq i32 %tmp, %a
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f7(i32 %a, i32 %b) {
+define i32 @f7(i32 %a, i32 %b) {
 ; CHECK-LABEL: f7:
 ; CHECK: cmp.w {{.*}}, r1, lsr #6
     %tmp = lshr i32 %b, 6
     %tmp1 = icmp ne i32 %tmp, %a
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f8(i32 %a, i32 %b) {
+define i32 @f8(i32 %a, i32 %b) {
 ; CHECK-LABEL: f8:
 ; CHECK: cmp.w {{.*}}, r1, asr #7
     %tmp = ashr i32 %b, 7
     %tmp1 = icmp eq i32 %a, %tmp
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f9(i32 %a, i32 %b) {
+define i32 @f9(i32 %a, i32 %b) {
 ; CHECK-LABEL: f9:
 ; CHECK: cmp.w {{.*}}, {{.*}}, ror #8
     %l8 = shl i32 %a, 24
     %r8 = lshr i32 %a, 8
     %tmp = or i32 %l8, %r8
     %tmp1 = icmp ne i32 %a, %tmp
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }

test/CodeGen/Thumb2/thumb2-teq.ll

 ; RUN: llc -mtriple=thumb-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - | FileCheck %s
 
 ; These tests would be improved by 'movs r0, #0' being rematerialized below the
 ; test as 'mov.w r0, #0'.
 
 ; 0x000000bb = 187
-define i1 @f2(i32 %a) {
+define i32 @f2(i32 %a) {
     %tmp = xor i32 %a, 187
     %tmp1 = icmp eq i32 0, %tmp
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f2:
 ; CHECK:  teq.w {{.*}}, #187
 
 ; 0x00aa00aa = 11141290
-define i1 @f3(i32 %a) {
+define i32 @f3(i32 %a) {
     %tmp = xor i32 %a, 11141290 
     %tmp1 = icmp eq i32 %tmp, 0
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f3:
 ; CHECK:  teq.w {{.*}}, #11141290
 
 ; 0xcc00cc00 = 3422604288
-define i1 @f6(i32 %a) {
+define i32 @f6(i32 %a) {
     %tmp = xor i32 %a, 3422604288
     %tmp1 = icmp eq i32 0, %tmp
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f6:
 ; CHECK:  teq.w {{.*}}, #-872363008
 
 ; 0xdddddddd = 3722304989
-define i1 @f7(i32 %a) {
+define i32 @f7(i32 %a) {
     %tmp = xor i32 %a, 3722304989
     %tmp1 = icmp eq i32 %tmp, 0
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f7:
 ; CHECK:  teq.w {{.*}}, #-572662307
 
+; 0x00110000 = 1114112
+define i32 @f10(i32 %a) {
+    %tmp = xor i32 %a, 1114112
+    %tmp1 = icmp eq i32 0, %tmp
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
+}
+; CHECK-LABEL: f10:
+; CHECK:  teq.w {{.*}}, #1114112
+
+; 0x000000bb = 187
+define i1 @f12(i32 %a) {
+    %tmp = xor i32 %a, 187
+    %tmp1 = icmp eq i32 0, %tmp
+    ret i1 %tmp1
+}
+; CHECK-LABEL: f12:
+; CHECK:      eor r0, r0, #187
+; CHECK-NEXT: clz r0, r0
+; CHECK-NEXT: lsrs r0, r0, #5
+
+; 0x00aa00aa = 11141290
+define i1 @f13(i32 %a) {
+    %tmp = xor i32 %a, 11141290 
+    %tmp1 = icmp eq i32 %tmp, 0
+    ret i1 %tmp1
+}
+; CHECK-LABEL: f13:
+; CHECK:      eor r0, r0, #11141290
+; CHECK-NEXT: clz r0, r0
+; CHECK-NEXT: lsrs r0, r0, #5
+
+; 0xcc00cc00 = 3422604288
+define i1 @f16(i32 %a) {
+    %tmp = xor i32 %a, 3422604288
+    %tmp1 = icmp eq i32 0, %tmp
+    ret i1 %tmp1
+}
+; CHECK-LABEL: f16:
+; CHECK:      eor r0, r0, #-872363008
+; CHECK-NEXT: clz r0, r0
+; CHECK-NEXT: lsrs r0, r0, #5
+
 ; 0xdddddddd = 3722304989
-define i1 @f8(i32 %a) {
+define i1 @f17(i32 %a) {
     %tmp = xor i32 %a, 3722304989
-    %tmp1 = icmp ne i32 0, %tmp
+    %tmp1 = icmp eq i32 %tmp, 0
     ret i1 %tmp1
 }
+; CHECK-LABEL: f17:
+; CHECK:      eor r0, r0, #-572662307
+; CHECK-NEXT: clz r0, r0
+; CHECK-NEXT: lsrs r0, r0, #5
 
 ; 0x00110000 = 1114112
-define i1 @f10(i32 %a) {
+define i1 @f18(i32 %a) {
     %tmp = xor i32 %a, 1114112
     %tmp1 = icmp eq i32 0, %tmp
     ret i1 %tmp1
 }
-; CHECK-LABEL: f10:
-; CHECK:  teq.w {{.*}}, #1114112
+; CHECK-LABEL: f18:
+; CHECK:      eor r0, r0, #1114112
+; CHECK-NEXT: clz r0, r0
+; CHECK-NEXT: lsrs r0, r0, #5
 

test/CodeGen/Thumb2/thumb2-teq2.ll

 ; RUN: llc -mtriple=thumb-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - | FileCheck %s
 
 ; These tests would be improved by 'movs r0, #0' being rematerialized below the
 ; tst as 'mov.w r0, #0'.
 
-define i1 @f2(i32 %a, i32 %b) {
+define i32 @f2(i32 %a, i32 %b) {
 ; CHECK: f2
 ; CHECK: teq.w {{.*}}, r1
     %tmp = xor i32 %a, %b
     %tmp1 = icmp eq i32 %tmp, 0
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f4(i32 %a, i32 %b) {
+define i32 @f4(i32 %a, i32 %b) {
 ; CHECK: f4
 ; CHECK: teq.w  {{.*}}, r1
     %tmp = xor i32 %a, %b
     %tmp1 = icmp eq i32 0, %tmp
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f6(i32 %a, i32 %b) {
+define i32 @f6(i32 %a, i32 %b) {
 ; CHECK: f6
 ; CHECK: teq.w  {{.*}}, r1, lsl #5
     %tmp = shl i32 %b, 5
     %tmp1 = xor i32 %a, %tmp
     %tmp2 = icmp eq i32 %tmp1, 0
-    ret i1 %tmp2
+    %ret = select i1 %tmp2, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f7(i32 %a, i32 %b) {
+define i32 @f7(i32 %a, i32 %b) {
 ; CHECK: f7
 ; CHECK: teq.w  {{.*}}, r1, lsr #6
     %tmp = lshr i32 %b, 6
     %tmp1 = xor i32 %a, %tmp
     %tmp2 = icmp eq i32 %tmp1, 0
-    ret i1 %tmp2
+    %ret = select i1 %tmp2, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f8(i32 %a, i32 %b) {
+define i32 @f8(i32 %a, i32 %b) {
 ; CHECK: f8
 ; CHECK: teq.w  {{.*}}, r1, asr #7
     %tmp = ashr i32 %b, 7
     %tmp1 = xor i32 %a, %tmp
     %tmp2 = icmp eq i32 %tmp1, 0
-    ret i1 %tmp2
+    %ret = select i1 %tmp2, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f9(i32 %a, i32 %b) {
+define i32 @f9(i32 %a, i32 %b) {
 ; CHECK: f9
 ; CHECK: teq.w  {{.*}}, {{.*}}, ror #8
     %l8 = shl i32 %a, 24
     %r8 = lshr i32 %a, 8
     %tmp = or i32 %l8, %r8
     %tmp1 = xor i32 %a, %tmp
     %tmp2 = icmp eq i32 %tmp1, 0
-    ret i1 %tmp2
+    %ret = select i1 %tmp2, i32 42, i32 24
+    ret i32 %ret
 }

test/CodeGen/Thumb2/thumb2-tst.ll

 ; RUN: llc -mtriple=thumb-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - | FileCheck %s
 
 ; These tests would be improved by 'movs r0, #0' being rematerialized below the
 ; tst as 'mov.w r0, #0'.
 
 ; 0x000000bb = 187
-define i1 @f2(i32 %a) {
+define i32 @f2(i32 %a) {
     %tmp = and i32 %a, 187
     %tmp1 = icmp eq i32 0, %tmp
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f2:
 ; CHECK:  tst.w {{.*}}, #187
 
 ; 0x00aa00aa = 11141290
-define i1 @f3(i32 %a) {
+define i32 @f3(i32 %a) {
     %tmp = and i32 %a, 11141290 
     %tmp1 = icmp eq i32 %tmp, 0
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f3:
 ; CHECK:  tst.w {{.*}}, #11141290
 
 ; 0xcc00cc00 = 3422604288
-define i1 @f6(i32 %a) {
+define i32 @f6(i32 %a) {
     %tmp = and i32 %a, 3422604288
     %tmp1 = icmp eq i32 0, %tmp
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f6:
 ; CHECK:  tst.w {{.*}}, #-872363008
 
 ; 0xdddddddd = 3722304989
-define i1 @f7(i32 %a) {
+define i32 @f7(i32 %a) {
     %tmp = and i32 %a, 3722304989
     %tmp1 = icmp eq i32 %tmp, 0
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f7:
 ; CHECK:  tst.w {{.*}}, #-572662307
 
 ; 0x00110000 = 1114112
-define i1 @f10(i32 %a) {
+define i32 @f10(i32 %a) {
     %tmp = and i32 %a, 1114112
     %tmp1 = icmp eq i32 0, %tmp
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 ; CHECK-LABEL: f10:
 ; CHECK:  tst.w {{.*}}, #1114112
+
+; 0x000000bb = 187
+define i1 @f12(i32 %a) {
+    %tmp = and i32 %a, 187
+    %tmp1 = icmp eq i32 0, %tmp
+    ret i1 %tmp1
+}
+; CHECK-LABEL: f12:
+; CHECK:    and r0, r0, #187
+; CHECK-NEXT: clz r0, r0
+; CHECK-NEXT: lsrs r0, r0, #5
+
+; 0x00aa00aa = 11141290
+define i1 @f13(i32 %a) {
+    %tmp = and i32 %a, 11141290
+    %tmp1 = icmp eq i32 %tmp, 0
+    ret i1 %tmp1
+}
+; CHECK-LABEL: f13:
+; CHECK:    and r0, r0, #11141290
+; CHECK-NEXT: clz r0, r0
+; CHECK-NEXT: lsrs r0, r0, #5
+
+; 0xcc00cc00 = 3422604288
+define i1 @f16(i32 %a) {
+    %tmp = and i32 %a, 3422604288
+    %tmp1 = icmp eq i32 0, %tmp
+    ret i1 %tmp1
+}
+; CHECK-LABEL: f16:
+; CHECK:    and r0, r0, #-872363008
+; CHECK-NEXT: clz r0, r0
+; CHECK-NEXT: lsrs r0, r0, #5
+
+; 0xdddddddd = 3722304989
+define i1 @f17(i32 %a) {
+    %tmp = and i32 %a, 3722304989
+    %tmp1 = icmp eq i32 %tmp, 0
+    ret i1 %tmp1
+}
+; CHECK-LABEL: f17:
+; CHECK:    bic r0, r0, #572662306
+; CHECK-NEXT: clz r0, r0
+; CHECK-NEXT: lsrs r0, r0, #5
+
+; 0x00110000 = 1114112
+define i1 @f18(i32 %a) {
+    %tmp = and i32 %a, 1114112
+    %tmp1 = icmp eq i32 0, %tmp
+    ret i1 %tmp1
+}
+; CHECK-LABEL: f18:
+; CHECK:    and r0, r0, #1114112
+; CHECK-NEXT: clz r0, r0
+; CHECK-NEXT: lsrs r0, r0, #5
+

test/CodeGen/Thumb2/thumb2-tst2.ll

 ; RUN: llc -mtriple=thumb-eabi -mcpu=arm1156t2-s -mattr=+thumb2 %s -o - | FileCheck %s
 
 ; These tests would be improved by 'movs r0, #0' being rematerialized below the
 ; tst as 'mov.w r0, #0'.
 
-define i1 @f2(i32 %a, i32 %b) {
+define i32 @f2(i32 %a, i32 %b) {
 ; CHECK-LABEL: f2:
 ; CHECK: tst {{.*}}, r1
     %tmp = and i32 %a, %b
     %tmp1 = icmp eq i32 %tmp, 0
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f4(i32 %a, i32 %b) {
+define i32 @f4(i32 %a, i32 %b) {
 ; CHECK-LABEL: f4:
 ; CHECK: tst {{.*}}, r1
     %tmp = and i32 %a, %b
     %tmp1 = icmp eq i32 0, %tmp
-    ret i1 %tmp1
+    %ret = select i1 %tmp1, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f6(i32 %a, i32 %b) {
+define i32 @f6(i32 %a, i32 %b) {
 ; CHECK-LABEL: f6:
 ; CHECK: tst.w {{.*}}, r1, lsl #5
     %tmp = shl i32 %b, 5
     %tmp1 = and i32 %a, %tmp
     %tmp2 = icmp eq i32 %tmp1, 0
-    ret i1 %tmp2
+    %ret = select i1 %tmp2, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f7(i32 %a, i32 %b) {
+define i32 @f7(i32 %a, i32 %b) {
 ; CHECK-LABEL: f7:
 ; CHECK: tst.w {{.*}}, r1, lsr #6
     %tmp = lshr i32 %b, 6
     %tmp1 = and i32 %a, %tmp
     %tmp2 = icmp eq i32 %tmp1, 0
-    ret i1 %tmp2
+    %ret = select i1 %tmp2, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f8(i32 %a, i32 %b) {
+define i32 @f8(i32 %a, i32 %b) {
 ; CHECK-LABEL: f8:
 ; CHECK: tst.w {{.*}}, r1, asr #7
     %tmp = ashr i32 %b, 7
     %tmp1 = and i32 %a, %tmp
     %tmp2 = icmp eq i32 %tmp1, 0
-    ret i1 %tmp2
+    %ret = select i1 %tmp2, i32 42, i32 24
+    ret i32 %ret
 }
 
-define i1 @f9(i32 %a, i32 %b) {
+define i32 @f9(i32 %a, i32 %b) {
 ; CHECK-LABEL: f9:
 ; CHECK: tst.w {{.*}}, {{.*}}, ror #8
     %l8 = shl i32 %a, 24
     %r8 = lshr i32 %a, 8
     %tmp = or i32 %l8, %r8
     %tmp1 = and i32 %a, %tmp
     %tmp2 = icmp eq i32 %tmp1, 0
-    ret i1 %tmp2
+    %ret = select i1 %tmp2, i32 42, i32 24
+    ret i32 %ret
 }