[AArch64] Support SETCCCARRY lowering

[fzhinkin]

Summary

Support SETCCCARRY lowering to SBCS instruction.

Related issue: https://github.com/llvm/llvm-project/issues/44629

[fzhinkin]

Updated tests

[fzhinkin]

llvm/test/CodeGen/AArch64/i128-cmp.ll

24

Also checking if it's possible to use setcccarry to lower setcc eq/ne. It seems like it may improve code on x86, ARM and AArch64, but I'd prefer fixing it separately.

[efriedma]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

417

Do we ever actually use SETCCCARRY for i32 on targets where i64 is legal?

llvm/test/CodeGen/AArch64/i128-cmp.ll

24

Not sure how you could use setcccarry for equality; the carry bit doesn't contain enough information. You need something more like the AArch64 ccmp.

[fzhinkin]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

417

SelectionDAGLegalize::LegalizeOp uses default case to handle SETCCCARRY and calls TLI.getOperationAction with SETCCCARRY's type (which is i32). I guess that this question never arose previously because for other targets there were either no 64-bit registers (ARM), or because various value types were supported (X86, M86k).

Not sure if it should be fixed along with this change or via separate commit.

llvm/test/CodeGen/AArch64/i128-cmp.ll

24

Correct me if I'm wrong, but SETCCCARRY is like a regular SETCC which takes carry flag into account when comparing values. So we can generate the carry flag by comparing/subtracting registers holding least significant bits of i128 and then use SETCCCARRY to compare registers holding most significant bits (exactly how it works for other CCs).

It would be something like:

cmp x2, x0
sbcs xzr, x3, x1
cset w0, eq

Of course, on AArch64 sbcs in this case could be replaced with ccmp, but by using SETCCCARRY we can also improve code generated for other targets (X86 in particular). For targets supporting conditional comparisons (ARM, AArch64) SETCCARRY could then be replaced with more appropriate instruction.

[efriedma]

LGTM

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

417

Oh, I see what you mean. The current way is fine; just wasn't really thinking about the output type.

llvm/test/CodeGen/AArch64/i128-cmp.ll

24

cmp x2, x0
sbcs xzr, x3, x1
cset w0, eq

Equality would generally mean x2 equals x0 and x3 equals x1. This throws away the equality bit of the first comparison, and adding the carry bit to the second comparison just means it returns "equal" in some cases where x3 and x1 aren't equal.

If you try to run some actual examples, I'm sure you'll see what I mean.

[fzhinkin]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

417

I'm not sure that it supposed to work this way, because while SelectionDAGLegalize::LegalizeOp looks at SETCCCARRY's output type, DAGTypeLegalizer::IntegerExpandSetCCOperands uses operand's type.

llvm/test/CodeGen/AArch64/i128-cmp.ll

24

Got it, thanks!

[efriedma]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

417

Oh, hmm. They should use the same type; that's probably worth fixing. (I don't think it causes a practical problem here, but it could in other cases.)

[fzhinkin]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

417

Sure, I'll fix it first then, thanks!

[fzhinkin]

Use operand's type to query SETCCCARRY legalize action

[fzhinkin]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

417

Added fix to this change as there were no other way to test its correctness.

[efriedma]

LGTM

llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

#endif
   case ISD::ATOMIC_STORE:
     Action = TLI.getOperationAction(Node->getOpcode(),
                                     Node->getOperand(2).getValueType());
     break;
   case ISD::SELECT_CC:
   case ISD::STRICT_FSETCC:
   case ISD::STRICT_FSETCCS:
   case ISD::SETCC:
+  case ISD::SETCCCARRY:
   case ISD::VP_SETCC:
   case ISD::BR_CC: {
     unsigned Opc = Node->getOpcode();
     unsigned CCOperand = Opc == ISD::SELECT_CC                         ? 4
                          : Opc == ISD::STRICT_FSETCC                   ? 3
                          : Opc == ISD::STRICT_FSETCCS                  ? 3
+                         : Opc == ISD::SETCCCARRY                      ? 3
                          : (Opc == ISD::SETCC || Opc == ISD::VP_SETCC) ? 2
                                                                        : 1;
     unsigned CompareOperand = Opc == ISD::BR_CC            ? 2
                               : Opc == ISD::STRICT_FSETCC  ? 1
                               : Opc == ISD::STRICT_FSETCCS ? 1
                                                            : 0;
     MVT OpVT = Node->getOperand(CompareOperand).getSimpleValueType();
     ISD::CondCode CCCode =

llvm/lib/Target/AArch64/AArch64ISelLowering.h

private:
   SDValue LowerDarwinGlobalTLSAddress(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerELFGlobalTLSAddress(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerELFTLSLocalExec(const GlobalValue *GV, SDValue ThreadBase,
                                const SDLoc &DL, SelectionDAG &DAG) const;
   SDValue LowerELFTLSDescCallSeq(SDValue SymAddr, const SDLoc &DL,
                                  SelectionDAG &DAG) const;
   SDValue LowerWindowsGlobalTLSAddress(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSETCC(SDValue Op, SelectionDAG &DAG) const;
+  SDValue LowerSETCCCARRY(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerBR_CC(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSELECT(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSELECT_CC(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSELECT_CC(ISD::CondCode CC, SDValue LHS, SDValue RHS,
                          SDValue TVal, SDValue FVal, const SDLoc &dl,
                          SelectionDAG &DAG) const;
   SDValue LowerJumpTable(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerBR_JT(SDValue Op, SelectionDAG &DAG) const;

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

AArch64TargetLowering::AArch64TargetLowering(const TargetMachine &TM,
   setOperationAction(ISD::SELECT_CC, MVT::i32, Custom);
   setOperationAction(ISD::SELECT_CC, MVT::i64, Custom);
   setOperationAction(ISD::SELECT_CC, MVT::f16, Custom);
   setOperationAction(ISD::SELECT_CC, MVT::bf16, Expand);
   setOperationAction(ISD::SELECT_CC, MVT::f32, Custom);
   setOperationAction(ISD::SELECT_CC, MVT::f64, Custom);
   setOperationAction(ISD::BR_JT, MVT::Other, Custom);
   setOperationAction(ISD::JumpTable, MVT::i64, Custom);
+  setOperationAction(ISD::SETCCCARRY, MVT::i64, Custom);
 
   setOperationAction(ISD::SHL_PARTS, MVT::i64, Custom);
   setOperationAction(ISD::SRA_PARTS, MVT::i64, Custom);
   setOperationAction(ISD::SRL_PARTS, MVT::i64, Custom);
 
   setOperationAction(ISD::FREM, MVT::f32, Expand);
   setOperationAction(ISD::FREM, MVT::f64, Expand);
   setOperationAction(ISD::FREM, MVT::f80, Expand);
SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
   case ISD::GlobalAddress:
     return LowerGlobalAddress(Op, DAG);
   case ISD::GlobalTLSAddress:
     return LowerGlobalTLSAddress(Op, DAG);
   case ISD::SETCC:
   case ISD::STRICT_FSETCC:
   case ISD::STRICT_FSETCCS:
     return LowerSETCC(Op, DAG);
+  case ISD::SETCCCARRY:
+    return LowerSETCCCARRY(Op, DAG);
   case ISD::BRCOND:
     return LowerBRCOND(Op, DAG);
   case ISD::BR_CC:
     return LowerBR_CC(Op, DAG);
   case ISD::SELECT:
     return LowerSELECT(Op, DAG);
   case ISD::SELECT_CC:
     return LowerSELECT_CC(Op, DAG);
SDValue CS1 =
         DAG.getNode(AArch64ISD::CSEL, dl, VT, TVal, FVal, CC1Val, Cmp);
 
     SDValue CC2Val = DAG.getConstant(CC2, dl, MVT::i32);
     Res = DAG.getNode(AArch64ISD::CSEL, dl, VT, TVal, CS1, CC2Val, Cmp);
   }
   return IsStrict ? DAG.getMergeValues({Res, Cmp.getValue(1)}, dl) : Res;
 }
 
+SDValue AArch64TargetLowering::LowerSETCCCARRY(SDValue Op,
+                                               SelectionDAG &DAG) const {
+
+  SDValue LHS = Op.getOperand(0);
+  SDValue RHS = Op.getOperand(1);
+  EVT VT = LHS.getValueType();
+  if (VT != MVT::i32 && VT != MVT::i64)
+    return SDValue();
+
+  SDLoc DL(Op);
+  SDValue Carry = Op.getOperand(2);
+  // SBCS uses a carry not a borrow so the carry flag should be inverted first.
+  SDValue InvCarry = valueToCarryFlag(Carry, DAG, true);
+  SDValue Cmp = DAG.getNode(AArch64ISD::SBCS, DL, DAG.getVTList(VT, MVT::Glue),
+                            LHS, RHS, InvCarry);
+
+  EVT OpVT = Op.getValueType();
+  SDValue TVal = DAG.getConstant(1, DL, OpVT);
+  SDValue FVal = DAG.getConstant(0, DL, OpVT);
+
+  ISD::CondCode Cond = cast<CondCodeSDNode>(Op.getOperand(3))->get();
+  ISD::CondCode CondInv = ISD::getSetCCInverse(Cond, VT);
+  SDValue CCVal =
+      DAG.getConstant(changeIntCCToAArch64CC(CondInv), DL, MVT::i32);
+  // Inputs are swapped because the condition is inverted. This will allow
+  // matching with a single CSINC instruction.
+  return DAG.getNode(AArch64ISD::CSEL, DL, OpVT, FVal, TVal, CCVal,
+                     Cmp.getValue(1));
+}
+
 SDValue AArch64TargetLowering::LowerSELECT_CC(ISD::CondCode CC, SDValue LHS,
                                               SDValue RHS, SDValue TVal,
                                               SDValue FVal, const SDLoc &dl,
                                               SelectionDAG &DAG) const {
   // Handle f128 first, because it will result in a comparison of some RTLIB
   // call result against zero.
   if (LHS.getValueType() == MVT::f128) {
     softenSetCCOperands(DAG, MVT::f128, LHS, RHS, CC, dl, LHS, RHS);

llvm/test/CodeGen/AArch64/arm64-atomic-128.ll

 }
 
 define void @fetch_and_min(i128* %p, i128 %bits) {
 ; CHECK-LABEL: fetch_and_min:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:  .LBB8_1: // %atomicrmw.start
 ; CHECK-NEXT:    // =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldaxp x9, x8, [x0]
-; CHECK-NEXT:    cmp x9, x2
-; CHECK-NEXT:    cset w10, ls
-; CHECK-NEXT:    cmp x8, x3
-; CHECK-NEXT:    cset w11, le
-; CHECK-NEXT:    csel w10, w10, w11, eq
-; CHECK-NEXT:    cmp w10, #0
-; CHECK-NEXT:    csel x10, x8, x3, ne
-; CHECK-NEXT:    csel x11, x9, x2, ne
+; CHECK-NEXT:    cmp x2, x9
+; CHECK-NEXT:    sbcs xzr, x3, x8
+; CHECK-NEXT:    csel x10, x8, x3, ge
+; CHECK-NEXT:    csel x11, x9, x2, ge
 ; CHECK-NEXT:    stlxp w12, x11, x10, [x0]
 ; CHECK-NEXT:    cbnz w12, .LBB8_1
 ; CHECK-NEXT:  // %bb.2: // %atomicrmw.end
 ; CHECK-NEXT:    adrp x10, :got:var
 ; CHECK-NEXT:    ldr x10, [x10, :got_lo12:var]
 ; CHECK-NEXT:    stp x9, x8, [x10]
 ; CHECK-NEXT:    ret
 
   %val = atomicrmw min i128* %p, i128 %bits seq_cst
   store i128 %val, i128* @var, align 16
   ret void
 }
 
 define void @fetch_and_max(i128* %p, i128 %bits) {
 ; CHECK-LABEL: fetch_and_max:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:  .LBB9_1: // %atomicrmw.start
 ; CHECK-NEXT:    // =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldaxp x9, x8, [x0]
-; CHECK-NEXT:    cmp x9, x2
-; CHECK-NEXT:    cset w10, hi
-; CHECK-NEXT:    cmp x8, x3
-; CHECK-NEXT:    cset w11, gt
-; CHECK-NEXT:    csel w10, w10, w11, eq
-; CHECK-NEXT:    cmp w10, #0
-; CHECK-NEXT:    csel x10, x8, x3, ne
-; CHECK-NEXT:    csel x11, x9, x2, ne
+; CHECK-NEXT:    cmp x2, x9
+; CHECK-NEXT:    sbcs xzr, x3, x8
+; CHECK-NEXT:    csel x10, x8, x3, lt
+; CHECK-NEXT:    csel x11, x9, x2, lt
 ; CHECK-NEXT:    stlxp w12, x11, x10, [x0]
 ; CHECK-NEXT:    cbnz w12, .LBB9_1
 ; CHECK-NEXT:  // %bb.2: // %atomicrmw.end
 ; CHECK-NEXT:    adrp x10, :got:var
 ; CHECK-NEXT:    ldr x10, [x10, :got_lo12:var]
 ; CHECK-NEXT:    stp x9, x8, [x10]
 ; CHECK-NEXT:    ret
 
   %val = atomicrmw max i128* %p, i128 %bits seq_cst
   store i128 %val, i128* @var, align 16
   ret void
 }
 
 define void @fetch_and_umin(i128* %p, i128 %bits) {
 ; CHECK-LABEL: fetch_and_umin:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:  .LBB10_1: // %atomicrmw.start
 ; CHECK-NEXT:    // =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldaxp x9, x8, [x0]
-; CHECK-NEXT:    cmp x9, x2
-; CHECK-NEXT:    cset w10, ls
-; CHECK-NEXT:    cmp x8, x3
-; CHECK-NEXT:    cset w11, ls
-; CHECK-NEXT:    csel w10, w10, w11, eq
-; CHECK-NEXT:    cmp w10, #0
-; CHECK-NEXT:    csel x10, x8, x3, ne
-; CHECK-NEXT:    csel x11, x9, x2, ne
+; CHECK-NEXT:    cmp x2, x9
+; CHECK-NEXT:    sbcs xzr, x3, x8
+; CHECK-NEXT:    csel x10, x8, x3, hs
+; CHECK-NEXT:    csel x11, x9, x2, hs
 ; CHECK-NEXT:    stlxp w12, x11, x10, [x0]
 ; CHECK-NEXT:    cbnz w12, .LBB10_1
 ; CHECK-NEXT:  // %bb.2: // %atomicrmw.end
 ; CHECK-NEXT:    adrp x10, :got:var
 ; CHECK-NEXT:    ldr x10, [x10, :got_lo12:var]
 ; CHECK-NEXT:    stp x9, x8, [x10]
 ; CHECK-NEXT:    ret
 
   %val = atomicrmw umin i128* %p, i128 %bits seq_cst
   store i128 %val, i128* @var, align 16
   ret void
 }
 
 define void @fetch_and_umax(i128* %p, i128 %bits) {
 ; CHECK-LABEL: fetch_and_umax:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:  .LBB11_1: // %atomicrmw.start
 ; CHECK-NEXT:    // =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldaxp x9, x8, [x0]
-; CHECK-NEXT:    cmp x9, x2
-; CHECK-NEXT:    cset w10, hi
-; CHECK-NEXT:    cmp x8, x3
-; CHECK-NEXT:    cset w11, hi
-; CHECK-NEXT:    csel w10, w10, w11, eq
-; CHECK-NEXT:    cmp w10, #0
-; CHECK-NEXT:    csel x10, x8, x3, ne
-; CHECK-NEXT:    csel x11, x9, x2, ne
+; CHECK-NEXT:    cmp x2, x9
+; CHECK-NEXT:    sbcs xzr, x3, x8
+; CHECK-NEXT:    csel x10, x8, x3, lo
+; CHECK-NEXT:    csel x11, x9, x2, lo
 ; CHECK-NEXT:    stlxp w12, x11, x10, [x0]
 ; CHECK-NEXT:    cbnz w12, .LBB11_1
 ; CHECK-NEXT:  // %bb.2: // %atomicrmw.end
 ; CHECK-NEXT:    adrp x10, :got:var
 ; CHECK-NEXT:    ldr x10, [x10, :got_lo12:var]
 ; CHECK-NEXT:    stp x9, x8, [x10]
 ; CHECK-NEXT:    ret
 

llvm/test/CodeGen/AArch64/fpclamptosat.ll

 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    bl __fixdfti
 ; CHECK-NEXT:    cmp x1, #1
 ; CHECK-NEXT:    csel x8, x0, xzr, lt
 ; CHECK-NEXT:    csinc x9, x1, xzr, lt
-; CHECK-NEXT:    cmp x8, #0
-; CHECK-NEXT:    cset w10, ne
-; CHECK-NEXT:    cmp x9, #0
-; CHECK-NEXT:    cset w9, gt
-; CHECK-NEXT:    csel w9, w10, w9, eq
-; CHECK-NEXT:    cmp w9, #0
-; CHECK-NEXT:    csel x0, x8, xzr, ne
+; CHECK-NEXT:    cmp xzr, x8
+; CHECK-NEXT:    ngcs xzr, x9
+; CHECK-NEXT:    csel x0, x8, xzr, lt
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
 entry:
   %conv = fptosi double %x to i128
   %0 = icmp slt i128 %conv, 18446744073709551616
   %spec.store.select = select i1 %0, i128 %conv, i128 18446744073709551616
   %1 = icmp sgt i128 %spec.store.select, 0
   %spec.store.select7 = select i1 %1, i128 %spec.store.select, i128 0

 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    bl __fixsfti
 ; CHECK-NEXT:    cmp x1, #1
 ; CHECK-NEXT:    csel x8, x0, xzr, lt
 ; CHECK-NEXT:    csinc x9, x1, xzr, lt
-; CHECK-NEXT:    cmp x8, #0
-; CHECK-NEXT:    cset w10, ne
-; CHECK-NEXT:    cmp x9, #0
-; CHECK-NEXT:    cset w9, gt
-; CHECK-NEXT:    csel w9, w10, w9, eq
-; CHECK-NEXT:    cmp w9, #0
-; CHECK-NEXT:    csel x0, x8, xzr, ne
+; CHECK-NEXT:    cmp xzr, x8
+; CHECK-NEXT:    ngcs xzr, x9
+; CHECK-NEXT:    csel x0, x8, xzr, lt
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
 entry:
   %conv = fptosi float %x to i128
   %0 = icmp slt i128 %conv, 18446744073709551616
   %spec.store.select = select i1 %0, i128 %conv, i128 18446744073709551616
   %1 = icmp sgt i128 %spec.store.select, 0
   %spec.store.select7 = select i1 %1, i128 %spec.store.select, i128 0

 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    bl __fixhfti
 ; CHECK-NEXT:    cmp x1, #1
 ; CHECK-NEXT:    csel x8, x0, xzr, lt
 ; CHECK-NEXT:    csinc x9, x1, xzr, lt
-; CHECK-NEXT:    cmp x8, #0
-; CHECK-NEXT:    cset w10, ne
-; CHECK-NEXT:    cmp x9, #0
-; CHECK-NEXT:    cset w9, gt
-; CHECK-NEXT:    csel w9, w10, w9, eq
-; CHECK-NEXT:    cmp w9, #0
-; CHECK-NEXT:    csel x0, x8, xzr, ne
+; CHECK-NEXT:    cmp xzr, x8
+; CHECK-NEXT:    ngcs xzr, x9
+; CHECK-NEXT:    csel x0, x8, xzr, lt
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
 entry:
   %conv = fptosi half %x to i128
   %0 = icmp slt i128 %conv, 18446744073709551616
   %spec.store.select = select i1 %0, i128 %conv, i128 18446744073709551616
   %1 = icmp sgt i128 %spec.store.select, 0
   %spec.store.select7 = select i1 %1, i128 %spec.store.select, i128 0

llvm/test/CodeGen/AArch64/fpclamptosat_vec.ll

 ; CHECK-NEXT:    sub sp, sp, #48
 ; CHECK-NEXT:    .cfi_def_cfa_offset 48
 ; CHECK-NEXT:    str x30, [sp, #16] // 8-byte Folded Spill
 ; CHECK-NEXT:    stp x20, x19, [sp, #32] // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w19, -8
 ; CHECK-NEXT:    .cfi_offset w20, -16
 ; CHECK-NEXT:    .cfi_offset w30, -32
 ; CHECK-NEXT:    str q0, [sp] // 16-byte Folded Spill
-; CHECK-NEXT:    mov d0, v0.d[1]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    bl __fixdfti
 ; CHECK-NEXT:    ldr q0, [sp] // 16-byte Folded Reload
 ; CHECK-NEXT:    mov x19, x0
 ; CHECK-NEXT:    mov x20, x1
-; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; CHECK-NEXT:    mov d0, v0.d[1]
 ; CHECK-NEXT:    bl __fixdfti
 ; CHECK-NEXT:    cmp x1, #1
 ; CHECK-NEXT:    ldr x30, [sp, #16] // 8-byte Folded Reload
 ; CHECK-NEXT:    csel x8, x0, xzr, lt
 ; CHECK-NEXT:    csinc x9, x1, xzr, lt
 ; CHECK-NEXT:    cmp x20, #1
 ; CHECK-NEXT:    csel x10, x19, xzr, lt
 ; CHECK-NEXT:    csinc x11, x20, xzr, lt
-; CHECK-NEXT:    cmp x10, #0
-; CHECK-NEXT:    cset w12, ne
-; CHECK-NEXT:    cmp x11, #0
-; CHECK-NEXT:    cset w11, gt
-; CHECK-NEXT:    csel w11, w12, w11, eq
-; CHECK-NEXT:    cmp x8, #0
-; CHECK-NEXT:    cset w12, ne
-; CHECK-NEXT:    cmp x9, #0
-; CHECK-NEXT:    cset w9, gt
-; CHECK-NEXT:    csel w9, w12, w9, eq
-; CHECK-NEXT:    cmp w9, #0
-; CHECK-NEXT:    csel x8, x8, xzr, ne
-; CHECK-NEXT:    cmp w11, #0
-; CHECK-NEXT:    csel x9, x10, xzr, ne
+; CHECK-NEXT:    cmp xzr, x10
+; CHECK-NEXT:    ngcs xzr, x11
+; CHECK-NEXT:    csel x10, x10, xzr, lt
+; CHECK-NEXT:    cmp xzr, x8
+; CHECK-NEXT:    ngcs xzr, x9
+; CHECK-NEXT:    csel x8, x8, xzr, lt
 ; CHECK-NEXT:    ldp x20, x19, [sp, #32] // 16-byte Folded Reload
-; CHECK-NEXT:    fmov d0, x8
-; CHECK-NEXT:    fmov d1, x9
+; CHECK-NEXT:    fmov d0, x10
+; CHECK-NEXT:    fmov d1, x8
 ; CHECK-NEXT:    mov v0.d[1], v1.d[0]
 ; CHECK-NEXT:    add sp, sp, #48
 ; CHECK-NEXT:    ret
 entry:
   %conv = fptosi <2 x double> %x to <2 x i128>
   %0 = icmp slt <2 x i128> %conv, <i128 18446744073709551616, i128 18446744073709551616>
   %spec.store.select = select <2 x i1> %0, <2 x i128> %conv, <2 x i128> <i128 18446744073709551616, i128 18446744073709551616>
   %1 = icmp sgt <2 x i128> %spec.store.select, zeroinitializer

 ; CHECK-NEXT:    .cfi_def_cfa_offset 48
 ; CHECK-NEXT:    str x30, [sp, #16] // 8-byte Folded Spill
 ; CHECK-NEXT:    stp x20, x19, [sp, #32] // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w19, -8
 ; CHECK-NEXT:    .cfi_offset w20, -16
 ; CHECK-NEXT:    .cfi_offset w30, -32
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    str q0, [sp] // 16-byte Folded Spill
-; CHECK-NEXT:    mov s0, v0.s[1]
+; CHECK-NEXT:    // kill: def $s0 killed $s0 killed $q0
 ; CHECK-NEXT:    bl __fixsfti
 ; CHECK-NEXT:    ldr q0, [sp] // 16-byte Folded Reload
 ; CHECK-NEXT:    mov x19, x0
 ; CHECK-NEXT:    mov x20, x1
-; CHECK-NEXT:    // kill: def $s0 killed $s0 killed $q0
+; CHECK-NEXT:    mov s0, v0.s[1]
 ; CHECK-NEXT:    bl __fixsfti
 ; CHECK-NEXT:    cmp x1, #1
 ; CHECK-NEXT:    ldr x30, [sp, #16] // 8-byte Folded Reload
 ; CHECK-NEXT:    csinc x8, x1, xzr, lt
 ; CHECK-NEXT:    csel x9, x0, xzr, lt
 ; CHECK-NEXT:    cmp x20, #1
 ; CHECK-NEXT:    csel x10, x19, xzr, lt
 ; CHECK-NEXT:    csinc x11, x20, xzr, lt
-; CHECK-NEXT:    cmp x10, #0
-; CHECK-NEXT:    cset w12, ne
-; CHECK-NEXT:    cmp x11, #0
-; CHECK-NEXT:    cset w11, gt
-; CHECK-NEXT:    csel w11, w12, w11, eq
-; CHECK-NEXT:    cmp x9, #0
-; CHECK-NEXT:    cset w12, ne
-; CHECK-NEXT:    cmp x8, #0
-; CHECK-NEXT:    cset w8, gt
-; CHECK-NEXT:    csel w8, w12, w8, eq
-; CHECK-NEXT:    cmp w8, #0
-; CHECK-NEXT:    csel x8, x9, xzr, ne
-; CHECK-NEXT:    cmp w11, #0
-; CHECK-NEXT:    csel x9, x10, xzr, ne
+; CHECK-NEXT:    cmp xzr, x10
+; CHECK-NEXT:    ngcs xzr, x11
+; CHECK-NEXT:    csel x10, x10, xzr, lt
+; CHECK-NEXT:    cmp xzr, x9
+; CHECK-NEXT:    ngcs xzr, x8
+; CHECK-NEXT:    csel x8, x9, xzr, lt
 ; CHECK-NEXT:    ldp x20, x19, [sp, #32] // 16-byte Folded Reload
-; CHECK-NEXT:    fmov d0, x8
-; CHECK-NEXT:    fmov d1, x9
+; CHECK-NEXT:    fmov d0, x10
+; CHECK-NEXT:    fmov d1, x8
 ; CHECK-NEXT:    mov v0.d[1], v1.d[0]
 ; CHECK-NEXT:    add sp, sp, #48
 ; CHECK-NEXT:    ret
 entry:
   %conv = fptosi <2 x float> %x to <2 x i128>
   %0 = icmp slt <2 x i128> %conv, <i128 18446744073709551616, i128 18446744073709551616>
   %spec.store.select = select <2 x i1> %0, <2 x i128> %conv, <2 x i128> <i128 18446744073709551616, i128 18446744073709551616>
   %1 = icmp sgt <2 x i128> %spec.store.select, zeroinitializer

 ; CHECK-NEXT:    .cfi_def_cfa_offset 48
 ; CHECK-NEXT:    str x30, [sp, #16] // 8-byte Folded Spill
 ; CHECK-NEXT:    stp x20, x19, [sp, #32] // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w19, -8
 ; CHECK-NEXT:    .cfi_offset w20, -16
 ; CHECK-NEXT:    .cfi_offset w30, -32
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    str q0, [sp] // 16-byte Folded Spill
-; CHECK-NEXT:    mov h0, v0.h[1]
+; CHECK-NEXT:    // kill: def $h0 killed $h0 killed $q0
 ; CHECK-NEXT:    bl __fixhfti
 ; CHECK-NEXT:    ldr q0, [sp] // 16-byte Folded Reload
 ; CHECK-NEXT:    mov x19, x0
 ; CHECK-NEXT:    mov x20, x1
-; CHECK-NEXT:    // kill: def $h0 killed $h0 killed $q0
+; CHECK-NEXT:    mov h0, v0.h[1]
 ; CHECK-NEXT:    bl __fixhfti
 ; CHECK-NEXT:    cmp x1, #1
 ; CHECK-NEXT:    ldr x30, [sp, #16] // 8-byte Folded Reload
 ; CHECK-NEXT:    csinc x8, x1, xzr, lt
 ; CHECK-NEXT:    csel x9, x0, xzr, lt
 ; CHECK-NEXT:    cmp x20, #1
 ; CHECK-NEXT:    csel x10, x19, xzr, lt
 ; CHECK-NEXT:    csinc x11, x20, xzr, lt
-; CHECK-NEXT:    cmp x10, #0
-; CHECK-NEXT:    cset w12, ne
-; CHECK-NEXT:    cmp x11, #0
-; CHECK-NEXT:    cset w11, gt
-; CHECK-NEXT:    csel w11, w12, w11, eq
-; CHECK-NEXT:    cmp x9, #0
-; CHECK-NEXT:    cset w12, ne
-; CHECK-NEXT:    cmp x8, #0
-; CHECK-NEXT:    cset w8, gt
-; CHECK-NEXT:    csel w8, w12, w8, eq
-; CHECK-NEXT:    cmp w8, #0
-; CHECK-NEXT:    csel x8, x9, xzr, ne
-; CHECK-NEXT:    cmp w11, #0
-; CHECK-NEXT:    csel x9, x10, xzr, ne
+; CHECK-NEXT:    cmp xzr, x10
+; CHECK-NEXT:    ngcs xzr, x11
+; CHECK-NEXT:    csel x10, x10, xzr, lt
+; CHECK-NEXT:    cmp xzr, x9
+; CHECK-NEXT:    ngcs xzr, x8
+; CHECK-NEXT:    csel x8, x9, xzr, lt
 ; CHECK-NEXT:    ldp x20, x19, [sp, #32] // 16-byte Folded Reload
-; CHECK-NEXT:    fmov d0, x8
-; CHECK-NEXT:    fmov d1, x9
+; CHECK-NEXT:    fmov d0, x10
+; CHECK-NEXT:    fmov d1, x8
 ; CHECK-NEXT:    mov v0.d[1], v1.d[0]
 ; CHECK-NEXT:    add sp, sp, #48
 ; CHECK-NEXT:    ret
 entry:
   %conv = fptosi <2 x half> %x to <2 x i128>
   %0 = icmp slt <2 x i128> %conv, <i128 18446744073709551616, i128 18446744073709551616>
   %spec.store.select = select <2 x i1> %0, <2 x i128> %conv, <2 x i128> <i128 18446744073709551616, i128 18446744073709551616>
   %1 = icmp sgt <2 x i128> %spec.store.select, zeroinitializer

llvm/test/CodeGen/AArch64/i128-cmp.ll

; CHECK-NEXT:    ret
     ret i1 %cmp
 }
 
 define i1 @cmp_i128_ne(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_ne:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    eor x8, x1, x3
 ; CHECK-NEXT:    eor x9, x0, x2
 ; CHECK-NEXT:    orr x8, x9, x8
 ; CHECK-NEXT:    cmp x8, #0
 ; CHECK-NEXT:    cset w0, ne
 ; CHECK-NEXT:    ret
     %cmp = icmp ne i128 %a, %b
     ret i1 %cmp
 }
 
 define i1 @cmp_i128_ugt(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_ugt:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, hi
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, hi
-; CHECK-NEXT:    csel w0, w8, w9, eq
+; CHECK-NEXT:    cmp x2, x0
+; CHECK-NEXT:    sbcs xzr, x3, x1
+; CHECK-NEXT:    cset w0, lo
 ; CHECK-NEXT:    ret
     %cmp = icmp ugt i128 %a, %b
     ret i1 %cmp
 }
 
 define i1 @cmp_i128_uge(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_uge:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, hs
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, hs
-; CHECK-NEXT:    csel w0, w8, w9, eq
+; CHECK-NEXT:    sbcs xzr, x1, x3
+; CHECK-NEXT:    cset w0, hs
 ; CHECK-NEXT:    ret
     %cmp = icmp uge i128 %a, %b
     ret i1 %cmp
 }
 
 define i1 @cmp_i128_ult(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_ult:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, lo
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, lo
-; CHECK-NEXT:    csel w0, w8, w9, eq
+; CHECK-NEXT:    sbcs xzr, x1, x3
+; CHECK-NEXT:    cset w0, lo
 ; CHECK-NEXT:    ret
     %cmp = icmp ult i128 %a, %b
     ret i1 %cmp
 }
 
 define i1 @cmp_i128_ule(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_ule:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, ls
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, ls
-; CHECK-NEXT:    csel w0, w8, w9, eq
+; CHECK-NEXT:    cmp x2, x0
+; CHECK-NEXT:    sbcs xzr, x3, x1
+; CHECK-NEXT:    cset w0, hs
 ; CHECK-NEXT:    ret
     %cmp = icmp ule i128 %a, %b
     ret i1 %cmp
 }
 
 define i1 @cmp_i128_sgt(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_sgt:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, hi
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, gt
-; CHECK-NEXT:    csel w0, w8, w9, eq
+; CHECK-NEXT:    cmp x2, x0
+; CHECK-NEXT:    sbcs xzr, x3, x1
+; CHECK-NEXT:    cset w0, lt
 ; CHECK-NEXT:    ret
     %cmp = icmp sgt i128 %a, %b
     ret i1 %cmp
 }
 
 define i1 @cmp_i128_sge(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_sge:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, hs
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, ge
-; CHECK-NEXT:    csel w0, w8, w9, eq
+; CHECK-NEXT:    sbcs xzr, x1, x3
+; CHECK-NEXT:    cset w0, ge
 ; CHECK-NEXT:    ret
     %cmp = icmp sge i128 %a, %b
     ret i1 %cmp
 }
 
 define i1 @cmp_i128_slt(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_slt:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, lo
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, lt
-; CHECK-NEXT:    csel w0, w8, w9, eq
+; CHECK-NEXT:    sbcs xzr, x1, x3
+; CHECK-NEXT:    cset w0, lt
 ; CHECK-NEXT:    ret
     %cmp = icmp slt i128 %a, %b
     ret i1 %cmp
 }
 
 define i1 @cmp_i128_sle(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_sle:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, ls
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, le
-; CHECK-NEXT:    csel w0, w8, w9, eq
+; CHECK-NEXT:    cmp x2, x0
+; CHECK-NEXT:    sbcs xzr, x3, x1
+; CHECK-NEXT:    cset w0, ge
 ; CHECK-NEXT:    ret
     %cmp = icmp sle i128 %a, %b
     ret i1 %cmp
 }
 
 define void @br_on_cmp_i128_eq(i128 %a, i128 %b) nounwind {
 ; CHECK-LABEL: br_on_cmp_i128_eq:
 ; CHECK:       // %bb.0:
call:
     br label %exit
 exit:
     ret void
 }
 
 define void @br_on_cmp_i128_ugt(i128 %a, i128 %b) nounwind {
 ; CHECK-LABEL: br_on_cmp_i128_ugt:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, ls
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, ls
-; CHECK-NEXT:    csel w8, w8, w9, eq
-; CHECK-NEXT:    tbnz w8, #0, .LBB12_2
+; CHECK-NEXT:    cmp x2, x0
+; CHECK-NEXT:    sbcs xzr, x3, x1
+; CHECK-NEXT:    b.hs .LBB12_2
 ; CHECK-NEXT:  // %bb.1: // %call
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    bl call
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:  .LBB12_2: // %exit
 ; CHECK-NEXT:    ret
     %cmp = icmp ugt i128 %a, %b
     br i1 %cmp, label %call, label %exit
 call:
     call void @call()
     br label %exit
 exit:
     ret void
 }
 
 define void @br_on_cmp_i128_uge(i128 %a, i128 %b) nounwind {
 ; CHECK-LABEL: br_on_cmp_i128_uge:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, lo
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, lo
-; CHECK-NEXT:    csel w8, w8, w9, eq
-; CHECK-NEXT:    tbnz w8, #0, .LBB13_2
+; CHECK-NEXT:    sbcs xzr, x1, x3
+; CHECK-NEXT:    b.lo .LBB13_2
 ; CHECK-NEXT:  // %bb.1: // %call
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    bl call
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:  .LBB13_2: // %exit
 ; CHECK-NEXT:    ret
     %cmp = icmp uge i128 %a, %b
     br i1 %cmp, label %call, label %exit
 call:
     call void @call()
     br label %exit
 exit:
     ret void
 }
 
 define void @br_on_cmp_i128_ult(i128 %a, i128 %b) nounwind {
 ; CHECK-LABEL: br_on_cmp_i128_ult:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, hs
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, hs
-; CHECK-NEXT:    csel w8, w8, w9, eq
-; CHECK-NEXT:    tbnz w8, #0, .LBB14_2
+; CHECK-NEXT:    sbcs xzr, x1, x3
+; CHECK-NEXT:    b.hs .LBB14_2
 ; CHECK-NEXT:  // %bb.1: // %call
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    bl call
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:  .LBB14_2: // %exit
 ; CHECK-NEXT:    ret
     %cmp = icmp ult i128 %a, %b
     br i1 %cmp, label %call, label %exit
 call:
     call void @call()
     br label %exit
 exit:
     ret void
 }
 
 define void @br_on_cmp_i128_ule(i128 %a, i128 %b) nounwind {
 ; CHECK-LABEL: br_on_cmp_i128_ule:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, hi
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, hi
-; CHECK-NEXT:    csel w8, w8, w9, eq
-; CHECK-NEXT:    tbnz w8, #0, .LBB15_2
+; CHECK-NEXT:    cmp x2, x0
+; CHECK-NEXT:    sbcs xzr, x3, x1
+; CHECK-NEXT:    b.lo .LBB15_2
 ; CHECK-NEXT:  // %bb.1: // %call
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    bl call
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:  .LBB15_2: // %exit
 ; CHECK-NEXT:    ret
     %cmp = icmp ule i128 %a, %b
     br i1 %cmp, label %call, label %exit
 call:
     call void @call()
     br label %exit
 exit:
     ret void
 }
 
 define void @br_on_cmp_i128_sgt(i128 %a, i128 %b) nounwind {
 ; CHECK-LABEL: br_on_cmp_i128_sgt:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, ls
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, le
-; CHECK-NEXT:    csel w8, w8, w9, eq
-; CHECK-NEXT:    tbnz w8, #0, .LBB16_2
+; CHECK-NEXT:    cmp x2, x0
+; CHECK-NEXT:    sbcs xzr, x3, x1
+; CHECK-NEXT:    b.ge .LBB16_2
 ; CHECK-NEXT:  // %bb.1: // %call
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    bl call
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:  .LBB16_2: // %exit
 ; CHECK-NEXT:    ret
     %cmp = icmp sgt i128 %a, %b
     br i1 %cmp, label %call, label %exit
 call:
     call void @call()
     br label %exit
 exit:
     ret void
 }
 
 define void @br_on_cmp_i128_sge(i128 %a, i128 %b) nounwind {
 ; CHECK-LABEL: br_on_cmp_i128_sge:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, lo
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, lt
-; CHECK-NEXT:    csel w8, w8, w9, eq
-; CHECK-NEXT:    tbnz w8, #0, .LBB17_2
+; CHECK-NEXT:    sbcs xzr, x1, x3
+; CHECK-NEXT:    b.lt .LBB17_2
 ; CHECK-NEXT:  // %bb.1: // %call
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    bl call
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:  .LBB17_2: // %exit
 ; CHECK-NEXT:    ret
     %cmp = icmp sge i128 %a, %b
     br i1 %cmp, label %call, label %exit
 call:
     call void @call()
     br label %exit
 exit:
     ret void
 }
 
 define void @br_on_cmp_i128_slt(i128 %a, i128 %b) nounwind {
 ; CHECK-LABEL: br_on_cmp_i128_slt:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, hs
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, ge
-; CHECK-NEXT:    csel w8, w8, w9, eq
-; CHECK-NEXT:    tbnz w8, #0, .LBB18_2
+; CHECK-NEXT:    sbcs xzr, x1, x3
+; CHECK-NEXT:    b.ge .LBB18_2
 ; CHECK-NEXT:  // %bb.1: // %call
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    bl call
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:  .LBB18_2: // %exit
 ; CHECK-NEXT:    ret
     %cmp = icmp slt i128 %a, %b
     br i1 %cmp, label %call, label %exit
 call:
     call void @call()
     br label %exit
 exit:
     ret void
 }
 
 define void @br_on_cmp_i128_sle(i128 %a, i128 %b) nounwind {
 ; CHECK-LABEL: br_on_cmp_i128_sle:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, hi
-; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    cset w9, gt
-; CHECK-NEXT:    csel w8, w8, w9, eq
-; CHECK-NEXT:    tbnz w8, #0, .LBB19_2
+; CHECK-NEXT:    cmp x2, x0
+; CHECK-NEXT:    sbcs xzr, x3, x1
+; CHECK-NEXT:    b.lt .LBB19_2
 ; CHECK-NEXT:  // %bb.1: // %call
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    bl call
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:  .LBB19_2: // %exit
 ; CHECK-NEXT:    ret
     %cmp = icmp sle i128 %a, %b
     br i1 %cmp, label %call, label %exit
 call:
     call void @call()
     br label %exit
 exit:
     ret void
 }