[AArch64] Optimize cmp chain before legalization

[Allen]

Summary

• For case bcmp9, there is extras AND and EXTEND int the chain of OR/XOR, which prevent the transform, so enable the optimize before legalization.

• The key IR frag related:

      t37: i32,ch = load<(load (s8) from %ir.4), anyext from i8> t0, t11, undef:i64
        t12: i64 = add t4, Constant:i64<8>
      t38: i32,ch = load<(load (s8) from %ir.5), anyext from i8> t0, t12, undef:i64
    t39: i32 = xor t37, t38
  t40: i64 = any_extend t39
t42: i64 = and t40, Constant:i64<255>

Depends on D138398 to fix combine_setcc_glue

[dmgreen]

There is code in DAGCombiner::BackwardsPropagateMask that can propagate And's back to loads, and would usually handle patterns like this but it can't look through any_extends. It has seemed to be useful in the past though.
You could also imagine transforming i64 and (any_extend(i32 x), mask) into i64 zext(i32 and(x, mask) under AArch64, as we know the zext will be free. I think that would run into other problems though, as the zext between the And isn't handled for all the BFI cases. Without improving BFI at the same time it would lead to other regressions.

So I'm not sure either of those methods would be better than this, even if they are more general. I think it would be useful to add deliberate tests for this though if we can, especially for the edge cases.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

11982 ↗

(On Diff #475108)

Why "Dup" in the name? Because there are two loads? When I see Dup I think of vector splats.

12013 ↗

(On Diff #475108)

Does it not need to check other bits about the Mask?

12030 ↗

(On Diff #475108)

Should XOR be Logicop.getOpcode()?

20666 ↗

(On Diff #475108)

This can be done separately.

[Allen]

In D137936#3930036, @dmgreen wrote:

There is code in DAGCombiner::BackwardsPropagateMask that can propagate And's back to loads, and would usually handle patterns like this but it can't look through any_extends. It has seemed to be useful in the past though.
You could also imagine transforming i64 and (any_extend(i32 x), mask) into i64 zext(i32 and(x, mask) under AArch64, as we know the zext will be free. I think that would run into other problems though, as the zext between the And isn't handled for all the BFI cases. Without improving BFI at the same time it would lead to other regressions.

So I'm not sure either of those methods would be better than this, even if they are more general. I think it would be useful to add deliberate tests for this though if we can, especially for the edge cases.

a) visitAND already try the i64 and (any_extend(i32 x), mask) into i64 zext(i32 and(x, mask)`, but fail as checking MaskedValueIsZero, https://github.com/llvm/llvm-project/blob/main/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp#L6302
b) I also tryed to adjust ISD::ANY_EXTEND --> ISD::ZERO_EXTEND in DAGTypeLegalizer::PromoteIntOp_ZERO_EXTEND, then we'll also get the expect zext without AND, but some x86 cases regressions for chains of ANDs, so I limited to both operands are Loads.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

11982 ↗

(On Diff #475108)

Yes, Both operands are Load. I'll rename it to CombineZExtLogicopDoubleExtLoad

12013 ↗

(On Diff #475108)

Done, thanks

12030 ↗

(On Diff #475108)

Oh, good catch. thanks

20666 ↗

(On Diff #475108)

ok, https://reviews.llvm.org/D138124

[bcl5980]

Maybe you still can try to enable the optimize before legalization to fix these issues like:

if ((Cond == ISD::SETEQ || Cond == ISD::SETNE) && isNullConstant(RHS) &&
    LHS->getOpcode() == ISD::OR && LHS->hasOneUse() &&
    isOrXorChain(LHS, NumXors, WorkList)) {
  SDValue XOR0, XOR1;
  std::tie(XOR0, XOR1) = WorkList[0];
  SDValue Cmp = DAG.getSetCC(DL, VT, XOR0, XOR1, ISD::SETNE);
  for (unsigned I = 1; I < WorkList.size(); I++) {
    std::tie(XOR0, XOR1) = WorkList[I];
    SDValue CmpChain = DAG.getSetCC(DL, VT, XOR0, XOR1, ISD::SETNE);
    Cmp = DAG.getNode(ISD::OR, DL, VT, Cmp, CmpChain);
  }

  // Exit early by inverting the condition, which help reduce indentations.
  return DAG.getSetCC(DL, VT, Cmp, DAG.getConstant(0, DL, VT), Cond);
}

After that you need remove if (!DCI.isBeforeLegalize()) for performOrXorChainCombine and remove the function call in lowerSetCC to make the code cleaner.
As @dmgreen mentioned before, maybe you can try to combine to AArch64ISD::SUBS + AArch64ISD::CCMP. But you may need to fix some type legalization issues to make it works.
Current patch looks too specific, I think.

[Allen]

In D137936#3936125, @bcl5980 wrote:

Maybe you still can try to enable the optimize before legalization to fix these issues like:

if ((Cond == ISD::SETEQ || Cond == ISD::SETNE) && isNullConstant(RHS) &&
    LHS->getOpcode() == ISD::OR && LHS->hasOneUse() &&
    isOrXorChain(LHS, NumXors, WorkList)) {
  SDValue XOR0, XOR1;
  std::tie(XOR0, XOR1) = WorkList[0];
  SDValue Cmp = DAG.getSetCC(DL, VT, XOR0, XOR1, ISD::SETNE);
  for (unsigned I = 1; I < WorkList.size(); I++) {
    std::tie(XOR0, XOR1) = WorkList[I];
    SDValue CmpChain = DAG.getSetCC(DL, VT, XOR0, XOR1, ISD::SETNE);
    Cmp = DAG.getNode(ISD::OR, DL, VT, Cmp, CmpChain);
  }

  // Exit early by inverting the condition, which help reduce indentations.
  return DAG.getSetCC(DL, VT, Cmp, DAG.getConstant(0, DL, VT), Cond);
}

After that you need remove if (!DCI.isBeforeLegalize()) for performOrXorChainCombine and remove the function call in lowerSetCC to make the code cleaner.
As @dmgreen mentioned before, maybe you can try to combine to AArch64ISD::SUBS + AArch64ISD::CCMP. But you may need to fix some type legalization issues to make it works.
Current patch looks too specific, I think.

Thanks @bcl5980 for your idea, apply your comment.

[bcl5980]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8574

Maybe the ISD::ZERO_EXTEND can remain here?

llvm/test/CodeGen/AArch64/dag-combine-setcc.ll

224

Looks regression for this case?

[bcl5980]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8614

It looks this can continue to be simplified to

unsigned LogicOp = (Cond == ISD::SETEQ) ? ISD::AND : ISD::OR;
SDValue Cmp = DAG.getSetCC(DL, VT, XOR0, XOR1, Cond);
for (unsigned I = 1; I < WorkList.size(); I++) {
  std::tie(XOR0, XOR1) = WorkList[I];
  SDValue CmpChain = DAG.getSetCC(DL, VT, XOR0, XOR1, Cond);
  Cmp = DAG.getNode(LogicOp, DL, VT, Cmp, CmpChain);
}

return Cmp;

Looks more cases can get benefit from it.

llvm/test/CodeGen/AArch64/dag-combine-setcc.ll

224

This regression cmp x3, x10 part is a little tricky. It depends on the tree search sequence. If we search RHS first it will disappear. But I don't know how to fix it by a elegant way.

[Allen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8574

In the current implementation, we generate the AND and ANY_EXTEND sequence when , so the ISD::ZERO_EXTEND is not required?

8614

Yes, this change fix the case @PR58675, while regression on case combine_setcc_glue, so I'll need more work on it.

+++ b/llvm/test/CodeGen/AArch64/dag-combine-setcc.ll
@@ -191,9 +191,11 @@ define i32 @combine_setcc_glue(i128 noundef %x, i128 noundef %y) {
 ; CHECK-LABEL: combine_setcc_glue:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    cmp x1, x3
-; CHECK-NEXT:    ccmp x0, x2, #0, eq
-; CHECK-NEXT:    ccmp x0, x2, #4, ne
-; CHECK-NEXT:    cset w0, eq
+; CHECK-NEXT:    cset w8, eq
+; CHECK-NEXT:    cmp x0, x2
+; CHECK-NEXT:    cset w9, eq
+; CHECK-NEXT:    and w8, w9, w8
+; CHECK-NEXT:    orr w0, w8, w9
 ; CHECK-NEXT:    ret

[bcl5980]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8574

The case like that can trigger the zext part. https://www.godbolt.org/z/bPrT3G7bh
And I think the cost is very low so we can add it.

8614

Don't worry about the combine_setcc_glue regression. D138401 already fixed that.

[bcl5980]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8614

Sorry, the change fix combine_setcc_glue is D138398

[Allen]

rebase base on D138398

[Allen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8614

Thanks for the fixing, I rebase on that.

[bcl5980]

LGTM, but please waiting for @dmgreen or someone else to approve it.

[dmgreen]

Can you add tests for chains or or/xor with some different types. Maybe i8/i16/i128 (assuming we already have them for i32/i64). Then something strange like an i42. If we are allowing the transform in many more places then it would be good to test them.

[Allen]

Add 5 more cases with zero_extend and different types (i8/i16/i128/i42)

[dmgreen]

llvm/test/CodeGen/AArch64/bcmp.ll

432

We don't usually put godbolt links in the source.

I see the new code is a little larger. They wouldn't be if the inputs were loaded or known to be smaller already though. And they don't fail, which was the main point of adding them.

[dmgreen]

OK lets go with this patch. Its not quite perfect, but the gains are good to see and we can fix the other issues if they cause problems. If you remove the godbolt link then LGTM.

Remember to upload with context. thanks

[Allen]

Remove the godbolt as comment

[Allen]

llvm/test/CodeGen/AArch64/bcmp.ll

432

Deleted, thanks

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (Num == MaxXors)
     return false;
 
   // The leaf node must be XOR
   if (N->getOpcode() == ISD::XOR) {
     WorkList.push_back(std::make_pair(N->getOperand(0), N->getOperand(1)));
     Num++;
     return true;
   }
 
   // All the non-leaf nodes must be OR.
   if (N->getOpcode() != ISD::OR || !N->hasOneUse())
     return false;
 
   if (isOrXorChain(N->getOperand(0), Num, WorkList) &&
       isOrXorChain(N->getOperand(1), Num, WorkList))
     return true;
   return false;
static SDValue performOrXorChainCombine(SDNode *N, SelectionDAG &DAG) {
 
   ISD::CondCode Cond = cast<CondCodeSDNode>(N->getOperand(2))->get();
   // Try to express conjunction "cmp 0 (or (xor A0 A1) (xor B0 B1))" as:
   // sub A0, A1; ccmp B0, B1, 0, eq; cmp inv(Cond) flag
   unsigned NumXors = 0;
   if ((Cond == ISD::SETEQ || Cond == ISD::SETNE) && isNullConstant(RHS) &&
       LHS->getOpcode() == ISD::OR && LHS->hasOneUse() &&
       isOrXorChain(LHS, NumXors, WorkList)) {
-    SDValue CCVal = DAG.getConstant(AArch64CC::EQ, DL, MVT_CC);
-    EVT TstVT = LHS->getValueType(0);
     SDValue XOR0, XOR1;
     std::tie(XOR0, XOR1) = WorkList[0];
-    SDValue Cmp = DAG.getNode(AArch64ISD::SUBS, DL,
-                              DAG.getVTList(TstVT, MVT::i32), XOR0, XOR1);
-    SDValue Overflow = Cmp.getValue(1);
-    SDValue CCmp;
+    SDValue Cmp = DAG.getSetCC(DL, VT, XOR0, XOR1, ISD::SETNE);
     for (unsigned I = 1; I < WorkList.size(); I++) {
       std::tie(XOR0, XOR1) = WorkList[I];
-      SDValue NZCVOp = DAG.getConstant(0, DL, MVT::i32);
-      CCmp = DAG.getNode(AArch64ISD::CCMP, DL, MVT_CC, XOR0, XOR1, NZCVOp,
-                         CCVal, Overflow);
-      Overflow = CCmp;
+      SDValue CmpChain = DAG.getSetCC(DL, VT, XOR0, XOR1, ISD::SETNE);
+      Cmp = DAG.getNode(ISD::OR, DL, VT, Cmp, CmpChain);
     }
 
     // Exit early by inverting the condition, which help reduce indentations.
-    SDValue TVal = DAG.getConstant(1, DL, VT);
-    SDValue FVal = DAG.getConstant(0, DL, VT);
-    AArch64CC::CondCode CC = changeIntCCToAArch64CC(Cond);
-    AArch64CC::CondCode InvCC = AArch64CC::getInvertedCondCode(CC);
-    return DAG.getNode(AArch64ISD::CSEL, DL, VT, FVal, TVal,
-                       DAG.getConstant(InvCC, DL, MVT::i32), CCmp);
+    return DAG.getSetCC(DL, VT, Cmp, DAG.getConstant(0, DL, VT), Cond);
   }
 
   return SDValue();
 }
 
 SDValue AArch64TargetLowering::LowerSETCC(SDValue Op, SelectionDAG &DAG) const {
 
   if (Op.getValueType().isVector())
if (LHS.getValueType() == MVT::f128) {
     // If softenSetCCOperands returned a scalar, use it.
     if (!RHS.getNode()) {
       assert(LHS.getValueType() == Op.getValueType() &&
              "Unexpected setcc expansion!");
       return IsStrict ? DAG.getMergeValues({LHS, Chain}, dl) : LHS;
     }
   }
 
-  // Address some cases folded And in the stage of `Optimized type-legalized
-  // selection`
-  if (SDValue V = performOrXorChainCombine(Op.getNode(), DAG))
-    return V;
-
   if (LHS.getValueType().isInteger()) {
     SDValue CCVal;
     SDValue Cmp = getAArch64Cmp(
         LHS, RHS, ISD::getSetCCInverse(CC, LHS.getValueType()), CCVal, DAG, dl);
 
     // Note that we inverted the condition above, so we reverse the order of
     // the true and false operands here.  This will allow the setcc to be
     // matched to a single CSINC instruction.
if (FromVT.isFixedLengthVector() &&
         FromVT.getVectorElementType() == MVT::i1) {
       LHS = DAG.getNode(ISD::VECREDUCE_OR, DL, MVT::i1, LHS->getOperand(0));
       LHS = DAG.getNode(ISD::ZERO_EXTEND, DL, ToVT, LHS);
       return DAG.getSetCC(DL, VT, LHS, RHS, Cond);
     }
   }
 
   // Try to perform the memcmp when the result is tested for [in]equality with 0
-  if (!DCI.isBeforeLegalize())
-    if (SDValue V = performOrXorChainCombine(N, DAG))
-      return V;
 
   return SDValue();
 }
 
 // Replace a flag-setting operator (eg ANDS) with the generic version
 // (eg AND) if the flag is unused.
 static SDValue performFlagSettingCombine(SDNode *N,
                                          TargetLowering::DAGCombinerInfo &DCI,

llvm/test/CodeGen/AArch64/atomicrmw-O0.ll

 ; NOLSE-NEXT:    str x9, [sp, #8] // 8-byte Folded Spill
 ; NOLSE-NEXT:    mov x10, x12
 ; NOLSE-NEXT:    str x10, [sp, #16] // 8-byte Folded Spill
 ; NOLSE-NEXT:    subs x12, x12, x13
 ; NOLSE-NEXT:    ccmp x8, x11, #0, eq
 ; NOLSE-NEXT:    cset w8, ne
 ; NOLSE-NEXT:    str x10, [sp, #32] // 8-byte Folded Spill
 ; NOLSE-NEXT:    str x9, [sp, #40] // 8-byte Folded Spill
-; NOLSE-NEXT:    tbnz w8, #0, .LBB4_1
+; NOLSE-NEXT:    cbnz w8, .LBB4_1
 ; NOLSE-NEXT:    b .LBB4_6
 ; NOLSE-NEXT:  .LBB4_6: // %atomicrmw.end
 ; NOLSE-NEXT:    ldr x1, [sp, #8] // 8-byte Folded Reload
 ; NOLSE-NEXT:    ldr x0, [sp, #16] // 8-byte Folded Reload
 ; NOLSE-NEXT:    add sp, sp, #48
 ; NOLSE-NEXT:    ret
 ;
 ; LSE-LABEL: test_rmw_add_128:

 ; LSE-NEXT:    str x9, [sp, #8] // 8-byte Folded Spill
 ; LSE-NEXT:    mov x10, x0
 ; LSE-NEXT:    str x10, [sp, #16] // 8-byte Folded Spill
 ; LSE-NEXT:    subs x11, x10, x11
 ; LSE-NEXT:    ccmp x9, x8, #0, eq
 ; LSE-NEXT:    cset w8, ne
 ; LSE-NEXT:    str x10, [sp, #32] // 8-byte Folded Spill
 ; LSE-NEXT:    str x9, [sp, #40] // 8-byte Folded Spill
-; LSE-NEXT:    tbnz w8, #0, .LBB4_1
+; LSE-NEXT:    cbnz w8, .LBB4_1
 ; LSE-NEXT:    b .LBB4_2
 ; LSE-NEXT:  .LBB4_2: // %atomicrmw.end
 ; LSE-NEXT:    ldr x1, [sp, #8] // 8-byte Folded Reload
 ; LSE-NEXT:    ldr x0, [sp, #16] // 8-byte Folded Reload
 ; LSE-NEXT:    add sp, sp, #48
 ; LSE-NEXT:    ret
 entry:
   %res = atomicrmw add i128* %dst, i128 1 seq_cst

 ; NOLSE-NEXT:    str x9, [sp, #8] // 8-byte Folded Spill
 ; NOLSE-NEXT:    mov x10, x12
 ; NOLSE-NEXT:    str x10, [sp, #16] // 8-byte Folded Spill
 ; NOLSE-NEXT:    subs x12, x12, x13
 ; NOLSE-NEXT:    ccmp x8, x11, #0, eq
 ; NOLSE-NEXT:    cset w8, ne
 ; NOLSE-NEXT:    str x10, [sp, #32] // 8-byte Folded Spill
 ; NOLSE-NEXT:    str x9, [sp, #40] // 8-byte Folded Spill
-; NOLSE-NEXT:    tbnz w8, #0, .LBB9_1
+; NOLSE-NEXT:    cbnz w8, .LBB9_1
 ; NOLSE-NEXT:    b .LBB9_6
 ; NOLSE-NEXT:  .LBB9_6: // %atomicrmw.end
 ; NOLSE-NEXT:    ldr x1, [sp, #8] // 8-byte Folded Reload
 ; NOLSE-NEXT:    ldr x0, [sp, #16] // 8-byte Folded Reload
 ; NOLSE-NEXT:    add sp, sp, #48
 ; NOLSE-NEXT:    ret
 ;
 ; LSE-LABEL: test_rmw_nand_128:

 ; LSE-NEXT:    str x9, [sp, #8] // 8-byte Folded Spill
 ; LSE-NEXT:    mov x10, x0
 ; LSE-NEXT:    str x10, [sp, #16] // 8-byte Folded Spill
 ; LSE-NEXT:    subs x11, x10, x11
 ; LSE-NEXT:    ccmp x9, x8, #0, eq
 ; LSE-NEXT:    cset w8, ne
 ; LSE-NEXT:    str x10, [sp, #32] // 8-byte Folded Spill
 ; LSE-NEXT:    str x9, [sp, #40] // 8-byte Folded Spill
-; LSE-NEXT:    tbnz w8, #0, .LBB9_1
+; LSE-NEXT:    cbnz w8, .LBB9_1
 ; LSE-NEXT:    b .LBB9_2
 ; LSE-NEXT:  .LBB9_2: // %atomicrmw.end
 ; LSE-NEXT:    ldr x1, [sp, #8] // 8-byte Folded Reload
 ; LSE-NEXT:    ldr x0, [sp, #16] // 8-byte Folded Reload
 ; LSE-NEXT:    add sp, sp, #48
 ; LSE-NEXT:    ret
 entry:
   %res = atomicrmw nand i128* %dst, i128 1 seq_cst
   ret i128 %res
 }

llvm/test/CodeGen/AArch64/bcmp.ll

 ; CHECK-NEXT:    cmp x8, x9
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 8)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
-; TODO: or (xor a, b), (and (xor c, d), C2)
+; or (xor a, b), (and (xor c, d), C2)
 define i1 @bcmp9(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp9:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ldrb w9, [x0, #8]
-; CHECK-NEXT:    ldrb w10, [x1, #8]
 ; CHECK-NEXT:    ldr x8, [x0]
-; CHECK-NEXT:    ldr x11, [x1]
-; CHECK-NEXT:    eor w9, w9, w10
-; CHECK-NEXT:    and x9, x9, #0xff
-; CHECK-NEXT:    eor x8, x8, x11
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    cmp x8, #0
+; CHECK-NEXT:    ldr x9, [x1]
+; CHECK-NEXT:    ldrb w10, [x0, #8]
+; CHECK-NEXT:    ldrb w11, [x1, #8]
+; CHECK-NEXT:    cmp x8, x9
+; CHECK-NEXT:    ccmp x10, x11, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 9)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
 define i1 @bcmp10(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp10:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ldrh w9, [x0, #8]
-; CHECK-NEXT:    ldrh w10, [x1, #8]
 ; CHECK-NEXT:    ldr x8, [x0]
-; CHECK-NEXT:    ldr x11, [x1]
-; CHECK-NEXT:    eor w9, w9, w10
-; CHECK-NEXT:    and x9, x9, #0xffff
-; CHECK-NEXT:    eor x8, x8, x11
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    cmp x8, #0
+; CHECK-NEXT:    ldr x9, [x1]
+; CHECK-NEXT:    ldrh w10, [x0, #8]
+; CHECK-NEXT:    ldrh w11, [x1, #8]
+; CHECK-NEXT:    cmp x8, x9
+; CHECK-NEXT:    ccmp x10, x11, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 10)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
 define i1 @bcmp11(ptr %a, ptr %b) {

 
 define i1 @bcmp12(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp12:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr x8, [x0]
 ; CHECK-NEXT:    ldr x9, [x1]
 ; CHECK-NEXT:    ldr w10, [x0, #8]
 ; CHECK-NEXT:    ldr w11, [x1, #8]
-; CHECK-NEXT:    eor x8, x8, x9
-; CHECK-NEXT:    eor w9, w10, w11
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    cmp x8, #0
+; CHECK-NEXT:    cmp x8, x9
+; CHECK-NEXT:    ccmp x10, x11, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 12)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
 define i1 @bcmp13(ptr %a, ptr %b) {

 }
 
 define i1 @bcmp20(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp20:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldp x8, x9, [x0]
 ; CHECK-NEXT:    ldp x10, x11, [x1]
 ; CHECK-NEXT:    ldr w12, [x0, #16]
-; CHECK-NEXT:    ldr w13, [x1, #16]
-; CHECK-NEXT:    eor x8, x8, x10
-; CHECK-NEXT:    eor x9, x9, x11
-; CHECK-NEXT:    eor w10, w12, w13
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    orr x8, x8, x10
-; CHECK-NEXT:    cmp x8, #0
+; CHECK-NEXT:    cmp x8, x10
+; CHECK-NEXT:    ldr w8, [x1, #16]
+; CHECK-NEXT:    ccmp x9, x11, #0, eq
+; CHECK-NEXT:    ccmp x12, x8, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 20)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
 define i1 @bcmp24(ptr %a, ptr %b) {

 }
 
 define i1 @bcmp28(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp28:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldp x8, x9, [x0]
 ; CHECK-NEXT:    ldp x10, x11, [x1]
 ; CHECK-NEXT:    ldr x12, [x0, #16]
-; CHECK-NEXT:    ldr x13, [x1, #16]
-; CHECK-NEXT:    ldr w14, [x0, #24]
-; CHECK-NEXT:    eor x8, x8, x10
-; CHECK-NEXT:    ldr w15, [x1, #24]
-; CHECK-NEXT:    eor x9, x9, x11
-; CHECK-NEXT:    eor x10, x12, x13
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    eor w11, w14, w15
-; CHECK-NEXT:    orr x9, x10, x11
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    cmp x8, #0
+; CHECK-NEXT:    cmp x8, x10
+; CHECK-NEXT:    ldr x8, [x1, #16]
+; CHECK-NEXT:    ccmp x9, x11, #0, eq
+; CHECK-NEXT:    ldr w9, [x0, #24]
+; CHECK-NEXT:    ldr w10, [x1, #24]
+; CHECK-NEXT:    ccmp x12, x8, #0, eq
+; CHECK-NEXT:    ccmp x9, x10, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 28)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
 define i1 @bcmp33(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp33:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldp x8, x9, [x0]
 ; CHECK-NEXT:    ldp x10, x11, [x1]
-; CHECK-NEXT:    ldp x12, x13, [x0, #16]
-; CHECK-NEXT:    ldp x14, x15, [x1, #16]
-; CHECK-NEXT:    eor x8, x8, x10
-; CHECK-NEXT:    eor x9, x9, x11
-; CHECK-NEXT:    ldrb w16, [x0, #32]
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    ldrb w17, [x1, #32]
-; CHECK-NEXT:    eor x10, x12, x14
-; CHECK-NEXT:    eor x11, x13, x15
-; CHECK-NEXT:    eor w12, w16, w17
-; CHECK-NEXT:    orr x9, x10, x11
-; CHECK-NEXT:    and x10, x12, #0xff
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    orr x8, x8, x10
-; CHECK-NEXT:    cmp x8, #0
+; CHECK-NEXT:    cmp x8, x10
+; CHECK-NEXT:    ccmp x9, x11, #0, eq
+; CHECK-NEXT:    ldrb w11, [x1, #32]
+; CHECK-NEXT:    ldp x8, x9, [x0, #16]
+; CHECK-NEXT:    ldp x12, x10, [x1, #16]
+; CHECK-NEXT:    ccmp x8, x12, #0, eq
+; CHECK-NEXT:    ldrb w8, [x0, #32]
+; CHECK-NEXT:    ccmp x9, x10, #0, eq
+; CHECK-NEXT:    ccmp x8, x11, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 33)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
 define i1 @bcmp38(ptr %a, ptr %b) {

 ; CHECK-NEXT:    cmp w0, #0
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 89)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 

llvm/test/CodeGen/AArch64/dag-combine-setcc.ll

11:                                               ; preds = %4, %9
   ret i32 %12
 }
 
 ; There may be issues with the CMP/CCMP with the scheduling of instructions
 ; that ISel will create out of the DAG
 define i32 @combine_setcc_glue(i128 noundef %x, i128 noundef %y) {
 ; CHECK-LABEL: combine_setcc_glue:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    cset w8, eq
-; CHECK-NEXT:    ccmp x1, x3, #0, eq
-; CHECK-NEXT:    cset w9, eq
-; CHECK-NEXT:    orr w0, w9, w8
+; CHECK-NEXT:    cmp x1, x3
+; CHECK-NEXT:    ccmp x0, x2, #0, eq
+; CHECK-NEXT:    ccmp x0, x2, #4, ne
+; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
 entry:
   %cmp3 = icmp eq i128 %x, %y
   %conv = trunc i128 %x to i64
   %conv1 = trunc i128 %y to i64
   %cmp = icmp eq i64 %conv, %conv1
   %or7 = or i1 %cmp3, %cmp
   %or = zext i1 %or7 to i32

 ; CHECK-NEXT:    cmp x0, x8
 ; CHECK-NEXT:    csel x10, x0, x8, lo
 ; CHECK-NEXT:    cmp x1, x9
 ; CHECK-NEXT:    csel x8, x0, x8, lo
 ; CHECK-NEXT:    csel x8, x10, x8, eq
 ; CHECK-NEXT:    csel x10, x1, x9, lo
 ; CHECK-NEXT:    subs x8, x2, x8
 ; CHECK-NEXT:    sbc x9, x3, x10
-; CHECK-NEXT:    ccmp x3, x10, #0, eq
-; CHECK-NEXT:    b.ne .LBB12_1
+; CHECK-NEXT:    cset w11, ne
+; CHECK-NEXT:    cmp x3, x10
+; CHECK-NEXT:    cset w10, ne
+; CHECK-NEXT:    orr w10, w11, w10
+; CHECK-NEXT:    cbnz w10, .LBB12_1
 ; CHECK-NEXT:  // %bb.2: // %do.end
 ; CHECK-NEXT:    mov x0, xzr
 ; CHECK-NEXT:    mov x1, xzr
 ; CHECK-NEXT:    ret
 entry:
   br label %do.body
 
 do.body:                                      ; preds = %do.body, %entry

llvm/test/CodeGen/AArch64/i128-cmp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-uknown-uknown -verify-machineinstrs -o - %s | FileCheck %s
 
 declare void @call()
 
 define i1 @cmp_i128_eq(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    ccmp x1, x3, #0, eq
+; CHECK-NEXT:    cmp x1, x3
+; CHECK-NEXT:    ccmp x0, x2, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
     %cmp = icmp eq i128 %a, %b
     ret i1 %cmp
 }
 
 define i1 @cmp_i128_ne(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_ne:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmp x0, x2
-; CHECK-NEXT:    ccmp x1, x3, #0, eq
+; CHECK-NEXT:    cmp x1, x3
+; CHECK-NEXT:    ccmp x0, x2, #0, eq
 ; CHECK-NEXT:    cset w0, ne
 ; CHECK-NEXT:    ret
     %cmp = icmp ne i128 %a, %b
     ret i1 %cmp
 }
 
 define i1 @cmp_i128_ugt(i128 %a, i128 %b) {
 ; CHECK-LABEL: cmp_i128_ugt:

llvm/test/CodeGen/AArch64/umulo-128-legalisation-lowering.ll

 ; AARCH-NEXT:    cinc x11, x14, hs
 ; AARCH-NEXT:    mul x0, x0, x2
 ; AARCH-NEXT:    adds x11, x13, x11
 ; AARCH-NEXT:    umulh x13, x8, x3
 ; AARCH-NEXT:    cset w14, hs
 ; AARCH-NEXT:    adds x11, x12, x11
 ; AARCH-NEXT:    adc x12, x13, x14
 ; AARCH-NEXT:    adds x10, x11, x10
-; AARCH-NEXT:    asr x11, x1, #63
 ; AARCH-NEXT:    adc x9, x12, x9
-; AARCH-NEXT:    cmp x10, x11
-; AARCH-NEXT:    ccmp x9, x11, #0, eq
+; AARCH-NEXT:    asr x11, x1, #63
+; AARCH-NEXT:    cmp x9, x11
+; AARCH-NEXT:    ccmp x10, x11, #0, eq
 ; AARCH-NEXT:    cset w9, ne
 ; AARCH-NEXT:    tbz x8, #63, .LBB1_2
 ; AARCH-NEXT:  // %bb.1: // %Entry
 ; AARCH-NEXT:    eor x8, x3, #0x8000000000000000
 ; AARCH-NEXT:    orr x8, x2, x8
 ; AARCH-NEXT:    cbz x8, .LBB1_3
 ; AARCH-NEXT:  .LBB1_2: // %Else2
 ; AARCH-NEXT:    cbz w9, .LBB1_4