[AArch64] Optimize more memcmp when the result is tested for [in]equality with 0

[Allen]

Summary

We already surpport the or (xor a, b), (xor c, d) with D136244, while it should
capture more cases than just bcmp according the comment on
https://reviews.llvm.org/D136672, so this patch try to fold continuous
comparison series.

Also add a new callsite in LowerSETCC to address some cases folded And in the
stage of Optimized type-legalized selection.

[dmgreen]

Looks like a good patch. I like the way it checks for chains of ORs.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

19655

The check for OR could be moved below the check for XOR, saving XOR from being checked twice.

19670

Could this be called something like performOrXorChainCombine. I'm not sure I love that name, but the transform is more general than just memcmp. It would be good to mention memcmp/bmp in the comments of the function perhaps, to explain the motivation.

19682

LLVM likes to exit early by inverting the condition. It can help reduce indentations.

19684

Perhaps rename isMemcmpPattern to isOrXorChain too.

19694

i -> I

19711

Could this do:

if (!DCI.isBeforeLegalize())
  if (SDValue V = performMemcmpCombine(N, DAG))
    return V;

It should avoid needing to pass DCI through to performMemcmpCombine.

llvm/test/CodeGen/AArch64/bcmp.ll

43

I think this might be fixed if performMemcmpCombine was called from lowerSETCC as well as from the combine. It looks like the issue is that it we do not call the performMemcmpCombine after the operands further up the tree have been simplified.

145

This And shouldn't be here - it should be pushed higher into the ldrb's. There is some code that already tried to push ands up into loads, but I've not looked into whether it could be extended to handle this too.

[Allen]

Address comments

[Allen]

llvm/test/CodeGen/AArch64/bcmp.ll

43

Thansk, apply with your commend

145

Ok, I'll try to figure out that with a following patch, add a TODO for this case

[bcl5980]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8571

Do we need add one-use check for or?
Or generate the depth of the node to determine the more than one use node can enable the opt or not?

8592

cmp A0, A1; ccmp B0, B1 ?

[bcl5980]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8565

Can ISD::SUB be leaf node also?

[Allen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8565

Thanks, But I doesn't sure this need be extended as the bcmp doesn't generate such IR after outlining?

8571

Yes, we can check the one-use to start, thanks.
If more complex scenarios are required in the actual code, then the depth may also need later.

8592

Done, thanks!

[bcl5980]

llvm/test/CodeGen/AArch64/bcmp.ll

415–428

I agree that cmp+ccmp chain is generally better but a little worry about this test case.
cmp chain need 8 cycles to do on every machine.
But 8 xor + 7 or + 1 cmp can run faster on high end cpu. For example a 4 width int alu port machine.
2 cycle for xor
3 cycle for or
1 cycle for cmp
total 6 cycle.

[Allen]

llvm/test/CodeGen/AArch64/bcmp.ll

415–428

Good catch. In general, all of the XOR, OR and CMP use ALU ports, so data dependency will become the bottleneck on high end CPU.
If so, an additional parameter is needed to guard the max number of xors ? Or some other suggestion?

[bcl5980]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8565

I believe the leaf node needn't one-use. It will not increase the instruction count.

llvm/test/CodeGen/AArch64/bcmp.ll

415–428

I'm also not sure if we need a max leaf node limitation. Max size of bcmp expand is 64bytes. So larger size also needn't worry about it.

[Allen]

Delete the one-use of leaf node XOR and add a limitation of xor nodes

[Allen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8565

Done, thanks

llvm/test/CodeGen/AArch64/bcmp.ll

415–428

Thanks, so I add the max limitation number 6 for xors now.
If we can get more schedule model info, we may relex th e condition later.

[dmgreen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8591

This code doesn't handle float compares so you shouldn't need the IsStrict stuff. Maybe only call this from LowerSETCC if the Opcode is ISD::SETCC or LHS.getValueType().isInteger().

llvm/test/CodeGen/AArch64/bcmp.ll

415–428

I can see what you mean, but I'm not sure we need to limit this case. In my experience this much of reduction in instruction count can be good for performance on its own, even if it turns the tree into a series of dependant ccmp. We could theoretically have larger trees though, so maybe put the limit to higher?

Unless you have performance results that actually shows it to be worse, we believe it is better on most cpus so I would have it perform the transform in this case. 8 less instructions is probably always worth 1 longer critical patch length.

[Allen]

Adjust the max limitation 6->16

[Allen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

8591

• ok. delete th IsStrict stuff

• the callsize from performSETCCCombine is still need as brcond+setcc will be combine into br_cc in the stage Optimized type-legalized selection, which is before the Legalized selection DAG, such as case br_on_cmp_i128_ne in file CodeGen/AArch64/i128-cmp.ll

llvm/test/CodeGen/AArch64/bcmp.ll

415–428

Thanks, I adjust the max limitation 6->16, and this case it enable the transform (I don't have a machine with 4 width ALU port to test this case)

[tschuett]

There is no GlobalISel code?

[Allen]

In D137721#3923121, @tschuett wrote:

There is no GlobalISel code?

I'm not familiar with GlobalISel, but I can try to complement this it with a new independent patch. It would be better if you could direct me which interface I need to focus on:)

[tschuett]

No worries. Their Instruction Selection Code is here:
https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

[dmgreen]

Thanks for the update. LGTM.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

 // in future when both implementations will be based off MGATHER rather
 // than the GLD1 nodes added for the SVE gather load intrinsics.
 static cl::opt<bool>
 EnableCombineMGatherIntrinsics("aarch64-enable-mgather-combine", cl::Hidden,
                                 cl::desc("Combine extends of AArch64 masked "
                                          "gather intrinsics"),
                                 cl::init(true));
 
+// All of the XOR, OR and CMP use ALU ports, and data dependency will become the
+// bottleneck after this transform on high end CPU. So this max leaf node
+// limitation is guard cmp+ccmp will be profitable.
+static cl::opt<unsigned> MaxXors("aarch64-max-xors", cl::init(16), cl::Hidden,
+                                 cl::desc("Maximum of xors"));
+
 /// Value type used for condition codes.
 static const MVT MVT_CC = MVT::i32;
 
 static inline EVT getPackedSVEVectorVT(EVT VT) {
   switch (VT.getSimpleVT().SimpleTy) {
   default:
     llvm_unreachable("unexpected element type for vector");
   case MVT::i8:
case MVT::v2i64: {
     break;
   }
   }
 
   return DAG.getNode(AArch64ISD::NVCAST, DL, VT,
                      DAG.getNode(ISD::BITREVERSE, DL, VST, REVB));
 }
 
+// Check whether the continuous comparison sequence.
+static bool
+isOrXorChain(SDValue N, unsigned &Num,
+             SmallVector<std::pair<SDValue, SDValue>, 16> &WorkList) {
+  if (Num == MaxXors)
+    return false;
+
+  // The leaf node must be XOR
+  if (N->getOpcode() == ISD::XOR) {
+    WorkList.push_back(std::make_pair(N->getOperand(0), N->getOperand(1)));
+    Num++;
+    return true;
+  }
+
+  // All the non-leaf nodes must be OR.
+  if (N->getOpcode() != ISD::OR || !N->hasOneUse())
+    return false;
+
+  if (isOrXorChain(N->getOperand(0), Num, WorkList) &&
+      isOrXorChain(N->getOperand(1), Num, WorkList))
+    return true;
+  return false;
+}
+
+// Transform chains of ORs and XORs, which usually outlined by memcmp/bmp.
+static SDValue performOrXorChainCombine(SDNode *N, SelectionDAG &DAG) {
+  SDValue LHS = N->getOperand(0);
+  SDValue RHS = N->getOperand(1);
+  SDLoc DL(N);
+  EVT VT = N->getValueType(0);
+  SmallVector<std::pair<SDValue, SDValue>, 16> WorkList;
+
+  // Only handle integer compares.
+  if (N->getOpcode() != ISD::SETCC)
+    return SDValue();
+
+  ISD::CondCode Cond = cast<CondCodeSDNode>(N->getOperand(2))->get();
+  // Try to express conjunction "cmp 0 (or (xor A0 A1) (xor B0 B1))" as:
+  // sub A0, A1; ccmp B0, B1, 0, eq; cmp inv(Cond) flag
+  unsigned NumXors = 0;
+  if ((Cond == ISD::SETEQ || Cond == ISD::SETNE) && isNullConstant(RHS) &&
+      LHS->getOpcode() == ISD::OR && LHS->hasOneUse() &&
+      isOrXorChain(LHS, NumXors, WorkList)) {
+    SDValue CCVal = DAG.getConstant(AArch64CC::EQ, DL, MVT_CC);
+    EVT TstVT = LHS->getValueType(0);
+    SDValue XOR0, XOR1;
+    std::tie(XOR0, XOR1) = WorkList[0];
+    SDValue Cmp = DAG.getNode(AArch64ISD::SUBS, DL,
+                              DAG.getVTList(TstVT, MVT::i32), XOR0, XOR1);
+    SDValue Overflow = Cmp.getValue(1);
+    SDValue CCmp;
+    for (unsigned I = 1; I < WorkList.size(); I++) {
+      std::tie(XOR0, XOR1) = WorkList[I];
+      SDValue NZCVOp = DAG.getConstant(0, DL, MVT::i32);
+      CCmp = DAG.getNode(AArch64ISD::CCMP, DL, MVT_CC, XOR0, XOR1, NZCVOp,
+                         CCVal, Overflow);
+      Overflow = CCmp;
+    }
+
+    // Exit early by inverting the condition, which help reduce indentations.
+    SDValue TVal = DAG.getConstant(1, DL, VT);
+    SDValue FVal = DAG.getConstant(0, DL, VT);
+    AArch64CC::CondCode CC = changeIntCCToAArch64CC(Cond);
+    AArch64CC::CondCode InvCC = AArch64CC::getInvertedCondCode(CC);
+    return DAG.getNode(AArch64ISD::CSEL, DL, VT, FVal, TVal,
+                       DAG.getConstant(InvCC, DL, MVT::i32), CCmp);
+  }
+
+  return SDValue();
+}
+
 SDValue AArch64TargetLowering::LowerSETCC(SDValue Op, SelectionDAG &DAG) const {
 
   if (Op.getValueType().isVector())
     return LowerVSETCC(Op, DAG);
 
   bool IsStrict = Op->isStrictFPOpcode();
   bool IsSignaling = Op.getOpcode() == ISD::STRICT_FSETCCS;
   unsigned OpNo = IsStrict ? 1 : 0;
if (LHS.getValueType() == MVT::f128) {
     // If softenSetCCOperands returned a scalar, use it.
     if (!RHS.getNode()) {
       assert(LHS.getValueType() == Op.getValueType() &&
              "Unexpected setcc expansion!");
       return IsStrict ? DAG.getMergeValues({LHS, Chain}, dl) : LHS;
     }
   }
 
+  // Address some cases folded And in the stage of `Optimized type-legalized
+  // selection`
+  if (SDValue V = performOrXorChainCombine(Op.getNode(), DAG))
+    return V;
+
   if (LHS.getValueType().isInteger()) {
     SDValue CCVal;
     SDValue Cmp = getAArch64Cmp(
         LHS, RHS, ISD::getSetCCInverse(CC, LHS.getValueType()), CCVal, DAG, dl);
 
     // Note that we inverted the condition above, so we reverse the order of
     // the true and false operands here.  This will allow the setcc to be
     // matched to a single CSINC instruction.
return DAG.getNode(ISD::SETCC, DL, UseMVT.changeVectorElementType(MVT::i1),
                      Op0ExtV, Op1ExtV, Op->getOperand(2));
 }
 
 static SDValue performSETCCCombine(SDNode *N,
                                    TargetLowering::DAGCombinerInfo &DCI,
                                    SelectionDAG &DAG) {
   assert(N->getOpcode() == ISD::SETCC && "Unexpected opcode!");
   SDValue LHS = N->getOperand(0);
   SDValue RHS = N->getOperand(1);
   ISD::CondCode Cond = cast<CondCodeSDNode>(N->getOperand(2))->get();
   SDLoc DL(N);
   EVT VT = N->getValueType(0);
 
   if (SDValue V = tryToWidenSetCCOperands(N, DAG))
     return V;
 
   // setcc (csel 0, 1, cond, X), 1, ne ==> csel 0, 1, !cond, X
   if (Cond == ISD::SETNE && isOneConstant(RHS) &&
       LHS->getOpcode() == AArch64ISD::CSEL &&
       isNullConstant(LHS->getOperand(0)) && isOneConstant(LHS->getOperand(1)) &&
       LHS->hasOneUse()) {
     // Invert CSEL's condition.
     auto *OpCC = cast<ConstantSDNode>(LHS.getOperand(2));
     auto OldCond = static_cast<AArch64CC::CondCode>(OpCC->getZExtValue());
     auto NewCond = getInvertedCondCode(OldCond);
 
     // csel 0, 1, !cond, X
     SDValue CSEL =
         DAG.getNode(AArch64ISD::CSEL, DL, LHS.getValueType(), LHS.getOperand(0),
                     LHS.getOperand(1), DAG.getConstant(NewCond, DL, MVT::i32),
                     LHS.getOperand(3));
     return DAG.getZExtOrTrunc(CSEL, DL, VT);
   }
 
   // setcc (srl x, imm), 0, ne ==> setcc (and x, (-1 << imm)), 0, ne
   if (Cond == ISD::SETNE && isNullConstant(RHS) &&
       LHS->getOpcode() == ISD::SRL && isa<ConstantSDNode>(LHS->getOperand(1)) &&
       LHS->hasOneUse()) {
     EVT TstVT = LHS->getValueType(0);
     if (TstVT.isScalarInteger() && TstVT.getFixedSizeInBits() <= 64) {
       // this pattern will get better opt in emitComparison
       uint64_t TstImm = -1ULL << LHS->getConstantOperandVal(1);
       SDValue TST = DAG.getNode(ISD::AND, DL, TstVT, LHS->getOperand(0),
                                 DAG.getConstant(TstImm, DL, TstVT));
       return DAG.getNode(ISD::SETCC, DL, VT, TST, RHS, N->getOperand(2));
     }
   }
 
   // setcc (iN (bitcast (vNi1 X))), 0, (eq|ne)
   //   ==> setcc (iN (zext (i1 (vecreduce_or (vNi1 X))))), 0, (eq|ne)
   if (DCI.isBeforeLegalize() && VT.isScalarInteger() &&
       (Cond == ISD::SETEQ || Cond == ISD::SETNE) && isNullConstant(RHS) &&
       LHS->getOpcode() == ISD::BITCAST) {
     EVT ToVT = LHS->getValueType(0);
     EVT FromVT = LHS->getOperand(0).getValueType();
     if (FromVT.isFixedLengthVector() &&
         FromVT.getVectorElementType() == MVT::i1) {
       LHS = DAG.getNode(ISD::VECREDUCE_OR, DL, MVT::i1, LHS->getOperand(0));
       LHS = DAG.getNode(ISD::ZERO_EXTEND, DL, ToVT, LHS);
       return DAG.getSetCC(DL, VT, LHS, RHS, Cond);
     }
   }
 
-  // Try to express conjunction "cmp 0 (or (xor A0 A1) (xor B0 B1))" as:
-  // cmp A0, A0; ccmp A0, B1, 0, eq; cmp inv(Cond) flag
-  if (!DCI.isBeforeLegalize() && VT.isScalarInteger() &&
-      (Cond == ISD::SETEQ || Cond == ISD::SETNE) && isNullConstant(RHS) &&
-      LHS->getOpcode() == ISD::OR &&
-      (LHS.getOperand(0)->getOpcode() == ISD::XOR &&
-       LHS.getOperand(1)->getOpcode() == ISD::XOR) &&
-      LHS.hasOneUse() && LHS.getOperand(0)->hasOneUse() &&
-      LHS.getOperand(1)->hasOneUse()) {
-    SDValue XOR0 = LHS.getOperand(0);
-    SDValue XOR1 = LHS.getOperand(1);
-    SDValue CCVal = DAG.getConstant(AArch64CC::EQ, DL, MVT_CC);
-    EVT TstVT = LHS->getValueType(0);
-    SDValue Cmp =
-        DAG.getNode(AArch64ISD::SUBS, DL, DAG.getVTList(TstVT, MVT::i32),
-                    XOR0.getOperand(0), XOR0.getOperand(1));
-    SDValue Overflow = Cmp.getValue(1);
-    SDValue NZCVOp = DAG.getConstant(0, DL, MVT::i32);
-    SDValue CCmp = DAG.getNode(AArch64ISD::CCMP, DL, MVT_CC, XOR1.getOperand(0),
-                               XOR1.getOperand(1), NZCVOp, CCVal, Overflow);
-    // Invert CSEL's operands.
-    SDValue TVal = DAG.getConstant(1, DL, VT);
-    SDValue FVal = DAG.getConstant(0, DL, VT);
-    AArch64CC::CondCode CC = changeIntCCToAArch64CC(Cond);
-    AArch64CC::CondCode InvCC = AArch64CC::getInvertedCondCode(CC);
-    return DAG.getNode(AArch64ISD::CSEL, DL, VT, FVal, TVal,
-                       DAG.getConstant(InvCC, DL, MVT::i32), CCmp);
-  }
+  // Try to perform the memcmp when the result is tested for [in]equality with 0
+  if (!DCI.isBeforeLegalize())
+    if (SDValue V = performOrXorChainCombine(N, DAG))
+      return V;
 
   return SDValue();
 }
 
 // Replace a flag-setting operator (eg ANDS) with the generic version
 // (eg AND) if the flag is unused.
 static SDValue performFlagSettingCombine(SDNode *N,
                                          TargetLowering::DAGCombinerInfo &DCI,

llvm/test/CodeGen/AArch64/bcmp-inline-small.ll

 }
 
 define i1 @test_bs(i8* %s1, i8* %s2) optsize {
 ; CHECKN-LABEL: test_bs:
 ; CHECKN:       // %bb.0: // %entry
 ; CHECKN-NEXT:    ldp x8, x9, [x0]
 ; CHECKN-NEXT:    ldp x10, x11, [x1]
 ; CHECKN-NEXT:    ldr x12, [x0, #16]
-; CHECKN-NEXT:    ldr x13, [x1, #16]
-; CHECKN-NEXT:    ldur x14, [x0, #23]
-; CHECKN-NEXT:    eor x8, x8, x10
-; CHECKN-NEXT:    ldur x15, [x1, #23]
-; CHECKN-NEXT:    eor x9, x9, x11
-; CHECKN-NEXT:    eor x10, x12, x13
-; CHECKN-NEXT:    orr x8, x8, x9
-; CHECKN-NEXT:    eor x11, x14, x15
-; CHECKN-NEXT:    orr x9, x10, x11
-; CHECKN-NEXT:    orr x8, x8, x9
-; CHECKN-NEXT:    cmp x8, #0
+; CHECKN-NEXT:    cmp x8, x10
+; CHECKN-NEXT:    ldr x8, [x1, #16]
+; CHECKN-NEXT:    ccmp x9, x11, #0, eq
+; CHECKN-NEXT:    ldur x9, [x0, #23]
+; CHECKN-NEXT:    ldur x10, [x1, #23]
+; CHECKN-NEXT:    ccmp x12, x8, #0, eq
+; CHECKN-NEXT:    ccmp x9, x10, #0, eq
 ; CHECKN-NEXT:    cset w0, eq
 ; CHECKN-NEXT:    ret
 ;
 ; CHECKS-LABEL: test_bs:
 ; CHECKS:       // %bb.0: // %entry
 ; CHECKS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECKS-NEXT:    .cfi_def_cfa_offset 16
 ; CHECKS-NEXT:    .cfi_offset w30, -16

llvm/test/CodeGen/AArch64/bcmp.ll

 ; CHECK-NEXT:    cmp w8, w9
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 2)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
+; or (and (xor a, b), C1), (and (xor c, d), C2)
 define i1 @bcmp3(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp3:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldrh w8, [x0]
 ; CHECK-NEXT:    ldrh w9, [x1]
 ; CHECK-NEXT:    ldrb w10, [x0, #2]
 ; CHECK-NEXT:    ldrb w11, [x1, #2]
-; CHECK-NEXT:    eor w8, w8, w9
-; CHECK-NEXT:    eor w9, w10, w11
-; CHECK-NEXT:    orr w8, w8, w9
-; CHECK-NEXT:    cmp w8, #0
+; CHECK-NEXT:    cmp w8, w9
+; CHECK-NEXT:    ccmp w10, w11, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 3)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
 define i1 @bcmp4(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp4:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr w8, [x0]
 ; CHECK-NEXT:    ldr w9, [x1]
 ; CHECK-NEXT:    cmp w8, w9
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 4)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
+; or (xor a, b), (and (xor c, d), C2)
 define i1 @bcmp5(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp5:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr w8, [x0]
 ; CHECK-NEXT:    ldr w9, [x1]
 ; CHECK-NEXT:    ldrb w10, [x0, #4]
 ; CHECK-NEXT:    ldrb w11, [x1, #4]
-; CHECK-NEXT:    eor w8, w8, w9
-; CHECK-NEXT:    eor w9, w10, w11
-; CHECK-NEXT:    orr w8, w8, w9
-; CHECK-NEXT:    cmp w8, #0
+; CHECK-NEXT:    cmp w8, w9
+; CHECK-NEXT:    ccmp w10, w11, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 5)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
+; or (xor a, b), (and (xor c, d), C2)
 define i1 @bcmp6(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp6:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr w8, [x0]
 ; CHECK-NEXT:    ldr w9, [x1]
 ; CHECK-NEXT:    ldrh w10, [x0, #4]
 ; CHECK-NEXT:    ldrh w11, [x1, #4]
-; CHECK-NEXT:    eor w8, w8, w9
-; CHECK-NEXT:    eor w9, w10, w11
-; CHECK-NEXT:    orr w8, w8, w9
-; CHECK-NEXT:    cmp w8, #0
+; CHECK-NEXT:    cmp w8, w9
+; CHECK-NEXT:    ccmp w10, w11, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 6)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
+; or (xor a, b), (xor c, d)
 define i1 @bcmp7(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp7:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr w8, [x0]
 ; CHECK-NEXT:    ldr w9, [x1]
 ; CHECK-NEXT:    ldur w10, [x0, #3]
 ; CHECK-NEXT:    ldur w11, [x1, #3]
 ; CHECK-NEXT:    cmp w8, w9

 ; CHECK-NEXT:    cmp x8, x9
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 8)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
+; TODO: or (xor a, b), (and (xor c, d), C2)
 define i1 @bcmp9(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp9:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldrb w9, [x0, #8]
 ; CHECK-NEXT:    ldrb w10, [x1, #8]
 ; CHECK-NEXT:    ldr x8, [x0]
 ; CHECK-NEXT:    ldr x11, [x1]
 ; CHECK-NEXT:    eor w9, w9, w10
 ; CHECK-NEXT:    and x9, x9, #0xff
 ; CHECK-NEXT:    eor x8, x8, x11
 ; CHECK-NEXT:    orr x8, x8, x9
 ; CHECK-NEXT:    cmp x8, #0
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 9)
   %r = icmp eq i32 %cr, 0
   ret i1 %r

 }
 
 define i1 @bcmp24(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp24:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldp x8, x9, [x0]
 ; CHECK-NEXT:    ldp x10, x11, [x1]
 ; CHECK-NEXT:    ldr x12, [x0, #16]
-; CHECK-NEXT:    ldr x13, [x1, #16]
-; CHECK-NEXT:    eor x8, x8, x10
-; CHECK-NEXT:    eor x9, x9, x11
-; CHECK-NEXT:    eor x10, x12, x13
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    orr x8, x8, x10
-; CHECK-NEXT:    cmp x8, #0
+; CHECK-NEXT:    cmp x8, x10
+; CHECK-NEXT:    ldr x8, [x1, #16]
+; CHECK-NEXT:    ccmp x9, x11, #0, eq
+; CHECK-NEXT:    ccmp x12, x8, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 24)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
 define i1 @bcmp28(ptr %a, ptr %b) {
; CHECK-NEXT:    ret
   ret i1 %r
 }
 
 define i1 @bcmp38(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp38:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldp x8, x9, [x0]
 ; CHECK-NEXT:    ldp x10, x11, [x1]
-; CHECK-NEXT:    ldp x12, x13, [x0, #16]
-; CHECK-NEXT:    ldp x14, x15, [x1, #16]
-; CHECK-NEXT:    eor x8, x8, x10
-; CHECK-NEXT:    eor x9, x9, x11
-; CHECK-NEXT:    ldur x10, [x0, #30]
-; CHECK-NEXT:    orr x8, x8, x9
+; CHECK-NEXT:    cmp x8, x10
+; CHECK-NEXT:    ccmp x9, x11, #0, eq
 ; CHECK-NEXT:    ldur x11, [x1, #30]
-; CHECK-NEXT:    eor x12, x12, x14
-; CHECK-NEXT:    eor x13, x13, x15
-; CHECK-NEXT:    orr x9, x12, x13
-; CHECK-NEXT:    eor x10, x10, x11
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    orr x8, x8, x10
-; CHECK-NEXT:    cmp x8, #0
+; CHECK-NEXT:    ldp x8, x9, [x0, #16]
+; CHECK-NEXT:    ldp x12, x10, [x1, #16]
+; CHECK-NEXT:    ccmp x8, x12, #0, eq
+; CHECK-NEXT:    ldur x8, [x0, #30]
+; CHECK-NEXT:    ccmp x9, x10, #0, eq
+; CHECK-NEXT:    ccmp x8, x11, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 38)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
 define i1 @bcmp45(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp45:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldp x8, x9, [x0]
 ; CHECK-NEXT:    ldp x10, x11, [x1]
-; CHECK-NEXT:    ldp x12, x13, [x0, #16]
-; CHECK-NEXT:    ldp x14, x15, [x1, #16]
-; CHECK-NEXT:    eor x8, x8, x10
-; CHECK-NEXT:    eor x9, x9, x11
-; CHECK-NEXT:    ldr x16, [x0, #32]
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    ldr x17, [x1, #32]
-; CHECK-NEXT:    ldur x18, [x0, #37]
-; CHECK-NEXT:    eor x10, x12, x14
-; CHECK-NEXT:    ldur x0, [x1, #37]
-; CHECK-NEXT:    eor x11, x13, x15
-; CHECK-NEXT:    eor x12, x16, x17
-; CHECK-NEXT:    orr x9, x10, x11
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    eor x13, x18, x0
-; CHECK-NEXT:    orr x10, x12, x13
-; CHECK-NEXT:    orr x8, x8, x10
-; CHECK-NEXT:    cmp x8, #0
+; CHECK-NEXT:    cmp x8, x10
+; CHECK-NEXT:    ccmp x9, x11, #0, eq
+; CHECK-NEXT:    ldr x11, [x1, #32]
+; CHECK-NEXT:    ldp x8, x9, [x0, #16]
+; CHECK-NEXT:    ldp x12, x10, [x1, #16]
+; CHECK-NEXT:    ccmp x8, x12, #0, eq
+; CHECK-NEXT:    ldr x8, [x0, #32]
+; CHECK-NEXT:    ccmp x9, x10, #0, eq
+; CHECK-NEXT:    ldur x9, [x0, #37]
+; CHECK-NEXT:    ldur x10, [x1, #37]
+; CHECK-NEXT:    ccmp x8, x11, #0, eq
+; CHECK-NEXT:    ccmp x9, x10, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 45)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
+; Although the large cmp chain may be not profitable on high end CPU, we
+; believe it is better on most cpus, so perform the transform now.
+; 8 xor + 7 or + 1 cmp only need 6 cycles on a 4 width ALU port machine
+;   2 cycle for xor
+;   3 cycle for or
+;   1 cycle for cmp
 define i1 @bcmp64(ptr %a, ptr %b) {
 ; CHECK-LABEL: bcmp64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldp x8, x9, [x0]
 ; CHECK-NEXT:    ldp x10, x11, [x1]
-; CHECK-NEXT:    ldp x12, x13, [x0, #16]
-; CHECK-NEXT:    ldp x14, x15, [x1, #16]
-; CHECK-NEXT:    eor x8, x8, x10
-; CHECK-NEXT:    eor x9, x9, x11
-; CHECK-NEXT:    ldp x16, x17, [x0, #32]
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    ldp x18, x2, [x1, #32]
-; CHECK-NEXT:    eor x12, x12, x14
-; CHECK-NEXT:    eor x13, x13, x15
-; CHECK-NEXT:    ldp x3, x0, [x0, #48]
-; CHECK-NEXT:    orr x9, x12, x13
-; CHECK-NEXT:    ldp x10, x11, [x1, #48]
-; CHECK-NEXT:    eor x14, x16, x18
-; CHECK-NEXT:    eor x15, x17, x2
-; CHECK-NEXT:    orr x12, x14, x15
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    eor x10, x3, x10
-; CHECK-NEXT:    eor x11, x0, x11
-; CHECK-NEXT:    orr x10, x10, x11
-; CHECK-NEXT:    orr x9, x12, x10
-; CHECK-NEXT:    orr x8, x8, x9
-; CHECK-NEXT:    cmp x8, #0
+; CHECK-NEXT:    cmp x8, x10
+; CHECK-NEXT:    ccmp x9, x11, #0, eq
+; CHECK-NEXT:    ldp x8, x9, [x0, #16]
+; CHECK-NEXT:    ldp x12, x10, [x1, #16]
+; CHECK-NEXT:    ccmp x8, x12, #0, eq
+; CHECK-NEXT:    ldp x8, x11, [x1, #32]
+; CHECK-NEXT:    ccmp x9, x10, #0, eq
+; CHECK-NEXT:    ldp x9, x10, [x0, #32]
+; CHECK-NEXT:    ccmp x9, x8, #0, eq
+; CHECK-NEXT:    ccmp x10, x11, #0, eq
+; CHECK-NEXT:    ldp x9, x10, [x0, #48]
+; CHECK-NEXT:    ldp x8, x11, [x1, #48]
+; CHECK-NEXT:    ccmp x9, x8, #0, eq
+; CHECK-NEXT:    ccmp x10, x11, #0, eq
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %cr = call i32 @bcmp(ptr %a, ptr %b, i64 64)
   %r = icmp eq i32 %cr, 0
   ret i1 %r
 }
 
 define i1 @bcmp89(ptr %a, ptr %b) {