[AArch64] Try to re-use extended operand for SETCC with vector ops.

[fhahn]

Summary

Try to re-use an already extended operand for SetCC with vector operands
feeding an extended select. Doing so avoids requiring another full
extension of the SET_CC result when lowering the select.

This improves lowering for certain extend/cmp/select patterns operating.
For example with v16i8, this replaces 6 instructions for the extra extension
with 4 separate selects.

This improves the generated code for loops like the one below in
combination with D96522.

int foo(uint8_t *p, int N) {
  unsigned long long sum = 0;
  for (int i = 0; i < N ; i++, p++) {
    unsigned int v = *p;
    sum += (v < 127) ? v : 256 - v;
  }
  return sum;
}

https://clang.godbolt.org/z/Wco866MjY

On the AArch64 cores I have access to, the patch improves performance of
the vector loop by ~10%.

This could be generalized per follow-ups, but the initial version
targets one of the more important cases in combination with D96522.

Alive2 modeling:

• sext EQ https://alive2.llvm.org/ce/z/5upBvb
• sext NE https://alive2.llvm.org/ce/z/zbEcJp
• zext EQ https://alive2.llvm.org/ce/z/_xMwof
• zext NE https://alive2.llvm.org/ce/z/5FwKfc
• zext unsigned predicate: https://alive2.llvm.org/ce/z/iEwLU3
• sext signed predicate: https://alive2.llvm.org/ce/z/aMBega

[dmgreen]

Sorry for the delay. It took a while to find the time to figure out what was wrong, but it turned out that was in the other patch.

The change sounds OK to me, although it might be useful to generalize it to handle more types. This seems like it is useful in a few more cases: https://godbolt.org/z/fo8MMrqrE. I'm not sure if this generalises to not just select uses.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

18075

Do you have a test for multiple uses?

19910

Whitespace

[fhahn]

Rebased on top of new test case, removed whitespace change.

In D120481#3353908, @dmgreen wrote:

Sorry for the delay. It took a while to find the time to figure out what was wrong, but it turned out that was in the other patch.

The change sounds OK to me, although it might be useful to generalize it to handle more types. This seems like it is useful in a few more cases: https://godbolt.org/z/fo8MMrqrE. I'm not sure if this generalises to not just select uses.

Thanks for taking a look! Agreed that it will be helpful for a couple of other type combinations. But maybe the generalization should be done as follow-ups?

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

18075

I added one in 3836003e87ee, thanks!

19910

Removed, thanks!

[dmgreen]

Thanks for taking a look! Agreed that it will be helpful for a couple of other type combinations. But maybe the generalization should be done as follow-ups?

Any reason to do that in a followup, as opposed to doing it here?

[fhahn]

Update to lift restrictions on specific vector types.

In D120481#3366872, @dmgreen wrote:

Thanks for taking a look! Agreed that it will be helpful for a couple of other type combinations. But maybe the generalization should be done as follow-ups?

Any reason to do that in a followup, as opposed to doing it here?

Mostly time constraints :)

I finally had some time to generalize this to support vector types in general

[dmgreen]

This sounds good as far as I can tell. If we are dealing with a dag combine, do we need to worry about odd types at all? And I think we can include eq and ne too, if alive is to be believed.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

18082

I think this can skip the getScalarType call: UseMVT.getScalarSizeInBits() <= Op0MVT.getScalarSizeInBits()

18104

Could these include eq and ne conditions too?

llvm/test/CodeGen/AArch64/vselect-ext.ll

156–157

Can you remove all of these with the lower cases.

[fhahn]

In D120481#3632283, @dmgreen wrote:

This sounds good as far as I can tell. If we are dealing with a dag combine, do we need to worry about odd types at all?

Do you mean worry in terms of whether it is optimal? There are a few test cases with odd types, and it seems like they are handled OK on that front.

And I think we can include eq and ne too, if alive is to be believed.

Thanks, I added support for EQ/NE. The type of extend shouldn't matter, as long as we chose the existing one.

• sext EQ https://alive2.llvm.org/ce/z/5upBvb
• sext NE https://alive2.llvm.org/ce/z/zbEcJp
• zext EQ https://alive2.llvm.org/ce/z/_xMwof
• zext NE https://alive2.llvm.org/ce/z/5FwKfc

[dmgreen]

In D120481#3634566, @fhahn wrote:

In D120481#3632283, @dmgreen wrote:

This sounds good as far as I can tell. If we are dealing with a dag combine, do we need to worry about odd types at all?

Do you mean worry in terms of whether it is optimal? There are a few test cases with odd types, and it seems like they are handled OK on that front.

I meant for awkward types like i23 and i128. They don't have to be optimal, so long as they work OK. Just so long as we have a couple of tests, to make sure it doesnt run into problems.

The change sounds good with a small simplification suggestion. LGTM

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

18103

I think this might be able to combine the if blocks together for the ne/eq conditions:

if (Op0SExt && (isSignedIntSetCC(CC) || isIntEqualitySetCC(CC))) {
  Op0ExtV = SDValue(Op0SExt, 0);
  Op1ExtV = DAG.getNode(ISD::SIGN_EXTEND, DL, UseMVT, Op->getOperand(1));
} else if (Op0ZExt && (isUnsignedIntSetCC(CC) || isIntEqualitySetCC(CC))) {
  Op0ExtV = SDValue(Op0ZExt, 0);
  Op1ExtV = DAG.getNode(ISD::ZERO_EXTEND, DL, UseMVT, Op->getOperand(1));
} else
  return SDValue();

[fhahn]

Rebase on top of additional tests.

[fhahn]

Simplify check as suggested.

[fhahn]

In D120481#3634726, @dmgreen wrote:

In D120481#3634566, @fhahn wrote:

In D120481#3632283, @dmgreen wrote:

This sounds good as far as I can tell. If we are dealing with a dag combine, do we need to worry about odd types at all?

Do you mean worry in terms of whether it is optimal? There are a few test cases with odd types, and it seems like they are handled OK on that front.

I meant for awkward types like i23 and i128. They don't have to be optimal, so long as they work OK. Just so long as we have a couple of tests, to make sure it doesnt run into problems.

Ah got it! I added a few tests with vectors of i13 and i15. Looks like this handled well, because they are first converted to legal vector types.

The change sounds good with a small simplification suggestion. LGTM

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

18082

Thanks, simplified!

18103

Simplified as suggested, thanks!

18104

Yes, updated!

llvm/test/CodeGen/AArch64/vselect-ext.ll

156–157

Yeah I removed those. They were added by accident :(

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

static SDValue performCSELCombine(SDNode *N,
   // CSEL 0, cttz(X), eq(X, 0) -> AND cttz bitwidth-1
   // CSEL cttz(X), 0, ne(X, 0) -> AND cttz bitwidth-1
   if (SDValue Folded = foldCSELofCTTZ(N, DAG))
     return Folded;
 
   return performCONDCombine(N, DCI, DAG, 2, 3);
 }
 
+// Try to re-use an already extended operand of a vector SetCC feeding a
+// extended select. Doing so avoids requiring another full extension of the
+// SET_CC result when lowering the select.
+static SDValue tryToWidenSetCCOperands(SDNode *Op, SelectionDAG &DAG) {
+  EVT Op0MVT = Op->getOperand(0).getValueType();
+  if (!Op0MVT.isVector() || Op->use_empty())
+    return SDValue();
+
+  // Make sure that all uses of Op are VSELECTs with result matching types where
+  // the result type has a larger element type than the SetCC operand.
+  SDNode *FirstUse = *Op->use_begin();
+  if (FirstUse->getOpcode() != ISD::VSELECT)
+    return SDValue();
+  EVT UseMVT = FirstUse->getValueType(0);
+  if (UseMVT.getScalarSizeInBits() <= Op0MVT.getScalarSizeInBits())
+    return SDValue();
+  if (any_of(Op->uses(), [&UseMVT](const SDNode *N) {
+        return N->getOpcode() != ISD::VSELECT || N->getValueType(0) != UseMVT;
+      }))
+    return SDValue();
+
+  APInt V;
+  if (!ISD::isConstantSplatVector(Op->getOperand(1).getNode(), V))
+    return SDValue();
+
+  SDLoc DL(Op);
+  SDValue Op0ExtV;
+  SDValue Op1ExtV;
+  ISD::CondCode CC = cast<CondCodeSDNode>(Op->getOperand(2))->get();
+  // Check if the first operand of the SET_CC is already extended. If it is,
+  // split the SET_CC and re-use the extended version of the operand.
+  SDNode *Op0SExt = DAG.getNodeIfExists(ISD::SIGN_EXTEND, DAG.getVTList(UseMVT),
+                                        Op->getOperand(0));
+  SDNode *Op0ZExt = DAG.getNodeIfExists(ISD::ZERO_EXTEND, DAG.getVTList(UseMVT),
+                                        Op->getOperand(0));
+  if (Op0SExt && isSignedIntSetCC(CC)) {
+    Op0ExtV = SDValue(Op0SExt, 0);
+    Op1ExtV = DAG.getNode(ISD::SIGN_EXTEND, DL, UseMVT, Op->getOperand(1));
+  } else if (Op0ZExt && isUnsignedIntSetCC(CC)) {
+    Op0ExtV = SDValue(Op0ZExt, 0);
+    Op1ExtV = DAG.getNode(ISD::ZERO_EXTEND, DL, UseMVT, Op->getOperand(1));
+  } else if (CC == ISD::SETNE || CC == ISD::SETEQ) {
+    if (Op0SExt) {
+      Op0ExtV = SDValue(Op0SExt, 0);
+      Op1ExtV = DAG.getNode(ISD::SIGN_EXTEND, DL, UseMVT, Op->getOperand(1));
+    } else if (Op0ZExt) {
+      Op0ExtV = SDValue(Op0ZExt, 0);
+      Op1ExtV = DAG.getNode(ISD::ZERO_EXTEND, DL, UseMVT, Op->getOperand(1));
+    } else
+      return SDValue();
+  } else
+    return SDValue();
+
+  return DAG.getNode(ISD::SETCC, DL, UseMVT.changeVectorElementType(MVT::i1),
+                     Op0ExtV, Op1ExtV, Op->getOperand(2));
+}
+
 static SDValue performSETCCCombine(SDNode *N, SelectionDAG &DAG) {
   assert(N->getOpcode() == ISD::SETCC && "Unexpected opcode!");
   SDValue LHS = N->getOperand(0);
   SDValue RHS = N->getOperand(1);
   ISD::CondCode Cond = cast<CondCodeSDNode>(N->getOperand(2))->get();
   SDLoc DL(N);
   EVT VT = N->getValueType(0);
 
+  if (SDValue V = tryToWidenSetCCOperands(N, DAG))
+    return V;
+
   // setcc (csel 0, 1, cond, X), 1, ne ==> csel 0, 1, !cond, X
   if (Cond == ISD::SETNE && isOneConstant(RHS) &&
       LHS->getOpcode() == AArch64ISD::CSEL &&
       isNullConstant(LHS->getOperand(0)) && isOneConstant(LHS->getOperand(1)) &&
       LHS->hasOneUse()) {
     // Invert CSEL's condition.
     auto *OpCC = cast<ConstantSDNode>(LHS.getOperand(2));
     auto OldCond = static_cast<AArch64CC::CondCode>(OpCC->getZExtValue());
void AArch64TargetLowering::ReplaceNodeResults(
   case ISD::VECREDUCE_UMAX:
   case ISD::VECREDUCE_UMIN:
     Results.push_back(LowerVECREDUCE(SDValue(N, 0), DAG));
     return;
   case ISD::ADD:
   case ISD::FADD:
     ReplaceAddWithADDP(N, Results, DAG, Subtarget);
     return;
 
   case ISD::CTPOP:
     if (SDValue Result = LowerCTPOP(SDValue(N, 0), DAG))
       Results.push_back(Result);
     return;
   case AArch64ISD::SADDV:
     ReplaceReductionResults(N, Results, DAG, ISD::ADD, AArch64ISD::SADDV);
     return;
   case AArch64ISD::UADDV:

llvm/test/CodeGen/AArch64/vselect-ext.ll

entry:
   %cmp = icmp sgt <16 x i8> %a, <i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10>
   %sel = select <16 x i1> %cmp, <16 x i32> %ext, <16 x i32> zeroinitializer
   ret <16 x i32> %sel
 }
 
 define <8 x i64> @same_zext_used_in_cmp_unsigned_pred_and_select_v8i64(<8 x i8> %a) {
 ; CHECK-LABEL: same_zext_used_in_cmp_unsigned_pred_and_select_v8i64:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.8b v1, #10
-; CHECK-NEXT:    ushll.8h v2, v0, #0
-; CHECK-NEXT:    cmhi.8b v0, v0, v1
-; CHECK-NEXT:    ushll.4s v1, v2, #0
-; CHECK-NEXT:    sshll.8h v0, v0, #0
-; CHECK-NEXT:    ushll2.4s v2, v2, #0
-; CHECK-NEXT:    sshll.4s v3, v0, #0
-; CHECK-NEXT:    sshll2.4s v0, v0, #0
-; CHECK-NEXT:    ushll.2d v4, v1, #0
-; CHECK-NEXT:    ushll.2d v5, v2, #0
-; CHECK-NEXT:    ushll2.2d v1, v1, #0
-; CHECK-NEXT:    ushll2.2d v2, v2, #0
-; CHECK-NEXT:    sshll.2d v6, v3, #0
-; CHECK-NEXT:    sshll.2d v7, v0, #0
-; CHECK-NEXT:    sshll2.2d v0, v0, #0
-; CHECK-NEXT:    sshll2.2d v16, v3, #0
-; CHECK-NEXT:    and.16b v3, v2, v0
-; CHECK-NEXT:    and.16b v1, v1, v16
-; CHECK-NEXT:    and.16b v2, v5, v7
-; CHECK-NEXT:    and.16b v0, v4, v6
+; CHECK-NEXT:    ushll.8h v0, v0, #0
+; CHECK-NEXT:    mov w8, #10
+; CHECK-NEXT:    ushll2.4s v2, v0, #0
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    dup.2d v1, x8
+; CHECK-NEXT:    ushll2.2d v3, v2, #0
+; CHECK-NEXT:    ushll2.2d v4, v0, #0
+; CHECK-NEXT:    ushll.2d v0, v0, #0
+; CHECK-NEXT:    ushll.2d v2, v2, #0
+; CHECK-NEXT:    cmhi.2d v5, v0, v1
+; CHECK-NEXT:    cmhi.2d v6, v2, v1
+; CHECK-NEXT:    cmhi.2d v7, v3, v1
+; CHECK-NEXT:    cmhi.2d v1, v4, v1
+; CHECK-NEXT:    and.16b v3, v3, v7
+; CHECK-NEXT:    and.16b v1, v4, v1
+; CHECK-NEXT:    and.16b v2, v2, v6
+; CHECK-NEXT:    and.16b v0, v0, v5
 ; CHECK-NEXT:    ret
   %ext = zext <8 x i8> %a to <8 x i64>
   %cmp = icmp ugt <8 x i8> %a, <i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10>
   %sel = select <8 x i1> %cmp, <8 x i64> %ext, <8 x i64> zeroinitializer
   ret <8 x i64> %sel
 }
 
 
 define <16 x i32> @same_zext_used_in_cmp_unsigned_pred_and_select_v16i32(<16 x i8> %a) {
 ; CHECK-LABEL: same_zext_used_in_cmp_unsigned_pred_and_select_v16i32:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.16b v1, #10
-; CHECK-NEXT:    ushll.8h v2, v0, #0
-; CHECK-NEXT:    ushll2.8h v3, v0, #0
-; CHECK-NEXT:    ushll.4s v4, v2, #0
-; CHECK-NEXT:    cmhi.16b v0, v0, v1
-; CHECK-NEXT:    ushll.4s v5, v3, #0
-; CHECK-NEXT:    ushll2.4s v1, v3, #0
-; CHECK-NEXT:    sshll.8h v3, v0, #0
-; CHECK-NEXT:    sshll2.8h v0, v0, #0
-; CHECK-NEXT:    ushll2.4s v2, v2, #0
-; CHECK-NEXT:    sshll.4s v6, v3, #0
-; CHECK-NEXT:    sshll.4s v7, v0, #0
-; CHECK-NEXT:    sshll2.4s v0, v0, #0
-; CHECK-NEXT:    sshll2.4s v16, v3, #0
-; CHECK-NEXT:    and.16b v3, v1, v0
-; CHECK-NEXT:    and.16b v1, v2, v16
-; CHECK-NEXT:    and.16b v2, v5, v7
-; CHECK-NEXT:    and.16b v0, v4, v6
+; CHECK-NEXT:    movi.4s v1, #10
+; CHECK-NEXT:    ushll2.8h v2, v0, #0
+; CHECK-NEXT:    ushll.8h v0, v0, #0
+; CHECK-NEXT:    ushll2.4s v3, v2, #0
+; CHECK-NEXT:    ushll2.4s v4, v0, #0
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    ushll.4s v2, v2, #0
+; CHECK-NEXT:    cmhi.4s v5, v0, v1
+; CHECK-NEXT:    cmhi.4s v6, v2, v1
+; CHECK-NEXT:    cmhi.4s v7, v3, v1
+; CHECK-NEXT:    cmhi.4s v1, v4, v1
+; CHECK-NEXT:    and.16b v3, v3, v7
+; CHECK-NEXT:    and.16b v1, v4, v1
+; CHECK-NEXT:    and.16b v2, v2, v6
+; CHECK-NEXT:    and.16b v0, v0, v5
 ; CHECK-NEXT:    ret
   %ext = zext <16 x i8> %a to <16 x i32>
   %cmp = icmp ugt <16 x i8> %a, <i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10>
   %sel = select <16 x i1> %cmp, <16 x i32> %ext, <16 x i32> zeroinitializer
   ret <16 x i32> %sel
 }
 
 define <8 x i32> @same_zext_used_in_cmp_unsigned_pred_and_select_v8i32(<8 x i8> %a) {
 ; CHECK-LABEL: same_zext_used_in_cmp_unsigned_pred_and_select_v8i32:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.8b v1, #10
-; CHECK-NEXT:    ushll.8h v2, v0, #0
-; CHECK-NEXT:    ushll.4s v3, v2, #0
-; CHECK-NEXT:    cmhi.8b v0, v0, v1
-; CHECK-NEXT:    ushll2.4s v1, v2, #0
-; CHECK-NEXT:    sshll.8h v0, v0, #0
-; CHECK-NEXT:    sshll2.4s v2, v0, #0
-; CHECK-NEXT:    sshll.4s v0, v0, #0
-; CHECK-NEXT:    and.16b v1, v1, v2
-; CHECK-NEXT:    and.16b v0, v3, v0
+; CHECK-NEXT:    movi.4s v1, #10
+; CHECK-NEXT:    ushll.8h v0, v0, #0
+; CHECK-NEXT:    ushll2.4s v2, v0, #0
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    cmhi.4s v3, v2, v1
+; CHECK-NEXT:    cmhi.4s v4, v0, v1
+; CHECK-NEXT:    and.16b v1, v2, v3
+; CHECK-NEXT:    and.16b v0, v0, v4
 ; CHECK-NEXT:    ret
   %ext = zext <8 x i8> %a to <8 x i32>
   %cmp = icmp ugt <8 x i8> %a, <i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10>
   %sel = select <8 x i1> %cmp, <8 x i32> %ext, <8 x i32> zeroinitializer
   ret <8 x i32> %sel
 }
 
 define <8 x i32> @same_zext_used_in_cmp_unsigned_pred_and_select_v8i32_2(<8 x i16> %a) {
 ; CHECK-LABEL: same_zext_used_in_cmp_unsigned_pred_and_select_v8i32_2:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.8h v1, #10
-; CHECK-NEXT:    ushll.4s v2, v0, #0
-; CHECK-NEXT:    cmhi.8h v1, v0, v1
-; CHECK-NEXT:    ushll2.4s v0, v0, #0
-; CHECK-NEXT:    sshll2.4s v3, v1, #0
-; CHECK-NEXT:    sshll.4s v4, v1, #0
-; CHECK-NEXT:    and.16b v1, v0, v3
-; CHECK-NEXT:    and.16b v0, v2, v4
+; CHECK-NEXT:    movi.4s v1, #10
+; CHECK-NEXT:    ushll2.4s v2, v0, #0
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    cmhi.4s v3, v2, v1
+; CHECK-NEXT:    cmhi.4s v4, v0, v1
+; CHECK-NEXT:    and.16b v1, v2, v3
+; CHECK-NEXT:    and.16b v0, v0, v4
 ; CHECK-NEXT:    ret
   %ext = zext <8 x i16> %a to <8 x i32>
   %cmp = icmp ugt <8 x i16> %a, <i16 10, i16 10, i16 10, i16 10, i16 10, i16 10, i16 10, i16 10>
   %sel = select <8 x i1> %cmp, <8 x i32> %ext, <8 x i32> zeroinitializer
   ret <8 x i32> %sel
 }
 
 
 define <8 x i32> @same_zext_used_in_cmp_unsigned_pred_and_select_v8i32_from_v8i15(<8 x i15> %a) {
 ; CHECK-LABEL: same_zext_used_in_cmp_unsigned_pred_and_select_v8i32_from_v8i15:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.8h v1, #10
+; CHECK-NEXT:    movi.4s v1, #10
 ; CHECK-NEXT:    bic.8h v0, #128, lsl #8
-; CHECK-NEXT:    ushll.4s v2, v0, #0
-; CHECK-NEXT:    cmhi.8h v1, v0, v1
-; CHECK-NEXT:    ushll2.4s v0, v0, #0
-; CHECK-NEXT:    ushll2.4s v3, v1, #0
-; CHECK-NEXT:    ushll.4s v1, v1, #0
-; CHECK-NEXT:    shl.4s v3, v3, #17
-; CHECK-NEXT:    shl.4s v1, v1, #17
-; CHECK-NEXT:    sshr.4s v3, v3, #17
-; CHECK-NEXT:    sshr.4s v4, v1, #17
-; CHECK-NEXT:    and.16b v1, v0, v3
-; CHECK-NEXT:    and.16b v0, v2, v4
+; CHECK-NEXT:    ushll2.4s v2, v0, #0
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    cmhi.4s v3, v2, v1
+; CHECK-NEXT:    cmhi.4s v4, v0, v1
+; CHECK-NEXT:    and.16b v1, v2, v3
+; CHECK-NEXT:    and.16b v0, v0, v4
 ; CHECK-NEXT:    ret
   %ext = zext <8 x i15> %a to <8 x i32>
   %cmp = icmp ugt <8 x i15> %a, <i15 10, i15 10, i15 10, i15 10, i15 10, i15 10, i15 10, i15 10>
   %sel = select <8 x i1> %cmp, <8 x i32> %ext, <8 x i32> zeroinitializer
   ret <8 x i32> %sel
 }
 
 define <7 x i32> @same_zext_used_in_cmp_unsigned_pred_and_select_v7i32(<7 x i16> %a) {
; CHECK-NEXT:    .loh AdrpLdr Lloh0, Lloh1
   %cmp = icmp ugt <3 x i8> %a, <i8 10, i8 10, i8 10>
   %sel = select <3 x i1> %cmp, <3 x i32> %ext, <3 x i32> zeroinitializer
   ret <3 x i32> %sel
 }
 
 define <4 x i32> @same_zext_used_in_cmp_unsigned_pred_and_select_v4i32(<4 x i16> %a) {
 ; CHECK-LABEL: same_zext_used_in_cmp_unsigned_pred_and_select_v4i32:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.4h v1, #10
-; CHECK-NEXT:    ushll.4s v2, v0, #0
-; CHECK-NEXT:    cmhi.4h v0, v0, v1
-; CHECK-NEXT:    sshll.4s v0, v0, #0
-; CHECK-NEXT:    and.16b v0, v2, v0
+; CHECK-NEXT:    movi.4s v1, #10
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    cmhi.4s v1, v0, v1
+; CHECK-NEXT:    and.16b v0, v0, v1
 ; CHECK-NEXT:    ret
   %ext = zext <4 x i16> %a to <4 x i32>
   %cmp = icmp ugt <4 x i16> %a, <i16 10, i16 10, i16 10, i16 10>
   %sel = select <4 x i1> %cmp, <4 x i32> %ext, <4 x i32> zeroinitializer
   ret <4 x i32> %sel
 }
 
 define <2 x i32> @same_zext_used_in_cmp_unsigned_pred_and_select_v2i32(<2 x i16> %a) {
; CHECK-NEXT:    ret
   %cmp = icmp ugt <2 x i16> %a, <i16 10, i16 10>
   %sel = select <2 x i1> %cmp, <2 x i32> %ext, <2 x i32> zeroinitializer
   ret <2 x i32> %sel
 }
 
 define <8 x i32> @same_zext_used_in_cmp_eq_and_select_v8i32(<8 x i16> %a) {
 ; CHECK-LABEL: same_zext_used_in_cmp_eq_and_select_v8i32:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.8h v1, #10
-; CHECK-NEXT:    ushll.4s v2, v0, #0
-; CHECK-NEXT:    cmeq.8h v1, v0, v1
-; CHECK-NEXT:    ushll2.4s v0, v0, #0
-; CHECK-NEXT:    sshll2.4s v3, v1, #0
-; CHECK-NEXT:    sshll.4s v4, v1, #0
-; CHECK-NEXT:    and.16b v1, v0, v3
-; CHECK-NEXT:    and.16b v0, v2, v4
+; CHECK-NEXT:    movi.4s v1, #10
+; CHECK-NEXT:    ushll2.4s v2, v0, #0
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    cmeq.4s v3, v2, v1
+; CHECK-NEXT:    cmeq.4s v4, v0, v1
+; CHECK-NEXT:    and.16b v1, v2, v3
+; CHECK-NEXT:    and.16b v0, v0, v4
 ; CHECK-NEXT:    ret
   %ext = zext <8 x i16> %a to <8 x i32>
   %cmp = icmp eq <8 x i16> %a, <i16 10, i16 10, i16 10, i16 10, i16 10, i16 10, i16 10, i16 10>
   %sel = select <8 x i1> %cmp, <8 x i32> %ext, <8 x i32> zeroinitializer
   ret <8 x i32> %sel
 }
 
 define <8 x i32> @same_zext_used_in_cmp_eq_and_select_v8i32_from_v8i13(<8 x i13> %a) {
 ; CHECK-LABEL: same_zext_used_in_cmp_eq_and_select_v8i32_from_v8i13:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.8h v1, #10
+; CHECK-NEXT:    movi.4s v1, #10
 ; CHECK-NEXT:    bic.8h v0, #224, lsl #8
-; CHECK-NEXT:    ushll.4s v2, v0, #0
-; CHECK-NEXT:    cmeq.8h v1, v0, v1
-; CHECK-NEXT:    ushll2.4s v0, v0, #0
-; CHECK-NEXT:    ushll2.4s v3, v1, #0
-; CHECK-NEXT:    ushll.4s v1, v1, #0
-; CHECK-NEXT:    shl.4s v3, v3, #19
-; CHECK-NEXT:    shl.4s v1, v1, #19
-; CHECK-NEXT:    sshr.4s v3, v3, #19
-; CHECK-NEXT:    sshr.4s v4, v1, #19
-; CHECK-NEXT:    and.16b v1, v0, v3
-; CHECK-NEXT:    and.16b v0, v2, v4
+; CHECK-NEXT:    ushll2.4s v2, v0, #0
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    cmeq.4s v3, v2, v1
+; CHECK-NEXT:    cmeq.4s v4, v0, v1
+; CHECK-NEXT:    and.16b v1, v2, v3
+; CHECK-NEXT:    and.16b v0, v0, v4
 ; CHECK-NEXT:    ret
   %ext = zext <8 x i13> %a to <8 x i32>
   %cmp = icmp eq <8 x i13> %a, <i13 10, i13 10, i13 10, i13 10, i13 10, i13 10, i13 10, i13 10>
   %sel = select <8 x i1> %cmp, <8 x i32> %ext, <8 x i32> zeroinitializer
   ret <8 x i32> %sel
 }
 
 define <16 x i32> @same_zext_used_in_cmp_ne_and_select_v8i32(<16 x i8> %a) {
 ; CHECK-LABEL: same_zext_used_in_cmp_ne_and_select_v8i32:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.16b v1, #10
-; CHECK-NEXT:    ushll.8h v2, v0, #0
-; CHECK-NEXT:    ushll2.8h v3, v0, #0
-; CHECK-NEXT:    ushll.4s v4, v2, #0
-; CHECK-NEXT:    cmeq.16b v0, v0, v1
-; CHECK-NEXT:    ushll.4s v5, v3, #0
-; CHECK-NEXT:    mvn.16b v0, v0
-; CHECK-NEXT:    ushll2.4s v1, v3, #0
-; CHECK-NEXT:    sshll.8h v3, v0, #0
-; CHECK-NEXT:    sshll2.8h v0, v0, #0
-; CHECK-NEXT:    ushll2.4s v2, v2, #0
-; CHECK-NEXT:    sshll.4s v6, v3, #0
-; CHECK-NEXT:    sshll.4s v7, v0, #0
-; CHECK-NEXT:    sshll2.4s v0, v0, #0
-; CHECK-NEXT:    sshll2.4s v16, v3, #0
-; CHECK-NEXT:    and.16b v3, v1, v0
-; CHECK-NEXT:    and.16b v1, v2, v16
-; CHECK-NEXT:    and.16b v2, v5, v7
-; CHECK-NEXT:    and.16b v0, v4, v6
+; CHECK-NEXT:    movi.4s v1, #10
+; CHECK-NEXT:    ushll2.8h v2, v0, #0
+; CHECK-NEXT:    ushll.8h v0, v0, #0
+; CHECK-NEXT:    ushll2.4s v3, v2, #0
+; CHECK-NEXT:    ushll2.4s v4, v0, #0
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    ushll.4s v2, v2, #0
+; CHECK-NEXT:    cmeq.4s v5, v0, v1
+; CHECK-NEXT:    cmeq.4s v6, v2, v1
+; CHECK-NEXT:    cmeq.4s v7, v3, v1
+; CHECK-NEXT:    cmeq.4s v1, v4, v1
+; CHECK-NEXT:    bic.16b v3, v3, v7
+; CHECK-NEXT:    bic.16b v1, v4, v1
+; CHECK-NEXT:    bic.16b v2, v2, v6
+; CHECK-NEXT:    bic.16b v0, v0, v5
 ; CHECK-NEXT:    ret
   %ext = zext <16 x i8> %a to <16 x i32>
   %cmp = icmp ne <16 x i8> %a, <i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10>
   %sel = select <16 x i1> %cmp, <16 x i32> %ext, <16 x i32> zeroinitializer
   ret <16 x i32> %sel
 }
 
 ; A variation of @same_zext_used_in_cmp_unsigned_pred_and_select, with with
entry:
   %sel.2 = select <16 x i1> %cmp, <16 x i64> %v, <16 x i64> zeroinitializer
   store <16 x i64> %sel.2, <16 x i64>* %ptr
   ret <16 x i32> %sel
 }
 
 define <16 x i32> @same_sext_used_in_cmp_signed_pred_and_select_v16i32(<16 x i8> %a) {
 ; CHECK-LABEL: same_sext_used_in_cmp_signed_pred_and_select_v16i32:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:    movi.16b v1, #10
-; CHECK-NEXT:    sshll.8h v3, v0, #0
+; CHECK-NEXT:    movi.4s v1, #10
 ; CHECK-NEXT:    sshll2.8h v2, v0, #0
-; CHECK-NEXT:    cmgt.16b v0, v0, v1
-; CHECK-NEXT:    ext.16b v1, v3, v3, #8
-; CHECK-NEXT:    sshll.8h v5, v0, #0
-; CHECK-NEXT:    sshll2.8h v0, v0, #0
-; CHECK-NEXT:    ext.16b v4, v2, v2, #8
-; CHECK-NEXT:    ext.16b v6, v5, v5, #8
-; CHECK-NEXT:    ext.16b v7, v0, v0, #8
-; CHECK-NEXT:    and.8b v0, v2, v0
-; CHECK-NEXT:    sshll.4s v2, v0, #0
-; CHECK-NEXT:    and.8b v0, v3, v5
-; CHECK-NEXT:    and.8b v1, v1, v6
-; CHECK-NEXT:    and.8b v3, v4, v7
+; CHECK-NEXT:    sshll.8h v0, v0, #0
+; CHECK-NEXT:    sshll2.4s v3, v2, #0
+; CHECK-NEXT:    sshll2.4s v4, v0, #0
 ; CHECK-NEXT:    sshll.4s v0, v0, #0
-; CHECK-NEXT:    sshll.4s v1, v1, #0
-; CHECK-NEXT:    sshll.4s v3, v3, #0
+; CHECK-NEXT:    sshll.4s v2, v2, #0
+; CHECK-NEXT:    cmgt.4s v5, v0, v1
+; CHECK-NEXT:    cmgt.4s v6, v2, v1
+; CHECK-NEXT:    cmgt.4s v7, v3, v1
+; CHECK-NEXT:    cmgt.4s v1, v4, v1
+; CHECK-NEXT:    and.16b v3, v3, v7
+; CHECK-NEXT:    and.16b v1, v4, v1
+; CHECK-NEXT:    and.16b v2, v2, v6
+; CHECK-NEXT:    and.16b v0, v0, v5
 ; CHECK-NEXT:    ret
 entry:
   %ext = sext <16 x i8> %a to <16 x i32>
   %cmp = icmp sgt <16 x i8> %a, <i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10>
   %sel = select <16 x i1> %cmp, <16 x i32> %ext, <16 x i32> zeroinitializer
   ret <16 x i32> %sel
 }
 
 define <8 x i32> @same_sext_used_in_cmp_eq_and_select_v8i32(<8 x i16> %a) {
 ; CHECK-LABEL: same_sext_used_in_cmp_eq_and_select_v8i32:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.8h v1, #10
-; CHECK-NEXT:    ext.16b v2, v0, v0, #8
-; CHECK-NEXT:    cmeq.8h v1, v0, v1
-; CHECK-NEXT:    ext.16b v3, v1, v1, #8
-; CHECK-NEXT:    and.8b v0, v0, v1
+; CHECK-NEXT:    movi.4s v1, #10
+; CHECK-NEXT:    sshll2.4s v2, v0, #0
 ; CHECK-NEXT:    sshll.4s v0, v0, #0
-; CHECK-NEXT:    and.8b v1, v2, v3
-; CHECK-NEXT:    sshll.4s v1, v1, #0
+; CHECK-NEXT:    cmeq.4s v3, v2, v1
+; CHECK-NEXT:    cmeq.4s v4, v0, v1
+; CHECK-NEXT:    and.16b v1, v2, v3
+; CHECK-NEXT:    and.16b v0, v0, v4
 ; CHECK-NEXT:    ret
   %ext = sext <8 x i16> %a to <8 x i32>
   %cmp = icmp eq <8 x i16> %a, <i16 10, i16 10, i16 10, i16 10, i16 10, i16 10, i16 10, i16 10>
   %sel = select <8 x i1> %cmp, <8 x i32> %ext, <8 x i32> zeroinitializer
   ret <8 x i32> %sel
 }
 
 define <8 x i32> @same_sext_used_in_cmp_eq_and_select_v8i32_from_v8i13(<8 x i13> %a) {
 ; CHECK-LABEL: same_sext_used_in_cmp_eq_and_select_v8i32_from_v8i13:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.8h v1, #10
-; CHECK-NEXT:    mov.16b v2, v0
-; CHECK-NEXT:    bic.8h v2, #224, lsl #8
-; CHECK-NEXT:    ushll2.4s v3, v0, #0
-; CHECK-NEXT:    cmeq.8h v1, v2, v1
-; CHECK-NEXT:    ushll.4s v0, v0, #0
-; CHECK-NEXT:    ushll.4s v2, v1, #0
-; CHECK-NEXT:    ushll2.4s v1, v1, #0
-; CHECK-NEXT:    shl.4s v3, v3, #19
-; CHECK-NEXT:    shl.4s v1, v1, #19
-; CHECK-NEXT:    shl.4s v0, v0, #19
+; CHECK-NEXT:    ushll2.4s v2, v0, #0
+; CHECK-NEXT:    ushll.4s v0, v0, #0
+; CHECK-NEXT:    movi.4s v1, #10
 ; CHECK-NEXT:    shl.4s v2, v2, #19
-; CHECK-NEXT:    sshr.4s v3, v3, #19
-; CHECK-NEXT:    sshr.4s v1, v1, #19
-; CHECK-NEXT:    sshr.4s v0, v0, #19
+; CHECK-NEXT:    shl.4s v0, v0, #19
 ; CHECK-NEXT:    sshr.4s v2, v2, #19
-; CHECK-NEXT:    and.16b v1, v3, v1
-; CHECK-NEXT:    and.16b v0, v0, v2
+; CHECK-NEXT:    sshr.4s v0, v0, #19
+; CHECK-NEXT:    cmeq.4s v3, v2, v1
+; CHECK-NEXT:    cmeq.4s v4, v0, v1
+; CHECK-NEXT:    and.16b v1, v2, v3
+; CHECK-NEXT:    and.16b v0, v0, v4
 ; CHECK-NEXT:    ret
   %ext = sext <8 x i13> %a to <8 x i32>
   %cmp = icmp eq <8 x i13> %a, <i13 10, i13 10, i13 10, i13 10, i13 10, i13 10, i13 10, i13 10>
   %sel = select <8 x i1> %cmp, <8 x i32> %ext, <8 x i32> zeroinitializer
   ret <8 x i32> %sel
 }
 
 define <16 x i32> @same_sext_used_in_cmp_ne_and_select_v8i32(<16 x i8> %a) {
 ; CHECK-LABEL: same_sext_used_in_cmp_ne_and_select_v8i32:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.16b v1, #10
-; CHECK-NEXT:    sshll.8h v3, v0, #0
+; CHECK-NEXT:    movi.4s v1, #10
 ; CHECK-NEXT:    sshll2.8h v2, v0, #0
-; CHECK-NEXT:    cmeq.16b v0, v0, v1
-; CHECK-NEXT:    ext.16b v1, v3, v3, #8
-; CHECK-NEXT:    mvn.16b v0, v0
-; CHECK-NEXT:    sshll.8h v5, v0, #0
-; CHECK-NEXT:    sshll2.8h v0, v0, #0
-; CHECK-NEXT:    ext.16b v4, v2, v2, #8
-; CHECK-NEXT:    ext.16b v6, v5, v5, #8
-; CHECK-NEXT:    ext.16b v7, v0, v0, #8
-; CHECK-NEXT:    and.8b v0, v2, v0
-; CHECK-NEXT:    sshll.4s v2, v0, #0
-; CHECK-NEXT:    and.8b v0, v3, v5
-; CHECK-NEXT:    and.8b v1, v1, v6
-; CHECK-NEXT:    and.8b v3, v4, v7
+; CHECK-NEXT:    sshll.8h v0, v0, #0
+; CHECK-NEXT:    sshll2.4s v3, v2, #0
+; CHECK-NEXT:    sshll2.4s v4, v0, #0
 ; CHECK-NEXT:    sshll.4s v0, v0, #0
-; CHECK-NEXT:    sshll.4s v1, v1, #0
-; CHECK-NEXT:    sshll.4s v3, v3, #0
+; CHECK-NEXT:    sshll.4s v2, v2, #0
+; CHECK-NEXT:    cmeq.4s v5, v0, v1
+; CHECK-NEXT:    cmeq.4s v6, v2, v1
+; CHECK-NEXT:    cmeq.4s v7, v3, v1
+; CHECK-NEXT:    cmeq.4s v1, v4, v1
+; CHECK-NEXT:    bic.16b v3, v3, v7
+; CHECK-NEXT:    bic.16b v1, v4, v1
+; CHECK-NEXT:    bic.16b v2, v2, v6
+; CHECK-NEXT:    bic.16b v0, v0, v5
 ; CHECK-NEXT:    ret
   %ext = sext <16 x i8> %a to <16 x i32>
   %cmp = icmp ne <16 x i8> %a, <i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10, i8 10>
   %sel = select <16 x i1> %cmp, <16 x i32> %ext, <16 x i32> zeroinitializer
   ret <16 x i32> %sel
 }
 
 define <8 x i32> @same_sext_used_in_cmp_signed_pred_and_select_v8i32(<8 x i16> %a) {
 ; CHECK-LABEL: same_sext_used_in_cmp_signed_pred_and_select_v8i32:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:    movi.8h v1, #10
-; CHECK-NEXT:    ext.16b v2, v0, v0, #8
-; CHECK-NEXT:    cmgt.8h v1, v0, v1
-; CHECK-NEXT:    ext.16b v3, v1, v1, #8
-; CHECK-NEXT:    and.8b v0, v0, v1
+; CHECK-NEXT:    movi.4s v1, #10
+; CHECK-NEXT:    sshll2.4s v2, v0, #0
 ; CHECK-NEXT:    sshll.4s v0, v0, #0
-; CHECK-NEXT:    and.8b v1, v2, v3
-; CHECK-NEXT:    sshll.4s v1, v1, #0
+; CHECK-NEXT:    cmgt.4s v3, v2, v1
+; CHECK-NEXT:    cmgt.4s v4, v0, v1
+; CHECK-NEXT:    and.16b v1, v2, v3
+; CHECK-NEXT:    and.16b v0, v0, v4
 ; CHECK-NEXT:    ret
 entry:
   %ext = sext <8 x i16> %a to <8 x i32>
   %cmp = icmp sgt <8 x i16> %a, <i16 10, i16 10, i16 10, i16 10, i16 10, i16 10, i16 10, i16 10>
   %sel = select <8 x i1> %cmp, <8 x i32> %ext, <8 x i32> zeroinitializer
   ret <8 x i32> %sel
 }
 
 define <8 x i32> @same_sext_used_in_cmp_unsigned_pred_and_select_v8i32_from_v8i15(<8 x i15> %a) {
 ; CHECK-LABEL: same_sext_used_in_cmp_unsigned_pred_and_select_v8i32_from_v8i15:
 ; CHECK:       ; %bb.0:
-; CHECK-NEXT:    movi.8h v1, #10
-; CHECK-NEXT:    shl.8h v2, v0, #1
-; CHECK-NEXT:    ushll2.4s v3, v0, #0
-; CHECK-NEXT:    sshr.8h v2, v2, #1
+; CHECK-NEXT:    ushll2.4s v2, v0, #0
 ; CHECK-NEXT:    ushll.4s v0, v0, #0
-; CHECK-NEXT:    shl.4s v3, v3, #17
-; CHECK-NEXT:    cmge.8h v1, v2, v1
-; CHECK-NEXT:    shl.4s v0, v0, #17
-; CHECK-NEXT:    ushll.4s v2, v1, #0
-; CHECK-NEXT:    sshr.4s v3, v3, #17
-; CHECK-NEXT:    ushll2.4s v1, v1, #0
+; CHECK-NEXT:    movi.4s v1, #10
 ; CHECK-NEXT:    shl.4s v2, v2, #17
-; CHECK-NEXT:    shl.4s v1, v1, #17
-; CHECK-NEXT:    sshr.4s v0, v0, #17
+; CHECK-NEXT:    shl.4s v0, v0, #17
 ; CHECK-NEXT:    sshr.4s v2, v2, #17
-; CHECK-NEXT:    sshr.4s v1, v1, #17
-; CHECK-NEXT:    and.16b v0, v0, v2
-; CHECK-NEXT:    and.16b v1, v3, v1
+; CHECK-NEXT:    sshr.4s v0, v0, #17
+; CHECK-NEXT:    cmge.4s v3, v2, v1
+; CHECK-NEXT:    cmge.4s v4, v0, v1
+; CHECK-NEXT:    and.16b v1, v2, v3
+; CHECK-NEXT:    and.16b v0, v0, v4
 ; CHECK-NEXT:    ret
   %ext = sext <8 x i15> %a to <8 x i32>
   %cmp = icmp sge <8 x i15> %a, <i15 10, i15 10, i15 10, i15 10, i15 10, i15 10, i15 10, i15 10>
   %sel = select <8 x i1> %cmp, <8 x i32> %ext, <8 x i32> zeroinitializer
   ret <8 x i32> %sel
 }
 
 define <16 x i32> @same_sext_used_in_cmp_unsigned_pred_and_select(<16 x i8> %a) {