[Codegen] (X & (C l>>/<< Y)) ==/!= 0 --> ((X <</l>> Y) & C) ==/!= 0 fold

[lebedev.ri]

Summary

This was originally reported in D62818.
https://rise4fun.com/Alive/oPH

InstCombine does the opposite fold, in hope that C l>>/<< Y expression
will be hoisted out of a loop if Y is invariant and X is not.
But as it is seen from the diffs here, if it didn't get hoisted,
the produced assembly is almost universally worse.

Much like with my recent "hoist add/sub by/from const" patches,
we should get almost universal win if we hoist constant,
there is almost always an "and/test by imm" instruction,
but "shift of imm" not so much, so we may avoid having to
materialize the immediate, and thus need one less register.
And since we now shift not by constant, but by something else,
the live-range of that something else may reduce.

Special care needs to be applied not to disturb x86 BT / hexagon tstbit
instruction pattern. And to not get into endless combine loop.

From what i can tell,

• PPC changes are all good
• AMDGPU neutral
• AArch64 good except vectors (pattern with lshr improves, but shl symmetrically degrades)
• ARM all good, at least in general?
• X86 look ok, the immediate gets encoded int test instruction (vectors are a mess regardless)

[lebedev.ri]

Forgot to add, please bikeshed on the function names.
I know these are horrible, and i can't come up with better ones.

[efriedma]

For the cases involving sign bits, are we actually expecting the IR to look like this when we reach SelectionDAG? With some form of D62818, I would expect we end up with "icmp slt"...

AArch64 good except vectors (pattern with lshr improves, but shl symmetrically degrades)

AArch64 doesn't have separate shift left and shift right instructions; instead, it's based on the sign of the shift amount. This can make the optimal pattern a little strange, yes.

[lebedev.ri]

In D62871#1529899, @efriedma wrote:

For the cases involving sign bits, are we actually expecting the IR to look like this when we reach SelectionDAG? With some form of D62818, I would expect we end up with "icmp slt"...

Probably not, but that is what we end up with currently https://godbolt.org/z/IDhcTo
That does look like missing instcombine fold (+a fold that doesn't fire),
not sure if we want to also handle it here in backend.

AArch64 good except vectors (pattern with lshr improves, but shl symmetrically degrades)

AArch64 doesn't have separate shift left and shift right instructions; instead, it's based on the sign of the shift amount. This can make the optimal pattern a little strange, yes.

[nhaehnle]

FWIW, the AMDGPU changes aren't entirely neutral because they shift the wait for previous memory loads up by one instructions, so it's a minor loss. However, that seems benign enough that I wouldn't object.

[RKSimon]

lib/Target/X86/X86ISelLowering.h

845

put this into X86ISelLowering.cpp, same for Hexagon

test/CodeGen/X86/hoist-and-by-const-from-shl-in-eqcmp-zero.ll

740

please can you run this through llvm-mca to compare perf?

[lebedev.ri]

test/CodeGen/X86/hoist-and-by-const-from-shl-in-eqcmp-zero.ll

740

While this isn't what you asked, i can tell you right away that both variants are very far from optimal:
https://godbolt.org/z/6dKBNR

[lebedev.ri]

test/CodeGen/X86/hoist-and-by-const-from-shl-in-eqcmp-zero.ll

740

Hm, i was looking at that wrong, of course that isn't identical,
because those sse/sse2 shifts all shift all elements by the same amount.

mca: https://godbolt.org/z/PQrkhj
(i did not ensure strictly sequential execution though)

[lebedev.ri]

Add X86 AVX2 runlines, move hasBitTest definitions into .cpp

test/CodeGen/X86/hoist-and-by-const-from-shl-in-eqcmp-zero.ll

740

It's interesting to note how, much like aarch64, these vector changes are symmetrical.
(X & (C << Y)) ==/!= 0 is results in best x86 pre-avx2 codegen
Should i just tune the TLI x86 hook?

[xbolva00]

Not sure, but this pattern regressed
https://godbolt.org/z/vu9C6F

Clang 8 OK, trunk not..

@nikic

[nikic]

@xbolva00 Looks like an instcombine fail:

define dso_local i32 @do_shift(i32, i32) local_unnamed_addr #0 {
  %3 = tail call i32 @llvm.usub.sat.i32(i32 %1, i32 %0)
  %4 = add i32 %3, %0
  ret i32 %4
}

usub.sat(a, b) + b should be canonicalized to umax(a, b).

[lebedev.ri]

Disable non-beneficial folds.

[lebedev.ri]

ping

[lebedev.ri]

ping

[lebedev.ri]

ping. i *think* i addressed all review notes?

[lebedev.ri]

Up

[lebedev.ri]

ping

[lebedev.ri]

ping

[lebedev.ri]

ping again.
I'm not aware of any blockers/review feedback that i need to address here.

[efriedma]

LGTM with one minor change.

include/llvm/CodeGen/TargetLowering.h

538

Maybe worth clarifying here that the point of this hook is to prevent DAGCombine from breaking the pattern?

[spatel]

include/llvm/CodeGen/TargetLowering.h

579

Is there some reason to send XC in as a parameter rather than asking if it is a constant within this function and giving that constant a local name?

[lebedev.ri]

include/llvm/CodeGen/TargetLowering.h

538

Or creating it if it has potential of being recognized, will add.

579

I'm pretty sure i have tried that.
As far as i recall, i can't do that because isConstOrConstSplat() is defined in DAGCombine.cpp,
which isn't guaranteed to be linked in every library where this hook could end up.

[spatel]

include/llvm/CodeGen/TargetLowering.h

579

Ah yes, we have a mess of misplaced helper functions like that. No need to hold this patch up any more then, but it would be good to add a comment somewhere with that explanation.

[lebedev.ri]

Thank you for taking a look!
Rebased, addressed notes about comments.
Care to stamp? :)

[spatel]

LGTM

[lebedev.ri]

In D62871#1600125, @spatel wrote:

LGTM

Thank you for the review!

include/llvm/CodeGen/TargetLowering.h

public:
   /// X = ~A & B
   /// This can be used to simplify select or other instructions.
   virtual bool hasAndNot(SDValue X) const {
     // If the target has the more complex version of this operation, assume that
     // it has this operation too.
     return hasAndNotCompare(X);
   }
 
+  /// Return true if the target has a bit-test instruction:
+  ///   (X & (1 << Y)) ==/!= 0
+  virtual bool hasBitTest(SDValue X, SDValue Y) const { return false; }
+
   /// There are two ways to clear extreme bits (either low or high):
   /// Mask:    x &  (-1 << y)  (the instcombine canonical form)
   /// Shifts:  x >> y << y
   /// Return true if the variant with 2 variable shifts is preferred.
   /// Return false if there is no preference.
   virtual bool shouldFoldMaskToVariableShiftPair(SDValue X) const {
     // By default, let's assume that no one prefers shifts.
     return false;
public:
   /// Return true if we should transform.
   /// Return false if there is no preference.
   virtual bool shouldTransformSignedTruncationCheck(EVT XVT,
                                                     unsigned KeptBits) const {
     // By default, let's assume that no one prefers shifts.
     return false;
   }
 
+  /// Given the pattern
+  ///   (X & (C l>>/<< Y)) ==/!= 0
+  /// return true if it should be transformed into:
+  ///   ((X <</l>> Y) & C) ==/!= 0
+  /// WARNING: if 'X' is a constant, the fold may deadlock!
+  virtual bool shouldProduceAndByConstByHoistingConstFromShiftsLHSOfAnd(
+      SDValue X, ConstantSDNode *XC, ConstantSDNode *CC, SDValue Y,
+      unsigned OldShiftOpcode, unsigned NewShiftOpcode) const {
+    if (hasBitTest(X, Y)) {
+      // One interesting pattern that we'd want to form is 'bit test':
+      //   ((1 << Y) & C) ==/!= 0
+      // But we also need to be careful not to try to reverse that fold.
+
+      // Is this '1 << Y' ?
+      if (OldShiftOpcode == ISD::SHL && CC->isOne())
+        return false; // Keep the 'bit test' pattern.
+
+      // Will it be '1 << Y' after the transform ?
+      if (XC && NewShiftOpcode == ISD::SHL && XC->isOne())
+        return true; // Do form the 'bit test' pattern.
+    }
+
+    // If 'X' is a constant, and we transform, then we will immediately
+    // try to undo the fold, thus causing endless combine loop.
+    // So by default, let's assume everyone prefers the fold
+    // iff 'X' is not a constant.
+    return !XC;
+  }
+
   /// Return true if the target wants to use the optimization that
   /// turns ext(promotableInst1(...(promotableInstN(load)))) into
   /// promotedInst1(...(promotedInstN(ext(load)))).
   bool enableExtLdPromotion() const { return EnableExtLdPromotion; }
 
   /// Return true if the target can combine store(extractelement VectorTy,
   /// Idx).
   /// \p Cost[out] gives the cost of that transformation when this is true.
SDValue foldSetCCWithAnd(EVT VT, SDValue N0, SDValue N1, ISD::CondCode Cond,
                            const SDLoc &DL, DAGCombinerInfo &DCI) const;
   SDValue foldSetCCWithBinOp(EVT VT, SDValue N0, SDValue N1, ISD::CondCode Cond,
                              const SDLoc &DL, DAGCombinerInfo &DCI) const;
 
   SDValue optimizeSetCCOfSignedTruncationCheck(EVT SCCVT, SDValue N0,
                                                SDValue N1, ISD::CondCode Cond,
                                                DAGCombinerInfo &DCI,
                                                const SDLoc &DL) const;
+
+  // (X & (C l>>/<< Y)) ==/!= 0  -->  ((X <</l>> Y) & C) ==/!= 0
+  SDValue optimizeSetCCByHoistingAndByConstFromLogicalShift(
+      EVT SCCVT, SDValue N0, SDValue N1C, ISD::CondCode Cond,
+      DAGCombinerInfo &DCI, const SDLoc &DL) const;
 };
 
 /// Given an LLVM IR type and return type attributes, compute the return value
 /// EVTs and flags, and optionally also the offsets, if the return value is
 /// being lowered to memory.
 void GetReturnInfo(CallingConv::ID CC, Type *ReturnType, AttributeList attr,
                    SmallVectorImpl<ISD::OutputArg> &Outs,
                    const TargetLowering &TLI, const DataLayout &DL);
 
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_TARGETLOWERING_H

lib/CodeGen/SelectionDAG/TargetLowering.cpp

SDValue TargetLowering::optimizeSetCCOfSignedTruncationCheck(
   SDValue ShiftAmt = DAG.getConstant(MaskedBits, DL, XVT);
   SDValue T0 = DAG.getNode(ISD::SHL, DL, XVT, X, ShiftAmt);
   SDValue T1 = DAG.getNode(ISD::SRA, DL, XVT, T0, ShiftAmt);
   SDValue T2 = DAG.getSetCC(DL, SCCVT, T1, X, NewCond);
 
   return T2;
 }
 
+// (X & (C l>>/<< Y)) ==/!= 0  -->  ((X <</l>> Y) & C) ==/!= 0
+SDValue TargetLowering::optimizeSetCCByHoistingAndByConstFromLogicalShift(
+    EVT SCCVT, SDValue N0, SDValue N1C, ISD::CondCode Cond,
+    DAGCombinerInfo &DCI, const SDLoc &DL) const {
+  assert(isConstOrConstSplat(N1C) &&
+         isConstOrConstSplat(N1C)->getAPIntValue().isNullValue() &&
+         "Should be a comparison with 0.");
+  assert((Cond == ISD::SETEQ || Cond == ISD::SETNE) &&
+         "Valid only for [in]equality comparisons.");
+
+  unsigned NewShiftOpcode;
+  SDValue X, C, Y;
+
+  SelectionDAG &DAG = DCI.DAG;
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+
+  // Look for '(C l>>/<< Y)'.
+  auto Match = [&NewShiftOpcode, &X, &C, &Y, &TLI](SDValue V) {
+    // The shift should be one-use.
+    if (!V.hasOneUse())
+      return false;
+    unsigned OldShiftOpcode = V.getOpcode();
+    switch (OldShiftOpcode) {
+    case ISD::SHL:
+      NewShiftOpcode = ISD::SRL;
+      break;
+    case ISD::SRL:
+      NewShiftOpcode = ISD::SHL;
+      break;
+    default:
+      return false; // must be a logical shift.
+    }
+    // We should be shifting a constant.
+    // FIXME: best to use isConstantOrConstantVector().
+    C = V.getOperand(0);
+    ConstantSDNode *CC =
+        isConstOrConstSplat(C, /*AllowUndefs=*/true, /*AllowTruncation=*/true);
+    if (!CC)
+      return false;
+    Y = V.getOperand(1);
+
+    ConstantSDNode *XC =
+        isConstOrConstSplat(X, /*AllowUndefs=*/true, /*AllowTruncation=*/true);
+    return TLI.shouldProduceAndByConstByHoistingConstFromShiftsLHSOfAnd(
+        X, XC, CC, Y, OldShiftOpcode, NewShiftOpcode);
+  };
+
+  // LHS of comparison should be an one-use 'and'.
+  if (N0.getOpcode() != ISD::AND || !N0.hasOneUse())
+    return SDValue();
+
+  X = N0.getOperand(0);
+  SDValue Mask = N0.getOperand(1);
+
+  // 'and' is commutative!
+  if (!Match(Mask)) {
+    std::swap(X, Mask);
+    if (!Match(Mask))
+      return SDValue();
+  }
+
+  EVT VT = X.getValueType();
+
+  // Produce:
+  // ((X 'OppositeShiftOpcode' Y) & C) Cond 0
+  SDValue T0 = DAG.getNode(NewShiftOpcode, DL, VT, X, Y);
+  SDValue T1 = DAG.getNode(ISD::AND, DL, VT, T0, C);
+  SDValue T2 = DAG.getSetCC(DL, SCCVT, T1, N1C, Cond);
+  return T2;
+}
+
 /// Try to fold an equality comparison with a {add/sub/xor} binary operation as
 /// the 1st operand (N0). Callers are expected to swap the N0/N1 parameters to
 /// handle the commuted versions of these patterns.
 SDValue TargetLowering::foldSetCCWithBinOp(EVT VT, SDValue N0, SDValue N1,
                                            ISD::CondCode Cond, const SDLoc &DL,
                                            DAGCombinerInfo &DCI) const {
   unsigned BOpcode = N0.getOpcode();
   assert((BOpcode == ISD::ADD || BOpcode == ISD::SUB || BOpcode == ISD::XOR) &&
if (Cond == ISD::SETGT || Cond == ISD::SETUGT) {
         // If we have setugt X, Max-1, turn it into seteq X, Max
         if (C1 == MaxVal-1)
           return DAG.getSetCC(dl, VT, N0,
                               DAG.getConstant(MaxVal, dl, N0.getValueType()),
                               ISD::SETEQ);
       }
     }
 
+    if (Cond == ISD::SETEQ || Cond == ISD::SETNE) {
+      // (X & (C l>>/<< Y)) ==/!= 0  -->  ((X <</l>> Y) & C) ==/!= 0
+      if (C1.isNullValue())
+        if (SDValue CC = optimizeSetCCByHoistingAndByConstFromLogicalShift(
+                VT, N0, N1, Cond, DCI, dl))
+          return CC;
+    }
+
     // If we have "setcc X, C0", check to see if we can shrink the immediate
     // by changing cc.
     // TODO: Support this for vectors after legalize ops.
     if (!VT.isVector() || DCI.isBeforeLegalizeOps()) {
       // SETUGT X, SINTMAX  -> SETLT X, 0
       if (Cond == ISD::SETUGT &&
           C1 == APInt::getSignedMaxValue(OperandBitSize))
         return DAG.getSetCC(dl, VT, N0,

lib/Target/Hexagon/HexagonISelLowering.h

public:
 
     bool isTruncateFree(Type *Ty1, Type *Ty2) const override;
     bool isTruncateFree(EVT VT1, EVT VT2) const override;
 
     bool isCheapToSpeculateCttz() const override { return true; }
     bool isCheapToSpeculateCtlz() const override { return true; }
     bool isCtlzFast() const override { return true; }
 
+    bool hasBitTest(SDValue X, SDValue Y) const override {
+      return X.getValueType().isScalarInteger(); // 'tstbit'
+    }
+
     bool allowTruncateForTailCall(Type *Ty1, Type *Ty2) const override;
 
     /// Return true if an FMA operation is faster than a pair of mul and add
     /// instructions. fmuladd intrinsics will be expanded to FMAs when this
     /// method returns true (and FMAs are legal), otherwise fmuladd is
     /// expanded to mul + add.
     bool isFMAFasterThanFMulAndFAdd(EVT) const override;
 

lib/Target/X86/X86ISelLowering.h

public:
     }
 
     bool isMaskAndCmp0FoldingBeneficial(const Instruction &AndI) const override;
 
     bool hasAndNotCompare(SDValue Y) const override;
 
     bool hasAndNot(SDValue Y) const override;
 
+    bool hasBitTest(SDValue X, SDValue Y) const override {
+      return X.getValueType().isScalarInteger(); // 'bt'
+    }
+
     bool shouldFoldConstantShiftPairToMask(const SDNode *N,
                                            CombineLevel Level) const override;
 
     bool shouldFoldMaskToVariableShiftPair(SDValue Y) const override;
 
     bool
     shouldTransformSignedTruncationCheck(EVT XVT,
                                          unsigned KeptBits) const override {

test/CodeGen/AArch64/hoist-and-by-const-from-lshr-in-eqcmp-zero.ll

 ; A few scalar test
 ;------------------------------------------------------------------------------;
 
 ; i8 scalar
 
 define i1 @scalar_i8_signbit_eq(i8 %x, i8 %y) nounwind {
 ; CHECK-LABEL: scalar_i8_signbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #128
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsr w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xff
+; CHECK-NEXT:    lsl w8, w0, w1
+; CHECK-NEXT:    tst w8, #0x80
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i8_lowestbit_eq(i8 %x, i8 %y) nounwind {
 ; CHECK-LABEL: scalar_i8_lowestbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #1
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsr w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xff
+; CHECK-NEXT:    lsl w8, w0, w1
+; CHECK-NEXT:    tst w8, #0x1
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i8 1, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i8_bitsinmiddle_eq(i8 %x, i8 %y) nounwind {
 ; CHECK-LABEL: scalar_i8_bitsinmiddle_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #24
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsr w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xff
+; CHECK-NEXT:    lsl w8, w0, w1
+; CHECK-NEXT:    tst w8, #0x18
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i8 24, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 ; i16 scalar
 
 define i1 @scalar_i16_signbit_eq(i16 %x, i16 %y) nounwind {
 ; CHECK-LABEL: scalar_i16_signbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #32768
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsr w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xffff
+; CHECK-NEXT:    lsl w8, w0, w1
+; CHECK-NEXT:    tst w8, #0x8000
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i16 32768, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i16_lowestbit_eq(i16 %x, i16 %y) nounwind {
 ; CHECK-LABEL: scalar_i16_lowestbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #1
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsr w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xffff
+; CHECK-NEXT:    lsl w8, w0, w1
+; CHECK-NEXT:    tst w8, #0x1
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i16 1, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i16_bitsinmiddle_eq(i16 %x, i16 %y) nounwind {
 ; CHECK-LABEL: scalar_i16_bitsinmiddle_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #4080
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsr w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xffff
+; CHECK-NEXT:    lsl w8, w0, w1
+; CHECK-NEXT:    tst w8, #0xff0
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i16 4080, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 ; i32 scalar
 
 define i1 @scalar_i32_signbit_eq(i32 %x, i32 %y) nounwind {
 ; CHECK-LABEL: scalar_i32_signbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #-2147483648
-; CHECK-NEXT:    lsr w8, w8, w1
-; CHECK-NEXT:    tst w8, w0
+; CHECK-NEXT:    lsl w8, w0, w1
+; CHECK-NEXT:    tst w8, #0x80000000
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i32 2147483648, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i32_lowestbit_eq(i32 %x, i32 %y) nounwind {
 ; CHECK-LABEL: scalar_i32_lowestbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #1
-; CHECK-NEXT:    lsr w8, w8, w1
-; CHECK-NEXT:    tst w8, w0
+; CHECK-NEXT:    lsl w8, w0, w1
+; CHECK-NEXT:    tst w8, #0x1
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i32 1, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i32_bitsinmiddle_eq(i32 %x, i32 %y) nounwind {
 ; CHECK-LABEL: scalar_i32_bitsinmiddle_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #16776960
-; CHECK-NEXT:    lsr w8, w8, w1
-; CHECK-NEXT:    tst w8, w0
+; CHECK-NEXT:    lsl w8, w0, w1
+; CHECK-NEXT:    tst w8, #0xffff00
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i32 16776960, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 ; i64 scalar
 
 define i1 @scalar_i64_signbit_eq(i64 %x, i64 %y) nounwind {
 ; CHECK-LABEL: scalar_i64_signbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov x8, #-9223372036854775808
-; CHECK-NEXT:    lsr x8, x8, x1
-; CHECK-NEXT:    tst x8, x0
+; CHECK-NEXT:    lsl x8, x0, x1
+; CHECK-NEXT:    tst x8, #0x8000000000000000
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i64 9223372036854775808, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i64_lowestbit_eq(i64 %x, i64 %y) nounwind {
 ; CHECK-LABEL: scalar_i64_lowestbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #1
-; CHECK-NEXT:    lsr x8, x8, x1
-; CHECK-NEXT:    tst x8, x0
+; CHECK-NEXT:    lsl x8, x0, x1
+; CHECK-NEXT:    tst x8, #0x1
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i64 1, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i64_bitsinmiddle_eq(i64 %x, i64 %y) nounwind {
 ; CHECK-LABEL: scalar_i64_bitsinmiddle_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov x8, #281474976645120
-; CHECK-NEXT:    lsr x8, x8, x1
-; CHECK-NEXT:    tst x8, x0
+; CHECK-NEXT:    lsl x8, x0, x1
+; CHECK-NEXT:    tst x8, #0xffffffff0000
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = lshr i64 281474976645120, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;
 ; A few trivial vector tests
 ;------------------------------------------------------------------------------;
 
 define <4 x i1> @vec_4xi32_splat_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; CHECK-LABEL: vec_4xi32_splat_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    neg v1.4s, v1.4s
 ; CHECK-NEXT:    movi v2.4s, #1
-; CHECK-NEXT:    ushl v1.4s, v2.4s, v1.4s
-; CHECK-NEXT:    and v0.16b, v1.16b, v0.16b
+; CHECK-NEXT:    ushl v0.4s, v0.4s, v1.4s
+; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    xtn v0.4h, v0.4s
 ; CHECK-NEXT:    ret
   %t0 = lshr <4 x i32> <i32 1, i32 1, i32 1, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
; CHECK-NEXT:    ret
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 
 define <4 x i1> @vec_4xi32_nonsplat_undef0_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; CHECK-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    neg v1.4s, v1.4s
 ; CHECK-NEXT:    movi v2.4s, #1
-; CHECK-NEXT:    ushl v1.4s, v2.4s, v1.4s
-; CHECK-NEXT:    and v0.16b, v1.16b, v0.16b
+; CHECK-NEXT:    ushl v0.4s, v0.4s, v1.4s
+; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    xtn v0.4h, v0.4s
 ; CHECK-NEXT:    ret
   %t0 = lshr <4 x i32> <i32 1, i32 1, i32 undef, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }

 
 ;------------------------------------------------------------------------------;
 ; A special tests
 ;------------------------------------------------------------------------------;
 
 define i1 @scalar_i8_signbit_ne(i8 %x, i8 %y) nounwind {
 ; CHECK-LABEL: scalar_i8_signbit_ne:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #128
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsr w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xff
-; CHECK-NEXT:    cset w0, ne
+; CHECK-NEXT:    lsl w8, w0, w1
+; CHECK-NEXT:    ubfx w0, w8, #7, #1
 ; CHECK-NEXT:    ret
   %t0 = lshr i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp ne i8 %t1, 0 ;  we are perfectly happy with 'ne' predicate
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;

test/CodeGen/AArch64/hoist-and-by-const-from-shl-in-eqcmp-zero.ll

 ; A few scalar test
 ;------------------------------------------------------------------------------;
 
 ; i8 scalar
 
 define i1 @scalar_i8_signbit_eq(i8 %x, i8 %y) nounwind {
 ; CHECK-LABEL: scalar_i8_signbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #-128
+; CHECK-NEXT:    and w8, w0, #0xff
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsl w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xff
+; CHECK-NEXT:    lsr w8, w8, w1
+; CHECK-NEXT:    tst w8, #0x80
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i8_lowestbit_eq(i8 %x, i8 %y) nounwind {
 ; CHECK-LABEL: scalar_i8_lowestbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    and w8, w0, #0xff
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsl w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xff
+; CHECK-NEXT:    lsr w8, w8, w1
+; CHECK-NEXT:    tst w8, #0x1
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i8 1, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i8_bitsinmiddle_eq(i8 %x, i8 %y) nounwind {
 ; CHECK-LABEL: scalar_i8_bitsinmiddle_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #24
+; CHECK-NEXT:    and w8, w0, #0xff
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsl w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xff
+; CHECK-NEXT:    lsr w8, w8, w1
+; CHECK-NEXT:    tst w8, #0x18
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i8 24, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 ; i16 scalar
 
 define i1 @scalar_i16_signbit_eq(i16 %x, i16 %y) nounwind {
 ; CHECK-LABEL: scalar_i16_signbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #-32768
+; CHECK-NEXT:    and w8, w0, #0xffff
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsl w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xffff
+; CHECK-NEXT:    lsr w8, w8, w1
+; CHECK-NEXT:    tst w8, #0x8000
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i16 32768, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i16_lowestbit_eq(i16 %x, i16 %y) nounwind {
 ; CHECK-LABEL: scalar_i16_lowestbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    and w8, w0, #0xffff
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsl w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xffff
+; CHECK-NEXT:    lsr w8, w8, w1
+; CHECK-NEXT:    tst w8, #0x1
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i16 1, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i16_bitsinmiddle_eq(i16 %x, i16 %y) nounwind {
 ; CHECK-LABEL: scalar_i16_bitsinmiddle_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #4080
+; CHECK-NEXT:    and w8, w0, #0xffff
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsl w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xffff
+; CHECK-NEXT:    lsr w8, w8, w1
+; CHECK-NEXT:    tst w8, #0xff0
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i16 4080, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 ; i32 scalar
 
 define i1 @scalar_i32_signbit_eq(i32 %x, i32 %y) nounwind {
 ; CHECK-LABEL: scalar_i32_signbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #-2147483648
-; CHECK-NEXT:    lsl w8, w8, w1
-; CHECK-NEXT:    tst w8, w0
+; CHECK-NEXT:    lsr w8, w0, w1
+; CHECK-NEXT:    tst w8, #0x80000000
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i32 2147483648, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i32_lowestbit_eq(i32 %x, i32 %y) nounwind {
 ; CHECK-LABEL: scalar_i32_lowestbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #1
-; CHECK-NEXT:    lsl w8, w8, w1
-; CHECK-NEXT:    tst w8, w0
+; CHECK-NEXT:    lsr w8, w0, w1
+; CHECK-NEXT:    tst w8, #0x1
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i32 1, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i32_bitsinmiddle_eq(i32 %x, i32 %y) nounwind {
 ; CHECK-LABEL: scalar_i32_bitsinmiddle_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #16776960
-; CHECK-NEXT:    lsl w8, w8, w1
-; CHECK-NEXT:    tst w8, w0
+; CHECK-NEXT:    lsr w8, w0, w1
+; CHECK-NEXT:    tst w8, #0xffff00
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i32 16776960, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 ; i64 scalar
 
 define i1 @scalar_i64_signbit_eq(i64 %x, i64 %y) nounwind {
 ; CHECK-LABEL: scalar_i64_signbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov x8, #-9223372036854775808
-; CHECK-NEXT:    lsl x8, x8, x1
-; CHECK-NEXT:    tst x8, x0
+; CHECK-NEXT:    lsr x8, x0, x1
+; CHECK-NEXT:    tst x8, #0x8000000000000000
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i64 9223372036854775808, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i64_lowestbit_eq(i64 %x, i64 %y) nounwind {
 ; CHECK-LABEL: scalar_i64_lowestbit_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #1
-; CHECK-NEXT:    lsl x8, x8, x1
-; CHECK-NEXT:    tst x8, x0
+; CHECK-NEXT:    lsr x8, x0, x1
+; CHECK-NEXT:    tst x8, #0x1
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i64 1, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i64_bitsinmiddle_eq(i64 %x, i64 %y) nounwind {
 ; CHECK-LABEL: scalar_i64_bitsinmiddle_eq:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov x8, #281474976645120
-; CHECK-NEXT:    lsl x8, x8, x1
-; CHECK-NEXT:    tst x8, x0
+; CHECK-NEXT:    lsr x8, x0, x1
+; CHECK-NEXT:    tst x8, #0xffffffff0000
 ; CHECK-NEXT:    cset w0, eq
 ; CHECK-NEXT:    ret
   %t0 = shl i64 281474976645120, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;
 ; A few trivial vector tests
 ;------------------------------------------------------------------------------;
 
 define <4 x i1> @vec_4xi32_splat_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; CHECK-LABEL: vec_4xi32_splat_eq:
 ; CHECK:       // %bb.0:
+; CHECK-NEXT:    neg v1.4s, v1.4s
 ; CHECK-NEXT:    movi v2.4s, #1
-; CHECK-NEXT:    ushl v1.4s, v2.4s, v1.4s
-; CHECK-NEXT:    and v0.16b, v1.16b, v0.16b
+; CHECK-NEXT:    ushl v0.4s, v0.4s, v1.4s
+; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    xtn v0.4h, v0.4s
 ; CHECK-NEXT:    ret
   %t0 = shl <4 x i32> <i32 1, i32 1, i32 1, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
; CHECK-NEXT:    ret
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 
 define <4 x i1> @vec_4xi32_nonsplat_undef0_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; CHECK-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; CHECK:       // %bb.0:
+; CHECK-NEXT:    neg v1.4s, v1.4s
 ; CHECK-NEXT:    movi v2.4s, #1
-; CHECK-NEXT:    ushl v1.4s, v2.4s, v1.4s
-; CHECK-NEXT:    and v0.16b, v1.16b, v0.16b
+; CHECK-NEXT:    ushl v0.4s, v0.4s, v1.4s
+; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    xtn v0.4h, v0.4s
 ; CHECK-NEXT:    ret
   %t0 = shl <4 x i32> <i32 1, i32 1, i32 undef, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }

 
 ;------------------------------------------------------------------------------;
 ; A special tests
 ;------------------------------------------------------------------------------;
 
 define i1 @scalar_i8_signbit_ne(i8 %x, i8 %y) nounwind {
 ; CHECK-LABEL: scalar_i8_signbit_ne:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, #-128
+; CHECK-NEXT:    and w8, w0, #0xff
 ; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
-; CHECK-NEXT:    lsl w8, w8, w1
-; CHECK-NEXT:    and w8, w8, w0
-; CHECK-NEXT:    tst w8, #0xff
-; CHECK-NEXT:    cset w0, ne
+; CHECK-NEXT:    lsr w8, w8, w1
+; CHECK-NEXT:    ubfx w0, w8, #7, #1
 ; CHECK-NEXT:    ret
   %t0 = shl i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp ne i8 %t1, 0 ;  we are perfectly happy with 'ne' predicate
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;

test/CodeGen/AMDGPU/commute-shifts.ll

 ; SI-NEXT:    s_mov_b32 s2, s0
 ; SI-NEXT:    s_mov_b32 s3, s0
 ; SI-NEXT:    s_mov_b32 s4, s0
 ; SI-NEXT:    s_mov_b32 s5, s0
 ; SI-NEXT:    s_mov_b32 s6, s0
 ; SI-NEXT:    s_mov_b32 s7, s0
 ; SI-NEXT:    image_load v2, v0, s[0:7] dmask:0x1 unorm
 ; SI-NEXT:    v_and_b32_e32 v0, 7, v0
-; SI-NEXT:    v_lshl_b32_e32 v0, 1, v0
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_and_b32_e32 v0, v2, v0
-; SI-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
+; SI-NEXT:    v_lshrrev_b32_e32 v0, v0, v2
+; SI-NEXT:    v_and_b32_e32 v0, 1, v0
+; SI-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v0
 ; SI-NEXT:    v_cndmask_b32_e32 v0, 0, v1, vcc
 ; SI-NEXT:    v_cvt_pkrtz_f16_f32_e32 v0, s0, v0
 ; SI-NEXT:    ; return to shader part epilog
 ;
 ; VI-LABEL: main:
 ; VI:       ; %bb.0: ; %bb
 ; VI-NEXT:    v_cvt_i32_f32_e32 v0, v0
 ; VI-NEXT:    s_mov_b32 s0, 0
 ; VI-NEXT:    s_mov_b32 s1, s0
 ; VI-NEXT:    s_mov_b32 s2, s0
 ; VI-NEXT:    s_mov_b32 s3, s0
 ; VI-NEXT:    s_mov_b32 s4, s0
 ; VI-NEXT:    s_mov_b32 s5, s0
 ; VI-NEXT:    s_mov_b32 s6, s0
 ; VI-NEXT:    s_mov_b32 s7, s0
 ; VI-NEXT:    image_load v2, v0, s[0:7] dmask:0x1 unorm
 ; VI-NEXT:    v_and_b32_e32 v0, 7, v0
-; VI-NEXT:    v_lshlrev_b32_e64 v0, v0, 1
 ; VI-NEXT:    s_waitcnt vmcnt(0)
-; VI-NEXT:    v_and_b32_e32 v0, v2, v0
-; VI-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
+; VI-NEXT:    v_lshrrev_b32_e32 v0, v0, v2
+; VI-NEXT:    v_and_b32_e32 v0, 1, v0
+; VI-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v0
 ; VI-NEXT:    v_cndmask_b32_e32 v0, 0, v1, vcc
 ; VI-NEXT:    v_cvt_pkrtz_f16_f32 v0, s0, v0
 ; VI-NEXT:    ; return to shader part epilog
 bb:
   %tmp = fptosi float %arg0 to i32
   %tmp1 = call <4 x float> @llvm.amdgcn.image.load.1d.v4f32.i32(i32 15, i32 undef, <8 x i32> undef, i32 0, i32 0)
   %tmp2.f = extractelement <4 x float> %tmp1, i32 0
   %tmp2 = bitcast float %tmp2.f to i32

test/CodeGen/ARM/hoist-and-by-const-from-lshr-in-eqcmp-zero.ll

 ;------------------------------------------------------------------------------;
 
 ; i8 scalar
 
 define i1 @scalar_i8_signbit_eq(i8 %x, i8 %y) nounwind {
 ; ARM-LABEL: scalar_i8_signbit_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    uxtb r1, r1
-; ARM-NEXT:    mov r2, #128
-; ARM-NEXT:    and r0, r0, r2, lsr r1
+; ARM-NEXT:    lsl r0, r0, r1
+; ARM-NEXT:    mov r1, #1
 ; ARM-NEXT:    uxtb r0, r0
-; ARM-NEXT:    clz r0, r0
-; ARM-NEXT:    lsr r0, r0, #5
+; ARM-NEXT:    eor r0, r1, r0, lsr #7
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i8_signbit_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxtb r1, r1
-; THUMB6-NEXT:    movs r2, #128
-; THUMB6-NEXT:    lsrs r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxtb r1, r2
+; THUMB6-NEXT:    lsls r0, r1
+; THUMB6-NEXT:    movs r1, #128
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
 ; THUMB78-LABEL: scalar_i8_signbit_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    uxtb r1, r1
-; THUMB78-NEXT:    movs r2, #128
-; THUMB78-NEXT:    lsr.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
+; THUMB78-NEXT:    lsls r0, r1
+; THUMB78-NEXT:    movs r1, #1
 ; THUMB78-NEXT:    uxtb r0, r0
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
+; THUMB78-NEXT:    eor.w r0, r1, r0, lsr #7
 ; THUMB78-NEXT:    bx lr
   %t0 = lshr i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i8_lowestbit_eq(i8 %x, i8 %y) nounwind {
 ; ARM-LABEL: scalar_i8_lowestbit_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    uxtb r1, r1
 ; ARM-NEXT:    mov r2, #1
-; ARM-NEXT:    and r0, r0, r2, lsr r1
-; ARM-NEXT:    uxtb r0, r0
-; ARM-NEXT:    clz r0, r0
-; ARM-NEXT:    lsr r0, r0, #5
+; ARM-NEXT:    bic r0, r2, r0, lsl r1
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i8_lowestbit_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxtb r1, r1
-; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    lsrs r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxtb r1, r2
+; THUMB6-NEXT:    lsls r0, r1
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
 ; THUMB78-LABEL: scalar_i8_lowestbit_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    uxtb r1, r1
-; THUMB78-NEXT:    movs r2, #1
-; THUMB78-NEXT:    lsr.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxtb r0, r0
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
+; THUMB78-NEXT:    lsls r0, r1
+; THUMB78-NEXT:    movs r1, #1
+; THUMB78-NEXT:    bic.w r0, r1, r0
 ; THUMB78-NEXT:    bx lr
   %t0 = lshr i8 1, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i8_bitsinmiddle_eq(i8 %x, i8 %y) nounwind {
 ; ARM-LABEL: scalar_i8_bitsinmiddle_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    uxtb r1, r1
 ; ARM-NEXT:    mov r2, #24
-; ARM-NEXT:    and r0, r0, r2, lsr r1
-; ARM-NEXT:    uxtb r0, r0
+; ARM-NEXT:    and r0, r2, r0, lsl r1
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i8_bitsinmiddle_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxtb r1, r1
-; THUMB6-NEXT:    movs r2, #24
-; THUMB6-NEXT:    lsrs r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxtb r1, r2
+; THUMB6-NEXT:    lsls r0, r1
+; THUMB6-NEXT:    movs r1, #24
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
 ; THUMB78-LABEL: scalar_i8_bitsinmiddle_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    uxtb r1, r1
-; THUMB78-NEXT:    movs r2, #24
-; THUMB78-NEXT:    lsr.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxtb r0, r0
+; THUMB78-NEXT:    lsls r0, r1
+; THUMB78-NEXT:    and r0, r0, #24
 ; THUMB78-NEXT:    clz r0, r0
 ; THUMB78-NEXT:    lsrs r0, r0, #5
 ; THUMB78-NEXT:    bx lr
   %t0 = lshr i8 24, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 ; i16 scalar
 
 define i1 @scalar_i16_signbit_eq(i16 %x, i16 %y) nounwind {
 ; ARM-LABEL: scalar_i16_signbit_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    uxth r1, r1
-; ARM-NEXT:    mov r2, #32768
-; ARM-NEXT:    and r0, r0, r2, lsr r1
+; ARM-NEXT:    lsl r0, r0, r1
+; ARM-NEXT:    mov r1, #1
 ; ARM-NEXT:    uxth r0, r0
-; ARM-NEXT:    clz r0, r0
-; ARM-NEXT:    lsr r0, r0, #5
+; ARM-NEXT:    eor r0, r1, r0, lsr #15
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i16_signbit_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxth r1, r1
-; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    lsls r2, r2, #15
-; THUMB6-NEXT:    lsrs r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxth r1, r2
+; THUMB6-NEXT:    lsls r0, r1
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    lsls r1, r1, #15
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
 ; THUMB78-LABEL: scalar_i16_signbit_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    uxth r1, r1
-; THUMB78-NEXT:    mov.w r2, #32768
-; THUMB78-NEXT:    lsr.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
+; THUMB78-NEXT:    lsls r0, r1
+; THUMB78-NEXT:    movs r1, #1
 ; THUMB78-NEXT:    uxth r0, r0
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
+; THUMB78-NEXT:    eor.w r0, r1, r0, lsr #15
 ; THUMB78-NEXT:    bx lr
   %t0 = lshr i16 32768, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i16_lowestbit_eq(i16 %x, i16 %y) nounwind {
 ; ARM-LABEL: scalar_i16_lowestbit_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    uxth r1, r1
 ; ARM-NEXT:    mov r2, #1
-; ARM-NEXT:    and r0, r0, r2, lsr r1
-; ARM-NEXT:    uxth r0, r0
-; ARM-NEXT:    clz r0, r0
-; ARM-NEXT:    lsr r0, r0, #5
+; ARM-NEXT:    bic r0, r2, r0, lsl r1
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i16_lowestbit_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxth r1, r1
-; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    lsrs r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxth r1, r2
+; THUMB6-NEXT:    lsls r0, r1
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
 ; THUMB78-LABEL: scalar_i16_lowestbit_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    uxth r1, r1
-; THUMB78-NEXT:    movs r2, #1
-; THUMB78-NEXT:    lsr.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxth r0, r0
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
+; THUMB78-NEXT:    lsls r0, r1
+; THUMB78-NEXT:    movs r1, #1
+; THUMB78-NEXT:    bic.w r0, r1, r0
 ; THUMB78-NEXT:    bx lr
   %t0 = lshr i16 1, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i16_bitsinmiddle_eq(i16 %x, i16 %y) nounwind {
 ; ARM-LABEL: scalar_i16_bitsinmiddle_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    uxth r1, r1
 ; ARM-NEXT:    mov r2, #4080
-; ARM-NEXT:    and r0, r0, r2, lsr r1
-; ARM-NEXT:    uxth r0, r0
+; ARM-NEXT:    and r0, r2, r0, lsl r1
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i16_bitsinmiddle_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxth r1, r1
-; THUMB6-NEXT:    movs r2, #255
-; THUMB6-NEXT:    lsls r2, r2, #4
-; THUMB6-NEXT:    lsrs r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxth r1, r2
+; THUMB6-NEXT:    lsls r0, r1
+; THUMB6-NEXT:    movs r1, #255
+; THUMB6-NEXT:    lsls r1, r1, #4
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
 ; THUMB78-LABEL: scalar_i16_bitsinmiddle_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    uxth r1, r1
-; THUMB78-NEXT:    mov.w r2, #4080
-; THUMB78-NEXT:    lsr.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxth r0, r0
+; THUMB78-NEXT:    lsls r0, r1
+; THUMB78-NEXT:    and r0, r0, #4080
 ; THUMB78-NEXT:    clz r0, r0
 ; THUMB78-NEXT:    lsrs r0, r0, #5
 ; THUMB78-NEXT:    bx lr
   %t0 = lshr i16 4080, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 ; i32 scalar
 
 define i1 @scalar_i32_signbit_eq(i32 %x, i32 %y) nounwind {
 ; ARM-LABEL: scalar_i32_signbit_eq:
 ; ARM:       @ %bb.0:
-; ARM-NEXT:    mov r2, #-2147483648
-; ARM-NEXT:    and r0, r0, r2, lsr r1
-; ARM-NEXT:    clz r0, r0
-; ARM-NEXT:    lsr r0, r0, #5
+; ARM-NEXT:    mvn r0, r0, lsl r1
+; ARM-NEXT:    lsr r0, r0, #31
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i32_signbit_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    lsls r2, r2, #31
-; THUMB6-NEXT:    lsrs r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    rsbs r0, r2, #0
-; THUMB6-NEXT:    adcs r0, r2
+; THUMB6-NEXT:    lsls r0, r1
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    lsls r1, r1, #31
+; THUMB6-NEXT:    ands r1, r0
+; THUMB6-NEXT:    rsbs r0, r1, #0
+; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
 ; THUMB78-LABEL: scalar_i32_signbit_eq:
 ; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    mov.w r2, #-2147483648
-; THUMB78-NEXT:    lsr.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
+; THUMB78-NEXT:    lsls r0, r1
+; THUMB78-NEXT:    mvns r0, r0
+; THUMB78-NEXT:    lsrs r0, r0, #31
 ; THUMB78-NEXT:    bx lr
   %t0 = lshr i32 2147483648, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i32_lowestbit_eq(i32 %x, i32 %y) nounwind {
 ; ARM-LABEL: scalar_i32_lowestbit_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    mov r2, #1
-; ARM-NEXT:    and r0, r0, r2, lsr r1
-; ARM-NEXT:    clz r0, r0
-; ARM-NEXT:    lsr r0, r0, #5
+; ARM-NEXT:    bic r0, r2, r0, lsl r1
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i32_lowestbit_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    lsrs r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    rsbs r0, r2, #0
-; THUMB6-NEXT:    adcs r0, r2
+; THUMB6-NEXT:    lsls r0, r1
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    ands r1, r0
+; THUMB6-NEXT:    rsbs r0, r1, #0
+; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
 ; THUMB78-LABEL: scalar_i32_lowestbit_eq:
 ; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    movs r2, #1
-; THUMB78-NEXT:    lsr.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
+; THUMB78-NEXT:    lsls r0, r1
+; THUMB78-NEXT:    movs r1, #1
+; THUMB78-NEXT:    bic.w r0, r1, r0
 ; THUMB78-NEXT:    bx lr
   %t0 = lshr i32 1, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i32_bitsinmiddle_eq(i32 %x, i32 %y) nounwind {
 ; ARM6-LABEL: scalar_i32_bitsinmiddle_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    mov r2, #65280
 ; ARM6-NEXT:    orr r2, r2, #16711680
-; ARM6-NEXT:    and r0, r0, r2, lsr r1
+; ARM6-NEXT:    and r0, r2, r0, lsl r1
 ; ARM6-NEXT:    clz r0, r0
 ; ARM6-NEXT:    lsr r0, r0, #5
 ; ARM6-NEXT:    bx lr
 ;
 ; ARM78-LABEL: scalar_i32_bitsinmiddle_eq:
 ; ARM78:       @ %bb.0:
 ; ARM78-NEXT:    movw r2, #65280
 ; ARM78-NEXT:    movt r2, #255
-; ARM78-NEXT:    and r0, r0, r2, lsr r1
+; ARM78-NEXT:    and r0, r2, r0, lsl r1
 ; ARM78-NEXT:    clz r0, r0
 ; ARM78-NEXT:    lsr r0, r0, #5
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i32_bitsinmiddle_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    ldr r2, .LCPI8_0
-; THUMB6-NEXT:    lsrs r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    rsbs r0, r2, #0
-; THUMB6-NEXT:    adcs r0, r2
+; THUMB6-NEXT:    lsls r0, r1
+; THUMB6-NEXT:    ldr r1, .LCPI8_0
+; THUMB6-NEXT:    ands r1, r0
+; THUMB6-NEXT:    rsbs r0, r1, #0
+; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ; THUMB6-NEXT:    .p2align 2
 ; THUMB6-NEXT:  @ %bb.1:
 ; THUMB6-NEXT:  .LCPI8_0:
 ; THUMB6-NEXT:    .long 16776960 @ 0xffff00
 ;
 ; THUMB78-LABEL: scalar_i32_bitsinmiddle_eq:
 ; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    movw r2, #65280
-; THUMB78-NEXT:    movt r2, #255
-; THUMB78-NEXT:    lsr.w r1, r2, r1
+; THUMB78-NEXT:    lsls r0, r1
+; THUMB78-NEXT:    movw r1, #65280
+; THUMB78-NEXT:    movt r1, #255
 ; THUMB78-NEXT:    ands r0, r1
 ; THUMB78-NEXT:    clz r0, r0
 ; THUMB78-NEXT:    lsrs r0, r0, #5
 ; THUMB78-NEXT:    bx lr
   %t0 = lshr i32 16776960, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 ; i64 scalar
 
 define i1 @scalar_i64_signbit_eq(i64 %x, i64 %y) nounwind {
-; ARM6-LABEL: scalar_i64_signbit_eq:
-; ARM6:       @ %bb.0:
-; ARM6-NEXT:    push {r11, lr}
-; ARM6-NEXT:    mov r12, #-2147483648
-; ARM6-NEXT:    sub lr, r2, #32
-; ARM6-NEXT:    lsr r3, r12, r2
-; ARM6-NEXT:    rsb r2, r2, #32
-; ARM6-NEXT:    cmp lr, #0
-; ARM6-NEXT:    lsl r2, r12, r2
-; ARM6-NEXT:    movge r3, #0
-; ARM6-NEXT:    lsrge r2, r12, lr
-; ARM6-NEXT:    and r1, r3, r1
-; ARM6-NEXT:    and r0, r2, r0
-; ARM6-NEXT:    orr r0, r0, r1
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    pop {r11, pc}
-;
-; ARM78-LABEL: scalar_i64_signbit_eq:
-; ARM78:       @ %bb.0:
-; ARM78-NEXT:    push {r11, lr}
-; ARM78-NEXT:    mov r12, #-2147483648
-; ARM78-NEXT:    sub lr, r2, #32
-; ARM78-NEXT:    lsr r3, r12, r2
-; ARM78-NEXT:    rsb r2, r2, #32
-; ARM78-NEXT:    cmp lr, #0
-; ARM78-NEXT:    lsl r2, r12, r2
-; ARM78-NEXT:    movwge r3, #0
-; ARM78-NEXT:    lsrge r2, r12, lr
-; ARM78-NEXT:    and r1, r3, r1
-; ARM78-NEXT:    and r0, r2, r0
-; ARM78-NEXT:    orr r0, r0, r1
-; ARM78-NEXT:    clz r0, r0
-; ARM78-NEXT:    lsr r0, r0, #5
-; ARM78-NEXT:    pop {r11, pc}
+; ARM-LABEL: scalar_i64_signbit_eq:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    rsb r3, r2, #32
+; ARM-NEXT:    lsr r3, r0, r3
+; ARM-NEXT:    orr r1, r3, r1, lsl r2
+; ARM-NEXT:    sub r2, r2, #32
+; ARM-NEXT:    cmp r2, #0
+; ARM-NEXT:    lslge r1, r0, r2
+; ARM-NEXT:    mvn r0, r1
+; ARM-NEXT:    lsr r0, r0, #31
+; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i64_signbit_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    mov r4, r1
-; THUMB6-NEXT:    mov r5, r0
+; THUMB6-NEXT:    push {r7, lr}
+; THUMB6-NEXT:    bl __ashldi3
 ; THUMB6-NEXT:    movs r0, #1
-; THUMB6-NEXT:    lsls r1, r0, #31
-; THUMB6-NEXT:    movs r0, #0
-; THUMB6-NEXT:    bl __lshrdi3
-; THUMB6-NEXT:    ands r1, r4
-; THUMB6-NEXT:    ands r0, r5
-; THUMB6-NEXT:    orrs r0, r1
-; THUMB6-NEXT:    rsbs r1, r0, #0
-; THUMB6-NEXT:    adcs r0, r1
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
-;
-; THUMB7-LABEL: scalar_i64_signbit_eq:
-; THUMB7:       @ %bb.0:
-; THUMB7-NEXT:    push {r7, lr}
-; THUMB7-NEXT:    rsb.w r3, r2, #32
-; THUMB7-NEXT:    mov.w r12, #-2147483648
-; THUMB7-NEXT:    sub.w lr, r2, #32
-; THUMB7-NEXT:    lsr.w r2, r12, r2
-; THUMB7-NEXT:    lsl.w r3, r12, r3
-; THUMB7-NEXT:    cmp.w lr, #0
-; THUMB7-NEXT:    it ge
-; THUMB7-NEXT:    lsrge.w r3, r12, lr
-; THUMB7-NEXT:    it ge
-; THUMB7-NEXT:    movge r2, #0
-; THUMB7-NEXT:    ands r0, r3
-; THUMB7-NEXT:    ands r1, r2
-; THUMB7-NEXT:    orrs r0, r1
-; THUMB7-NEXT:    clz r0, r0
-; THUMB7-NEXT:    lsrs r0, r0, #5
-; THUMB7-NEXT:    pop {r7, pc}
+; THUMB6-NEXT:    lsls r2, r0, #31
+; THUMB6-NEXT:    ands r2, r1
+; THUMB6-NEXT:    rsbs r0, r2, #0
+; THUMB6-NEXT:    adcs r0, r2
+; THUMB6-NEXT:    pop {r7, pc}
 ;
-; THUMB8-LABEL: scalar_i64_signbit_eq:
-; THUMB8:       @ %bb.0:
-; THUMB8-NEXT:    .save {r7, lr}
-; THUMB8-NEXT:    push {r7, lr}
-; THUMB8-NEXT:    rsb.w r3, r2, #32
-; THUMB8-NEXT:    sub.w lr, r2, #32
-; THUMB8-NEXT:    mov.w r12, #-2147483648
-; THUMB8-NEXT:    cmp.w lr, #0
-; THUMB8-NEXT:    lsl.w r3, r12, r3
-; THUMB8-NEXT:    lsr.w r2, r12, r2
-; THUMB8-NEXT:    it ge
-; THUMB8-NEXT:    lsrge.w r3, r12, lr
-; THUMB8-NEXT:    it ge
-; THUMB8-NEXT:    movge r2, #0
-; THUMB8-NEXT:    ands r0, r3
-; THUMB8-NEXT:    ands r1, r2
-; THUMB8-NEXT:    orrs r0, r1
-; THUMB8-NEXT:    clz r0, r0
-; THUMB8-NEXT:    lsrs r0, r0, #5
-; THUMB8-NEXT:    pop {r7, pc}
+; THUMB78-LABEL: scalar_i64_signbit_eq:
+; THUMB78:       @ %bb.0:
+; THUMB78-NEXT:    rsb.w r3, r2, #32
+; THUMB78-NEXT:    lsls r1, r2
+; THUMB78-NEXT:    subs r2, #32
+; THUMB78-NEXT:    lsr.w r3, r0, r3
+; THUMB78-NEXT:    orrs r1, r3
+; THUMB78-NEXT:    cmp r2, #0
+; THUMB78-NEXT:    it ge
+; THUMB78-NEXT:    lslge.w r1, r0, r2
+; THUMB78-NEXT:    mvns r0, r1
+; THUMB78-NEXT:    lsrs r0, r0, #31
+; THUMB78-NEXT:    bx lr
   %t0 = lshr i64 9223372036854775808, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i64_lowestbit_eq(i64 %x, i64 %y) nounwind {
 ; ARM6-LABEL: scalar_i64_lowestbit_eq:
 ; ARM6:       @ %bb.0:
+; ARM6-NEXT:    sub r1, r2, #32
+; ARM6-NEXT:    lsl r0, r0, r2
+; ARM6-NEXT:    cmp r1, #0
 ; ARM6-NEXT:    mov r1, #1
-; ARM6-NEXT:    lsr r1, r1, r2
-; ARM6-NEXT:    sub r2, r2, #32
-; ARM6-NEXT:    cmp r2, #0
-; ARM6-NEXT:    movge r1, #0
-; ARM6-NEXT:    and r0, r1, r0
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    lsr r0, r0, #5
+; ARM6-NEXT:    movge r0, #0
+; ARM6-NEXT:    bic r0, r1, r0
 ; ARM6-NEXT:    bx lr
 ;
 ; ARM78-LABEL: scalar_i64_lowestbit_eq:
 ; ARM78:       @ %bb.0:
+; ARM78-NEXT:    sub r1, r2, #32
+; ARM78-NEXT:    lsl r0, r0, r2
+; ARM78-NEXT:    cmp r1, #0
 ; ARM78-NEXT:    mov r1, #1
-; ARM78-NEXT:    lsr r1, r1, r2
-; ARM78-NEXT:    sub r2, r2, #32
-; ARM78-NEXT:    cmp r2, #0
-; ARM78-NEXT:    movwge r1, #0
-; ARM78-NEXT:    and r0, r1, r0
-; ARM78-NEXT:    clz r0, r0
-; ARM78-NEXT:    lsr r0, r0, #5
+; ARM78-NEXT:    movwge r0, #0
+; ARM78-NEXT:    bic r0, r1, r0
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i64_lowestbit_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    mov r4, r1
-; THUMB6-NEXT:    mov r5, r0
-; THUMB6-NEXT:    movs r0, #1
-; THUMB6-NEXT:    movs r1, #0
-; THUMB6-NEXT:    bl __lshrdi3
-; THUMB6-NEXT:    ands r1, r4
-; THUMB6-NEXT:    ands r0, r5
-; THUMB6-NEXT:    orrs r0, r1
-; THUMB6-NEXT:    rsbs r1, r0, #0
+; THUMB6-NEXT:    push {r7, lr}
+; THUMB6-NEXT:    bl __ashldi3
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    ands r1, r0
+; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    pop {r7, pc}
 ;
-; THUMB78-LABEL: scalar_i64_lowestbit_eq:
-; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    movs r1, #1
-; THUMB78-NEXT:    lsrs r1, r2
-; THUMB78-NEXT:    subs r2, #32
-; THUMB78-NEXT:    cmp r2, #0
-; THUMB78-NEXT:    it ge
-; THUMB78-NEXT:    movge r1, #0
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
-; THUMB78-NEXT:    bx lr
+; THUMB7-LABEL: scalar_i64_lowestbit_eq:
+; THUMB7:       @ %bb.0:
+; THUMB7-NEXT:    sub.w r1, r2, #32
+; THUMB7-NEXT:    lsls r0, r2
+; THUMB7-NEXT:    cmp r1, #0
+; THUMB7-NEXT:    it ge
+; THUMB7-NEXT:    movge r0, #0
+; THUMB7-NEXT:    movs r1, #1
+; THUMB7-NEXT:    bic.w r0, r1, r0
+; THUMB7-NEXT:    bx lr
+;
+; THUMB8-LABEL: scalar_i64_lowestbit_eq:
+; THUMB8:       @ %bb.0:
+; THUMB8-NEXT:    lsls r0, r2
+; THUMB8-NEXT:    sub.w r1, r2, #32
+; THUMB8-NEXT:    cmp r1, #0
+; THUMB8-NEXT:    it ge
+; THUMB8-NEXT:    movge r0, #0
+; THUMB8-NEXT:    movs r1, #1
+; THUMB8-NEXT:    bic.w r0, r1, r0
+; THUMB8-NEXT:    bx lr
   %t0 = lshr i64 1, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i64_bitsinmiddle_eq(i64 %x, i64 %y) nounwind {
 ; ARM6-LABEL: scalar_i64_bitsinmiddle_eq:
 ; ARM6:       @ %bb.0:
-; ARM6-NEXT:    push {r11, lr}
-; ARM6-NEXT:    mov r12, #255
-; ARM6-NEXT:    sub lr, r2, #32
-; ARM6-NEXT:    orr r12, r12, #65280
-; ARM6-NEXT:    cmp lr, #0
-; ARM6-NEXT:    lsr r3, r12, r2
-; ARM6-NEXT:    movge r3, #0
-; ARM6-NEXT:    and r1, r3, r1
-; ARM6-NEXT:    mov r3, #16711680
-; ARM6-NEXT:    cmp lr, #0
-; ARM6-NEXT:    orr r3, r3, #-16777216
-; ARM6-NEXT:    lsr r3, r3, r2
-; ARM6-NEXT:    rsb r2, r2, #32
-; ARM6-NEXT:    orr r2, r3, r12, lsl r2
-; ARM6-NEXT:    lsrge r2, r12, lr
-; ARM6-NEXT:    and r0, r2, r0
-; ARM6-NEXT:    orr r0, r0, r1
+; ARM6-NEXT:    rsb r3, r2, #32
+; ARM6-NEXT:    lsr r3, r0, r3
+; ARM6-NEXT:    orr r1, r3, r1, lsl r2
+; ARM6-NEXT:    sub r3, r2, #32
+; ARM6-NEXT:    cmp r3, #0
+; ARM6-NEXT:    lslge r1, r0, r3
+; ARM6-NEXT:    lsl r0, r0, r2
+; ARM6-NEXT:    movge r0, #0
+; ARM6-NEXT:    pkhbt r0, r1, r0
 ; ARM6-NEXT:    clz r0, r0
 ; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    pop {r11, pc}
+; ARM6-NEXT:    bx lr
 ;
 ; ARM78-LABEL: scalar_i64_bitsinmiddle_eq:
 ; ARM78:       @ %bb.0:
-; ARM78-NEXT:    push {r11, lr}
-; ARM78-NEXT:    movw r12, #65535
-; ARM78-NEXT:    sub lr, r2, #32
-; ARM78-NEXT:    lsr r3, r12, r2
-; ARM78-NEXT:    cmp lr, #0
-; ARM78-NEXT:    movwge r3, #0
-; ARM78-NEXT:    and r1, r3, r1
-; ARM78-NEXT:    movw r3, #0
-; ARM78-NEXT:    cmp lr, #0
-; ARM78-NEXT:    movt r3, #65535
-; ARM78-NEXT:    lsr r3, r3, r2
-; ARM78-NEXT:    rsb r2, r2, #32
-; ARM78-NEXT:    orr r2, r3, r12, lsl r2
-; ARM78-NEXT:    lsrge r2, r12, lr
-; ARM78-NEXT:    and r0, r2, r0
-; ARM78-NEXT:    orr r0, r0, r1
+; ARM78-NEXT:    rsb r3, r2, #32
+; ARM78-NEXT:    lsr r3, r0, r3
+; ARM78-NEXT:    orr r1, r3, r1, lsl r2
+; ARM78-NEXT:    sub r3, r2, #32
+; ARM78-NEXT:    cmp r3, #0
+; ARM78-NEXT:    lslge r1, r0, r3
+; ARM78-NEXT:    lsl r0, r0, r2
+; ARM78-NEXT:    movwge r0, #0
+; ARM78-NEXT:    pkhbt r0, r1, r0
 ; ARM78-NEXT:    clz r0, r0
 ; ARM78-NEXT:    lsr r0, r0, #5
-; ARM78-NEXT:    pop {r11, pc}
+; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i64_bitsinmiddle_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    mov r4, r1
-; THUMB6-NEXT:    mov r5, r0
-; THUMB6-NEXT:    ldr r0, .LCPI11_0
-; THUMB6-NEXT:    ldr r1, .LCPI11_1
-; THUMB6-NEXT:    bl __lshrdi3
-; THUMB6-NEXT:    ands r1, r4
-; THUMB6-NEXT:    ands r0, r5
-; THUMB6-NEXT:    orrs r0, r1
-; THUMB6-NEXT:    rsbs r1, r0, #0
+; THUMB6-NEXT:    push {r7, lr}
+; THUMB6-NEXT:    bl __ashldi3
+; THUMB6-NEXT:    ldr r2, .LCPI11_0
+; THUMB6-NEXT:    ands r2, r0
+; THUMB6-NEXT:    uxth r0, r1
+; THUMB6-NEXT:    adds r1, r2, r0
+; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    pop {r7, pc}
 ; THUMB6-NEXT:    .p2align 2
 ; THUMB6-NEXT:  @ %bb.1:
 ; THUMB6-NEXT:  .LCPI11_0:
 ; THUMB6-NEXT:    .long 4294901760 @ 0xffff0000
-; THUMB6-NEXT:  .LCPI11_1:
-; THUMB6-NEXT:    .long 65535 @ 0xffff
 ;
-; THUMB7-LABEL: scalar_i64_bitsinmiddle_eq:
-; THUMB7:       @ %bb.0:
-; THUMB7-NEXT:    push {r7, lr}
-; THUMB7-NEXT:    movs r3, #0
-; THUMB7-NEXT:    movw lr, #65535
-; THUMB7-NEXT:    movt r3, #65535
-; THUMB7-NEXT:    lsr.w r12, r3, r2
-; THUMB7-NEXT:    rsb.w r3, r2, #32
-; THUMB7-NEXT:    lsl.w r3, lr, r3
-; THUMB7-NEXT:    orr.w r3, r3, r12
-; THUMB7-NEXT:    sub.w r12, r2, #32
-; THUMB7-NEXT:    lsr.w r2, lr, r2
-; THUMB7-NEXT:    cmp.w r12, #0
-; THUMB7-NEXT:    it ge
-; THUMB7-NEXT:    lsrge.w r3, lr, r12
-; THUMB7-NEXT:    it ge
-; THUMB7-NEXT:    movge r2, #0
-; THUMB7-NEXT:    ands r0, r3
-; THUMB7-NEXT:    ands r1, r2
-; THUMB7-NEXT:    orrs r0, r1
-; THUMB7-NEXT:    clz r0, r0
-; THUMB7-NEXT:    lsrs r0, r0, #5
-; THUMB7-NEXT:    pop {r7, pc}
-;
-; THUMB8-LABEL: scalar_i64_bitsinmiddle_eq:
-; THUMB8:       @ %bb.0:
-; THUMB8-NEXT:    .save {r7, lr}
-; THUMB8-NEXT:    push {r7, lr}
-; THUMB8-NEXT:    movs r3, #0
-; THUMB8-NEXT:    movw lr, #65535
-; THUMB8-NEXT:    movt r3, #65535
-; THUMB8-NEXT:    lsr.w r12, r3, r2
-; THUMB8-NEXT:    rsb.w r3, r2, #32
-; THUMB8-NEXT:    lsl.w r3, lr, r3
-; THUMB8-NEXT:    orr.w r3, r3, r12
-; THUMB8-NEXT:    sub.w r12, r2, #32
-; THUMB8-NEXT:    cmp.w r12, #0
-; THUMB8-NEXT:    lsr.w r2, lr, r2
-; THUMB8-NEXT:    it ge
-; THUMB8-NEXT:    lsrge.w r3, lr, r12
-; THUMB8-NEXT:    it ge
-; THUMB8-NEXT:    movge r2, #0
-; THUMB8-NEXT:    ands r0, r3
-; THUMB8-NEXT:    ands r1, r2
-; THUMB8-NEXT:    orrs r0, r1
-; THUMB8-NEXT:    clz r0, r0
-; THUMB8-NEXT:    lsrs r0, r0, #5
-; THUMB8-NEXT:    pop {r7, pc}
+; THUMB78-LABEL: scalar_i64_bitsinmiddle_eq:
+; THUMB78:       @ %bb.0:
+; THUMB78-NEXT:    rsb.w r3, r2, #32
+; THUMB78-NEXT:    lsls r1, r2
+; THUMB78-NEXT:    lsr.w r3, r0, r3
+; THUMB78-NEXT:    orrs r1, r3
+; THUMB78-NEXT:    sub.w r3, r2, #32
+; THUMB78-NEXT:    cmp r3, #0
+; THUMB78-NEXT:    it ge
+; THUMB78-NEXT:    lslge.w r1, r0, r3
+; THUMB78-NEXT:    lsl.w r0, r0, r2
+; THUMB78-NEXT:    it ge
+; THUMB78-NEXT:    movge r0, #0
+; THUMB78-NEXT:    pkhbt r0, r1, r0
+; THUMB78-NEXT:    clz r0, r0
+; THUMB78-NEXT:    lsrs r0, r0, #5
+; THUMB78-NEXT:    bx lr
   %t0 = lshr i64 281474976645120, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;
 ; A few trivial vector tests
 ;------------------------------------------------------------------------------;
 
 define <4 x i1> @vec_4xi32_splat_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; ARM6-LABEL: vec_4xi32_splat_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    push {r11, lr}
 ; ARM6-NEXT:    ldr r12, [sp, #8]
 ; ARM6-NEXT:    mov lr, #1
-; ARM6-NEXT:    and r0, r0, lr, lsr r12
+; ARM6-NEXT:    bic r0, lr, r0, lsl r12
 ; ARM6-NEXT:    ldr r12, [sp, #12]
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    and r1, r1, lr, lsr r12
+; ARM6-NEXT:    bic r1, lr, r1, lsl r12
 ; ARM6-NEXT:    ldr r12, [sp, #16]
-; ARM6-NEXT:    clz r1, r1
-; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    and r2, r2, lr, lsr r12
+; ARM6-NEXT:    bic r2, lr, r2, lsl r12
 ; ARM6-NEXT:    ldr r12, [sp, #20]
-; ARM6-NEXT:    clz r2, r2
-; ARM6-NEXT:    lsr r1, r1, #5
-; ARM6-NEXT:    and r3, r3, lr, lsr r12
-; ARM6-NEXT:    lsr r2, r2, #5
-; ARM6-NEXT:    clz r3, r3
-; ARM6-NEXT:    lsr r3, r3, #5
+; ARM6-NEXT:    bic r3, lr, r3, lsl r12
 ; ARM6-NEXT:    pop {r11, pc}
 ;
 ; ARM78-LABEL: vec_4xi32_splat_eq:
 ; ARM78:       @ %bb.0:
+; ARM78-NEXT:    vmov d17, r2, r3
 ; ARM78-NEXT:    mov r12, sp
-; ARM78-NEXT:    vld1.64 {d16, d17}, [r12]
-; ARM78-NEXT:    vmov.i32 q9, #0x1
-; ARM78-NEXT:    vneg.s32 q8, q8
-; ARM78-NEXT:    vshl.u32 q8, q9, q8
-; ARM78-NEXT:    vmov d19, r2, r3
-; ARM78-NEXT:    vmov d18, r0, r1
-; ARM78-NEXT:    vtst.32 q8, q8, q9
+; ARM78-NEXT:    vld1.64 {d18, d19}, [r12]
+; ARM78-NEXT:    vmov d16, r0, r1
+; ARM78-NEXT:    vmov.i32 q10, #0x1
+; ARM78-NEXT:    vshl.u32 q8, q8, q9
+; ARM78-NEXT:    vtst.32 q8, q8, q10
 ; ARM78-NEXT:    vmvn q8, q8
 ; ARM78-NEXT:    vmovn.i32 d16, q8
 ; ARM78-NEXT:    vmov r0, r1, d16
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: vec_4xi32_splat_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r6, lr}
-; THUMB6-NEXT:    ldr r5, [sp, #16]
+; THUMB6-NEXT:    push {r4, r5, r7, lr}
+; THUMB6-NEXT:    ldr r4, [sp, #16]
+; THUMB6-NEXT:    lsls r0, r4
 ; THUMB6-NEXT:    movs r4, #1
-; THUMB6-NEXT:    mov r6, r4
-; THUMB6-NEXT:    lsrs r6, r5
-; THUMB6-NEXT:    ands r6, r0
-; THUMB6-NEXT:    rsbs r0, r6, #0
-; THUMB6-NEXT:    adcs r0, r6
+; THUMB6-NEXT:    ands r0, r4
+; THUMB6-NEXT:    rsbs r5, r0, #0
+; THUMB6-NEXT:    adcs r0, r5
 ; THUMB6-NEXT:    ldr r5, [sp, #20]
-; THUMB6-NEXT:    mov r6, r4
-; THUMB6-NEXT:    lsrs r6, r5
-; THUMB6-NEXT:    ands r6, r1
-; THUMB6-NEXT:    rsbs r1, r6, #0
-; THUMB6-NEXT:    adcs r1, r6
+; THUMB6-NEXT:    lsls r1, r5
+; THUMB6-NEXT:    ands r1, r4
+; THUMB6-NEXT:    rsbs r5, r1, #0
+; THUMB6-NEXT:    adcs r1, r5
 ; THUMB6-NEXT:    ldr r5, [sp, #24]
-; THUMB6-NEXT:    mov r6, r4
-; THUMB6-NEXT:    lsrs r6, r5
-; THUMB6-NEXT:    ands r6, r2
-; THUMB6-NEXT:    rsbs r2, r6, #0
-; THUMB6-NEXT:    adcs r2, r6
+; THUMB6-NEXT:    lsls r2, r5
+; THUMB6-NEXT:    ands r2, r4
+; THUMB6-NEXT:    rsbs r5, r2, #0
+; THUMB6-NEXT:    adcs r2, r5
 ; THUMB6-NEXT:    ldr r5, [sp, #28]
-; THUMB6-NEXT:    lsrs r4, r5
-; THUMB6-NEXT:    ands r4, r3
-; THUMB6-NEXT:    rsbs r3, r4, #0
+; THUMB6-NEXT:    lsls r3, r5
+; THUMB6-NEXT:    ands r3, r4
+; THUMB6-NEXT:    rsbs r4, r3, #0
 ; THUMB6-NEXT:    adcs r3, r4
-; THUMB6-NEXT:    pop {r4, r5, r6, pc}
+; THUMB6-NEXT:    pop {r4, r5, r7, pc}
 ;
 ; THUMB78-LABEL: vec_4xi32_splat_eq:
 ; THUMB78:       @ %bb.0:
+; THUMB78-NEXT:    vmov d17, r2, r3
 ; THUMB78-NEXT:    mov r12, sp
-; THUMB78-NEXT:    vld1.64 {d16, d17}, [r12]
-; THUMB78-NEXT:    vmov.i32 q9, #0x1
-; THUMB78-NEXT:    vneg.s32 q8, q8
-; THUMB78-NEXT:    vshl.u32 q8, q9, q8
-; THUMB78-NEXT:    vmov d19, r2, r3
-; THUMB78-NEXT:    vmov d18, r0, r1
-; THUMB78-NEXT:    vtst.32 q8, q8, q9
+; THUMB78-NEXT:    vld1.64 {d18, d19}, [r12]
+; THUMB78-NEXT:    vmov d16, r0, r1
+; THUMB78-NEXT:    vmov.i32 q10, #0x1
+; THUMB78-NEXT:    vshl.u32 q8, q8, q9
+; THUMB78-NEXT:    vtst.32 q8, q8, q10
 ; THUMB78-NEXT:    vmvn q8, q8
 ; THUMB78-NEXT:    vmovn.i32 d16, q8
 ; THUMB78-NEXT:    vmov r0, r1, d16
 ; THUMB78-NEXT:    bx lr
   %t0 = lshr <4 x i32> <i32 1, i32 1, i32 1, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 
 define <4 x i1> @vec_4xi32_nonsplat_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; ARM6-LABEL: vec_4xi32_nonsplat_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    ldr r12, [sp, #4]
 ; ARM6-NEXT:    mov r0, #1
-; ARM6-NEXT:    and r0, r1, r0, lsr r12
+; ARM6-NEXT:    bic r1, r0, r1, lsl r12
 ; ARM6-NEXT:    ldr r12, [sp, #8]
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    lsr r1, r0, #5
 ; ARM6-NEXT:    mov r0, #65280
 ; ARM6-NEXT:    orr r0, r0, #16711680
-; ARM6-NEXT:    and r0, r2, r0, lsr r12
-; ARM6-NEXT:    ldr r12, [sp, #12]
+; ARM6-NEXT:    and r0, r0, r2, lsl r12
 ; ARM6-NEXT:    clz r0, r0
 ; ARM6-NEXT:    lsr r2, r0, #5
-; ARM6-NEXT:    mov r0, #-2147483648
-; ARM6-NEXT:    and r0, r3, r0, lsr r12
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    lsr r3, r0, #5
+; ARM6-NEXT:    ldr r0, [sp, #12]
+; ARM6-NEXT:    mvn r0, r3, lsl r0
+; ARM6-NEXT:    lsr r3, r0, #31
 ; ARM6-NEXT:    mov r0, #1
 ; ARM6-NEXT:    bx lr
 ;
 ; ARM78-LABEL: vec_4xi32_nonsplat_eq:
 ; ARM78:       @ %bb.0:
 ; ARM78-NEXT:    mov r12, sp
 ; ARM78-NEXT:    vld1.64 {d16, d17}, [r12]
 ; ARM78-NEXT:    adr r12, .LCPI13_0

 ; ARM78-NEXT:  .LCPI13_0:
 ; ARM78-NEXT:    .long 0 @ 0x0
 ; ARM78-NEXT:    .long 1 @ 0x1
 ; ARM78-NEXT:    .long 16776960 @ 0xffff00
 ; ARM78-NEXT:    .long 2147483648 @ 0x80000000
 ;
 ; THUMB6-LABEL: vec_4xi32_nonsplat_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    ldr r4, [sp, #20]
+; THUMB6-NEXT:    push {r4, lr}
+; THUMB6-NEXT:    ldr r0, [sp, #12]
+; THUMB6-NEXT:    lsls r1, r0
 ; THUMB6-NEXT:    movs r0, #1
-; THUMB6-NEXT:    mov r5, r0
-; THUMB6-NEXT:    lsrs r5, r4
-; THUMB6-NEXT:    ands r5, r1
-; THUMB6-NEXT:    rsbs r1, r5, #0
-; THUMB6-NEXT:    adcs r1, r5
-; THUMB6-NEXT:    ldr r4, [sp, #24]
-; THUMB6-NEXT:    ldr r5, .LCPI13_0
-; THUMB6-NEXT:    lsrs r5, r4
-; THUMB6-NEXT:    ands r5, r2
-; THUMB6-NEXT:    rsbs r2, r5, #0
-; THUMB6-NEXT:    adcs r2, r5
+; THUMB6-NEXT:    ands r1, r0
+; THUMB6-NEXT:    rsbs r4, r1, #0
+; THUMB6-NEXT:    adcs r1, r4
+; THUMB6-NEXT:    ldr r4, [sp, #16]
+; THUMB6-NEXT:    lsls r2, r4
+; THUMB6-NEXT:    ldr r4, .LCPI13_0
+; THUMB6-NEXT:    ands r4, r2
+; THUMB6-NEXT:    rsbs r2, r4, #0
+; THUMB6-NEXT:    adcs r2, r4
+; THUMB6-NEXT:    ldr r4, [sp, #20]
+; THUMB6-NEXT:    lsls r3, r4
 ; THUMB6-NEXT:    lsls r4, r0, #31
-; THUMB6-NEXT:    ldr r5, [sp, #28]
-; THUMB6-NEXT:    lsrs r4, r5
 ; THUMB6-NEXT:    ands r4, r3
 ; THUMB6-NEXT:    rsbs r3, r4, #0
 ; THUMB6-NEXT:    adcs r3, r4
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    pop {r4, pc}
 ; THUMB6-NEXT:    .p2align 2
 ; THUMB6-NEXT:  @ %bb.1:
 ; THUMB6-NEXT:  .LCPI13_0:
 ; THUMB6-NEXT:    .long 16776960 @ 0xffff00
 ;
 ; THUMB78-LABEL: vec_4xi32_nonsplat_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    mov r12, sp

 
 define <4 x i1> @vec_4xi32_nonsplat_undef0_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; ARM6-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    push {r11, lr}
 ; ARM6-NEXT:    ldr r2, [sp, #12]
 ; ARM6-NEXT:    mov lr, #1
 ; ARM6-NEXT:    ldr r12, [sp, #8]
-; ARM6-NEXT:    and r1, r1, lr, lsr r2
+; ARM6-NEXT:    bic r1, lr, r1, lsl r2
 ; ARM6-NEXT:    ldr r2, [sp, #20]
-; ARM6-NEXT:    and r0, r0, lr, lsr r12
-; ARM6-NEXT:    clz r1, r1
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    and r2, r3, lr, lsr r2
-; ARM6-NEXT:    lsr r1, r1, #5
-; ARM6-NEXT:    clz r2, r2
-; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    lsr r3, r2, #5
+; ARM6-NEXT:    bic r0, lr, r0, lsl r12
+; ARM6-NEXT:    bic r3, lr, r3, lsl r2
 ; ARM6-NEXT:    mov r2, #1
 ; ARM6-NEXT:    pop {r11, pc}
 ;
 ; ARM78-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; ARM78:       @ %bb.0:
+; ARM78-NEXT:    vmov d17, r2, r3
 ; ARM78-NEXT:    mov r12, sp
-; ARM78-NEXT:    vld1.64 {d16, d17}, [r12]
-; ARM78-NEXT:    vmov.i32 q9, #0x1
-; ARM78-NEXT:    vneg.s32 q8, q8
-; ARM78-NEXT:    vshl.u32 q8, q9, q8
-; ARM78-NEXT:    vmov d19, r2, r3
-; ARM78-NEXT:    vmov d18, r0, r1
-; ARM78-NEXT:    vtst.32 q8, q8, q9
+; ARM78-NEXT:    vld1.64 {d18, d19}, [r12]
+; ARM78-NEXT:    vmov d16, r0, r1
+; ARM78-NEXT:    vmov.i32 q10, #0x1
+; ARM78-NEXT:    vshl.u32 q8, q8, q9
+; ARM78-NEXT:    vtst.32 q8, q8, q10
 ; ARM78-NEXT:    vmvn q8, q8
 ; ARM78-NEXT:    vmovn.i32 d16, q8
 ; ARM78-NEXT:    vmov r0, r1, d16
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    ldr r4, [sp, #16]
+; THUMB6-NEXT:    push {r4, lr}
+; THUMB6-NEXT:    ldr r2, [sp, #8]
+; THUMB6-NEXT:    lsls r0, r2
 ; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsrs r5, r4
-; THUMB6-NEXT:    ands r5, r0
-; THUMB6-NEXT:    rsbs r0, r5, #0
-; THUMB6-NEXT:    adcs r0, r5
+; THUMB6-NEXT:    ands r0, r2
+; THUMB6-NEXT:    rsbs r4, r0, #0
+; THUMB6-NEXT:    adcs r0, r4
+; THUMB6-NEXT:    ldr r4, [sp, #12]
+; THUMB6-NEXT:    lsls r1, r4
+; THUMB6-NEXT:    ands r1, r2
+; THUMB6-NEXT:    rsbs r4, r1, #0
+; THUMB6-NEXT:    adcs r1, r4
 ; THUMB6-NEXT:    ldr r4, [sp, #20]
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsrs r5, r4
-; THUMB6-NEXT:    ands r5, r1
-; THUMB6-NEXT:    rsbs r1, r5, #0
-; THUMB6-NEXT:    adcs r1, r5
-; THUMB6-NEXT:    ldr r4, [sp, #28]
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsrs r5, r4
-; THUMB6-NEXT:    ands r5, r3
-; THUMB6-NEXT:    rsbs r3, r5, #0
-; THUMB6-NEXT:    adcs r3, r5
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    lsls r3, r4
+; THUMB6-NEXT:    ands r3, r2
+; THUMB6-NEXT:    rsbs r4, r3, #0
+; THUMB6-NEXT:    adcs r3, r4
+; THUMB6-NEXT:    pop {r4, pc}
 ;
 ; THUMB78-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; THUMB78:       @ %bb.0:
+; THUMB78-NEXT:    vmov d17, r2, r3
 ; THUMB78-NEXT:    mov r12, sp
-; THUMB78-NEXT:    vld1.64 {d16, d17}, [r12]
-; THUMB78-NEXT:    vmov.i32 q9, #0x1
-; THUMB78-NEXT:    vneg.s32 q8, q8
-; THUMB78-NEXT:    vshl.u32 q8, q9, q8
-; THUMB78-NEXT:    vmov d19, r2, r3
-; THUMB78-NEXT:    vmov d18, r0, r1
-; THUMB78-NEXT:    vtst.32 q8, q8, q9
+; THUMB78-NEXT:    vld1.64 {d18, d19}, [r12]
+; THUMB78-NEXT:    vmov d16, r0, r1
+; THUMB78-NEXT:    vmov.i32 q10, #0x1
+; THUMB78-NEXT:    vshl.u32 q8, q8, q9
+; THUMB78-NEXT:    vtst.32 q8, q8, q10
 ; THUMB78-NEXT:    vmvn q8, q8
 ; THUMB78-NEXT:    vmovn.i32 d16, q8
 ; THUMB78-NEXT:    vmov r0, r1, d16
 ; THUMB78-NEXT:    bx lr
   %t0 = lshr <4 x i32> <i32 1, i32 1, i32 undef, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 define <4 x i1> @vec_4xi32_nonsplat_undef1_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; ARM6-LABEL: vec_4xi32_nonsplat_undef1_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    push {r11, lr}
 ; ARM6-NEXT:    ldr r2, [sp, #12]
 ; ARM6-NEXT:    mov lr, #1
 ; ARM6-NEXT:    ldr r12, [sp, #8]
-; ARM6-NEXT:    and r1, r1, lr, lsr r2
+; ARM6-NEXT:    bic r1, lr, r1, lsl r2
 ; ARM6-NEXT:    ldr r2, [sp, #20]
-; ARM6-NEXT:    and r0, r0, lr, lsr r12
-; ARM6-NEXT:    clz r1, r1
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    and r2, r3, lr, lsr r2
-; ARM6-NEXT:    lsr r1, r1, #5
-; ARM6-NEXT:    clz r2, r2
-; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    lsr r3, r2, #5
+; ARM6-NEXT:    bic r0, lr, r0, lsl r12
+; ARM6-NEXT:    bic r3, lr, r3, lsl r2
 ; ARM6-NEXT:    pop {r11, pc}
 ;
 ; ARM78-LABEL: vec_4xi32_nonsplat_undef1_eq:
 ; ARM78:       @ %bb.0:
 ; ARM78-NEXT:    mov r12, sp
 ; ARM78-NEXT:    vld1.64 {d16, d17}, [r12]
 ; ARM78-NEXT:    vmov.i32 q9, #0x1
 ; ARM78-NEXT:    vneg.s32 q8, q8
 ; ARM78-NEXT:    vshl.u32 q8, q9, q8
 ; ARM78-NEXT:    vmov d19, r2, r3
 ; ARM78-NEXT:    vmov d18, r0, r1
 ; ARM78-NEXT:    vtst.32 q8, q8, q9
 ; ARM78-NEXT:    vmvn q8, q8
 ; ARM78-NEXT:    vmovn.i32 d16, q8
 ; ARM78-NEXT:    vmov r0, r1, d16
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: vec_4xi32_nonsplat_undef1_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    ldr r4, [sp, #16]
+; THUMB6-NEXT:    push {r4, lr}
+; THUMB6-NEXT:    ldr r2, [sp, #8]
+; THUMB6-NEXT:    lsls r0, r2
 ; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsrs r5, r4
-; THUMB6-NEXT:    ands r5, r0
-; THUMB6-NEXT:    rsbs r0, r5, #0
-; THUMB6-NEXT:    adcs r0, r5
+; THUMB6-NEXT:    ands r0, r2
+; THUMB6-NEXT:    rsbs r4, r0, #0
+; THUMB6-NEXT:    adcs r0, r4
+; THUMB6-NEXT:    ldr r4, [sp, #12]
+; THUMB6-NEXT:    lsls r1, r4
+; THUMB6-NEXT:    ands r1, r2
+; THUMB6-NEXT:    rsbs r4, r1, #0
+; THUMB6-NEXT:    adcs r1, r4
 ; THUMB6-NEXT:    ldr r4, [sp, #20]
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsrs r5, r4
-; THUMB6-NEXT:    ands r5, r1
-; THUMB6-NEXT:    rsbs r1, r5, #0
-; THUMB6-NEXT:    adcs r1, r5
-; THUMB6-NEXT:    ldr r4, [sp, #28]
-; THUMB6-NEXT:    lsrs r2, r4
-; THUMB6-NEXT:    ands r2, r3
-; THUMB6-NEXT:    rsbs r3, r2, #0
+; THUMB6-NEXT:    lsls r3, r4
+; THUMB6-NEXT:    ands r3, r2
+; THUMB6-NEXT:    rsbs r2, r3, #0
 ; THUMB6-NEXT:    adcs r3, r2
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    pop {r4, pc}
 ;
 ; THUMB78-LABEL: vec_4xi32_nonsplat_undef1_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    mov r12, sp
 ; THUMB78-NEXT:    vld1.64 {d16, d17}, [r12]
 ; THUMB78-NEXT:    vmov.i32 q9, #0x1
 ; THUMB78-NEXT:    vneg.s32 q8, q8
 ; THUMB78-NEXT:    vshl.u32 q8, q9, q8

 }
 define <4 x i1> @vec_4xi32_nonsplat_undef2_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; ARM6-LABEL: vec_4xi32_nonsplat_undef2_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    push {r11, lr}
 ; ARM6-NEXT:    ldr r2, [sp, #12]
 ; ARM6-NEXT:    mov lr, #1
 ; ARM6-NEXT:    ldr r12, [sp, #8]
-; ARM6-NEXT:    and r1, r1, lr, lsr r2
+; ARM6-NEXT:    bic r1, lr, r1, lsl r2
 ; ARM6-NEXT:    ldr r2, [sp, #20]
-; ARM6-NEXT:    and r0, r0, lr, lsr r12
-; ARM6-NEXT:    clz r1, r1
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    and r2, r3, lr, lsr r2
-; ARM6-NEXT:    lsr r1, r1, #5
-; ARM6-NEXT:    clz r2, r2
-; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    lsr r3, r2, #5
+; ARM6-NEXT:    bic r0, lr, r0, lsl r12
+; ARM6-NEXT:    bic r3, lr, r3, lsl r2
 ; ARM6-NEXT:    pop {r11, pc}
 ;
 ; ARM78-LABEL: vec_4xi32_nonsplat_undef2_eq:
 ; ARM78:       @ %bb.0:
 ; ARM78-NEXT:    mov r12, sp
 ; ARM78-NEXT:    vld1.64 {d16, d17}, [r12]
 ; ARM78-NEXT:    vmov.i32 q9, #0x1
 ; ARM78-NEXT:    vneg.s32 q8, q8
 ; ARM78-NEXT:    vshl.u32 q8, q9, q8
 ; ARM78-NEXT:    vmov d19, r2, r3
 ; ARM78-NEXT:    vmov d18, r0, r1
 ; ARM78-NEXT:    vtst.32 q8, q8, q9
 ; ARM78-NEXT:    vmvn q8, q8
 ; ARM78-NEXT:    vmovn.i32 d16, q8
 ; ARM78-NEXT:    vmov r0, r1, d16
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: vec_4xi32_nonsplat_undef2_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    ldr r4, [sp, #16]
+; THUMB6-NEXT:    push {r4, lr}
+; THUMB6-NEXT:    ldr r2, [sp, #8]
+; THUMB6-NEXT:    lsls r0, r2
 ; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsrs r5, r4
-; THUMB6-NEXT:    ands r5, r0
-; THUMB6-NEXT:    rsbs r0, r5, #0
-; THUMB6-NEXT:    adcs r0, r5
+; THUMB6-NEXT:    ands r0, r2
+; THUMB6-NEXT:    rsbs r4, r0, #0
+; THUMB6-NEXT:    adcs r0, r4
+; THUMB6-NEXT:    ldr r4, [sp, #12]
+; THUMB6-NEXT:    lsls r1, r4
+; THUMB6-NEXT:    ands r1, r2
+; THUMB6-NEXT:    rsbs r4, r1, #0
+; THUMB6-NEXT:    adcs r1, r4
 ; THUMB6-NEXT:    ldr r4, [sp, #20]
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsrs r5, r4
-; THUMB6-NEXT:    ands r5, r1
-; THUMB6-NEXT:    rsbs r1, r5, #0
-; THUMB6-NEXT:    adcs r1, r5
-; THUMB6-NEXT:    ldr r4, [sp, #28]
-; THUMB6-NEXT:    lsrs r2, r4
-; THUMB6-NEXT:    ands r2, r3
-; THUMB6-NEXT:    rsbs r3, r2, #0
+; THUMB6-NEXT:    lsls r3, r4
+; THUMB6-NEXT:    ands r3, r2
+; THUMB6-NEXT:    rsbs r2, r3, #0
 ; THUMB6-NEXT:    adcs r3, r2
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    pop {r4, pc}
 ;
 ; THUMB78-LABEL: vec_4xi32_nonsplat_undef2_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    mov r12, sp
 ; THUMB78-NEXT:    vld1.64 {d16, d17}, [r12]
 ; THUMB78-NEXT:    vmov.i32 q9, #0x1
 ; THUMB78-NEXT:    vneg.s32 q8, q8
 ; THUMB78-NEXT:    vshl.u32 q8, q9, q8
; THUMB78-NEXT:    bx lr
   ret <4 x i1> %res
 }
 
 ;------------------------------------------------------------------------------;
 ; A special tests
 ;------------------------------------------------------------------------------;
 
 define i1 @scalar_i8_signbit_ne(i8 %x, i8 %y) nounwind {
-; ARM6-LABEL: scalar_i8_signbit_ne:
-; ARM6:       @ %bb.0:
-; ARM6-NEXT:    uxtb r1, r1
-; ARM6-NEXT:    mov r2, #128
-; ARM6-NEXT:    and r0, r0, r2, lsr r1
-; ARM6-NEXT:    uxtb r0, r0
-; ARM6-NEXT:    cmp r0, #0
-; ARM6-NEXT:    movne r0, #1
-; ARM6-NEXT:    bx lr
-;
-; ARM78-LABEL: scalar_i8_signbit_ne:
-; ARM78:       @ %bb.0:
-; ARM78-NEXT:    uxtb r1, r1
-; ARM78-NEXT:    mov r2, #128
-; ARM78-NEXT:    and r0, r0, r2, lsr r1
-; ARM78-NEXT:    uxtb r0, r0
-; ARM78-NEXT:    cmp r0, #0
-; ARM78-NEXT:    movwne r0, #1
-; ARM78-NEXT:    bx lr
-;
-; THUMB6-LABEL: scalar_i8_signbit_ne:
-; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    uxtb r1, r1
-; THUMB6-NEXT:    movs r2, #128
-; THUMB6-NEXT:    lsrs r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxtb r0, r2
-; THUMB6-NEXT:    subs r1, r0, #1
-; THUMB6-NEXT:    sbcs r0, r1
-; THUMB6-NEXT:    bx lr
+; ARM-LABEL: scalar_i8_signbit_ne:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    uxtb r1, r1
+; ARM-NEXT:    lsl r0, r0, r1
+; ARM-NEXT:    uxtb r0, r0
+; ARM-NEXT:    lsr r0, r0, #7
+; ARM-NEXT:    bx lr
 ;
-; THUMB78-LABEL: scalar_i8_signbit_ne:
-; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    uxtb r1, r1
-; THUMB78-NEXT:    movs r2, #128
-; THUMB78-NEXT:    lsr.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxtb r0, r0
-; THUMB78-NEXT:    cmp r0, #0
-; THUMB78-NEXT:    it ne
-; THUMB78-NEXT:    movne r0, #1
-; THUMB78-NEXT:    bx lr
+; THUMB-LABEL: scalar_i8_signbit_ne:
+; THUMB:       @ %bb.0:
+; THUMB-NEXT:    uxtb r1, r1
+; THUMB-NEXT:    lsls r0, r1
+; THUMB-NEXT:    uxtb r0, r0
+; THUMB-NEXT:    lsrs r0, r0, #7
+; THUMB-NEXT:    bx lr
   %t0 = lshr i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp ne i8 %t1, 0 ;  we are perfectly happy with 'ne' predicate
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;
 ; What if X is a constant too?

test/CodeGen/ARM/hoist-and-by-const-from-shl-in-eqcmp-zero.ll

 ;------------------------------------------------------------------------------;
 
 ; i8 scalar
 
 define i1 @scalar_i8_signbit_eq(i8 %x, i8 %y) nounwind {
 ; ARM-LABEL: scalar_i8_signbit_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    uxtb r1, r1
-; ARM-NEXT:    mvn r2, #127
-; ARM-NEXT:    and r0, r0, r2, lsl r1
 ; ARM-NEXT:    uxtb r0, r0
-; ARM-NEXT:    clz r0, r0
-; ARM-NEXT:    lsr r0, r0, #5
+; ARM-NEXT:    lsr r0, r0, r1
+; ARM-NEXT:    mov r1, #1
+; ARM-NEXT:    uxtb r0, r0
+; ARM-NEXT:    eor r0, r1, r0, lsr #7
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i8_signbit_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxtb r1, r1
-; THUMB6-NEXT:    movs r2, #127
-; THUMB6-NEXT:    mvns r2, r2
-; THUMB6-NEXT:    lsls r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxtb r1, r2
+; THUMB6-NEXT:    uxtb r0, r0
+; THUMB6-NEXT:    lsrs r0, r1
+; THUMB6-NEXT:    movs r1, #128
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
-; THUMB78-LABEL: scalar_i8_signbit_eq:
-; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    uxtb r1, r1
-; THUMB78-NEXT:    mvn r2, #127
-; THUMB78-NEXT:    lsl.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxtb r0, r0
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
-; THUMB78-NEXT:    bx lr
+; THUMB7-LABEL: scalar_i8_signbit_eq:
+; THUMB7:       @ %bb.0:
+; THUMB7-NEXT:    uxtb r1, r1
+; THUMB7-NEXT:    uxtb r0, r0
+; THUMB7-NEXT:    lsrs r0, r1
+; THUMB7-NEXT:    movs r1, #1
+; THUMB7-NEXT:    uxtb r0, r0
+; THUMB7-NEXT:    eor.w r0, r1, r0, lsr #7
+; THUMB7-NEXT:    bx lr
+;
+; THUMB8-LABEL: scalar_i8_signbit_eq:
+; THUMB8:       @ %bb.0:
+; THUMB8-NEXT:    uxtb r0, r0
+; THUMB8-NEXT:    uxtb r1, r1
+; THUMB8-NEXT:    lsrs r0, r1
+; THUMB8-NEXT:    movs r1, #1
+; THUMB8-NEXT:    uxtb r0, r0
+; THUMB8-NEXT:    eor.w r0, r1, r0, lsr #7
+; THUMB8-NEXT:    bx lr
   %t0 = shl i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i8_lowestbit_eq(i8 %x, i8 %y) nounwind {
 ; ARM-LABEL: scalar_i8_lowestbit_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    uxtb r1, r1
-; ARM-NEXT:    mov r2, #1
-; ARM-NEXT:    and r0, r0, r2, lsl r1
 ; ARM-NEXT:    uxtb r0, r0
-; ARM-NEXT:    clz r0, r0
-; ARM-NEXT:    lsr r0, r0, #5
+; ARM-NEXT:    mov r2, #1
+; ARM-NEXT:    bic r0, r2, r0, lsr r1
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i8_lowestbit_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxtb r1, r1
-; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    lsls r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxtb r1, r2
+; THUMB6-NEXT:    uxtb r0, r0
+; THUMB6-NEXT:    lsrs r0, r1
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
-; THUMB78-LABEL: scalar_i8_lowestbit_eq:
-; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    uxtb r1, r1
-; THUMB78-NEXT:    movs r2, #1
-; THUMB78-NEXT:    lsl.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxtb r0, r0
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
-; THUMB78-NEXT:    bx lr
+; THUMB7-LABEL: scalar_i8_lowestbit_eq:
+; THUMB7:       @ %bb.0:
+; THUMB7-NEXT:    uxtb r1, r1
+; THUMB7-NEXT:    uxtb r0, r0
+; THUMB7-NEXT:    lsrs r0, r1
+; THUMB7-NEXT:    movs r1, #1
+; THUMB7-NEXT:    bic.w r0, r1, r0
+; THUMB7-NEXT:    bx lr
+;
+; THUMB8-LABEL: scalar_i8_lowestbit_eq:
+; THUMB8:       @ %bb.0:
+; THUMB8-NEXT:    uxtb r0, r0
+; THUMB8-NEXT:    uxtb r1, r1
+; THUMB8-NEXT:    lsrs r0, r1
+; THUMB8-NEXT:    movs r1, #1
+; THUMB8-NEXT:    bic.w r0, r1, r0
+; THUMB8-NEXT:    bx lr
   %t0 = shl i8 1, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i8_bitsinmiddle_eq(i8 %x, i8 %y) nounwind {
 ; ARM-LABEL: scalar_i8_bitsinmiddle_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    uxtb r1, r1
-; ARM-NEXT:    mov r2, #24
-; ARM-NEXT:    and r0, r0, r2, lsl r1
 ; ARM-NEXT:    uxtb r0, r0
+; ARM-NEXT:    mov r2, #24
+; ARM-NEXT:    and r0, r2, r0, lsr r1
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i8_bitsinmiddle_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxtb r1, r1
-; THUMB6-NEXT:    movs r2, #24
-; THUMB6-NEXT:    lsls r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxtb r1, r2
+; THUMB6-NEXT:    uxtb r0, r0
+; THUMB6-NEXT:    lsrs r0, r1
+; THUMB6-NEXT:    movs r1, #24
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
-; THUMB78-LABEL: scalar_i8_bitsinmiddle_eq:
-; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    uxtb r1, r1
-; THUMB78-NEXT:    movs r2, #24
-; THUMB78-NEXT:    lsl.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxtb r0, r0
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
-; THUMB78-NEXT:    bx lr
+; THUMB7-LABEL: scalar_i8_bitsinmiddle_eq:
+; THUMB7:       @ %bb.0:
+; THUMB7-NEXT:    uxtb r1, r1
+; THUMB7-NEXT:    uxtb r0, r0
+; THUMB7-NEXT:    lsrs r0, r1
+; THUMB7-NEXT:    and r0, r0, #24
+; THUMB7-NEXT:    clz r0, r0
+; THUMB7-NEXT:    lsrs r0, r0, #5
+; THUMB7-NEXT:    bx lr
+;
+; THUMB8-LABEL: scalar_i8_bitsinmiddle_eq:
+; THUMB8:       @ %bb.0:
+; THUMB8-NEXT:    uxtb r0, r0
+; THUMB8-NEXT:    uxtb r1, r1
+; THUMB8-NEXT:    lsrs r0, r1
+; THUMB8-NEXT:    and r0, r0, #24
+; THUMB8-NEXT:    clz r0, r0
+; THUMB8-NEXT:    lsrs r0, r0, #5
+; THUMB8-NEXT:    bx lr
   %t0 = shl i8 24, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 ; i16 scalar
 
 define i1 @scalar_i16_signbit_eq(i16 %x, i16 %y) nounwind {
-; ARM6-LABEL: scalar_i16_signbit_eq:
-; ARM6:       @ %bb.0:
-; ARM6-NEXT:    ldr r2, .LCPI3_0
-; ARM6-NEXT:    uxth r1, r1
-; ARM6-NEXT:    and r0, r0, r2, lsl r1
-; ARM6-NEXT:    uxth r0, r0
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    bx lr
-; ARM6-NEXT:    .p2align 2
-; ARM6-NEXT:  @ %bb.1:
-; ARM6-NEXT:  .LCPI3_0:
-; ARM6-NEXT:    .long 4294934528 @ 0xffff8000
-;
-; ARM78-LABEL: scalar_i16_signbit_eq:
-; ARM78:       @ %bb.0:
-; ARM78-NEXT:    movw r2, #32768
-; ARM78-NEXT:    uxth r1, r1
-; ARM78-NEXT:    movt r2, #65535
-; ARM78-NEXT:    and r0, r0, r2, lsl r1
-; ARM78-NEXT:    uxth r0, r0
-; ARM78-NEXT:    clz r0, r0
-; ARM78-NEXT:    lsr r0, r0, #5
-; ARM78-NEXT:    bx lr
+; ARM-LABEL: scalar_i16_signbit_eq:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    uxth r1, r1
+; ARM-NEXT:    uxth r0, r0
+; ARM-NEXT:    lsr r0, r0, r1
+; ARM-NEXT:    mov r1, #1
+; ARM-NEXT:    uxth r0, r0
+; ARM-NEXT:    eor r0, r1, r0, lsr #15
+; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i16_signbit_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxth r1, r1
-; THUMB6-NEXT:    ldr r2, .LCPI3_0
-; THUMB6-NEXT:    lsls r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxth r1, r2
+; THUMB6-NEXT:    uxth r0, r0
+; THUMB6-NEXT:    lsrs r0, r1
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    lsls r1, r1, #15
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
-; THUMB6-NEXT:    .p2align 2
-; THUMB6-NEXT:  @ %bb.1:
-; THUMB6-NEXT:  .LCPI3_0:
-; THUMB6-NEXT:    .long 4294934528 @ 0xffff8000
 ;
-; THUMB78-LABEL: scalar_i16_signbit_eq:
-; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    movw r2, #32768
-; THUMB78-NEXT:    uxth r1, r1
-; THUMB78-NEXT:    movt r2, #65535
-; THUMB78-NEXT:    lsl.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxth r0, r0
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
-; THUMB78-NEXT:    bx lr
+; THUMB7-LABEL: scalar_i16_signbit_eq:
+; THUMB7:       @ %bb.0:
+; THUMB7-NEXT:    uxth r1, r1
+; THUMB7-NEXT:    uxth r0, r0
+; THUMB7-NEXT:    lsrs r0, r1
+; THUMB7-NEXT:    movs r1, #1
+; THUMB7-NEXT:    uxth r0, r0
+; THUMB7-NEXT:    eor.w r0, r1, r0, lsr #15
+; THUMB7-NEXT:    bx lr
+;
+; THUMB8-LABEL: scalar_i16_signbit_eq:
+; THUMB8:       @ %bb.0:
+; THUMB8-NEXT:    uxth r0, r0
+; THUMB8-NEXT:    uxth r1, r1
+; THUMB8-NEXT:    lsrs r0, r1
+; THUMB8-NEXT:    movs r1, #1
+; THUMB8-NEXT:    uxth r0, r0
+; THUMB8-NEXT:    eor.w r0, r1, r0, lsr #15
+; THUMB8-NEXT:    bx lr
   %t0 = shl i16 32768, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i16_lowestbit_eq(i16 %x, i16 %y) nounwind {
 ; ARM-LABEL: scalar_i16_lowestbit_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    uxth r1, r1
-; ARM-NEXT:    mov r2, #1
-; ARM-NEXT:    and r0, r0, r2, lsl r1
 ; ARM-NEXT:    uxth r0, r0
-; ARM-NEXT:    clz r0, r0
-; ARM-NEXT:    lsr r0, r0, #5
+; ARM-NEXT:    mov r2, #1
+; ARM-NEXT:    bic r0, r2, r0, lsr r1
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i16_lowestbit_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxth r1, r1
-; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    lsls r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxth r1, r2
+; THUMB6-NEXT:    uxth r0, r0
+; THUMB6-NEXT:    lsrs r0, r1
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
-; THUMB78-LABEL: scalar_i16_lowestbit_eq:
-; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    uxth r1, r1
-; THUMB78-NEXT:    movs r2, #1
-; THUMB78-NEXT:    lsl.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxth r0, r0
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
-; THUMB78-NEXT:    bx lr
+; THUMB7-LABEL: scalar_i16_lowestbit_eq:
+; THUMB7:       @ %bb.0:
+; THUMB7-NEXT:    uxth r1, r1
+; THUMB7-NEXT:    uxth r0, r0
+; THUMB7-NEXT:    lsrs r0, r1
+; THUMB7-NEXT:    movs r1, #1
+; THUMB7-NEXT:    bic.w r0, r1, r0
+; THUMB7-NEXT:    bx lr
+;
+; THUMB8-LABEL: scalar_i16_lowestbit_eq:
+; THUMB8:       @ %bb.0:
+; THUMB8-NEXT:    uxth r0, r0
+; THUMB8-NEXT:    uxth r1, r1
+; THUMB8-NEXT:    lsrs r0, r1
+; THUMB8-NEXT:    movs r1, #1
+; THUMB8-NEXT:    bic.w r0, r1, r0
+; THUMB8-NEXT:    bx lr
   %t0 = shl i16 1, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i16_bitsinmiddle_eq(i16 %x, i16 %y) nounwind {
 ; ARM-LABEL: scalar_i16_bitsinmiddle_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    uxth r1, r1
-; ARM-NEXT:    mov r2, #4080
-; ARM-NEXT:    and r0, r0, r2, lsl r1
 ; ARM-NEXT:    uxth r0, r0
+; ARM-NEXT:    mov r2, #4080
+; ARM-NEXT:    and r0, r2, r0, lsr r1
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i16_bitsinmiddle_eq:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxth r1, r1
-; THUMB6-NEXT:    movs r2, #255
-; THUMB6-NEXT:    lsls r2, r2, #4
-; THUMB6-NEXT:    lsls r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxth r1, r2
+; THUMB6-NEXT:    uxth r0, r0
+; THUMB6-NEXT:    lsrs r0, r1
+; THUMB6-NEXT:    movs r1, #255
+; THUMB6-NEXT:    lsls r1, r1, #4
+; THUMB6-NEXT:    ands r1, r0
 ; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
-; THUMB78-LABEL: scalar_i16_bitsinmiddle_eq:
-; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    uxth r1, r1
-; THUMB78-NEXT:    mov.w r2, #4080
-; THUMB78-NEXT:    lsl.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxth r0, r0
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
-; THUMB78-NEXT:    bx lr
+; THUMB7-LABEL: scalar_i16_bitsinmiddle_eq:
+; THUMB7:       @ %bb.0:
+; THUMB7-NEXT:    uxth r1, r1
+; THUMB7-NEXT:    uxth r0, r0
+; THUMB7-NEXT:    lsrs r0, r1
+; THUMB7-NEXT:    and r0, r0, #4080
+; THUMB7-NEXT:    clz r0, r0
+; THUMB7-NEXT:    lsrs r0, r0, #5
+; THUMB7-NEXT:    bx lr
+;
+; THUMB8-LABEL: scalar_i16_bitsinmiddle_eq:
+; THUMB8:       @ %bb.0:
+; THUMB8-NEXT:    uxth r0, r0
+; THUMB8-NEXT:    uxth r1, r1
+; THUMB8-NEXT:    lsrs r0, r1
+; THUMB8-NEXT:    and r0, r0, #4080
+; THUMB8-NEXT:    clz r0, r0
+; THUMB8-NEXT:    lsrs r0, r0, #5
+; THUMB8-NEXT:    bx lr
   %t0 = shl i16 4080, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 ; i32 scalar
 
 define i1 @scalar_i32_signbit_eq(i32 %x, i32 %y) nounwind {
 ; ARM-LABEL: scalar_i32_signbit_eq:
 ; ARM:       @ %bb.0:
-; ARM-NEXT:    mov r2, #-2147483648
-; ARM-NEXT:    and r0, r0, r2, lsl r1
-; ARM-NEXT:    clz r0, r0
-; ARM-NEXT:    lsr r0, r0, #5
+; ARM-NEXT:    mvn r0, r0, lsr r1
+; ARM-NEXT:    lsr r0, r0, #31
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i32_signbit_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    lsls r2, r2, #31
-; THUMB6-NEXT:    lsls r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    rsbs r0, r2, #0
-; THUMB6-NEXT:    adcs r0, r2
+; THUMB6-NEXT:    lsrs r0, r1
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    lsls r1, r1, #31
+; THUMB6-NEXT:    ands r1, r0
+; THUMB6-NEXT:    rsbs r0, r1, #0
+; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
 ; THUMB78-LABEL: scalar_i32_signbit_eq:
 ; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    mov.w r2, #-2147483648
-; THUMB78-NEXT:    lsl.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
+; THUMB78-NEXT:    lsrs r0, r1
+; THUMB78-NEXT:    mvns r0, r0
+; THUMB78-NEXT:    lsrs r0, r0, #31
 ; THUMB78-NEXT:    bx lr
   %t0 = shl i32 2147483648, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i32_lowestbit_eq(i32 %x, i32 %y) nounwind {
 ; ARM-LABEL: scalar_i32_lowestbit_eq:
 ; ARM:       @ %bb.0:
 ; ARM-NEXT:    mov r2, #1
-; ARM-NEXT:    and r0, r0, r2, lsl r1
-; ARM-NEXT:    clz r0, r0
-; ARM-NEXT:    lsr r0, r0, #5
+; ARM-NEXT:    bic r0, r2, r0, lsr r1
 ; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i32_lowestbit_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    lsls r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    rsbs r0, r2, #0
-; THUMB6-NEXT:    adcs r0, r2
+; THUMB6-NEXT:    lsrs r0, r1
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    ands r1, r0
+; THUMB6-NEXT:    rsbs r0, r1, #0
+; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ;
 ; THUMB78-LABEL: scalar_i32_lowestbit_eq:
 ; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    movs r2, #1
-; THUMB78-NEXT:    lsl.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
+; THUMB78-NEXT:    lsrs r0, r1
+; THUMB78-NEXT:    movs r1, #1
+; THUMB78-NEXT:    bic.w r0, r1, r0
 ; THUMB78-NEXT:    bx lr
   %t0 = shl i32 1, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i32_bitsinmiddle_eq(i32 %x, i32 %y) nounwind {
 ; ARM6-LABEL: scalar_i32_bitsinmiddle_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    mov r2, #65280
 ; ARM6-NEXT:    orr r2, r2, #16711680
-; ARM6-NEXT:    and r0, r0, r2, lsl r1
+; ARM6-NEXT:    and r0, r2, r0, lsr r1
 ; ARM6-NEXT:    clz r0, r0
 ; ARM6-NEXT:    lsr r0, r0, #5
 ; ARM6-NEXT:    bx lr
 ;
 ; ARM78-LABEL: scalar_i32_bitsinmiddle_eq:
 ; ARM78:       @ %bb.0:
 ; ARM78-NEXT:    movw r2, #65280
 ; ARM78-NEXT:    movt r2, #255
-; ARM78-NEXT:    and r0, r0, r2, lsl r1
+; ARM78-NEXT:    and r0, r2, r0, lsr r1
 ; ARM78-NEXT:    clz r0, r0
 ; ARM78-NEXT:    lsr r0, r0, #5
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i32_bitsinmiddle_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    ldr r2, .LCPI8_0
-; THUMB6-NEXT:    lsls r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    rsbs r0, r2, #0
-; THUMB6-NEXT:    adcs r0, r2
+; THUMB6-NEXT:    lsrs r0, r1
+; THUMB6-NEXT:    ldr r1, .LCPI8_0
+; THUMB6-NEXT:    ands r1, r0
+; THUMB6-NEXT:    rsbs r0, r1, #0
+; THUMB6-NEXT:    adcs r0, r1
 ; THUMB6-NEXT:    bx lr
 ; THUMB6-NEXT:    .p2align 2
 ; THUMB6-NEXT:  @ %bb.1:
 ; THUMB6-NEXT:  .LCPI8_0:
 ; THUMB6-NEXT:    .long 16776960 @ 0xffff00
 ;
 ; THUMB78-LABEL: scalar_i32_bitsinmiddle_eq:
 ; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    movw r2, #65280
-; THUMB78-NEXT:    movt r2, #255
-; THUMB78-NEXT:    lsl.w r1, r2, r1
+; THUMB78-NEXT:    lsrs r0, r1
+; THUMB78-NEXT:    movw r1, #65280
+; THUMB78-NEXT:    movt r1, #255
 ; THUMB78-NEXT:    ands r0, r1
 ; THUMB78-NEXT:    clz r0, r0
 ; THUMB78-NEXT:    lsrs r0, r0, #5
 ; THUMB78-NEXT:    bx lr
   %t0 = shl i32 16776960, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 ; i64 scalar
 
 define i1 @scalar_i64_signbit_eq(i64 %x, i64 %y) nounwind {
 ; ARM6-LABEL: scalar_i64_signbit_eq:
 ; ARM6:       @ %bb.0:
-; ARM6-NEXT:    mov r0, #-2147483648
-; ARM6-NEXT:    lsl r0, r0, r2
-; ARM6-NEXT:    sub r2, r2, #32
-; ARM6-NEXT:    cmp r2, #0
+; ARM6-NEXT:    lsr r0, r1, r2
+; ARM6-NEXT:    sub r1, r2, #32
+; ARM6-NEXT:    cmp r1, #0
 ; ARM6-NEXT:    movge r0, #0
-; ARM6-NEXT:    and r0, r0, r1
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    lsr r0, r0, #5
+; ARM6-NEXT:    mvn r0, r0
+; ARM6-NEXT:    lsr r0, r0, #31
 ; ARM6-NEXT:    bx lr
 ;
 ; ARM78-LABEL: scalar_i64_signbit_eq:
 ; ARM78:       @ %bb.0:
-; ARM78-NEXT:    mov r0, #-2147483648
-; ARM78-NEXT:    lsl r0, r0, r2
-; ARM78-NEXT:    sub r2, r2, #32
-; ARM78-NEXT:    cmp r2, #0
+; ARM78-NEXT:    lsr r0, r1, r2
+; ARM78-NEXT:    sub r1, r2, #32
+; ARM78-NEXT:    cmp r1, #0
 ; ARM78-NEXT:    movwge r0, #0
-; ARM78-NEXT:    and r0, r0, r1
-; ARM78-NEXT:    clz r0, r0
-; ARM78-NEXT:    lsr r0, r0, #5
+; ARM78-NEXT:    mvn r0, r0
+; ARM78-NEXT:    lsr r0, r0, #31
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i64_signbit_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    mov r4, r1
-; THUMB6-NEXT:    mov r5, r0
+; THUMB6-NEXT:    push {r7, lr}
+; THUMB6-NEXT:    bl __lshrdi3
 ; THUMB6-NEXT:    movs r0, #1
-; THUMB6-NEXT:    lsls r1, r0, #31
-; THUMB6-NEXT:    movs r0, #0
-; THUMB6-NEXT:    bl __ashldi3
-; THUMB6-NEXT:    ands r1, r4
-; THUMB6-NEXT:    ands r0, r5
-; THUMB6-NEXT:    orrs r0, r1
-; THUMB6-NEXT:    rsbs r1, r0, #0
-; THUMB6-NEXT:    adcs r0, r1
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    lsls r2, r0, #31
+; THUMB6-NEXT:    ands r2, r1
+; THUMB6-NEXT:    rsbs r0, r2, #0
+; THUMB6-NEXT:    adcs r0, r2
+; THUMB6-NEXT:    pop {r7, pc}
 ;
 ; THUMB78-LABEL: scalar_i64_signbit_eq:
 ; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    mov.w r0, #-2147483648
-; THUMB78-NEXT:    lsls r0, r2
-; THUMB78-NEXT:    subs r2, #32
-; THUMB78-NEXT:    cmp r2, #0
+; THUMB78-NEXT:    lsr.w r0, r1, r2
+; THUMB78-NEXT:    sub.w r1, r2, #32
+; THUMB78-NEXT:    cmp r1, #0
 ; THUMB78-NEXT:    it ge
 ; THUMB78-NEXT:    movge r0, #0
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    clz r0, r0
-; THUMB78-NEXT:    lsrs r0, r0, #5
+; THUMB78-NEXT:    mvns r0, r0
+; THUMB78-NEXT:    lsrs r0, r0, #31
 ; THUMB78-NEXT:    bx lr
   %t0 = shl i64 9223372036854775808, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i64_lowestbit_eq(i64 %x, i64 %y) nounwind {
-; ARM6-LABEL: scalar_i64_lowestbit_eq:
-; ARM6:       @ %bb.0:
-; ARM6-NEXT:    push {r11, lr}
-; ARM6-NEXT:    mov r12, #1
-; ARM6-NEXT:    sub lr, r2, #32
-; ARM6-NEXT:    lsl r3, r12, r2
-; ARM6-NEXT:    rsb r2, r2, #32
-; ARM6-NEXT:    cmp lr, #0
-; ARM6-NEXT:    lsr r2, r12, r2
-; ARM6-NEXT:    movge r3, #0
-; ARM6-NEXT:    lslge r2, r12, lr
-; ARM6-NEXT:    and r0, r3, r0
-; ARM6-NEXT:    and r1, r2, r1
-; ARM6-NEXT:    orr r0, r0, r1
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    pop {r11, pc}
-;
-; ARM78-LABEL: scalar_i64_lowestbit_eq:
-; ARM78:       @ %bb.0:
-; ARM78-NEXT:    push {r11, lr}
-; ARM78-NEXT:    mov r12, #1
-; ARM78-NEXT:    sub lr, r2, #32
-; ARM78-NEXT:    lsl r3, r12, r2
-; ARM78-NEXT:    rsb r2, r2, #32
-; ARM78-NEXT:    cmp lr, #0
-; ARM78-NEXT:    lsr r2, r12, r2
-; ARM78-NEXT:    movwge r3, #0
-; ARM78-NEXT:    lslge r2, r12, lr
-; ARM78-NEXT:    and r0, r3, r0
-; ARM78-NEXT:    and r1, r2, r1
-; ARM78-NEXT:    orr r0, r0, r1
-; ARM78-NEXT:    clz r0, r0
-; ARM78-NEXT:    lsr r0, r0, #5
-; ARM78-NEXT:    pop {r11, pc}
+; ARM-LABEL: scalar_i64_lowestbit_eq:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    rsb r3, r2, #32
+; ARM-NEXT:    lsr r0, r0, r2
+; ARM-NEXT:    sub r2, r2, #32
+; ARM-NEXT:    orr r0, r0, r1, lsl r3
+; ARM-NEXT:    cmp r2, #0
+; ARM-NEXT:    lsrge r0, r1, r2
+; ARM-NEXT:    mov r1, #1
+; ARM-NEXT:    bic r0, r1, r0
+; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i64_lowestbit_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    mov r4, r1
-; THUMB6-NEXT:    mov r5, r0
-; THUMB6-NEXT:    movs r0, #1
-; THUMB6-NEXT:    movs r1, #0
-; THUMB6-NEXT:    bl __ashldi3
-; THUMB6-NEXT:    ands r1, r4
-; THUMB6-NEXT:    ands r0, r5
-; THUMB6-NEXT:    orrs r0, r1
-; THUMB6-NEXT:    rsbs r1, r0, #0
+; THUMB6-NEXT:    push {r7, lr}
+; THUMB6-NEXT:    bl __lshrdi3
+; THUMB6-NEXT:    movs r1, #1
+; THUMB6-NEXT:    ands r1, r0
+; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    pop {r7, pc}
 ;
-; THUMB7-LABEL: scalar_i64_lowestbit_eq:
-; THUMB7:       @ %bb.0:
-; THUMB7-NEXT:    push {r7, lr}
-; THUMB7-NEXT:    rsb.w r3, r2, #32
-; THUMB7-NEXT:    mov.w r12, #1
-; THUMB7-NEXT:    sub.w lr, r2, #32
-; THUMB7-NEXT:    lsl.w r2, r12, r2
-; THUMB7-NEXT:    lsr.w r3, r12, r3
-; THUMB7-NEXT:    cmp.w lr, #0
-; THUMB7-NEXT:    it ge
-; THUMB7-NEXT:    lslge.w r3, r12, lr
-; THUMB7-NEXT:    it ge
-; THUMB7-NEXT:    movge r2, #0
-; THUMB7-NEXT:    ands r1, r3
-; THUMB7-NEXT:    ands r0, r2
-; THUMB7-NEXT:    orrs r0, r1
-; THUMB7-NEXT:    clz r0, r0
-; THUMB7-NEXT:    lsrs r0, r0, #5
-; THUMB7-NEXT:    pop {r7, pc}
-;
-; THUMB8-LABEL: scalar_i64_lowestbit_eq:
-; THUMB8:       @ %bb.0:
-; THUMB8-NEXT:    .save {r7, lr}
-; THUMB8-NEXT:    push {r7, lr}
-; THUMB8-NEXT:    rsb.w r3, r2, #32
-; THUMB8-NEXT:    sub.w lr, r2, #32
-; THUMB8-NEXT:    mov.w r12, #1
-; THUMB8-NEXT:    cmp.w lr, #0
-; THUMB8-NEXT:    lsr.w r3, r12, r3
-; THUMB8-NEXT:    lsl.w r2, r12, r2
-; THUMB8-NEXT:    it ge
-; THUMB8-NEXT:    lslge.w r3, r12, lr
-; THUMB8-NEXT:    it ge
-; THUMB8-NEXT:    movge r2, #0
-; THUMB8-NEXT:    ands r1, r3
-; THUMB8-NEXT:    ands r0, r2
-; THUMB8-NEXT:    orrs r0, r1
-; THUMB8-NEXT:    clz r0, r0
-; THUMB8-NEXT:    lsrs r0, r0, #5
-; THUMB8-NEXT:    pop {r7, pc}
+; THUMB78-LABEL: scalar_i64_lowestbit_eq:
+; THUMB78:       @ %bb.0:
+; THUMB78-NEXT:    rsb.w r3, r2, #32
+; THUMB78-NEXT:    lsrs r0, r2
+; THUMB78-NEXT:    subs r2, #32
+; THUMB78-NEXT:    lsl.w r3, r1, r3
+; THUMB78-NEXT:    orrs r0, r3
+; THUMB78-NEXT:    cmp r2, #0
+; THUMB78-NEXT:    it ge
+; THUMB78-NEXT:    lsrge.w r0, r1, r2
+; THUMB78-NEXT:    movs r1, #1
+; THUMB78-NEXT:    bic.w r0, r1, r0
+; THUMB78-NEXT:    bx lr
   %t0 = shl i64 1, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i64_bitsinmiddle_eq(i64 %x, i64 %y) nounwind {
 ; ARM6-LABEL: scalar_i64_bitsinmiddle_eq:
 ; ARM6:       @ %bb.0:
-; ARM6-NEXT:    push {r4, lr}
-; ARM6-NEXT:    mov r12, #16711680
-; ARM6-NEXT:    sub lr, r2, #32
-; ARM6-NEXT:    orr r12, r12, #-16777216
-; ARM6-NEXT:    cmp lr, #0
-; ARM6-NEXT:    mov r4, #255
-; ARM6-NEXT:    lsl r3, r12, r2
-; ARM6-NEXT:    orr r4, r4, #65280
-; ARM6-NEXT:    movge r3, #0
-; ARM6-NEXT:    and r0, r3, r0
 ; ARM6-NEXT:    rsb r3, r2, #32
-; ARM6-NEXT:    cmp lr, #0
-; ARM6-NEXT:    lsr r3, r12, r3
-; ARM6-NEXT:    orr r2, r3, r4, lsl r2
-; ARM6-NEXT:    lslge r2, r12, lr
-; ARM6-NEXT:    and r1, r2, r1
-; ARM6-NEXT:    orr r0, r0, r1
+; ARM6-NEXT:    lsr r0, r0, r2
+; ARM6-NEXT:    orr r0, r0, r1, lsl r3
+; ARM6-NEXT:    sub r3, r2, #32
+; ARM6-NEXT:    cmp r3, #0
+; ARM6-NEXT:    lsrge r0, r1, r3
+; ARM6-NEXT:    lsr r1, r1, r2
+; ARM6-NEXT:    movge r1, #0
+; ARM6-NEXT:    pkhbt r0, r1, r0
 ; ARM6-NEXT:    clz r0, r0
 ; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    pop {r4, pc}
+; ARM6-NEXT:    bx lr
 ;
 ; ARM78-LABEL: scalar_i64_bitsinmiddle_eq:
 ; ARM78:       @ %bb.0:
-; ARM78-NEXT:    push {r4, lr}
-; ARM78-NEXT:    movw r12, #0
-; ARM78-NEXT:    sub lr, r2, #32
-; ARM78-NEXT:    movt r12, #65535
-; ARM78-NEXT:    cmp lr, #0
-; ARM78-NEXT:    lsl r3, r12, r2
-; ARM78-NEXT:    movw r4, #65535
-; ARM78-NEXT:    movwge r3, #0
-; ARM78-NEXT:    and r0, r3, r0
 ; ARM78-NEXT:    rsb r3, r2, #32
-; ARM78-NEXT:    cmp lr, #0
-; ARM78-NEXT:    lsr r3, r12, r3
-; ARM78-NEXT:    orr r2, r3, r4, lsl r2
-; ARM78-NEXT:    lslge r2, r12, lr
-; ARM78-NEXT:    and r1, r2, r1
-; ARM78-NEXT:    orr r0, r0, r1
+; ARM78-NEXT:    lsr r0, r0, r2
+; ARM78-NEXT:    orr r0, r0, r1, lsl r3
+; ARM78-NEXT:    sub r3, r2, #32
+; ARM78-NEXT:    cmp r3, #0
+; ARM78-NEXT:    lsrge r0, r1, r3
+; ARM78-NEXT:    lsr r1, r1, r2
+; ARM78-NEXT:    movwge r1, #0
+; ARM78-NEXT:    pkhbt r0, r1, r0
 ; ARM78-NEXT:    clz r0, r0
 ; ARM78-NEXT:    lsr r0, r0, #5
-; ARM78-NEXT:    pop {r4, pc}
+; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i64_bitsinmiddle_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    mov r4, r1
-; THUMB6-NEXT:    mov r5, r0
-; THUMB6-NEXT:    ldr r0, .LCPI11_0
-; THUMB6-NEXT:    ldr r1, .LCPI11_1
-; THUMB6-NEXT:    bl __ashldi3
-; THUMB6-NEXT:    ands r1, r4
-; THUMB6-NEXT:    ands r0, r5
-; THUMB6-NEXT:    orrs r0, r1
-; THUMB6-NEXT:    rsbs r1, r0, #0
+; THUMB6-NEXT:    push {r7, lr}
+; THUMB6-NEXT:    bl __lshrdi3
+; THUMB6-NEXT:    ldr r2, .LCPI11_0
+; THUMB6-NEXT:    ands r2, r0
+; THUMB6-NEXT:    uxth r0, r1
+; THUMB6-NEXT:    adds r1, r2, r0
+; THUMB6-NEXT:    rsbs r0, r1, #0
 ; THUMB6-NEXT:    adcs r0, r1
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    pop {r7, pc}
 ; THUMB6-NEXT:    .p2align 2
 ; THUMB6-NEXT:  @ %bb.1:
 ; THUMB6-NEXT:  .LCPI11_0:
 ; THUMB6-NEXT:    .long 4294901760 @ 0xffff0000
-; THUMB6-NEXT:  .LCPI11_1:
-; THUMB6-NEXT:    .long 65535 @ 0xffff
 ;
-; THUMB7-LABEL: scalar_i64_bitsinmiddle_eq:
-; THUMB7:       @ %bb.0:
-; THUMB7-NEXT:    push {r7, lr}
-; THUMB7-NEXT:    movw r3, #65535
-; THUMB7-NEXT:    movw lr, #0
-; THUMB7-NEXT:    lsl.w r12, r3, r2
-; THUMB7-NEXT:    rsb.w r3, r2, #32
-; THUMB7-NEXT:    movt lr, #65535
-; THUMB7-NEXT:    lsr.w r3, lr, r3
-; THUMB7-NEXT:    orr.w r3, r3, r12
-; THUMB7-NEXT:    sub.w r12, r2, #32
-; THUMB7-NEXT:    lsl.w r2, lr, r2
-; THUMB7-NEXT:    cmp.w r12, #0
-; THUMB7-NEXT:    it ge
-; THUMB7-NEXT:    lslge.w r3, lr, r12
-; THUMB7-NEXT:    it ge
-; THUMB7-NEXT:    movge r2, #0
-; THUMB7-NEXT:    ands r1, r3
-; THUMB7-NEXT:    ands r0, r2
-; THUMB7-NEXT:    orrs r0, r1
-; THUMB7-NEXT:    clz r0, r0
-; THUMB7-NEXT:    lsrs r0, r0, #5
-; THUMB7-NEXT:    pop {r7, pc}
-;
-; THUMB8-LABEL: scalar_i64_bitsinmiddle_eq:
-; THUMB8:       @ %bb.0:
-; THUMB8-NEXT:    .save {r7, lr}
-; THUMB8-NEXT:    push {r7, lr}
-; THUMB8-NEXT:    movw r3, #65535
-; THUMB8-NEXT:    movw lr, #0
-; THUMB8-NEXT:    lsl.w r12, r3, r2
-; THUMB8-NEXT:    rsb.w r3, r2, #32
-; THUMB8-NEXT:    movt lr, #65535
-; THUMB8-NEXT:    lsr.w r3, lr, r3
-; THUMB8-NEXT:    orr.w r3, r3, r12
-; THUMB8-NEXT:    sub.w r12, r2, #32
-; THUMB8-NEXT:    cmp.w r12, #0
-; THUMB8-NEXT:    lsl.w r2, lr, r2
-; THUMB8-NEXT:    it ge
-; THUMB8-NEXT:    lslge.w r3, lr, r12
-; THUMB8-NEXT:    it ge
-; THUMB8-NEXT:    movge r2, #0
-; THUMB8-NEXT:    ands r1, r3
-; THUMB8-NEXT:    ands r0, r2
-; THUMB8-NEXT:    orrs r0, r1
-; THUMB8-NEXT:    clz r0, r0
-; THUMB8-NEXT:    lsrs r0, r0, #5
-; THUMB8-NEXT:    pop {r7, pc}
+; THUMB78-LABEL: scalar_i64_bitsinmiddle_eq:
+; THUMB78:       @ %bb.0:
+; THUMB78-NEXT:    rsb.w r3, r2, #32
+; THUMB78-NEXT:    lsrs r0, r2
+; THUMB78-NEXT:    lsl.w r3, r1, r3
+; THUMB78-NEXT:    orrs r0, r3
+; THUMB78-NEXT:    sub.w r3, r2, #32
+; THUMB78-NEXT:    cmp r3, #0
+; THUMB78-NEXT:    it ge
+; THUMB78-NEXT:    lsrge.w r0, r1, r3
+; THUMB78-NEXT:    lsr.w r1, r1, r2
+; THUMB78-NEXT:    it ge
+; THUMB78-NEXT:    movge r1, #0
+; THUMB78-NEXT:    pkhbt r0, r1, r0
+; THUMB78-NEXT:    clz r0, r0
+; THUMB78-NEXT:    lsrs r0, r0, #5
+; THUMB78-NEXT:    bx lr
   %t0 = shl i64 281474976645120, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;
 ; A few trivial vector tests
 ;------------------------------------------------------------------------------;
 
 define <4 x i1> @vec_4xi32_splat_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; ARM6-LABEL: vec_4xi32_splat_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    push {r11, lr}
 ; ARM6-NEXT:    ldr r12, [sp, #8]
 ; ARM6-NEXT:    mov lr, #1
-; ARM6-NEXT:    and r0, r0, lr, lsl r12
+; ARM6-NEXT:    bic r0, lr, r0, lsr r12
 ; ARM6-NEXT:    ldr r12, [sp, #12]
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    and r1, r1, lr, lsl r12
+; ARM6-NEXT:    bic r1, lr, r1, lsr r12
 ; ARM6-NEXT:    ldr r12, [sp, #16]
-; ARM6-NEXT:    clz r1, r1
-; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    and r2, r2, lr, lsl r12
+; ARM6-NEXT:    bic r2, lr, r2, lsr r12
 ; ARM6-NEXT:    ldr r12, [sp, #20]
-; ARM6-NEXT:    clz r2, r2
-; ARM6-NEXT:    lsr r1, r1, #5
-; ARM6-NEXT:    and r3, r3, lr, lsl r12
-; ARM6-NEXT:    lsr r2, r2, #5
-; ARM6-NEXT:    clz r3, r3
-; ARM6-NEXT:    lsr r3, r3, #5
+; ARM6-NEXT:    bic r3, lr, r3, lsr r12
 ; ARM6-NEXT:    pop {r11, pc}
 ;
 ; ARM78-LABEL: vec_4xi32_splat_eq:
 ; ARM78:       @ %bb.0:
-; ARM78-NEXT:    vmov.i32 q8, #0x1
 ; ARM78-NEXT:    mov r12, sp
-; ARM78-NEXT:    vld1.64 {d18, d19}, [r12]
-; ARM78-NEXT:    vshl.u32 q8, q8, q9
+; ARM78-NEXT:    vld1.64 {d16, d17}, [r12]
 ; ARM78-NEXT:    vmov d19, r2, r3
+; ARM78-NEXT:    vneg.s32 q8, q8
 ; ARM78-NEXT:    vmov d18, r0, r1
-; ARM78-NEXT:    vtst.32 q8, q8, q9
+; ARM78-NEXT:    vmov.i32 q10, #0x1
+; ARM78-NEXT:    vshl.u32 q8, q9, q8
+; ARM78-NEXT:    vtst.32 q8, q8, q10
 ; ARM78-NEXT:    vmvn q8, q8
 ; ARM78-NEXT:    vmovn.i32 d16, q8
 ; ARM78-NEXT:    vmov r0, r1, d16
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: vec_4xi32_splat_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r6, lr}
-; THUMB6-NEXT:    ldr r5, [sp, #16]
+; THUMB6-NEXT:    push {r4, r5, r7, lr}
+; THUMB6-NEXT:    ldr r4, [sp, #16]
+; THUMB6-NEXT:    lsrs r0, r4
 ; THUMB6-NEXT:    movs r4, #1
-; THUMB6-NEXT:    mov r6, r4
-; THUMB6-NEXT:    lsls r6, r5
-; THUMB6-NEXT:    ands r6, r0
-; THUMB6-NEXT:    rsbs r0, r6, #0
-; THUMB6-NEXT:    adcs r0, r6
+; THUMB6-NEXT:    ands r0, r4
+; THUMB6-NEXT:    rsbs r5, r0, #0
+; THUMB6-NEXT:    adcs r0, r5
 ; THUMB6-NEXT:    ldr r5, [sp, #20]
-; THUMB6-NEXT:    mov r6, r4
-; THUMB6-NEXT:    lsls r6, r5
-; THUMB6-NEXT:    ands r6, r1
-; THUMB6-NEXT:    rsbs r1, r6, #0
-; THUMB6-NEXT:    adcs r1, r6
+; THUMB6-NEXT:    lsrs r1, r5
+; THUMB6-NEXT:    ands r1, r4
+; THUMB6-NEXT:    rsbs r5, r1, #0
+; THUMB6-NEXT:    adcs r1, r5
 ; THUMB6-NEXT:    ldr r5, [sp, #24]
-; THUMB6-NEXT:    mov r6, r4
-; THUMB6-NEXT:    lsls r6, r5
-; THUMB6-NEXT:    ands r6, r2
-; THUMB6-NEXT:    rsbs r2, r6, #0
-; THUMB6-NEXT:    adcs r2, r6
+; THUMB6-NEXT:    lsrs r2, r5
+; THUMB6-NEXT:    ands r2, r4
+; THUMB6-NEXT:    rsbs r5, r2, #0
+; THUMB6-NEXT:    adcs r2, r5
 ; THUMB6-NEXT:    ldr r5, [sp, #28]
-; THUMB6-NEXT:    lsls r4, r5
-; THUMB6-NEXT:    ands r4, r3
-; THUMB6-NEXT:    rsbs r3, r4, #0
+; THUMB6-NEXT:    lsrs r3, r5
+; THUMB6-NEXT:    ands r3, r4
+; THUMB6-NEXT:    rsbs r4, r3, #0
 ; THUMB6-NEXT:    adcs r3, r4
-; THUMB6-NEXT:    pop {r4, r5, r6, pc}
+; THUMB6-NEXT:    pop {r4, r5, r7, pc}
 ;
 ; THUMB78-LABEL: vec_4xi32_splat_eq:
 ; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    vmov.i32 q8, #0x1
 ; THUMB78-NEXT:    mov r12, sp
-; THUMB78-NEXT:    vld1.64 {d18, d19}, [r12]
-; THUMB78-NEXT:    vshl.u32 q8, q8, q9
+; THUMB78-NEXT:    vld1.64 {d16, d17}, [r12]
 ; THUMB78-NEXT:    vmov d19, r2, r3
+; THUMB78-NEXT:    vneg.s32 q8, q8
 ; THUMB78-NEXT:    vmov d18, r0, r1
-; THUMB78-NEXT:    vtst.32 q8, q8, q9
+; THUMB78-NEXT:    vmov.i32 q10, #0x1
+; THUMB78-NEXT:    vshl.u32 q8, q9, q8
+; THUMB78-NEXT:    vtst.32 q8, q8, q10
 ; THUMB78-NEXT:    vmvn q8, q8
 ; THUMB78-NEXT:    vmovn.i32 d16, q8
 ; THUMB78-NEXT:    vmov r0, r1, d16
 ; THUMB78-NEXT:    bx lr
   %t0 = shl <4 x i32> <i32 1, i32 1, i32 1, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 
 define <4 x i1> @vec_4xi32_nonsplat_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; ARM6-LABEL: vec_4xi32_nonsplat_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    ldr r12, [sp, #4]
 ; ARM6-NEXT:    mov r0, #1
-; ARM6-NEXT:    and r0, r1, r0, lsl r12
+; ARM6-NEXT:    bic r1, r0, r1, lsr r12
 ; ARM6-NEXT:    ldr r12, [sp, #8]
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    lsr r1, r0, #5
 ; ARM6-NEXT:    mov r0, #65280
 ; ARM6-NEXT:    orr r0, r0, #16711680
-; ARM6-NEXT:    and r0, r2, r0, lsl r12
-; ARM6-NEXT:    ldr r12, [sp, #12]
+; ARM6-NEXT:    and r0, r0, r2, lsr r12
 ; ARM6-NEXT:    clz r0, r0
 ; ARM6-NEXT:    lsr r2, r0, #5
-; ARM6-NEXT:    mov r0, #-2147483648
-; ARM6-NEXT:    and r0, r3, r0, lsl r12
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    lsr r3, r0, #5
+; ARM6-NEXT:    ldr r0, [sp, #12]
+; ARM6-NEXT:    mvn r0, r3, lsr r0
+; ARM6-NEXT:    lsr r3, r0, #31
 ; ARM6-NEXT:    mov r0, #1
 ; ARM6-NEXT:    bx lr
 ;
 ; ARM78-LABEL: vec_4xi32_nonsplat_eq:
 ; ARM78:       @ %bb.0:
 ; ARM78-NEXT:    mov r12, sp
 ; ARM78-NEXT:    vld1.64 {d16, d17}, [r12]
 ; ARM78-NEXT:    adr r12, .LCPI13_0

 ; ARM78-NEXT:  .LCPI13_0:
 ; ARM78-NEXT:    .long 0 @ 0x0
 ; ARM78-NEXT:    .long 1 @ 0x1
 ; ARM78-NEXT:    .long 16776960 @ 0xffff00
 ; ARM78-NEXT:    .long 2147483648 @ 0x80000000
 ;
 ; THUMB6-LABEL: vec_4xi32_nonsplat_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    ldr r4, [sp, #20]
+; THUMB6-NEXT:    push {r4, lr}
+; THUMB6-NEXT:    ldr r0, [sp, #12]
+; THUMB6-NEXT:    lsrs r1, r0
 ; THUMB6-NEXT:    movs r0, #1
-; THUMB6-NEXT:    mov r5, r0
-; THUMB6-NEXT:    lsls r5, r4
-; THUMB6-NEXT:    ands r5, r1
-; THUMB6-NEXT:    rsbs r1, r5, #0
-; THUMB6-NEXT:    adcs r1, r5
-; THUMB6-NEXT:    ldr r4, [sp, #24]
-; THUMB6-NEXT:    ldr r5, .LCPI13_0
-; THUMB6-NEXT:    lsls r5, r4
-; THUMB6-NEXT:    ands r5, r2
-; THUMB6-NEXT:    rsbs r2, r5, #0
-; THUMB6-NEXT:    adcs r2, r5
+; THUMB6-NEXT:    ands r1, r0
+; THUMB6-NEXT:    rsbs r4, r1, #0
+; THUMB6-NEXT:    adcs r1, r4
+; THUMB6-NEXT:    ldr r4, [sp, #16]
+; THUMB6-NEXT:    lsrs r2, r4
+; THUMB6-NEXT:    ldr r4, .LCPI13_0
+; THUMB6-NEXT:    ands r4, r2
+; THUMB6-NEXT:    rsbs r2, r4, #0
+; THUMB6-NEXT:    adcs r2, r4
+; THUMB6-NEXT:    ldr r4, [sp, #20]
+; THUMB6-NEXT:    lsrs r3, r4
 ; THUMB6-NEXT:    lsls r4, r0, #31
-; THUMB6-NEXT:    ldr r5, [sp, #28]
-; THUMB6-NEXT:    lsls r4, r5
 ; THUMB6-NEXT:    ands r4, r3
 ; THUMB6-NEXT:    rsbs r3, r4, #0
 ; THUMB6-NEXT:    adcs r3, r4
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    pop {r4, pc}
 ; THUMB6-NEXT:    .p2align 2
 ; THUMB6-NEXT:  @ %bb.1:
 ; THUMB6-NEXT:  .LCPI13_0:
 ; THUMB6-NEXT:    .long 16776960 @ 0xffff00
 ;
 ; THUMB78-LABEL: vec_4xi32_nonsplat_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    mov r12, sp

 
 define <4 x i1> @vec_4xi32_nonsplat_undef0_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; ARM6-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    push {r11, lr}
 ; ARM6-NEXT:    ldr r2, [sp, #12]
 ; ARM6-NEXT:    mov lr, #1
 ; ARM6-NEXT:    ldr r12, [sp, #8]
-; ARM6-NEXT:    and r1, r1, lr, lsl r2
+; ARM6-NEXT:    bic r1, lr, r1, lsr r2
 ; ARM6-NEXT:    ldr r2, [sp, #20]
-; ARM6-NEXT:    and r0, r0, lr, lsl r12
-; ARM6-NEXT:    clz r1, r1
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    and r2, r3, lr, lsl r2
-; ARM6-NEXT:    lsr r1, r1, #5
-; ARM6-NEXT:    clz r2, r2
-; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    lsr r3, r2, #5
+; ARM6-NEXT:    bic r0, lr, r0, lsr r12
+; ARM6-NEXT:    bic r3, lr, r3, lsr r2
 ; ARM6-NEXT:    mov r2, #1
 ; ARM6-NEXT:    pop {r11, pc}
 ;
 ; ARM78-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; ARM78:       @ %bb.0:
-; ARM78-NEXT:    vmov.i32 q8, #0x1
 ; ARM78-NEXT:    mov r12, sp
-; ARM78-NEXT:    vld1.64 {d18, d19}, [r12]
-; ARM78-NEXT:    vshl.u32 q8, q8, q9
+; ARM78-NEXT:    vld1.64 {d16, d17}, [r12]
 ; ARM78-NEXT:    vmov d19, r2, r3
+; ARM78-NEXT:    vneg.s32 q8, q8
 ; ARM78-NEXT:    vmov d18, r0, r1
-; ARM78-NEXT:    vtst.32 q8, q8, q9
+; ARM78-NEXT:    vmov.i32 q10, #0x1
+; ARM78-NEXT:    vshl.u32 q8, q9, q8
+; ARM78-NEXT:    vtst.32 q8, q8, q10
 ; ARM78-NEXT:    vmvn q8, q8
 ; ARM78-NEXT:    vmovn.i32 d16, q8
 ; ARM78-NEXT:    vmov r0, r1, d16
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    ldr r4, [sp, #16]
+; THUMB6-NEXT:    push {r4, lr}
+; THUMB6-NEXT:    ldr r2, [sp, #8]
+; THUMB6-NEXT:    lsrs r0, r2
 ; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsls r5, r4
-; THUMB6-NEXT:    ands r5, r0
-; THUMB6-NEXT:    rsbs r0, r5, #0
-; THUMB6-NEXT:    adcs r0, r5
+; THUMB6-NEXT:    ands r0, r2
+; THUMB6-NEXT:    rsbs r4, r0, #0
+; THUMB6-NEXT:    adcs r0, r4
+; THUMB6-NEXT:    ldr r4, [sp, #12]
+; THUMB6-NEXT:    lsrs r1, r4
+; THUMB6-NEXT:    ands r1, r2
+; THUMB6-NEXT:    rsbs r4, r1, #0
+; THUMB6-NEXT:    adcs r1, r4
 ; THUMB6-NEXT:    ldr r4, [sp, #20]
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsls r5, r4
-; THUMB6-NEXT:    ands r5, r1
-; THUMB6-NEXT:    rsbs r1, r5, #0
-; THUMB6-NEXT:    adcs r1, r5
-; THUMB6-NEXT:    ldr r4, [sp, #28]
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsls r5, r4
-; THUMB6-NEXT:    ands r5, r3
-; THUMB6-NEXT:    rsbs r3, r5, #0
-; THUMB6-NEXT:    adcs r3, r5
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    lsrs r3, r4
+; THUMB6-NEXT:    ands r3, r2
+; THUMB6-NEXT:    rsbs r4, r3, #0
+; THUMB6-NEXT:    adcs r3, r4
+; THUMB6-NEXT:    pop {r4, pc}
 ;
 ; THUMB78-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    vmov.i32 q8, #0x1
 ; THUMB78-NEXT:    mov r12, sp
-; THUMB78-NEXT:    vld1.64 {d18, d19}, [r12]
-; THUMB78-NEXT:    vshl.u32 q8, q8, q9
+; THUMB78-NEXT:    vld1.64 {d16, d17}, [r12]
 ; THUMB78-NEXT:    vmov d19, r2, r3
+; THUMB78-NEXT:    vneg.s32 q8, q8
 ; THUMB78-NEXT:    vmov d18, r0, r1
-; THUMB78-NEXT:    vtst.32 q8, q8, q9
+; THUMB78-NEXT:    vmov.i32 q10, #0x1
+; THUMB78-NEXT:    vshl.u32 q8, q9, q8
+; THUMB78-NEXT:    vtst.32 q8, q8, q10
 ; THUMB78-NEXT:    vmvn q8, q8
 ; THUMB78-NEXT:    vmovn.i32 d16, q8
 ; THUMB78-NEXT:    vmov r0, r1, d16
 ; THUMB78-NEXT:    bx lr
   %t0 = shl <4 x i32> <i32 1, i32 1, i32 undef, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 define <4 x i1> @vec_4xi32_nonsplat_undef1_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; ARM6-LABEL: vec_4xi32_nonsplat_undef1_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    push {r11, lr}
 ; ARM6-NEXT:    ldr r2, [sp, #12]
 ; ARM6-NEXT:    mov lr, #1
 ; ARM6-NEXT:    ldr r12, [sp, #8]
-; ARM6-NEXT:    and r1, r1, lr, lsl r2
+; ARM6-NEXT:    bic r1, lr, r1, lsr r2
 ; ARM6-NEXT:    ldr r2, [sp, #20]
-; ARM6-NEXT:    and r0, r0, lr, lsl r12
-; ARM6-NEXT:    clz r1, r1
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    and r2, r3, lr, lsl r2
-; ARM6-NEXT:    lsr r1, r1, #5
-; ARM6-NEXT:    clz r2, r2
-; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    lsr r3, r2, #5
+; ARM6-NEXT:    bic r0, lr, r0, lsr r12
+; ARM6-NEXT:    bic r3, lr, r3, lsr r2
 ; ARM6-NEXT:    pop {r11, pc}
 ;
 ; ARM78-LABEL: vec_4xi32_nonsplat_undef1_eq:
 ; ARM78:       @ %bb.0:
 ; ARM78-NEXT:    vmov.i32 q8, #0x1
 ; ARM78-NEXT:    mov r12, sp
 ; ARM78-NEXT:    vld1.64 {d18, d19}, [r12]
 ; ARM78-NEXT:    vshl.u32 q8, q8, q9
 ; ARM78-NEXT:    vmov d19, r2, r3
 ; ARM78-NEXT:    vmov d18, r0, r1
 ; ARM78-NEXT:    vtst.32 q8, q8, q9
 ; ARM78-NEXT:    vmvn q8, q8
 ; ARM78-NEXT:    vmovn.i32 d16, q8
 ; ARM78-NEXT:    vmov r0, r1, d16
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: vec_4xi32_nonsplat_undef1_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    ldr r4, [sp, #16]
+; THUMB6-NEXT:    push {r4, lr}
+; THUMB6-NEXT:    ldr r2, [sp, #8]
+; THUMB6-NEXT:    lsrs r0, r2
 ; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsls r5, r4
-; THUMB6-NEXT:    ands r5, r0
-; THUMB6-NEXT:    rsbs r0, r5, #0
-; THUMB6-NEXT:    adcs r0, r5
+; THUMB6-NEXT:    ands r0, r2
+; THUMB6-NEXT:    rsbs r4, r0, #0
+; THUMB6-NEXT:    adcs r0, r4
+; THUMB6-NEXT:    ldr r4, [sp, #12]
+; THUMB6-NEXT:    lsrs r1, r4
+; THUMB6-NEXT:    ands r1, r2
+; THUMB6-NEXT:    rsbs r4, r1, #0
+; THUMB6-NEXT:    adcs r1, r4
 ; THUMB6-NEXT:    ldr r4, [sp, #20]
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsls r5, r4
-; THUMB6-NEXT:    ands r5, r1
-; THUMB6-NEXT:    rsbs r1, r5, #0
-; THUMB6-NEXT:    adcs r1, r5
-; THUMB6-NEXT:    ldr r4, [sp, #28]
-; THUMB6-NEXT:    lsls r2, r4
-; THUMB6-NEXT:    ands r2, r3
-; THUMB6-NEXT:    rsbs r3, r2, #0
+; THUMB6-NEXT:    lsrs r3, r4
+; THUMB6-NEXT:    ands r3, r2
+; THUMB6-NEXT:    rsbs r2, r3, #0
 ; THUMB6-NEXT:    adcs r3, r2
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    pop {r4, pc}
 ;
 ; THUMB78-LABEL: vec_4xi32_nonsplat_undef1_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    vmov.i32 q8, #0x1
 ; THUMB78-NEXT:    mov r12, sp
 ; THUMB78-NEXT:    vld1.64 {d18, d19}, [r12]
 ; THUMB78-NEXT:    vshl.u32 q8, q8, q9
 ; THUMB78-NEXT:    vmov d19, r2, r3

 }
 define <4 x i1> @vec_4xi32_nonsplat_undef2_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; ARM6-LABEL: vec_4xi32_nonsplat_undef2_eq:
 ; ARM6:       @ %bb.0:
 ; ARM6-NEXT:    push {r11, lr}
 ; ARM6-NEXT:    ldr r2, [sp, #12]
 ; ARM6-NEXT:    mov lr, #1
 ; ARM6-NEXT:    ldr r12, [sp, #8]
-; ARM6-NEXT:    and r1, r1, lr, lsl r2
+; ARM6-NEXT:    bic r1, lr, r1, lsr r2
 ; ARM6-NEXT:    ldr r2, [sp, #20]
-; ARM6-NEXT:    and r0, r0, lr, lsl r12
-; ARM6-NEXT:    clz r1, r1
-; ARM6-NEXT:    clz r0, r0
-; ARM6-NEXT:    and r2, r3, lr, lsl r2
-; ARM6-NEXT:    lsr r1, r1, #5
-; ARM6-NEXT:    clz r2, r2
-; ARM6-NEXT:    lsr r0, r0, #5
-; ARM6-NEXT:    lsr r3, r2, #5
+; ARM6-NEXT:    bic r0, lr, r0, lsr r12
+; ARM6-NEXT:    bic r3, lr, r3, lsr r2
 ; ARM6-NEXT:    pop {r11, pc}
 ;
 ; ARM78-LABEL: vec_4xi32_nonsplat_undef2_eq:
 ; ARM78:       @ %bb.0:
 ; ARM78-NEXT:    vmov.i32 q8, #0x1
 ; ARM78-NEXT:    mov r12, sp
 ; ARM78-NEXT:    vld1.64 {d18, d19}, [r12]
 ; ARM78-NEXT:    vshl.u32 q8, q8, q9
 ; ARM78-NEXT:    vmov d19, r2, r3
 ; ARM78-NEXT:    vmov d18, r0, r1
 ; ARM78-NEXT:    vtst.32 q8, q8, q9
 ; ARM78-NEXT:    vmvn q8, q8
 ; ARM78-NEXT:    vmovn.i32 d16, q8
 ; ARM78-NEXT:    vmov r0, r1, d16
 ; ARM78-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: vec_4xi32_nonsplat_undef2_eq:
 ; THUMB6:       @ %bb.0:
-; THUMB6-NEXT:    push {r4, r5, r7, lr}
-; THUMB6-NEXT:    ldr r4, [sp, #16]
+; THUMB6-NEXT:    push {r4, lr}
+; THUMB6-NEXT:    ldr r2, [sp, #8]
+; THUMB6-NEXT:    lsrs r0, r2
 ; THUMB6-NEXT:    movs r2, #1
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsls r5, r4
-; THUMB6-NEXT:    ands r5, r0
-; THUMB6-NEXT:    rsbs r0, r5, #0
-; THUMB6-NEXT:    adcs r0, r5
+; THUMB6-NEXT:    ands r0, r2
+; THUMB6-NEXT:    rsbs r4, r0, #0
+; THUMB6-NEXT:    adcs r0, r4
+; THUMB6-NEXT:    ldr r4, [sp, #12]
+; THUMB6-NEXT:    lsrs r1, r4
+; THUMB6-NEXT:    ands r1, r2
+; THUMB6-NEXT:    rsbs r4, r1, #0
+; THUMB6-NEXT:    adcs r1, r4
 ; THUMB6-NEXT:    ldr r4, [sp, #20]
-; THUMB6-NEXT:    mov r5, r2
-; THUMB6-NEXT:    lsls r5, r4
-; THUMB6-NEXT:    ands r5, r1
-; THUMB6-NEXT:    rsbs r1, r5, #0
-; THUMB6-NEXT:    adcs r1, r5
-; THUMB6-NEXT:    ldr r4, [sp, #28]
-; THUMB6-NEXT:    lsls r2, r4
-; THUMB6-NEXT:    ands r2, r3
-; THUMB6-NEXT:    rsbs r3, r2, #0
+; THUMB6-NEXT:    lsrs r3, r4
+; THUMB6-NEXT:    ands r3, r2
+; THUMB6-NEXT:    rsbs r2, r3, #0
 ; THUMB6-NEXT:    adcs r3, r2
-; THUMB6-NEXT:    pop {r4, r5, r7, pc}
+; THUMB6-NEXT:    pop {r4, pc}
 ;
 ; THUMB78-LABEL: vec_4xi32_nonsplat_undef2_eq:
 ; THUMB78:       @ %bb.0:
 ; THUMB78-NEXT:    vmov.i32 q8, #0x1
 ; THUMB78-NEXT:    mov r12, sp
 ; THUMB78-NEXT:    vld1.64 {d18, d19}, [r12]
 ; THUMB78-NEXT:    vshl.u32 q8, q8, q9
 ; THUMB78-NEXT:    vmov d19, r2, r3
; THUMB78-NEXT:    bx lr
   ret <4 x i1> %res
 }
 
 ;------------------------------------------------------------------------------;
 ; A special tests
 ;------------------------------------------------------------------------------;
 
 define i1 @scalar_i8_signbit_ne(i8 %x, i8 %y) nounwind {
-; ARM6-LABEL: scalar_i8_signbit_ne:
-; ARM6:       @ %bb.0:
-; ARM6-NEXT:    uxtb r1, r1
-; ARM6-NEXT:    mvn r2, #127
-; ARM6-NEXT:    and r0, r0, r2, lsl r1
-; ARM6-NEXT:    uxtb r0, r0
-; ARM6-NEXT:    cmp r0, #0
-; ARM6-NEXT:    movne r0, #1
-; ARM6-NEXT:    bx lr
-;
-; ARM78-LABEL: scalar_i8_signbit_ne:
-; ARM78:       @ %bb.0:
-; ARM78-NEXT:    uxtb r1, r1
-; ARM78-NEXT:    mvn r2, #127
-; ARM78-NEXT:    and r0, r0, r2, lsl r1
-; ARM78-NEXT:    uxtb r0, r0
-; ARM78-NEXT:    cmp r0, #0
-; ARM78-NEXT:    movwne r0, #1
-; ARM78-NEXT:    bx lr
+; ARM-LABEL: scalar_i8_signbit_ne:
+; ARM:       @ %bb.0:
+; ARM-NEXT:    uxtb r1, r1
+; ARM-NEXT:    uxtb r0, r0
+; ARM-NEXT:    lsr r0, r0, r1
+; ARM-NEXT:    uxtb r0, r0
+; ARM-NEXT:    lsr r0, r0, #7
+; ARM-NEXT:    bx lr
 ;
 ; THUMB6-LABEL: scalar_i8_signbit_ne:
 ; THUMB6:       @ %bb.0:
 ; THUMB6-NEXT:    uxtb r1, r1
-; THUMB6-NEXT:    movs r2, #127
-; THUMB6-NEXT:    mvns r2, r2
-; THUMB6-NEXT:    lsls r2, r1
-; THUMB6-NEXT:    ands r2, r0
-; THUMB6-NEXT:    uxtb r0, r2
-; THUMB6-NEXT:    subs r1, r0, #1
-; THUMB6-NEXT:    sbcs r0, r1
+; THUMB6-NEXT:    uxtb r0, r0
+; THUMB6-NEXT:    lsrs r0, r1
+; THUMB6-NEXT:    uxtb r0, r0
+; THUMB6-NEXT:    lsrs r0, r0, #7
 ; THUMB6-NEXT:    bx lr
 ;
-; THUMB78-LABEL: scalar_i8_signbit_ne:
-; THUMB78:       @ %bb.0:
-; THUMB78-NEXT:    uxtb r1, r1
-; THUMB78-NEXT:    mvn r2, #127
-; THUMB78-NEXT:    lsl.w r1, r2, r1
-; THUMB78-NEXT:    ands r0, r1
-; THUMB78-NEXT:    uxtb r0, r0
-; THUMB78-NEXT:    cmp r0, #0
-; THUMB78-NEXT:    it ne
-; THUMB78-NEXT:    movne r0, #1
-; THUMB78-NEXT:    bx lr
+; THUMB7-LABEL: scalar_i8_signbit_ne:
+; THUMB7:       @ %bb.0:
+; THUMB7-NEXT:    uxtb r1, r1
+; THUMB7-NEXT:    uxtb r0, r0
+; THUMB7-NEXT:    lsrs r0, r1
+; THUMB7-NEXT:    uxtb r0, r0
+; THUMB7-NEXT:    lsrs r0, r0, #7
+; THUMB7-NEXT:    bx lr
+;
+; THUMB8-LABEL: scalar_i8_signbit_ne:
+; THUMB8:       @ %bb.0:
+; THUMB8-NEXT:    uxtb r0, r0
+; THUMB8-NEXT:    uxtb r1, r1
+; THUMB8-NEXT:    lsrs r0, r1
+; THUMB8-NEXT:    uxtb r0, r0
+; THUMB8-NEXT:    lsrs r0, r0, #7
+; THUMB8-NEXT:    bx lr
   %t0 = shl i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp ne i8 %t1, 0 ;  we are perfectly happy with 'ne' predicate
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;
 ; What if X is a constant too?

test/CodeGen/PowerPC/shift-cmp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s | FileCheck %s
 target datalayout = "e-m:e-i64:64-n32:64"
 target triple = "powerpc64le-unknown-linux-gnu"
 
 define i1 @and_cmp_variable_power_of_two(i32 %x, i32 %y) {
 ; CHECK-LABEL: and_cmp_variable_power_of_two:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    subfic 4, 4, 32
-; CHECK-NEXT:    rlwnm 3, 3, 4, 31, 31
+; CHECK-NEXT:    srw 3, 3, 4
 ; CHECK-NEXT:    blr
   %shl = shl i32 1, %y
   %and = and i32 %x, %shl
   %cmp = icmp eq i32 %and, %shl
   ret i1 %cmp
 }
 
 define i1 @and_cmp_variable_power_of_two_64(i64 %x, i64 %y) {
 ; CHECK-LABEL: and_cmp_variable_power_of_two_64:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    subfic 4, 4, 64
-; CHECK-NEXT:    rldcl 3, 3, 4, 63
+; CHECK-NEXT:    srd 3, 3, 4
 ; CHECK-NEXT:    blr
   %shl = shl i64 1, %y
   %and = and i64 %x, %shl
   %cmp = icmp eq i64 %and, %shl
   ret i1 %cmp
 }
 
 define i1 @and_ncmp_variable_power_of_two(i32 %x, i32 %y) {
 ; CHECK-LABEL: and_ncmp_variable_power_of_two:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    subfic 4, 4, 32
-; CHECK-NEXT:    nor 3, 3, 3
-; CHECK-NEXT:    rlwnm 3, 3, 4, 31, 31
+; CHECK-NEXT:    srw 3, 3, 4
+; CHECK-NEXT:    xori 3, 3, 1
 ; CHECK-NEXT:    blr
   %shl = shl i32 1, %y
   %and = and i32 %x, %shl
   %cmp = icmp ne i32 %and, %shl
   ret i1 %cmp
 }
 
 define i1 @and_ncmp_variable_power_of_two_64(i64 %x, i64 %y) {
 ; CHECK-LABEL: and_ncmp_variable_power_of_two_64:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    not 3, 3
-; CHECK-NEXT:    subfic 4, 4, 64
-; CHECK-NEXT:    rldcl 3, 3, 4, 63
+; CHECK-NEXT:    srd 3, 3, 4
+; CHECK-NEXT:    xori 3, 3, 1
 ; CHECK-NEXT:    blr
   %shl = shl i64 1, %y
   %and = and i64 %x, %shl
   %cmp = icmp ne i64 %and, %shl
   ret i1 %cmp
 }

test/CodeGen/X86/hoist-and-by-const-from-lshr-in-eqcmp-zero.ll

 ;------------------------------------------------------------------------------;
 
 ; i8 scalar
 
 define i1 @scalar_i8_signbit_eq(i8 %x, i8 %y) nounwind {
 ; X86-LABEL: scalar_i8_signbit_eq:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NEXT:    movb $-128, %al
-; X86-NEXT:    shrb %cl, %al
-; X86-NEXT:    testb %al, {{[0-9]+}}(%esp)
+; X86-NEXT:    movb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    shlb %cl, %al
+; X86-NEXT:    testb $-128, %al
 ; X86-NEXT:    sete %al
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: scalar_i8_signbit_eq:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %esi, %ecx
-; X64-NEXT:    movb $-128, %al
 ; X64-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NEXT:    shrb %cl, %al
-; X64-NEXT:    testb %dil, %al
+; X64-NEXT:    shlb %cl, %dil
+; X64-NEXT:    testb $-128, %dil
 ; X64-NEXT:    sete %al
 ; X64-NEXT:    retq
   %t0 = lshr i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i8_lowestbit_eq(i8 %x, i8 %y) nounwind {
 ; X86-LABEL: scalar_i8_lowestbit_eq:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NEXT:    movb $1, %al
-; X86-NEXT:    shrb %cl, %al
-; X86-NEXT:    testb %al, {{[0-9]+}}(%esp)
+; X86-NEXT:    movb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    shlb %cl, %al
+; X86-NEXT:    testb $1, %al
 ; X86-NEXT:    sete %al
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: scalar_i8_lowestbit_eq:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %esi, %ecx
-; X64-NEXT:    movb $1, %al
 ; X64-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NEXT:    shrb %cl, %al
-; X64-NEXT:    testb %dil, %al
+; X64-NEXT:    shlb %cl, %dil
+; X64-NEXT:    testb $1, %dil
 ; X64-NEXT:    sete %al
 ; X64-NEXT:    retq
   %t0 = lshr i8 1, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i8_bitsinmiddle_eq(i8 %x, i8 %y) nounwind {
 ; X86-LABEL: scalar_i8_bitsinmiddle_eq:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NEXT:    movb $24, %al
-; X86-NEXT:    shrb %cl, %al
-; X86-NEXT:    testb %al, {{[0-9]+}}(%esp)
+; X86-NEXT:    movb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    shlb %cl, %al
+; X86-NEXT:    testb $24, %al
 ; X86-NEXT:    sete %al
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: scalar_i8_bitsinmiddle_eq:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %esi, %ecx
-; X64-NEXT:    movb $24, %al
 ; X64-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NEXT:    shrb %cl, %al
-; X64-NEXT:    testb %dil, %al
+; X64-NEXT:    shlb %cl, %dil
+; X64-NEXT:    testb $24, %dil
 ; X64-NEXT:    sete %al
 ; X64-NEXT:    retq
   %t0 = lshr i8 24, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 ; i16 scalar
 
 define i1 @scalar_i16_signbit_eq(i16 %x, i16 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i16_signbit_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $32768, %eax # imm = 0x8000
-; X86-NOBMI-NEXT:    shrl %cl, %eax
-; X86-NOBMI-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shll %cl, %eax
+; X86-NOBMI-NEXT:    testl $32768, %eax # imm = 0x8000
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i16_signbit_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $32768, %eax # imm = 0x8000
-; X86-BMI1-NEXT:    shrl %cl, %eax
-; X86-BMI1-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shll %cl, %eax
+; X86-BMI1-NEXT:    testl $32768, %eax # imm = 0x8000
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i16_signbit_eq:
 ; X86-BMI12:       # %bb.0:
 ; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X86-BMI12-NEXT:    movl $32768, %ecx # imm = 0x8000
-; X86-BMI12-NEXT:    shrxl %eax, %ecx, %eax
-; X86-BMI12-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-BMI12-NEXT:    shlxl %eax, {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    testl $32768, %eax # imm = 0x8000
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i16_signbit_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movl %esi, %ecx
-; X64-NOBMI-NEXT:    movl $32768, %eax # imm = 0x8000
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NOBMI-NEXT:    shrl %cl, %eax
-; X64-NOBMI-NEXT:    testw %di, %ax
+; X64-NOBMI-NEXT:    shll %cl, %edi
+; X64-NOBMI-NEXT:    testl $32768, %edi # imm = 0x8000
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i16_signbit_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movl %esi, %ecx
-; X64-BMI1-NEXT:    movl $32768, %eax # imm = 0x8000
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-BMI1-NEXT:    shrl %cl, %eax
-; X64-BMI1-NEXT:    testw %di, %ax
+; X64-BMI1-NEXT:    shll %cl, %edi
+; X64-BMI1-NEXT:    testl $32768, %edi # imm = 0x8000
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i16_signbit_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movl $32768, %eax # imm = 0x8000
-; X64-BMI12-NEXT:    shrxl %esi, %eax, %eax
-; X64-BMI12-NEXT:    testw %di, %ax
+; X64-BMI12-NEXT:    shlxl %esi, %edi, %eax
+; X64-BMI12-NEXT:    testl $32768, %eax # imm = 0x8000
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = lshr i16 32768, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i16_lowestbit_eq(i16 %x, i16 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i16_lowestbit_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $1, %eax
-; X86-NOBMI-NEXT:    shrl %cl, %eax
-; X86-NOBMI-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shll %cl, %eax
+; X86-NOBMI-NEXT:    testb $1, %al
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i16_lowestbit_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $1, %eax
-; X86-BMI1-NEXT:    shrl %cl, %eax
-; X86-BMI1-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shll %cl, %eax
+; X86-BMI1-NEXT:    testb $1, %al
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i16_lowestbit_eq:
 ; X86-BMI12:       # %bb.0:
 ; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X86-BMI12-NEXT:    movl $1, %ecx
-; X86-BMI12-NEXT:    shrxl %eax, %ecx, %eax
-; X86-BMI12-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-BMI12-NEXT:    shlxl %eax, {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    testb $1, %al
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i16_lowestbit_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movl %esi, %ecx
-; X64-NOBMI-NEXT:    movl $1, %eax
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NOBMI-NEXT:    shrl %cl, %eax
-; X64-NOBMI-NEXT:    testw %di, %ax
+; X64-NOBMI-NEXT:    shll %cl, %edi
+; X64-NOBMI-NEXT:    testb $1, %dil
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i16_lowestbit_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movl %esi, %ecx
-; X64-BMI1-NEXT:    movl $1, %eax
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-BMI1-NEXT:    shrl %cl, %eax
-; X64-BMI1-NEXT:    testw %di, %ax
+; X64-BMI1-NEXT:    shll %cl, %edi
+; X64-BMI1-NEXT:    testb $1, %dil
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i16_lowestbit_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movl $1, %eax
-; X64-BMI12-NEXT:    shrxl %esi, %eax, %eax
-; X64-BMI12-NEXT:    testw %di, %ax
+; X64-BMI12-NEXT:    shlxl %esi, %edi, %eax
+; X64-BMI12-NEXT:    testb $1, %al
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = lshr i16 1, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i16_bitsinmiddle_eq(i16 %x, i16 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $4080, %eax # imm = 0xFF0
-; X86-NOBMI-NEXT:    shrl %cl, %eax
-; X86-NOBMI-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shll %cl, %eax
+; X86-NOBMI-NEXT:    testl $4080, %eax # imm = 0xFF0
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $4080, %eax # imm = 0xFF0
-; X86-BMI1-NEXT:    shrl %cl, %eax
-; X86-BMI1-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shll %cl, %eax
+; X86-BMI1-NEXT:    testl $4080, %eax # imm = 0xFF0
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X86-BMI12:       # %bb.0:
 ; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X86-BMI12-NEXT:    movl $4080, %ecx # imm = 0xFF0
-; X86-BMI12-NEXT:    shrxl %eax, %ecx, %eax
-; X86-BMI12-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-BMI12-NEXT:    shlxl %eax, {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    testl $4080, %eax # imm = 0xFF0
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movl %esi, %ecx
-; X64-NOBMI-NEXT:    movl $4080, %eax # imm = 0xFF0
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NOBMI-NEXT:    shrl %cl, %eax
-; X64-NOBMI-NEXT:    testw %di, %ax
+; X64-NOBMI-NEXT:    shll %cl, %edi
+; X64-NOBMI-NEXT:    testl $4080, %edi # imm = 0xFF0
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movl %esi, %ecx
-; X64-BMI1-NEXT:    movl $4080, %eax # imm = 0xFF0
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-BMI1-NEXT:    shrl %cl, %eax
-; X64-BMI1-NEXT:    testw %di, %ax
+; X64-BMI1-NEXT:    shll %cl, %edi
+; X64-BMI1-NEXT:    testl $4080, %edi # imm = 0xFF0
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movl $4080, %eax # imm = 0xFF0
-; X64-BMI12-NEXT:    shrxl %esi, %eax, %eax
-; X64-BMI12-NEXT:    testw %di, %ax
+; X64-BMI12-NEXT:    shlxl %esi, %edi, %eax
+; X64-BMI12-NEXT:    testl $4080, %eax # imm = 0xFF0
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = lshr i16 4080, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 ; i32 scalar
 
 define i1 @scalar_i32_signbit_eq(i32 %x, i32 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i32_signbit_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
-; X86-NOBMI-NEXT:    shrl %cl, %eax
-; X86-NOBMI-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shll %cl, %eax
+; X86-NOBMI-NEXT:    testl $-2147483648, %eax # imm = 0x80000000
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i32_signbit_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
-; X86-BMI1-NEXT:    shrl %cl, %eax
-; X86-BMI1-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shll %cl, %eax
+; X86-BMI1-NEXT:    testl $-2147483648, %eax # imm = 0x80000000
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i32_signbit_eq:
 ; X86-BMI12:       # %bb.0:
 ; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X86-BMI12-NEXT:    movl $-2147483648, %ecx # imm = 0x80000000
-; X86-BMI12-NEXT:    shrxl %eax, %ecx, %eax
-; X86-BMI12-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-BMI12-NEXT:    shlxl %eax, {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    testl $-2147483648, %eax # imm = 0x80000000
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i32_signbit_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movl %esi, %ecx
-; X64-NOBMI-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NOBMI-NEXT:    shrl %cl, %eax
-; X64-NOBMI-NEXT:    testl %edi, %eax
+; X64-NOBMI-NEXT:    shll %cl, %edi
+; X64-NOBMI-NEXT:    testl $-2147483648, %edi # imm = 0x80000000
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i32_signbit_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movl %esi, %ecx
-; X64-BMI1-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-BMI1-NEXT:    shrl %cl, %eax
-; X64-BMI1-NEXT:    testl %edi, %eax
+; X64-BMI1-NEXT:    shll %cl, %edi
+; X64-BMI1-NEXT:    testl $-2147483648, %edi # imm = 0x80000000
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i32_signbit_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
-; X64-BMI12-NEXT:    shrxl %esi, %eax, %eax
-; X64-BMI12-NEXT:    testl %edi, %eax
+; X64-BMI12-NEXT:    shlxl %esi, %edi, %eax
+; X64-BMI12-NEXT:    testl $-2147483648, %eax # imm = 0x80000000
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = lshr i32 2147483648, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i32_lowestbit_eq(i32 %x, i32 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i32_lowestbit_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $1, %eax
-; X86-NOBMI-NEXT:    shrl %cl, %eax
-; X86-NOBMI-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shll %cl, %eax
+; X86-NOBMI-NEXT:    testb $1, %al
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i32_lowestbit_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $1, %eax
-; X86-BMI1-NEXT:    shrl %cl, %eax
-; X86-BMI1-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shll %cl, %eax
+; X86-BMI1-NEXT:    testb $1, %al
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i32_lowestbit_eq:
 ; X86-BMI12:       # %bb.0:
 ; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X86-BMI12-NEXT:    movl $1, %ecx
-; X86-BMI12-NEXT:    shrxl %eax, %ecx, %eax
-; X86-BMI12-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-BMI12-NEXT:    shlxl %eax, {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    testb $1, %al
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i32_lowestbit_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movl %esi, %ecx
-; X64-NOBMI-NEXT:    movl $1, %eax
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NOBMI-NEXT:    shrl %cl, %eax
-; X64-NOBMI-NEXT:    testl %edi, %eax
+; X64-NOBMI-NEXT:    shll %cl, %edi
+; X64-NOBMI-NEXT:    testb $1, %dil
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i32_lowestbit_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movl %esi, %ecx
-; X64-BMI1-NEXT:    movl $1, %eax
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-BMI1-NEXT:    shrl %cl, %eax
-; X64-BMI1-NEXT:    testl %edi, %eax
+; X64-BMI1-NEXT:    shll %cl, %edi
+; X64-BMI1-NEXT:    testb $1, %dil
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i32_lowestbit_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movl $1, %eax
-; X64-BMI12-NEXT:    shrxl %esi, %eax, %eax
-; X64-BMI12-NEXT:    testl %edi, %eax
+; X64-BMI12-NEXT:    shlxl %esi, %edi, %eax
+; X64-BMI12-NEXT:    testb $1, %al
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = lshr i32 1, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i32_bitsinmiddle_eq(i32 %x, i32 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $16776960, %eax # imm = 0xFFFF00
-; X86-NOBMI-NEXT:    shrl %cl, %eax
-; X86-NOBMI-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shll %cl, %eax
+; X86-NOBMI-NEXT:    testl $16776960, %eax # imm = 0xFFFF00
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $16776960, %eax # imm = 0xFFFF00
-; X86-BMI1-NEXT:    shrl %cl, %eax
-; X86-BMI1-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shll %cl, %eax
+; X86-BMI1-NEXT:    testl $16776960, %eax # imm = 0xFFFF00
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X86-BMI12:       # %bb.0:
 ; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X86-BMI12-NEXT:    movl $16776960, %ecx # imm = 0xFFFF00
-; X86-BMI12-NEXT:    shrxl %eax, %ecx, %eax
-; X86-BMI12-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-BMI12-NEXT:    shlxl %eax, {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    testl $16776960, %eax # imm = 0xFFFF00
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movl %esi, %ecx
-; X64-NOBMI-NEXT:    movl $16776960, %eax # imm = 0xFFFF00
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NOBMI-NEXT:    shrl %cl, %eax
-; X64-NOBMI-NEXT:    testl %edi, %eax
+; X64-NOBMI-NEXT:    shll %cl, %edi
+; X64-NOBMI-NEXT:    testl $16776960, %edi # imm = 0xFFFF00
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movl %esi, %ecx
-; X64-BMI1-NEXT:    movl $16776960, %eax # imm = 0xFFFF00
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-BMI1-NEXT:    shrl %cl, %eax
-; X64-BMI1-NEXT:    testl %edi, %eax
+; X64-BMI1-NEXT:    shll %cl, %edi
+; X64-BMI1-NEXT:    testl $16776960, %edi # imm = 0xFFFF00
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movl $16776960, %eax # imm = 0xFFFF00
-; X64-BMI12-NEXT:    shrxl %esi, %eax, %eax
-; X64-BMI12-NEXT:    testl %edi, %eax
+; X64-BMI12-NEXT:    shlxl %esi, %edi, %eax
+; X64-BMI12-NEXT:    testl $16776960, %eax # imm = 0xFFFF00
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = lshr i32 16776960, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 ; i64 scalar
 
 define i1 @scalar_i64_signbit_eq(i64 %x, i64 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i64_signbit_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    pushl %esi
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
-; X86-NOBMI-NEXT:    xorl %edx, %edx
-; X86-NOBMI-NEXT:    xorl %esi, %esi
-; X86-NOBMI-NEXT:    shrdl %cl, %eax, %esi
-; X86-NOBMI-NEXT:    shrl %cl, %eax
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NOBMI-NEXT:    movl %eax, %esi
+; X86-NOBMI-NEXT:    shll %cl, %esi
+; X86-NOBMI-NEXT:    shldl %cl, %eax, %edx
 ; X86-NOBMI-NEXT:    testb $32, %cl
-; X86-NOBMI-NEXT:    cmovnel %eax, %esi
-; X86-NOBMI-NEXT:    cmovnel %edx, %eax
-; X86-NOBMI-NEXT:    andl {{[0-9]+}}(%esp), %esi
-; X86-NOBMI-NEXT:    andl {{[0-9]+}}(%esp), %eax
-; X86-NOBMI-NEXT:    orl %esi, %eax
+; X86-NOBMI-NEXT:    cmovnel %esi, %edx
+; X86-NOBMI-NEXT:    testl $-2147483648, %edx # imm = 0x80000000
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    popl %esi
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i64_signbit_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    pushl %esi
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
-; X86-BMI1-NEXT:    xorl %edx, %edx
-; X86-BMI1-NEXT:    xorl %esi, %esi
-; X86-BMI1-NEXT:    shrdl %cl, %eax, %esi
-; X86-BMI1-NEXT:    shrl %cl, %eax
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-BMI1-NEXT:    movl %eax, %esi
+; X86-BMI1-NEXT:    shll %cl, %esi
+; X86-BMI1-NEXT:    shldl %cl, %eax, %edx
 ; X86-BMI1-NEXT:    testb $32, %cl
-; X86-BMI1-NEXT:    cmovnel %eax, %esi
-; X86-BMI1-NEXT:    cmovnel %edx, %eax
-; X86-BMI1-NEXT:    andl {{[0-9]+}}(%esp), %esi
-; X86-BMI1-NEXT:    andl {{[0-9]+}}(%esp), %eax
-; X86-BMI1-NEXT:    orl %esi, %eax
+; X86-BMI1-NEXT:    cmovnel %esi, %edx
+; X86-BMI1-NEXT:    testl $-2147483648, %edx # imm = 0x80000000
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    popl %esi
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i64_signbit_eq:
 ; X86-BMI12:       # %bb.0:
-; X86-BMI12-NEXT:    pushl %esi
 ; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI12-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
-; X86-BMI12-NEXT:    xorl %edx, %edx
-; X86-BMI12-NEXT:    xorl %esi, %esi
-; X86-BMI12-NEXT:    shrdl %cl, %eax, %esi
-; X86-BMI12-NEXT:    shrxl %ecx, %eax, %eax
+; X86-BMI12-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-BMI12-NEXT:    shldl %cl, %eax, %edx
+; X86-BMI12-NEXT:    shlxl %ecx, %eax, %eax
 ; X86-BMI12-NEXT:    testb $32, %cl
-; X86-BMI12-NEXT:    cmovnel %eax, %esi
-; X86-BMI12-NEXT:    cmovnel %edx, %eax
-; X86-BMI12-NEXT:    andl {{[0-9]+}}(%esp), %esi
-; X86-BMI12-NEXT:    andl {{[0-9]+}}(%esp), %eax
-; X86-BMI12-NEXT:    orl %esi, %eax
+; X86-BMI12-NEXT:    cmovel %edx, %eax
+; X86-BMI12-NEXT:    testl $-2147483648, %eax # imm = 0x80000000
 ; X86-BMI12-NEXT:    sete %al
-; X86-BMI12-NEXT:    popl %esi
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i64_signbit_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movq %rsi, %rcx
-; X64-NOBMI-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $rcx
-; X64-NOBMI-NEXT:    shrq %cl, %rax
-; X64-NOBMI-NEXT:    testq %rdi, %rax
+; X64-NOBMI-NEXT:    shlq %cl, %rdi
+; X64-NOBMI-NEXT:    shrq $63, %rdi
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i64_signbit_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movq %rsi, %rcx
-; X64-BMI1-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $rcx
-; X64-BMI1-NEXT:    shrq %cl, %rax
-; X64-BMI1-NEXT:    testq %rdi, %rax
+; X64-BMI1-NEXT:    shlq %cl, %rdi
+; X64-BMI1-NEXT:    shrq $63, %rdi
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i64_signbit_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
-; X64-BMI12-NEXT:    shrxq %rsi, %rax, %rax
-; X64-BMI12-NEXT:    testq %rdi, %rax
+; X64-BMI12-NEXT:    shlxq %rsi, %rdi, %rax
+; X64-BMI12-NEXT:    shrq $63, %rax
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = lshr i64 9223372036854775808, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i64_lowestbit_eq(i64 %x, i64 %y) nounwind {
-; X86-LABEL: scalar_i64_lowestbit_eq:
-; X86:       # %bb.0:
-; X86-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NEXT:    xorl %eax, %eax
-; X86-NEXT:    movl $1, %edx
-; X86-NEXT:    shrdl %cl, %eax, %edx
-; X86-NEXT:    testb $32, %cl
-; X86-NEXT:    cmovnel %eax, %edx
-; X86-NEXT:    andl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    orl $0, %edx
-; X86-NEXT:    sete %al
-; X86-NEXT:    retl
+; X86-NOBMI-LABEL: scalar_i64_lowestbit_eq:
+; X86-NOBMI:       # %bb.0:
+; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shll %cl, %eax
+; X86-NOBMI-NEXT:    xorl %edx, %edx
+; X86-NOBMI-NEXT:    testb $32, %cl
+; X86-NOBMI-NEXT:    cmovel %eax, %edx
+; X86-NOBMI-NEXT:    testb $1, %dl
+; X86-NOBMI-NEXT:    sete %al
+; X86-NOBMI-NEXT:    retl
+;
+; X86-BMI1-LABEL: scalar_i64_lowestbit_eq:
+; X86-BMI1:       # %bb.0:
+; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shll %cl, %eax
+; X86-BMI1-NEXT:    xorl %edx, %edx
+; X86-BMI1-NEXT:    testb $32, %cl
+; X86-BMI1-NEXT:    cmovel %eax, %edx
+; X86-BMI1-NEXT:    testb $1, %dl
+; X86-BMI1-NEXT:    sete %al
+; X86-BMI1-NEXT:    retl
+;
+; X86-BMI12-LABEL: scalar_i64_lowestbit_eq:
+; X86-BMI12:       # %bb.0:
+; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
+; X86-BMI12-NEXT:    shlxl %eax, {{[0-9]+}}(%esp), %ecx
+; X86-BMI12-NEXT:    xorl %edx, %edx
+; X86-BMI12-NEXT:    testb $32, %al
+; X86-BMI12-NEXT:    cmovel %ecx, %edx
+; X86-BMI12-NEXT:    testb $1, %dl
+; X86-BMI12-NEXT:    sete %al
+; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i64_lowestbit_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movq %rsi, %rcx
-; X64-NOBMI-NEXT:    movl $1, %eax
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $rcx
-; X64-NOBMI-NEXT:    shrq %cl, %rax
-; X64-NOBMI-NEXT:    testq %rdi, %rax
+; X64-NOBMI-NEXT:    shlq %cl, %rdi
+; X64-NOBMI-NEXT:    testb $1, %dil
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i64_lowestbit_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movq %rsi, %rcx
-; X64-BMI1-NEXT:    movl $1, %eax
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $rcx
-; X64-BMI1-NEXT:    shrq %cl, %rax
-; X64-BMI1-NEXT:    testq %rdi, %rax
+; X64-BMI1-NEXT:    shlq %cl, %rdi
+; X64-BMI1-NEXT:    testb $1, %dil
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i64_lowestbit_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movl $1, %eax
-; X64-BMI12-NEXT:    shrxq %rsi, %rax, %rax
-; X64-BMI12-NEXT:    testq %rdi, %rax
+; X64-BMI12-NEXT:    shlxq %rsi, %rdi, %rax
+; X64-BMI12-NEXT:    testb $1, %al
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = lshr i64 1, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i64_bitsinmiddle_eq(i64 %x, i64 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    pushl %esi
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $65535, %eax # imm = 0xFFFF
-; X86-NOBMI-NEXT:    movl $-65536, %edx # imm = 0xFFFF0000
-; X86-NOBMI-NEXT:    shrdl %cl, %eax, %edx
-; X86-NOBMI-NEXT:    shrl %cl, %eax
-; X86-NOBMI-NEXT:    xorl %esi, %esi
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NOBMI-NEXT:    movl %eax, %esi
+; X86-NOBMI-NEXT:    shll %cl, %esi
+; X86-NOBMI-NEXT:    shldl %cl, %eax, %edx
+; X86-NOBMI-NEXT:    xorl %eax, %eax
 ; X86-NOBMI-NEXT:    testb $32, %cl
-; X86-NOBMI-NEXT:    cmovnel %eax, %edx
-; X86-NOBMI-NEXT:    cmovel %eax, %esi
-; X86-NOBMI-NEXT:    andl {{[0-9]+}}(%esp), %edx
-; X86-NOBMI-NEXT:    andl {{[0-9]+}}(%esp), %esi
-; X86-NOBMI-NEXT:    orl %edx, %esi
+; X86-NOBMI-NEXT:    cmovnel %esi, %edx
+; X86-NOBMI-NEXT:    movzwl %dx, %ecx
+; X86-NOBMI-NEXT:    cmovel %esi, %eax
+; X86-NOBMI-NEXT:    andl $-65536, %eax # imm = 0xFFFF0000
+; X86-NOBMI-NEXT:    orl %ecx, %eax
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    popl %esi
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    pushl %esi
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $65535, %eax # imm = 0xFFFF
-; X86-BMI1-NEXT:    movl $-65536, %edx # imm = 0xFFFF0000
-; X86-BMI1-NEXT:    shrdl %cl, %eax, %edx
-; X86-BMI1-NEXT:    shrl %cl, %eax
-; X86-BMI1-NEXT:    xorl %esi, %esi
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-BMI1-NEXT:    movl %eax, %esi
+; X86-BMI1-NEXT:    shll %cl, %esi
+; X86-BMI1-NEXT:    shldl %cl, %eax, %edx
+; X86-BMI1-NEXT:    xorl %eax, %eax
 ; X86-BMI1-NEXT:    testb $32, %cl
-; X86-BMI1-NEXT:    cmovnel %eax, %edx
-; X86-BMI1-NEXT:    cmovel %eax, %esi
-; X86-BMI1-NEXT:    andl {{[0-9]+}}(%esp), %edx
-; X86-BMI1-NEXT:    andl {{[0-9]+}}(%esp), %esi
-; X86-BMI1-NEXT:    orl %edx, %esi
+; X86-BMI1-NEXT:    cmovnel %esi, %edx
+; X86-BMI1-NEXT:    movzwl %dx, %ecx
+; X86-BMI1-NEXT:    cmovel %esi, %eax
+; X86-BMI1-NEXT:    andl $-65536, %eax # imm = 0xFFFF0000
+; X86-BMI1-NEXT:    orl %ecx, %eax
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    popl %esi
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X86-BMI12:       # %bb.0:
 ; X86-BMI12-NEXT:    pushl %esi
 ; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI12-NEXT:    movl $65535, %eax # imm = 0xFFFF
-; X86-BMI12-NEXT:    movl $-65536, %edx # imm = 0xFFFF0000
-; X86-BMI12-NEXT:    shrdl %cl, %eax, %edx
-; X86-BMI12-NEXT:    shrxl %ecx, %eax, %eax
+; X86-BMI12-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-BMI12-NEXT:    shldl %cl, %eax, %edx
+; X86-BMI12-NEXT:    shlxl %ecx, %eax, %eax
 ; X86-BMI12-NEXT:    xorl %esi, %esi
 ; X86-BMI12-NEXT:    testb $32, %cl
 ; X86-BMI12-NEXT:    cmovnel %eax, %edx
+; X86-BMI12-NEXT:    movzwl %dx, %ecx
 ; X86-BMI12-NEXT:    cmovel %eax, %esi
-; X86-BMI12-NEXT:    andl {{[0-9]+}}(%esp), %edx
-; X86-BMI12-NEXT:    andl {{[0-9]+}}(%esp), %esi
-; X86-BMI12-NEXT:    orl %edx, %esi
+; X86-BMI12-NEXT:    andl $-65536, %esi # imm = 0xFFFF0000
+; X86-BMI12-NEXT:    orl %ecx, %esi
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    popl %esi
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movq %rsi, %rcx
-; X64-NOBMI-NEXT:    movabsq $281474976645120, %rax # imm = 0xFFFFFFFF0000
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $rcx
-; X64-NOBMI-NEXT:    shrq %cl, %rax
-; X64-NOBMI-NEXT:    testq %rdi, %rax
+; X64-NOBMI-NEXT:    shlq %cl, %rdi
+; X64-NOBMI-NEXT:    movabsq $281474976645120, %rax # imm = 0xFFFFFFFF0000
+; X64-NOBMI-NEXT:    testq %rax, %rdi
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movq %rsi, %rcx
-; X64-BMI1-NEXT:    movabsq $281474976645120, %rax # imm = 0xFFFFFFFF0000
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $rcx
-; X64-BMI1-NEXT:    shrq %cl, %rax
-; X64-BMI1-NEXT:    testq %rdi, %rax
+; X64-BMI1-NEXT:    shlq %cl, %rdi
+; X64-BMI1-NEXT:    movabsq $281474976645120, %rax # imm = 0xFFFFFFFF0000
+; X64-BMI1-NEXT:    testq %rax, %rdi
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movabsq $281474976645120, %rax # imm = 0xFFFFFFFF0000
-; X64-BMI12-NEXT:    shrxq %rsi, %rax, %rax
-; X64-BMI12-NEXT:    testq %rdi, %rax
+; X64-BMI12-NEXT:    shlxq %rsi, %rdi, %rax
+; X64-BMI12-NEXT:    movabsq $281474976645120, %rcx # imm = 0xFFFFFFFF0000
+; X64-BMI12-NEXT:    testq %rcx, %rax
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = lshr i64 281474976645120, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;
 ; A few trivial vector tests
 ;------------------------------------------------------------------------------;
 
 define <4 x i1> @vec_4xi32_splat_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
-; CHECK-LABEL: vec_4xi32_splat_eq:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    pshuflw {{.*#+}} xmm2 = xmm1[2,3,3,3,4,5,6,7]
-; CHECK-NEXT:    movdqa {{.*#+}} xmm3 = [1,1,1,1]
-; CHECK-NEXT:    movdqa %xmm3, %xmm4
-; CHECK-NEXT:    psrld %xmm2, %xmm4
-; CHECK-NEXT:    pshuflw {{.*#+}} xmm2 = xmm1[0,1,1,1,4,5,6,7]
-; CHECK-NEXT:    movdqa %xmm3, %xmm5
-; CHECK-NEXT:    psrld %xmm2, %xmm5
-; CHECK-NEXT:    punpcklqdq {{.*#+}} xmm5 = xmm5[0],xmm4[0]
-; CHECK-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,3,0,1]
-; CHECK-NEXT:    pshuflw {{.*#+}} xmm2 = xmm1[2,3,3,3,4,5,6,7]
-; CHECK-NEXT:    movdqa %xmm3, %xmm4
-; CHECK-NEXT:    psrld %xmm2, %xmm4
-; CHECK-NEXT:    pshuflw {{.*#+}} xmm1 = xmm1[0,1,1,1,4,5,6,7]
-; CHECK-NEXT:    psrld %xmm1, %xmm3
-; CHECK-NEXT:    punpckhqdq {{.*#+}} xmm3 = xmm3[1],xmm4[1]
-; CHECK-NEXT:    shufps {{.*#+}} xmm5 = xmm5[0,3],xmm3[0,3]
-; CHECK-NEXT:    andps %xmm5, %xmm0
-; CHECK-NEXT:    pxor %xmm1, %xmm1
-; CHECK-NEXT:    pcmpeqd %xmm1, %xmm0
-; CHECK-NEXT:    ret{{[l|q]}}
+; X86-LABEL: vec_4xi32_splat_eq:
+; X86:       # %bb.0:
+; X86-NEXT:    pxor %xmm2, %xmm2
+; X86-NEXT:    pslld $23, %xmm1
+; X86-NEXT:    paddd {{\.LCPI.*}}, %xmm1
+; X86-NEXT:    cvttps2dq %xmm1, %xmm1
+; X86-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[1,1,3,3]
+; X86-NEXT:    pmuludq %xmm1, %xmm0
+; X86-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
+; X86-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
+; X86-NEXT:    pmuludq %xmm3, %xmm1
+; X86-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[0,2,2,3]
+; X86-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; X86-NEXT:    pand {{\.LCPI.*}}, %xmm0
+; X86-NEXT:    pcmpeqd %xmm2, %xmm0
+; X86-NEXT:    retl
+;
+; X64-LABEL: vec_4xi32_splat_eq:
+; X64:       # %bb.0:
+; X64-NEXT:    pxor %xmm2, %xmm2
+; X64-NEXT:    pslld $23, %xmm1
+; X64-NEXT:    paddd {{.*}}(%rip), %xmm1
+; X64-NEXT:    cvttps2dq %xmm1, %xmm1
+; X64-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[1,1,3,3]
+; X64-NEXT:    pmuludq %xmm1, %xmm0
+; X64-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
+; X64-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
+; X64-NEXT:    pmuludq %xmm3, %xmm1
+; X64-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[0,2,2,3]
+; X64-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; X64-NEXT:    pand {{.*}}(%rip), %xmm0
+; X64-NEXT:    pcmpeqd %xmm2, %xmm0
+; X64-NEXT:    retq
   %t0 = lshr <4 x i32> <i32 1, i32 1, i32 1, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 
 define <4 x i1> @vec_4xi32_nonsplat_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; CHECK-LABEL: vec_4xi32_nonsplat_eq:

 ; CHECK-NEXT:    ret{{[l|q]}}
   %t0 = lshr <4 x i32> <i32 0, i32 1, i32 16776960, i32 2147483648>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 
 define <4 x i1> @vec_4xi32_nonsplat_undef0_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
-; CHECK-LABEL: vec_4xi32_nonsplat_undef0_eq:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    pshuflw {{.*#+}} xmm2 = xmm1[2,3,3,3,4,5,6,7]
-; CHECK-NEXT:    movdqa {{.*#+}} xmm3 = <1,1,u,1>
-; CHECK-NEXT:    movdqa %xmm3, %xmm4
-; CHECK-NEXT:    psrld %xmm2, %xmm4
-; CHECK-NEXT:    pshuflw {{.*#+}} xmm2 = xmm1[0,1,1,1,4,5,6,7]
-; CHECK-NEXT:    movdqa %xmm3, %xmm5
-; CHECK-NEXT:    psrld %xmm2, %xmm5
-; CHECK-NEXT:    punpcklqdq {{.*#+}} xmm5 = xmm5[0],xmm4[0]
-; CHECK-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,3,0,1]
-; CHECK-NEXT:    pshuflw {{.*#+}} xmm2 = xmm1[2,3,3,3,4,5,6,7]
-; CHECK-NEXT:    movdqa %xmm3, %xmm4
-; CHECK-NEXT:    psrld %xmm2, %xmm4
-; CHECK-NEXT:    pshuflw {{.*#+}} xmm1 = xmm1[0,1,1,1,4,5,6,7]
-; CHECK-NEXT:    psrld %xmm1, %xmm3
-; CHECK-NEXT:    punpckhqdq {{.*#+}} xmm3 = xmm3[1],xmm4[1]
-; CHECK-NEXT:    shufps {{.*#+}} xmm5 = xmm5[0,3],xmm3[0,3]
-; CHECK-NEXT:    andps %xmm5, %xmm0
-; CHECK-NEXT:    pxor %xmm1, %xmm1
-; CHECK-NEXT:    pcmpeqd %xmm1, %xmm0
-; CHECK-NEXT:    ret{{[l|q]}}
+; X86-LABEL: vec_4xi32_nonsplat_undef0_eq:
+; X86:       # %bb.0:
+; X86-NEXT:    pxor %xmm2, %xmm2
+; X86-NEXT:    pslld $23, %xmm1
+; X86-NEXT:    paddd {{\.LCPI.*}}, %xmm1
+; X86-NEXT:    cvttps2dq %xmm1, %xmm1
+; X86-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[1,1,3,3]
+; X86-NEXT:    pmuludq %xmm1, %xmm0
+; X86-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
+; X86-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
+; X86-NEXT:    pmuludq %xmm3, %xmm1
+; X86-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[0,2,2,3]
+; X86-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; X86-NEXT:    pand {{\.LCPI.*}}, %xmm0
+; X86-NEXT:    pcmpeqd %xmm2, %xmm0
+; X86-NEXT:    retl
+;
+; X64-LABEL: vec_4xi32_nonsplat_undef0_eq:
+; X64:       # %bb.0:
+; X64-NEXT:    pxor %xmm2, %xmm2
+; X64-NEXT:    pslld $23, %xmm1
+; X64-NEXT:    paddd {{.*}}(%rip), %xmm1
+; X64-NEXT:    cvttps2dq %xmm1, %xmm1
+; X64-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[1,1,3,3]
+; X64-NEXT:    pmuludq %xmm1, %xmm0
+; X64-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
+; X64-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
+; X64-NEXT:    pmuludq %xmm3, %xmm1
+; X64-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[0,2,2,3]
+; X64-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; X64-NEXT:    pand {{.*}}(%rip), %xmm0
+; X64-NEXT:    pcmpeqd %xmm2, %xmm0
+; X64-NEXT:    retq
   %t0 = lshr <4 x i32> <i32 1, i32 1, i32 undef, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 define <4 x i1> @vec_4xi32_nonsplat_undef1_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; CHECK-LABEL: vec_4xi32_nonsplat_undef1_eq:
 ; CHECK:       # %bb.0:

 ;------------------------------------------------------------------------------;
 ; A special tests
 ;------------------------------------------------------------------------------;
 
 define i1 @scalar_i8_signbit_ne(i8 %x, i8 %y) nounwind {
 ; X86-LABEL: scalar_i8_signbit_ne:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NEXT:    movb $-128, %al
-; X86-NEXT:    shrb %cl, %al
-; X86-NEXT:    testb %al, {{[0-9]+}}(%esp)
-; X86-NEXT:    setne %al
+; X86-NEXT:    movb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    shlb %cl, %al
+; X86-NEXT:    shrb $7, %al
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: scalar_i8_signbit_ne:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %esi, %ecx
-; X64-NEXT:    movb $-128, %al
+; X64-NEXT:    movl %edi, %eax
 ; X64-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NEXT:    shrb %cl, %al
-; X64-NEXT:    testb %dil, %al
-; X64-NEXT:    setne %al
+; X64-NEXT:    shlb %cl, %al
+; X64-NEXT:    shrb $7, %al
+; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %t0 = lshr i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp ne i8 %t1, 0 ;  we are perfectly happy with 'ne' predicate
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;

test/CodeGen/X86/hoist-and-by-const-from-shl-in-eqcmp-zero.ll

 ;------------------------------------------------------------------------------;
 
 ; i8 scalar
 
 define i1 @scalar_i8_signbit_eq(i8 %x, i8 %y) nounwind {
 ; X86-LABEL: scalar_i8_signbit_eq:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NEXT:    movb $-128, %al
-; X86-NEXT:    shlb %cl, %al
-; X86-NEXT:    testb %al, {{[0-9]+}}(%esp)
+; X86-NEXT:    movb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    shrb %cl, %al
+; X86-NEXT:    testb $-128, %al
 ; X86-NEXT:    sete %al
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: scalar_i8_signbit_eq:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %esi, %ecx
-; X64-NEXT:    movb $-128, %al
 ; X64-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NEXT:    shlb %cl, %al
-; X64-NEXT:    testb %dil, %al
+; X64-NEXT:    shrb %cl, %dil
+; X64-NEXT:    testb $-128, %dil
 ; X64-NEXT:    sete %al
 ; X64-NEXT:    retq
   %t0 = shl i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
; X64-NEXT:    retq
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i8_bitsinmiddle_eq(i8 %x, i8 %y) nounwind {
 ; X86-LABEL: scalar_i8_bitsinmiddle_eq:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NEXT:    movb $24, %al
-; X86-NEXT:    shlb %cl, %al
-; X86-NEXT:    testb %al, {{[0-9]+}}(%esp)
+; X86-NEXT:    movb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    shrb %cl, %al
+; X86-NEXT:    testb $24, %al
 ; X86-NEXT:    sete %al
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: scalar_i8_bitsinmiddle_eq:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %esi, %ecx
-; X64-NEXT:    movb $24, %al
 ; X64-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NEXT:    shlb %cl, %al
-; X64-NEXT:    testb %dil, %al
+; X64-NEXT:    shrb %cl, %dil
+; X64-NEXT:    testb $24, %dil
 ; X64-NEXT:    sete %al
 ; X64-NEXT:    retq
   %t0 = shl i8 24, %y
   %t1 = and i8 %t0, %x
   %res = icmp eq i8 %t1, 0
   ret i1 %res
 }
 
 ; i16 scalar
 
 define i1 @scalar_i16_signbit_eq(i16 %x, i16 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i16_signbit_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $-32768, %eax # imm = 0x8000
-; X86-NOBMI-NEXT:    shll %cl, %eax
-; X86-NOBMI-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-NOBMI-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shrl %cl, %eax
+; X86-NOBMI-NEXT:    testl $32768, %eax # imm = 0x8000
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i16_signbit_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $-32768, %eax # imm = 0x8000
-; X86-BMI1-NEXT:    shll %cl, %eax
-; X86-BMI1-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-BMI1-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shrl %cl, %eax
+; X86-BMI1-NEXT:    testl $32768, %eax # imm = 0x8000
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i16_signbit_eq:
 ; X86-BMI12:       # %bb.0:
-; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X86-BMI12-NEXT:    movl $-32768, %ecx # imm = 0x8000
-; X86-BMI12-NEXT:    shlxl %eax, %ecx, %eax
-; X86-BMI12-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-BMI12-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %cl
+; X86-BMI12-NEXT:    shrxl %ecx, %eax, %eax
+; X86-BMI12-NEXT:    testl $32768, %eax # imm = 0x8000
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i16_signbit_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movl %esi, %ecx
-; X64-NOBMI-NEXT:    movl $-32768, %eax # imm = 0x8000
+; X64-NOBMI-NEXT:    movzwl %di, %eax
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NOBMI-NEXT:    shll %cl, %eax
-; X64-NOBMI-NEXT:    testw %di, %ax
+; X64-NOBMI-NEXT:    shrl %cl, %eax
+; X64-NOBMI-NEXT:    testl $32768, %eax # imm = 0x8000
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i16_signbit_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movl %esi, %ecx
-; X64-BMI1-NEXT:    movl $-32768, %eax # imm = 0x8000
+; X64-BMI1-NEXT:    movzwl %di, %eax
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-BMI1-NEXT:    shll %cl, %eax
-; X64-BMI1-NEXT:    testw %di, %ax
+; X64-BMI1-NEXT:    shrl %cl, %eax
+; X64-BMI1-NEXT:    testl $32768, %eax # imm = 0x8000
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i16_signbit_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movl $-32768, %eax # imm = 0x8000
-; X64-BMI12-NEXT:    shlxl %esi, %eax, %eax
-; X64-BMI12-NEXT:    testw %di, %ax
+; X64-BMI12-NEXT:    movzwl %di, %eax
+; X64-BMI12-NEXT:    shrxl %esi, %eax, %eax
+; X64-BMI12-NEXT:    testl $32768, %eax # imm = 0x8000
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = shl i16 32768, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
; X64-NEXT:    retq
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i16_bitsinmiddle_eq(i16 %x, i16 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $4080, %eax # imm = 0xFF0
-; X86-NOBMI-NEXT:    shll %cl, %eax
-; X86-NOBMI-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-NOBMI-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shrl %cl, %eax
+; X86-NOBMI-NEXT:    testl $4080, %eax # imm = 0xFF0
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $4080, %eax # imm = 0xFF0
-; X86-BMI1-NEXT:    shll %cl, %eax
-; X86-BMI1-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-BMI1-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shrl %cl, %eax
+; X86-BMI1-NEXT:    testl $4080, %eax # imm = 0xFF0
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X86-BMI12:       # %bb.0:
-; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X86-BMI12-NEXT:    movl $4080, %ecx # imm = 0xFF0
-; X86-BMI12-NEXT:    shlxl %eax, %ecx, %eax
-; X86-BMI12-NEXT:    testw %ax, {{[0-9]+}}(%esp)
+; X86-BMI12-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %cl
+; X86-BMI12-NEXT:    shrxl %ecx, %eax, %eax
+; X86-BMI12-NEXT:    testl $4080, %eax # imm = 0xFF0
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movl %esi, %ecx
-; X64-NOBMI-NEXT:    movl $4080, %eax # imm = 0xFF0
+; X64-NOBMI-NEXT:    movzwl %di, %eax
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NOBMI-NEXT:    shll %cl, %eax
-; X64-NOBMI-NEXT:    testw %di, %ax
+; X64-NOBMI-NEXT:    shrl %cl, %eax
+; X64-NOBMI-NEXT:    testl $4080, %eax # imm = 0xFF0
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movl %esi, %ecx
-; X64-BMI1-NEXT:    movl $4080, %eax # imm = 0xFF0
+; X64-BMI1-NEXT:    movzwl %di, %eax
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-BMI1-NEXT:    shll %cl, %eax
-; X64-BMI1-NEXT:    testw %di, %ax
+; X64-BMI1-NEXT:    shrl %cl, %eax
+; X64-BMI1-NEXT:    testl $4080, %eax # imm = 0xFF0
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i16_bitsinmiddle_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movl $4080, %eax # imm = 0xFF0
-; X64-BMI12-NEXT:    shlxl %esi, %eax, %eax
-; X64-BMI12-NEXT:    testw %di, %ax
+; X64-BMI12-NEXT:    movzwl %di, %eax
+; X64-BMI12-NEXT:    shrxl %esi, %eax, %eax
+; X64-BMI12-NEXT:    testl $4080, %eax # imm = 0xFF0
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = shl i16 4080, %y
   %t1 = and i16 %t0, %x
   %res = icmp eq i16 %t1, 0
   ret i1 %res
 }
 
 ; i32 scalar
 
 define i1 @scalar_i32_signbit_eq(i32 %x, i32 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i32_signbit_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
-; X86-NOBMI-NEXT:    shll %cl, %eax
-; X86-NOBMI-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shrl %cl, %eax
+; X86-NOBMI-NEXT:    testl $-2147483648, %eax # imm = 0x80000000
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i32_signbit_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
-; X86-BMI1-NEXT:    shll %cl, %eax
-; X86-BMI1-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shrl %cl, %eax
+; X86-BMI1-NEXT:    testl $-2147483648, %eax # imm = 0x80000000
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i32_signbit_eq:
 ; X86-BMI12:       # %bb.0:
 ; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X86-BMI12-NEXT:    movl $-2147483648, %ecx # imm = 0x80000000
-; X86-BMI12-NEXT:    shlxl %eax, %ecx, %eax
-; X86-BMI12-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-BMI12-NEXT:    shrxl %eax, {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    testl $-2147483648, %eax # imm = 0x80000000
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i32_signbit_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movl %esi, %ecx
-; X64-NOBMI-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NOBMI-NEXT:    shll %cl, %eax
-; X64-NOBMI-NEXT:    testl %edi, %eax
+; X64-NOBMI-NEXT:    shrl %cl, %edi
+; X64-NOBMI-NEXT:    testl $-2147483648, %edi # imm = 0x80000000
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i32_signbit_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movl %esi, %ecx
-; X64-BMI1-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-BMI1-NEXT:    shll %cl, %eax
-; X64-BMI1-NEXT:    testl %edi, %eax
+; X64-BMI1-NEXT:    shrl %cl, %edi
+; X64-BMI1-NEXT:    testl $-2147483648, %edi # imm = 0x80000000
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i32_signbit_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movl $-2147483648, %eax # imm = 0x80000000
-; X64-BMI12-NEXT:    shlxl %esi, %eax, %eax
-; X64-BMI12-NEXT:    testl %edi, %eax
+; X64-BMI12-NEXT:    shrxl %esi, %edi, %eax
+; X64-BMI12-NEXT:    testl $-2147483648, %eax # imm = 0x80000000
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = shl i32 2147483648, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
; X64-NEXT:    retq
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i32_bitsinmiddle_eq(i32 %x, i32 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $16776960, %eax # imm = 0xFFFF00
-; X86-NOBMI-NEXT:    shll %cl, %eax
-; X86-NOBMI-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shrl %cl, %eax
+; X86-NOBMI-NEXT:    testl $16776960, %eax # imm = 0xFFFF00
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $16776960, %eax # imm = 0xFFFF00
-; X86-BMI1-NEXT:    shll %cl, %eax
-; X86-BMI1-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shrl %cl, %eax
+; X86-BMI1-NEXT:    testl $16776960, %eax # imm = 0xFFFF00
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X86-BMI12:       # %bb.0:
 ; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X86-BMI12-NEXT:    movl $16776960, %ecx # imm = 0xFFFF00
-; X86-BMI12-NEXT:    shlxl %eax, %ecx, %eax
-; X86-BMI12-NEXT:    testl %eax, {{[0-9]+}}(%esp)
+; X86-BMI12-NEXT:    shrxl %eax, {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    testl $16776960, %eax # imm = 0xFFFF00
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movl %esi, %ecx
-; X64-NOBMI-NEXT:    movl $16776960, %eax # imm = 0xFFFF00
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NOBMI-NEXT:    shll %cl, %eax
-; X64-NOBMI-NEXT:    testl %edi, %eax
+; X64-NOBMI-NEXT:    shrl %cl, %edi
+; X64-NOBMI-NEXT:    testl $16776960, %edi # imm = 0xFFFF00
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movl %esi, %ecx
-; X64-BMI1-NEXT:    movl $16776960, %eax # imm = 0xFFFF00
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-BMI1-NEXT:    shll %cl, %eax
-; X64-BMI1-NEXT:    testl %edi, %eax
+; X64-BMI1-NEXT:    shrl %cl, %edi
+; X64-BMI1-NEXT:    testl $16776960, %edi # imm = 0xFFFF00
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i32_bitsinmiddle_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movl $16776960, %eax # imm = 0xFFFF00
-; X64-BMI12-NEXT:    shlxl %esi, %eax, %eax
-; X64-BMI12-NEXT:    testl %edi, %eax
+; X64-BMI12-NEXT:    shrxl %esi, %edi, %eax
+; X64-BMI12-NEXT:    testl $16776960, %eax # imm = 0xFFFF00
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = shl i32 16776960, %y
   %t1 = and i32 %t0, %x
   %res = icmp eq i32 %t1, 0
   ret i1 %res
 }
 
 ; i64 scalar
 
 define i1 @scalar_i64_signbit_eq(i64 %x, i64 %y) nounwind {
-; X86-LABEL: scalar_i64_signbit_eq:
-; X86:       # %bb.0:
-; X86-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NEXT:    xorl %eax, %eax
-; X86-NEXT:    movl $-2147483648, %edx # imm = 0x80000000
-; X86-NEXT:    shldl %cl, %eax, %edx
-; X86-NEXT:    testb $32, %cl
-; X86-NEXT:    cmovnel %eax, %edx
-; X86-NEXT:    andl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    orl $0, %edx
-; X86-NEXT:    sete %al
-; X86-NEXT:    retl
+; X86-NOBMI-LABEL: scalar_i64_signbit_eq:
+; X86-NOBMI:       # %bb.0:
+; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    shrl %cl, %eax
+; X86-NOBMI-NEXT:    xorl %edx, %edx
+; X86-NOBMI-NEXT:    testb $32, %cl
+; X86-NOBMI-NEXT:    cmovel %eax, %edx
+; X86-NOBMI-NEXT:    testl $-2147483648, %edx # imm = 0x80000000
+; X86-NOBMI-NEXT:    sete %al
+; X86-NOBMI-NEXT:    retl
+;
+; X86-BMI1-LABEL: scalar_i64_signbit_eq:
+; X86-BMI1:       # %bb.0:
+; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    shrl %cl, %eax
+; X86-BMI1-NEXT:    xorl %edx, %edx
+; X86-BMI1-NEXT:    testb $32, %cl
+; X86-BMI1-NEXT:    cmovel %eax, %edx
+; X86-BMI1-NEXT:    testl $-2147483648, %edx # imm = 0x80000000
+; X86-BMI1-NEXT:    sete %al
+; X86-BMI1-NEXT:    retl
+;
+; X86-BMI12-LABEL: scalar_i64_signbit_eq:
+; X86-BMI12:       # %bb.0:
+; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %al
+; X86-BMI12-NEXT:    shrxl %eax, {{[0-9]+}}(%esp), %ecx
+; X86-BMI12-NEXT:    xorl %edx, %edx
+; X86-BMI12-NEXT:    testb $32, %al
+; X86-BMI12-NEXT:    cmovel %ecx, %edx
+; X86-BMI12-NEXT:    testl $-2147483648, %edx # imm = 0x80000000
+; X86-BMI12-NEXT:    sete %al
+; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i64_signbit_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movq %rsi, %rcx
-; X64-NOBMI-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $rcx
-; X64-NOBMI-NEXT:    shlq %cl, %rax
-; X64-NOBMI-NEXT:    testq %rdi, %rax
-; X64-NOBMI-NEXT:    sete %al
+; X64-NOBMI-NEXT:    shrq %cl, %rdi
+; X64-NOBMI-NEXT:    btq $63, %rdi
+; X64-NOBMI-NEXT:    setae %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i64_signbit_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movq %rsi, %rcx
-; X64-BMI1-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $rcx
-; X64-BMI1-NEXT:    shlq %cl, %rax
-; X64-BMI1-NEXT:    testq %rdi, %rax
-; X64-BMI1-NEXT:    sete %al
+; X64-BMI1-NEXT:    shrq %cl, %rdi
+; X64-BMI1-NEXT:    btq $63, %rdi
+; X64-BMI1-NEXT:    setae %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i64_signbit_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movabsq $-9223372036854775808, %rax # imm = 0x8000000000000000
-; X64-BMI12-NEXT:    shlxq %rsi, %rax, %rax
-; X64-BMI12-NEXT:    testq %rdi, %rax
-; X64-BMI12-NEXT:    sete %al
+; X64-BMI12-NEXT:    shrxq %rsi, %rdi, %rax
+; X64-BMI12-NEXT:    btq $63, %rax
+; X64-BMI12-NEXT:    setae %al
 ; X64-BMI12-NEXT:    retq
   %t0 = shl i64 9223372036854775808, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 define i1 @scalar_i64_lowestbit_eq(i64 %x, i64 %y) nounwind {
; X64-NEXT:    retq
   ret i1 %res
 }
 
 define i1 @scalar_i64_bitsinmiddle_eq(i64 %x, i64 %y) nounwind {
 ; X86-NOBMI-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X86-NOBMI:       # %bb.0:
 ; X86-NOBMI-NEXT:    pushl %esi
 ; X86-NOBMI-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NOBMI-NEXT:    movl $-65536, %eax # imm = 0xFFFF0000
-; X86-NOBMI-NEXT:    movl $65535, %edx # imm = 0xFFFF
-; X86-NOBMI-NEXT:    shldl %cl, %eax, %edx
-; X86-NOBMI-NEXT:    shll %cl, %eax
-; X86-NOBMI-NEXT:    xorl %esi, %esi
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOBMI-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NOBMI-NEXT:    movl %edx, %esi
+; X86-NOBMI-NEXT:    shrl %cl, %esi
+; X86-NOBMI-NEXT:    shrdl %cl, %edx, %eax
+; X86-NOBMI-NEXT:    xorl %edx, %edx
 ; X86-NOBMI-NEXT:    testb $32, %cl
-; X86-NOBMI-NEXT:    cmovnel %eax, %edx
-; X86-NOBMI-NEXT:    cmovel %eax, %esi
-; X86-NOBMI-NEXT:    andl {{[0-9]+}}(%esp), %edx
-; X86-NOBMI-NEXT:    andl {{[0-9]+}}(%esp), %esi
-; X86-NOBMI-NEXT:    orl %edx, %esi
+; X86-NOBMI-NEXT:    cmovnel %esi, %eax
+; X86-NOBMI-NEXT:    cmovel %esi, %edx
+; X86-NOBMI-NEXT:    andl $-65536, %eax # imm = 0xFFFF0000
+; X86-NOBMI-NEXT:    movzwl %dx, %ecx
+; X86-NOBMI-NEXT:    orl %eax, %ecx
 ; X86-NOBMI-NEXT:    sete %al
 ; X86-NOBMI-NEXT:    popl %esi
 ; X86-NOBMI-NEXT:    retl
 ;
 ; X86-BMI1-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X86-BMI1:       # %bb.0:
 ; X86-BMI1-NEXT:    pushl %esi
 ; X86-BMI1-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI1-NEXT:    movl $-65536, %eax # imm = 0xFFFF0000
-; X86-BMI1-NEXT:    movl $65535, %edx # imm = 0xFFFF
-; X86-BMI1-NEXT:    shldl %cl, %eax, %edx
-; X86-BMI1-NEXT:    shll %cl, %eax
-; X86-BMI1-NEXT:    xorl %esi, %esi
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-BMI1-NEXT:    movl %edx, %esi
+; X86-BMI1-NEXT:    shrl %cl, %esi
+; X86-BMI1-NEXT:    shrdl %cl, %edx, %eax
+; X86-BMI1-NEXT:    xorl %edx, %edx
 ; X86-BMI1-NEXT:    testb $32, %cl
-; X86-BMI1-NEXT:    cmovnel %eax, %edx
-; X86-BMI1-NEXT:    cmovel %eax, %esi
-; X86-BMI1-NEXT:    andl {{[0-9]+}}(%esp), %edx
-; X86-BMI1-NEXT:    andl {{[0-9]+}}(%esp), %esi
-; X86-BMI1-NEXT:    orl %edx, %esi
+; X86-BMI1-NEXT:    cmovnel %esi, %eax
+; X86-BMI1-NEXT:    cmovel %esi, %edx
+; X86-BMI1-NEXT:    andl $-65536, %eax # imm = 0xFFFF0000
+; X86-BMI1-NEXT:    movzwl %dx, %ecx
+; X86-BMI1-NEXT:    orl %eax, %ecx
 ; X86-BMI1-NEXT:    sete %al
 ; X86-BMI1-NEXT:    popl %esi
 ; X86-BMI1-NEXT:    retl
 ;
 ; X86-BMI12-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X86-BMI12:       # %bb.0:
 ; X86-BMI12-NEXT:    pushl %esi
 ; X86-BMI12-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-BMI12-NEXT:    movl $-65536, %eax # imm = 0xFFFF0000
-; X86-BMI12-NEXT:    movl $65535, %edx # imm = 0xFFFF
-; X86-BMI12-NEXT:    shldl %cl, %eax, %edx
-; X86-BMI12-NEXT:    shlxl %ecx, %eax, %eax
+; X86-BMI12-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI12-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-BMI12-NEXT:    shrdl %cl, %edx, %eax
+; X86-BMI12-NEXT:    shrxl %ecx, %edx, %edx
 ; X86-BMI12-NEXT:    xorl %esi, %esi
 ; X86-BMI12-NEXT:    testb $32, %cl
-; X86-BMI12-NEXT:    cmovnel %eax, %edx
-; X86-BMI12-NEXT:    cmovel %eax, %esi
-; X86-BMI12-NEXT:    andl {{[0-9]+}}(%esp), %edx
-; X86-BMI12-NEXT:    andl {{[0-9]+}}(%esp), %esi
-; X86-BMI12-NEXT:    orl %edx, %esi
+; X86-BMI12-NEXT:    cmovnel %edx, %eax
+; X86-BMI12-NEXT:    cmovel %edx, %esi
+; X86-BMI12-NEXT:    andl $-65536, %eax # imm = 0xFFFF0000
+; X86-BMI12-NEXT:    movzwl %si, %ecx
+; X86-BMI12-NEXT:    orl %eax, %ecx
 ; X86-BMI12-NEXT:    sete %al
 ; X86-BMI12-NEXT:    popl %esi
 ; X86-BMI12-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X64-NOBMI:       # %bb.0:
 ; X64-NOBMI-NEXT:    movq %rsi, %rcx
-; X64-NOBMI-NEXT:    movabsq $281474976645120, %rax # imm = 0xFFFFFFFF0000
 ; X64-NOBMI-NEXT:    # kill: def $cl killed $cl killed $rcx
-; X64-NOBMI-NEXT:    shlq %cl, %rax
-; X64-NOBMI-NEXT:    testq %rdi, %rax
+; X64-NOBMI-NEXT:    shrq %cl, %rdi
+; X64-NOBMI-NEXT:    movabsq $281474976645120, %rax # imm = 0xFFFFFFFF0000
+; X64-NOBMI-NEXT:    testq %rax, %rdi
 ; X64-NOBMI-NEXT:    sete %al
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMI1-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X64-BMI1:       # %bb.0:
 ; X64-BMI1-NEXT:    movq %rsi, %rcx
-; X64-BMI1-NEXT:    movabsq $281474976645120, %rax # imm = 0xFFFFFFFF0000
 ; X64-BMI1-NEXT:    # kill: def $cl killed $cl killed $rcx
-; X64-BMI1-NEXT:    shlq %cl, %rax
-; X64-BMI1-NEXT:    testq %rdi, %rax
+; X64-BMI1-NEXT:    shrq %cl, %rdi
+; X64-BMI1-NEXT:    movabsq $281474976645120, %rax # imm = 0xFFFFFFFF0000
+; X64-BMI1-NEXT:    testq %rax, %rdi
 ; X64-BMI1-NEXT:    sete %al
 ; X64-BMI1-NEXT:    retq
 ;
 ; X64-BMI12-LABEL: scalar_i64_bitsinmiddle_eq:
 ; X64-BMI12:       # %bb.0:
-; X64-BMI12-NEXT:    movabsq $281474976645120, %rax # imm = 0xFFFFFFFF0000
-; X64-BMI12-NEXT:    shlxq %rsi, %rax, %rax
-; X64-BMI12-NEXT:    testq %rdi, %rax
+; X64-BMI12-NEXT:    shrxq %rsi, %rdi, %rax
+; X64-BMI12-NEXT:    movabsq $281474976645120, %rcx # imm = 0xFFFFFFFF0000
+; X64-BMI12-NEXT:    testq %rcx, %rax
 ; X64-BMI12-NEXT:    sete %al
 ; X64-BMI12-NEXT:    retq
   %t0 = shl i64 281474976645120, %y
   %t1 = and i64 %t0, %x
   %res = icmp eq i64 %t1, 0
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;
 ; A few trivial vector tests
 ;------------------------------------------------------------------------------;
 
 define <4 x i1> @vec_4xi32_splat_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; X86-LABEL: vec_4xi32_splat_eq:
 ; X86:       # %bb.0:
-; X86-NEXT:    pslld $23, %xmm1
-; X86-NEXT:    paddd {{\.LCPI.*}}, %xmm1
-; X86-NEXT:    cvttps2dq %xmm1, %xmm1
-; X86-NEXT:    movdqa {{.*#+}} xmm2 = [1,1,1,1]
-; X86-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[1,1,3,3]
-; X86-NEXT:    pmuludq %xmm2, %xmm3
-; X86-NEXT:    pmuludq %xmm1, %xmm2
-; X86-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[0,2,2,3]
-; X86-NEXT:    pshufd {{.*#+}} xmm2 = xmm3[0,2,2,3]
-; X86-NEXT:    punpckldq {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1]
-; X86-NEXT:    pand %xmm1, %xmm0
-; X86-NEXT:    pxor %xmm1, %xmm1
-; X86-NEXT:    pcmpeqd %xmm1, %xmm0
+; X86-NEXT:    pxor %xmm3, %xmm3
+; X86-NEXT:    pshuflw {{.*#+}} xmm2 = xmm1[2,3,3,3,4,5,6,7]
+; X86-NEXT:    movdqa %xmm0, %xmm4
+; X86-NEXT:    psrld %xmm2, %xmm4
+; X86-NEXT:    pshuflw {{.*#+}} xmm5 = xmm1[0,1,1,1,4,5,6,7]
+; X86-NEXT:    movdqa %xmm0, %xmm2
+; X86-NEXT:    psrld %xmm5, %xmm2
+; X86-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm4[0]
+; X86-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,3,0,1]
+; X86-NEXT:    pshuflw {{.*#+}} xmm4 = xmm1[2,3,3,3,4,5,6,7]
+; X86-NEXT:    movdqa %xmm0, %xmm5
+; X86-NEXT:    psrld %xmm4, %xmm5
+; X86-NEXT:    pshuflw {{.*#+}} xmm1 = xmm1[0,1,1,1,4,5,6,7]
+; X86-NEXT:    psrld %xmm1, %xmm0
+; X86-NEXT:    punpckhqdq {{.*#+}} xmm0 = xmm0[1],xmm5[1]
+; X86-NEXT:    shufps {{.*#+}} xmm2 = xmm2[0,3],xmm0[0,3]
+; X86-NEXT:    andps {{\.LCPI.*}}, %xmm2
+; X86-NEXT:    pcmpeqd %xmm3, %xmm2
+; X86-NEXT:    movdqa %xmm2, %xmm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: vec_4xi32_splat_eq:
 ; X64:       # %bb.0:
-; X64-NEXT:    pslld $23, %xmm1
-; X64-NEXT:    paddd {{.*}}(%rip), %xmm1
-; X64-NEXT:    cvttps2dq %xmm1, %xmm1
-; X64-NEXT:    movdqa {{.*#+}} xmm2 = [1,1,1,1]
-; X64-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[1,1,3,3]
-; X64-NEXT:    pmuludq %xmm2, %xmm3
-; X64-NEXT:    pmuludq %xmm1, %xmm2
-; X64-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[0,2,2,3]
-; X64-NEXT:    pshufd {{.*#+}} xmm2 = xmm3[0,2,2,3]
-; X64-NEXT:    punpckldq {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1]
-; X64-NEXT:    pand %xmm1, %xmm0
-; X64-NEXT:    pxor %xmm1, %xmm1
-; X64-NEXT:    pcmpeqd %xmm1, %xmm0
+; X64-NEXT:    pxor %xmm3, %xmm3
+; X64-NEXT:    pshuflw {{.*#+}} xmm2 = xmm1[2,3,3,3,4,5,6,7]
+; X64-NEXT:    movdqa %xmm0, %xmm4
+; X64-NEXT:    psrld %xmm2, %xmm4
+; X64-NEXT:    pshuflw {{.*#+}} xmm5 = xmm1[0,1,1,1,4,5,6,7]
+; X64-NEXT:    movdqa %xmm0, %xmm2
+; X64-NEXT:    psrld %xmm5, %xmm2
+; X64-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm4[0]
+; X64-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,3,0,1]
+; X64-NEXT:    pshuflw {{.*#+}} xmm4 = xmm1[2,3,3,3,4,5,6,7]
+; X64-NEXT:    movdqa %xmm0, %xmm5
+; X64-NEXT:    psrld %xmm4, %xmm5
+; X64-NEXT:    pshuflw {{.*#+}} xmm1 = xmm1[0,1,1,1,4,5,6,7]
+; X64-NEXT:    psrld %xmm1, %xmm0
+; X64-NEXT:    punpckhqdq {{.*#+}} xmm0 = xmm0[1],xmm5[1]
+; X64-NEXT:    shufps {{.*#+}} xmm2 = xmm2[0,3],xmm0[0,3]
+; X64-NEXT:    andps {{.*}}(%rip), %xmm2
+; X64-NEXT:    pcmpeqd %xmm3, %xmm2
+; X64-NEXT:    movdqa %xmm2, %xmm0
 ; X64-NEXT:    retq
   %t0 = shl <4 x i32> <i32 1, i32 1, i32 1, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 
 define <4 x i1> @vec_4xi32_nonsplat_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
; X64-NEXT:    retq
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 
 define <4 x i1> @vec_4xi32_nonsplat_undef0_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; X86-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; X86:       # %bb.0:
-; X86-NEXT:    pslld $23, %xmm1
-; X86-NEXT:    paddd {{\.LCPI.*}}, %xmm1
-; X86-NEXT:    cvttps2dq %xmm1, %xmm1
-; X86-NEXT:    movdqa {{.*#+}} xmm2 = [1,1,1,1]
-; X86-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[1,1,3,3]
-; X86-NEXT:    pmuludq %xmm2, %xmm3
-; X86-NEXT:    pmuludq %xmm1, %xmm2
-; X86-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[0,2,2,3]
-; X86-NEXT:    pshufd {{.*#+}} xmm2 = xmm3[0,2,2,3]
-; X86-NEXT:    punpckldq {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1]
-; X86-NEXT:    pand %xmm1, %xmm0
-; X86-NEXT:    pxor %xmm1, %xmm1
-; X86-NEXT:    pcmpeqd %xmm1, %xmm0
+; X86-NEXT:    pxor %xmm3, %xmm3
+; X86-NEXT:    pshuflw {{.*#+}} xmm2 = xmm1[2,3,3,3,4,5,6,7]
+; X86-NEXT:    movdqa %xmm0, %xmm4
+; X86-NEXT:    psrld %xmm2, %xmm4
+; X86-NEXT:    pshuflw {{.*#+}} xmm5 = xmm1[0,1,1,1,4,5,6,7]
+; X86-NEXT:    movdqa %xmm0, %xmm2
+; X86-NEXT:    psrld %xmm5, %xmm2
+; X86-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm4[0]
+; X86-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,3,0,1]
+; X86-NEXT:    pshuflw {{.*#+}} xmm4 = xmm1[2,3,3,3,4,5,6,7]
+; X86-NEXT:    movdqa %xmm0, %xmm5
+; X86-NEXT:    psrld %xmm4, %xmm5
+; X86-NEXT:    pshuflw {{.*#+}} xmm1 = xmm1[0,1,1,1,4,5,6,7]
+; X86-NEXT:    psrld %xmm1, %xmm0
+; X86-NEXT:    punpckhqdq {{.*#+}} xmm0 = xmm0[1],xmm5[1]
+; X86-NEXT:    shufps {{.*#+}} xmm2 = xmm2[0,3],xmm0[0,3]
+; X86-NEXT:    andps {{\.LCPI.*}}, %xmm2
+; X86-NEXT:    pcmpeqd %xmm3, %xmm2
+; X86-NEXT:    movdqa %xmm2, %xmm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: vec_4xi32_nonsplat_undef0_eq:
 ; X64:       # %bb.0:
-; X64-NEXT:    pslld $23, %xmm1
-; X64-NEXT:    paddd {{.*}}(%rip), %xmm1
-; X64-NEXT:    cvttps2dq %xmm1, %xmm1
-; X64-NEXT:    movdqa {{.*#+}} xmm2 = [1,1,1,1]
-; X64-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[1,1,3,3]
-; X64-NEXT:    pmuludq %xmm2, %xmm3
-; X64-NEXT:    pmuludq %xmm1, %xmm2
-; X64-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[0,2,2,3]
-; X64-NEXT:    pshufd {{.*#+}} xmm2 = xmm3[0,2,2,3]
-; X64-NEXT:    punpckldq {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1]
-; X64-NEXT:    pand %xmm1, %xmm0
-; X64-NEXT:    pxor %xmm1, %xmm1
-; X64-NEXT:    pcmpeqd %xmm1, %xmm0
+; X64-NEXT:    pxor %xmm3, %xmm3
+; X64-NEXT:    pshuflw {{.*#+}} xmm2 = xmm1[2,3,3,3,4,5,6,7]
+; X64-NEXT:    movdqa %xmm0, %xmm4
+; X64-NEXT:    psrld %xmm2, %xmm4
+; X64-NEXT:    pshuflw {{.*#+}} xmm5 = xmm1[0,1,1,1,4,5,6,7]
+; X64-NEXT:    movdqa %xmm0, %xmm2
+; X64-NEXT:    psrld %xmm5, %xmm2
+; X64-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm4[0]
+; X64-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,3,0,1]
+; X64-NEXT:    pshuflw {{.*#+}} xmm4 = xmm1[2,3,3,3,4,5,6,7]
+; X64-NEXT:    movdqa %xmm0, %xmm5
+; X64-NEXT:    psrld %xmm4, %xmm5
+; X64-NEXT:    pshuflw {{.*#+}} xmm1 = xmm1[0,1,1,1,4,5,6,7]
+; X64-NEXT:    psrld %xmm1, %xmm0
+; X64-NEXT:    punpckhqdq {{.*#+}} xmm0 = xmm0[1],xmm5[1]
+; X64-NEXT:    shufps {{.*#+}} xmm2 = xmm2[0,3],xmm0[0,3]
+; X64-NEXT:    andps {{.*}}(%rip), %xmm2
+; X64-NEXT:    pcmpeqd %xmm3, %xmm2
+; X64-NEXT:    movdqa %xmm2, %xmm0
 ; X64-NEXT:    retq
   %t0 = shl <4 x i32> <i32 1, i32 1, i32 undef, i32 1>, %y
   %t1 = and <4 x i32> %t0, %x
   %res = icmp eq <4 x i32> %t1, <i32 0, i32 0, i32 0, i32 0>
   ret <4 x i1> %res
 }
 define <4 x i1> @vec_4xi32_nonsplat_undef1_eq(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; X86-LABEL: vec_4xi32_nonsplat_undef1_eq:

 ;------------------------------------------------------------------------------;
 ; A special tests
 ;------------------------------------------------------------------------------;
 
 define i1 @scalar_i8_signbit_ne(i8 %x, i8 %y) nounwind {
 ; X86-LABEL: scalar_i8_signbit_ne:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movb {{[0-9]+}}(%esp), %cl
-; X86-NEXT:    movb $-128, %al
-; X86-NEXT:    shlb %cl, %al
-; X86-NEXT:    testb %al, {{[0-9]+}}(%esp)
-; X86-NEXT:    setne %al
+; X86-NEXT:    movb {{[0-9]+}}(%esp), %al
+; X86-NEXT:    shrb %cl, %al
+; X86-NEXT:    shrb $7, %al
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: scalar_i8_signbit_ne:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %esi, %ecx
-; X64-NEXT:    movb $-128, %al
+; X64-NEXT:    movl %edi, %eax
 ; X64-NEXT:    # kill: def $cl killed $cl killed $ecx
-; X64-NEXT:    shlb %cl, %al
-; X64-NEXT:    testb %dil, %al
-; X64-NEXT:    setne %al
+; X64-NEXT:    shrb %cl, %al
+; X64-NEXT:    shrb $7, %al
+; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %t0 = shl i8 128, %y
   %t1 = and i8 %t0, %x
   %res = icmp ne i8 %t1, 0 ;  we are perfectly happy with 'ne' predicate
   ret i1 %res
 }
 
 ;------------------------------------------------------------------------------;