[SLP] Consider alternatives for cost of select instructions.

[fhahn]

Summary

Some architectures do not have general vector select instructions (e.g.
AArch64). But some cmp/select patterns can be vectorized using other
instructions/intrinsics.

One example is using min/max instructions for certain patterns.

This patch updates the cost calculations for selects in the SLP
vectorizer to consider using min/max intrinsics.

This patch does not change SLP vectorizer's codegen itself to actually
generate those intrinsics, but relies on the backends to lower the
vector cmps & selects. This keeps things simple on the SLP side and
works well in practice for AArch64.

This exposes additional SLP vectorization opportunities in some
benchmarks on AArch64 (-O3 -flto).

Metric: SLP.NumVectorInstructions

Program base slp diff
test-suite...ications/JM/ldecod/ldecod.test 502.00 697.00 38.8%
test-suite...ications/JM/lencod/lencod.test 1023.00 1414.00 38.2%
test-suite...-typeset/consumer-typeset.test 56.00 65.00 16.1%
test-suite...6/464.h264ref/464.h264ref.test 804.00 822.00 2.2%
test-suite...006/453.povray/453.povray.test 3335.00 3357.00 0.7%
test-suite...CFP2000/177.mesa/177.mesa.test 2110.00 2121.00 0.5%
test-suite...:: External/Povray/povray.test 2378.00 2382.00 0.2%

[xbolva00]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

3553

Maybe add TODO to remove/adjust this code when LLVM starts using u/smin/max fully?

3568

intrinsic::not_intrinsic

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

3559–3595

Use match() functions to match the pattern. Also, what's the criterion for generating intrinsic or vector instruction? Can we rely on it unconditionally and just choose the minimum cost?

[dmgreen]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

3559–3595

There's something called matchSelectPattern that might help quite a bit too.

[fhahn]

Use matchSelectPattern instead of manually looking for min/max patterns.

[fhahn]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

3559–3595

. Also, what's the criterion for generating intrinsic or vector instruction? Can we rely on it unconditionally and just choose the minimum cost?

Currently for costing, it just picks whatever has the cheaper cost (select lowering or min/max version). The code generated by the SLP vectorizer is currently unchanged and it relies on the backend to pick whichever version is more profitable.

This works well on AArch64, but there could be a case where for example SLP vectorizer decides it is only profitable with min/max, but the backend is missing a corresponding fold and won't be able to generate the right instructions.

We could add a general fold to VectorCombine as follow up, to behave more predictable, if that is desired, or adjust the SLP codegen.

I am also planing a follow-up that supports additional select patterns that can be lowered more efficiently than currently on AArch64 that are not min/max.

There's something called matchSelectPattern that might help quite a bit too.

That's useful, thanks! Updated the code to use that. Unfortunately it seems like we still need some code to turn the pattern into an intrinsic, but maybe the code should be moved to SelectPatternResult?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

3550–3567

Maybe outline all this as a separate function, something like static int getVectorMinMaxCost(...)?

3590

unexpected

[fhahn]

Outlined cost computation to getVectorMinMaxCost helper function, fix typo. Thanks!

[fhahn]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

3550–3567

Done, it was getting quite big.

[samparker]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

3378

Maybe this should be part of TTI? There's already static methods for reduction matching and could this logic not also be useful for the loop vectorizer?

[RKSimon]

Why the reluctance to actually generate the min/max intrinsics?

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

3378

Yes - it'd be great to move most of this into TTI or Utils/Local.cpp

[fhahn]

Updated to move matching code to ValueTracking where the other related match code lives.

In D89969#2349241, @RKSimon wrote:

Why the reluctance to actually generate the min/max intrinsics?

Nothing other than trying to keep the patch as small as possible. If people think it's better to generate the intrinsics directly, I can adjust SLP codegen.

[fhahn]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

3378

Sounds good. I moved the matching part to ValueTracking. It does not seem like the ideal location, but it is where the related SelectPatternResult helpers live.

could this logic not also be useful for the loop vectorizer?

yep, this is something I am planning on looking into shortly as a follow up.

[fhahn]

Remove remnants of getVectorMinMaxCost

[spatel]

I haven't looked at the details here, but I just want to chime in that I'm cautiously optimistic that we'll have the basic cost model for intrinsics straightened out soon ( example: c963bde0152a ).
That should then allow us to canonicalize to min/max/abs intrinsics in instcombine. I'm not sure if this patch makes that goal harder, easier, or neither. We will eventually want the vectorizers to recognize and produce those intrinsics directly.

[fhahn]

In D89969#2350457, @spatel wrote:

I haven't looked at the details here, but I just want to chime in that I'm cautiously optimistic that we'll have the basic cost model for intrinsics straightened out soon ( example: c963bde0152a ).
That should then allow us to canonicalize to min/max/abs intrinsics in instcombine. I'm not sure if this patch makes that goal harder, easier, or neither. We will eventually want the vectorizers to recognize and produce those intrinsics directly.

I don't think this patch would make things harder. The change itself may become less important, if some of those scalar selects are turned into min/max before SLPVectorizer sees them. We can remove it, as @xbolva00 suggested, if it turns out it is not needed any more.

[samparker]

Probably best to wait a little while for other's feedback, but this LGTM.

[RKSimon]

LGTM with one minor

llvm/lib/Analysis/ValueTracking.cpp

5999

Update the comment?

[dmajor]

This commit triggered an assertion failure on our 32-bit bots:

reduced.c:

a, b, c;
l() {
  int e = a, f = l, g, h, i, j;
  float *d = c, *k = b;
  for (;;)
    for (; g < f; g++) {
      k[h] = d[i];
      k[h - 1] = d[j];
      h += e << 1;
      i += e;
    }
}

clang -cc1 -triple i386-unknown-linux-gnu -emit-obj -target-cpu pentium-m -O1 -vectorize-loops -vectorize-slp reduced.c

llvm::Type *llvm::Type::getWithNewBitWidth(unsigned int) const: Assertion `isIntOrIntVectorTy() && "Original type expected to be a vector of integers or a scalar integer."' failed.

[fhahn]

Interesting, thanks for sharing! I reverted the patch while I investigate.

llvm/lib/Analysis/ValueTracking.cpp

5999

Thanks, I updated this in the committed version.

llvm/include/llvm/Analysis/ValueTracking.h

constexpr unsigned MaxAnalysisRecursionDepth = 6;
   /// Return the inverse minimum/maximum flavor of the specified flavor.
   /// For example, signed minimum is the inverse of signed maximum.
   SelectPatternFlavor getInverseMinMaxFlavor(SelectPatternFlavor SPF);
 
   /// Return the canonical inverse comparison predicate for the specified
   /// minimum/maximum flavor.
   CmpInst::Predicate getInverseMinMaxPred(SelectPatternFlavor SPF);
 
+  /// Check if the values in \p VL are select instructions that can be converted
+  /// to a min or max (vector) intrinsic. Returns the intrinsic ID, if such a
+  /// conversion is possible, together with a bool indicating whether all select
+  /// conditions are only used by the selects. Otherwise return
+  /// Intrinsic::not_intrinsic.
+  std::pair<Intrinsic::ID, bool>
+  canConvertToMinOrMaxIntrinsic(ArrayRef<Value *> VL);
+
   /// Return true if RHS is known to be implied true by LHS.  Return false if
   /// RHS is known to be implied false by LHS.  Otherwise, return None if no
   /// implication can be made.
   /// A & B must be i1 (boolean) values or a vector of such values. Note that
   /// the truth table for implication is the same as <=u on i1 values (but not
   /// <=s!).  The truth table for both is:
   ///    | T | F (B)
   ///  T | T | F

llvm/lib/Analysis/ValueTracking.cpp

SelectPatternFlavor llvm::getInverseMinMaxFlavor(SelectPatternFlavor SPF) {
   if (SPF == SPF_UMAX) return SPF_UMIN;
   llvm_unreachable("unhandled!");
 }
 
 CmpInst::Predicate llvm::getInverseMinMaxPred(SelectPatternFlavor SPF) {
   return getMinMaxPred(getInverseMinMaxFlavor(SPF));
 }
 
+std::pair<Intrinsic::ID, bool>
+llvm::canConvertToMinOrMaxIntrinsic(ArrayRef<Value *> VL) {
+  // Check if VL contains select instructions that can be folded into a min/max
+  // vector intrinsic and return the intrinsic if it is possible.
+  // TODO: Support floating point min/max.
+  bool AllCmpSingleUse = true;
+  SelectPatternResult SelectPattern;
+  SelectPattern.Flavor = SPF_UNKNOWN;
+  if (all_of(VL, [&SelectPattern, &AllCmpSingleUse](Value *I) {
+        Value *LHS, *RHS;
+        auto CurrentPattern = matchSelectPattern(I, LHS, RHS);
+        if (!SelectPatternResult::isMinOrMax(CurrentPattern.Flavor) ||
+            CurrentPattern.Flavor == SPF_FMINNUM ||
+            CurrentPattern.Flavor == SPF_FMAXNUM)
+          return false;
+        if (SelectPattern.Flavor != SPF_UNKNOWN &&
+            SelectPattern.Flavor != CurrentPattern.Flavor)
+          return false;
+        SelectPattern = CurrentPattern;
+        AllCmpSingleUse &=
+            match(I, m_Select(m_OneUse(m_Value()), m_Value(), m_Value()));
+        return true;
+      })) {
+    switch (SelectPattern.Flavor) {
+    case SPF_SMIN:
+      return {Intrinsic::smin, AllCmpSingleUse};
+    case SPF_UMIN:
+      return {Intrinsic::umin, AllCmpSingleUse};
+    case SPF_SMAX:
+      return {Intrinsic::smax, AllCmpSingleUse};
+    case SPF_UMAX:
+      return {Intrinsic::umax, AllCmpSingleUse};
+    default:
+      llvm_unreachable("unexpected select pattern flavor");
+    }
+  }
+  return {Intrinsic::not_intrinsic, false};
+}
+
 /// Return true if "icmp Pred LHS RHS" is always true.
 static bool isTruePredicate(CmpInst::Predicate Pred, const Value *LHS,
                             const Value *RHS, const DataLayout &DL,
                             unsigned Depth) {
   assert(!LHS->getType()->isVectorTy() && "TODO: extend to handle vectors!");
   if (ICmpInst::isTrueWhenEqual(Pred) && LHS == RHS)
     return true;
 

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

if (!CI->isNoBuiltin() && VecFunc) {
     LibCost = TTI->getCallInstrCost(nullptr, VecTy, VecTys,
                                     TTI::TCK_RecipThroughput);
   }
   return {IntrinsicCost, LibCost};
 }
 
 int BoUpSLP::getEntryCost(TreeEntry *E) {
   ArrayRef<Value*> VL = E->Scalars;
 
   Type *ScalarTy = VL[0]->getType();
   if (StoreInst *SI = dyn_cast<StoreInst>(VL[0]))
     ScalarTy = SI->getValueOperand()->getType();
   else if (CmpInst *CI = dyn_cast<CmpInst>(VL[0]))
     ScalarTy = CI->getOperand(0)->getType();
   auto *VecTy = FixedVectorType::get(ScalarTy, VL.size());
   TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput;
 
case Instruction::Select: {
       int ScalarEltCost = TTI->getCmpSelInstrCost(E->getOpcode(), ScalarTy,
                                                   Builder.getInt1Ty(),
                                                   CostKind, VL0);
       if (NeedToShuffleReuses) {
         ReuseShuffleCost -= (ReuseShuffleNumbers - VL.size()) * ScalarEltCost;
       }
       auto *MaskTy = FixedVectorType::get(Builder.getInt1Ty(), VL.size());
       int ScalarCost = VecTy->getNumElements() * ScalarEltCost;
       int VecCost = TTI->getCmpSelInstrCost(E->getOpcode(), VecTy, MaskTy,
                                             CostKind, VL0);
+      // Check if it is possible and profitable to use min/max for selects in
+      // VL.
+      //
+      auto IntrinsicAndUse = canConvertToMinOrMaxIntrinsic(VL);
+      if (IntrinsicAndUse.first != Intrinsic::not_intrinsic) {
+        IntrinsicCostAttributes CostAttrs(IntrinsicAndUse.first, VecTy,
+                                          {VecTy, VecTy});
+        int IntrinsicCost = TTI->getIntrinsicInstrCost(CostAttrs, CostKind);
+        // If the selects are the only uses of the compares, they will be dead
+        // and we can adjust the cost by removing their cost.
+        if (IntrinsicAndUse.second)
+          IntrinsicCost -= TTI->getCmpSelInstrCost(Instruction::ICmp, VecTy,
+                                                   MaskTy, CostKind);
+        VecCost = std::min(VecCost, IntrinsicCost);
+      }
       return ReuseShuffleCost + VecCost - ScalarCost;
     }
     case Instruction::FNeg:
     case Instruction::Add:
     case Instruction::FAdd:
     case Instruction::Sub:
     case Instruction::FSub:
     case Instruction::Mul:
     case Instruction::FMul:
     case Instruction::UDiv:
     case Instruction::SDiv:
     case Instruction::FDiv:
     case Instruction::URem:
     case Instruction::SRem:
     case Instruction::FRem:
     case Instruction::Shl:
     case Instruction::LShr:
     case Instruction::AShr:
     case Instruction::And:
     case Instruction::Or:
     case Instruction::Xor: {
       // Certain instructions can be cheaper to vectorize if they have a
       // constant second vector operand.
       TargetTransformInfo::OperandValueKind Op1VK =
           TargetTransformInfo::OK_AnyValue;
       TargetTransformInfo::OperandValueKind Op2VK =
           TargetTransformInfo::OK_UniformConstantValue;
       TargetTransformInfo::OperandValueProperties Op1VP =
           TargetTransformInfo::OP_None;
       TargetTransformInfo::OperandValueProperties Op2VP =
           TargetTransformInfo::OP_PowerOf2;
 
       // If all operands are exactly the same ConstantInt then set the
       // operand kind to OK_UniformConstantValue.
       // If instead not all operands are constants, then set the operand kind
       // to OK_AnyValue. If all operands are constants but not the same,
       // then set the operand kind to OK_NonUniformConstantValue.

llvm/test/Transforms/SLPVectorizer/AArch64/vectorizable-selects-min-max.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -slp-vectorizer -S %s | FileCheck %s
-; RUN: opt -passes='slp-vectorizer' -S %s | FileCheck %s
+; RUN: opt -aa-pipeline=basic-aa -passes='slp-vectorizer' -S %s | FileCheck %s
 
 target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
 target triple = "arm64-apple-ios5.0.0"
 
 define void @select_umin_8xi16(i16* %ptr, i16 %x) {
 ; CHECK-LABEL: @select_umin_8xi16(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i16, i16* [[PTR:%.*]], align 2
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp ult i16 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i16 [[L_0]], i16 16383
-; CHECK-NEXT:    store i16 [[S_0]], i16* [[PTR]], align 2
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i16, i16* [[GEP_1]], align 2
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i16 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i16 [[L_1]], i16 16383
-; CHECK-NEXT:    store i16 [[S_1]], i16* [[GEP_1]], align 2
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR:%.*]], i16 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i16, i16* [[GEP_2]], align 2
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ult i16 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i16 [[L_2]], i16 16383
-; CHECK-NEXT:    store i16 [[S_2]], i16* [[GEP_2]], align 2
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i16, i16* [[GEP_3]], align 2
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp ult i16 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i16 [[L_3]], i16 16383
-; CHECK-NEXT:    store i16 [[S_3]], i16* [[GEP_3]], align 2
 ; CHECK-NEXT:    [[GEP_4:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 4
-; CHECK-NEXT:    [[L_4:%.*]] = load i16, i16* [[GEP_4]], align 2
-; CHECK-NEXT:    [[CMP_4:%.*]] = icmp ult i16 [[L_4]], 16383
-; CHECK-NEXT:    [[S_4:%.*]] = select i1 [[CMP_4]], i16 [[L_4]], i16 16383
-; CHECK-NEXT:    store i16 [[S_4]], i16* [[GEP_4]], align 2
 ; CHECK-NEXT:    [[GEP_5:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 5
-; CHECK-NEXT:    [[L_5:%.*]] = load i16, i16* [[GEP_5]], align 2
-; CHECK-NEXT:    [[CMP_5:%.*]] = icmp ult i16 [[L_5]], 16383
-; CHECK-NEXT:    [[S_5:%.*]] = select i1 [[CMP_5]], i16 [[L_5]], i16 16383
-; CHECK-NEXT:    store i16 [[S_5]], i16* [[GEP_5]], align 2
 ; CHECK-NEXT:    [[GEP_6:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 6
-; CHECK-NEXT:    [[L_6:%.*]] = load i16, i16* [[GEP_6]], align 2
-; CHECK-NEXT:    [[CMP_6:%.*]] = icmp ult i16 [[L_6]], 16383
-; CHECK-NEXT:    [[S_6:%.*]] = select i1 [[CMP_6]], i16 [[L_6]], i16 16383
-; CHECK-NEXT:    store i16 [[S_6]], i16* [[GEP_6]], align 2
 ; CHECK-NEXT:    [[GEP_7:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 7
-; CHECK-NEXT:    [[L_7:%.*]] = load i16, i16* [[GEP_7]], align 2
-; CHECK-NEXT:    [[CMP_7:%.*]] = icmp ult i16 [[L_7]], 16383
-; CHECK-NEXT:    [[S_7:%.*]] = select i1 [[CMP_7]], i16 [[L_7]], i16 16383
-; CHECK-NEXT:    store i16 [[S_7]], i16* [[GEP_7]], align 2
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i16>, <8 x i16>* [[TMP0]], align 2
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ult <8 x i16> [[TMP1]], <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <8 x i1> [[TMP2]], <8 x i16> [[TMP1]], <8 x i16> <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    store <8 x i16> [[TMP3]], <8 x i16>* [[TMP4]], align 2
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i16, i16* %ptr
   %cmp.0 = icmp ult i16 %l.0, 16383
   %s.0 = select i1 %cmp.0, i16 %l.0, i16 16383
   store i16 %s.0, i16* %ptr, align 2
 
entry:
   %s.7 = select i1 %cmp.7, i16 %l.7, i16 16383
   store i16 %s.7, i16* %gep.7, align 2
   ret void
 }
 
 define void @select_umin_4xi32(i32* %ptr, i32 %x) {
 ; CHECK-LABEL: @select_umin_4xi32(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i32, i32* [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp ult i32 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i32 [[L_0]], i32 16383
-; CHECK-NEXT:    store i32 [[S_0]], i32* [[PTR]], align 4
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i32, i32* [[GEP_1]], align 4
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ult i32 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i32 [[L_1]], i32 16383
-; CHECK-NEXT:    store i32 [[S_1]], i32* [[GEP_1]], align 4
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR:%.*]], i32 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i32, i32* [[GEP_2]], align 4
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ult i32 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i32 [[L_2]], i32 16383
-; CHECK-NEXT:    store i32 [[S_2]], i32* [[GEP_2]], align 4
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i32, i32* [[GEP_3]], align 4
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp ult i32 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i32 [[L_3]], i32 16383
-; CHECK-NEXT:    store i32 [[S_3]], i32* [[GEP_3]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ult <4 x i32> [[TMP1]], <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <4 x i1> [[TMP2]], <4 x i32> [[TMP1]], <4 x i32> <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP3]], <4 x i32>* [[TMP4]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i32, i32* %ptr
   %cmp.0 = icmp ult i32 %l.0, 16383
   %s.0 = select i1 %cmp.0, i32 %l.0, i32 16383
   store i32 %s.0, i32* %ptr, align 4
 
entry:
 
   ret void
 }
 
 
 define void @select_umin_ule_8xi16(i16* %ptr, i16 %x) {
 ; CHECK-LABEL: @select_umin_ule_8xi16(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i16, i16* [[PTR:%.*]], align 2
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp ule i16 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i16 [[L_0]], i16 16383
-; CHECK-NEXT:    store i16 [[S_0]], i16* [[PTR]], align 2
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i16, i16* [[GEP_1]], align 2
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ule i16 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i16 [[L_1]], i16 16383
-; CHECK-NEXT:    store i16 [[S_1]], i16* [[GEP_1]], align 2
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR:%.*]], i16 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i16, i16* [[GEP_2]], align 2
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ule i16 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i16 [[L_2]], i16 16383
-; CHECK-NEXT:    store i16 [[S_2]], i16* [[GEP_2]], align 2
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i16, i16* [[GEP_3]], align 2
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp ule i16 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i16 [[L_3]], i16 16383
-; CHECK-NEXT:    store i16 [[S_3]], i16* [[GEP_3]], align 2
 ; CHECK-NEXT:    [[GEP_4:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 4
-; CHECK-NEXT:    [[L_4:%.*]] = load i16, i16* [[GEP_4]], align 2
-; CHECK-NEXT:    [[CMP_4:%.*]] = icmp ule i16 [[L_4]], 16383
-; CHECK-NEXT:    [[S_4:%.*]] = select i1 [[CMP_4]], i16 [[L_4]], i16 16383
-; CHECK-NEXT:    store i16 [[S_4]], i16* [[GEP_4]], align 2
 ; CHECK-NEXT:    [[GEP_5:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 5
-; CHECK-NEXT:    [[L_5:%.*]] = load i16, i16* [[GEP_5]], align 2
-; CHECK-NEXT:    [[CMP_5:%.*]] = icmp ule i16 [[L_5]], 16383
-; CHECK-NEXT:    [[S_5:%.*]] = select i1 [[CMP_5]], i16 [[L_5]], i16 16383
-; CHECK-NEXT:    store i16 [[S_5]], i16* [[GEP_5]], align 2
 ; CHECK-NEXT:    [[GEP_6:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 6
-; CHECK-NEXT:    [[L_6:%.*]] = load i16, i16* [[GEP_6]], align 2
-; CHECK-NEXT:    [[CMP_6:%.*]] = icmp ule i16 [[L_6]], 16383
-; CHECK-NEXT:    [[S_6:%.*]] = select i1 [[CMP_6]], i16 [[L_6]], i16 16383
-; CHECK-NEXT:    store i16 [[S_6]], i16* [[GEP_6]], align 2
 ; CHECK-NEXT:    [[GEP_7:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 7
-; CHECK-NEXT:    [[L_7:%.*]] = load i16, i16* [[GEP_7]], align 2
-; CHECK-NEXT:    [[CMP_7:%.*]] = icmp ule i16 [[L_7]], 16383
-; CHECK-NEXT:    [[S_7:%.*]] = select i1 [[CMP_7]], i16 [[L_7]], i16 16383
-; CHECK-NEXT:    store i16 [[S_7]], i16* [[GEP_7]], align 2
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i16>, <8 x i16>* [[TMP0]], align 2
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ule <8 x i16> [[TMP1]], <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <8 x i1> [[TMP2]], <8 x i16> [[TMP1]], <8 x i16> <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    store <8 x i16> [[TMP3]], <8 x i16>* [[TMP4]], align 2
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i16, i16* %ptr
   %cmp.0 = icmp ule i16 %l.0, 16383
   %s.0 = select i1 %cmp.0, i16 %l.0, i16 16383
   store i16 %s.0, i16* %ptr, align 2
 
entry:
   %s.7 = select i1 %cmp.7, i16 %l.7, i16 16383
   store i16 %s.7, i16* %gep.7, align 2
   ret void
 }
 
 define void @select_umin_ule_4xi32(i32* %ptr, i32 %x) {
 ; CHECK-LABEL: @select_umin_ule_4xi32(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i32, i32* [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp ule i32 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i32 [[L_0]], i32 16383
-; CHECK-NEXT:    store i32 [[S_0]], i32* [[PTR]], align 4
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i32, i32* [[GEP_1]], align 4
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ule i32 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i32 [[L_1]], i32 16383
-; CHECK-NEXT:    store i32 [[S_1]], i32* [[GEP_1]], align 4
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR:%.*]], i32 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i32, i32* [[GEP_2]], align 4
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ule i32 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i32 [[L_2]], i32 16383
-; CHECK-NEXT:    store i32 [[S_2]], i32* [[GEP_2]], align 4
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i32, i32* [[GEP_3]], align 4
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp ule i32 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i32 [[L_3]], i32 16383
-; CHECK-NEXT:    store i32 [[S_3]], i32* [[GEP_3]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ule <4 x i32> [[TMP1]], <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <4 x i1> [[TMP2]], <4 x i32> [[TMP1]], <4 x i32> <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP3]], <4 x i32>* [[TMP4]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i32, i32* %ptr
   %cmp.0 = icmp ule i32 %l.0, 16383
   %s.0 = select i1 %cmp.0, i32 %l.0, i32 16383
   store i32 %s.0, i32* %ptr, align 4
 
entry:
   store i64 %s.1, i64* %gep.1, align 4
 
   ret void
 }
 
 define void @select_smin_8xi16(i16* %ptr, i16 %x) {
 ; CHECK-LABEL: @select_smin_8xi16(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i16, i16* [[PTR:%.*]], align 2
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp slt i16 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i16 [[L_0]], i16 16383
-; CHECK-NEXT:    store i16 [[S_0]], i16* [[PTR]], align 2
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i16, i16* [[GEP_1]], align 2
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp slt i16 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i16 [[L_1]], i16 16383
-; CHECK-NEXT:    store i16 [[S_1]], i16* [[GEP_1]], align 2
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR:%.*]], i16 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i16, i16* [[GEP_2]], align 2
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp slt i16 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i16 [[L_2]], i16 16383
-; CHECK-NEXT:    store i16 [[S_2]], i16* [[GEP_2]], align 2
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i16, i16* [[GEP_3]], align 2
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp slt i16 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i16 [[L_3]], i16 16383
-; CHECK-NEXT:    store i16 [[S_3]], i16* [[GEP_3]], align 2
 ; CHECK-NEXT:    [[GEP_4:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 4
-; CHECK-NEXT:    [[L_4:%.*]] = load i16, i16* [[GEP_4]], align 2
-; CHECK-NEXT:    [[CMP_4:%.*]] = icmp slt i16 [[L_4]], 16383
-; CHECK-NEXT:    [[S_4:%.*]] = select i1 [[CMP_4]], i16 [[L_4]], i16 16383
-; CHECK-NEXT:    store i16 [[S_4]], i16* [[GEP_4]], align 2
 ; CHECK-NEXT:    [[GEP_5:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 5
-; CHECK-NEXT:    [[L_5:%.*]] = load i16, i16* [[GEP_5]], align 2
-; CHECK-NEXT:    [[CMP_5:%.*]] = icmp slt i16 [[L_5]], 16383
-; CHECK-NEXT:    [[S_5:%.*]] = select i1 [[CMP_5]], i16 [[L_5]], i16 16383
-; CHECK-NEXT:    store i16 [[S_5]], i16* [[GEP_5]], align 2
 ; CHECK-NEXT:    [[GEP_6:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 6
-; CHECK-NEXT:    [[L_6:%.*]] = load i16, i16* [[GEP_6]], align 2
-; CHECK-NEXT:    [[CMP_6:%.*]] = icmp slt i16 [[L_6]], 16383
-; CHECK-NEXT:    [[S_6:%.*]] = select i1 [[CMP_6]], i16 [[L_6]], i16 16383
-; CHECK-NEXT:    store i16 [[S_6]], i16* [[GEP_6]], align 2
 ; CHECK-NEXT:    [[GEP_7:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 7
-; CHECK-NEXT:    [[L_7:%.*]] = load i16, i16* [[GEP_7]], align 2
-; CHECK-NEXT:    [[CMP_7:%.*]] = icmp slt i16 [[L_7]], 16383
-; CHECK-NEXT:    [[S_7:%.*]] = select i1 [[CMP_7]], i16 [[L_7]], i16 16383
-; CHECK-NEXT:    store i16 [[S_7]], i16* [[GEP_7]], align 2
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i16>, <8 x i16>* [[TMP0]], align 2
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp slt <8 x i16> [[TMP1]], <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <8 x i1> [[TMP2]], <8 x i16> [[TMP1]], <8 x i16> <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    store <8 x i16> [[TMP3]], <8 x i16>* [[TMP4]], align 2
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i16, i16* %ptr
   %cmp.0 = icmp slt i16 %l.0, 16383
   %s.0 = select i1 %cmp.0, i16 %l.0, i16 16383
   store i16 %s.0, i16* %ptr, align 2
 
entry:
   %s.7 = select i1 %cmp.7, i16 %l.7, i16 16383
   store i16 %s.7, i16* %gep.7, align 2
   ret void
 }
 
 define void @select_smin_4xi32(i32* %ptr, i32 %x) {
 ; CHECK-LABEL: @select_smin_4xi32(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i32, i32* [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp slt i32 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i32 [[L_0]], i32 16383
-; CHECK-NEXT:    store i32 [[S_0]], i32* [[PTR]], align 4
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i32, i32* [[GEP_1]], align 4
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp slt i32 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i32 [[L_1]], i32 16383
-; CHECK-NEXT:    store i32 [[S_1]], i32* [[GEP_1]], align 4
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR:%.*]], i32 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i32, i32* [[GEP_2]], align 4
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp slt i32 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i32 [[L_2]], i32 16383
-; CHECK-NEXT:    store i32 [[S_2]], i32* [[GEP_2]], align 4
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i32, i32* [[GEP_3]], align 4
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp slt i32 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i32 [[L_3]], i32 16383
-; CHECK-NEXT:    store i32 [[S_3]], i32* [[GEP_3]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp slt <4 x i32> [[TMP1]], <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <4 x i1> [[TMP2]], <4 x i32> [[TMP1]], <4 x i32> <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP3]], <4 x i32>* [[TMP4]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i32, i32* %ptr
   %cmp.0 = icmp slt i32 %l.0, 16383
   %s.0 = select i1 %cmp.0, i32 %l.0, i32 16383
   store i32 %s.0, i32* %ptr, align 4
 
entry:
   store i64 %s.1, i64* %gep.1, align 4
 
   ret void
 }
 
 define void @select_smin_sle_8xi16(i16* %ptr, i16 %x) {
 ; CHECK-LABEL: @select_smin_sle_8xi16(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i16, i16* [[PTR:%.*]], align 2
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp sle i16 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i16 [[L_0]], i16 16383
-; CHECK-NEXT:    store i16 [[S_0]], i16* [[PTR]], align 2
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i16, i16* [[GEP_1]], align 2
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sle i16 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i16 [[L_1]], i16 16383
-; CHECK-NEXT:    store i16 [[S_1]], i16* [[GEP_1]], align 2
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR:%.*]], i16 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i16, i16* [[GEP_2]], align 2
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sle i16 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i16 [[L_2]], i16 16383
-; CHECK-NEXT:    store i16 [[S_2]], i16* [[GEP_2]], align 2
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i16, i16* [[GEP_3]], align 2
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp sle i16 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i16 [[L_3]], i16 16383
-; CHECK-NEXT:    store i16 [[S_3]], i16* [[GEP_3]], align 2
 ; CHECK-NEXT:    [[GEP_4:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 4
-; CHECK-NEXT:    [[L_4:%.*]] = load i16, i16* [[GEP_4]], align 2
-; CHECK-NEXT:    [[CMP_4:%.*]] = icmp sle i16 [[L_4]], 16383
-; CHECK-NEXT:    [[S_4:%.*]] = select i1 [[CMP_4]], i16 [[L_4]], i16 16383
-; CHECK-NEXT:    store i16 [[S_4]], i16* [[GEP_4]], align 2
 ; CHECK-NEXT:    [[GEP_5:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 5
-; CHECK-NEXT:    [[L_5:%.*]] = load i16, i16* [[GEP_5]], align 2
-; CHECK-NEXT:    [[CMP_5:%.*]] = icmp sle i16 [[L_5]], 16383
-; CHECK-NEXT:    [[S_5:%.*]] = select i1 [[CMP_5]], i16 [[L_5]], i16 16383
-; CHECK-NEXT:    store i16 [[S_5]], i16* [[GEP_5]], align 2
 ; CHECK-NEXT:    [[GEP_6:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 6
-; CHECK-NEXT:    [[L_6:%.*]] = load i16, i16* [[GEP_6]], align 2
-; CHECK-NEXT:    [[CMP_6:%.*]] = icmp sle i16 [[L_6]], 16383
-; CHECK-NEXT:    [[S_6:%.*]] = select i1 [[CMP_6]], i16 [[L_6]], i16 16383
-; CHECK-NEXT:    store i16 [[S_6]], i16* [[GEP_6]], align 2
 ; CHECK-NEXT:    [[GEP_7:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 7
-; CHECK-NEXT:    [[L_7:%.*]] = load i16, i16* [[GEP_7]], align 2
-; CHECK-NEXT:    [[CMP_7:%.*]] = icmp sle i16 [[L_7]], 16383
-; CHECK-NEXT:    [[S_7:%.*]] = select i1 [[CMP_7]], i16 [[L_7]], i16 16383
-; CHECK-NEXT:    store i16 [[S_7]], i16* [[GEP_7]], align 2
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i16>, <8 x i16>* [[TMP0]], align 2
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp sle <8 x i16> [[TMP1]], <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <8 x i1> [[TMP2]], <8 x i16> [[TMP1]], <8 x i16> <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    store <8 x i16> [[TMP3]], <8 x i16>* [[TMP4]], align 2
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i16, i16* %ptr
   %cmp.0 = icmp sle i16 %l.0, 16383
   %s.0 = select i1 %cmp.0, i16 %l.0, i16 16383
   store i16 %s.0, i16* %ptr, align 2
 
entry:
   %s.7 = select i1 %cmp.7, i16 %l.7, i16 16383
   store i16 %s.7, i16* %gep.7, align 2
   ret void
 }
 
 define void @select_smin_sle_4xi32(i32* %ptr, i32 %x) {
 ; CHECK-LABEL: @select_smin_sle_4xi32(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i32, i32* [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp sle i32 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i32 [[L_0]], i32 16383
-; CHECK-NEXT:    store i32 [[S_0]], i32* [[PTR]], align 4
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i32, i32* [[GEP_1]], align 4
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sle i32 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i32 [[L_1]], i32 16383
-; CHECK-NEXT:    store i32 [[S_1]], i32* [[GEP_1]], align 4
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR:%.*]], i32 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i32, i32* [[GEP_2]], align 4
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sle i32 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i32 [[L_2]], i32 16383
-; CHECK-NEXT:    store i32 [[S_2]], i32* [[GEP_2]], align 4
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i32, i32* [[GEP_3]], align 4
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp sle i32 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i32 [[L_3]], i32 16383
-; CHECK-NEXT:    store i32 [[S_3]], i32* [[GEP_3]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp sle <4 x i32> [[TMP1]], <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <4 x i1> [[TMP2]], <4 x i32> [[TMP1]], <4 x i32> <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP3]], <4 x i32>* [[TMP4]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i32, i32* %ptr
   %cmp.0 = icmp sle i32 %l.0, 16383
   %s.0 = select i1 %cmp.0, i32 %l.0, i32 16383
   store i32 %s.0, i32* %ptr, align 4
 
entry:
   %s.1 = select i1 %cmp.1, i64 %l.1, i64 16383
   store i64 %s.1, i64* %gep.1, align 4
 
   ret void
 }
 define void @select_umax_8xi16(i16* %ptr, i16 %x) {
 ; CHECK-LABEL: @select_umax_8xi16(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i16, i16* [[PTR:%.*]], align 2
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp ugt i16 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i16 [[L_0]], i16 16383
-; CHECK-NEXT:    store i16 [[S_0]], i16* [[PTR]], align 2
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i16, i16* [[GEP_1]], align 2
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ugt i16 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i16 [[L_1]], i16 16383
-; CHECK-NEXT:    store i16 [[S_1]], i16* [[GEP_1]], align 2
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR:%.*]], i16 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i16, i16* [[GEP_2]], align 2
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ugt i16 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i16 [[L_2]], i16 16383
-; CHECK-NEXT:    store i16 [[S_2]], i16* [[GEP_2]], align 2
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i16, i16* [[GEP_3]], align 2
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp ugt i16 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i16 [[L_3]], i16 16383
-; CHECK-NEXT:    store i16 [[S_3]], i16* [[GEP_3]], align 2
 ; CHECK-NEXT:    [[GEP_4:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 4
-; CHECK-NEXT:    [[L_4:%.*]] = load i16, i16* [[GEP_4]], align 2
-; CHECK-NEXT:    [[CMP_4:%.*]] = icmp ugt i16 [[L_4]], 16383
-; CHECK-NEXT:    [[S_4:%.*]] = select i1 [[CMP_4]], i16 [[L_4]], i16 16383
-; CHECK-NEXT:    store i16 [[S_4]], i16* [[GEP_4]], align 2
 ; CHECK-NEXT:    [[GEP_5:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 5
-; CHECK-NEXT:    [[L_5:%.*]] = load i16, i16* [[GEP_5]], align 2
-; CHECK-NEXT:    [[CMP_5:%.*]] = icmp ugt i16 [[L_5]], 16383
-; CHECK-NEXT:    [[S_5:%.*]] = select i1 [[CMP_5]], i16 [[L_5]], i16 16383
-; CHECK-NEXT:    store i16 [[S_5]], i16* [[GEP_5]], align 2
 ; CHECK-NEXT:    [[GEP_6:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 6
-; CHECK-NEXT:    [[L_6:%.*]] = load i16, i16* [[GEP_6]], align 2
-; CHECK-NEXT:    [[CMP_6:%.*]] = icmp ugt i16 [[L_6]], 16383
-; CHECK-NEXT:    [[S_6:%.*]] = select i1 [[CMP_6]], i16 [[L_6]], i16 16383
-; CHECK-NEXT:    store i16 [[S_6]], i16* [[GEP_6]], align 2
 ; CHECK-NEXT:    [[GEP_7:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 7
-; CHECK-NEXT:    [[L_7:%.*]] = load i16, i16* [[GEP_7]], align 2
-; CHECK-NEXT:    [[CMP_7:%.*]] = icmp ugt i16 [[L_7]], 16383
-; CHECK-NEXT:    [[S_7:%.*]] = select i1 [[CMP_7]], i16 [[L_7]], i16 16383
-; CHECK-NEXT:    store i16 [[S_7]], i16* [[GEP_7]], align 2
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i16>, <8 x i16>* [[TMP0]], align 2
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ugt <8 x i16> [[TMP1]], <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <8 x i1> [[TMP2]], <8 x i16> [[TMP1]], <8 x i16> <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    store <8 x i16> [[TMP3]], <8 x i16>* [[TMP4]], align 2
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i16, i16* %ptr
   %cmp.0 = icmp ugt i16 %l.0, 16383
   %s.0 = select i1 %cmp.0, i16 %l.0, i16 16383
   store i16 %s.0, i16* %ptr, align 2
 
entry:
   %s.7 = select i1 %cmp.7, i16 %l.7, i16 16383
   store i16 %s.7, i16* %gep.7, align 2
   ret void
 }
 
 define void @select_umax_4xi32(i32* %ptr, i32 %x) {
 ; CHECK-LABEL: @select_umax_4xi32(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i32, i32* [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp ugt i32 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i32 [[L_0]], i32 16383
-; CHECK-NEXT:    store i32 [[S_0]], i32* [[PTR]], align 4
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i32, i32* [[GEP_1]], align 4
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ugt i32 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i32 [[L_1]], i32 16383
-; CHECK-NEXT:    store i32 [[S_1]], i32* [[GEP_1]], align 4
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR:%.*]], i32 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i32, i32* [[GEP_2]], align 4
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ugt i32 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i32 [[L_2]], i32 16383
-; CHECK-NEXT:    store i32 [[S_2]], i32* [[GEP_2]], align 4
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i32, i32* [[GEP_3]], align 4
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp ugt i32 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i32 [[L_3]], i32 16383
-; CHECK-NEXT:    store i32 [[S_3]], i32* [[GEP_3]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp ugt <4 x i32> [[TMP1]], <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <4 x i1> [[TMP2]], <4 x i32> [[TMP1]], <4 x i32> <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP3]], <4 x i32>* [[TMP4]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i32, i32* %ptr
   %cmp.0 = icmp ugt i32 %l.0, 16383
   %s.0 = select i1 %cmp.0, i32 %l.0, i32 16383
   store i32 %s.0, i32* %ptr, align 4
 
entry:
   store i64 %s.1, i64* %gep.1, align 4
 
   ret void
 }
 
 define void @select_umax_uge_8xi16(i16* %ptr, i16 %x) {
 ; CHECK-LABEL: @select_umax_uge_8xi16(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i16, i16* [[PTR:%.*]], align 2
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp uge i16 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i16 [[L_0]], i16 16383
-; CHECK-NEXT:    store i16 [[S_0]], i16* [[PTR]], align 2
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i16, i16* [[GEP_1]], align 2
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp uge i16 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i16 [[L_1]], i16 16383
-; CHECK-NEXT:    store i16 [[S_1]], i16* [[GEP_1]], align 2
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR:%.*]], i16 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i16, i16* [[GEP_2]], align 2
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp uge i16 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i16 [[L_2]], i16 16383
-; CHECK-NEXT:    store i16 [[S_2]], i16* [[GEP_2]], align 2
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i16, i16* [[GEP_3]], align 2
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp uge i16 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i16 [[L_3]], i16 16383
-; CHECK-NEXT:    store i16 [[S_3]], i16* [[GEP_3]], align 2
 ; CHECK-NEXT:    [[GEP_4:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 4
-; CHECK-NEXT:    [[L_4:%.*]] = load i16, i16* [[GEP_4]], align 2
-; CHECK-NEXT:    [[CMP_4:%.*]] = icmp uge i16 [[L_4]], 16383
-; CHECK-NEXT:    [[S_4:%.*]] = select i1 [[CMP_4]], i16 [[L_4]], i16 16383
-; CHECK-NEXT:    store i16 [[S_4]], i16* [[GEP_4]], align 2
 ; CHECK-NEXT:    [[GEP_5:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 5
-; CHECK-NEXT:    [[L_5:%.*]] = load i16, i16* [[GEP_5]], align 2
-; CHECK-NEXT:    [[CMP_5:%.*]] = icmp uge i16 [[L_5]], 16383
-; CHECK-NEXT:    [[S_5:%.*]] = select i1 [[CMP_5]], i16 [[L_5]], i16 16383
-; CHECK-NEXT:    store i16 [[S_5]], i16* [[GEP_5]], align 2
 ; CHECK-NEXT:    [[GEP_6:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 6
-; CHECK-NEXT:    [[L_6:%.*]] = load i16, i16* [[GEP_6]], align 2
-; CHECK-NEXT:    [[CMP_6:%.*]] = icmp uge i16 [[L_6]], 16383
-; CHECK-NEXT:    [[S_6:%.*]] = select i1 [[CMP_6]], i16 [[L_6]], i16 16383
-; CHECK-NEXT:    store i16 [[S_6]], i16* [[GEP_6]], align 2
 ; CHECK-NEXT:    [[GEP_7:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 7
-; CHECK-NEXT:    [[L_7:%.*]] = load i16, i16* [[GEP_7]], align 2
-; CHECK-NEXT:    [[CMP_7:%.*]] = icmp uge i16 [[L_7]], 16383
-; CHECK-NEXT:    [[S_7:%.*]] = select i1 [[CMP_7]], i16 [[L_7]], i16 16383
-; CHECK-NEXT:    store i16 [[S_7]], i16* [[GEP_7]], align 2
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i16>, <8 x i16>* [[TMP0]], align 2
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp uge <8 x i16> [[TMP1]], <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <8 x i1> [[TMP2]], <8 x i16> [[TMP1]], <8 x i16> <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    store <8 x i16> [[TMP3]], <8 x i16>* [[TMP4]], align 2
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i16, i16* %ptr
   %cmp.0 = icmp uge i16 %l.0, 16383
   %s.0 = select i1 %cmp.0, i16 %l.0, i16 16383
   store i16 %s.0, i16* %ptr, align 2
 
entry:
   %s.7 = select i1 %cmp.7, i16 %l.7, i16 16383
   store i16 %s.7, i16* %gep.7, align 2
   ret void
 }
 
 define void @select_umax_uge_4xi32(i32* %ptr, i32 %x) {
 ; CHECK-LABEL: @select_umax_uge_4xi32(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i32, i32* [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp uge i32 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i32 [[L_0]], i32 16383
-; CHECK-NEXT:    store i32 [[S_0]], i32* [[PTR]], align 4
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i32, i32* [[GEP_1]], align 4
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp uge i32 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i32 [[L_1]], i32 16383
-; CHECK-NEXT:    store i32 [[S_1]], i32* [[GEP_1]], align 4
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR:%.*]], i32 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i32, i32* [[GEP_2]], align 4
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp uge i32 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i32 [[L_2]], i32 16383
-; CHECK-NEXT:    store i32 [[S_2]], i32* [[GEP_2]], align 4
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i32, i32* [[GEP_3]], align 4
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp uge i32 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i32 [[L_3]], i32 16383
-; CHECK-NEXT:    store i32 [[S_3]], i32* [[GEP_3]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp uge <4 x i32> [[TMP1]], <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <4 x i1> [[TMP2]], <4 x i32> [[TMP1]], <4 x i32> <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP3]], <4 x i32>* [[TMP4]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i32, i32* %ptr
   %cmp.0 = icmp uge i32 %l.0, 16383
   %s.0 = select i1 %cmp.0, i32 %l.0, i32 16383
   store i32 %s.0, i32* %ptr, align 4
 
entry:
   store i64 %s.1, i64* %gep.1, align 4
 
   ret void
 }
 
 define void @select_smax_8xi16(i16* %ptr, i16 %x) {
 ; CHECK-LABEL: @select_smax_8xi16(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i16, i16* [[PTR:%.*]], align 2
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp sgt i16 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i16 [[L_0]], i16 16383
-; CHECK-NEXT:    store i16 [[S_0]], i16* [[PTR]], align 2
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i16, i16* [[GEP_1]], align 2
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sgt i16 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i16 [[L_1]], i16 16383
-; CHECK-NEXT:    store i16 [[S_1]], i16* [[GEP_1]], align 2
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR:%.*]], i16 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i16, i16* [[GEP_2]], align 2
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sgt i16 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i16 [[L_2]], i16 16383
-; CHECK-NEXT:    store i16 [[S_2]], i16* [[GEP_2]], align 2
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i16, i16* [[GEP_3]], align 2
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp sgt i16 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i16 [[L_3]], i16 16383
-; CHECK-NEXT:    store i16 [[S_3]], i16* [[GEP_3]], align 2
 ; CHECK-NEXT:    [[GEP_4:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 4
-; CHECK-NEXT:    [[L_4:%.*]] = load i16, i16* [[GEP_4]], align 2
-; CHECK-NEXT:    [[CMP_4:%.*]] = icmp sgt i16 [[L_4]], 16383
-; CHECK-NEXT:    [[S_4:%.*]] = select i1 [[CMP_4]], i16 [[L_4]], i16 16383
-; CHECK-NEXT:    store i16 [[S_4]], i16* [[GEP_4]], align 2
 ; CHECK-NEXT:    [[GEP_5:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 5
-; CHECK-NEXT:    [[L_5:%.*]] = load i16, i16* [[GEP_5]], align 2
-; CHECK-NEXT:    [[CMP_5:%.*]] = icmp sgt i16 [[L_5]], 16383
-; CHECK-NEXT:    [[S_5:%.*]] = select i1 [[CMP_5]], i16 [[L_5]], i16 16383
-; CHECK-NEXT:    store i16 [[S_5]], i16* [[GEP_5]], align 2
 ; CHECK-NEXT:    [[GEP_6:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 6
-; CHECK-NEXT:    [[L_6:%.*]] = load i16, i16* [[GEP_6]], align 2
-; CHECK-NEXT:    [[CMP_6:%.*]] = icmp sgt i16 [[L_6]], 16383
-; CHECK-NEXT:    [[S_6:%.*]] = select i1 [[CMP_6]], i16 [[L_6]], i16 16383
-; CHECK-NEXT:    store i16 [[S_6]], i16* [[GEP_6]], align 2
 ; CHECK-NEXT:    [[GEP_7:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 7
-; CHECK-NEXT:    [[L_7:%.*]] = load i16, i16* [[GEP_7]], align 2
-; CHECK-NEXT:    [[CMP_7:%.*]] = icmp sgt i16 [[L_7]], 16383
-; CHECK-NEXT:    [[S_7:%.*]] = select i1 [[CMP_7]], i16 [[L_7]], i16 16383
-; CHECK-NEXT:    store i16 [[S_7]], i16* [[GEP_7]], align 2
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i16>, <8 x i16>* [[TMP0]], align 2
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp sgt <8 x i16> [[TMP1]], <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <8 x i1> [[TMP2]], <8 x i16> [[TMP1]], <8 x i16> <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    store <8 x i16> [[TMP3]], <8 x i16>* [[TMP4]], align 2
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i16, i16* %ptr
   %cmp.0 = icmp sgt i16 %l.0, 16383
   %s.0 = select i1 %cmp.0, i16 %l.0, i16 16383
   store i16 %s.0, i16* %ptr, align 2
 
entry:
   %s.7 = select i1 %cmp.7, i16 %l.7, i16 16383
   store i16 %s.7, i16* %gep.7, align 2
   ret void
 }
 
 define void @select_smax_4xi32(i32* %ptr, i32 %x) {
 ; CHECK-LABEL: @select_smax_4xi32(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i32, i32* [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp sgt i32 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i32 [[L_0]], i32 16383
-; CHECK-NEXT:    store i32 [[S_0]], i32* [[PTR]], align 4
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i32, i32* [[GEP_1]], align 4
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sgt i32 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i32 [[L_1]], i32 16383
-; CHECK-NEXT:    store i32 [[S_1]], i32* [[GEP_1]], align 4
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR:%.*]], i32 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i32, i32* [[GEP_2]], align 4
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sgt i32 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i32 [[L_2]], i32 16383
-; CHECK-NEXT:    store i32 [[S_2]], i32* [[GEP_2]], align 4
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i32, i32* [[GEP_3]], align 4
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp sgt i32 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i32 [[L_3]], i32 16383
-; CHECK-NEXT:    store i32 [[S_3]], i32* [[GEP_3]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp sgt <4 x i32> [[TMP1]], <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <4 x i1> [[TMP2]], <4 x i32> [[TMP1]], <4 x i32> <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP3]], <4 x i32>* [[TMP4]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i32, i32* %ptr
   %cmp.0 = icmp sgt i32 %l.0, 16383
   %s.0 = select i1 %cmp.0, i32 %l.0, i32 16383
   store i32 %s.0, i32* %ptr, align 4
 
entry:
 
   ret void
 }
 
 
 define void @select_smax_sge_8xi16(i16* %ptr, i16 %x) {
 ; CHECK-LABEL: @select_smax_sge_8xi16(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i16, i16* [[PTR:%.*]], align 2
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp sge i16 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i16 [[L_0]], i16 16383
-; CHECK-NEXT:    store i16 [[S_0]], i16* [[PTR]], align 2
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i16, i16* [[GEP_1]], align 2
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sge i16 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i16 [[L_1]], i16 16383
-; CHECK-NEXT:    store i16 [[S_1]], i16* [[GEP_1]], align 2
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i16, i16* [[PTR:%.*]], i16 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i16, i16* [[GEP_2]], align 2
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sge i16 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i16 [[L_2]], i16 16383
-; CHECK-NEXT:    store i16 [[S_2]], i16* [[GEP_2]], align 2
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i16, i16* [[GEP_3]], align 2
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp sge i16 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i16 [[L_3]], i16 16383
-; CHECK-NEXT:    store i16 [[S_3]], i16* [[GEP_3]], align 2
 ; CHECK-NEXT:    [[GEP_4:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 4
-; CHECK-NEXT:    [[L_4:%.*]] = load i16, i16* [[GEP_4]], align 2
-; CHECK-NEXT:    [[CMP_4:%.*]] = icmp sge i16 [[L_4]], 16383
-; CHECK-NEXT:    [[S_4:%.*]] = select i1 [[CMP_4]], i16 [[L_4]], i16 16383
-; CHECK-NEXT:    store i16 [[S_4]], i16* [[GEP_4]], align 2
 ; CHECK-NEXT:    [[GEP_5:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 5
-; CHECK-NEXT:    [[L_5:%.*]] = load i16, i16* [[GEP_5]], align 2
-; CHECK-NEXT:    [[CMP_5:%.*]] = icmp sge i16 [[L_5]], 16383
-; CHECK-NEXT:    [[S_5:%.*]] = select i1 [[CMP_5]], i16 [[L_5]], i16 16383
-; CHECK-NEXT:    store i16 [[S_5]], i16* [[GEP_5]], align 2
 ; CHECK-NEXT:    [[GEP_6:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 6
-; CHECK-NEXT:    [[L_6:%.*]] = load i16, i16* [[GEP_6]], align 2
-; CHECK-NEXT:    [[CMP_6:%.*]] = icmp sge i16 [[L_6]], 16383
-; CHECK-NEXT:    [[S_6:%.*]] = select i1 [[CMP_6]], i16 [[L_6]], i16 16383
-; CHECK-NEXT:    store i16 [[S_6]], i16* [[GEP_6]], align 2
 ; CHECK-NEXT:    [[GEP_7:%.*]] = getelementptr inbounds i16, i16* [[PTR]], i16 7
-; CHECK-NEXT:    [[L_7:%.*]] = load i16, i16* [[GEP_7]], align 2
-; CHECK-NEXT:    [[CMP_7:%.*]] = icmp sge i16 [[L_7]], 16383
-; CHECK-NEXT:    [[S_7:%.*]] = select i1 [[CMP_7]], i16 [[L_7]], i16 16383
-; CHECK-NEXT:    store i16 [[S_7]], i16* [[GEP_7]], align 2
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i16>, <8 x i16>* [[TMP0]], align 2
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp sge <8 x i16> [[TMP1]], <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <8 x i1> [[TMP2]], <8 x i16> [[TMP1]], <8 x i16> <i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383, i16 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i16* [[PTR]] to <8 x i16>*
+; CHECK-NEXT:    store <8 x i16> [[TMP3]], <8 x i16>* [[TMP4]], align 2
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i16, i16* %ptr
   %cmp.0 = icmp sge i16 %l.0, 16383
   %s.0 = select i1 %cmp.0, i16 %l.0, i16 16383
   store i16 %s.0, i16* %ptr, align 2
 
entry:
   %s.7 = select i1 %cmp.7, i16 %l.7, i16 16383
   store i16 %s.7, i16* %gep.7, align 2
   ret void
 }
 
 define void @select_smax_sge_4xi32(i32* %ptr, i32 %x) {
 ; CHECK-LABEL: @select_smax_sge_4xi32(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i32, i32* [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp sge i32 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i32 [[L_0]], i32 16383
-; CHECK-NEXT:    store i32 [[S_0]], i32* [[PTR]], align 4
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i32, i32* [[GEP_1]], align 4
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp sge i32 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i32 [[L_1]], i32 16383
-; CHECK-NEXT:    store i32 [[S_1]], i32* [[GEP_1]], align 4
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR:%.*]], i32 1
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i32, i32* [[GEP_2]], align 4
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp sge i32 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i32 [[L_2]], i32 16383
-; CHECK-NEXT:    store i32 [[S_2]], i32* [[GEP_2]], align 4
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i32, i32* [[GEP_3]], align 4
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp sge i32 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i32 [[L_3]], i32 16383
-; CHECK-NEXT:    store i32 [[S_3]], i32* [[GEP_3]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp sge <4 x i32> [[TMP1]], <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP3:%.*]] = select <4 x i1> [[TMP2]], <4 x i32> [[TMP1]], <4 x i32> <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP3]], <4 x i32>* [[TMP4]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i32, i32* %ptr
   %cmp.0 = icmp sge i32 %l.0, 16383
   %s.0 = select i1 %cmp.0, i32 %l.0, i32 16383
   store i32 %s.0, i32* %ptr, align 4