[x86, SSE] AVX1 PR28129

[dtemirbulatov]

Summary

Further perf tests on Jaguar indicate that:

vxorps  %ymm0, %ymm0, %ymm0
vcmpps  $15, %ymm0, %ymm0, %ymm0

is consistently faster (by about 9%) than:

vpcmpeqd  %xmm0, %xmm0, %xmm0
vinsertf128  $1, %xmm0, %ymm0, %ymm0

Testing equivalent code on a SandyBridge (E5-2640) puts it slightly (~3%) faster as well.
OK to commit, this change is only related to AVX1?

[dtemirbulatov]

here is test-case:
define <4 x double> @cmp256_domain(<4 x double> %a) {

%cmp = fcmp oeq <4 x double> zeroinitializer, zeroinitializer
%sext = sext <4 x i1> %cmp to <4 x i64>
%mask = bitcast <4 x i64> %sext to <4 x double>
%add = fadd <4 x double> %a, %mask
ret <4 x double> %add

}
So, we are changing "immAllOnesV" for 256-bit vectors only for AVX1 machine.

[spatel]

lib/Target/X86/X86InstrSSE.td

7761–7762

This comment should be updated to match the new code.

Is it correct that this pattern won't apply to most integer code for an AVX target because that would already be legalized to v4i32/v2i64? If that's true, I think it's also worth mentioning here.

I'm imagining cases like this:

define <8 x i32> @cmpeq_v8i32(<8 x i32> %a) nounwind {
 %cmp = icmp eq <8 x i32> %a, <i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1>
  %res = sext <8 x i1> %cmp to <8 x i32>
 ret <8 x i32> %res
}

define <8 x i32> @cmpne_v8i32(<8 x i32> %a) nounwind {
  %cmp = icmp ne <8 x i32> %a, <i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1>
  %res = sext <8 x i1> %cmp to <8 x i32>
  ret <8 x i32> %res
}

define <8 x i32> @sub1_v8i32(<8 x i32> %a) nounwind {
  %add = add <8 x i32> %a, <i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1>
  ret <8 x i32> %add
}

7762–7766

It's not clear why we require a zero operand. Would a dummy (undef) register also work? Should we allow that when optimizing for size so the vxorps is not needed?

test/CodeGen/X86/vector-pcmp.ll

151–153

That's an interesting case...that we probably can't answer at the DAG level. Would it be better to use two 128-bit vpxor instructions instead of incurring a potential domain-crossing penalty with the one 256-bit vxorps?

[RKSimon]

lib/Target/X86/X86InstrSSE.td

7762–7766

There isn't a fast path for vcmptrue (despite it ignoring the inputs) on Jaguar/SandyBridge - I mentioned in https://bugs.llvm.org/show_bug.cgi?id=28129#c8 that using undef vars causes dependency regressions. Zeroing the register breaks the dependency.

[RKSimon]

lib/Target/X86/X86InstrSSE.td

7761–7762

This comment should be updated to match the new code.

+1

test/CodeGen/X86/vector-pcmp.ll

151–153

Do you mean this?

vextractf128 $1, %ymm0, %xmm1
vpxor %xmm2, %xmm2, %xmm2
vpcmpgtb %xmm1, %xmm2, %xmm1
vpcmpgtb %xmm0, %xmm2, %xmm0
vcmpeqd %xmm2, %xmm2, %xmm2
vpxor %xmm2, %xmm1, %xmm1
vpxor %xmm2, %xmm0, %xmm0
vinsertf128 $1, %xmm1, %ymm0, %ymm0

[spatel]

test/CodeGen/X86/vector-pcmp.ll

151–153

Yes - I remember reading somewhere (and not sure how widely this applies) that the 'insertX128' insts may not actually have domain-crossing penalties. The other variable in this mix (thinking about Jaguar here) is that the 256-bit ops may be cracked and double-pumped anyway, so if we have that + domain-crossing penalty, then the two 128-bit insts should be faster?

[RKSimon]

test/CodeGen/X86/vector-pcmp.ll

151–153

A quick hot loop test suggests that the old vpcmpeqd+vinsertf128+xor approach takes 8cy, the 256-bit xor+vcmptrueps+xor approach takes 7cy and the 128-bit vpcmpeqd+2*xor takes 6cy on Jaguar.

It might be worth looking at splitting some 256-bit bitwise operations that take concatenated 128-bit operations, but I don't think it should get in the way of this patch.

[spatel]

test/CodeGen/X86/vector-pcmp.ll

151–153

Agreed - the splitting problem is separate:
https://bugs.llvm.org/show_bug.cgi?id=32790

[RKSimon]

Almost there - the only other improvement I can think of would be to use vcmptrueps(undef, undef) for OptForSize - doesn't break the dependency so slight perf regression in exchange for no vxorps.

[zvi]

Here's what IACA reports for Sandybridge:

Before:

| Num Of |              Ports pressure in cycles              |      |
|  Uops    |  0  - DV  |  1  |  2  -  D  |  3  -  D  |  4  |  5  |      |
|   1        |                | 0.3 |              |              |      | 0.7 |       | vpcmpeqd xmm1, xmm1, xmm1
|   1        |               |       |               |              |      | 1.0 |      | vinsertf128 ymm1, ymm1, xmm1, 0x1
|   1        | 1.0         |       |               |              |      |       | CP | vblendps ymm0, ymm1, ymm0, 0xaa
|   1        |               | 1.0 |               |              |      |       | CP | vcvtdq2ps ymm0, ymm0

After:

| Num Of |              Ports pressure in cycles              |      |

Uops	0 - DV	1	2 - D	3 - D	4	5
1*								vxorps ymm1, ymm1, ymm1
1		1.0						vcmpps ymm1, ymm1, ymm1, 0xf
1	0.5					0.5	CP	vblendps ymm0, ymm1, ymm0, 0xaa
1		1.0					CP	vcvtdq2ps ymm0, ymm0

\* - instruction micro-ops not bound to a port

[dtemirbulatov]

update after review process.

[dtemirbulatov]

another update after issue with -Os.

[dtemirbulatov]

update after issue -Os was cleared.

[spatel]

This looks almost done. I have one question about -Os vs. -Oz, and one nit about a code comment.

lib/Target/X86/X86InstrSSE.td

489

Should this be "OptForMinSize"? Ie, I was just looking at some other code patterns, and we are inserting xorps/xorpd even at -Os.

In fact, we're doing that even at -Oz which I think is a mistake.

7761–7762

I'd add a little more explanation here just to make it clearer:
"To create a 256-bit all ones value, we use VCMPTRUEPS and zero out the fake register operand to avoid false dependencies."

[spatel]

lib/Target/X86/X86InstrSSE.td

489

After thinking about this again, I'm now wondering why we would make the zero reg input a part of the patterns here. Are there any other td patterns that do this? For the most part, I think we handle the xor generation using ExecutionDepsFix.

[dtemirbulatov]

Rebased and changed OptForSize to OptForMinSize.

[spatel]

test/CodeGen/X86/all-ones-vector.ll

1

Many cosmetic diffs have been introduced because you used a different script. Please use "update_llc_test_checks.py" to regenerate these and upload the patch again. I need to "fix" the other script to not work with llc. :)

[dtemirbulatov]

updated tests after review with proper update_llc_test_checks.py.

[RKSimon]

LGTM - with a future patch to investigate using ExecutionDepsFix

lib/Target/X86/X86InstrInfo.cpp

case X86::AVX512_256_SET0: {
     return Expand2AddrUndef(MIB, get(X86::VPXORDZrr));
   }
   case X86::AVX512_512_SET0:
     return Expand2AddrUndef(MIB, get(X86::VPXORDZrr));
   case X86::V_SETALLONES:
     return Expand2AddrUndef(MIB, get(HasAVX ? X86::VPCMPEQDrr : X86::PCMPEQDrr));
   case X86::AVX2_SETALLONES:
     return Expand2AddrUndef(MIB, get(X86::VPCMPEQDYrr));
+  case X86::AVX1_SETALLONES: {
+    unsigned Reg = MIB->getOperand(0).getReg();
+    // VCMPPSYrri with an immediate 0xf should
+    // produce VCMPTRUEPS.
+    MIB->setDesc(get(X86::VCMPPSYrri));
+    MIB.addReg(Reg, RegState::Undef).addReg(Reg, RegState::Undef).addImm(0xf);
+    return true;
+  }
   case X86::AVX512_512_SETALLONES: {
     unsigned Reg = MIB->getOperand(0).getReg();
     MIB->setDesc(get(X86::VPTERNLOGDZrri));
     // VPTERNLOGD needs 3 register inputs and an immediate.
     // 0xff will return 1s for any input.
     MIB.addReg(Reg, RegState::Undef).addReg(Reg, RegState::Undef)
        .addReg(Reg, RegState::Undef).addImm(0xff);
     return true;
if (LoadMI.hasOneMemOperand())
     Alignment = (*LoadMI.memoperands_begin())->getAlignment();
   else
     switch (LoadMI.getOpcode()) {
     case X86::AVX512_512_SET0:
     case X86::AVX512_512_SETALLONES:
       Alignment = 64;
       break;
     case X86::AVX2_SETALLONES:
+    case X86::AVX1_SETALLONES:
     case X86::AVX_SET0:
     case X86::AVX512_256_SET0:
       Alignment = 32;
       break;
     case X86::V_SET0:
     case X86::V_SETALLONES:
     case X86::AVX512_128_SET0:
       Alignment = 16;
MachineInstr *X86InstrInfo::foldMemoryOperandImpl(
   if (LoadMI.getOperand(0).getSubReg() != MI.getOperand(Ops[0]).getSubReg())
     return nullptr;
 
   SmallVector<MachineOperand,X86::AddrNumOperands> MOs;
   switch (LoadMI.getOpcode()) {
   case X86::V_SET0:
   case X86::V_SETALLONES:
   case X86::AVX2_SETALLONES:
+  case X86::AVX1_SETALLONES:
   case X86::AVX_SET0:
   case X86::AVX512_128_SET0:
   case X86::AVX512_256_SET0:
   case X86::AVX512_512_SET0:
   case X86::AVX512_512_SETALLONES:
   case X86::FsFLD0SD:
   case X86::AVX512_FsFLD0SD:
   case X86::FsFLD0SS:
case X86::AVX512_FsFLD0SS: {
     unsigned Opc = LoadMI.getOpcode();
     if (Opc == X86::FsFLD0SS || Opc == X86::AVX512_FsFLD0SS)
       Ty = Type::getFloatTy(MF.getFunction()->getContext());
     else if (Opc == X86::FsFLD0SD || Opc == X86::AVX512_FsFLD0SD)
       Ty = Type::getDoubleTy(MF.getFunction()->getContext());
     else if (Opc == X86::AVX512_512_SET0 || Opc == X86::AVX512_512_SETALLONES)
       Ty = VectorType::get(Type::getInt32Ty(MF.getFunction()->getContext()),16);
     else if (Opc == X86::AVX2_SETALLONES || Opc == X86::AVX_SET0 ||
-             Opc == X86::AVX512_256_SET0)
+             Opc == X86::AVX512_256_SET0 || Opc == X86::AVX1_SETALLONES)
       Ty = VectorType::get(Type::getInt32Ty(MF.getFunction()->getContext()), 8);
     else
       Ty = VectorType::get(Type::getInt32Ty(MF.getFunction()->getContext()), 4);
 
     bool IsAllOnes = (Opc == X86::V_SETALLONES || Opc == X86::AVX2_SETALLONES ||
-                      Opc == X86::AVX512_512_SETALLONES);
+                      Opc == X86::AVX512_512_SETALLONES || 
+                      Opc == X86::AVX1_SETALLONES);
     const Constant *C = IsAllOnes ? Constant::getAllOnesValue(Ty) :
                                     Constant::getNullValue(Ty);
     unsigned CPI = MCP.getConstantPoolIndex(C, Alignment);
 
     // Create operands to load from the constant pool entry.
     MOs.push_back(MachineOperand::CreateReg(PICBase, false));
     MOs.push_back(MachineOperand::CreateImm(1));
     MOs.push_back(MachineOperand::CreateReg(0, false));

lib/Target/X86/X86InstrSSE.td

 }
 
 // We set canFoldAsLoad because this can be converted to a constant-pool
 // load of an all-ones value if folding it would be beneficial.
 let isReMaterializable = 1, isAsCheapAsAMove = 1, canFoldAsLoad = 1,
     isPseudo = 1, SchedRW = [WriteZero] in {
   def V_SETALLONES : I<0, Pseudo, (outs VR128:$dst), (ins), "",
                        [(set VR128:$dst, (v4i32 immAllOnesV))]>;
+  // TODO: there is an issue in triggering OptForSize with vector code.
+  let Predicates = [HasAVX1Only, OptForSize] in {
+  def AVX1_SETALLONES: I<0, Pseudo, (outs VR256:$dst), (ins), "",
+                          [(set VR256:$dst, (v8i32 immAllOnesV))]>;
+  }
   let Predicates = [HasAVX2] in
   def AVX2_SETALLONES : I<0, Pseudo, (outs VR256:$dst), (ins), "",
                           [(set VR256:$dst, (v8i32 immAllOnesV))]>;
 }
 
 //===----------------------------------------------------------------------===//
 // SSE 1 & 2 - Move FP Scalar Instructions
 //
def VINSERTF128rr : AVXAIi8<0x18, MRMSrcReg, (outs VR256:$dst),
           "vinsertf128\t{$src3, $src2, $src1, $dst|$dst, $src1, $src2, $src3}",
           []>, Sched<[WriteFShuffle]>, VEX_4V, VEX_L;
 let mayLoad = 1 in
 def VINSERTF128rm : AVXAIi8<0x18, MRMSrcMem, (outs VR256:$dst),
           (ins VR256:$src1, f128mem:$src2, u8imm:$src3),
           "vinsertf128\t{$src3, $src2, $src1, $dst|$dst, $src1, $src2, $src3}",
           []>, Sched<[WriteFShuffleLd, ReadAfterLd]>, VEX_4V, VEX_L;
 }
 
-
-// Without AVX2 we need to concat two v4i32 V_SETALLONES to create a 256-bit
-// all ones value.
-let Predicates = [HasAVX1Only] in
-def : Pat<(v8i32 immAllOnesV),
-          (VINSERTF128rr
-           (INSERT_SUBREG (v8i32 (IMPLICIT_DEF)), (V_SETALLONES), sub_xmm),
-           (V_SETALLONES), 1)>;
+// To create a 256-bit all ones value, we should produce VCMPTRUEPS
+// with YMM register containing zero.
+let Predicates = [HasAVX1Only] in {
+def : Pat<(v8i32 immAllOnesV), (VCMPPSYrri (AVX_SET0), (AVX_SET0), 0xf)>;
+}
 
 multiclass vinsert_lowering<string InstrStr, ValueType From, ValueType To,
                             PatFrag memop_frag> {
   def : Pat<(vinsert128_insert:$ins (To VR256:$src1), (From VR128:$src2),
                                    (iPTR imm)),
             (!cast<Instruction>(InstrStr#rr) VR256:$src1, VR128:$src2,
                                        (INSERT_get_vinsert128_imm VR256:$ins))>;
   def : Pat<(vinsert128_insert:$ins (To VR256:$src1),

test/CodeGen/X86/all-ones-vector.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i386-unknown -mattr=+sse2 | FileCheck %s --check-prefix=X32-SSE
 ; RUN: llc < %s -mtriple=i386-unknown -mattr=+avx  | FileCheck %s --check-prefix=X32-AVX --check-prefix=X32-AVX1
 ; RUN: llc < %s -mtriple=i386-unknown -mattr=+avx2 | FileCheck %s --check-prefix=X32-AVX --check-prefix=X32-AVX256 --check-prefix=X32-AVX2
 ; RUN: llc < %s -mtriple=i386-unknown -mcpu=knl | FileCheck %s --check-prefix=X32-AVX --check-prefix=X32-AVX256 --check-prefix=X32-AVX512 --check-prefix=X32-KNL
 ; RUN: llc < %s -mtriple=i386-unknown -mcpu=skx | FileCheck %s --check-prefix=X32-AVX --check-prefix=X32-AVX256 --check-prefix=X32-AVX512 --check-prefix=X32-SKX
 ; RUN: llc < %s -mtriple=x86_64-unknown -mattr=+sse2 | FileCheck %s --check-prefix=X64-SSE
 ; RUN: llc < %s -mtriple=x86_64-unknown -mattr=+avx  | FileCheck %s --check-prefix=X64-AVX --check-prefix=X64-AVX1
 ; RUN: llc < %s -mtriple=x86_64-unknown -mattr=+avx2 | FileCheck %s --check-prefix=X64-AVX --check-prefix=X64-AVX256 --check-prefix=X64-AVX2

 ; X32-SSE-LABEL: allones_v32i8:
 ; X32-SSE:       # BB#0:
 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v32i8:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX256-LABEL: allones_v32i8:
 ; X32-AVX256:       # BB#0:
 ; X32-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX256-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v32i8:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v32i8:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX256-LABEL: allones_v32i8:
 ; X64-AVX256:       # BB#0:
 ; X64-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX256-NEXT:    retq
   ret <32 x i8> <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
 }
 
 define <16 x i16> @allones_v16i16() nounwind {
 ; X32-SSE-LABEL: allones_v16i16:
 ; X32-SSE:       # BB#0:
 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v16i16:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX256-LABEL: allones_v16i16:
 ; X32-AVX256:       # BB#0:
 ; X32-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX256-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v16i16:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v16i16:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX256-LABEL: allones_v16i16:
 ; X64-AVX256:       # BB#0:
 ; X64-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX256-NEXT:    retq
   ret <16 x i16> <i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1>
 }
 
 define <8 x i32> @allones_v8i32() nounwind {
 ; X32-SSE-LABEL: allones_v8i32:
 ; X32-SSE:       # BB#0:
 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v8i32:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX256-LABEL: allones_v8i32:
 ; X32-AVX256:       # BB#0:
 ; X32-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX256-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v8i32:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v8i32:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX256-LABEL: allones_v8i32:
 ; X64-AVX256:       # BB#0:
 ; X64-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX256-NEXT:    retq
   ret <8 x i32> <i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1>
 }
 
 define <4 x i64> @allones_v4i64() nounwind {
 ; X32-SSE-LABEL: allones_v4i64:
 ; X32-SSE:       # BB#0:
 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v4i64:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX256-LABEL: allones_v4i64:
 ; X32-AVX256:       # BB#0:
 ; X32-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX256-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v4i64:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v4i64:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX256-LABEL: allones_v4i64:
 ; X64-AVX256:       # BB#0:
 ; X64-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX256-NEXT:    retq
   ret <4 x i64> <i64 -1, i64 -1, i64 -1, i64 -1>
 }
 
 define <4 x double> @allones_v4f64() nounwind {
 ; X32-SSE-LABEL: allones_v4f64:
 ; X32-SSE:       # BB#0:
 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v4f64:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX256-LABEL: allones_v4f64:
 ; X32-AVX256:       # BB#0:
 ; X32-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX256-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v4f64:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v4f64:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX256-LABEL: allones_v4f64:
 ; X64-AVX256:       # BB#0:
 ; X64-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX256-NEXT:    retq
   ret <4 x double> <double 0xffffffffffffffff, double 0xffffffffffffffff, double 0xffffffffffffffff, double 0xffffffffffffffff>
 }
 
 define <4 x double> @allones_v4f64_optsize() nounwind optsize {
 ; X32-SSE-LABEL: allones_v4f64_optsize:
 ; X32-SSE:       # BB#0:
 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v4f64_optsize:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX256-LABEL: allones_v4f64_optsize:
 ; X32-AVX256:       # BB#0:
 ; X32-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX256-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v4f64_optsize:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v4f64_optsize:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX256-LABEL: allones_v4f64_optsize:
 ; X64-AVX256:       # BB#0:
 ; X64-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX256-NEXT:    retq
   ret <4 x double> <double 0xffffffffffffffff, double 0xffffffffffffffff, double 0xffffffffffffffff, double 0xffffffffffffffff>
 }
 
 define <8 x float> @allones_v8f32() nounwind {
 ; X32-SSE-LABEL: allones_v8f32:
 ; X32-SSE:       # BB#0:
 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v8f32:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX256-LABEL: allones_v8f32:
 ; X32-AVX256:       # BB#0:
 ; X32-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX256-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v8f32:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v8f32:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX256-LABEL: allones_v8f32:
 ; X64-AVX256:       # BB#0:
 ; X64-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX256-NEXT:    retq
   ret <8 x float> <float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000>
 }
 
 define <8 x float> @allones_v8f32_optsize() nounwind optsize {
 ; X32-SSE-LABEL: allones_v8f32_optsize:
 ; X32-SSE:       # BB#0:
 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v8f32_optsize:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX256-LABEL: allones_v8f32_optsize:
 ; X32-AVX256:       # BB#0:
 ; X32-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX256-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v8f32_optsize:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v8f32_optsize:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX256-LABEL: allones_v8f32_optsize:
 ; X64-AVX256:       # BB#0:
 ; X64-AVX256-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX256-NEXT:    retq
   ret <8 x float> <float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000>
 }
 
 define <64 x i8> @allones_v64i8() nounwind {
 ; X32-SSE-LABEL: allones_v64i8:
 ; X32-SSE:       # BB#0:
 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X32-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v64i8:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX2-LABEL: allones_v64i8:
 ; X32-AVX2:       # BB#0:
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X32-AVX2-NEXT:    retl

 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X64-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v64i8:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: allones_v64i8:
 ; X64-AVX2:       # BB#0:
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X64-AVX2-NEXT:    retq

 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X32-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v32i16:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX2-LABEL: allones_v32i16:
 ; X32-AVX2:       # BB#0:
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X32-AVX2-NEXT:    retl

 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X64-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v32i16:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: allones_v32i16:
 ; X64-AVX2:       # BB#0:
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X64-AVX2-NEXT:    retq

 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X32-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v16i32:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX2-LABEL: allones_v16i32:
 ; X32-AVX2:       # BB#0:
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X32-AVX2-NEXT:    retl
 ;
 ; X32-AVX512-LABEL: allones_v16i32:
 ; X32-AVX512:       # BB#0:
 ; X32-AVX512-NEXT:    vpternlogd $255, %zmm0, %zmm0, %zmm0
 ; X32-AVX512-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v16i32:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X64-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v16i32:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: allones_v16i32:
 ; X64-AVX2:       # BB#0:
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X64-AVX2-NEXT:    retq

 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X32-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v8i64:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX2-LABEL: allones_v8i64:
 ; X32-AVX2:       # BB#0:
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X32-AVX2-NEXT:    retl
 ;
 ; X32-AVX512-LABEL: allones_v8i64:
 ; X32-AVX512:       # BB#0:
 ; X32-AVX512-NEXT:    vpternlogd $255, %zmm0, %zmm0, %zmm0
 ; X32-AVX512-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v8i64:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X64-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v8i64:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: allones_v8i64:
 ; X64-AVX2:       # BB#0:
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X64-AVX2-NEXT:    retq

 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X32-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v8f64:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX2-LABEL: allones_v8f64:
 ; X32-AVX2:       # BB#0:
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X32-AVX2-NEXT:    retl
 ;
 ; X32-AVX512-LABEL: allones_v8f64:
 ; X32-AVX512:       # BB#0:
 ; X32-AVX512-NEXT:    vpternlogd $255, %zmm0, %zmm0, %zmm0
 ; X32-AVX512-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v8f64:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X64-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v8f64:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: allones_v8f64:
 ; X64-AVX2:       # BB#0:
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X64-AVX2-NEXT:    retq

 ; X32-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X32-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X32-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X32-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X32-SSE-NEXT:    retl
 ;
 ; X32-AVX1-LABEL: allones_v16f32:
 ; X32-AVX1:       # BB#0:
-; X32-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X32-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX2-LABEL: allones_v16f32:
 ; X32-AVX2:       # BB#0:
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X32-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X32-AVX2-NEXT:    retl
 ;
 ; X32-AVX512-LABEL: allones_v16f32:
 ; X32-AVX512:       # BB#0:
 ; X32-AVX512-NEXT:    vpternlogd $255, %zmm0, %zmm0, %zmm0
 ; X32-AVX512-NEXT:    retl
 ;
 ; X64-SSE-LABEL: allones_v16f32:
 ; X64-SSE:       # BB#0:
 ; X64-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
 ; X64-SSE-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-SSE-NEXT:    pcmpeqd %xmm2, %xmm2
 ; X64-SSE-NEXT:    pcmpeqd %xmm3, %xmm3
 ; X64-SSE-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: allones_v16f32:
 ; X64-AVX1:       # BB#0:
-; X64-AVX1-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vxorps %ymm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; X64-AVX1-NEXT:    vmovaps %ymm0, %ymm1
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: allones_v16f32:
 ; X64-AVX2:       # BB#0:
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; X64-AVX2-NEXT:    retq
 ;
 ; X64-AVX512-LABEL: allones_v16f32:
 ; X64-AVX512:       # BB#0:
 ; X64-AVX512-NEXT:    vpternlogd $255, %zmm0, %zmm0, %zmm0
 ; X64-AVX512-NEXT:    retq
   ret <16 x float> <float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000, float 0xffffffffe0000000>
 }

test/CodeGen/X86/avx-basic.ll

; CHECK-NEXT:    retq
   store <8 x float> zeroinitializer, <8 x float>* @x, align 32
   store <4 x double> zeroinitializer, <4 x double>* @y, align 32
   ret void
 }
 
 define void @ones([0 x float]* nocapture %RET, [0 x float]* nocapture %aFOO) nounwind {
 ; CHECK-LABEL: ones:
 ; CHECK:       ## BB#0: ## %allocas
-; CHECK-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; CHECK-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; CHECK-NEXT:    vxorps  %ymm0, %ymm0, %ymm0
+; CHECK-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; CHECK-NEXT:    vmovaps %ymm0, (%rdi)
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 allocas:
   %ptr2vec615 = bitcast [0 x float]* %RET to <8 x float>*
   store <8 x float> <float 0xFFFFFFFFE0000000, float 0xFFFFFFFFE0000000, float
 0xFFFFFFFFE0000000, float 0xFFFFFFFFE0000000, float 0xFFFFFFFFE0000000, float
 0xFFFFFFFFE0000000, float 0xFFFFFFFFE0000000, float 0xFFFFFFFFE0000000>, <8 x
 float>* %ptr2vec615, align 32
   ret void
 }
 
 define void @ones2([0 x i32]* nocapture %RET, [0 x i32]* nocapture %aFOO) nounwind {
 ; CHECK-LABEL: ones2:
 ; CHECK:       ## BB#0: ## %allocas
-; CHECK-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; CHECK-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; CHECK-NEXT:    vxorps  %ymm0, %ymm0, %ymm0
+; CHECK-NEXT:    vcmptrueps %ymm0, %ymm0, %ymm0
 ; CHECK-NEXT:    vmovaps %ymm0, (%rdi)
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 allocas:
   %ptr2vec615 = bitcast [0 x i32]* %RET to <8 x i32>*
   store <8 x i32> <i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1>, <8 x i32>* %ptr2vec615, align 32
   ret void
 }

test/CodeGen/X86/avx-cvt-3.ll

; X64-NEXT:    retq
   %1 = shufflevector <8 x i32> %a0, <8 x i32> <i32 0, i32 undef, i32 0, i32 undef, i32 0, i32 undef, i32 0, i32 undef>, <8 x i32> <i32 8, i32 1, i32 10, i32 3, i32 12, i32 5, i32 14, i32 7>
   %2 = sitofp <8 x i32> %1 to <8 x float>
   ret <8 x float> %2
 }
 
 define <8 x float> @sitofp_insert_allbits_v8i32(<8 x i32> %a0) {
 ; X86-LABEL: sitofp_insert_allbits_v8i32:
 ; X86:       # BB#0:
-; X86-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X86-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X86-NEXT:    vxorps %ymm1, %ymm1, %ymm1
+; X86-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X86-NEXT:    vblendps {{.*#+}} ymm0 = ymm1[0],ymm0[1],ymm1[2],ymm0[3],ymm1[4,5],ymm0[6,7]
 ; X86-NEXT:    vcvtdq2ps %ymm0, %ymm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: sitofp_insert_allbits_v8i32:
 ; X64:       # BB#0:
-; X64-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X64-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X64-NEXT:    vxorps  %ymm1, %ymm1, %ymm1
+; X64-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X64-NEXT:    vblendps {{.*#+}} ymm0 = ymm1[0],ymm0[1],ymm1[2],ymm0[3],ymm1[4,5],ymm0[6,7]
 ; X64-NEXT:    vcvtdq2ps %ymm0, %ymm0
 ; X64-NEXT:    retq
   %1 = insertelement <8 x i32> %a0, i32 -1, i32 0
   %2 = insertelement <8 x i32>  %1, i32 -1, i32 2
   %3 = insertelement <8 x i32>  %2, i32 -1, i32 4
   %4 = insertelement <8 x i32>  %3, i32 -1, i32 5
   %5 = sitofp <8 x i32> %4 to <8 x float>
   ret <8 x float> %5
 }
 
 define <8 x float> @sitofp_shuffle_allbits_v8i32(<8 x i32> %a0) {
 ; X86-LABEL: sitofp_shuffle_allbits_v8i32:
 ; X86:       # BB#0:
-; X86-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X86-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X86-NEXT:    vxorps  %ymm1, %ymm1, %ymm1
+; X86-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X86-NEXT:    vblendps {{.*#+}} ymm0 = ymm1[0],ymm0[1],ymm1[2],ymm0[3],ymm1[4],ymm0[5],ymm1[6],ymm0[7]
 ; X86-NEXT:    vcvtdq2ps %ymm0, %ymm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: sitofp_shuffle_allbits_v8i32:
 ; X64:       # BB#0:
-; X64-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X64-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X64-NEXT:    vxorps %ymm1, %ymm1, %ymm1
+; X64-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X64-NEXT:    vblendps {{.*#+}} ymm0 = ymm1[0],ymm0[1],ymm1[2],ymm0[3],ymm1[4],ymm0[5],ymm1[6],ymm0[7]
 ; X64-NEXT:    vcvtdq2ps %ymm0, %ymm0
 ; X64-NEXT:    retq
   %1 = shufflevector <8 x i32> %a0, <8 x i32> <i32 -1, i32 undef, i32 -1, i32 undef, i32 -1, i32 undef, i32 -1, i32 undef>, <8 x i32> <i32 8, i32 1, i32 10, i32 3, i32 12, i32 5, i32 14, i32 7>
   %2 = sitofp <8 x i32> %1 to <8 x float>
   ret <8 x float> %2
 }
 
 define <8 x float> @sitofp_insert_constants_v8i32(<8 x i32> %a0) {
 ; X86-LABEL: sitofp_insert_constants_v8i32:
 ; X86:       # BB#0:
 ; X86-NEXT:    vxorps %ymm1, %ymm1, %ymm1
 ; X86-NEXT:    vblendps {{.*#+}} ymm0 = ymm1[0],ymm0[1,2,3,4,5,6,7]
-; X86-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X86-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X86-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X86-NEXT:    vblendps {{.*#+}} ymm0 = ymm0[0,1],ymm1[2],ymm0[3,4,5,6,7]
 ; X86-NEXT:    vextractf128 $1, %ymm0, %xmm1
 ; X86-NEXT:    movl $2, %eax
 ; X86-NEXT:    vpinsrd $0, %eax, %xmm1, %xmm1
 ; X86-NEXT:    movl $-3, %eax
 ; X86-NEXT:    vpinsrd $1, %eax, %xmm1, %xmm1
 ; X86-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; X86-NEXT:    vcvtdq2ps %ymm0, %ymm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: sitofp_insert_constants_v8i32:
 ; X64:       # BB#0:
 ; X64-NEXT:    vxorps %ymm1, %ymm1, %ymm1
 ; X64-NEXT:    vblendps {{.*#+}} ymm0 = ymm1[0],ymm0[1,2,3,4,5,6,7]
-; X64-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X64-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X64-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X64-NEXT:    vblendps {{.*#+}} ymm0 = ymm0[0,1],ymm1[2],ymm0[3,4,5,6,7]
 ; X64-NEXT:    vextractf128 $1, %ymm0, %xmm1
 ; X64-NEXT:    movl $2, %eax
 ; X64-NEXT:    vpinsrd $0, %eax, %xmm1, %xmm1
 ; X64-NEXT:    movl $-3, %eax
 ; X64-NEXT:    vpinsrd $1, %eax, %xmm1, %xmm1
 ; X64-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; X64-NEXT:    vcvtdq2ps %ymm0, %ymm0

test/CodeGen/X86/avx-intrinsics-fast-isel.ll

; X64-NEXT:    retq
   %res = and <8 x i32> %1, %2
   %bc = bitcast <8 x i32> %res to <8 x float>
   ret <8 x float> %bc
 }
 
 define <4 x double> @test_mm256_andnot_pd(<4 x double> %a0, <4 x double> %a1) nounwind {
 ; X32-LABEL: test_mm256_andnot_pd:
 ; X32:       # BB#0:
-; X32-NEXT:    vpcmpeqd %xmm2, %xmm2, %xmm2
-; X32-NEXT:    vinsertf128 $1, %xmm2, %ymm2, %ymm2
+; X32-NEXT:    vxorps %ymm2, %ymm2, %ymm2
+; X32-NEXT:    vcmptrueps %ymm2, %ymm2, %ymm2
 ; X32-NEXT:    vxorps %ymm2, %ymm0, %ymm0
 ; X32-NEXT:    vandps %ymm1, %ymm0, %ymm0
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_andnot_pd:
 ; X64:       # BB#0:
-; X64-NEXT:    vpcmpeqd %xmm2, %xmm2, %xmm2
-; X64-NEXT:    vinsertf128 $1, %xmm2, %ymm2, %ymm2
+; X64-NEXT:    vxorps %ymm2, %ymm2, %ymm2
+; X64-NEXT:    vcmptrueps %ymm2, %ymm2, %ymm2
 ; X64-NEXT:    vxorps %ymm2, %ymm0, %ymm0
 ; X64-NEXT:    vandps %ymm1, %ymm0, %ymm0
 ; X64-NEXT:    retq
   %1 = bitcast <4 x double> %a0 to <4 x i64>
   %2 = bitcast <4 x double> %a1 to <4 x i64>
   %3 = xor <4 x i64> %1, <i64 -1, i64 -1, i64 -1, i64 -1>
   %res = and <4 x i64> %3, %2
   %bc = bitcast <4 x i64> %res to <4 x double>

test/CodeGen/X86/pr28129.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=avx | FileCheck %s --check-prefix=X86
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx | FileCheck %s --check-prefix=X64
 
 define <4 x double> @cmp4f64_domain(<4 x double> %a) {
 ; X86-LABEL: cmp4f64_domain:
 ; X86:       # BB#0:
-; X86-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X86-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X86-NEXT:    vxorps %ymm1, %ymm1, %ymm1
+; X86-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X86-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: cmp4f64_domain:
 ; X64:       # BB#0:
-; X64-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X64-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X64-NEXT:    vxorps %ymm1, %ymm1, %ymm1
+; X64-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X64-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
 ; X64-NEXT:    retq
   %cmp = fcmp oeq <4 x double> zeroinitializer, zeroinitializer
   %sext = sext <4 x i1> %cmp to <4 x i64>
   %mask = bitcast <4 x i64> %sext to <4 x double>
   %add = fadd <4 x double> %a, %mask
   ret <4 x double> %add
 }
 
 define <4 x double> @cmp4f64_domain_optsize(<4 x double> %a) optsize {
 ; X86-LABEL: cmp4f64_domain_optsize:
 ; X86:       # BB#0:
-; X86-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X86-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X86-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X86-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: cmp4f64_domain_optsize:
 ; X64:       # BB#0:
-; X64-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X64-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X64-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X64-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
 ; X64-NEXT:    retq
   %cmp = fcmp oeq <4 x double> zeroinitializer, zeroinitializer
   %sext = sext <4 x i1> %cmp to <4 x i64>
   %mask = bitcast <4 x i64> %sext to <4 x double>
   %add = fadd <4 x double> %a, %mask
   ret <4 x double> %add
 }
 
 define <8 x float> @cmp8f32_domain(<8 x float> %a) {
 ; X86-LABEL: cmp8f32_domain:
 ; X86:       # BB#0:
-; X86-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X86-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X86-NEXT:    vxorps %ymm1, %ymm1, %ymm1
+; X86-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X86-NEXT:    vaddps %ymm1, %ymm0, %ymm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: cmp8f32_domain:
 ; X64:       # BB#0:
-; X64-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X64-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X64-NEXT:    vxorps %ymm1, %ymm1, %ymm1
+; X64-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X64-NEXT:    vaddps %ymm1, %ymm0, %ymm0
 ; X64-NEXT:    retq
   %cmp = fcmp oeq <8 x float> zeroinitializer, zeroinitializer
   %sext = sext <8 x i1> %cmp to <8 x i32>
   %mask = bitcast <8 x i32> %sext to <8 x float>
   %add = fadd <8 x float> %a, %mask
   ret <8 x float> %add
 }
 
 define <8 x float> @cmp8f32_domain_optsize(<8 x float> %a) optsize {
 ; X86-LABEL: cmp8f32_domain_optsize:
 ; X86:       # BB#0:
-; X86-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X86-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X86-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X86-NEXT:    vaddps %ymm1, %ymm0, %ymm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: cmp8f32_domain_optsize:
 ; X64:       # BB#0:
-; X64-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; X64-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; X64-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; X64-NEXT:    vaddps %ymm1, %ymm0, %ymm0
 ; X64-NEXT:    retq
   %cmp = fcmp oeq <8 x float> zeroinitializer, zeroinitializer
   %sext = sext <8 x i1> %cmp to <8 x i32>
   %mask = bitcast <8 x i32> %sext to <8 x float>
   %add = fadd <8 x float> %a, %mask
   ret <8 x float> %add
 }

test/CodeGen/X86/vector-pcmp.ll

 ;
 ; AVX1-LABEL: test_pcmpgtb_256:
 ; AVX1:       # BB#0:
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm1
 ; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX1-NEXT:    vpcmpgtb %xmm1, %xmm2, %xmm1
 ; AVX1-NEXT:    vpcmpgtb %xmm0, %xmm2, %xmm0
 ; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
-; AVX1-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; AVX1-NEXT:    vxorps %ymm1, %ymm1, %ymm1
+; AVX1-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; AVX1-NEXT:    vxorps %ymm1, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_pcmpgtb_256:
 ; AVX2:       # BB#0:
 ; AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; AVX2-NEXT:    vpcmpgtb %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    retq
   %sign = ashr <32 x i8> %x, <i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7>

 ; SSE-NEXT:    retq
 ;
 ; AVX1-LABEL: test_pcmpgtw_256:
 ; AVX1:       # BB#0:
 ; AVX1-NEXT:    vpsraw $15, %xmm0, %xmm1
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vpsraw $15, %xmm0, %xmm0
 ; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm0
-; AVX1-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; AVX1-NEXT:    vxorps %ymm1, %ymm1, %ymm1
+; AVX1-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; AVX1-NEXT:    vxorps %ymm1, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_pcmpgtw_256:
 ; AVX2:       # BB#0:
 ; AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; AVX2-NEXT:    vpcmpgtw %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    retq

 ; SSE-NEXT:    retq
 ;
 ; AVX1-LABEL: test_pcmpgtd_256:
 ; AVX1:       # BB#0:
 ; AVX1-NEXT:    vpsrad $31, %xmm0, %xmm1
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vpsrad $31, %xmm0, %xmm0
 ; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm0
-; AVX1-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; AVX1-NEXT:    vxorps %ymm1, %ymm1, %ymm1
+; AVX1-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; AVX1-NEXT:    vxorps %ymm1, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_pcmpgtd_256:
 ; AVX2:       # BB#0:
 ; AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; AVX2-NEXT:    vpcmpgtd %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    retq

 ; AVX1-LABEL: test_pcmpgtq_256:
 ; AVX1:       # BB#0:
 ; AVX1-NEXT:    vpsrad $31, %xmm0, %xmm1
 ; AVX1-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vpsrad $31, %xmm0, %xmm0
 ; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[1,1,3,3]
 ; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm0
-; AVX1-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm1, %ymm1
+; AVX1-NEXT:    vxorps %ymm1, %ymm1, %ymm1
+; AVX1-NEXT:    vcmptrueps %ymm1, %ymm1, %ymm1
 ; AVX1-NEXT:    vxorps %ymm1, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_pcmpgtq_256:
 ; AVX2:       # BB#0:
 ; AVX2-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
 ; AVX2-NEXT:    vpcmpgtq %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    retq

test/CodeGen/X86/xop-intrinsics-fast-isel.ll

; X64-NEXT:    retq
   %res = call <2 x i64> @llvm.x86.xop.vpcmov(<2 x i64> %a0, <2 x i64> %a1, <2 x i64> %a2)
   ret <2 x i64> %res
 }
 declare <2 x i64> @llvm.x86.xop.vpcmov(<2 x i64>, <2 x i64>, <2 x i64>) nounwind readnone
 
 define <4 x i64> @test_mm256_cmov_si256(<4 x i64> %a0, <4 x i64> %a1, <4 x i64> %a2) {
 ; X32-LABEL: test_mm256_cmov_si256:
 ; X32:       # BB#0:
-; X32-NEXT:    vpcmpeqd %xmm3, %xmm3, %xmm3
-; X32-NEXT:    vinsertf128 $1, %xmm3, %ymm3, %ymm3
+; X32-NEXT:    vxorps %ymm3, %ymm3, %ymm3
+; X32-NEXT:    vcmptrueps %ymm3, %ymm3, %ymm3
 ; X32-NEXT:    vxorps %ymm3, %ymm2, %ymm3
 ; X32-NEXT:    vandps %ymm2, %ymm0, %ymm0
 ; X32-NEXT:    vandps %ymm3, %ymm1, %ymm1
 ; X32-NEXT:    vorps %ymm1, %ymm0, %ymm0
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_cmov_si256:
 ; X64:       # BB#0:
-; X64-NEXT:    vpcmpeqd %xmm3, %xmm3, %xmm3
-; X64-NEXT:    vinsertf128 $1, %xmm3, %ymm3, %ymm3
+; X64-NEXT:    vxorps %ymm3, %ymm3, %ymm3
+; X64-NEXT:    vcmptrueps %ymm3, %ymm3, %ymm3
 ; X64-NEXT:    vxorps %ymm3, %ymm2, %ymm3
 ; X64-NEXT:    vandps %ymm2, %ymm0, %ymm0
 ; X64-NEXT:    vandps %ymm3, %ymm1, %ymm1
 ; X64-NEXT:    vorps %ymm1, %ymm0, %ymm0
 ; X64-NEXT:    retq
   %res = call <4 x i64> @llvm.x86.xop.vpcmov.256(<4 x i64> %a0, <4 x i64> %a1, <4 x i64> %a2)
   ret <4 x i64> %res
 }