[DAGCombiner] loosen restriction for creating narrow vector load from extract(wide load)

[spatel]

Summary

This is a follow-up to the change in D33578 that introduced this transform:
(extract_subvector (load wide vector)) --> (load narrow vector)

[niravd]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

14674

This is condition looks like it's going to trigger differently on different indexes combining consecutive subvectors extracted from the same larger vector. This is why the vec_int_to_fp case still has a load to zmm0 (see other comment)

It seems like what we'd like to do is check that all uses of Ld are cheap if there is more than one use (and then convert all uses simulatenously). That said, I think checking freeness/cheapness for each possible ExtIdxValue is the way to go.

test/CodeGen/X86/vec_int_to_fp.ll

3669

We're only partially converting the load-extracts here. there should only be a load to zmmX and extracts or 4 direct loads to xmmX.

[spatel]

test/CodeGen/X86/vec_int_to_fp.ll

3669

Agreed - that's what I meant in the description when I said that these diffs might be seen as bugs in isExtractSubvectorCheap().

In this case, x86 has made it cheap to extract from index 0 or one other index:

return (Index == 0 || Index == ResVT.getVectorNumElements());

Clearly, this was only tested with cases where we are extracting a half-sized vector. So it misses 2 out of the N/4 possibilities for AVX512 in this test.

I think this change is still an improvement (but not ideal of course), but my goal with this patch was really to answer the questions for the non-x86 diffs. I could just skip this step and post the more liberal patch with more test diffs if that seems better.

[tstellar]

All of the AMDGPU test changes are regressions.

[niravd]

It looks like most of the AMDGPU cases fail because:

• TLI.isExtractSubvectorCheap(VT, ExtIdxValue) is not defined for AMDGPU.
• Legalization breaks sign-/zero-extended vectors into a concat of smaller subvectors.

The former seems easy for someone who knows AMDGPU to correct.

[spatel]

In D33866#772755, @niravd wrote:

It looks like most of the AMDGPU cases fail because:

• TLI.isExtractSubvectorCheap(VT, ExtIdxValue) is not defined for AMDGPU.
• Legalization breaks sign-/zero-extended vectors into a concat of smaller subvectors.

The former seems easy for someone who knows AMDGPU to correct.

Actually, I see another way out. I missed this TLI hook:

// Return true if it is profitable to reduce the given load node to a smaller
// type.
//
// e.g. (i16 (trunc (i32 (load x))) -> i16 load x should be performed
virtual bool shouldReduceLoadWidth(SDNode *Load,
                                   ISD::LoadExtType ExtTy,
                                   EVT NewVT) const {
  return true;
}

This was originally added for AMDGPU (rL224084), so that should prevent the regressions.

[spatel]

Patch updated:

1. Remove the one-use restriction.
2. Add the TLI..shouldReduceLoadWidth() predicate.

So now we see the full effect on x86, sidestep the AMDGPU problems, but seem to have introduced some ARM regressions.

AFAICT, the x86 diffs are all wins. This includes an improvement to select non-temporal loads where we failed to do so before.

[spatel]

Patch updated:
Rebased after rL304718 - the AVX1 non-temporal isel got fixed there, so now we just see different scheduling in those tests.

[efriedma]

The diffs to the ARM tests are clearly no good: you're splitting 128-bit vector loads into two 64-bit vector loads for no benefit.

You're generating fewer instructions on x86, but it's not obvious it's beneficial; you get rid of the EXTRACT_SUBVECTOR operations, but the end result is a lot more instructions with memory operands.

test/CodeGen/AArch64/arm64-vabs.ll

141

We need to generate more complete checks for these tests... but I would guess this is adding extra instructions.

[RKSimon]

is this still relevant? abandon?

[spatel]

In D33866#1653879, @RKSimon wrote:

is this still relevant? abandon?

Abandoning. It's too big of a change even with the predicating TLI hook. We've probably already improved some of the x86 tests with other patches.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

 /// If we are extracting a subvector from a wide vector load, convert to a
 /// narrow load to eliminate the extraction:
 /// (extract_subvector (load wide vector)) --> (load narrow vector)
 static SDValue narrowExtractedVectorLoad(SDNode *Extract, SelectionDAG &DAG) {
   // TODO: Add support for big-endian. The offset calculation must be adjusted.
   if (DAG.getDataLayout().isBigEndian())
     return SDValue();
 
-  // TODO: The one-use check is overly conservative. Check the cost of the
-  // extract instead or remove that condition entirely.
   auto *Ld = dyn_cast<LoadSDNode>(Extract->getOperand(0));
   auto *ExtIdx = dyn_cast<ConstantSDNode>(Extract->getOperand(1));
-  if (!Ld || !Ld->hasOneUse() || Ld->getExtensionType() || Ld->isVolatile() ||
-      !ExtIdx)
+  if (!Ld || Ld->getExtensionType() || Ld->isVolatile() || !ExtIdx)
+    return SDValue();
+
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+  unsigned ExtIdxValue = ExtIdx->getZExtValue();
+  EVT VT = Extract->getValueType(0);
+
+  // TODO: It would be better to ask if the extract is free, rather than cheap,
+  // or just eliminate this check entirely. Using a narrower load directly
+  // reduces the dependency chain and may reduce register pressure.
+  if (!Ld->hasOneUse() && TLI.isExtractSubvectorCheap(VT, ExtIdxValue))
     return SDValue();
 
   // The narrow load will be offset from the base address of the old load if
   // we are extracting from something besides index 0 (little-endian).
-  EVT VT = Extract->getValueType(0);
   SDLoc DL(Extract);
   SDValue BaseAddr = Ld->getOperand(1);
-  unsigned Offset = ExtIdx->getZExtValue() * VT.getScalarType().getStoreSize();
+  unsigned Offset = ExtIdxValue * VT.getScalarType().getStoreSize();
 
   // TODO: Use "BaseIndexOffset" to make this more effective.
   SDValue NewAddr = DAG.getMemBasePlusOffset(BaseAddr, Offset, DL);
   MachineFunction &MF = DAG.getMachineFunction();
   MachineMemOperand *MMO = MF.getMachineMemOperand(Ld->getMemOperand(), Offset,
                                                    VT.getStoreSize());
   SDValue NewLd = DAG.getLoad(VT, DL, Ld->getChain(), NewAddr, MMO);
 

test/CodeGen/AArch64/arm64-vabs.ll

;CHECK: uabdl.2d
   %tmp4 = zext <2 x i32> %tmp3 to <2 x i64>
   ret <2 x i64> %tmp4
 }
 
 declare i16 @llvm.experimental.vector.reduce.add.i16.v16i16(<16 x i16>)
 
 define i16 @uabdl8h_rdx(<16 x i8>* %a, <16 x i8>* %b) {
 ; CHECK-LABEL: uabdl8h_rdx
-; CHECK: uabdl2.8h
+; CHECK: uabdl.8h
 ; CHECK: uabdl.8h
   %aload = load <16 x i8>, <16 x i8>* %a, align 1
   %bload = load <16 x i8>, <16 x i8>* %b, align 1
   %aext = zext <16 x i8> %aload to <16 x i16>
   %bext = zext <16 x i8> %bload to <16 x i16>
   %abdiff = sub nsw <16 x i16> %aext, %bext
   %abcmp = icmp slt <16 x i16> %abdiff, zeroinitializer
   %ababs = sub nsw <16 x i16> zeroinitializer, %abdiff
   %absel = select <16 x i1> %abcmp, <16 x i16> %ababs, <16 x i16> %abdiff
   %reduced_v = call i16 @llvm.experimental.vector.reduce.add.i16.v16i16(<16 x i16> %absel)
   ret i16 %reduced_v
 }
 
 declare i32 @llvm.experimental.vector.reduce.add.i32.v8i32(<8 x i32>)
 
 define i32 @uabdl4s_rdx(<8 x i16>* %a, <8 x i16>* %b) {
 ; CHECK-LABEL: uabdl4s_rdx
-; CHECK: uabdl2.4s
+; CHECK: uabdl.4s
 ; CHECK: uabdl.4s
   %aload = load <8 x i16>, <8 x i16>* %a, align 1
   %bload = load <8 x i16>, <8 x i16>* %b, align 1
   %aext = zext <8 x i16> %aload to <8 x i32>
   %bext = zext <8 x i16> %bload to <8 x i32>
   %abdiff = sub nsw <8 x i32> %aext, %bext
   %abcmp = icmp slt <8 x i32> %abdiff, zeroinitializer
   %ababs = sub nsw <8 x i32> zeroinitializer, %abdiff
   %absel = select <8 x i1> %abcmp, <8 x i32> %ababs, <8 x i32> %abdiff
   %reduced_v = call i32 @llvm.experimental.vector.reduce.add.i32.v8i32(<8 x i32> %absel)
   ret i32 %reduced_v
 }
 
 declare i64 @llvm.experimental.vector.reduce.add.i64.v4i64(<4 x i64>)
 
 define i64 @uabdl2d_rdx(<4 x i32>* %a, <4 x i32>* %b, i32 %h) {
 ; CHECK: uabdl2d_rdx
-; CHECK: uabdl2.2d
+; CHECK: uabdl.2d
 ; CHECK: uabdl.2d
   %aload = load <4 x i32>, <4 x i32>* %a, align 1
   %bload = load <4 x i32>, <4 x i32>* %b, align 1
   %aext = zext <4 x i32> %aload to <4 x i64>
   %bext = zext <4 x i32> %bload to <4 x i64>
   %abdiff = sub nsw <4 x i64> %aext, %bext
   %abcmp = icmp slt <4 x i64> %abdiff, zeroinitializer
   %ababs = sub nsw <4 x i64> zeroinitializer, %abdiff

test/CodeGen/AArch64/merge-store.ll

 ; RUN: llc < %s -mtriple=aarch64-unknown-unknown -mcpu=cyclone | FileCheck %s --check-prefix=CYCLONE --check-prefix=CHECK
 ; RUN: llc < %s -mtriple=aarch64-eabi -mattr=-slow-misaligned-128store | FileCheck %s --check-prefix=MISALIGNED --check-prefix=CHECK
 
 @g0 = external global <3 x float>, align 16
 @g1 = external global <3 x float>, align 4
 
-; CHECK: ldr q[[R0:[0-9]+]], {{\[}}[[R1:x[0-9]+]], :lo12:g0
+; CHECK: ldr d[[R0:[0-9]+]], {{\[}}[[R1:x[0-9]+]], :lo12:g0
 ; CHECK: str d[[R0]]
 
 define void @blam() {
   %tmp4 = getelementptr inbounds <3 x float>, <3 x float>* @g1, i64 0, i64 0
   %tmp5 = load <3 x float>, <3 x float>* @g0, align 16
   %tmp6 = extractelement <3 x float> %tmp5, i64 0
   store float %tmp6, float* %tmp4
   %tmp7 = getelementptr inbounds float, float* %tmp4, i64 1

test/CodeGen/AMDGPU/half.ll

 define amdgpu_kernel void @global_truncstore_v3f32_to_v3f16(<3 x half> addrspace(1)* %out, <3 x float> addrspace(1)* %in) #0 {
   %val = load <3 x float>, <3 x float> addrspace(1)* %in
   %cvt = fptrunc <3 x float> %val to <3 x half>
   store <3 x half> %cvt, <3 x half> addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}global_truncstore_v4f32_to_v4f16:
-; GCN: buffer_load_dwordx4
+; GCN: buffer_load_dwordx2
+; GCN: buffer_load_dwordx2
 ; GCN-DAG: v_cvt_f16_f32_e32
 ; SI-DAG:  v_cvt_f16_f32_e32
 ; SI-DAG:  v_cvt_f16_f32_e32
 ; VI-DAG:  v_cvt_f16_f32_sdwa
 ; VI-DAG:  v_cvt_f16_f32_sdwa
 ; GCN-DAG: v_cvt_f16_f32_e32
 ; GCN: buffer_store_dwordx2
 ; GCN: s_endpgm
 define amdgpu_kernel void @global_truncstore_v4f32_to_v4f16(<4 x half> addrspace(1)* %out, <4 x float> addrspace(1)* %in) #0 {
   %val = load <4 x float>, <4 x float> addrspace(1)* %in
   %cvt = fptrunc <4 x float> %val to <4 x half>
   store <4 x half> %cvt, <4 x half> addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}global_truncstore_v8f32_to_v8f16:
-; GCN: buffer_load_dwordx4
-; GCN: buffer_load_dwordx4
+; GCN: buffer_load_dwordx2
+; GCN: buffer_load_dwordx2
+; GCN: buffer_load_dwordx2
+; GCN: buffer_load_dwordx2
 ; SI:  v_cvt_f16_f32_e32
 ; SI:  v_cvt_f16_f32_e32
 ; SI:  v_cvt_f16_f32_e32
 ; SI:  v_cvt_f16_f32_e32
 ; SI:  v_cvt_f16_f32_e32
 ; SI:  v_cvt_f16_f32_e32
 ; SI:  v_cvt_f16_f32_e32
 ; SI:  v_cvt_f16_f32_e32

 define amdgpu_kernel void @global_truncstore_v8f32_to_v8f16(<8 x half> addrspace(1)* %out, <8 x float> addrspace(1)* %in) #0 {
   %val = load <8 x float>, <8 x float> addrspace(1)* %in
   %cvt = fptrunc <8 x float> %val to <8 x half>
   store <8 x half> %cvt, <8 x half> addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}global_truncstore_v16f32_to_v16f16:
-; GCN: buffer_load_dwordx4
-; GCN: buffer_load_dwordx4
-; GCN: buffer_load_dwordx4
-; GCN: buffer_load_dwordx4
+; GCN: buffer_load_dwordx2
+; GCN: buffer_load_dwordx2
+; GCN: buffer_load_dwordx2
+; GCN: buffer_load_dwordx2
+; GCN: buffer_load_dwordx2
+; GCN: buffer_load_dwordx2
+; GCN: buffer_load_dwordx2
+; GCN: buffer_load_dwordx2
 ; GCN-DAG: v_cvt_f16_f32_e32
 ; GCN-DAG: v_cvt_f16_f32_e32
 ; GCN-DAG: v_cvt_f16_f32_e32
 ; GCN-DAG: v_cvt_f16_f32_e32
 ; GCN-DAG: v_cvt_f16_f32_e32
 ; GCN-DAG: v_cvt_f16_f32_e32
 ; GCN-DAG: v_cvt_f16_f32_e32
 ; GCN-DAG: v_cvt_f16_f32_e32

test/CodeGen/AMDGPU/load-constant-i32.ll

entry:
   %ld = load <2 x i32>, <2 x i32> addrspace(2)* %in
   store <2 x i32> %ld, <2 x i32> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_load_v3i32:
 ; GCN: s_load_dwordx4
 
-; EG: VTX_READ_128
+; EG: VTX_READ_64
+; EG: VTX_READ_32
 define amdgpu_kernel void @constant_load_v3i32(<3 x i32> addrspace(1)* %out, <3 x i32> addrspace(2)* %in) #0 {
 entry:
   %ld = load <3 x i32>, <3 x i32> addrspace(2)* %in
   store <3 x i32> %ld, <3 x i32> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_load_v4i32:

 define amdgpu_kernel void @constant_sextload_v2i32_to_v2i64(<2 x i64> addrspace(1)* %out, <2 x i32> addrspace(2)* %in) #0 {
   %ld = load <2 x i32>, <2 x i32> addrspace(2)* %in
   %ext = sext <2 x i32> %ld to <2 x i64>
   store <2 x i64> %ext, <2 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_zextload_v4i32_to_v4i64:
-; GCN: s_load_dwordx4
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
 
 ; GCN: store_dwordx4
 ; GCN: store_dwordx4
 define amdgpu_kernel void @constant_zextload_v4i32_to_v4i64(<4 x i64> addrspace(1)* %out, <4 x i32> addrspace(2)* %in) #0 {
   %ld = load <4 x i32>, <4 x i32> addrspace(2)* %in
   %ext = zext <4 x i32> %ld to <4 x i64>
   store <4 x i64> %ext, <4 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_sextload_v4i32_to_v4i64:
-; GCN: s_load_dwordx4
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
 
 ; GCN: s_ashr_i32
 ; GCN: s_ashr_i32
 ; GCN: s_ashr_i32
 ; GCN: s_ashr_i32
 
 ; GCN: store_dwordx4
 ; GCN: store_dwordx4
 define amdgpu_kernel void @constant_sextload_v4i32_to_v4i64(<4 x i64> addrspace(1)* %out, <4 x i32> addrspace(2)* %in) #0 {
   %ld = load <4 x i32>, <4 x i32> addrspace(2)* %in
   %ext = sext <4 x i32> %ld to <4 x i64>
   store <4 x i64> %ext, <4 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_zextload_v8i32_to_v8i64:
-; GCN: s_load_dwordx8
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
 
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 
 ; GCN-HSA-DAG: flat_store_dwordx4
 ; GCN-HSA-DAG: flat_store_dwordx4
 ; GCN-SA-DAG: flat_store_dwordx4
 ; GCN-HSA-DAG: flat_store_dwordx4
 define amdgpu_kernel void @constant_zextload_v8i32_to_v8i64(<8 x i64> addrspace(1)* %out, <8 x i32> addrspace(2)* %in) #0 {
   %ld = load <8 x i32>, <8 x i32> addrspace(2)* %in
   %ext = zext <8 x i32> %ld to <8 x i64>
   store <8 x i64> %ext, <8 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_sextload_v8i32_to_v8i64:
-; GCN: s_load_dwordx8
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
 
 ; GCN: s_ashr_i32
 ; GCN: s_ashr_i32
 ; GCN: s_ashr_i32
 ; GCN: s_ashr_i32
 ; GCN: s_ashr_i32
 ; GCN: s_ashr_i32
 ; GCN: s_ashr_i32

 define amdgpu_kernel void @constant_sextload_v8i32_to_v8i64(<8 x i64> addrspace(1)* %out, <8 x i32> addrspace(2)* %in) #0 {
   %ld = load <8 x i32>, <8 x i32> addrspace(2)* %in
   %ext = sext <8 x i32> %ld to <8 x i64>
   store <8 x i64> %ext, <8 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_sextload_v16i32_to_v16i64:
-; GCN: s_load_dwordx16
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
 
 
 ; GCN-DAG: s_ashr_i32
 
 ; GCN: store_dwordx4
 ; GCN: store_dwordx4
 ; GCN: store_dwordx4
 ; GCN: store_dwordx4
 ; GCN: store_dwordx4
 ; GCN: store_dwordx4
 ; GCN: store_dwordx4
 ; GCN: store_dwordx4
 define amdgpu_kernel void @constant_sextload_v16i32_to_v16i64(<16 x i64> addrspace(1)* %out, <16 x i32> addrspace(2)* %in) #0 {
   %ld = load <16 x i32>, <16 x i32> addrspace(2)* %in
   %ext = sext <16 x i32> %ld to <16 x i64>
   store <16 x i64> %ext, <16 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_zextload_v16i32_to_v16i64
-; GCN: s_load_dwordx16
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
 
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 
 ; GCN-HSA: flat_store_dwordx4
 ; GCN-HSA: flat_store_dwordx4
 ; GCN-HSA: flat_store_dwordx4
 ; GCN-HSA: flat_store_dwordx4
 ; GCN-HSA: flat_store_dwordx4
 ; GCN-HSA: flat_store_dwordx4
-; GCN-HSA: flat_store_dwordx4
-; GCN-HSA: flat_store_dwordx4
 define amdgpu_kernel void @constant_zextload_v16i32_to_v16i64(<16 x i64> addrspace(1)* %out, <16 x i32> addrspace(2)* %in) #0 {
   %ld = load <16 x i32>, <16 x i32> addrspace(2)* %in
   %ext = zext <16 x i32> %ld to <16 x i64>
   store <16 x i64> %ext, <16 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_sextload_v32i32_to_v32i64:
 
-; GCN: s_load_dwordx16
-; GCN-DAG: s_load_dwordx16
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN-DAG: s_load_dwordx2
+; GCN-DAG: s_load_dwordx2
+; GCN-DAG: s_load_dwordx2
+; GCN-DAG: s_load_dwordx2
+; GCN-DAG: s_load_dwordx2
+; GCN-DAG: s_load_dwordx2
+; GCN-DAG: s_load_dwordx2
+; GCN-DAG: s_load_dwordx2
 
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4

 define amdgpu_kernel void @constant_sextload_v32i32_to_v32i64(<32 x i64> addrspace(1)* %out, <32 x i32> addrspace(2)* %in) #0 {
   %ld = load <32 x i32>, <32 x i32> addrspace(2)* %in
   %ext = sext <32 x i32> %ld to <32 x i64>
   store <32 x i64> %ext, <32 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_zextload_v32i32_to_v32i64:
-; GCN: s_load_dwordx16
-; GCN: s_load_dwordx16
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
+; GCN: s_load_dwordx2
 
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4

test/CodeGen/AMDGPU/load-global-f32.ll

entry:
   store <2 x float> %tmp0, <2 x float> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_load_v3f32:
 ; GCN-NOHSA: buffer_load_dwordx4
 ; GCN-HSA: flat_load_dwordx4
 
-; R600: VTX_READ_128
+; R600: VTX_READ_64
+; R600: VTX_READ_32
 define amdgpu_kernel void @global_load_v3f32(<3 x float> addrspace(1)* %out, <3 x float> addrspace(1)* %in) #0 {
 entry:
   %tmp0 = load <3 x float>, <3 x float> addrspace(1)* %in
   store <3 x float> %tmp0, <3 x float> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_load_v4f32:

test/CodeGen/AMDGPU/load-global-i32.ll

entry:
   store <2 x i32> %ld, <2 x i32> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_load_v3i32:
 ; GCN-NOHSA: buffer_load_dwordx4
 ; GCN-HSA: flat_load_dwordx4
 
-; EG: VTX_READ_128
+; EG: VTX_READ_64
+; EG: VTX_READ_32
 define amdgpu_kernel void @global_load_v3i32(<3 x i32> addrspace(1)* %out, <3 x i32> addrspace(1)* %in) #0 {
 entry:
   %ld = load <3 x i32>, <3 x i32> addrspace(1)* %in
   store <3 x i32> %ld, <3 x i32> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_load_v4i32:

 define amdgpu_kernel void @global_sextload_v2i32_to_v2i64(<2 x i64> addrspace(1)* %out, <2 x i32> addrspace(1)* %in) #0 {
   %ld = load <2 x i32>, <2 x i32> addrspace(1)* %in
   %ext = sext <2 x i32> %ld to <2 x i64>
   store <2 x i64> %ext, <2 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_zextload_v4i32_to_v4i64:
-; GCN-NOHSA: buffer_load_dwordx4
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 
-; GCN-HSA: flat_load_dwordx4
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
 ; GCN-HSA: flat_store_dwordx4
 ; GCN-HSA: flat_store_dwordx4
 define amdgpu_kernel void @global_zextload_v4i32_to_v4i64(<4 x i64> addrspace(1)* %out, <4 x i32> addrspace(1)* %in) #0 {
   %ld = load <4 x i32>, <4 x i32> addrspace(1)* %in
   %ext = zext <4 x i32> %ld to <4 x i64>
   store <4 x i64> %ext, <4 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_sextload_v4i32_to_v4i64:
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
 
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 
 ; GCN-HSA-DAG: flat_store_dwordx4
 ; GCN-HSA-DAG: flat_store_dwordx4
 define amdgpu_kernel void @global_sextload_v4i32_to_v4i64(<4 x i64> addrspace(1)* %out, <4 x i32> addrspace(1)* %in) #0 {
   %ld = load <4 x i32>, <4 x i32> addrspace(1)* %in
   %ext = sext <4 x i32> %ld to <4 x i64>
   store <4 x i64> %ext, <4 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_zextload_v8i32_to_v8i64:
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
 
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
 
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 
 ; GCN-HSA-DAG: flat_store_dwordx4
 ; GCN-HSA-DAG: flat_store_dwordx4
 ; GCN-SA-DAG: flat_store_dwordx4
 ; GCN-HSA-DAG: flat_store_dwordx4
 define amdgpu_kernel void @global_zextload_v8i32_to_v8i64(<8 x i64> addrspace(1)* %out, <8 x i32> addrspace(1)* %in) #0 {
   %ld = load <8 x i32>, <8 x i32> addrspace(1)* %in
   %ext = zext <8 x i32> %ld to <8 x i64>
   store <8 x i64> %ext, <8 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_sextload_v8i32_to_v8i64:
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
 
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
 
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32

 define amdgpu_kernel void @global_sextload_v8i32_to_v8i64(<8 x i64> addrspace(1)* %out, <8 x i32> addrspace(1)* %in) #0 {
   %ld = load <8 x i32>, <8 x i32> addrspace(1)* %in
   %ext = sext <8 x i32> %ld to <8 x i64>
   store <8 x i64> %ext, <8 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_sextload_v16i32_to_v16i64:
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
 
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
 
 
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-HSA-DAG: flat_store_dwordx4

 define amdgpu_kernel void @global_sextload_v16i32_to_v16i64(<16 x i64> addrspace(1)* %out, <16 x i32> addrspace(1)* %in) #0 {
   %ld = load <16 x i32>, <16 x i32> addrspace(1)* %in
   %ext = sext <16 x i32> %ld to <16 x i64>
   store <16 x i64> %ext, <16 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_zextload_v16i32_to_v16i64
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
 
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
 
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
define amdgpu_kernel void @global_zextload_v16i32_to_v16i64(<16 x i64> addrspace(1)* %out, <16 x i32> addrspace(1)* %in) #0 {
   %ld = load <16 x i32>, <16 x i32> addrspace(1)* %in
   %ext = zext <16 x i32> %ld to <16 x i64>
   store <16 x i64> %ext, <16 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_sextload_v32i32_to_v32i64:
 
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA-DAG: buffer_load_dwordx4
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA-DAG: buffer_load_dwordx2
+; GCN-NOHSA-DAG: buffer_load_dwordx2
 
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
 
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32
 ; GCN-DAG: v_ashrrev_i32

 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 ; GCN-NOHSA: buffer_store_dwordx4
 
-; GCN-HSA: flat_store_dwordx4
-; GCN-HSA: flat_store_dwordx4
-; GCN-HSA: flat_store_dwordx4
-; GCN-HSA: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
 
-; GCN-HSA: flat_store_dwordx4
-; GCN-HSA: flat_store_dwordx4
-; GCN-HSA: flat_store_dwordx4
-; GCN-HSA: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
 
-; GCN-HSA: flat_store_dwordx4
-; GCN-HSA: flat_store_dwordx4
-; GCN-HSA: flat_store_dwordx4
-; GCN-HSA: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
+; GCN-HSA-DAG: flat_store_dwordx4
 
 ; GCN-HSA: flat_store_dwordx4
 ; GCN-HSA: flat_store_dwordx4
 ; GCN-HSA: flat_store_dwordx4
 ; GCN-HSA: flat_store_dwordx4
 
 define amdgpu_kernel void @global_sextload_v32i32_to_v32i64(<32 x i64> addrspace(1)* %out, <32 x i32> addrspace(1)* %in) #0 {
   %ld = load <32 x i32>, <32 x i32> addrspace(1)* %in
   %ext = sext <32 x i32> %ld to <32 x i64>
   store <32 x i64> %ext, <32 x i64> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_zextload_v32i32_to_v32i64:
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
-; GCN-NOHSA: buffer_load_dwordx4
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
+; GCN-NOHSA: buffer_load_dwordx2
 
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
-; GCN-HSA: flat_load_dwordx4
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
+; GCN-HSA: flat_load_dwordx2
 
 
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 ; GCN-NOHSA-DAG: buffer_store_dwordx4
 
 ; GCN-NOHSA-DAG: buffer_store_dwordx4

test/CodeGen/AMDGPU/sad.ll

define amdgpu_kernel void @v_sad_u32_vector_pat1(<4 x i32> addrspace(1)* %out, <4 x i32> %a, <4 x i32> %b, <4 x i32> %c) {
 
   %ret0 = sub <4 x i32> %t0, %t1
   %ret = add <4 x i32> %ret0, %c
 
   store <4 x i32> %ret, <4 x i32> addrspace(1)* %out
   ret void
 }
 
+; FIXME: This should lower to sad?
+
 ; GCN-LABEL: {{^}}v_sad_u32_vector_pat2:
-; GCN: v_sad_u32 v{{[0-9]+}}, s{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GCN: v_sad_u32 v{{[0-9]+}}, s{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GCN: v_sad_u32 v{{[0-9]+}}, s{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
-; GCN: v_sad_u32 v{{[0-9]+}}, s{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
+; GCN: v_cmp_gt_u32_e32 vcc, s{{[0-9]+}}, v{{[0-9]+}}
+; GCN: v_cmp_gt_u32_e64 s[{{[0-9]+}}:{{[0-9]+}}], s{{[0-9]+}}, v{{[0-9]+}}
+; GCN: v_cmp_gt_u32_e64 s[{{[0-9]+}}:{{[0-9]+}}], s{{[0-9]+}}, v{{[0-9]+}}
+; GCN: v_cmp_gt_u32_e64 s[{{[0-9]+}}:{{[0-9]+}}], s{{[0-9]+}}, v{{[0-9]+}}
 define amdgpu_kernel void @v_sad_u32_vector_pat2(<4 x i32> addrspace(1)* %out, <4 x i32> %a, <4 x i32> %b, <4 x i32> %c) {
   %icmp0 = icmp ugt <4 x i32> %a, %b
   %sub0 = sub <4 x i32> %a, %b
   %sub1 = sub <4 x i32> %b, %a
   %ret0 = select <4 x i1> %icmp0, <4 x i32> %sub0, <4 x i32> %sub1
 
   %ret = add <4 x i32> %ret0, %c
 

test/CodeGen/AMDGPU/vectorize-global-local.ll

 ; RUN: llc -march=amdgcn -mcpu=tonga -verify-machineinstrs < %s | FileCheck %s
-; CHECK-DAG: flat_load_dwordx4
-; CHECK-DAG: flat_load_dwordx4
-; CHECK-DAG: flat_load_dwordx4
-; CHECK-DAG: flat_load_dwordx4
+; CHECK-DAG: flat_load_dwordx2
+; CHECK-DAG: flat_load_dwordx2
+; CHECK-DAG: flat_load_dwordx2
+; CHECK-DAG: flat_load_dwordx2
+; CHECK-DAG: flat_load_dwordx2
+; CHECK-DAG: flat_load_dwordx2
+; CHECK-DAG: flat_load_dwordx2
+; CHECK-DAG: flat_load_dwordx2
 ; CHECK-DAG: ds_write2_b32
 ; CHECK-DAG: ds_write2_b32
 ; CHECK-DAG: ds_write2_b32
 ; CHECK-DAG: ds_write2_b32
 ; CHECK-DAG: ds_write2_b32
 ; CHECK-DAG: ds_write2_b32
 ; CHECK-DAG: ds_write2_b32
 ; CHECK-DAG: ds_write2_b32

test/CodeGen/X86/vec_int_to_fp.ll

 ; AVX2-NEXT:    vcvtsi2ssq %rax, %xmm4, %xmm0
 ; AVX2-NEXT:    vinsertps {{.*#+}} xmm0 = xmm2[0,1,2],xmm0[0]
 ; AVX2-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512F-LABEL: sitofp_load_8i64_to_8f32:
 ; AVX512F:       # BB#0:
 ; AVX512F-NEXT:    vmovdqa64 (%rdi), %zmm0
-; AVX512F-NEXT:    vextracti32x4 $2, %zmm0, %xmm1
+; AVX512F-NEXT:    vmovdqa 32(%rdi), %xmm1
 ; AVX512F-NEXT:    vpextrq $1, %xmm1, %rax
 ; AVX512F-NEXT:    vcvtsi2ssq %rax, %xmm2, %xmm2
 ; AVX512F-NEXT:    vmovq %xmm1, %rax
 ; AVX512F-NEXT:    vcvtsi2ssq %rax, %xmm3, %xmm1
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[2,3]
-; AVX512F-NEXT:    vextracti32x4 $3, %zmm0, %xmm2
+; AVX512F-NEXT:    vmovdqa 48(%rdi), %xmm2
 ; AVX512F-NEXT:    vmovq %xmm2, %rax
 ; AVX512F-NEXT:    vcvtsi2ssq %rax, %xmm3, %xmm3
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1],xmm3[0],xmm1[3]
 ; AVX512F-NEXT:    vpextrq $1, %xmm2, %rax
 ; AVX512F-NEXT:    vcvtsi2ssq %rax, %xmm4, %xmm2
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1,2],xmm2[0]
 ; AVX512F-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX512F-NEXT:    vcvtsi2ssq %rax, %xmm4, %xmm2
 ; AVX512F-NEXT:    vmovq %xmm0, %rax
 ; AVX512F-NEXT:    vcvtsi2ssq %rax, %xmm4, %xmm3
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm2 = xmm3[0],xmm2[0],xmm3[2,3]
 ; AVX512F-NEXT:    vextracti32x4 $1, %zmm0, %xmm0
 ; AVX512F-NEXT:    vmovq %xmm0, %rax
 ; AVX512F-NEXT:    vcvtsi2ssq %rax, %xmm4, %xmm3
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1],xmm3[0],xmm2[3]
 ; AVX512F-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX512F-NEXT:    vcvtsi2ssq %rax, %xmm4, %xmm0
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm0 = xmm2[0,1,2],xmm0[0]
 ; AVX512F-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; AVX512F-NEXT:    retq
 ;
 ; AVX512VL-LABEL: sitofp_load_8i64_to_8f32:
 ; AVX512VL:       # BB#0:
 ; AVX512VL-NEXT:    vmovdqa64 (%rdi), %zmm0
-; AVX512VL-NEXT:    vextracti32x4 $2, %zmm0, %xmm1
+; AVX512VL-NEXT:    vmovdqa 32(%rdi), %xmm1
 ; AVX512VL-NEXT:    vpextrq $1, %xmm1, %rax
 ; AVX512VL-NEXT:    vcvtsi2ssq %rax, %xmm2, %xmm2
 ; AVX512VL-NEXT:    vmovq %xmm1, %rax
 ; AVX512VL-NEXT:    vcvtsi2ssq %rax, %xmm3, %xmm1
 ; AVX512VL-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[2,3]
-; AVX512VL-NEXT:    vextracti32x4 $3, %zmm0, %xmm2
+; AVX512VL-NEXT:    vmovdqa 48(%rdi), %xmm2
 ; AVX512VL-NEXT:    vmovq %xmm2, %rax
 ; AVX512VL-NEXT:    vcvtsi2ssq %rax, %xmm3, %xmm3
 ; AVX512VL-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1],xmm3[0],xmm1[3]
 ; AVX512VL-NEXT:    vpextrq $1, %xmm2, %rax
 ; AVX512VL-NEXT:    vcvtsi2ssq %rax, %xmm4, %xmm2
 ; AVX512VL-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1,2],xmm2[0]
 ; AVX512VL-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX512VL-NEXT:    vcvtsi2ssq %rax, %xmm4, %xmm2

 ; AVX2-NEXT:  .LBB80_24:
 ; AVX2-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1,2],xmm2[0]
 ; AVX2-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512F-LABEL: uitofp_load_8i64_to_8f32:
 ; AVX512F:       # BB#0:
 ; AVX512F-NEXT:    vmovdqa64 (%rdi), %zmm0
-; AVX512F-NEXT:    vextracti32x4 $2, %zmm0, %xmm1
+; AVX512F-NEXT:    vmovdqa 32(%rdi), %xmm1
 ; AVX512F-NEXT:    vpextrq $1, %xmm1, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm2, %xmm2
 ; AVX512F-NEXT:    vmovq %xmm1, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm3, %xmm1
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[2,3]
-; AVX512F-NEXT:    vextracti32x4 $3, %zmm0, %xmm2
+; AVX512F-NEXT:    vmovdqa 48(%rdi), %xmm2
 ; AVX512F-NEXT:    vmovq %xmm2, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm3, %xmm3
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1],xmm3[0],xmm1[3]
 ; AVX512F-NEXT:    vpextrq $1, %xmm2, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm4, %xmm2
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1,2],xmm2[0]
 ; AVX512F-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm4, %xmm2
 ; AVX512F-NEXT:    vmovq %xmm0, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm4, %xmm3
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm2 = xmm3[0],xmm2[0],xmm3[2,3]
 ; AVX512F-NEXT:    vextracti32x4 $1, %zmm0, %xmm0
 ; AVX512F-NEXT:    vmovq %xmm0, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm4, %xmm3
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1],xmm3[0],xmm2[3]
 ; AVX512F-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm4, %xmm0
 ; AVX512F-NEXT:    vinsertps {{.*#+}} xmm0 = xmm2[0,1,2],xmm0[0]
 ; AVX512F-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; AVX512F-NEXT:    retq
 ;
 ; AVX512VL-LABEL: uitofp_load_8i64_to_8f32:
 ; AVX512VL:       # BB#0:
 ; AVX512VL-NEXT:    vmovdqa64 (%rdi), %zmm0
-; AVX512VL-NEXT:    vextracti32x4 $2, %zmm0, %xmm1
+; AVX512VL-NEXT:    vmovdqa 32(%rdi), %xmm1
 ; AVX512VL-NEXT:    vpextrq $1, %xmm1, %rax
 ; AVX512VL-NEXT:    vcvtusi2ssq %rax, %xmm2, %xmm2
 ; AVX512VL-NEXT:    vmovq %xmm1, %rax
 ; AVX512VL-NEXT:    vcvtusi2ssq %rax, %xmm3, %xmm1
 ; AVX512VL-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[2,3]
-; AVX512VL-NEXT:    vextracti32x4 $3, %zmm0, %xmm2
+; AVX512VL-NEXT:    vmovdqa 48(%rdi), %xmm2
 ; AVX512VL-NEXT:    vmovq %xmm2, %rax
 ; AVX512VL-NEXT:    vcvtusi2ssq %rax, %xmm3, %xmm3
 ; AVX512VL-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1],xmm3[0],xmm1[3]
 ; AVX512VL-NEXT:    vpextrq $1, %xmm2, %rax
 ; AVX512VL-NEXT:    vcvtusi2ssq %rax, %xmm4, %xmm2
 ; AVX512VL-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1,2],xmm2[0]
 ; AVX512VL-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX512VL-NEXT:    vcvtusi2ssq %rax, %xmm4, %xmm2