[LegalizeVectorTypes] Allow single loads and stores for more short vectors

[tauril]

Summary

When lowering a load or store for TypeWidenVector, the type legalizer
would use a single load or store if the associated integer type was legal
or promoted. E.g. it loads a v4i8 as an i32 if i32 is legal/promotable.
(See https://reviews.llvm.org/rL236528 for reference.)

This patch applies that behaviour to vector types. If the vector type
is TypePromoteInteger, the element type is going to be
TypePromoteInteger as well, which will lead to have a single promoting
load rather than N individual promoting loads. For instance, if we have
a v3i1, we would now have a load of v4i1 instead of 3 loads of i1.

I don't have any knowledge with AMDGPU and it seems that this commit
introduces some less performant code with the R600 architecture. I would
appreciate if someone knowledgeable with that architecture could enlighten
me on how bad the changes are.

[jvesely]

Hi,

sorry for the delay. afaik r600 does not do any special handling wrt to coalescing loads. There is a general load/store vectorizer by @arsenm, so it looks like this patch is interfering with it, but I'd expect the same to happen for GCN as well.
I'm OK with these pessimizations, R600 loads/stores have bigger problems.

jan

test/CodeGen/AMDGPU/load-constant-i16.ll

197 ↗

(On Diff #179849)

is there any reason to remove these lines from the test?

test/CodeGen/AMDGPU/load-global-i16.ll

201 ↗

(On Diff #179849)

have the extra moves been eliminated by this patch? if not, is there another reason to remove these lines?

222 ↗

(On Diff #179849)

same here

[tauril]

test/CodeGen/AMDGPU/load-constant-i16.ll

197 ↗

(On Diff #179849)

I removed these lines from the test because the generated code no longer has them in it.

test/CodeGen/AMDGPU/load-global-i16.ll

201 ↗

(On Diff #179849)

Exactly, the moves have been eliminated by the patch!

222 ↗

(On Diff #179849)

Same thing here, the patch removed these lines from the generated code.

[jvesely]

test/CodeGen/AMDGPU/load-constant-i16.ll

197 ↗

(On Diff #179849)

I see. If the values are directly loaded to registers used in STORE_RAW, it'd be nice to preserve the tracking instead of using generic {{T[0-9]\.[XYZW]}}
i.e:
VTX_READ_16 [[ST_LO]].X, ... 0, #1
VTX_READ_16 [[ST_LO]].Y, ... 2, #1
VTX_READ_16 [[ST_HI]].X, ... 4, #1

[tauril]

I modified some AMDGPU tests to track more registers where possible as @jvesely suggested, and I added some missing new relevant generated instructions (BFE_INT).

[jvesely]

In D56201#1356228, @tauril wrote:

I modified some AMDGPU tests to track more registers where possible as @jvesely suggested, and I added some missing new relevant generated instructions (BFE_INT).

thanks. the r600 test changes LGTM.
I can't comment on the x86 ones.

[lebedev.ri]

In D56201#1356312, @jvesely wrote:

In D56201#1356228, @tauril wrote:

I modified some AMDGPU tests to track more registers where possible as @jvesely suggested, and I added some missing new relevant generated instructions (BFE_INT).

I can't comment on the x86 ones.

test/CodeGen/X86/load-local-i1.ll should be precommitted i guess, so the change is actually visible.

[tauril]

I renamed the new test from test/CodeGen/X86/load-local-i1.ll to test/CodeGen/X86/load-local-v3i1.ll
because I made a mistake the first time.
I also precommitted that same new test so that we can see the difference that this patch introduces with it,
as suggested by @lebedev.ri.
The relevant information to notice is the removal of the negb instruction followed by the filling of the
arguments of the masked_load_v3 function with the same value (as it is explained in the beginning of
the test).

Because this updated patch now considers that the new test already existed, I don't know how this will be
handled if this patch gets accepted, when trying to commit it.

[tauril]

Ping.

[craig.topper]

test/CodeGen/X86/load-local-v3i1.ll

28 ↗

(On Diff #181601)

Can you add 'nounwind' to this to drop the .cfi lines

[tauril]

I added the nounwind attribute to the new test function's definition as @craig.topper suggested.

[craig.topper]

X86 changes look good to me

[tauril]

Ping @bogner.

I also want to point out I don't have commit access if the patch were to be accepted.

[arsenm]

The R600 code is worse, but it seems to just be for the 3x vector cases. I don't expect much out of r600, particularly for edge cases like this so I wouldn't worry about it too much

[tauril]

Ping @bogner.

[tauril]

If no one else has any remark regarding this patch, could someone with the rights to approve the patch? @bogner

[tauril]

Ping.

[tauril]

Ping.

[bogner]

Sorry for the delay, I somehow completely missed this. The change looks good - I'll commit it for you today or tomorrow.

[bogner]

I hit a couple of minor issues trying to apply the patch:

1. We hit an assertion in test/CodeGen/AMDGPU/kernel-args.ll. Can you take a look?
2. I don't have test/CodeGen/X86/load-local-v3i1.ll in my tree, so that part doesn't apply. Is it reasonable to drop this part of the change?
3. A couple of the X86 tests have changed slightly - these are easy enough to upgrade with update_llc_test_checks though.

[tauril]

Hello @bogner, thanks for accepting this patch!

1. We hit an assertion in test/CodeGen/AMDGPU/kernel-args.ll. Can you take a look?

The problem came from the R600ISelLowering where apparently the global addresspace is not supported for vector truncating stores. (@jvesely would it be possible that you have a look on the changes I made in R600ISelLowering.cpp to avoid the assert?). The failing test (v5i16_arg) was introduced in https://reviews.llvm.org/D58928 by @tpr.

2. I don't have test/CodeGen/X86/load-local-v3i1.ll in my tree, so that part doesn't apply. Is it reasonable to drop this part of the change?

About that, it was requested by @lebedev.ri to have the file pre-committed to see how my modifications changed it. I've now removed the pre-committed file so that it is created by this patch.

3. A couple of the X86 tests have changed slightly - these are easy enough to upgrade with update_llc_test_checks though.

I have updated the tests accordingly!

I also ran ninja check-llvm on a build with asserts activated to ensure everything passed.

llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

if (MemVT.getSizeInBits() <= WidenEltWidth)
       break;
     auto Action = TLI.getTypeAction(*DAG.getContext(), MemVT);
     if ((Action == TargetLowering::TypeLegal ||
          Action == TargetLowering::TypePromoteInteger) &&
         (WidenWidth % MemVTWidth) == 0 &&
         isPowerOf2_32(WidenWidth / MemVTWidth) &&
         (MemVTWidth <= Width ||
          (Align!=0 && MemVTWidth<=AlignInBits && MemVTWidth<=Width+WidenEx))) {
+      if (MemVTWidth == WidenWidth)
+        return MemVT;
       RetVT = MemVT;
       break;
     }
   }
 
   // See if there is a larger vector type to load/store that has the same vector
   // element type and is evenly divisible with the WidenVT.
   for (VT = (unsigned)MVT::LAST_VECTOR_VALUETYPE;
        VT >= (unsigned)MVT::FIRST_VECTOR_VALUETYPE; --VT) {
     EVT MemVT = (MVT::SimpleValueType) VT;
     unsigned MemVTWidth = MemVT.getSizeInBits();
-    if (TLI.isTypeLegal(MemVT) && WidenEltVT == MemVT.getVectorElementType() &&
+    auto Action = TLI.getTypeAction(*DAG.getContext(), MemVT);
+    if ((Action == TargetLowering::TypeLegal ||
+         Action == TargetLowering::TypePromoteInteger) &&
+        WidenEltVT == MemVT.getVectorElementType() &&
         (WidenWidth % MemVTWidth) == 0 &&
         isPowerOf2_32(WidenWidth / MemVTWidth) &&
         (MemVTWidth <= Width ||
          (Align!=0 && MemVTWidth<=AlignInBits && MemVTWidth<=Width+WidenEx))) {
       if (RetVT.getSizeInBits() < MemVTWidth || MemVT == WidenVT)
         return MemVT;
     }
   }

llvm/trunk/lib/Target/AMDGPU/R600ISelLowering.cpp

SDValue R600TargetLowering::LowerSTORE(SDValue Op, SelectionDAG &DAG) const {
   SDValue Value = StoreNode->getValue();
 
   EVT VT = Value.getValueType();
   EVT MemVT = StoreNode->getMemoryVT();
   EVT PtrVT = Ptr.getValueType();
 
   SDLoc DL(Op);
 
+  const bool TruncatingStore = StoreNode->isTruncatingStore();
+
   // Neither LOCAL nor PRIVATE can do vectors at the moment
-  if ((AS == AMDGPUAS::LOCAL_ADDRESS || AS == AMDGPUAS::PRIVATE_ADDRESS) &&
+  if ((AS == AMDGPUAS::LOCAL_ADDRESS || AS == AMDGPUAS::PRIVATE_ADDRESS ||
+       TruncatingStore) &&
       VT.isVector()) {
-    if ((AS == AMDGPUAS::PRIVATE_ADDRESS) &&
-         StoreNode->isTruncatingStore()) {
+    if ((AS == AMDGPUAS::PRIVATE_ADDRESS) && TruncatingStore) {
       // Add an extra level of chain to isolate this vector
       SDValue NewChain = DAG.getNode(AMDGPUISD::DUMMY_CHAIN, DL, MVT::Other, Chain);
       // TODO: can the chain be replaced without creating a new store?
       SDValue NewStore = DAG.getTruncStore(
           NewChain, DL, Value, Ptr, StoreNode->getPointerInfo(),
           MemVT, StoreNode->getAlignment(),
           StoreNode->getMemOperand()->getFlags(), StoreNode->getAAInfo());
       StoreNode = cast<StoreSDNode>(NewStore);
SDValue R600TargetLowering::LowerSTORE(SDValue Op, SelectionDAG &DAG) const {
   }
 
   SDValue DWordAddr = DAG.getNode(ISD::SRL, DL, PtrVT, Ptr,
                                   DAG.getConstant(2, DL, PtrVT));
 
   if (AS == AMDGPUAS::GLOBAL_ADDRESS) {
     // It is beneficial to create MSKOR here instead of combiner to avoid
     // artificial dependencies introduced by RMW
-    if (StoreNode->isTruncatingStore()) {
+    if (TruncatingStore) {
       assert(VT.bitsLE(MVT::i32));
       SDValue MaskConstant;
       if (MemVT == MVT::i8) {
         MaskConstant = DAG.getConstant(0xFF, DL, MVT::i32);
       } else {
         assert(MemVT == MVT::i16);
         assert(StoreNode->getAlignment() >= 2);
         MaskConstant = DAG.getConstant(0xFFFF, DL, MVT::i32);
if (TruncatingStore) {
       SDValue Args[3] = { Chain, Input, DWordAddr };
       return DAG.getMemIntrinsicNode(AMDGPUISD::STORE_MSKOR, DL,
                                      Op->getVTList(), Args, MemVT,
                                      StoreNode->getMemOperand());
     } else if (Ptr->getOpcode() != AMDGPUISD::DWORDADDR && VT.bitsGE(MVT::i32)) {
       // Convert pointer from byte address to dword address.
       Ptr = DAG.getNode(AMDGPUISD::DWORDADDR, DL, PtrVT, DWordAddr);
 
-      if (StoreNode->isTruncatingStore() || StoreNode->isIndexed()) {
-        llvm_unreachable("Truncated and indexed stores not supported yet");
+      if (StoreNode->isIndexed()) {
+        llvm_unreachable("Indexed stores not supported yet");
       } else {
         Chain = DAG.getStore(Chain, DL, Value, Ptr, StoreNode->getMemOperand());
       }
       return Chain;
     }
   }
 
   // GLOBAL_ADDRESS has been handled above, LOCAL_ADDRESS allows all sizes

llvm/trunk/test/CodeGen/AMDGPU/load-constant-i16.ll

entry:
   %ld = load <2 x i16>, <2 x i16> addrspace(4)* %in
   store <2 x i16> %ld, <2 x i16> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_load_v3i16:
 ; GCN: s_load_dwordx2 s
 
-; EG-DAG: VTX_READ_32 T{{[0-9]+}}.X, T{{[0-9]+}}.X, 0, #1
+; EG-DAG: VTX_READ_16 T{{[0-9]+}}.X, T{{[0-9]+}}.X, 0, #1
+; EG-DAG: VTX_READ_16 T{{[0-9]+}}.X, T{{[0-9]+}}.X, 2, #1
 ; EG-DAG: VTX_READ_16 T{{[0-9]+}}.X, T{{[0-9]+}}.X, 4, #1
 define amdgpu_kernel void @constant_load_v3i16(<3 x i16> addrspace(1)* %out, <3 x i16> addrspace(4)* %in) {
 entry:
   %ld = load <3 x i16>, <3 x i16> addrspace(4)* %in
   store <3 x i16> %ld, <3 x i16> addrspace(1)* %out
   ret void
 }
 

 
 ; FUNC-LABEL: {{^}}constant_zextload_v3i16_to_v3i32:
 ; GCN: s_load_dwordx2
 
 ; v3i16 is naturally 8 byte aligned
 ; EG-DAG: MEM_RAT_CACHELESS STORE_RAW [[ST_LO:T[0-9]]].XY, {{T[0-9].[XYZW]}},
 ; EG-DAG: MEM_RAT_CACHELESS STORE_RAW [[ST_HI:T[0-9]]].X, {{T[0-9].[XYZW]}},
 ; EG: CF_END
-; EG-DAG: VTX_READ_32 [[DST_LO:T[0-9]\.[XYZW]]], {{T[0-9]\.[XYZW]}}, 0, #1
-; EG-DAG: VTX_READ_16 [[DST_HI:T[0-9]\.[XYZW]]], {{T[0-9]\.[XYZW]}}, 4, #1
-; TODO: This should use DST, but for some there are redundant MOVs
-; EG-DAG: LSHR {{[* ]*}}[[ST_LO]].Y, {{T[0-9]\.[XYZW]}}, literal
+; EG-DAG: VTX_READ_16 [[ST_LO]].X, [[SRC:T[0-9]\.[XYZW]]], 0, #1
+; EG-DAG: VTX_READ_16 {{T[0-9]\.[XYZW]}}, [[SRC]], 2, #1
+; EG-DAG: VTX_READ_16 [[ST_HI]].X, [[SRC]], 4, #1
+; EG-DAG: LSHR {{[* ]*}}{{T[0-9]\.[XYZW]}}, {{T[0-9]\.[XYZW]}}, literal
 ; EG-DAG: 16
-; EG-DAG: AND_INT {{[* ]*}}[[ST_LO]].X, {{T[0-9]\.[XYZW]}}, literal
-; EG-DAG: AND_INT {{[* ]*}}[[ST_HI]].X, {{T[0-9]\.[XYZW]}}, literal
-; EG-DAG: 65535
-; EG-DAG: 65535
 define amdgpu_kernel void @constant_zextload_v3i16_to_v3i32(<3 x i32> addrspace(1)* %out, <3 x i16> addrspace(4)* %in) {
 entry:
   %ld = load <3 x i16>, <3 x i16> addrspace(4)* %in
   %ext = zext <3 x i16> %ld to <3 x i32>
   store <3 x i32> %ext, <3 x i32> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}constant_sextload_v3i16_to_v3i32:
 ; GCN: s_load_dwordx2
 
 ; EG-DAG: MEM_RAT_CACHELESS STORE_RAW [[ST_LO:T[0-9]]].XY, {{T[0-9].[XYZW]}},
 ; EG-DAG: MEM_RAT_CACHELESS STORE_RAW [[ST_HI:T[0-9]]].X, {{T[0-9].[XYZW]}},
 ; v3i16 is naturally 8 byte aligned
-; EG-DAG: VTX_READ_32 [[DST_HI:T[0-9]\.[XYZW]]], [[PTR:T[0-9]\.[XYZW]]], 0, #1
-; EG-DAG: VTX_READ_16 [[DST_LO:T[0-9]\.[XYZW]]], {{T[0-9]\.[XYZW]}}, 4, #1
-; EG-DAG: ASHR {{[* ]*}}[[ST_LO]].Y, {{T[0-9]\.[XYZW]}}, literal
-; EG-DAG: BFE_INT {{[* ]*}}[[ST_LO]].X, {{T[0-9]\.[XYZW]}}, 0.0, literal
-; EG-DAG: BFE_INT {{[* ]*}}[[ST_HI]].X, {{T[0-9]\.[XYZW]}}, 0.0, literal
+; EG-DAG: VTX_READ_16 [[ST_LO]].X, [[SRC:T[0-9]\.[XYZW]]], 0, #1
+; EG-DAG: VTX_READ_16 [[DST_MID:T[0-9]\.[XYZW]]], [[SRC]], 2, #1
+; EG-DAG: VTX_READ_16 [[ST_HI]].X, [[SRC]], 4, #1
+; EG-DAG: BFE_INT {{[* ]*}}[[ST_LO]].X, [[ST_LO]].X, 0.0, literal
+; EG-DAG: BFE_INT {{[* ]*}}[[ST_LO]].Y, [[DST_MID]], 0.0, literal
+; EG-DAG: BFE_INT {{[* ]*}}[[ST_HI]].X, [[ST_HI]].X, 0.0, literal
 ; EG-DAG: 16
 ; EG-DAG: 16
 define amdgpu_kernel void @constant_sextload_v3i16_to_v3i32(<3 x i32> addrspace(1)* %out, <3 x i16> addrspace(4)* %in) {
 entry:
   %ld = load <3 x i16>, <3 x i16> addrspace(4)* %in
   %ext = sext <3 x i16> %ld to <3 x i32>
   store <3 x i32> %ext, <3 x i32> addrspace(1)* %out
   ret void

llvm/trunk/test/CodeGen/AMDGPU/load-global-i16.ll

entry:
   store <2 x i16> %ld, <2 x i16> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_load_v3i16:
 ; GCN-NOHSA: buffer_load_dwordx2 v
 ; GCN-HSA: flat_load_dwordx2 v
 
-; EGCM-DAG: VTX_READ_32 T{{[0-9]+}}.X, T{{[0-9]+}}.X, 0, #1
+; EGCM-DAG: VTX_READ_16 T{{[0-9]+}}.X, T{{[0-9]+}}.X, 0, #1
+; EGCM-DAG: VTX_READ_16 T{{[0-9]+}}.X, T{{[0-9]+}}.X, 2, #1
 ; EGCM-DAG: VTX_READ_16 T{{[0-9]+}}.X, T{{[0-9]+}}.X, 4, #1
 define amdgpu_kernel void @global_load_v3i16(<3 x i16> addrspace(1)* %out, <3 x i16> addrspace(1)* %in) {
 entry:
   %ld = load <3 x i16>, <3 x i16> addrspace(1)* %in
   store <3 x i16> %ld, <3 x i16> addrspace(1)* %out
   ret void
 }
 

 ; FUNC-LABEL: {{^}}global_zextload_v3i16_to_v3i32:
 ; GCN-NOHSA: buffer_load_dwordx2
 ; GCN-HSA: flat_load_dwordx2
 
 ; CM: MEM_RAT_CACHELESS STORE_DWORD [[ST_LO:T[0-9]]], {{T[0-9]\.[XYZW]}}
 ; CM: MEM_RAT_CACHELESS STORE_DWORD [[ST_HI:T[0-9]]].X, {{T[0-9]\.[XYZW]}}
 ; EG: MEM_RAT_CACHELESS STORE_RAW [[ST_HI:T[0-9]]].X, {{T[0-9]\.[XYZW]}},
 ; EG: MEM_RAT_CACHELESS STORE_RAW [[ST_LO:T[0-9]]].XY, {{T[0-9]\.[XYZW]}},
-; EGCM-DAG: VTX_READ_32 [[DST_LO:T[0-9]\.[XYZW]]], {{T[0-9]\.[XYZW]}}, 0, #1
-; EGCM-DAG: VTX_READ_16 [[DST_HI:T[0-9]\.[XYZW]]], {{T[0-9]\.[XYZW]}}, 4, #1
-; TODO: This should use DST, but for some there are redundant MOVs
-; EGCM: LSHR {{[* ]*}}[[ST_LO]].Y, {{T[0-9]\.[XYZW]}}, literal
+; EGCM-DAG: VTX_READ_16 [[ST_LO]].X, [[SRC:T[0-9]\.[XYZW]]], 0, #1
+; EGCM-DAG: VTX_READ_16 {{T[0-9]\.[XYZW]}}, [[SRC]], 2, #1
+; EGCM-DAG: VTX_READ_16 [[ST_HI]].X, [[SRC]], 4, #1
 ; EGCM: 16
 define amdgpu_kernel void @global_zextload_v3i16_to_v3i32(<3 x i32> addrspace(1)* %out, <3 x i16> addrspace(1)* %in) {
 entry:
   %ld = load <3 x i16>, <3 x i16> addrspace(1)* %in
   %ext = zext <3 x i16> %ld to <3 x i32>
   store <3 x i32> %ext, <3 x i32> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}global_sextload_v3i16_to_v3i32:
 ; GCN-NOHSA: buffer_load_dwordx2
 ; GCN-HSA: flat_load_dwordx2
 
 ; CM: MEM_RAT_CACHELESS STORE_DWORD [[ST_LO:T[0-9]]], {{T[0-9]\.[XYZW]}}
 ; CM: MEM_RAT_CACHELESS STORE_DWORD [[ST_HI:T[0-9]]].X, {{T[0-9]\.[XYZW]}}
 ; EG: MEM_RAT_CACHELESS STORE_RAW [[ST_HI:T[0-9]]].X, {{T[0-9]\.[XYZW]}},
 ; EG: MEM_RAT_CACHELESS STORE_RAW [[ST_LO:T[0-9]]].XY, {{T[0-9]\.[XYZW]}},
-; EGCM-DAG: VTX_READ_32 [[DST_LO:T[0-9]\.[XYZW]]], {{T[0-9].[XYZW]}}, 0, #1
-; EGCM-DAG: VTX_READ_16 [[DST_HI:T[0-9]\.[XYZW]]], {{T[0-9].[XYZW]}}, 4, #1
-; TODO: This should use DST, but for some there are redundant MOVs
-; EGCM-DAG: ASHR {{[* ]*}}[[ST_LO]].Y, {{T[0-9]\.[XYZW]}}, literal
-; EGCM-DAG: BFE_INT {{[* ]*}}[[ST_LO]].X, {{T[0-9]\.[XYZW]}}, 0.0, literal
+; EGCM-DAG: VTX_READ_16 [[DST_LO:T[0-9]\.[XYZW]]], [[SRC:T[0-9].[XYZW]]], 0, #1
+; EGCM-DAG: VTX_READ_16 [[DST_MID:T[0-9]\.[XYZW]]], [[SRC]], 2, #1
+; EGCM-DAG: VTX_READ_16 [[DST_HI:T[0-9]\.[XYZW]]], [[SRC]], 4, #1
+; EGCM-DAG: BFE_INT {{[* ]*}}[[ST_LO]].X, [[DST_LO]], 0.0, literal
+; EGCM-DAG: BFE_INT {{[* ]*}}[[ST_LO]].Y, [[DST_MID]], 0.0, literal
 ; EGCM-DAG: BFE_INT {{[* ]*}}[[ST_HI]].X, [[DST_HI]], 0.0, literal
 ; EGCM-DAG: 16
 ; EGCM-DAG: 16
 define amdgpu_kernel void @global_sextload_v3i16_to_v3i32(<3 x i32> addrspace(1)* %out, <3 x i16> addrspace(1)* %in) {
 entry:
   %ld = load <3 x i16>, <3 x i16> addrspace(1)* %in
   %ext = sext <3 x i16> %ld to <3 x i32>
   store <3 x i32> %ext, <3 x i32> addrspace(1)* %out

llvm/trunk/test/CodeGen/AMDGPU/load-local-i16.ll

 ; GFX9-NOT: m0
 ; SICIVI: s_mov_b32 m0
 
 ; GCN: ds_read_b64
 ; GCN-DAG: ds_write_b32
 ; GCN-DAG: ds_write_b16
 
 ; EG-DAG: LDS_USHORT_READ_RET
-; EG-DAG: LDS_READ_RET
+; EG-DAG: LDS_USHORT_READ_RET
 define amdgpu_kernel void @local_load_v3i16(<3 x i16> addrspace(3)* %out, <3 x i16> addrspace(3)* %in) {
 entry:
   %ld = load <3 x i16>, <3 x i16> addrspace(3)* %in
   store <3 x i16> %ld, <3 x i16> addrspace(3)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}local_load_v4i16:

 ; FUNC-LABEL: {{^}}local_local_zextload_v3i16_to_v3i32:
 ; GFX9-NOT: m0
 ; SICIVI: s_mov_b32 m0
 
 ; GCN: ds_read_b64
 ; GCN-DAG: ds_write_b32
 ; GCN-DAG: ds_write_b64
 
-; EG: LDS_READ_RET
+; EG: LDS_USHORT_READ_RET
+; EG: LDS_USHORT_READ_RET
+; EG: LDS_USHORT_READ_RET
 define amdgpu_kernel void @local_local_zextload_v3i16_to_v3i32(<3 x i32> addrspace(3)* %out, <3 x i16> addrspace(3)* %in) {
 entry:
   %ld = load <3 x i16>, <3 x i16> addrspace(3)* %in
   %ext = zext <3 x i16> %ld to <3 x i32>
   store <3 x i32> %ext, <3 x i32> addrspace(3)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}local_local_sextload_v3i16_to_v3i32:
 ; GFX9-NOT: m0
 ; SICIVI: s_mov_b32 m0
 
 ; GCN: ds_read_b64
 ; GCN-DAG: ds_write_b32
 ; GCN-DAG: ds_write_b64
 
-; EG: LDS_READ_RET
+; EG: LDS_USHORT_READ_RET
+; EG: LDS_USHORT_READ_RET
+; EG: LDS_USHORT_READ_RET
 ; EG-DAG: BFE_INT
 ; EG-DAG: BFE_INT
 ; EG-DAG: BFE_INT
 define amdgpu_kernel void @local_local_sextload_v3i16_to_v3i32(<3 x i32> addrspace(3)* %out, <3 x i16> addrspace(3)* %in) {
 entry:
   %ld = load <3 x i16>, <3 x i16> addrspace(3)* %in
   %ext = sext <3 x i16> %ld to <3 x i32>
   store <3 x i32> %ext, <3 x i32> addrspace(3)* %out

llvm/trunk/test/CodeGen/X86/load-local-v3i1.ll

+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.2 | FileCheck %s
+
+; widen a v3i1 to v4i1 to do a vector load/store. We would previously
+; reconstruct the said v3i1 from the first element of the vector by filling all
+; the lanes of the vector with that first element, which was obviously wrong.
+; This was done in the type-legalizing of the DAG, when legalizing the load.
+
+; Function Attrs: argmemonly nounwind readonly
+declare <3 x i32> @llvm.masked.load.v3i32.p1v3i32(<3 x i32> addrspace(1)*, i32, <3 x i1>, <3 x i32>)
+
+; Function Attrs: argmemonly nounwind
+declare void @llvm.masked.store.v3i32.p1v3i32(<3 x i32>, <3 x i32> addrspace(1)*, i32, <3 x i1>)
+
+define  <3 x i32> @masked_load_v3(i32 addrspace(1)*, <3 x i1>) {
+entry:
+  %2 = bitcast i32 addrspace(1)* %0 to <3 x i32> addrspace(1)*
+  %3 = call <3 x i32> @llvm.masked.load.v3i32.p1v3i32(<3 x i32> addrspace(1)* %2, i32 4, <3 x i1> %1, <3 x i32> undef)
+  ret <3 x i32> %3
+}
+
+define void @masked_store4_v3(<3 x i32>, i32 addrspace(1)*, <3 x i1>) {
+entry:
+  %3 = bitcast i32 addrspace(1)* %1 to <3 x i32> addrspace(1)*
+  call void @llvm.masked.store.v3i32.p1v3i32(<3 x i32> %0, <3 x i32> addrspace(1)* %3, i32 4, <3 x i1> %2)
+  ret void
+}
+
+define void @local_load_v3i1(i32 addrspace(1)* %out, i32 addrspace(1)* %in, <3 x i1>* %predicate_ptr) nounwind {
+; CHECK-LABEL: local_load_v3i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT: pushq   %rbp
+; CHECK-NEXT: pushq   %r15
+; CHECK-NEXT: pushq   %r14
+; CHECK-NEXT: pushq   %rbx
+; CHECK-NEXT: pushq   %rax
+; CHECK-NEXT: movq    %rdi, %r14
+; CHECK-NEXT: movzbl  (%rdx), %ebp
+; CHECK-NEXT: movl    %ebp, %eax
+; CHECK-NEXT: shrl    %eax
+; CHECK-NEXT: andl    $1, %eax
+; CHECK-NEXT: movl    %ebp, %ecx
+; CHECK-NEXT: andl    $1, %ecx
+; CHECK-NEXT: movd    %ecx, %xmm0
+; CHECK-NEXT: pinsrd  $1, %eax, %xmm0
+; CHECK-NEXT: shrl    $2, %ebp
+; CHECK-NEXT: andl    $1, %ebp
+; CHECK-NEXT: pinsrd  $2, %ebp, %xmm0
+; CHECK-NEXT: movd    %xmm0, %ebx
+; CHECK-NEXT: pextrd  $1, %xmm0, %r15d
+; CHECK-NEXT: movq    %rsi, %rdi
+; CHECK-NEXT: movl    %ebx, %esi
+; CHECK-NEXT: movl    %r15d, %edx
+; CHECK-NEXT: movl    %ebp, %ecx
+; CHECK-NEXT: callq   masked_load_v3
+; CHECK-NEXT: movq    %r14, %rdi
+; CHECK-NEXT: movl    %ebx, %esi
+; CHECK-NEXT: movl    %r15d, %edx
+; CHECK-NEXT: movl    %ebp, %ecx
+; CHECK-NEXT: callq   masked_store4_v3
+; CHECK-NEXT: addq    $8, %rsp
+; CHECK-NEXT: popq    %rbx
+; CHECK-NEXT: popq    %r14
+; CHECK-NEXT: popq    %r15
+; CHECK-NEXT: popq    %rbp
+; CHECK-NEXT: retq
+  %predicate = load <3 x i1>, <3 x i1>* %predicate_ptr
+  %load1 = call <3 x i32> @masked_load_v3(i32 addrspace(1)* %in, <3 x i1> %predicate)
+  call void @masked_store4_v3(<3 x i32> %load1, i32 addrspace(1)* %out, <3 x i1> %predicate)
+  ret void
+}

llvm/trunk/test/CodeGen/X86/widen_arith-3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+sse4.2 -post-RA-scheduler=true | FileCheck %s
 
 ; Widen a v3i16 to v8i16 to do a vector add
 
 @.str = internal constant [4 x i8] c"%d \00"
 @.str1 = internal constant [2 x i8] c"\0A\00"
 
 define void @update(<3 x i16>* %dst, <3 x i16>* %src, i32 %n) nounwind {
 ; CHECK-LABEL: update:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushl %ebp
 ; CHECK-NEXT:    movl %esp, %ebp
 ; CHECK-NEXT:    andl $-8, %esp
-; CHECK-NEXT:    subl $40, %esp
+; CHECK-NEXT:    subl $32, %esp
 ; CHECK-NEXT:    movl {{\.LCPI.*}}, %eax
 ; CHECK-NEXT:    movdqa {{.*#+}} xmm1 = [0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15]
 ; CHECK-NEXT:    pcmpeqd %xmm0, %xmm0
 ; CHECK-NEXT:    movw $1, {{[0-9]+}}(%esp)
 ; CHECK-NEXT:    movl $0, {{[0-9]+}}(%esp)
 ; CHECK-NEXT:    movl %eax, {{[0-9]+}}(%esp)
 ; CHECK-NEXT:    jmp .LBB0_1
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_2: # %forbody
 ; CHECK-NEXT:    # in Loop: Header=BB0_1 Depth=1
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    movl 12(%ebp), %edx
 ; CHECK-NEXT:    movl 8(%ebp), %ecx
-; CHECK-NEXT:    movd {{.*#+}} xmm2 = mem[0],zero,zero,zero
-; CHECK-NEXT:    pmovzxwd {{.*#+}} xmm2 = xmm2[0],zero,xmm2[1],zero,xmm2[2],zero,xmm2[3],zero
-; CHECK-NEXT:    pinsrd $2, 4(%edx,%eax,8), %xmm2
+; CHECK-NEXT:    pmovzxwd {{.*#+}} xmm2 = mem[0],zero,mem[1],zero,mem[2],zero,mem[3],zero
 ; CHECK-NEXT:    psubd %xmm0, %xmm2
 ; CHECK-NEXT:    pextrw $4, %xmm2, 4(%ecx,%eax,8)
 ; CHECK-NEXT:    pshufb %xmm1, %xmm2
 ; CHECK-NEXT:    movd %xmm2, (%ecx,%eax,8)
 ; CHECK-NEXT:    incl {{[0-9]+}}(%esp)
 ; CHECK-NEXT:  .LBB0_1: # %forcond
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax

llvm/trunk/test/CodeGen/X86/widen_cast-2.ll

 ; CHECK-NEXT:    movl (%esp), %eax
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; CHECK-NEXT:    shll $5, %eax
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; CHECK-NEXT:    movdqa (%edx,%eax), %xmm1
 ; CHECK-NEXT:    movdqa 16(%edx,%eax), %xmm2
 ; CHECK-NEXT:    psubw %xmm0, %xmm1
 ; CHECK-NEXT:    psubw %xmm0, %xmm2
-; CHECK-NEXT:    movd %xmm2, 16(%ecx,%eax)
-; CHECK-NEXT:    pextrd $1, %xmm2, 20(%ecx,%eax)
 ; CHECK-NEXT:    pextrd $2, %xmm2, 24(%ecx,%eax)
+; CHECK-NEXT:    movq %xmm2, 16(%ecx,%eax)
 ; CHECK-NEXT:    movdqa %xmm1, (%ecx,%eax)
 ; CHECK-NEXT:    incl (%esp)
 ; CHECK-NEXT:    cmpl $3, (%esp)
 ; CHECK-NEXT:    jle .LBB0_2
 ; CHECK-NEXT:  .LBB0_3: # %afterfor
 ; CHECK-NEXT:    popl %eax
 ; CHECK-NEXT:    retl
 entry:

llvm/trunk/test/CodeGen/X86/widen_cast-3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+sse4.2 | FileCheck %s --check-prefix=X86
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.2 | FileCheck %s --check-prefix=X64
 
 ; bitcast v12i8 to v3i32
 
 define void @convert(<12 x i8>* %dst.addr, <3 x i32> %src) nounwind {
 ; X86-LABEL: convert:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X86-NEXT:    psubd %xmm1, %xmm0
 ; X86-NEXT:    pextrd $2, %xmm0, 8(%eax)
-; X86-NEXT:    pextrd $1, %xmm0, 4(%eax)
-; X86-NEXT:    movd %xmm0, (%eax)
+; X86-NEXT:    movq %xmm0, (%eax)
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: convert:
 ; X64:       # %bb.0:
 ; X64-NEXT:    pcmpeqd %xmm1, %xmm1
 ; X64-NEXT:    psubd %xmm1, %xmm0
 ; X64-NEXT:    pextrd $2, %xmm0, 8(%rdi)
 ; X64-NEXT:    movq %xmm0, (%rdi)
 ; X64-NEXT:    retq
   %add = add <3 x i32> %src, < i32 1, i32 1, i32 1 >
   %conv = bitcast <3 x i32> %add to <12 x i8>
   store <12 x i8> %conv, <12 x i8>* %dst.addr
   ret void
 }

llvm/trunk/test/CodeGen/X86/widen_load-2.ll

 ; X86-LABEL: add3i32:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    movdqa (%edx), %xmm0
 ; X86-NEXT:    paddd (%ecx), %xmm0
 ; X86-NEXT:    pextrd $2, %xmm0, 8(%eax)
-; X86-NEXT:    pextrd $1, %xmm0, 4(%eax)
-; X86-NEXT:    movd %xmm0, (%eax)
+; X86-NEXT:    movq %xmm0, (%eax)
 ; X86-NEXT:    retl $4
 ;
 ; X64-LABEL: add3i32:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    movdqa (%rsi), %xmm0
 ; X64-NEXT:    paddd (%rdx), %xmm0
 ; X64-NEXT:    pextrd $2, %xmm0, 8(%rdi)
 ; X64-NEXT:    movq %xmm0, (%rdi)
 ; X64-NEXT:    retq
   %a = load %i32vec3, %i32vec3* %ap, align 16
   %b = load %i32vec3, %i32vec3* %bp, align 16
   %x = add %i32vec3 %a, %b
   store %i32vec3 %x, %i32vec3* %ret, align 16
   ret void
 }
 
 define void @add3i32_2(%i32vec3*  sret %ret, %i32vec3* %ap, %i32vec3* %bp)  {
 ; X86-LABEL: add3i32_2:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; X86-NEXT:    pinsrd $1, 4(%edx), %xmm0
+; X86-NEXT:    movq {{.*#+}} xmm0 = mem[0],zero
 ; X86-NEXT:    pinsrd $2, 8(%edx), %xmm0
-; X86-NEXT:    movd {{.*#+}} xmm1 = mem[0],zero,zero,zero
-; X86-NEXT:    pinsrd $1, 4(%ecx), %xmm1
+; X86-NEXT:    movq {{.*#+}} xmm1 = mem[0],zero
 ; X86-NEXT:    pinsrd $2, 8(%ecx), %xmm1
 ; X86-NEXT:    paddd %xmm0, %xmm1
-; X86-NEXT:    pextrd $1, %xmm1, 4(%eax)
+; X86-NEXT:    movq %xmm1, (%eax)
 ; X86-NEXT:    pextrd $2, %xmm1, 8(%eax)
-; X86-NEXT:    movd %xmm1, (%eax)
 ; X86-NEXT:    retl $4
 ;
 ; X64-LABEL: add3i32_2:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    movq {{.*#+}} xmm0 = mem[0],zero
 ; X64-NEXT:    pinsrd $2, 8(%rsi), %xmm0
 ; X64-NEXT:    movq {{.*#+}} xmm1 = mem[0],zero

 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    movdqa (%edx), %xmm0
 ; X86-NEXT:    movdqa 16(%edx), %xmm1
 ; X86-NEXT:    paddd (%ecx), %xmm0
 ; X86-NEXT:    paddd 16(%ecx), %xmm1
-; X86-NEXT:    movd %xmm1, 16(%eax)
-; X86-NEXT:    pextrd $1, %xmm1, 20(%eax)
 ; X86-NEXT:    pextrd $2, %xmm1, 24(%eax)
+; X86-NEXT:    movq %xmm1, 16(%eax)
 ; X86-NEXT:    movdqa %xmm0, (%eax)
 ; X86-NEXT:    retl $4
 ;
 ; X64-LABEL: add7i32:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    movdqa (%rsi), %xmm0
 ; X64-NEXT:    movdqa 16(%rsi), %xmm1

 
 %i16vec3 = type <3 x i16>
 define void @add3i16(%i16vec3* nocapture sret %ret, %i16vec3* %ap, %i16vec3* %bp) nounwind {
 ; X86-LABEL: add3i16:
 ; X86:       # %bb.0:
 ; X86-NEXT:    pushl %ebp
 ; X86-NEXT:    movl %esp, %ebp
 ; X86-NEXT:    andl $-8, %esp
-; X86-NEXT:    subl $24, %esp
+; X86-NEXT:    subl $8, %esp
 ; X86-NEXT:    movl 8(%ebp), %eax
 ; X86-NEXT:    movl 16(%ebp), %ecx
 ; X86-NEXT:    movl 12(%ebp), %edx
-; X86-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; X86-NEXT:    pmovzxwd {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero
-; X86-NEXT:    pinsrd $2, 4(%edx), %xmm0
-; X86-NEXT:    movd {{.*#+}} xmm1 = mem[0],zero,zero,zero
-; X86-NEXT:    pmovzxwd {{.*#+}} xmm1 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero
-; X86-NEXT:    pinsrd $2, 4(%ecx), %xmm1
+; X86-NEXT:    pmovzxwd {{.*#+}} xmm0 = mem[0],zero,mem[1],zero,mem[2],zero,mem[3],zero
+; X86-NEXT:    pmovzxwd {{.*#+}} xmm1 = mem[0],zero,mem[1],zero,mem[2],zero,mem[3],zero
 ; X86-NEXT:    paddd %xmm0, %xmm1
 ; X86-NEXT:    pextrw $4, %xmm1, 4(%eax)
 ; X86-NEXT:    pshufb {{.*#+}} xmm1 = xmm1[0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15]
 ; X86-NEXT:    movd %xmm1, (%eax)
 ; X86-NEXT:    movl %ebp, %esp
 ; X86-NEXT:    popl %ebp
 ; X86-NEXT:    retl $4
 ;

 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    movdqa (%edx), %xmm0
 ; X86-NEXT:    movdqa 16(%edx), %xmm1
 ; X86-NEXT:    paddw (%ecx), %xmm0
 ; X86-NEXT:    paddw 16(%ecx), %xmm1
-; X86-NEXT:    movd %xmm1, 16(%eax)
-; X86-NEXT:    pextrd $1, %xmm1, 20(%eax)
+; X86-NEXT:    movq %xmm1, 16(%eax)
 ; X86-NEXT:    movdqa %xmm0, (%eax)
 ; X86-NEXT:    retl $4
 ;
 ; X64-LABEL: add12i16:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    movdqa (%rsi), %xmm0
 ; X64-NEXT:    movdqa 16(%rsi), %xmm1

 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    movdqa (%edx), %xmm0
 ; X86-NEXT:    movdqa 16(%edx), %xmm1
 ; X86-NEXT:    paddb (%ecx), %xmm0
 ; X86-NEXT:    paddb 16(%ecx), %xmm1
-; X86-NEXT:    movd %xmm1, 16(%eax)
-; X86-NEXT:    pextrd $1, %xmm1, 20(%eax)
 ; X86-NEXT:    pextrd $2, %xmm1, 24(%eax)
 ; X86-NEXT:    pextrw $6, %xmm1, 28(%eax)
 ; X86-NEXT:    pextrb $14, %xmm1, 30(%eax)
+; X86-NEXT:    movq %xmm1, 16(%eax)
 ; X86-NEXT:    movdqa %xmm0, (%eax)
 ; X86-NEXT:    retl $4
 ;
 ; X64-LABEL: add31i8:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    movdqa (%rsi), %xmm0
 ; X64-NEXT:    movdqa 16(%rsi), %xmm1