[AMDGPU][GlobalISel] Fix 96 and 128 local loads and stores

[mbrkusanin]

Summary

Fix local ds_read/write_b96/b128 so they can be selected if the alignment
allows. Otherwise, either pick appropriate ds_read2/write2 instructions or break
them down.

[arsenm]

Other than on SI, there are 96-bit DS read/write. Is this just working around some later selection problem?

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

275 ↗

(On Diff #270081)

The address space doesn't make this special? This willl break SI?

[mbrkusanin]

Yes, it basically avoids problems of not being able to select 3x32 for local address space. SDag was breaking these down to a ds_read_b64 and ds_read_b32 so I did the same thing for GlobalISel.

I've looked at .td files and it seems that the following pattern can be added so ds_read_b96 can be selected

foreach vt = VReg_64.RegTypes in {
defm : DSReadPat_mc <DS_READ_B64, vt, "load_alignX_local">;
}

but I'm not sure what should the minimal alignment (X) be for this specific instruction. Any idea? For alignment of 4 every test will pass but, otherwise we'll need to break some cases to b64, b32 pairs.

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

275 ↗

(On Diff #270081)

SI will be fine because hasDwordx3LoadStores will be false. Local address space uses ds_read and ds_write so the name is slightly confusing.

[arsenm]

In D81638#2089681, @mbrkusanin wrote:

Yes, it basically avoids problems of not being able to select 3x32 for local address space. SDag was breaking these down to a ds_read_b64 and ds_read_b32 so I did the same thing for GlobalISel.

I've looked at .td files and it seems that the following pattern can be added so ds_read_b96 can be selected

foreach vt = VReg_64.RegTypes in {
defm : DSReadPat_mc <DS_READ_B64, vt, "load_alignX_local">;
}

but I'm not sure what should the minimal alignment (X) be for this specific instruction. Any idea? For alignment of 4 every test will pass but, otherwise we'll need to break some cases to b64, b32 pairs.

I'm not sure what the alignment requirement is (I think it also depends on whether unaligned DS access is supported, which we never fully implemented in the compiler).

FYI I am working on completely rewriting the load/store legalization rules and moving this into custom lowering

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

275 ↗

(On Diff #270081)

I mean all the logic here should work without special casing the address space here

[mbrkusanin]

Looking at ISA .pdf docs for SI (gfx6) and onward I have not found any requirements for alignments on local loads and stores. There are mentions of dword alignment for reads and writes of dword and larger for buffer instructions but nothing more specific for LDS or GDS. SDag likes to break down ds_read/write_b128 in certain cases but does not know about b96. It seems to me that the code was not updated since SI.
Now b96 and b128 will be picked for align 4 and larger (align 2 and 1 are broken down same way as before). Furthermore, there are several Vulkan conformance tests that have align 4 loads and stores (96 and 128) that will now pass.

Alternative solution would be to break them down to ds_read/write_b64/b32 when we're not sure. But I don't have a test to check if this is necessary.

Since all changes are in .td files this should not cause you any problems with rewriting load/store legality rules.

[arsenm]

In D81638#2096030, @mbrkusanin wrote:

Looking at ISA .pdf docs for SI (gfx6) and onward I have not found any requirements for alignments on local loads and stores. There are mentions of dword alignment for reads and writes of dword and larger for buffer instructions but nothing more specific for LDS or GDS. SDag likes to break down ds_read/write_b128 in certain cases but does not know about b96. It seems to me that the code was not updated since SI.
Now b96 and b128 will be picked for align 4 and larger (align 2 and 1 are broken down same way as before). Furthermore, there are several Vulkan conformance tests that have align 4 loads and stores (96 and 128) that will now pass.

They definitely have some alignment requirements, but it's poorly documented. I'm pretty confident b128 requires 16-byte alignment, and b64 requires 8. I think gfx9 added unaligned access support (dependent on some config registers), but I think we never fully handled all of these changes. I think the linux driver hardcodes this to allow unaligned access.
The code was updated since SI, but probably not since 96-bit types were added to MVT.

Alternative solution would be to break them down to ds_read/write_b64/b32 when we're not sure. But I don't have a test to check if this is necessary.

Since all changes are in .td files this should not cause you any problems with rewriting load/store legality rules.

[mbrkusanin]

Sorry, I was away for a few days.

Now for CI and VI, b96 and b128 have minimal alignment of 16, gfx9 and onward has minimal alignment of 4 (anything lower then 4 is already broken down for all targets).

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUInstructions.td

501

I think the Aligned<> subclasses didn't actually work for some reason, but I only half fixed the patterns maybe?

llvm/lib/Target/AMDGPU/DSInstructions.td

683–706

You shouldn't need to re-consider the legalization logic. The selector can mostly assume legal inputs. If the less aligned version wasn't legal, it should have been broken down. This also depends more specifically on the unaligned features, rather than gfx78

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

1365–1386 ↗

(On Diff #272441)

Can you commit this as a separate patch? This changes the DAG path too. I also don't think "unaligned" here means requires 4 byte alignment

[mbrkusanin]

Code that was changing alignment requirements from SITargetLowering::allowsMisalignedMemoryAccessesImpl in now in D82788.

[mbrkusanin]

llvm/lib/Target/AMDGPU/AMDGPUInstructions.td

501

I changed it so now Aligned<> subclasses are used for both load and store. They seem to work fine.

llvm/lib/Target/AMDGPU/DSInstructions.td

683–706

I couldn't get rid of subtarget predicates because of the way SDag uses allowsMisalignedMemoryAccessesImpl. For example on gfx7/8, ds_read_b128 requires alignment of 16, but we need to say that alignment of 8 is also okay because we can pick ds_read2_b64. GISel however just sees that alignment of 8 is okay and picks ds_read_b128 instead of ds_read2_b64. If both are acceptable according to DSInstructions.td then GIsel will pick the first one (If i change the order in .td file and move it up it will actually pick ds_read2_b64 but that breaks any structure that file had).

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

1365–1386 ↗

(On Diff #272441)

Moved to D82788

[mbrkusanin]

llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp

1328–1407

This is currently unsed but is required because of DS128Bit8ByteAligned pattern. Without the patternt GIsel does not know how to pick ds_read2/write2_b64.

Currently if ds_read/write_b128 are not legal because of alignment they will be broken down to 4 ds_read/write_b32 instructions and later combined in SILoadStoreOptimizer. It seems to me that is should be possible to tell ISel's lowerLoad and lowerStore to pick ds_read2/write2_b64 when appropriate with this pattern. But it would be easier if that was a separate patch.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp

1328

This is basically the same as the splitting the 64-bit case into read2. Can you factor that to avoid duplicating all of this

[mbrkusanin]

Reduced duplicated code for SelectDS64Bit4ByteAligned and SelectDS128Bit8ByteAligned.

[mbrkusanin]

• Rebase
• Updated to reflect the changes in parent revision.

[mbrkusanin]

• Rebase

[mbrkusanin]

Moved tests from here to parent revision (mistakenly put them in wrong patch):

• llvm/test/Transforms/LoadStoreVectorizer/AMDGPU/merge-stores.ll
• llvm/test/Transforms/LoadStoreVectorizer/AMDGPU/multiple_tails.ll

[mbrkusanin]

• Rebase
• Ping

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUInstructions.td

501

Did you double check the generated matcher table? The problem was the current emitter only checks one of these predicates at a time, so it successfully imports but then doesn't actually perform the check. It's silently ignored

[mbrkusanin]

llvm/lib/Target/AMDGPU/AMDGPUInstructions.td

501

• "GIM_CheckMemoryAlignment, /*MI*/0, /*MMO*/0, /*MinAlign*/16," and "8" does show up throughout .inc files (AMDGPUGenGlobalISel.inc) for ds_read/write instructions.
• Aligned<> does affect produced code and other tests.
• Regardless off what is used, "let MinAlignment = X;" or "Aligned<X>", identical .inc files are produced.

[mbrkusanin]

• Updated llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-load-local-128.mir

llvm/lib/Target/AMDGPU/AMDGPU.td

def isGFX7Only :
   Predicate<"Subtarget->getGeneration() == AMDGPUSubtarget::SEA_ISLANDS">,
   AssemblerPredicate<(all_of (not FeatureGCN3Encoding), FeatureCIInsts, (not FeatureGFX10Insts))>;
 
 def isGFX7GFX10 :
   Predicate<"Subtarget->getGeneration() == AMDGPUSubtarget::SEA_ISLANDS ||"
             "Subtarget->getGeneration() == AMDGPUSubtarget::GFX10">,
   AssemblerPredicate<(all_of (not FeatureGCN3Encoding), FeatureCIInsts)>;
 
+def isGFX7GFX8 :
+  Predicate<"Subtarget->getGeneration() == AMDGPUSubtarget::SEA_ISLANDS ||"
+            "Subtarget->getGeneration() == AMDGPUSubtarget::VOLCANIC_ISLANDS">,
+  AssemblerPredicate<(all_of FeatureSouthernIslands, FeatureCIInsts)>;
+
 def isGFX7GFX8GFX9 :
   Predicate<"Subtarget->getGeneration() == AMDGPUSubtarget::SEA_ISLANDS ||"
             "Subtarget->getGeneration() == AMDGPUSubtarget::VOLCANIC_ISLANDS ||"
             "Subtarget->getGeneration() == AMDGPUSubtarget::GFX9">,
   AssemblerPredicate<(all_of FeatureGFX7GFX8GFX9Insts)>;
 
 def isGFX6GFX7GFX8GFX9 :
   Predicate<"Subtarget->getGeneration() == AMDGPUSubtarget::SOUTHERN_ISLANDS ||"

llvm/lib/Target/AMDGPU/AMDGPUGISel.td

 def gi_ds_1addr_1offset :
     GIComplexOperandMatcher<s32, "selectDS1Addr1Offset">,
     GIComplexPatternEquiv<DS1Addr1Offset>;
 
 def gi_ds_64bit_4byte_aligned :
     GIComplexOperandMatcher<s64, "selectDS64Bit4ByteAligned">,
     GIComplexPatternEquiv<DS64Bit4ByteAligned>;
 
+def gi_ds_128bit_8byte_aligned :
+    GIComplexOperandMatcher<s64, "selectDS128Bit8ByteAligned">,
+    GIComplexPatternEquiv<DS128Bit8ByteAligned>;
+
 def gi_mubuf_addr64 :
     GIComplexOperandMatcher<s64, "selectMUBUFAddr64">,
     GIComplexPatternEquiv<MUBUFAddr64>;
 
 def gi_mubuf_offset :
     GIComplexOperandMatcher<s64, "selectMUBUFOffset">,
     GIComplexPatternEquiv<MUBUFOffset>;
 

llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp

private:
   const TargetRegisterClass *getOperandRegClass(SDNode *N, unsigned OpNo) const;
   virtual bool SelectADDRVTX_READ(SDValue Addr, SDValue &Base, SDValue &Offset);
   virtual bool SelectADDRIndirect(SDValue Addr, SDValue &Base, SDValue &Offset);
   bool isDSOffsetLegal(SDValue Base, unsigned Offset,
                        unsigned OffsetBits) const;
   bool SelectDS1Addr1Offset(SDValue Ptr, SDValue &Base, SDValue &Offset) const;
   bool SelectDS64Bit4ByteAligned(SDValue Ptr, SDValue &Base, SDValue &Offset0,
                                  SDValue &Offset1) const;
+  bool SelectDS128Bit8ByteAligned(SDValue Ptr, SDValue &Base, SDValue &Offset0,
+                                  SDValue &Offset1) const;
+  bool SelectDSReadWrite2(SDValue Ptr, SDValue &Base, SDValue &Offset0,
+                          SDValue &Offset1, bool IsDS128) const;
   bool SelectMUBUF(SDValue Addr, SDValue &SRsrc, SDValue &VAddr,
                    SDValue &SOffset, SDValue &Offset, SDValue &Offen,
                    SDValue &Idxen, SDValue &Addr64, SDValue &GLC, SDValue &SLC,
                    SDValue &TFE, SDValue &DLC, SDValue &SWZ) const;
   bool SelectMUBUFAddr64(SDValue Addr, SDValue &SRsrc, SDValue &VAddr,
                          SDValue &SOffset, SDValue &Offset, SDValue &GLC,
                          SDValue &SLC, SDValue &TFE, SDValue &DLC,
                          SDValue &SWZ) const;
bool AMDGPUDAGToDAGISel::SelectDS1Addr1Offset(SDValue Addr, SDValue &Base,
   Offset = CurDAG->getTargetConstant(0, SDLoc(Addr), MVT::i16);
   return true;
 }
 
 // TODO: If offset is too big, put low 16-bit into offset.
 bool AMDGPUDAGToDAGISel::SelectDS64Bit4ByteAligned(SDValue Addr, SDValue &Base,
                                                    SDValue &Offset0,
                                                    SDValue &Offset1) const {
+  return SelectDSReadWrite2(Addr, Base, Offset0, Offset1, false);
+}
+
+bool AMDGPUDAGToDAGISel::SelectDS128Bit8ByteAligned(SDValue Addr, SDValue &Base,
+                                                    SDValue &Offset0,
+                                                    SDValue &Offset1) const {
+  return SelectDSReadWrite2(Addr, Base, Offset0, Offset1, true);
+}
+
+bool AMDGPUDAGToDAGISel::SelectDSReadWrite2(SDValue Addr, SDValue &Base,
+                                            SDValue &Offset0, SDValue &Offset1,
+                                            bool IsDS128) const {
   SDLoc DL(Addr);
+  unsigned Align = IsDS128 ? 8 : 4;
 
   if (CurDAG->isBaseWithConstantOffset(Addr)) {
     SDValue N0 = Addr.getOperand(0);
     SDValue N1 = Addr.getOperand(1);
     ConstantSDNode *C1 = cast<ConstantSDNode>(N1);
-    unsigned DWordOffset0 = C1->getZExtValue() / 4;
-    unsigned DWordOffset1 = DWordOffset0 + 1;
+    unsigned OffsetValue0 = C1->getZExtValue() / Align;
+    unsigned OffsetValue1 = OffsetValue0 + 1;
     // (add n0, c0)
-    if (isDSOffsetLegal(N0, DWordOffset1, 8)) {
+    if (isDSOffsetLegal(N0, OffsetValue1, 8)) {
       Base = N0;
-      Offset0 = CurDAG->getTargetConstant(DWordOffset0, DL, MVT::i8);
-      Offset1 = CurDAG->getTargetConstant(DWordOffset1, DL, MVT::i8);
+      Offset0 = CurDAG->getTargetConstant(OffsetValue0, DL, MVT::i8);
+      Offset1 = CurDAG->getTargetConstant(OffsetValue1, DL, MVT::i8);
       return true;
     }
   } else if (Addr.getOpcode() == ISD::SUB) {
     // sub C, x -> add (sub 0, x), C
-    if (const ConstantSDNode *C = dyn_cast<ConstantSDNode>(Addr.getOperand(0))) {
-      unsigned DWordOffset0 = C->getZExtValue() / 4;
-      unsigned DWordOffset1 = DWordOffset0 + 1;
+    if (const ConstantSDNode *C =
+            dyn_cast<ConstantSDNode>(Addr.getOperand(0))) {
+      unsigned OffsetValue0 = C->getZExtValue() / Align;
+      unsigned OffsetValue1 = OffsetValue0 + 1;
 
-      if (isUInt<8>(DWordOffset0)) {
+      if (isUInt<8>(OffsetValue0)) {
         SDLoc DL(Addr);
         SDValue Zero = CurDAG->getTargetConstant(0, DL, MVT::i32);
 
         // XXX - This is kind of hacky. Create a dummy sub node so we can check
         // the known bits in isDSOffsetLegal. We need to emit the selected node
         // here, so this is thrown away.
-        SDValue Sub = CurDAG->getNode(ISD::SUB, DL, MVT::i32,
-                                      Zero, Addr.getOperand(1));
+        SDValue Sub =
+            CurDAG->getNode(ISD::SUB, DL, MVT::i32, Zero, Addr.getOperand(1));
 
-        if (isDSOffsetLegal(Sub, DWordOffset1, 8)) {
+        if (isDSOffsetLegal(Sub, OffsetValue1, 8)) {
           SmallVector<SDValue, 3> Opnds;
           Opnds.push_back(Zero);
           Opnds.push_back(Addr.getOperand(1));
           unsigned SubOp = AMDGPU::V_SUB_CO_U32_e32;
           if (Subtarget->hasAddNoCarry()) {
             SubOp = AMDGPU::V_SUB_U32_e64;
             Opnds.push_back(
                 CurDAG->getTargetConstant(0, {}, MVT::i1)); // clamp bit
           }
 
-          MachineSDNode *MachineSub
-            = CurDAG->getMachineNode(SubOp, DL, MVT::i32, Opnds);
+          MachineSDNode *MachineSub = CurDAG->getMachineNode(
+              SubOp, DL, (IsDS128 ? MVT::i64 : MVT::i32), Opnds);
 
           Base = SDValue(MachineSub, 0);
-          Offset0 = CurDAG->getTargetConstant(DWordOffset0, DL, MVT::i8);
-          Offset1 = CurDAG->getTargetConstant(DWordOffset1, DL, MVT::i8);
+          Offset0 = CurDAG->getTargetConstant(OffsetValue0, DL, MVT::i8);
+          Offset1 = CurDAG->getTargetConstant(OffsetValue1, DL, MVT::i8);
           return true;
         }
       }
     }
   } else if (const ConstantSDNode *CAddr = dyn_cast<ConstantSDNode>(Addr)) {
-    unsigned DWordOffset0 = CAddr->getZExtValue() / 4;
-    unsigned DWordOffset1 = DWordOffset0 + 1;
-    assert(4 * DWordOffset0 == CAddr->getZExtValue());
+    unsigned OffsetValue0 = CAddr->getZExtValue() / Align;
+    unsigned OffsetValue1 = OffsetValue0 + 1;
+    assert(Align * OffsetValue0 == CAddr->getZExtValue());
 
-    if (isUInt<8>(DWordOffset0) && isUInt<8>(DWordOffset1)) {
+    if (isUInt<8>(OffsetValue0) && isUInt<8>(OffsetValue1)) {
       SDValue Zero = CurDAG->getTargetConstant(0, DL, MVT::i32);
-      MachineSDNode *MovZero
-        = CurDAG->getMachineNode(AMDGPU::V_MOV_B32_e32,
-                                 DL, MVT::i32, Zero);
+      MachineSDNode *MovZero =
+          CurDAG->getMachineNode(AMDGPU::V_MOV_B32_e32, DL, MVT::i32, Zero);
       Base = SDValue(MovZero, 0);
-      Offset0 = CurDAG->getTargetConstant(DWordOffset0, DL, MVT::i8);
-      Offset1 = CurDAG->getTargetConstant(DWordOffset1, DL, MVT::i8);
+      Offset0 = CurDAG->getTargetConstant(OffsetValue0, DL, MVT::i8);
+      Offset1 = CurDAG->getTargetConstant(OffsetValue1, DL, MVT::i8);
       return true;
     }
   }
 
   // default case
 
   Base = Addr;
   Offset0 = CurDAG->getTargetConstant(0, DL, MVT::i8);
   Offset1 = CurDAG->getTargetConstant(1, DL, MVT::i8);
   return true;
 }
 
 bool AMDGPUDAGToDAGISel::SelectMUBUF(SDValue Addr, SDValue &Ptr,
                                      SDValue &VAddr, SDValue &SOffset,
                                      SDValue &Offset, SDValue &Offen,
                                      SDValue &Idxen, SDValue &Addr64,
                                      SDValue &GLC, SDValue &SLC,
                                      SDValue &TFE, SDValue &DLC,
                                      SDValue &SWZ) const {
   // Subtarget prefers to use flat instruction
   // FIXME: This should be a pattern predicate and not reach here
   if (Subtarget->useFlatForGlobal())
     return false;
 
   SDLoc DL(Addr);
 
   if (!GLC.getNode())
     GLC = CurDAG->getTargetConstant(0, DL, MVT::i1);
   if (!SLC.getNode())
     SLC = CurDAG->getTargetConstant(0, DL, MVT::i1);
   TFE = CurDAG->getTargetConstant(0, DL, MVT::i1);
   DLC = CurDAG->getTargetConstant(0, DL, MVT::i1);
   SWZ = CurDAG->getTargetConstant(0, DL, MVT::i1);
 
   Idxen = CurDAG->getTargetConstant(0, DL, MVT::i1);
   Offen = CurDAG->getTargetConstant(0, DL, MVT::i1);
   Addr64 = CurDAG->getTargetConstant(0, DL, MVT::i1);
   SOffset = CurDAG->getTargetConstant(0, DL, MVT::i32);
 
   ConstantSDNode *C1 = nullptr;
   SDValue N0 = Addr;
   if (CurDAG->isBaseWithConstantOffset(Addr)) {
     C1 = cast<ConstantSDNode>(Addr.getOperand(1));
     if (isUInt<32>(C1->getZExtValue()))
       N0 = Addr.getOperand(0);
     else
       C1 = nullptr;
   }
 
   if (N0.getOpcode() == ISD::ADD) {
     // (add N2, N3) -> addr64, or
     // (add (add N2, N3), C1) -> addr64
     SDValue N2 = N0.getOperand(0);
     SDValue N3 = N0.getOperand(1);
     Addr64 = CurDAG->getTargetConstant(1, DL, MVT::i1);
 
     if (N2->isDivergent()) {
       if (N3->isDivergent()) {
         // Both N2 and N3 are divergent. Use N0 (the result of the add) as the
         // addr64, and construct the resource from a 0 address.
         Ptr = SDValue(buildSMovImm64(DL, 0, MVT::v2i32), 0);
         VAddr = N0;
       } else {
         // N2 is divergent, N3 is not.
         Ptr = N3;
         VAddr = N2;
       }
     } else {
       // N2 is not divergent.
       Ptr = N2;
       VAddr = N3;
     }
     Offset = CurDAG->getTargetConstant(0, DL, MVT::i16);
   } else if (N0->isDivergent()) {
     // N0 is divergent. Use it as the addr64, and construct the resource from a
     // 0 address.
     Ptr = SDValue(buildSMovImm64(DL, 0, MVT::v2i32), 0);
     VAddr = N0;
     Addr64 = CurDAG->getTargetConstant(1, DL, MVT::i1);
   } else {
     // N0 -> offset, or
     // (N0 + C1) -> offset
     VAddr = CurDAG->getTargetConstant(0, DL, MVT::i32);
     Ptr = N0;
   }
 
   if (!C1) {
     // No offset.
     Offset = CurDAG->getTargetConstant(0, DL, MVT::i16);
     return true;
   }
 
   if (SIInstrInfo::isLegalMUBUFImmOffset(C1->getZExtValue())) {
     // Legal offset for instruction.
     Offset = CurDAG->getTargetConstant(C1->getZExtValue(), DL, MVT::i16);
     return true;
   }
 
   // Illegal offset, store it in soffset.
   Offset = CurDAG->getTargetConstant(0, DL, MVT::i16);

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h

private:
   bool isDSOffsetLegal(Register Base, int64_t Offset,
                        unsigned OffsetBits) const;
 
   std::pair<Register, unsigned>
   selectDS1Addr1OffsetImpl(MachineOperand &Root) const;
   InstructionSelector::ComplexRendererFns
   selectDS1Addr1Offset(MachineOperand &Root) const;
 
-  std::pair<Register, unsigned>
-  selectDS64Bit4ByteAlignedImpl(MachineOperand &Root) const;
   InstructionSelector::ComplexRendererFns
   selectDS64Bit4ByteAligned(MachineOperand &Root) const;
 
+  InstructionSelector::ComplexRendererFns
+  selectDS128Bit8ByteAligned(MachineOperand &Root) const;
+
+  std::pair<Register, unsigned>
+  selectDSReadWrite2Impl(MachineOperand &Root, bool IsDS128) const;
+  InstructionSelector::ComplexRendererFns
+  selectDSReadWrite2(MachineOperand &Root, bool IsDS128) const;
+
   std::pair<Register, int64_t>
   getPtrBaseWithConstantOffset(Register Root,
                                const MachineRegisterInfo &MRI) const;
 
   // Parse out a chain of up to two g_ptr_add instructions.
   // g_ptr_add (n0, _)
   // g_ptr_add (n0, (n1 = g_ptr_add n2, n3))
   struct MUBUFAddressData {

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

AMDGPUInstructionSelector::selectDS1Addr1Offset(MachineOperand &Root) const {
   return {{
       [=](MachineInstrBuilder &MIB) { MIB.addReg(Reg); },
       [=](MachineInstrBuilder &MIB) { MIB.addImm(Offset); }
     }};
 }
 
 InstructionSelector::ComplexRendererFns
 AMDGPUInstructionSelector::selectDS64Bit4ByteAligned(MachineOperand &Root) const {
+  return selectDSReadWrite2(Root, false);
+}
+
+InstructionSelector::ComplexRendererFns
+AMDGPUInstructionSelector::selectDS128Bit8ByteAligned(MachineOperand &Root) const {
+  return selectDSReadWrite2(Root, true);
+}
+
+InstructionSelector::ComplexRendererFns
+AMDGPUInstructionSelector::selectDSReadWrite2(MachineOperand &Root,
+                                              bool IsDS128) const {
   Register Reg;
   unsigned Offset;
-  std::tie(Reg, Offset) = selectDS64Bit4ByteAlignedImpl(Root);
+  std::tie(Reg, Offset) = selectDSReadWrite2Impl(Root, IsDS128);
   return {{
       [=](MachineInstrBuilder &MIB) { MIB.addReg(Reg); },
       [=](MachineInstrBuilder &MIB) { MIB.addImm(Offset); },
       [=](MachineInstrBuilder &MIB) { MIB.addImm(Offset+1); }
     }};
 }
 
 std::pair<Register, unsigned>
-AMDGPUInstructionSelector::selectDS64Bit4ByteAlignedImpl(MachineOperand &Root) const {
+AMDGPUInstructionSelector::selectDSReadWrite2Impl(MachineOperand &Root,
+                                                  bool IsDS128) const {
   const MachineInstr *RootDef = MRI->getVRegDef(Root.getReg());
   if (!RootDef)
     return std::make_pair(Root.getReg(), 0);
 
   int64_t ConstAddr = 0;
 
   Register PtrBase;
   int64_t Offset;
   std::tie(PtrBase, Offset) =
     getPtrBaseWithConstantOffset(Root.getReg(), *MRI);
 
   if (Offset) {
-    int64_t DWordOffset0 = Offset / 4;
-    int64_t DWordOffset1 = DWordOffset0 + 1;
-    if (isDSOffsetLegal(PtrBase, DWordOffset1, 8)) {
+    int64_t OffsetValue0 = Offset / (IsDS128 ? 8 : 4);
+    int64_t OffsetValue1 = OffsetValue0 + 1;
+    if (isDSOffsetLegal(PtrBase, OffsetValue1, (IsDS128 ? 16 : 8))) {
       // (add n0, c0)
-      return std::make_pair(PtrBase, DWordOffset0);
+      return std::make_pair(PtrBase, OffsetValue0);
     }
   } else if (RootDef->getOpcode() == AMDGPU::G_SUB) {
     // TODO
 
   } else if (mi_match(Root.getReg(), *MRI, m_ICst(ConstAddr))) {
     // TODO
 
   }

llvm/lib/Target/AMDGPU/AMDGPUInstructions.td

 defm atomic_load_or : ret_noret_binary_atomic_op<atomic_load_or>;
 defm atomic_load_sub : ret_noret_binary_atomic_op<atomic_load_sub>;
 defm atomic_load_umax : ret_noret_binary_atomic_op<atomic_load_umax>;
 defm atomic_load_umin : ret_noret_binary_atomic_op<atomic_load_umin>;
 defm atomic_load_xor : ret_noret_binary_atomic_op<atomic_load_xor>;
 defm atomic_load_fadd : ret_noret_binary_atomic_op<atomic_load_fadd, 0>;
 defm AMDGPUatomic_cmp_swap : ret_noret_binary_atomic_op<AMDGPUatomic_cmp_swap>;
 
-
-def load_align8_local : PatFrag <(ops node:$ptr), (load_local node:$ptr)> {
+def load_align8_local : PatFrag<(ops node:$ptr), (load_local node:$ptr)>,
+                        Aligned<8> {
   let IsLoad = 1;
   let IsNonExtLoad = 1;
-  let MinAlignment = 8;
 }
 
-def load_align16_local : PatFrag <(ops node:$ptr), (load_local node:$ptr)> {
+def load_align16_local : PatFrag<(ops node:$ptr), (load_local node:$ptr)>,
+                        Aligned<16> {
   let IsLoad = 1;
   let IsNonExtLoad = 1;
-  let MinAlignment = 16;
 }
 
 def store_align8_local: PatFrag<(ops node:$val, node:$ptr),
                                 (store_local node:$val, node:$ptr)>, Aligned<8> {
   let IsStore = 1;
   let IsTruncStore = 0;
 }
 
 def store_align16_local: PatFrag<(ops node:$val, node:$ptr),
                                 (store_local node:$val, node:$ptr)>, Aligned<16> {
   let IsStore = 1;
   let IsTruncStore = 0;

llvm/lib/Target/AMDGPU/DSInstructions.td

 defm : DSReadPat_mc <DS_READ_B64, i64, "atomic_load_64_local">;
 
 let AddedComplexity = 100 in {
 
 foreach vt = VReg_64.RegTypes in {
 defm : DSReadPat_mc <DS_READ_B64, vt, "load_align8_local">;
 }
 
-defm : DSReadPat_mc <DS_READ_B128, v4i32, "load_align16_local">;
+let SubtargetPredicate = isGFX7GFX8 in {
+
+foreach vt = VReg_96.RegTypes in {
+defm : DSReadPat_mc <DS_READ_B96, vt, "load_align16_local">;
+}
+
+foreach vt = VReg_128.RegTypes in {
+defm : DSReadPat_mc <DS_READ_B128, vt, "load_align16_local">;
+}
+
+}
+
+let SubtargetPredicate = isGFX9Plus in {
+
+foreach vt = VReg_96.RegTypes in {
+defm : DSReadPat_mc <DS_READ_B96, vt, "load_local">;
+}
+
+foreach vt = VReg_128.RegTypes in {
+defm : DSReadPat_mc <DS_READ_B128, vt, "load_local">;
+}
+
+}
 
 } // End AddedComplexity = 100
 
 let OtherPredicates = [D16PreservesUnusedBits] in {
 def : DSReadPat_D16<DS_READ_U16_D16_HI, load_d16_hi_local, v2i16>;
 def : DSReadPat_D16<DS_READ_U16_D16_HI, load_d16_hi_local, v2f16>;
 def : DSReadPat_D16<DS_READ_U8_D16_HI, az_extloadi8_d16_hi_local, v2i16>;
 def : DSReadPat_D16<DS_READ_U8_D16_HI, az_extloadi8_d16_hi_local, v2f16>;
 def : DSReadPat_D16<DS_READ_I8_D16_HI, sextloadi8_d16_hi_local, v2i16>;

 
 class DS64Bit4ByteAlignedWritePat<DS_Pseudo inst, ValueType vt, PatFrag frag> : GCNPat<
   (frag vt:$value, (DS64Bit4ByteAligned i32:$ptr, i8:$offset0, i8:$offset1)),
   (inst $ptr, (i32 (EXTRACT_SUBREG VReg_64:$value, sub0)),
               (i32 (EXTRACT_SUBREG VReg_64:$value, sub1)), $offset0, $offset1,
               (i1 0))
 >;
 
+class DS128Bit8ByteAlignedReadPat<DS_Pseudo inst, ValueType vt, PatFrag frag> : GCNPat <
+  (vt:$value (frag (DS128Bit8ByteAligned i32:$ptr, i8:$offset0, i8:$offset1))),
+  (inst $ptr, $offset0, $offset1, (i1 0))
+>;
+
+class DS128Bit8ByteAlignedWritePat<DS_Pseudo inst, ValueType vt, PatFrag frag> : GCNPat<
+  (frag vt:$value, (DS128Bit8ByteAligned i32:$ptr, i8:$offset0, i8:$offset1)),
+  (inst $ptr, (i64 (EXTRACT_SUBREG VReg_128:$value, sub0_sub1)),
+              (i64 (EXTRACT_SUBREG VReg_128:$value, sub2_sub3)), $offset0, $offset1,
+              (i1 0))
+>;
+
 multiclass DS64Bit4ByteAlignedPat_mc<ValueType vt> {
   let OtherPredicates = [LDSRequiresM0Init, isGFX7Plus] in {
     def : DS64Bit4ByteAlignedReadPat<DS_READ2_B32, vt, load_local_m0>;
     def : DS64Bit4ByteAlignedWritePat<DS_WRITE2_B32, vt, store_local_m0>;
   }
 
   let OtherPredicates = [NotLDSRequiresM0Init] in {
     def : DS64Bit4ByteAlignedReadPat<DS_READ2_B32_gfx9, vt, load_local>;
     def : DS64Bit4ByteAlignedWritePat<DS_WRITE2_B32_gfx9, vt, store_local>;
   }
 }
 
+multiclass DS128Bit8ByteAlignedPat_mc<ValueType vt> {
+  let OtherPredicates = [LDSRequiresM0Init, isGFX7Plus] in {
+    def : DS128Bit8ByteAlignedReadPat<DS_READ2_B64, vt, load_local_m0>;
+    def : DS128Bit8ByteAlignedWritePat<DS_WRITE2_B64, vt, store_local_m0>;
+  }
+
+  let OtherPredicates = [NotLDSRequiresM0Init] in {
+    def : DS128Bit8ByteAlignedReadPat<DS_READ2_B64_gfx9, vt, load_local>;
+    def : DS128Bit8ByteAlignedWritePat<DS_WRITE2_B64_gfx9, vt, store_local>;
+  }
+}
+
 // v2i32 loads are split into i32 loads on SI during lowering, due to a bug
 // related to bounds checking.
 foreach vt = VReg_64.RegTypes in {
 defm : DS64Bit4ByteAlignedPat_mc<vt>;
 }
 
+foreach vt = VReg_128.RegTypes in {
+defm : DS128Bit8ByteAlignedPat_mc<vt>;
+}
+
 let AddedComplexity = 100 in {
 
 foreach vt = VReg_64.RegTypes in {
 defm : DSWritePat_mc <DS_WRITE_B64, vt, "store_align8_local">;
 }
 
-defm : DSWritePat_mc <DS_WRITE_B128, v4i32, "store_align16_local">;
+let SubtargetPredicate = isGFX7GFX8 in {
+
+foreach vt = VReg_96.RegTypes in {
+defm : DSWritePat_mc <DS_WRITE_B96, vt, "store_align16_local">;
+}
+
+foreach vt = VReg_128.RegTypes in {
+defm : DSWritePat_mc <DS_WRITE_B128, vt, "store_align16_local">;
+}
+
+}
+
+let SubtargetPredicate = isGFX9Plus in {
+
+foreach vt = VReg_96.RegTypes in {
+defm : DSWritePat_mc <DS_WRITE_B96, vt, "store_local">;
+}
+
+foreach vt = VReg_128.RegTypes in {
+defm : DSWritePat_mc <DS_WRITE_B128, vt, "store_local">;
+}
+
+}
 
 } // End AddedComplexity = 100
 class DSAtomicRetPat<DS_Pseudo inst, ValueType vt, PatFrag frag, bit gds=0> : GCNPat <
   (frag (DS1Addr1Offset i32:$ptr, i16:$offset), vt:$value),
   (inst $ptr, getVregSrcForVT<vt>.ret:$value, offset:$offset, (i1 gds))
 >;
 
 multiclass DSAtomicRetPat_mc<DS_Pseudo inst, ValueType vt, string frag> {

llvm/lib/Target/AMDGPU/SIInstrInfo.td

 
 let MemoryVT = i16 in {
 def extloadi16_local_m0 : PatFrag<(ops node:$ptr), (extloadi16_glue node:$ptr)>;
 def sextloadi16_local_m0 : PatFrag<(ops node:$ptr), (sextloadi16_glue node:$ptr)>;
 def zextloadi16_local_m0 : PatFrag<(ops node:$ptr), (zextloadi16_glue node:$ptr)>;
 }
 
 def load_align8_local_m0 : PatFrag<(ops node:$ptr),
-                                   (load_local_m0 node:$ptr)> {
+                                   (load_local_m0 node:$ptr)>, Aligned<8> {
   let IsLoad = 1;
   let IsNonExtLoad = 1;
-  let MinAlignment = 8;
 }
+
 def load_align16_local_m0 : PatFrag<(ops node:$ptr),
-                                    (load_local_m0 node:$ptr)> {
+                                   (load_local_m0 node:$ptr)>, Aligned<16> {
   let IsLoad = 1;
   let IsNonExtLoad = 1;
-  let MinAlignment = 16;
 }
 
 } // End IsLoad = 1
 
 let IsAtomic = 1, AddressSpaces = LoadAddress_local.AddrSpaces in {
 def atomic_load_32_local_m0 : PatFrag<(ops node:$ptr),
                                       (atomic_load_32_glue node:$ptr)> {
   let MemoryVT = i32;

 
 def truncstorei16_local_m0 : PatFrag<(ops node:$val, node:$ptr),
                                     (unindexedstore_glue node:$val, node:$ptr)> {
   let IsStore = 1;
   let MemoryVT = i16;
 }
 }
 
-def store_align16_local_m0 : PatFrag <
-  (ops node:$value, node:$ptr),
-  (store_local_m0 node:$value, node:$ptr)> {
+def store_align8_local_m0 : PatFrag <(ops node:$value, node:$ptr),
+                                     (store_local_m0 node:$value, node:$ptr)>,
+                            Aligned<8> {
   let IsStore = 1;
   let IsTruncStore = 0;
-  let MinAlignment = 16;
 }
 
-def store_align8_local_m0 : PatFrag <
-  (ops node:$value, node:$ptr),
-  (store_local_m0 node:$value, node:$ptr)> {
+def store_align16_local_m0 : PatFrag <(ops node:$value, node:$ptr),
+                                     (store_local_m0 node:$value, node:$ptr)>,
+                            Aligned<16> {
   let IsStore = 1;
   let IsTruncStore = 0;
-  let MinAlignment = 8;
 }
 
 let AddressSpaces = StoreAddress_local.AddrSpaces in {
 
 def atomic_store_local_32_m0 : PatFrag <
   (ops node:$value, node:$ptr),
   (AMDGPUatomic_st_glue node:$value, node:$ptr)> {
   let IsAtomic = 1;

 def PackedI16InputMods : PackedIntInputMods<PackedI16InputModsMatchClass>;
 
 //===----------------------------------------------------------------------===//
 // Complex patterns
 //===----------------------------------------------------------------------===//
 
 def DS1Addr1Offset : ComplexPattern<i32, 2, "SelectDS1Addr1Offset">;
 def DS64Bit4ByteAligned : ComplexPattern<i32, 3, "SelectDS64Bit4ByteAligned">;
+def DS128Bit8ByteAligned : ComplexPattern<i64, 3, "SelectDS128Bit8ByteAligned">;
 
 def MOVRELOffset : ComplexPattern<i32, 2, "SelectMOVRELOffset">;
 
 def VOP3Mods0 : ComplexPattern<untyped, 4, "SelectVOP3Mods0">;
 def VOP3Mods0Clamp : ComplexPattern<untyped, 3, "SelectVOP3Mods0Clamp">;
 def VOP3Mods  : ComplexPattern<untyped, 2, "SelectVOP3Mods">;
 def VOP3NoMods : ComplexPattern<untyped, 1, "SelectVOP3NoMods">;
 // VOP3Mods, but the input source is known to never be NaN.

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-load-local-128.mir

 tracksRegLiveness: true
 
 body: |
   bb.0:
     liveins:  $vgpr0
 
     ; GFX7-LABEL: name: load_local_v4s32_align_8
     ; GFX7: liveins: $vgpr0
-    ; GFX7: [[COPY:%[0-9]+]]:vgpr(p3) = COPY $vgpr0
+    ; GFX7: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX7: $m0 = S_MOV_B32 -1
-    ; GFX7: [[LOAD:%[0-9]+]]:vreg_128(<4 x s32>) = G_LOAD [[COPY]](p3) :: (load 16, align 8, addrspace 3)
-    ; GFX7: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[LOAD]](<4 x s32>)
+    ; GFX7: [[DS_READ2_B64_:%[0-9]+]]:vreg_128 = DS_READ2_B64 [[COPY]], 0, 1, 0, implicit $m0, implicit $exec :: (load 16, align 8, addrspace 3)
+    ; GFX7: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[DS_READ2_B64_]]
     ; GFX9-LABEL: name: load_local_v4s32_align_8
     ; GFX9: liveins: $vgpr0
-    ; GFX9: [[COPY:%[0-9]+]]:vgpr(p3) = COPY $vgpr0
-    ; GFX9: [[LOAD:%[0-9]+]]:vreg_128(<4 x s32>) = G_LOAD [[COPY]](p3) :: (load 16, align 8, addrspace 3)
-    ; GFX9: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[LOAD]](<4 x s32>)
+    ; GFX9: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
+    ; GFX9: [[DS_READ_B128_gfx9_:%[0-9]+]]:vreg_128 = DS_READ_B128_gfx9 [[COPY]], 0, 0, implicit $exec :: (load 16, align 8, addrspace 3)
+    ; GFX9: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[DS_READ_B128_gfx9_]]
     %0:vgpr(p3) = COPY $vgpr0
     %1:vgpr(<4 x  s32>) = G_LOAD %0 :: (load 16, align 8, addrspace 3)
     $vgpr0_vgpr1_vgpr2_vgpr3 = COPY %1
 
 ...
 
 ---
 
 name: load_local_v2s64
 legalized:       true
 regBankSelected: true
 tracksRegLiveness: true
 
 body: |
   bb.0:
     liveins:  $vgpr0
 
     ; GFX7-LABEL: name: load_local_v2s64
     ; GFX7: liveins: $vgpr0
-    ; GFX7: [[COPY:%[0-9]+]]:vgpr(p3) = COPY $vgpr0
+    ; GFX7: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX7: $m0 = S_MOV_B32 -1
-    ; GFX7: [[LOAD:%[0-9]+]]:vreg_128(<2 x s64>) = G_LOAD [[COPY]](p3) :: (load 16, align 8, addrspace 3)
-    ; GFX7: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[LOAD]](<2 x s64>)
+    ; GFX7: [[DS_READ2_B64_:%[0-9]+]]:vreg_128 = DS_READ2_B64 [[COPY]], 0, 1, 0, implicit $m0, implicit $exec :: (load 16, align 8, addrspace 3)
+    ; GFX7: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[DS_READ2_B64_]]
     ; GFX9-LABEL: name: load_local_v2s64
     ; GFX9: liveins: $vgpr0
-    ; GFX9: [[COPY:%[0-9]+]]:vgpr(p3) = COPY $vgpr0
-    ; GFX9: [[LOAD:%[0-9]+]]:vreg_128(<2 x s64>) = G_LOAD [[COPY]](p3) :: (load 16, align 8, addrspace 3)
-    ; GFX9: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[LOAD]](<2 x s64>)
+    ; GFX9: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
+    ; GFX9: [[DS_READ_B128_gfx9_:%[0-9]+]]:vreg_128 = DS_READ_B128_gfx9 [[COPY]], 0, 0, implicit $exec :: (load 16, align 8, addrspace 3)
+    ; GFX9: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[DS_READ_B128_gfx9_]]
     %0:vgpr(p3) = COPY $vgpr0
     %1:vgpr(<2 x s64>) = G_LOAD %0 :: (load 16, align 8, addrspace 3)
     $vgpr0_vgpr1_vgpr2_vgpr3 = COPY %1
 
 ...
 
 ---
 

 tracksRegLiveness: true
 
 body: |
   bb.0:
     liveins:  $vgpr0
 
     ; GFX7-LABEL: name: load_local_s128
     ; GFX7: liveins: $vgpr0
-    ; GFX7: [[COPY:%[0-9]+]]:vgpr(p3) = COPY $vgpr0
+    ; GFX7: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX7: $m0 = S_MOV_B32 -1
-    ; GFX7: [[LOAD:%[0-9]+]]:vreg_128(s128) = G_LOAD [[COPY]](p3) :: (load 16, align 8, addrspace 3)
-    ; GFX7: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[LOAD]](s128)
+    ; GFX7: [[DS_READ2_B64_:%[0-9]+]]:vreg_128 = DS_READ2_B64 [[COPY]], 0, 1, 0, implicit $m0, implicit $exec :: (load 16, align 8, addrspace 3)
+    ; GFX7: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[DS_READ2_B64_]]
     ; GFX9-LABEL: name: load_local_s128
     ; GFX9: liveins: $vgpr0
-    ; GFX9: [[COPY:%[0-9]+]]:vgpr(p3) = COPY $vgpr0
-    ; GFX9: [[LOAD:%[0-9]+]]:vreg_128(s128) = G_LOAD [[COPY]](p3) :: (load 16, align 8, addrspace 3)
-    ; GFX9: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[LOAD]](s128)
+    ; GFX9: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
+    ; GFX9: [[DS_READ_B128_gfx9_:%[0-9]+]]:vreg_128 = DS_READ_B128_gfx9 [[COPY]], 0, 0, implicit $exec :: (load 16, align 8, addrspace 3)
+    ; GFX9: $vgpr0_vgpr1_vgpr2_vgpr3 = COPY [[DS_READ_B128_gfx9_]]
     %0:vgpr(p3) = COPY $vgpr0
     %1:vgpr(s128) = G_LOAD %0 :: (load 16, align 8, addrspace 3)
     $vgpr0_vgpr1_vgpr2_vgpr3 = COPY %1
 
 ...
 
 ---
 

llvm/test/CodeGen/AMDGPU/GlobalISel/load-local.128.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX9 %s
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=hawaii -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX7 %s
+
+; FIXME:
+; XUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=tahiti -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX6 %s
+
+define <4 x i32> @load_lds_v4i32(<4 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v4i32:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_b128 v[0:3], v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v4i32:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    ds_read_b128 v[0:3], v0
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <4 x i32>, <4 x i32> addrspace(3)* %ptr
+  ret <4 x i32> %load
+}
+
+define <4 x i32> @load_lds_v4i32_align1(<4 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v4i32_align1:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_u8 v1, v0
+; GFX9-NEXT:    ds_read_u8 v2, v0 offset:1
+; GFX9-NEXT:    ds_read_u8 v4, v0 offset:2
+; GFX9-NEXT:    ds_read_u8 v5, v0 offset:3
+; GFX9-NEXT:    ds_read_u8 v6, v0 offset:4
+; GFX9-NEXT:    ds_read_u8 v7, v0 offset:5
+; GFX9-NEXT:    ds_read_u8 v8, v0 offset:6
+; GFX9-NEXT:    ds_read_u8 v9, v0 offset:7
+; GFX9-NEXT:    s_mov_b32 s5, 8
+; GFX9-NEXT:    s_movk_i32 s4, 0xff
+; GFX9-NEXT:    s_waitcnt lgkmcnt(6)
+; GFX9-NEXT:    v_lshlrev_b32_sdwa v2, s5, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-NEXT:    v_and_or_b32 v1, v1, s4, v2
+; GFX9-NEXT:    s_waitcnt lgkmcnt(5)
+; GFX9-NEXT:    v_and_b32_e32 v2, s4, v4
+; GFX9-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX9-NEXT:    v_and_b32_e32 v4, s4, v5
+; GFX9-NEXT:    v_mov_b32_e32 v3, 0xff
+; GFX9-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 24, v4
+; GFX9-NEXT:    v_or3_b32 v4, v1, v2, v4
+; GFX9-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX9-NEXT:    v_lshlrev_b32_sdwa v1, s5, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(1)
+; GFX9-NEXT:    v_and_b32_e32 v2, v8, v3
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_and_b32_e32 v5, v9, v3
+; GFX9-NEXT:    v_and_or_b32 v1, v6, s4, v1
+; GFX9-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; GFX9-NEXT:    v_lshlrev_b32_e32 v5, 24, v5
+; GFX9-NEXT:    v_or3_b32 v1, v1, v2, v5
+; GFX9-NEXT:    ds_read_u8 v2, v0 offset:8
+; GFX9-NEXT:    ds_read_u8 v6, v0 offset:9
+; GFX9-NEXT:    ds_read_u8 v7, v0 offset:10
+; GFX9-NEXT:    ds_read_u8 v8, v0 offset:11
+; GFX9-NEXT:    ds_read_u8 v9, v0 offset:12
+; GFX9-NEXT:    ds_read_u8 v10, v0 offset:13
+; GFX9-NEXT:    ds_read_u8 v11, v0 offset:14
+; GFX9-NEXT:    ds_read_u8 v0, v0 offset:15
+; GFX9-NEXT:    v_mov_b32_e32 v5, 8
+; GFX9-NEXT:    s_waitcnt lgkmcnt(6)
+; GFX9-NEXT:    v_lshlrev_b32_sdwa v6, v5, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-NEXT:    v_and_or_b32 v2, v2, v3, v6
+; GFX9-NEXT:    s_waitcnt lgkmcnt(5)
+; GFX9-NEXT:    v_and_b32_e32 v6, v7, v3
+; GFX9-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX9-NEXT:    v_and_b32_e32 v7, v8, v3
+; GFX9-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
+; GFX9-NEXT:    v_lshlrev_b32_e32 v7, 24, v7
+; GFX9-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX9-NEXT:    v_lshlrev_b32_sdwa v5, v5, v10 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_and_b32_e32 v0, v0, v3
+; GFX9-NEXT:    v_or3_b32 v2, v2, v6, v7
+; GFX9-NEXT:    v_and_b32_e32 v6, v11, v3
+; GFX9-NEXT:    v_and_or_b32 v5, v9, v3, v5
+; GFX9-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 24, v0
+; GFX9-NEXT:    v_or3_b32 v3, v5, v6, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, v4
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v4i32_align1:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_movk_i32 s4, 0xff
+; GFX7-NEXT:    ds_read_u8 v1, v0
+; GFX7-NEXT:    ds_read_u8 v2, v0 offset:1
+; GFX7-NEXT:    ds_read_u8 v4, v0 offset:2
+; GFX7-NEXT:    ds_read_u8 v5, v0 offset:3
+; GFX7-NEXT:    ds_read_u8 v6, v0 offset:4
+; GFX7-NEXT:    ds_read_u8 v7, v0 offset:5
+; GFX7-NEXT:    ds_read_u8 v8, v0 offset:6
+; GFX7-NEXT:    ds_read_u8 v9, v0 offset:7
+; GFX7-NEXT:    s_waitcnt lgkmcnt(6)
+; GFX7-NEXT:    v_and_b32_e32 v2, s4, v2
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v1
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 8, v2
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(5)
+; GFX7-NEXT:    v_and_b32_e32 v2, s4, v4
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX7-NEXT:    v_and_b32_e32 v2, s4, v5
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 24, v2
+; GFX7-NEXT:    v_mov_b32_e32 v3, 0xff
+; GFX7-NEXT:    v_or_b32_e32 v4, v1, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX7-NEXT:    v_and_b32_e32 v2, v7, v3
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v6
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 8, v2
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(1)
+; GFX7-NEXT:    v_and_b32_e32 v2, v8, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_and_b32_e32 v2, v9, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 24, v2
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    ds_read_u8 v2, v0 offset:8
+; GFX7-NEXT:    ds_read_u8 v5, v0 offset:9
+; GFX7-NEXT:    ds_read_u8 v6, v0 offset:10
+; GFX7-NEXT:    ds_read_u8 v7, v0 offset:11
+; GFX7-NEXT:    ds_read_u8 v8, v0 offset:12
+; GFX7-NEXT:    ds_read_u8 v9, v0 offset:13
+; GFX7-NEXT:    ds_read_u8 v10, v0 offset:14
+; GFX7-NEXT:    ds_read_u8 v0, v0 offset:15
+; GFX7-NEXT:    s_waitcnt lgkmcnt(6)
+; GFX7-NEXT:    v_and_b32_e32 v5, v5, v3
+; GFX7-NEXT:    v_and_b32_e32 v2, v2, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 8, v5
+; GFX7-NEXT:    v_or_b32_e32 v2, v2, v5
+; GFX7-NEXT:    s_waitcnt lgkmcnt(5)
+; GFX7-NEXT:    v_and_b32_e32 v5, v6, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
+; GFX7-NEXT:    v_or_b32_e32 v2, v2, v5
+; GFX7-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX7-NEXT:    v_and_b32_e32 v5, v7, v3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX7-NEXT:    v_and_b32_e32 v6, v9, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 24, v5
+; GFX7-NEXT:    v_or_b32_e32 v2, v2, v5
+; GFX7-NEXT:    v_and_b32_e32 v5, v8, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 8, v6
+; GFX7-NEXT:    v_or_b32_e32 v5, v5, v6
+; GFX7-NEXT:    s_waitcnt lgkmcnt(1)
+; GFX7-NEXT:    v_and_b32_e32 v6, v10, v3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_and_b32_e32 v0, v0, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
+; GFX7-NEXT:    v_or_b32_e32 v5, v5, v6
+; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 24, v0
+; GFX7-NEXT:    v_or_b32_e32 v3, v5, v0
+; GFX7-NEXT:    v_mov_b32_e32 v0, v4
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <4 x i32>, <4 x i32> addrspace(3)* %ptr, align 1
+  ret <4 x i32> %load
+}
+
+define <4 x i32> @load_lds_v4i32_align2(<4 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v4i32_align2:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s4, 0xffff
+; GFX9-NEXT:    ds_read_u16 v1, v0
+; GFX9-NEXT:    ds_read_u16 v2, v0 offset:2
+; GFX9-NEXT:    ds_read_u16 v3, v0 offset:4
+; GFX9-NEXT:    ds_read_u16 v4, v0 offset:6
+; GFX9-NEXT:    ds_read_u16 v5, v0 offset:8
+; GFX9-NEXT:    ds_read_u16 v6, v0 offset:10
+; GFX9-NEXT:    ds_read_u16 v7, v0 offset:12
+; GFX9-NEXT:    ds_read_u16 v8, v0 offset:14
+; GFX9-NEXT:    s_waitcnt lgkmcnt(6)
+; GFX9-NEXT:    v_and_b32_e32 v0, s4, v2
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
+; GFX9-NEXT:    v_and_or_b32 v0, v1, s4, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX9-NEXT:    v_and_b32_e32 v1, s4, v4
+; GFX9-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; GFX9-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX9-NEXT:    v_and_b32_e32 v2, s4, v6
+; GFX9-NEXT:    v_and_or_b32 v1, v3, s4, v1
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_and_b32_e32 v3, s4, v8
+; GFX9-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; GFX9-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; GFX9-NEXT:    v_and_or_b32 v2, v5, s4, v2
+; GFX9-NEXT:    v_and_or_b32 v3, v7, s4, v3
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v4i32_align2:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    ds_read_u16 v1, v0
+; GFX7-NEXT:    ds_read_u16 v2, v0 offset:2
+; GFX7-NEXT:    ds_read_u16 v3, v0 offset:4
+; GFX7-NEXT:    ds_read_u16 v4, v0 offset:6
+; GFX7-NEXT:    ds_read_u16 v5, v0 offset:8
+; GFX7-NEXT:    ds_read_u16 v6, v0 offset:10
+; GFX7-NEXT:    ds_read_u16 v7, v0 offset:12
+; GFX7-NEXT:    ds_read_u16 v8, v0 offset:14
+; GFX7-NEXT:    s_mov_b32 s4, 0xffff
+; GFX7-NEXT:    s_waitcnt lgkmcnt(7)
+; GFX7-NEXT:    v_and_b32_e32 v0, s4, v1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(6)
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX7-NEXT:    v_and_b32_e32 v2, s4, v4
+; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; GFX7-NEXT:    v_or_b32_e32 v0, v0, v1
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX7-NEXT:    v_and_b32_e32 v3, s4, v6
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_and_b32_e32 v4, 0xffff, v8
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    v_and_b32_e32 v2, s4, v5
+; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; GFX7-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX7-NEXT:    v_and_b32_e32 v3, s4, v7
+; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
+; GFX7-NEXT:    v_or_b32_e32 v3, v3, v4
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <4 x i32>, <4 x i32> addrspace(3)* %ptr, align 2
+  ret <4 x i32> %load
+}
+
+define <4 x i32> @load_lds_v4i32_align4(<4 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v4i32_align4:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_b128 v[0:3], v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v4i32_align4:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v2, v0
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    ds_read2_b32 v[0:1], v0 offset1:1
+; GFX7-NEXT:    ds_read2_b32 v[2:3], v2 offset0:2 offset1:3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <4 x i32>, <4 x i32> addrspace(3)* %ptr, align 4
+  ret <4 x i32> %load
+}
+
+define <4 x i32> @load_lds_v4i32_align8(<4 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v4i32_align8:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_b128 v[0:3], v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v4i32_align8:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    ds_read2_b64 v[0:3], v0 offset1:1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <4 x i32>, <4 x i32> addrspace(3)* %ptr, align 8
+  ret <4 x i32> %load
+}
+
+define <4 x i32> @load_lds_v4i32_align16(<4 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v4i32_align16:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_b128 v[0:3], v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v4i32_align16:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    ds_read_b128 v[0:3], v0
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <4 x i32>, <4 x i32> addrspace(3)* %ptr, align 16
+  ret <4 x i32> %load
+}

llvm/test/CodeGen/AMDGPU/GlobalISel/load-local.96.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX9 %s
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=hawaii -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX7 %s
+
+; FIXME:
+; XUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=tahiti -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX6 %s
+
+define <3 x i32> @load_lds_v3i32(<3 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v3i32:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_b96 v[0:2], v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v3i32:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    ds_read_b96 v[0:2], v0
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <3 x i32>, <3 x i32> addrspace(3)* %ptr
+  ret <3 x i32> %load
+}
+
+define <3 x i32> @load_lds_v3i32_align1(<3 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v3i32_align1:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v2, v0
+; GFX9-NEXT:    ds_read_u8 v0, v0
+; GFX9-NEXT:    ds_read_u8 v1, v2 offset:1
+; GFX9-NEXT:    ds_read_u8 v4, v2 offset:2
+; GFX9-NEXT:    ds_read_u8 v5, v2 offset:3
+; GFX9-NEXT:    ds_read_u8 v6, v2 offset:4
+; GFX9-NEXT:    ds_read_u8 v7, v2 offset:5
+; GFX9-NEXT:    ds_read_u8 v8, v2 offset:6
+; GFX9-NEXT:    ds_read_u8 v9, v2 offset:7
+; GFX9-NEXT:    s_mov_b32 s5, 8
+; GFX9-NEXT:    s_movk_i32 s4, 0xff
+; GFX9-NEXT:    s_waitcnt lgkmcnt(6)
+; GFX9-NEXT:    v_lshlrev_b32_sdwa v1, s5, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-NEXT:    v_and_or_b32 v0, v0, s4, v1
+; GFX9-NEXT:    s_waitcnt lgkmcnt(5)
+; GFX9-NEXT:    v_and_b32_e32 v1, s4, v4
+; GFX9-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX9-NEXT:    v_and_b32_e32 v4, s4, v5
+; GFX9-NEXT:    v_mov_b32_e32 v3, 0xff
+; GFX9-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 24, v4
+; GFX9-NEXT:    v_or3_b32 v0, v0, v1, v4
+; GFX9-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX9-NEXT:    v_lshlrev_b32_sdwa v1, s5, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(1)
+; GFX9-NEXT:    v_and_b32_e32 v4, v8, v3
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_and_b32_e32 v5, v9, v3
+; GFX9-NEXT:    v_and_or_b32 v1, v6, s4, v1
+; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
+; GFX9-NEXT:    v_lshlrev_b32_e32 v5, 24, v5
+; GFX9-NEXT:    v_or3_b32 v1, v1, v4, v5
+; GFX9-NEXT:    ds_read_u8 v4, v2 offset:8
+; GFX9-NEXT:    ds_read_u8 v5, v2 offset:9
+; GFX9-NEXT:    ds_read_u8 v6, v2 offset:10
+; GFX9-NEXT:    ds_read_u8 v2, v2 offset:11
+; GFX9-NEXT:    v_mov_b32_e32 v7, 8
+; GFX9-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX9-NEXT:    v_lshlrev_b32_sdwa v5, v7, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-NEXT:    v_and_or_b32 v4, v4, v3, v5
+; GFX9-NEXT:    s_waitcnt lgkmcnt(1)
+; GFX9-NEXT:    v_and_b32_e32 v5, v6, v3
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_and_b32_e32 v2, v2, v3
+; GFX9-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
+; GFX9-NEXT:    v_lshlrev_b32_e32 v2, 24, v2
+; GFX9-NEXT:    v_or3_b32 v2, v4, v5, v2
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v3i32_align1:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    v_mov_b32_e32 v2, v0
+; GFX7-NEXT:    s_movk_i32 s4, 0xff
+; GFX7-NEXT:    ds_read_u8 v0, v0
+; GFX7-NEXT:    ds_read_u8 v1, v2 offset:1
+; GFX7-NEXT:    ds_read_u8 v4, v2 offset:2
+; GFX7-NEXT:    ds_read_u8 v5, v2 offset:3
+; GFX7-NEXT:    ds_read_u8 v6, v2 offset:4
+; GFX7-NEXT:    ds_read_u8 v7, v2 offset:5
+; GFX7-NEXT:    ds_read_u8 v8, v2 offset:6
+; GFX7-NEXT:    ds_read_u8 v9, v2 offset:7
+; GFX7-NEXT:    s_waitcnt lgkmcnt(6)
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v1
+; GFX7-NEXT:    v_and_b32_e32 v0, s4, v0
+; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 8, v1
+; GFX7-NEXT:    v_or_b32_e32 v0, v0, v1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(5)
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v4
+; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; GFX7-NEXT:    v_mov_b32_e32 v3, 0xff
+; GFX7-NEXT:    v_or_b32_e32 v0, v0, v1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v5
+; GFX7-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX7-NEXT:    v_and_b32_e32 v4, v7, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 24, v1
+; GFX7-NEXT:    v_or_b32_e32 v0, v0, v1
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v6
+; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 8, v4
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v4
+; GFX7-NEXT:    s_waitcnt lgkmcnt(1)
+; GFX7-NEXT:    v_and_b32_e32 v4, v8, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v4
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_and_b32_e32 v4, v9, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 24, v4
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v4
+; GFX7-NEXT:    ds_read_u8 v4, v2 offset:8
+; GFX7-NEXT:    ds_read_u8 v5, v2 offset:9
+; GFX7-NEXT:    ds_read_u8 v6, v2 offset:10
+; GFX7-NEXT:    ds_read_u8 v2, v2 offset:11
+; GFX7-NEXT:    s_waitcnt lgkmcnt(3)
+; GFX7-NEXT:    v_and_b32_e32 v4, v4, v3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX7-NEXT:    v_and_b32_e32 v5, v5, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 8, v5
+; GFX7-NEXT:    v_or_b32_e32 v4, v4, v5
+; GFX7-NEXT:    s_waitcnt lgkmcnt(1)
+; GFX7-NEXT:    v_and_b32_e32 v5, v6, v3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_and_b32_e32 v2, v2, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
+; GFX7-NEXT:    v_or_b32_e32 v4, v4, v5
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 24, v2
+; GFX7-NEXT:    v_or_b32_e32 v2, v4, v2
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <3 x i32>, <3 x i32> addrspace(3)* %ptr, align 1
+  ret <3 x i32> %load
+}
+
+define <3 x i32> @load_lds_v3i32_align2(<3 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v3i32_align2:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_u16 v1, v0
+; GFX9-NEXT:    ds_read_u16 v2, v0 offset:2
+; GFX9-NEXT:    ds_read_u16 v3, v0 offset:4
+; GFX9-NEXT:    ds_read_u16 v4, v0 offset:6
+; GFX9-NEXT:    ds_read_u16 v5, v0 offset:8
+; GFX9-NEXT:    ds_read_u16 v6, v0 offset:10
+; GFX9-NEXT:    s_mov_b32 s4, 0xffff
+; GFX9-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX9-NEXT:    v_and_b32_e32 v0, s4, v2
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
+; GFX9-NEXT:    v_and_or_b32 v0, v1, s4, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX9-NEXT:    v_and_b32_e32 v1, s4, v4
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_and_b32_e32 v2, s4, v6
+; GFX9-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; GFX9-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; GFX9-NEXT:    v_and_or_b32 v1, v3, s4, v1
+; GFX9-NEXT:    v_and_or_b32 v2, v5, s4, v2
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v3i32_align2:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    ds_read_u16 v1, v0
+; GFX7-NEXT:    ds_read_u16 v2, v0 offset:2
+; GFX7-NEXT:    ds_read_u16 v3, v0 offset:4
+; GFX7-NEXT:    ds_read_u16 v4, v0 offset:6
+; GFX7-NEXT:    ds_read_u16 v5, v0 offset:8
+; GFX7-NEXT:    ds_read_u16 v6, v0 offset:10
+; GFX7-NEXT:    s_mov_b32 s4, 0xffff
+; GFX7-NEXT:    s_waitcnt lgkmcnt(5)
+; GFX7-NEXT:    v_and_b32_e32 v0, s4, v1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX7-NEXT:    v_and_b32_e32 v2, s4, v4
+; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; GFX7-NEXT:    v_or_b32_e32 v0, v0, v1
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_and_b32_e32 v3, s4, v6
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    v_and_b32_e32 v2, s4, v5
+; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; GFX7-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <3 x i32>, <3 x i32> addrspace(3)* %ptr, align 2
+  ret <3 x i32> %load
+}
+
+define <3 x i32> @load_lds_v3i32_align4(<3 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v3i32_align4:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_b96 v[0:2], v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v3i32_align4:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v2, v0
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    ds_read2_b32 v[0:1], v0 offset1:1
+; GFX7-NEXT:    ds_read_b32 v2, v2 offset:8
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <3 x i32>, <3 x i32> addrspace(3)* %ptr, align 4
+  ret <3 x i32> %load
+}
+
+define <3 x i32> @load_lds_v3i32_align8(<3 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v3i32_align8:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_b96 v[0:2], v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v3i32_align8:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v2, v0
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    ds_read_b64 v[0:1], v0
+; GFX7-NEXT:    ds_read_b32 v2, v2 offset:8
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <3 x i32>, <3 x i32> addrspace(3)* %ptr, align 8
+  ret <3 x i32> %load
+}
+
+define <3 x i32> @load_lds_v3i32_align16(<3 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v3i32_align16:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_b96 v[0:2], v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v3i32_align16:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    ds_read_b96 v[0:2], v0
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <3 x i32>, <3 x i32> addrspace(3)* %ptr, align 16
+  ret <3 x i32> %load
+}

llvm/test/CodeGen/AMDGPU/GlobalISel/load-unaligned.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=gfx900 -mattr=+unaligned-access-mode -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX9 %s
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=hawaii -mattr=+unaligned-access-mode -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX7 %s
+
+; Unaligned DS access in available from GFX9 onwards.
+; LDS alignment enforcement is controlled by a configuration register:
+; SH_MEM_CONFIG.alignment_mode
+
+define <4 x i32> @load_lds_v4i32_align1(<4 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v4i32_align1:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_b128 v[0:3], v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v4i32_align1:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_movk_i32 s4, 0xff
+; GFX7-NEXT:    ds_read_u8 v1, v0
+; GFX7-NEXT:    ds_read_u8 v2, v0 offset:1
+; GFX7-NEXT:    ds_read_u8 v4, v0 offset:2
+; GFX7-NEXT:    ds_read_u8 v5, v0 offset:3
+; GFX7-NEXT:    ds_read_u8 v6, v0 offset:4
+; GFX7-NEXT:    ds_read_u8 v7, v0 offset:5
+; GFX7-NEXT:    ds_read_u8 v8, v0 offset:6
+; GFX7-NEXT:    ds_read_u8 v9, v0 offset:7
+; GFX7-NEXT:    s_waitcnt lgkmcnt(6)
+; GFX7-NEXT:    v_and_b32_e32 v2, s4, v2
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v1
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 8, v2
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(5)
+; GFX7-NEXT:    v_and_b32_e32 v2, s4, v4
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX7-NEXT:    v_and_b32_e32 v2, s4, v5
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 24, v2
+; GFX7-NEXT:    v_mov_b32_e32 v3, 0xff
+; GFX7-NEXT:    v_or_b32_e32 v4, v1, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX7-NEXT:    v_and_b32_e32 v2, v7, v3
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v6
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 8, v2
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(1)
+; GFX7-NEXT:    v_and_b32_e32 v2, v8, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_and_b32_e32 v2, v9, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 24, v2
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    ds_read_u8 v2, v0 offset:8
+; GFX7-NEXT:    ds_read_u8 v5, v0 offset:9
+; GFX7-NEXT:    ds_read_u8 v6, v0 offset:10
+; GFX7-NEXT:    ds_read_u8 v7, v0 offset:11
+; GFX7-NEXT:    ds_read_u8 v8, v0 offset:12
+; GFX7-NEXT:    ds_read_u8 v9, v0 offset:13
+; GFX7-NEXT:    ds_read_u8 v10, v0 offset:14
+; GFX7-NEXT:    ds_read_u8 v0, v0 offset:15
+; GFX7-NEXT:    s_waitcnt lgkmcnt(6)
+; GFX7-NEXT:    v_and_b32_e32 v5, v5, v3
+; GFX7-NEXT:    v_and_b32_e32 v2, v2, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 8, v5
+; GFX7-NEXT:    v_or_b32_e32 v2, v2, v5
+; GFX7-NEXT:    s_waitcnt lgkmcnt(5)
+; GFX7-NEXT:    v_and_b32_e32 v5, v6, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
+; GFX7-NEXT:    v_or_b32_e32 v2, v2, v5
+; GFX7-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX7-NEXT:    v_and_b32_e32 v5, v7, v3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX7-NEXT:    v_and_b32_e32 v6, v9, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 24, v5
+; GFX7-NEXT:    v_or_b32_e32 v2, v2, v5
+; GFX7-NEXT:    v_and_b32_e32 v5, v8, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 8, v6
+; GFX7-NEXT:    v_or_b32_e32 v5, v5, v6
+; GFX7-NEXT:    s_waitcnt lgkmcnt(1)
+; GFX7-NEXT:    v_and_b32_e32 v6, v10, v3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_and_b32_e32 v0, v0, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
+; GFX7-NEXT:    v_or_b32_e32 v5, v5, v6
+; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 24, v0
+; GFX7-NEXT:    v_or_b32_e32 v3, v5, v0
+; GFX7-NEXT:    v_mov_b32_e32 v0, v4
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <4 x i32>, <4 x i32> addrspace(3)* %ptr, align 1
+  ret <4 x i32> %load
+}
+
+define <3 x i32> @load_lds_v3i32_align1(<3 x i32> addrspace(3)* %ptr) {
+; GFX9-LABEL: load_lds_v3i32_align1:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_read_b96 v[0:2], v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: load_lds_v3i32_align1:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    v_mov_b32_e32 v2, v0
+; GFX7-NEXT:    s_movk_i32 s4, 0xff
+; GFX7-NEXT:    ds_read_u8 v0, v0
+; GFX7-NEXT:    ds_read_u8 v1, v2 offset:1
+; GFX7-NEXT:    ds_read_u8 v4, v2 offset:2
+; GFX7-NEXT:    ds_read_u8 v5, v2 offset:3
+; GFX7-NEXT:    ds_read_u8 v6, v2 offset:4
+; GFX7-NEXT:    ds_read_u8 v7, v2 offset:5
+; GFX7-NEXT:    ds_read_u8 v8, v2 offset:6
+; GFX7-NEXT:    ds_read_u8 v9, v2 offset:7
+; GFX7-NEXT:    s_waitcnt lgkmcnt(6)
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v1
+; GFX7-NEXT:    v_and_b32_e32 v0, s4, v0
+; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 8, v1
+; GFX7-NEXT:    v_or_b32_e32 v0, v0, v1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(5)
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v4
+; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; GFX7-NEXT:    v_mov_b32_e32 v3, 0xff
+; GFX7-NEXT:    v_or_b32_e32 v0, v0, v1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(4)
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v5
+; GFX7-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX7-NEXT:    v_and_b32_e32 v4, v7, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 24, v1
+; GFX7-NEXT:    v_or_b32_e32 v0, v0, v1
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v6
+; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 8, v4
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v4
+; GFX7-NEXT:    s_waitcnt lgkmcnt(1)
+; GFX7-NEXT:    v_and_b32_e32 v4, v8, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v4
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_and_b32_e32 v4, v9, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 24, v4
+; GFX7-NEXT:    v_or_b32_e32 v1, v1, v4
+; GFX7-NEXT:    ds_read_u8 v4, v2 offset:8
+; GFX7-NEXT:    ds_read_u8 v5, v2 offset:9
+; GFX7-NEXT:    ds_read_u8 v6, v2 offset:10
+; GFX7-NEXT:    ds_read_u8 v2, v2 offset:11
+; GFX7-NEXT:    s_waitcnt lgkmcnt(3)
+; GFX7-NEXT:    v_and_b32_e32 v4, v4, v3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(2)
+; GFX7-NEXT:    v_and_b32_e32 v5, v5, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 8, v5
+; GFX7-NEXT:    v_or_b32_e32 v4, v4, v5
+; GFX7-NEXT:    s_waitcnt lgkmcnt(1)
+; GFX7-NEXT:    v_and_b32_e32 v5, v6, v3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_and_b32_e32 v2, v2, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
+; GFX7-NEXT:    v_or_b32_e32 v4, v4, v5
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 24, v2
+; GFX7-NEXT:    v_or_b32_e32 v2, v4, v2
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  %load = load <3 x i32>, <3 x i32> addrspace(3)* %ptr, align 1
+  ret <3 x i32> %load
+}
+
+define void @store_lds_v4i32_align1(<4 x i32> addrspace(3)* %out, <4 x i32> %x) {
+; GFX9-LABEL: store_lds_v4i32_align1:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_write_b128 v0, v[1:4]
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: store_lds_v4i32_align1:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 8, v1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 16, v1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 24, v1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 8, v2
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v9, 16, v2
+; GFX7-NEXT:    v_lshrrev_b32_e32 v10, 24, v2
+; GFX7-NEXT:    ds_write_b8 v0, v1
+; GFX7-NEXT:    ds_write_b8 v0, v5 offset:1
+; GFX7-NEXT:    ds_write_b8 v0, v6 offset:2
+; GFX7-NEXT:    ds_write_b8 v0, v7 offset:3
+; GFX7-NEXT:    ds_write_b8 v0, v2 offset:4
+; GFX7-NEXT:    ds_write_b8 v0, v8 offset:5
+; GFX7-NEXT:    ds_write_b8 v0, v9 offset:6
+; GFX7-NEXT:    ds_write_b8 v0, v10 offset:7
+; GFX7-NEXT:    v_lshrrev_b32_e32 v1, 8, v3
+; GFX7-NEXT:    v_lshrrev_b32_e32 v2, 16, v3
+; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 24, v3
+; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 8, v4
+; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 16, v4
+; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 24, v4
+; GFX7-NEXT:    ds_write_b8 v0, v3 offset:8
+; GFX7-NEXT:    ds_write_b8 v0, v1 offset:9
+; GFX7-NEXT:    ds_write_b8 v0, v2 offset:10
+; GFX7-NEXT:    ds_write_b8 v0, v5 offset:11
+; GFX7-NEXT:    ds_write_b8 v0, v4 offset:12
+; GFX7-NEXT:    ds_write_b8 v0, v6 offset:13
+; GFX7-NEXT:    ds_write_b8 v0, v7 offset:14
+; GFX7-NEXT:    ds_write_b8 v0, v8 offset:15
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  store <4 x i32> %x, <4 x i32> addrspace(3)* %out, align 1
+  ret void
+}
+
+define void @store_lds_v3i32_align1(<3 x i32> addrspace(3)* %out, <3 x i32> %x) {
+; GFX9-LABEL: store_lds_v3i32_align1:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-NEXT:    ds_write_b96 v0, v[1:3]
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX7-LABEL: store_lds_v3i32_align1:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX7-NEXT:    v_lshrrev_b32_e32 v4, 8, v1
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 16, v1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 24, v1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 8, v2
+; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 16, v2
+; GFX7-NEXT:    v_lshrrev_b32_e32 v9, 24, v2
+; GFX7-NEXT:    ds_write_b8 v0, v1
+; GFX7-NEXT:    ds_write_b8 v0, v4 offset:1
+; GFX7-NEXT:    ds_write_b8 v0, v5 offset:2
+; GFX7-NEXT:    ds_write_b8 v0, v6 offset:3
+; GFX7-NEXT:    ds_write_b8 v0, v2 offset:4
+; GFX7-NEXT:    ds_write_b8 v0, v7 offset:5
+; GFX7-NEXT:    ds_write_b8 v0, v8 offset:6
+; GFX7-NEXT:    ds_write_b8 v0, v9 offset:7
+; GFX7-NEXT:    v_lshrrev_b32_e32 v1, 8, v3
+; GFX7-NEXT:    v_lshrrev_b32_e32 v2, 16, v3
+; GFX7-NEXT:    v_lshrrev_b32_e32 v4, 24, v3
+; GFX7-NEXT:    ds_write_b8 v0, v3 offset:8
+; GFX7-NEXT:    ds_write_b8 v0, v1 offset:9
+; GFX7-NEXT:    ds_write_b8 v0, v2 offset:10
+; GFX7-NEXT:    ds_write_b8 v0, v4 offset:11
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_setpc_b64 s[30:31]
+  store <3 x i32> %x, <3 x i32> addrspace(3)* %out, align 1
+  ret void
+}

llvm/test/CodeGen/AMDGPU/GlobalISel/store-local.128.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX9 %s
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=hawaii -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX7 %s
+
+; FIXME:
+; XUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=tahiti -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX6 %s
+
+define amdgpu_kernel void @store_lds_v4i32(<4 x i32> addrspace(3)* %out, <4 x i32> %x) {
+; GFX9-LABEL: store_lds_v4i32:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v4, s4
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-NEXT:    ds_write_b128 v4, v[0:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v4i32:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v4, s4
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s1
+; GFX7-NEXT:    v_mov_b32_e32 v2, s2
+; GFX7-NEXT:    v_mov_b32_e32 v3, s3
+; GFX7-NEXT:    ds_write_b128 v4, v[0:3]
+; GFX7-NEXT:    s_endpgm
+  store <4 x i32> %x, <4 x i32> addrspace(3)* %out
+  ret void
+}
+
+define amdgpu_kernel void @store_lds_v4i32_align1(<4 x i32> addrspace(3)* %out, <4 x i32> %x) {
+; GFX9-LABEL: store_lds_v4i32_align1:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v1, s4
+; GFX9-NEXT:    s_lshr_b32 s5, s0, 8
+; GFX9-NEXT:    s_lshr_b32 s6, s0, 16
+; GFX9-NEXT:    s_lshr_b32 s7, s0, 24
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    s_lshr_b32 s0, s1, 8
+; GFX9-NEXT:    v_mov_b32_e32 v2, s5
+; GFX9-NEXT:    s_lshr_b32 s4, s1, 16
+; GFX9-NEXT:    s_lshr_b32 s5, s1, 24
+; GFX9-NEXT:    v_mov_b32_e32 v5, s1
+; GFX9-NEXT:    v_mov_b32_e32 v6, s0
+; GFX9-NEXT:    v_mov_b32_e32 v3, s6
+; GFX9-NEXT:    s_lshr_b32 s0, s2, 8
+; GFX9-NEXT:    s_lshr_b32 s1, s2, 16
+; GFX9-NEXT:    v_mov_b32_e32 v7, s4
+; GFX9-NEXT:    v_mov_b32_e32 v4, s7
+; GFX9-NEXT:    v_mov_b32_e32 v8, s5
+; GFX9-NEXT:    ds_write_b8 v1, v0
+; GFX9-NEXT:    ds_write_b8 v1, v2 offset:1
+; GFX9-NEXT:    ds_write_b8 v1, v3 offset:2
+; GFX9-NEXT:    ds_write_b8 v1, v4 offset:3
+; GFX9-NEXT:    ds_write_b8 v1, v5 offset:4
+; GFX9-NEXT:    ds_write_b8 v1, v6 offset:5
+; GFX9-NEXT:    ds_write_b8 v1, v7 offset:6
+; GFX9-NEXT:    ds_write_b8 v1, v8 offset:7
+; GFX9-NEXT:    s_lshr_b32 s4, s2, 24
+; GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GFX9-NEXT:    v_mov_b32_e32 v2, s0
+; GFX9-NEXT:    v_mov_b32_e32 v3, s1
+; GFX9-NEXT:    s_lshr_b32 s0, s3, 8
+; GFX9-NEXT:    s_lshr_b32 s1, s3, 16
+; GFX9-NEXT:    s_lshr_b32 s2, s3, 24
+; GFX9-NEXT:    v_mov_b32_e32 v4, s4
+; GFX9-NEXT:    v_mov_b32_e32 v5, s3
+; GFX9-NEXT:    v_mov_b32_e32 v6, s0
+; GFX9-NEXT:    v_mov_b32_e32 v7, s1
+; GFX9-NEXT:    v_mov_b32_e32 v8, s2
+; GFX9-NEXT:    ds_write_b8 v1, v0 offset:8
+; GFX9-NEXT:    ds_write_b8 v1, v2 offset:9
+; GFX9-NEXT:    ds_write_b8 v1, v3 offset:10
+; GFX9-NEXT:    ds_write_b8 v1, v4 offset:11
+; GFX9-NEXT:    ds_write_b8 v1, v5 offset:12
+; GFX9-NEXT:    ds_write_b8 v1, v6 offset:13
+; GFX9-NEXT:    ds_write_b8 v1, v7 offset:14
+; GFX9-NEXT:    ds_write_b8 v1, v8 offset:15
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v4i32_align1:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v1, s4
+; GFX7-NEXT:    s_lshr_b32 s5, s0, 8
+; GFX7-NEXT:    s_lshr_b32 s6, s0, 16
+; GFX7-NEXT:    s_lshr_b32 s7, s0, 24
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    s_lshr_b32 s0, s1, 8
+; GFX7-NEXT:    v_mov_b32_e32 v2, s5
+; GFX7-NEXT:    s_lshr_b32 s4, s1, 16
+; GFX7-NEXT:    s_lshr_b32 s5, s1, 24
+; GFX7-NEXT:    v_mov_b32_e32 v5, s1
+; GFX7-NEXT:    v_mov_b32_e32 v6, s0
+; GFX7-NEXT:    v_mov_b32_e32 v3, s6
+; GFX7-NEXT:    s_lshr_b32 s0, s2, 8
+; GFX7-NEXT:    s_lshr_b32 s1, s2, 16
+; GFX7-NEXT:    v_mov_b32_e32 v7, s4
+; GFX7-NEXT:    v_mov_b32_e32 v4, s7
+; GFX7-NEXT:    v_mov_b32_e32 v8, s5
+; GFX7-NEXT:    ds_write_b8 v1, v0
+; GFX7-NEXT:    ds_write_b8 v1, v2 offset:1
+; GFX7-NEXT:    ds_write_b8 v1, v3 offset:2
+; GFX7-NEXT:    ds_write_b8 v1, v4 offset:3
+; GFX7-NEXT:    ds_write_b8 v1, v5 offset:4
+; GFX7-NEXT:    ds_write_b8 v1, v6 offset:5
+; GFX7-NEXT:    ds_write_b8 v1, v7 offset:6
+; GFX7-NEXT:    ds_write_b8 v1, v8 offset:7
+; GFX7-NEXT:    s_lshr_b32 s4, s2, 24
+; GFX7-NEXT:    v_mov_b32_e32 v0, s2
+; GFX7-NEXT:    v_mov_b32_e32 v2, s0
+; GFX7-NEXT:    v_mov_b32_e32 v3, s1
+; GFX7-NEXT:    s_lshr_b32 s0, s3, 8
+; GFX7-NEXT:    s_lshr_b32 s1, s3, 16
+; GFX7-NEXT:    s_lshr_b32 s2, s3, 24
+; GFX7-NEXT:    v_mov_b32_e32 v4, s4
+; GFX7-NEXT:    v_mov_b32_e32 v5, s3
+; GFX7-NEXT:    v_mov_b32_e32 v6, s0
+; GFX7-NEXT:    v_mov_b32_e32 v7, s1
+; GFX7-NEXT:    v_mov_b32_e32 v8, s2
+; GFX7-NEXT:    ds_write_b8 v1, v0 offset:8
+; GFX7-NEXT:    ds_write_b8 v1, v2 offset:9
+; GFX7-NEXT:    ds_write_b8 v1, v3 offset:10
+; GFX7-NEXT:    ds_write_b8 v1, v4 offset:11
+; GFX7-NEXT:    ds_write_b8 v1, v5 offset:12
+; GFX7-NEXT:    ds_write_b8 v1, v6 offset:13
+; GFX7-NEXT:    ds_write_b8 v1, v7 offset:14
+; GFX7-NEXT:    ds_write_b8 v1, v8 offset:15
+; GFX7-NEXT:    s_endpgm
+  store <4 x i32> %x, <4 x i32> addrspace(3)* %out, align 1
+  ret void
+}
+
+define amdgpu_kernel void @store_lds_v4i32_align2(<4 x i32> addrspace(3)* %out, <4 x i32> %x) {
+; GFX9-LABEL: store_lds_v4i32_align2:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v1, s4
+; GFX9-NEXT:    s_lshr_b32 s5, s0, 16
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    s_lshr_b32 s0, s1, 16
+; GFX9-NEXT:    v_mov_b32_e32 v4, s0
+; GFX9-NEXT:    s_lshr_b32 s0, s2, 16
+; GFX9-NEXT:    v_mov_b32_e32 v6, s0
+; GFX9-NEXT:    s_lshr_b32 s0, s3, 16
+; GFX9-NEXT:    v_mov_b32_e32 v2, s5
+; GFX9-NEXT:    v_mov_b32_e32 v3, s1
+; GFX9-NEXT:    v_mov_b32_e32 v5, s2
+; GFX9-NEXT:    v_mov_b32_e32 v7, s3
+; GFX9-NEXT:    v_mov_b32_e32 v8, s0
+; GFX9-NEXT:    ds_write_b16 v1, v0
+; GFX9-NEXT:    ds_write_b16 v1, v2 offset:2
+; GFX9-NEXT:    ds_write_b16 v1, v3 offset:4
+; GFX9-NEXT:    ds_write_b16 v1, v4 offset:6
+; GFX9-NEXT:    ds_write_b16 v1, v5 offset:8
+; GFX9-NEXT:    ds_write_b16 v1, v6 offset:10
+; GFX9-NEXT:    ds_write_b16 v1, v7 offset:12
+; GFX9-NEXT:    ds_write_b16 v1, v8 offset:14
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v4i32_align2:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v1, s4
+; GFX7-NEXT:    s_lshr_b32 s5, s0, 16
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    s_lshr_b32 s0, s1, 16
+; GFX7-NEXT:    v_mov_b32_e32 v4, s0
+; GFX7-NEXT:    s_lshr_b32 s0, s2, 16
+; GFX7-NEXT:    v_mov_b32_e32 v6, s0
+; GFX7-NEXT:    s_lshr_b32 s0, s3, 16
+; GFX7-NEXT:    v_mov_b32_e32 v2, s5
+; GFX7-NEXT:    v_mov_b32_e32 v3, s1
+; GFX7-NEXT:    v_mov_b32_e32 v5, s2
+; GFX7-NEXT:    v_mov_b32_e32 v7, s3
+; GFX7-NEXT:    v_mov_b32_e32 v8, s0
+; GFX7-NEXT:    ds_write_b16 v1, v0
+; GFX7-NEXT:    ds_write_b16 v1, v2 offset:2
+; GFX7-NEXT:    ds_write_b16 v1, v3 offset:4
+; GFX7-NEXT:    ds_write_b16 v1, v4 offset:6
+; GFX7-NEXT:    ds_write_b16 v1, v5 offset:8
+; GFX7-NEXT:    ds_write_b16 v1, v6 offset:10
+; GFX7-NEXT:    ds_write_b16 v1, v7 offset:12
+; GFX7-NEXT:    ds_write_b16 v1, v8 offset:14
+; GFX7-NEXT:    s_endpgm
+  store <4 x i32> %x, <4 x i32> addrspace(3)* %out, align 2
+  ret void
+}
+
+define amdgpu_kernel void @store_lds_v4i32_align4(<4 x i32> addrspace(3)* %out, <4 x i32> %x) {
+; GFX9-LABEL: store_lds_v4i32_align4:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v4, s4
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-NEXT:    ds_write_b128 v4, v[0:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v4i32_align4:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v1, s4
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    v_mov_b32_e32 v2, s1
+; GFX7-NEXT:    ds_write2_b32 v1, v0, v2 offset1:1
+; GFX7-NEXT:    v_mov_b32_e32 v0, s2
+; GFX7-NEXT:    v_mov_b32_e32 v2, s3
+; GFX7-NEXT:    ds_write2_b32 v1, v0, v2 offset0:2 offset1:3
+; GFX7-NEXT:    s_endpgm
+  store <4 x i32> %x, <4 x i32> addrspace(3)* %out, align 4
+  ret void
+}
+
+define amdgpu_kernel void @store_lds_v4i32_align8(<4 x i32> addrspace(3)* %out, <4 x i32> %x) {
+; GFX9-LABEL: store_lds_v4i32_align8:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v4, s4
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-NEXT:    ds_write_b128 v4, v[0:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v4i32_align8:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v4, s4
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s1
+; GFX7-NEXT:    v_mov_b32_e32 v2, s2
+; GFX7-NEXT:    v_mov_b32_e32 v3, s3
+; GFX7-NEXT:    ds_write2_b64 v4, v[0:1], v[2:3] offset1:1
+; GFX7-NEXT:    s_endpgm
+  store <4 x i32> %x, <4 x i32> addrspace(3)* %out, align 8
+  ret void
+}
+
+define amdgpu_kernel void @store_lds_v4i32_align16(<4 x i32> addrspace(3)* %out, <4 x i32> %x) {
+; GFX9-LABEL: store_lds_v4i32_align16:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v4, s4
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-NEXT:    ds_write_b128 v4, v[0:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v4i32_align16:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v4, s4
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s1
+; GFX7-NEXT:    v_mov_b32_e32 v2, s2
+; GFX7-NEXT:    v_mov_b32_e32 v3, s3
+; GFX7-NEXT:    ds_write_b128 v4, v[0:3]
+; GFX7-NEXT:    s_endpgm
+  store <4 x i32> %x, <4 x i32> addrspace(3)* %out, align 16
+  ret void
+}

llvm/test/CodeGen/AMDGPU/GlobalISel/store-local.96.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX9 %s
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=hawaii -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX7 %s
+
+; FIXME:
+; XUN: llc -global-isel -mtriple=amdgcn-amd-amdpal -mcpu=tahiti -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX6 %s
+
+define amdgpu_kernel void @store_lds_v3i32(<3 x i32> addrspace(3)* %out, <3 x i32> %x) {
+; GFX9-LABEL: store_lds_v3i32:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v3, s4
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; GFX9-NEXT:    ds_write_b96 v3, v[0:2]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v3i32:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v3, s4
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s1
+; GFX7-NEXT:    v_mov_b32_e32 v2, s2
+; GFX7-NEXT:    ds_write_b96 v3, v[0:2]
+; GFX7-NEXT:    s_endpgm
+  store <3 x i32> %x, <3 x i32> addrspace(3)* %out
+  ret void
+}
+
+define amdgpu_kernel void @store_lds_v3i32_align1(<3 x i32> addrspace(3)* %out, <3 x i32> %x) {
+; GFX9-LABEL: store_lds_v3i32_align1:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v1, s4
+; GFX9-NEXT:    s_lshr_b32 s3, s0, 8
+; GFX9-NEXT:    s_lshr_b32 s5, s0, 16
+; GFX9-NEXT:    s_lshr_b32 s6, s0, 24
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s3
+; GFX9-NEXT:    s_lshr_b32 s0, s1, 8
+; GFX9-NEXT:    s_lshr_b32 s3, s1, 16
+; GFX9-NEXT:    s_lshr_b32 s4, s1, 24
+; GFX9-NEXT:    v_mov_b32_e32 v5, s1
+; GFX9-NEXT:    v_mov_b32_e32 v6, s0
+; GFX9-NEXT:    v_mov_b32_e32 v7, s3
+; GFX9-NEXT:    v_mov_b32_e32 v3, s5
+; GFX9-NEXT:    v_mov_b32_e32 v4, s6
+; GFX9-NEXT:    v_mov_b32_e32 v8, s4
+; GFX9-NEXT:    ds_write_b8 v1, v0
+; GFX9-NEXT:    ds_write_b8 v1, v2 offset:1
+; GFX9-NEXT:    ds_write_b8 v1, v3 offset:2
+; GFX9-NEXT:    ds_write_b8 v1, v4 offset:3
+; GFX9-NEXT:    ds_write_b8 v1, v5 offset:4
+; GFX9-NEXT:    ds_write_b8 v1, v6 offset:5
+; GFX9-NEXT:    ds_write_b8 v1, v7 offset:6
+; GFX9-NEXT:    ds_write_b8 v1, v8 offset:7
+; GFX9-NEXT:    s_lshr_b32 s0, s2, 8
+; GFX9-NEXT:    s_lshr_b32 s1, s2, 16
+; GFX9-NEXT:    s_lshr_b32 s3, s2, 24
+; GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GFX9-NEXT:    v_mov_b32_e32 v2, s0
+; GFX9-NEXT:    v_mov_b32_e32 v3, s1
+; GFX9-NEXT:    v_mov_b32_e32 v4, s3
+; GFX9-NEXT:    ds_write_b8 v1, v0 offset:8
+; GFX9-NEXT:    ds_write_b8 v1, v2 offset:9
+; GFX9-NEXT:    ds_write_b8 v1, v3 offset:10
+; GFX9-NEXT:    ds_write_b8 v1, v4 offset:11
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v3i32_align1:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v1, s4
+; GFX7-NEXT:    s_lshr_b32 s3, s0, 8
+; GFX7-NEXT:    s_lshr_b32 s5, s0, 16
+; GFX7-NEXT:    s_lshr_b32 s6, s0, 24
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    v_mov_b32_e32 v2, s3
+; GFX7-NEXT:    s_lshr_b32 s0, s1, 8
+; GFX7-NEXT:    s_lshr_b32 s3, s1, 16
+; GFX7-NEXT:    s_lshr_b32 s4, s1, 24
+; GFX7-NEXT:    v_mov_b32_e32 v5, s1
+; GFX7-NEXT:    v_mov_b32_e32 v6, s0
+; GFX7-NEXT:    v_mov_b32_e32 v7, s3
+; GFX7-NEXT:    v_mov_b32_e32 v3, s5
+; GFX7-NEXT:    v_mov_b32_e32 v4, s6
+; GFX7-NEXT:    v_mov_b32_e32 v8, s4
+; GFX7-NEXT:    ds_write_b8 v1, v0
+; GFX7-NEXT:    ds_write_b8 v1, v2 offset:1
+; GFX7-NEXT:    ds_write_b8 v1, v3 offset:2
+; GFX7-NEXT:    ds_write_b8 v1, v4 offset:3
+; GFX7-NEXT:    ds_write_b8 v1, v5 offset:4
+; GFX7-NEXT:    ds_write_b8 v1, v6 offset:5
+; GFX7-NEXT:    ds_write_b8 v1, v7 offset:6
+; GFX7-NEXT:    ds_write_b8 v1, v8 offset:7
+; GFX7-NEXT:    s_lshr_b32 s0, s2, 8
+; GFX7-NEXT:    s_lshr_b32 s1, s2, 16
+; GFX7-NEXT:    s_lshr_b32 s3, s2, 24
+; GFX7-NEXT:    v_mov_b32_e32 v0, s2
+; GFX7-NEXT:    v_mov_b32_e32 v2, s0
+; GFX7-NEXT:    v_mov_b32_e32 v3, s1
+; GFX7-NEXT:    v_mov_b32_e32 v4, s3
+; GFX7-NEXT:    ds_write_b8 v1, v0 offset:8
+; GFX7-NEXT:    ds_write_b8 v1, v2 offset:9
+; GFX7-NEXT:    ds_write_b8 v1, v3 offset:10
+; GFX7-NEXT:    ds_write_b8 v1, v4 offset:11
+; GFX7-NEXT:    s_endpgm
+  store <3 x i32> %x, <3 x i32> addrspace(3)* %out, align 1
+  ret void
+}
+
+define amdgpu_kernel void @store_lds_v3i32_align2(<3 x i32> addrspace(3)* %out, <3 x i32> %x) {
+; GFX9-LABEL: store_lds_v3i32_align2:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v1, s4
+; GFX9-NEXT:    s_lshr_b32 s3, s0, 16
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    s_lshr_b32 s0, s1, 16
+; GFX9-NEXT:    v_mov_b32_e32 v4, s0
+; GFX9-NEXT:    s_lshr_b32 s0, s2, 16
+; GFX9-NEXT:    v_mov_b32_e32 v2, s3
+; GFX9-NEXT:    v_mov_b32_e32 v3, s1
+; GFX9-NEXT:    v_mov_b32_e32 v5, s2
+; GFX9-NEXT:    v_mov_b32_e32 v6, s0
+; GFX9-NEXT:    ds_write_b16 v1, v0
+; GFX9-NEXT:    ds_write_b16 v1, v2 offset:2
+; GFX9-NEXT:    ds_write_b16 v1, v3 offset:4
+; GFX9-NEXT:    ds_write_b16 v1, v4 offset:6
+; GFX9-NEXT:    ds_write_b16 v1, v5 offset:8
+; GFX9-NEXT:    ds_write_b16 v1, v6 offset:10
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v3i32_align2:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v1, s4
+; GFX7-NEXT:    s_lshr_b32 s3, s0, 16
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    s_lshr_b32 s0, s1, 16
+; GFX7-NEXT:    v_mov_b32_e32 v4, s0
+; GFX7-NEXT:    s_lshr_b32 s0, s2, 16
+; GFX7-NEXT:    v_mov_b32_e32 v2, s3
+; GFX7-NEXT:    v_mov_b32_e32 v3, s1
+; GFX7-NEXT:    v_mov_b32_e32 v5, s2
+; GFX7-NEXT:    v_mov_b32_e32 v6, s0
+; GFX7-NEXT:    ds_write_b16 v1, v0
+; GFX7-NEXT:    ds_write_b16 v1, v2 offset:2
+; GFX7-NEXT:    ds_write_b16 v1, v3 offset:4
+; GFX7-NEXT:    ds_write_b16 v1, v4 offset:6
+; GFX7-NEXT:    ds_write_b16 v1, v5 offset:8
+; GFX7-NEXT:    ds_write_b16 v1, v6 offset:10
+; GFX7-NEXT:    s_endpgm
+  store <3 x i32> %x, <3 x i32> addrspace(3)* %out, align 2
+  ret void
+}
+
+define amdgpu_kernel void @store_lds_v3i32_align4(<3 x i32> addrspace(3)* %out, <3 x i32> %x) {
+; GFX9-LABEL: store_lds_v3i32_align4:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v3, s4
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; GFX9-NEXT:    ds_write_b96 v3, v[0:2]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v3i32_align4:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v2, s4
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s1
+; GFX7-NEXT:    v_mov_b32_e32 v3, s2
+; GFX7-NEXT:    ds_write2_b32 v2, v0, v1 offset1:1
+; GFX7-NEXT:    ds_write_b32 v2, v3 offset:8
+; GFX7-NEXT:    s_endpgm
+  store <3 x i32> %x, <3 x i32> addrspace(3)* %out, align 4
+  ret void
+}
+
+define amdgpu_kernel void @store_lds_v3i32_align8(<3 x i32> addrspace(3)* %out, <3 x i32> %x) {
+; GFX9-LABEL: store_lds_v3i32_align8:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v3, s4
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; GFX9-NEXT:    ds_write_b96 v3, v[0:2]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v3i32_align8:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v2, s4
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s1
+; GFX7-NEXT:    v_mov_b32_e32 v3, s2
+; GFX7-NEXT:    ds_write_b64 v2, v[0:1]
+; GFX7-NEXT:    ds_write_b32 v2, v3 offset:8
+; GFX7-NEXT:    s_endpgm
+  store <3 x i32> %x, <3 x i32> addrspace(3)* %out, align 8
+  ret void
+}
+
+define amdgpu_kernel void @store_lds_v3i32_align16(<3 x i32> addrspace(3)* %out, <3 x i32> %x) {
+; GFX9-LABEL: store_lds_v3i32_align16:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v3, s4
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; GFX9-NEXT:    ds_write_b96 v3, v[0:2]
+; GFX9-NEXT:    s_endpgm
+;
+; GFX7-LABEL: store_lds_v3i32_align16:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dword s4, s[0:1], 0x9
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0xd
+; GFX7-NEXT:    s_mov_b32 m0, -1
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    v_mov_b32_e32 v3, s4
+; GFX7-NEXT:    v_mov_b32_e32 v0, s0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s1
+; GFX7-NEXT:    v_mov_b32_e32 v2, s2
+; GFX7-NEXT:    ds_write_b96 v3, v[0:2]
+; GFX7-NEXT:    s_endpgm
+  store <3 x i32> %x, <3 x i32> addrspace(3)* %out, align 16
+  ret void
+}