[AMDGPU] Legalize VGPR Rsrc operands for MUBUF instructions

[scott.linder]

Summary

Emit a waterfall loop in the general case for a potentially-divergent Rsrc operand. When possible, avoid this by using Addr64 instructions.

The old legalization assumed Addr64 was available, and only handled the _ADDR64 and _OFFSET variants of MUBUF instructions.

I am not sure what the best way to identify Addr64 MUBUF opcodes is, but since we need the InstrMapping in order to convert _OFFSET to _ADDR64 anyway I tried to reuse that mapping.

[arsenm]

In this case I think we should have some pure-IR tests since we'll need to worry about this in GlobalISel as well.

You are missing some tests involving control flow that are likely to be broken. Some cases I would worry about:

• Expansion outside of the entry block, with values live across block boundaries
• Expansion of two adjacent operations that introduce control flow
• Expansion in multiple blocks in the function

I think there were a few other problematic cases I ran into for the vector indexing waterfall

lib/Target/AMDGPU/SIInstrInfo.cpp

3481–3486

I was hoping for this to a be a more general function that could be re-used for other operations in the future, and probably sharable with the vector indexing code.

I was thinking about something like a list of operands that need to be SGPRs, and iterating over all of the subregisters rather than assuming the SGPR128 + sometimes SGPR32 the buffer operations use

3490

This is a bool

3546–3548

Can you post a separate patch to fix the vector indexing loop not using this?

test/CodeGen/AMDGPU/mubuf-legalize-operands.mir

2

I think this needs a -verify-machineinstrs

17–18

If you're not going to write out the variable checks for these, you're probably better off just using update_mir_test_checks

[arsenm]

I think with this we can also fix the broken definition for const-32bit address space of always giving a uniform result

[scott.linder]

I will add some IR tests for at least those cases.

I don't understand what you mean about the const-32bit addrspace; can you explain more?

lib/Target/AMDGPU/SIInstrInfo.cpp

3481–3486

I can work on this; would you prefer I do that in this patch?

3490

I thought isUndef was a bool and getUndefRegState returned flags.

3546–3548

Yes, I was planning on doing that. Alternatively if I unify the two in this patch that will also handle it.

test/CodeGen/AMDGPU/mubuf-legalize-operands.mir

17–18

I will add the variable checks here.

[arsenm]

In D50982#1207797, @scott.linder wrote:

I will add some IR tests for at least those cases.

I don't understand what you mean about the const-32bit addrspace; can you explain more?

I think part of the shader assume uniform hack is we claim that loads from the 32-bit const address space are uniform, such that it makes it "ok" to use the single readlane. This was always a broken concept though. See isSDNodeAlwaysUniform handling of load, and the documentation

lib/Target/AMDGPU/SIInstrInfo.cpp

3481–3486

That can be a second patch

3490

Yes, you're right

[scott.linder]

Use variables for registers in MIR tests. Add IR tests, trying to cover problem cases around introducing control flow. There is another case in the indirect-addressing tests with an SGPR def between the two operations, but I'm not sure what it is testing. Are the tests I added useful, and are there still more you can think of to add?

I will start working on a patch with a generic waterfall emitter, and I will talk to the graphics devs about how to fix the readfirstlane shortcut and the addrspace hack.

[nhaehnle]

I find it a bit concerning that we seem to have different semantics for these instructions depending on the environment.

Graphics really wants to legalize the instructions using a single v_readfirstlane (without the loop overhead), because that's what the relevant semantics are there. This patch doesn't change that, so it's not too bad, but please keep it in mind.

[arsenm]

We should probably have some real executing tests that stress this in piglit or somewhere else.

A solution to the graphics semantics problem might be to use a readfirstlane intrinsic call as an input as an assertion the input is known uniform. We can then optimize out patterns that look like V_READFIRSTLANE (COPY SGPR to VGPR)

[arsenm]

test/CodeGen/AMDGPU/mubuf-legalize-operands.ll

2

Probably should have an -O0 run line to make sure spills work correctly. Full check lines for every test for this is probably too exhausting for doing it manually, so maybe just a case with a cross block use

12

What's this?

[scott.linder]

I think even in the simplest case my code is wrong. The $soffset operand to the BUFFER_LOAD remains killed but the def is not moved into the LoopBB. I am essentially transforming:

%soffset:sreg_32_xm0 = S_MOV_B32 0
BUFFER_LOAD ..., killed %soffset:sreg_32_xm0, ...

Into:

%soffset:sreg_32_xm0 = S_MOV_B32 0
bb1:
BUFFER_LOAD ..., killed %soffset:sreg_32_xm0, ...
S_CBRANCH bb1

Is it correct to just clear the killed flag on the operand? Should I do the same for the $vindex/$voffset/$vaddr operands?

test/CodeGen/AMDGPU/mubuf-legalize-operands.ll

12

This is me trying to be sure the uses occurs after the defs, even though the order within each doesn't matter. My understanding is that without the divider the cmp's could appear before the readfirstlanes and because they are adjacent -DAG it would still match. The chances of it seem low, and maybe there is a better way than a dummy CHECK-NOT to prevent it?

[arsenm]

You can just unconditionally remove kill flags. They are more or less deprecated, and fully recomputed later before RA. We do make use of them for a few things. Pre-RA they are only really useful for a few places that care about physical registers

[arsenm]

test/CodeGen/AMDGPU/mubuf-legalize-operands.ll

12

I'm not sure that's how CHECK-NOT works, although there were some recent changes to try to clarify the behavior of CHECK-DAG I haven't looked into

[scott.linder]

Clear killed flag on uses, add test at -O0

[arsenm]

lib/Target/AMDGPU/SIInstrInfo.cpp

3571–3575

Use MRI.clearKillFlags

[scott.linder]

Use MRI.clearKillFlags

[scott.linder]

Ping

[arsenm]

LGTM

lib/Target/AMDGPU/SIInstrInfo.h

namespace AMDGPU {
   int getCommuteRev(uint16_t Opcode);
 
   LLVM_READONLY
   int getCommuteOrig(uint16_t Opcode);
 
   LLVM_READONLY
   int getAddr64Inst(uint16_t Opcode);
 
+  /// Check if \p Opcode is an Addr64 opcode.
+  ///
+  /// \returns \p Opcode if it is an Addr64 opcode, otherwise -1.
+  LLVM_READONLY
+  int getIfAddr64Inst(uint16_t Opcode);
+
   LLVM_READONLY
   int getMUBUFNoLdsInst(uint16_t Opcode);
 
   LLVM_READONLY
   int getAtomicRetOp(uint16_t Opcode);
 
   LLVM_READONLY
   int getAtomicNoRetOp(uint16_t Opcode);

lib/Target/AMDGPU/SIInstrInfo.cpp

void SIInstrInfo::legalizeGenericOperand(MachineBasicBlock &InsertMBB,
   if (!Def)
     return;
 
   // Try to eliminate the copy if it is copying an immediate value.
   if (Def->isMoveImmediate())
     FoldImmediate(*Copy, *Def, OpReg, &MRI);
 }
 
+// Emit the actual waterfall loop, executing the wrapped instruction for each
+// unique value of \p Rsrc across all lanes. In the best case we execute 1
+// iteration, in the worst case we execute 64 (once per lane).
+static void emitLoadSRsrcFromVGPRLoop(const SIInstrInfo &TII,
+                                      MachineRegisterInfo &MRI,
+                                      MachineBasicBlock &OrigBB,
+                                      MachineBasicBlock &LoopBB,
+                                      const DebugLoc &DL,
+                                      MachineOperand &Rsrc) {
+  MachineBasicBlock::iterator I = LoopBB.begin();
+
+  unsigned VRsrc = Rsrc.getReg();
+  unsigned VRsrcUndef = getUndefRegState(Rsrc.isUndef());
+
+  unsigned SaveExec = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass);
+  unsigned CondReg0 = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass);
+  unsigned CondReg1 = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass);
+  unsigned AndCond = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass);
+  unsigned SRsrcSub0 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
+  unsigned SRsrcSub1 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
+  unsigned SRsrcSub2 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
+  unsigned SRsrcSub3 = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
+  unsigned SRsrc = MRI.createVirtualRegister(&AMDGPU::SReg_128RegClass);
+
+  // Beginning of the loop, read the next Rsrc variant.
+  BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_READFIRSTLANE_B32), SRsrcSub0)
+    .addReg(VRsrc, VRsrcUndef, AMDGPU::sub0);
+  BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_READFIRSTLANE_B32), SRsrcSub1)
+    .addReg(VRsrc, VRsrcUndef, AMDGPU::sub1);
+  BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_READFIRSTLANE_B32), SRsrcSub2)
+    .addReg(VRsrc, VRsrcUndef, AMDGPU::sub2);
+  BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_READFIRSTLANE_B32), SRsrcSub3)
+    .addReg(VRsrc, VRsrcUndef, AMDGPU::sub3);
+
+  BuildMI(LoopBB, I, DL, TII.get(AMDGPU::REG_SEQUENCE), SRsrc)
+    .addReg(SRsrcSub0)
+    .addImm(AMDGPU::sub0)
+    .addReg(SRsrcSub1)
+    .addImm(AMDGPU::sub1)
+    .addReg(SRsrcSub2)
+    .addImm(AMDGPU::sub2)
+    .addReg(SRsrcSub3)
+    .addImm(AMDGPU::sub3);
+
+  // Update Rsrc operand to use the SGPR Rsrc.
+  Rsrc.setReg(SRsrc);
+
+  // Identify all lanes with identical Rsrc operands in their VGPRs.
+  BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_CMP_EQ_U64_e64), CondReg0)
+    .addReg(SRsrc, 0, AMDGPU::sub0_sub1)
+    .addReg(VRsrc, 0, AMDGPU::sub0_sub1);
+  BuildMI(LoopBB, I, DL, TII.get(AMDGPU::V_CMP_EQ_U64_e64), CondReg1)
+    .addReg(SRsrc, 0, AMDGPU::sub2_sub3)
+    .addReg(VRsrc, 0, AMDGPU::sub2_sub3);
+  BuildMI(LoopBB, I, DL, TII.get(AMDGPU::S_AND_B64), AndCond)
+    .addReg(CondReg0)
+    .addReg(CondReg1);
+
+  MRI.setSimpleHint(SaveExec, AndCond);
+
+  // Update EXEC to matching lanes, saving original to SaveExec.
+  BuildMI(LoopBB, I, DL, TII.get(AMDGPU::S_AND_SAVEEXEC_B64), SaveExec)
+    .addReg(AndCond, RegState::Kill);
+
+  // The original instruction is here; we insert the terminators after it.
+  I = LoopBB.end();
+
+  // Update EXEC, switch all done bits to 0 and all todo bits to 1.
+  BuildMI(LoopBB, I, DL, TII.get(AMDGPU::S_XOR_B64_term), AMDGPU::EXEC)
+    .addReg(AMDGPU::EXEC)
+    .addReg(SaveExec);
+  BuildMI(LoopBB, I, DL, TII.get(AMDGPU::S_CBRANCH_EXECNZ))
+    .addMBB(&LoopBB);
+}
+
+// Build a waterfall loop around \p MI, replacing the VGPR \p Rsrc register
+// with SGPRs by iterating over all unique values across all lanes.
+static void loadSRsrcFromVGPR(const SIInstrInfo &TII,
+                              MachineInstr &MI,
+                              MachineOperand &Rsrc) {
+  MachineBasicBlock &MBB = *MI.getParent();
+  MachineFunction &MF = *MBB.getParent();
+  MachineRegisterInfo &MRI = MF.getRegInfo();
+  MachineBasicBlock::iterator I(&MI);
+  const DebugLoc &DL = MI.getDebugLoc();
+
+  unsigned SaveExec = MRI.createVirtualRegister(&AMDGPU::SReg_64_XEXECRegClass);
+
+  // Save the EXEC mask
+  BuildMI(MBB, I, DL, TII.get(AMDGPU::S_MOV_B64), SaveExec)
+    .addReg(AMDGPU::EXEC);
+
+  // To insert the loop we need to split the block. Move everything after this
+  // point to a new block, and insert a new empty block between the two.
+  MachineBasicBlock *LoopBB = MF.CreateMachineBasicBlock();
+  MachineBasicBlock *RemainderBB = MF.CreateMachineBasicBlock();
+  MachineFunction::iterator MBBI(MBB);
+  ++MBBI;
+
+  MF.insert(MBBI, LoopBB);
+  MF.insert(MBBI, RemainderBB);
+
+  LoopBB->addSuccessor(LoopBB);
+  LoopBB->addSuccessor(RemainderBB);
+
+  // Move MI to the LoopBB, and the remainder of the block to RemainderBB.
+  MachineBasicBlock::iterator J = I++;
+  RemainderBB->transferSuccessorsAndUpdatePHIs(&MBB);
+  RemainderBB->splice(RemainderBB->begin(), &MBB, I, MBB.end());
+  LoopBB->splice(LoopBB->begin(), &MBB, J);
+
+  MBB.addSuccessor(LoopBB);
+
+  emitLoadSRsrcFromVGPRLoop(TII, MRI, MBB, *LoopBB, DL, Rsrc);
+
+  // Restore the EXEC mask
+  MachineBasicBlock::iterator First = RemainderBB->begin();
+  BuildMI(*RemainderBB, First, DL, TII.get(AMDGPU::S_MOV_B64), AMDGPU::EXEC)
+    .addReg(SaveExec);
+}
+
+// Extract pointer from Rsrc and return a zero-value Rsrc replacement.
+static std::tuple<unsigned, unsigned> extractRsrcPtr(const SIInstrInfo &TII,
+                                                     MachineInstr &MI,
+                                                     MachineOperand &Rsrc) {
+  MachineBasicBlock &MBB = *MI.getParent();
+  MachineFunction &MF = *MBB.getParent();
+  MachineRegisterInfo &MRI = MF.getRegInfo();
+
+  // Extract the ptr from the resource descriptor.
+  unsigned RsrcPtr = TII.buildExtractSubReg(MI, MRI, Rsrc,
+    &AMDGPU::VReg_128RegClass, AMDGPU::sub0_sub1, &AMDGPU::VReg_64RegClass);
+
+  // Create an empty resource descriptor
+  unsigned Zero64 = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass);
+  unsigned SRsrcFormatLo = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
+  unsigned SRsrcFormatHi = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
+  unsigned NewSRsrc = MRI.createVirtualRegister(&AMDGPU::SReg_128RegClass);
+  uint64_t RsrcDataFormat = TII.getDefaultRsrcDataFormat();
+
+  // Zero64 = 0
+  BuildMI(MBB, MI, MI.getDebugLoc(), TII.get(AMDGPU::S_MOV_B64), Zero64)
+      .addImm(0);
+
+  // SRsrcFormatLo = RSRC_DATA_FORMAT{31-0}
+  BuildMI(MBB, MI, MI.getDebugLoc(), TII.get(AMDGPU::S_MOV_B32), SRsrcFormatLo)
+      .addImm(RsrcDataFormat & 0xFFFFFFFF);
+
+  // SRsrcFormatHi = RSRC_DATA_FORMAT{63-32}
+  BuildMI(MBB, MI, MI.getDebugLoc(), TII.get(AMDGPU::S_MOV_B32), SRsrcFormatHi)
+      .addImm(RsrcDataFormat >> 32);
+
+  // NewSRsrc = {Zero64, SRsrcFormat}
+  BuildMI(MBB, MI, MI.getDebugLoc(), TII.get(AMDGPU::REG_SEQUENCE), NewSRsrc)
+      .addReg(Zero64)
+      .addImm(AMDGPU::sub0_sub1)
+      .addReg(SRsrcFormatLo)
+      .addImm(AMDGPU::sub2)
+      .addReg(SRsrcFormatHi)
+      .addImm(AMDGPU::sub3);
+
+  return std::tie(RsrcPtr, NewSRsrc);
+}
+
 void SIInstrInfo::legalizeOperands(MachineInstr &MI) const {
   MachineFunction &MF = *MI.getParent()->getParent();
   MachineRegisterInfo &MRI = MF.getRegInfo();
 
   // Legalize VOP2
   if (isVOP2(MI) || isVOPC(MI)) {
     legalizeOperandsVOP2(MRI, MI);
     return;
if (isMIMG(MI) ||
     MachineOperand *SSamp = getNamedOperand(MI, AMDGPU::OpName::ssamp);
     if (SSamp && !RI.isSGPRClass(MRI.getRegClass(SSamp->getReg()))) {
       unsigned SGPR = readlaneVGPRToSGPR(SSamp->getReg(), MI, MRI);
       SSamp->setReg(SGPR);
     }
     return;
   }
 
-  // Legalize MUBUF* instructions by converting to addr64 form.
-  // FIXME: If we start using the non-addr64 instructions for compute, we
-  // may need to legalize them as above. This especially applies to the
-  // buffer_load_format_* variants and variants with idxen (or bothen).
-  int SRsrcIdx =
+  // Legalize MUBUF* instructions.
+  int RsrcIdx =
       AMDGPU::getNamedOperandIdx(MI.getOpcode(), AMDGPU::OpName::srsrc);
-  if (SRsrcIdx != -1) {
+  if (RsrcIdx != -1) {
     // We have an MUBUF instruction
-    MachineOperand *SRsrc = &MI.getOperand(SRsrcIdx);
-    unsigned SRsrcRC = get(MI.getOpcode()).OpInfo[SRsrcIdx].RegClass;
-    if (RI.getCommonSubClass(MRI.getRegClass(SRsrc->getReg()),
-                                             RI.getRegClass(SRsrcRC))) {
+    MachineOperand *Rsrc = &MI.getOperand(RsrcIdx);
+    unsigned RsrcRC = get(MI.getOpcode()).OpInfo[RsrcIdx].RegClass;
+    if (RI.getCommonSubClass(MRI.getRegClass(Rsrc->getReg()),
+                                             RI.getRegClass(RsrcRC))) {
       // The operands are legal.
       // FIXME: We may need to legalize operands besided srsrc.
       return;
     }
 
-    MachineBasicBlock &MBB = *MI.getParent();
-
-    // Extract the ptr from the resource descriptor.
-    unsigned SRsrcPtr = buildExtractSubReg(MI, MRI, *SRsrc,
-      &AMDGPU::VReg_128RegClass, AMDGPU::sub0_sub1, &AMDGPU::VReg_64RegClass);
-
-    // Create an empty resource descriptor
-    unsigned Zero64 = MRI.createVirtualRegister(&AMDGPU::SReg_64RegClass);
-    unsigned SRsrcFormatLo = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
-    unsigned SRsrcFormatHi = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
-    unsigned NewSRsrc = MRI.createVirtualRegister(&AMDGPU::SReg_128RegClass);
-    uint64_t RsrcDataFormat = getDefaultRsrcDataFormat();
-
-    // Zero64 = 0
-    BuildMI(MBB, MI, MI.getDebugLoc(), get(AMDGPU::S_MOV_B64), Zero64)
-        .addImm(0);
-
-    // SRsrcFormatLo = RSRC_DATA_FORMAT{31-0}
-    BuildMI(MBB, MI, MI.getDebugLoc(), get(AMDGPU::S_MOV_B32), SRsrcFormatLo)
-        .addImm(RsrcDataFormat & 0xFFFFFFFF);
-
-    // SRsrcFormatHi = RSRC_DATA_FORMAT{63-32}
-    BuildMI(MBB, MI, MI.getDebugLoc(), get(AMDGPU::S_MOV_B32), SRsrcFormatHi)
-        .addImm(RsrcDataFormat >> 32);
+    // Legalize a VGPR Rsrc.
+    //
+    // If the instruction is _ADDR64, we can avoid a waterfall by extracting
+    // the base pointer from the VGPR Rsrc, adding it to the VAddr, then using
+    // a zero-value SRsrc.
+    //
+    // If the instruction is _OFFSET (both idxen and offen disabled), and we
+    // support ADDR64 instructions, we can convert to ADDR64 and do the same as
+    // above.
+    //
+    // Otherwise we are on non-ADDR64 hardware, and/or we have
+    // idxen/offen/bothen and we fall back to a waterfall loop.
 
-    // NewSRsrc = {Zero64, SRsrcFormat}
-    BuildMI(MBB, MI, MI.getDebugLoc(), get(AMDGPU::REG_SEQUENCE), NewSRsrc)
-        .addReg(Zero64)
-        .addImm(AMDGPU::sub0_sub1)
-        .addReg(SRsrcFormatLo)
-        .addImm(AMDGPU::sub2)
-        .addReg(SRsrcFormatHi)
-        .addImm(AMDGPU::sub3);
+    MachineBasicBlock &MBB = *MI.getParent();
 
     MachineOperand *VAddr = getNamedOperand(MI, AMDGPU::OpName::vaddr);
-    unsigned NewVAddr = MRI.createVirtualRegister(&AMDGPU::VReg_64RegClass);
-    if (VAddr) {
+    if (VAddr && AMDGPU::getIfAddr64Inst(MI.getOpcode()) != -1) {
       // This is already an ADDR64 instruction so we need to add the pointer
       // extracted from the resource descriptor to the current value of VAddr.
       unsigned NewVAddrLo = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
       unsigned NewVAddrHi = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
+      unsigned NewVAddr = MRI.createVirtualRegister(&AMDGPU::VReg_64RegClass);
+
+      unsigned RsrcPtr, NewSRsrc;
+      std::tie(RsrcPtr, NewSRsrc) = extractRsrcPtr(*this, MI, *Rsrc);
 
-      // NewVaddrLo = SRsrcPtr:sub0 + VAddr:sub0
+      // NewVaddrLo = RsrcPtr:sub0 + VAddr:sub0
       DebugLoc DL = MI.getDebugLoc();
       BuildMI(MBB, MI, DL, get(AMDGPU::V_ADD_I32_e32), NewVAddrLo)
-        .addReg(SRsrcPtr, 0, AMDGPU::sub0)
+        .addReg(RsrcPtr, 0, AMDGPU::sub0)
         .addReg(VAddr->getReg(), 0, AMDGPU::sub0);
 
-      // NewVaddrHi = SRsrcPtr:sub1 + VAddr:sub1
+      // NewVaddrHi = RsrcPtr:sub1 + VAddr:sub1
       BuildMI(MBB, MI, DL, get(AMDGPU::V_ADDC_U32_e32), NewVAddrHi)
-        .addReg(SRsrcPtr, 0, AMDGPU::sub1)
+        .addReg(RsrcPtr, 0, AMDGPU::sub1)
         .addReg(VAddr->getReg(), 0, AMDGPU::sub1);
 
       // NewVaddr = {NewVaddrHi, NewVaddrLo}
       BuildMI(MBB, MI, MI.getDebugLoc(), get(AMDGPU::REG_SEQUENCE), NewVAddr)
           .addReg(NewVAddrLo)
           .addImm(AMDGPU::sub0)
           .addReg(NewVAddrHi)
           .addImm(AMDGPU::sub1);
-    } else {
+
+      VAddr->setReg(NewVAddr);
+      Rsrc->setReg(NewSRsrc);
+    } else if (!VAddr && ST.hasAddr64()) {
       // This instructions is the _OFFSET variant, so we need to convert it to
       // ADDR64.
       assert(MBB.getParent()->getSubtarget<GCNSubtarget>().getGeneration()
              < AMDGPUSubtarget::VOLCANIC_ISLANDS &&
              "FIXME: Need to emit flat atomics here");
 
+      unsigned RsrcPtr, NewSRsrc;
+      std::tie(RsrcPtr, NewSRsrc) = extractRsrcPtr(*this, MI, *Rsrc);
+
+      unsigned NewVAddr = MRI.createVirtualRegister(&AMDGPU::VReg_64RegClass);
       MachineOperand *VData = getNamedOperand(MI, AMDGPU::OpName::vdata);
       MachineOperand *Offset = getNamedOperand(MI, AMDGPU::OpName::offset);
       MachineOperand *SOffset = getNamedOperand(MI, AMDGPU::OpName::soffset);
       unsigned Addr64Opcode = AMDGPU::getAddr64Inst(MI.getOpcode());
 
       // Atomics rith return have have an additional tied operand and are
       // missing some of the special bits.
       MachineOperand *VDataIn = getNamedOperand(MI, AMDGPU::OpName::vdata_in);
       MachineInstr *Addr64;
 
       if (!VDataIn) {
         // Regular buffer load / store.
         MachineInstrBuilder MIB =
             BuildMI(MBB, MI, MI.getDebugLoc(), get(Addr64Opcode))
                 .add(*VData)
-                .addReg(AMDGPU::NoRegister) // Dummy value for vaddr.
-                // This will be replaced later
-                // with the new value of vaddr.
-                .add(*SRsrc)
+                .addReg(NewVAddr)
+                .addReg(NewSRsrc)
                 .add(*SOffset)
                 .add(*Offset);
 
         // Atomics do not have this operand.
         if (const MachineOperand *GLC =
                 getNamedOperand(MI, AMDGPU::OpName::glc)) {
           MIB.addImm(GLC->getImm());
         }
 
         MIB.addImm(getNamedImmOperand(MI, AMDGPU::OpName::slc));
 
         if (const MachineOperand *TFE =
                 getNamedOperand(MI, AMDGPU::OpName::tfe)) {
           MIB.addImm(TFE->getImm());
         }
 
         MIB.cloneMemRefs(MI);
         Addr64 = MIB;
       } else {
         // Atomics with return.
         Addr64 = BuildMI(MBB, MI, MI.getDebugLoc(), get(Addr64Opcode))
                      .add(*VData)
                      .add(*VDataIn)
-                     .addReg(AMDGPU::NoRegister) // Dummy value for vaddr.
-                     // This will be replaced later
-                     // with the new value of vaddr.
-                     .add(*SRsrc)
+                     .addReg(NewVAddr)
+                     .addReg(NewSRsrc)
                      .add(*SOffset)
                      .add(*Offset)
                      .addImm(getNamedImmOperand(MI, AMDGPU::OpName::slc))
                      .cloneMemRefs(MI);
       }
 
       MI.removeFromParent();
 
       // NewVaddr = {NewVaddrHi, NewVaddrLo}
       BuildMI(MBB, Addr64, Addr64->getDebugLoc(), get(AMDGPU::REG_SEQUENCE),
               NewVAddr)
-          .addReg(SRsrcPtr, 0, AMDGPU::sub0)
+          .addReg(RsrcPtr, 0, AMDGPU::sub0)
           .addImm(AMDGPU::sub0)
-          .addReg(SRsrcPtr, 0, AMDGPU::sub1)
+          .addReg(RsrcPtr, 0, AMDGPU::sub1)
           .addImm(AMDGPU::sub1);
-
-      VAddr = getNamedOperand(*Addr64, AMDGPU::OpName::vaddr);
-      SRsrc = getNamedOperand(*Addr64, AMDGPU::OpName::srsrc);
+    } else {
+      // This is another variant; legalize Rsrc with waterfall loop from VGPRs
+      // to SGPRs.
+      loadSRsrcFromVGPR(*this, MI, *Rsrc);
     }
-
-    // Update the instruction to use NewVaddr
-    VAddr->setReg(NewVAddr);
-    // Update the instruction to use NewSRsrc
-    SRsrc->setReg(NewSRsrc);
   }
 }
 
 void SIInstrInfo::moveToVALU(MachineInstr &TopInst) const {
   SetVectorType Worklist;
   Worklist.insert(&TopInst);
 
   while (!Worklist.empty()) {

lib/Target/AMDGPU/SIInstrInfo.td

 def getAddr64Inst : InstrMapping {
   let FilterClass = "MUBUFAddr64Table";
   let RowFields = ["OpName"];
   let ColFields = ["IsAddr64"];
   let KeyCol = ["0"];
   let ValueCols = [["1"]];
 }
 
+def getIfAddr64Inst : InstrMapping {
+  let FilterClass = "MUBUFAddr64Table";
+  let RowFields = ["OpName"];
+  let ColFields = ["IsAddr64"];
+  let KeyCol = ["1"];
+  let ValueCols = [["1"]];
+}
+
 def getMUBUFNoLdsInst : InstrMapping {
   let FilterClass = "MUBUFLdsTable";
   let RowFields = ["OpName"];
   let ColFields = ["IsLds"];
   let KeyCol = ["1"];
   let ValueCols = [["0"]];
 }
 

test/CodeGen/AMDGPU/mubuf-legalize-operands.ll

+; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs -o - %s | FileCheck %s
+
+; Test that we correctly legalize VGPR Rsrc operands in MUBUF instructions.
+
+; CHECK-LABEL: mubuf_vgpr
+; CHECK: s_mov_b64 [[SAVEEXEC:s\[[0-9]+:[0-9]+\]]], exec
+; CHECK: [[LOOPBB:BB[0-9]+_[0-9]+]]:
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC0:[0-9]+]], v0
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC1:[0-9]+]], v1
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC2:[0-9]+]], v2
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC3:[0-9]+]], v3
+; CHECK-NOT: DAG-DIVIDER
+; CHECK-DAG: v_cmp_eq_u64_e32 vcc, s{{\[}}[[SRSRC0]]:[[SRSRC1]]{{\]}}, v[0:1]
+; CHECK-DAG: v_cmp_eq_u64_e64 [[CMP0:s\[[0-9]+:[0-9]+\]]], s{{\[}}[[SRSRC2]]:[[SRSRC3]]{{\]}}, v[2:3]
+; CHECK: s_and_b64 [[CMP:s\[[0-9]+:[0-9]+\]]], vcc, [[CMP0]]
+; CHECK: s_waitcnt vmcnt(0)
+; CHECK: buffer_load_format_x [[RES:v[0-9]+]], v4, s{{\[}}[[SRSRC0]]:[[SRSRC3]]{{\]}}, 0 idxen
+; CHECK: s_xor_b64 exec, exec, [[CMP]]
+; CHECK: s_cbranch_execnz [[LOOPBB]]
+; CHECK: s_mov_b64 exec, [[SAVEEXEC]]
+; CHECK: v_mov_b32_e32 v0, [[RES]]
+define float @mubuf_vgpr(<4 x i32> %i, i32 %c) #0 {
+  %call = call float @llvm.amdgcn.buffer.load.format.f32(<4 x i32> %i, i32 %c, i32 0, i1 zeroext false, i1 zeroext false) #1
+  ret float %call
+}
+
+; CHECK-LABEL: mubuf_vgpr_adjacent_in_block
+
+; CHECK: s_mov_b64 [[SAVEEXEC:s\[[0-9]+:[0-9]+\]]], exec
+; CHECK: [[LOOPBB0:BB[0-9]+_[0-9]+]]:
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC0:[0-9]+]], v0
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC1:[0-9]+]], v1
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC2:[0-9]+]], v2
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC3:[0-9]+]], v3
+; CHECK-NOT: DAG-DIVIDER
+; CHECK-DAG: v_cmp_eq_u64_e32 vcc, s{{\[}}[[SRSRC0]]:[[SRSRC1]]{{\]}}, v[0:1]
+; CHECK-DAG: v_cmp_eq_u64_e64 [[CMP0:s\[[0-9]+:[0-9]+\]]], s{{\[}}[[SRSRC2]]:[[SRSRC3]]{{\]}}, v[2:3]
+; CHECK: s_and_b64 [[CMP:s\[[0-9]+:[0-9]+\]]], vcc, [[CMP0]]
+; CHECK: s_waitcnt vmcnt(0)
+; CHECK: buffer_load_format_x [[RES0:v[0-9]+]], v8, s{{\[}}[[SRSRC0]]:[[SRSRC3]]{{\]}}, 0 idxen
+; CHECK: s_xor_b64 exec, exec, [[CMP]]
+; CHECK: s_cbranch_execnz [[LOOPBB0]]
+
+; CHECK: s_mov_b64 exec, [[SAVEEXEC]]
+; FIXME: redundant s_mov
+; CHECK: s_mov_b64 [[SAVEEXEC:s\[[0-9]+:[0-9]+\]]], exec
+
+; CHECK: [[LOOPBB1:BB[0-9]+_[0-9]+]]:
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC0:[0-9]+]], v4
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC1:[0-9]+]], v5
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC2:[0-9]+]], v6
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC3:[0-9]+]], v7
+; CHECK-NOT: DAG-DIVIDER
+; CHECK-DAG: v_cmp_eq_u64_e32 vcc, s{{\[}}[[SRSRC0]]:[[SRSRC1]]{{\]}}, v[4:5]
+; CHECK-DAG: v_cmp_eq_u64_e64 [[CMP0:s\[[0-9]+:[0-9]+\]]], s{{\[}}[[SRSRC2]]:[[SRSRC3]]{{\]}}, v[6:7]
+; CHECK: s_and_b64 [[CMP:s\[[0-9]+:[0-9]+\]]], vcc, [[CMP0]]
+; CHECK: s_waitcnt vmcnt(0)
+; CHECK: buffer_load_format_x [[RES1:v[0-9]+]], v8, s{{\[}}[[SRSRC0]]:[[SRSRC3]]{{\]}}, 0 idxen
+; CHECK: s_xor_b64 exec, exec, [[CMP]]
+; CHECK: s_cbranch_execnz [[LOOPBB1]]
+
+; CHECK: s_mov_b64 exec, [[SAVEEXEC]]
+; CHECK-DAG: global_store_dword v[9:10], [[RES0]], off
+; CHECK-DAG: global_store_dword v[11:12], [[RES1]], off
+
+define void @mubuf_vgpr_adjacent_in_block(<4 x i32> %i, <4 x i32> %j, i32 %c, float addrspace(1)* %out0, float addrspace(1)* %out1) #0 {
+entry:
+  %val0 = call float @llvm.amdgcn.buffer.load.format.f32(<4 x i32> %i, i32 %c, i32 0, i1 zeroext false, i1 zeroext false) #1
+  %val1 = call float @llvm.amdgcn.buffer.load.format.f32(<4 x i32> %j, i32 %c, i32 0, i1 zeroext false, i1 zeroext false) #1
+  store volatile float %val0, float addrspace(1)* %out0
+  store volatile float %val1, float addrspace(1)* %out1
+  ret void
+}
+
+; CHECK-LABEL: mubuf_vgpr_outside_entry
+
+; CHECK-DAG: v_mov_b32_e32 [[IDX:v[0-9]+]], s4
+; CHECK-DAG: s_mov_b64 [[SAVEEXEC:s\[[0-9]+:[0-9]+\]]], exec
+
+; CHECK: [[LOOPBB0:BB[0-9]+_[0-9]+]]:
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC0:[0-9]+]], v0
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC1:[0-9]+]], v1
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC2:[0-9]+]], v2
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC3:[0-9]+]], v3
+; CHECK-NOT: DAG-DIVIDER
+; CHECK-DAG: v_cmp_eq_u64_e32 vcc, s{{\[}}[[SRSRC0]]:[[SRSRC1]]{{\]}}, v[0:1]
+; CHECK-DAG: v_cmp_eq_u64_e64 [[CMP0:s\[[0-9]+:[0-9]+\]]], s{{\[}}[[SRSRC2]]:[[SRSRC3]]{{\]}}, v[2:3]
+; CHECK: s_and_b64 [[CMP:s\[[0-9]+:[0-9]+\]]], vcc, [[CMP0]]
+; CHECK: s_waitcnt vmcnt(0)
+; CHECK: buffer_load_format_x [[RES:v[0-9]+]], [[IDX]], s{{\[}}[[SRSRC0]]:[[SRSRC3]]{{\]}}, 0 idxen
+; CHECK: s_xor_b64 exec, exec, [[CMP]]
+; CHECK: s_cbranch_execnz [[LOOPBB0]]
+
+; CHECK: s_mov_b64 exec, [[SAVEEXEC]]
+; CHECK: s_cbranch_execz [[TERMBB:BB[0-9]+_[0-9]+]]
+
+; CHECK: BB{{[0-9]+_[0-9]+}}:
+; CHECK-DAG: v_mov_b32_e32 [[IDX:v[0-9]+]], s4
+; CHECK-DAG: s_mov_b64 [[SAVEEXEC:s\[[0-9]+:[0-9]+\]]], exec
+
+; CHECK: [[LOOPBB1:BB[0-9]+_[0-9]+]]:
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC0:[0-9]+]], v4
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC1:[0-9]+]], v5
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC2:[0-9]+]], v6
+; CHECK-DAG: v_readfirstlane_b32 s[[SRSRC3:[0-9]+]], v7
+; CHECK-NOT: DAG-DIVIDER
+; CHECK-DAG: v_cmp_eq_u64_e32 vcc, s{{\[}}[[SRSRC0]]:[[SRSRC1]]{{\]}}, v[4:5]
+; CHECK-DAG: v_cmp_eq_u64_e64 [[CMP0:s\[[0-9]+:[0-9]+\]]], s{{\[}}[[SRSRC2]]:[[SRSRC3]]{{\]}}, v[6:7]
+; CHECK: s_and_b64 [[CMP:s\[[0-9]+:[0-9]+\]]], vcc, [[CMP0]]
+; CHECK: s_waitcnt vmcnt(0)
+; CHECK: buffer_load_format_x [[RES]], [[IDX]], s{{\[}}[[SRSRC0]]:[[SRSRC3]]{{\]}}, 0 idxen
+; CHECK: s_xor_b64 exec, exec, [[CMP]]
+; CHECK: s_cbranch_execnz [[LOOPBB1]]
+
+; CHECK: s_mov_b64 exec, [[SAVEEXEC]]
+
+; CHECK: [[TERMBB]]:
+; global_store_dword v[11:12], [[RES]], off
+
+define void @mubuf_vgpr_outside_entry(<4 x i32> %i, <4 x i32> %j, i32 %c, float addrspace(1)* %in, float addrspace(1)* %out) #0 {
+entry:
+  %live.out.reg = call i32 asm sideeffect "s_mov_b32 $0, 17", "={s4}" ()
+  %val0 = call float @llvm.amdgcn.buffer.load.format.f32(<4 x i32> %i, i32 %live.out.reg, i32 0, i1 zeroext false, i1 zeroext false) #1
+  %idx = call i32 @llvm.amdgcn.workitem.id.x() #1
+  %cmp = icmp eq i32 %idx, 0
+  br i1 %cmp, label %bb1, label %bb2
+
+bb1:
+  %val1 = call float @llvm.amdgcn.buffer.load.format.f32(<4 x i32> %j, i32 %live.out.reg, i32 0, i1 zeroext false, i1 zeroext false) #1
+  br label %bb2
+
+bb2:
+  %val = phi float [ %val0, %entry ], [ %val1, %bb1 ]
+  store volatile float %val, float addrspace(1)* %out
+  ret void
+}
+
+declare i32 @llvm.amdgcn.workitem.id.x() #1
+declare float @llvm.amdgcn.buffer.load.format.f32(<4 x i32>, i32, i32, i1, i1) #1
+
+attributes #0 = { nounwind }
+attributes #1 = { nounwind readnone }

test/CodeGen/AMDGPU/mubuf-legalize-operands.mir

+# RUN: llc -march=amdgcn -mcpu=gfx700 -verify-machineinstrs --run-pass=si-fix-sgpr-copies -o - %s | FileCheck %s --check-prefixes=COMMON,ADDR64
+# RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs --run-pass=si-fix-sgpr-copies -o - %s | FileCheck %s --check-prefixes=COMMON,NO-ADDR64
+
+# Test that we correctly legalize VGPR Rsrc operands in MUBUF instructions.
+#
+# On ADDR64 hardware we optimize the _ADDR64 and _OFFSET cases to avoid
+# needing a waterfall. For all other instruction variants, and when we are
+# on non-ADDR64 hardware, we emit a waterfall loop.
+
+# COMMON-LABEL: name: idxen
+# COMMON-LABEL:  bb.0:
+# COMMON-NEXT: successors: %bb.1({{.*}})
+# COMMON: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
+# COMMON: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
+# COMMON-LABEL: bb.1:
+# COMMON-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
+# COMMON: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
+# COMMON: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
+# COMMON: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
+# COMMON: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
+# COMMON: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
+# COMMON: [[CMP0:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
+# COMMON: [[CMP1:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
+# COMMON: [[CMP:%[0-9]+]]:sreg_64 = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc
+# COMMON: [[TMPEXEC:%[0-9]+]]:sreg_64 = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
+# COMMON: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, implicit $exec
+# COMMON: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc
+# COMMON: S_CBRANCH_EXECNZ %bb.1, implicit $exec
+# COMMON-LABEL  bb.2:
+# COMMON: $exec = S_MOV_B64 [[SAVEEXEC]]
+---
+name:            idxen
+liveins:
+  - { reg: '$vgpr0', virtual-reg: '%0' }
+  - { reg: '$vgpr1', virtual-reg: '%1' }
+  - { reg: '$vgpr2', virtual-reg: '%2' }
+  - { reg: '$vgpr3', virtual-reg: '%3' }
+  - { reg: '$vgpr4', virtual-reg: '%4' }
+  - { reg: '$sgpr30_sgpr31', virtual-reg: '%5' }
+body:             |
+  bb.0:
+    liveins: $vgpr0, $vgpr1, $vgpr2, $vgpr3, $vgpr4, $sgpr30_sgpr31
+    %5:sreg_64 = COPY $sgpr30_sgpr31
+    %4:vgpr_32 = COPY $vgpr4
+    %3:vgpr_32 = COPY $vgpr3
+    %2:vgpr_32 = COPY $vgpr2
+    %1:vgpr_32 = COPY $vgpr1
+    %0:vgpr_32 = COPY $vgpr0
+    %6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
+    %7:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN %4, killed %6, 0, 0, 0, 0, 0, implicit $exec
+    $sgpr30_sgpr31 = COPY %5
+    $vgpr0 = COPY %7
+    S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
+...
+
+# COMMON-LABEL: name: offen
+# COMMON-LABEL:  bb.0:
+# COMMON-NEXT: successors: %bb.1({{.*}})
+# COMMON: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
+# COMMON: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
+# COMMON-LABEL: bb.1:
+# COMMON-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
+# COMMON: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
+# COMMON: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
+# COMMON: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
+# COMMON: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
+# COMMON: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
+# COMMON: [[CMP0:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
+# COMMON: [[CMP1:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
+# COMMON: [[CMP:%[0-9]+]]:sreg_64 = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc
+# COMMON: [[TMPEXEC:%[0-9]+]]:sreg_64 = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
+# COMMON: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, implicit $exec
+# COMMON: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc
+# COMMON: S_CBRANCH_EXECNZ %bb.1, implicit $exec
+# COMMON-LABEL  bb.2:
+# COMMON: $exec = S_MOV_B64 [[SAVEEXEC]]
+---
+name:            offen
+liveins:
+  - { reg: '$vgpr0', virtual-reg: '%0' }
+  - { reg: '$vgpr1', virtual-reg: '%1' }
+  - { reg: '$vgpr2', virtual-reg: '%2' }
+  - { reg: '$vgpr3', virtual-reg: '%3' }
+  - { reg: '$vgpr4', virtual-reg: '%4' }
+  - { reg: '$sgpr30_sgpr31', virtual-reg: '%5' }
+body:             |
+  bb.0:
+    liveins: $vgpr0, $vgpr1, $vgpr2, $vgpr3, $vgpr4, $sgpr30_sgpr31
+    %5:sreg_64 = COPY $sgpr30_sgpr31
+    %4:vgpr_32 = COPY $vgpr4
+    %3:vgpr_32 = COPY $vgpr3
+    %2:vgpr_32 = COPY $vgpr2
+    %1:vgpr_32 = COPY $vgpr1
+    %0:vgpr_32 = COPY $vgpr0
+    %6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
+    %7:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFEN %4, killed %6, 0, 0, 0, 0, 0, implicit $exec
+    $sgpr30_sgpr31 = COPY %5
+    $vgpr0 = COPY %7
+    S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
+...
+
+# COMMON-LABEL: name: bothen
+# COMMON-LABEL:  bb.0:
+# COMMON-NEXT: successors: %bb.1({{.*}})
+# COMMON: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
+# COMMON: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
+# COMMON-LABEL: bb.1:
+# COMMON-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
+# COMMON: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
+# COMMON: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
+# COMMON: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
+# COMMON: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
+# COMMON: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
+# COMMON: [[CMP0:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
+# COMMON: [[CMP1:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
+# COMMON: [[CMP:%[0-9]+]]:sreg_64 = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc
+# COMMON: [[TMPEXEC:%[0-9]+]]:sreg_64 = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
+# COMMON: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_BOTHEN %4, killed [[SRSRC]], 0, 0, 0, 0, 0, implicit $exec
+# COMMON: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc
+# COMMON: S_CBRANCH_EXECNZ %bb.1, implicit $exec
+# COMMON-LABEL  bb.2:
+# COMMON: $exec = S_MOV_B64 [[SAVEEXEC]]
+---
+name:            bothen
+liveins:
+  - { reg: '$vgpr0', virtual-reg: '%0' }
+  - { reg: '$vgpr1', virtual-reg: '%1' }
+  - { reg: '$vgpr2', virtual-reg: '%2' }
+  - { reg: '$vgpr3', virtual-reg: '%3' }
+  - { reg: '$vgpr4_vgpr5', virtual-reg: '%4' }
+  - { reg: '$sgpr30_sgpr31', virtual-reg: '%5' }
+body:             |
+  bb.0:
+    liveins: $vgpr0, $vgpr1, $vgpr2, $vgpr3, $vgpr4, $sgpr30_sgpr31
+    %5:sreg_64 = COPY $sgpr30_sgpr31
+    %4:vreg_64 = COPY $vgpr4_vgpr5
+    %3:vgpr_32 = COPY $vgpr3
+    %2:vgpr_32 = COPY $vgpr2
+    %1:vgpr_32 = COPY $vgpr1
+    %0:vgpr_32 = COPY $vgpr0
+    %6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
+    %7:vgpr_32 = BUFFER_LOAD_FORMAT_X_BOTHEN %4, killed %6, 0, 0, 0, 0, 0, implicit $exec
+    $sgpr30_sgpr31 = COPY %5
+    $vgpr0 = COPY %7
+    S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
+...
+
+# COMMON-LABEL: name: addr64
+# COMMON-LABEL:  bb.0:
+# COMMON: %12:vreg_64 = COPY %8.sub0_sub1
+# COMMON: %13:sreg_64 = S_MOV_B64 0
+# COMMON: %14:sgpr_32 = S_MOV_B32 0
+# COMMON: %15:sgpr_32 = S_MOV_B32 61440
+# COMMON: %16:sreg_128 = REG_SEQUENCE %13, %subreg.sub0_sub1, %14, %subreg.sub2, %15, %subreg.sub3
+# COMMON: %9:vgpr_32 = V_ADD_I32_e32 %12.sub0, %4.sub0, implicit-def $vcc, implicit $exec
+# COMMON: %10:vgpr_32 = V_ADDC_U32_e32 %12.sub1, %4.sub1, implicit-def $vcc, implicit $vcc, implicit $exec
+# COMMON: %11:vreg_64 = REG_SEQUENCE %9, %subreg.sub0, %10, %subreg.sub1
+# COMMON: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_ADDR64 %11, killed %16, 0, 0, 0, 0, 0, implicit $exec
+---
+name:            addr64
+liveins:
+  - { reg: '$vgpr0', virtual-reg: '%0' }
+  - { reg: '$vgpr1', virtual-reg: '%1' }
+  - { reg: '$vgpr2', virtual-reg: '%2' }
+  - { reg: '$vgpr3', virtual-reg: '%3' }
+  - { reg: '$vgpr4_vgpr5', virtual-reg: '%4' }
+  - { reg: '$sgpr30_sgpr31', virtual-reg: '%5' }
+body:             |
+  bb.0:
+    liveins: $vgpr0, $vgpr1, $vgpr2, $vgpr3, $vgpr4, $sgpr30_sgpr31
+    %5:sreg_64 = COPY $sgpr30_sgpr31
+    %4:vreg_64 = COPY $vgpr4_vgpr5
+    %3:vgpr_32 = COPY $vgpr3
+    %2:vgpr_32 = COPY $vgpr2
+    %1:vgpr_32 = COPY $vgpr1
+    %0:vgpr_32 = COPY $vgpr0
+    %6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
+    %7:vgpr_32 = BUFFER_LOAD_FORMAT_X_ADDR64 %4, killed %6, 0, 0, 0, 0, 0, implicit $exec
+    $sgpr30_sgpr31 = COPY %5
+    $vgpr0 = COPY %7
+    S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
+...
+
+# COMMON-LABEL: name: offset
+# COMMON-LABEL:  bb.0:
+
+# NO-ADDR64-NEXT: successors: %bb.1({{.*}})
+# NO-ADDR64: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
+# NO-ADDR64: [[SAVEEXEC:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
+# NO-ADDR64-LABEL: bb.1:
+# NO-ADDR64-NEXT: successors: %bb.1({{.*}}), %bb.2({{.*}})
+# NO-ADDR64: [[SRSRC0:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub0, implicit $exec
+# NO-ADDR64: [[SRSRC1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub1, implicit $exec
+# NO-ADDR64: [[SRSRC2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub2, implicit $exec
+# NO-ADDR64: [[SRSRC3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[VRSRC]].sub3, implicit $exec
+# NO-ADDR64: [[SRSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[SRSRC0]], %subreg.sub0, [[SRSRC1]], %subreg.sub1, [[SRSRC2]], %subreg.sub2, [[SRSRC3]], %subreg.sub3
+# NO-ADDR64: [[CMP0:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub0_sub1, [[VRSRC]].sub0_sub1, implicit $exec
+# NO-ADDR64: [[CMP1:%[0-9]+]]:sreg_64 = V_CMP_EQ_U64_e64 [[SRSRC]].sub2_sub3, [[VRSRC]].sub2_sub3, implicit $exec
+# NO-ADDR64: [[CMP:%[0-9]+]]:sreg_64 = S_AND_B64 [[CMP0]], [[CMP1]], implicit-def $scc
+# NO-ADDR64: [[TMPEXEC:%[0-9]+]]:sreg_64 = S_AND_SAVEEXEC_B64 killed [[CMP]], implicit-def $exec, implicit-def $scc, implicit $exec
+# NO-ADDR64: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFSET killed [[SRSRC]], 0, 0, 0, 0, 0, implicit $exec
+# NO-ADDR64: $exec = S_XOR_B64_term $exec, [[TMPEXEC]], implicit-def $scc
+# NO-ADDR64: S_CBRANCH_EXECNZ %bb.1, implicit $exec
+# NO-ADDR64-LABEL  bb.2:
+# NO-ADDR64: $exec = S_MOV_B64 [[SAVEEXEC]]
+
+# ADDR64: [[VRSRC:%[0-9]+]]:vreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
+# ADDR64: [[RSRCPTR:%[0-9]+]]:vreg_64 = COPY [[VRSRC]].sub0_sub1
+# ADDR64: [[ZERO64:%[0-9]+]]:sreg_64 = S_MOV_B64 0
+# ADDR64: [[RSRCFMTLO:%[0-9]+]]:sgpr_32 = S_MOV_B32 0
+# ADDR64: [[RSRCFMTHI:%[0-9]+]]:sgpr_32 = S_MOV_B32 61440
+# ADDR64: [[ZERORSRC:%[0-9]+]]:sreg_128 = REG_SEQUENCE [[ZERO64]], %subreg.sub0_sub1, [[RSRCFMTLO]], %subreg.sub2, [[RSRCFMTHI]], %subreg.sub3
+# ADDR64: [[VADDR64:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[RSRCPTR]].sub0, %subreg.sub0, [[RSRCPTR]].sub1, %subreg.sub1
+# ADDR64: {{[0-9]+}}:vgpr_32 = BUFFER_LOAD_FORMAT_X_ADDR64 [[VADDR64]], [[ZERORSRC]], 0, 0, 0, 0, 0, implicit $exec
+
+---
+name:            offset
+liveins:
+  - { reg: '$vgpr0', virtual-reg: '%0' }
+  - { reg: '$vgpr1', virtual-reg: '%1' }
+  - { reg: '$vgpr2', virtual-reg: '%2' }
+  - { reg: '$vgpr3', virtual-reg: '%3' }
+  - { reg: '$vgpr4_vgpr5', virtual-reg: '%4' }
+  - { reg: '$sgpr30_sgpr31', virtual-reg: '%5' }
+body:             |
+  bb.0:
+    liveins: $vgpr0, $vgpr1, $vgpr2, $vgpr3, $vgpr4, $sgpr30_sgpr31
+    %5:sreg_64 = COPY $sgpr30_sgpr31
+    %4:vreg_64 = COPY $vgpr4_vgpr5
+    %3:vgpr_32 = COPY $vgpr3
+    %2:vgpr_32 = COPY $vgpr2
+    %1:vgpr_32 = COPY $vgpr1
+    %0:vgpr_32 = COPY $vgpr0
+    %6:sreg_128 = REG_SEQUENCE %0, %subreg.sub0, %1, %subreg.sub1, %2, %subreg.sub2, %3, %subreg.sub3
+    %7:vgpr_32 = BUFFER_LOAD_FORMAT_X_OFFSET killed %6, 0, 0, 0, 0, 0, implicit $exec
+    $sgpr30_sgpr31 = COPY %5
+    $vgpr0 = COPY %7
+    S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
+...