AMDGPU: Fold readlane from copy of SGPR or imm

[arsenm]

Summary

These may be inserted to assert uniformity somewhere.

[rampitec]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

I do not think it is correct. If first lane is off you will fold wrong value. In both cases of readfirstlane and readlane you need to check a specific lane, which I am not sure even possible.

[arsenm]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

readlane ignores exec. readfirstlane only reads exec to find the first lane, and still returns a result when exec is 0. The source value is a constant, so it doesn't matter what the lane is

[rampitec]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

Readlane ignores exec, but v_mov_b32 does not. So you may have an old value of vgpr instead of immediate.
For readfirstlane you will get wrong value on the sgpr if exec=0 I believe.

[arsenm]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

It would matter if there was an exec def between the copy and the use, which I want to ban within a block

[rampitec]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

But it is not banned yet, right? Are we sure that only happens within a block? Why block cannot be executed with a zero exec? And then at the very least we need to be sure register is virtual.

[arsenm]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

We force execz skip branches around blocks that have readlane/readfirstlane in them. exec = 0 should also be illegal on function entry

[rampitec]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

I thought about it. From the ISA perspective it is illegal if you are reading an inactive lane. This can be legal here on the grounds that register is virtual and we are in SSA. Are both conditions true?

[arsenm]

Check for exec def

[arsenm]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

SGPR spilling does read/write VGPRs for inactive lanes, but the use of an SSA value for an invalid lane doesn't make sense. I'm not sure if the exec check is strictly necessary, but I've added it.

[rampitec]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

I do not think this check really changes anything. Look: you have lane 8 disabled, both at def and use. Then you do readlane for the lane 8. It will pass the check and yield wrong value. So this check is not necessary and does not help too.

There are just two conditions which may make it legal: 1) register is virtual 2) we are in SSA. If both a satisfied you can do it, because there can be no other valid def anyway.

[rampitec]

This cannot work as written, at least readlane part. Maybe readfirstlane can, providing skip is always inserted around and def is in the same block. Otherwise it needs to be a virtual register and SSA.

[arsenm]

In D63225#1544639, @rampitec wrote:

This cannot work as written, at least readlane part. Maybe readfirstlane can, providing skip is always inserted around and def is in the same block. Otherwise it needs to be a virtual register and SSA.

SIFoldOperands is SSA only, and physical registers are filtered out already

[rampitec]

In D63225#1544642, @arsenm wrote:

In D63225#1544639, @rampitec wrote:

This cannot work as written, at least readlane part. Maybe readfirstlane can, providing skip is always inserted around and def is in the same block. Otherwise it needs to be a virtual register and SSA.

SIFoldOperands is SSA only, and physical registers are filtered out already

Why do we check for virtual registers all over the place place then, including the few lines above?

Also note, if virtual and SSA, then check for exec modification is not needed because that must be the only def anyway. Otherwise it reads undef that way or another.

[nhaehnle]

Thanks, this is very helpful!

Does this collapse redundant readfirstlanes? I.e., the case where the result of a readfirstlane is immediately fed to another readfirstlane. I think it might fix that -- could you please add a corresponding test?

lib/Target/AMDGPU/SIFoldOperands.cpp

525

You do need the EXEC mask check either way because the use could be observing a def inside a divergent loop.

For the readfirstlane case, that check should be sufficient. For the readlane case, I agree that you need to know the register is virtual and in SSA. But with that knowledge, I believe the code should be good as-is.

[rampitec]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

Right. Exec check probably needed, at any rate it doesn't hurt. But readlane case needs either be removed or additional check needed.

[arsenm]

Add test for multiple folds

lib/Target/AMDGPU/SIFoldOperands.cpp

525

Both of those are known. This pass is only used SSA, and skips physical registers before this point

[rampitec]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

It is SSA, granted. But I do not see that we check for virtual register at all possible foldOperand() invocations, only in one of the cases.

[arsenm]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

The search for users begins from a virtual register def. foldInstOperand is never called

[arsenm]

Add a few more examples of physical registers not folding

[rampitec]

lib/Target/AMDGPU/SIFoldOperands.cpp

525

There are quite a few places where we call foldOperand(), including recursively. Only foldInstOperand() is guarded with the check. It is not immediately obvious none of that calls will end up with a virtual register only. Moreover, we are checking for virtual destination registers in the very same function. It will be safe to add the check here, otherwise I cannot prove to myself it is safe.

[rampitec]

LGTM if precheckin will succeed for this and for D63456.

[arsenm]

r363670

lib/Target/AMDGPU/SIFoldOperands.cpp

if (UseMI->isCopy() && OpToFold.isReg() &&
       UseMI->getOperand(1).setReg(OpToFold.getReg());
       UseMI->getOperand(1).setSubReg(OpToFold.getSubReg());
       UseMI->getOperand(1).setIsKill(false);
       CopiesToReplace.push_back(UseMI);
       OpToFold.setIsKill(false);
       return;
     }
 
+    unsigned UseOpc = UseMI->getOpcode();
+    if (UseOpc == AMDGPU::V_READFIRSTLANE_B32 ||
+        (UseOpc == AMDGPU::V_READLANE_B32 &&
+         (int)UseOpIdx ==
+         AMDGPU::getNamedOperandIdx(UseOpc, AMDGPU::OpName::src0))) {
+      // %vgpr = V_MOV_B32 imm
+      // %sgpr = V_READFIRSTLANE_B32 %vgpr
+      // =>
+      // %sgpr = S_MOV_B32 imm
+      if (FoldingImm) {
+        if (!isEXECMaskConstantBetweenDefAndUses(
+              UseMI->getOperand(UseOpIdx).getReg(), *MRI))
+          return;
+
+        UseMI->setDesc(TII->get(AMDGPU::S_MOV_B32));
+        UseMI->getOperand(1).ChangeToImmediate(OpToFold.getImm());
+        UseMI->RemoveOperand(2); // Remove exec read (or src1 for readlane)
+        return;
+      }
+
+      if (OpToFold.isReg() && TRI->isSGPRReg(*MRI, OpToFold.getReg())) {
+        if (!isEXECMaskConstantBetweenDefAndUses(
+              UseMI->getOperand(UseOpIdx).getReg(), *MRI))
+          return;
+
+        // %vgpr = COPY %sgpr0
+        // %sgpr1 = V_READFIRSTLANE_B32 %vgpr
+        // =>
+        // %sgpr1 = COPY %sgpr0
+        UseMI->setDesc(TII->get(AMDGPU::COPY));
+        UseMI->RemoveOperand(2); // Remove exec read (or src1 for readlane)
+        return;
+      }
+    }
+
     const MCInstrDesc &UseDesc = UseMI->getDesc();
 
     // Don't fold into target independent nodes.  Target independent opcodes
     // don't have defined register classes.
     if (UseDesc.isVariadic() ||
         UseOp.isImplicit() ||
         UseDesc.OpInfo[UseOpIdx].RegClass == -1)
       return;

lib/Target/AMDGPU/SIInstructions.td

 // Undo sub x, c -> add x, -c canonicalization since c is more likely
 // an inline immediate than -c.
 // TODO: Also do for 64-bit.
 def : GCNPat<
   (add i32:$src0, (i32 NegSubInlineConst32:$src1)),
   (S_SUB_I32 $src0, NegSubInlineConst32:$src1)
 >;
 
+// Avoid pointlessly materializing a constant in VGPR.
+// FIXME: Should also do this for readlane, but tablegen crashes on
+// the ignored src1.
+def : GCNPat<
+  (int_amdgcn_readfirstlane (i32 imm:$src)),
+  (S_MOV_B32 $src)
+>;
 
 multiclass BFMPatterns <ValueType vt, InstSI BFM, InstSI MOV> {
   def : GCNPat <
     (vt (shl (vt (add (vt (shl 1, vt:$a)), -1)), vt:$b)),
     (BFM $a, $b)
   >;
 
   def : GCNPat <

test/CodeGen/AMDGPU/fold-readlane.mir

+# RUN: llc -march=amdgcn -run-pass si-fold-operands -verify-machineinstrs %s -o - | FileCheck -check-prefix=GCN %s
+
+# GCN-LABEL: name: fold-imm-readfirstlane{{$}}
+# GCN: %1:sreg_32_xm0 = S_MOV_B32 123
+---
+name: fold-imm-readfirstlane
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    %0:vgpr_32 = V_MOV_B32_e32 123, implicit $exec
+    %1:sreg_32_xm0 = V_READFIRSTLANE_B32 %0, implicit $exec
+...
+
+# GCN-LABEL: name: no-fold-imm-readfirstlane-physreg{{$}}
+# GCN: $vgpr0 = V_MOV_B32_e32 123, implicit $exec
+# GCN-NEXT: V_READFIRSTLANE_B32 $vgpr0, implicit $exec
+
+---
+name: no-fold-imm-readfirstlane-physreg
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    $vgpr0 = V_MOV_B32_e32 123, implicit $exec
+    %0:sreg_32_xm0 = V_READFIRSTLANE_B32 $vgpr0, implicit $exec
+...
+
+# TODO: This could be folded, if the search for exec modifications was
+# smarter.
+
+# GCN-LABEL: name: fold-imm-readfirstlane-cross-block{{$}}
+# GCN: V_MOV_B32
+# GCN: V_READFIRSTLANE_B32
+---
+name: fold-imm-readfirstlane-cross-block
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    %0:vgpr_32 = V_MOV_B32_e32 123, implicit $exec
+
+  bb.1:
+    %1:sreg_32_xm0 = V_READFIRSTLANE_B32 %0, implicit $exec
+...
+
+# TODO: This could be folded, if the search for exec modifications was
+# smarter.
+
+# GCN-LABEL: name: fold-copy-readfirstlane-cross-block{{$}}
+# GCN: V_MOV_B32
+# GCN: V_READFIRSTLANE_B32
+---
+name: fold-copy-readfirstlane-cross-block
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr12
+    %0:sreg_32_xm0 = COPY $sgpr12
+    %1:vgpr_32 = V_MOV_B32_e32 %0, implicit $exec
+
+  bb.1:
+    %2:sreg_32_xm0 = V_READFIRSTLANE_B32 %1, implicit $exec
+...
+
+# GCN-LABEL: name: fold-copy-readfirstlane-cross-block-exec-def{{$}}
+# GCN: V_MOV_B32
+# GCN: $exec = S_MOV_B64_term
+# GCN: V_READFIRSTLANE_B32
+---
+name: fold-copy-readfirstlane-cross-block-exec-def
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr10_sgpr11, $sgpr12
+    %0:sreg_32_xm0 = COPY $sgpr12
+    %1:vgpr_32 = V_MOV_B32_e32 %0, implicit $exec
+    $exec = S_MOV_B64_term $sgpr10_sgpr11
+
+  bb.1:
+    %2:sreg_32_xm0 = V_READFIRSTLANE_B32 %1, implicit $exec
+...
+
+# GCN-LABEL: name: fold-copy-readfirstlane-same-block-exec-def{{$}}
+# GCN: COPY
+# GCN-NEXT: %1:vgpr_32 = COPY %0
+# GCN-NEXT: $exec = S_MOV_B64
+# GCN-NEXT: V_READFIRSTLANE_B32
+---
+name: fold-copy-readfirstlane-same-block-exec-def
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr10_sgpr11, $sgpr12
+    %0:sreg_32_xm0 = COPY $sgpr12
+    %1:vgpr_32 = COPY %0, implicit $exec
+    $exec = S_MOV_B64 $sgpr10_sgpr11
+    %2:sreg_32_xm0 = V_READFIRSTLANE_B32 %1, implicit $exec
+
+...
+
+# GCN-LABEL: name: fold-imm-readfirstlane-cross-block-exec-def{{$}}
+# GCN: V_MOV_B32
+# GCN: $exec = S_MOV_B64
+# GCN: V_READFIRSTLANE_B32
+
+---
+name: fold-imm-readfirstlane-cross-block-exec-def
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr10_sgpr11, $sgpr12_sgpr13
+    %0:vgpr_32 = V_MOV_B32_e32 123, implicit $exec
+    $exec = S_MOV_B64_term $sgpr10_sgpr11
+
+  bb.1:
+    %1:sreg_32_xm0 = V_READFIRSTLANE_B32 %0, implicit $exec
+...
+
+# GCN-LABEL: name: fold-imm-readfirstlane-same-block-exec-def{{$}}
+# GCN: V_MOV_B32
+# GCN-NEXT: $exec = S_MOV_B64
+# GCN-NEXT: V_READFIRSTLANE_B32
+---
+name: fold-imm-readfirstlane-same-block-exec-def
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr10_sgpr11
+    %0:vgpr_32 = V_MOV_B32_e32 123, implicit $exec
+    $exec = S_MOV_B64 $sgpr10_sgpr11
+    %1:sreg_32_xm0 = V_READFIRSTLANE_B32 %0, implicit $exec
+
+...
+
+# GCN-LABEL: name: fold-sgpr-copy-readfirstlane-same-block-exec-def{{$}}
+# GCN: COPY
+# GCN-NEXT: $exec = S_MOV_B64
+# GCN-NEXT: V_READFIRSTLANE_B32
+---
+name: fold-sgpr-copy-readfirstlane-same-block-exec-def
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $sgpr10_sgpr11, $sgpr12
+    %0:vgpr_32 = COPY $sgpr12
+    $exec = S_MOV_B64 $sgpr10_sgpr11
+    %1:sreg_32_xm0 = V_READFIRSTLANE_B32 %0, implicit $exec
+...
+
+# GCN-LABEL: name: fold-imm-readfirstlane-user{{$}}
+# GCN: %3:sreg_32_xm0 = S_MOV_B32 123
+---
+name: fold-imm-readfirstlane-user
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0, $sgpr0_sgpr1
+    %0:vgpr_32 = V_MOV_B32_e32 123, implicit $exec
+    %1:sreg_32_xm0 = V_READFIRSTLANE_B32 %0, implicit $exec
+    %2:sreg_32_xm0 = COPY %1
+    %3:sreg_32_xm0 = COPY %2
+    S_ENDPGM 0, implicit %3
+...
+
+# GCN-LABEL: name: fold-imm-readlane{{$}}
+# GCN: %1:sreg_32_xm0 = S_MOV_B32 123
+---
+name: fold-imm-readlane
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0, $sgpr0_sgpr1
+    %0:vgpr_32 = V_MOV_B32_e32 123, implicit $exec
+    %1:sreg_32_xm0 = V_READLANE_B32 %0, 0, implicit $exec
+...
+
+# GCN-LABEL: name: fold-imm-readlane-src1{{$}}
+# GCN: %0:vgpr_32 = COPY $vgpr0
+# GCN: V_READLANE_B32 %0, 12, implicit $exec
+---
+name: fold-imm-readlane-src1
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    %0:vgpr_32 = COPY $vgpr0
+    %1:sreg_32_xm0 = S_MOV_B32 12
+    %2:sreg_32_xm0 = V_READLANE_B32 %0, %1, implicit $exec
+...

test/CodeGen/AMDGPU/llvm.amdgcn.readfirstlane.ll

-; RUN: llc -mtriple=amdgcn--amdhsa -mcpu=fiji -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=amdgcn--amdhsa -mcpu=fiji -verify-machineinstrs < %s | FileCheck -enable-var-scope %s
 
 declare i32 @llvm.amdgcn.readfirstlane(i32) #0
 
 ; CHECK-LABEL: {{^}}test_readfirstlane:
-; CHECK: v_readfirstlane_b32 s{{[0-9]+}}, v{{[0-9]+}}
-define amdgpu_kernel void @test_readfirstlane(i32 addrspace(1)* %out, i32 %src) #1 {
+; CHECK: v_readfirstlane_b32 s{{[0-9]+}}, v2
+define void @test_readfirstlane(i32 addrspace(1)* %out, i32 %src) #1 {
   %readfirstlane = call i32 @llvm.amdgcn.readfirstlane(i32 %src)
   store i32 %readfirstlane, i32 addrspace(1)* %out, align 4
   ret void
 }
 
 ; CHECK-LABEL: {{^}}test_readfirstlane_imm:
-; CHECK: v_mov_b32_e32 [[VVAL:v[0-9]]], 32
-; CHECK: v_readfirstlane_b32 s{{[0-9]+}}, [[VVAL]]
+; CHECK: s_mov_b32 [[SGPR_VAL:s[0-9]]], 32
+; CHECK-NOT: [[SGPR_VAL]]
+; CHECK: ; use [[SGPR_VAL]]
 define amdgpu_kernel void @test_readfirstlane_imm(i32 addrspace(1)* %out) #1 {
   %readfirstlane = call i32 @llvm.amdgcn.readfirstlane(i32 32)
+  call void asm sideeffect "; use $0", "s"(i32 %readfirstlane)
+  ret void
+}
+
+; CHECK-LABEL: {{^}}test_readfirstlane_imm_fold:
+; CHECK: v_mov_b32_e32 [[VVAL:v[0-9]]], 32
+; CHECK-NOT: [[VVAL]]
+; CHECK: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[VVAL]]
+define amdgpu_kernel void @test_readfirstlane_imm_fold(i32 addrspace(1)* %out) #1 {
+  %readfirstlane = call i32 @llvm.amdgcn.readfirstlane(i32 32)
   store i32 %readfirstlane, i32 addrspace(1)* %out, align 4
   ret void
 }
 
 ; TODO: m0 should be folded.
 ; CHECK-LABEL: {{^}}test_readfirstlane_m0:
 ; CHECK: s_mov_b32 m0, -1
 ; CHECK: s_mov_b32 [[COPY_M0:s[0-9]+]], m0
 ; CHECK: v_mov_b32_e32 [[VVAL:v[0-9]]], [[COPY_M0]]
-; CHECK: v_readfirstlane_b32 s{{[0-9]+}}, [[VVAL]]
+; CHECK: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[VVAL]]
 define amdgpu_kernel void @test_readfirstlane_m0(i32 addrspace(1)* %out) #1 {
   %m0 = call i32 asm "s_mov_b32 m0, -1", "={M0}"()
   %readfirstlane = call i32 @llvm.amdgcn.readfirstlane(i32 %m0)
   store i32 %readfirstlane, i32 addrspace(1)* %out, align 4
   ret void
 }
 
+; CHECK-LABEL: {{^}}test_readfirstlane_copy_from_sgpr:
+; CHECK: ;;#ASMSTART
+; CHECK-NEXT: s_mov_b32 [[SGPR:s[0-9]+]]
+; CHECK: ;;#ASMEND
+; CHECK-NOT: [[SGPR]]
+; CHECK-NOT: readfirstlane
+; CHECK: v_mov_b32_e32 [[VCOPY:v[0-9]+]], [[SGPR]]
+; CHECK: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[VCOPY]]
+define amdgpu_kernel void @test_readfirstlane_copy_from_sgpr(i32 addrspace(1)* %out) #1 {
+  %sgpr = call i32 asm "s_mov_b32 $0, 0", "=s"()
+  %readfirstlane = call i32 @llvm.amdgcn.readfirstlane(i32 %sgpr)
+  store i32 %readfirstlane, i32 addrspace(1)* %out, align 4
+  ret void
+}
+
+; Make sure this doesn't crash.
+; CHECK-LABEL: {{^}}test_readfirstlane_fi:
+; CHECK: v_mov_b32_e32 [[FIVAL:v[0-9]]], 4
+; CHECK: v_readfirstlane_b32 s{{[0-9]+}}, [[FIVAL]]
+define amdgpu_kernel void @test_readfirstlane_fi(i32 addrspace(1)* %out) #1 {
+  %alloca = alloca i32, addrspace(5)
+  %int = ptrtoint i32 addrspace(5)* %alloca to i32
+  %readfirstlane = call i32 @llvm.amdgcn.readfirstlane(i32 %int)
+  call void asm sideeffect "; use $0", "s"(i32 %readfirstlane)
+  ret void
+}
+
 attributes #0 = { nounwind readnone convergent }
 attributes #1 = { nounwind }

test/CodeGen/AMDGPU/llvm.amdgcn.readlane.ll

-; RUN: llc -mtriple=amdgcn--amdhsa -mcpu=fiji -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=amdgcn--amdhsa -mcpu=fiji -verify-machineinstrs < %s | FileCheck -enable-var-scope %s
 
 declare i32 @llvm.amdgcn.readlane(i32, i32) #0
 
-; CHECK-LABEL: {{^}}test_readlane_sreg:
-; CHECK: v_readlane_b32 s{{[0-9]+}}, v{{[0-9]+}}, s{{[0-9]+}}
-define amdgpu_kernel void @test_readlane_sreg(i32 addrspace(1)* %out, i32 %src0, i32 %src1) #1 {
+; CHECK-LABEL: {{^}}test_readlane_sreg_sreg:
+; CHECK-NOT: v_readlane_b32
+define amdgpu_kernel void @test_readlane_sreg_sreg(i32 %src0, i32 %src1) #1 {
   %readlane = call i32 @llvm.amdgcn.readlane(i32 %src0, i32 %src1)
-  store i32 %readlane, i32 addrspace(1)* %out, align 4
+  call void asm sideeffect "; use $0", "s"(i32 %readlane)
+  ret void
+}
+
+; CHECK-LABEL: {{^}}test_readlane_vreg_sreg:
+; CHECK: v_readlane_b32 s{{[0-9]+}}, v{{[0-9]+}}, s{{[0-9]+}}
+define amdgpu_kernel void @test_readlane_vreg_sreg(i32 %src0, i32 %src1) #1 {
+  %vgpr = call i32 asm sideeffect "; def $0", "=v"()
+  %readlane = call i32 @llvm.amdgcn.readlane(i32 %vgpr, i32 %src1)
+  call void asm sideeffect "; use $0", "s"(i32 %readlane)
   ret void
 }
 
 ; CHECK-LABEL: {{^}}test_readlane_imm_sreg:
-; CHECK: v_mov_b32_e32 [[VVAL:v[0-9]]], 32
-; CHECK: v_readlane_b32 s{{[0-9]+}}, [[VVAL]], s{{[0-9]+}}
+; CHECK-NOT: v_readlane_b32
 define amdgpu_kernel void @test_readlane_imm_sreg(i32 addrspace(1)* %out, i32 %src1) #1 {
   %readlane = call i32 @llvm.amdgcn.readlane(i32 32, i32 %src1)
   store i32 %readlane, i32 addrspace(1)* %out, align 4
   ret void
 }
 
 ; CHECK-LABEL: {{^}}test_readlane_vregs:
 ; CHECK: v_readfirstlane_b32 [[LANE:s[0-9]+]], v{{[0-9]+}}
define amdgpu_kernel void @test_readlane_vregs(i32 addrspace(1)* %out, <2 x i32> addrspace(1)* %in) #1 {
   ret void
 }
 
 ; TODO: m0 should be folded.
 ; CHECK-LABEL: {{^}}test_readlane_m0_sreg:
 ; CHECK: s_mov_b32 m0, -1
 ; CHECK: s_mov_b32 [[COPY_M0:s[0-9]+]], m0
 ; CHECK: v_mov_b32_e32 [[VVAL:v[0-9]]], [[COPY_M0]]
-; CHECK: v_readlane_b32 s{{[0-9]+}}, [[VVAL]], s{{[0-9]+}}
+; CHECK: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[VVAL]]
 define amdgpu_kernel void @test_readlane_m0_sreg(i32 addrspace(1)* %out, i32 %src1) #1 {
   %m0 = call i32 asm "s_mov_b32 m0, -1", "={M0}"()
   %readlane = call i32 @llvm.amdgcn.readlane(i32 %m0, i32 %src1)
   store i32 %readlane, i32 addrspace(1)* %out, align 4
   ret void
 }
 
-; CHECK-LABEL: {{^}}test_readlane_imm:
+; CHECK-LABEL: {{^}}test_readlane_vgpr_imm:
 ; CHECK: v_readlane_b32 s{{[0-9]+}}, v{{[0-9]+}}, 32
-define amdgpu_kernel void @test_readlane_imm(i32 addrspace(1)* %out, i32 %src0) #1 {
-  %readlane = call i32 @llvm.amdgcn.readlane(i32 %src0, i32 32) #0
+define amdgpu_kernel void @test_readlane_vgpr_imm(i32 addrspace(1)* %out) #1 {
+  %vgpr = call i32 asm sideeffect "; def $0", "=v"()
+  %readlane = call i32 @llvm.amdgcn.readlane(i32 %vgpr, i32 32) #0
   store i32 %readlane, i32 addrspace(1)* %out, align 4
   ret void
 }
 
+; CHECK-LABEL: {{^}}test_readlane_copy_from_sgpr:
+; CHECK: ;;#ASMSTART
+; CHECK-NEXT: s_mov_b32 [[SGPR:s[0-9]+]]
+; CHECK: ;;#ASMEND
+; CHECK-NOT: [[SGPR]]
+; CHECK-NOT: readlane
+; CHECK: v_mov_b32_e32 [[VCOPY:v[0-9]+]], [[SGPR]]
+; CHECK: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[VCOPY]]
+define amdgpu_kernel void @test_readlane_copy_from_sgpr(i32 addrspace(1)* %out) #1 {
+  %sgpr = call i32 asm "s_mov_b32 $0, 0", "=s"()
+  %readfirstlane = call i32 @llvm.amdgcn.readlane(i32 %sgpr, i32 7)
+  store i32 %readfirstlane, i32 addrspace(1)* %out, align 4
+  ret void
+}
+
 declare i32 @llvm.amdgcn.workitem.id.x() #2
 
 attributes #0 = { nounwind readnone convergent }
 attributes #1 = { nounwind }
 attributes #2 = { nounwind readnone }