[AMDGPU][GlobalISel] Widen 1 and 2 byte scalar loads

[vangthao]

Summary

Widen 1 and 2 byte scalar loads to 4 bytes when sufficiently
aligned to avoid using a global load.

[arsenm]

Needs some pure MIR tests too (also including negative tests)

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

448

There's no point in checking the size anymore

1159

I don't understand this getSize check. It's illegal for the memory size to exceed the loaded type

1160

You could also handle G_SEXTLOAD and G_ZEXTLOAD, but would require inserting some instructions to set the high bits appropriately. This also makes me think this should be done earlier

1171

Needs to elaborate more on when this is valid

1174

Should use trunc. I'm trying to eliminate G_EXTRACT usage

[vangthao]

Handle G_SEXTLOAD and G_ZEXTLOAD and added more tests.

[foad]

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

1181

Use B.buildZExtInReg.

[vangthao]

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

1181

I attempted to use B.buildZExtInReg but this function creates a new virtual register as the destination instead of using MI.getOperand(0)'s register when passing it as the first argument. This is unlike buildSExtInReg which uses the first argument as the destination.

[foad]

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

1181

Ah, that's a bug in the implementation:

--- a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
@@ -492,7 +492,7 @@ MachineInstrBuilder MachineIRBuilder::buildZExtInReg(const DstOp &Res,
   LLT ResTy = Res.getLLTTy(*getMRI());
   auto Mask = buildConstant(
       ResTy, APInt::getLowBitsSet(ResTy.getScalarSizeInBits(), ImmOp));
-  return buildAnd(ResTy, Op, Mask);
+  return buildAnd(Res, Op, Mask);
 }
 
 MachineInstrBuilder MachineIRBuilder::buildCast(const DstOp &Dst,

[vangthao]

Fix destination register bug in buildZExtInReg and use it for widening scalar G_ZEXTLOAD memory access.

[arsenm]

llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp

495 ↗

(On Diff #338904)

Should split this into a separate patch

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

1166

Grammar, widen->widened

1170–1171

This comment is worded confusingly so it doesn't explain what is going on. We are widening the memory access, not the result register type

1174

Should just hoist the S32 definition instead of repeating it

[vangthao]

Reverted buildZExtInReg changes for later patch. Addressed comments.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

447–448

The old comment was more informative

llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-widen-scalar-loads.mir

245 ↗

(On Diff #339248)

Should add some dummy uses, e.g. S_ENDPGM 0, implicit %1

Just in case regbankselect decides to start dropping dead uses someday

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-widen-scalar-loads.mir

289 ↗

(On Diff #339248)

Maybe add a negative case for the memory not being invariant

[vangthao]

Updated mir test and comment.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

1179–1181

MachineIRBuilder has a buildZExtInReg helper to figure out the mask and create the and for you

1181

S32.getScalarSizeInBits() is a complicated way of just saying 32

[vangthao]

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

1179–1181

I used this previously but there was an issue with the implementation in buildZExtInReg where it would create a new destination register instead of using the original destination register passed to it. Jay mentioned this bug in a previous comment.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

1179–1181

Yes, buildZExtInReg is broken and needs to be fixed. You should fix it and continue using it

[vangthao]

Rebase and use buildZExtInReg

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

 //===- AMDGPURegisterBankInfo.cpp -------------------------------*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
static bool isScalarLoadLegal(const MachineInstr &MI) {
   if (!MI.hasOneMemOperand())
     return false;
 
   const MachineMemOperand *MMO = *MI.memoperands_begin();
   const unsigned AS = MMO->getAddrSpace();
   const bool IsConst = AS == AMDGPUAS::CONSTANT_ADDRESS ||
                        AS == AMDGPUAS::CONSTANT_ADDRESS_32BIT;
 
-  // There are no extending SMRD/SMEM loads, and they require 4-byte alignment.
-  return MMO->getSize() >= 4 && MMO->getAlign() >= Align(4) &&
+  // Require 4-byte alignment.
+  return MMO->getSize() >= 1 && MMO->getAlign() >= Align(4) &&
          // Can't do a scalar atomic load.
          !MMO->isAtomic() &&
          // Don't use scalar loads for volatile accesses to non-constant address
          // spaces.
          (IsConst || !MMO->isVolatile()) &&
          // Memory must be known constant, or not written before this load.
          (IsConst || MMO->isInvariant() || memOpHasNoClobbered(MMO)) &&
          AMDGPUInstrInfo::isUniformMMO(MMO);
bool AMDGPURegisterBankInfo::applyMappingLoad(MachineInstr &MI,
   Register DstReg = MI.getOperand(0).getReg();
   const LLT LoadTy = MRI.getType(DstReg);
   unsigned LoadSize = LoadTy.getSizeInBits();
   const unsigned MaxNonSmrdLoadSize = 128;
 
   const RegisterBank *PtrBank =
     OpdMapper.getInstrMapping().getOperandMapping(1).BreakDown[0].RegBank;
   if (PtrBank == &AMDGPU::SGPRRegBank) {
-    // If the pointer is an SGPR, we ordinarily have nothing to do.
-    if (LoadSize != 96)
+    MachineMemOperand *MMO = *MI.memoperands_begin();
+    // There are some special cases that we need to look at for 32 bit and 96
+    // bit SGPR loads otherwise we have nothing to do.
+    if (LoadSize != 32 && LoadSize != 96)
+      return false;
+
+    // Scalar loads of size 8 or 16 bit with proper alignment may be widen to 32
+    // bit.
+    if (LoadSize == 32 && (MMO->getSize() >= 4 ||
+          MI.getOpcode() != AMDGPU::G_LOAD ||
+          LoadTy.isVector() ||
+          !isScalarLoadLegal(MI)))
       return false;
 
-    MachineMemOperand *MMO = *MI.memoperands_begin();
     Register PtrReg = MI.getOperand(1).getReg();
-    // 96-bit loads are only available for vector loads. We need to split this
-    // into a 64-bit part, and 32 (unless we can widen to a 128-bit load).
 
     ApplyRegBankMapping O(*this, MRI, &AMDGPU::SGPRRegBank);
     MachineIRBuilder B(MI, O);
 
+    if (LoadSize == 32) {
+      // Widen memory access for 8 or 16 bit scalar loads to 32 bit.
+      LLT WiderTy = LLT::scalar(32);
+      auto WideLoad = B.buildLoadFromOffset(WiderTy, PtrReg, *MMO, 0);
+      B.buildExtract(MI.getOperand(0), WideLoad, 0);
+    } else {
+      // 96-bit loads are only available for vector loads. We need to split this
+      // into a 64-bit part, and 32 (unless we can widen to a 128-bit load).
       if (MMO->getAlign() < Align(16)) {
         LLT Part64, Part32;
         std::tie(Part64, Part32) = splitUnequalType(LoadTy, 64);
         auto Load0 = B.buildLoadFromOffset(Part64, PtrReg, *MMO, 0);
         auto Load1 = B.buildLoadFromOffset(Part32, PtrReg, *MMO, 8);
 
         auto Undef = B.buildUndef(LoadTy);
         auto Ins0 = B.buildInsert(LoadTy, Undef, Load0, 0);
         B.buildInsert(MI.getOperand(0), Ins0, Load1, 64);
       } else {
         LLT WiderTy = widen96To128(LoadTy);
         auto WideLoad = B.buildLoadFromOffset(WiderTy, PtrReg, *MMO, 0);
         B.buildExtract(MI.getOperand(0), WideLoad, 0);
       }
+    }
 
     MI.eraseFromParent();
     return true;
   }
 
   // 128-bit loads are supported for all instruction types.
   if (LoadSize <= MaxNonSmrdLoadSize)
     return false;

llvm/test/CodeGen/AMDGPU/GlobalISel/frem.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN:  llc -global-isel -amdgpu-scalarize-global-loads=false -enable-misched=0 -march=amdgcn -mcpu=bonaire -verify-machineinstrs < %s | FileCheck --check-prefix=CI %s
 ; RUN:  llc -global-isel -amdgpu-scalarize-global-loads=false -enable-misched=0 -march=amdgcn -mcpu=tonga -verify-machineinstrs < %s | FileCheck --check-prefix=VI %s
 
 define amdgpu_kernel void @frem_f16(half addrspace(1)* %out, half addrspace(1)* %in1, half addrspace(1)* %in2) #0 {
 ; CI-LABEL: frem_f16:
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
-; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
-; CI-NEXT:    s_mov_b32 s10, -1
-; CI-NEXT:    s_mov_b32 s11, 0xf000
-; CI-NEXT:    s_mov_b64 s[2:3], s[10:11]
-; CI-NEXT:    s_waitcnt lgkmcnt(0)
-; CI-NEXT:    s_mov_b64 s[8:9], s[6:7]
-; CI-NEXT:    buffer_load_ushort v0, off, s[8:11], 0
-; CI-NEXT:    buffer_load_ushort v1, off, s[0:3], 0 offset:8
-; CI-NEXT:    s_waitcnt vmcnt(1)
-; CI-NEXT:    v_cvt_f32_f16_e32 v0, v0
-; CI-NEXT:    s_waitcnt vmcnt(0)
-; CI-NEXT:    v_cvt_f32_f16_e32 v1, v1
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dword s0, s[6:7], 0x0
+; CI-NEXT:    s_load_dword s1, s[8:9], 0x2
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_cvt_f32_f16_e32 v0, s0
+; CI-NEXT:    v_cvt_f32_f16_e32 v1, s1
 ; CI-NEXT:    v_div_scale_f32 v2, s[0:1], v1, v1, v0
 ; CI-NEXT:    v_div_scale_f32 v3, vcc, v0, v1, v0
 ; CI-NEXT:    v_rcp_f32_e32 v4, v2
 ; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
 ; CI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
 ; CI-NEXT:    v_fma_f32 v4, v5, v4, v4
 ; CI-NEXT:    v_mul_f32_e32 v5, v3, v4
 ; CI-NEXT:    v_fma_f32 v6, -v2, v5, v3
 ; CI-NEXT:    v_fma_f32 v5, v6, v4, v5
 ; CI-NEXT:    v_fma_f32 v2, -v2, v5, v3
 ; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
 ; CI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
-; CI-NEXT:    s_mov_b64 s[6:7], s[10:11]
+; CI-NEXT:    s_mov_b32 s6, -1
+; CI-NEXT:    s_mov_b32 s7, 0xf000
 ; CI-NEXT:    v_div_fixup_f32 v2, v2, v1, v0
 ; CI-NEXT:    v_trunc_f32_e32 v2, v2
 ; CI-NEXT:    v_fma_f32 v0, -v2, v1, v0
 ; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 2, 2), 0
 ; CI-NEXT:    v_cvt_f16_f32_e32 v0, v0
 ; CI-NEXT:    buffer_store_short v0, off, s[4:7], 0
 ; CI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: frem_f16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_mov_b32_e32 v0, s6
-; VI-NEXT:    s_add_u32 s0, s8, 8
-; VI-NEXT:    v_mov_b32_e32 v1, s7
-; VI-NEXT:    s_addc_u32 s1, s9, 0
-; VI-NEXT:    flat_load_ushort v2, v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    s_load_dword s0, s[6:7], 0x0
+; VI-NEXT:    s_load_dword s1, s[8:9], 0x8
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_cvt_f32_f16_e32 v0, s0
+; VI-NEXT:    v_cvt_f32_f16_e32 v2, s1
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    flat_load_ushort v0, v[0:1]
-; VI-NEXT:    s_waitcnt vmcnt(1)
-; VI-NEXT:    v_cvt_f32_f16_e32 v1, v2
-; VI-NEXT:    s_waitcnt vmcnt(0)
-; VI-NEXT:    v_cvt_f32_f16_e32 v3, v0
-; VI-NEXT:    v_rcp_f32_e32 v3, v3
-; VI-NEXT:    v_mul_f32_e32 v1, v1, v3
-; VI-NEXT:    v_cvt_f16_f32_e32 v1, v1
-; VI-NEXT:    v_div_fixup_f16 v1, v1, v0, v2
-; VI-NEXT:    v_trunc_f16_e32 v1, v1
-; VI-NEXT:    v_fma_f16 v2, -v1, v0, v2
+; VI-NEXT:    v_rcp_f32_e32 v2, v2
+; VI-NEXT:    v_mul_f32_e32 v0, v0, v2
+; VI-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; VI-NEXT:    v_div_fixup_f16 v0, v0, v1, s0
+; VI-NEXT:    v_trunc_f16_e32 v0, v0
+; VI-NEXT:    v_fma_f16 v2, -v0, v1, s0
 ; VI-NEXT:    v_mov_b32_e32 v0, s4
 ; VI-NEXT:    v_mov_b32_e32 v1, s5
 ; VI-NEXT:    flat_store_short v[0:1], v2
 ; VI-NEXT:    s_endpgm
    %gep2 = getelementptr half, half addrspace(1)* %in2, i32 4
    %r0 = load half, half addrspace(1)* %in1, align 4
    %r1 = load half, half addrspace(1)* %gep2, align 4
    %r2 = frem half %r0, %r1
    store half %r2, half addrspace(1)* %out, align 4
    ret void
 }
 
 define amdgpu_kernel void @fast_frem_f16(half addrspace(1)* %out, half addrspace(1)* %in1, half addrspace(1)* %in2) #0 {
 ; CI-LABEL: fast_frem_f16:
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
-; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
-; CI-NEXT:    s_mov_b32 s10, -1
-; CI-NEXT:    s_mov_b32 s11, 0xf000
-; CI-NEXT:    s_mov_b64 s[2:3], s[10:11]
-; CI-NEXT:    s_waitcnt lgkmcnt(0)
-; CI-NEXT:    s_mov_b64 s[8:9], s[6:7]
-; CI-NEXT:    buffer_load_ushort v1, off, s[0:3], 0 offset:8
-; CI-NEXT:    buffer_load_ushort v0, off, s[8:11], 0
-; CI-NEXT:    s_mov_b64 s[6:7], s[10:11]
-; CI-NEXT:    s_waitcnt vmcnt(1)
-; CI-NEXT:    v_cvt_f32_f16_e32 v1, v1
-; CI-NEXT:    s_waitcnt vmcnt(0)
-; CI-NEXT:    v_cvt_f32_f16_e32 v0, v0
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dword s0, s[6:7], 0x0
+; CI-NEXT:    s_load_dword s1, s[8:9], 0x2
+; CI-NEXT:    s_mov_b32 s6, -1
+; CI-NEXT:    s_mov_b32 s7, 0xf000
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_cvt_f32_f16_e32 v0, s0
+; CI-NEXT:    v_cvt_f32_f16_e32 v1, s1
 ; CI-NEXT:    v_rcp_f32_e32 v2, v1
 ; CI-NEXT:    v_mul_f32_e32 v2, v0, v2
 ; CI-NEXT:    v_trunc_f32_e32 v2, v2
 ; CI-NEXT:    v_fma_f32 v0, -v2, v1, v0
 ; CI-NEXT:    v_cvt_f16_f32_e32 v0, v0
 ; CI-NEXT:    buffer_store_short v0, off, s[4:7], 0
 ; CI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: fast_frem_f16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_mov_b32_e32 v0, s6
-; VI-NEXT:    s_add_u32 s0, s8, 8
-; VI-NEXT:    v_mov_b32_e32 v1, s7
-; VI-NEXT:    s_addc_u32 s1, s9, 0
-; VI-NEXT:    flat_load_ushort v2, v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    flat_load_ushort v0, v[0:1]
-; VI-NEXT:    s_waitcnt vmcnt(0)
-; VI-NEXT:    v_rcp_f16_e32 v1, v0
-; VI-NEXT:    v_mul_f16_e32 v1, v2, v1
-; VI-NEXT:    v_trunc_f16_e32 v1, v1
-; VI-NEXT:    v_fma_f16 v2, -v1, v0, v2
+; VI-NEXT:    s_load_dword s0, s[6:7], 0x0
+; VI-NEXT:    s_load_dword s1, s[8:9], 0x8
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v1, s0
+; VI-NEXT:    v_rcp_f16_e32 v0, s1
+; VI-NEXT:    v_mul_f16_e32 v0, s0, v0
+; VI-NEXT:    v_trunc_f16_e32 v0, v0
+; VI-NEXT:    v_fma_f16 v2, -v0, s1, v1
 ; VI-NEXT:    v_mov_b32_e32 v0, s4
 ; VI-NEXT:    v_mov_b32_e32 v1, s5
 ; VI-NEXT:    flat_store_short v[0:1], v2
 ; VI-NEXT:    s_endpgm
    %gep2 = getelementptr half, half addrspace(1)* %in2, i32 4
    %r0 = load half, half addrspace(1)* %in1, align 4
    %r1 = load half, half addrspace(1)* %gep2, align 4
    %r2 = frem fast half %r0, %r1
    store half %r2, half addrspace(1)* %out, align 4
    ret void
 }
 
 define amdgpu_kernel void @unsafe_frem_f16(half addrspace(1)* %out, half addrspace(1)* %in1, half addrspace(1)* %in2) #1 {
 ; CI-LABEL: unsafe_frem_f16:
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
-; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
-; CI-NEXT:    s_mov_b32 s10, -1
-; CI-NEXT:    s_mov_b32 s11, 0xf000
-; CI-NEXT:    s_mov_b64 s[2:3], s[10:11]
-; CI-NEXT:    s_waitcnt lgkmcnt(0)
-; CI-NEXT:    s_mov_b64 s[8:9], s[6:7]
-; CI-NEXT:    buffer_load_ushort v1, off, s[0:3], 0 offset:8
-; CI-NEXT:    buffer_load_ushort v0, off, s[8:11], 0
-; CI-NEXT:    s_mov_b64 s[6:7], s[10:11]
-; CI-NEXT:    s_waitcnt vmcnt(1)
-; CI-NEXT:    v_cvt_f32_f16_e32 v1, v1
-; CI-NEXT:    s_waitcnt vmcnt(0)
-; CI-NEXT:    v_cvt_f32_f16_e32 v0, v0
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dword s0, s[6:7], 0x0
+; CI-NEXT:    s_load_dword s1, s[8:9], 0x2
+; CI-NEXT:    s_mov_b32 s6, -1
+; CI-NEXT:    s_mov_b32 s7, 0xf000
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_cvt_f32_f16_e32 v0, s0
+; CI-NEXT:    v_cvt_f32_f16_e32 v1, s1
 ; CI-NEXT:    v_rcp_f32_e32 v2, v1
 ; CI-NEXT:    v_mul_f32_e32 v2, v0, v2
 ; CI-NEXT:    v_trunc_f32_e32 v2, v2
 ; CI-NEXT:    v_fma_f32 v0, -v2, v1, v0
 ; CI-NEXT:    v_cvt_f16_f32_e32 v0, v0
 ; CI-NEXT:    buffer_store_short v0, off, s[4:7], 0
 ; CI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: unsafe_frem_f16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_mov_b32_e32 v0, s6
-; VI-NEXT:    s_add_u32 s0, s8, 8
-; VI-NEXT:    v_mov_b32_e32 v1, s7
-; VI-NEXT:    s_addc_u32 s1, s9, 0
-; VI-NEXT:    flat_load_ushort v2, v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    flat_load_ushort v0, v[0:1]
-; VI-NEXT:    s_waitcnt vmcnt(0)
-; VI-NEXT:    v_rcp_f16_e32 v1, v0
-; VI-NEXT:    v_mul_f16_e32 v1, v2, v1
-; VI-NEXT:    v_trunc_f16_e32 v1, v1
-; VI-NEXT:    v_fma_f16 v2, -v1, v0, v2
+; VI-NEXT:    s_load_dword s0, s[6:7], 0x0
+; VI-NEXT:    s_load_dword s1, s[8:9], 0x8
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v1, s0
+; VI-NEXT:    v_rcp_f16_e32 v0, s1
+; VI-NEXT:    v_mul_f16_e32 v0, s0, v0
+; VI-NEXT:    v_trunc_f16_e32 v0, v0
+; VI-NEXT:    v_fma_f16 v2, -v0, s1, v1
 ; VI-NEXT:    v_mov_b32_e32 v0, s4
 ; VI-NEXT:    v_mov_b32_e32 v1, s5
 ; VI-NEXT:    flat_store_short v[0:1], v2
 ; VI-NEXT:    s_endpgm
    %gep2 = getelementptr half, half addrspace(1)* %in2, i32 4
    %r0 = load half, half addrspace(1)* %in1, align 4
    %r1 = load half, half addrspace(1)* %gep2, align 4
    %r2 = frem half %r0, %r1

llvm/test/CodeGen/AMDGPU/GlobalISel/widen-i8-i16-scalar-loads.ll

+; RUN: llc -global-isel -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck %s
+
+define amdgpu_kernel void @constant_load_i8(i8 addrspace (1)* %out, i8 addrspace(4)* %in) #0 {
+; CHECK-LABEL: {{^}}constant_load_i8:
+; CHECK:    s_load_dword s2, s[2:3], 0x0
+; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
+  %ld = load i8, i8 addrspace(4)* %in, align 4
+  store i8 %ld, i8 addrspace(1)* %out, align 4
+  ret void
+}
+
+define amdgpu_kernel void @constant_load_i16(i16 addrspace (1)* %out, i16 addrspace(4)* %in) #0 {
+; CHECK-LABEL: {{^}}constant_load_i16:
+; CHECK:    s_load_dword s2, s[2:3], 0x0
+; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
+  %ld = load i16, i16 addrspace(4)* %in, align 4
+  store i16 %ld, i16 addrspace(1)* %out, align 4
+  ret void
+}
+