This is an archive of the discontinued LLVM Phabricator instance.

Differential D152909

[AMDGPU][GFX11] Add test coverage for 16-bit conversions, part 16.
AbandonedPublic

Authored by kosarev on Jun 14 2023, 5:43 AM.

Details

Reviewers
arsenm
Joe_Nash
rampitec
foad

Diff Detail

Event Timeline

kosarev created this revision.Jun 14 2023, 5:43 AM
Herald added a project: Restricted Project. · View Herald TranscriptJun 14 2023, 5:43 AM
Herald added subscribers: StephenFan, kerbowa, tpr and 4 others. · View Herald Transcript
kosarev requested review of this revision.Jun 14 2023, 5:43 AM
Herald added a project: Restricted Project. · View Herald TranscriptJun 14 2023, 5:43 AM
Herald added subscribers: llvm-commits, wdng. · View Herald Transcript
Harbormaster completed remote builds in B238778: Diff 531292.Jun 14 2023, 7:15 AM
Joe_Nash added inline comments.Jun 14 2023, 7:48 AM
llvm/test/CodeGen/AMDGPU/v_mac.ll
9

The tests in these files appear to check for usage of mac or mad. So we would need to know whether equivalent fma/fmac should be expected on GFX11, and why they are not present. There are other tests for fma on GFX11, though I'm not sure if they cover similar cases. Additionally, the hand written tests basically look for a single instruction. It seems a shame to reduce that clarity. It might be better to skip this patch.

kosarev added inline comments.Jun 15 2023, 4:30 AM
llvm/test/CodeGen/AMDGPU/v_mac.ll
9

Or, alternatively, maybe file a ticket, add a TODO and let the test catch the changes in the output?

Joe_Nash added inline comments.Jun 15 2023, 7:25 AM
llvm/test/CodeGen/AMDGPU/v_mac.ll
9

That is potentially an option. But if we knew at this moment there was no way to produce fma on GFX11 from the input IR, I would prefer not to land this patch, and there would be no ticket to file.

kosarev added inline comments.Jun 15 2023, 8:12 AM
llvm/test/CodeGen/AMDGPU/v_mac.ll
9

Jay @foad, Matt @arsenm, do you know if there are good reasons for v_fma not to be generated for these inputs on GFX11?

arsenm added inline comments.Jun 15 2023, 8:14 AM
llvm/test/CodeGen/AMDGPU/v_mac.ll
9

It's not legal because these don't have the contract flag. mad requires no denormals but doesn't require contract. fma requires contract but respects the denormal mode

kosarev mentioned this in D153269: [AMDGPU][GFX11] Add test coverage for FMA instructions..Jun 19 2023, 5:44 AM

I'm going to abandon this in favour of D153269, if no objections.

Joe_Nash added a comment.Jun 19 2023, 9:18 AM
In D152909#4432481, @kosarev wrote:

I'm going to abandon this in favour of D153269, if no objections.

Sounds good

kosarev abandoned this revision.Jun 20 2023, 2:51 AM

Revision Contents

PathSize
llvm/
test/
CodeGen/
AMDGPU/
1438 lines
2658 lines

Diff 531292

llvm/test/CodeGen/AMDGPU/v_mac.ll

; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mattr=+mad-mac-f32-insts -denormal-fp-math-f32=preserve-sign -verify-machineinstrs < %s | FileCheck --check-prefixes=SI,GCN %s ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=tonga -denormal-fp-math=preserve-sign -denormal-fp-math-f32=preserve-sign -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck --check-prefixes=VI-FLUSH,GCN %s ; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mattr=+mad-mac-f32-insts -denormal-fp-math-f32=preserve-sign -verify-machineinstrs < %s | FileCheck --check-prefixes=SI %s
; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=tonga -denormal-fp-math=ieee -denormal-fp-math-f32=preserve-sign -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s ; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=tonga -denormal-fp-math=preserve-sign -denormal-fp-math-f32=preserve-sign -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck --check-prefixes=VI-FLUSH %s
; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=tonga -denormal-fp-math=ieee -denormal-fp-math-f32=preserve-sign -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefixes=VI %s
; GCN-LABEL: {{^}}mac_vvv: ; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=gfx1100 -denormal-fp-math=preserve-sign -denormal-fp-math-f32=preserve-sign -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck --check-prefixes=GFX11-FLUSH %s
; GCN: buffer_load_dword [[A:v[0-9]+]], off, s[{{[0-9]+:[0-9]+}}], 0 glc{{$}} ; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=gfx1100 -denormal-fp-math=ieee -denormal-fp-math-f32=preserve-sign -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX11 %s
; GCN: buffer_load_dword [[B:v[0-9]+]], off, s[{{[0-9]+:[0-9]+}}], 0 offset:4
; GCN: buffer_load_dword [[C:v[0-9]+]], off, s[{{[0-9]+:[0-9]+}}], 0 offset:8
; GCN: v_mac_f32_e32 [[C]], [[A]], [[B]]
Joe_NashUnsubmitted
Not Done
ReplyInline Actions

The tests in these files appear to check for usage of mac or mad. So we would need to know whether equivalent fma/fmac should be expected on GFX11, and why they are not present. There are other tests for fma on GFX11, though I'm not sure if they cover similar cases. Additionally, the hand written tests basically look for a single instruction. It seems a shame to reduce that clarity. It might be better to skip this patch.

Joe_Nash: The tests in these files appear to check for usage of mac or mad. So we would need to know…
kosarevAuthorUnsubmitted
Done
ReplyInline Actions

Or, alternatively, maybe file a ticket, add a TODO and let the test catch the changes in the output?

kosarev: Or, alternatively, maybe file a ticket, add a TODO and let the test catch the changes in the…
Joe_NashUnsubmitted
Not Done
ReplyInline Actions

That is potentially an option. But if we knew at this moment there was no way to produce fma on GFX11 from the input IR, I would prefer not to land this patch, and there would be no ticket to file.

Joe_Nash: That is potentially an option. But if we knew at this moment there was no way to produce fma on…
kosarevAuthorUnsubmitted
Done
ReplyInline Actions

Jay @foad, Matt @arsenm, do you know if there are good reasons for v_fma not to be generated for these inputs on GFX11?

kosarev: Jay @foad, Matt @arsenm, do you know if there are good reasons for v_fma not to be generated…
arsenmUnsubmitted
Not Done
ReplyInline Actions

It's not legal because these don't have the contract flag. mad requires no denormals but doesn't require contract. fma requires contract but respects the denormal mode

arsenm: It's not legal because these don't have the contract flag. mad requires no denormals but…
; GCN: buffer_store_dword [[C]]
define amdgpu_kernel void @mac_vvv(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 { define amdgpu_kernel void @mac_vvv(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
; SI-LABEL: mac_vvv:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s7, 0xf000
; SI-NEXT: s_mov_b32 s6, -1
; SI-NEXT: s_mov_b32 s10, s6
; SI-NEXT: s_mov_b32 s11, s7
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s8, s2
; SI-NEXT: s_mov_b32 s9, s3
; SI-NEXT: buffer_load_dword v0, off, s[8:11], 0 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: buffer_load_dword v1, off, s[8:11], 0 offset:4 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:8 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: s_mov_b32 s4, s0
; SI-NEXT: s_mov_b32 s5, s1
; SI-NEXT: v_mac_f32_e32 v2, v0, v1
; SI-NEXT: buffer_store_dword v2, off, s[4:7], 0
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: mac_vvv:
; VI-FLUSH: ; %bb.0: ; %entry
; VI-FLUSH-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-FLUSH-NEXT: s_mov_b32 s7, 0xf000
; VI-FLUSH-NEXT: s_mov_b32 s6, -1
; VI-FLUSH-NEXT: s_mov_b32 s10, s6
; VI-FLUSH-NEXT: s_mov_b32 s11, s7
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s8, s2
; VI-FLUSH-NEXT: s_mov_b32 s9, s3
; VI-FLUSH-NEXT: buffer_load_dword v0, off, s[8:11], 0 glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: buffer_load_dword v1, off, s[8:11], 0 offset:4 glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:8 glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s4, s0
; VI-FLUSH-NEXT: s_mov_b32 s5, s1
; VI-FLUSH-NEXT: v_mac_f32_e32 v2, v0, v1
; VI-FLUSH-NEXT: buffer_store_dword v2, off, s[4:7], 0
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: mac_vvv:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s7, 0xf000
; VI-NEXT: s_mov_b32 s6, -1
; VI-NEXT: s_mov_b32 s10, s6
; VI-NEXT: s_mov_b32 s11, s7
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s8, s2
; VI-NEXT: s_mov_b32 s9, s3
; VI-NEXT: buffer_load_dword v0, off, s[8:11], 0 glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: buffer_load_dword v1, off, s[8:11], 0 offset:4 glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:8 glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: s_mov_b32 s4, s0
; VI-NEXT: s_mov_b32 s5, s1
; VI-NEXT: v_mac_f32_e32 v2, v0, v1
; VI-NEXT: buffer_store_dword v2, off, s[4:7], 0
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: mac_vvv:
; GFX11-FLUSH: ; %bb.0: ; %entry
; GFX11-FLUSH-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_mov_b32 s6, -1
; GFX11-FLUSH-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-FLUSH-NEXT: s_mov_b32 s10, s6
; GFX11-FLUSH-NEXT: s_mov_b32 s11, s7
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s8, s2
; GFX11-FLUSH-NEXT: s_mov_b32 s9, s3
; GFX11-FLUSH-NEXT: buffer_load_b32 v0, off, s[8:11], 0 glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: buffer_load_b32 v1, off, s[8:11], 0 offset:4 glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: buffer_load_b32 v2, off, s[8:11], 0 offset:8 glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s4, s0
; GFX11-FLUSH-NEXT: s_mov_b32 s5, s1
; GFX11-FLUSH-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_add_f32_e32 v0, v0, v2
; GFX11-FLUSH-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: mac_vvv:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s6, -1
; GFX11-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-NEXT: s_mov_b32 s10, s6
; GFX11-NEXT: s_mov_b32 s11, s7
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s8, s2
; GFX11-NEXT: s_mov_b32 s9, s3
; GFX11-NEXT: buffer_load_b32 v0, off, s[8:11], 0 glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: buffer_load_b32 v1, off, s[8:11], 0 offset:4 glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: buffer_load_b32 v2, off, s[8:11], 0 offset:8 glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_mov_b32 s4, s0
; GFX11-NEXT: s_mov_b32 s5, s1
; GFX11-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_add_f32_e32 v0, v0, v2
; GFX11-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
entry: entry:
%b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1 %b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1
%c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2 %c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2
%a = load volatile float, ptr addrspace(1) %in %a = load volatile float, ptr addrspace(1) %in
%b = load volatile float, ptr addrspace(1) %b_ptr %b = load volatile float, ptr addrspace(1) %b_ptr
%c = load volatile float, ptr addrspace(1) %c_ptr %c = load volatile float, ptr addrspace(1) %c_ptr
%tmp0 = fmul float %a, %b %tmp0 = fmul float %a, %b
%tmp1 = fadd float %tmp0, %c %tmp1 = fadd float %tmp0, %c
store float %tmp1, ptr addrspace(1) %out store float %tmp1, ptr addrspace(1) %out
ret void ret void
} }
; GCN-LABEL: {{^}}mad_inline_sgpr_inline:
; GCN-NOT: v_mac_f32
; GCN: v_mad_f32 v{{[0-9]}}, s{{[0-9]+}}, 0.5, 0.5
define amdgpu_kernel void @mad_inline_sgpr_inline(ptr addrspace(1) %out, float %in) #0 { define amdgpu_kernel void @mad_inline_sgpr_inline(ptr addrspace(1) %out, float %in) #0 {
; SI-LABEL: mad_inline_sgpr_inline:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dword s4, s[0:1], 0xb
; SI-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s3, 0xf000
; SI-NEXT: s_mov_b32 s2, -1
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: v_mad_f32 v0, s4, 0.5, 0.5
; SI-NEXT: buffer_store_dword v0, off, s[0:3], 0
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: mad_inline_sgpr_inline:
; VI-FLUSH: ; %bb.0: ; %entry
; VI-FLUSH-NEXT: s_load_dword s4, s[0:1], 0x2c
; VI-FLUSH-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x24
; VI-FLUSH-NEXT: s_mov_b32 s3, 0xf000
; VI-FLUSH-NEXT: s_mov_b32 s2, -1
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: v_mad_f32 v0, s4, 0.5, 0.5
; VI-FLUSH-NEXT: buffer_store_dword v0, off, s[0:3], 0
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: mad_inline_sgpr_inline:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dword s4, s[0:1], 0x2c
; VI-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s3, 0xf000
; VI-NEXT: s_mov_b32 s2, -1
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: v_mad_f32 v0, s4, 0.5, 0.5
; VI-NEXT: buffer_store_dword v0, off, s[0:3], 0
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: mad_inline_sgpr_inline:
; GFX11-FLUSH: ; %bb.0: ; %entry
; GFX11-FLUSH-NEXT: s_clause 0x1
; GFX11-FLUSH-NEXT: s_load_b32 s2, s[0:1], 0x2c
; GFX11-FLUSH-NEXT: s_load_b64 s[0:1], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_mov_b32 s3, 0x31016000
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: v_mul_f32_e64 v0, s2, 0.5
; GFX11-FLUSH-NEXT: s_mov_b32 s2, -1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_add_f32_e32 v0, 0.5, v0
; GFX11-FLUSH-NEXT: buffer_store_b32 v0, off, s[0:3], 0
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: mad_inline_sgpr_inline:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_clause 0x1
; GFX11-NEXT: s_load_b32 s2, s[0:1], 0x2c
; GFX11-NEXT: s_load_b64 s[0:1], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s3, 0x31016000
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: v_mul_f32_e64 v0, s2, 0.5
; GFX11-NEXT: s_mov_b32 s2, -1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_add_f32_e32 v0, 0.5, v0
; GFX11-NEXT: buffer_store_b32 v0, off, s[0:3], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
entry: entry:
%tmp0 = fmul float 0.5, %in %tmp0 = fmul float 0.5, %in
%tmp1 = fadd float %tmp0, 0.5 %tmp1 = fadd float %tmp0, 0.5
store float %tmp1, ptr addrspace(1) %out store float %tmp1, ptr addrspace(1) %out
ret void ret void
} }
; GCN-LABEL: {{^}}mad_vvs:
; GCN-NOT: v_mac_f32
; GCN: v_mad_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, s{{[0-9]+}}
define amdgpu_kernel void @mad_vvs(ptr addrspace(1) %out, ptr addrspace(1) %in, float %c) #0 { define amdgpu_kernel void @mad_vvs(ptr addrspace(1) %out, ptr addrspace(1) %in, float %c) #0 {
; SI-LABEL: mad_vvs:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s3, 0xf000
; SI-NEXT: s_mov_b32 s2, -1
; SI-NEXT: s_load_dword s12, s[0:1], 0xd
; SI-NEXT: s_mov_b32 s10, s2
; SI-NEXT: s_mov_b32 s11, s3
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s8, s6
; SI-NEXT: s_mov_b32 s9, s7
; SI-NEXT: buffer_load_dwordx2 v[0:1], off, s[8:11], 0
; SI-NEXT: s_mov_b32 s0, s4
; SI-NEXT: s_mov_b32 s1, s5
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_mad_f32 v0, v0, v1, s12
; SI-NEXT: buffer_store_dword v0, off, s[0:3], 0
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: mad_vvs:
; VI-FLUSH: ; %bb.0: ; %entry
; VI-FLUSH-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x24
; VI-FLUSH-NEXT: s_load_dword s12, s[0:1], 0x34
; VI-FLUSH-NEXT: s_mov_b32 s3, 0xf000
; VI-FLUSH-NEXT: s_mov_b32 s2, -1
; VI-FLUSH-NEXT: s_mov_b32 s10, s2
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s8, s6
; VI-FLUSH-NEXT: s_mov_b32 s9, s7
; VI-FLUSH-NEXT: s_mov_b32 s11, s3
; VI-FLUSH-NEXT: buffer_load_dwordx2 v[0:1], off, s[8:11], 0
; VI-FLUSH-NEXT: s_mov_b32 s0, s4
; VI-FLUSH-NEXT: s_mov_b32 s1, s5
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: v_mad_f32 v0, v0, v1, s12
; VI-FLUSH-NEXT: buffer_store_dword v0, off, s[0:3], 0
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: mad_vvs:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x24
; VI-NEXT: s_load_dword s12, s[0:1], 0x34
; VI-NEXT: s_mov_b32 s3, 0xf000
; VI-NEXT: s_mov_b32 s2, -1
; VI-NEXT: s_mov_b32 s10, s2
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s8, s6
; VI-NEXT: s_mov_b32 s9, s7
; VI-NEXT: s_mov_b32 s11, s3
; VI-NEXT: buffer_load_dwordx2 v[0:1], off, s[8:11], 0
; VI-NEXT: s_mov_b32 s0, s4
; VI-NEXT: s_mov_b32 s1, s5
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f32 v0, v0, v1, s12
; VI-NEXT: buffer_store_dword v0, off, s[0:3], 0
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: mad_vvs:
; GFX11-FLUSH: ; %bb.0: ; %entry
; GFX11-FLUSH-NEXT: s_load_b128 s[4:7], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_mov_b32 s3, 0x31016000
; GFX11-FLUSH-NEXT: s_mov_b32 s2, -1
; GFX11-FLUSH-NEXT: s_mov_b32 s11, s3
; GFX11-FLUSH-NEXT: s_mov_b32 s10, s2
; GFX11-FLUSH-NEXT: s_load_b32 s0, s[0:1], 0x34
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s8, s6
; GFX11-FLUSH-NEXT: s_mov_b32 s9, s7
; GFX11-FLUSH-NEXT: s_mov_b32 s1, s5
; GFX11-FLUSH-NEXT: buffer_load_b64 v[0:1], off, s[8:11], 0
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_add_f32_e32 v0, s0, v0
; GFX11-FLUSH-NEXT: s_mov_b32 s0, s4
; GFX11-FLUSH-NEXT: buffer_store_b32 v0, off, s[0:3], 0
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: mad_vvs:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b128 s[4:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s3, 0x31016000
; GFX11-NEXT: s_mov_b32 s2, -1
; GFX11-NEXT: s_mov_b32 s11, s3
; GFX11-NEXT: s_mov_b32 s10, s2
; GFX11-NEXT: s_load_b32 s0, s[0:1], 0x34
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s8, s6
; GFX11-NEXT: s_mov_b32 s9, s7
; GFX11-NEXT: s_mov_b32 s1, s5
; GFX11-NEXT: buffer_load_b64 v[0:1], off, s[8:11], 0
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_add_f32_e32 v0, s0, v0
; GFX11-NEXT: s_mov_b32 s0, s4
; GFX11-NEXT: buffer_store_b32 v0, off, s[0:3], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
entry: entry:
%b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1 %b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1
%a = load float, ptr addrspace(1) %in %a = load float, ptr addrspace(1) %in
%b = load float, ptr addrspace(1) %b_ptr %b = load float, ptr addrspace(1) %b_ptr
%tmp0 = fmul float %a, %b %tmp0 = fmul float %a, %b
%tmp1 = fadd float %tmp0, %c %tmp1 = fadd float %tmp0, %c
store float %tmp1, ptr addrspace(1) %out store float %tmp1, ptr addrspace(1) %out
ret void ret void
} }
; GCN-LABEL: {{^}}mac_ssv:
; GCN: v_mac_f32_e64 v{{[0-9]+}}, s{{[0-9]+}}, s{{[0-9]+}}
define amdgpu_kernel void @mac_ssv(ptr addrspace(1) %out, ptr addrspace(1) %in, float %a) #0 { define amdgpu_kernel void @mac_ssv(ptr addrspace(1) %out, ptr addrspace(1) %in, float %a) #0 {
; SI-LABEL: mac_ssv:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s3, 0xf000
; SI-NEXT: s_mov_b32 s2, -1
; SI-NEXT: s_load_dword s12, s[0:1], 0xd
; SI-NEXT: s_mov_b32 s10, s2
; SI-NEXT: s_mov_b32 s11, s3
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s8, s6
; SI-NEXT: s_mov_b32 s9, s7
; SI-NEXT: buffer_load_dword v0, off, s[8:11], 0
; SI-NEXT: s_mov_b32 s0, s4
; SI-NEXT: s_mov_b32 s1, s5
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_mac_f32_e64 v0, s12, s12
; SI-NEXT: buffer_store_dword v0, off, s[0:3], 0
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: mac_ssv:
; VI-FLUSH: ; %bb.0: ; %entry
; VI-FLUSH-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x24
; VI-FLUSH-NEXT: s_load_dword s12, s[0:1], 0x34
; VI-FLUSH-NEXT: s_mov_b32 s3, 0xf000
; VI-FLUSH-NEXT: s_mov_b32 s2, -1
; VI-FLUSH-NEXT: s_mov_b32 s10, s2
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s8, s6
; VI-FLUSH-NEXT: s_mov_b32 s9, s7
; VI-FLUSH-NEXT: s_mov_b32 s11, s3
; VI-FLUSH-NEXT: buffer_load_dword v0, off, s[8:11], 0
; VI-FLUSH-NEXT: s_mov_b32 s0, s4
; VI-FLUSH-NEXT: s_mov_b32 s1, s5
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: v_mac_f32_e64 v0, s12, s12
; VI-FLUSH-NEXT: buffer_store_dword v0, off, s[0:3], 0
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: mac_ssv:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x24
; VI-NEXT: s_load_dword s12, s[0:1], 0x34
; VI-NEXT: s_mov_b32 s3, 0xf000
; VI-NEXT: s_mov_b32 s2, -1
; VI-NEXT: s_mov_b32 s10, s2
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s8, s6
; VI-NEXT: s_mov_b32 s9, s7
; VI-NEXT: s_mov_b32 s11, s3
; VI-NEXT: buffer_load_dword v0, off, s[8:11], 0
; VI-NEXT: s_mov_b32 s0, s4
; VI-NEXT: s_mov_b32 s1, s5
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mac_f32_e64 v0, s12, s12
; VI-NEXT: buffer_store_dword v0, off, s[0:3], 0
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: mac_ssv:
; GFX11-FLUSH: ; %bb.0: ; %entry
; GFX11-FLUSH-NEXT: s_load_b128 s[4:7], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_mov_b32 s3, 0x31016000
; GFX11-FLUSH-NEXT: s_mov_b32 s2, -1
; GFX11-FLUSH-NEXT: s_mov_b32 s11, s3
; GFX11-FLUSH-NEXT: s_mov_b32 s10, s2
; GFX11-FLUSH-NEXT: s_load_b32 s0, s[0:1], 0x34
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s8, s6
; GFX11-FLUSH-NEXT: s_mov_b32 s9, s7
; GFX11-FLUSH-NEXT: s_mov_b32 s1, s5
; GFX11-FLUSH-NEXT: buffer_load_b32 v0, off, s[8:11], 0
; GFX11-FLUSH-NEXT: v_mul_f32_e64 v1, s0, s0
; GFX11-FLUSH-NEXT: s_mov_b32 s0, s4
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_add_f32_e32 v0, v1, v0
; GFX11-FLUSH-NEXT: buffer_store_b32 v0, off, s[0:3], 0
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: mac_ssv:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b128 s[4:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s3, 0x31016000
; GFX11-NEXT: s_mov_b32 s2, -1
; GFX11-NEXT: s_mov_b32 s11, s3
; GFX11-NEXT: s_mov_b32 s10, s2
; GFX11-NEXT: s_load_b32 s0, s[0:1], 0x34
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s8, s6
; GFX11-NEXT: s_mov_b32 s9, s7
; GFX11-NEXT: s_mov_b32 s1, s5
; GFX11-NEXT: buffer_load_b32 v0, off, s[8:11], 0
; GFX11-NEXT: v_mul_f32_e64 v1, s0, s0
; GFX11-NEXT: s_mov_b32 s0, s4
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_add_f32_e32 v0, v1, v0
; GFX11-NEXT: buffer_store_b32 v0, off, s[0:3], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
entry: entry:
%c = load float, ptr addrspace(1) %in %c = load float, ptr addrspace(1) %in
%tmp0 = fmul float %a, %a %tmp0 = fmul float %a, %a
%tmp1 = fadd float %tmp0, %c %tmp1 = fadd float %tmp0, %c
store float %tmp1, ptr addrspace(1) %out store float %tmp1, ptr addrspace(1) %out
ret void ret void
} }
; GCN-LABEL: {{^}}mac_mad_same_add:
; GCN: v_mad_f32 v{{[0-9]}}, v{{[0-9]+}}, v{{[0-9]+}}, [[ADD:v[0-9]+]]
; GCN: v_mac_f32_e32 [[ADD]], v{{[0-9]+}}, v{{[0-9]+}}
define amdgpu_kernel void @mac_mad_same_add(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 { define amdgpu_kernel void @mac_mad_same_add(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
; SI-LABEL: mac_mad_same_add:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s7, 0xf000
; SI-NEXT: s_mov_b32 s6, -1
; SI-NEXT: s_mov_b32 s10, s6
; SI-NEXT: s_mov_b32 s11, s7
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s8, s2
; SI-NEXT: s_mov_b32 s9, s3
; SI-NEXT: buffer_load_dword v0, off, s[8:11], 0 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:4 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: buffer_load_dword v1, off, s[8:11], 0 offset:8 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: buffer_load_dword v3, off, s[8:11], 0 offset:12 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: buffer_load_dword v4, off, s[8:11], 0 offset:16 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: s_mov_b32 s4, s0
; SI-NEXT: s_mov_b32 s5, s1
; SI-NEXT: v_mad_f32 v0, v0, v2, v1
; SI-NEXT: v_mac_f32_e32 v1, v3, v4
; SI-NEXT: buffer_store_dwordx2 v[0:1], off, s[4:7], 0
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: mac_mad_same_add:
; VI-FLUSH: ; %bb.0: ; %entry
; VI-FLUSH-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-FLUSH-NEXT: s_mov_b32 s7, 0xf000
; VI-FLUSH-NEXT: s_mov_b32 s6, -1
; VI-FLUSH-NEXT: s_mov_b32 s10, s6
; VI-FLUSH-NEXT: s_mov_b32 s11, s7
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s8, s2
; VI-FLUSH-NEXT: s_mov_b32 s9, s3
; VI-FLUSH-NEXT: buffer_load_dword v0, off, s[8:11], 0 glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:4 glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: buffer_load_dword v1, off, s[8:11], 0 offset:8 glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: buffer_load_dword v3, off, s[8:11], 0 offset:12 glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: buffer_load_dword v4, off, s[8:11], 0 offset:16 glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s4, s0
; VI-FLUSH-NEXT: s_mov_b32 s5, s1
; VI-FLUSH-NEXT: v_mad_f32 v0, v0, v2, v1
; VI-FLUSH-NEXT: v_mac_f32_e32 v1, v3, v4
; VI-FLUSH-NEXT: buffer_store_dwordx2 v[0:1], off, s[4:7], 0
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: mac_mad_same_add:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s7, 0xf000
; VI-NEXT: s_mov_b32 s6, -1
; VI-NEXT: s_mov_b32 s10, s6
; VI-NEXT: s_mov_b32 s11, s7
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s8, s2
; VI-NEXT: s_mov_b32 s9, s3
; VI-NEXT: buffer_load_dword v0, off, s[8:11], 0 glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:4 glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: buffer_load_dword v1, off, s[8:11], 0 offset:8 glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: buffer_load_dword v3, off, s[8:11], 0 offset:12 glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: buffer_load_dword v4, off, s[8:11], 0 offset:16 glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: s_mov_b32 s4, s0
; VI-NEXT: s_mov_b32 s5, s1
; VI-NEXT: v_mad_f32 v0, v0, v2, v1
; VI-NEXT: v_mac_f32_e32 v1, v3, v4
; VI-NEXT: buffer_store_dwordx2 v[0:1], off, s[4:7], 0
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: mac_mad_same_add:
; GFX11-FLUSH: ; %bb.0: ; %entry
; GFX11-FLUSH-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_mov_b32 s6, -1
; GFX11-FLUSH-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-FLUSH-NEXT: s_mov_b32 s10, s6
; GFX11-FLUSH-NEXT: s_mov_b32 s11, s7
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s8, s2
; GFX11-FLUSH-NEXT: s_mov_b32 s9, s3
; GFX11-FLUSH-NEXT: buffer_load_b32 v0, off, s[8:11], 0 glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: buffer_load_b32 v1, off, s[8:11], 0 offset:4 glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: buffer_load_b32 v2, off, s[8:11], 0 offset:8 glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: buffer_load_b32 v3, off, s[8:11], 0 offset:12 glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: buffer_load_b32 v4, off, s[8:11], 0 offset:16 glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s4, s0
; GFX11-FLUSH-NEXT: s_mov_b32 s5, s1
; GFX11-FLUSH-NEXT: v_dual_mul_f32 v0, v0, v1 :: v_dual_mul_f32 v1, v3, v4
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX11-FLUSH-NEXT: v_add_f32_e32 v0, v0, v2
; GFX11-FLUSH-NEXT: v_add_f32_e32 v1, v1, v2
; GFX11-FLUSH-NEXT: buffer_store_b64 v[0:1], off, s[4:7], 0
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: mac_mad_same_add:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s6, -1
; GFX11-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-NEXT: s_mov_b32 s10, s6
; GFX11-NEXT: s_mov_b32 s11, s7
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s8, s2
; GFX11-NEXT: s_mov_b32 s9, s3
; GFX11-NEXT: buffer_load_b32 v0, off, s[8:11], 0 glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: buffer_load_b32 v1, off, s[8:11], 0 offset:4 glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: buffer_load_b32 v2, off, s[8:11], 0 offset:8 glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: buffer_load_b32 v3, off, s[8:11], 0 offset:12 glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: buffer_load_b32 v4, off, s[8:11], 0 offset:16 glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_mov_b32 s4, s0
; GFX11-NEXT: s_mov_b32 s5, s1
; GFX11-NEXT: v_dual_mul_f32 v0, v0, v1 :: v_dual_mul_f32 v1, v3, v4
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX11-NEXT: v_add_f32_e32 v0, v0, v2
; GFX11-NEXT: v_add_f32_e32 v1, v1, v2
; GFX11-NEXT: buffer_store_b64 v[0:1], off, s[4:7], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
entry: entry:
%b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1 %b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1
%c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2 %c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2
%d_ptr = getelementptr float, ptr addrspace(1) %in, i32 3 %d_ptr = getelementptr float, ptr addrspace(1) %in, i32 3
%e_ptr = getelementptr float, ptr addrspace(1) %in, i32 4 %e_ptr = getelementptr float, ptr addrspace(1) %in, i32 4
%a = load volatile float, ptr addrspace(1) %in %a = load volatile float, ptr addrspace(1) %in
%b = load volatile float, ptr addrspace(1) %b_ptr %b = load volatile float, ptr addrspace(1) %b_ptr
Show All 10 Linesentry:
%out1 = getelementptr float, ptr addrspace(1) %out, i32 1 %out1 = getelementptr float, ptr addrspace(1) %out, i32 1
store float %tmp1, ptr addrspace(1) %out store float %tmp1, ptr addrspace(1) %out
store float %tmp3, ptr addrspace(1) %out1 store float %tmp3, ptr addrspace(1) %out1
ret void ret void
} }
; There is no advantage to using v_mac when one of the operands is negated ; There is no advantage to using v_mac when one of the operands is negated
; and v_mad accepts more operand types. ; and v_mad accepts more operand types.
; GCN-LABEL: {{^}}mad_neg_src0:
; GCN-NOT: v_mac_f32
; GCN: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[-0-9]}}
define amdgpu_kernel void @mad_neg_src0(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 { define amdgpu_kernel void @mad_neg_src0(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
; SI-LABEL: mad_neg_src0:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s7, 0xf000
; SI-NEXT: s_mov_b32 s6, -1
; SI-NEXT: s_mov_b32 s10, s6
; SI-NEXT: s_mov_b32 s11, s7
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s8, s2
; SI-NEXT: s_mov_b32 s9, s3
; SI-NEXT: buffer_load_dwordx2 v[0:1], off, s[8:11], 0
; SI-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:8
; SI-NEXT: s_mov_b32 s4, s0
; SI-NEXT: s_mov_b32 s5, s1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_mad_f32 v0, -v0, v1, v2
; SI-NEXT: buffer_store_dword v0, off, s[4:7], 0
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: mad_neg_src0:
; VI-FLUSH: ; %bb.0: ; %entry
; VI-FLUSH-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-FLUSH-NEXT: s_mov_b32 s7, 0xf000
; VI-FLUSH-NEXT: s_mov_b32 s6, -1
; VI-FLUSH-NEXT: s_mov_b32 s10, s6
; VI-FLUSH-NEXT: s_mov_b32 s11, s7
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s8, s2
; VI-FLUSH-NEXT: s_mov_b32 s9, s3
; VI-FLUSH-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-FLUSH-NEXT: s_mov_b32 s4, s0
; VI-FLUSH-NEXT: s_mov_b32 s5, s1
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: v_mad_f32 v0, -v0, v1, v2
; VI-FLUSH-NEXT: buffer_store_dword v0, off, s[4:7], 0
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: mad_neg_src0:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s7, 0xf000
; VI-NEXT: s_mov_b32 s6, -1
; VI-NEXT: s_mov_b32 s10, s6
; VI-NEXT: s_mov_b32 s11, s7
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s8, s2
; VI-NEXT: s_mov_b32 s9, s3
; VI-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-NEXT: s_mov_b32 s4, s0
; VI-NEXT: s_mov_b32 s5, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f32 v0, -v0, v1, v2
; VI-NEXT: buffer_store_dword v0, off, s[4:7], 0
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: mad_neg_src0:
; GFX11-FLUSH: ; %bb.0: ; %entry
; GFX11-FLUSH-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_mov_b32 s6, -1
; GFX11-FLUSH-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-FLUSH-NEXT: s_mov_b32 s10, s6
; GFX11-FLUSH-NEXT: s_mov_b32 s11, s7
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s8, s2
; GFX11-FLUSH-NEXT: s_mov_b32 s9, s3
; GFX11-FLUSH-NEXT: s_mov_b32 s4, s0
; GFX11-FLUSH-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-FLUSH-NEXT: s_mov_b32 s5, s1
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_sub_f32_e32 v0, v2, v0
; GFX11-FLUSH-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: mad_neg_src0:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s6, -1
; GFX11-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-NEXT: s_mov_b32 s10, s6
; GFX11-NEXT: s_mov_b32 s11, s7
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s8, s2
; GFX11-NEXT: s_mov_b32 s9, s3
; GFX11-NEXT: s_mov_b32 s4, s0
; GFX11-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-NEXT: s_mov_b32 s5, s1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_sub_f32_e32 v0, v2, v0
; GFX11-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
entry: entry:
%b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1 %b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1
%c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2 %c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2
%a = load float, ptr addrspace(1) %in %a = load float, ptr addrspace(1) %in
%b = load float, ptr addrspace(1) %b_ptr %b = load float, ptr addrspace(1) %b_ptr
%c = load float, ptr addrspace(1) %c_ptr %c = load float, ptr addrspace(1) %c_ptr
%neg_a = fneg float %a %neg_a = fneg float %a
%tmp0 = fmul float %neg_a, %b %tmp0 = fmul float %neg_a, %b
%tmp1 = fadd float %tmp0, %c %tmp1 = fadd float %tmp0, %c
store float %tmp1, ptr addrspace(1) %out store float %tmp1, ptr addrspace(1) %out
ret void ret void
} }
; GCN-LABEL: {{^}}nsz_mad_sub0_src0:
; GCN-NOT: v_mac_f32
; GCN: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[-0-9]}}
define amdgpu_kernel void @nsz_mad_sub0_src0(ptr addrspace(1) %out, ptr addrspace(1) %in) #1 { define amdgpu_kernel void @nsz_mad_sub0_src0(ptr addrspace(1) %out, ptr addrspace(1) %in) #1 {
; SI-LABEL: nsz_mad_sub0_src0:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s7, 0xf000
; SI-NEXT: s_mov_b32 s6, -1
; SI-NEXT: s_mov_b32 s10, s6
; SI-NEXT: s_mov_b32 s11, s7
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s8, s2
; SI-NEXT: s_mov_b32 s9, s3
; SI-NEXT: buffer_load_dwordx2 v[0:1], off, s[8:11], 0
; SI-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:8
; SI-NEXT: s_mov_b32 s4, s0
; SI-NEXT: s_mov_b32 s5, s1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_mad_f32 v0, -v0, v1, v2
; SI-NEXT: buffer_store_dword v0, off, s[4:7], 0
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: nsz_mad_sub0_src0:
; VI-FLUSH: ; %bb.0: ; %entry
; VI-FLUSH-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-FLUSH-NEXT: s_mov_b32 s7, 0xf000
; VI-FLUSH-NEXT: s_mov_b32 s6, -1
; VI-FLUSH-NEXT: s_mov_b32 s10, s6
; VI-FLUSH-NEXT: s_mov_b32 s11, s7
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s8, s2
; VI-FLUSH-NEXT: s_mov_b32 s9, s3
; VI-FLUSH-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-FLUSH-NEXT: s_mov_b32 s4, s0
; VI-FLUSH-NEXT: s_mov_b32 s5, s1
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: v_mad_f32 v0, -v0, v1, v2
; VI-FLUSH-NEXT: buffer_store_dword v0, off, s[4:7], 0
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: nsz_mad_sub0_src0:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s7, 0xf000
; VI-NEXT: s_mov_b32 s6, -1
; VI-NEXT: s_mov_b32 s10, s6
; VI-NEXT: s_mov_b32 s11, s7
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s8, s2
; VI-NEXT: s_mov_b32 s9, s3
; VI-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-NEXT: s_mov_b32 s4, s0
; VI-NEXT: s_mov_b32 s5, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f32 v0, -v0, v1, v2
; VI-NEXT: buffer_store_dword v0, off, s[4:7], 0
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: nsz_mad_sub0_src0:
; GFX11-FLUSH: ; %bb.0: ; %entry
; GFX11-FLUSH-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_mov_b32 s6, -1
; GFX11-FLUSH-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-FLUSH-NEXT: s_mov_b32 s10, s6
; GFX11-FLUSH-NEXT: s_mov_b32 s11, s7
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s8, s2
; GFX11-FLUSH-NEXT: s_mov_b32 s9, s3
; GFX11-FLUSH-NEXT: s_mov_b32 s4, s0
; GFX11-FLUSH-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-FLUSH-NEXT: s_mov_b32 s5, s1
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_sub_f32_e32 v0, v2, v0
; GFX11-FLUSH-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: nsz_mad_sub0_src0:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s6, -1
; GFX11-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-NEXT: s_mov_b32 s10, s6
; GFX11-NEXT: s_mov_b32 s11, s7
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s8, s2
; GFX11-NEXT: s_mov_b32 s9, s3
; GFX11-NEXT: s_mov_b32 s4, s0
; GFX11-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-NEXT: s_mov_b32 s5, s1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_sub_f32_e32 v0, v2, v0
; GFX11-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
entry: entry:
%b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1 %b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1
%c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2 %c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2
%a = load float, ptr addrspace(1) %in %a = load float, ptr addrspace(1) %in
%b = load float, ptr addrspace(1) %b_ptr %b = load float, ptr addrspace(1) %b_ptr
%c = load float, ptr addrspace(1) %c_ptr %c = load float, ptr addrspace(1) %c_ptr
%neg_a = fsub float 0.0, %a %neg_a = fsub float 0.0, %a
%tmp0 = fmul float %neg_a, %b %tmp0 = fmul float %neg_a, %b
%tmp1 = fadd float %tmp0, %c %tmp1 = fadd float %tmp0, %c
store float %tmp1, ptr addrspace(1) %out store float %tmp1, ptr addrspace(1) %out
ret void ret void
} }
; GCN-LABEL: {{^}}safe_mad_sub0_src0:
; GCN: v_sub_f32_e32 [[SUB0:v[0-9]+]], 0,
; GCN: v_ma{{[cd]}}_f32{{[_e32]*}} v{{[0-9]+}}, [[SUB0]], v{{[0-9]+}}
define amdgpu_kernel void @safe_mad_sub0_src0(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 { define amdgpu_kernel void @safe_mad_sub0_src0(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
; SI-LABEL: safe_mad_sub0_src0:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s7, 0xf000
; SI-NEXT: s_mov_b32 s6, -1
; SI-NEXT: s_mov_b32 s10, s6
; SI-NEXT: s_mov_b32 s11, s7
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s8, s2
; SI-NEXT: s_mov_b32 s9, s3
; SI-NEXT: buffer_load_dwordx2 v[0:1], off, s[8:11], 0
; SI-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:8
; SI-NEXT: s_mov_b32 s4, s0
; SI-NEXT: s_mov_b32 s5, s1
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_sub_f32_e32 v0, 0, v0
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_mac_f32_e32 v2, v0, v1
; SI-NEXT: buffer_store_dword v2, off, s[4:7], 0
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: safe_mad_sub0_src0:
; VI-FLUSH: ; %bb.0: ; %entry
; VI-FLUSH-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-FLUSH-NEXT: s_mov_b32 s7, 0xf000
; VI-FLUSH-NEXT: s_mov_b32 s6, -1
; VI-FLUSH-NEXT: s_mov_b32 s10, s6
; VI-FLUSH-NEXT: s_mov_b32 s11, s7
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s8, s2
; VI-FLUSH-NEXT: s_mov_b32 s9, s3
; VI-FLUSH-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-FLUSH-NEXT: s_mov_b32 s4, s0
; VI-FLUSH-NEXT: s_mov_b32 s5, s1
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: v_sub_f32_e32 v0, 0, v0
; VI-FLUSH-NEXT: v_mac_f32_e32 v2, v0, v1
; VI-FLUSH-NEXT: buffer_store_dword v2, off, s[4:7], 0
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: safe_mad_sub0_src0:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s7, 0xf000
; VI-NEXT: s_mov_b32 s6, -1
; VI-NEXT: s_mov_b32 s10, s6
; VI-NEXT: s_mov_b32 s11, s7
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s8, s2
; VI-NEXT: s_mov_b32 s9, s3
; VI-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-NEXT: s_mov_b32 s4, s0
; VI-NEXT: s_mov_b32 s5, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_sub_f32_e32 v0, 0, v0
; VI-NEXT: v_mac_f32_e32 v2, v0, v1
; VI-NEXT: buffer_store_dword v2, off, s[4:7], 0
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: safe_mad_sub0_src0:
; GFX11-FLUSH: ; %bb.0: ; %entry
; GFX11-FLUSH-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_mov_b32 s6, -1
; GFX11-FLUSH-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-FLUSH-NEXT: s_mov_b32 s10, s6
; GFX11-FLUSH-NEXT: s_mov_b32 s11, s7
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s8, s2
; GFX11-FLUSH-NEXT: s_mov_b32 s9, s3
; GFX11-FLUSH-NEXT: s_mov_b32 s4, s0
; GFX11-FLUSH-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-FLUSH-NEXT: s_mov_b32 s5, s1
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: v_sub_f32_e32 v0, 0, v0
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-FLUSH-NEXT: v_add_f32_e32 v0, v0, v2
; GFX11-FLUSH-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: safe_mad_sub0_src0:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s6, -1
; GFX11-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-NEXT: s_mov_b32 s10, s6
; GFX11-NEXT: s_mov_b32 s11, s7
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s8, s2
; GFX11-NEXT: s_mov_b32 s9, s3
; GFX11-NEXT: s_mov_b32 s4, s0
; GFX11-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-NEXT: s_mov_b32 s5, s1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_sub_f32_e32 v0, 0, v0
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX11-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-NEXT: v_add_f32_e32 v0, v0, v2
; GFX11-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
entry: entry:
%b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1 %b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1
%c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2 %c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2
%a = load float, ptr addrspace(1) %in %a = load float, ptr addrspace(1) %in
%b = load float, ptr addrspace(1) %b_ptr %b = load float, ptr addrspace(1) %b_ptr
%c = load float, ptr addrspace(1) %c_ptr %c = load float, ptr addrspace(1) %c_ptr
%neg_a = fsub float 0.0, %a %neg_a = fsub float 0.0, %a
%tmp0 = fmul float %neg_a, %b %tmp0 = fmul float %neg_a, %b
%tmp1 = fadd float %tmp0, %c %tmp1 = fadd float %tmp0, %c
store float %tmp1, ptr addrspace(1) %out store float %tmp1, ptr addrspace(1) %out
ret void ret void
} }
; GCN-LABEL: {{^}}mad_neg_src1:
; GCN-NOT: v_mac_f32
; GCN: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[-0-9]}}
define amdgpu_kernel void @mad_neg_src1(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 { define amdgpu_kernel void @mad_neg_src1(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
; SI-LABEL: mad_neg_src1:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s7, 0xf000
; SI-NEXT: s_mov_b32 s6, -1
; SI-NEXT: s_mov_b32 s10, s6
; SI-NEXT: s_mov_b32 s11, s7
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s8, s2
; SI-NEXT: s_mov_b32 s9, s3
; SI-NEXT: buffer_load_dwordx2 v[0:1], off, s[8:11], 0
; SI-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:8
; SI-NEXT: s_mov_b32 s4, s0
; SI-NEXT: s_mov_b32 s5, s1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_mad_f32 v0, -v0, v1, v2
; SI-NEXT: buffer_store_dword v0, off, s[4:7], 0
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: mad_neg_src1:
; VI-FLUSH: ; %bb.0: ; %entry
; VI-FLUSH-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-FLUSH-NEXT: s_mov_b32 s7, 0xf000
; VI-FLUSH-NEXT: s_mov_b32 s6, -1
; VI-FLUSH-NEXT: s_mov_b32 s10, s6
; VI-FLUSH-NEXT: s_mov_b32 s11, s7
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s8, s2
; VI-FLUSH-NEXT: s_mov_b32 s9, s3
; VI-FLUSH-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-FLUSH-NEXT: s_mov_b32 s4, s0
; VI-FLUSH-NEXT: s_mov_b32 s5, s1
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: v_mad_f32 v0, -v0, v1, v2
; VI-FLUSH-NEXT: buffer_store_dword v0, off, s[4:7], 0
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: mad_neg_src1:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s7, 0xf000
; VI-NEXT: s_mov_b32 s6, -1
; VI-NEXT: s_mov_b32 s10, s6
; VI-NEXT: s_mov_b32 s11, s7
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s8, s2
; VI-NEXT: s_mov_b32 s9, s3
; VI-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-NEXT: s_mov_b32 s4, s0
; VI-NEXT: s_mov_b32 s5, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f32 v0, -v0, v1, v2
; VI-NEXT: buffer_store_dword v0, off, s[4:7], 0
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: mad_neg_src1:
; GFX11-FLUSH: ; %bb.0: ; %entry
; GFX11-FLUSH-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_mov_b32 s6, -1
; GFX11-FLUSH-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-FLUSH-NEXT: s_mov_b32 s10, s6
; GFX11-FLUSH-NEXT: s_mov_b32 s11, s7
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s8, s2
; GFX11-FLUSH-NEXT: s_mov_b32 s9, s3
; GFX11-FLUSH-NEXT: s_mov_b32 s4, s0
; GFX11-FLUSH-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-FLUSH-NEXT: s_mov_b32 s5, s1
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_sub_f32_e32 v0, v2, v0
; GFX11-FLUSH-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: mad_neg_src1:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s6, -1
; GFX11-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-NEXT: s_mov_b32 s10, s6
; GFX11-NEXT: s_mov_b32 s11, s7
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s8, s2
; GFX11-NEXT: s_mov_b32 s9, s3
; GFX11-NEXT: s_mov_b32 s4, s0
; GFX11-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-NEXT: s_mov_b32 s5, s1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_sub_f32_e32 v0, v2, v0
; GFX11-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
entry: entry:
%b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1 %b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1
%c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2 %c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2
%a = load float, ptr addrspace(1) %in %a = load float, ptr addrspace(1) %in
%b = load float, ptr addrspace(1) %b_ptr %b = load float, ptr addrspace(1) %b_ptr
%c = load float, ptr addrspace(1) %c_ptr %c = load float, ptr addrspace(1) %c_ptr
%neg_b = fneg float %b %neg_b = fneg float %b
%tmp0 = fmul float %a, %neg_b %tmp0 = fmul float %a, %neg_b
%tmp1 = fadd float %tmp0, %c %tmp1 = fadd float %tmp0, %c
store float %tmp1, ptr addrspace(1) %out store float %tmp1, ptr addrspace(1) %out
ret void ret void
} }
; GCN-LABEL: {{^}}nsz_mad_sub0_src1:
; GCN-NOT: v_mac_f32
; GCN: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[-0-9]}}
define amdgpu_kernel void @nsz_mad_sub0_src1(ptr addrspace(1) %out, ptr addrspace(1) %in) #1 { define amdgpu_kernel void @nsz_mad_sub0_src1(ptr addrspace(1) %out, ptr addrspace(1) %in) #1 {
; SI-LABEL: nsz_mad_sub0_src1:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s7, 0xf000
; SI-NEXT: s_mov_b32 s6, -1
; SI-NEXT: s_mov_b32 s10, s6
; SI-NEXT: s_mov_b32 s11, s7
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s8, s2
; SI-NEXT: s_mov_b32 s9, s3
; SI-NEXT: buffer_load_dwordx2 v[0:1], off, s[8:11], 0
; SI-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:8
; SI-NEXT: s_mov_b32 s4, s0
; SI-NEXT: s_mov_b32 s5, s1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_mad_f32 v0, -v0, v1, v2
; SI-NEXT: buffer_store_dword v0, off, s[4:7], 0
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: nsz_mad_sub0_src1:
; VI-FLUSH: ; %bb.0: ; %entry
; VI-FLUSH-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-FLUSH-NEXT: s_mov_b32 s7, 0xf000
; VI-FLUSH-NEXT: s_mov_b32 s6, -1
; VI-FLUSH-NEXT: s_mov_b32 s10, s6
; VI-FLUSH-NEXT: s_mov_b32 s11, s7
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s8, s2
; VI-FLUSH-NEXT: s_mov_b32 s9, s3
; VI-FLUSH-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-FLUSH-NEXT: s_mov_b32 s4, s0
; VI-FLUSH-NEXT: s_mov_b32 s5, s1
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: v_mad_f32 v0, -v0, v1, v2
; VI-FLUSH-NEXT: buffer_store_dword v0, off, s[4:7], 0
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: nsz_mad_sub0_src1:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s7, 0xf000
; VI-NEXT: s_mov_b32 s6, -1
; VI-NEXT: s_mov_b32 s10, s6
; VI-NEXT: s_mov_b32 s11, s7
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s8, s2
; VI-NEXT: s_mov_b32 s9, s3
; VI-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-NEXT: s_mov_b32 s4, s0
; VI-NEXT: s_mov_b32 s5, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f32 v0, -v0, v1, v2
; VI-NEXT: buffer_store_dword v0, off, s[4:7], 0
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: nsz_mad_sub0_src1:
; GFX11-FLUSH: ; %bb.0: ; %entry
; GFX11-FLUSH-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_mov_b32 s6, -1
; GFX11-FLUSH-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-FLUSH-NEXT: s_mov_b32 s10, s6
; GFX11-FLUSH-NEXT: s_mov_b32 s11, s7
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s8, s2
; GFX11-FLUSH-NEXT: s_mov_b32 s9, s3
; GFX11-FLUSH-NEXT: s_mov_b32 s4, s0
; GFX11-FLUSH-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-FLUSH-NEXT: s_mov_b32 s5, s1
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_sub_f32_e32 v0, v2, v0
; GFX11-FLUSH-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: nsz_mad_sub0_src1:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s6, -1
; GFX11-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-NEXT: s_mov_b32 s10, s6
; GFX11-NEXT: s_mov_b32 s11, s7
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s8, s2
; GFX11-NEXT: s_mov_b32 s9, s3
; GFX11-NEXT: s_mov_b32 s4, s0
; GFX11-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-NEXT: s_mov_b32 s5, s1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_sub_f32_e32 v0, v2, v0
; GFX11-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
entry: entry:
%b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1 %b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1
%c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2 %c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2
%a = load float, ptr addrspace(1) %in %a = load float, ptr addrspace(1) %in
%b = load float, ptr addrspace(1) %b_ptr %b = load float, ptr addrspace(1) %b_ptr
%c = load float, ptr addrspace(1) %c_ptr %c = load float, ptr addrspace(1) %c_ptr
%neg_b = fsub float 0.0, %b %neg_b = fsub float 0.0, %b
%tmp0 = fmul float %a, %neg_b %tmp0 = fmul float %a, %neg_b
%tmp1 = fadd float %tmp0, %c %tmp1 = fadd float %tmp0, %c
store float %tmp1, ptr addrspace(1) %out store float %tmp1, ptr addrspace(1) %out
ret void ret void
} }
; GCN-LABEL: {{^}}mad_neg_src2:
; GCN-NOT: v_mac
; GCN: v_mad_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[-0-9]}}
define amdgpu_kernel void @mad_neg_src2(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 { define amdgpu_kernel void @mad_neg_src2(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
; SI-LABEL: mad_neg_src2:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s7, 0xf000
; SI-NEXT: s_mov_b32 s6, -1
; SI-NEXT: s_mov_b32 s10, s6
; SI-NEXT: s_mov_b32 s11, s7
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s8, s2
; SI-NEXT: s_mov_b32 s9, s3
; SI-NEXT: buffer_load_dwordx2 v[0:1], off, s[8:11], 0
; SI-NEXT: buffer_load_dword v2, off, s[8:11], 0 offset:8
; SI-NEXT: s_mov_b32 s4, s0
; SI-NEXT: s_mov_b32 s5, s1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_mad_f32 v0, v0, v1, -v2
; SI-NEXT: buffer_store_dword v0, off, s[4:7], 0
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: mad_neg_src2:
; VI-FLUSH: ; %bb.0: ; %entry
; VI-FLUSH-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-FLUSH-NEXT: s_mov_b32 s7, 0xf000
; VI-FLUSH-NEXT: s_mov_b32 s6, -1
; VI-FLUSH-NEXT: s_mov_b32 s10, s6
; VI-FLUSH-NEXT: s_mov_b32 s11, s7
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: s_mov_b32 s8, s2
; VI-FLUSH-NEXT: s_mov_b32 s9, s3
; VI-FLUSH-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-FLUSH-NEXT: s_mov_b32 s4, s0
; VI-FLUSH-NEXT: s_mov_b32 s5, s1
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: v_mad_f32 v0, v0, v1, -v2
; VI-FLUSH-NEXT: buffer_store_dword v0, off, s[4:7], 0
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: mad_neg_src2:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s7, 0xf000
; VI-NEXT: s_mov_b32 s6, -1
; VI-NEXT: s_mov_b32 s10, s6
; VI-NEXT: s_mov_b32 s11, s7
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s8, s2
; VI-NEXT: s_mov_b32 s9, s3
; VI-NEXT: buffer_load_dwordx3 v[0:2], off, s[8:11], 0
; VI-NEXT: s_mov_b32 s4, s0
; VI-NEXT: s_mov_b32 s5, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f32 v0, v0, v1, -v2
; VI-NEXT: buffer_store_dword v0, off, s[4:7], 0
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: mad_neg_src2:
; GFX11-FLUSH: ; %bb.0: ; %entry
; GFX11-FLUSH-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_mov_b32 s6, -1
; GFX11-FLUSH-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-FLUSH-NEXT: s_mov_b32 s10, s6
; GFX11-FLUSH-NEXT: s_mov_b32 s11, s7
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: s_mov_b32 s8, s2
; GFX11-FLUSH-NEXT: s_mov_b32 s9, s3
; GFX11-FLUSH-NEXT: s_mov_b32 s4, s0
; GFX11-FLUSH-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-FLUSH-NEXT: s_mov_b32 s5, s1
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_sub_f32_e32 v0, v0, v2
; GFX11-FLUSH-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: mad_neg_src2:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s6, -1
; GFX11-NEXT: s_mov_b32 s7, 0x31016000
; GFX11-NEXT: s_mov_b32 s10, s6
; GFX11-NEXT: s_mov_b32 s11, s7
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s8, s2
; GFX11-NEXT: s_mov_b32 s9, s3
; GFX11-NEXT: s_mov_b32 s4, s0
; GFX11-NEXT: buffer_load_b96 v[0:2], off, s[8:11], 0
; GFX11-NEXT: s_mov_b32 s5, s1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_mul_f32_e32 v0, v0, v1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_sub_f32_e32 v0, v0, v2
; GFX11-NEXT: buffer_store_b32 v0, off, s[4:7], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
entry: entry:
%b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1 %b_ptr = getelementptr float, ptr addrspace(1) %in, i32 1
%c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2 %c_ptr = getelementptr float, ptr addrspace(1) %in, i32 2
%a = load float, ptr addrspace(1) %in %a = load float, ptr addrspace(1) %in
%b = load float, ptr addrspace(1) %b_ptr %b = load float, ptr addrspace(1) %b_ptr
%c = load float, ptr addrspace(1) %c_ptr %c = load float, ptr addrspace(1) %c_ptr
%neg_c = fneg float %c %neg_c = fneg float %c
%tmp0 = fmul float %a, %b %tmp0 = fmul float %a, %b
%tmp1 = fadd float %tmp0, %neg_c %tmp1 = fadd float %tmp0, %neg_c
store float %tmp1, ptr addrspace(1) %out store float %tmp1, ptr addrspace(1) %out
ret void ret void
} }
; Without special casing the inline constant check for v_mac_f32's ; Without special casing the inline constant check for v_mac_f32's
; src2, this fails to fold the 1.0 into a mad. ; src2, this fails to fold the 1.0 into a mad.
; GCN-LABEL: {{^}}fold_inline_imm_into_mac_src2_f32:
; GCN: {{buffer|flat}}_load_dword [[A:v[0-9]+]]
; GCN: {{buffer|flat}}_load_dword [[B:v[0-9]+]]
; GCN: v_add_f32_e32 [[TMP2:v[0-9]+]], [[A]], [[A]]
; GCN: v_mad_f32 v{{[0-9]+}}, [[TMP2]], -4.0, 1.0
define amdgpu_kernel void @fold_inline_imm_into_mac_src2_f32(ptr addrspace(1) %out, ptr addrspace(1) %a, ptr addrspace(1) %b) #3 { define amdgpu_kernel void @fold_inline_imm_into_mac_src2_f32(ptr addrspace(1) %out, ptr addrspace(1) %a, ptr addrspace(1) %b) #3 {
; SI-LABEL: fold_inline_imm_into_mac_src2_f32:
; SI: ; %bb.0: ; %bb
; SI-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x9
; SI-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0xd
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, 0
; SI-NEXT: v_lshlrev_b32_e32 v0, 2, v0
; SI-NEXT: v_mov_b32_e32 v1, 0
; SI-NEXT: s_mov_b64 s[2:3], s[10:11]
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b64 s[8:9], s[6:7]
; SI-NEXT: buffer_load_dword v2, v[0:1], s[8:11], 0 addr64 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: buffer_load_dword v3, v[0:1], s[0:3], 0 addr64 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: s_mov_b64 s[6:7], s[10:11]
; SI-NEXT: v_add_f32_e32 v4, v2, v2
; SI-NEXT: v_add_f32_e32 v5, v3, v3
; SI-NEXT: v_mad_f32 v4, v4, -4.0, 1.0
; SI-NEXT: v_add_f32_e32 v3, v4, v3
; SI-NEXT: v_mad_f32 v2, -v5, v2, 1.0
; SI-NEXT: v_mac_f32_e32 v3, 0x41000000, v2
; SI-NEXT: buffer_store_dword v3, v[0:1], s[4:7], 0 addr64
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: fold_inline_imm_into_mac_src2_f32:
; VI-FLUSH: ; %bb.0: ; %bb
; VI-FLUSH-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x24
; VI-FLUSH-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x34
; VI-FLUSH-NEXT: v_lshlrev_b32_e32 v4, 2, v0
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: v_mov_b32_e32 v1, s7
; VI-FLUSH-NEXT: v_add_u32_e32 v0, vcc, s6, v4
; VI-FLUSH-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-FLUSH-NEXT: v_mov_b32_e32 v3, s1
; VI-FLUSH-NEXT: v_add_u32_e32 v2, vcc, s0, v4
; VI-FLUSH-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc
; VI-FLUSH-NEXT: flat_load_dword v5, v[0:1] glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: flat_load_dword v2, v[2:3] glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: v_add_u32_e32 v0, vcc, s4, v4
; VI-FLUSH-NEXT: v_mov_b32_e32 v1, s5
; VI-FLUSH-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-FLUSH-NEXT: v_add_f32_e32 v3, v5, v5
; VI-FLUSH-NEXT: v_add_f32_e32 v4, v2, v2
; VI-FLUSH-NEXT: v_mad_f32 v3, v3, -4.0, 1.0
; VI-FLUSH-NEXT: v_add_f32_e32 v2, v3, v2
; VI-FLUSH-NEXT: v_mad_f32 v3, -v4, v5, 1.0
; VI-FLUSH-NEXT: v_mac_f32_e32 v2, 0x41000000, v3
; VI-FLUSH-NEXT: flat_store_dword v[0:1], v2
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: fold_inline_imm_into_mac_src2_f32:
; VI: ; %bb.0: ; %bb
; VI-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x24
; VI-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x34
; VI-NEXT: v_lshlrev_b32_e32 v4, 2, v0
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: v_mov_b32_e32 v1, s7
; VI-NEXT: v_add_u32_e32 v0, vcc, s6, v4
; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-NEXT: v_mov_b32_e32 v3, s1
; VI-NEXT: v_add_u32_e32 v2, vcc, s0, v4
; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc
; VI-NEXT: flat_load_dword v5, v[0:1] glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: flat_load_dword v2, v[2:3] glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_add_u32_e32 v0, vcc, s4, v4
; VI-NEXT: v_mov_b32_e32 v1, s5
; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-NEXT: v_add_f32_e32 v3, v5, v5
; VI-NEXT: v_add_f32_e32 v4, v2, v2
; VI-NEXT: v_mad_f32 v3, v3, -4.0, 1.0
; VI-NEXT: v_add_f32_e32 v2, v3, v2
; VI-NEXT: v_mad_f32 v3, -v4, v5, 1.0
; VI-NEXT: v_mac_f32_e32 v2, 0x41000000, v3
; VI-NEXT: flat_store_dword v[0:1], v2
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: fold_inline_imm_into_mac_src2_f32:
; GFX11-FLUSH: ; %bb.0: ; %bb
; GFX11-FLUSH-NEXT: s_clause 0x1
; GFX11-FLUSH-NEXT: s_load_b128 s[4:7], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_load_b64 s[0:1], s[0:1], 0x34
; GFX11-FLUSH-NEXT: v_lshlrev_b32_e32 v0, 2, v0
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: global_load_b32 v1, v0, s[6:7] glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: global_load_b32 v2, v0, s[0:1] glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: v_dual_add_f32 v3, v1, v1 :: v_dual_add_f32 v4, v2, v2
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_mul_f32_e32 v3, -4.0, v3
; GFX11-FLUSH-NEXT: v_add_f32_e32 v3, 1.0, v3
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_dual_mul_f32 v1, v4, v1 :: v_dual_add_f32 v2, v3, v2
; GFX11-FLUSH-NEXT: v_sub_f32_e32 v1, 1.0, v1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_mul_f32_e32 v1, 0x41000000, v1
; GFX11-FLUSH-NEXT: v_add_f32_e32 v1, v2, v1
; GFX11-FLUSH-NEXT: global_store_b32 v0, v1, s[4:5]
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: fold_inline_imm_into_mac_src2_f32:
; GFX11: ; %bb.0: ; %bb
; GFX11-NEXT: s_clause 0x1
; GFX11-NEXT: s_load_b128 s[4:7], s[0:1], 0x24
; GFX11-NEXT: s_load_b64 s[0:1], s[0:1], 0x34
; GFX11-NEXT: v_lshlrev_b32_e32 v0, 2, v0
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: global_load_b32 v1, v0, s[6:7] glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: global_load_b32 v2, v0, s[0:1] glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_dual_add_f32 v3, v1, v1 :: v_dual_add_f32 v4, v2, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX11-NEXT: v_mul_f32_e32 v3, -4.0, v3
; GFX11-NEXT: v_add_f32_e32 v3, 1.0, v3
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX11-NEXT: v_dual_mul_f32 v1, v4, v1 :: v_dual_add_f32 v2, v3, v2
; GFX11-NEXT: v_sub_f32_e32 v1, 1.0, v1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX11-NEXT: v_mul_f32_e32 v1, 0x41000000, v1
; GFX11-NEXT: v_add_f32_e32 v1, v2, v1
; GFX11-NEXT: global_store_b32 v0, v1, s[4:5]
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
bb: bb:
%tid = call i32 @llvm.amdgcn.workitem.id.x() %tid = call i32 @llvm.amdgcn.workitem.id.x()
%tid.ext = sext i32 %tid to i64 %tid.ext = sext i32 %tid to i64
%gep.a = getelementptr inbounds float, ptr addrspace(1) %a, i64 %tid.ext %gep.a = getelementptr inbounds float, ptr addrspace(1) %a, i64 %tid.ext
%gep.b = getelementptr inbounds float, ptr addrspace(1) %b, i64 %tid.ext %gep.b = getelementptr inbounds float, ptr addrspace(1) %b, i64 %tid.ext
%gep.out = getelementptr inbounds float, ptr addrspace(1) %out, i64 %tid.ext %gep.out = getelementptr inbounds float, ptr addrspace(1) %out, i64 %tid.ext
%tmp = load volatile float, ptr addrspace(1) %gep.a %tmp = load volatile float, ptr addrspace(1) %gep.a
%tmp1 = load volatile float, ptr addrspace(1) %gep.b %tmp1 = load volatile float, ptr addrspace(1) %gep.b
%tmp2 = fadd float %tmp, %tmp %tmp2 = fadd float %tmp, %tmp
%tmp3 = fmul float %tmp2, 4.0 %tmp3 = fmul float %tmp2, 4.0
%tmp4 = fsub float 1.0, %tmp3 %tmp4 = fsub float 1.0, %tmp3
%tmp5 = fadd float %tmp4, %tmp1 %tmp5 = fadd float %tmp4, %tmp1
%tmp6 = fadd float %tmp1, %tmp1 %tmp6 = fadd float %tmp1, %tmp1
%tmp7 = fmul float %tmp6, %tmp %tmp7 = fmul float %tmp6, %tmp
%tmp8 = fsub float 1.0, %tmp7 %tmp8 = fsub float 1.0, %tmp7
%tmp9 = fmul float %tmp8, 8.0 %tmp9 = fmul float %tmp8, 8.0
%tmp10 = fadd float %tmp5, %tmp9 %tmp10 = fadd float %tmp5, %tmp9
store float %tmp10, ptr addrspace(1) %gep.out store float %tmp10, ptr addrspace(1) %gep.out
ret void ret void
} }
; GCN-LABEL: {{^}}fold_inline_imm_into_mac_src2_f16:
; GCN: {{buffer|flat}}_load_ushort [[A:v[0-9]+]]
; GCN: {{buffer|flat}}_load_ushort [[B:v[0-9]+]]
; SI-DAG: v_cvt_f32_f16_e32 [[CVT_A:v[0-9]+]], [[A]]
; SI-DAG: v_cvt_f32_f16_e32 [[CVT_B:v[0-9]+]], [[B]]
; SI: v_add_f32_e32 [[TMP2:v[0-9]+]], [[CVT_A]], [[CVT_A]]
; SI: v_mad_f32 v{{[0-9]+}}, [[TMP2]], -4.0, 1.0
; SI: v_mac_f32_e32 v{{[0-9]+}}, 0x41000000, v{{[0-9]+}}
; VI-FLUSH: v_add_f16_e32 [[TMP2:v[0-9]+]], [[A]], [[A]]
; VI-FLUSH: v_mad_f16 v{{[0-9]+}}, [[TMP2]], -4.0, 1.0
define amdgpu_kernel void @fold_inline_imm_into_mac_src2_f16(ptr addrspace(1) %out, ptr addrspace(1) %a, ptr addrspace(1) %b) #3 { define amdgpu_kernel void @fold_inline_imm_into_mac_src2_f16(ptr addrspace(1) %out, ptr addrspace(1) %a, ptr addrspace(1) %b) #3 {
; SI-LABEL: fold_inline_imm_into_mac_src2_f16:
; SI: ; %bb.0: ; %bb
; SI-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x9
; SI-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0xd
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, 0
; SI-NEXT: v_lshlrev_b32_e32 v0, 1, v0
; SI-NEXT: v_mov_b32_e32 v1, 0
; SI-NEXT: s_mov_b64 s[2:3], s[10:11]
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b64 s[8:9], s[6:7]
; SI-NEXT: buffer_load_ushort v2, v[0:1], s[8:11], 0 addr64 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: buffer_load_ushort v3, v[0:1], s[0:3], 0 addr64 glc
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: s_mov_b64 s[6:7], s[10:11]
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_cvt_f32_f16_e32 v3, v3
; SI-NEXT: v_add_f32_e32 v4, v2, v2
; SI-NEXT: v_add_f32_e32 v5, v3, v3
; SI-NEXT: v_mad_f32 v4, v4, -4.0, 1.0
; SI-NEXT: v_add_f32_e32 v3, v4, v3
; SI-NEXT: v_mad_f32 v2, -v5, v2, 1.0
; SI-NEXT: v_mac_f32_e32 v3, 0x41000000, v2
; SI-NEXT: v_cvt_f16_f32_e32 v2, v3
; SI-NEXT: buffer_store_short v2, v[0:1], s[4:7], 0 addr64
; SI-NEXT: s_endpgm
;
; VI-FLUSH-LABEL: fold_inline_imm_into_mac_src2_f16:
; VI-FLUSH: ; %bb.0: ; %bb
; VI-FLUSH-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x24
; VI-FLUSH-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x34
; VI-FLUSH-NEXT: v_lshlrev_b32_e32 v4, 1, v0
; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; VI-FLUSH-NEXT: v_mov_b32_e32 v1, s7
; VI-FLUSH-NEXT: v_add_u32_e32 v0, vcc, s6, v4
; VI-FLUSH-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-FLUSH-NEXT: v_mov_b32_e32 v3, s1
; VI-FLUSH-NEXT: v_add_u32_e32 v2, vcc, s0, v4
; VI-FLUSH-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc
; VI-FLUSH-NEXT: flat_load_ushort v5, v[0:1] glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: flat_load_ushort v2, v[2:3] glc
; VI-FLUSH-NEXT: s_waitcnt vmcnt(0)
; VI-FLUSH-NEXT: v_add_u32_e32 v0, vcc, s4, v4
; VI-FLUSH-NEXT: v_mov_b32_e32 v1, s5
; VI-FLUSH-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-FLUSH-NEXT: v_add_f16_e32 v3, v5, v5
; VI-FLUSH-NEXT: v_add_f16_e32 v4, v2, v2
; VI-FLUSH-NEXT: v_mad_f16 v3, v3, -4.0, 1.0
; VI-FLUSH-NEXT: v_add_f16_e32 v2, v3, v2
; VI-FLUSH-NEXT: v_mad_f16 v3, -v4, v5, 1.0
; VI-FLUSH-NEXT: v_mac_f16_e32 v2, 0x4800, v3
; VI-FLUSH-NEXT: flat_store_short v[0:1], v2
; VI-FLUSH-NEXT: s_endpgm
;
; VI-LABEL: fold_inline_imm_into_mac_src2_f16:
; VI: ; %bb.0: ; %bb
; VI-NEXT: s_load_dwordx4 s[4:7], s[0:1], 0x24
; VI-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x34
; VI-NEXT: v_lshlrev_b32_e32 v4, 1, v0
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: v_mov_b32_e32 v1, s7
; VI-NEXT: v_add_u32_e32 v0, vcc, s6, v4
; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-NEXT: v_mov_b32_e32 v3, s1
; VI-NEXT: v_add_u32_e32 v2, vcc, s0, v4
; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc
; VI-NEXT: flat_load_ushort v5, v[0:1] glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: flat_load_ushort v2, v[2:3] glc
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_add_u32_e32 v0, vcc, s4, v4
; VI-NEXT: v_mov_b32_e32 v1, s5
; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-NEXT: v_add_f16_e32 v3, v5, v5
; VI-NEXT: v_add_f16_e32 v4, v2, v2
; VI-NEXT: v_mul_f16_e32 v3, -4.0, v3
; VI-NEXT: v_mul_f16_e32 v4, v4, v5
; VI-NEXT: v_add_f16_e32 v3, 1.0, v3
; VI-NEXT: v_sub_f16_e32 v4, 1.0, v4
; VI-NEXT: v_add_f16_e32 v2, v3, v2
; VI-NEXT: v_mul_f16_e32 v3, 0x4800, v4
; VI-NEXT: v_add_f16_e32 v2, v2, v3
; VI-NEXT: flat_store_short v[0:1], v2
; VI-NEXT: s_endpgm
;
; GFX11-FLUSH-LABEL: fold_inline_imm_into_mac_src2_f16:
; GFX11-FLUSH: ; %bb.0: ; %bb
; GFX11-FLUSH-NEXT: s_clause 0x1
; GFX11-FLUSH-NEXT: s_load_b128 s[4:7], s[0:1], 0x24
; GFX11-FLUSH-NEXT: s_load_b64 s[0:1], s[0:1], 0x34
; GFX11-FLUSH-NEXT: v_lshlrev_b32_e32 v0, 1, v0
; GFX11-FLUSH-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-FLUSH-NEXT: global_load_u16 v1, v0, s[6:7] glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: global_load_u16 v2, v0, s[0:1] glc dlc
; GFX11-FLUSH-NEXT: s_waitcnt vmcnt(0)
; GFX11-FLUSH-NEXT: v_add_f16_e32 v3, v1, v1
; GFX11-FLUSH-NEXT: v_add_f16_e32 v4, v2, v2
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX11-FLUSH-NEXT: v_mul_f16_e32 v3, -4.0, v3
; GFX11-FLUSH-NEXT: v_mul_f16_e32 v1, v4, v1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX11-FLUSH-NEXT: v_add_f16_e32 v3, 1.0, v3
; GFX11-FLUSH-NEXT: v_sub_f16_e32 v1, 1.0, v1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX11-FLUSH-NEXT: v_add_f16_e32 v2, v3, v2
; GFX11-FLUSH-NEXT: v_mul_f16_e32 v1, 0x4800, v1
; GFX11-FLUSH-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-FLUSH-NEXT: v_add_f16_e32 v1, v2, v1
; GFX11-FLUSH-NEXT: global_store_b16 v0, v1, s[4:5]
; GFX11-FLUSH-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-FLUSH-NEXT: s_endpgm
;
; GFX11-LABEL: fold_inline_imm_into_mac_src2_f16:
; GFX11: ; %bb.0: ; %bb
; GFX11-NEXT: s_clause 0x1
; GFX11-NEXT: s_load_b128 s[4:7], s[0:1], 0x24
; GFX11-NEXT: s_load_b64 s[0:1], s[0:1], 0x34
; GFX11-NEXT: v_lshlrev_b32_e32 v0, 1, v0
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: global_load_u16 v1, v0, s[6:7] glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: global_load_u16 v2, v0, s[0:1] glc dlc
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_add_f16_e32 v3, v1, v1
; GFX11-NEXT: v_add_f16_e32 v4, v2, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX11-NEXT: v_mul_f16_e32 v3, -4.0, v3
; GFX11-NEXT: v_mul_f16_e32 v1, v4, v1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX11-NEXT: v_add_f16_e32 v3, 1.0, v3
; GFX11-NEXT: v_sub_f16_e32 v1, 1.0, v1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX11-NEXT: v_add_f16_e32 v2, v3, v2
; GFX11-NEXT: v_mul_f16_e32 v1, 0x4800, v1
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_add_f16_e32 v1, v2, v1
; GFX11-NEXT: global_store_b16 v0, v1, s[4:5]
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
bb: bb:
%tid = call i32 @llvm.amdgcn.workitem.id.x() %tid = call i32 @llvm.amdgcn.workitem.id.x()
%tid.ext = sext i32 %tid to i64 %tid.ext = sext i32 %tid to i64
%gep.a = getelementptr inbounds half, ptr addrspace(1) %a, i64 %tid.ext %gep.a = getelementptr inbounds half, ptr addrspace(1) %a, i64 %tid.ext
%gep.b = getelementptr inbounds half, ptr addrspace(1) %b, i64 %tid.ext %gep.b = getelementptr inbounds half, ptr addrspace(1) %b, i64 %tid.ext
%gep.out = getelementptr inbounds half, ptr addrspace(1) %out, i64 %tid.ext %gep.out = getelementptr inbounds half, ptr addrspace(1) %out, i64 %tid.ext
%tmp = load volatile half, ptr addrspace(1) %gep.a %tmp = load volatile half, ptr addrspace(1) %gep.a
%tmp1 = load volatile half, ptr addrspace(1) %gep.b %tmp1 = load volatile half, ptr addrspace(1) %gep.b
Show All 19 Lines

llvm/test/CodeGen/AMDGPU/v_mac_f16.ll

; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=tahiti -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefix=GCN -check-prefix=SI %s ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=fiji -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefix=GCN -check-prefix=VI %s ; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=tahiti -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefixes=SI %s
; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=fiji -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefixes=VI %s
; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=gfx1100 -mattr=-flat-for-global -verify-machineinstrs < %s | FileCheck -allow-deprecated-dag-overlap -check-prefixes=GFX11 %s
; GCN-LABEL: {{^}}mac_f16:
; GCN: {{buffer|flat}}_load_ushort v[[A_F16:[0-9]+]]
; GCN: {{buffer|flat}}_load_ushort v[[B_F16:[0-9]+]]
; GCN: {{buffer|flat}}_load_ushort v[[C_F16:[0-9]+]]
; SI: v_cvt_f32_f16_e32 v[[A_F32:[0-9]+]], v[[A_F16]]
; SI: v_cvt_f32_f16_e32 v[[B_F32:[0-9]+]], v[[B_F16]]
; SI: v_cvt_f32_f16_e32 v[[C_F32:[0-9]+]], v[[C_F16]]
; SI: v_mac_f32_e32 v[[C_F32]], v[[A_F32]], v[[B_F32]]
; SI: v_cvt_f16_f32_e32 v[[R_F16:[0-9]+]], v[[C_F32]]
; SI: buffer_store_short v[[R_F16]]
; VI: v_mac_f16_e32 v[[C_F16]], v[[A_F16]], v[[B_F16]]
; VI: buffer_store_short v[[C_F16]]
; GCN: s_endpgm
define amdgpu_kernel void @mac_f16( define amdgpu_kernel void @mac_f16(
; SI-LABEL: mac_f16:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; SI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; SI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mac_f32_e32 v2, v0, v1
; SI-NEXT: v_cvt_f16_f32_e32 v0, v2
; SI-NEXT: buffer_store_short v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_f16:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; VI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; VI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mac_f16_e32 v2, v0, v1
; VI-NEXT: buffer_store_short v2, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_f16:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_u16 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_u16 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_u16 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_mul_f16_e32 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_add_f16_e32 v0, v0, v2
; GFX11-NEXT: buffer_store_b16 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load half, ptr addrspace(1) %a %a.val = load half, ptr addrspace(1) %a
%b.val = load half, ptr addrspace(1) %b %b.val = load half, ptr addrspace(1) %b
%c.val = load half, ptr addrspace(1) %c %c.val = load half, ptr addrspace(1) %c
%t.val = fmul half %a.val, %b.val %t.val = fmul half %a.val, %b.val
%r.val = fadd half %t.val, %c.val %r.val = fadd half %t.val, %c.val
store half %r.val, ptr addrspace(1) %r store half %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_f16_same_add:
; SI: v_mad_f32 v{{[0-9]}}, v{{[0-9]+}}, v{{[0-9]+}}, [[ADD:v[0-9]+]]
; SI: v_mac_f32_e32 [[ADD]], v{{[0-9]+}}, v{{[0-9]+}}
; VI: v_mad_f16 v{{[0-9]}}, v{{[0-9]+}}, v{{[0-9]+}}, [[ADD:v[0-9]+]]
; VI: v_mac_f16_e32 [[ADD]], v{{[0-9]+}}, v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_f16_same_add( define amdgpu_kernel void @mac_f16_same_add(
; SI-LABEL: mac_f16_same_add:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[4:11], s[0:1], 0x9
; SI-NEXT: s_load_dwordx4 s[12:15], s[0:1], 0x11
; SI-NEXT: s_load_dwordx2 s[16:17], s[0:1], 0x15
; SI-NEXT: s_mov_b32 s3, 0xf000
; SI-NEXT: s_mov_b32 s2, -1
; SI-NEXT: s_mov_b32 s18, s2
; SI-NEXT: s_mov_b32 s19, s3
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s20, s8
; SI-NEXT: s_mov_b32 s21, s9
; SI-NEXT: s_mov_b32 s22, s2
; SI-NEXT: s_mov_b32 s23, s3
; SI-NEXT: s_mov_b32 s8, s10
; SI-NEXT: s_mov_b32 s9, s11
; SI-NEXT: s_mov_b32 s10, s2
; SI-NEXT: s_mov_b32 s11, s3
; SI-NEXT: s_mov_b32 s24, s12
; SI-NEXT: s_mov_b32 s25, s13
; SI-NEXT: s_mov_b32 s26, s2
; SI-NEXT: s_mov_b32 s27, s3
; SI-NEXT: s_mov_b32 s12, s14
; SI-NEXT: s_mov_b32 s13, s15
; SI-NEXT: s_mov_b32 s14, s2
; SI-NEXT: s_mov_b32 s15, s3
; SI-NEXT: buffer_load_ushort v0, off, s[16:19], 0
; SI-NEXT: buffer_load_ushort v1, off, s[20:23], 0
; SI-NEXT: buffer_load_ushort v2, off, s[8:11], 0
; SI-NEXT: buffer_load_ushort v3, off, s[24:27], 0
; SI-NEXT: buffer_load_ushort v4, off, s[12:15], 0
; SI-NEXT: s_mov_b32 s0, s4
; SI-NEXT: s_mov_b32 s1, s5
; SI-NEXT: s_mov_b32 s8, s6
; SI-NEXT: s_mov_b32 s9, s7
; SI-NEXT: s_waitcnt vmcnt(4)
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(3)
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v3, v3
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v4, v4
; SI-NEXT: v_mad_f32 v1, v1, v2, v3
; SI-NEXT: v_mac_f32_e32 v3, v4, v0
; SI-NEXT: v_cvt_f16_f32_e32 v0, v1
; SI-NEXT: v_cvt_f16_f32_e32 v1, v3
; SI-NEXT: buffer_store_short v0, off, s[0:3], 0
; SI-NEXT: buffer_store_short v1, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_f16_same_add:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[4:11], s[0:1], 0x24
; VI-NEXT: s_load_dwordx4 s[12:15], s[0:1], 0x44
; VI-NEXT: s_load_dwordx2 s[16:17], s[0:1], 0x54
; VI-NEXT: s_mov_b32 s3, 0xf000
; VI-NEXT: s_mov_b32 s2, -1
; VI-NEXT: s_mov_b32 s22, s2
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s20, s8
; VI-NEXT: s_mov_b32 s21, s9
; VI-NEXT: s_mov_b32 s23, s3
; VI-NEXT: s_mov_b32 s8, s10
; VI-NEXT: s_mov_b32 s9, s11
; VI-NEXT: s_mov_b32 s10, s2
; VI-NEXT: s_mov_b32 s11, s3
; VI-NEXT: s_mov_b32 s24, s12
; VI-NEXT: s_mov_b32 s25, s13
; VI-NEXT: s_mov_b32 s26, s2
; VI-NEXT: s_mov_b32 s27, s3
; VI-NEXT: s_mov_b32 s12, s14
; VI-NEXT: s_mov_b32 s13, s15
; VI-NEXT: s_mov_b32 s14, s2
; VI-NEXT: s_mov_b32 s15, s3
; VI-NEXT: s_mov_b32 s18, s2
; VI-NEXT: s_mov_b32 s19, s3
; VI-NEXT: buffer_load_ushort v0, off, s[20:23], 0
; VI-NEXT: buffer_load_ushort v1, off, s[8:11], 0
; VI-NEXT: buffer_load_ushort v2, off, s[24:27], 0
; VI-NEXT: buffer_load_ushort v3, off, s[16:19], 0
; VI-NEXT: buffer_load_ushort v4, off, s[12:15], 0
; VI-NEXT: s_mov_b32 s0, s4
; VI-NEXT: s_mov_b32 s1, s5
; VI-NEXT: s_mov_b32 s8, s6
; VI-NEXT: s_mov_b32 s9, s7
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_mad_f16 v0, v0, v1, v2
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mac_f16_e32 v2, v4, v3
; VI-NEXT: buffer_store_short v0, off, s[0:3], 0
; VI-NEXT: buffer_store_short v2, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_f16_same_add:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_clause 0x2
; GFX11-NEXT: s_load_b256 s[4:11], s[0:1], 0x24
; GFX11-NEXT: s_load_b128 s[12:15], s[0:1], 0x44
; GFX11-NEXT: s_load_b64 s[16:17], s[0:1], 0x54
; GFX11-NEXT: s_mov_b32 s2, -1
; GFX11-NEXT: s_mov_b32 s3, 0x31016000
; GFX11-NEXT: s_mov_b32 s18, s2
; GFX11-NEXT: s_mov_b32 s19, s3
; GFX11-NEXT: s_mov_b32 s22, s2
; GFX11-NEXT: s_mov_b32 s23, s3
; GFX11-NEXT: s_mov_b32 s26, s2
; GFX11-NEXT: s_mov_b32 s27, s3
; GFX11-NEXT: s_mov_b32 s38, s2
; GFX11-NEXT: s_mov_b32 s39, s3
; GFX11-NEXT: s_mov_b32 s30, s2
; GFX11-NEXT: s_mov_b32 s31, s3
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s20, s8
; GFX11-NEXT: s_mov_b32 s21, s9
; GFX11-NEXT: s_mov_b32 s24, s10
; GFX11-NEXT: s_mov_b32 s25, s11
; GFX11-NEXT: s_mov_b32 s36, s14
; GFX11-NEXT: s_mov_b32 s37, s15
; GFX11-NEXT: buffer_load_u16 v0, off, s[16:19], 0
; GFX11-NEXT: buffer_load_u16 v1, off, s[20:23], 0
; GFX11-NEXT: buffer_load_u16 v2, off, s[24:27], 0
; GFX11-NEXT: buffer_load_u16 v3, off, s[36:39], 0
; GFX11-NEXT: s_mov_b32 s28, s12
; GFX11-NEXT: s_mov_b32 s29, s13
; GFX11-NEXT: s_mov_b32 s0, s4
; GFX11-NEXT: buffer_load_u16 v4, off, s[28:31], 0
; GFX11-NEXT: s_mov_b32 s1, s5
; GFX11-NEXT: s_mov_b32 s10, s2
; GFX11-NEXT: s_mov_b32 s11, s3
; GFX11-NEXT: s_mov_b32 s8, s6
; GFX11-NEXT: s_mov_b32 s9, s7
; GFX11-NEXT: s_waitcnt vmcnt(2)
; GFX11-NEXT: v_mul_f16_e32 v1, v1, v2
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_mul_f16_e32 v0, v3, v0
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX11-NEXT: v_add_f16_e32 v1, v1, v4
; GFX11-NEXT: v_add_f16_e32 v0, v0, v4
; GFX11-NEXT: buffer_store_b16 v1, off, s[0:3], 0
; GFX11-NEXT: buffer_store_b16 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r0, ptr addrspace(1) %r0,
ptr addrspace(1) %r1, ptr addrspace(1) %r1,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c, ptr addrspace(1) %c,
ptr addrspace(1) %d, ptr addrspace(1) %d,
ptr addrspace(1) %e) #0 { ptr addrspace(1) %e) #0 {
entry: entry:
Show All 9 Linesentry:
%t1.val = fmul half %d.val, %e.val %t1.val = fmul half %d.val, %e.val
%r1.val = fadd half %t1.val, %c.val %r1.val = fadd half %t1.val, %c.val
store half %r0.val, ptr addrspace(1) %r0 store half %r0.val, ptr addrspace(1) %r0
store half %r1.val, ptr addrspace(1) %r1 store half %r1.val, ptr addrspace(1) %r1
ret void ret void
} }
; GCN-LABEL: {{^}}mac_f16_neg_a:
; SI: v_cvt_f32_f16_e32 [[CVT_A:v[0-9]+]], v{{[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT_B:v[0-9]+]], v{{[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT_C:v[0-9]+]], v{{[0-9]+}}
; SI: v_mad_f32 v{{[0-9]+}}, -[[CVT_A]], [[CVT_B]], [[CVT_C]]
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_f16_neg_a( define amdgpu_kernel void @mac_f16_neg_a(
; SI-LABEL: mac_f16_neg_a:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; SI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; SI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, -v0, v1, v2
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: buffer_store_short v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_f16_neg_a:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; VI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; VI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f16 v0, -v0, v1, v2
; VI-NEXT: buffer_store_short v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_f16_neg_a:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_u16 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_u16 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_u16 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_mul_f16_e32 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_sub_f16_e32 v0, v2, v0
; GFX11-NEXT: buffer_store_b16 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load half, ptr addrspace(1) %a %a.val = load half, ptr addrspace(1) %a
%b.val = load half, ptr addrspace(1) %b %b.val = load half, ptr addrspace(1) %b
%c.val = load half, ptr addrspace(1) %c %c.val = load half, ptr addrspace(1) %c
%a.neg = fneg half %a.val %a.neg = fneg half %a.val
%t.val = fmul half %a.neg, %b.val %t.val = fmul half %a.neg, %b.val
%r.val = fadd half %t.val, %c.val %r.val = fadd half %t.val, %c.val
store half %r.val, ptr addrspace(1) %r store half %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_f16_neg_b:
; SI: v_cvt_f32_f16_e32 [[CVT_A:v[0-9]+]], v{{[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT_B:v[0-9]+]], v{{[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT_C:v[0-9]+]], v{{[0-9]+}}
; SI: v_mad_f32 v{{[0-9]+}}, -[[CVT_A]], [[CVT_B]], [[CVT_C]]
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_f16_neg_b( define amdgpu_kernel void @mac_f16_neg_b(
; SI-LABEL: mac_f16_neg_b:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; SI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; SI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, -v0, v1, v2
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: buffer_store_short v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_f16_neg_b:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; VI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; VI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f16 v0, -v0, v1, v2
; VI-NEXT: buffer_store_short v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_f16_neg_b:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_u16 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_u16 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_u16 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_mul_f16_e32 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_sub_f16_e32 v0, v2, v0
; GFX11-NEXT: buffer_store_b16 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load half, ptr addrspace(1) %a %a.val = load half, ptr addrspace(1) %a
%b.val = load half, ptr addrspace(1) %b %b.val = load half, ptr addrspace(1) %b
%c.val = load half, ptr addrspace(1) %c %c.val = load half, ptr addrspace(1) %c
%b.neg = fneg half %b.val %b.neg = fneg half %b.val
%t.val = fmul half %a.val, %b.neg %t.val = fmul half %a.val, %b.neg
%r.val = fadd half %t.val, %c.val %r.val = fadd half %t.val, %c.val
store half %r.val, ptr addrspace(1) %r store half %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_f16_neg_c:
; SI: v_cvt_f32_f16_e32
; SI: v_cvt_f32_f16_e32
; SI: v_cvt_f32_f16_e32
; SI: v_mad_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[0-9]+}}
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_f16_neg_c( define amdgpu_kernel void @mac_f16_neg_c(
; SI-LABEL: mac_f16_neg_c:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; SI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; SI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, v0, v1, -v2
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: buffer_store_short v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_f16_neg_c:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; VI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; VI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f16 v0, v0, v1, -v2
; VI-NEXT: buffer_store_short v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_f16_neg_c:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_u16 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_u16 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_u16 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_mul_f16_e32 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_sub_f16_e32 v0, v0, v2
; GFX11-NEXT: buffer_store_b16 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load half, ptr addrspace(1) %a %a.val = load half, ptr addrspace(1) %a
%b.val = load half, ptr addrspace(1) %b %b.val = load half, ptr addrspace(1) %b
%c.val = load half, ptr addrspace(1) %c %c.val = load half, ptr addrspace(1) %c
%c.neg = fneg half %c.val %c.neg = fneg half %c.val
%t.val = fmul half %a.val, %b.val %t.val = fmul half %a.val, %b.val
%r.val = fadd half %t.val, %c.neg %r.val = fadd half %t.val, %c.neg
store half %r.val, ptr addrspace(1) %r store half %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_f16_neg_a_safe_fp_math:
; SI: v_sub_f32_e32 v[[NEG_A:[0-9]+]], 0, v{{[0-9]+}}
; SI: v_mac_f32_e32 v{{[0-9]+}}, v[[NEG_A]], v{{[0-9]+}}
; VI: v_sub_f16_e32 v[[NEG_A:[0-9]+]], 0, v{{[0-9]+}}
; VI: v_mac_f16_e32 v{{[0-9]+}}, v[[NEG_A]], v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_f16_neg_a_safe_fp_math( define amdgpu_kernel void @mac_f16_neg_a_safe_fp_math(
; SI-LABEL: mac_f16_neg_a_safe_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; SI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; SI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_sub_f32_e32 v0, 0, v0
; SI-NEXT: v_mac_f32_e32 v2, v0, v1
; SI-NEXT: v_cvt_f16_f32_e32 v0, v2
; SI-NEXT: buffer_store_short v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_f16_neg_a_safe_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; VI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; VI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_sub_f16_e32 v0, 0, v0
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mac_f16_e32 v2, v0, v1
; VI-NEXT: buffer_store_short v2, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_f16_neg_a_safe_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: buffer_load_u16 v0, off, s[12:15], 0
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: buffer_load_u16 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_u16 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(2)
; GFX11-NEXT: v_sub_f16_e32 v0, 0, v0
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
; GFX11-NEXT: v_mul_f16_e32 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_add_f16_e32 v0, v0, v2
; GFX11-NEXT: buffer_store_b16 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load half, ptr addrspace(1) %a %a.val = load half, ptr addrspace(1) %a
%b.val = load half, ptr addrspace(1) %b %b.val = load half, ptr addrspace(1) %b
%c.val = load half, ptr addrspace(1) %c %c.val = load half, ptr addrspace(1) %c
%a.neg = fsub half 0.0, %a.val %a.neg = fsub half 0.0, %a.val
%t.val = fmul half %a.neg, %b.val %t.val = fmul half %a.neg, %b.val
%r.val = fadd half %t.val, %c.val %r.val = fadd half %t.val, %c.val
store half %r.val, ptr addrspace(1) %r store half %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_f16_neg_b_safe_fp_math:
; SI: v_sub_f32_e32 v[[NEG_A:[0-9]+]], 0, v{{[0-9]+}}
; SI: v_mac_f32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v[[NEG_A]]
; VI: v_sub_f16_e32 v[[NEG_A:[0-9]+]], 0, v{{[0-9]+}}
; VI: v_mac_f16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v[[NEG_A]]
; GCN: s_endpgm
define amdgpu_kernel void @mac_f16_neg_b_safe_fp_math( define amdgpu_kernel void @mac_f16_neg_b_safe_fp_math(
; SI-LABEL: mac_f16_neg_b_safe_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; SI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; SI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_sub_f32_e32 v1, 0, v1
; SI-NEXT: v_mac_f32_e32 v2, v0, v1
; SI-NEXT: v_cvt_f16_f32_e32 v0, v2
; SI-NEXT: buffer_store_short v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_f16_neg_b_safe_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; VI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; VI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(1)
; VI-NEXT: v_sub_f16_e32 v1, 0, v1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mac_f16_e32 v2, v0, v1
; VI-NEXT: buffer_store_short v2, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_f16_neg_b_safe_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s4
; GFX11-NEXT: s_mov_b32 s13, s5
; GFX11-NEXT: s_mov_b32 s16, s2
; GFX11-NEXT: buffer_load_u16 v0, off, s[12:15], 0
; GFX11-NEXT: s_mov_b32 s17, s3
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: buffer_load_u16 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_u16 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(2)
; GFX11-NEXT: v_sub_f16_e32 v0, 0, v0
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
; GFX11-NEXT: v_mul_f16_e32 v0, v1, v0
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_add_f16_e32 v0, v0, v2
; GFX11-NEXT: buffer_store_b16 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load half, ptr addrspace(1) %a %a.val = load half, ptr addrspace(1) %a
%b.val = load half, ptr addrspace(1) %b %b.val = load half, ptr addrspace(1) %b
%c.val = load half, ptr addrspace(1) %c %c.val = load half, ptr addrspace(1) %c
%b.neg = fsub half 0.0, %b.val %b.neg = fsub half 0.0, %b.val
%t.val = fmul half %a.val, %b.neg %t.val = fmul half %a.val, %b.neg
%r.val = fadd half %t.val, %c.val %r.val = fadd half %t.val, %c.val
store half %r.val, ptr addrspace(1) %r store half %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_f16_neg_c_safe_fp_math:
; SI: v_sub_f32_e32 v[[NEG_A:[0-9]+]], 0, v{{[0-9]+}}
; SI: v_mac_f32_e32 v[[NEG_A]], v{{[0-9]+}}, v{{[0-9]+}}
; VI: v_sub_f16_e32 v[[NEG_A:[0-9]+]], 0, v{{[0-9]+}}
; VI: v_mac_f16_e32 v[[NEG_A]], v{{[0-9]+}}, v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_f16_neg_c_safe_fp_math( define amdgpu_kernel void @mac_f16_neg_c_safe_fp_math(
; SI-LABEL: mac_f16_neg_c_safe_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; SI-NEXT: buffer_load_ushort v1, off, s[4:7], 0
; SI-NEXT: buffer_load_ushort v2, off, s[16:19], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_sub_f32_e32 v1, 0, v1
; SI-NEXT: v_mac_f32_e32 v1, v0, v2
; SI-NEXT: v_cvt_f16_f32_e32 v0, v1
; SI-NEXT: buffer_store_short v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_f16_neg_c_safe_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; VI-NEXT: buffer_load_ushort v1, off, s[4:7], 0
; VI-NEXT: buffer_load_ushort v2, off, s[16:19], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(1)
; VI-NEXT: v_sub_f16_e32 v1, 0, v1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mac_f16_e32 v1, v0, v2
; VI-NEXT: buffer_store_short v1, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_f16_neg_c_safe_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s22, s10
; GFX11-NEXT: s_mov_b32 s23, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s20, s6
; GFX11-NEXT: s_mov_b32 s21, s7
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_u16 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_u16 v1, off, s[20:23], 0
; GFX11-NEXT: buffer_load_u16 v2, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_sub_f16_e32 v1, 0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_mul_f16_e32 v0, v0, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_add_f16_e32 v0, v0, v1
; GFX11-NEXT: buffer_store_b16 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load half, ptr addrspace(1) %a %a.val = load half, ptr addrspace(1) %a
%b.val = load half, ptr addrspace(1) %b %b.val = load half, ptr addrspace(1) %b
%c.val = load half, ptr addrspace(1) %c %c.val = load half, ptr addrspace(1) %c
%c.neg = fsub half 0.0, %c.val %c.neg = fsub half 0.0, %c.val
%t.val = fmul half %a.val, %b.val %t.val = fmul half %a.val, %b.val
%r.val = fadd half %t.val, %c.neg %r.val = fadd half %t.val, %c.neg
store half %r.val, ptr addrspace(1) %r store half %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_f16_neg_a_nsz_fp_math:
; SI: v_cvt_f32_f16_e32 [[CVT_A:v[0-9]+]], v{{[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT_B:v[0-9]+]], v{{[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT_C:v[0-9]+]], v{{[0-9]+}}
; SI: v_mad_f32 v{{[0-9]+}}, -[[CVT_A]], [[CVT_B]], [[CVT_C]]
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_f16_neg_a_nsz_fp_math( define amdgpu_kernel void @mac_f16_neg_a_nsz_fp_math(
; SI-LABEL: mac_f16_neg_a_nsz_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; SI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; SI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, -v0, v1, v2
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: buffer_store_short v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_f16_neg_a_nsz_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; VI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; VI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f16 v0, -v0, v1, v2
; VI-NEXT: buffer_store_short v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_f16_neg_a_nsz_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_u16 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_u16 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_u16 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_mul_f16_e32 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_sub_f16_e32 v0, v2, v0
; GFX11-NEXT: buffer_store_b16 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #1 { ptr addrspace(1) %c) #1 {
entry: entry:
%a.val = load half, ptr addrspace(1) %a %a.val = load half, ptr addrspace(1) %a
%b.val = load half, ptr addrspace(1) %b %b.val = load half, ptr addrspace(1) %b
%c.val = load half, ptr addrspace(1) %c %c.val = load half, ptr addrspace(1) %c
%a.neg = fsub half 0.0, %a.val %a.neg = fsub half 0.0, %a.val
%t.val = fmul half %a.neg, %b.val %t.val = fmul half %a.neg, %b.val
%r.val = fadd half %t.val, %c.val %r.val = fadd half %t.val, %c.val
store half %r.val, ptr addrspace(1) %r store half %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_f16_neg_b_nsz_fp_math:
; SI: v_cvt_f32_f16_e32 [[CVT_A:v[0-9]+]], v{{[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT_B:v[0-9]+]], v{{[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT_C:v[0-9]+]], v{{[0-9]+}}
; SI: v_mad_f32 v{{[0-9]+}}, -[[CVT_A]], [[CVT_B]], [[CVT_C]]
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_f16_neg_b_nsz_fp_math( define amdgpu_kernel void @mac_f16_neg_b_nsz_fp_math(
; SI-LABEL: mac_f16_neg_b_nsz_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; SI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; SI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, -v0, v1, v2
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: buffer_store_short v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_f16_neg_b_nsz_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; VI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; VI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f16 v0, -v0, v1, v2
; VI-NEXT: buffer_store_short v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_f16_neg_b_nsz_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_u16 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_u16 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_u16 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_mul_f16_e32 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_sub_f16_e32 v0, v2, v0
; GFX11-NEXT: buffer_store_b16 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #1 { ptr addrspace(1) %c) #1 {
entry: entry:
%a.val = load half, ptr addrspace(1) %a %a.val = load half, ptr addrspace(1) %a
%b.val = load half, ptr addrspace(1) %b %b.val = load half, ptr addrspace(1) %b
%c.val = load half, ptr addrspace(1) %c %c.val = load half, ptr addrspace(1) %c
%b.neg = fsub half 0.0, %b.val %b.neg = fsub half 0.0, %b.val
%t.val = fmul half %a.val, %b.neg %t.val = fmul half %a.val, %b.neg
%r.val = fadd half %t.val, %c.val %r.val = fadd half %t.val, %c.val
store half %r.val, ptr addrspace(1) %r store half %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_f16_neg_c_nsz_fp_math:
; SI: v_cvt_f32_f16_e32 [[CVT_A:v[0-9]+]], v{{[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT_B:v[0-9]+]], v{{[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT_C:v[0-9]+]], v{{[0-9]+}}
; SI: v_mad_f32 v{{[0-9]+}}, [[CVT_A]], [[CVT_B]], -[[CVT_C]]
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[0-9]}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_f16_neg_c_nsz_fp_math( define amdgpu_kernel void @mac_f16_neg_c_nsz_fp_math(
; SI-LABEL: mac_f16_neg_c_nsz_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; SI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; SI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, v0, v1, -v2
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: buffer_store_short v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_f16_neg_c_nsz_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_ushort v0, off, s[12:15], 0
; VI-NEXT: buffer_load_ushort v1, off, s[16:19], 0
; VI-NEXT: buffer_load_ushort v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mad_f16 v0, v0, v1, -v2
; VI-NEXT: buffer_store_short v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_f16_neg_c_nsz_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_u16 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_u16 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_u16 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_mul_f16_e32 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_sub_f16_e32 v0, v0, v2
; GFX11-NEXT: buffer_store_b16 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #1 { ptr addrspace(1) %c) #1 {
entry: entry:
%a.val = load half, ptr addrspace(1) %a %a.val = load half, ptr addrspace(1) %a
%b.val = load half, ptr addrspace(1) %b %b.val = load half, ptr addrspace(1) %b
%c.val = load half, ptr addrspace(1) %c %c.val = load half, ptr addrspace(1) %c
%c.neg = fsub half 0.0, %c.val %c.neg = fsub half 0.0, %c.val
%t.val = fmul half %a.val, %b.val %t.val = fmul half %a.val, %b.val
%r.val = fadd half %t.val, %c.neg %r.val = fadd half %t.val, %c.neg
store half %r.val, ptr addrspace(1) %r store half %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_v2f16:
; GCN: {{buffer|flat}}_load_dword v[[A_V2_F16:[0-9]+]]
; GCN: {{buffer|flat}}_load_dword v[[B_V2_F16:[0-9]+]]
; GCN: {{buffer|flat}}_load_dword v[[C_V2_F16:[0-9]+]]
; SI: v_cvt_f32_f16_e32 v[[A_F32_0:[0-9]+]], v[[A_V2_F16]]
; SI-DAG: v_lshrrev_b32_e32 v[[A_F16_1:[0-9]+]], 16, v[[A_V2_F16]]
; SI-DAG: v_cvt_f32_f16_e32 v[[A_F32_1:[0-9]+]], v[[A_F16_1]]
; SI-DAG: v_lshrrev_b32_e32 v[[B_F16_1:[0-9]+]], 16, v[[B_V2_F16]]
; SI-DAG: v_cvt_f32_f16_e32 v[[B_F32_1:[0-9]+]], v[[B_F16_1]]
; SI-DAG: v_cvt_f32_f16_e32 v[[B_F32_0:[0-9]+]], v[[B_V2_F16]]
; SI-DAG: v_lshrrev_b32_e32 v[[C_F16_1:[0-9]+]], 16, v[[C_V2_F16]]
; SI-DAG: v_cvt_f32_f16_e32 v[[C_F32_1:[0-9]+]], v[[C_F16_1]]
; SI-DAG: v_cvt_f32_f16_e32 v[[C_F32_0:[0-9]+]], v[[C_V2_F16]]
; SI-DAG: v_mac_f32_e32 v[[C_F32_0]], v[[A_F32_0]], v[[B_F32_0]]
; SI-DAG: v_cvt_f16_f32_e32 v[[R_F16_LO:[0-9]+]], v[[C_F32_0]]
; SI-DAG: v_mac_f32_e32 v[[C_F32_1]], v[[A_F32_1]], v[[B_F32_1]]
; SI-DAG: v_cvt_f16_f32_e32 v[[R_F16_1:[0-9]+]], v[[C_F32_1]]
; SI: v_lshlrev_b32_e32 v[[R_F16_HI:[0-9]+]], 16, v[[R_F16_1]]
; VI-NOT: and
; SI: v_or_b32_e32 v[[R_V2_F16:[0-9]+]], v[[R_F16_LO]], v[[R_F16_HI]]
; VI-DAG: v_lshrrev_b32_e32 v[[C_F16_1:[0-9]+]], 16, v[[C_V2_F16]]
; VI-DAG: v_mac_f16_sdwa v[[C_F16_1]], v[[A_V2_F16]], v[[B_V2_F16]] dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
; VI-DAG: v_mac_f16_e32 v[[C_V2_F16]], v[[A_V2_F16]], v[[B_V2_F16]]
; VI-DAG: v_lshlrev_b32_e32 v[[R_F16_HI:[0-9]+]], 16, v[[C_F16_1]]
; VI-NOT: and
; VI: v_or_b32_e32 v[[R_V2_F16:[0-9]+]], v[[C_V2_F16]], v[[R_F16_HI]]
; GCN: {{buffer|flat}}_store_dword v[[R_V2_F16]]
; GCN: s_endpgm
define amdgpu_kernel void @mac_v2f16( define amdgpu_kernel void @mac_v2f16(
; SI-LABEL: mac_v2f16:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: s_barrier
; SI-NEXT: buffer_load_dword v1, off, s[16:19], 0
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: s_barrier
; SI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: v_cvt_f32_f16_e32 v3, v0
; SI-NEXT: v_lshrrev_b32_e32 v0, 16, v0
; SI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; SI-NEXT: v_cvt_f32_f16_e32 v4, v4
; SI-NEXT: v_cvt_f32_f16_e32 v5, v5
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mac_f32_e32 v5, v0, v4
; SI-NEXT: v_cvt_f16_f32_e32 v0, v5
; SI-NEXT: v_mac_f32_e32 v2, v3, v1
; SI-NEXT: v_cvt_f16_f32_e32 v1, v2
; SI-NEXT: v_lshlrev_b32_e32 v0, 16, v0
; SI-NEXT: v_or_b32_e32 v0, v1, v0
; SI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_v2f16:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: s_barrier
; VI-NEXT: buffer_load_dword v1, off, s[16:19], 0
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: s_barrier
; VI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_lshrrev_b32_e32 v3, 16, v2
; VI-NEXT: v_mac_f16_sdwa v3, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
; VI-NEXT: v_lshlrev_b32_e32 v3, 16, v3
; VI-NEXT: v_mac_f16_e32 v2, v0, v1
; VI-NEXT: v_or_b32_e32 v0, v2, v3
; VI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_v2f16:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_b32 v0, off, s[12:15], 0
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_barrier
; GFX11-NEXT: buffer_load_b32 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_barrier
; GFX11-NEXT: buffer_load_b32 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: v_pk_mul_f16 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_pk_add_f16 v0, v0, v2
; GFX11-NEXT: buffer_store_b32 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load <2 x half>, ptr addrspace(1) %a %a.val = load <2 x half>, ptr addrspace(1) %a
call void @llvm.amdgcn.s.barrier() #2 call void @llvm.amdgcn.s.barrier() #2
%b.val = load <2 x half>, ptr addrspace(1) %b %b.val = load <2 x half>, ptr addrspace(1) %b
call void @llvm.amdgcn.s.barrier() #2 call void @llvm.amdgcn.s.barrier() #2
%c.val = load <2 x half>, ptr addrspace(1) %c %c.val = load <2 x half>, ptr addrspace(1) %c
%t.val = fmul <2 x half> %a.val, %b.val %t.val = fmul <2 x half> %a.val, %b.val
%r.val = fadd <2 x half> %t.val, %c.val %r.val = fadd <2 x half> %t.val, %c.val
store <2 x half> %r.val, ptr addrspace(1) %r store <2 x half> %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_v2f16_same_add:
; SI-DAG: v_mad_f32 v{{[0-9]}}, v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; SI-DAG: v_mac_f32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; SI-DAG: v_mad_f32 v{{[0-9]}}, v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; SI-DAG: v_mac_f32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; VI-DAG: v_mac_f16_sdwa v{{[0-9]}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
; VI-DAG: v_mad_f16 v{{[0-9]}}, v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; VI-DAG: v_mac_f16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
; VI-DAG: v_mac_f16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_v2f16_same_add( define amdgpu_kernel void @mac_v2f16_same_add(
; SI-LABEL: mac_v2f16_same_add:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[4:11], s[0:1], 0x9
; SI-NEXT: s_load_dwordx4 s[12:15], s[0:1], 0x11
; SI-NEXT: s_load_dwordx2 s[16:17], s[0:1], 0x15
; SI-NEXT: s_mov_b32 s3, 0xf000
; SI-NEXT: s_mov_b32 s2, -1
; SI-NEXT: s_mov_b32 s22, s2
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s20, s8
; SI-NEXT: s_mov_b32 s21, s9
; SI-NEXT: s_mov_b32 s23, s3
; SI-NEXT: s_mov_b32 s8, s10
; SI-NEXT: s_mov_b32 s9, s11
; SI-NEXT: s_mov_b32 s10, s2
; SI-NEXT: s_mov_b32 s11, s3
; SI-NEXT: s_mov_b32 s24, s12
; SI-NEXT: s_mov_b32 s25, s13
; SI-NEXT: s_mov_b32 s26, s2
; SI-NEXT: s_mov_b32 s27, s3
; SI-NEXT: s_mov_b32 s18, s2
; SI-NEXT: s_mov_b32 s19, s3
; SI-NEXT: s_mov_b32 s12, s14
; SI-NEXT: buffer_load_dword v1, off, s[20:23], 0
; SI-NEXT: s_mov_b32 s13, s15
; SI-NEXT: buffer_load_dword v2, off, s[8:11], 0
; SI-NEXT: s_mov_b32 s14, s2
; SI-NEXT: buffer_load_dword v3, off, s[24:27], 0
; SI-NEXT: s_mov_b32 s15, s3
; SI-NEXT: buffer_load_dword v0, off, s[16:19], 0
; SI-NEXT: buffer_load_dword v4, off, s[12:15], 0
; SI-NEXT: s_mov_b32 s0, s4
; SI-NEXT: s_mov_b32 s1, s5
; SI-NEXT: s_mov_b32 s8, s6
; SI-NEXT: s_mov_b32 s9, s7
; SI-NEXT: s_waitcnt vmcnt(4)
; SI-NEXT: v_cvt_f32_f16_e32 v6, v1
; SI-NEXT: v_lshrrev_b32_e32 v1, 16, v1
; SI-NEXT: s_waitcnt vmcnt(3)
; SI-NEXT: v_cvt_f32_f16_e32 v7, v2
; SI-NEXT: v_lshrrev_b32_e32 v2, 16, v2
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v8, v3
; SI-NEXT: v_lshrrev_b32_e32 v3, 16, v3
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v5, v0
; SI-NEXT: v_lshrrev_b32_e32 v0, 16, v0
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_cvt_f32_f16_e32 v9, v4
; SI-NEXT: v_lshrrev_b32_e32 v4, 16, v4
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_cvt_f32_f16_e32 v3, v3
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: v_cvt_f32_f16_e32 v4, v4
; SI-NEXT: v_mad_f32 v6, v6, v7, v8
; SI-NEXT: v_mad_f32 v1, v1, v2, v3
; SI-NEXT: v_mac_f32_e32 v8, v9, v5
; SI-NEXT: v_mac_f32_e32 v3, v4, v0
; SI-NEXT: v_cvt_f16_f32_e32 v0, v1
; SI-NEXT: v_cvt_f16_f32_e32 v1, v3
; SI-NEXT: v_cvt_f16_f32_e32 v2, v6
; SI-NEXT: v_cvt_f16_f32_e32 v3, v8
; SI-NEXT: v_lshlrev_b32_e32 v0, 16, v0
; SI-NEXT: v_lshlrev_b32_e32 v1, 16, v1
; SI-NEXT: v_or_b32_e32 v0, v2, v0
; SI-NEXT: v_or_b32_e32 v1, v3, v1
; SI-NEXT: buffer_store_dword v0, off, s[0:3], 0
; SI-NEXT: buffer_store_dword v1, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_v2f16_same_add:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[4:11], s[0:1], 0x24
; VI-NEXT: s_load_dwordx4 s[12:15], s[0:1], 0x44
; VI-NEXT: s_load_dwordx2 s[16:17], s[0:1], 0x54
; VI-NEXT: s_mov_b32 s3, 0xf000
; VI-NEXT: s_mov_b32 s2, -1
; VI-NEXT: s_mov_b32 s22, s2
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s20, s8
; VI-NEXT: s_mov_b32 s21, s9
; VI-NEXT: s_mov_b32 s23, s3
; VI-NEXT: s_mov_b32 s8, s10
; VI-NEXT: s_mov_b32 s9, s11
; VI-NEXT: s_mov_b32 s10, s2
; VI-NEXT: s_mov_b32 s11, s3
; VI-NEXT: s_mov_b32 s24, s12
; VI-NEXT: s_mov_b32 s25, s13
; VI-NEXT: s_mov_b32 s26, s2
; VI-NEXT: s_mov_b32 s27, s3
; VI-NEXT: s_mov_b32 s12, s14
; VI-NEXT: s_mov_b32 s13, s15
; VI-NEXT: s_mov_b32 s14, s2
; VI-NEXT: s_mov_b32 s15, s3
; VI-NEXT: s_mov_b32 s18, s2
; VI-NEXT: s_mov_b32 s19, s3
; VI-NEXT: buffer_load_dword v0, off, s[20:23], 0
; VI-NEXT: buffer_load_dword v1, off, s[8:11], 0
; VI-NEXT: buffer_load_dword v2, off, s[24:27], 0
; VI-NEXT: buffer_load_dword v3, off, s[16:19], 0
; VI-NEXT: buffer_load_dword v4, off, s[12:15], 0
; VI-NEXT: s_mov_b32 s0, s4
; VI-NEXT: s_mov_b32 s1, s5
; VI-NEXT: s_mov_b32 s8, s6
; VI-NEXT: s_mov_b32 s9, s7
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; VI-NEXT: v_mov_b32_e32 v7, v5
; VI-NEXT: v_mac_f16_sdwa v7, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
; VI-NEXT: v_mad_f16 v6, v0, v1, v2
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mac_f16_sdwa v5, v4, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
; VI-NEXT: v_lshlrev_b32_e32 v1, 16, v7
; VI-NEXT: v_mac_f16_e32 v2, v4, v3
; VI-NEXT: v_lshlrev_b32_e32 v0, 16, v5
; VI-NEXT: v_or_b32_e32 v1, v6, v1
; VI-NEXT: v_or_b32_e32 v0, v2, v0
; VI-NEXT: buffer_store_dword v1, off, s[0:3], 0
; VI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_v2f16_same_add:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_clause 0x2
; GFX11-NEXT: s_load_b256 s[4:11], s[0:1], 0x24
; GFX11-NEXT: s_load_b128 s[12:15], s[0:1], 0x44
; GFX11-NEXT: s_load_b64 s[16:17], s[0:1], 0x54
; GFX11-NEXT: s_mov_b32 s2, -1
; GFX11-NEXT: s_mov_b32 s3, 0x31016000
; GFX11-NEXT: s_mov_b32 s18, s2
; GFX11-NEXT: s_mov_b32 s19, s3
; GFX11-NEXT: s_mov_b32 s22, s2
; GFX11-NEXT: s_mov_b32 s23, s3
; GFX11-NEXT: s_mov_b32 s26, s2
; GFX11-NEXT: s_mov_b32 s27, s3
; GFX11-NEXT: s_mov_b32 s38, s2
; GFX11-NEXT: s_mov_b32 s39, s3
; GFX11-NEXT: s_mov_b32 s30, s2
; GFX11-NEXT: s_mov_b32 s31, s3
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s20, s8
; GFX11-NEXT: s_mov_b32 s21, s9
; GFX11-NEXT: s_mov_b32 s24, s10
; GFX11-NEXT: s_mov_b32 s25, s11
; GFX11-NEXT: s_mov_b32 s36, s14
; GFX11-NEXT: s_mov_b32 s37, s15
; GFX11-NEXT: buffer_load_b32 v0, off, s[16:19], 0
; GFX11-NEXT: buffer_load_b32 v1, off, s[20:23], 0
; GFX11-NEXT: buffer_load_b32 v2, off, s[24:27], 0
; GFX11-NEXT: buffer_load_b32 v3, off, s[36:39], 0
; GFX11-NEXT: s_mov_b32 s28, s12
; GFX11-NEXT: s_mov_b32 s29, s13
; GFX11-NEXT: s_mov_b32 s0, s4
; GFX11-NEXT: buffer_load_b32 v4, off, s[28:31], 0
; GFX11-NEXT: s_mov_b32 s1, s5
; GFX11-NEXT: s_mov_b32 s10, s2
; GFX11-NEXT: s_mov_b32 s11, s3
; GFX11-NEXT: s_mov_b32 s8, s6
; GFX11-NEXT: s_mov_b32 s9, s7
; GFX11-NEXT: s_waitcnt vmcnt(2)
; GFX11-NEXT: v_pk_mul_f16 v1, v1, v2
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_pk_mul_f16 v0, v3, v0
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX11-NEXT: v_pk_add_f16 v1, v1, v4
; GFX11-NEXT: v_pk_add_f16 v0, v0, v4
; GFX11-NEXT: buffer_store_b32 v1, off, s[0:3], 0
; GFX11-NEXT: buffer_store_b32 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r0, ptr addrspace(1) %r0,
ptr addrspace(1) %r1, ptr addrspace(1) %r1,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c, ptr addrspace(1) %c,
ptr addrspace(1) %d, ptr addrspace(1) %d,
ptr addrspace(1) %e) #0 { ptr addrspace(1) %e) #0 {
entry: entry:
Show All 9 Linesentry:
%t1.val = fmul <2 x half> %d.val, %e.val %t1.val = fmul <2 x half> %d.val, %e.val
%r1.val = fadd <2 x half> %t1.val, %c.val %r1.val = fadd <2 x half> %t1.val, %c.val
store <2 x half> %r0.val, ptr addrspace(1) %r0 store <2 x half> %r0.val, ptr addrspace(1) %r0
store <2 x half> %r1.val, ptr addrspace(1) %r1 store <2 x half> %r1.val, ptr addrspace(1) %r1
ret void ret void
} }
; GCN-LABEL: {{^}}mac_v2f16_neg_a:
; SI: v_cvt_f32_f16_e32 [[CVT0:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT1:v[0-9]+]], {{v[0-9]+}}
; SI: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; SI: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_v2f16_neg_a( define amdgpu_kernel void @mac_v2f16_neg_a(
; SI-LABEL: mac_v2f16_neg_a:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_dword v1, off, s[16:19], 0
; SI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v3, v0
; SI-NEXT: v_lshrrev_b32_e32 v0, 16, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: v_cvt_f32_f16_e32 v4, v4
; SI-NEXT: v_cvt_f32_f16_e32 v5, v5
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, -v0, v4, v5
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: v_mad_f32 v1, -v3, v1, v2
; SI-NEXT: v_cvt_f16_f32_e32 v1, v1
; SI-NEXT: v_lshlrev_b32_e32 v0, 16, v0
; SI-NEXT: v_or_b32_e32 v0, v1, v0
; SI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_v2f16_neg_a:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_dword v0, off, s[16:19], 0
; VI-NEXT: buffer_load_dword v1, off, s[12:15], 0
; VI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_lshrrev_b32_e32 v3, 16, v0
; VI-NEXT: s_waitcnt vmcnt(1)
; VI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; VI-NEXT: v_mad_f16 v3, -v4, v3, v5
; VI-NEXT: v_lshlrev_b32_e32 v3, 16, v3
; VI-NEXT: v_mad_f16 v0, -v1, v0, v2
; VI-NEXT: v_or_b32_e32 v0, v0, v3
; VI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_v2f16_neg_a:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_b32 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_b32 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_b32 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_pk_mul_f16 v0, v0, v1 neg_lo:[0,1] neg_hi:[0,1]
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_pk_add_f16 v0, v2, v0
; GFX11-NEXT: buffer_store_b32 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load <2 x half>, ptr addrspace(1) %a %a.val = load <2 x half>, ptr addrspace(1) %a
%b.val = load <2 x half>, ptr addrspace(1) %b %b.val = load <2 x half>, ptr addrspace(1) %b
%c.val = load <2 x half>, ptr addrspace(1) %c %c.val = load <2 x half>, ptr addrspace(1) %c
%a.neg = fneg <2 x half> %a.val %a.neg = fneg <2 x half> %a.val
%t.val = fmul <2 x half> %a.neg, %b.val %t.val = fmul <2 x half> %a.neg, %b.val
%r.val = fadd <2 x half> %t.val, %c.val %r.val = fadd <2 x half> %t.val, %c.val
store <2 x half> %r.val, ptr addrspace(1) %r store <2 x half> %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_v2f16_neg_b
; SI: v_cvt_f32_f16_e32 [[CVT0:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT1:v[0-9]+]], {{v[0-9]+}}
; SI: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; SI: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_v2f16_neg_b( define amdgpu_kernel void @mac_v2f16_neg_b(
; SI-LABEL: mac_v2f16_neg_b:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_dword v1, off, s[16:19], 0
; SI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v3, v0
; SI-NEXT: v_lshrrev_b32_e32 v0, 16, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: v_cvt_f32_f16_e32 v4, v4
; SI-NEXT: v_cvt_f32_f16_e32 v5, v5
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, -v0, v4, v5
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: v_mad_f32 v1, -v3, v1, v2
; SI-NEXT: v_cvt_f16_f32_e32 v1, v1
; SI-NEXT: v_lshlrev_b32_e32 v0, 16, v0
; SI-NEXT: v_or_b32_e32 v0, v1, v0
; SI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_v2f16_neg_b:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_dword v0, off, s[16:19], 0
; VI-NEXT: buffer_load_dword v1, off, s[12:15], 0
; VI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_lshrrev_b32_e32 v3, 16, v0
; VI-NEXT: s_waitcnt vmcnt(1)
; VI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; VI-NEXT: v_mad_f16 v3, -v4, v3, v5
; VI-NEXT: v_lshlrev_b32_e32 v3, 16, v3
; VI-NEXT: v_mad_f16 v0, -v1, v0, v2
; VI-NEXT: v_or_b32_e32 v0, v0, v3
; VI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_v2f16_neg_b:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_b32 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_b32 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_b32 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_pk_mul_f16 v0, v0, v1 neg_lo:[0,1] neg_hi:[0,1]
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_pk_add_f16 v0, v2, v0
; GFX11-NEXT: buffer_store_b32 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load <2 x half>, ptr addrspace(1) %a %a.val = load <2 x half>, ptr addrspace(1) %a
%b.val = load <2 x half>, ptr addrspace(1) %b %b.val = load <2 x half>, ptr addrspace(1) %b
%c.val = load <2 x half>, ptr addrspace(1) %c %c.val = load <2 x half>, ptr addrspace(1) %c
%b.neg = fneg <2 x half> %b.val %b.neg = fneg <2 x half> %b.val
%t.val = fmul <2 x half> %a.val, %b.neg %t.val = fmul <2 x half> %a.val, %b.neg
%r.val = fadd <2 x half> %t.val, %c.val %r.val = fadd <2 x half> %t.val, %c.val
store <2 x half> %r.val, ptr addrspace(1) %r store <2 x half> %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_v2f16_neg_c:
; SI: v_cvt_f32_f16_e32 [[CVT0:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT1:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT2:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT3:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT4:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT5:v[0-9]+]], {{v[0-9]+}}
; SI-DAG: v_mad_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[0-9]+}}
; SI-DAG: v_mad_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[0-9]+}}
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[0-9]+}}
; VI: v_mad_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_v2f16_neg_c( define amdgpu_kernel void @mac_v2f16_neg_c(
; SI-LABEL: mac_v2f16_neg_c:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_dword v1, off, s[16:19], 0
; SI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v3, v0
; SI-NEXT: v_lshrrev_b32_e32 v0, 16, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: v_cvt_f32_f16_e32 v4, v4
; SI-NEXT: v_cvt_f32_f16_e32 v5, v5
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, v0, v4, -v5
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: v_mad_f32 v1, v3, v1, -v2
; SI-NEXT: v_cvt_f16_f32_e32 v1, v1
; SI-NEXT: v_lshlrev_b32_e32 v0, 16, v0
; SI-NEXT: v_or_b32_e32 v0, v1, v0
; SI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_v2f16_neg_c:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_dword v0, off, s[16:19], 0
; VI-NEXT: buffer_load_dword v1, off, s[12:15], 0
; VI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_lshrrev_b32_e32 v3, 16, v0
; VI-NEXT: s_waitcnt vmcnt(1)
; VI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; VI-NEXT: v_mad_f16 v3, v4, v3, -v5
; VI-NEXT: v_lshlrev_b32_e32 v3, 16, v3
; VI-NEXT: v_mad_f16 v0, v1, v0, -v2
; VI-NEXT: v_or_b32_e32 v0, v0, v3
; VI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_v2f16_neg_c:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_b32 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_b32 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_b32 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_pk_mul_f16 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_pk_add_f16 v0, v0, v2 neg_lo:[0,1] neg_hi:[0,1]
; GFX11-NEXT: buffer_store_b32 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load <2 x half>, ptr addrspace(1) %a %a.val = load <2 x half>, ptr addrspace(1) %a
%b.val = load <2 x half>, ptr addrspace(1) %b %b.val = load <2 x half>, ptr addrspace(1) %b
%c.val = load <2 x half>, ptr addrspace(1) %c %c.val = load <2 x half>, ptr addrspace(1) %c
%c.neg = fneg <2 x half> %c.val %c.neg = fneg <2 x half> %c.val
%t.val = fmul <2 x half> %a.val, %b.val %t.val = fmul <2 x half> %a.val, %b.val
%r.val = fadd <2 x half> %t.val, %c.neg %r.val = fadd <2 x half> %t.val, %c.neg
store <2 x half> %r.val, ptr addrspace(1) %r store <2 x half> %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_v2f16_neg_a_safe_fp_math:
; SI: v_sub_f32_e32 v[[NEG_A0:[0-9]+]], 0, v{{[0-9]+}}
; SI: v_sub_f32_e32 v[[NEG_A1:[0-9]+]], 0, v{{[0-9]+}}
; SI-DAG: v_mac_f32_e32 v{{[0-9]+}}, v[[NEG_A0]], v{{[0-9]+}}
; SI-DAG: v_mac_f32_e32 v{{[0-9]+}}, v[[NEG_A1]], v{{[0-9]+}}
; VI-DAG: v_mov_b32_e32 [[ZERO:v[0-9]+]], 0
; VI-DAG: v_sub_f16_e32 v[[NEG_A1:[0-9]+]], 0, v{{[0-9]+}}
; VI-DAG: v_sub_f16_sdwa v[[NEG_A0:[0-9]+]], [[ZERO]], v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
; VI-DAG: v_mac_f16_sdwa v{{[0-9]+}}, v[[NEG_A0]], v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
; VI-DAG: v_mac_f16_e32 v{{[0-9]+}}, v[[NEG_A1]], v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_v2f16_neg_a_safe_fp_math( define amdgpu_kernel void @mac_v2f16_neg_a_safe_fp_math(
; SI-LABEL: mac_v2f16_neg_a_safe_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_dword v1, off, s[16:19], 0
; SI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_lshrrev_b32_e32 v3, 16, v0
; SI-NEXT: v_cvt_f32_f16_e32 v3, v3
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v4, v1
; SI-NEXT: v_lshrrev_b32_e32 v1, 16, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: v_cvt_f32_f16_e32 v5, v5
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_sub_f32_e32 v3, 0, v3
; SI-NEXT: v_sub_f32_e32 v0, 0, v0
; SI-NEXT: v_mac_f32_e32 v5, v3, v1
; SI-NEXT: v_cvt_f16_f32_e32 v1, v5
; SI-NEXT: v_mac_f32_e32 v2, v0, v4
; SI-NEXT: v_cvt_f16_f32_e32 v0, v2
; SI-NEXT: v_lshlrev_b32_e32 v1, 16, v1
; SI-NEXT: v_or_b32_e32 v0, v0, v1
; SI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_v2f16_neg_a_safe_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: buffer_load_dword v1, off, s[4:7], 0
; VI-NEXT: buffer_load_dword v2, off, s[16:19], 0
; VI-NEXT: v_mov_b32_e32 v3, 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_sub_f16_e32 v4, 0, v0
; VI-NEXT: v_sub_f16_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
; VI-NEXT: s_waitcnt vmcnt(1)
; VI-NEXT: v_lshrrev_b32_e32 v3, 16, v1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mac_f16_sdwa v3, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
; VI-NEXT: v_lshlrev_b32_e32 v0, 16, v3
; VI-NEXT: v_mac_f16_e32 v1, v4, v2
; VI-NEXT: v_or_b32_e32 v0, v1, v0
; VI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_v2f16_neg_a_safe_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: buffer_load_b32 v0, off, s[12:15], 0
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: buffer_load_b32 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_b32 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(2)
; GFX11-NEXT: v_pk_add_f16 v0, v0, 0 neg_lo:[1,0] neg_hi:[1,0]
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
; GFX11-NEXT: v_pk_mul_f16 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_pk_add_f16 v0, v0, v2
; GFX11-NEXT: buffer_store_b32 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load <2 x half>, ptr addrspace(1) %a %a.val = load <2 x half>, ptr addrspace(1) %a
%b.val = load <2 x half>, ptr addrspace(1) %b %b.val = load <2 x half>, ptr addrspace(1) %b
%c.val = load <2 x half>, ptr addrspace(1) %c %c.val = load <2 x half>, ptr addrspace(1) %c
%a.neg = fsub <2 x half> <half 0.0, half 0.0>, %a.val %a.neg = fsub <2 x half> <half 0.0, half 0.0>, %a.val
%t.val = fmul <2 x half> %a.neg, %b.val %t.val = fmul <2 x half> %a.neg, %b.val
%r.val = fadd <2 x half> %t.val, %c.val %r.val = fadd <2 x half> %t.val, %c.val
store <2 x half> %r.val, ptr addrspace(1) %r store <2 x half> %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_v2f16_neg_b_safe_fp_math:
; SI: v_sub_f32_e32 v[[NEG_A0:[0-9]+]], 0, v{{[0-9]+}}
; SI: v_sub_f32_e32 v[[NEG_A1:[0-9]+]], 0, v{{[0-9]+}}
; SI-DAG: v_mac_f32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v[[NEG_A0]]
; SI-DAG: v_mac_f32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v[[NEG_A1]]
; VI: v_mov_b32_e32 [[ZERO:v[0-9]+]], 0
; VI: v_sub_f16_e32 v[[NEG_A1:[0-9]+]], 0, v{{[0-9]+}}
; VI: v_sub_f16_sdwa v[[NEG_A0:[0-9]+]], [[ZERO]], v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
; VI-DAG: v_mac_f16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v[[NEG_A0]] dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
; VI-DAG: v_mac_f16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v[[NEG_A1]]
; GCN: s_endpgm
define amdgpu_kernel void @mac_v2f16_neg_b_safe_fp_math( define amdgpu_kernel void @mac_v2f16_neg_b_safe_fp_math(
; SI-LABEL: mac_v2f16_neg_b_safe_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: buffer_load_dword v1, off, s[16:19], 0
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; SI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; SI-NEXT: v_cvt_f32_f16_e32 v4, v4
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_cvt_f32_f16_e32 v3, v0
; SI-NEXT: v_lshrrev_b32_e32 v0, 16, v0
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: v_cvt_f32_f16_e32 v5, v5
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_sub_f32_e32 v4, 0, v4
; SI-NEXT: v_sub_f32_e32 v1, 0, v1
; SI-NEXT: v_mac_f32_e32 v5, v0, v4
; SI-NEXT: v_cvt_f16_f32_e32 v0, v5
; SI-NEXT: v_mac_f32_e32 v2, v3, v1
; SI-NEXT: v_cvt_f16_f32_e32 v1, v2
; SI-NEXT: v_lshlrev_b32_e32 v0, 16, v0
; SI-NEXT: v_or_b32_e32 v0, v1, v0
; SI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_v2f16_neg_b_safe_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: buffer_load_dword v0, off, s[16:19], 0
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: buffer_load_dword v1, off, s[4:7], 0
; VI-NEXT: buffer_load_dword v2, off, s[12:15], 0
; VI-NEXT: v_mov_b32_e32 v3, 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_sub_f16_e32 v4, 0, v0
; VI-NEXT: v_sub_f16_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
; VI-NEXT: s_waitcnt vmcnt(1)
; VI-NEXT: v_lshrrev_b32_e32 v3, 16, v1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mac_f16_sdwa v3, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
; VI-NEXT: v_lshlrev_b32_e32 v0, 16, v3
; VI-NEXT: v_mac_f16_e32 v1, v2, v4
; VI-NEXT: v_or_b32_e32 v0, v1, v0
; VI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_v2f16_neg_b_safe_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s4
; GFX11-NEXT: s_mov_b32 s13, s5
; GFX11-NEXT: s_mov_b32 s16, s2
; GFX11-NEXT: buffer_load_b32 v0, off, s[12:15], 0
; GFX11-NEXT: s_mov_b32 s17, s3
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: buffer_load_b32 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_b32 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(2)
; GFX11-NEXT: v_pk_add_f16 v0, v0, 0 neg_lo:[1,0] neg_hi:[1,0]
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
; GFX11-NEXT: v_pk_mul_f16 v0, v1, v0
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_pk_add_f16 v0, v0, v2
; GFX11-NEXT: buffer_store_b32 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load <2 x half>, ptr addrspace(1) %a %a.val = load <2 x half>, ptr addrspace(1) %a
%b.val = load <2 x half>, ptr addrspace(1) %b %b.val = load <2 x half>, ptr addrspace(1) %b
%c.val = load <2 x half>, ptr addrspace(1) %c %c.val = load <2 x half>, ptr addrspace(1) %c
%b.neg = fsub <2 x half> <half 0.0, half 0.0>, %b.val %b.neg = fsub <2 x half> <half 0.0, half 0.0>, %b.val
%t.val = fmul <2 x half> %a.val, %b.neg %t.val = fmul <2 x half> %a.val, %b.neg
%r.val = fadd <2 x half> %t.val, %c.val %r.val = fadd <2 x half> %t.val, %c.val
store <2 x half> %r.val, ptr addrspace(1) %r store <2 x half> %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_v2f16_neg_c_safe_fp_math:
; SI: v_sub_f32_e32 v[[NEG_A0:[0-9]+]], 0, v{{[0-9]+}}
; SI: v_sub_f32_e32 v[[NEG_A1:[0-9]+]], 0, v{{[0-9]+}}
; SI-DAG: v_mac_f32_e32 v[[NEG_A0]], v{{[0-9]+}}, v{{[0-9]+}}
; SI-DAG: v_mac_f32_e32 v[[NEG_A1]], v{{[0-9]+}}, v{{[0-9]+}}
; VI: v_mov_b32_e32 [[ZERO:v[0-9]+]], 0
; VI: v_sub_f16_e32 v[[NEG_A1:[0-9]+]], 0, v{{[0-9]+}}
; VI: v_sub_f16_sdwa v[[NEG_A0:[0-9]+]], [[ZERO]], v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
; VI-DAG: v_mac_f16_sdwa v[[NEG_A0]], v{{[0-9]+}}, v{{[0-9]+}} dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
; VI-DAG: v_mac_f16_e32 v[[NEG_A1]], v{{[0-9]+}}, v{{[0-9]+}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_v2f16_neg_c_safe_fp_math( define amdgpu_kernel void @mac_v2f16_neg_c_safe_fp_math(
; SI-LABEL: mac_v2f16_neg_c_safe_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_dword v1, off, s[16:19], 0
; SI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v3, v0
; SI-NEXT: v_lshrrev_b32_e32 v0, 16, v0
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; SI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; SI-NEXT: v_cvt_f32_f16_e32 v5, v5
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: v_cvt_f32_f16_e32 v4, v4
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: v_sub_f32_e32 v5, 0, v5
; SI-NEXT: v_sub_f32_e32 v2, 0, v2
; SI-NEXT: v_mac_f32_e32 v5, v0, v4
; SI-NEXT: v_cvt_f16_f32_e32 v0, v5
; SI-NEXT: v_mac_f32_e32 v2, v3, v1
; SI-NEXT: v_cvt_f16_f32_e32 v1, v2
; SI-NEXT: v_lshlrev_b32_e32 v0, 16, v0
; SI-NEXT: v_or_b32_e32 v0, v1, v0
; SI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_v2f16_neg_c_safe_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: buffer_load_dword v0, off, s[4:7], 0
; VI-NEXT: buffer_load_dword v1, off, s[12:15], 0
; VI-NEXT: buffer_load_dword v2, off, s[16:19], 0
; VI-NEXT: v_mov_b32_e32 v3, 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_sub_f16_e32 v4, 0, v0
; VI-NEXT: v_sub_f16_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_mac_f16_sdwa v0, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
; VI-NEXT: v_lshlrev_b32_e32 v0, 16, v0
; VI-NEXT: v_mac_f16_e32 v4, v1, v2
; VI-NEXT: v_or_b32_e32 v0, v4, v0
; VI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_v2f16_neg_c_safe_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s22, s10
; GFX11-NEXT: s_mov_b32 s23, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s20, s6
; GFX11-NEXT: s_mov_b32 s21, s7
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_b32 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_b32 v1, off, s[20:23], 0
; GFX11-NEXT: buffer_load_b32 v2, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_pk_add_f16 v1, v1, 0 neg_lo:[1,0] neg_hi:[1,0]
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_pk_mul_f16 v0, v0, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_pk_add_f16 v0, v0, v1
; GFX11-NEXT: buffer_store_b32 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #0 { ptr addrspace(1) %c) #0 {
entry: entry:
%a.val = load <2 x half>, ptr addrspace(1) %a %a.val = load <2 x half>, ptr addrspace(1) %a
%b.val = load <2 x half>, ptr addrspace(1) %b %b.val = load <2 x half>, ptr addrspace(1) %b
%c.val = load <2 x half>, ptr addrspace(1) %c %c.val = load <2 x half>, ptr addrspace(1) %c
%c.neg = fsub <2 x half> <half 0.0, half 0.0>, %c.val %c.neg = fsub <2 x half> <half 0.0, half 0.0>, %c.val
%t.val = fmul <2 x half> %a.val, %b.val %t.val = fmul <2 x half> %a.val, %b.val
%r.val = fadd <2 x half> %t.val, %c.neg %r.val = fadd <2 x half> %t.val, %c.neg
store <2 x half> %r.val, ptr addrspace(1) %r store <2 x half> %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_v2f16_neg_a_nsz_fp_math:
; SI: v_cvt_f32_f16_e32 [[CVT0:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT1:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT2:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT3:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT4:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT5:v[0-9]+]], {{v[0-9]+}}
; SI-DAG: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; SI-DAG: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[-0-9]}}
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[-0-9]}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_v2f16_neg_a_nsz_fp_math( define amdgpu_kernel void @mac_v2f16_neg_a_nsz_fp_math(
; SI-LABEL: mac_v2f16_neg_a_nsz_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_dword v1, off, s[16:19], 0
; SI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v3, v0
; SI-NEXT: v_lshrrev_b32_e32 v0, 16, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: v_cvt_f32_f16_e32 v4, v4
; SI-NEXT: v_cvt_f32_f16_e32 v5, v5
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, -v0, v4, v5
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: v_mad_f32 v1, -v3, v1, v2
; SI-NEXT: v_cvt_f16_f32_e32 v1, v1
; SI-NEXT: v_lshlrev_b32_e32 v0, 16, v0
; SI-NEXT: v_or_b32_e32 v0, v1, v0
; SI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_v2f16_neg_a_nsz_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_dword v0, off, s[16:19], 0
; VI-NEXT: buffer_load_dword v1, off, s[12:15], 0
; VI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_lshrrev_b32_e32 v3, 16, v0
; VI-NEXT: s_waitcnt vmcnt(1)
; VI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; VI-NEXT: v_mad_f16 v3, -v4, v3, v5
; VI-NEXT: v_lshlrev_b32_e32 v3, 16, v3
; VI-NEXT: v_mad_f16 v0, -v1, v0, v2
; VI-NEXT: v_or_b32_e32 v0, v0, v3
; VI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_v2f16_neg_a_nsz_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_b32 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_b32 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_b32 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_pk_mul_f16 v0, v0, v1 neg_lo:[0,1] neg_hi:[0,1]
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_pk_add_f16 v0, v2, v0
; GFX11-NEXT: buffer_store_b32 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #1 { ptr addrspace(1) %c) #1 {
entry: entry:
%a.val = load <2 x half>, ptr addrspace(1) %a %a.val = load <2 x half>, ptr addrspace(1) %a
%b.val = load <2 x half>, ptr addrspace(1) %b %b.val = load <2 x half>, ptr addrspace(1) %b
%c.val = load <2 x half>, ptr addrspace(1) %c %c.val = load <2 x half>, ptr addrspace(1) %c
%a.neg = fsub <2 x half> <half 0.0, half 0.0>, %a.val %a.neg = fsub <2 x half> <half 0.0, half 0.0>, %a.val
%t.val = fmul <2 x half> %a.neg, %b.val %t.val = fmul <2 x half> %a.neg, %b.val
%r.val = fadd <2 x half> %t.val, %c.val %r.val = fadd <2 x half> %t.val, %c.val
store <2 x half> %r.val, ptr addrspace(1) %r store <2 x half> %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_v2f16_neg_b_nsz_fp_math:
; SI: v_cvt_f32_f16_e32 [[CVT0:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT1:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT2:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT3:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT4:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT5:v[0-9]+]], {{v[0-9]+}}
; SI-DAG: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; SI-DAG: v_mad_f32 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[-0-9]}}
; VI: v_mad_f16 v{{[0-9]+}}, -v{{[0-9]+}}, v{{[0-9]+}}, v{{[-0-9]}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_v2f16_neg_b_nsz_fp_math( define amdgpu_kernel void @mac_v2f16_neg_b_nsz_fp_math(
; SI-LABEL: mac_v2f16_neg_b_nsz_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_dword v1, off, s[16:19], 0
; SI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v3, v0
; SI-NEXT: v_lshrrev_b32_e32 v0, 16, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: v_cvt_f32_f16_e32 v4, v4
; SI-NEXT: v_cvt_f32_f16_e32 v5, v5
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, -v0, v4, v5
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: v_mad_f32 v1, -v3, v1, v2
; SI-NEXT: v_cvt_f16_f32_e32 v1, v1
; SI-NEXT: v_lshlrev_b32_e32 v0, 16, v0
; SI-NEXT: v_or_b32_e32 v0, v1, v0
; SI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_v2f16_neg_b_nsz_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_dword v0, off, s[16:19], 0
; VI-NEXT: buffer_load_dword v1, off, s[12:15], 0
; VI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_lshrrev_b32_e32 v3, 16, v0
; VI-NEXT: s_waitcnt vmcnt(1)
; VI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; VI-NEXT: v_mad_f16 v3, -v4, v3, v5
; VI-NEXT: v_lshlrev_b32_e32 v3, 16, v3
; VI-NEXT: v_mad_f16 v0, -v1, v0, v2
; VI-NEXT: v_or_b32_e32 v0, v0, v3
; VI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_v2f16_neg_b_nsz_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_b32 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_b32 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_b32 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_pk_mul_f16 v0, v0, v1 neg_lo:[0,1] neg_hi:[0,1]
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_pk_add_f16 v0, v2, v0
; GFX11-NEXT: buffer_store_b32 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #1 { ptr addrspace(1) %c) #1 {
entry: entry:
%a.val = load <2 x half>, ptr addrspace(1) %a %a.val = load <2 x half>, ptr addrspace(1) %a
%b.val = load <2 x half>, ptr addrspace(1) %b %b.val = load <2 x half>, ptr addrspace(1) %b
%c.val = load <2 x half>, ptr addrspace(1) %c %c.val = load <2 x half>, ptr addrspace(1) %c
%b.neg = fsub <2 x half> <half 0.0, half 0.0>, %b.val %b.neg = fsub <2 x half> <half 0.0, half 0.0>, %b.val
%t.val = fmul <2 x half> %a.val, %b.neg %t.val = fmul <2 x half> %a.val, %b.neg
%r.val = fadd <2 x half> %t.val, %c.val %r.val = fadd <2 x half> %t.val, %c.val
store <2 x half> %r.val, ptr addrspace(1) %r store <2 x half> %r.val, ptr addrspace(1) %r
ret void ret void
} }
; GCN-LABEL: {{^}}mac_v2f16_neg_c_nsz_fp_math:
; SI: v_cvt_f32_f16_e32 [[CVT0:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT1:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT2:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT3:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT4:v[0-9]+]], {{v[0-9]+}}
; SI: v_cvt_f32_f16_e32 [[CVT5:v[0-9]+]], {{v[0-9]+}}
; SI-DAG: v_mad_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[0-9]+}}
; SI-DAG: v_mad_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[0-9]+}}
; VI-NOT: v_mac_f16
; VI: v_mad_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[-0-9]}}
; VI: v_mad_f16 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}, -v{{[-0-9]}}
; GCN: s_endpgm
define amdgpu_kernel void @mac_v2f16_neg_c_nsz_fp_math( define amdgpu_kernel void @mac_v2f16_neg_c_nsz_fp_math(
; SI-LABEL: mac_v2f16_neg_c_nsz_fp_math:
; SI: ; %bb.0: ; %entry
; SI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x9
; SI-NEXT: s_mov_b32 s11, 0xf000
; SI-NEXT: s_mov_b32 s10, -1
; SI-NEXT: s_mov_b32 s14, s10
; SI-NEXT: s_mov_b32 s15, s11
; SI-NEXT: s_waitcnt lgkmcnt(0)
; SI-NEXT: s_mov_b32 s12, s2
; SI-NEXT: s_mov_b32 s13, s3
; SI-NEXT: s_mov_b32 s16, s4
; SI-NEXT: s_mov_b32 s17, s5
; SI-NEXT: s_mov_b32 s18, s10
; SI-NEXT: s_mov_b32 s19, s11
; SI-NEXT: s_mov_b32 s4, s6
; SI-NEXT: s_mov_b32 s5, s7
; SI-NEXT: s_mov_b32 s6, s10
; SI-NEXT: buffer_load_dword v0, off, s[12:15], 0
; SI-NEXT: s_mov_b32 s7, s11
; SI-NEXT: buffer_load_dword v1, off, s[16:19], 0
; SI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; SI-NEXT: s_mov_b32 s8, s0
; SI-NEXT: s_mov_b32 s9, s1
; SI-NEXT: s_waitcnt vmcnt(2)
; SI-NEXT: v_cvt_f32_f16_e32 v3, v0
; SI-NEXT: v_lshrrev_b32_e32 v0, 16, v0
; SI-NEXT: s_waitcnt vmcnt(1)
; SI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; SI-NEXT: s_waitcnt vmcnt(0)
; SI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; SI-NEXT: v_cvt_f32_f16_e32 v0, v0
; SI-NEXT: v_cvt_f32_f16_e32 v4, v4
; SI-NEXT: v_cvt_f32_f16_e32 v5, v5
; SI-NEXT: v_cvt_f32_f16_e32 v1, v1
; SI-NEXT: v_cvt_f32_f16_e32 v2, v2
; SI-NEXT: v_mad_f32 v0, v0, v4, -v5
; SI-NEXT: v_cvt_f16_f32_e32 v0, v0
; SI-NEXT: v_mad_f32 v1, v3, v1, -v2
; SI-NEXT: v_cvt_f16_f32_e32 v1, v1
; SI-NEXT: v_lshlrev_b32_e32 v0, 16, v0
; SI-NEXT: v_or_b32_e32 v0, v1, v0
; SI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; SI-NEXT: s_endpgm
;
; VI-LABEL: mac_v2f16_neg_c_nsz_fp_math:
; VI: ; %bb.0: ; %entry
; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x24
; VI-NEXT: s_mov_b32 s11, 0xf000
; VI-NEXT: s_mov_b32 s10, -1
; VI-NEXT: s_mov_b32 s18, s10
; VI-NEXT: s_mov_b32 s19, s11
; VI-NEXT: s_waitcnt lgkmcnt(0)
; VI-NEXT: s_mov_b32 s16, s4
; VI-NEXT: s_mov_b32 s17, s5
; VI-NEXT: s_mov_b32 s14, s10
; VI-NEXT: s_mov_b32 s12, s2
; VI-NEXT: s_mov_b32 s13, s3
; VI-NEXT: s_mov_b32 s15, s11
; VI-NEXT: s_mov_b32 s4, s6
; VI-NEXT: s_mov_b32 s5, s7
; VI-NEXT: s_mov_b32 s6, s10
; VI-NEXT: s_mov_b32 s7, s11
; VI-NEXT: buffer_load_dword v0, off, s[16:19], 0
; VI-NEXT: buffer_load_dword v1, off, s[12:15], 0
; VI-NEXT: buffer_load_dword v2, off, s[4:7], 0
; VI-NEXT: s_mov_b32 s8, s0
; VI-NEXT: s_mov_b32 s9, s1
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_lshrrev_b32_e32 v3, 16, v0
; VI-NEXT: s_waitcnt vmcnt(1)
; VI-NEXT: v_lshrrev_b32_e32 v4, 16, v1
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_lshrrev_b32_e32 v5, 16, v2
; VI-NEXT: v_mad_f16 v3, v4, v3, -v5
; VI-NEXT: v_lshlrev_b32_e32 v3, 16, v3
; VI-NEXT: v_mad_f16 v0, v1, v0, -v2
; VI-NEXT: v_or_b32_e32 v0, v0, v3
; VI-NEXT: buffer_store_dword v0, off, s[8:11], 0
; VI-NEXT: s_endpgm
;
; GFX11-LABEL: mac_v2f16_neg_c_nsz_fp_math:
; GFX11: ; %bb.0: ; %entry
; GFX11-NEXT: s_load_b256 s[0:7], s[0:1], 0x24
; GFX11-NEXT: s_mov_b32 s10, -1
; GFX11-NEXT: s_mov_b32 s11, 0x31016000
; GFX11-NEXT: s_mov_b32 s14, s10
; GFX11-NEXT: s_mov_b32 s15, s11
; GFX11-NEXT: s_mov_b32 s18, s10
; GFX11-NEXT: s_mov_b32 s19, s11
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_mov_b32 s12, s2
; GFX11-NEXT: s_mov_b32 s13, s3
; GFX11-NEXT: s_mov_b32 s16, s4
; GFX11-NEXT: s_mov_b32 s17, s5
; GFX11-NEXT: buffer_load_b32 v0, off, s[12:15], 0
; GFX11-NEXT: buffer_load_b32 v1, off, s[16:19], 0
; GFX11-NEXT: s_mov_b32 s4, s6
; GFX11-NEXT: s_mov_b32 s5, s7
; GFX11-NEXT: s_mov_b32 s6, s10
; GFX11-NEXT: s_mov_b32 s7, s11
; GFX11-NEXT: s_mov_b32 s8, s0
; GFX11-NEXT: buffer_load_b32 v2, off, s[4:7], 0
; GFX11-NEXT: s_mov_b32 s9, s1
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_pk_mul_f16 v0, v0, v1
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX11-NEXT: v_pk_add_f16 v0, v0, v2 neg_lo:[0,1] neg_hi:[0,1]
; GFX11-NEXT: buffer_store_b32 v0, off, s[8:11], 0
; GFX11-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
; GFX11-NEXT: s_endpgm
ptr addrspace(1) %r, ptr addrspace(1) %r,
ptr addrspace(1) %a, ptr addrspace(1) %a,
ptr addrspace(1) %b, ptr addrspace(1) %b,
ptr addrspace(1) %c) #1 { ptr addrspace(1) %c) #1 {
entry: entry:
%a.val = load <2 x half>, ptr addrspace(1) %a %a.val = load <2 x half>, ptr addrspace(1) %a
%b.val = load <2 x half>, ptr addrspace(1) %b %b.val = load <2 x half>, ptr addrspace(1) %b
%c.val = load <2 x half>, ptr addrspace(1) %c %c.val = load <2 x half>, ptr addrspace(1) %c
Show All 14 Lines
llvm/test/CodeGen/AMDGPU/v_mac.ll