AMDGPU/GlobalISel: Lower G_FREM

[Petar.Avramovic]

Summary

Add custom lower for G_FREM.

[arsenm]

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

2737 ↗

(On Diff #279811)

buildFDiv? These are all dropping the flags too

2738 ↗

(On Diff #279811)

buildFFloor?

2738 ↗

(On Diff #279811)

Is this a correct handling of frem? The AMDGPU dag expansion uses ISD::FTRUNC, but I'm not sure that was ever correct

[arsenm]

I tried fixing the existing one to use ffloor instead of ftrunc; OpenCL conformance still fails when I plug frem into fmod

[Petar.Avramovic]

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

2738 ↗

(On Diff #279811)

The G_FPTRUNC complains about src and dst being same size, I hit

assert(DstTy.getSizeInBits() < SrcTy.getSizeInBits() &&
       "invalid widening trunc");

from the variable name I thought that FFloor could work but I guess that it works only when operands have same sign. (btw vulkan cts tests where I saw this passed).
Dag expansion seems correct from the description of fmod/frem.
This generic instruction here should discard digits after decimal point, do we have such instruction?

[arsenm]

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

2738 ↗

(On Diff #279811)

ISD::FTRUNC is G_INTRINSIC_TRUNC

I'm not really clear on what frem really is, or if it's really supposed to be the same as OpenCL fmod

[Petar.Avramovic]

Preserve flags, and use G_INTRINSIC_TRUNC.

[arsenm]

I don't think copy the DAG path was necessarily the right choice. The correct thing to do might be to make the DAG path use floor? Is either even correct if this fails conformance?

I don't think this should go in generic code unless we're more sure this is the correct operation

[foad]

In D84324#2168127, @arsenm wrote:

I don't think copy the DAG path was necessarily the right choice. The correct thing to do might be to make the DAG path use floor? Is either even correct if this fails conformance?

I don't think this should go in generic code unless we're more sure this is the correct operation

I am convinced that trunc (not floor) is what you need here to implement IR's frem instruction, where the result has the same sign as the dividend (same as the C library fmod).

See also the OpenCL fmod spec which is pretty clear on this: http://man.opencl.org/fmod.html

[foad]

LGTM if Matt has no further comments.

[arsenm]

I'd still like to understand why this is failing conformance if I use frem for opencl fmod. My current suspicion is the fsub + fmul really needs to be an FMA

[foad]

I'd still like to understand why this is failing conformance if I use frem for opencl fmod.

What is the alternative to using frem, that passes conformance?

[arsenm]

In D84324#2169289, @foad wrote:

I'd still like to understand why this is failing conformance if I use frem for opencl fmod.

What is the alternative to using frem, that passes conformance?

A huge expansion that involves loops:
https://github.com/RadeonOpenCompute/ROCm-Device-Libs/blob/amd-stg-open/ocml/src/remainderF_base.h#L38

[foad]

In D84324#2169293, @arsenm wrote:

In D84324#2169289, @foad wrote:

I'd still like to understand why this is failing conformance if I use frem for opencl fmod.

What is the alternative to using frem, that passes conformance?

A huge expansion that involves loops:
https://github.com/RadeonOpenCompute/ROCm-Device-Libs/blob/amd-stg-open/ocml/src/remainderF_base.h#L38

Then it needs debugging. Perhaps there are cases where the simple expansion gives fmod(x,y)==y, even though the result is supposed to have magnitude strictly less than y. Or perhaps it doesn't handle nans or infinities correctly.

[arsenm]

In D84324#2169348, @foad wrote:

In D84324#2169293, @arsenm wrote:

In D84324#2169289, @foad wrote:

I'd still like to understand why this is failing conformance if I use frem for opencl fmod.

What is the alternative to using frem, that passes conformance?

A huge expansion that involves loops:
https://github.com/RadeonOpenCompute/ROCm-Device-Libs/blob/amd-stg-open/ocml/src/remainderF_base.h#L38

Then it needs debugging. Perhaps there are cases where the simple expansion gives fmod(x,y)==y, even though the result is supposed to have magnitude strictly less than y. Or perhaps it doesn't handle nans or infinities correctly.

The errors aren't small, and aren't just edge cases:
ERROR: fmod: inf ulp error at {-0x1.7a1ba8p+111 (0xf73d0dd4), -0x1.5b9526p-97 (0x8f2dca93)}: *-0x1.c5f348p-98 vs. inf (0x7f800000) at index: 3

ERROR: fmod: -inf ulp error at {0x1.80bb0ep+70 (0x62c05d87), 0x1.08e51ap-82 (0x1684728d)}: *0x1.9d1d8cp-83 vs. -inf (0xff800000) at index: 0

ERROR: fmod: -134961856.000000 ulp error at {-0x1.f47464p-69 (0x9d7a3a32), -0x1.9bdef4p-97 (0x8f4def7a)}: *-0x1.682a78p-98 vs. -0x1.17eep-94 (0x908bf700) at index: 0

ERROR: fmod: inf ulp error at {-0x1.50c0b6p+67 (0xe128605b), 0x1.80b7ep-90 (0x12c05bf0)}: *-0x1.fa594p-91 vs. inf (0x7f800000) at index: 3

ERROR: fmod: -672311475662299076755456.000000 ulp error at {0x1.f8807ep+111 (0x777c403f), -0x1.7711aap+32 (0xcfbb88d5)}: *0x1.b2ab38p+30 vs. -0x1.1cbc28p+86 (0xea8e5e14) at index: 1

ERROR: fmod: 20258841443692227182914624580021649408.000000 ulp error at {-0x1.0107a4p+41 (0xd40083d2), -0x1.144c6ap-86 (0x948a2635)}: *-0x1.85d4acp-87 vs. 0x1.e7b6cp+13 (0x4673db60) at index: 2

ERROR: fmod: -29506071830531670016.000000 ulp error at {0x1.804b2cp+9 (0x44402596), -0x1.477f18p-57 (0xa323bf8c)}: *0x1.41dc88p-57 vs. -0x1.997aap-16 (0xb7ccbd50) at index: 4

ERROR: fmod: 211623838063271919251058575015936.000000 ulp error at {0x1.4d90a4p+41 (0x5426c852), 0x1.96784ap-65 (0x1f4b3c25)}: *0x1.61a4fp-68 vs. 0x1.4de23p+16 (0x47a6f118) at index: 3

ERROR: fmod: -inf ulp error at {0x1.9aedb8p+83 (0x694d76dc), 0x1.4c23f2p-119 (0x42611f9)}: *0x1.bba984p-120 vs. -inf (0xff800000) at index: 0

ERROR: fmod: 100959080964579999364158242467109404672.000000 ulp error at {0x1.aa412p+26 (0x4cd52090), -0x1.92be06p-101 (0x8d495f03)}: *0x1.59267p-101 vs. 0x1.2fd00cp+2 (0x4097e806) at index: 0

ERROR: fmod: inf ulp error at {-0x1.6ebdap+99 (0xf1375ed0), -0x1.fe72bp-36 (0xadff3958)}: *-0x1.39174p-38 vs. inf (0x7f800000) at index: 0

ERROR: fmod: -101767765295104.000000 ulp error at {-0x1.7f0c94p+21 (0xca3f864a), 0x1.8df2eap-29 (0x3146f975)}: *-0x1.191792p-29 vs. -0x1.723aap-6 (0xbcb91d50) at index: 4

ERROR: fmod: 5685162310369280.000000 ulp error at {-0x1.8ccd58p-14 (0xb8c666ac), 0x1.b15ed6p-71 (0x1c58af6b)}: *-0x1.b6d34p-74 vs. 0x1.432ap-45 (0x29219500) at index: 2

ERROR: fmod: inf ulp error at {-0x1.5e45dep+81 (0xe82f22ef), -0x1.4bae4cp-112 (0x87a5d726)}: *-0x1.56f99p-114 vs. inf (0x7f800000) at index: 1

ERROR: fmod: -inf ulp error at {0x1.cd2da6p+126 (0x7ee696d3), -0x1.647a26p-40 (0xabb23d13)}: *0x1.03ad2ap-40 vs. -inf (0xff800000) at index: 3

ERROR: fmod: -50753958115442425856.000000 ulp error at {-0x1.abb154p+118 (0xfad5d8aa), 0x1.8eb61ap+61 (0x5e475b0d)}: *-0x1.eeecp+52 vs. -0x1.602d24p+94 (0xeeb01692) at index: 0

ERROR: fmod: 65524330144892614344704.000000 ulp error at {0x1.ce604ap+11 (0x45673025), 0x1.0bbbaap-65 (0x1f05ddd5)}: *0x1.e25edp-66 vs. 0x1.bc0298p-14 (0x38de014c) at index: 1

ERROR: fmod: -inf ulp error at {0x1.c98fd6p+122 (0x7ce4c7eb), 0x1.fe4a44p-116 (0x5ff2522)}: *0x1.98780cp-116 vs. -inf (0xff800000) at index: 0

ERROR: fmod: inf ulp error at {-0x1.3132dep+84 (0xe998996f), -0x1.a91dcep-110 (0x88d48ee7)}: *-0x1.9270eap-110 vs. inf (0x7f800000) at index: 4

ERROR: fmod: -inf ulp error at {0x1.24c0e6p+36 (0x51926073), -0x1.a5528ap-122 (0x82d2a945)}: *0x1.86bd68p-122 vs. -inf (0xff800000) at index: 1

ERROR: fmod: 328137422309548672426878384939712118784.000000 ulp error at {-0x1.f16f6cp+68 (0xe1f8b7b6), 0x1.34d296p-58 (0x229a694b)}: *-0x1.5fa6ap-61 vs. 0x1.edb9fp+43 (0x5576dcf8) at index: 2

ERROR: fmod: -355087763374080.000000 ulp error at {0x1.b2ab98p+95 (0x6f5955cc), -0x1.ffc7fcp+45 (0xd67fe3fe)}: *0x1.fe7558p+45 vs. -0x1.42f35p+70 (0xe2a179a8) at index: 7

ERROR: fmod: -22169001879878959779282944.000000 ulp error at {-0x1.92d1bap-15 (0xb84968dd), -0x1.a4b14cp-100 (0x8dd258a6)}: *-0x1.38fb4p-103 vs. -0x1.25678p-42 (0xaa92b3c0) at index: 1

ERROR: fmod: inf ulp error at {-0x1.080adep+92 (0xed84056f), -0x1.c2f17cp-77 (0x996178be)}: *-0x1.cd9d5p-79 vs. inf (0x7f800000) at index: 0

ERROR: fmod: inf ulp error at {-0x1.17f76ap+95 (0xef0bfbb5), -0x1.4e0fe4p-108 (0x89a707f2)}: *-0x1.a2f8c8p-109 vs. inf (0x7f800000) at index: 2

ERROR: fmod: -6874084436590301837445300224.000000 ulp error at {0x1.697b38p+54 (0x5ab4bd9c), 0x1.f3988cp-39 (0x2c79cc46)}: *0x1.b4c1ap-40 vs. -0x1.6361cp+29 (0xce31b0e0) at index: 1

ERROR: fmod: -8738995383377592320.000000 ulp error at {0x1.496b64p+85 (0x6a24b5b2), 0x1.bae466p+24 (0x4bdd7233)}: *0x1.7a58cp+20 vs. -0x1.e51c98p+59 (0xdd728e4c) at index: 1

ERROR: fmod: 343065079314668060672.000000 ulp error at {-0x1.21b74ap-47 (0xa810dba5), 0x1.fdb82ep-117 (0x57edc17)}: *-0x1.606e5ap-117 vs. 0x1.298fcp-72 (0x1b94c7e0) at index: 4

ERROR: fmod: -inf ulp error at {0x1.8cdda2p+60 (0x5dc66ed1), 0x1.aeea96p-109 (0x957754b)}: *0x1.6ac0ep-112 vs. -inf (0xff800000) at index: 2

ERROR: fmod: -2991471016484578263040000.000000 ulp error at {-0x1.6a71bep+46 (0xd6b538df), -0x1.4091d2p-34 (0xaea048e9)}: *-0x1.7398e8p-36 vs. -0x1.3cbbfcp+22 (0xca9e5dfe) at index: 3

ERROR: fmod: -inf ulp error at {0x1.ec3bf8p+71 (0x63761dfc), -0x1.36870ep-68 (0x9d9b4387)}: *0x1.06b884p-68 vs. -inf (0xff800000) at index: 0

ERROR: fmod: -1561864113370783809536.000000 ulp error at {0x1.c99b3ap+127 (0x7f64cd9d), -0x1.8f9088p+58 (0xdcc7c844)}: *0x1.aed24p+55 vs. -0x1.52acep+102 (0xf2a95670) at index: 0

[foad]

In D84324#2169378, @arsenm wrote:

The errors aren't small, and aren't just edge cases:

The ones that return +/- inf are because the division overflows. The others look like rounding error in the division when the result of x/y is large but doesn't overflow - this can easily lead to a result with the wrong sign, or with magnitude larger than y. I don't think it's realistic to try to fix these problems with an inline expansion. It really needs a library function.

I suppose the question is: is this patch still a useful default implementation of frem?

[arsenm]

In D84324#2169535, @foad wrote:

In D84324#2169378, @arsenm wrote:

The errors aren't small, and aren't just edge cases:

The ones that return +/- inf are because the division overflows. The others look like rounding error in the division when the result of x/y is large but doesn't overflow - this can easily lead to a result with the wrong sign, or with magnitude larger than y. I don't think it's realistic to try to fix these problems with an inline expansion. It really needs a library function.

I suppose the question is: is this patch still a useful default implementation of frem?

For something that doesn't work perfectly, I don't think it belongs in the generic code. It would be more palatable to keep this in AMDGPU to match the DAG behavior

[arsenm]

In D84324#2170042, @arsenm wrote:

In D84324#2169535, @foad wrote:

In D84324#2169378, @arsenm wrote:

The errors aren't small, and aren't just edge cases:

The ones that return +/- inf are because the division overflows. The others look like rounding error in the division when the result of x/y is large but doesn't overflow - this can easily lead to a result with the wrong sign, or with magnitude larger than y. I don't think it's realistic to try to fix these problems with an inline expansion. It really needs a library function.

I suppose the question is: is this patch still a useful default implementation of frem?

For something that doesn't work perfectly, I don't think it belongs in the generic code. It would be more palatable to keep this in AMDGPU to match the DAG behavior

Maybe it should also fail to legalize if it's not afn?

[foad]

In D84324#2170042, @arsenm wrote:

In D84324#2169535, @foad wrote:

In D84324#2169378, @arsenm wrote:

The errors aren't small, and aren't just edge cases:

The ones that return +/- inf are because the division overflows. The others look like rounding error in the division when the result of x/y is large but doesn't overflow - this can easily lead to a result with the wrong sign, or with magnitude larger than y. I don't think it's realistic to try to fix these problems with an inline expansion. It really needs a library function.

I suppose the question is: is this patch still a useful default implementation of frem?

For something that doesn't work perfectly, I don't think it belongs in the generic code. It would be more palatable to keep this in AMDGPU to match the DAG behavior

Sounds reasonable.

[Petar.Avramovic]

Switch to custom lowering and update to match changes in dag custom lowering for frem (and use same lowering for s16 also).

[foad]

Looks OK to me but please wait to hear from @arsenm too.

llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h

1668–1674

Maybe put this declaration next to buildFAdd / buildFSub ?

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

703–705

Does this need to be conditional on ST.has16BitInsts ?

1869–1870

Use buildIntrinsicTrunc?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

703–705

It doesn't strictly have to be, but it would produce a better result to force promotion to 32-bit first

[Petar.Avramovic]

Addressed review comments.

[foad]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

703–705

I assume Matt meant to force promotion to 32-bit first if the subtarget doesn't have 16-bit instructions. Compared to the previous version of your patch, the code for fast_frem_f16 has got better for CI but worse for VI.

[Petar.Avramovic]

Force promotion to 32-bit first when subtarget doesn't have 16-bit instruction.

llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h

 
   /// Build and insert \p Res = G_FSUB \p Op0, \p Op1
   MachineInstrBuilder buildFSub(const DstOp &Dst, const SrcOp &Src0,
                                 const SrcOp &Src1,
                                 Optional<unsigned> Flags = None) {
     return buildInstr(TargetOpcode::G_FSUB, {Dst}, {Src0, Src1}, Flags);
   }
 
+  /// Build and insert \p Res = G_FDIV \p Op0, \p Op1
+  MachineInstrBuilder buildFDiv(const DstOp &Dst, const SrcOp &Src0,
+                                const SrcOp &Src1,
+                                Optional<unsigned> Flags = None) {
+    return buildInstr(TargetOpcode::G_FDIV, {Dst}, {Src0, Src1}, Flags);
+  }
+
   /// Build and insert \p Res = G_FMA \p Op0, \p Op1, \p Op2
   MachineInstrBuilder buildFMA(const DstOp &Dst, const SrcOp &Src0,
                                const SrcOp &Src1, const SrcOp &Src2,
                                Optional<unsigned> Flags = None) {
     return buildInstr(TargetOpcode::G_FMA, {Dst}, {Src0, Src1, Src2}, Flags);
   }
 
   /// Build and insert \p Res = G_FMAD \p Op0, \p Op1, \p Op2

   }
 
   /// Build and insert \p Res = G_UMAX \p Op0, \p Op1
   MachineInstrBuilder buildUMax(const DstOp &Dst, const SrcOp &Src0,
                                 const SrcOp &Src1) {
     return buildInstr(TargetOpcode::G_UMAX, {Dst}, {Src0, Src1});
   }
 
   /// Build and insert \p Res = G_JUMP_TABLE \p JTI
   ///
   /// G_JUMP_TABLE sets \p Res to the address of the jump table specified by
   /// the jump table index \p JTI.
   ///
   /// \return a MachineInstrBuilder for the newly created instruction.
   MachineInstrBuilder buildJumpTable(const LLT PtrTy, unsigned JTI);
 
   virtual MachineInstrBuilder buildInstr(unsigned Opc, ArrayRef<DstOp> DstOps,
                                          ArrayRef<SrcOp> SrcOps,
                                          Optional<unsigned> Flags = None);
 };
 
 } // End namespace llvm.
 #endif // LLVM_CODEGEN_GLOBALISEL_MACHINEIRBUILDER_H

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.h

Register getSegmentAperture(unsigned AddrSpace,
                               MachineIRBuilder &B) const;
 
   bool legalizeAddrSpaceCast(MachineInstr &MI, MachineRegisterInfo &MRI,
                              MachineIRBuilder &B) const;
   bool legalizeFrint(MachineInstr &MI, MachineRegisterInfo &MRI,
                      MachineIRBuilder &B) const;
   bool legalizeFceil(MachineInstr &MI, MachineRegisterInfo &MRI,
                      MachineIRBuilder &B) const;
+  bool legalizeFrem(MachineInstr &MI, MachineRegisterInfo &MRI,
+                    MachineIRBuilder &B) const;
   bool legalizeIntrinsicTrunc(MachineInstr &MI, MachineRegisterInfo &MRI,
                               MachineIRBuilder &B) const;
   bool legalizeITOFP(MachineInstr &MI, MachineRegisterInfo &MRI,
                      MachineIRBuilder &B, bool Signed) const;
   bool legalizeFPTOI(MachineInstr &MI, MachineRegisterInfo &MRI,
                      MachineIRBuilder &B, bool Signed) const;
   bool legalizeMinNumMaxNum(LegalizerHelper &Helper, MachineInstr &MI) const;
   bool legalizeExtractVectorElt(MachineInstr &MI, MachineRegisterInfo &MRI,

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

if (ST.hasMadF16() && ST.hasMadMacF32Insts())
     FMad.customFor({S32, S16});
   else if (ST.hasMadMacF32Insts())
     FMad.customFor({S32});
   else if (ST.hasMadF16())
     FMad.customFor({S16});
   FMad.scalarize(0)
       .lower();
 
+  getActionDefinitionsBuilder(G_FREM)
+    .minScalar(0, S32)
+    .customFor({S32, S64})
+    .scalarize(0);
+
   // TODO: Do we need to clamp maximum bitwidth?
   getActionDefinitionsBuilder(G_TRUNC)
     .legalIf(isScalar(0))
     .legalFor({{V2S16, V2S32}})
     .clampMaxNumElements(0, S16, 2)
     // Avoid scalarizing in cases that should be truly illegal. In unresolvable
     // situations (like an invalid implicit use), we don't want to infinite loop
     // in the legalizer.
bool AMDGPULegalizerInfo::legalizeCustom(LegalizerHelper &Helper,
 
   switch (MI.getOpcode()) {
   case TargetOpcode::G_ADDRSPACE_CAST:
     return legalizeAddrSpaceCast(MI, MRI, B);
   case TargetOpcode::G_FRINT:
     return legalizeFrint(MI, MRI, B);
   case TargetOpcode::G_FCEIL:
     return legalizeFceil(MI, MRI, B);
+  case TargetOpcode::G_FREM:
+    return legalizeFrem(MI, MRI, B);
   case TargetOpcode::G_INTRINSIC_TRUNC:
     return legalizeIntrinsicTrunc(MI, MRI, B);
   case TargetOpcode::G_SITOFP:
     return legalizeITOFP(MI, MRI, B, true);
   case TargetOpcode::G_UITOFP:
     return legalizeITOFP(MI, MRI, B, false);
   case TargetOpcode::G_FPTOSI:
     return legalizeFPTOI(MI, MRI, B, true);
bool AMDGPULegalizerInfo::legalizeFceil(
   auto And = B.buildAnd(S1, Lt0, NeTrunc);
   auto Add = B.buildSelect(S64, And, One, Zero);
 
   // TODO: Should this propagate fast-math-flags?
   B.buildFAdd(MI.getOperand(0).getReg(), Trunc, Add);
   return true;
 }
 
+bool AMDGPULegalizerInfo::legalizeFrem(
+  MachineInstr &MI, MachineRegisterInfo &MRI,
+  MachineIRBuilder &B) const {
+    Register DstReg = MI.getOperand(0).getReg();
+    Register Src0Reg = MI.getOperand(1).getReg();
+    Register Src1Reg = MI.getOperand(2).getReg();
+    auto Flags = MI.getFlags();
+    LLT Ty = MRI.getType(DstReg);
+
+    auto Div = B.buildFDiv(Ty, Src0Reg, Src1Reg, Flags);
+    auto Trunc = B.buildIntrinsicTrunc(Ty, Div, Flags);
+    auto Neg = B.buildFNeg(Ty, Trunc, Flags);
+    B.buildFMA(DstReg, Neg, Src1Reg, Src0Reg, Flags);
+    MI.eraseFromParent();
+    return true;
+}
+
 static MachineInstrBuilder extractF64Exponent(Register Hi,
                                               MachineIRBuilder &B) {
   const unsigned FractBits = 52;
   const unsigned ExpBits = 11;
   LLT S32 = LLT::scalar(32);
 
   auto Const0 = B.buildConstant(S32, FractBits - 32);
   auto Const1 = B.buildConstant(S32, ExpBits);

llvm/test/CodeGen/AMDGPU/GlobalISel/frem.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN:  llc -global-isel -amdgpu-scalarize-global-loads=false -enable-misched=0 -march=amdgcn -mcpu=bonaire -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,CI %s
+; RUN:  llc -global-isel -amdgpu-scalarize-global-loads=false -enable-misched=0 -march=amdgcn -mcpu=tonga -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,VI %s
+
+define amdgpu_kernel void @frem_f16(half addrspace(1)* %out, half addrspace(1)* %in1, half addrspace(1)* %in2) #0 {
+; CI-LABEL: frem_f16:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
+; CI-NEXT:    s_mov_b32 s10, -1
+; CI-NEXT:    s_mov_b32 s11, 0xf000
+; CI-NEXT:    s_mov_b64 s[2:3], s[10:11]
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_mov_b64 s[8:9], s[6:7]
+; CI-NEXT:    buffer_load_ushort v0, off, s[8:11], 0
+; CI-NEXT:    buffer_load_ushort v1, off, s[0:3], 0 offset:8
+; CI-NEXT:    s_waitcnt vmcnt(1)
+; CI-NEXT:    v_cvt_f32_f16_e32 v0, v0
+; CI-NEXT:    s_waitcnt vmcnt(0)
+; CI-NEXT:    v_cvt_f32_f16_e32 v1, v1
+; CI-NEXT:    v_div_scale_f32 v2, s[0:1], v1, v1, v0
+; CI-NEXT:    v_div_scale_f32 v3, vcc, v0, v1, v0
+; CI-NEXT:    v_rcp_f32_e32 v4, v2
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; CI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; CI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; CI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; CI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; CI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; CI-NEXT:    s_mov_b64 s[6:7], s[10:11]
+; CI-NEXT:    v_div_fixup_f32 v2, v2, v1, v0
+; CI-NEXT:    v_trunc_f32_e32 v2, v2
+; CI-NEXT:    v_fma_f32 v0, -v2, v1, v0
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 2, 2), 0
+; CI-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; CI-NEXT:    buffer_store_short v0, off, s[4:7], 0
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: frem_f16:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s6
+; VI-NEXT:    s_add_u32 s0, s8, 8
+; VI-NEXT:    v_mov_b32_e32 v1, s7
+; VI-NEXT:    s_addc_u32 s1, s9, 0
+; VI-NEXT:    flat_load_ushort v2, v[0:1]
+; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    v_mov_b32_e32 v1, s1
+; VI-NEXT:    flat_load_ushort v0, v[0:1]
+; VI-NEXT:    s_waitcnt vmcnt(1) lgkmcnt(1)
+; VI-NEXT:    v_cvt_f32_f16_e32 v1, v2
+; VI-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; VI-NEXT:    v_cvt_f32_f16_e32 v0, v0
+; VI-NEXT:    v_div_scale_f32 v2, s[0:1], v0, v0, v1
+; VI-NEXT:    v_div_scale_f32 v3, vcc, v1, v0, v1
+; VI-NEXT:    v_rcp_f32_e32 v4, v2
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; VI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; VI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; VI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; VI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; VI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; VI-NEXT:    v_div_fixup_f32 v2, v2, v0, v1
+; VI-NEXT:    v_trunc_f32_e32 v2, v2
+; VI-NEXT:    v_fma_f32 v0, -v2, v0, v1
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 2, 2), 0
+; VI-NEXT:    v_cvt_f16_f32_e32 v2, v0
+; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    v_mov_b32_e32 v1, s5
+; VI-NEXT:    flat_store_short v[0:1], v2
+; VI-NEXT:    s_endpgm
+   %gep2 = getelementptr half, half addrspace(1)* %in2, i32 4
+   %r0 = load half, half addrspace(1)* %in1, align 4
+   %r1 = load half, half addrspace(1)* %gep2, align 4
+   %r2 = frem half %r0, %r1
+   store half %r2, half addrspace(1)* %out, align 4
+   ret void
+}
+
+define amdgpu_kernel void @fast_frem_f16(half addrspace(1)* %out, half addrspace(1)* %in1, half addrspace(1)* %in2) #0 {
+; CI-LABEL: fast_frem_f16:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
+; CI-NEXT:    s_mov_b32 s10, -1
+; CI-NEXT:    s_mov_b32 s11, 0xf000
+; CI-NEXT:    s_mov_b64 s[2:3], s[10:11]
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_mov_b64 s[8:9], s[6:7]
+; CI-NEXT:    buffer_load_ushort v1, off, s[0:3], 0 offset:8
+; CI-NEXT:    buffer_load_ushort v0, off, s[8:11], 0
+; CI-NEXT:    s_mov_b64 s[6:7], s[10:11]
+; CI-NEXT:    s_waitcnt vmcnt(1)
+; CI-NEXT:    v_cvt_f32_f16_e32 v1, v1
+; CI-NEXT:    s_waitcnt vmcnt(0)
+; CI-NEXT:    v_cvt_f32_f16_e32 v0, v0
+; CI-NEXT:    v_rcp_f32_e32 v2, v1
+; CI-NEXT:    v_mul_f32_e32 v2, v0, v2
+; CI-NEXT:    v_trunc_f32_e32 v2, v2
+; CI-NEXT:    v_fma_f32 v0, -v2, v1, v0
+; CI-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; CI-NEXT:    buffer_store_short v0, off, s[4:7], 0
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: fast_frem_f16:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s6
+; VI-NEXT:    s_add_u32 s0, s8, 8
+; VI-NEXT:    v_mov_b32_e32 v1, s7
+; VI-NEXT:    s_addc_u32 s1, s9, 0
+; VI-NEXT:    flat_load_ushort v2, v[0:1]
+; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    v_mov_b32_e32 v1, s1
+; VI-NEXT:    flat_load_ushort v0, v[0:1]
+; VI-NEXT:    s_waitcnt vmcnt(1) lgkmcnt(1)
+; VI-NEXT:    v_cvt_f32_f16_e32 v1, v2
+; VI-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; VI-NEXT:    v_cvt_f32_f16_e32 v0, v0
+; VI-NEXT:    v_rcp_f32_e32 v2, v0
+; VI-NEXT:    v_mul_f32_e32 v2, v1, v2
+; VI-NEXT:    v_trunc_f32_e32 v2, v2
+; VI-NEXT:    v_fma_f32 v0, -v2, v0, v1
+; VI-NEXT:    v_cvt_f16_f32_e32 v2, v0
+; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    v_mov_b32_e32 v1, s5
+; VI-NEXT:    flat_store_short v[0:1], v2
+; VI-NEXT:    s_endpgm
+   %gep2 = getelementptr half, half addrspace(1)* %in2, i32 4
+   %r0 = load half, half addrspace(1)* %in1, align 4
+   %r1 = load half, half addrspace(1)* %gep2, align 4
+   %r2 = frem fast half %r0, %r1
+   store half %r2, half addrspace(1)* %out, align 4
+   ret void
+}
+
+define amdgpu_kernel void @unsafe_frem_f16(half addrspace(1)* %out, half addrspace(1)* %in1, half addrspace(1)* %in2) #1 {
+; CI-LABEL: unsafe_frem_f16:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
+; CI-NEXT:    s_mov_b32 s10, -1
+; CI-NEXT:    s_mov_b32 s11, 0xf000
+; CI-NEXT:    s_mov_b64 s[2:3], s[10:11]
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_mov_b64 s[8:9], s[6:7]
+; CI-NEXT:    buffer_load_ushort v0, off, s[8:11], 0
+; CI-NEXT:    buffer_load_ushort v1, off, s[0:3], 0 offset:8
+; CI-NEXT:    s_waitcnt vmcnt(1)
+; CI-NEXT:    v_cvt_f32_f16_e32 v0, v0
+; CI-NEXT:    s_waitcnt vmcnt(0)
+; CI-NEXT:    v_cvt_f32_f16_e32 v1, v1
+; CI-NEXT:    v_div_scale_f32 v2, s[0:1], v1, v1, v0
+; CI-NEXT:    v_div_scale_f32 v3, vcc, v0, v1, v0
+; CI-NEXT:    v_rcp_f32_e32 v4, v2
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; CI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; CI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; CI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; CI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; CI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; CI-NEXT:    s_mov_b64 s[6:7], s[10:11]
+; CI-NEXT:    v_div_fixup_f32 v2, v2, v1, v0
+; CI-NEXT:    v_trunc_f32_e32 v2, v2
+; CI-NEXT:    v_fma_f32 v0, -v2, v1, v0
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 2, 2), 0
+; CI-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; CI-NEXT:    buffer_store_short v0, off, s[4:7], 0
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: unsafe_frem_f16:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s6
+; VI-NEXT:    s_add_u32 s0, s8, 8
+; VI-NEXT:    v_mov_b32_e32 v1, s7
+; VI-NEXT:    s_addc_u32 s1, s9, 0
+; VI-NEXT:    flat_load_ushort v2, v[0:1]
+; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    v_mov_b32_e32 v1, s1
+; VI-NEXT:    flat_load_ushort v0, v[0:1]
+; VI-NEXT:    s_waitcnt vmcnt(1) lgkmcnt(1)
+; VI-NEXT:    v_cvt_f32_f16_e32 v1, v2
+; VI-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; VI-NEXT:    v_cvt_f32_f16_e32 v0, v0
+; VI-NEXT:    v_div_scale_f32 v2, s[0:1], v0, v0, v1
+; VI-NEXT:    v_div_scale_f32 v3, vcc, v1, v0, v1
+; VI-NEXT:    v_rcp_f32_e32 v4, v2
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; VI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; VI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; VI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; VI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; VI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; VI-NEXT:    v_div_fixup_f32 v2, v2, v0, v1
+; VI-NEXT:    v_trunc_f32_e32 v2, v2
+; VI-NEXT:    v_fma_f32 v0, -v2, v0, v1
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 2, 2), 0
+; VI-NEXT:    v_cvt_f16_f32_e32 v2, v0
+; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    v_mov_b32_e32 v1, s5
+; VI-NEXT:    flat_store_short v[0:1], v2
+; VI-NEXT:    s_endpgm
+   %gep2 = getelementptr half, half addrspace(1)* %in2, i32 4
+   %r0 = load half, half addrspace(1)* %in1, align 4
+   %r1 = load half, half addrspace(1)* %gep2, align 4
+   %r2 = frem half %r0, %r1
+   store half %r2, half addrspace(1)* %out, align 4
+   ret void
+}
+
+define amdgpu_kernel void @frem_f32(float addrspace(1)* %out, float addrspace(1)* %in1, float addrspace(1)* %in2) #0 {
+; CI-LABEL: frem_f32:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dword s2, s[6:7], 0x0
+; CI-NEXT:    s_load_dword s0, s[8:9], 0x4
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_mov_b32_e32 v0, s0
+; CI-NEXT:    v_div_scale_f32 v1, s[0:1], v0, v0, s2
+; CI-NEXT:    v_div_scale_f32 v2, vcc, s2, v0, s2
+; CI-NEXT:    v_rcp_f32_e32 v3, v1
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v4, -v1, v3, 1.0
+; CI-NEXT:    v_fma_f32 v3, v4, v3, v3
+; CI-NEXT:    v_mul_f32_e32 v4, v2, v3
+; CI-NEXT:    v_fma_f32 v5, -v1, v4, v2
+; CI-NEXT:    v_fma_f32 v4, v5, v3, v4
+; CI-NEXT:    v_fma_f32 v1, -v1, v4, v2
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v1, v1, v3, v4
+; CI-NEXT:    s_mov_b32 s6, -1
+; CI-NEXT:    s_mov_b32 s7, 0xf000
+; CI-NEXT:    v_div_fixup_f32 v1, v1, v0, s2
+; CI-NEXT:    v_trunc_f32_e32 v1, v1
+; CI-NEXT:    v_fma_f32 v0, -v1, v0, s2
+; CI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: frem_f32:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dword s2, s[6:7], 0x0
+; VI-NEXT:    s_load_dword s0, s[8:9], 0x10
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    v_div_scale_f32 v1, s[0:1], v0, v0, s2
+; VI-NEXT:    v_div_scale_f32 v2, vcc, s2, v0, s2
+; VI-NEXT:    v_rcp_f32_e32 v3, v1
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v4, -v1, v3, 1.0
+; VI-NEXT:    v_fma_f32 v3, v4, v3, v3
+; VI-NEXT:    v_mul_f32_e32 v4, v2, v3
+; VI-NEXT:    v_fma_f32 v5, -v1, v4, v2
+; VI-NEXT:    v_fma_f32 v4, v5, v3, v4
+; VI-NEXT:    v_fma_f32 v1, -v1, v4, v2
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v1, v1, v3, v4
+; VI-NEXT:    v_div_fixup_f32 v1, v1, v0, s2
+; VI-NEXT:    v_trunc_f32_e32 v1, v1
+; VI-NEXT:    v_fma_f32 v2, -v1, v0, s2
+; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    v_mov_b32_e32 v1, s5
+; VI-NEXT:    flat_store_dword v[0:1], v2
+; VI-NEXT:    s_endpgm
+   %gep2 = getelementptr float, float addrspace(1)* %in2, i32 4
+   %r0 = load float, float addrspace(1)* %in1, align 4
+   %r1 = load float, float addrspace(1)* %gep2, align 4
+   %r2 = frem float %r0, %r1
+   store float %r2, float addrspace(1)* %out, align 4
+   ret void
+}
+
+define amdgpu_kernel void @fast_frem_f32(float addrspace(1)* %out, float addrspace(1)* %in1, float addrspace(1)* %in2) #0 {
+; CI-LABEL: fast_frem_f32:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dword s0, s[6:7], 0x0
+; CI-NEXT:    s_load_dword s1, s[8:9], 0x4
+; CI-NEXT:    s_mov_b32 s6, -1
+; CI-NEXT:    s_mov_b32 s7, 0xf000
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_mov_b32_e32 v1, s0
+; CI-NEXT:    v_rcp_f32_e32 v0, s1
+; CI-NEXT:    v_mul_f32_e32 v0, s0, v0
+; CI-NEXT:    v_trunc_f32_e32 v0, v0
+; CI-NEXT:    v_fma_f32 v0, -v0, s1, v1
+; CI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: fast_frem_f32:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dword s0, s[6:7], 0x0
+; VI-NEXT:    s_load_dword s1, s[8:9], 0x10
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v1, s0
+; VI-NEXT:    v_rcp_f32_e32 v0, s1
+; VI-NEXT:    v_mul_f32_e32 v0, s0, v0
+; VI-NEXT:    v_trunc_f32_e32 v0, v0
+; VI-NEXT:    v_fma_f32 v2, -v0, s1, v1
+; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    v_mov_b32_e32 v1, s5
+; VI-NEXT:    flat_store_dword v[0:1], v2
+; VI-NEXT:    s_endpgm
+   %gep2 = getelementptr float, float addrspace(1)* %in2, i32 4
+   %r0 = load float, float addrspace(1)* %in1, align 4
+   %r1 = load float, float addrspace(1)* %gep2, align 4
+   %r2 = frem fast float %r0, %r1
+   store float %r2, float addrspace(1)* %out, align 4
+   ret void
+}
+
+define amdgpu_kernel void @unsafe_frem_f32(float addrspace(1)* %out, float addrspace(1)* %in1, float addrspace(1)* %in2) #1 {
+; CI-LABEL: unsafe_frem_f32:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dword s2, s[6:7], 0x0
+; CI-NEXT:    s_load_dword s0, s[8:9], 0x4
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_mov_b32_e32 v0, s0
+; CI-NEXT:    v_div_scale_f32 v1, s[0:1], v0, v0, s2
+; CI-NEXT:    v_div_scale_f32 v2, vcc, s2, v0, s2
+; CI-NEXT:    v_rcp_f32_e32 v3, v1
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v4, -v1, v3, 1.0
+; CI-NEXT:    v_fma_f32 v3, v4, v3, v3
+; CI-NEXT:    v_mul_f32_e32 v4, v2, v3
+; CI-NEXT:    v_fma_f32 v5, -v1, v4, v2
+; CI-NEXT:    v_fma_f32 v4, v5, v3, v4
+; CI-NEXT:    v_fma_f32 v1, -v1, v4, v2
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v1, v1, v3, v4
+; CI-NEXT:    s_mov_b32 s6, -1
+; CI-NEXT:    s_mov_b32 s7, 0xf000
+; CI-NEXT:    v_div_fixup_f32 v1, v1, v0, s2
+; CI-NEXT:    v_trunc_f32_e32 v1, v1
+; CI-NEXT:    v_fma_f32 v0, -v1, v0, s2
+; CI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: unsafe_frem_f32:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dword s2, s[6:7], 0x0
+; VI-NEXT:    s_load_dword s0, s[8:9], 0x10
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    v_div_scale_f32 v1, s[0:1], v0, v0, s2
+; VI-NEXT:    v_div_scale_f32 v2, vcc, s2, v0, s2
+; VI-NEXT:    v_rcp_f32_e32 v3, v1
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v4, -v1, v3, 1.0
+; VI-NEXT:    v_fma_f32 v3, v4, v3, v3
+; VI-NEXT:    v_mul_f32_e32 v4, v2, v3
+; VI-NEXT:    v_fma_f32 v5, -v1, v4, v2
+; VI-NEXT:    v_fma_f32 v4, v5, v3, v4
+; VI-NEXT:    v_fma_f32 v1, -v1, v4, v2
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v1, v1, v3, v4
+; VI-NEXT:    v_div_fixup_f32 v1, v1, v0, s2
+; VI-NEXT:    v_trunc_f32_e32 v1, v1
+; VI-NEXT:    v_fma_f32 v2, -v1, v0, s2
+; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    v_mov_b32_e32 v1, s5
+; VI-NEXT:    flat_store_dword v[0:1], v2
+; VI-NEXT:    s_endpgm
+   %gep2 = getelementptr float, float addrspace(1)* %in2, i32 4
+   %r0 = load float, float addrspace(1)* %in1, align 4
+   %r1 = load float, float addrspace(1)* %gep2, align 4
+   %r2 = frem float %r0, %r1
+   store float %r2, float addrspace(1)* %out, align 4
+   ret void
+}
+
+define amdgpu_kernel void @frem_f64(double addrspace(1)* %out, double addrspace(1)* %in1, double addrspace(1)* %in2) #0 {
+; CI-LABEL: frem_f64:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
+; CI-NEXT:    s_load_dwordx2 s[2:3], s[8:9], 0x0
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_mov_b32_e32 v0, s2
+; CI-NEXT:    v_mov_b32_e32 v1, s3
+; CI-NEXT:    v_div_scale_f64 v[2:3], s[2:3], v[0:1], v[0:1], s[0:1]
+; CI-NEXT:    v_div_scale_f64 v[8:9], vcc, s[0:1], v[0:1], s[0:1]
+; CI-NEXT:    v_rcp_f64_e32 v[4:5], v[2:3]
+; CI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; CI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; CI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; CI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; CI-NEXT:    v_mul_f64 v[6:7], v[8:9], v[4:5]
+; CI-NEXT:    v_fma_f64 v[2:3], -v[2:3], v[6:7], v[8:9]
+; CI-NEXT:    v_div_fmas_f64 v[2:3], v[2:3], v[4:5], v[6:7]
+; CI-NEXT:    v_div_fixup_f64 v[2:3], v[2:3], v[0:1], s[0:1]
+; CI-NEXT:    v_trunc_f64_e32 v[2:3], v[2:3]
+; CI-NEXT:    v_fma_f64 v[0:1], -v[2:3], v[0:1], s[0:1]
+; CI-NEXT:    v_mov_b32_e32 v2, s4
+; CI-NEXT:    v_mov_b32_e32 v3, s5
+; CI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: frem_f64:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
+; VI-NEXT:    s_load_dwordx2 s[2:3], s[8:9], 0x0
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s2
+; VI-NEXT:    v_mov_b32_e32 v1, s3
+; VI-NEXT:    v_div_scale_f64 v[2:3], s[2:3], v[0:1], v[0:1], s[0:1]
+; VI-NEXT:    v_div_scale_f64 v[8:9], vcc, s[0:1], v[0:1], s[0:1]
+; VI-NEXT:    v_rcp_f64_e32 v[4:5], v[2:3]
+; VI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; VI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; VI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; VI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; VI-NEXT:    v_mul_f64 v[6:7], v[8:9], v[4:5]
+; VI-NEXT:    v_fma_f64 v[2:3], -v[2:3], v[6:7], v[8:9]
+; VI-NEXT:    v_div_fmas_f64 v[2:3], v[2:3], v[4:5], v[6:7]
+; VI-NEXT:    v_div_fixup_f64 v[2:3], v[2:3], v[0:1], s[0:1]
+; VI-NEXT:    v_trunc_f64_e32 v[2:3], v[2:3]
+; VI-NEXT:    v_fma_f64 v[0:1], -v[2:3], v[0:1], s[0:1]
+; VI-NEXT:    v_mov_b32_e32 v2, s4
+; VI-NEXT:    v_mov_b32_e32 v3, s5
+; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; VI-NEXT:    s_endpgm
+   %r0 = load double, double addrspace(1)* %in1, align 8
+   %r1 = load double, double addrspace(1)* %in2, align 8
+   %r2 = frem double %r0, %r1
+   store double %r2, double addrspace(1)* %out, align 8
+   ret void
+}
+
+define amdgpu_kernel void @fast_frem_f64(double addrspace(1)* %out, double addrspace(1)* %in1, double addrspace(1)* %in2) #0 {
+; CI-LABEL: fast_frem_f64:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
+; CI-NEXT:    s_load_dwordx2 s[2:3], s[8:9], 0x0
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_mov_b32_e32 v0, s2
+; CI-NEXT:    v_mov_b32_e32 v1, s3
+; CI-NEXT:    v_div_scale_f64 v[2:3], s[2:3], v[0:1], v[0:1], s[0:1]
+; CI-NEXT:    v_div_scale_f64 v[8:9], vcc, s[0:1], v[0:1], s[0:1]
+; CI-NEXT:    v_rcp_f64_e32 v[4:5], v[2:3]
+; CI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; CI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; CI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; CI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; CI-NEXT:    v_mul_f64 v[6:7], v[8:9], v[4:5]
+; CI-NEXT:    v_fma_f64 v[2:3], -v[2:3], v[6:7], v[8:9]
+; CI-NEXT:    v_div_fmas_f64 v[2:3], v[2:3], v[4:5], v[6:7]
+; CI-NEXT:    v_div_fixup_f64 v[2:3], v[2:3], v[0:1], s[0:1]
+; CI-NEXT:    v_trunc_f64_e32 v[2:3], v[2:3]
+; CI-NEXT:    v_fma_f64 v[0:1], -v[2:3], v[0:1], s[0:1]
+; CI-NEXT:    v_mov_b32_e32 v2, s4
+; CI-NEXT:    v_mov_b32_e32 v3, s5
+; CI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: fast_frem_f64:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
+; VI-NEXT:    s_load_dwordx2 s[2:3], s[8:9], 0x0
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s2
+; VI-NEXT:    v_mov_b32_e32 v1, s3
+; VI-NEXT:    v_div_scale_f64 v[2:3], s[2:3], v[0:1], v[0:1], s[0:1]
+; VI-NEXT:    v_div_scale_f64 v[8:9], vcc, s[0:1], v[0:1], s[0:1]
+; VI-NEXT:    v_rcp_f64_e32 v[4:5], v[2:3]
+; VI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; VI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; VI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; VI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; VI-NEXT:    v_mul_f64 v[6:7], v[8:9], v[4:5]
+; VI-NEXT:    v_fma_f64 v[2:3], -v[2:3], v[6:7], v[8:9]
+; VI-NEXT:    v_div_fmas_f64 v[2:3], v[2:3], v[4:5], v[6:7]
+; VI-NEXT:    v_div_fixup_f64 v[2:3], v[2:3], v[0:1], s[0:1]
+; VI-NEXT:    v_trunc_f64_e32 v[2:3], v[2:3]
+; VI-NEXT:    v_fma_f64 v[0:1], -v[2:3], v[0:1], s[0:1]
+; VI-NEXT:    v_mov_b32_e32 v2, s4
+; VI-NEXT:    v_mov_b32_e32 v3, s5
+; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; VI-NEXT:    s_endpgm
+   %r0 = load double, double addrspace(1)* %in1, align 8
+   %r1 = load double, double addrspace(1)* %in2, align 8
+   %r2 = frem fast double %r0, %r1
+   store double %r2, double addrspace(1)* %out, align 8
+   ret void
+}
+
+define amdgpu_kernel void @unsafe_frem_f64(double addrspace(1)* %out, double addrspace(1)* %in1,
+; CI-LABEL: unsafe_frem_f64:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
+; CI-NEXT:    s_load_dwordx2 s[2:3], s[8:9], 0x0
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_mov_b32_e32 v0, s2
+; CI-NEXT:    v_mov_b32_e32 v1, s3
+; CI-NEXT:    v_div_scale_f64 v[2:3], s[2:3], v[0:1], v[0:1], s[0:1]
+; CI-NEXT:    v_div_scale_f64 v[8:9], vcc, s[0:1], v[0:1], s[0:1]
+; CI-NEXT:    v_rcp_f64_e32 v[4:5], v[2:3]
+; CI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; CI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; CI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; CI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; CI-NEXT:    v_mul_f64 v[6:7], v[8:9], v[4:5]
+; CI-NEXT:    v_fma_f64 v[2:3], -v[2:3], v[6:7], v[8:9]
+; CI-NEXT:    v_div_fmas_f64 v[2:3], v[2:3], v[4:5], v[6:7]
+; CI-NEXT:    v_div_fixup_f64 v[2:3], v[2:3], v[0:1], s[0:1]
+; CI-NEXT:    v_trunc_f64_e32 v[2:3], v[2:3]
+; CI-NEXT:    v_fma_f64 v[0:1], -v[2:3], v[0:1], s[0:1]
+; CI-NEXT:    v_mov_b32_e32 v2, s4
+; CI-NEXT:    v_mov_b32_e32 v3, s5
+; CI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: unsafe_frem_f64:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
+; VI-NEXT:    s_load_dwordx2 s[2:3], s[8:9], 0x0
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s2
+; VI-NEXT:    v_mov_b32_e32 v1, s3
+; VI-NEXT:    v_div_scale_f64 v[2:3], s[2:3], v[0:1], v[0:1], s[0:1]
+; VI-NEXT:    v_div_scale_f64 v[8:9], vcc, s[0:1], v[0:1], s[0:1]
+; VI-NEXT:    v_rcp_f64_e32 v[4:5], v[2:3]
+; VI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; VI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; VI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; VI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; VI-NEXT:    v_mul_f64 v[6:7], v[8:9], v[4:5]
+; VI-NEXT:    v_fma_f64 v[2:3], -v[2:3], v[6:7], v[8:9]
+; VI-NEXT:    v_div_fmas_f64 v[2:3], v[2:3], v[4:5], v[6:7]
+; VI-NEXT:    v_div_fixup_f64 v[2:3], v[2:3], v[0:1], s[0:1]
+; VI-NEXT:    v_trunc_f64_e32 v[2:3], v[2:3]
+; VI-NEXT:    v_fma_f64 v[0:1], -v[2:3], v[0:1], s[0:1]
+; VI-NEXT:    v_mov_b32_e32 v2, s4
+; VI-NEXT:    v_mov_b32_e32 v3, s5
+; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; VI-NEXT:    s_endpgm
+                             double addrspace(1)* %in2) #1 {
+   %r0 = load double, double addrspace(1)* %in1, align 8
+   %r1 = load double, double addrspace(1)* %in2, align 8
+   %r2 = frem double %r0, %r1
+   store double %r2, double addrspace(1)* %out, align 8
+   ret void
+}
+
+define amdgpu_kernel void @frem_v2f16(<2 x half> addrspace(1)* %out, <2 x half> addrspace(1)* %in1, <2 x half> addrspace(1)* %in2) #0 {
+; CI-LABEL: frem_v2f16:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dword s0, s[6:7], 0x0
+; CI-NEXT:    s_load_dword s1, s[8:9], 0x4
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_cvt_f32_f16_e32 v0, s0
+; CI-NEXT:    v_cvt_f32_f16_e32 v1, s1
+; CI-NEXT:    s_lshr_b32 s2, s0, 16
+; CI-NEXT:    s_lshr_b32 s3, s1, 16
+; CI-NEXT:    v_div_scale_f32 v2, s[0:1], v1, v1, v0
+; CI-NEXT:    v_div_scale_f32 v3, vcc, v0, v1, v0
+; CI-NEXT:    v_rcp_f32_e32 v4, v2
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; CI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; CI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; CI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; CI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; CI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; CI-NEXT:    v_div_fixup_f32 v2, v2, v1, v0
+; CI-NEXT:    v_trunc_f32_e32 v2, v2
+; CI-NEXT:    v_fma_f32 v0, -v2, v1, v0
+; CI-NEXT:    v_cvt_f32_f16_e32 v1, s2
+; CI-NEXT:    v_cvt_f32_f16_e32 v2, s3
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 2, 2), 0
+; CI-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; CI-NEXT:    v_div_scale_f32 v3, s[0:1], v2, v2, v1
+; CI-NEXT:    v_div_scale_f32 v4, vcc, v1, v2, v1
+; CI-NEXT:    v_rcp_f32_e32 v5, v3
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v6, -v3, v5, 1.0
+; CI-NEXT:    v_fma_f32 v5, v6, v5, v5
+; CI-NEXT:    v_mul_f32_e32 v6, v4, v5
+; CI-NEXT:    v_fma_f32 v7, -v3, v6, v4
+; CI-NEXT:    v_fma_f32 v6, v7, v5, v6
+; CI-NEXT:    v_fma_f32 v3, -v3, v6, v4
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v3, v3, v5, v6
+; CI-NEXT:    v_bfe_u32 v0, v0, 0, 16
+; CI-NEXT:    v_div_fixup_f32 v3, v3, v2, v1
+; CI-NEXT:    v_trunc_f32_e32 v3, v3
+; CI-NEXT:    v_fma_f32 v1, -v3, v2, v1
+; CI-NEXT:    v_cvt_f16_f32_e32 v1, v1
+; CI-NEXT:    v_bfe_u32 v1, v1, 0, 16
+; CI-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; CI-NEXT:    v_or_b32_e32 v2, v0, v1
+; CI-NEXT:    v_mov_b32_e32 v0, s4
+; CI-NEXT:    v_mov_b32_e32 v1, s5
+; CI-NEXT:    flat_store_dword v[0:1], v2
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: frem_v2f16:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dword s0, s[6:7], 0x0
+; VI-NEXT:    s_load_dword s1, s[8:9], 0x10
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_cvt_f32_f16_e32 v0, s0
+; VI-NEXT:    v_cvt_f32_f16_e32 v1, s1
+; VI-NEXT:    s_lshr_b32 s2, s0, 16
+; VI-NEXT:    s_lshr_b32 s3, s1, 16
+; VI-NEXT:    v_div_scale_f32 v2, s[0:1], v1, v1, v0
+; VI-NEXT:    v_div_scale_f32 v3, vcc, v0, v1, v0
+; VI-NEXT:    v_rcp_f32_e32 v4, v2
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; VI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; VI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; VI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; VI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; VI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; VI-NEXT:    v_div_fixup_f32 v2, v2, v1, v0
+; VI-NEXT:    v_trunc_f32_e32 v2, v2
+; VI-NEXT:    v_fma_f32 v0, -v2, v1, v0
+; VI-NEXT:    v_cvt_f32_f16_e32 v1, s2
+; VI-NEXT:    v_cvt_f32_f16_e32 v2, s3
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 2, 2), 0
+; VI-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; VI-NEXT:    v_div_scale_f32 v3, s[0:1], v2, v2, v1
+; VI-NEXT:    v_div_scale_f32 v4, vcc, v1, v2, v1
+; VI-NEXT:    v_rcp_f32_e32 v5, v3
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v6, -v3, v5, 1.0
+; VI-NEXT:    v_fma_f32 v5, v6, v5, v5
+; VI-NEXT:    v_mul_f32_e32 v6, v4, v5
+; VI-NEXT:    v_fma_f32 v7, -v3, v6, v4
+; VI-NEXT:    v_fma_f32 v6, v7, v5, v6
+; VI-NEXT:    v_fma_f32 v3, -v3, v6, v4
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v3, v3, v5, v6
+; VI-NEXT:    v_div_fixup_f32 v3, v3, v2, v1
+; VI-NEXT:    v_trunc_f32_e32 v3, v3
+; VI-NEXT:    v_fma_f32 v1, -v3, v2, v1
+; VI-NEXT:    v_cvt_f16_f32_e32 v1, v1
+; VI-NEXT:    v_mov_b32_e32 v2, 16
+; VI-NEXT:    v_lshlrev_b32_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
+; VI-NEXT:    v_or_b32_sdwa v2, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    v_mov_b32_e32 v1, s5
+; VI-NEXT:    flat_store_dword v[0:1], v2
+; VI-NEXT:    s_endpgm
+   %gep2 = getelementptr <2 x half>, <2 x half> addrspace(1)* %in2, i32 4
+   %r0 = load <2 x half>, <2 x half> addrspace(1)* %in1, align 8
+   %r1 = load <2 x half>, <2 x half> addrspace(1)* %gep2, align 8
+   %r2 = frem <2 x half> %r0, %r1
+   store <2 x half> %r2, <2 x half> addrspace(1)* %out, align 8
+   ret void
+}
+
+define amdgpu_kernel void @frem_v4f16(<4 x half> addrspace(1)* %out, <4 x half> addrspace(1)* %in1, <4 x half> addrspace(1)* %in2) #0 {
+; CI-LABEL: frem_v4f16:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
+; CI-NEXT:    s_load_dwordx2 s[2:3], s[8:9], 0x8
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_cvt_f32_f16_e32 v0, s0
+; CI-NEXT:    v_cvt_f32_f16_e32 v1, s2
+; CI-NEXT:    s_lshr_b32 s8, s0, 16
+; CI-NEXT:    s_lshr_b32 s9, s1, 16
+; CI-NEXT:    s_lshr_b32 s10, s2, 16
+; CI-NEXT:    v_div_scale_f32 v2, s[6:7], v1, v1, v0
+; CI-NEXT:    s_lshr_b32 s11, s3, 16
+; CI-NEXT:    v_div_scale_f32 v3, vcc, v0, v1, v0
+; CI-NEXT:    v_rcp_f32_e32 v4, v2
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; CI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; CI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; CI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; CI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; CI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; CI-NEXT:    v_div_fixup_f32 v2, v2, v1, v0
+; CI-NEXT:    v_trunc_f32_e32 v2, v2
+; CI-NEXT:    v_fma_f32 v0, -v2, v1, v0
+; CI-NEXT:    v_cvt_f32_f16_e32 v1, s8
+; CI-NEXT:    v_cvt_f32_f16_e32 v2, s10
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 2, 2), 0
+; CI-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; CI-NEXT:    v_div_scale_f32 v3, s[6:7], v2, v2, v1
+; CI-NEXT:    v_div_scale_f32 v4, vcc, v1, v2, v1
+; CI-NEXT:    v_rcp_f32_e32 v5, v3
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v6, -v3, v5, 1.0
+; CI-NEXT:    v_fma_f32 v5, v6, v5, v5
+; CI-NEXT:    v_mul_f32_e32 v6, v4, v5
+; CI-NEXT:    v_fma_f32 v7, -v3, v6, v4
+; CI-NEXT:    v_fma_f32 v6, v7, v5, v6
+; CI-NEXT:    v_fma_f32 v3, -v3, v6, v4
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v3, v3, v5, v6
+; CI-NEXT:    v_div_fixup_f32 v3, v3, v2, v1
+; CI-NEXT:    v_trunc_f32_e32 v3, v3
+; CI-NEXT:    v_fma_f32 v1, -v3, v2, v1
+; CI-NEXT:    v_cvt_f32_f16_e32 v2, s1
+; CI-NEXT:    v_cvt_f32_f16_e32 v3, s3
+; CI-NEXT:    v_cvt_f16_f32_e32 v1, v1
+; CI-NEXT:    v_div_scale_f32 v4, s[0:1], v3, v3, v2
+; CI-NEXT:    v_div_scale_f32 v5, vcc, v2, v3, v2
+; CI-NEXT:    v_rcp_f32_e32 v6, v4
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v7, -v4, v6, 1.0
+; CI-NEXT:    v_fma_f32 v6, v7, v6, v6
+; CI-NEXT:    v_mul_f32_e32 v7, v5, v6
+; CI-NEXT:    v_fma_f32 v8, -v4, v7, v5
+; CI-NEXT:    v_fma_f32 v7, v8, v6, v7
+; CI-NEXT:    v_fma_f32 v4, -v4, v7, v5
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v4, v4, v6, v7
+; CI-NEXT:    v_div_fixup_f32 v4, v4, v3, v2
+; CI-NEXT:    v_trunc_f32_e32 v4, v4
+; CI-NEXT:    v_fma_f32 v2, -v4, v3, v2
+; CI-NEXT:    v_cvt_f32_f16_e32 v3, s9
+; CI-NEXT:    v_cvt_f32_f16_e32 v4, s11
+; CI-NEXT:    v_cvt_f16_f32_e32 v2, v2
+; CI-NEXT:    v_div_scale_f32 v5, s[0:1], v4, v4, v3
+; CI-NEXT:    v_div_scale_f32 v6, vcc, v3, v4, v3
+; CI-NEXT:    v_rcp_f32_e32 v7, v5
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v8, -v5, v7, 1.0
+; CI-NEXT:    v_fma_f32 v7, v8, v7, v7
+; CI-NEXT:    v_mul_f32_e32 v8, v6, v7
+; CI-NEXT:    v_fma_f32 v9, -v5, v8, v6
+; CI-NEXT:    v_fma_f32 v8, v9, v7, v8
+; CI-NEXT:    v_fma_f32 v5, -v5, v8, v6
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v5, v5, v7, v8
+; CI-NEXT:    v_bfe_u32 v1, v1, 0, 16
+; CI-NEXT:    v_bfe_u32 v0, v0, 0, 16
+; CI-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; CI-NEXT:    v_or_b32_e32 v0, v0, v1
+; CI-NEXT:    v_bfe_u32 v1, v2, 0, 16
+; CI-NEXT:    v_div_fixup_f32 v5, v5, v4, v3
+; CI-NEXT:    v_trunc_f32_e32 v5, v5
+; CI-NEXT:    v_fma_f32 v3, -v5, v4, v3
+; CI-NEXT:    v_cvt_f16_f32_e32 v3, v3
+; CI-NEXT:    v_bfe_u32 v2, v3, 0, 16
+; CI-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; CI-NEXT:    v_or_b32_e32 v1, v1, v2
+; CI-NEXT:    v_mov_b32_e32 v2, s4
+; CI-NEXT:    v_mov_b32_e32 v3, s5
+; CI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: frem_v4f16:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
+; VI-NEXT:    s_load_dwordx2 s[2:3], s[8:9], 0x20
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_cvt_f32_f16_e32 v0, s0
+; VI-NEXT:    v_cvt_f32_f16_e32 v1, s2
+; VI-NEXT:    s_lshr_b32 s8, s0, 16
+; VI-NEXT:    s_lshr_b32 s9, s1, 16
+; VI-NEXT:    s_lshr_b32 s10, s2, 16
+; VI-NEXT:    v_div_scale_f32 v2, s[6:7], v1, v1, v0
+; VI-NEXT:    s_lshr_b32 s11, s3, 16
+; VI-NEXT:    v_div_scale_f32 v3, vcc, v0, v1, v0
+; VI-NEXT:    v_rcp_f32_e32 v4, v2
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; VI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; VI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; VI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; VI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; VI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; VI-NEXT:    v_div_fixup_f32 v2, v2, v1, v0
+; VI-NEXT:    v_trunc_f32_e32 v2, v2
+; VI-NEXT:    v_fma_f32 v0, -v2, v1, v0
+; VI-NEXT:    v_cvt_f32_f16_e32 v1, s8
+; VI-NEXT:    v_cvt_f32_f16_e32 v2, s10
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 2, 2), 0
+; VI-NEXT:    v_cvt_f16_f32_e32 v0, v0
+; VI-NEXT:    v_div_scale_f32 v3, s[6:7], v2, v2, v1
+; VI-NEXT:    v_div_scale_f32 v4, vcc, v1, v2, v1
+; VI-NEXT:    v_rcp_f32_e32 v5, v3
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v6, -v3, v5, 1.0
+; VI-NEXT:    v_fma_f32 v5, v6, v5, v5
+; VI-NEXT:    v_mul_f32_e32 v6, v4, v5
+; VI-NEXT:    v_fma_f32 v7, -v3, v6, v4
+; VI-NEXT:    v_fma_f32 v6, v7, v5, v6
+; VI-NEXT:    v_fma_f32 v3, -v3, v6, v4
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v3, v3, v5, v6
+; VI-NEXT:    v_div_fixup_f32 v3, v3, v2, v1
+; VI-NEXT:    v_trunc_f32_e32 v3, v3
+; VI-NEXT:    v_fma_f32 v1, -v3, v2, v1
+; VI-NEXT:    v_cvt_f32_f16_e32 v2, s1
+; VI-NEXT:    v_cvt_f32_f16_e32 v3, s3
+; VI-NEXT:    v_cvt_f16_f32_e32 v1, v1
+; VI-NEXT:    v_div_scale_f32 v4, s[0:1], v3, v3, v2
+; VI-NEXT:    v_div_scale_f32 v5, vcc, v2, v3, v2
+; VI-NEXT:    v_rcp_f32_e32 v6, v4
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v7, -v4, v6, 1.0
+; VI-NEXT:    v_fma_f32 v6, v7, v6, v6
+; VI-NEXT:    v_mul_f32_e32 v7, v5, v6
+; VI-NEXT:    v_fma_f32 v8, -v4, v7, v5
+; VI-NEXT:    v_fma_f32 v7, v8, v6, v7
+; VI-NEXT:    v_fma_f32 v4, -v4, v7, v5
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v4, v4, v6, v7
+; VI-NEXT:    v_div_fixup_f32 v4, v4, v3, v2
+; VI-NEXT:    v_trunc_f32_e32 v4, v4
+; VI-NEXT:    v_fma_f32 v2, -v4, v3, v2
+; VI-NEXT:    v_cvt_f32_f16_e32 v3, s9
+; VI-NEXT:    v_cvt_f32_f16_e32 v4, s11
+; VI-NEXT:    v_cvt_f16_f32_e32 v2, v2
+; VI-NEXT:    v_div_scale_f32 v5, s[0:1], v4, v4, v3
+; VI-NEXT:    v_div_scale_f32 v6, vcc, v3, v4, v3
+; VI-NEXT:    v_rcp_f32_e32 v7, v5
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v8, -v5, v7, 1.0
+; VI-NEXT:    v_fma_f32 v7, v8, v7, v7
+; VI-NEXT:    v_mul_f32_e32 v8, v6, v7
+; VI-NEXT:    v_fma_f32 v9, -v5, v8, v6
+; VI-NEXT:    v_fma_f32 v8, v9, v7, v8
+; VI-NEXT:    v_fma_f32 v5, -v5, v8, v6
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v5, v5, v7, v8
+; VI-NEXT:    v_div_fixup_f32 v5, v5, v4, v3
+; VI-NEXT:    v_trunc_f32_e32 v5, v5
+; VI-NEXT:    v_fma_f32 v3, -v5, v4, v3
+; VI-NEXT:    v_cvt_f16_f32_e32 v3, v3
+; VI-NEXT:    v_mov_b32_e32 v4, 16
+; VI-NEXT:    v_lshlrev_b32_sdwa v1, v4, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
+; VI-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; VI-NEXT:    v_lshlrev_b32_sdwa v1, v4, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
+; VI-NEXT:    v_or_b32_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; VI-NEXT:    v_mov_b32_e32 v2, s4
+; VI-NEXT:    v_mov_b32_e32 v3, s5
+; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; VI-NEXT:    s_endpgm
+   %gep2 = getelementptr <4 x half>, <4 x half> addrspace(1)* %in2, i32 4
+   %r0 = load <4 x half>, <4 x half> addrspace(1)* %in1, align 16
+   %r1 = load <4 x half>, <4 x half> addrspace(1)* %gep2, align 16
+   %r2 = frem <4 x half> %r0, %r1
+   store <4 x half> %r2, <4 x half> addrspace(1)* %out, align 16
+   ret void
+}
+
+define amdgpu_kernel void @frem_v2f32(<2 x float> addrspace(1)* %out, <2 x float> addrspace(1)* %in1, <2 x float> addrspace(1)* %in2) #0 {
+; CI-LABEL: frem_v2f32:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
+; CI-NEXT:    s_load_dwordx2 s[2:3], s[8:9], 0x8
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_mov_b32_e32 v0, s2
+; CI-NEXT:    v_div_scale_f32 v1, s[6:7], v0, v0, s0
+; CI-NEXT:    v_div_scale_f32 v2, vcc, s0, v0, s0
+; CI-NEXT:    v_rcp_f32_e32 v3, v1
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v4, -v1, v3, 1.0
+; CI-NEXT:    v_fma_f32 v3, v4, v3, v3
+; CI-NEXT:    v_mul_f32_e32 v4, v2, v3
+; CI-NEXT:    v_fma_f32 v5, -v1, v4, v2
+; CI-NEXT:    v_fma_f32 v4, v5, v3, v4
+; CI-NEXT:    v_fma_f32 v1, -v1, v4, v2
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v1, v1, v3, v4
+; CI-NEXT:    v_div_fixup_f32 v1, v1, v0, s0
+; CI-NEXT:    v_trunc_f32_e32 v1, v1
+; CI-NEXT:    v_fma_f32 v0, -v1, v0, s0
+; CI-NEXT:    v_mov_b32_e32 v1, s3
+; CI-NEXT:    v_div_scale_f32 v2, s[2:3], v1, v1, s1
+; CI-NEXT:    v_div_scale_f32 v3, vcc, s1, v1, s1
+; CI-NEXT:    v_rcp_f32_e32 v4, v2
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; CI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; CI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; CI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; CI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; CI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; CI-NEXT:    s_mov_b32 s6, -1
+; CI-NEXT:    s_mov_b32 s7, 0xf000
+; CI-NEXT:    v_div_fixup_f32 v2, v2, v1, s1
+; CI-NEXT:    v_trunc_f32_e32 v2, v2
+; CI-NEXT:    v_fma_f32 v1, -v2, v1, s1
+; CI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: frem_v2f32:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
+; VI-NEXT:    s_load_dwordx2 s[2:3], s[8:9], 0x20
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s2
+; VI-NEXT:    v_div_scale_f32 v1, s[6:7], v0, v0, s0
+; VI-NEXT:    v_div_scale_f32 v2, vcc, s0, v0, s0
+; VI-NEXT:    v_rcp_f32_e32 v3, v1
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v4, -v1, v3, 1.0
+; VI-NEXT:    v_fma_f32 v3, v4, v3, v3
+; VI-NEXT:    v_mul_f32_e32 v4, v2, v3
+; VI-NEXT:    v_fma_f32 v5, -v1, v4, v2
+; VI-NEXT:    v_fma_f32 v4, v5, v3, v4
+; VI-NEXT:    v_fma_f32 v1, -v1, v4, v2
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v1, v1, v3, v4
+; VI-NEXT:    v_div_fixup_f32 v1, v1, v0, s0
+; VI-NEXT:    v_trunc_f32_e32 v1, v1
+; VI-NEXT:    v_fma_f32 v0, -v1, v0, s0
+; VI-NEXT:    v_mov_b32_e32 v1, s3
+; VI-NEXT:    v_div_scale_f32 v2, s[2:3], v1, v1, s1
+; VI-NEXT:    v_div_scale_f32 v3, vcc, s1, v1, s1
+; VI-NEXT:    v_rcp_f32_e32 v4, v2
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; VI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; VI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; VI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; VI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; VI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; VI-NEXT:    v_div_fixup_f32 v2, v2, v1, s1
+; VI-NEXT:    v_trunc_f32_e32 v2, v2
+; VI-NEXT:    v_fma_f32 v1, -v2, v1, s1
+; VI-NEXT:    v_mov_b32_e32 v2, s4
+; VI-NEXT:    v_mov_b32_e32 v3, s5
+; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; VI-NEXT:    s_endpgm
+   %gep2 = getelementptr <2 x float>, <2 x float> addrspace(1)* %in2, i32 4
+   %r0 = load <2 x float>, <2 x float> addrspace(1)* %in1, align 8
+   %r1 = load <2 x float>, <2 x float> addrspace(1)* %gep2, align 8
+   %r2 = frem <2 x float> %r0, %r1
+   store <2 x float> %r2, <2 x float> addrspace(1)* %out, align 8
+   ret void
+}
+
+define amdgpu_kernel void @frem_v4f32(<4 x float> addrspace(1)* %out, <4 x float> addrspace(1)* %in1, <4 x float> addrspace(1)* %in2) #0 {
+; CI-LABEL: frem_v4f32:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dwordx4 s[0:3], s[6:7], 0x0
+; CI-NEXT:    s_load_dwordx4 s[8:11], s[8:9], 0x10
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_mov_b32_e32 v0, s8
+; CI-NEXT:    v_div_scale_f32 v1, s[6:7], v0, v0, s0
+; CI-NEXT:    v_div_scale_f32 v2, vcc, s0, v0, s0
+; CI-NEXT:    v_rcp_f32_e32 v3, v1
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v4, -v1, v3, 1.0
+; CI-NEXT:    v_fma_f32 v3, v4, v3, v3
+; CI-NEXT:    v_mul_f32_e32 v4, v2, v3
+; CI-NEXT:    v_fma_f32 v5, -v1, v4, v2
+; CI-NEXT:    v_fma_f32 v4, v5, v3, v4
+; CI-NEXT:    v_fma_f32 v1, -v1, v4, v2
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v1, v1, v3, v4
+; CI-NEXT:    v_div_fixup_f32 v1, v1, v0, s0
+; CI-NEXT:    v_trunc_f32_e32 v1, v1
+; CI-NEXT:    v_fma_f32 v0, -v1, v0, s0
+; CI-NEXT:    v_mov_b32_e32 v1, s9
+; CI-NEXT:    v_div_scale_f32 v2, s[6:7], v1, v1, s1
+; CI-NEXT:    v_div_scale_f32 v3, vcc, s1, v1, s1
+; CI-NEXT:    v_rcp_f32_e32 v4, v2
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; CI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; CI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; CI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; CI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; CI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; CI-NEXT:    v_div_fixup_f32 v2, v2, v1, s1
+; CI-NEXT:    v_trunc_f32_e32 v2, v2
+; CI-NEXT:    v_fma_f32 v1, -v2, v1, s1
+; CI-NEXT:    v_mov_b32_e32 v2, s10
+; CI-NEXT:    v_div_scale_f32 v3, s[0:1], v2, v2, s2
+; CI-NEXT:    v_div_scale_f32 v4, vcc, s2, v2, s2
+; CI-NEXT:    v_rcp_f32_e32 v5, v3
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v6, -v3, v5, 1.0
+; CI-NEXT:    v_fma_f32 v5, v6, v5, v5
+; CI-NEXT:    v_mul_f32_e32 v6, v4, v5
+; CI-NEXT:    v_fma_f32 v7, -v3, v6, v4
+; CI-NEXT:    v_fma_f32 v6, v7, v5, v6
+; CI-NEXT:    v_fma_f32 v3, -v3, v6, v4
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v3, v3, v5, v6
+; CI-NEXT:    v_div_fixup_f32 v3, v3, v2, s2
+; CI-NEXT:    v_trunc_f32_e32 v3, v3
+; CI-NEXT:    v_fma_f32 v2, -v3, v2, s2
+; CI-NEXT:    v_mov_b32_e32 v3, s11
+; CI-NEXT:    v_div_scale_f32 v4, s[0:1], v3, v3, s3
+; CI-NEXT:    v_div_scale_f32 v5, vcc, s3, v3, s3
+; CI-NEXT:    v_rcp_f32_e32 v6, v4
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; CI-NEXT:    v_fma_f32 v7, -v4, v6, 1.0
+; CI-NEXT:    v_fma_f32 v6, v7, v6, v6
+; CI-NEXT:    v_mul_f32_e32 v7, v5, v6
+; CI-NEXT:    v_fma_f32 v8, -v4, v7, v5
+; CI-NEXT:    v_fma_f32 v7, v8, v6, v7
+; CI-NEXT:    v_fma_f32 v4, -v4, v7, v5
+; CI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; CI-NEXT:    v_div_fmas_f32 v4, v4, v6, v7
+; CI-NEXT:    s_mov_b32 s6, -1
+; CI-NEXT:    s_mov_b32 s7, 0xf000
+; CI-NEXT:    v_div_fixup_f32 v4, v4, v3, s3
+; CI-NEXT:    v_trunc_f32_e32 v4, v4
+; CI-NEXT:    v_fma_f32 v3, -v4, v3, s3
+; CI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: frem_v4f32:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dwordx4 s[0:3], s[6:7], 0x0
+; VI-NEXT:    s_load_dwordx4 s[8:11], s[8:9], 0x40
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s8
+; VI-NEXT:    v_div_scale_f32 v1, s[6:7], v0, v0, s0
+; VI-NEXT:    v_div_scale_f32 v2, vcc, s0, v0, s0
+; VI-NEXT:    v_rcp_f32_e32 v3, v1
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v4, -v1, v3, 1.0
+; VI-NEXT:    v_fma_f32 v3, v4, v3, v3
+; VI-NEXT:    v_mul_f32_e32 v4, v2, v3
+; VI-NEXT:    v_fma_f32 v5, -v1, v4, v2
+; VI-NEXT:    v_fma_f32 v4, v5, v3, v4
+; VI-NEXT:    v_fma_f32 v1, -v1, v4, v2
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v1, v1, v3, v4
+; VI-NEXT:    v_div_fixup_f32 v1, v1, v0, s0
+; VI-NEXT:    v_trunc_f32_e32 v1, v1
+; VI-NEXT:    v_fma_f32 v0, -v1, v0, s0
+; VI-NEXT:    v_mov_b32_e32 v1, s9
+; VI-NEXT:    v_div_scale_f32 v2, s[6:7], v1, v1, s1
+; VI-NEXT:    v_div_scale_f32 v3, vcc, s1, v1, s1
+; VI-NEXT:    v_rcp_f32_e32 v4, v2
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v5, -v2, v4, 1.0
+; VI-NEXT:    v_fma_f32 v4, v5, v4, v4
+; VI-NEXT:    v_mul_f32_e32 v5, v3, v4
+; VI-NEXT:    v_fma_f32 v6, -v2, v5, v3
+; VI-NEXT:    v_fma_f32 v5, v6, v4, v5
+; VI-NEXT:    v_fma_f32 v2, -v2, v5, v3
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v2, v2, v4, v5
+; VI-NEXT:    v_div_fixup_f32 v2, v2, v1, s1
+; VI-NEXT:    v_trunc_f32_e32 v2, v2
+; VI-NEXT:    v_fma_f32 v1, -v2, v1, s1
+; VI-NEXT:    v_mov_b32_e32 v2, s10
+; VI-NEXT:    v_div_scale_f32 v3, s[0:1], v2, v2, s2
+; VI-NEXT:    v_div_scale_f32 v4, vcc, s2, v2, s2
+; VI-NEXT:    v_rcp_f32_e32 v5, v3
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v6, -v3, v5, 1.0
+; VI-NEXT:    v_fma_f32 v5, v6, v5, v5
+; VI-NEXT:    v_mul_f32_e32 v6, v4, v5
+; VI-NEXT:    v_fma_f32 v7, -v3, v6, v4
+; VI-NEXT:    v_fma_f32 v6, v7, v5, v6
+; VI-NEXT:    v_fma_f32 v3, -v3, v6, v4
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v3, v3, v5, v6
+; VI-NEXT:    v_div_fixup_f32 v3, v3, v2, s2
+; VI-NEXT:    v_trunc_f32_e32 v3, v3
+; VI-NEXT:    v_fma_f32 v2, -v3, v2, s2
+; VI-NEXT:    v_mov_b32_e32 v3, s11
+; VI-NEXT:    v_div_scale_f32 v4, s[0:1], v3, v3, s3
+; VI-NEXT:    v_div_scale_f32 v5, vcc, s3, v3, s3
+; VI-NEXT:    v_rcp_f32_e32 v6, v4
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; VI-NEXT:    v_fma_f32 v7, -v4, v6, 1.0
+; VI-NEXT:    v_fma_f32 v6, v7, v6, v6
+; VI-NEXT:    v_mul_f32_e32 v7, v5, v6
+; VI-NEXT:    v_fma_f32 v8, -v4, v7, v5
+; VI-NEXT:    v_fma_f32 v7, v8, v6, v7
+; VI-NEXT:    v_fma_f32 v4, -v4, v7, v5
+; VI-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; VI-NEXT:    v_div_fmas_f32 v4, v4, v6, v7
+; VI-NEXT:    v_div_fixup_f32 v4, v4, v3, s3
+; VI-NEXT:    v_trunc_f32_e32 v4, v4
+; VI-NEXT:    v_fma_f32 v3, -v4, v3, s3
+; VI-NEXT:    v_mov_b32_e32 v4, s4
+; VI-NEXT:    v_mov_b32_e32 v5, s5
+; VI-NEXT:    flat_store_dwordx4 v[4:5], v[0:3]
+; VI-NEXT:    s_endpgm
+   %gep2 = getelementptr <4 x float>, <4 x float> addrspace(1)* %in2, i32 4
+   %r0 = load <4 x float>, <4 x float> addrspace(1)* %in1, align 16
+   %r1 = load <4 x float>, <4 x float> addrspace(1)* %gep2, align 16
+   %r2 = frem <4 x float> %r0, %r1
+   store <4 x float> %r2, <4 x float> addrspace(1)* %out, align 16
+   ret void
+}
+
+define amdgpu_kernel void @frem_v2f64(<2 x double> addrspace(1)* %out, <2 x double> addrspace(1)* %in1, <2 x double> addrspace(1)* %in2) #0 {
+; CI-LABEL: frem_v2f64:
+; CI:       ; %bb.0:
+; CI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; CI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xd
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    s_load_dwordx4 s[0:3], s[6:7], 0x0
+; CI-NEXT:    s_load_dwordx4 s[8:11], s[8:9], 0x10
+; CI-NEXT:    s_waitcnt lgkmcnt(0)
+; CI-NEXT:    v_mov_b32_e32 v0, s8
+; CI-NEXT:    v_mov_b32_e32 v1, s9
+; CI-NEXT:    v_div_scale_f64 v[2:3], s[6:7], v[0:1], v[0:1], s[0:1]
+; CI-NEXT:    v_div_scale_f64 v[8:9], vcc, s[0:1], v[0:1], s[0:1]
+; CI-NEXT:    v_rcp_f64_e32 v[4:5], v[2:3]
+; CI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; CI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; CI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; CI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; CI-NEXT:    v_mul_f64 v[6:7], v[8:9], v[4:5]
+; CI-NEXT:    v_fma_f64 v[2:3], -v[2:3], v[6:7], v[8:9]
+; CI-NEXT:    v_div_fmas_f64 v[2:3], v[2:3], v[4:5], v[6:7]
+; CI-NEXT:    v_div_fixup_f64 v[2:3], v[2:3], v[0:1], s[0:1]
+; CI-NEXT:    v_trunc_f64_e32 v[2:3], v[2:3]
+; CI-NEXT:    v_fma_f64 v[0:1], -v[2:3], v[0:1], s[0:1]
+; CI-NEXT:    v_mov_b32_e32 v2, s10
+; CI-NEXT:    v_mov_b32_e32 v3, s11
+; CI-NEXT:    v_div_scale_f64 v[4:5], s[0:1], v[2:3], v[2:3], s[2:3]
+; CI-NEXT:    v_div_scale_f64 v[10:11], vcc, s[2:3], v[2:3], s[2:3]
+; CI-NEXT:    v_rcp_f64_e32 v[6:7], v[4:5]
+; CI-NEXT:    v_fma_f64 v[8:9], -v[4:5], v[6:7], 1.0
+; CI-NEXT:    v_fma_f64 v[6:7], v[6:7], v[8:9], v[6:7]
+; CI-NEXT:    v_fma_f64 v[8:9], -v[4:5], v[6:7], 1.0
+; CI-NEXT:    v_fma_f64 v[6:7], v[6:7], v[8:9], v[6:7]
+; CI-NEXT:    v_mul_f64 v[8:9], v[10:11], v[6:7]
+; CI-NEXT:    v_fma_f64 v[4:5], -v[4:5], v[8:9], v[10:11]
+; CI-NEXT:    v_div_fmas_f64 v[4:5], v[4:5], v[6:7], v[8:9]
+; CI-NEXT:    v_div_fixup_f64 v[4:5], v[4:5], v[2:3], s[2:3]
+; CI-NEXT:    v_trunc_f64_e32 v[4:5], v[4:5]
+; CI-NEXT:    v_fma_f64 v[2:3], -v[4:5], v[2:3], s[2:3]
+; CI-NEXT:    v_mov_b32_e32 v4, s4
+; CI-NEXT:    v_mov_b32_e32 v5, s5
+; CI-NEXT:    flat_store_dwordx4 v[4:5], v[0:3]
+; CI-NEXT:    s_endpgm
+;
+; VI-LABEL: frem_v2f64:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_load_dwordx4 s[0:3], s[6:7], 0x0
+; VI-NEXT:    s_load_dwordx4 s[8:11], s[8:9], 0x40
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    v_mov_b32_e32 v0, s8
+; VI-NEXT:    v_mov_b32_e32 v1, s9
+; VI-NEXT:    v_div_scale_f64 v[2:3], s[6:7], v[0:1], v[0:1], s[0:1]
+; VI-NEXT:    v_div_scale_f64 v[8:9], vcc, s[0:1], v[0:1], s[0:1]
+; VI-NEXT:    v_rcp_f64_e32 v[4:5], v[2:3]
+; VI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; VI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; VI-NEXT:    v_fma_f64 v[6:7], -v[2:3], v[4:5], 1.0
+; VI-NEXT:    v_fma_f64 v[4:5], v[4:5], v[6:7], v[4:5]
+; VI-NEXT:    v_mul_f64 v[6:7], v[8:9], v[4:5]
+; VI-NEXT:    v_fma_f64 v[2:3], -v[2:3], v[6:7], v[8:9]
+; VI-NEXT:    v_div_fmas_f64 v[2:3], v[2:3], v[4:5], v[6:7]
+; VI-NEXT:    v_div_fixup_f64 v[2:3], v[2:3], v[0:1], s[0:1]
+; VI-NEXT:    v_trunc_f64_e32 v[2:3], v[2:3]
+; VI-NEXT:    v_fma_f64 v[0:1], -v[2:3], v[0:1], s[0:1]
+; VI-NEXT:    v_mov_b32_e32 v2, s10
+; VI-NEXT:    v_mov_b32_e32 v3, s11
+; VI-NEXT:    v_div_scale_f64 v[4:5], s[0:1], v[2:3], v[2:3], s[2:3]
+; VI-NEXT:    v_div_scale_f64 v[10:11], vcc, s[2:3], v[2:3], s[2:3]
+; VI-NEXT:    v_rcp_f64_e32 v[6:7], v[4:5]
+; VI-NEXT:    v_fma_f64 v[8:9], -v[4:5], v[6:7], 1.0
+; VI-NEXT:    v_fma_f64 v[6:7], v[6:7], v[8:9], v[6:7]
+; VI-NEXT:    v_fma_f64 v[8:9], -v[4:5], v[6:7], 1.0
+; VI-NEXT:    v_fma_f64 v[6:7], v[6:7], v[8:9], v[6:7]
+; VI-NEXT:    v_mul_f64 v[8:9], v[10:11], v[6:7]
+; VI-NEXT:    v_fma_f64 v[4:5], -v[4:5], v[8:9], v[10:11]
+; VI-NEXT:    v_div_fmas_f64 v[4:5], v[4:5], v[6:7], v[8:9]
+; VI-NEXT:    v_div_fixup_f64 v[4:5], v[4:5], v[2:3], s[2:3]
+; VI-NEXT:    v_trunc_f64_e32 v[4:5], v[4:5]
+; VI-NEXT:    v_fma_f64 v[2:3], -v[4:5], v[2:3], s[2:3]
+; VI-NEXT:    v_mov_b32_e32 v4, s4
+; VI-NEXT:    v_mov_b32_e32 v5, s5
+; VI-NEXT:    flat_store_dwordx4 v[4:5], v[0:3]
+; VI-NEXT:    s_endpgm
+   %gep2 = getelementptr <2 x double>, <2 x double> addrspace(1)* %in2, i32 4
+   %r0 = load <2 x double>, <2 x double> addrspace(1)* %in1, align 16
+   %r1 = load <2 x double>, <2 x double> addrspace(1)* %gep2, align 16
+   %r2 = frem <2 x double> %r0, %r1
+   store <2 x double> %r2, <2 x double> addrspace(1)* %out, align 16
+   ret void
+}
+
+attributes #0 = { nounwind "unsafe-fp-math"="false" "denormal-fp-math-f32"="preserve-sign,preserve-sign" }
+attributes #1 = { nounwind "unsafe-fp-math"="true" "denormal-fp-math-f32"="preserve-sign,preserve-sign" }