AMDGPU/GlobalISel: Do not fcanonicalize const splat padded with undef

[Petar.Avramovic]

Summary

Recognize constant splat padded with undef in isCanonicalized.
Fcanonicalize will be removed by RemoveFcanonicalize in post-legalizer
combiner. We will treat undef as value that will result in a splat
in clamp combine after regbankselect.

[arsenm]

This is incorrect. Undef could be anything, so it could be a nan

[Petar.Avramovic]

This is required to match splat that was padded with undef into clamp D90052. When undef is used to pad with element that does not affect the result we interpret it in a way to clamp other elements.

[arsenm]

In D104408#2822733, @Petar.Avramovic wrote:

This is required to match splat that was padded with undef into clamp D90052. When undef is used to pad with element that does not affect the result we interpret it in a way to clamp other elements.

This at least needs a parameter to ignore undefs. It's not really true to the name of the function otherwise

[foad]

matchUnaryPredicate already has an AllowUndefs argument, so you could do the same.

Incidentally ValueTracking's isKnownNeverNaN treats undef as not-nan, but only when it appears as an element of a vector constant.

[Petar.Avramovic]

Move to isCanonicalized and post-legalizer combiner.

[foad]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

9723

What if SplatReg is a signaling nan here?

[Petar.Avramovic]

Fix cases of SNaN and QNaN constant splats padded with undef, also add test for them.

[foad]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

9723

What about NaNs? This seems like it should be combined with the G_FCONSTANT case above.

[Petar.Avramovic]

Merge with G_FCONSTANT case above.

[arsenm]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

9714

If the matcher is finding it's a constant, why does it return the register instead of the direct APFloat?

[Petar.Avramovic]

Use matcher that returns both register and APFloat from G_FCONSTANT.

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

 #include "SIMachineFunctionInfo.h"
 #include "SIRegisterInfo.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/LegacyDivergenceAnalysis.h"
 #include "llvm/BinaryFormat/ELF.h"
 #include "llvm/CodeGen/Analysis.h"
 #include "llvm/CodeGen/FunctionLoweringInfo.h"
 #include "llvm/CodeGen/GlobalISel/GISelKnownBits.h"
+#include "llvm/CodeGen/GlobalISel/MIPatternMatch.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/IntrinsicsAMDGPU.h"
 #include "llvm/IR/IntrinsicsR600.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/KnownBits.h"
 
bool SITargetLowering::isCanonicalized(Register Reg, MachineFunction &MF,
                                        unsigned MaxDepth) const {
   MachineRegisterInfo &MRI = MF.getRegInfo();
   MachineInstr *MI = MRI.getVRegDef(Reg);
   unsigned Opcode = MI->getOpcode();
 
   if (Opcode == AMDGPU::G_FCANONICALIZE)
     return true;
 
-  if (Opcode == AMDGPU::G_FCONSTANT) {
-    auto F = MI->getOperand(1).getFPImm()->getValueAPF();
-    if (F.isNaN() && F.isSignaling())
+  Register FCst;
+  // Constant splat (can be padded with undef) or scalar constant.
+  if (mi_match(Reg, MRI, MIPatternMatch::FCstOrSplatFCstRegMatch(FCst))) {
+    auto F = getConstantFPVRegVal(FCst, MRI)->getValueAPF();
+    if (F.isSignaling())
       return false;
-    return !F.isDenormal() || denormalsEnabledForType(MRI.getType(Reg), MF);
+    return !F.isDenormal() || denormalsEnabledForType(MRI.getType(FCst), MF);
   }
 
   if (MaxDepth == 0)
     return false;
 
   switch (Opcode) {
   case AMDGPU::G_FMINNUM_IEEE:
   case AMDGPU::G_FMAXNUM_IEEE: {
     if (Subtarget->supportsMinMaxDenormModes() ||
         denormalsEnabledForType(MRI.getType(Reg), MF))
       return true;
     for (unsigned I = 1, E = MI->getNumOperands(); I != E; ++I) {
       if (!isCanonicalized(MI->getOperand(I).getReg(), MF, MaxDepth - 1))
         return false;

llvm/test/CodeGen/AMDGPU/GlobalISel/combine-fcanonicalize.mir

bb.0:
     %6:_(s32) = G_FCANONICALIZE %0
     %7:_(s32) = G_FCANONICALIZE %1
     %2:_(s32) = G_FMINNUM_IEEE %6, %7
     %4:_(s32) = G_FCANONICALIZE %2
     %5:_(s32) = G_FCANONICALIZE %2
     %3:_(s32) = G_FMAXNUM_IEEE %4, %5
     $vgpr0 = COPY %3(s32)
 ...
+
+---
+name: test_splat_padded_with_undef
+tracksRegLiveness: true
+legalized: true
+machineFunctionInfo:
+  mode:
+    ieee: true
+body: |
+  bb.0 :
+    liveins: $vgpr0
+
+    ; CHECK-LABEL: name: test_splat_padded_with_undef
+    ; CHECK: liveins: $vgpr0
+    ; CHECK: [[COPY:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr0
+    ; CHECK: [[C:%[0-9]+]]:_(s16) = G_FCONSTANT half 0xH4000
+    ; CHECK: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[C]](s16)
+    ; CHECK: [[BUILD_VECTOR_TRUNC:%[0-9]+]]:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC [[ANYEXT]](s32), [[ANYEXT]](s32)
+    ; CHECK: [[C1:%[0-9]+]]:_(s16) = G_FCONSTANT half 0xH0000
+    ; CHECK: [[ANYEXT1:%[0-9]+]]:_(s32) = G_ANYEXT [[C1]](s16)
+    ; CHECK: [[DEF:%[0-9]+]]:_(s32) = G_IMPLICIT_DEF
+    ; CHECK: [[BUILD_VECTOR_TRUNC1:%[0-9]+]]:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC [[ANYEXT1]](s32), [[DEF]](s32)
+    ; CHECK: [[C2:%[0-9]+]]:_(s16) = G_FCONSTANT half 0xH3C00
+    ; CHECK: [[ANYEXT2:%[0-9]+]]:_(s32) = G_ANYEXT [[C2]](s16)
+    ; CHECK: [[BUILD_VECTOR_TRUNC2:%[0-9]+]]:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC [[ANYEXT2]](s32), [[ANYEXT2]](s32)
+    ; CHECK: [[FMUL:%[0-9]+]]:_(<2 x s16>) = G_FMUL [[COPY]], [[BUILD_VECTOR_TRUNC]]
+    ; CHECK: [[FCANONICALIZE:%[0-9]+]]:_(<2 x s16>) = G_FCANONICALIZE [[FMUL]]
+    ; CHECK: [[FMAXNUM_IEEE:%[0-9]+]]:_(<2 x s16>) = G_FMAXNUM_IEEE [[BUILD_VECTOR_TRUNC1]], [[FCANONICALIZE]]
+    ; CHECK: [[FMINNUM_IEEE:%[0-9]+]]:_(<2 x s16>) = G_FMINNUM_IEEE [[BUILD_VECTOR_TRUNC2]], [[FMAXNUM_IEEE]]
+    ; CHECK: $vgpr0 = COPY [[FMINNUM_IEEE]](<2 x s16>)
+    %0:_(<2 x s16>) = COPY $vgpr0
+    %3:_(s16) = G_FCONSTANT half 0xH4000
+    %16:_(s32) = G_ANYEXT %3(s16)
+    %2:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC %16(s32), %16(s32)
+    %6:_(s16) = G_FCONSTANT half 0xH0000
+    %17:_(s32) = G_ANYEXT %6(s16)
+    %18:_(s32) = G_IMPLICIT_DEF
+    %5:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC %17(s32), %18(s32)
+    %10:_(s16) = G_FCONSTANT half 0xH3C00
+    %19:_(s32) = G_ANYEXT %10(s16)
+    %9:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC %19(s32), %19(s32)
+    %4:_(<2 x s16>) = G_FMUL %0, %2
+    %14:_(<2 x s16>) = G_FCANONICALIZE %5
+    %15:_(<2 x s16>) = G_FCANONICALIZE %4
+    %8:_(<2 x s16>) = G_FMAXNUM_IEEE %14, %15
+    %13:_(<2 x s16>) = G_FCANONICALIZE %8
+    %11:_(<2 x s16>) = G_FMINNUM_IEEE %9, %13
+    $vgpr0 = COPY %11(<2 x s16>)
+...
+
+---
+# 0xH7E01 is SNaN and 0xH7E01 is QNaN
+name: test_splat_SNaN_and_QNaN_padded_with_undef
+tracksRegLiveness: true
+legalized: true
+machineFunctionInfo:
+  mode:
+    ieee: true
+body: |
+  bb.0 :
+    liveins: $vgpr0
+
+    ; CHECK-LABEL: name: test_splat_SNaN_and_QNaN_padded_with_undef
+    ; CHECK: liveins: $vgpr0
+    ; CHECK: [[COPY:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr0
+    ; CHECK: [[C:%[0-9]+]]:_(s16) = G_FCONSTANT half 0xH4000
+    ; CHECK: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[C]](s16)
+    ; CHECK: [[BUILD_VECTOR_TRUNC:%[0-9]+]]:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC [[ANYEXT]](s32), [[ANYEXT]](s32)
+    ; CHECK: [[C1:%[0-9]+]]:_(s16) = G_FCONSTANT half 0xH7C01
+    ; CHECK: [[ANYEXT1:%[0-9]+]]:_(s32) = G_ANYEXT [[C1]](s16)
+    ; CHECK: [[DEF:%[0-9]+]]:_(s32) = G_IMPLICIT_DEF
+    ; CHECK: [[BUILD_VECTOR_TRUNC1:%[0-9]+]]:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC [[ANYEXT1]](s32), [[DEF]](s32)
+    ; CHECK: [[C2:%[0-9]+]]:_(s16) = G_FCONSTANT half 0xH7E01
+    ; CHECK: [[ANYEXT2:%[0-9]+]]:_(s32) = G_ANYEXT [[C2]](s16)
+    ; CHECK: [[BUILD_VECTOR_TRUNC2:%[0-9]+]]:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC [[ANYEXT2]](s32), [[ANYEXT2]](s32)
+    ; CHECK: [[FMUL:%[0-9]+]]:_(<2 x s16>) = G_FMUL [[COPY]], [[BUILD_VECTOR_TRUNC]]
+    ; CHECK: [[FCANONICALIZE:%[0-9]+]]:_(<2 x s16>) = G_FCANONICALIZE [[BUILD_VECTOR_TRUNC1]]
+    ; CHECK: [[FCANONICALIZE1:%[0-9]+]]:_(<2 x s16>) = G_FCANONICALIZE [[FMUL]]
+    ; CHECK: [[FMAXNUM_IEEE:%[0-9]+]]:_(<2 x s16>) = G_FMAXNUM_IEEE [[FCANONICALIZE]], [[FCANONICALIZE1]]
+    ; CHECK: [[FMINNUM_IEEE:%[0-9]+]]:_(<2 x s16>) = G_FMINNUM_IEEE [[BUILD_VECTOR_TRUNC2]], [[FMAXNUM_IEEE]]
+    ; CHECK: $vgpr0 = COPY [[FMINNUM_IEEE]](<2 x s16>)
+    %0:_(<2 x s16>) = COPY $vgpr0
+    %3:_(s16) = G_FCONSTANT half 0xH4000
+    %16:_(s32) = G_ANYEXT %3(s16)
+    %2:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC %16(s32), %16(s32)
+    %6:_(s16) = G_FCONSTANT half 0xH7C01
+    %17:_(s32) = G_ANYEXT %6(s16)
+    %18:_(s32) = G_IMPLICIT_DEF
+    %5:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC %17(s32), %18(s32)
+    %10:_(s16) = G_FCONSTANT half 0xH7E01
+    %19:_(s32) = G_ANYEXT %10(s16)
+    %9:_(<2 x s16>) = G_BUILD_VECTOR_TRUNC %19(s32), %19(s32)
+    %4:_(<2 x s16>) = G_FMUL %0, %2
+    %14:_(<2 x s16>) = G_FCANONICALIZE %5
+    %15:_(<2 x s16>) = G_FCANONICALIZE %4
+    %8:_(<2 x s16>) = G_FMAXNUM_IEEE %14, %15
+    %13:_(<2 x s16>) = G_FCANONICALIZE %8
+    %11:_(<2 x s16>) = G_FMINNUM_IEEE %9, %13
+    $vgpr0 = COPY %11(<2 x s16>)
+...