Use llvm.is_fpclass to implement FP classification functions

[sepavloff]

Summary

Builtin floating-point number classification functions:

• __builtin_isnan,
• __builtin_isinf,
• __builtin_finite, and
• __builtin_isnormal

now are implemented using llvm.is_fpclass. New builtin functions were
added:

• __builtin_issubnormal,
• __builtin_iszero, and
• __builtin_issignaling.

They represent corresponding standard C library functions and are also
implemented using llvm.is_fpclass.

A new builting function __builtin_isfpclass is added. It is called as:

• __builtin_isfpclass(<test>, <floating point value>)

and returns an integer value, which is non-zero if the floating point
argument belongs to one of the classes specified by the first argument,
and zero otherwise. The set of classes is an integer value, where each
value class is represented by a bit. There are ten data classes, as
defined by the IEEE-754 standard.

This change makes the target callback TargetCodeGenInfo::testFPKind
unneeded. It is preserved in this change and should be removed later.

[sepavloff]

Updated patch because the base revision is updated

[qiucf]

This patch looks good and llvm.is.fpclass will by default be expanded (except SystemZ which has their own lowering). Is there any blocker for this?

[sepavloff]

Prepare the patch for review

In this patch the callback TargetCodeGenInfo::testFPKind is
preserved, to make this change safer.

[kpn]

clang/test/CodeGen/strictfp_builtins.c

160 ↗

(On Diff #481901)

Is there a way to test that we're using llvm.is.fpclass() in this case?

[sepavloff]

clang/test/CodeGen/strictfp_builtins.c

160 ↗

(On Diff #481901)

This is a bit different case than binary predicate. isinf_sign returns 0, 1 or -1. It could be implemented using calls of is_fpclass and signbit, but the implementation should be efficient for various targets and for vector data also. It is worth a separate patch.

[arsenm]

I think the __builtin_isfpclass part should be a separate patch from the rest of the functions. Also needs sema tests for the handling of the immediate argument

clang/include/clang/Basic/Builtins.def

471

Should use I to mark the isfpclass test mask given you don't try to handle the variable case

clang/lib/CodeGen/CGBuiltin.cpp

3154–3155

Should enforce constant argument with "I" builtin constraint

[sepavloff]

Remove __builtin_isfpclass

[arsenm]

This will produce worse codegen in many situations. I'm working on a stack of patches to convert is.fpclass back to fcmp when legal

[sepavloff]

Update patch

• Rebase,
• Add test that checks inlining.

[arsenm]

clang/include/clang/Basic/Builtins.def

468–470

Sorry, I should have clarified, all the new builtins should be added separately from the change of the implementation of the existing builtins

[sepavloff]

Remove support of __builtin_is{subnormal,zero,signaling}, rebased

[arsenm]

This change itself LGTM but I think it should wait until after we get more optimizations in to go back to fcmp, and after the release branch

[arsenm]

In D112932#4043545, @arsenm wrote:

This change itself LGTM but I think it should wait until after we get more optimizations in to go back to fcmp, and after the release branch

I think these optimizations are mostly in a good state. I still have some outstanding is.fpclass codegen patches (e.g. D143201 and related)

[arsenm]

In D112932#4005339, @sepavloff wrote:

Remove __builtin_isfpclass

I am interested in adding a __builtin_isfpclass, just in a separate patch

[arsenm]

clang/lib/CodeGen/CGBuiltin.cpp

2253–2255

Might as well put this in IRBuilder

[sepavloff]

Add createIsFPClass to IRBuilder

[arsenm]

LGTM

clang/test/CodeGen/isfpclass.c

2 ↗

(On Diff #529156)

While already covered by existing tests, it might be nicer to test all the builtins in the same file with and without fenv access enabled

[arsenm]

Description now needs updating since this diff no longer adds all the split out pieces

[qiucf]

Is there any blocker for this to land?

clang/include/clang/Basic/Builtins.def

 BUILTIN(__builtin_isgreaterequal, "i.", "Fnct")
 BUILTIN(__builtin_isless        , "i.", "Fnct")
 BUILTIN(__builtin_islessequal   , "i.", "Fnct")
 BUILTIN(__builtin_islessgreater , "i.", "Fnct")
 BUILTIN(__builtin_isunordered   , "i.", "Fnct")
 
 // Unary FP classification
 BUILTIN(__builtin_fpclassify, "iiiiii.", "Fnct")
 BUILTIN(__builtin_isfinite,   "i.", "Fnct")
 BUILTIN(__builtin_isinf,      "i.", "Fnct")
 BUILTIN(__builtin_isinf_sign, "i.", "Fnct")
 BUILTIN(__builtin_isnan,      "i.", "Fnct")
 BUILTIN(__builtin_isnormal,   "i.", "Fnct")
+BUILTIN(__builtin_issubnormal, "i.", "Fnct")
+BUILTIN(__builtin_iszero,      "i.", "Fnct")
+BUILTIN(__builtin_issignaling, "i.", "Fnct")
+BUILTIN(__builtin_isfpclass,   "iCi.", "Fnct")
 
 // FP signbit builtins
 BUILTIN(__builtin_signbit, "i.", "Fnct")
 BUILTIN(__builtin_signbitf, "if", "Fnc")
 BUILTIN(__builtin_signbitl, "iLd", "Fnc")
 
 // Special FP builtins.
 BUILTIN(__builtin_canonicalize, "dd", "nc")

clang/lib/CodeGen/CGBuiltin.cpp

 #include "clang/AST/Attr.h"
 #include "clang/AST/Decl.h"
 #include "clang/AST/OSLog.h"
 #include "clang/Basic/TargetBuiltins.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/CodeGen/CGFunctionInfo.h"
 #include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/APInt.h"
+#include "llvm/ADT/FloatingPointMode.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Analysis/ValueTracking.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/IntrinsicsAArch64.h"
 #include "llvm/IR/IntrinsicsAMDGPU.h"
#define MUTATE_LDBL(func) \
   MUTATE_LDBL(tgamma)
   MUTATE_LDBL(lgamma)
 #undef MUTATE_LDBL
   default:
     return BuiltinID;
   }
 }
 
+static Value *createCallToIsFPClass(CodeGenFunction *CGF, const CallExpr *Call,
+                                    unsigned Test) {
+  auto TestV = llvm::ConstantInt::get(CGF->Int32Ty, Test);
+  const Expr *Arg = Call->getNumArgs() == 1 ? Call->getArg(0) : Call->getArg(1);
+  CodeGenFunction::CGFPOptionsRAII FPOptsRAII(*CGF, Call);
+  Value *V = CGF->EmitScalarExpr(Arg);
+  Function *F = CGF->CGM.getIntrinsic(Intrinsic::is_fpclass, V->getType());
+  return CGF->Builder.CreateCall(F, {V, TestV});
+}
+
 RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
                                         const CallExpr *E,
                                         ReturnValueSlot ReturnValue) {
   const FunctionDecl *FD = GD.getDecl()->getAsFunction();
   // See if we can constant fold this builtin.  If so, don't emit it at all.
   Expr::EvalResult Result;
   if (E->EvaluateAsRValue(Result, CGM.getContext()) &&
       !Result.hasSideEffects()) {
case Builtin::BI__builtin_islessgreater:
       break;
     case Builtin::BI__builtin_isunordered:
       LHS = Builder.CreateFCmpUNO(LHS, RHS, "cmp");
       break;
     }
     // ZExt bool to int type.
     return RValue::get(Builder.CreateZExt(LHS, ConvertType(E->getType())));
   }
-  case Builtin::BI__builtin_isnan: {
-    CodeGenFunction::CGFPOptionsRAII FPOptsRAII(*this, E);
-    Value *V = EmitScalarExpr(E->getArg(0));
-    llvm::Type *Ty = V->getType();
-    const llvm::fltSemantics &Semantics = Ty->getFltSemantics();
-    if (!Builder.getIsFPConstrained() ||
-        Builder.getDefaultConstrainedExcept() == fp::ebIgnore ||
-        !Ty->isIEEE()) {
-      V = Builder.CreateFCmpUNO(V, V, "cmp");
-      return RValue::get(Builder.CreateZExt(V, ConvertType(E->getType())));
-    }
 
-    if (Value *Result = getTargetHooks().testFPKind(V, BuiltinID, Builder, CGM))
-      return RValue::get(Result);
+  case Builtin::BI__builtin_isnan:
+    return RValue::get(
+        Builder.CreateZExt(createCallToIsFPClass(this, E, FPClassTest::fcNan),
+                           ConvertType(E->getType())));
 
-    // NaN has all exp bits set and a non zero significand. Therefore:
-    // isnan(V) == ((exp mask - (abs(V) & exp mask)) < 0)
-    unsigned bitsize = Ty->getScalarSizeInBits();
-    llvm::IntegerType *IntTy = Builder.getIntNTy(bitsize);
-    Value *IntV = Builder.CreateBitCast(V, IntTy);
-    APInt AndMask = APInt::getSignedMaxValue(bitsize);
-    Value *AbsV =
-        Builder.CreateAnd(IntV, llvm::ConstantInt::get(IntTy, AndMask));
-    APInt ExpMask = APFloat::getInf(Semantics).bitcastToAPInt();
-    Value *Sub =
-        Builder.CreateSub(llvm::ConstantInt::get(IntTy, ExpMask), AbsV);
-    // V = sign bit (Sub) <=> V = (Sub < 0)
-    V = Builder.CreateLShr(Sub, llvm::ConstantInt::get(IntTy, bitsize - 1));
-    if (bitsize > 32)
-      V = Builder.CreateTrunc(V, ConvertType(E->getType()));
-    return RValue::get(V);
+  case Builtin::BI__builtin_issignaling:
+    return RValue::get(
+        Builder.CreateZExt(createCallToIsFPClass(this, E, FPClassTest::fcSNan),
+                           ConvertType(E->getType())));
+
+  case Builtin::BI__builtin_isinf:
+    return RValue::get(
+        Builder.CreateZExt(createCallToIsFPClass(this, E, FPClassTest::fcInf),
+                           ConvertType(E->getType())));
+
+  case Builtin::BIfinite:
+  case Builtin::BI__finite:
+  case Builtin::BIfinitef:
+  case Builtin::BI__finitef:
+  case Builtin::BIfinitel:
+  case Builtin::BI__finitel:
+  case Builtin::BI__builtin_isfinite:
+    return RValue::get(Builder.CreateZExt(
+        createCallToIsFPClass(this, E, FPClassTest::fcFinite),
+        ConvertType(E->getType())));
+
+  case Builtin::BI__builtin_isnormal:
+    return RValue::get(Builder.CreateZExt(
+        createCallToIsFPClass(this, E, FPClassTest::fcNormal),
+        ConvertType(E->getType())));
+
+  case Builtin::BI__builtin_issubnormal:
+    return RValue::get(Builder.CreateZExt(
+        createCallToIsFPClass(this, E, FPClassTest::fcSubnormal),
+        ConvertType(E->getType())));
+
+  case Builtin::BI__builtin_iszero:
+    return RValue::get(
+        Builder.CreateZExt(createCallToIsFPClass(this, E, FPClassTest::fcZero),
+                           ConvertType(E->getType())));
+
+  case Builtin::BI__builtin_isfpclass: {
+    Expr::EvalResult Result;
+    if (!E->getArg(0)->EvaluateAsInt(Result, CGM.getContext()))
+      break;
+    uint64_t Test = Result.Val.getInt().getLimitedValue();
+    return RValue::get(Builder.CreateZExt(createCallToIsFPClass(this, E, Test),
+                                          ConvertType(E->getType())));
   }
 
   case Builtin::BI__builtin_elementwise_abs: {
     Value *Result;
     QualType QT = E->getArg(0)->getType();
 
     if (auto *VecTy = QT->getAs<VectorType>())
       QT = VecTy->getElementType();
     if (QT->isIntegerType())
case Builtin::BI__builtin_matrix_column_major_store: {
     EmitNonNullArgCheck(RValue::get(Dst.getPointer()), E->getArg(1)->getType(),
                         E->getArg(1)->getExprLoc(), FD, 0);
     Value *Result = MB.CreateColumnMajorStore(
         Matrix, Dst.getPointer(), Align(Dst.getAlignment().getQuantity()),
         Stride, IsVolatile, MatrixTy->getNumRows(), MatrixTy->getNumColumns());
     return RValue::get(Result);
   }
 
-  case Builtin::BIfinite:
-  case Builtin::BI__finite:
-  case Builtin::BIfinitef:
-  case Builtin::BI__finitef:
-  case Builtin::BIfinitel:
-  case Builtin::BI__finitel:
-  case Builtin::BI__builtin_isinf:
-  case Builtin::BI__builtin_isfinite: {
-    // isinf(x)    --> fabs(x) == infinity
-    // isfinite(x) --> fabs(x) != infinity
-    // x != NaN via the ordered compare in either case.
-    CodeGenFunction::CGFPOptionsRAII FPOptsRAII(*this, E);
-    Value *V = EmitScalarExpr(E->getArg(0));
-    llvm::Type *Ty = V->getType();
-    if (!Builder.getIsFPConstrained() ||
-        Builder.getDefaultConstrainedExcept() == fp::ebIgnore ||
-        !Ty->isIEEE()) {
-      Value *Fabs = EmitFAbs(*this, V);
-      Constant *Infinity = ConstantFP::getInfinity(V->getType());
-      CmpInst::Predicate Pred = (BuiltinID == Builtin::BI__builtin_isinf)
-                                    ? CmpInst::FCMP_OEQ
-                                    : CmpInst::FCMP_ONE;
-      Value *FCmp = Builder.CreateFCmp(Pred, Fabs, Infinity, "cmpinf");
-      return RValue::get(Builder.CreateZExt(FCmp, ConvertType(E->getType())));
-    }
-
-    if (Value *Result = getTargetHooks().testFPKind(V, BuiltinID, Builder, CGM))
-      return RValue::get(Result);
-
-    // Inf values have all exp bits set and a zero significand. Therefore:
-    // isinf(V) == ((V << 1) == ((exp mask) << 1))
-    // isfinite(V) == ((V << 1) < ((exp mask) << 1)) using unsigned comparison
-    unsigned bitsize = Ty->getScalarSizeInBits();
-    llvm::IntegerType *IntTy = Builder.getIntNTy(bitsize);
-    Value *IntV = Builder.CreateBitCast(V, IntTy);
-    Value *Shl1 = Builder.CreateShl(IntV, 1);
-    const llvm::fltSemantics &Semantics = Ty->getFltSemantics();
-    APInt ExpMask = APFloat::getInf(Semantics).bitcastToAPInt();
-    Value *ExpMaskShl1 = llvm::ConstantInt::get(IntTy, ExpMask.shl(1));
-    if (BuiltinID == Builtin::BI__builtin_isinf)
-      V = Builder.CreateICmpEQ(Shl1, ExpMaskShl1);
-    else
-      V = Builder.CreateICmpULT(Shl1, ExpMaskShl1);
-    return RValue::get(Builder.CreateZExt(V, ConvertType(E->getType())));
-  }
-
   case Builtin::BI__builtin_isinf_sign: {
     // isinf_sign(x) -> fabs(x) == infinity ? (signbit(x) ? -1 : 1) : 0
     CodeGenFunction::CGFPOptionsRAII FPOptsRAII(*this, E);
     // FIXME: for strictfp/IEEE-754 we need to not trap on SNaN here.
     Value *Arg = EmitScalarExpr(E->getArg(0));
     Value *AbsArg = EmitFAbs(*this, Arg);
     Value *IsInf = Builder.CreateFCmpOEQ(
         AbsArg, ConstantFP::getInfinity(Arg->getType()), "isinf");
     Value *IsNeg = EmitSignBit(*this, Arg);
 
     llvm::Type *IntTy = ConvertType(E->getType());
     Value *Zero = Constant::getNullValue(IntTy);
     Value *One = ConstantInt::get(IntTy, 1);
     Value *NegativeOne = ConstantInt::get(IntTy, -1);
     Value *SignResult = Builder.CreateSelect(IsNeg, NegativeOne, One);
     Value *Result = Builder.CreateSelect(IsInf, SignResult, Zero);
     return RValue::get(Result);
   }
 
-  case Builtin::BI__builtin_isnormal: {
-    // isnormal(x) --> x == x && fabsf(x) < infinity && fabsf(x) >= float_min
-    CodeGenFunction::CGFPOptionsRAII FPOptsRAII(*this, E);
-    // FIXME: for strictfp/IEEE-754 we need to not trap on SNaN here.
-    Value *V = EmitScalarExpr(E->getArg(0));
-    Value *Eq = Builder.CreateFCmpOEQ(V, V, "iseq");
-
-    Value *Abs = EmitFAbs(*this, V);
-    Value *IsLessThanInf =
-      Builder.CreateFCmpULT(Abs, ConstantFP::getInfinity(V->getType()),"isinf");
-    APFloat Smallest = APFloat::getSmallestNormalized(
-                   getContext().getFloatTypeSemantics(E->getArg(0)->getType()));
-    Value *IsNormal =
-      Builder.CreateFCmpUGE(Abs, ConstantFP::get(V->getContext(), Smallest),
-                            "isnormal");
-    V = Builder.CreateAnd(Eq, IsLessThanInf, "and");
-    V = Builder.CreateAnd(V, IsNormal, "and");
-    return RValue::get(Builder.CreateZExt(V, ConvertType(E->getType())));
-  }
-
   case Builtin::BI__builtin_flt_rounds: {
     Function *F = CGM.getIntrinsic(Intrinsic::flt_rounds);
 
     llvm::Type *ResultType = ConvertType(E->getType());
     Value *Result = Builder.CreateCall(F);
     if (Result->getType() != ResultType)
       Result = Builder.CreateIntCast(Result, ResultType, /*isSigned*/true,
                                      "cast");

clang/lib/CodeGen/TargetInfo.h

public:
   /// by the system in a return slot.
   ///
   /// Used by __builtin_frob_return_addr().
   virtual llvm::Value *encodeReturnAddress(CodeGen::CodeGenFunction &CGF,
                                            llvm::Value *Address) const {
     return Address;
   }
 
-  /// Performs a target specific test of a floating point value for things
-  /// like IsNaN, Infinity, ... Nullptr is returned if no implementation
-  /// exists.
-  virtual llvm::Value *
-  testFPKind(llvm::Value *V, unsigned BuiltinID, CGBuilderTy &Builder,
-             CodeGenModule &CGM) const {
-    assert(V->getType()->isFloatingPointTy() && "V should have an FP type.");
-    return nullptr;
-  }
-
   /// Corrects the low-level LLVM type for a given constraint and "usual"
   /// type.
   ///
   /// \returns A pointer to a new LLVM type, possibly the same as the original
   /// on success; 0 on failure.
   virtual llvm::Type *adjustInlineAsmType(CodeGen::CodeGenFunction &CGF,
                                           StringRef Constraint,
                                           llvm::Type *Ty) const {

clang/lib/CodeGen/TargetInfo.cpp

public:
   }
 };
 
 class SystemZTargetCodeGenInfo : public TargetCodeGenInfo {
 public:
   SystemZTargetCodeGenInfo(CodeGenTypes &CGT, bool HasVector, bool SoftFloatABI)
       : TargetCodeGenInfo(
             std::make_unique<SystemZABIInfo>(CGT, HasVector, SoftFloatABI)) {}
-
-  llvm::Value *testFPKind(llvm::Value *V, unsigned BuiltinID,
-                          CGBuilderTy &Builder,
-                          CodeGenModule &CGM) const override {
-    assert(V->getType()->isFloatingPointTy() && "V should have an FP type.");
-    // Only use TDC in constrained FP mode.
-    if (!Builder.getIsFPConstrained())
-      return nullptr;
-
-    llvm::Type *Ty = V->getType();
-    if (Ty->isFloatTy() || Ty->isDoubleTy() || Ty->isFP128Ty()) {
-      llvm::Module &M = CGM.getModule();
-      auto &Ctx = M.getContext();
-      llvm::Function *TDCFunc =
-          llvm::Intrinsic::getDeclaration(&M, llvm::Intrinsic::s390_tdc, Ty);
-      unsigned TDCBits = 0;
-      switch (BuiltinID) {
-      case Builtin::BI__builtin_isnan:
-        TDCBits = 0xf;
-        break;
-      case Builtin::BIfinite:
-      case Builtin::BI__finite:
-      case Builtin::BIfinitef:
-      case Builtin::BI__finitef:
-      case Builtin::BIfinitel:
-      case Builtin::BI__finitel:
-      case Builtin::BI__builtin_isfinite:
-        TDCBits = 0xfc0;
-        break;
-      case Builtin::BI__builtin_isinf:
-        TDCBits = 0x30;
-        break;
-      default:
-        break;
-      }
-      if (TDCBits)
-        return Builder.CreateCall(
-            TDCFunc,
-            {V, llvm::ConstantInt::get(llvm::Type::getInt64Ty(Ctx), TDCBits)});
-    }
-    return nullptr;
-  }
 };
 }
 
 bool SystemZABIInfo::isPromotableIntegerTypeForABI(QualType Ty) const {
   // Treat an enum type as its underlying type.
   if (const EnumType *EnumTy = Ty->getAs<EnumType>())
     Ty = EnumTy->getDecl()->getIntegerType();