[FPEnv] Intrinsic for setting rounding mode

[sepavloff]

Summary

To set non-default rounding mode user usually calls function 'fesetround'
from standard C library. This way has some disadvantages.

• It creates unnecessary dependency on libc. On the other hand, setting rounding mode requires few instruction and could be made by compiler. Sometimes standard C library even is not available, like in the case of GPU or AI cores that execute small kernels.
• Compiler could generate more effective code if it know that particular call just sets rounding mode.

This change introduces new IR intrinsic, namely 'llvm.set.rounding', which
sets current rounding mode, similar to 'fesetround'. It however differs
from the latter, because it is a lower level facility:

• 'llvm.set.rounding' does not return any value, whereas 'fesetround' returns non-zero value in the case of failure. In glibc 'fesetround' reports failure if its argument is invalid or unsupported or if floating point operations are unavailable on the hardware. Compiler usually knows what core it generates code for and it can validate arguments in many cases.
• Rounding mode is specified in 'fesetround' using constants like 'FE_TONEAREST', which are target dependent. It is inconvenient to work with such constants at IR level.

C standard provides a target-independent way to specify rounding mode, it
is used in FLT_ROUNDS, however it does not define standard way to set
rounding mode using this encoding.

This change implements only IR intrinsic. Lowering it to machine code is
target-specific and will be implemented latter. Mapping of 'fesetround'
to 'llvm.set.rounding' is also not implemented here.

[sepavloff]

Rebased patch

[sepavloff]

Add missed change

[kpn]

Are there GPUs or AI cores _targeted by llvm_ that support changing the rounding mode at run-time?

[arsenm]

In D74729#2005777, @kpn wrote:

Are there GPUs or AI cores _targeted by llvm_ that support changing the rounding mode at run-time?

AMDGPU can

[kpn]

In D74729#2005862, @arsenm wrote:

In D74729#2005777, @kpn wrote:

Are there GPUs or AI cores _targeted by llvm_ that support changing the rounding mode at run-time?

AMDGPU can

Is the fpsetround() function available on AMDGPU? Put another way, is this intrinsic needed on AMDGPU?

[arsenm]

In D74729#2005862, @arsenm wrote:

In D74729#2005777, @kpn wrote:

Are there GPUs or AI cores _targeted by llvm_ that support changing the rounding mode at run-time?

AMDGPU can

Originally we had 1 instruction that can change all aspects of the FP environment (which has a heavy runtime cost). The newest subtargets also have 2 additional and faster instructions that can separately set the rounding mode, and denormal mode. Therefore the decision of how to lower this is different per-subtarget, and therefore an abstracted and legalizable intrinsic is useful

[arsenm]

In D74729#2005867, @kpn wrote:

In D74729#2005862, @arsenm wrote:

In D74729#2005777, @kpn wrote:

Are there GPUs or AI cores _targeted by llvm_ that support changing the rounding mode at run-time?

AMDGPU can

Is the fpsetround() function available on AMDGPU? Put another way, is this intrinsic needed on AMDGPU?

We have no ISA libcalls of any kind, everything is through instructions (with different handling per-subtarget depending on what you're setting)

[kpn]

I think we need to be clear in the documentation that this is _not_ a substitute for the constrained FP intrinsics. Can you please add that to your new documentation? Include a link to the "Floating-Point Environment" section in the language ref.

Add to the IR verifier checks that the input is a constant that matches the documentation. The type and input values all need to be checked.

[arsenm]

llvm/docs/LangRef.rst

18727–18731

I'm wondering if this should be more opaque, and broader for the entire FP environment (not just the rounding mode). For AMDGPU we have a number of additional bits in the FP environment. We also have the denormal mode, enabling FP exceptions, and a few more exotic target specific FP mode bits.

[RKSimon]

llvm/docs/LangRef.rst

18701

Remove "read or " ?

18727–18731

@arsenm Would these extra bits be exclusive modes or would you need this to support target specific mode combos?

[sepavloff]

In D74729#2005958, @kpn wrote:

I think we need to be clear in the documentation that this is _not_ a substitute for the constrained FP intrinsics. Can you please add that to your new documentation? Include a link to the "Floating-Point Environment" section in the language ref.

It cannot be such substitute. IIUC constrained intrinsics were introduced to represent variants of corresponding C functions that operate in non-default FP environment to distinguish them from "ordinary" variants, that are pure functions. Functions like set_rounding always access FP environment, it always have side effect and must be properly ordered.

Add to the IR verifier checks that the input is a constant that matches the documentation. The type and input values all need to be checked.

The type of input value is already checked by generic code. As for values, there are two notes.

1. Input values may be a variable, not constant.
2. A target may support non-standard rounding modes.

However constant values are limited by 3 bits, of which one (Dynamic) cannot be used as argument. So adding check to IR verifier makes sense.

llvm/docs/LangRef.rst

18701

This is a section for group of intrinsics. Now there is only one intrinsic in it, which indeed only writes FP environment. It makes sense to implement intrinsics fo standard C functions, like fegetmode, fetestexcept and others.

Actually the intrinsic flt_rounds may be documented here. I will add documentation for it.

18727–18731

I'm wondering if this should be more opaque, and broader for the entire FP environment (not just the rounding mode).

C library defined function fesetmode, which sets all control modes, not just rounding. It make sense to introduce intrinsic for it, which would serve these purposes.

[kpn]

In D74729#2008093, @sepavloff wrote:

In D74729#2005958, @kpn wrote:

I think we need to be clear in the documentation that this is _not_ a substitute for the constrained FP intrinsics. Can you please add that to your new documentation? Include a link to the "Floating-Point Environment" section in the language ref.

It cannot be such substitute. IIUC constrained intrinsics were introduced to represent variants of corresponding C functions that operate in non-default FP environment to distinguish them from "ordinary" variants, that are pure functions. Functions like set_rounding always access FP environment, it always have side effect and must be properly ordered.

Sure, you know that, and I know that, but someone who hasn't been following along the past couple of years may not know that. A sentence or two tying things together won't hurt. Something along the lines of "Altering the rounding mode requires special care. See 'Floating-Point Environment'.", with a link to that section of the documentation.

[sepavloff]

Updated patch

• Added verification code,
• Added note to documentation.

[sepavloff]

In D74729#2008437, @kpn wrote:

In D74729#2008093, @sepavloff wrote:

In D74729#2005958, @kpn wrote:

I think we need to be clear in the documentation that this is _not_ a substitute for the constrained FP intrinsics. Can you please add that to your new documentation? Include a link to the "Floating-Point Environment" section in the language ref.

It cannot be such substitute. IIUC constrained intrinsics were introduced to represent variants of corresponding C functions that operate in non-default FP environment to distinguish them from "ordinary" variants, that are pure functions. Functions like set_rounding always access FP environment, it always have side effect and must be properly ordered.

Sure, you know that, and I know that, but someone who hasn't been following along the past couple of years may not know that. A sentence or two tying things together won't hurt. Something along the lines of "Altering the rounding mode requires special care. See 'Floating-Point Environment'.", with a link to that section of the documentation.

Ah, got it! Added note to the documentation. Thank you!

llvm/docs/LangRef.rst

18701

Added flt_rounds in D79322.

[arsenm]

llvm/docs/LangRef.rst

18727–18731

We have the rounding mode controls as presented here, however they are broken down by FP type. We can separately set the rounding mode for f32 and f64/f16, so there are two different settings. We also have the denormal mode, for inputs and outputs, also broken down by type in the same way. The denormal handling and per-type handling I think deserve consideration here

We have 2 additional target specific FP bits nothing else would need to really think about, but it would be nice if you could set the exact mode you want in a single intrinsic call. I'm less interested in these though

18727–18731

Oh, we also have a bit to turn on/off fp exceptions which is probably generally interesting

[arsenm]

llvm/docs/LangRef.rst

18727–18731

And by a bit, I mean a mask of bits for different FP exception types

[sepavloff]

Rebased patch

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6226

Don't you need to call getRoot not DAG.getRoot()?

llvm/lib/IR/Verifier.cpp

5031

Is the argument intended to always be a constant or we're just verifying it when we can? The latter seems unusual.

[sepavloff]

Updated patch

• Use getRoot() instead of DAG.getRoot().

[sepavloff]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6226

Indeed. Thank you!

llvm/lib/IR/Verifier.cpp

5031

The argument may be a variable. If it is a constant, it must be a valid rounding mode. It is expected to be a value of type RoundingMode. Values from 0 to 4 denote IEEE rounding modes, they may be followed by target-specific rounding modes. The argument value must be less than RoundingMode::Dynamic, which now if 7.

I am hesitating if this code is useful enough, as even for constant argument its validity cannot be verified due to non-IEEE rounding modes. Probably we should remove this check.

[sepavloff]

Rebased patch

[sepavloff]

I'm wondering if this should be more opaque, and broader for the entire FP environment (not just the rounding mode). For AMDGPU we have a number of additional bits in the FP environment. We also have the denormal mode, enabling FP exceptions, and a few more exotic target specific FP mode bits.

Instrinsics get.fpmode and set.fpmode introduced in D82525 may be used for this case.

[sepavloff]

Rebased patch

[sepavloff]

Rebased patch

[sepavloff]

Rebased patch

[sepavloff]

Updated patch

[sepavloff]

Removed clang-tidy warning

[sepavloff]

Any feedback is appreciated.

[RKSimon]

llvm/lib/IR/Verifier.cpp

5031

I'm OK with this being dropped - @craig.topper @kpn ?

[craig.topper]

llvm/lib/IR/Verifier.cpp

5031

I'm fine removing it

[sepavloff]

Removed check from Verifier

[sepavloff]

Ping.

[RKSimon]

llvm/include/llvm/CodeGen/ISDOpcodes.h

732

Sorry for the bikeshedding - but if SET_ROUNDING is supposed to match FLT_ROUNDS - shouldn't it have a more similar name?

[sepavloff]

llvm/include/llvm/CodeGen/ISDOpcodes.h

732

It is FLT_ROUNDS_ that has "wrong" name. It is named after the macro FLT_ROUNDS, which is defined by C99. To get better names FLT_ROUNDS_ must be renamed not SET_ROUNDING.

[RKSimon]

ok, I've no more questions @arsenm @kpn?

llvm/include/llvm/CodeGen/ISDOpcodes.h

732

OK - add a TODO comment by FLT_ROUNDS_ then?

[kpn]

In D74729#2354279, @RKSimon wrote:

ok, I've no more questions @arsenm @kpn?

Nothing from me.

[sepavloff]

Added TODO.

[sepavloff]

llvm/include/llvm/CodeGen/ISDOpcodes.h

732

Done.

[craig.topper]

llvm/include/llvm/IR/IRBuilder.h

897

Do we need this? I don't think we provide IRBuilder for all intrinsics. Just common ones.

[craig.topper]

Do we need type legalization support for targets like RISCV where i32 isn't a legal type in SelectionDAG?

[sepavloff]

Removed method IRBuilderBase::createSetRounding.

[sepavloff]

In D74729#2358110, @craig.topper wrote:

Do we need type legalization support for targets like RISCV where i32 isn't a legal type in SelectionDAG?

Lowering of the intrinsic is anyway custom. Default promotion to i64 is OK.

[craig.topper]

In D74729#2358796, @sepavloff wrote:

In D74729#2358110, @craig.topper wrote:

Do we need type legalization support for targets like RISCV where i32 isn't a legal type in SelectionDAG?

Lowering of the intrinsic is anyway custom. Default promotion to i64 is OK.

There is no default promotion code. Each opcode that needs to be promoted by type legalization must be handled in LegalizeIntegerTypes.cpp

[sepavloff]

Add support in DAGTypeLegalizer::PromoteIntegerOperand

[sepavloff]

In D74729#2359928, @craig.topper wrote:

In D74729#2358796, @sepavloff wrote:

In D74729#2358110, @craig.topper wrote:

Do we need type legalization support for targets like RISCV where i32 isn't a legal type in SelectionDAG?

Lowering of the intrinsic is anyway custom. Default promotion to i64 is OK.

There is no default promotion code. Each opcode that needs to be promoted by type legalization must be handled in LegalizeIntegerTypes.cpp

Handling of SET_ROUNDING on RISCV is implemented in D91242.

[sepavloff]

Ping.

[RKSimon]

LGTM - any other comments? @craig.topper is the RISCV followup at D91242 OK do you think? It'll probably be what other targets end up using as reference.

llvm/docs/LangRef.rst

 
 This function returns the same values as the libm ``trunc`` functions
 would and handles error conditions in the same way.
 
 
 Floating Point Environment Manipulation intrinsics
 --------------------------------------------------
 
 These functions read or write floating point environment, such as rounding
 mode or state of floating point exceptions. Altering the floating point
 environment requires special care. See :ref:`Floating Point Environment <floatenv>`.
 
 '``llvm.flt.rounds``' Intrinsic
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""

 
 Semantics:
 """"""""""
 
 The '``llvm.flt.rounds``' intrinsic returns the current rounding mode.
 Encoding of the returned values is same as the result of ``FLT_ROUNDS``,
 specified by C standard:
 
 ::
 
     0  - toward zero
     1  - to nearest, ties to even
     2  - toward positive infinity
     3  - toward negative infinity
     4  - to nearest, ties away from zero
 
 Other values may be used to represent additional rounding modes, supported by a
 target. These values are target-specific.
 
+
+'``llvm.set.rounding``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+
+::
+
+      declare void @llvm.set.rounding(i32 <val>)
+
+Overview:
+"""""""""
+
+The '``llvm.set.rounding``' intrinsic sets current rounding mode.
+
+Arguments:
+""""""""""
+
+The argument is the required rounding mode. Encoding of rounding mode is
+the same as used by '``llvm.flt.rounds``'.
+
+Semantics:
+""""""""""
+
+The '``llvm.set.rounding``' intrinsic sets the current rounding mode. It is
+similar to C library function 'fesetround', however this intrinsic does not
+return any value and uses platform-independent representation of IEEE rounding
+modes.
+
+
 General Intrinsics
 ------------------
 
 This class of intrinsics is designed to be generic and has no specific
 purpose.
 
 '``llvm.var.annotation``' Intrinsic
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

llvm/include/llvm/CodeGen/ISDOpcodes.h

enum NodeType {
   ///
   /// The TRUNC = 1 case is used in cases where we know that the value will
   /// not be modified by the node, because Y is not using any of the extra
   /// precision of source type.  This allows certain transformations like
   /// FP_EXTEND(FP_ROUND(X,1)) -> X which are not safe for
   /// FP_EXTEND(FP_ROUND(X,0)) because the extra bits aren't removed.
   FP_ROUND,
 
-  /// FLT_ROUNDS_ - Returns current rounding mode:
+  /// Returns current rounding mode:
   /// -1 Undefined
   ///  0 Round to 0
-  ///  1 Round to nearest
+  ///  1 Round to nearest, ties to even
   ///  2 Round to +inf
   ///  3 Round to -inf
+  ///  4 Round to nearest, ties to zero
   /// Result is rounding mode and chain. Input is a chain.
   FLT_ROUNDS_,
 
+  /// Set rounding mode.
+  /// The first operand is a chain pointer. The second specifies the required
+  /// rounding mode, encoded in the same way as used in '``FLT_ROUNDS_``'.
+  SET_ROUNDING,
+
   /// X = FP_EXTEND(Y) - Extend a smaller FP type into a larger FP type.
   FP_EXTEND,
 
   /// BITCAST - This operator converts between integer, vector and FP
   /// values, as if the value was stored to memory with one type and loaded
   /// from the same address with the other type (or equivalently for vector
   /// format conversions, etc).  The source and result are required to have
   /// the same bit size (e.g.  f32 <-> i32).  This can also be used for

llvm/include/llvm/IR/IRBuilder.h

CallInst *CreateMinimum(Value *LHS, Value *RHS, const Twine &Name = "") {
     return CreateBinaryIntrinsic(Intrinsic::minimum, LHS, RHS, nullptr, Name);
   }
 
   /// Create call to the maximum intrinsic.
   CallInst *CreateMaximum(Value *LHS, Value *RHS, const Twine &Name = "") {
     return CreateBinaryIntrinsic(Intrinsic::maximum, LHS, RHS, nullptr, Name);
   }
 
+  /// Create call to the set_rounding intrinsic.
+  CallInst *CreateSetRounding(Value *RM, const Twine &Name = "") {
+    return CreateIntrinsic(Intrinsic::set_rounding, {}, {RM}, nullptr, Name);
+  }
+
 private:
   /// Create a call to a masked intrinsic with given Id.
   CallInst *CreateMaskedIntrinsic(Intrinsic::ID Id, ArrayRef<Value *> Ops,
                                   ArrayRef<Type *> OverloadedTypes,
                                   const Twine &Name = "");
 
   Value *getCastedInt8PtrValue(Value *Ptr);
 

llvm/include/llvm/IR/Intrinsics.td

def int_objectsize : Intrinsic<[llvm_anyint_ty],
                                 ImmArg<ArgIndex<3>>]>,
                                GCCBuiltin<"__builtin_object_size">;
 
 //===--------------- Access to Floating Point Environment -----------------===//
 //
 
 let IntrProperties = [IntrInaccessibleMemOnly, IntrWillReturn] in {
   def int_flt_rounds    : Intrinsic<[llvm_i32_ty], []>;
+  def int_set_rounding  : Intrinsic<[], [llvm_i32_ty]>;
 }
 
 //===--------------- Constrained Floating Point Intrinsics ----------------===//
 //
 
 let IntrProperties = [IntrInaccessibleMemOnly, IntrWillReturn] in {
   def int_experimental_constrained_fadd : Intrinsic<[ llvm_anyfloat_ty ],
                                                     [ LLVMMatchType<0>,

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

setValue(&I, DAG.getNode(ISD::BITCAST, sdl, MVT::i16,
                                                                MVT::i32))));
     return;
   case Intrinsic::convert_from_fp16:
     setValue(&I, DAG.getNode(ISD::FP_EXTEND, sdl,
                              TLI.getValueType(DAG.getDataLayout(), I.getType()),
                              DAG.getNode(ISD::BITCAST, sdl, MVT::f16,
                                          getValue(I.getArgOperand(0)))));
     return;
+  case Intrinsic::set_rounding:
+    Res = DAG.getNode(ISD::SET_ROUNDING, sdl, MVT::Other,
+                      {getRoot(), getValue(I.getArgOperand(0))});
+    setValue(&I, Res);
+    DAG.setRoot(Res.getValue(0));
+    return;
   case Intrinsic::pcmarker: {
     SDValue Tmp = getValue(I.getArgOperand(0));
     DAG.setRoot(DAG.getNode(ISD::PCMARKER, sdl, MVT::Other, getRoot(), Tmp));
     return;
   }
   case Intrinsic::readcyclecounter: {
     SDValue Op = getRoot();
     Res = DAG.getNode(ISD::READCYCLECOUNTER, sdl,

llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp

 //===- SelectionDAGDumper.cpp - Implement SelectionDAG::dump() ------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This implements the SelectionDAG::dump method and friends.
#endif
   case ISD::ANY_EXTEND:                 return "any_extend";
   case ISD::SIGN_EXTEND_INREG:          return "sign_extend_inreg";
   case ISD::ANY_EXTEND_VECTOR_INREG:    return "any_extend_vector_inreg";
   case ISD::SIGN_EXTEND_VECTOR_INREG:   return "sign_extend_vector_inreg";
   case ISD::ZERO_EXTEND_VECTOR_INREG:   return "zero_extend_vector_inreg";
   case ISD::TRUNCATE:                   return "truncate";
   case ISD::FP_ROUND:                   return "fp_round";
   case ISD::STRICT_FP_ROUND:            return "strict_fp_round";
-  case ISD::FLT_ROUNDS_:                return "flt_rounds";
   case ISD::FP_EXTEND:                  return "fp_extend";
   case ISD::STRICT_FP_EXTEND:           return "strict_fp_extend";
 
   case ISD::SINT_TO_FP:                 return "sint_to_fp";
   case ISD::STRICT_SINT_TO_FP:          return "strict_sint_to_fp";
   case ISD::UINT_TO_FP:                 return "uint_to_fp";
   case ISD::STRICT_UINT_TO_FP:          return "strict_uint_to_fp";
   case ISD::FP_TO_SINT:                 return "fp_to_sint";
#endif
   case ISD::GC_TRANSITION_END:          return "gc_transition.end";
   case ISD::GET_DYNAMIC_AREA_OFFSET:    return "get.dynamic.area.offset";
   case ISD::FREEZE:                     return "freeze";
   case ISD::PREALLOCATED_SETUP:
     return "call_setup";
   case ISD::PREALLOCATED_ARG:
     return "call_alloc";
 
+  // Floating point environment manipulation
+  case ISD::FLT_ROUNDS_:                return "flt_rounds";
+  case ISD::SET_ROUNDING:               return "set_rounding";
+
   // Bit manipulation
   case ISD::ABS:                        return "abs";
   case ISD::BITREVERSE:                 return "bitreverse";
   case ISD::BSWAP:                      return "bswap";
   case ISD::CTPOP:                      return "ctpop";
   case ISD::CTTZ:                       return "cttz";
   case ISD::CTTZ_ZERO_UNDEF:            return "cttz_zero_undef";
   case ISD::CTLZ:                       return "ctlz";

llvm/lib/IR/Verifier.cpp

case Intrinsic::matrix_column_major_store: {
     default:
       llvm_unreachable("unexpected intrinsic");
     }
     Assert(TypeToCheck->getNumElements() ==
                NumRows->getZExtValue() * NumColumns->getZExtValue(),
            "result of a matrix operation does not fit in the returned vector");
     break;
   }
+  case Intrinsic::set_rounding: {
+    if (auto RM = dyn_cast<ConstantInt>(Call.getArgOperand(0))) {
+      Assert(RM->getZExtValue() < static_cast<unsigned>(RoundingMode::Dynamic),
+             "invalid value of rounding mode");
+    }
+  }
   };
 }
 
 /// Carefully grab the subprogram from a local scope.
 ///
 /// This carefully grabs the subprogram from a local scope, avoiding the
 /// built-in assertions that would typically fire.
 static DISubprogram *getSubprogram(Metadata *LocalScope) {

llvm/unittests/IR/IRBuilderTest.cpp

TEST_F(IRBuilderTest, Intrinsics) {
   EXPECT_TRUE(II->hasNoInfs());
   EXPECT_FALSE(II->hasNoNaNs());
 
   Call = Builder.CreateUnaryIntrinsic(Intrinsic::roundeven, V);
   II = cast<IntrinsicInst>(Call);
   EXPECT_EQ(II->getIntrinsicID(), Intrinsic::roundeven);
   EXPECT_FALSE(II->hasNoInfs());
   EXPECT_FALSE(II->hasNoNaNs());
+
+  Call = Builder.CreateSetRounding(
+      Builder.getInt32(static_cast<uint32_t>(RoundingMode::TowardZero)));
+  II = cast<IntrinsicInst>(Call);
+  EXPECT_EQ(II->getIntrinsicID(), Intrinsic::set_rounding);
 }
 
 TEST_F(IRBuilderTest, IntrinsicsWithScalableVectors) {
   IRBuilder<> Builder(BB);
   CallInst *Call;
   FunctionType *FTy;
 
   // Test scalable flag isn't dropped for intrinsic that is explicitly defined