[ConstantFolding] Fold constrained compare intrinsics

[sepavloff]

Summary

The change implements constant folding of ‘llvm.experimental.constrained.fcmp’
and ‘llvm.experimental.constrained.fcmps’ intrinsics.

[bjope]

Sorry, I don't know much about fcmp.

[sepavloff]

Add check lines to the test

[kpn]

llvm/lib/Analysis/ConstantFolding.cpp

1798

This still worries me. Are _analysis_ passes allowed to change the input IR? What if the caller decides to not do a fold after calling this function to see if a fold is possible? And why is the ebIgnore case different from the other two?

Constant folding won't add this ReadNone attribute when the constant folding code doesn't have an Instruction to alter. I don't know what to do about that.

llvm/lib/IR/Instructions.cpp

4134 ↗

(On Diff #374511)

Besides moving this code to a new function, are there any changes to it? It's hard to tell.

llvm/test/Transforms/InstSimplify/constfold-constrained.ll

420

There are no "maytrap" tests here. For the sake of future readers it would probably be useful to show that "maytrap" was taken into account and is working correctly.

[sepavloff]

Added tests with "maytrap"

[nikic]

llvm/lib/Analysis/ConstantFolding.cpp

1798

Analysis passes are indeed not allowed to change the IR. And this isn't a harmless change either if it gets left behind -- e.g. it invalidates MemorySSA.

[sepavloff]

llvm/lib/Analysis/ConstantFolding.cpp

1798

Analysis passes are indeed not allowed to change the IR.

Constant folding is not an analysis, it changes IR. I don't know why this file is in Analysis directory.

What if the caller decides to not do a fold after calling this function to see if a fold is possible?

Nothing bad would happen. Only side effect would be removed, but we know for sure that it is absent, we just evaluated the operation. The function call would not be removed if its result is used.

And why is the ebIgnore case different from the other two?

If exception behavior is ebIgnore, such calls will get attribute SDNodeFlags::NoFPExcept in DAG and such instructions do not have side effects. But setting ReadNone for instructions with ebIgnore allows removal of these instructions earlier, at IR level, which could have positive effect.

And this isn't a harmless change either if it gets left behind -- e.g. it invalidates MemorySSA.

Comparison of two floating point numbers do not use memory access. But it can change bits in the floating point state register (only Invalid bit can be set). This change is emulated as memory access so that instruction be ordered correctly. This is why constrained intrinsics declared with attribute IntrInaccessibleMemOnly. As no actual memory access occurs, it is harmless to set ReadNone in this case.

llvm/lib/IR/Instructions.cpp

4134 ↗

(On Diff #374511)

No, this is only a code moving.

llvm/test/Transforms/InstSimplify/constfold-constrained.ll

420

Added tests with "maytrap".

[sepavloff]

Set ReadNone for all intrinsics if they can be evaluated without raising signals

[nikic]

llvm/lib/Analysis/ConstantFolding.cpp

1798

Constant folding is not an analysis, it changes IR. I don't know why this file is in Analysis directory.

The constant folding analysis does not change IR. Users of the constant folding analysis change IR based on the analysis result.

Comparison of two floating point numbers do not use memory access. But it can change bits in the floating point state register (only Invalid bit can be set). This change is emulated as memory access so that instruction be ordered correctly. This is why constrained intrinsics declared with attribute IntrInaccessibleMemOnly. As no actual memory access occurs, it is harmless to set ReadNone in this case.

My point here was that adding a readnone attribute invalidates MemorySSA, because it means that the instruction should no longer have a MemoryAccess -- it would result in a verification failure. Just calling ConstFold/InstSimplify should never have this kind of effect.

Unless @spatel or @lebedev.ri tell me I'm wrong here, I believe this should be considered a blocker for further work in this area.

[lebedev.ri]

Unless @spatel or @lebedev.ri tell me I'm wrong here, I believe this should be considered a blocker for further work in this area.

[sepavloff]

Move attribute modification to InstSimplify pass

[sepavloff]

Any feedback?

[craig.topper]

Should the change that effects fadd, fma, etc. tests be moved to a different patch that is a pre-requisite of the compare change?

llvm/lib/Analysis/ConstantFolding.cpp

2312

Does ConstrainedFPCmpIntrinsic have any method for determining fcmp vs fcmps? If not should it?

llvm/lib/IR/Instructions.cpp

4243 ↗

(On Diff #388475)

This feels a little like it shouldn't be part of FCmpInst, but I don't have a concrete suggestion of where to put it instead. ConstantFold.cpp feels like the right home, but unfortunatley the header for that isn't visible to ConstantFolding.cpp.

[sepavloff]

Addressed reviewer's notes

[sepavloff]

In D110322#3159399, @craig.topper wrote:

Should the change that effects fadd, fma, etc. tests be moved to a different patch that is a pre-requisite of the compare change?

Yes, it is a good idea to move this debatable change into a separate patch, it is here: D114766. This patch does not depend on it.

llvm/lib/Analysis/ConstantFolding.cpp

2312

Probably, but it does not help here. The interface is designed to evaluate constant value without construction of a node. ConstrainedFPCmpIntrinsic can be extracted from Call but checking intrinsic ID seems simpler.

llvm/lib/IR/Instructions.cpp

4243 ↗

(On Diff #388475)

I moved ConstantFold to include directory in dependency patch and put evaluatePredicate there.

[nikic]

llvm/lib/Analysis/ConstantFolding.cpp

2312

FYI recently ICmpInst::compare() was added for the corresponding operation on ICmpInsts (in https://github.com/llvm/llvm-project/commit/25043c8276644e684f8d14cd4cadaa87a7e99b0e), so it might make sense to have the same method on FCmpInst. (No strong opinion on placement though.)

[craig.topper]

llvm/lib/Analysis/ConstantFolding.cpp

2310

auto *FCmp I believe LLVM coding standards prefer to keep the * around with auto so things that are pointers are obvious.

2312

The ConstrainedFPCmpIntrinsic is already being extracted from the call to get the predicate. So I thought it would make sense to get the signaling state from it as well and not pass the intrinsic ID at all.

[sepavloff]

Changed arguments of evaluateCompare and rebased

[sepavloff]

llvm/lib/Analysis/ConstantFolding.cpp

2312

Changed signature of evaluateCompare and fixed interface of ConstrainedFPCmpIntrinsic accordingly.

[sepavloff]

Is there anything I can do to make this patch acceptable?

[kpn]

My guess:

Removing the addition of the readnone attribute? It's a correctness issue. We don't have a solution to the performance issue in generated code, but correctness beats performance every time.

[sepavloff]

In D110322#3195092, @kpn wrote:

My guess:

Removing the addition of the readnone attribute? It's a correctness issue. We don't have a solution to the performance issue in generated code, but correctness beats performance every time.

This patch now does not touch this attribute, this code is already removed.

[kpn]

llvm/lib/Analysis/ConstantFolding.cpp

1798

Here. These two lines still need to be removed. Along with the comment, of course.

[sepavloff]

llvm/lib/Analysis/ConstantFolding.cpp

1798

Done in the separate patch: https://reviews.llvm.org/D115870

[spatel]

llvm/lib/IR/ConstantFold.cpp

2021 ↗

(On Diff #392020)

This was suggested in an earlier comment - can we add this next to:
https://github.com/llvm/llvm-project/blob/80b1d472d6f7e10403e37cc74bfda278372ecb97/llvm/lib/IR/Instructions.cpp#L4135

(and adjust the names/parameters for consistency)

That can be a preliminary/NFC commit if I'm seeing it correctly.

[sepavloff]

llvm/lib/IR/ConstantFold.cpp

2021 ↗

(On Diff #392020)

Previously such method existed but I was recommended to remove it: https://reviews.llvm.org/D110322?id=388475#3159399.

Otherwise there is no problem to restore that implementation.

[spatel]

I have no objections to this patch now. I'd pre-commit the baseline tests though, and let's see if there's consensus on where to house the helper function.

llvm/lib/IR/ConstantFold.cpp

2021 ↗

(On Diff #392020)

Ah, sorry I didn't see that earlier comment/revision.
@craig.topper - do you still prefer the current structure?

[craig.topper]

llvm/lib/IR/ConstantFold.cpp

2021 ↗

(On Diff #392020)

I think the integer change happened while this patch was in review. I'm fine with restoring the previous implementation to make this consistent with integer.

[sepavloff]

Put back method of FCmpInst for constant comparison evaluation.

[spatel]

Please pre-commit the baseline tests and FCmpInst::compare(), so we're left with only real logic changes and tests that show the change in behavior.

[sepavloff]

FCmpInst::compare is moved to separate patch

[sepavloff]

Rebased patch

[sepavloff]

Any feedback?

[kpn]

LGTM

[sepavloff]

Thanks!

[bgraur]

Hi,

This commit causes clang to crash when compiling the attached reproducer with the following compilation command:

clang -cc1 \
  -emit-obj \
  -target-feature +sse4.2 \
  -frounding-math \
  -O1 \
  -std=gnu++17 \
  -fsized-deallocation \
  -o /tmp/repro.o -x c++ repro.cc

Please note that the crash only reproduces something like 60% of cases (just re-run a few times).
Please revert.

repro.cc633 BDownload

[sepavloff]

In D110322#3346560, @bgraur wrote:

Hi,

This commit causes clang to crash when compiling the attached reproducer with the following compilation command:

clang -cc1 \
  -emit-obj \
  -target-feature +sse4.2 \
  -frounding-math \
  -O1 \
  -std=gnu++17 \
  -fsized-deallocation \
  -o /tmp/repro.o -x c++ repro.cc

Please note that the crash only reproduces something like 60% of cases (just re-run a few times).
Please revert.

repro.cc633 BDownload

Thank you for the report. There was an issue with treatment of vector constants. Commit https://reviews.llvm.org/rG6982c38cb120 fixes it.

llvm/include/llvm/IR/IntrinsicInst.h

static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
   }
 };
 
 /// Constrained floating point compare intrinsics.
 class ConstrainedFPCmpIntrinsic : public ConstrainedFPIntrinsic {
 public:
   FCmpInst::Predicate getPredicate() const;
+  bool isSignaling() const {
+    return getIntrinsicID() == Intrinsic::experimental_constrained_fcmps;
+  }
 
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static bool classof(const IntrinsicInst *I) {
     switch (I->getIntrinsicID()) {
     case Intrinsic::experimental_constrained_fcmp:
     case Intrinsic::experimental_constrained_fcmps:
       return true;
     default:

llvm/lib/Analysis/ConstantFolding.cpp

bool llvm::canConstantFoldCallTo(const CallBase *Call, const Function *F) {
   case Intrinsic::experimental_constrained_frem:
   case Intrinsic::experimental_constrained_ceil:
   case Intrinsic::experimental_constrained_floor:
   case Intrinsic::experimental_constrained_round:
   case Intrinsic::experimental_constrained_roundeven:
   case Intrinsic::experimental_constrained_trunc:
   case Intrinsic::experimental_constrained_nearbyint:
   case Intrinsic::experimental_constrained_rint:
+  case Intrinsic::experimental_constrained_fcmp:
+  case Intrinsic::experimental_constrained_fcmps:
     return true;
   default:
     return false;
   case Intrinsic::not_intrinsic: break;
   }
 
   if (!F->hasName() || Call->isStrictFP())
     return false;

 /// \param St Exception flags raised during constant evaluation.
 static bool mayFoldConstrained(ConstrainedFPIntrinsic *CI,
                                APFloat::opStatus St) {
   Optional<RoundingMode> ORM = CI->getRoundingMode();
   Optional<fp::ExceptionBehavior> EB = CI->getExceptionBehavior();
 
   // If the operation does not change exception status flags, it is safe
   // to fold.
   if (St == APFloat::opStatus::opOK)
     return true;
 
   // If evaluation raised FP exception, the result can depend on rounding
   // mode. If the latter is unknown, folding is not possible.
-  if (!ORM || *ORM == RoundingMode::Dynamic)
+  if (ORM && *ORM == RoundingMode::Dynamic)
     return false;
 
   // If FP exceptions are ignored, fold the call, even if such exception is
   // raised.
-  if (!EB || *EB != fp::ExceptionBehavior::ebStrict)
+  if (EB && *EB != fp::ExceptionBehavior::ebStrict)
     return true;
 
   // Leave the calculation for runtime so that exception flags be correctly set
   // in hardware.
   return false;
 }
 
 /// Returns the rounding mode that should be used for constant evaluation.
case Intrinsic::x86_sse2_cvttsd2si64:
                                            /*IsSigned*/true);
       break;
     }
   }
 
   return nullptr;
 }
 
+static Constant *evaluateCompare(const ConstrainedFPIntrinsic *Call) {
+  APFloat::opStatus St = APFloat::opOK;
+  auto *FCmp = cast<ConstrainedFPCmpIntrinsic>(Call);
+  FCmpInst::Predicate Cond = FCmp->getPredicate();
+  const APFloat &Op1 = cast<ConstantFP>(FCmp->getOperand(0))->getValueAPF();
+  const APFloat &Op2 = cast<ConstantFP>(FCmp->getOperand(1))->getValueAPF();
+  if (FCmp->isSignaling()) {
+    if (Op1.isNaN() || Op2.isNaN())
+      St = APFloat::opInvalidOp;
+  } else {
+    if (Op1.isSignaling() || Op2.isSignaling())
+      St = APFloat::opInvalidOp;
+  }
+  bool Result = FCmpInst::compare(Op1, Op2, Cond);
+  if (mayFoldConstrained(const_cast<ConstrainedFPCmpIntrinsic *>(FCmp), St))
+    return ConstantInt::get(Call->getType(), Result);
+  return nullptr;
+}
+
 static Constant *ConstantFoldScalarCall2(StringRef Name,
                                          Intrinsic::ID IntrinsicID,
                                          Type *Ty,
                                          ArrayRef<Constant *> Operands,
                                          const TargetLibraryInfo *TLI,
                                          const CallBase *Call) {
   assert(Operands.size() == 2 && "Wrong number of operands.");
 
case Intrinsic::minimum:
         return Operands[1];
       if (IsOp1Undef)
         return Operands[0];
       break;
     }
   }
 
   if (const auto *Op1 = dyn_cast<ConstantFP>(Operands[0])) {
-    if (!Ty->isFloatingPointTy())
-      return nullptr;
     const APFloat &Op1V = Op1->getValueAPF();
 
     if (const auto *Op2 = dyn_cast<ConstantFP>(Operands[1])) {
       if (Op2->getType() != Op1->getType())
         return nullptr;
       const APFloat &Op2V = Op2->getValueAPF();
 
       if (const auto *ConstrIntr = dyn_cast<ConstrainedFPIntrinsic>(Call)) {
if (const auto *Op2 = dyn_cast<ConstantFP>(Operands[1])) {
           St = Res.multiply(Op2V, RM);
           break;
         case Intrinsic::experimental_constrained_fdiv:
           St = Res.divide(Op2V, RM);
           break;
         case Intrinsic::experimental_constrained_frem:
           St = Res.mod(Op2V);
           break;
+        case Intrinsic::experimental_constrained_fcmp:
+        case Intrinsic::experimental_constrained_fcmps:
+          return evaluateCompare(ConstrIntr);
         }
         if (mayFoldConstrained(const_cast<ConstrainedFPIntrinsic *>(ConstrIntr),
                                St))
           return ConstantFP::get(Ty->getContext(), Res);
         return nullptr;
       }
 
       switch (IntrinsicID) {

llvm/test/Transforms/InstSimplify/constfold-constrained.ll

 ; CHECK-NEXT:    ret double 5.000000e+00
 ;
 entry:
   %result = call double @llvm.experimental.constrained.fmuladd.f64(double 1.0, double 2.0, double 3.0, metadata !"round.dynamic", metadata !"fpexcept.ignore") #0
   ret double %result
 }
 
 
 ; When exceptions are ignored, comparison of constants can be folded, even for (signaling) NaNs.
 define i1 @cmp_eq_01() #0 {
 ; CHECK-LABEL: @cmp_eq_01(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RESULT:%.*]] = call i1 @llvm.experimental.constrained.fcmp.f64(double 1.000000e+00, double 2.000000e+00, metadata !"oeq", metadata !"fpexcept.ignore") #[[ATTR0]]
-; CHECK-NEXT:    ret i1 [[RESULT]]
+; CHECK-NEXT:    ret i1 false
 ;
 entry:
   %result = call i1 @llvm.experimental.constrained.fcmp.f64(double 1.0, double 2.0, metadata !"oeq", metadata !"fpexcept.ignore") #0
   ret i1 %result
 }
 
 define i1 @cmp_eq_02() #0 {
 ; CHECK-LABEL: @cmp_eq_02(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RESULT:%.*]] = call i1 @llvm.experimental.constrained.fcmp.f64(double 2.000000e+00, double 2.000000e+00, metadata !"oeq", metadata !"fpexcept.ignore") #[[ATTR0]]
-; CHECK-NEXT:    ret i1 [[RESULT]]
+; CHECK-NEXT:    ret i1 true
 ;
 entry:
   %result = call i1 @llvm.experimental.constrained.fcmp.f64(double 2.0, double 2.0, metadata !"oeq", metadata !"fpexcept.ignore") #0
   ret i1 %result
 }
 
 define i1 @cmp_eq_03() #0 {
 ; CHECK-LABEL: @cmp_eq_03(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RESULT:%.*]] = call i1 @llvm.experimental.constrained.fcmp.f64(double 2.000000e+00, double 0x7FF8000000000000, metadata !"oeq", metadata !"fpexcept.ignore") #[[ATTR0]]
-; CHECK-NEXT:    ret i1 [[RESULT]]
+; CHECK-NEXT:    ret i1 false
 ;
 entry:
   %result = call i1 @llvm.experimental.constrained.fcmp.f64(double 2.0, double 0x7ff8000000000000, metadata !"oeq", metadata !"fpexcept.ignore") #0
   ret i1 %result
 }
 
 define i1 @cmp_eq_04() #0 {
 ; CHECK-LABEL: @cmp_eq_04(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RESULT:%.*]] = call i1 @llvm.experimental.constrained.fcmp.f64(double 2.000000e+00, double 0x7FF4000000000000, metadata !"oeq", metadata !"fpexcept.ignore") #[[ATTR0]]
-; CHECK-NEXT:    ret i1 [[RESULT]]
+; CHECK-NEXT:    ret i1 false
 ;
 entry:
   %result = call i1 @llvm.experimental.constrained.fcmp.f64(double 2.0, double 0x7ff4000000000000, metadata !"oeq", metadata !"fpexcept.ignore") #0
   ret i1 %result
 }
 
 define i1 @cmp_eq_05() #0 {
 ; CHECK-LABEL: @cmp_eq_05(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RESULT:%.*]] = call i1 @llvm.experimental.constrained.fcmps.f64(double 2.000000e+00, double 0x7FF8000000000000, metadata !"oeq", metadata !"fpexcept.ignore") #[[ATTR0]]
-; CHECK-NEXT:    ret i1 [[RESULT]]
+; CHECK-NEXT:    ret i1 false
 ;
 entry:
   %result = call i1 @llvm.experimental.constrained.fcmps.f64(double 2.0, double 0x7ff8000000000000, metadata !"oeq", metadata !"fpexcept.ignore") #0
   ret i1 %result
 }
 
 define i1 @cmp_eq_06() #0 {
 ; CHECK-LABEL: @cmp_eq_06(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[RESULT:%.*]] = call i1 @llvm.experimental.constrained.fcmps.f64(double 2.000000e+00, double 0x7FF4000000000000, metadata !"oeq", metadata !"fpexcept.ignore") #[[ATTR0]]
-; CHECK-NEXT:    ret i1 [[RESULT]]
+; CHECK-NEXT:    ret i1 false
 ;
 entry:
   %result = call i1 @llvm.experimental.constrained.fcmps.f64(double 2.0, double 0x7ff4000000000000, metadata !"oeq", metadata !"fpexcept.ignore") #0
   ret i1 %result
 }
 
 ; Compare with SNAN is NOT folded if the exception behavior mode is not 'ignore'.
 define i1 @cmp_eq_nan_01() #0 {
entry:
   ret i1 %result
 }
 
 ; Compare with QNAN is folded for fcmp but is NOT folded for fcmps if the exception behavior mode is not 'ignore'.
 define i1 @cmp_eq_nan_03() #0 {
 ; CHECK-LABEL: @cmp_eq_nan_03(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[RESULT:%.*]] = call i1 @llvm.experimental.constrained.fcmp.f64(double 0x7FF8000000000000, double 1.000000e+00, metadata !"oeq", metadata !"fpexcept.strict") #[[ATTR0]]
-; CHECK-NEXT:    ret i1 [[RESULT]]
+; CHECK-NEXT:    ret i1 false
 ;
 entry:
   %result = call i1 @llvm.experimental.constrained.fcmp.f64(double 0x7ff8000000000000, double 1.0, metadata !"oeq", metadata !"fpexcept.strict") #0
   ret i1 %result
 }
 
 define i1 @cmp_eq_nan_04() #0 {
 ; CHECK-LABEL: @cmp_eq_nan_04(