Constrained fp OpBundles

[simoll]

Summary

This adds support for the cfp-except and cfp-round operand bundles. These bundles can be tagged on to floating-point intrinsics to specify a non-default floating environment.

The first application, we are preparing with this patch, is to support non-default fp environments in VP intrinsics (eg llvm.vp.fadd). However, constrained fp bundles could replace all redundant intrinsics that are just another intrinsic plus metadata params for the fp environment (eg llvm.experimental.constrained.sqrt).

This patch won't be committed before floating-point VP intrinsics are in (unless people would like to use these bundles right away for constrained fp, of course).

[simoll]

• NFC. Fixed spelling, typos.

[simoll]

nfc, comment cleanup.

[simoll]

• Fixed a failing tests (there is now two more op bundles)
• Rebased.

[kpn]

Do you need tests for the verifier part?

Would replacing some of the constrained intrinsics with operand bundles on other intrinsics have the same safety guarantee in the middle end? Currently the middle end doesn't know about the constrained intrinsics so it handles code conservatively. What about intrinsics the middle end already knows about? Would the new operand bundle be as safe as the constrained intrinsics are now?

[simoll]

In D93455#2460765, @kpn wrote:

Do you need tests for the verifier part?

Probably. With this patch standalone, we can only do "catch-22 testing": all uses of the bundle are illegal. We can extend that once the VP floating-point intrinsics are upstream that actually support these bundles.

Would replacing some of the constrained intrinsics with operand bundles on other intrinsics have the same safety guarantee in the middle end? Currently the middle end doesn't know about the constrained intrinsics so it handles code conservatively.

Not easily. This has the same re-occurring issue as enabling predication on all instructions: if we allow the constrained bundle on regular fp intrinsics overnight, existing optimization code will ignore the bundles and potentially break code.
An elegant way out for this would be to introduce an FPMathIntrinsic helper class that only matches intrinsics in the default fp environment (that is without the bundle). All code that looks for a specific fp math function can then no longer just take the intrinsic id but needs to cast the intrinsic to that helper class first.
Btw, the FPMathIntrinsic helper also fits well into the generalized pattern matching framework (D92086).

What about intrinsics the middle end already knows about?

Constrained bundles and the existing constrained intrinsics can exist side by side if all places that reason about constrained fp do so through the ConstrainedFPIntrinsic class. Once that is the case and we want to switch on bundles on some intrinsics, we only need to wire that up in ConstrainedFPIntrinsic .

Would the new operand bundle be as safe as the constrained intrinsics are now?

Yes, it is illegal to drop operand bundles (see the LangRef) . The bundle is just as stable as the regular operand list of the intrinsics. One thing to take care of is proper use of callsite attributes: when the bundle is used with non-default settings, the call site must have the strictfp attribute and potentially others depending on the fp configuration (eg it may trap, access memory (fp config registers), ..).

[kpn]

In D93455#2462358, @simoll wrote:

In D93455#2460765, @kpn wrote:

Do you need tests for the verifier part?

Probably. With this patch standalone, we can only do "catch-22 testing": all uses of the bundle are illegal. We can extend that once the VP floating-point intrinsics are upstream that actually support these bundles.

Would replacing some of the constrained intrinsics with operand bundles on other intrinsics have the same safety guarantee in the middle end? Currently the middle end doesn't know about the constrained intrinsics so it handles code conservatively.

Not easily. This has the same re-occurring issue as enabling predication on all instructions: if we allow the constrained bundle on regular fp intrinsics overnight, existing optimization code will ignore the bundles and potentially break code.

That's a non-starter. People are relying on the strictfp options and #pragmas to work correctly already. That's why I had to add to clang the ability to turn it off the command-line option on targets that don't support it yet. So regressing and making strictfp unsafe again is just not an option.

An elegant way out for this would be to introduce an FPMathIntrinsic helper class that only matches intrinsics in the default fp environment (that is without the bundle). All code that looks for a specific fp math function can then no longer just take the intrinsic id but needs to cast the intrinsic to that helper class first.
Btw, the FPMathIntrinsic helper also fits well into the generalized pattern matching framework (D92086).

If there's a way to replace the constrained intrinsics with other, existing intrinsics plus operand bundles _while_ keeping the safety guarantee that we currently have then that sounds good. But the safety _must_ be the default until we get to updating optimizations to understand strictfp.

What about intrinsics the middle end already knows about?

Constrained bundles and the existing constrained intrinsics can exist side by side if all places that reason about constrained fp do so through the ConstrainedFPIntrinsic class. Once that is the case and we want to switch on bundles on some intrinsics, we only need to wire that up in ConstrainedFPIntrinsic .

I see no reason to keep duplicate intrinsics. Once other intrinsics are able to replace the constrained ones then we should drop the llvm.experimental.constrained.* set. We do need to make sure there's an easy way to check that an intrinsic call is one of these constrained-replacement intrinsics, if only for the IR Verifier, but hopefully that'll be straightforward and not error prone.

Would the new operand bundle be as safe as the constrained intrinsics are now?

Yes, it is illegal to drop operand bundles (see the LangRef) . The bundle is just as stable as the regular operand list of the intrinsics. One thing to take care of is proper use of callsite attributes: when the bundle is used with non-default settings, the call site must have the strictfp attribute and potentially others depending on the fp configuration (eg it may trap, access memory (fp config registers), ..).

I still have D68233 open, but I'm blocked because clang's C++ support isn't ready yet. And clang's handling of the AST isn't ready yet either because it drops metadata. Plus, I have the successor to D68233 still that verifies that non-strict and strictfp are never mixed in a function.

[simoll]

In D93455#2463506, @kpn wrote:

In D93455#2462358, @simoll wrote:

In D93455#2460765, @kpn wrote:

Do you need tests for the verifier part?

Probably. With this patch standalone, we can only do "catch-22 testing": all uses of the bundle are illegal. We can extend that once the VP floating-point intrinsics are upstream that actually support these bundles.

Would replacing some of the constrained intrinsics with operand bundles on other intrinsics have the same safety guarantee in the middle end? Currently the middle end doesn't know about the constrained intrinsics so it handles code conservatively.

Not easily. This has the same re-occurring issue as enabling predication on all instructions: if we allow the constrained bundle on regular fp intrinsics overnight, existing optimization code will ignore the bundles and potentially break code.

That's a non-starter. People are relying on the strictfp options and #pragmas to work correctly already. That's why I had to add to clang the ability to turn it off the command-line option on targets that don't support it yet. So regressing and making strictfp unsafe again is just not an option.

An elegant way out for this would be to introduce an FPMathIntrinsic helper class that only matches intrinsics in the default fp environment (that is without the bundle). All code that looks for a specific fp math function can then no longer just take the intrinsic id but needs to cast the intrinsic to that helper class first.
Btw, the FPMathIntrinsic helper also fits well into the generalized pattern matching framework (D92086).

If there's a way to replace the constrained intrinsics with other, existing intrinsics plus operand bundles _while_ keeping the safety guarantee that we currently have then that sounds good. But the safety _must_ be the default until we get to updating optimizations to understand strictfp.

That's why i am proposing to initially use these bundles only for the VP intrinsics and keep the constrained fp intrinsics in place. When we are sure that all optimizations use the proper abstractions, we can move to replace constrained fp intrinsics with the bundled version instead.

What about intrinsics the middle end already knows about?

Constrained bundles and the existing constrained intrinsics can exist side by side if all places that reason about constrained fp do so through the ConstrainedFPIntrinsic class. Once that is the case and we want to switch on bundles on some intrinsics, we only need to wire that up in ConstrainedFPIntrinsic .

I see no reason to keep duplicate intrinsics. Once other intrinsics are able to replace the constrained ones then we should drop the llvm.experimental.constrained.* set. We do need to make sure there's an easy way to check that an intrinsic call is one of these constrained-replacement intrinsics, if only for the IR Verifier, but hopefully that'll be straightforward and not error prone.

If we can cast it to a ConstrainedFPIntrinsic, it is legal either because it is an existring intrinsic + bundle or an intrinsics of the llvm.experimental.constrained.* kind. And i agree, we should retire the constrained intrinsics as soon as it is safe to do so.

Would the new operand bundle be as safe as the constrained intrinsics are now?

Yes, it is illegal to drop operand bundles (see the LangRef) . The bundle is just as stable as the regular operand list of the intrinsics. One thing to take care of is proper use of callsite attributes: when the bundle is used with non-default settings, the call site must have the strictfp attribute and potentially others depending on the fp configuration (eg it may trap, access memory (fp config registers), ..).

I still have D68233 open, but I'm blocked because clang's C++ support isn't ready yet. And clang's handling of the AST isn't ready yet either because it drops metadata. Plus, I have the successor to D68233 still that verifies that non-strict and strictfp are never mixed in a function.

Wait. So you are saying clang drop the metadata arguments in constrained fp intrinsics?

[kpn]

In D93455#2465495, @simoll wrote:

In D93455#2463506, @kpn wrote:

In D93455#2462358, @simoll wrote:

In D93455#2460765, @kpn wrote:

Do you need tests for the verifier part?

Probably. With this patch standalone, we can only do "catch-22 testing": all uses of the bundle are illegal. We can extend that once the VP floating-point intrinsics are upstream that actually support these bundles.

Would replacing some of the constrained intrinsics with operand bundles on other intrinsics have the same safety guarantee in the middle end? Currently the middle end doesn't know about the constrained intrinsics so it handles code conservatively.

Not easily. This has the same re-occurring issue as enabling predication on all instructions: if we allow the constrained bundle on regular fp intrinsics overnight, existing optimization code will ignore the bundles and potentially break code.

That's a non-starter. People are relying on the strictfp options and #pragmas to work correctly already. That's why I had to add to clang the ability to turn it off the command-line option on targets that don't support it yet. So regressing and making strictfp unsafe again is just not an option.

An elegant way out for this would be to introduce an FPMathIntrinsic helper class that only matches intrinsics in the default fp environment (that is without the bundle). All code that looks for a specific fp math function can then no longer just take the intrinsic id but needs to cast the intrinsic to that helper class first.
Btw, the FPMathIntrinsic helper also fits well into the generalized pattern matching framework (D92086).

If there's a way to replace the constrained intrinsics with other, existing intrinsics plus operand bundles _while_ keeping the safety guarantee that we currently have then that sounds good. But the safety _must_ be the default until we get to updating optimizations to understand strictfp.

That's why i am proposing to initially use these bundles only for the VP intrinsics and keep the constrained fp intrinsics in place. When we are sure that all optimizations use the proper abstractions, we can move to replace constrained fp intrinsics with the bundled version instead.

OK, that sounds like a good plan. And clang uses the IRBuilder with its easy support for constrained floating point, so the number of places to touch is smaller than one might think. Mostly just CGBuiltins.cpp would need to be changed.

Would the new operand bundle be as safe as the constrained intrinsics are now?

Yes, it is illegal to drop operand bundles (see the LangRef) . The bundle is just as stable as the regular operand list of the intrinsics. One thing to take care of is proper use of callsite attributes: when the bundle is used with non-default settings, the call site must have the strictfp attribute and potentially others depending on the fp configuration (eg it may trap, access memory (fp config registers), ..).

I still have D68233 open, but I'm blocked because clang's C++ support isn't ready yet. And clang's handling of the AST isn't ready yet either because it drops metadata. Plus, I have the successor to D68233 still that verifies that non-strict and strictfp are never mixed in a function.

Wait. So you are saying clang drop the metadata arguments in constrained fp intrinsics?

It's not quite that simple. First, if clang is in strictfp mode for a function then it stays in strictfp mode for that function. So we'll still get constrained intrinsics. There's no issue there.

If the command line arguments are used to enable strictfp AND no #pragma is used to affect the compiler's idea of the FP environment then there's no problem. It will use the global metadata which will be in the AST so, again, no problem.

However, if strictfp is enabled on the command line and #pragma is used on the FP environment then we're running into places where the global metadata is present in the AST _instead_ _of_ the correct metadata based on the #pragma. You can see this in tests that set "maytrap" on the RUN line and then use a #pragma to go fully strict at the top of the test. Broken tests will have a mixture of strict and maytrap. Note that only a handful of tests have been changed to do that so far.

It's a work in progress to fix, and I still haven't figured out a good way to prevent regressions.

llvm/docs/LangRef.rst

 A "gc-live" operand bundle is only valid on a :ref:`gc.statepoint <gc_statepoint>`
 intrinsic. The operand bundle must contain every pointer to a garbage collected
 object which potentially needs to be updated by the garbage collector.
 
 When lowered, any relocated value will be recorded in the corresponding
 :ref:`stackmap entry <statepoint-stackmap-format>`.  See the intrinsic description
 for further details.
 
+.. _ob_cfp_round:
+
+Floating-point Rounding Mode Operand Bundles
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Rounding mode operand bundles are characterized by the ``"cfp-round"`` operand
+bundle tag. They have one operand which is a metadata string. The possible
+rounding mode strings are the same as for
+:ref:`Constrained Floating-Point Intrinsics <constrainedfp>`.
+An operation with a rounding mode operand bundle must have the `strictfp`
+attribute.
+
+.. code-block:: llvm
+
+      ...
+
+      %t = call <8 x double> strictfp @llvm.vp.fadd.v8f64(<8 x double> %x, <8 x double> %y, <8 x i1> %mask, i32 %avl) [ "cfp-round"(metadata !"round.downard") ]
+
+.. _ob_cfp_except:
+
+Floating-point Exception Behavior Operand Bundles
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Exception behavior operand bundles are characterized by the ``"cfp-except"``
+operand bundle tag.  They have one operand which is a metadata string. The
+possible exception behavior strings are the same as for :ref:`Constrained
+Floating-Point Intrinsics <constrainedfp>`.
+An operation with a exception behavior operand bundle must have the `strictfp`
+attribute.
+
+
+.. code-block:: llvm
+
+      ...
+
+      %t = call <8 x double> strictfp @llvm.vp.fdiv.v8f64(<8 x double> %x, <8 x double> %y, <8 x i1> %mask, i32 %avl) [ "cfp-except"(metadata !"fpexcept.strict") ]
+
 .. _moduleasm:
 
 Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
 to the GCC "file scope inline asm" blocks. These blocks are internally
 concatenated by LLVM and treated as a single unit, but may be separated

llvm/include/llvm/IR/IntrinsicInst.h

public:
   static Optional<int> GetVectorLengthParamPos(Intrinsic::ID IntrinsicID);
 
   /// The llvm.vp.* intrinsics for this instruction Opcode
   static Intrinsic::ID GetForOpcode(unsigned OC);
 
   // Whether \p ID is a VP intrinsic ID.
   static bool IsVPIntrinsic(Intrinsic::ID);
 
+  /// Constrained FP {
+
+  // Whether the VP intrinsic \p ID can have a rounding mode bundle.
+  static bool HasRoundingMode(Intrinsic::ID);
+  Optional<RoundingMode> getRoundingMode() const;
+
+  // Whether the VP intrinsic \p ID can have a exception behavior
+  // bundle.
+  static bool HasExceptionBehavior(Intrinsic::ID);
+  Optional<fp::ExceptionBehavior> getExceptionBehavior() const;
+
+  /// } Constrained FP
+
   /// \return the mask parameter or nullptr.
   Value *getMaskParam() const;
 
   /// \return the vector length parameter or nullptr.
   Value *getVectorLengthParam() const;
 
   /// \return whether the vector length param can be ignored.
   bool canIgnoreVectorLengthParam() const;

llvm/include/llvm/IR/LLVMContext.h

#undef LLVM_FIXED_MD_KIND
   /// LLVMContext::LLVMContext().
   enum : unsigned {
     OB_deopt = 0,         // "deopt"
     OB_funclet = 1,       // "funclet"
     OB_gc_transition = 2, // "gc-transition"
     OB_cfguardtarget = 3, // "cfguardtarget"
     OB_preallocated = 4,  // "preallocated"
     OB_gc_live = 5,       // "gc-live"
+    OB_cfp_round = 6,     // "cfp-round"
+    OB_cfp_except = 7,    // "cfp-except"
   };
 
   /// getMDKindID - Return a unique non-zero ID for the specified metadata kind.
   /// This ID is uniqued across modules in the current LLVMContext.
   unsigned getMDKindID(StringRef Name) const;
 
   /// getMDKindNames - Populate client supplied SmallVector with the name for
   /// custom metadata IDs registered in this LLVMContext.

llvm/lib/AsmParser/LLParser.cpp

while (Lex.getKind() != lltok::rsquare) {
     std::vector<Value *> Inputs;
     while (Lex.getKind() != lltok::rparen) {
       // If this isn't the first input, we need a comma.
       if (!Inputs.empty() &&
           parseToken(lltok::comma, "expected ',' in input list"))
         return true;
 
       Type *Ty = nullptr;
+      if (parseType(Ty))
+        return true;
+
       Value *Input = nullptr;
-      if (parseType(Ty) || parseValue(Ty, Input, PFS))
+      if (Ty->isMetadataTy()) {
+        if (parseMetadataAsValue(Input, PFS))
           return true;
+      } else if (parseValue(Ty, Input, PFS))
+        return true;
+
       Inputs.push_back(Input);
     }
 
     BundleList.emplace_back(std::move(Tag), std::move(Inputs));
 
     Lex.Lex(); // Lex the ')'.
   }
 

llvm/lib/IR/IntrinsicInst.cpp

 #define BEGIN_REGISTER_VP_INTRINSIC(VPID, MASKPOS, VLENPOS)                    \
   case Intrinsic::VPID:                                                        \
     break;
 #include "llvm/IR/VPIntrinsics.def"
   }
   return true;
 }
 
+bool VPIntrinsic::HasRoundingMode(Intrinsic::ID) {
+  return false; // TODO enable for fp ops.
+}
+
+bool VPIntrinsic::HasExceptionBehavior(Intrinsic::ID) {
+  return false; // TODO enable for fp ops.
+}
+
+Optional<RoundingMode> VPIntrinsic::getRoundingMode() const {
+  auto Bundle = this->getOperandBundle("cfp-round");
+  if (!Bundle)
+    return None;
+  Metadata *MD = cast<MetadataAsValue>(Bundle->Inputs[0])->getMetadata();
+  if (!MD || !isa<MDString>(MD))
+    return None;
+  return StrToRoundingMode(cast<MDString>(MD)->getString());
+}
+
+Optional<fp::ExceptionBehavior> VPIntrinsic::getExceptionBehavior() const {
+  auto Bundle = this->getOperandBundle("cfp-except");
+  if (!Bundle)
+    return None;
+  Metadata *MD = cast<MetadataAsValue>(Bundle->Inputs[0])->getMetadata();
+  if (!MD || !isa<MDString>(MD))
+    return None;
+
+  return StrToExceptionBehavior(cast<MDString>(MD)->getString());
+}
+
 // Equivalent non-predicated opcode
 unsigned VPIntrinsic::GetFunctionalOpcodeForVP(Intrinsic::ID ID) {
   unsigned FunctionalOC = Instruction::Call;
   switch (ID) {
   default:
     break;
 #define BEGIN_REGISTER_VP_INTRINSIC(VPID, ...) case Intrinsic::VPID:
 #define HANDLE_VP_TO_OPC(OPC) FunctionalOC = Instruction::OPC;

llvm/lib/IR/LLVMContext.cpp

assert(PreallocatedEntry->second == LLVMContext::OB_preallocated &&
          "preallocated operand bundle id drifted!");
   (void)PreallocatedEntry;
 
   auto *GCLiveEntry = pImpl->getOrInsertBundleTag("gc-live");
   assert(GCLiveEntry->second == LLVMContext::OB_gc_live &&
          "gc-transition operand bundle id drifted!");
   (void)GCLiveEntry;
 
+  auto *CFPRoundEntry = pImpl->getOrInsertBundleTag("cfp-round");
+  assert(CFPRoundEntry->second == LLVMContext::OB_cfp_round &&
+         "cfp-round operand bundle id drifted!");
+  (void)CFPRoundEntry;
+
+  auto *CFPExceptEntry = pImpl->getOrInsertBundleTag("cfp-except");
+  assert(CFPExceptEntry->second == LLVMContext::OB_cfp_except &&
+         "cfp-except operand bundle id drifted!");
+  (void)CFPExceptEntry;
+
   SyncScope::ID SingleThreadSSID =
       pImpl->getOrInsertSyncScopeID("singlethread");
   assert(SingleThreadSSID == SyncScope::SingleThread &&
          "singlethread synchronization scope ID drifted!");
   (void)SingleThreadSSID;
 
   SyncScope::ID SystemSSID =
       pImpl->getOrInsertSyncScopeID("");

llvm/lib/IR/Verifier.cpp

 #include "llvm/IR/ConstantRange.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DebugInfo.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Dominators.h"
+#include "llvm/IR/FPEnv.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalAlias.h"
 #include "llvm/IR/GlobalValue.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/InstVisitor.h"
 #include "llvm/IR/InstrTypes.h"
 #include "llvm/IR/Instruction.h"
void verifyFunctionAttrs(FunctionType *FT, AttributeList Attrs,
                            const Value *V, bool IsIntrinsic);
   void verifyFunctionMetadata(ArrayRef<std::pair<unsigned, MDNode *>> MDs);
 
   void visitConstantExprsRecursively(const Constant *EntryC);
   void visitConstantExpr(const ConstantExpr *CE);
   void verifyStatepoint(const CallBase &Call);
   void verifyFrameRecoverIndices();
   void verifySiblingFuncletUnwinds();
+  void verifyConstrainedFPBundles(const Instruction &);
 
   void verifyFragmentExpression(const DbgVariableIntrinsic &I);
   template <typename ValueOrMetadata>
   void verifyFragmentExpression(const DIVariable &V,
                                 DIExpression::FragmentInfo Fragment,
                                 ValueOrMetadata *Desc);
   void verifyFnArgs(const DbgVariableIntrinsic &I);
   void verifyNotEntryValue(const DbgVariableIntrinsic &I);
do {
       Active.insert(PredPad);
     } while (true);
     // Each node only has one successor, so we've walked all the active
     // nodes' successors.
     Active.clear();
   }
 }
 
+void Verifier::verifyConstrainedFPBundles(const Instruction &I) {
+  auto *CB = dyn_cast<CallBase>(&I);
+  if (!CB)
+    return;
+  auto ExceptBundle = CB->getOperandBundle("cfp-except");
+  auto RoundBundle = CB->getOperandBundle("cfp-round");
+  if (!ExceptBundle && !RoundBundle)
+    return;
+
+  auto *VPIntrin = dyn_cast<VPIntrinsic>(&I);
+  Assert(
+      VPIntrin,
+      "Constrained FP bundles only enabled for Vector Predication Intrinsics",
+      VPIntrin);
+  Assert(!RoundBundle ||
+             VPIntrinsic::HasRoundingMode(VPIntrin->getIntrinsicID()),
+         "Intrinsic does not accept a constrained fp rounding mode.", VPIntrin);
+  Assert(!ExceptBundle ||
+             VPIntrinsic::HasExceptionBehavior(VPIntrin->getIntrinsicID()),
+         "Intrinsic does not accept a constrained fp exception behavior.",
+         VPIntrin);
+
+  Assert(VPIntrin->hasFnAttr(Attribute::StrictFP),
+         "Constrained fp operation must have the 'strictfp' attribute.",
+         VPIntrin);
+
+  if (RoundBundle) {
+    Assert(RoundBundle->Inputs.size() == 1,
+           "Constrained fp rounding mode bundle must have exactly one operand.",
+           VPIntrin);
+    auto &RoundInput = *RoundBundle->Inputs[0];
+    Assert(isa<MetadataAsValue>(RoundInput),
+           "Constrained fp rounding mode is not a metadata string.",
+           RoundInput);
+    auto *RoundString =
+        dyn_cast<MDString>(cast<MetadataAsValue>(RoundInput).getMetadata());
+    Assert(RoundString,
+           "Constrained fp rounding mode is not a metadata string.",
+           RoundInput);
+    auto RoundOpt = VPIntrin->getRoundingMode();
+    Assert(RoundOpt.hasValue(), "Invalid rounding mode metadata.", RoundString);
+  }
+
+  if (ExceptBundle) {
+    Assert(ExceptBundle->Inputs.size() == 1,
+           "Constrained fp exception behavior bundle must have exactly only "
+           "one operand.",
+           VPIntrin);
+    auto &ExceptInput = *ExceptBundle->Inputs[0];
+    Assert(isa<MetadataAsValue>(ExceptInput),
+           "Constrained fp exception behavior is not a metadata string.",
+           ExceptInput);
+    auto *ExceptString =
+        dyn_cast<MDString>(cast<MetadataAsValue>(ExceptInput).getMetadata());
+    Assert(ExceptString,
+           "Constrained fp exception behavior is not a metadata string.",
+           ExceptInput);
+    auto ExceptOpt = VPIntrin->getExceptionBehavior();
+    Assert(ExceptOpt.hasValue(), "Invalid exception behavior metadata.",
+           ExceptString);
+  }
+}
+
 // visitFunction - Verify that a function is ok.
 //
 void Verifier::visitFunction(const Function &F) {
   visitGlobalValue(F);
 
   // Check function arguments.
   FunctionType *FT = F.getFunctionType();
   unsigned NumArgs = F.arg_size();
if (Function *F = Call.getCalledFunction())
     if (Intrinsic::ID ID = (Intrinsic::ID)F->getIntrinsicID())
       visitIntrinsicCall(ID, Call);
 
   // Verify that a callsite has at most one "deopt", at most one "funclet", at
   // most one "gc-transition", at most one "cfguardtarget",
   // and at most one "preallocated" operand bundle.
   bool FoundDeoptBundle = false, FoundFuncletBundle = false,
        FoundGCTransitionBundle = false, FoundCFGuardTargetBundle = false,
-       FoundPreallocatedBundle = false, FoundGCLiveBundle = false;;
+       FoundPreallocatedBundle = false, FoundGCLiveBundle = false,
+       FoundCFPExceptBundle = false, FoundCFPRoundBundle = false;
   for (unsigned i = 0, e = Call.getNumOperandBundles(); i < e; ++i) {
     OperandBundleUse BU = Call.getOperandBundleAt(i);
     uint32_t Tag = BU.getTagID();
     if (Tag == LLVMContext::OB_deopt) {
       Assert(!FoundDeoptBundle, "Multiple deopt operand bundles", Call);
       FoundDeoptBundle = true;
     } else if (Tag == LLVMContext::OB_gc_transition) {
       Assert(!FoundGCTransitionBundle, "Multiple gc-transition operand bundles",
if (Tag == LLVMContext::OB_deopt) {
                  Input->getIntrinsicID() == Intrinsic::call_preallocated_setup,
              "\"preallocated\" argument must be a token from "
              "llvm.call.preallocated.setup",
              Call);
     } else if (Tag == LLVMContext::OB_gc_live) {
       Assert(!FoundGCLiveBundle, "Multiple gc-live operand bundles",
              Call);
       FoundGCLiveBundle = true;
+    } else if (Tag == LLVMContext::OB_cfp_round) {
+      Assert(!FoundCFPRoundBundle, "Multiple cfp-round operand bundles", Call);
+      FoundCFPRoundBundle = true;
+    } else if (Tag == LLVMContext::OB_cfp_except) {
+      Assert(!FoundCFPExceptBundle, "Multiple cfp-except operand bundles",
+             Call);
+      FoundCFPExceptBundle = true;
     }
   }
 
   // Verify that each inlinable callsite of a debug-info-bearing function in a
   // debug-info-bearing function has a debug location attached to it. Failure to
   // do so causes assertion failures when the inliner sets up inline scope info.
   if (Call.getFunction()->getSubprogram() && Call.getCalledFunction() &&
       Call.getCalledFunction()->getSubprogram())
if (MDNode *N = I.getDebugLoc().getAsMDNode()) {
     visitMDNode(*N, AreDebugLocsAllowed::Yes);
   }
 
   if (auto *DII = dyn_cast<DbgVariableIntrinsic>(&I)) {
     verifyFragmentExpression(*DII);
     verifyNotEntryValue(*DII);
   }
 
+  verifyConstrainedFPBundles(I);
+
   SmallVector<std::pair<unsigned, MDNode *>, 4> MDs;
   I.getAllMetadata(MDs);
   for (auto Attachment : MDs) {
     unsigned Kind = Attachment.first;
     auto AllowLocs =
         (Kind == LLVMContext::MD_dbg || Kind == LLVMContext::MD_loop)
             ? AreDebugLocsAllowed::Yes
             : AreDebugLocsAllowed::No;

llvm/test/Bitcode/operand-bundles-bc-analyzer.ll

 ; RUN: llvm-as < %s | llvm-bcanalyzer -dump -disable-histogram | FileCheck %s
 
 ; CHECK:  <OPERAND_BUNDLE_TAGS_BLOCK
 ; CHECK-NEXT:    <OPERAND_BUNDLE_TAG
 ; CHECK-NEXT:    <OPERAND_BUNDLE_TAG
 ; CHECK-NEXT:    <OPERAND_BUNDLE_TAG
 ; CHECK-NEXT:    <OPERAND_BUNDLE_TAG
 ; CHECK-NEXT:    <OPERAND_BUNDLE_TAG
 ; CHECK-NEXT:    <OPERAND_BUNDLE_TAG
 ; CHECK-NEXT:    <OPERAND_BUNDLE_TAG
 ; CHECK-NEXT:    <OPERAND_BUNDLE_TAG
+; CHECK-NEXT:    <OPERAND_BUNDLE_TAG
+; CHECK-NEXT:    <OPERAND_BUNDLE_TAG
 ; CHECK-NEXT:  </OPERAND_BUNDLE_TAGS_BLOCK
 
 ; CHECK:   <FUNCTION_BLOCK
 ; CHECK:    <OPERAND_BUNDLE
 ; CHECK:    <OPERAND_BUNDLE
 ; CHECK-NOT: <OPERAND_BUNDLE
 ; CHECK:  </FUNCTION_BLOCK