[FPEnv] Constrained FCmp intrinsics

[uweigand]

Summary

This is a continuation of https://reviews.llvm.org/D54649, which was abandoned by @cameron.mcinally

The major differences to Cameron's approach include:

• I did not actually require any TableGen changes, but was able to represent the overloaded intrinsic types using existing mechanisms. This solution looks correct to me, but if I'm missing anything here, please let me know ...
• This adds a full set of compare intrinsics for all comparison codes.
• I've added (mostly) complete SystemZ back-end support to actually generate correct code for all of them -- this is also to verify that the use of Custom expansion of strict operations actually allows the back-end to do what it needs to do.

This patch is still not complete, but I wanted to show it now to ask for feedback.

Some areas that definitely need more work are:

• Vector type legalization of invalid vector types involving STRICT_FSETCC doesn't work yet.
• Signalling comparisons are not supported yet.
• We may also need strict versions of SELECT_CC and BR_CC. I haven't implemented those yet since they aren't really necessary for SystemZ, but some other platforms may require them.
• The X86 back-end changes are incomplete, they're right now the bare minimum to keep Cameron's two test cases working.
• There are many SETCC-related transformations and optimizations in the SelectionDAG codegen. Many of these are also valid for the strict FP case. I haven't found a simple way yet to write those transformations so that can easily handle both cases without duplicating a lot of boilerplate code ... Right now the patch implements the minimum set of those to get the SystemZ test cases to work (at least generating the same code as the non-strict versions for extremely simple use cases).

[cameron.mcinally]

I did not actually require any TableGen changes, but was able to represent the overloaded intrinsic types using existing mechanisms. This solution looks correct to me, but if I'm missing anything here, please let me know ...

That sounds right. LLVMScalarOrSameVectorWidth did not exist when D54649 was proposed, IINM.

This adds a full set of compare intrinsics for all comparison codes.

There's really no reasonable alternative, correct? I believe that negating < and > would get messy wrt signals when a NaN is present. I don't think that savings is worth the effort (e.g. readability).

The X86 back-end changes are incomplete, they're right now the bare minimum to keep Cameron's two test cases working.

I'd be ok with leaving these changes to a later patch. Your call...

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

7073

These DAG.getNode changes could probably be broken out into a standalone Diff.

lib/Target/SystemZ/SystemZISelLowering.cpp

2171

This is probably worthy of a comment.

2655

I notice the static_cast<SystemZISD::NodeType> has been dropped. Is this ok?

lib/Target/SystemZ/SystemZISelLowering.h

248

preoduce typo could use a fix.

[uweigand]

In D69281#1723640, @cameron.mcinally wrote:

I did not actually require any TableGen changes, but was able to represent the overloaded intrinsic types using existing mechanisms. This solution looks correct to me, but if I'm missing anything here, please let me know ...

That sounds right. LLVMScalarOrSameVectorWidth did not exist when D54649 was proposed, IINM.

OK, thanks for your review!

This adds a full set of compare intrinsics for all comparison codes.

There's really no reasonable alternative, correct? I believe that negating < and > would get messy wrt signals when a NaN is present. I don't think that savings is worth the effort (e.g. readability).

Agreed.

The X86 back-end changes are incomplete, they're right now the bare minimum to keep Cameron's two test cases working.

I'd be ok with leaving these changes to a later patch. Your call...

OK, I'm dropping the X86-related changes for now.

Due to the switch to the github monorepo, it seems all the inline comments disappeared -- sorry for that.

Anyway, I believe I've anyway addressed them all:

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp:7073
These DAG.getNode changes could probably be broken out into a standalone Diff.

Agreed. Since this is just a simple refactoring, I've checked this in as 664f84e246478db82be2871f36fd1a523d9f2731.

lib/Target/SystemZ/SystemZISelLowering.cpp:2171
This is probably worthy of a comment.

Agreed. Added comment.

lib/Target/SystemZ/SystemZISelLowering.cpp:2655
I notice the static_cast<SystemZISD::NodeType> has been dropped. Is this ok?

Since the return value of that function was just plain int anyway, that static_cast didn't really have any point as far as I can see ...

lib/Target/SystemZ/SystemZISelLowering.h:248
preoduce typo could use a fix.

Checked in separately as d4a7855b68d4d53f121209333d5f2796731ab1f5

[andrew.w.kaylor]

This looks like a very good start. I've talked to @pengfei about the x86 backend support for this. As long as x86 doesn't fail horribly, I'd be OK with the x86 work being done in a separate patch that depends on this one.

Do you have a plan for handling quiet versus signaling predicates?

[cameron.mcinally]

Hmm, something is weird (with my Diff, at least). It looks like changes are listed twice (e.g. lib/IR/Verifier.cpp and llvm/lib/IR/Verifier.cpp). Maybe this is an SVN->GIT side-effect and the patch needs a rebase?

[uweigand]

In D69281#1740932, @cameron.mcinally wrote:

Hmm, something is weird (with my Diff, at least). It looks like changes are listed twice (e.g. lib/IR/Verifier.cpp and llvm/lib/IR/Verifier.cpp). Maybe this is an SVN->GIT side-effect and the patch needs a rebase?

In the current diff I only see the new llvm/lib/... paths. It is indeed the case that the diff I submitted originally had the SVN-style lib/... paths, but I updated it with a new patch after the GIT transition.

[cameron.mcinally]

Oh, it's gone now. Maybe it was a temporary hiccup. Will review this afternoon...

[cameron.mcinally]

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

1420 ↗

(On Diff #227715)

This throws away strict-ness for the expanded scalar selects. Is that deliberate?

I would have expected this to produce scalar STRICT_FSETCC nodes instead.

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

3832 ↗

(On Diff #227715)

This else can be removed.

llvm/lib/Target/SystemZ/SystemZISelLowering.cpp

2514 ↗

(On Diff #227715)

Missing space before ?.

[uweigand]

Thanks for the review!

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

1420 ↗

(On Diff #227715)

I don't see how this throws away strict-ness. Note that we do produce scalar STRICT_FSETCC nodes (that's in the ScalarOp = DAG.getNode(Op->getOpcode() ...) line). The additional select just ensures the (integral) output has the same value that a vector SETCC would have, instead of the (difference) value that a scalar SETCC has. Since this is completely an integer operation, strict-ness doesn't apply.

[cameron.mcinally]

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

1420 ↗

(On Diff #227715)

Oh, I see it now. I misread this change...

[simoll]

llvm/include/llvm/IR/Intrinsics.td

750–791 ↗

(On Diff #227715)

Out of curiosity: what is your motivation to have one intrinsic per fcmp predicate instead of, say, an i8 immarg?

[uweigand]

llvm/include/llvm/IR/Intrinsics.td

750–791 ↗

(On Diff #227715)

I didn't really make the decision, just took it over from the original patch. But I agree it seems to make sense simply from a perspective of being explicit in the IR: if there's just a random numeric value, it would make the IR harder to read/write.

[cameron.mcinally]

llvm/include/llvm/IR/Intrinsics.td

750–791 ↗

(On Diff #227715)

It would be hard to read -- unless we wrote an IR decoding routine somewhere. That seems weird though.

[andrew.w.kaylor]

llvm/include/llvm/IR/Intrinsics.td

750–791 ↗

(On Diff #227715)

This is why we used metadata strings for the constraint arguments. I'm still not entirely satisfied with that. Would it be reasonable to make these arguments constants and add a custom printer and parser for these intrinsics? What about token arguments?

[simoll]

llvm/include/llvm/IR/Intrinsics.td

750–791 ↗

(On Diff #227715)

This is why we used metadata strings for the constraint arguments. I'm still not entirely satisfied with that.

You could invent some fpenv operand bundle for fpexcept, fpround, as in:

%p = llvm.experimental.constrained.fcmp_ugt(%x, %y) [ "fpenv"(metadata !"fpround.tonearest", metadata !"fpexcept.strict") ]

Omission would then imply the default fp env (i'd like that for the VP fp intrinsics).

From https://llvm.org/docs/LangRef.html#operand-bundles

An operand bundle at a call site cannot change the implementation of the called function.

I am not sure what to make out of that statement. I guess it should be ok to also use operand bundles also for the comparison predicate? Otw, it might be possible to change to OpBound specification to allow different implementations.

Personally, i'd prefer we had a more flexible attribute mechanism, so we could tag-on all of these things:

%p = fcmp %x, %y cmp(ugt) fpround(tonearest) fpexcept(strict)

[andrew.w.kaylor]

llvm/include/llvm/IR/Intrinsics.td

750–791 ↗

(On Diff #227715)

I like the idea of operand bundles for the constraint arguments. They seem naturally separate. I'm less comfortable with the comparison predicate being in the operand bundle because it is essential to the semantic meaning of the operation.

I started an experiment yesterday using token arguments instead of metadata to represent the rounding mode and exception behavior. I can easily shift the tokens to an operand bundle. I think we can use a token argument for the fcmp intrinsic predicate. I'll post an RFC to the mailing list soon to solicit opinions about using token arguments this way.

Regarding more flexible attributes, the problem we were trying to solve with the intrinsics was introducing new semantics in a way that wouldn't break existing optimizations that didn't know about the new semantics. Once these intrinsics are well established and well supported we might be able to merge their behavior back into the instructions.

[andrew.w.kaylor]

llvm/include/llvm/IR/Intrinsics.td

750–791 ↗

(On Diff #227715)

I've just uploaded D70261, showing what I have in mind for implementing the FP constraints as tokens in an operand bundle. I think the fcmp predicates could be represented by a new token type in the same way, but as I said previously I'd prefer the predicate to be a proper argument to the constrained fcmp intrinsic.

[uweigand]

Updated patch to only use a single llvm.experimental.constrained.fcmp intrinsic, where the predicate is passed as argument.

I agree that it does look preferable to have a single intrinsic, if only to make potential future IR optimizations simpler structurally to the corresponding operations on fcmp statements.

For now I'm simply using a metadata string to hold the predicate; if we come up with some form of "enum constant" in the IR in the future, it should be straightforward to change this (users are encapsulated).

[uweigand]

This is now a functionally complete patch to implement constrained floating-point comparison intrinsics.

The major differences to the prior version are:

• Added documentation for the new intrinsics.
• Added support for signaling comparisons.
• Updated for common-code changes (ConstrainedOps.def).
• Removed (incorrect and unnecessary) optimization in DAGCombine.
• Reviewed SystemZ back-end optimizations to make sure they respect strict semantics.

There are still opportunities to implement more optimizations in common code, but that can wait for later patches.

To handling signaling comparisons I now added a second intrinsic llvm.experimental.constrained.fcmps, which works otherwise the same as llvm.experiemental.constrained.fcmp. (Another design choice could have been to use a single intrinsics with an extra boolean argument, but that seemed to be less useful.)

[andrew.w.kaylor]

I would prefer a single intrinsic/opcode with additional predicates to handle signalling vs. quiet. My main reason is consistency with how the IEEE-754 spec describes comparisons, but that's a weak argument. @craig.topper mentioned to me that there are some additional opcodes that might need to be duplicated if you treat signalling and quiet as separate operations.

[craig.topper]

In D69281#1759494, @andrew.w.kaylor wrote:

I would prefer a single intrinsic/opcode with additional predicates to handle signalling vs. quiet. My main reason is consistency with how the IEEE-754 spec describes comparisons, but that's a weak argument. @craig.topper mentioned to me that there are some additional opcodes that might need to be duplicated if you treat signalling and quiet as separate operations.

I was referring to SELECT_CC and BR_CC. If we have a separate STRICT_FSETCC for signalling and quiet, then we we'll need separate STRICT_SELECT_CC/BR_CC if for signalling and quiet.

[uweigand]

In the end, the reason why I chose to separate STRICT_FSETCC and STRICT_FSETCCS is that this makes it straightforward for the back-end to signal availability of the underlying instructions (which may be distinct).

Specifically, on SystemZ we have quiet vector compare instructions in z13, but we only got the signaling vector compare instructions in z14. This means on z13 we have to scalarize signaling (but not quiet) vector compares. With two DAG opcodes, the back end only has to specify this like so:

setOperationAction(ISD::STRICT_FSETCC, VT, Custom);
if (Subtarget.hasVectorEnhancements1())
  setOperationAction(ISD::STRICT_FSETCCS, VT, Custom);

and everything is just handled correctly automatically.

Having just a single DAG opcode would make this more complicated.

On the other hand, I found handling two opcodes in common code quite straightforward, usually simply adding another case statement to a switch. Adding an extra argument to the DAGnode (sort of like FP_ROUND) also need special code all over the place to handle ...

[craig.topper]

In D69281#1759817, @uweigand wrote:

In the end, the reason why I chose to separate STRICT_FSETCC and STRICT_FSETCCS is that this makes it straightforward for the back-end to signal availability of the underlying instructions (which may be distinct).

Specifically, on SystemZ we have quiet vector compare instructions in z13, but we only got the signaling vector compare instructions in z14. This means on z13 we have to scalarize signaling (but not quiet) vector compares. With two DAG opcodes, the back end only has to specify this like so:

setOperationAction(ISD::STRICT_FSETCC, VT, Custom);
if (Subtarget.hasVectorEnhancements1())
  setOperationAction(ISD::STRICT_FSETCCS, VT, Custom);

and everything is just handled correctly automatically.

Having just a single DAG opcode would make this more complicated.

On the other hand, I found handling two opcodes in common code quite straightforward, usually simply adding another case statement to a switch. Adding an extra argument to the DAGnode (sort of like FP_ROUND) also need special code all over the place to handle ...

Just a thought if we merged signaling into the predicate, could we use the setCondCodeAction(might have the name wrong). On X86 for vectors we have signaling on qnan for LT/LE/GT/GE, but eq/ne/ord/unord only signal on snan. At least prior to AVX. Of course since it’s X86 we probably need custom handling for other reasons so maybe it doesn’t matter.

[uweigand]

In D69281#1759830, @craig.topper wrote:

Just a thought if we merged signaling into the predicate, could we use the setCondCodeAction(might have the name wrong).

Hmm, right now it appears this is only checked for scalar operations, not vector.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

1329 ↗

(On Diff #230775)

The || should be on the line before

1364 ↗

(On Diff #230775)

The || on the line before

[craig.topper]

I didn't look at the SystemZ specifics, but the rest seemed fine

[andrew.w.kaylor]

llvm/docs/LangRef.rst

15728 ↗

(On Diff #230775)

Can we say "will" rather than "may"?

[uweigand]

Address review comments.

[cameron.mcinally]

In D69281#1759494, @andrew.w.kaylor wrote:

I would prefer a single intrinsic/opcode with additional predicates to handle signalling vs. quiet. My main reason is consistency with how the IEEE-754 spec describes comparisons, but that's a weak argument. @craig.topper mentioned to me that there are some additional opcodes that might need to be duplicated if you treat signalling and quiet as separate operations.

I don't have a strong opinion on this, but this would get weird for compareQuietUnordered and compareQuietOrdered. There are no signaling counterparts for those.

[cameron.mcinally]

Oh, or maybe I misunderstood. I assumed Andy meant a single intrinsics with an extra boolean argument, but now I'm not so sure...

If we're adding new predicates, e.g. SGT/QGT, then that's probably fine.

[uweigand]

I would prefer to model the complete set of operations both in a signaling and quiet form. It is true that IEEE does not define a named operation for each of these combinations, but it would still be useful:

• so we can model the capabilities of the hardware ISA (e.g. SystemZ and Power do have an instruction for any of these)
• and the compiler could in fact make use of them via optimization (e.g. an "isSignalingEqual" could be result of an optimization of "x <= y && x >= y").

Note that even today, not all of the predicate codes match an IEEE named operation, e.g. there is none corresponding to "one".

Now, assuming we want to model the complete set of operations, there would still be different ways to do so:

• use distinct intrinsics / opcodes to distinguish signaling vs. quiet operations, each carrying a predicate operand (like this patch currently does)
• use a single intrinsic / opcode with two operands (the predicate / condition code, and a boolean to indicate signaling vs. quiet)
• encode signaling vs. quiet status into the condition code, and use a single intrinsic / opcode with just one operand, the extended condition code

These ways would be generally equivalent, so it is a question of what seems easiest to implement. I had initially experimented with using a single intrinsic / opcode with two operands, but in the end found the current solution with two opcodes simpler overall (in particular because of the setOperationAction issue discussed above). I have not tried extended condition codes -- I'm not sure if this would run into issues elsewhere if those "leak" into other code currently handling condition code that is not aware of the new ones ...

[andrew.w.kaylor]

To clarify, I was suggesting a single intrinsic with quiet/signaling encoded in the predicate. I think the IR definition is a bit cleaner this way and it maps to both the IEEE specification and at least the most common way (I think) this is implemented in x86 hardware and probably others. That said, if doing it this way makes the implementation more complicated I could go along with separate intrinsics and ISD opcodes.

[craig.topper]

In D69281#1762150, @andrew.w.kaylor wrote:

To clarify, I was suggesting a single intrinsic with quiet/signaling encoded in the predicate. I think the IR definition is a bit cleaner this way and it maps to both the IEEE specification and at least the most common way (I think) this is implemented in x86 hardware and probably others. That said, if doing it this way makes the implementation more complicated I could go along with separate intrinsics and ISD opcodes.

Legacy X87 has separate instructions for signalling and quiet. Originally with FCOM/FUCOM that write to the condition code bits in FPSW. Later with FCOMI and FUCOMI that write to EFLAGS. SSE added similar scalar comparisons that are split between signalling and quiet that produce an EFLAGS result, COMISS/COMISD and UCOMISS/UCOMISD. The flags are defined in a weird way that requires two branches or two setcc instructions and a logic op to implement oeq and une.

SSE also added scalar and vector instructions, CMPPS/CMPD/CMPSS/CMPSD, that produce a alls 0s or all 1s result in each element of a vector register. These instructions have 8 encodings, some that are signalling and some that are quiet. And for some behaviors we have to commute operands. They can't represent all 32 possible behaviors. So for vector comparisons we'll need to scalarize when we don't have the right encoding. With AVX the 8 encodings where extended to 32 encodings that support all possible behaviors.

Having two different ISD opcodes is straightforward for FCOM/FCOMI/FUCOMI/FCOMI/COMISS/COMISD/UCOMISS/UCOMISD selection. A single ISD opcode with merged predicate would also be straightforward.

The CMPPS/CMPD/CMPSS/CMPSD selection in X86 is already in custom code due to the 8 encoding issue. So two ISD opcodes or one probably doesn't make a lot of difference there either.

[uweigand]

How can we get to a decision on this? At some point we need to move ahead one way or the other.

I'd still argue that using separate ISD nodes for signaling vs. quiet compares is preferable, both from a complexity of implementation perspective and also conceptually: I feel that a signaling and a quiet compare implement the same predicates (and also return the same values in the absence of traps), it's just that operation is different in which exceptions are signaled ...

[pengfei]

Adding an extra argument to the DAGnode (sort of like FP_ROUND) also need special code all over the place to handle ...

Hi @uweigand, I don't quite understand why we need special code all over the place to handle. From what I can see, STRICT_FSETCCS always have the same action with STRICT_FSETCC in common code.

[uweigand]

In D69281#1770668, @pengfei wrote:

Adding an extra argument to the DAGnode (sort of like FP_ROUND) also need special code all over the place to handle ...

Hi @uweigand, I don't quite understand why we need special code all over the place to handle. From what I can see, STRICT_FSETCCS always have the same action with STRICT_FSETCC in common code.

I had the variant with the extra operand implemented initially, and what I was seeing was that made the node "special" because now we don't have the same set of operands between SETCC and STRICT_FSETCC any more. Usually, the STRICT_ opcodes have the same set of operands as the regular operands, with the exception of the chain. Code tends to assume that this is the case (e.g. when morphing a strict node into a regular node, or when writing a single legalization or other transformation that applies to both the strict and the regular node). Having not just the chain, but one additional extra operand required code changes to handle that operand in various places. Instead, with two DAG opcodes, you usually just have to add two more cases to a switch and that's it.

But those were really minor issues; the primary reason for me to switch was that I needed a way to tell common code which operations are legal, where the ISA has differences between signaling and quiet operations. (On Z we always have both for scalar types, but for vector types we got the signaling operations at a later ISA level than the quiet ones). This is trivial with two opcodes, but would require quite a bit of custom code if when using just a single one.

[craig.topper]

For X86, I think the one opcode or two is a wash. So I think this is fine.

[craig.topper]

LGTM

[pengfei]

In D69281#1771143, @uweigand wrote:

In D69281#1770668, @pengfei wrote:

Adding an extra argument to the DAGnode (sort of like FP_ROUND) also need special code all over the place to handle ...

Hi @uweigand, I don't quite understand why we need special code all over the place to handle. From what I can see, STRICT_FSETCCS always have the same action with STRICT_FSETCC in common code.

I had the variant with the extra operand implemented initially, and what I was seeing was that made the node "special" because now we don't have the same set of operands between SETCC and STRICT_FSETCC any more. Usually, the STRICT_ opcodes have the same set of operands as the regular operands, with the exception of the chain. Code tends to assume that this is the case (e.g. when morphing a strict node into a regular node, or when writing a single legalization or other transformation that applies to both the strict and the regular node). Having not just the chain, but one additional extra operand required code changes to handle that operand in various places. Instead, with two DAG opcodes, you usually just have to add two more cases to a switch and that's it.

But those were really minor issues; the primary reason for me to switch was that I needed a way to tell common code which operations are legal, where the ISA has differences between signaling and quiet operations. (On Z we always have both for scalar types, but for vector types we got the signaling operations at a later ISA level than the quiet ones). This is trivial with two opcodes, but would require quite a bit of custom code if when using just a single one.

Got it, thanks!

include/llvm/CodeGen/ISDOpcodes.h

enum NodeType {
     /// It is used to limit optimizations while the DAG is being optimized.
     STRICT_FP_ROUND,
 
     /// X = STRICT_FP_EXTEND(Y) - Extend a smaller FP type into a larger FP
     /// type.
     /// It is used to limit optimizations while the DAG is being optimized.
     STRICT_FP_EXTEND,
 
+    /// STRICT_FSETCC - Constrained version of SETCC, used for floating-point
+    /// operands only.
+    STRICT_FSETCC,
+
     /// FMA - Perform a * b + c with no intermediate rounding step.
     FMA,
 
     /// FMAD - Perform a * b + c, while getting the same result as the
     /// separately rounded operations.
     FMAD,
 
     /// FCOPYSIGN(X, Y) - Return the value of X with the sign of Y.  NOTE: This

include/llvm/CodeGen/SelectionDAGNodes.h

switch (NodeType) {
       case ISD::STRICT_LROUND:
       case ISD::STRICT_LLROUND:
       case ISD::STRICT_FROUND:
       case ISD::STRICT_FTRUNC:
       case ISD::STRICT_FP_TO_SINT:
       case ISD::STRICT_FP_TO_UINT:
       case ISD::STRICT_FP_ROUND:
       case ISD::STRICT_FP_EXTEND:
+      case ISD::STRICT_FSETCC:
         return true;
     }
   }
 
   /// Test if this node has a post-isel opcode, directly
   /// corresponding to a MachineInstr opcode.
   bool isMachineOpcode() const { return NodeType < 0; }
 

include/llvm/CodeGen/TargetLowering.h

switch (Op) {
       case ISD::STRICT_LROUND: EqOpc = ISD::LROUND; break;
       case ISD::STRICT_LLROUND: EqOpc = ISD::LLROUND; break;
       case ISD::STRICT_FROUND: EqOpc = ISD::FROUND; break;
       case ISD::STRICT_FTRUNC: EqOpc = ISD::FTRUNC; break;
       case ISD::STRICT_FP_TO_SINT: EqOpc = ISD::FP_TO_SINT; break;
       case ISD::STRICT_FP_TO_UINT: EqOpc = ISD::FP_TO_UINT; break;
       case ISD::STRICT_FP_ROUND: EqOpc = ISD::FP_ROUND; break;
       case ISD::STRICT_FP_EXTEND: EqOpc = ISD::FP_EXTEND; break;
+      case ISD::STRICT_FSETCC: EqOpc = ISD::SETCC; break;
     }
 
     return getOperationAction(EqOpc, VT);
   }
 
   /// Return true if the specified operation is legal on this target or can be
   /// made legal with custom lowering. This is used to help guide high-level
   /// lowering decisions.

include/llvm/IR/IntrinsicInst.h

static bool classof(const IntrinsicInst *I) {
       case Intrinsic::experimental_constrained_maxnum:
       case Intrinsic::experimental_constrained_minnum:
       case Intrinsic::experimental_constrained_ceil:
       case Intrinsic::experimental_constrained_floor:
       case Intrinsic::experimental_constrained_lround:
       case Intrinsic::experimental_constrained_llround:
       case Intrinsic::experimental_constrained_round:
       case Intrinsic::experimental_constrained_trunc:
+      case Intrinsic::experimental_constrained_fcmpoeq:
+      case Intrinsic::experimental_constrained_fcmpogt:
+      case Intrinsic::experimental_constrained_fcmpoge:
+      case Intrinsic::experimental_constrained_fcmpolt:
+      case Intrinsic::experimental_constrained_fcmpole:
+      case Intrinsic::experimental_constrained_fcmpone:
+      case Intrinsic::experimental_constrained_fcmpord:
+      case Intrinsic::experimental_constrained_fcmpueq:
+      case Intrinsic::experimental_constrained_fcmpugt:
+      case Intrinsic::experimental_constrained_fcmpuge:
+      case Intrinsic::experimental_constrained_fcmpult:
+      case Intrinsic::experimental_constrained_fcmpule:
+      case Intrinsic::experimental_constrained_fcmpune:
+      case Intrinsic::experimental_constrained_fcmpuno:
         return true;
       default: return false;
       }
     }
     static bool classof(const Value *V) {
       return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
     }
   };

include/llvm/IR/Intrinsics.td

let IntrProperties = [IntrInaccessibleMemOnly, IntrWillReturn] in {
   def int_experimental_constrained_round : Intrinsic<[ llvm_anyfloat_ty ],
                                                      [ LLVMMatchType<0>,
                                                       llvm_metadata_ty,
                                                       llvm_metadata_ty ]>;
   def int_experimental_constrained_trunc : Intrinsic<[ llvm_anyfloat_ty ],
                                                      [ LLVMMatchType<0>,
                                                        llvm_metadata_ty,
                                                        llvm_metadata_ty ]>;
+
+  // Comparison intrinsics.  These correspond to the non-strict "fcmp"
+  // operation, but instead of taking the condition code as argument,
+  // a separate intrinsic is provided for each condition code.
+  def int_experimental_constrained_fcmpoeq
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpogt
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpoge
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpolt
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpole
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpone
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpord
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpueq
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpugt
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpuge
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpult
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpule
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpune
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
+  def int_experimental_constrained_fcmpuno
+      : Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+                  [ llvm_anyfloat_ty, LLVMMatchType<0>, llvm_metadata_ty ]>;
 }
 // FIXME: Add intrinsic for fcmp.
 // FIXME: Consider maybe adding intrinsics for sitofp, uitofp.
 
 //===------------------------- Expect Intrinsics --------------------------===//
 //
 def int_expect : Intrinsic<[llvm_anyint_ty],
   [LLVMMatchType<0>, LLVMMatchType<0>], [IntrNoMem, IntrWillReturn]>;

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

if (!VT.isVector() && !TLI.convertSelectOfConstantsToMath(VT)) {
       if (SetCCVT.getScalarSizeInBits() != 1 &&
           (!LegalOperations || TLI.isOperationLegal(ISD::SETCC, N00VT))) {
         SDValue SetCC = DAG.getSetCC(DL, SetCCVT, N00, N01, CC);
         return DAG.getSelect(DL, VT, SetCC, ExtTrueVal, Zero);
       }
     }
   }
 
+  // Some of the transformations above are also valid for STRICT_FSETCC.
+  if (N0.getOpcode() == ISD::STRICT_FSETCC) {
+    SDValue Chain = N0.getOperand(0);
+    SDValue N00 = N0.getOperand(1);
+    SDValue N01 = N0.getOperand(2);
+    SDValue N02 = N0.getOperand(3);
+    EVT N00VT = N0.getOperand(1).getValueType();
+
+    // sext(strict_fsetcc) -> sext_in_reg(strict_fsetcc) for vectors.
+    if (VT.isVector() && !LegalOperations &&
+        TLI.getBooleanContents(N00VT) ==
+            TargetLowering::ZeroOrNegativeOneBooleanContent) {
+      EVT SVT = getSetCCResultType(N00VT);
+
+      if (SVT != N0.getValueType()) {
+        if (VT.getSizeInBits() == SVT.getSizeInBits()) {
+          SDVTList VTs = DAG.getVTList(VT, MVT::Other);
+          SDValue VSetCC = DAG.getNode(ISD::STRICT_FSETCC, DL, VTs,
+                                       Chain, N00, N01, N02);
+          DAG.ReplaceAllUsesOfValueWith(N0.getValue(1), VSetCC.getValue(1));
+          return VSetCC;
+        }
+
+        EVT MatchingVecType = N00VT.changeVectorElementTypeToInteger();
+        if (SVT == MatchingVecType) {
+          SDVTList VTs = DAG.getVTList(MatchingVecType, MVT::Other);
+          SDValue VSetCC = DAG.getNode(ISD::STRICT_FSETCC, DL, VTs,
+                                       Chain, N00, N01, N02);
+          DAG.ReplaceAllUsesOfValueWith(N0.getValue(1), VSetCC.getValue(1));
+          return DAG.getSExtOrTrunc(VSetCC, DL, VT);
+        }
+      }
+    }
+  }
+
   // fold (sext x) -> (zext x) if the sign bit is known zero.
   if ((!LegalOperations || TLI.isOperationLegal(ISD::ZERO_EXTEND, VT)) &&
       DAG.SignBitIsZero(N0))
     return DAG.getNode(ISD::ZERO_EXTEND, DL, VT, N0);
 
   if (SDValue NewVSel = matchVSelectOpSizesWithSetCC(N))
     return NewVSel;
 

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

case ISD::SIGN_EXTEND_INREG: {
     Action = TLI.getOperationAction(Node->getOpcode(), InnerType);
     break;
   }
   case ISD::ATOMIC_STORE:
     Action = TLI.getOperationAction(Node->getOpcode(),
                                     Node->getOperand(2).getValueType());
     break;
   case ISD::SELECT_CC:
+  case ISD::STRICT_FSETCC:
   case ISD::SETCC:
   case ISD::BR_CC: {
     unsigned CCOperand = Node->getOpcode() == ISD::SELECT_CC ? 4 :
+                         Node->getOpcode() == ISD::STRICT_FSETCC ? 3 :
                          Node->getOpcode() == ISD::SETCC ? 2 : 1;
-    unsigned CompareOperand = Node->getOpcode() == ISD::BR_CC ? 2 : 0;
+    unsigned CompareOperand = Node->getOpcode() == ISD::BR_CC ? 2 :
+                              Node->getOpcode() == ISD::STRICT_FSETCC ? 1 : 0;
     MVT OpVT = Node->getOperand(CompareOperand).getSimpleValueType();
     ISD::CondCode CCCode =
         cast<CondCodeSDNode>(Node->getOperand(CCOperand))->get();
     Action = TLI.getCondCodeAction(CCCode, OpVT);
     if (Action == TargetLowering::Legal) {
       if (Node->getOpcode() == ISD::SELECT_CC)
         Action = TLI.getOperationAction(Node->getOpcode(),
                                         Node->getValueType(0));

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

#endif
   case ISD::LOAD:        Res = PromoteIntRes_LOAD(cast<LoadSDNode>(N)); break;
   case ISD::MLOAD:       Res = PromoteIntRes_MLOAD(cast<MaskedLoadSDNode>(N));
     break;
   case ISD::MGATHER:     Res = PromoteIntRes_MGATHER(cast<MaskedGatherSDNode>(N));
     break;
   case ISD::SELECT:      Res = PromoteIntRes_SELECT(N); break;
   case ISD::VSELECT:     Res = PromoteIntRes_VSELECT(N); break;
   case ISD::SELECT_CC:   Res = PromoteIntRes_SELECT_CC(N); break;
+  case ISD::STRICT_FSETCC:
   case ISD::SETCC:       Res = PromoteIntRes_SETCC(N); break;
   case ISD::SMIN:
   case ISD::SMAX:        Res = PromoteIntRes_SExtIntBinOp(N); break;
   case ISD::UMIN:
   case ISD::UMAX:        Res = PromoteIntRes_ZExtIntBinOp(N); break;
 
   case ISD::SHL:         Res = PromoteIntRes_SHL(N); break;
   case ISD::SIGN_EXTEND_INREG:
SDValue DAGTypeLegalizer::PromoteIntRes_SELECT_CC(SDNode *N) {
   SDValue LHS = GetPromotedInteger(N->getOperand(2));
   SDValue RHS = GetPromotedInteger(N->getOperand(3));
   return DAG.getNode(ISD::SELECT_CC, SDLoc(N),
                      LHS.getValueType(), N->getOperand(0),
                      N->getOperand(1), LHS, RHS, N->getOperand(4));
 }
 
 SDValue DAGTypeLegalizer::PromoteIntRes_SETCC(SDNode *N) {
-  EVT InVT = N->getOperand(0).getValueType();
+  bool IsStrict = N->isStrictFPOpcode();
+  int InOpNo = IsStrict? 1 : 0;
+  EVT InVT = N->getOperand(InOpNo).getValueType();
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
 
   EVT SVT = getSetCCResultType(InVT);
 
   // If we got back a type that needs to be promoted, this likely means the
   // the input type also needs to be promoted. So get the promoted type for
   // the input and try the query again.
   if (getTypeAction(SVT) == TargetLowering::TypePromoteInteger) {
     if (getTypeAction(InVT) == TargetLowering::TypePromoteInteger) {
       InVT = TLI.getTypeToTransformTo(*DAG.getContext(), InVT);
       SVT = getSetCCResultType(InVT);
     } else {
       // Input type isn't promoted, just use the default promoted type.
       SVT = NVT;
     }
   }
 
   SDLoc dl(N);
-  assert(SVT.isVector() == N->getOperand(0).getValueType().isVector() &&
+  assert(SVT.isVector() == N->getOperand(InOpNo).getValueType().isVector() &&
          "Vector compare must return a vector result!");
 
   // Get the SETCC result using the canonical SETCC type.
-  SDValue SetCC = DAG.getNode(N->getOpcode(), dl, SVT, N->getOperand(0),
-                              N->getOperand(1), N->getOperand(2));
+  SDValue SetCC;
+  if (IsStrict) {
+    EVT VTs[] = {SVT, MVT::Other};
+    SDValue Opers[] = {N->getOperand(0), N->getOperand(1),
+                       N->getOperand(2), N->getOperand(3)};
+    SetCC = DAG.getNode(N->getOpcode(), dl, VTs, Opers);
+    // Legalize the chain result - switch anything that used the old chain to
+    // use the new one.
+    ReplaceValueWith(SDValue(N, 1), SetCC.getValue(1));
+  } else
+    SetCC = DAG.getNode(N->getOpcode(), dl, SVT, N->getOperand(0),
 
   // Convert to the expected type.
   return DAG.getSExtOrTrunc(SetCC, dl, NVT);
 }
 
 SDValue DAGTypeLegalizer::PromoteIntRes_SHL(SDNode *N) {
   SDValue LHS = GetPromotedInteger(N->getOperand(0));
   SDValue RHS = N->getOperand(1);

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

SDValue VectorLegalizer::LegalizeOp(SDValue Op) {
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
   case ISD::STRICT_FP_TO_SINT:
   case ISD::STRICT_FP_TO_UINT:
   case ISD::STRICT_FP_ROUND:
   case ISD::STRICT_FP_EXTEND:
+  case ISD::STRICT_FSETCC:
     Action = TLI.getOperationAction(Node->getOpcode(), Node->getValueType(0));
     // If we're asked to expand a strict vector floating-point operation,
     // by default we're going to simply unroll it.  That is usually the
     // best approach, except in the case where the resulting strict (scalar)
     // operations would themselves use the fallback mutation to non-strict.
     // In that specific case, just do the fallback on the vector op.
     if (Action == TargetLowering::Expand &&
         TLI.getStrictFPOperationAction(Node->getOpcode(),
SDValue VectorLegalizer::Expand(SDValue Op) {
   case ISD::STRICT_FMAXNUM:
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
   case ISD::STRICT_FP_TO_SINT:
   case ISD::STRICT_FP_TO_UINT:
+  case ISD::STRICT_FSETCC:
     return ExpandStrictFPOp(Op);
   case ISD::VECREDUCE_ADD:
   case ISD::VECREDUCE_MUL:
   case ISD::VECREDUCE_AND:
   case ISD::VECREDUCE_OR:
   case ISD::VECREDUCE_XOR:
   case ISD::VECREDUCE_SMAX:
   case ISD::VECREDUCE_SMIN:

 }
 
 SDValue VectorLegalizer::ExpandStrictFPOp(SDValue Op) {
   EVT VT = Op.getValueType();
   EVT EltVT = VT.getVectorElementType();
   unsigned NumElems = VT.getVectorNumElements();
   unsigned NumOpers = Op.getNumOperands();
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
-  EVT ValueVTs[] = {EltVT, MVT::Other};
+
+  EVT TmpEltVT = EltVT;
+  if (Op->getOpcode() == ISD::STRICT_FSETCC)
+    TmpEltVT = TLI.getSetCCResultType(DAG.getDataLayout(),
+                                      *DAG.getContext(), TmpEltVT);
+
+  EVT ValueVTs[] = {TmpEltVT, MVT::Other};
   SDValue Chain = Op.getOperand(0);
   SDLoc dl(Op);
 
   SmallVector<SDValue, 32> OpValues;
   SmallVector<SDValue, 32> OpChains;
   for (unsigned i = 0; i < NumElems; ++i) {
     SmallVector<SDValue, 4> Opers;
     SDValue Idx = DAG.getConstant(i, dl,
for (unsigned j = 1; j < NumOpers; ++j) {
       if (OperVT.isVector())
         Oper = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl,
                            OperVT.getVectorElementType(), Oper, Idx);
 
       Opers.push_back(Oper);
     }
 
     SDValue ScalarOp = DAG.getNode(Op->getOpcode(), dl, ValueVTs, Opers);
+    SDValue ScalarResult = ScalarOp.getValue(0);
+    SDValue ScalarChain = ScalarOp.getValue(1);
+
+    if (Op->getOpcode() == ISD::STRICT_FSETCC)
+      ScalarResult = DAG.getSelect(dl, EltVT, ScalarResult,
+                           DAG.getConstant(APInt::getAllOnesValue
+                                           (EltVT.getSizeInBits()), dl, EltVT),
+                           DAG.getConstant(0, dl, EltVT));
 
-    OpValues.push_back(ScalarOp.getValue(0));
-    OpChains.push_back(ScalarOp.getValue(1));
+    OpValues.push_back(ScalarResult);
+    OpChains.push_back(ScalarChain);
   }
 
   SDValue Result = DAG.getBuildVector(VT, dl, OpValues);
   SDValue NewChain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, OpChains);
 
   AddLegalizedOperand(Op.getValue(0), Result);
   AddLegalizedOperand(Op.getValue(1), NewChain);
 

lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

#endif
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
   case ISD::STRICT_FP_TO_SINT:
   case ISD::STRICT_FP_TO_UINT:
   case ISD::STRICT_FP_EXTEND:
+  case ISD::STRICT_FSETCC:
     R = ScalarizeVecRes_StrictFPOp(N);
     break;
   case ISD::UADDO:
   case ISD::SADDO:
   case ISD::USUBO:
   case ISD::SSUBO:
   case ISD::UMULO:
   case ISD::SMULO:
#endif
   case ISD::STRICT_FRINT:
   case ISD::STRICT_FNEARBYINT:
   case ISD::STRICT_FMAXNUM:
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
+  case ISD::STRICT_FSETCC:
     SplitVecRes_StrictFPOp(N, Lo, Hi);
     break;
   case ISD::UADDO:
   case ISD::SADDO:
   case ISD::USUBO:
   case ISD::SSUBO:
   case ISD::UMULO:
   case ISD::SMULO:
#endif
   case ISD::STRICT_FRINT:
   case ISD::STRICT_FNEARBYINT:
   case ISD::STRICT_FMAXNUM:
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
+  case ISD::STRICT_FSETCC:
     Res = WidenVecRes_StrictFP(N);
     break;
 
   case ISD::UADDO:
   case ISD::SADDO:
   case ISD::USUBO:
   case ISD::SSUBO:
   case ISD::UMULO:

 }
 
 SDValue DAGTypeLegalizer::WidenVecRes_SCALAR_TO_VECTOR(SDNode *N) {
   EVT WidenVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
   return DAG.getNode(ISD::SCALAR_TO_VECTOR, SDLoc(N),
                      WidenVT, N->getOperand(0));
 }
 
+// Return true is this is a SETCC node or a strict version of it.
+static inline bool isSETCCOp(unsigned Opcode) {
+  switch (Opcode) {
+  case ISD::SETCC:
+  case ISD::STRICT_FSETCC:
+    return true;
+  }
+  return false;
+}
+
 // Return true if this is a node that could have two SETCCs as operands.
 static inline bool isLogicalMaskOp(unsigned Opcode) {
   switch (Opcode) {
   case ISD::AND:
   case ISD::OR:
   case ISD::XOR:
     return true;
   }
   return false;
 }
 
+// If N is a SETCC or a strict variant of it, return the type
+// of the compare operands.
+static inline EVT getSETCCOperandType(SDValue N) {
+  if (N->isStrictFPOpcode())
+    return N->getOperand(1).getValueType();
+  else
+    return N->getOperand(0).getValueType();
+}
+
 // This is used just for the assert in convertMask(). Check that this either
 // a SETCC or a previously handled SETCC by convertMask().
 #ifndef NDEBUG
 static inline bool isSETCCorConvertedSETCC(SDValue N) {
   if (N.getOpcode() == ISD::EXTRACT_SUBVECTOR)
     N = N.getOperand(0);
   else if (N.getOpcode() == ISD::CONCAT_VECTORS) {
     for (unsigned i = 1; i < N->getNumOperands(); ++i)
       if (!N->getOperand(i)->isUndef())
         return false;
     N = N.getOperand(0);
   }
 
   if (N.getOpcode() == ISD::TRUNCATE)
     N = N.getOperand(0);
   else if (N.getOpcode() == ISD::SIGN_EXTEND)
     N = N.getOperand(0);
 
   if (isLogicalMaskOp(N.getOpcode()))
     return isSETCCorConvertedSETCC(N.getOperand(0)) &&
            isSETCCorConvertedSETCC(N.getOperand(1));
 
-  return (N.getOpcode() == ISD::SETCC ||
+  return (isSETCCOp(N.getOpcode()) ||
           ISD::isBuildVectorOfConstantSDNodes(N.getNode()));
 }
 #endif
 
 // Return a mask of vector type MaskVT to replace InMask. Also adjust MaskVT
 // to ToMaskVT if needed with vector extension or truncation.
 SDValue DAGTypeLegalizer::convertMask(SDValue InMask, EVT MaskVT,
                                       EVT ToMaskVT) {
   // Currently a SETCC or a AND/OR/XOR with two SETCCs are handled.
   // FIXME: This code seems to be too restrictive, we might consider
   // generalizing it or dropping it.
   assert(isSETCCorConvertedSETCC(InMask) && "Unexpected mask argument.");
 
   // Make a new Mask node, with a legal result VT.
+  SDValue Mask;
   SmallVector<SDValue, 4> Ops;
   for (unsigned i = 0, e = InMask->getNumOperands(); i < e; ++i)
     Ops.push_back(InMask->getOperand(i));
-  SDValue Mask = DAG.getNode(InMask->getOpcode(), SDLoc(InMask), MaskVT, Ops);
+  if (InMask->isStrictFPOpcode()) {
+    Mask = DAG.getNode(InMask->getOpcode(), SDLoc(InMask),
+                       { MaskVT, MVT::Other }, Ops);
+    ReplaceValueWith(InMask.getValue(1), Mask.getValue(1));
+  }
+  else
+    Mask = DAG.getNode(InMask->getOpcode(), SDLoc(InMask), MaskVT, Ops);
 
   // If MaskVT has smaller or bigger elements than ToMaskVT, a vector sign
   // extend or truncate is needed.
   LLVMContext &Ctx = *DAG.getContext();
   unsigned MaskScalarBits = MaskVT.getScalarSizeInBits();
   unsigned ToMaskScalBits = ToMaskVT.getScalarSizeInBits();
   if (MaskScalarBits < ToMaskScalBits) {
     EVT ExtVT = EVT::getVectorVT(Ctx, ToMaskVT.getVectorElementType(),

 // scalarization of the SETCC, with many unnecessary instructions.
 SDValue DAGTypeLegalizer::WidenVSELECTAndMask(SDNode *N) {
   LLVMContext &Ctx = *DAG.getContext();
   SDValue Cond = N->getOperand(0);
 
   if (N->getOpcode() != ISD::VSELECT)
     return SDValue();
 
-  if (Cond->getOpcode() != ISD::SETCC && !isLogicalMaskOp(Cond->getOpcode()))
+  if (!isSETCCOp(Cond->getOpcode()) && !isLogicalMaskOp(Cond->getOpcode()))
     return SDValue();
 
   // If this is a splitted VSELECT that was previously already handled, do
   // nothing.
   EVT CondVT = Cond->getValueType(0);
   if (CondVT.getScalarSizeInBits() != 1)
     return SDValue();
 
   EVT VSelVT = N->getValueType(0);
   // Only handle vector types which are a power of 2.
   if (!isPowerOf2_64(VSelVT.getSizeInBits()))
     return SDValue();
 
   // Don't touch if this will be scalarized.
   EVT FinalVT = VSelVT;
   while (getTypeAction(FinalVT) == TargetLowering::TypeSplitVector)
     FinalVT = FinalVT.getHalfNumVectorElementsVT(Ctx);
 
   if (FinalVT.getVectorNumElements() == 1)
     return SDValue();
 
   // If there is support for an i1 vector mask, don't touch.
-  if (Cond.getOpcode() == ISD::SETCC) {
-    EVT SetCCOpVT = Cond->getOperand(0).getValueType();
+  if (isSETCCOp(Cond.getOpcode())) {
+    EVT SetCCOpVT = getSETCCOperandType(Cond);
     while (TLI.getTypeAction(Ctx, SetCCOpVT) != TargetLowering::TypeLegal)
       SetCCOpVT = TLI.getTypeToTransformTo(Ctx, SetCCOpVT);
     EVT SetCCResVT = getSetCCResultType(SetCCOpVT);
     if (SetCCResVT.getScalarSizeInBits() == 1)
       return SDValue();
   } else if (CondVT.getScalarType() == MVT::i1) {
     // If there is support for an i1 vector mask (or only scalar i1 conditions),
     // don't touch.
SDValue DAGTypeLegalizer::WidenVSELECTAndMask(SDNode *N) {
   }
 
   // The mask of the VSELECT should have integer elements.
   EVT ToMaskVT = VSelVT;
   if (!ToMaskVT.getScalarType().isInteger())
     ToMaskVT = ToMaskVT.changeVectorElementTypeToInteger();
 
   SDValue Mask;
-  if (Cond->getOpcode() == ISD::SETCC) {
-    EVT MaskVT = getSetCCResultType(Cond.getOperand(0).getValueType());
+  if (isSETCCOp(Cond->getOpcode())) {
+    EVT MaskVT = getSetCCResultType(getSETCCOperandType(Cond));
     Mask = convertMask(Cond, MaskVT, ToMaskVT);
   } else if (isLogicalMaskOp(Cond->getOpcode()) &&
-             Cond->getOperand(0).getOpcode() == ISD::SETCC &&
-             Cond->getOperand(1).getOpcode() == ISD::SETCC) {
+             isSETCCOp(Cond->getOperand(0).getOpcode()) &&
+             isSETCCOp(Cond->getOperand(1).getOpcode())) {
     // Cond is (AND/OR/XOR (SETCC, SETCC))
     SDValue SETCC0 = Cond->getOperand(0);
     SDValue SETCC1 = Cond->getOperand(1);
-    EVT VT0 = getSetCCResultType(SETCC0.getOperand(0).getValueType());
-    EVT VT1 = getSetCCResultType(SETCC1.getOperand(0).getValueType());
+    EVT VT0 = getSetCCResultType(getSETCCOperandType(SETCC0));
+    EVT VT1 = getSetCCResultType(getSETCCOperandType(SETCC1));
     unsigned ScalarBits0 = VT0.getScalarSizeInBits();
     unsigned ScalarBits1 = VT1.getScalarSizeInBits();
     unsigned ScalarBits_ToMask = ToMaskVT.getScalarSizeInBits();
     EVT MaskVT;
     // If the two SETCCs have different VTs, either extend/truncate one of
     // them to the other "towards" ToMaskVT, or truncate one and extend the
     // other to ToMaskVT.
     if (ScalarBits0 != ScalarBits1) {

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

SDNode* SelectionDAG::mutateStrictFPToFP(SDNode *Node) {
   case ISD::STRICT_LROUND:     NewOpc = ISD::LROUND;     break;
   case ISD::STRICT_LLROUND:    NewOpc = ISD::LLROUND;    break;
   case ISD::STRICT_FROUND:     NewOpc = ISD::FROUND;     break;
   case ISD::STRICT_FTRUNC:     NewOpc = ISD::FTRUNC;     break;
   case ISD::STRICT_FP_ROUND:   NewOpc = ISD::FP_ROUND;   break;
   case ISD::STRICT_FP_EXTEND:  NewOpc = ISD::FP_EXTEND;  break;
   case ISD::STRICT_FP_TO_SINT: NewOpc = ISD::FP_TO_SINT; break;
   case ISD::STRICT_FP_TO_UINT: NewOpc = ISD::FP_TO_UINT; break;
+  case ISD::STRICT_FSETCC:     NewOpc = ISD::SETCC;      break;
   }
 
   assert(Node->getNumValues() == 2 && "Unexpected number of results!");
 
   // We're taking this node out of the chain, so we need to re-link things.
   SDValue InputChain = Node->getOperand(0);
   SDValue OutputChain = SDValue(Node, 1);
   ReplaceAllUsesOfValueWith(OutputChain, InputChain);

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
   case Intrinsic::experimental_constrained_maxnum:
   case Intrinsic::experimental_constrained_minnum:
   case Intrinsic::experimental_constrained_ceil:
   case Intrinsic::experimental_constrained_floor:
   case Intrinsic::experimental_constrained_lround:
   case Intrinsic::experimental_constrained_llround:
   case Intrinsic::experimental_constrained_round:
   case Intrinsic::experimental_constrained_trunc:
+  case Intrinsic::experimental_constrained_fcmpoeq:
+  case Intrinsic::experimental_constrained_fcmpogt:
+  case Intrinsic::experimental_constrained_fcmpoge:
+  case Intrinsic::experimental_constrained_fcmpolt:
+  case Intrinsic::experimental_constrained_fcmpole:
+  case Intrinsic::experimental_constrained_fcmpone:
+  case Intrinsic::experimental_constrained_fcmpord:
+  case Intrinsic::experimental_constrained_fcmpueq:
+  case Intrinsic::experimental_constrained_fcmpugt:
+  case Intrinsic::experimental_constrained_fcmpuge:
+  case Intrinsic::experimental_constrained_fcmpult:
+  case Intrinsic::experimental_constrained_fcmpule:
+  case Intrinsic::experimental_constrained_fcmpune:
+  case Intrinsic::experimental_constrained_fcmpuno:
     visitConstrainedFPIntrinsic(cast<ConstrainedFPIntrinsic>(I));
     return;
   case Intrinsic::fmuladd: {
     EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType());
     if (TM.Options.AllowFPOpFusion != FPOpFusion::Strict &&
         TLI.isFMAFasterThanFMulAndFAdd(VT)) {
       setValue(&I, DAG.getNode(ISD::FMA, sdl,
                                getValue(I.getArgOperand(0)).getValueType(),
case Intrinsic::ptrmask: {
     return;
   }
   }
 }
 
 void SelectionDAGBuilder::visitConstrainedFPIntrinsic(
     const ConstrainedFPIntrinsic &FPI) {
   SDLoc sdl = getCurSDLoc();
+
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+  SmallVector<EVT, 4> ValueVTs;
+  ComputeValueVTs(TLI, DAG.getDataLayout(), FPI.getType(), ValueVTs);
+  ValueVTs.push_back(MVT::Other); // Out chain
+
+  SDValue Chain = getRoot();
+  SmallVector<SDValue, 4> Opers;
+  Opers.push_back(Chain);
+  if (FPI.isUnaryOp()) {
+    Opers.push_back(getValue(FPI.getArgOperand(0)));
+  } else if (FPI.isTernaryOp()) {
+    Opers.push_back(getValue(FPI.getArgOperand(0)));
+    Opers.push_back(getValue(FPI.getArgOperand(1)));
+    Opers.push_back(getValue(FPI.getArgOperand(2)));
+  } else {
+    Opers.push_back(getValue(FPI.getArgOperand(0)));
+    Opers.push_back(getValue(FPI.getArgOperand(1)));
+  }
+
   unsigned Opcode;
   switch (FPI.getIntrinsicID()) {
   default: llvm_unreachable("Impossible intrinsic");  // Can't reach here.
   case Intrinsic::experimental_constrained_fadd:
     Opcode = ISD::STRICT_FADD;
     break;
   case Intrinsic::experimental_constrained_fsub:
     Opcode = ISD::STRICT_FSUB;
void SelectionDAGBuilder::visitConstrainedFPIntrinsic(
   case Intrinsic::experimental_constrained_fptosi:
     Opcode = ISD::STRICT_FP_TO_SINT;
     break;
   case Intrinsic::experimental_constrained_fptoui:
     Opcode = ISD::STRICT_FP_TO_UINT;
     break;
   case Intrinsic::experimental_constrained_fptrunc:
     Opcode = ISD::STRICT_FP_ROUND;
+    Opers.push_back(DAG.getTargetConstant(0, sdl,
+                      TLI.getPointerTy(DAG.getDataLayout())));
     break;
   case Intrinsic::experimental_constrained_fpext:
     Opcode = ISD::STRICT_FP_EXTEND;
     break;
   case Intrinsic::experimental_constrained_sqrt:
     Opcode = ISD::STRICT_FSQRT;
     break;
   case Intrinsic::experimental_constrained_pow:
case Intrinsic::experimental_constrained_llround:
     Opcode = ISD::STRICT_LLROUND;
     break;
   case Intrinsic::experimental_constrained_round:
     Opcode = ISD::STRICT_FROUND;
     break;
   case Intrinsic::experimental_constrained_trunc:
     Opcode = ISD::STRICT_FTRUNC;
     break;
+  case Intrinsic::experimental_constrained_fcmpoeq:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETOEQ));
+    break;
+  case Intrinsic::experimental_constrained_fcmpogt:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETOGT));
+    break;
+  case Intrinsic::experimental_constrained_fcmpoge:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETOGE));
+    break;
+  case Intrinsic::experimental_constrained_fcmpolt:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETOLT));
+    break;
+  case Intrinsic::experimental_constrained_fcmpole:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETOLE));
+    break;
+  case Intrinsic::experimental_constrained_fcmpone:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETONE));
+    break;
+  case Intrinsic::experimental_constrained_fcmpord:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETO));
+    break;
+  case Intrinsic::experimental_constrained_fcmpueq:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETUEQ));
+    break;
+  case Intrinsic::experimental_constrained_fcmpugt:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETUGT));
+    break;
+  case Intrinsic::experimental_constrained_fcmpuge:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETUGE));
+    break;
+  case Intrinsic::experimental_constrained_fcmpult:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETULT));
+    break;
+  case Intrinsic::experimental_constrained_fcmpule:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETULE));
+    break;
+  case Intrinsic::experimental_constrained_fcmpune:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETUNE));
+    break;
+  case Intrinsic::experimental_constrained_fcmpuno:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETUO));
+    break;
   }
-  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
-  SDValue Chain = getRoot();
-  SmallVector<EVT, 4> ValueVTs;
-  ComputeValueVTs(TLI, DAG.getDataLayout(), FPI.getType(), ValueVTs);
-  ValueVTs.push_back(MVT::Other); // Out chain
 
   SDVTList VTs = DAG.getVTList(ValueVTs);
-  SDValue Result;
-  if (Opcode == ISD::STRICT_FP_ROUND)
-    Result = DAG.getNode(Opcode, sdl, VTs,
-                          { Chain, getValue(FPI.getArgOperand(0)),
-                               DAG.getTargetConstant(0, sdl,
-                               TLI.getPointerTy(DAG.getDataLayout())) });
-  else if (FPI.isUnaryOp())
-    Result = DAG.getNode(Opcode, sdl, VTs,
-                         { Chain, getValue(FPI.getArgOperand(0)) });
-  else if (FPI.isTernaryOp())
-    Result = DAG.getNode(Opcode, sdl, VTs,
-                         { Chain, getValue(FPI.getArgOperand(0)),
-                                  getValue(FPI.getArgOperand(1)),
-                                  getValue(FPI.getArgOperand(2)) });
-  else
-    Result = DAG.getNode(Opcode, sdl, VTs,
-                         { Chain, getValue(FPI.getArgOperand(0)),
-                           getValue(FPI.getArgOperand(1))  });
+  SDValue Result = DAG.getNode(Opcode, sdl, VTs, Opers);
 
   if (FPI.getExceptionBehavior() !=
       ConstrainedFPIntrinsic::ExceptionBehavior::ebIgnore) {
     SDNodeFlags Flags;
     Flags.setFPExcept(true);
     Result->setFlags(Flags);
   }
 

lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp

#endif
   case ISD::SMAX:                       return "smax";
   case ISD::UMIN:                       return "umin";
   case ISD::UMAX:                       return "umax";
 
   case ISD::FPOWI:                      return "fpowi";
   case ISD::STRICT_FPOWI:               return "strict_fpowi";
   case ISD::SETCC:                      return "setcc";
   case ISD::SETCCCARRY:                 return "setcccarry";
+  case ISD::STRICT_FSETCC:              return "strict_fsetcc";
   case ISD::SELECT:                     return "select";
   case ISD::VSELECT:                    return "vselect";
   case ISD::SELECT_CC:                  return "select_cc";
   case ISD::INSERT_VECTOR_ELT:          return "insert_vector_elt";
   case ISD::EXTRACT_VECTOR_ELT:         return "extract_vector_elt";
   case ISD::CONCAT_VECTORS:             return "concat_vectors";
   case ISD::INSERT_SUBVECTOR:           return "insert_subvector";
   case ISD::EXTRACT_SUBVECTOR:          return "extract_subvector";

lib/CodeGen/TargetLoweringBase.cpp

for (MVT VT : MVT::all_valuetypes()) {
     setOperationAction(ISD::STRICT_FROUND, VT, Expand);
     setOperationAction(ISD::STRICT_FTRUNC, VT, Expand);
     setOperationAction(ISD::STRICT_FMAXNUM, VT, Expand);
     setOperationAction(ISD::STRICT_FMINNUM, VT, Expand);
     setOperationAction(ISD::STRICT_FP_ROUND, VT, Expand);
     setOperationAction(ISD::STRICT_FP_EXTEND, VT, Expand);
     setOperationAction(ISD::STRICT_FP_TO_SINT, VT, Expand);
     setOperationAction(ISD::STRICT_FP_TO_UINT, VT, Expand);
+    setOperationAction(ISD::STRICT_FSETCC, VT, Expand);
 
     // For most targets @llvm.get.dynamic.area.offset just returns 0.
     setOperationAction(ISD::GET_DYNAMIC_AREA_OFFSET, VT, Expand);
 
     // Vector reduction default to expand.
     setOperationAction(ISD::VECREDUCE_FADD, VT, Expand);
     setOperationAction(ISD::VECREDUCE_FMUL, VT, Expand);
     setOperationAction(ISD::VECREDUCE_ADD, VT, Expand);

lib/IR/Verifier.cpp

void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
   case Intrinsic::experimental_constrained_maxnum:
   case Intrinsic::experimental_constrained_minnum:
   case Intrinsic::experimental_constrained_ceil:
   case Intrinsic::experimental_constrained_floor:
   case Intrinsic::experimental_constrained_lround:
   case Intrinsic::experimental_constrained_llround:
   case Intrinsic::experimental_constrained_round:
   case Intrinsic::experimental_constrained_trunc:
+  case Intrinsic::experimental_constrained_fcmpoeq:
+  case Intrinsic::experimental_constrained_fcmpogt:
+  case Intrinsic::experimental_constrained_fcmpoge:
+  case Intrinsic::experimental_constrained_fcmpolt:
+  case Intrinsic::experimental_constrained_fcmpole:
+  case Intrinsic::experimental_constrained_fcmpone:
+  case Intrinsic::experimental_constrained_fcmpord:
+  case Intrinsic::experimental_constrained_fcmpueq:
+  case Intrinsic::experimental_constrained_fcmpugt:
+  case Intrinsic::experimental_constrained_fcmpuge:
+  case Intrinsic::experimental_constrained_fcmpult:
+  case Intrinsic::experimental_constrained_fcmpule:
+  case Intrinsic::experimental_constrained_fcmpune:
+  case Intrinsic::experimental_constrained_fcmpuno:
     visitConstrainedFPIntrinsic(cast<ConstrainedFPIntrinsic>(Call));
     break;
   case Intrinsic::dbg_declare: // llvm.dbg.declare
     Assert(isa<MetadataAsValue>(Call.getArgOperand(0)),
            "invalid llvm.dbg.declare intrinsic call 1", Call);
     visitDbgIntrinsic("declare", cast<DbgVariableIntrinsic>(Call));
     break;
   case Intrinsic::dbg_addr: // llvm.dbg.addr
void Verifier::visitConstrainedFPIntrinsic(ConstrainedFPIntrinsic &FPI) {
   case Intrinsic::experimental_constrained_maxnum:
   case Intrinsic::experimental_constrained_minnum:
     Assert((NumOperands == 4), "invalid arguments for constrained FP intrinsic",
            &FPI);
     HasExceptionMD = true;
     HasRoundingMD = true;
     break;
 
+  case Intrinsic::experimental_constrained_fcmpoeq:
+  case Intrinsic::experimental_constrained_fcmpogt:
+  case Intrinsic::experimental_constrained_fcmpoge:
+  case Intrinsic::experimental_constrained_fcmpolt:
+  case Intrinsic::experimental_constrained_fcmpole:
+  case Intrinsic::experimental_constrained_fcmpone:
+  case Intrinsic::experimental_constrained_fcmpord:
+  case Intrinsic::experimental_constrained_fcmpueq:
+  case Intrinsic::experimental_constrained_fcmpugt:
+  case Intrinsic::experimental_constrained_fcmpuge:
+  case Intrinsic::experimental_constrained_fcmpult:
+  case Intrinsic::experimental_constrained_fcmpule:
+  case Intrinsic::experimental_constrained_fcmpune:
+  case Intrinsic::experimental_constrained_fcmpuno:
+    Assert((NumOperands == 3), "invalid arguments for constrained FP intrinsic",
+           &FPI);
+    HasExceptionMD = true;
+    break;
+
   case Intrinsic::experimental_constrained_fptosi:
   case Intrinsic::experimental_constrained_fptoui: { 
     Assert((NumOperands == 2),
            "invalid arguments for constrained FP intrinsic", &FPI);
     HasExceptionMD = true;
 
     Value *Operand = FPI.getArgOperand(0);
     uint64_t NumSrcElem = 0;

lib/Target/SystemZ/SystemZISelLowering.h

enum NodeType : unsigned {
   // Integer absolute.
   IABS,
 
   // Integer comparisons.  There are three operands: the two values
   // to compare, and an integer of type SystemZICMP.
   ICMP,
 
   // Floating-point comparisons.  The two operands are the values to compare.
-  FCMP,
+  FCMP, STRICT_FCMP,
 
   // Test under mask.  The first operand is ANDed with the second operand
   // and the condition codes are set on the result.  The third operand is
   // a boolean that is true if the condition codes need to distinguish
   // between CCMASK_TM_MIXED_MSB_0 and CCMASK_TM_MIXED_MSB_1 (which the
   // register forms do but the memory forms don't).
   TM,
 
enum NodeType : unsigned {
   VICMPH,
   VICMPHL,
 
   // Likewise, but also set the condition codes on the result.
   VICMPES,
   VICMPHS,
   VICMPHLS,
 
   // Compare floating-point vector operands 0 and 1 to preoduce the usual 0/-1
   // vector result.  VFCMPE is for "ordered and equal", VFCMPH for "ordered and
   // greater than" and VFCMPHE for "ordered and greater than or equal to".
-  VFCMPE,
-  VFCMPH,
-  VFCMPHE,
+  VFCMPE, STRICT_VFCMPE,
+  VFCMPH, STRICT_VFCMPH,
+  VFCMPHE, STRICT_VFCMPHE,
 
   // Likewise, but also set the condition codes on the result.
   VFCMPES,
   VFCMPHS,
   VFCMPHES,
 
   // Test floating-point data class for vectors.
   VFTCI,
 
   // Extend the even f32 elements of vector operand 0 to produce a vector
   // of f64 elements.
-  VEXTEND,
+  VEXTEND, STRICT_VEXTEND,
 
   // Round the f64 elements of vector operand 0 to f32s and store them in the
   // even elements of the result.
   VROUND,
 
   // AND the two vector operands together and set CC based on the result.
   VTM,
 
public:
   }
 
 private:
   const SystemZSubtarget &Subtarget;
 
   // Implement LowerOperation for individual opcodes.
   SDValue getVectorCmp(SelectionDAG &DAG, unsigned Opcode,
                        const SDLoc &DL, EVT VT,
-                       SDValue CmpOp0, SDValue CmpOp1) const;
+                       SDValue CmpOp0, SDValue CmpOp1, SDValue Chain) const;
   SDValue lowerVectorSETCC(SelectionDAG &DAG, const SDLoc &DL,
                            EVT VT, ISD::CondCode CC,
-                           SDValue CmpOp0, SDValue CmpOp1) const;
+                           SDValue CmpOp0, SDValue CmpOp1,
+                           SDValue Chain = SDValue()) const;
   SDValue lowerSETCC(SDValue Op, SelectionDAG &DAG) const;
+  SDValue lowerSTRICT_FSETCC(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerBR_CC(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerSELECT_CC(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerGlobalAddress(GlobalAddressSDNode *Node,
                              SelectionDAG &DAG) const;
   SDValue lowerTLSGetOffset(GlobalAddressSDNode *Node,
                             SelectionDAG &DAG, unsigned Opcode,
                             SDValue GOTOffset) const;
   SDValue lowerThreadPointer(const SDLoc &DL, SelectionDAG &DAG) const;

lib/Target/SystemZ/SystemZISelLowering.cpp

 
 using namespace llvm;
 
 #define DEBUG_TYPE "systemz-lower"
 
 namespace {
 // Represents information about a comparison.
 struct Comparison {
-  Comparison(SDValue Op0In, SDValue Op1In)
-    : Op0(Op0In), Op1(Op1In), Opcode(0), ICmpType(0), CCValid(0), CCMask(0) {}
+  Comparison(SDValue Op0In, SDValue Op1In, SDValue ChainIn)
+    : Op0(Op0In), Op1(Op1In), Chain(ChainIn),
+      Opcode(0), ICmpType(0), CCValid(0), CCMask(0) {}
 
   // The operands to the comparison.
   SDValue Op0, Op1;
 
+  // Chain if this is a strict floating-point comparison.
+  SDValue Chain;
+
   // The opcode that should be used to compare Op0 and Op1.
   unsigned Opcode;
 
   // A SystemZICMP value.  Only used for integer comparisons.
   unsigned ICmpType;
 
   // The mask of CC values that Opcode can produce.
   unsigned CCValid;
SystemZTargetLowering::SystemZTargetLowering(const TargetMachine &TM,
   // Handle operations that are handled in a similar way for all types.
   for (unsigned I = MVT::FIRST_INTEGER_VALUETYPE;
        I <= MVT::LAST_FP_VALUETYPE;
        ++I) {
     MVT VT = MVT::SimpleValueType(I);
     if (isTypeLegal(VT)) {
       // Lower SET_CC into an IPM-based sequence.
       setOperationAction(ISD::SETCC, VT, Custom);
+      setOperationAction(ISD::STRICT_FSETCC, VT, Custom);
 
       // Expand SELECT(C, A, B) into SELECT_CC(X, 0, A, B, NE).
       setOperationAction(ISD::SELECT, VT, Expand);
 
       // Lower SELECT_CC and BR_CC into separate comparisons and branches.
       setOperationAction(ISD::SELECT_CC, VT, Custom);
       setOperationAction(ISD::BR_CC,     VT, Custom);
     }
if (isTypeLegal(VT)) {
       // At present ROTL isn't matched by DAGCombiner.  ROTR should be
       // converted into ROTL.
       setOperationAction(ISD::ROTL, VT, Expand);
       setOperationAction(ISD::ROTR, VT, Expand);
 
       // Map SETCCs onto one of VCE, VCH or VCHL, swapping the operands
       // and inverting the result as necessary.
       setOperationAction(ISD::SETCC, VT, Custom);
+      setOperationAction(ISD::STRICT_FSETCC, VT, Custom);
     }
   }
 
   if (Subtarget.hasVector()) {
     // There should be no need to check for float types other than v2f64
     // since <2 x f32> isn't a legal type.
     setOperationAction(ISD::FP_TO_SINT, MVT::v2i64, Legal);
     setOperationAction(ISD::FP_TO_SINT, MVT::v2f64, Legal);
static void adjustForSubtraction(SelectionDAG &DAG, const SDLoc &DL,
   }
 }
 
 // Check whether C compares a floating-point value with zero and if that
 // floating-point value is also negated.  In this case we can use the
 // negation to set CC, so avoiding separate LOAD AND TEST and
 // LOAD (NEGATIVE/COMPLEMENT) instructions.
 static void adjustForFNeg(Comparison &C) {
+  if (C.Chain)
+    return;
   auto *C1 = dyn_cast<ConstantFPSDNode>(C.Op1);
   if (C1 && C1->isZero()) {
     for (auto I = C.Op0->use_begin(), E = C.Op0->use_end(); I != E; ++I) {
       SDNode *N = *I;
       if (N->getOpcode() == ISD::FNEG) {
         C.Op0 = SDValue(N, 0);
         C.CCMask = reverseCCMask(C.CCMask);
         return;

 
 // Return a Comparison that tests the condition-code result of intrinsic
 // node Call against constant integer CC using comparison code Cond.
 // Opcode is the opcode of the SystemZISD operation for the intrinsic
 // and CCValid is the set of possible condition-code results.
 static Comparison getIntrinsicCmp(SelectionDAG &DAG, unsigned Opcode,
                                   SDValue Call, unsigned CCValid, uint64_t CC,
                                   ISD::CondCode Cond) {
-  Comparison C(Call, SDValue());
+  Comparison C(Call, SDValue(), SDValue());
   C.Opcode = Opcode;
   C.CCValid = CCValid;
   if (Cond == ISD::SETEQ)
     // bit 3 for CC==0, bit 0 for CC==3, always false for CC>3.
     C.CCMask = CC < 4 ? 1 << (3 - CC) : 0;
   else if (Cond == ISD::SETNE)
     // ...and the inverse of that.
     C.CCMask = CC < 4 ? ~(1 << (3 - CC)) : -1;
static Comparison getIntrinsicCmp(SelectionDAG &DAG, unsigned Opcode,
   else
     llvm_unreachable("Unexpected integer comparison type");
   C.CCMask &= CCValid;
   return C;
 }
 
 // Decide how to implement a comparison of type Cond between CmpOp0 with CmpOp1.
 static Comparison getCmp(SelectionDAG &DAG, SDValue CmpOp0, SDValue CmpOp1,
-                         ISD::CondCode Cond, const SDLoc &DL) {
+                         ISD::CondCode Cond, const SDLoc &DL,
+                         SDValue Chain = SDValue()) {
   if (CmpOp1.getOpcode() == ISD::Constant) {
+    assert(!Chain);
     uint64_t Constant = cast<ConstantSDNode>(CmpOp1)->getZExtValue();
     unsigned Opcode, CCValid;
     if (CmpOp0.getOpcode() == ISD::INTRINSIC_W_CHAIN &&
         CmpOp0.getResNo() == 0 && CmpOp0->hasNUsesOfValue(1, 0) &&
         isIntrinsicWithCCAndChain(CmpOp0, Opcode, CCValid))
       return getIntrinsicCmp(DAG, Opcode, CmpOp0, CCValid, Constant, Cond);
     if (CmpOp0.getOpcode() == ISD::INTRINSIC_WO_CHAIN &&
         CmpOp0.getResNo() == CmpOp0->getNumValues() - 1 &&
         isIntrinsicWithCC(CmpOp0, Opcode, CCValid))
       return getIntrinsicCmp(DAG, Opcode, CmpOp0, CCValid, Constant, Cond);
   }
-  Comparison C(CmpOp0, CmpOp1);
+  Comparison C(CmpOp0, CmpOp1, Chain);
   C.CCMask = CCMaskForCondCode(Cond);
   if (C.Op0.getValueType().isFloatingPoint()) {
     C.CCValid = SystemZ::CCMASK_FCMP;
-    C.Opcode = SystemZISD::FCMP;
+    C.Opcode = C.Chain? SystemZISD::STRICT_FCMP : SystemZISD::FCMP;
     adjustForFNeg(C);
   } else {
+    assert(!C.Chain);
     C.CCValid = SystemZ::CCMASK_ICMP;
     C.Opcode = SystemZISD::ICMP;
     // Choose the type of comparison.  Equality and inequality tests can
     // use either signed or unsigned comparisons.  The choice also doesn't
     // matter if both sign bits are known to be clear.  In those cases we
     // want to give the main isel code the freedom to choose whichever
     // form fits best.
     if (C.CCMask == SystemZ::CCMASK_CMP_EQ ||
if (C.Opcode == SystemZISD::ICMP)
     return DAG.getNode(SystemZISD::ICMP, DL, MVT::i32, C.Op0, C.Op1,
                        DAG.getTargetConstant(C.ICmpType, DL, MVT::i32));
   if (C.Opcode == SystemZISD::TM) {
     bool RegisterOnly = (bool(C.CCMask & SystemZ::CCMASK_TM_MIXED_MSB_0) !=
                          bool(C.CCMask & SystemZ::CCMASK_TM_MIXED_MSB_1));
     return DAG.getNode(SystemZISD::TM, DL, MVT::i32, C.Op0, C.Op1,
                        DAG.getTargetConstant(RegisterOnly, DL, MVT::i32));
   }
+  if (C.Chain) {
+    SDVTList VTs = DAG.getVTList(MVT::i32, MVT::Other);
+    return DAG.getNode(C.Opcode, DL, VTs, C.Chain, C.Op0, C.Op1);
+  }
   return DAG.getNode(C.Opcode, DL, MVT::i32, C.Op0, C.Op1);
 }
 
 // Implement a 32-bit *MUL_LOHI operation by extending both operands to
 // 64 bits.  Extend is the extension type to use.  Store the high part
 // in Hi and the low part in Lo.
 static void lowerMUL_LOHI32(SelectionDAG &DAG, const SDLoc &DL, unsigned Extend,
                             SDValue Op0, SDValue Op1, SDValue &Hi,
static SDValue emitSETCC(SelectionDAG &DAG, const SDLoc &DL, SDValue CCReg,
   SDValue Ops[] = {DAG.getConstant(1, DL, MVT::i32),
                    DAG.getConstant(0, DL, MVT::i32),
                    DAG.getTargetConstant(CCValid, DL, MVT::i32),
                    DAG.getTargetConstant(CCMask, DL, MVT::i32), CCReg};
   return DAG.getNode(SystemZISD::SELECT_CCMASK, DL, MVT::i32, Ops);
 }
 
 // Return the SystemISD vector comparison operation for CC, or 0 if it cannot
-// be done directly.  IsFP is true if CC is for a floating-point rather than
-// integer comparison.
-static unsigned getVectorComparison(ISD::CondCode CC, bool IsFP) {
+// be done directly.  Mode is CmpMode::Int for integer comparisons, CmpMode::FP
+// for regular floating-point comparisons, and CmpMode::StrictFP for strict
+// floating-point comparisons.
+enum class CmpMode { Int, FP, StrictFP };
+static unsigned getVectorComparison(ISD::CondCode CC, CmpMode Mode) {
   switch (CC) {
   case ISD::SETOEQ:
   case ISD::SETEQ:
-    return IsFP ? SystemZISD::VFCMPE : SystemZISD::VICMPE;
+    switch (Mode) {
+    case CmpMode::Int:      return SystemZISD::VICMPE;
+    case CmpMode::FP:       return SystemZISD::VFCMPE;
+    case CmpMode::StrictFP: return SystemZISD::STRICT_VFCMPE;
+    default:                llvm_unreachable("Bad mode");
+    }
 
   case ISD::SETOGE:
   case ISD::SETGE:
-    return IsFP ? SystemZISD::VFCMPHE : static_cast<SystemZISD::NodeType>(0);
+    switch (Mode) {
+    case CmpMode::Int:      return 0;
+    case CmpMode::FP:       return SystemZISD::VFCMPHE;
+    case CmpMode::StrictFP: return SystemZISD::STRICT_VFCMPHE;
+    default:                llvm_unreachable("Bad mode");
+    }
 
   case ISD::SETOGT:
   case ISD::SETGT:
-    return IsFP ? SystemZISD::VFCMPH : SystemZISD::VICMPH;
+    switch (Mode) {
+    case CmpMode::Int:      return SystemZISD::VICMPH;
+    case CmpMode::FP:       return SystemZISD::VFCMPH;
+    case CmpMode::StrictFP: return SystemZISD::STRICT_VFCMPH;
+    default:                llvm_unreachable("Bad mode");
+    }
 
   case ISD::SETUGT:
-    return IsFP ? static_cast<SystemZISD::NodeType>(0) : SystemZISD::VICMPHL;
+    switch (Mode) {
+    case CmpMode::Int:      return SystemZISD::VICMPHL;
+    case CmpMode::FP:       return 0;
+    case CmpMode::StrictFP: return 0;
+    default:                llvm_unreachable("Bad mode");
+    }
 
   default:
     return 0;
   }
 }
 
 // Return the SystemZISD vector comparison operation for CC or its inverse,
 // or 0 if neither can be done directly.  Indicate in Invert whether the
-// result is for the inverse of CC.  IsFP is true if CC is for a
-// floating-point rather than integer comparison.
-static unsigned getVectorComparisonOrInvert(ISD::CondCode CC, bool IsFP,
+// result is for the inverse of CC.  Mode is as above.
+static unsigned getVectorComparisonOrInvert(ISD::CondCode CC, CmpMode Mode,
                                             bool &Invert) {
-  if (unsigned Opcode = getVectorComparison(CC, IsFP)) {
+  if (unsigned Opcode = getVectorComparison(CC, Mode)) {
     Invert = false;
     return Opcode;
   }
 
-  CC = ISD::getSetCCInverse(CC, !IsFP);
-  if (unsigned Opcode = getVectorComparison(CC, IsFP)) {
+  CC = ISD::getSetCCInverse(CC, Mode == CmpMode::Int);
+  if (unsigned Opcode = getVectorComparison(CC, Mode)) {
     Invert = true;
     return Opcode;
   }
 
   return 0;
 }
 
 // Return a v2f64 that contains the extended form of elements Start and Start+1
-// of v4f32 value Op.
+// of v4f32 value Op.  If Chain is nonnull, return the strict form.
 static SDValue expandV4F32ToV2F64(SelectionDAG &DAG, int Start, const SDLoc &DL,
-                                  SDValue Op) {
+                                  SDValue Op, SDValue Chain) {
   int Mask[] = { Start, -1, Start + 1, -1 };
   Op = DAG.getVectorShuffle(MVT::v4f32, DL, Op, DAG.getUNDEF(MVT::v4f32), Mask);
+  if (Chain) {
+    SDVTList VTs = DAG.getVTList(MVT::v2f64, MVT::Other);
+    return DAG.getNode(SystemZISD::STRICT_VEXTEND, DL, VTs, Chain, Op);
+  }
   return DAG.getNode(SystemZISD::VEXTEND, DL, MVT::v2f64, Op);
 }
 
 // Build a comparison of vectors CmpOp0 and CmpOp1 using opcode Opcode,
-// producing a result of type VT.
+// producing a result of type VT.  If Chain is nonnull, return the strict form.
 SDValue SystemZTargetLowering::getVectorCmp(SelectionDAG &DAG, unsigned Opcode,
                                             const SDLoc &DL, EVT VT,
                                             SDValue CmpOp0,
-                                            SDValue CmpOp1) const {
+                                            SDValue CmpOp1,
+                                            SDValue Chain) const {
   // There is no hardware support for v4f32 (unless we have the vector
   // enhancements facility 1), so extend the vector into two v2f64s
   // and compare those.
   if (CmpOp0.getValueType() == MVT::v4f32 &&
       !Subtarget.hasVectorEnhancements1()) {
-    SDValue H0 = expandV4F32ToV2F64(DAG, 0, DL, CmpOp0);
-    SDValue L0 = expandV4F32ToV2F64(DAG, 2, DL, CmpOp0);
-    SDValue H1 = expandV4F32ToV2F64(DAG, 0, DL, CmpOp1);
-    SDValue L1 = expandV4F32ToV2F64(DAG, 2, DL, CmpOp1);
+    SDValue H0 = expandV4F32ToV2F64(DAG, 0, DL, CmpOp0, Chain);
+    SDValue L0 = expandV4F32ToV2F64(DAG, 2, DL, CmpOp0, Chain);
+    SDValue H1 = expandV4F32ToV2F64(DAG, 0, DL, CmpOp1, Chain);
+    SDValue L1 = expandV4F32ToV2F64(DAG, 2, DL, CmpOp1, Chain);
+    if (Chain) {
+      SDVTList VTs = DAG.getVTList(MVT::v2i64, MVT::Other);
+      SDValue HRes = DAG.getNode(Opcode, DL, VTs, Chain, H0, H1);
+      SDValue LRes = DAG.getNode(Opcode, DL, VTs, Chain, L0, L1);
+      SDValue Res = DAG.getNode(SystemZISD::PACK, DL, VT, HRes, LRes);
+      SDValue Chains[6] = { H0.getValue(1), L0.getValue(1),
+                            H1.getValue(1), L1.getValue(1),
+                            HRes.getValue(1), LRes.getValue(1) };
+      SDValue NewChain = DAG.getNode(ISD::TokenFactor, DL, MVT::Other, Chains);
+      SDValue Ops[2] = { Res, NewChain };
+      return DAG.getMergeValues(Ops, DL);
+    }
     SDValue HRes = DAG.getNode(Opcode, DL, MVT::v2i64, H0, H1);
     SDValue LRes = DAG.getNode(Opcode, DL, MVT::v2i64, L0, L1);
     return DAG.getNode(SystemZISD::PACK, DL, VT, HRes, LRes);
   }
+  if (Chain) {
+    SDVTList VTs = DAG.getVTList(VT, MVT::Other);
+    return DAG.getNode(Opcode, DL, VTs, Chain, CmpOp0, CmpOp1);
+  }
   return DAG.getNode(Opcode, DL, VT, CmpOp0, CmpOp1);
 }
 
 // Lower a vector comparison of type CC between CmpOp0 and CmpOp1, producing
-// an integer mask of type VT.
+// an integer mask of type VT.  If Chain is nonnull, we have a strict
+// floating-point comparison.
 SDValue SystemZTargetLowering::lowerVectorSETCC(SelectionDAG &DAG,
                                                 const SDLoc &DL, EVT VT,
                                                 ISD::CondCode CC,
                                                 SDValue CmpOp0,
-                                                SDValue CmpOp1) const {
+                                                SDValue CmpOp1,
+                                                SDValue Chain) const {
   bool IsFP = CmpOp0.getValueType().isFloatingPoint();
+  assert (!Chain || IsFP);
+  CmpMode Mode = Chain ? CmpMode::StrictFP : IsFP ? CmpMode::FP : CmpMode::Int;
   bool Invert = false;
   SDValue Cmp;
   switch (CC) {
     // Handle tests for order using (or (ogt y x) (oge x y)).
   case ISD::SETUO:
     Invert = true;
     LLVM_FALLTHROUGH;
   case ISD::SETO: {
     assert(IsFP && "Unexpected integer comparison");
-    SDValue LT = getVectorCmp(DAG, SystemZISD::VFCMPH, DL, VT, CmpOp1, CmpOp0);
-    SDValue GE = getVectorCmp(DAG, SystemZISD::VFCMPHE, DL, VT, CmpOp0, CmpOp1);
+    SDValue LT = getVectorCmp(DAG, getVectorComparison(ISD::SETOGT, Mode),
+                              DL, VT, CmpOp1, CmpOp0, Chain);
+    SDValue GE = getVectorCmp(DAG, getVectorComparison(ISD::SETOGE, Mode),
+                              DL, VT, CmpOp0, CmpOp1, Chain);
     Cmp = DAG.getNode(ISD::OR, DL, VT, LT, GE);
+    if (Chain)
+      Chain = DAG.getNode(ISD::TokenFactor, DL, MVT::Other,
+                          LT.getValue(1), GE.getValue(1));
     break;
   }
 
     // Handle <> tests using (or (ogt y x) (ogt x y)).
   case ISD::SETUEQ:
     Invert = true;
     LLVM_FALLTHROUGH;
   case ISD::SETONE: {
     assert(IsFP && "Unexpected integer comparison");
-    SDValue LT = getVectorCmp(DAG, SystemZISD::VFCMPH, DL, VT, CmpOp1, CmpOp0);
-    SDValue GT = getVectorCmp(DAG, SystemZISD::VFCMPH, DL, VT, CmpOp0, CmpOp1);
+    SDValue LT = getVectorCmp(DAG, getVectorComparison(ISD::SETOGT, Mode),
+                              DL, VT, CmpOp1, CmpOp0, Chain);
+    SDValue GT = getVectorCmp(DAG, getVectorComparison(ISD::SETOGT, Mode),
+                              DL, VT, CmpOp0, CmpOp1, Chain);
     Cmp = DAG.getNode(ISD::OR, DL, VT, LT, GT);
+    if (Chain)
+      Chain = DAG.getNode(ISD::TokenFactor, DL, MVT::Other,
+                          LT.getValue(1), GT.getValue(1));
     break;
   }
 
     // Otherwise a single comparison is enough.  It doesn't really
     // matter whether we try the inversion or the swap first, since
     // there are no cases where both work.
   default:
-    if (unsigned Opcode = getVectorComparisonOrInvert(CC, IsFP, Invert))
-      Cmp = getVectorCmp(DAG, Opcode, DL, VT, CmpOp0, CmpOp1);
+    if (unsigned Opcode = getVectorComparisonOrInvert(CC, Mode, Invert))
+      Cmp = getVectorCmp(DAG, Opcode, DL, VT, CmpOp0, CmpOp1, Chain);
     else {
       CC = ISD::getSetCCSwappedOperands(CC);
-      if (unsigned Opcode = getVectorComparisonOrInvert(CC, IsFP, Invert))
-        Cmp = getVectorCmp(DAG, Opcode, DL, VT, CmpOp1, CmpOp0);
+      if (unsigned Opcode = getVectorComparisonOrInvert(CC, Mode, Invert))
+        Cmp = getVectorCmp(DAG, Opcode, DL, VT, CmpOp1, CmpOp0, Chain);
       else
         llvm_unreachable("Unhandled comparison");
     }
+    if (Chain)
+      Chain = Cmp.getValue(1);
     break;
   }
   if (Invert) {
     SDValue Mask =
       DAG.getSplatBuildVector(VT, DL, DAG.getConstant(-1, DL, MVT::i64));
     Cmp = DAG.getNode(ISD::XOR, DL, VT, Cmp, Mask);
   }
+  if (Chain && Chain.getNode() != Cmp.getNode()) {
+    SDValue Ops[2] = { Cmp, Chain };
+    Cmp = DAG.getMergeValues(Ops, DL);
+  }
   return Cmp;
 }
 
 SDValue SystemZTargetLowering::lowerSETCC(SDValue Op,
                                           SelectionDAG &DAG) const {
   SDValue CmpOp0   = Op.getOperand(0);
   SDValue CmpOp1   = Op.getOperand(1);
   ISD::CondCode CC = cast<CondCodeSDNode>(Op.getOperand(2))->get();
   SDLoc DL(Op);
   EVT VT = Op.getValueType();
   if (VT.isVector())
     return lowerVectorSETCC(DAG, DL, VT, CC, CmpOp0, CmpOp1);
 
   Comparison C(getCmp(DAG, CmpOp0, CmpOp1, CC, DL));
   SDValue CCReg = emitCmp(DAG, DL, C);
   return emitSETCC(DAG, DL, CCReg, C.CCValid, C.CCMask);
 }
 
+SDValue SystemZTargetLowering::lowerSTRICT_FSETCC(SDValue Op,
+                                                  SelectionDAG &DAG) const {
+  SDValue Chain    = Op.getOperand(0);
+  SDValue CmpOp0   = Op.getOperand(1);
+  SDValue CmpOp1   = Op.getOperand(2);
+  ISD::CondCode CC = cast<CondCodeSDNode>(Op.getOperand(3))->get();
+  SDLoc DL(Op);
+  EVT VT = Op.getNode()->getValueType(0);
+  if (VT.isVector()) {
+    SDValue Res = lowerVectorSETCC(DAG, DL, VT, CC, CmpOp0, CmpOp1, Chain);
+    return Res.getValue(Op.getResNo());
+  }
+
+  Comparison C(getCmp(DAG, CmpOp0, CmpOp1, CC, DL, Chain));
+  SDValue CCReg = emitCmp(DAG, DL, C);
+  CCReg->setFlags(Op->getFlags());
+  SDValue Result = emitSETCC(DAG, DL, CCReg, C.CCValid, C.CCMask);
+  SDValue Ops[2] = { Result, CCReg.getValue(1) };
+  return DAG.getMergeValues(Ops, DL);
+}
+
 SDValue SystemZTargetLowering::lowerBR_CC(SDValue Op, SelectionDAG &DAG) const {
   ISD::CondCode CC = cast<CondCodeSDNode>(Op.getOperand(1))->get();
   SDValue CmpOp0   = Op.getOperand(2);
   SDValue CmpOp1   = Op.getOperand(3);
   SDValue Dest     = Op.getOperand(4);
   SDLoc DL(Op);
 
   Comparison C(getCmp(DAG, CmpOp0, CmpOp1, CC, DL));
SDValue SystemZTargetLowering::LowerOperation(SDValue Op,
   case ISD::RETURNADDR:
     return lowerRETURNADDR(Op, DAG);
   case ISD::BR_CC:
     return lowerBR_CC(Op, DAG);
   case ISD::SELECT_CC:
     return lowerSELECT_CC(Op, DAG);
   case ISD::SETCC:
     return lowerSETCC(Op, DAG);
+  case ISD::STRICT_FSETCC:
+    return lowerSTRICT_FSETCC(Op, DAG);
   case ISD::GlobalAddress:
     return lowerGlobalAddress(cast<GlobalAddressSDNode>(Op), DAG);
   case ISD::GlobalTLSAddress:
     return lowerGlobalTLSAddress(cast<GlobalAddressSDNode>(Op), DAG);
   case ISD::BlockAddress:
     return lowerBlockAddress(cast<BlockAddressSDNode>(Op), DAG);
   case ISD::JumpTable:
     return lowerJumpTable(cast<JumpTableSDNode>(Op), DAG);
switch ((SystemZISD::NodeType)Opcode) {
     OPCODE(SIBCALL);
     OPCODE(TLS_GDCALL);
     OPCODE(TLS_LDCALL);
     OPCODE(PCREL_WRAPPER);
     OPCODE(PCREL_OFFSET);
     OPCODE(IABS);
     OPCODE(ICMP);
     OPCODE(FCMP);
+    OPCODE(STRICT_FCMP);
     OPCODE(TM);
     OPCODE(BR_CCMASK);
     OPCODE(SELECT_CCMASK);
     OPCODE(ADJDYNALLOC);
     OPCODE(POPCNT);
     OPCODE(SMUL_LOHI);
     OPCODE(UMUL_LOHI);
     OPCODE(SDIVREM);
switch ((SystemZISD::NodeType)Opcode) {
     OPCODE(VSUM);
     OPCODE(VICMPE);
     OPCODE(VICMPH);
     OPCODE(VICMPHL);
     OPCODE(VICMPES);
     OPCODE(VICMPHS);
     OPCODE(VICMPHLS);
     OPCODE(VFCMPE);
+    OPCODE(STRICT_VFCMPE);
     OPCODE(VFCMPH);
+    OPCODE(STRICT_VFCMPH);
     OPCODE(VFCMPHE);
+    OPCODE(STRICT_VFCMPHE);
     OPCODE(VFCMPES);
     OPCODE(VFCMPHS);
     OPCODE(VFCMPHES);
     OPCODE(VFTCI);
     OPCODE(VEXTEND);
+    OPCODE(STRICT_VEXTEND);
     OPCODE(VROUND);
     OPCODE(VTM);
     OPCODE(VFAE_CC);
     OPCODE(VFAEZ_CC);
     OPCODE(VFEE_CC);
     OPCODE(VFEEZ_CC);
     OPCODE(VFENE_CC);
     OPCODE(VFENEZ_CC);

lib/Target/SystemZ/SystemZInstrFP.td

let Uses = [FPC], mayRaiseFPException = 1, Defs = [CC] in {
   def DIDBR : TernaryRRFb<"didbr", 0xB35B, FP64, FP64, FP64>;
 }
 
 //===----------------------------------------------------------------------===//
 // Comparisons
 //===----------------------------------------------------------------------===//
 
 let Uses = [FPC], mayRaiseFPException = 1, Defs = [CC], CCValues = 0xF in {
-  def CEBR : CompareRRE<"cebr", 0xB309, z_fcmp, FP32,  FP32>;
-  def CDBR : CompareRRE<"cdbr", 0xB319, z_fcmp, FP64,  FP64>;
-  def CXBR : CompareRRE<"cxbr", 0xB349, z_fcmp, FP128, FP128>;
+  def CEBR : CompareRRE<"cebr", 0xB309, z_any_fcmp, FP32,  FP32>;
+  def CDBR : CompareRRE<"cdbr", 0xB319, z_any_fcmp, FP64,  FP64>;
+  def CXBR : CompareRRE<"cxbr", 0xB349, z_any_fcmp, FP128, FP128>;
 
-  def CEB : CompareRXE<"ceb", 0xED09, z_fcmp, FP32, load, 4>;
-  def CDB : CompareRXE<"cdb", 0xED19, z_fcmp, FP64, load, 8>;
+  def CEB : CompareRXE<"ceb", 0xED09, z_any_fcmp, FP32, load, 4>;
+  def CDB : CompareRXE<"cdb", 0xED19, z_any_fcmp, FP64, load, 8>;
 
   def KEBR : CompareRRE<"kebr", 0xB308, null_frag, FP32,  FP32>;
   def KDBR : CompareRRE<"kdbr", 0xB318, null_frag, FP64,  FP64>;
   def KXBR : CompareRRE<"kxbr", 0xB348, null_frag, FP128, FP128>;
 
   def KEB : CompareRXE<"keb", 0xED08, null_frag, FP32, load, 4>;
   def KDB : CompareRXE<"kdb", 0xED18, null_frag, FP64, load, 8>;
 }

lib/Target/SystemZ/SystemZInstrVector.td

let Predicates = [FeatureVectorEnhancements1] in {
     defm : VectorRounding<VFISB, v128sb>;
     defm : VectorRounding<WFISB, v32sb>;
     defm : VectorRounding<WFIXB, v128xb>;
   }
 
   // Load lengthened.
   let Uses = [FPC], mayRaiseFPException = 1 in {
     def VLDE  : UnaryVRRaFloatGeneric<"vlde", 0xE7C4>;
-    def VLDEB : UnaryVRRa<"vldeb", 0xE7C4, z_vextend, v128db, v128sb, 2, 0>;
+    def VLDEB : UnaryVRRa<"vldeb", 0xE7C4, z_any_vextend, v128db, v128sb, 2, 0>;
     def WLDEB : UnaryVRRa<"wldeb", 0xE7C4, any_fpextend, v64db, v32sb, 2, 8>;
   }
   let Predicates = [FeatureVectorEnhancements1] in {
     let Uses = [FPC], mayRaiseFPException = 1 in {
       let isAsmParserOnly = 1 in {
         def VFLL  : UnaryVRRaFloatGeneric<"vfll", 0xE7C4>;
         def VFLLS : UnaryVRRa<"vflls", 0xE7C4, null_frag, v128db, v128sb, 2, 0>;
         def WFLLS : UnaryVRRa<"wflls", 0xE7C4, null_frag, v64db, v32sb, 2, 8>;

 //===----------------------------------------------------------------------===//
 // Floating-point comparison
 //===----------------------------------------------------------------------===//
 
 let Predicates = [FeatureVector] in {
   // Compare scalar.
   let Uses = [FPC], mayRaiseFPException = 1, Defs = [CC] in {
     def WFC   : CompareVRRaFloatGeneric<"wfc", 0xE7CB>;
-    def WFCDB : CompareVRRa<"wfcdb", 0xE7CB, z_fcmp, v64db, 3>;
+    def WFCDB : CompareVRRa<"wfcdb", 0xE7CB, z_any_fcmp, v64db, 3>;
     let Predicates = [FeatureVectorEnhancements1] in {
-      def WFCSB : CompareVRRa<"wfcsb", 0xE7CB, z_fcmp, v32sb, 2>;
-      def WFCXB : CompareVRRa<"wfcxb", 0xE7CB, z_fcmp, v128xb, 4>;
+      def WFCSB : CompareVRRa<"wfcsb", 0xE7CB, z_any_fcmp, v32sb, 2>;
+      def WFCXB : CompareVRRa<"wfcxb", 0xE7CB, z_any_fcmp, v128xb, 4>;
     }
   }
 
   // Compare and signal scalar.
   let Uses = [FPC], mayRaiseFPException = 1, Defs = [CC] in {
     def WFK   : CompareVRRaFloatGeneric<"wfk", 0xE7CA>;
     def WFKDB : CompareVRRa<"wfkdb", 0xE7CA, null_frag, v64db, 3>;
     let Predicates = [FeatureVectorEnhancements1] in {
       def WFKSB : CompareVRRa<"wfksb", 0xE7CA, null_frag, v32sb, 2>;
       def WFKXB : CompareVRRa<"wfkxb", 0xE7CA, null_frag, v128xb, 4>;
     }
   }
 
   // Compare equal.
   let Uses = [FPC], mayRaiseFPException = 1 in {
     def  VFCE   : BinaryVRRcSPairFloatGeneric<"vfce", 0xE7E8>;
-    defm VFCEDB : BinaryVRRcSPair<"vfcedb", 0xE7E8, z_vfcmpe, z_vfcmpes,
+    defm VFCEDB : BinaryVRRcSPair<"vfcedb", 0xE7E8, z_any_vfcmpe, z_vfcmpes,
                                   v128g, v128db, 3, 0>;
     defm WFCEDB : BinaryVRRcSPair<"wfcedb", 0xE7E8, null_frag, null_frag,
                                   v64g, v64db, 3, 8>;
     let Predicates = [FeatureVectorEnhancements1] in {
-      defm VFCESB : BinaryVRRcSPair<"vfcesb", 0xE7E8, z_vfcmpe, z_vfcmpes,
+      defm VFCESB : BinaryVRRcSPair<"vfcesb", 0xE7E8, z_any_vfcmpe, z_vfcmpes,
                                     v128f, v128sb, 2, 0>;
       defm WFCESB : BinaryVRRcSPair<"wfcesb", 0xE7E8, null_frag, null_frag,
                                     v32f, v32sb, 2, 8>;
       defm WFCEXB : BinaryVRRcSPair<"wfcexb", 0xE7E8, null_frag, null_frag,
                                     v128q, v128xb, 4, 8>;
     }
   }
 
defm WFKESB : BinaryVRRcSPair<"wfkesb", 0xE7E8, null_frag, null_frag,
                                   v32f, v32sb, 2, 12>;
     defm WFKEXB : BinaryVRRcSPair<"wfkexb", 0xE7E8, null_frag, null_frag,
                                   v128q, v128xb, 4, 12>;
   }
 
   // Compare high.
   let Uses = [FPC], mayRaiseFPException = 1 in {
     def  VFCH   : BinaryVRRcSPairFloatGeneric<"vfch", 0xE7EB>;
-    defm VFCHDB : BinaryVRRcSPair<"vfchdb", 0xE7EB, z_vfcmph, z_vfcmphs,
+    defm VFCHDB : BinaryVRRcSPair<"vfchdb", 0xE7EB, z_any_vfcmph, z_vfcmphs,
                                   v128g, v128db, 3, 0>;
     defm WFCHDB : BinaryVRRcSPair<"wfchdb", 0xE7EB, null_frag, null_frag,
                                   v64g, v64db, 3, 8>;
     let Predicates = [FeatureVectorEnhancements1] in {
-      defm VFCHSB : BinaryVRRcSPair<"vfchsb", 0xE7EB, z_vfcmph, z_vfcmphs,
+      defm VFCHSB : BinaryVRRcSPair<"vfchsb", 0xE7EB, z_any_vfcmph, z_vfcmphs,
                                     v128f, v128sb, 2, 0>;
       defm WFCHSB : BinaryVRRcSPair<"wfchsb", 0xE7EB, null_frag, null_frag,
                                     v32f, v32sb, 2, 8>;
       defm WFCHXB : BinaryVRRcSPair<"wfchxb", 0xE7EB, null_frag, null_frag,
                                     v128q, v128xb, 4, 8>;
     }
   }
 
defm WFKHSB : BinaryVRRcSPair<"wfkhsb", 0xE7EB, null_frag, null_frag,
                                   v32f, v32sb, 2, 12>;
     defm WFKHXB : BinaryVRRcSPair<"wfkhxb", 0xE7EB, null_frag, null_frag,
                                   v128q, v128xb, 4, 12>;
   }
 
   // Compare high or equal.
   let Uses = [FPC], mayRaiseFPException = 1 in {
     def  VFCHE   : BinaryVRRcSPairFloatGeneric<"vfche", 0xE7EA>;
-    defm VFCHEDB : BinaryVRRcSPair<"vfchedb", 0xE7EA, z_vfcmphe, z_vfcmphes,
+    defm VFCHEDB : BinaryVRRcSPair<"vfchedb", 0xE7EA, z_any_vfcmphe, z_vfcmphes,
                                    v128g, v128db, 3, 0>;
     defm WFCHEDB : BinaryVRRcSPair<"wfchedb", 0xE7EA, null_frag, null_frag,
                                    v64g, v64db, 3, 8>;
     let Predicates = [FeatureVectorEnhancements1] in {
-      defm VFCHESB : BinaryVRRcSPair<"vfchesb", 0xE7EA, z_vfcmphe, z_vfcmphes,
+      defm VFCHESB : BinaryVRRcSPair<"vfchesb", 0xE7EA, z_any_vfcmphe, z_vfcmphes,
                                      v128f, v128sb, 2, 0>;
       defm WFCHESB : BinaryVRRcSPair<"wfchesb", 0xE7EA, null_frag, null_frag,
                                      v32f, v32sb, 2, 8>;
       defm WFCHEXB : BinaryVRRcSPair<"wfchexb", 0xE7EA, null_frag, null_frag,
                                      v128q, v128xb, 4, 8>;
     }
   }
 

lib/Target/SystemZ/SystemZOperators.td

def z_tls_ldcall        : SDNode<"SystemZISD::TLS_LDCALL", SDT_ZCall,
                                  [SDNPHasChain, SDNPInGlue, SDNPOutGlue,
                                   SDNPVariadic]>;
 def z_pcrel_wrapper     : SDNode<"SystemZISD::PCREL_WRAPPER", SDT_ZWrapPtr, []>;
 def z_pcrel_offset      : SDNode<"SystemZISD::PCREL_OFFSET",
                                  SDT_ZWrapOffset, []>;
 def z_iabs              : SDNode<"SystemZISD::IABS", SDTIntUnaryOp, []>;
 def z_icmp              : SDNode<"SystemZISD::ICMP", SDT_ZICmp>;
 def z_fcmp              : SDNode<"SystemZISD::FCMP", SDT_ZCmp>;
+def z_strict_fcmp       : SDNode<"SystemZISD::STRICT_FCMP", SDT_ZCmp,
+                                 [SDNPHasChain]>;
 def z_tm                : SDNode<"SystemZISD::TM", SDT_ZICmp>;
 def z_br_ccmask_1       : SDNode<"SystemZISD::BR_CCMASK", SDT_ZBRCCMask,
                                  [SDNPHasChain]>;
 def z_select_ccmask_1   : SDNode<"SystemZISD::SELECT_CCMASK",
                                  SDT_ZSelectCCMask>;
 def z_ipm_1             : SDNode<"SystemZISD::IPM", SDT_ZIPM>;
 def z_adjdynalloc       : SDNode<"SystemZISD::ADJDYNALLOC", SDT_ZAdjDynAlloc>;
 def z_popcnt            : SDNode<"SystemZISD::POPCNT", SDTIntUnaryOp>;

 def z_vsum              : SDNode<"SystemZISD::VSUM", SDT_ZVecBinaryConv>;
 def z_vicmpe            : SDNode<"SystemZISD::VICMPE", SDT_ZVecBinary>;
 def z_vicmph            : SDNode<"SystemZISD::VICMPH", SDT_ZVecBinary>;
 def z_vicmphl           : SDNode<"SystemZISD::VICMPHL", SDT_ZVecBinary>;
 def z_vicmpes           : SDNode<"SystemZISD::VICMPES", SDT_ZVecBinaryCC>;
 def z_vicmphs           : SDNode<"SystemZISD::VICMPHS", SDT_ZVecBinaryCC>;
 def z_vicmphls          : SDNode<"SystemZISD::VICMPHLS", SDT_ZVecBinaryCC>;
 def z_vfcmpe            : SDNode<"SystemZISD::VFCMPE", SDT_ZVecBinaryConv>;
+def z_strict_vfcmpe     : SDNode<"SystemZISD::STRICT_VFCMPE",
+                                 SDT_ZVecBinaryConv, [SDNPHasChain]>;
 def z_vfcmph            : SDNode<"SystemZISD::VFCMPH", SDT_ZVecBinaryConv>;
+def z_strict_vfcmph     : SDNode<"SystemZISD::STRICT_VFCMPH",
+                                 SDT_ZVecBinaryConv, [SDNPHasChain]>;
 def z_vfcmphe           : SDNode<"SystemZISD::VFCMPHE", SDT_ZVecBinaryConv>;
+def z_strict_vfcmphe    : SDNode<"SystemZISD::STRICT_VFCMPHE",
+                                 SDT_ZVecBinaryConv, [SDNPHasChain]>;
 def z_vfcmpes           : SDNode<"SystemZISD::VFCMPES", SDT_ZVecBinaryConvCC>;
 def z_vfcmphs           : SDNode<"SystemZISD::VFCMPHS", SDT_ZVecBinaryConvCC>;
 def z_vfcmphes          : SDNode<"SystemZISD::VFCMPHES", SDT_ZVecBinaryConvCC>;
 def z_vextend           : SDNode<"SystemZISD::VEXTEND", SDT_ZVecUnaryConv>;
+def z_strict_vextend    : SDNode<"SystemZISD::STRICT_VEXTEND",
+                                 SDT_ZVecUnaryConv, [SDNPHasChain]>;
 def z_vround            : SDNode<"SystemZISD::VROUND", SDT_ZVecUnaryConv>;
 def z_vtm               : SDNode<"SystemZISD::VTM", SDT_ZCmp>;
 def z_vfae_cc           : SDNode<"SystemZISD::VFAE_CC", SDT_ZVecTernaryIntCC>;
 def z_vfaez_cc          : SDNode<"SystemZISD::VFAEZ_CC", SDT_ZVecTernaryIntCC>;
 def z_vfee_cc           : SDNode<"SystemZISD::VFEE_CC", SDT_ZVecBinaryCC>;
 def z_vfeez_cc          : SDNode<"SystemZISD::VFEEZ_CC", SDT_ZVecBinaryCC>;
 def z_vfene_cc          : SDNode<"SystemZISD::VFENE_CC", SDT_ZVecBinaryCC>;
 def z_vfenez_cc         : SDNode<"SystemZISD::VFENEZ_CC", SDT_ZVecBinaryCC>;

 def any_fnma : PatFrag<(ops node:$src1, node:$src2, node:$src3),
                        (fneg (any_fma node:$src1, node:$src2, node:$src3))>;
 def any_fnms : PatFrag<(ops node:$src1, node:$src2, node:$src3),
                        (fneg (any_fms node:$src1, node:$src2, node:$src3))>;
 
 // Floating-point negative absolute.
 def fnabs : PatFrag<(ops node:$ptr), (fneg (fabs node:$ptr))>;
 
+// Strict floating-point fragments.
+def z_any_fcmp    : PatFrags<(ops node:$lhs, node:$rhs),
+                             [(z_strict_fcmp node:$lhs, node:$rhs),
+                              (z_fcmp node:$lhs, node:$rhs)]>;
+def z_any_vfcmpe  : PatFrags<(ops node:$lhs, node:$rhs),
+                             [(z_strict_vfcmpe node:$lhs, node:$rhs),
+                              (z_vfcmpe node:$lhs, node:$rhs)]>;
+def z_any_vfcmph  : PatFrags<(ops node:$lhs, node:$rhs),
+                             [(z_strict_vfcmph node:$lhs, node:$rhs),
+                              (z_vfcmph node:$lhs, node:$rhs)]>;
+def z_any_vfcmphe : PatFrags<(ops node:$lhs, node:$rhs),
+                             [(z_strict_vfcmphe node:$lhs, node:$rhs),
+                              (z_vfcmphe node:$lhs, node:$rhs)]>;
+def z_any_vextend : PatFrags<(ops node:$src),
+                             [(z_strict_vextend node:$src),
+                              (z_vextend node:$src)]>;
+
 // Create a unary operator that loads from memory and then performs
 // the given operation on it.
 class loadu<SDPatternOperator operator, SDPatternOperator load = load>
   : PatFrag<(ops node:$addr), (operator (load node:$addr))>;
 
 // Create a store operator that performs the given unary operation
 // on the value before storing it.
 class storeu<SDPatternOperator operator, SDPatternOperator store = store>

lib/Target/SystemZ/SystemZPatterns.td

multiclass BlockLoadStore<SDPatternOperator load, ValueType vt,
   defm : BinaryLoadStore<block_xor1, load, vt, xc, length>;
   defm : BinaryLoadStore<block_xor2, load, vt, xc, length>;
 }
 
 // Record that INSN is a LOAD AND TEST that can be used to compare
 // registers in CLS against zero.  The instruction has separate R1 and R2
 // operands, but they must be the same when the instruction is used like this.
 multiclass CompareZeroFP<Instruction insn, RegisterOperand cls> {
-  def : Pat<(z_fcmp cls:$reg, (fpimm0)), (insn cls:$reg, cls:$reg)>;
+  def : Pat<(z_any_fcmp cls:$reg, (fpimm0)), (insn cls:$reg, cls:$reg)>;
   // The sign of the zero makes no difference.
-  def : Pat<(z_fcmp cls:$reg, (fpimmneg0)), (insn cls:$reg, cls:$reg)>;
+  def : Pat<(z_any_fcmp cls:$reg, (fpimmneg0)), (insn cls:$reg, cls:$reg)>;
 }
 
 // Use INSN for performing binary operation OPERATION of type VT
 // on registers of class CLS.
 class BinaryRRWithType<Instruction insn, RegisterOperand cls,
                        SDPatternOperator operator, ValueType vt>
   : Pat<(vt (operator cls:$x, cls:$y)), (insn cls:$x, cls:$y)>;
 

lib/Target/X86/X86ISelLowering.cpp

if (!Subtarget.hasMOVBE())
     setOperationAction(ISD::BSWAP          , MVT::i16  , Expand);
 
   // These should be promoted to a larger select which is supported.
   setOperationAction(ISD::SELECT          , MVT::i1   , Promote);
   // X86 wants to expand cmov itself.
   for (auto VT : { MVT::f32, MVT::f64, MVT::f80, MVT::f128 }) {
     setOperationAction(ISD::SELECT, VT, Custom);
     setOperationAction(ISD::SETCC, VT, Custom);
+    setOperationAction(ISD::STRICT_FSETCC, VT, Custom);
   }
   for (auto VT : { MVT::i8, MVT::i16, MVT::i32, MVT::i64 }) {
     if (VT == MVT::i64 && !Subtarget.is64Bit())
       continue;
     setOperationAction(ISD::SELECT, VT, Custom);
     setOperationAction(ISD::SETCC,  VT, Custom);
   }
 
if (isTypeLegal(MVT::f64)) {
       setOperationAction(ISD::STRICT_FP_ROUND, MVT::f64, Custom);
     }
     if (isTypeLegal(MVT::f80)) {
       setOperationAction(ISD::FP_ROUND, MVT::f80, Custom);
       setOperationAction(ISD::STRICT_FP_ROUND, MVT::f80, Custom);
     }
 
     setOperationAction(ISD::SETCC, MVT::f128, Custom);
+    setOperationAction(ISD::STRICT_FSETCC, MVT::f128, Custom);
 
     setLoadExtAction(ISD::EXTLOAD, MVT::f128, MVT::f32, Expand);
     setLoadExtAction(ISD::EXTLOAD, MVT::f128, MVT::f64, Expand);
     setLoadExtAction(ISD::EXTLOAD, MVT::f128, MVT::f80, Expand);
     setTruncStoreAction(MVT::f128, MVT::f32, Expand);
     setTruncStoreAction(MVT::f128, MVT::f64, Expand);
     setTruncStoreAction(MVT::f128, MVT::f80, Expand);
   }
SDValue X86TargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   case ISD::FADD:
   case ISD::FSUB:               return lowerFaddFsub(Op, DAG);
   case ISD::FMUL:               return LowerF128Call(Op, DAG, RTLIB::MUL_F128);
   case ISD::FDIV:               return LowerF128Call(Op, DAG, RTLIB::DIV_F128);
   case ISD::FABS:
   case ISD::FNEG:               return LowerFABSorFNEG(Op, DAG);
   case ISD::FCOPYSIGN:          return LowerFCOPYSIGN(Op, DAG);
   case ISD::FGETSIGN:           return LowerFGETSIGN(Op, DAG);
+  case ISD::STRICT_FSETCC:
+    // FIXME: Just ignore constrained floating-point semantics for now.
+    Op = SDValue(DAG.mutateStrictFPToFP(Op.getNode()), 0);
+    LLVM_FALLTHROUGH;
   case ISD::SETCC:              return LowerSETCC(Op, DAG);
   case ISD::SETCCCARRY:         return LowerSETCCCARRY(Op, DAG);
   case ISD::SELECT:             return LowerSELECT(Op, DAG);
   case ISD::BRCOND:             return LowerBRCOND(Op, DAG);
   case ISD::JumpTable:          return LowerJumpTable(Op, DAG);
   case ISD::VASTART:            return LowerVASTART(Op, DAG);
   case ISD::VAARG:              return LowerVAARG(Op, DAG);
   case ISD::VACOPY:             return LowerVACOPY(Op, Subtarget, DAG);

test/CodeGen/SystemZ/fp-strict-cmp-01.ll

+; Test 32-bit floating-point comparison.  The tests assume a z10 implementation
+; of select, using conditional branches rather than LOCGR.
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z10 \
+; RUN:   | FileCheck -check-prefix=CHECK -check-prefix=CHECK-SCALAR %s
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14 \
+; RUN:   | FileCheck -check-prefix=CHECK -check-prefix=CHECK-VECTOR %s
+
+declare float @foo()
+
+; Check comparison with registers.
+define i64 @f1(i64 %a, i64 %b, float %f1, float %f2) #0 {
+; CHECK-LABEL: f1:
+; CHECK: cebr %f0, %f2
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check the low end of the CEB range.
+define i64 @f2(i64 %a, i64 %b, float %f1, float *%ptr) #0 {
+; CHECK-LABEL: f2:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %f2 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check the high end of the aligned CEB range.
+define i64 @f3(i64 %a, i64 %b, float %f1, float *%base) #0 {
+; CHECK-LABEL: f3:
+; CHECK: ceb %f0, 4092(%r4)
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %ptr = getelementptr float, float *%base, i64 1023
+  %f2 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check the next word up, which needs separate address logic.
+; Other sequences besides this one would be OK.
+define i64 @f4(i64 %a, i64 %b, float %f1, float *%base) #0 {
+; CHECK-LABEL: f4:
+; CHECK: aghi %r4, 4096
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %ptr = getelementptr float, float *%base, i64 1024
+  %f2 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check negative displacements, which also need separate address logic.
+define i64 @f5(i64 %a, i64 %b, float %f1, float *%base) #0 {
+; CHECK-LABEL: f5:
+; CHECK: aghi %r4, -4
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %ptr = getelementptr float, float *%base, i64 -1
+  %f2 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check that CEB allows indices.
+define i64 @f6(i64 %a, i64 %b, float %f1, float *%base, i64 %index) #0 {
+; CHECK-LABEL: f6:
+; CHECK: sllg %r1, %r5, 2
+; CHECK: ceb %f0, 400(%r1,%r4)
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %ptr1 = getelementptr float, float *%base, i64 %index
+  %ptr2 = getelementptr float, float *%ptr1, i64 100
+  %f2 = load float, float *%ptr2
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check that comparisons of spilled values can use CEB rather than CEBR.
+define float @f7(float *%ptr0) #0 {
+; CHECK-LABEL: f7:
+; CHECK: brasl %r14, foo@PLT
+; CHECK-SCALAR: ceb {{%f[0-9]+}}, 16{{[04]}}(%r15)
+; CHECK: br %r14
+  %ptr1 = getelementptr float, float *%ptr0, i64 2
+  %ptr2 = getelementptr float, float *%ptr0, i64 4
+  %ptr3 = getelementptr float, float *%ptr0, i64 6
+  %ptr4 = getelementptr float, float *%ptr0, i64 8
+  %ptr5 = getelementptr float, float *%ptr0, i64 10
+  %ptr6 = getelementptr float, float *%ptr0, i64 12
+  %ptr7 = getelementptr float, float *%ptr0, i64 14
+  %ptr8 = getelementptr float, float *%ptr0, i64 16
+  %ptr9 = getelementptr float, float *%ptr0, i64 18
+  %ptr10 = getelementptr float, float *%ptr0, i64 20
+
+  %val0 = load float, float *%ptr0
+  %val1 = load float, float *%ptr1
+  %val2 = load float, float *%ptr2
+  %val3 = load float, float *%ptr3
+  %val4 = load float, float *%ptr4
+  %val5 = load float, float *%ptr5
+  %val6 = load float, float *%ptr6
+  %val7 = load float, float *%ptr7
+  %val8 = load float, float *%ptr8
+  %val9 = load float, float *%ptr9
+  %val10 = load float, float *%ptr10
+
+  %ret = call float @foo() #0
+
+  %cmp0 = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %ret, float %val0,
+                                               metadata !"fpexcept.strict") #0
+  %cmp1 = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %ret, float %val1,
+                                               metadata !"fpexcept.strict") #0
+  %cmp2 = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %ret, float %val2,
+                                               metadata !"fpexcept.strict") #0
+  %cmp3 = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %ret, float %val3,
+                                               metadata !"fpexcept.strict") #0
+  %cmp4 = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %ret, float %val4,
+                                               metadata !"fpexcept.strict") #0
+  %cmp5 = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %ret, float %val5,
+                                               metadata !"fpexcept.strict") #0
+  %cmp6 = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %ret, float %val6,
+                                               metadata !"fpexcept.strict") #0
+  %cmp7 = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %ret, float %val7,
+                                               metadata !"fpexcept.strict") #0
+  %cmp8 = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %ret, float %val8,
+                                               metadata !"fpexcept.strict") #0
+  %cmp9 = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %ret, float %val9,
+                                               metadata !"fpexcept.strict") #0
+  %cmp10 = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %ret, float %val10,
+                                               metadata !"fpexcept.strict") #0
+
+  %sel0 = select i1 %cmp0, float %ret, float 0.0
+  %sel1 = select i1 %cmp1, float %sel0, float 1.0
+  %sel2 = select i1 %cmp2, float %sel1, float 2.0
+  %sel3 = select i1 %cmp3, float %sel2, float 3.0
+  %sel4 = select i1 %cmp4, float %sel3, float 4.0
+  %sel5 = select i1 %cmp5, float %sel4, float 5.0
+  %sel6 = select i1 %cmp6, float %sel5, float 6.0
+  %sel7 = select i1 %cmp7, float %sel6, float 7.0
+  %sel8 = select i1 %cmp8, float %sel7, float 8.0
+  %sel9 = select i1 %cmp9, float %sel8, float 9.0
+  %sel10 = select i1 %cmp10, float %sel9, float 10.0
+
+  ret float %sel10
+}
+
+; Check comparison with zero.
+define i64 @f8(i64 %a, i64 %b, float %f) #0 {
+; CHECK-LABEL: f8:
+; CHECK: ltebr %f0, %f0
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f32(
+                                               float %f, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check the comparison can be reversed if that allows CEB to be used,
+; first with oeq.
+define i64 @f9(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f9:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; ...then one.
+define i64 @f10(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f10:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: blhr %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrnlh %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpone.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; ...then olt.
+define i64 @f11(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f11:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: bhr %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrnh %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; ...then ole.
+define i64 @f12(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f12:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: bher %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrnhe %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpole.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; ...then oge.
+define i64 @f13(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f13:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: bler %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrnle %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoge.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; ...then ogt.
+define i64 @f14(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f14:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: blr %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrnl %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpogt.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; ...then ueq.
+define i64 @f15(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f15:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: bnlhr %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrlh %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpueq.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; ...then une.
+define i64 @f16(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f16:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: bner %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgre %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpune.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; ...then ult.
+define i64 @f17(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f17:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: bnler %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrle %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpult.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; ...then ule.
+define i64 @f18(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f18:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: bnlr %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrl %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpule.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; ...then uge.
+define i64 @f19(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f19:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: bnhr %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrh %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpuge.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; ...then ugt.
+define i64 @f20(i64 %a, i64 %b, float %f2, float *%ptr) #0 {
+; CHECK-LABEL: f20:
+; CHECK: ceb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: bnher %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrhe %r2, %r3
+; CHECK: br %r14
+  %f1 = load float, float *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpugt.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+attributes #0 = { strictfp }
+
+declare i1 @llvm.experimental.constrained.fcmpoeq.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpone.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpogt.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpoge.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpolt.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpole.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpueq.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpune.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpugt.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpuge.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpult.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpule.f32(float, float, metadata)

test/CodeGen/SystemZ/fp-strict-cmp-02.ll

+; Test 64-bit floating-point comparison.  The tests assume a z10 implementation
+; of select, using conditional branches rather than LOCGR.
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z10 \
+; RUN:   | FileCheck -check-prefix=CHECK -check-prefix=CHECK-SCALAR %s
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z13 -verify-machineinstrs\
+; RUN:   | FileCheck -check-prefix=CHECK -check-prefix=CHECK-VECTOR %s
+
+declare double @foo()
+
+; Check comparison with registers.
+define i64 @f1(i64 %a, i64 %b, double %f1, double %f2) #0 {
+; CHECK-LABEL: f1:
+; CHECK: cdbr %f0, %f2
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f64(
+                                               double %f1, double %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check the low end of the CDB range.
+define i64 @f2(i64 %a, i64 %b, double %f1, double *%ptr) #0 {
+; CHECK-LABEL: f2:
+; CHECK: cdb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %f2 = load double, double *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f64(
+                                               double %f1, double %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check the high end of the aligned CDB range.
+define i64 @f3(i64 %a, i64 %b, double %f1, double *%base) #0 {
+; CHECK-LABEL: f3:
+; CHECK: cdb %f0, 4088(%r4)
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %ptr = getelementptr double, double *%base, i64 511
+  %f2 = load double, double *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f64(
+                                               double %f1, double %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check the next doubleword up, which needs separate address logic.
+; Other sequences besides this one would be OK.
+define i64 @f4(i64 %a, i64 %b, double %f1, double *%base) #0 {
+; CHECK-LABEL: f4:
+; CHECK: aghi %r4, 4096
+; CHECK: cdb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %ptr = getelementptr double, double *%base, i64 512
+  %f2 = load double, double *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f64(
+                                               double %f1, double %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check negative displacements, which also need separate address logic.
+define i64 @f5(i64 %a, i64 %b, double %f1, double *%base) #0 {
+; CHECK-LABEL: f5:
+; CHECK: aghi %r4, -8
+; CHECK: cdb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %ptr = getelementptr double, double *%base, i64 -1
+  %f2 = load double, double *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f64(
+                                               double %f1, double %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check that CDB allows indices.
+define i64 @f6(i64 %a, i64 %b, double %f1, double *%base, i64 %index) #0 {
+; CHECK-LABEL: f6:
+; CHECK: sllg %r1, %r5, 3
+; CHECK: cdb %f0, 800(%r1,%r4)
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %ptr1 = getelementptr double, double *%base, i64 %index
+  %ptr2 = getelementptr double, double *%ptr1, i64 100
+  %f2 = load double, double *%ptr2
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f64(
+                                               double %f1, double %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check that comparisons of spilled values can use CDB rather than CDBR.
+define double @f7(double *%ptr0) #0 {
+; CHECK-LABEL: f7:
+; CHECK: brasl %r14, foo@PLT
+; CHECK-SCALAR: cdb {{%f[0-9]+}}, 160(%r15)
+; CHECK: br %r14
+  %ptr1 = getelementptr double, double *%ptr0, i64 2
+  %ptr2 = getelementptr double, double *%ptr0, i64 4
+  %ptr3 = getelementptr double, double *%ptr0, i64 6
+  %ptr4 = getelementptr double, double *%ptr0, i64 8
+  %ptr5 = getelementptr double, double *%ptr0, i64 10
+  %ptr6 = getelementptr double, double *%ptr0, i64 12
+  %ptr7 = getelementptr double, double *%ptr0, i64 14
+  %ptr8 = getelementptr double, double *%ptr0, i64 16
+  %ptr9 = getelementptr double, double *%ptr0, i64 18
+  %ptr10 = getelementptr double, double *%ptr0, i64 20
+
+  %val0 = load double, double *%ptr0
+  %val1 = load double, double *%ptr1
+  %val2 = load double, double *%ptr2
+  %val3 = load double, double *%ptr3
+  %val4 = load double, double *%ptr4
+  %val5 = load double, double *%ptr5
+  %val6 = load double, double *%ptr6
+  %val7 = load double, double *%ptr7
+  %val8 = load double, double *%ptr8
+  %val9 = load double, double *%ptr9
+  %val10 = load double, double *%ptr10
+
+  %ret = call double @foo() #0
+
+  %cmp0 = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %ret, double %val0,
+                                               metadata !"fpexcept.strict") #0
+  %cmp1 = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %ret, double %val1,
+                                               metadata !"fpexcept.strict") #0
+  %cmp2 = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %ret, double %val2,
+                                               metadata !"fpexcept.strict") #0
+  %cmp3 = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %ret, double %val3,
+                                               metadata !"fpexcept.strict") #0
+  %cmp4 = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %ret, double %val4,
+                                               metadata !"fpexcept.strict") #0
+  %cmp5 = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %ret, double %val5,
+                                               metadata !"fpexcept.strict") #0
+  %cmp6 = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %ret, double %val6,
+                                               metadata !"fpexcept.strict") #0
+  %cmp7 = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %ret, double %val7,
+                                               metadata !"fpexcept.strict") #0
+  %cmp8 = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %ret, double %val8,
+                                               metadata !"fpexcept.strict") #0
+  %cmp9 = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %ret, double %val9,
+                                               metadata !"fpexcept.strict") #0
+  %cmp10 = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %ret, double %val10,
+                                               metadata !"fpexcept.strict") #0
+
+  %sel0 = select i1 %cmp0, double %ret, double 0.0
+  %sel1 = select i1 %cmp1, double %sel0, double 1.0
+  %sel2 = select i1 %cmp2, double %sel1, double 2.0
+  %sel3 = select i1 %cmp3, double %sel2, double 3.0
+  %sel4 = select i1 %cmp4, double %sel3, double 4.0
+  %sel5 = select i1 %cmp5, double %sel4, double 5.0
+  %sel6 = select i1 %cmp6, double %sel5, double 6.0
+  %sel7 = select i1 %cmp7, double %sel6, double 7.0
+  %sel8 = select i1 %cmp8, double %sel7, double 8.0
+  %sel9 = select i1 %cmp9, double %sel8, double 9.0
+  %sel10 = select i1 %cmp10, double %sel9, double 10.0
+
+  ret double %sel10
+}
+
+; Check comparison with zero.
+define i64 @f8(i64 %a, i64 %b, double %f) #0 {
+; CHECK-LABEL: f8:
+; CHECK-SCALAR: ltdbr %f0, %f0
+; CHECK-SCALAR-NEXT: ber %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR: ltdbr %f0, %f0
+; CHECK-VECTOR-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f64(
+                                               double %f, double 0.0,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check the comparison can be reversed if that allows CDB to be used,
+define i64 @f9(i64 %a, i64 %b, double %f2, double *%ptr) #0 {
+; CHECK-LABEL: f9:
+; CHECK: cdb %f0, 0(%r4)
+; CHECK-SCALAR-NEXT: blr %r14
+; CHECK-SCALAR: lgr %r2, %r3
+; CHECK-VECTOR-NEXT: locgrnl %r2, %r3
+; CHECK: br %r14
+  %f1 = load double, double *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpogt.f64(
+                                               double %f1, double %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+attributes #0 = { strictfp }
+
+declare i1 @llvm.experimental.constrained.fcmpoeq.f64(double, double, metadata)
+declare i1 @llvm.experimental.constrained.fcmpolt.f64(double, double, metadata)
+declare i1 @llvm.experimental.constrained.fcmpogt.f64(double, double, metadata)
+

test/CodeGen/SystemZ/fp-strict-cmp-03.ll

+; Test 128-bit floating-point comparison.  The tests assume a z10 implementation
+; of select, using conditional branches rather than LOCGR.
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z10 | FileCheck %s
+
+; There is no memory form of 128-bit comparison.
+define i64 @f1(i64 %a, i64 %b, fp128 *%ptr, float %f2) #0 {
+; CHECK-LABEL: f1:
+; CHECK-DAG: lxebr %f0, %f0
+; CHECK-DAG: ld %f1, 0(%r4)
+; CHECK-DAG: ld %f3, 8(%r4)
+; CHECK: cxbr %f1, %f0
+; CHECK-NEXT: ber %r14
+; CHECK: lgr %r2, %r3
+; CHECK: br %r14
+  %f2x = fpext float %f2 to fp128
+  %f1 = load fp128, fp128 *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f128(
+                                               fp128 %f1, fp128 %f2x,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check comparison with zero.
+define i64 @f2(i64 %a, i64 %b, fp128 *%ptr) #0 {
+; CHECK-LABEL: f2:
+; CHECK: ld %f0, 0(%r4)
+; CHECK: ld %f2, 8(%r4)
+; CHECK: ltxbr %f0, %f0
+; CHECK-NEXT: ber %r14
+; CHECK: lgr %r2, %r3
+; CHECK: br %r14
+  %f = load fp128, fp128 *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f128(
+                                               fp128 %f, fp128 0xL00000000000000000000000000000000,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+attributes #0 = { strictfp }
+
+declare i1 @llvm.experimental.constrained.fcmpoeq.f128(fp128, fp128, metadata)
+

test/CodeGen/SystemZ/fp-strict-cmp-04.ll

+; Test that floating-point compares are omitted if CC already has the
+; right value.
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z10 -no-integrated-as | FileCheck %s
+
+declare float @llvm.fabs.f32(float %f)
+
+; Test addition followed by EQ, which can use the CC result of the addition.
+define float @f1(float %a, float %b, float *%dest) #0 {
+; CHECK-LABEL: f1:
+; CHECK: aebr %f0, %f2
+; CHECK-NEXT: ber %r14
+; CHECK: br %r14
+entry:
+  %res = call float @llvm.experimental.constrained.fadd.f32(
+                        float %a, float %b,
+                        metadata !"round.dynamic",
+                        metadata !"fpexcept.strict") #0
+  %cmp = call i1 @llvm.experimental.constrained.fcmpoeq.f32(
+                                               float %res, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %b, float *%dest
+  br label %exit
+
+exit:
+  ret float %res
+}
+
+; ...and again with LT.
+define float @f2(float %a, float %b, float *%dest) #0 {
+; CHECK-LABEL: f2:
+; CHECK: aebr %f0, %f2
+; CHECK-NEXT: blr %r14
+; CHECK: br %r14
+entry:
+  %res = call float @llvm.experimental.constrained.fadd.f32(
+                        float %a, float %b,
+                        metadata !"round.dynamic",
+                        metadata !"fpexcept.strict") #0
+  %cmp = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %res, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %b, float *%dest
+  br label %exit
+
+exit:
+  ret float %res
+}
+
+; ...and again with GT.
+define float @f3(float %a, float %b, float *%dest) #0 {
+; CHECK-LABEL: f3:
+; CHECK: aebr %f0, %f2
+; CHECK-NEXT: bhr %r14
+; CHECK: br %r14
+entry:
+  %res = call float @llvm.experimental.constrained.fadd.f32(
+                        float %a, float %b,
+                        metadata !"round.dynamic",
+                        metadata !"fpexcept.strict") #0
+  %cmp = call i1 @llvm.experimental.constrained.fcmpogt.f32(
+                                               float %res, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %b, float *%dest
+  br label %exit
+
+exit:
+  ret float %res
+}
+
+; ...and again with UEQ.
+define float @f4(float %a, float %b, float *%dest) #0 {
+; CHECK-LABEL: f4:
+; CHECK: aebr %f0, %f2
+; CHECK-NEXT: bnlhr %r14
+; CHECK: br %r14
+entry:
+  %res = call float @llvm.experimental.constrained.fadd.f32(
+                        float %a, float %b,
+                        metadata !"round.dynamic",
+                        metadata !"fpexcept.strict") #0
+  %cmp = call i1 @llvm.experimental.constrained.fcmpueq.f32(
+                                               float %res, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %b, float *%dest
+  br label %exit
+
+exit:
+  ret float %res
+}
+
+; Subtraction also provides a zero-based CC value.
+define float @f5(float %a, float %b, float *%dest) {
+; CHECK-LABEL: f5:
+; CHECK: seb %f0, 0(%r2)
+; CHECK-NEXT: bnher %r14
+; CHECK: br %r14
+entry:
+  %cur = load float, float *%dest
+  %res = call float @llvm.experimental.constrained.fsub.f32(
+                        float %a, float %cur,
+                        metadata !"round.dynamic",
+                        metadata !"fpexcept.strict") #0
+  %cmp = call i1 @llvm.experimental.constrained.fcmpult.f32(
+                                               float %res, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %b, float *%dest
+  br label %exit
+
+exit:
+  ret float %res
+}
+
+; Test the result of LOAD POSITIVE.
+define float @f6(float %dummy, float %a, float *%dest) #0 {
+; CHECK-LABEL: f6:
+; CHECK: lpebr %f0, %f2
+; CHECK-NEXT: bhr %r14
+; CHECK: br %r14
+entry:
+  %res = call float @llvm.fabs.f32(float %a)
+  %cmp = call i1 @llvm.experimental.constrained.fcmpogt.f32(
+                                               float %res, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %res, float *%dest
+  br label %exit
+
+exit:
+  ret float %res
+}
+
+; Test the result of LOAD NEGATIVE.
+define float @f7(float %dummy, float %a, float *%dest) #0 {
+; CHECK-LABEL: f7:
+; CHECK: lnebr %f0, %f2
+; CHECK-NEXT: blr %r14
+; CHECK: br %r14
+entry:
+  %abs = call float @llvm.fabs.f32(float %a)
+  %res = fsub float -0.0, %abs
+  %cmp = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %res, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %res, float *%dest
+  br label %exit
+
+exit:
+  ret float %res
+}
+
+; Test the result of LOAD COMPLEMENT.
+define float @f8(float %dummy, float %a, float *%dest) #0 {
+; CHECK-LABEL: f8:
+; CHECK: lcebr %f0, %f2
+; CHECK-NEXT: bler %r14
+; CHECK: br %r14
+entry:
+  %res = fsub float -0.0, %a
+  %cmp = call i1 @llvm.experimental.constrained.fcmpole.f32(
+                                               float %res, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %res, float *%dest
+  br label %exit
+
+exit:
+  ret float %res
+}
+
+; Multiplication (for example) does not modify CC.
+define float @f9(float %a, float %b, float *%dest) #0 {
+; CHECK-LABEL: f9:
+; CHECK: meebr %f0, %f2
+; CHECK-NEXT: ltebr %f0, %f0
+; CHECK-NEXT: blhr %r14
+; CHECK: br %r14
+entry:
+  %res = call float @llvm.experimental.constrained.fmul.f32(
+                        float %a, float %b,
+                        metadata !"round.dynamic",
+                        metadata !"fpexcept.strict") #0
+  %cmp = call i1 @llvm.experimental.constrained.fcmpone.f32(
+                                               float %res, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %b, float *%dest
+  br label %exit
+
+exit:
+  ret float %res
+}
+
+; Test a combination involving a CC-setting instruction followed by
+; a non-CC-setting instruction.
+define float @f10(float %a, float %b, float %c, float *%dest) #0 {
+; CHECK-LABEL: f10:
+; CHECK: aebr %f0, %f2
+; CHECK-NEXT: debr %f0, %f4
+; CHECK-NEXT: ltebr %f0, %f0
+; CHECK-NEXT: bner %r14
+; CHECK: br %r14
+entry:
+  %add = call float @llvm.experimental.constrained.fadd.f32(
+                        float %a, float %b,
+                        metadata !"round.dynamic",
+                        metadata !"fpexcept.strict") #0
+  %res = call float @llvm.experimental.constrained.fdiv.f32(
+                        float %add, float %c,
+                        metadata !"round.dynamic",
+                        metadata !"fpexcept.strict") #0
+  %cmp = call i1 @llvm.experimental.constrained.fcmpune.f32(
+                                               float %res, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %b, float *%dest
+  br label %exit
+
+exit:
+  ret float %res
+}
+
+; Test a case where CC is set based on a different register from the
+; compare input.
+define float @f11(float %a, float %b, float %c, float *%dest1, float *%dest2) #0 {
+; CHECK-LABEL: f11:
+; CHECK: aebr %f0, %f2
+; CHECK-NEXT: sebr %f4, %f0
+; CHECK-DAG: ste %f4, 0(%r2)
+; CHECK-DAG: ltebr %f0, %f0
+; CHECK-NEXT: ber %r14
+; CHECK: br %r14
+entry:
+  %add = call float @llvm.experimental.constrained.fadd.f32(
+                        float %a, float %b,
+                        metadata !"round.dynamic",
+                        metadata !"fpexcept.strict") #0
+  %sub = call float @llvm.experimental.constrained.fsub.f32(
+                        float %c, float %add,
+                        metadata !"round.dynamic",
+                        metadata !"fpexcept.strict") #0
+  store float %sub, float *%dest1
+  %cmp = call i1 @llvm.experimental.constrained.fcmpoeq.f32(
+                                               float %add, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %sub, float *%dest2
+  br label %exit
+
+exit:
+  ret float %add
+}
+
+; Test that LER gets converted to LTEBR where useful.
+define float @f12(float %dummy, float %val, float *%dest) #0 {
+; CHECK-LABEL: f12:
+; CHECK: ltebr %f0, %f2
+; CHECK-NEXT: #APP
+; CHECK-NEXT: blah %f0
+; CHECK-NEXT: #NO_APP
+; CHECK-NEXT: blr %r14
+; CHECK: br %r14
+entry:
+  call void asm sideeffect "blah $0", "{f0}"(float %val)
+  %cmp = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %val, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %val, float *%dest
+  br label %exit
+
+exit:
+  ret float %val
+}
+
+; Test that LDR gets converted to LTDBR where useful.
+define double @f13(double %dummy, double %val, double *%dest) #0 {
+; CHECK-LABEL: f13:
+; CHECK: ltdbr %f0, %f2
+; CHECK-NEXT: #APP
+; CHECK-NEXT: blah %f0
+; CHECK-NEXT: #NO_APP
+; CHECK-NEXT: blr %r14
+; CHECK: br %r14
+entry:
+  call void asm sideeffect "blah $0", "{f0}"(double %val)
+  %cmp = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %val, double 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store double %val, double *%dest
+  br label %exit
+
+exit:
+  ret double %val
+}
+
+; Test that LXR gets converted to LTXBR where useful.
+define void @f14(fp128 *%ptr1, fp128 *%ptr2) #0 {
+; CHECK-LABEL: f14:
+; CHECK: ltxbr
+; CHECK-NEXT: dxbr
+; CHECK-NEXT: std
+; CHECK-NEXT: std
+; CHECK-NEXT: mxbr
+; CHECK-NEXT: std
+; CHECK-NEXT: std
+; CHECK-NEXT: blr %r14
+; CHECK: br %r14
+entry:
+  %val1 = load fp128, fp128 *%ptr1
+  %val2 = load fp128, fp128 *%ptr2
+  %div = fdiv fp128 %val1, %val2
+  store fp128 %div, fp128 *%ptr1
+  %mul = fmul fp128 %val1, %val2
+  store fp128 %mul, fp128 *%ptr2
+  %cmp = call i1 @llvm.experimental.constrained.fcmpolt.f128(
+                                               fp128 %val1, fp128 0xL00000000000000000000000000000000,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  call void asm sideeffect "blah", ""()
+  br label %exit
+
+exit:
+  ret void
+}
+
+; Test a case where it is the source rather than destination of LER that
+; we need.
+define float @f15(float %val, float %dummy, float *%dest) #0 {
+; CHECK-LABEL: f15:
+; CHECK: ltebr %f2, %f0
+; CHECK-NEXT: #APP
+; CHECK-NEXT: blah %f2
+; CHECK-NEXT: #NO_APP
+; CHECK-NEXT: blr %r14
+; CHECK: br %r14
+entry:
+  call void asm sideeffect "blah $0", "{f2}"(float %val)
+  %cmp = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %val, float 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %val, float *%dest
+  br label %exit
+
+exit:
+  ret float %val
+}
+
+; Test a case where it is the source rather than destination of LDR that
+; we need.
+define double @f16(double %val, double %dummy, double *%dest) #0 {
+; CHECK-LABEL: f16:
+; CHECK: ltdbr %f2, %f0
+; CHECK-NEXT: #APP
+; CHECK-NEXT: blah %f2
+; CHECK-NEXT: #NO_APP
+; CHECK-NEXT: blr %r14
+; CHECK: br %r14
+entry:
+  call void asm sideeffect "blah $0", "{f2}"(double %val)
+  %cmp = call i1 @llvm.experimental.constrained.fcmpolt.f64(
+                                               double %val, double 0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store double %val, double *%dest
+  br label %exit
+
+exit:
+  ret double %val
+}
+
+; Repeat f2 with a comparison against -0.
+define float @f17(float %a, float %b, float *%dest) #0 {
+; CHECK-LABEL: f17:
+; CHECK: aebr %f0, %f2
+; CHECK-NEXT: blr %r14
+; CHECK: br %r14
+entry:
+  %res = call float @llvm.experimental.constrained.fadd.f32(
+                        float %a, float %b,
+                        metadata !"round.dynamic",
+                        metadata !"fpexcept.strict") #0
+  %cmp = call i1 @llvm.experimental.constrained.fcmpolt.f32(
+                                               float %res, float -0.0,
+                                               metadata !"fpexcept.strict") #0
+  br i1 %cmp, label %exit, label %store
+
+store:
+  store float %b, float *%dest
+  br label %exit
+
+exit:
+  ret float %res
+}
+
+attributes #0 = { strictfp }
+
+declare float @llvm.experimental.constrained.fadd.f32(float, float, metadata, metadata)
+declare float @llvm.experimental.constrained.fsub.f32(float, float, metadata, metadata)
+declare float @llvm.experimental.constrained.fmul.f32(float, float, metadata, metadata)
+declare float @llvm.experimental.constrained.fdiv.f32(float, float, metadata, metadata)
+declare i1 @llvm.experimental.constrained.fcmpoeq.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpone.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpolt.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpogt.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpole.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpueq.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpune.f32(float, float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpult.f32(float, float, metadata)
+
+declare i1 @llvm.experimental.constrained.fcmpolt.f64(double, double, metadata)
+declare i1 @llvm.experimental.constrained.fcmpolt.f128(fp128, fp128, metadata)

test/CodeGen/SystemZ/fp-strict-cmp-06.ll

+; Test f128 comparisons on z14.
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14 | FileCheck %s
+
+; There is no memory form of 128-bit comparison.
+define i64 @f1(i64 %a, i64 %b, fp128 *%ptr1, fp128 *%ptr2) #0 {
+; CHECK-LABEL: f1:
+; CHECK-DAG: vl [[REG1:%v[0-9]+]], 0(%r4)
+; CHECK-DAG: vl [[REG2:%v[0-9]+]], 0(%r5)
+; CHECK: wfcxb [[REG1]], [[REG2]]
+; CHECK-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %f1 = load fp128, fp128 *%ptr1
+  %f2 = load fp128, fp128 *%ptr2
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f128(
+                                               fp128 %f1, fp128 %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+; Check comparison with zero -- it is not worthwhile to copy to
+; FP pairs just so we can use LTXBR, so simply load up a zero.
+define i64 @f2(i64 %a, i64 %b, fp128 *%ptr) #0 {
+; CHECK-LABEL: f2:
+; CHECK-DAG: vl [[REG1:%v[0-9]+]], 0(%r4)
+; CHECK-DAG: vzero [[REG2:%v[0-9]+]]
+; CHECK: wfcxb [[REG1]], [[REG2]]
+; CHECK-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %f = load fp128, fp128 *%ptr
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f128(
+                                               fp128 %f, fp128 0xL00000000000000000000000000000000,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+attributes #0 = { strictfp }
+
+declare i1 @llvm.experimental.constrained.fcmpoeq.f128(fp128, fp128, metadata)
+

test/CodeGen/SystemZ/vec-strict-cmp-05.ll

+; Test strict v4f32 comparisons.
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z13 | FileCheck %s
+
+; Test oeq.
+define <4 x i32> @f1(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f1:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfcedb [[HIGHRES:%v[0-9]+]], [[HIGH0D]], [[HIGH1D]]
+; CHECK-DAG: vfcedb [[LOWRES:%v[0-9]+]], [[LOW0D]], [[LOW1D]]
+; CHECK: vpkg %v24, [[HIGHRES]], [[LOWRES]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpoeq.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test one.
+define <4 x i32> @f2(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f2:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchdb [[HIGHRES0:%v[0-9]+]], [[HIGH0D]], [[HIGH1D]]
+; CHECK-DAG: vfchdb [[LOWRES0:%v[0-9]+]], [[LOW0D]], [[LOW1D]]
+; CHECK-DAG: vfchdb [[HIGHRES1:%v[0-9]+]], [[HIGH1D]], [[HIGH0D]]
+; CHECK-DAG: vfchdb [[LOWRES1:%v[0-9]+]], [[LOW1D]], [[LOW0D]]
+; CHECK-DAG: vpkg [[RES0:%v[0-9]+]], [[HIGHRES0]], [[LOWRES0]]
+; CHECK-DAG: vpkg [[RES1:%v[0-9]+]], [[HIGHRES1]], [[LOWRES1]]
+; CHECK: vo %v24, [[RES1]], [[RES0]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpone.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ogt.
+define <4 x i32> @f3(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f3:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchdb [[HIGHRES:%v[0-9]+]], [[HIGH0D]], [[HIGH1D]]
+; CHECK-DAG: vfchdb [[LOWRES:%v[0-9]+]], [[LOW0D]], [[LOW1D]]
+; CHECK: vpkg %v24, [[HIGHRES]], [[LOWRES]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpogt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test oge.
+define <4 x i32> @f4(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f4:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchedb [[HIGHRES:%v[0-9]+]], [[HIGH0D]], [[HIGH1D]]
+; CHECK-DAG: vfchedb [[LOWRES:%v[0-9]+]], [[LOW0D]], [[LOW1D]]
+; CHECK: vpkg %v24, [[HIGHRES]], [[LOWRES]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpoge.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ole.
+define <4 x i32> @f5(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f5:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchedb [[HIGHRES:%v[0-9]+]], [[HIGH1D]], [[HIGH0D]]
+; CHECK-DAG: vfchedb [[LOWRES:%v[0-9]+]], [[LOW1D]], [[LOW0D]]
+; CHECK: vpkg %v24, [[HIGHRES]], [[LOWRES]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpole.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test olt.
+define <4 x i32> @f6(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f6:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchdb [[HIGHRES:%v[0-9]+]], [[HIGH1D]], [[HIGH0D]]
+; CHECK-DAG: vfchdb [[LOWRES:%v[0-9]+]], [[LOW1D]], [[LOW0D]]
+; CHECK: vpkg %v24, [[HIGHRES]], [[LOWRES]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpolt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ueq.
+define <4 x i32> @f7(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f7:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchdb [[HIGHRES0:%v[0-9]+]], [[HIGH0D]], [[HIGH1D]]
+; CHECK-DAG: vfchdb [[LOWRES0:%v[0-9]+]], [[LOW0D]], [[LOW1D]]
+; CHECK-DAG: vfchdb [[HIGHRES1:%v[0-9]+]], [[HIGH1D]], [[HIGH0D]]
+; CHECK-DAG: vfchdb [[LOWRES1:%v[0-9]+]], [[LOW1D]], [[LOW0D]]
+; CHECK-DAG: vpkg [[RES0:%v[0-9]+]], [[HIGHRES0]], [[LOWRES0]]
+; CHECK-DAG: vpkg [[RES1:%v[0-9]+]], [[HIGHRES1]], [[LOWRES1]]
+; CHECK: vno %v24, [[RES1]], [[RES0]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpueq.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test une.
+define <4 x i32> @f8(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f8:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfcedb [[HIGHRES:%v[0-9]+]], [[HIGH0D]], [[HIGH1D]]
+; CHECK-DAG: vfcedb [[LOWRES:%v[0-9]+]], [[LOW0D]], [[LOW1D]]
+; CHECK: vpkg [[RES:%v[0-9]+]], [[HIGHRES]], [[LOWRES]]
+; CHECK-NEXT: vno %v24, [[RES]], [[RES]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpune.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ugt.
+define <4 x i32> @f9(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f9:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchedb [[HIGHRES:%v[0-9]+]], [[HIGH1D]], [[HIGH0D]]
+; CHECK-DAG: vfchedb [[LOWRES:%v[0-9]+]], [[LOW1D]], [[LOW0D]]
+; CHECK: vpkg [[RES:%v[0-9]+]], [[HIGHRES]], [[LOWRES]]
+; CHECK-NEXT: vno %v24, [[RES]], [[RES]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpugt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test uge.
+define <4 x i32> @f10(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f10:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchdb [[HIGHRES:%v[0-9]+]], [[HIGH1D]], [[HIGH0D]]
+; CHECK-DAG: vfchdb [[LOWRES:%v[0-9]+]], [[LOW1D]], [[LOW0D]]
+; CHECK: vpkg [[RES:%v[0-9]+]], [[HIGHRES]], [[LOWRES]]
+; CHECK-NEXT: vno %v24, [[RES]], [[RES]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpuge.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ule.
+define <4 x i32> @f11(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f11:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchdb [[HIGHRES:%v[0-9]+]], [[HIGH0D]], [[HIGH1D]]
+; CHECK-DAG: vfchdb [[LOWRES:%v[0-9]+]], [[LOW0D]], [[LOW1D]]
+; CHECK: vpkg [[RES:%v[0-9]+]], [[HIGHRES]], [[LOWRES]]
+; CHECK-NEXT: vno %v24, [[RES]], [[RES]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpule.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ult.
+define <4 x i32> @f12(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f12:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchedb [[HIGHRES:%v[0-9]+]], [[HIGH0D]], [[HIGH1D]]
+; CHECK-DAG: vfchedb [[LOWRES:%v[0-9]+]], [[LOW0D]], [[LOW1D]]
+; CHECK: vpkg [[RES:%v[0-9]+]], [[HIGHRES]], [[LOWRES]]
+; CHECK-NEXT: vno %v24, [[RES]], [[RES]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpult.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ord.
+define <4 x i32> @f13(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f13:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchedb [[HIGHRES0:%v[0-9]+]], [[HIGH0D]], [[HIGH1D]]
+; CHECK-DAG: vfchedb [[LOWRES0:%v[0-9]+]], [[LOW0D]], [[LOW1D]]
+; CHECK-DAG: vfchdb [[HIGHRES1:%v[0-9]+]], [[HIGH1D]], [[HIGH0D]]
+; CHECK-DAG: vfchdb [[LOWRES1:%v[0-9]+]], [[LOW1D]], [[LOW0D]]
+; CHECK-DAG: vpkg [[RES0:%v[0-9]+]], [[HIGHRES0]], [[LOWRES0]]
+; CHECK-DAG: vpkg [[RES1:%v[0-9]+]], [[HIGHRES1]], [[LOWRES1]]
+; CHECK: vo %v24, [[RES1]], [[RES0]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpord.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test uno.
+define <4 x i32> @f14(<4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f14:
+; CHECK-DAG: vmrhf [[HIGH0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrlf [[LOW0E:%v[0-9]+]], %v24, %v24
+; CHECK-DAG: vmrhf [[HIGH1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vmrlf [[LOW1E:%v[0-9]+]], %v26, %v26
+; CHECK-DAG: vldeb [[HIGH0D:%v[0-9]+]], [[HIGH0E]]
+; CHECK-DAG: vldeb [[HIGH1D:%v[0-9]+]], [[HIGH1E]]
+; CHECK-DAG: vldeb [[LOW0D:%v[0-9]+]], [[LOW0E]]
+; CHECK-DAG: vldeb [[LOW1D:%v[0-9]+]], [[LOW1E]]
+; CHECK-DAG: vfchedb [[HIGHRES0:%v[0-9]+]], [[HIGH0D]], [[HIGH1D]]
+; CHECK-DAG: vfchedb [[LOWRES0:%v[0-9]+]], [[LOW0D]], [[LOW1D]]
+; CHECK-DAG: vfchdb [[HIGHRES1:%v[0-9]+]], [[HIGH1D]], [[HIGH0D]]
+; CHECK-DAG: vfchdb [[LOWRES1:%v[0-9]+]], [[LOW1D]], [[LOW0D]]
+; CHECK-DAG: vpkg [[RES0:%v[0-9]+]], [[HIGHRES0]], [[LOWRES0]]
+; CHECK-DAG: vpkg [[RES1:%v[0-9]+]], [[HIGHRES1]], [[LOWRES1]]
+; CHECK: vno %v24, [[RES1]], [[RES0]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpuno.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test oeq selects.
+define <4 x float> @f15(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f15:
+; CHECK: vpkg [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpoeq.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test one selects.
+define <4 x float> @f16(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f16:
+; CHECK: vo [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpone.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ogt selects.
+define <4 x float> @f17(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f17:
+; CHECK: vpkg [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpogt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test oge selects.
+define <4 x float> @f18(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f18:
+; CHECK: vpkg [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpoge.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ole selects.
+define <4 x float> @f19(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f19:
+; CHECK: vpkg [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpole.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test olt selects.
+define <4 x float> @f20(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f20:
+; CHECK: vpkg [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpolt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ueq selects.
+define <4 x float> @f21(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f21:
+; CHECK: vo [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpueq.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test une selects.
+define <4 x float> @f22(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f22:
+; CHECK: vpkg [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpune.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ugt selects.
+define <4 x float> @f23(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f23:
+; CHECK: vpkg [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpugt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test uge selects.
+define <4 x float> @f24(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f24:
+; CHECK: vpkg [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpuge.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ule selects.
+define <4 x float> @f25(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f25:
+; CHECK: vpkg [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpule.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ult selects.
+define <4 x float> @f26(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f26:
+; CHECK: vpkg [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpult.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ord selects.
+define <4 x float> @f27(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f27:
+; CHECK: vo [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpord.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test uno selects.
+define <4 x float> @f28(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f28:
+; CHECK: vo [[REG:%v[0-9]+]],
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpuno.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+attributes #0 = { strictfp }
+
+declare <4 x i1> @llvm.experimental.constrained.fcmpoeq.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpone.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpogt.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpoge.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpolt.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpole.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpueq.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpune.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpugt.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpuge.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpult.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpule.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpord.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpuno.v4f32(<4 x float>, <4 x float>, metadata)
+

test/CodeGen/SystemZ/vec-strict-cmp-06.ll

+; Test f64 and v2f64 strict comparisons.
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z13 | FileCheck %s
+
+; Test oeq.
+define <2 x i64> @f1(<2 x i64> %dummy, <2 x double> %val1, <2 x double> %val2) #0 {
+; CHECK-LABEL: f1:
+; CHECK: vfcedb %v24, %v26, %v28
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpoeq.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test one.
+define <2 x i64> @f2(<2 x i64> %dummy, <2 x double> %val1, <2 x double> %val2) #0 {
+; CHECK-LABEL: f2:
+; CHECK-DAG: vfchdb [[REG1:%v[0-9]+]], %v28, %v26
+; CHECK-DAG: vfchdb [[REG2:%v[0-9]+]], %v26, %v28
+; CHECK: vo %v24, [[REG1]], [[REG2]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpone.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test ogt.
+define <2 x i64> @f3(<2 x i64> %dummy, <2 x double> %val1, <2 x double> %val2) #0 {
+; CHECK-LABEL: f3:
+; CHECK: vfchdb %v24, %v26, %v28
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpogt.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test oge.
+define <2 x i64> @f4(<2 x i64> %dummy, <2 x double> %val1, <2 x double> %val2) #0 {
+; CHECK-LABEL: f4:
+; CHECK: vfchedb %v24, %v26, %v28
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpoge.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test ole.
+define <2 x i64> @f5(<2 x i64> %dummy, <2 x double> %val1, <2 x double> %val2) #0 {
+; CHECK-LABEL: f5:
+; CHECK: vfchedb %v24, %v28, %v26
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpole.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test olt.
+define <2 x i64> @f6(<2 x i64> %dummy, <2 x double> %val1, <2 x double> %val2) #0 {
+; CHECK-LABEL: f6:
+; CHECK: vfchdb %v24, %v28, %v26
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpolt.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test ueq.
+define <2 x i64> @f7(<2 x i64> %dummy, <2 x double> %val1, <2 x double> %val2) #0 {
+; CHECK-LABEL: f7:
+; CHECK-DAG: vfchdb [[REG1:%v[0-9]+]], %v28, %v26
+; CHECK-DAG: vfchdb [[REG2:%v[0-9]+]], %v26, %v28
+; CHECK: vno %v24, [[REG1]], [[REG2]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpueq.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test une.
+define <2 x i64> @f8(<2 x i64> %dummy, <2 x double> %val1, <2 x double> %val2) #0 {
+; CHECK-LABEL: f8:
+; CHECK: vfcedb [[REG:%v[0-9]+]], %v26, %v28
+; CHECK-NEXT: vno %v24, [[REG]], [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpune.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test ugt.
+define <2 x i64> @f9(<2 x i64> %dummy, <2 x double> %val1, <2 x double> %val2) #0 {
+; CHECK-LABEL: f9:
+; CHECK: vfchedb [[REG:%v[0-9]+]], %v28, %v26
+; CHECK-NEXT: vno %v24, [[REG]], [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpugt.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test uge.
+define <2 x i64> @f10(<2 x i64> %dummy, <2 x double> %val1,
+                      <2 x double> %val2) #0 {
+; CHECK-LABEL: f10:
+; CHECK: vfchdb [[REG:%v[0-9]+]], %v28, %v26
+; CHECK-NEXT: vno %v24, [[REG]], [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpuge.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test ule.
+define <2 x i64> @f11(<2 x i64> %dummy, <2 x double> %val1,
+                      <2 x double> %val2) #0 {
+; CHECK-LABEL: f11:
+; CHECK: vfchdb [[REG:%v[0-9]+]], %v26, %v28
+; CHECK-NEXT: vno %v24, [[REG]], [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpule.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test ult.
+define <2 x i64> @f12(<2 x i64> %dummy, <2 x double> %val1,
+                      <2 x double> %val2) #0 {
+; CHECK-LABEL: f12:
+; CHECK: vfchedb [[REG:%v[0-9]+]], %v26, %v28
+; CHECK-NEXT: vno %v24, [[REG]], [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpult.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test ord.
+define <2 x i64> @f13(<2 x i64> %dummy, <2 x double> %val1,
+                      <2 x double> %val2) #0 {
+; CHECK-LABEL: f13:
+; CHECK-DAG: vfchdb [[REG1:%v[0-9]+]], %v28, %v26
+; CHECK-DAG: vfchedb [[REG2:%v[0-9]+]], %v26, %v28
+; CHECK: vo %v24, [[REG1]], [[REG2]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpord.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test uno.
+define <2 x i64> @f14(<2 x i64> %dummy, <2 x double> %val1,
+                      <2 x double> %val2) #0 {
+; CHECK-LABEL: f14:
+; CHECK-DAG: vfchdb [[REG1:%v[0-9]+]], %v28, %v26
+; CHECK-DAG: vfchedb [[REG2:%v[0-9]+]], %v26, %v28
+; CHECK: vno %v24, [[REG1]], [[REG2]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpuno.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <2 x i1> %cmp to <2 x i64>
+  ret <2 x i64> %ret
+}
+
+; Test oeq selects.
+define <2 x double> @f15(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f15:
+; CHECK: vfcedb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpoeq.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test one selects.
+define <2 x double> @f16(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f16:
+; CHECK-DAG: vfchdb [[REG1:%v[0-9]+]], %v26, %v24
+; CHECK-DAG: vfchdb [[REG2:%v[0-9]+]], %v24, %v26
+; CHECK: vo [[REG:%v[0-9]+]], [[REG1]], [[REG2]]
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpone.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test ogt selects.
+define <2 x double> @f17(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f17:
+; CHECK: vfchdb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpogt.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test oge selects.
+define <2 x double> @f18(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f18:
+; CHECK: vfchedb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpoge.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test ole selects.
+define <2 x double> @f19(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f19:
+; CHECK: vfchedb [[REG:%v[0-9]+]], %v26, %v24
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpole.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test olt selects.
+define <2 x double> @f20(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f20:
+; CHECK: vfchdb [[REG:%v[0-9]+]], %v26, %v24
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpolt.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test ueq selects.
+define <2 x double> @f21(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f21:
+; CHECK-DAG: vfchdb [[REG1:%v[0-9]+]], %v26, %v24
+; CHECK-DAG: vfchdb [[REG2:%v[0-9]+]], %v24, %v26
+; CHECK: vo [[REG:%v[0-9]+]], [[REG1]], [[REG2]]
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpueq.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test une selects.
+define <2 x double> @f22(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f22:
+; CHECK: vfcedb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpune.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test ugt selects.
+define <2 x double> @f23(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f23:
+; CHECK: vfchedb [[REG:%v[0-9]+]], %v26, %v24
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpugt.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test uge selects.
+define <2 x double> @f24(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f24:
+; CHECK: vfchdb [[REG:%v[0-9]+]], %v26, %v24
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpuge.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test ule selects.
+define <2 x double> @f25(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f25:
+; CHECK: vfchdb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpule.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test ult selects.
+define <2 x double> @f26(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f26:
+; CHECK: vfchedb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpult.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test ord selects.
+define <2 x double> @f27(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f27:
+; CHECK-DAG: vfchdb [[REG1:%v[0-9]+]], %v26, %v24
+; CHECK-DAG: vfchedb [[REG2:%v[0-9]+]], %v24, %v26
+; CHECK: vo [[REG:%v[0-9]+]], [[REG1]], [[REG2]]
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpord.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test uno selects.
+define <2 x double> @f28(<2 x double> %val1, <2 x double> %val2,
+                         <2 x double> %val3, <2 x double> %val4) #0 {
+; CHECK-LABEL: f28:
+; CHECK-DAG: vfchdb [[REG1:%v[0-9]+]], %v26, %v24
+; CHECK-DAG: vfchedb [[REG2:%v[0-9]+]], %v24, %v26
+; CHECK: vo [[REG:%v[0-9]+]], [[REG1]], [[REG2]]
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <2 x i1> @llvm.experimental.constrained.fcmpuno.v2f64(
+                                               <2 x double> %val1, <2 x double> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <2 x i1> %cmp, <2 x double> %val3, <2 x double> %val4
+  ret <2 x double> %ret
+}
+
+; Test an f64 comparison that uses vector registers.
+define i64 @f29(i64 %a, i64 %b, double %f1, <2 x double> %vec) #0 {
+; CHECK-LABEL: f29:
+; CHECK: wfcdb %f0, %v24
+; CHECK-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %f2 = extractelement <2 x double> %vec, i32 0
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f64(
+                                               double %f1, double %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+attributes #0 = { strictfp }
+
+declare <2 x i1> @llvm.experimental.constrained.fcmpoeq.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpone.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpogt.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpoge.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpolt.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpole.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpueq.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpune.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpugt.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpuge.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpult.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpule.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpord.v2f64(<2 x double>, <2 x double>, metadata)
+declare <2 x i1> @llvm.experimental.constrained.fcmpuno.v2f64(<2 x double>, <2 x double>, metadata)
+
+declare i1 @llvm.experimental.constrained.fcmpoeq.f64(double, double, metadata)
+

test/CodeGen/SystemZ/vec-strict-cmp-07.ll

+; Test strict f32 and v4f32 comparisons on z14.
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14 | FileCheck %s
+
+; Test oeq.
+define <4 x i32> @f1(<4 x i32> %dummy, <4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f1:
+; CHECK: vfcesb %v24, %v26, %v28
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpoeq.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test one.
+define <4 x i32> @f2(<4 x i32> %dummy, <4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f2:
+; CHECK-DAG: vfchsb [[REG1:%v[0-9]+]], %v28, %v26
+; CHECK-DAG: vfchsb [[REG2:%v[0-9]+]], %v26, %v28
+; CHECK: vo %v24, [[REG1]], [[REG2]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpone.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ogt.
+define <4 x i32> @f3(<4 x i32> %dummy, <4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f3:
+; CHECK: vfchsb %v24, %v26, %v28
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpogt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test oge.
+define <4 x i32> @f4(<4 x i32> %dummy, <4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f4:
+; CHECK: vfchesb %v24, %v26, %v28
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpoge.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ole.
+define <4 x i32> @f5(<4 x i32> %dummy, <4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f5:
+; CHECK: vfchesb %v24, %v28, %v26
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpole.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test olt.
+define <4 x i32> @f6(<4 x i32> %dummy, <4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f6:
+; CHECK: vfchsb %v24, %v28, %v26
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpolt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ueq.
+define <4 x i32> @f7(<4 x i32> %dummy, <4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f7:
+; CHECK-DAG: vfchsb [[REG1:%v[0-9]+]], %v28, %v26
+; CHECK-DAG: vfchsb [[REG2:%v[0-9]+]], %v26, %v28
+; CHECK: vno %v24, [[REG1]], [[REG2]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpueq.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test une.
+define <4 x i32> @f8(<4 x i32> %dummy, <4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f8:
+; CHECK: vfcesb [[REG:%v[0-9]+]], %v26, %v28
+; CHECK-NEXT: vno %v24, [[REG]], [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpune.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ugt.
+define <4 x i32> @f9(<4 x i32> %dummy, <4 x float> %val1, <4 x float> %val2) #0 {
+; CHECK-LABEL: f9:
+; CHECK: vfchesb [[REG:%v[0-9]+]], %v28, %v26
+; CHECK-NEXT: vno %v24, [[REG]], [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpugt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test uge.
+define <4 x i32> @f10(<4 x i32> %dummy, <4 x float> %val1,
+                      <4 x float> %val2) #0 {
+; CHECK-LABEL: f10:
+; CHECK: vfchsb [[REG:%v[0-9]+]], %v28, %v26
+; CHECK-NEXT: vno %v24, [[REG]], [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpuge.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ule.
+define <4 x i32> @f11(<4 x i32> %dummy, <4 x float> %val1,
+                      <4 x float> %val2) #0 {
+; CHECK-LABEL: f11:
+; CHECK: vfchsb [[REG:%v[0-9]+]], %v26, %v28
+; CHECK-NEXT: vno %v24, [[REG]], [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpule.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ult.
+define <4 x i32> @f12(<4 x i32> %dummy, <4 x float> %val1,
+                      <4 x float> %val2) #0 {
+; CHECK-LABEL: f12:
+; CHECK: vfchesb [[REG:%v[0-9]+]], %v26, %v28
+; CHECK-NEXT: vno %v24, [[REG]], [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpult.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test ord.
+define <4 x i32> @f13(<4 x i32> %dummy, <4 x float> %val1,
+                      <4 x float> %val2) #0 {
+; CHECK-LABEL: f13:
+; CHECK-DAG: vfchsb [[REG1:%v[0-9]+]], %v28, %v26
+; CHECK-DAG: vfchesb [[REG2:%v[0-9]+]], %v26, %v28
+; CHECK: vo %v24, [[REG1]], [[REG2]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpord.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test uno.
+define <4 x i32> @f14(<4 x i32> %dummy, <4 x float> %val1,
+                      <4 x float> %val2) #0 {
+; CHECK-LABEL: f14:
+; CHECK-DAG: vfchsb [[REG1:%v[0-9]+]], %v28, %v26
+; CHECK-DAG: vfchesb [[REG2:%v[0-9]+]], %v26, %v28
+; CHECK: vno %v24, [[REG1]], [[REG2]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpuno.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = sext <4 x i1> %cmp to <4 x i32>
+  ret <4 x i32> %ret
+}
+
+; Test oeq selects.
+define <4 x float> @f15(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f15:
+; CHECK: vfcesb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpoeq.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test one selects.
+define <4 x float> @f16(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f16:
+; CHECK-DAG: vfchsb [[REG1:%v[0-9]+]], %v26, %v24
+; CHECK-DAG: vfchsb [[REG2:%v[0-9]+]], %v24, %v26
+; CHECK: vo [[REG:%v[0-9]+]], [[REG1]], [[REG2]]
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpone.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ogt selects.
+define <4 x float> @f17(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f17:
+; CHECK: vfchsb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpogt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test oge selects.
+define <4 x float> @f18(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f18:
+; CHECK: vfchesb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpoge.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ole selects.
+define <4 x float> @f19(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f19:
+; CHECK: vfchesb [[REG:%v[0-9]+]], %v26, %v24
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpole.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test olt selects.
+define <4 x float> @f20(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f20:
+; CHECK: vfchsb [[REG:%v[0-9]+]], %v26, %v24
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpolt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ueq selects.
+define <4 x float> @f21(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f21:
+; CHECK-DAG: vfchsb [[REG1:%v[0-9]+]], %v26, %v24
+; CHECK-DAG: vfchsb [[REG2:%v[0-9]+]], %v24, %v26
+; CHECK: vo [[REG:%v[0-9]+]], [[REG1]], [[REG2]]
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpueq.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test une selects.
+define <4 x float> @f22(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f22:
+; CHECK: vfcesb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpune.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ugt selects.
+define <4 x float> @f23(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f23:
+; CHECK: vfchesb [[REG:%v[0-9]+]], %v26, %v24
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpugt.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test uge selects.
+define <4 x float> @f24(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f24:
+; CHECK: vfchsb [[REG:%v[0-9]+]], %v26, %v24
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpuge.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ule selects.
+define <4 x float> @f25(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f25:
+; CHECK: vfchsb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpule.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ult selects.
+define <4 x float> @f26(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f26:
+; CHECK: vfchesb [[REG:%v[0-9]+]], %v24, %v26
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpult.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test ord selects.
+define <4 x float> @f27(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f27:
+; CHECK-DAG: vfchsb [[REG1:%v[0-9]+]], %v26, %v24
+; CHECK-DAG: vfchesb [[REG2:%v[0-9]+]], %v24, %v26
+; CHECK: vo [[REG:%v[0-9]+]], [[REG1]], [[REG2]]
+; CHECK-NEXT: vsel %v24, %v28, %v30, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpord.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test uno selects.
+define <4 x float> @f28(<4 x float> %val1, <4 x float> %val2,
+                        <4 x float> %val3, <4 x float> %val4) #0 {
+; CHECK-LABEL: f28:
+; CHECK-DAG: vfchsb [[REG1:%v[0-9]+]], %v26, %v24
+; CHECK-DAG: vfchesb [[REG2:%v[0-9]+]], %v24, %v26
+; CHECK: vo [[REG:%v[0-9]+]], [[REG1]], [[REG2]]
+; CHECK-NEXT: vsel %v24, %v30, %v28, [[REG]]
+; CHECK-NEXT: br %r14
+  %cmp = call <4 x i1> @llvm.experimental.constrained.fcmpuno.v4f32(
+                                               <4 x float> %val1, <4 x float> %val2,
+                                               metadata !"fpexcept.strict") #0
+  %ret = select <4 x i1> %cmp, <4 x float> %val3, <4 x float> %val4
+  ret <4 x float> %ret
+}
+
+; Test an f32 comparison that uses vector registers.
+define i64 @f29(i64 %a, i64 %b, float %f1, <4 x float> %vec) #0 {
+; CHECK-LABEL: f29:
+; CHECK: wfcsb %f0, %v24
+; CHECK-NEXT: locgrne %r2, %r3
+; CHECK: br %r14
+  %f2 = extractelement <4 x float> %vec, i32 0
+  %cond = call i1 @llvm.experimental.constrained.fcmpoeq.f32(
+                                               float %f1, float %f2,
+                                               metadata !"fpexcept.strict") #0
+  %res = select i1 %cond, i64 %a, i64 %b
+  ret i64 %res
+}
+
+attributes #0 = { strictfp }
+
+declare <4 x i1> @llvm.experimental.constrained.fcmpoeq.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpone.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpogt.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpoge.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpolt.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpole.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpueq.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpune.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpugt.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpuge.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpult.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpule.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpord.v4f32(<4 x float>, <4 x float>, metadata)
+declare <4 x i1> @llvm.experimental.constrained.fcmpuno.v4f32(<4 x float>, <4 x float>, metadata)
+
+declare i1 @llvm.experimental.constrained.fcmpoeq.f32(float, float, metadata)
+

test/CodeGen/X86/fp-intrinsics.ll

 ; COMMON: jmp llroundf
 define i64 @f30(float %x) #0 {
 entry:
   %result = call i64 @llvm.experimental.constrained.llround.i64.f32(float %x,
                                                metadata !"fpexcept.strict") #0
   ret i64 %result
 }
 
+; CHECK-LABEL: f31
+; COMMON: ucomisd
+define i1 @f31() {
+entry:
+  %cmp = call i1 @llvm.experimental.constrained.fcmpueq.f64(
+                                               double 1.000000e+00,
+                                               double 1.000000e+01,
+                                               metadata !"fpexcept.strict") #0
+  ret i1 %cmp
+}
+
+
 attributes #0 = { strictfp }
 
 @llvm.fp.env = thread_local global i8 zeroinitializer, section "llvm.metadata"
 declare double @llvm.experimental.constrained.fadd.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fsub.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fmul.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fdiv.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.frem.f64(double, double, metadata, metadata)

 declare i32 @llvm.experimental.constrained.lrint.i32.f64(double, metadata, metadata)
 declare i32 @llvm.experimental.constrained.lrint.i32.f32(float, metadata, metadata)
 declare i64 @llvm.experimental.constrained.llrint.i64.f64(double, metadata, metadata)
 declare i64 @llvm.experimental.constrained.llrint.i64.f32(float, metadata, metadata)
 declare i32 @llvm.experimental.constrained.lround.i32.f64(double, metadata)
 declare i32 @llvm.experimental.constrained.lround.i32.f32(float, metadata)
 declare i64 @llvm.experimental.constrained.llround.i64.f64(double, metadata)
 declare i64 @llvm.experimental.constrained.llround.i64.f32(float, metadata)
+declare i1 @llvm.experimental.constrained.fcmpueq.f64(double, double, metadata)

test/CodeGen/X86/vector-constrained-fp-intrinsics-cmp.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -O3 -mtriple=x86_64-pc-linux < %s | FileCheck %s
+
+define <2 x i1> @constrained_vector_fcmpueq_v2f64() {
+; CHECK-LABEL: constrained_vector_fcmpueq_v2f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
+; CHECK-NEXT:    movsd {{.*#+}} xmm1 = mem[0],zero
+; CHECK-NEXT:    ucomisd %xmm0, %xmm1
+; CHECK-NEXT:    movq $-1, %rcx
+; CHECK-NEXT:    movl $0, %edx
+; CHECK-NEXT:    cmoveq %rcx, %rdx
+; CHECK-NEXT:    movq %rdx, %xmm2
+; CHECK-NEXT:    ucomisd %xmm1, %xmm0
+; CHECK-NEXT:    cmoveq %rcx, %rax
+; CHECK-NEXT:    movq %rax, %xmm0
+; CHECK-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
+; CHECK-NEXT:    retq
+entry:
+  %fcmp = call <2 x i1> @llvm.experimental.constrained.fcmpueq.v2f64(
+           <2 x double> <double 1.0, double 2.0>,
+           <2 x double> <double 2.0, double 1.0>,
+           metadata !"fpexcept.strict")
+  ret <2 x i1> %fcmp
+}
+
+; Single width declarations
+declare <2 x i1> @llvm.experimental.constrained.fcmpueq.v2f64(<2 x double>, <2 x double>, metadata)
+