[FPEnv] Rough out constrained FCmp intrinsics

[cameron.mcinally]

Summary

This is a continuation of the abandoned D54121...

Here's a big patch for the constrained FCmp intrinsics. There's a lot here, so I did not include the full set of constrained FCmp intrinsics or tests yet. I intend to add those in a separate patch.

These intrinsics will require some auxiliary constructs in TableGen, which make up the bulk of this code. Those are:

• any_bool_ty
• LLVMScalarOrVectorSameWidth

I'll also point out some things that I'm unsure about inline. Hopefully others are more familiar with that code.

[cameron.mcinally]

include/llvm/CodeGen/ValueTypes.td

153

I'm not sure if it's safe to reorder these values. I had to cluster these entries together since there's a Duff's device in utils/TableGen/IntrinsicEmitter.cpp that depends on it.

Unfortunately, the new bAny entry could not be added to the end of the *Any defs list, since the last value is 255 and the container is an 8-bit unsigned int.

include/llvm/IR/Intrinsics.td

167

This should probably be more strict. We could end up with something like <2 <2 x double>> as an argument. There's some C++ code later that should catch that case, but we could probably come up with a better class to extend here.

include/llvm/Support/MachineValueType.h

219

This is the same value reordering mentioned in the comment for include/llvm/CodeGen/ValueTypes.td.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

1226

Has anyone given any thought to how we will Custom lower Strict nodes? This is punting too soon, but I wanted to get the code out there to review before the changeset became too large.

[nhaehnle]

include/llvm/IR/Intrinsics.td

167

Either way, it needs a comment that explains what it does :)

[nhaehnle]

include/llvm/IR/Intrinsics.td

167

I should say that my first thought at the name was that it's about bit width, which it's not, so maybe the name could be improved as well? And the num parameter doesn't currently go anywhere, which means that this can't work. The class should instead derive from LLVMMatchType<num>.

[uweigand]

In general, I think going forward the back-ends should be explicitly aware of the STRICT_ FP nodes. Anyway, that's the direction my proposed SystemZ patch takes: https://reviews.llvm.org/D45576

While the above patch currently only forwards Legal STRICT_ nodes to the back-end, it would only be logical to also forward Custom STRICT_ nodes, so that the target can handle them as necessary.

So I'm not sure why you're adding special code to handle Custom STRICT_ nodes in common code -- that seems to somewhat contradict the very notion of Custom. I believe if the target does mark any STRICT_ node as Custom, it expects to handle those for itself.

(A different issue may be where a STRICT_ node is marked as Expand and therefore translated to a non-strict node -- but that non-strict node was already marked Custom by the target. We may have to handle that case specially somehow, I'm not sure ...)

[cameron.mcinally]

Update TableGen changes, as suggested by @nhaehnle.

Also, renamed LLVMScalarOrVectorSameWidth to LLVMVectorSameLengthOrScalar. Is that easier to parse? [Note that I previously copied "Width" from existing code, but it should really be "Length".]

[cameron.mcinally]

@uweigand, what about something like this patch? The STRICT_FSETCC isn't handled all the way through the target, but it's stubbed out for now.

The bulk of this patch is getting any_bool_ty and LLVMVectorSameLengthOrScalar constructs in place. There's still a number of constrained FCMP intrinsics to add later, so I was thinking about adding the target support with those.

[uweigand]

In D54649#1310341, @cameron.mcinally wrote:

@uweigand, what about something like this patch? The STRICT_FSETCC isn't handled all the way through the target, but it's stubbed out for now.

See inline comments. Otherwise, I think this makes sense ...

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

81

Just to clarify: the reason why we need to handle STRICT_FSETCC here but not any of the other STRICT_ opcodes is that only FSETCC involves an integer type (as return type), right?

lib/Target/X86/X86ISelLowering.cpp

26106

Why do you have this here? It seems that this can never be called right now (at least without something like my D45576, right?)

[cameron.mcinally]

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

81

That's correct.

lib/Target/X86/X86ISelLowering.cpp

26106

It actually will be called. Some SETCCs are always custom lowered on x86, even for scalars.

The other constrained vector operations are all legal or expanded to legal scalar operations. SETCC is the first operation that needs the special lowering.

Just to be pedantic, I do realize this code is punting right now. The TableGen changes in this patch are quite involved. So I'd like for us to focus on those first. I will handle the STRICT_FSETCC peculiarities in a future patch.

[uweigand]

lib/Target/X86/X86ISelLowering.cpp

26106

But still, I don't see any place where common code would even call the back-end's getOperationAction on a STRICT_FSETCC.

E.g. when calling getStrictFPOperationAction with a STRICT_FSETCC, it will actually call the back-end with SETCC (not STRICT_FSETCC). So this should already return Custom for you ...

[cameron.mcinally]

lib/Target/X86/X86ISelLowering.cpp

26106

But still, I don't see any place where common code would even call the back-end's getOperationAction on a STRICT_FSETCC.

I agree with this statement. We call getStrictFPOperationAction(...) which will return Custom from SETCC (for the STRICT_FSETCC node).

E.g. when calling getStrictFPOperationAction with a STRICT_FSETCC, it will actually call the back-end with SETCC (not STRICT_FSETCC). So this should already return Custom for you ...

I also agree with this, but when the backend returns Custom for SETCC, we have to then Custom lower the STRICT_FSETCC node. We do not mutate the STRICT_FSETCC node into SETCC node in LegalizeOp(), where we call getStrictFPOperationAction().

I'm not sure I fully understand your point though (although I may admittedly be making a fundamental error). If you comment out this line in the code, you'll see the backtrace leading up to the problem.

[uweigand]

lib/Target/X86/X86ISelLowering.cpp

26106

Ah OK, now I see it.

I guess what confused me most is that you added the setOperationAction statement for STRICT_FSETCC further up. That statement really is the one that doesn't seem to have any effect.

This statement does have effect, it's just a bit weird that the reason why LowerOperation is called with a STRICT_FSETCC node is that setOperationAction for SETCC (not STRICT_FSETCC!) is Custom.

Taking a step back, I think the original thought behind the getStrictFPOperationAction logic was that if the target doesn't do anything special, all strict FP operations will just transparently work as if they were regular FP operations. But that is not (any longer?) the case if the regular FP operation is marked as Custom: there will have to be target support, or else it doesn't work at all.

The question is, is it worth trying to preseve that original thought? I guess one way to do that would be for common code to handle STRICT_ nodes where the corresponding regular node is Custom by mutating the node before calling LowerOperation on the regular node. (Later, when we add support for the target to actually handle STRICT_ nodes directly, we'd first check whether the STRICT_ node itself is marked Custom, and then pass it directly to the back-end.)

[cameron.mcinally]

lib/Target/X86/X86ISelLowering.cpp

26106

I guess what confused me most is that you added the setOperationAction statement for STRICT_FSETCC further up. That statement really is the one that doesn't seem to have any effect.

You're right. I'll correct it now. Sorry about that.

Taking a step back, I think the original thought behind the getStrictFPOperationAction logic was that if the target doesn't do anything special, all strict FP operations will just transparently work as if they were regular FP operations. But that is not (any longer?) the case if the regular FP operation is marked as Custom: there will have to be target support, or else it doesn't work at all.

Yes. This is the first StrictFP node that always needs to be Custom lowered. The others were general enough to not cause any problems.

The question is, is it worth trying to preseve that original thought? I guess one way to do that would be for common code to handle STRICT_ nodes where the corresponding regular node is Custom by mutating the node before calling LowerOperation on the regular node. (Later, when we add support for the target to actually handle STRICT_ nodes directly, we'd first check whether the STRICT_ node itself is marked Custom, and then pass it directly to the back-end.)

That was what I originally did with the patch, but then changed it to this. :D

I don't know if there's a huge difference between the two implementations. Both are weird (and really not right). They mutate the StrictFP node too soon.

In any case, this problem originated from not having backend support for strict nodes in place. So either solution is temporary. When we have backend support for strict nodes, this code will go away.

[cameron.mcinally]

Remove setting of STRICT_FSETCC Custom lowering, as @uweigand suggested.

[uweigand]

lib/Target/X86/X86ISelLowering.cpp

26106

OK, thanks for the explanation. I'm not really sure if I'd prefer this current implementation over your original one, but as you say, they're really both wrong anyway, so I wouldn't want to hold things up any further over this.

[cameron.mcinally]

lib/Target/X86/X86ISelLowering.cpp

26106

I still have the old patch, so can resurrect it if needed.

There is one drawback from that old patch though. It really kludged up the generic code for TargetLowering::Custom in LegalizeOp(). That code is built around a switch fall through and mutating the StrictFP node was awkward.

That said, the one obvious strength of that old patch is the Custom StrictFP node lowering is pretty general. If/when more Custom lowered StrictFP nodes are seen, they should just do the right thing without additional effort.

I'll post a snippet of that patch so you can see what I mean...

[cameron.mcinally]

Old patch:

Index: lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
===================================================================
--- lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
+++ lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
@@ -1198,30 +1199,36 @@
     case TargetLowering::Legal:
       LLVM_DEBUG(dbgs() << "Legal node: nothing to do\n");
       return;
-    case TargetLowering::Custom:
+    case TargetLowering::Custom: {
       LLVM_DEBUG(dbgs() << "Trying custom legalization\n");
+      SDNode *N = Node;
+
+      if (N->isStrictFPOpcode())
+        N = DAG.mutateStrictFPToFP(Node); 
+
       // FIXME: The handling for custom lowering with multiple results is
       // a complete mess.
-      if (SDValue Res = TLI.LowerOperation(SDValue(Node, 0), DAG)) {
-        if (!(Res.getNode() != Node || Res.getResNo() != 0))
+      if (SDValue Res = TLI.LowerOperation(SDValue(N, 0), DAG)) {
+        if (!(Res.getNode() != N || Res.getResNo() != 0))
           return;
 
-        if (Node->getNumValues() == 1) {
+        if (N->getNumValues() == 1) {
           LLVM_DEBUG(dbgs() << "Successfully custom legalized node\n");
           // We can just directly replace this node with the lowered value.
-          ReplaceNode(SDValue(Node, 0), Res);
+          ReplaceNode(SDValue(N, 0), Res);
           return;
         }
 
         SmallVector<SDValue, 8> ResultVals;
-        for (unsigned i = 0, e = Node->getNumValues(); i != e; ++i)
+        for (unsigned i = 0, e = N->getNumValues(); i != e; ++i)
           ResultVals.push_back(Res.getValue(i));
         LLVM_DEBUG(dbgs() << "Successfully custom legalized node\n");
-        ReplaceNode(Node, ResultVals.data());
+        ReplaceNode(N, ResultVals.data());
         return;
       }

I think I prefer the backend solution since it hides the weirdness in the LowerSTRICT_FSETCC(...) function:

SDValue X86TargetLowering::LowerSTRICT_FSETCC(SDValue Op, SelectionDAG &DAG) const {
  // Mutate away for now...
  Op = SDValue(DAG.mutateStrictFPToFP(Op.getNode()), 0);
  return LowerSETCC(Op, DAG);
}

[uweigand]

The one problem with your "new" code is that it now forces back-ends to implement something, or else code involving constrained intrinsics will trigger internal compiler errors. It might be preferable to avoid those ...

But I also agree that your "old" code is a bit awkward. Ideally, I'd have preferred to see the default handling of strict opcodes in the absence of target support to work just via regular legalization, i.e. remove all the special-cased getStrictFPOperationAction and mutateStrictFPToFP stuff. Instead, the strict opcodes would be treated like everything else, except they all have a default action of Expand, and the associated expand action (in the same place where all the other expand actions are) would simply replace the strict opcode by the corresponding normal opcode.

If that normal opcode then has an operation action of Custom, that should just automatically work I think (since the result of an Expand rule is just normally run through legalization again).

And any target that wants to actually properly handle strict opcodes would simply overwrite the default action by either Legal (if it can directly handle the strict opcode) or Custom (if it wants LowerOperation to be called with the strict opcode to do anything special).

But that's not where we are today, and I'm not sure if it is really worthwhile to rework that code at this point ...

[cameron.mcinally]

In D54649#1318675, @uweigand wrote:

The one problem with your "new" code is that it now forces back-ends to implement something, or else code involving constrained intrinsics will trigger internal compiler errors. It might be preferable to avoid those ...

Ah, I see. This is a problem.

But I also agree that your "old" code is a bit awkward. Ideally, I'd have preferred to see the default handling of strict opcodes in the absence of target support to work just via regular legalization, i.e. remove all the special-cased getStrictFPOperationAction and mutateStrictFPToFP stuff. Instead, the strict opcodes would be treated like everything else, except they all have a default action of Expand, and the associated expand action (in the same place where all the other expand actions are) would simply replace the strict opcode by the corresponding normal opcode.

If that normal opcode then has an operation action of Custom, that should just automatically work I think (since the result of an Expand rule is just normally run through legalization again).

And any target that wants to actually properly handle strict opcodes would simply overwrite the default action by either Legal (if it can directly handle the strict opcode) or Custom (if it wants LowerOperation to be called with the strict opcode to do anything special).

Looking closer, I may have a better solution for the SETCC case. It won't solve the general problem of nodes needing to be Custom lowered though. Will report back soon...

[cameron.mcinally]

Revert back to mutating a temp before Custom lowering.

[cameron.mcinally]

In D54649#1318762, @cameron.mcinally wrote:

Looking closer, I may have a better solution for the SETCC case. It won't solve the general problem of nodes needing to be Custom lowered though. Will report back soon...

This was a dead end...

[cameron.mcinally]

A quiet ping on this, since this is my last working day of the year...

This problem is awkward, so I understand if there is hesitation to move forward. There is a need to get this problem solved for targets that will be slow to adopt the STRICT_ nodes though.

[andrew.w.kaylor]

I'm a bit out of my depth with the tablegen type handling stuff. I have some minor comments regarding naming consistency. Everything looks reasonable to me.

Perhaps it would be worth renaming this review to make clear that it is primarily an extension to the tablegen type handling and only implements part of the strict FCMP stuff in order to have a use for the new type.

include/llvm/IR/Intrinsics.h

104

This should probably be called SameVectorWidthOrScalarArgument for consistency.

include/llvm/IR/Intrinsics.td

608

You have a problem with line wrap here.

lib/IR/Function.cpp

693

IIT_SAME_VEC_WIDTH_OR_SCALAR_ARG

[cameron.mcinally]

include/llvm/IR/Intrinsics.h

104

Ah, so we actually discussed this in one of the first comments. My view is that Vector Width describes the bit-width of the vector. And Vector Length describes the number of elements in the vector. If everyone agrees, then it would be correct as VecLength.

[andrew.w.kaylor]

include/llvm/IR/Intrinsics.h

104

Does 'length' mean the same thing that 'width' means in SameVecWidthArgument on the previous line? I'm happy with either wording as long as they're consistent.

[cameron.mcinally]

include/llvm/IR/Intrinsics.h

104

It does not. E.g. <2 x i32> has a length of 2 and width of 64.

[andrew.w.kaylor]

include/llvm/IR/Intrinsics.h

104

Is that what we want? For instance, would you want to be able to compare a <4 x f32> with a <2 x f64>?

What I was expecting was something that would require the vector to have the same element type and the same number of elements.

[cameron.mcinally]

include/llvm/IR/Intrinsics.h

104

Let's look at a hard example:

def int_experimental_constrained_fcmpeq : Intrinsic<[ llvm_anybool_ty ],
  [ LLVMVectorSameLengthOrScalar<0, llvm_anyfloat_ty>,
    LLVMVectorSameLengthOrScalar<0, llvm_anyfloat_ty>,
    llvm_metadata_ty,
    llvm_metadata_ty ]>;

We want the vector length to match between the <N x i1> return type and the <N x fp> operands. Both input arguments have the same type.

Now that I look at it, the 2nd LLVMVectorSameLengthOrScalar could probably just be a:

LLVMMatchType<1>

[andrew.w.kaylor]

include/llvm/IR/Intrinsics.h

104

Oh, I'm sorry. I read your explanation backward. For some reason I had it in my head that the vector width (128/256 or whatever) was what you were introducing.

Now I understand, and your choice makes perfect sense.

[cameron.mcinally]

include/llvm/IR/Intrinsics.h

104

No sweat.

I'll try to address your other comments in the near future. Some higher priority things have come up on my end. Stay tuned...

[cameron.mcinally]

Bah, the following patch poked holes in this Diff:

------------------------------------------------------------------------
r350421 | spatel | 2019-01-04 11:48:13 -0600 (Fri, 04 Jan 2019) | 18 lines

[x86] lower extracted fadd/fsub to horizontal vector math; 2nd try

The 1st try for this was at rL350369, but it caused IR-level diffs because
our cost models differentiate custom vs. legal/promote lowering. So that was
reverted at rL350373. The cost models were fixed independently at rL350403,
so this is effectively the same patch as last time.

Original commit message:
This would show up if we fix horizontal reductions to narrow as they go along,
but it's an improvement for size and/or Jaguar (fast-hops) independent of that.

We need to do this late to not interfere with other pattern matching of larger
horizontal sequences.

We can extend this to integer ops in a follow-up patch.

Differential Revision: https://reviews.llvm.org/D56011

r350421 is now custom lowering FADD and FSUB, and it breaks the generic Custom lowering changes I made to handle STRICT nodes.

Perhaps we should wait until D55506 has landed, since that would allow us to handle STRICT_FSETCC directly in the backend? I'll dig into this more and post soon...

[cameron.mcinally]

This is just taking up space, so abandoning it. I'll resurrect the useful parts if needed.

[uweigand]

In D54649#1356137, @cameron.mcinally wrote:

r350421 is now custom lowering FADD and FSUB, and it breaks the generic Custom lowering changes I made to handle STRICT nodes.

Perhaps we should wait until D55506 has landed, since that would allow us to handle STRICT_FSETCC directly in the backend? I'll dig into this more and post soon...

Now that this has landed, are you planning to have another look at this?

[cameron.mcinally]

In D54649#1561237, @uweigand wrote:

Now that this has landed, are you planning to have another look at this?

Thanks, Ulrich. I will dig into this.

I'm just wrapping up (I think) some other work with unary FNeg. If someone would like to pick this Diff up in the meantime, I have no objections...

include/llvm/CodeGen/ISDOpcodes.h

enum NodeType {
     /// Constrained versions of libm-equivalent floating point intrinsics.
     /// These will be lowered to the equivalent non-constrained pseudo-op
     /// (or expanded to the equivalent library call) before final selection.
     /// They are used to limit optimizations while the DAG is being optimized.
     STRICT_FSQRT, STRICT_FPOW, STRICT_FPOWI, STRICT_FSIN, STRICT_FCOS,
     STRICT_FEXP, STRICT_FEXP2, STRICT_FLOG, STRICT_FLOG10, STRICT_FLOG2,
     STRICT_FRINT, STRICT_FNEARBYINT, STRICT_FMAXNUM, STRICT_FMINNUM,
     STRICT_FCEIL, STRICT_FFLOOR, STRICT_FROUND, STRICT_FTRUNC,
+    STRICT_FSETCC,
 
     /// FMA - Perform a * b + c with no intermediate rounding step.
     FMA,
 
     /// FMAD - Perform a * b + c, while getting the same result as the
     /// separately rounded operations.
     FMAD,
 

include/llvm/CodeGen/SelectionDAGNodes.h

switch (NodeType) {
       case ISD::STRICT_FRINT:
       case ISD::STRICT_FNEARBYINT:
       case ISD::STRICT_FMAXNUM:
       case ISD::STRICT_FMINNUM:
       case ISD::STRICT_FCEIL:
       case ISD::STRICT_FFLOOR:
       case ISD::STRICT_FROUND:
       case ISD::STRICT_FTRUNC:
+      case ISD::STRICT_FSETCC:
         return true;
     }
   }
 
   /// Test if this node has a post-isel opcode, directly
   /// corresponding to a MachineInstr opcode.
   bool isMachineOpcode() const { return NodeType < 0; }
 

include/llvm/CodeGen/TargetLowering.h

switch (Op) {
       case ISD::STRICT_FRINT: EqOpc = ISD::FRINT; break;
       case ISD::STRICT_FNEARBYINT: EqOpc = ISD::FNEARBYINT; break;
       case ISD::STRICT_FMAXNUM: EqOpc = ISD::FMAXNUM; break;
       case ISD::STRICT_FMINNUM: EqOpc = ISD::FMINNUM; break;
       case ISD::STRICT_FCEIL: EqOpc = ISD::FCEIL; break;
       case ISD::STRICT_FFLOOR: EqOpc = ISD::FFLOOR; break;
       case ISD::STRICT_FROUND: EqOpc = ISD::FROUND; break;
       case ISD::STRICT_FTRUNC: EqOpc = ISD::FTRUNC; break;
+      case ISD::STRICT_FSETCC: EqOpc = ISD::SETCC; break;
     }
 
     auto Action = getOperationAction(EqOpc, VT);
 
-    // We don't currently handle Custom or Promote for strict FP pseudo-ops.
+    // We don't currently handle Promote for strict FP pseudo-ops.
     // For now, we just expand for those cases.
-    if (Action != Legal)
+    if (Action != Legal && Action != Custom)
       Action = Expand;
 
     return Action;
   }
 
   /// Return true if the specified operation is legal on this target or can be
   /// made legal with custom lowering. This is used to help guide high-level
   /// lowering decisions.

include/llvm/CodeGen/ValueTypes.h

public:
 
     /// Return true if this is a 2048-bit vector type.
     bool is2048BitVector() const {
       return isSimple() ? V.is2048BitVector() : isExtended2048BitVector();
     }
 
     /// Return true if this is an overloaded type for TableGen.
     bool isOverloaded() const {
-      return (V==MVT::iAny || V==MVT::fAny || V==MVT::vAny || V==MVT::iPTRAny);
+      return (V==MVT::bAny || V==MVT::iAny || V==MVT::fAny || 
+              V==MVT::vAny || V==MVT::iPTRAny);
     }
 
     /// Return true if the bit size is a multiple of 8.
     bool isByteSized() const {
       return (getSizeInBits() & 7) == 0;
     }
 
     /// Return true if the size is a power-of-two number of bytes.

include/llvm/CodeGen/ValueTypes.td

 def nxv4f64  : ValueType<256, 107>; // n x  4 x f64 vector value
 def nxv8f64  : ValueType<512, 108>; // n x  8 x f64 vector value
 
 def x86mmx : ValueType<64 , 109>;   // X86 MMX value
 def FlagVT : ValueType<0  , 110>;   // Pre-RA sched glue
 def isVoid : ValueType<0  , 111>;   // Produces no value
 def untyped: ValueType<8  , 112>;   // Produces an untyped value
 def ExceptRef: ValueType<0, 113>;   // WebAssembly's except_ref type
-def token  : ValueType<0  , 248>;   // TokenTy
-def MetadataVT: ValueType<0, 249>;  // Metadata
+def token  : ValueType<0  , 247>;   // TokenTy
+def MetadataVT: ValueType<0, 248>;  // Metadata
+
+// Pseudo valuetype to represent "bool of any format"
+def bAny  : ValueType<0  , 249>;
 
 // Pseudo valuetype mapped to the current pointer size to any address space.
 // Should only be used in TableGen.
 def iPTRAny   : ValueType<0, 250>;
 
 // Pseudo valuetype to represent "vector of any size"
 def vAny   : ValueType<0  , 251>;
 

include/llvm/IR/IntrinsicInst.h

static bool classof(const IntrinsicInst *I) {
       case Intrinsic::experimental_constrained_rint:
       case Intrinsic::experimental_constrained_nearbyint:
       case Intrinsic::experimental_constrained_maxnum:
       case Intrinsic::experimental_constrained_minnum:
       case Intrinsic::experimental_constrained_ceil:
       case Intrinsic::experimental_constrained_floor:
       case Intrinsic::experimental_constrained_round:
       case Intrinsic::experimental_constrained_trunc:
+      case Intrinsic::experimental_constrained_fcmpeq:
         return true;
       default: return false;
       }
     }
     static bool classof(const Value *V) {
       return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
     }
   };

include/llvm/IR/Intrinsics.h

#undef GET_INTRINSIC_ENUM_VALUES
 
   /// This is a type descriptor which explains the type requirements of an
   /// intrinsic. This is returned by getIntrinsicInfoTableEntries.
   struct IITDescriptor {
     enum IITDescriptorKind {
       Void, VarArg, MMX, Token, Metadata, Half, Float, Double, Quad,
       Integer, Vector, Pointer, Struct,
       Argument, ExtendArgument, TruncArgument, HalfVecArgument,
-      SameVecWidthArgument, PtrToArgument, PtrToElt, VecOfAnyPtrsToElt
+      SameVecWidthArgument, PtrToArgument, PtrToElt, VecOfAnyPtrsToElt,
+      SameVecLengthOrScalarArgument
     } Kind;
 
     union {
       unsigned Integer_Width;
       unsigned Float_Width;
       unsigned Vector_Width;
       unsigned Pointer_AddressSpace;
       unsigned Struct_NumElements;
       unsigned Argument_Info;
     };
 
     enum ArgKind {
       AK_Any,
       AK_AnyInteger,
       AK_AnyFloat,
       AK_AnyVector,
-      AK_AnyPointer
+      AK_AnyPointer,
+      AK_AnyBoolean
     };
 
     unsigned getArgumentNumber() const {
       assert(Kind == Argument || Kind == ExtendArgument ||
              Kind == TruncArgument || Kind == HalfVecArgument ||
              Kind == SameVecWidthArgument || Kind == PtrToArgument ||
-             Kind == PtrToElt);
+             Kind == PtrToElt || Kind == SameVecLengthOrScalarArgument);
       return Argument_Info >> 3;
     }
     ArgKind getArgumentKind() const {
       assert(Kind == Argument || Kind == ExtendArgument ||
              Kind == TruncArgument || Kind == HalfVecArgument ||
-             Kind == SameVecWidthArgument || Kind == PtrToArgument);
+             Kind == SameVecWidthArgument || Kind == PtrToArgument ||
+             Kind == SameVecLengthOrScalarArgument);
       return (ArgKind)(Argument_Info & 7);
     }
 
     // VecOfAnyPtrsToElt uses both an overloaded argument (for address space)
     // and a reference argument (for matching vector width and element types)
     unsigned getOverloadArgNumber() const {
-      assert(Kind == VecOfAnyPtrsToElt);
+      assert(Kind == VecOfAnyPtrsToElt || Kind == SameVecLengthOrScalarArgument);
       return Argument_Info >> 16;
     }
     unsigned getRefArgNumber() const {
-      assert(Kind == VecOfAnyPtrsToElt);
+      assert(Kind == VecOfAnyPtrsToElt || Kind == SameVecLengthOrScalarArgument);
       return Argument_Info & 0xFFFF;
     }
 
     static IITDescriptor get(IITDescriptorKind K, unsigned Field) {
       IITDescriptor Result = { K, { Field } };
       return Result;
     }
 

include/llvm/IR/Intrinsics.td

 // be twice as wide or half as wide as the other type.  This is only useful when
 // the intrinsic is overloaded, so the matched type should be declared as iAny.
 class LLVMExtendedType<int num> : LLVMMatchType<num>;
 class LLVMTruncatedType<int num> : LLVMMatchType<num>;
 class LLVMVectorSameWidth<int num, LLVMType elty>
   : LLVMMatchType<num> {
   ValueType ElTy = elty.VT;
 }
+
+class LLVMVectorSameLengthOrScalar<int num, LLVMType elty>
+  : LLVMMatchType<num> { 
+  LLVMType ElTy = elty;
+}
+
 class LLVMPointerTo<int num> : LLVMMatchType<num>;
 class LLVMPointerToElt<int num> : LLVMMatchType<num>;
 class LLVMVectorOfAnyPointersToElt<int num> : LLVMMatchType<num>;
 
 // Match the type of another intrinsic parameter that is expected to be a
 // vector type, but change the element count to be half as many
 class LLVMHalfElementsVectorType<int num> : LLVMMatchType<num>;
 
 def llvm_void_ty       : LLVMType<isVoid>;
 let isAny = 1 in {
   def llvm_any_ty        : LLVMType<Any>;
+  def llvm_anybool_ty    : LLVMType<bAny>;
   def llvm_anyint_ty     : LLVMType<iAny>;
   def llvm_anyfloat_ty   : LLVMType<fAny>;
   def llvm_anyvector_ty  : LLVMType<vAny>;
 }
 def llvm_i1_ty         : LLVMType<i1>;
 def llvm_i8_ty         : LLVMType<i8>;
 def llvm_i16_ty        : LLVMType<i16>;
 def llvm_i32_ty        : LLVMType<i32>;
let IntrProperties = [IntrInaccessibleMemOnly] in {
   def int_experimental_constrained_round : Intrinsic<[ llvm_anyfloat_ty ],
                                                      [ LLVMMatchType<0>,
                                                       llvm_metadata_ty,
                                                       llvm_metadata_ty ]>;
   def int_experimental_constrained_trunc : Intrinsic<[ llvm_anyfloat_ty ],
                                                      [ LLVMMatchType<0>,
                                                        llvm_metadata_ty,
                                                        llvm_metadata_ty ]>;
+
+  def int_experimental_constrained_fcmpeq : Intrinsic<[ llvm_anybool_ty ],
+                                                      [ LLVMVectorSameLengthOrScalar<0, llvm_anyfloat_ty>,
+                                                        LLVMVectorSameLengthOrScalar<0, llvm_anyfloat_ty>,
+                                                        llvm_metadata_ty,
+                                                        llvm_metadata_ty ]>;
 }
 // FIXME: Add intrinsics for fcmp, fptrunc, fpext, fptoui and fptosi.
 // FIXME: Add intrinsics for fabs and copysign? 
 
 
 //===------------------------- Expect Intrinsics --------------------------===//
 //
 def int_expect : Intrinsic<[llvm_anyint_ty], [LLVMMatchType<0>,

include/llvm/Support/MachineValueType.h

enum SimpleValueType : uint8_t {
       Other          =   1,   // This is a non-standard value
       i1             =   2,   // This is a 1 bit integer value
       i8             =   3,   // This is an 8 bit integer value
       i16            =   4,   // This is a 16 bit integer value
       i32            =   5,   // This is a 32 bit integer value
       i64            =   6,   // This is a 64 bit integer value
       i128           =   7,   // This is a 128 bit integer value
 
+      FIRST_BOOLEAN_VALUETYPE = i1,
+      LAST_BOOLEAN_VALUETYPE  = i1,
+
       FIRST_INTEGER_VALUETYPE = i1,
       LAST_INTEGER_VALUETYPE  = i128,
 
       f16            =   8,   // This is a 16 bit floating point value
       f32            =   9,   // This is a 32 bit floating point value
       f64            =  10,   // This is a 64 bit floating point value
       f80            =  11,   // This is a 80 bit floating point value
       f128           =  12,   // This is a 128 bit floating point value
enum SimpleValueType : uint8_t {
 
       nxv1i64        =  79,   // n x  1 x i64
       nxv2i64        =  80,   // n x  2 x i64
       nxv4i64        =  81,   // n x  4 x i64
       nxv8i64        =  82,   // n x  8 x i64
       nxv16i64       =  83,   // n x 16 x i64
       nxv32i64       =  84,   // n x 32 x i64
 
+      FIRST_BOOLEAN_VECTOR_VALUETYPE = v1i1,
+      LAST_BOOLEAN_VECTOR_VALUETYPE = v1024i1,
+
       FIRST_INTEGER_VECTOR_VALUETYPE = v1i1,
       LAST_INTEGER_VECTOR_VALUETYPE = nxv32i64,
 
       FIRST_INTEGER_SCALABLE_VALUETYPE = nxv1i1,
       LAST_INTEGER_SCALABLE_VALUETYPE = nxv32i64,
 
       v2f16          =  85,   //  2 x f16
       v4f16          =  86,   //  4 x f16
enum SimpleValueType : uint8_t {
       LAST_VALUETYPE =  114,   // This always remains at the end of the list.
 
       // This is the current maximum for LAST_VALUETYPE.
       // MVT::MAX_ALLOWED_VALUETYPE is used for asserts and to size bit vectors
       // This value must be a multiple of 32.
       MAX_ALLOWED_VALUETYPE = 128,
 
       // A value of type llvm::TokenTy
-      token          = 248,
+      token          = 247,
 
       // This is MDNode or MDString.
-      Metadata       = 249,
+      Metadata       = 248,
+
+      // An i1 or vector i1 value. This is used for intrinsics
+      // that have overloadings based on integer bit widths.
+      // This is only for tblgen's consumption!
+      bAny          = 249,
 
       // An int value the size of the pointer of the current
       // target to any address space. This must only be used internal to
       // tblgen. Other than for overloading, we treat iPTRAny the same as iPTR.
       iPTRAny        = 250,
 
       // A vector with any length and element size. This is used
       // for intrinsics that have overloadings based on vector types.
public:
     /// Return true if this is a 2048-bit vector type.
     bool is2048BitVector() const {
       return (SimpleTy == MVT::v256i8 || SimpleTy == MVT::v128i16 ||
               SimpleTy == MVT::v64i32 || SimpleTy == MVT::v32i64);
     }
 
     /// Return true if this is an overloaded type for TableGen.
     bool isOverloaded() const {
-      return (SimpleTy==MVT::Any  ||
+      return (SimpleTy==MVT::Any  || SimpleTy==MVT::bAny ||
               SimpleTy==MVT::iAny || SimpleTy==MVT::fAny ||
               SimpleTy==MVT::vAny || SimpleTy==MVT::iPTRAny);
     }
 
     /// Returns true if the given vector is a power of 2.
     bool isPow2VectorType() const {
       unsigned NElts = getVectorNumElements();
       return !(NElts & (NElts - 1));
unsigned getSizeInBits() const {
       switch (SimpleTy) {
       default:
         llvm_unreachable("getSizeInBits called on extended MVT.");
       case Other:
         llvm_unreachable("Value type is non-standard value, Other.");
       case iPTR:
         llvm_unreachable("Value type size is target-dependent. Ask TLI.");
       case iPTRAny:
+      case bAny:
       case iAny:
       case fAny:
       case vAny:
       case Any:
         llvm_unreachable("Value type is overloaded.");
       case token:
         llvm_unreachable("Token type is a sentinel that cannot be used "
                          "in codegen and has no size");
namespace llvm {
 
   public:
     /// SimpleValueType Iteration
     /// @{
     static mvt_range all_valuetypes() {
       return mvt_range(MVT::FIRST_VALUETYPE, MVT::LAST_VALUETYPE);
     }
 
+    static mvt_range boolean_valuetypes() {
+      return mvt_range(MVT::FIRST_BOOLEAN_VALUETYPE,
+                       (MVT::SimpleValueType)(MVT::LAST_BOOLEAN_VALUETYPE + 1));
+    }
+
     static mvt_range integer_valuetypes() {
       return mvt_range(MVT::FIRST_INTEGER_VALUETYPE,
                        (MVT::SimpleValueType)(MVT::LAST_INTEGER_VALUETYPE + 1));
     }
 
     static mvt_range fp_valuetypes() {
       return mvt_range(MVT::FIRST_FP_VALUETYPE,
                        (MVT::SimpleValueType)(MVT::LAST_FP_VALUETYPE + 1));
     }
 
     static mvt_range vector_valuetypes() {
       return mvt_range(MVT::FIRST_VECTOR_VALUETYPE,
                        (MVT::SimpleValueType)(MVT::LAST_VECTOR_VALUETYPE + 1));
     }
 
+    static mvt_range boolean_vector_valuetypes() {
+      return mvt_range(
+          MVT::FIRST_BOOLEAN_VECTOR_VALUETYPE,
+          (MVT::SimpleValueType)(MVT::LAST_BOOLEAN_VECTOR_VALUETYPE + 1));
+    }
+
     static mvt_range integer_vector_valuetypes() {
       return mvt_range(
           MVT::FIRST_INTEGER_VECTOR_VALUETYPE,
           (MVT::SimpleValueType)(MVT::LAST_INTEGER_VECTOR_VALUETYPE + 1));
     }
 
     static mvt_range fp_vector_valuetypes() {
       return mvt_range(

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

#endif
   case ISD::STRICT_FRINT:
   case ISD::STRICT_FNEARBYINT:
   case ISD::STRICT_FMAXNUM:
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
+  case ISD::STRICT_FSETCC:
     // These pseudo-ops get legalized as if they were their non-strict
     // equivalent.  For instance, if ISD::FSQRT is legal then ISD::STRICT_FSQRT
     // is also legal, but if ISD::FSQRT requires expansion then so does
     // ISD::STRICT_FSQRT.
     Action = TLI.getStrictFPOperationAction(Node->getOpcode(),
                                             Node->getValueType(0));
     break;
   case ISD::SADDSAT:
case TargetLowering::Custom:
         for (unsigned i = 0, e = Node->getNumValues(); i != e; ++i)
           ResultVals.push_back(Res.getValue(i));
         LLVM_DEBUG(dbgs() << "Successfully custom legalized node\n");
         ReplaceNode(Node, ResultVals.data());
         return;
       }
       LLVM_DEBUG(dbgs() << "Could not custom legalize node\n");
       LLVM_FALLTHROUGH;
     case TargetLowering::Expand:
       if (ExpandNode(Node))
         return;
       LLVM_FALLTHROUGH;
     case TargetLowering::LibCall:
       ConvertNodeToLibcall(Node);
       return;
     case TargetLowering::Promote:
       PromoteNode(Node);

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

#endif
   case ISD::MLOAD:       Res = PromoteIntRes_MLOAD(cast<MaskedLoadSDNode>(N));
     break;
   case ISD::MGATHER:     Res = PromoteIntRes_MGATHER(cast<MaskedGatherSDNode>(N));
     break;
   case ISD::SELECT:      Res = PromoteIntRes_SELECT(N); break;
   case ISD::VSELECT:     Res = PromoteIntRes_VSELECT(N); break;
   case ISD::SELECT_CC:   Res = PromoteIntRes_SELECT_CC(N); break;
   case ISD::SETCC:       Res = PromoteIntRes_SETCC(N); break;
+  case ISD::STRICT_FSETCC: 
+                         Res = PromoteIntRes_StrictFSETCC(N); break;
   case ISD::SMIN:
   case ISD::SMAX:        Res = PromoteIntRes_SExtIntBinOp(N); break;
   case ISD::UMIN:
   case ISD::UMAX:        Res = PromoteIntRes_ZExtIntBinOp(N); break;
 
   case ISD::SHL:         Res = PromoteIntRes_SHL(N); break;
   case ISD::SIGN_EXTEND_INREG:
                          Res = PromoteIntRes_SIGN_EXTEND_INREG(N); break;

 SDValue DAGTypeLegalizer::PromoteIntRes_SELECT_CC(SDNode *N) {
   SDValue LHS = GetPromotedInteger(N->getOperand(2));
   SDValue RHS = GetPromotedInteger(N->getOperand(3));
   return DAG.getNode(ISD::SELECT_CC, SDLoc(N),
                      LHS.getValueType(), N->getOperand(0),
                      N->getOperand(1), LHS, RHS, N->getOperand(4));
 }
 
+SDValue DAGTypeLegalizer::PromoteIntRes_StrictFSETCC(SDNode *N) {
+  // Operand 0 is the Chain.
+  EVT InVT = N->getOperand(1).getValueType();
+  EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
+
+  EVT SVT = getSetCCResultType(InVT);
+
+  // If we got back a type that needs to be promoted, this likely means the
+  // the input type also needs to be promoted. So get the promoted type for
+  // the input and try the query again.
+  if (getTypeAction(SVT) == TargetLowering::TypePromoteInteger) {
+    if (getTypeAction(InVT) == TargetLowering::TypePromoteInteger) {
+      InVT = TLI.getTypeToTransformTo(*DAG.getContext(), InVT);
+      SVT = getSetCCResultType(InVT);
+    } else {
+      // Input type isn't promoted, just use the default promoted type.
+      SVT = NVT;
+    }
+  }
+
+  SDLoc dl(N);
+  assert(SVT.isVector() == N->getOperand(1).getValueType().isVector() &&
+         "Vector compare must return a vector result!");
+
+  // Get the SETCC result using the canonical SETCC type.
+  EVT VTs[] = {SVT, MVT::Other};
+  SDValue Opers[] = {N->getOperand(0), N->getOperand(1), 
+                     N->getOperand(2), N->getOperand(3)};
+  SDValue SetCC = DAG.getNode(N->getOpcode(), dl, VTs, Opers);
+
+  // Convert to the expected type.
+  SDValue Res = DAG.getSExtOrTrunc(SetCC, dl, NVT);
+ 
+  // Legalize the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Res.getValue(1));
+
+  return Res;
+}
+
 SDValue DAGTypeLegalizer::PromoteIntRes_SETCC(SDNode *N) {
   EVT InVT = N->getOperand(0).getValueType();
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
 
   EVT SVT = getSetCCResultType(InVT);
 
   // If we got back a type that needs to be promoted, this likely means the
   // the input type also needs to be promoted. So get the promoted type for

lib/CodeGen/SelectionDAG/LegalizeTypes.h

private:
   SDValue PromoteIntRes_MLOAD(MaskedLoadSDNode *N);
   SDValue PromoteIntRes_MGATHER(MaskedGatherSDNode *N);
   SDValue PromoteIntRes_Overflow(SDNode *N);
   SDValue PromoteIntRes_SADDSUBO(SDNode *N, unsigned ResNo);
   SDValue PromoteIntRes_SELECT(SDNode *N);
   SDValue PromoteIntRes_VSELECT(SDNode *N);
   SDValue PromoteIntRes_SELECT_CC(SDNode *N);
   SDValue PromoteIntRes_SETCC(SDNode *N);
+  SDValue PromoteIntRes_StrictFSETCC(SDNode *N);
   SDValue PromoteIntRes_SHL(SDNode *N);
   SDValue PromoteIntRes_SimpleIntBinOp(SDNode *N);
   SDValue PromoteIntRes_ZExtIntBinOp(SDNode *N);
   SDValue PromoteIntRes_SExtIntBinOp(SDNode *N);
   SDValue PromoteIntRes_SIGN_EXTEND_INREG(SDNode *N);
   SDValue PromoteIntRes_SRA(SDNode *N);
   SDValue PromoteIntRes_SRL(SDNode *N);
   SDValue PromoteIntRes_TRUNCATE(SDNode *N);

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

SDValue VectorLegalizer::LegalizeOp(SDValue Op) {
   case ISD::STRICT_FRINT:
   case ISD::STRICT_FNEARBYINT:
   case ISD::STRICT_FMAXNUM:
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
+  case ISD::STRICT_FSETCC:
     // These pseudo-ops get legalized as if they were their non-strict
     // equivalent.  For instance, if ISD::FSQRT is legal then ISD::STRICT_FSQRT
     // is also legal, but if ISD::FSQRT requires expansion then so does
     // ISD::STRICT_FSQRT.
     Action = TLI.getStrictFPOperationAction(Node->getOpcode(),
                                             Node->getValueType(0));
     break;
   case ISD::ADD:
SDValue VectorLegalizer::Expand(SDValue Op) {
   case ISD::STRICT_FRINT:
   case ISD::STRICT_FNEARBYINT:
   case ISD::STRICT_FMAXNUM:
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
+  case ISD::STRICT_FSETCC:
     return ExpandStrictFPOp(Op);
   default:
     return DAG.UnrollVectorOp(Op.getNode());
   }
 }
 
 SDValue VectorLegalizer::ExpandSELECT(SDValue Op) {
   // Lower a select instruction where the condition is a scalar and the

lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

#endif
   case ISD::STRICT_FRINT:
   case ISD::STRICT_FNEARBYINT:
   case ISD::STRICT_FMAXNUM:
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
+  case ISD::STRICT_FSETCC:
     R = ScalarizeVecRes_StrictFPOp(N);
     break;
   }
 
   // If R is null, the sub-method took care of registering the result.
   if (R.getNode())
     SetScalarizedVector(SDValue(N, ResNo), R);
 }
#endif
   case ISD::STRICT_FRINT:
   case ISD::STRICT_FNEARBYINT:
   case ISD::STRICT_FMAXNUM:
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
+  case ISD::STRICT_FSETCC:
     SplitVecRes_StrictFPOp(N, Lo, Hi);
     break;
   }
 
   // If Lo/Hi is null, the sub-method took care of registering results etc.
   if (Lo.getNode())
     SetSplitVector(SDValue(N, ResNo), Lo, Hi);
 }
#endif
   case ISD::STRICT_FRINT:
   case ISD::STRICT_FNEARBYINT:
   case ISD::STRICT_FMAXNUM:
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
+  case ISD::STRICT_FSETCC:
     Res = WidenVecRes_StrictFP(N);
     break;
 
   case ISD::FCOPYSIGN:
     Res = WidenVecRes_FCOPYSIGN(N);
     break;
 
   case ISD::FPOWI:

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

case ISD::STRICT_FNEARBYINT:
     IsUnary = true;
     break;
   case ISD::STRICT_FMAXNUM: NewOpc = ISD::FMAXNUM; break;
   case ISD::STRICT_FMINNUM: NewOpc = ISD::FMINNUM; break;
   case ISD::STRICT_FCEIL: NewOpc = ISD::FCEIL; IsUnary = true; break;
   case ISD::STRICT_FFLOOR: NewOpc = ISD::FFLOOR; IsUnary = true; break;
   case ISD::STRICT_FROUND: NewOpc = ISD::FROUND; IsUnary = true; break;
   case ISD::STRICT_FTRUNC: NewOpc = ISD::FTRUNC; IsUnary = true; break;
+  case ISD::STRICT_FSETCC: NewOpc = ISD::SETCC; IsTernary = true; break;
   }
 
   // We're taking this node out of the chain, so we need to re-link things.
   SDValue InputChain = Node->getOperand(0);
   SDValue OutputChain = SDValue(Node, 1);
   ReplaceAllUsesOfValueWith(OutputChain, InputChain);
 
-  SDVTList VTs = getVTList(Node->getOperand(1).getValueType());
+  SDVTList VTs = getVTList(Node->getValueType(0));
   SDNode *Res = nullptr;
   if (IsUnary)
     Res = MorphNodeTo(Node, NewOpc, VTs, { Node->getOperand(1) });
   else if (IsTernary)
     Res = MorphNodeTo(Node, NewOpc, VTs, { Node->getOperand(1),
                                            Node->getOperand(2),
-                                           Node->getOperand(3)});
+                                           Node->getOperand(3) });
   else
     Res = MorphNodeTo(Node, NewOpc, VTs, { Node->getOperand(1),
                                            Node->getOperand(2) });
 
   // MorphNodeTo can operate in two ways: if an existing node with the
   // specified operands exists, it can just return it.  Otherwise, it
   // updates the node in place to have the requested operands.
   if (Res == Node) {

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I, unsigned Intrinsic) {
   case Intrinsic::experimental_constrained_rint:
   case Intrinsic::experimental_constrained_nearbyint:
   case Intrinsic::experimental_constrained_maxnum:
   case Intrinsic::experimental_constrained_minnum:
   case Intrinsic::experimental_constrained_ceil:
   case Intrinsic::experimental_constrained_floor:
   case Intrinsic::experimental_constrained_round:
   case Intrinsic::experimental_constrained_trunc:
+  case Intrinsic::experimental_constrained_fcmpeq:
     visitConstrainedFPIntrinsic(cast<ConstrainedFPIntrinsic>(I));
     return nullptr;
   case Intrinsic::fmuladd: {
     EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType());
     if (TM.Options.AllowFPOpFusion != FPOpFusion::Strict &&
         TLI.isFMAFasterThanFMulAndFAdd(VT)) {
       setValue(&I, DAG.getNode(ISD::FMA, sdl,
                                getValue(I.getArgOperand(0)).getValueType(),
case Intrinsic::wasm_landingpad_index:
     // delete it now.
     return nullptr;
   }
 }
 
 void SelectionDAGBuilder::visitConstrainedFPIntrinsic(
     const ConstrainedFPIntrinsic &FPI) {
   SDLoc sdl = getCurSDLoc();
+
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+  SmallVector<EVT, 4> ValueVTs;
+  ComputeValueVTs(TLI, DAG.getDataLayout(), FPI.getType(), ValueVTs);
+  ValueVTs.push_back(MVT::Other); // Out chain
+
+  SDValue Chain = getRoot();
+  SmallVector<SDValue, 4> Opers;
+  Opers.push_back(Chain);
+  if (FPI.isUnaryOp()) {
+    Opers.push_back(getValue(FPI.getArgOperand(0)));
+  } else if (FPI.isTernaryOp()) {
+    Opers.push_back(getValue(FPI.getArgOperand(0)));
+    Opers.push_back(getValue(FPI.getArgOperand(1)));
+    Opers.push_back(getValue(FPI.getArgOperand(2)));
+  } else {
+    Opers.push_back(getValue(FPI.getArgOperand(0)));
+    Opers.push_back(getValue(FPI.getArgOperand(1)));
+  }
+
   unsigned Opcode;
   switch (FPI.getIntrinsicID()) {
   default: llvm_unreachable("Impossible intrinsic");  // Can't reach here.
   case Intrinsic::experimental_constrained_fadd:
     Opcode = ISD::STRICT_FADD;
     break;
   case Intrinsic::experimental_constrained_fsub:
     Opcode = ISD::STRICT_FSUB;
case Intrinsic::experimental_constrained_floor:
     Opcode = ISD::STRICT_FFLOOR;
     break;
   case Intrinsic::experimental_constrained_round:
     Opcode = ISD::STRICT_FROUND;
     break;
   case Intrinsic::experimental_constrained_trunc:
     Opcode = ISD::STRICT_FTRUNC;
     break;
+  case Intrinsic::experimental_constrained_fcmpeq:
+    Opcode = ISD::STRICT_FSETCC;
+    Opers.push_back(DAG.getCondCode(ISD::SETUEQ));
+    break;
   }
-  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
-  SDValue Chain = getRoot();
-  SmallVector<EVT, 4> ValueVTs;
-  ComputeValueVTs(TLI, DAG.getDataLayout(), FPI.getType(), ValueVTs);
-  ValueVTs.push_back(MVT::Other); // Out chain
 
   SDVTList VTs = DAG.getVTList(ValueVTs);
-  SDValue Result;
-  if (FPI.isUnaryOp())
-    Result = DAG.getNode(Opcode, sdl, VTs,
-                         { Chain, getValue(FPI.getArgOperand(0)) });
-  else if (FPI.isTernaryOp())
-    Result = DAG.getNode(Opcode, sdl, VTs,
-                         { Chain, getValue(FPI.getArgOperand(0)),
-                                  getValue(FPI.getArgOperand(1)),
-                                  getValue(FPI.getArgOperand(2)) });
-  else
-    Result = DAG.getNode(Opcode, sdl, VTs,
-                         { Chain, getValue(FPI.getArgOperand(0)),
-                           getValue(FPI.getArgOperand(1))  });
+  SDValue Result = DAG.getNode(Opcode, sdl, VTs, Opers);
 
   assert(Result.getNode()->getNumValues() == 2);
   SDValue OutChain = Result.getValue(1);
   DAG.setRoot(OutChain);
   SDValue FPResult = Result.getValue(0);
   setValue(&FPI, FPResult);
 }
 

lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp

#endif
   case ISD::SMAX:                       return "smax";
   case ISD::UMIN:                       return "umin";
   case ISD::UMAX:                       return "umax";
 
   case ISD::FPOWI:                      return "fpowi";
   case ISD::STRICT_FPOWI:               return "strict_fpowi";
   case ISD::SETCC:                      return "setcc";
   case ISD::SETCCCARRY:                 return "setcccarry";
+  case ISD::STRICT_FSETCC:              return "strict_fsetcc";
   case ISD::SELECT:                     return "select";
   case ISD::VSELECT:                    return "vselect";
   case ISD::SELECT_CC:                  return "select_cc";
   case ISD::INSERT_VECTOR_ELT:          return "insert_vector_elt";
   case ISD::EXTRACT_VECTOR_ELT:         return "extract_vector_elt";
   case ISD::CONCAT_VECTORS:             return "concat_vectors";
   case ISD::INSERT_SUBVECTOR:           return "insert_subvector";
   case ISD::EXTRACT_SUBVECTOR:          return "extract_subvector";

lib/IR/Function.cpp

enum IIT_Info {
   IIT_PTR_TO_ELT = 33,
   IIT_VEC_OF_ANYPTRS_TO_ELT = 34,
   IIT_I128 = 35,
   IIT_V512 = 36,
   IIT_V1024 = 37,
   IIT_STRUCT6 = 38,
   IIT_STRUCT7 = 39,
   IIT_STRUCT8 = 40,
-  IIT_F128 = 41
+  IIT_F128 = 41,
+  IIT_SAME_VEC_LENGTH_OR_SCALAR_ARG = 42
 };
 
 static void DecodeIITType(unsigned &NextElt, ArrayRef<unsigned char> Infos,
                       SmallVectorImpl<Intrinsic::IITDescriptor> &OutputTable) {
   using namespace Intrinsic;
 
   IIT_Info Info = IIT_Info(Infos[NextElt++]);
   unsigned StructElts = 2;
case IIT_TRUNC_ARG: {
     return;
   }
   case IIT_HALF_VEC_ARG: {
     unsigned ArgInfo = (NextElt == Infos.size() ? 0 : Infos[NextElt++]);
     OutputTable.push_back(IITDescriptor::get(IITDescriptor::HalfVecArgument,
                                              ArgInfo));
     return;
   }
+  case IIT_SAME_VEC_LENGTH_OR_SCALAR_ARG: {
+    unsigned ArgNo = (NextElt == Infos.size() ? 0 : Infos[NextElt++]);
+    unsigned short RefNo = (NextElt == Infos.size() ? 0 : Infos[NextElt++]);
+    OutputTable.push_back(IITDescriptor::get(IITDescriptor::SameVecLengthOrScalarArgument,
+                                             ArgNo, RefNo));
+    return;
+  }
   case IIT_SAME_VEC_WIDTH_ARG: {
     unsigned ArgInfo = (NextElt == Infos.size() ? 0 : Infos[NextElt++]);
     OutputTable.push_back(IITDescriptor::get(IITDescriptor::SameVecWidthArgument,
                                              ArgInfo));
     return;
   }
   case IIT_PTR_TO_ARG: {
     unsigned ArgInfo = (NextElt == Infos.size() ? 0 : Infos[NextElt++]);
case IITDescriptor::TruncArgument: {
 
     IntegerType *ITy = cast<IntegerType>(Ty);
     assert(ITy->getBitWidth() % 2 == 0);
     return IntegerType::get(Context, ITy->getBitWidth() / 2);
   }
   case IITDescriptor::HalfVecArgument:
     return VectorType::getHalfElementsVectorType(cast<VectorType>(
                                                   Tys[D.getArgumentNumber()]));
+  case IITDescriptor::SameVecLengthOrScalarArgument:
+    // Return the overloaded type
+    return Tys[D.getOverloadArgNumber()];
   case IITDescriptor::SameVecWidthArgument: {
     Type *EltTy = DecodeFixedType(Infos, Tys, Context);
     Type *Ty = Tys[D.getArgumentNumber()];
     if (VectorType *VTy = dyn_cast<VectorType>(Ty)) {
       return VectorType::get(EltTy, VTy->getNumElements());
     }
     llvm_unreachable("unhandled");
   }
case IITDescriptor::Argument:
           ArgTys.push_back(Ty);
 
           switch (D.getArgumentKind()) {
             case IITDescriptor::AK_Any:        return false; // Success
             case IITDescriptor::AK_AnyInteger: return !Ty->isIntOrIntVectorTy();
             case IITDescriptor::AK_AnyFloat:   return !Ty->isFPOrFPVectorTy();
             case IITDescriptor::AK_AnyVector:  return !isa<VectorType>(Ty);
             case IITDescriptor::AK_AnyPointer: return !isa<PointerType>(Ty);
+            case IITDescriptor::AK_AnyBoolean: return !Ty->isIntOrIntVectorTy(1); 
           }
           llvm_unreachable("all argument kinds not covered");
 
     case IITDescriptor::ExtendArgument: {
       // This may only be used when referring to a previous vector argument.
       if (D.getArgumentNumber() >= ArgTys.size())
         return true;
 
case IITDescriptor::TruncArgument: {
       return Ty != NewTy;
     }
     case IITDescriptor::HalfVecArgument:
       // This may only be used when referring to a previous vector argument.
       return D.getArgumentNumber() >= ArgTys.size() ||
              !isa<VectorType>(ArgTys[D.getArgumentNumber()]) ||
              VectorType::getHalfElementsVectorType(
                      cast<VectorType>(ArgTys[D.getArgumentNumber()])) != Ty;
+    case IITDescriptor::SameVecLengthOrScalarArgument: {
+      unsigned RefArgNumber = D.getRefArgNumber(); 
+      if (RefArgNumber >= ArgTys.size())
+        return true;
+     
+      // Record the overloaded type
+      assert(D.getOverloadArgNumber() == ArgTys.size() &&
+             "Table consistency error");
+      ArgTys.push_back(Ty);
+ 
+      // Handle Vectors
+      if (Ty->isVectorTy() && ArgTys[RefArgNumber]->isVectorTy()) {
+        VectorType * ReferenceType =
+          dyn_cast<VectorType>(ArgTys[RefArgNumber]); 
+        VectorType *ThisArgType = dyn_cast<VectorType>(Ty);
+        if (!ThisArgType || !ReferenceType ||
+            (ReferenceType->getVectorNumElements() !=
+             ThisArgType->getVectorNumElements()))
+          return true;
+      } else {
+        // Handle Scalars
+        if (Ty->isVectorTy() || ArgTys[RefArgNumber]->isVectorTy()) 
+          return true;
+      }
+
+      return false;
+    }
     case IITDescriptor::SameVecWidthArgument: {
       if (D.getArgumentNumber() >= ArgTys.size())
         return true;
       VectorType * ReferenceType =
         dyn_cast<VectorType>(ArgTys[D.getArgumentNumber()]);
       VectorType *ThisArgType = dyn_cast<VectorType>(Ty);
       if (!ThisArgType || !ReferenceType ||
           (ReferenceType->getVectorNumElements() !=

lib/IR/Verifier.cpp

void Verifier::visitIntrinsicCallSite(Intrinsic::ID ID, CallSite CS) {
   case Intrinsic::experimental_constrained_rint:
   case Intrinsic::experimental_constrained_nearbyint:
   case Intrinsic::experimental_constrained_maxnum:
   case Intrinsic::experimental_constrained_minnum:
   case Intrinsic::experimental_constrained_ceil:
   case Intrinsic::experimental_constrained_floor:
   case Intrinsic::experimental_constrained_round:
   case Intrinsic::experimental_constrained_trunc:
+  case Intrinsic::experimental_constrained_fcmpeq:
     visitConstrainedFPIntrinsic(
         cast<ConstrainedFPIntrinsic>(*CS.getInstruction()));
     break;
   case Intrinsic::dbg_declare: // llvm.dbg.declare
     Assert(isa<MetadataAsValue>(CS.getArgOperand(0)),
            "invalid llvm.dbg.declare intrinsic call 1", CS);
     visitDbgIntrinsic("declare", cast<DbgVariableIntrinsic>(*CS.getInstruction()));
     break;

lib/Target/X86/X86ISelLowering.h

private:
     SDValue LowerGlobalAddress(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerGlobalTLSAddress(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerExternalSymbol(SDValue Op, SelectionDAG &DAG) const;
 
     SDValue LowerSINT_TO_FP(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerUINT_TO_FP(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerTRUNCATE(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerFP_TO_INT(SDValue Op, SelectionDAG &DAG) const;
+    SDValue LowerSTRICT_FSETCC(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerSETCC(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerSETCCCARRY(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerSELECT(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerBRCOND(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerJumpTable(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerDYNAMIC_STACKALLOC(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerVASTART(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerVAARG(SDValue Op, SelectionDAG &DAG) const;

lib/Target/X86/X86ISelLowering.cpp

if (!Subtarget.hasMOVBE())
     setOperationAction(ISD::BSWAP          , MVT::i16  , Expand);
 
   // These should be promoted to a larger select which is supported.
   setOperationAction(ISD::SELECT          , MVT::i1   , Promote);
   // X86 wants to expand cmov itself.
   for (auto VT : { MVT::f32, MVT::f64, MVT::f80, MVT::f128 }) {
     setOperationAction(ISD::SELECT, VT, Custom);
     setOperationAction(ISD::SETCC, VT, Custom);
+    setOperationAction(ISD::STRICT_FSETCC, VT, Custom);
   }
   for (auto VT : { MVT::i8, MVT::i16, MVT::i32, MVT::i64 }) {
     if (VT == MVT::i64 && !Subtarget.is64Bit())
       continue;
     setOperationAction(ISD::SELECT, VT, Custom);
     setOperationAction(ISD::SETCC,  VT, Custom);
   }
 
if (Op0.getOpcode() == ISD::OR && Op0.hasOneUse()) {
     LHS = Op0.getOperand(0);
     RHS = Op0.getOperand(1);
   }
 
   SDValue KORTEST = DAG.getNode(X86ISD::KORTEST, dl, MVT::i32, LHS, RHS);
   return getSETCC(X86CC, KORTEST, dl, DAG);
 }
 
+SDValue X86TargetLowering::LowerSTRICT_FSETCC(SDValue Op, SelectionDAG &DAG) const {
+  // Mutate away for now...
+  Op = SDValue(DAG.mutateStrictFPToFP(Op.getNode()), 0);
+  return LowerSETCC(Op, DAG);
+}
+
 SDValue X86TargetLowering::LowerSETCC(SDValue Op, SelectionDAG &DAG) const {
 
   MVT VT = Op.getSimpleValueType();
 
   if (VT.isVector()) return LowerVSETCC(Op, Subtarget, DAG);
 
   assert(VT == MVT::i8 && "SetCC type must be 8-bit integer");
   SDValue Op0 = Op.getOperand(0);
SDValue X86TargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   case ISD::FP_EXTEND:          return LowerFP_EXTEND(Op, DAG);
   case ISD::LOAD:               return LowerLoad(Op, Subtarget, DAG);
   case ISD::STORE:              return LowerStore(Op, Subtarget, DAG);
   case ISD::FABS:
   case ISD::FNEG:               return LowerFABSorFNEG(Op, DAG);
   case ISD::FCOPYSIGN:          return LowerFCOPYSIGN(Op, DAG);
   case ISD::FGETSIGN:           return LowerFGETSIGN(Op, DAG);
   case ISD::SETCC:              return LowerSETCC(Op, DAG);
+  case ISD::STRICT_FSETCC:      return LowerSTRICT_FSETCC(Op, DAG);
   case ISD::SETCCCARRY:         return LowerSETCCCARRY(Op, DAG);
   case ISD::SELECT:             return LowerSELECT(Op, DAG);
   case ISD::BRCOND:             return LowerBRCOND(Op, DAG);
   case ISD::JumpTable:          return LowerJumpTable(Op, DAG);
   case ISD::VASTART:            return LowerVASTART(Op, DAG);
   case ISD::VAARG:              return LowerVAARG(Op, DAG);
   case ISD::VACOPY:             return LowerVACOPY(Op, Subtarget, DAG);
   case ISD::INTRINSIC_WO_CHAIN: return LowerINTRINSIC_WO_CHAIN(Op, DAG);

test/CodeGen/X86/fp-intrinsics.ll

entry:
   %rem = call double @llvm.experimental.constrained.frem.f64(
                                                double 1.000000e+00,
                                                double 1.000000e+01,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %rem
 }
 
+; CHECK-LABEL: f20
+; COMMON: ucomisd
+define i1 @f20() {
+entry:
+  %cmp = call i1 @llvm.experimental.constrained.fcmpeq.f64(
+                                               double 1.000000e+00,
+                                               double 1.000000e+01,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret i1 %cmp
+}
+
 @llvm.fp.env = thread_local global i8 zeroinitializer, section "llvm.metadata"
 declare double @llvm.experimental.constrained.fadd.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fsub.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fmul.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fdiv.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.frem.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.sqrt.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.pow.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.powi.f64(double, i32, metadata, metadata)
 declare double @llvm.experimental.constrained.sin.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.cos.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.exp.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.exp2.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.log.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.log10.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.log2.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.rint.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.nearbyint.f64(double, metadata, metadata)
+declare i1 @llvm.experimental.constrained.fcmpeq.f64(double, double, metadata, metadata)
 declare float @llvm.experimental.constrained.fma.f32(float, float, float, metadata, metadata)
 declare double @llvm.experimental.constrained.fma.f64(double, double, double, metadata, metadata)

test/CodeGen/X86/vector-constrained-fp-intrinsics-cmp.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -O3 -mtriple=x86_64-pc-linux < %s | FileCheck %s
+
+define <2 x i1> @constrained_vector_fcmpeq_v2f64() {
+; CHECK-LABEL: constrained_vector_fcmpeq_v2f64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movapd {{.*#+}} xmm1 = [2.0E+0,1.0E+0]
+; CHECK-NEXT:    movapd {{.*#+}} xmm0 = [1.0E+0,2.0E+0]
+; CHECK-NEXT:    movapd %xmm0, %xmm2
+; CHECK-NEXT:    cmpeqpd %xmm1, %xmm2
+; CHECK-NEXT:    cmpunordpd %xmm1, %xmm0
+; CHECK-NEXT:    orpd %xmm2, %xmm0
+; CHECK-NEXT:    retq
+entry:
+  %fcmp = call <2 x i1> @llvm.experimental.constrained.fcmpeq.v2f64(
+           <2 x double> <double 1.0, double 2.0>,
+           <2 x double> <double 2.0, double 1.0>,
+           metadata !"round.dynamic",
+           metadata !"fpexcept.strict")
+  ret <2 x i1> %fcmp
+}
+
+; Single width declarations
+declare <2 x i1> @llvm.experimental.constrained.fcmpeq.v2f64(<2 x double>, <2 x double>, metadata, metadata)
+

utils/TableGen/CodeGenDAGPatterns.cpp

for (MVT T : Out) {
 
     Ovs.insert(T);
     // MachineValueTypeSet allows iteration and erasing.
     Out.erase(T);
   }
 
   for (MVT Ov : Ovs) {
     switch (Ov.SimpleTy) {
+      case MVT::bAny:
+        for (MVT T : MVT::boolean_valuetypes())
+          if (Legal.count(T))
+            Out.insert(T);
+        for (MVT T : MVT::boolean_vector_valuetypes())
+          if (Legal.count(T))
+            Out.insert(T);
+        return;
       case MVT::iPTRAny:
         Out.insert(MVT::iPTR);
         return;
       case MVT::iAny:
         for (MVT T : MVT::integer_valuetypes())
           if (Legal.count(T))
             Out.insert(T);
         for (MVT T : MVT::integer_vector_valuetypes())

utils/TableGen/CodeGenTarget.cpp

StringRef llvm::getEnumName(MVT::SimpleValueType T) {
   case MVT::Other:    return "MVT::Other";
   case MVT::i1:       return "MVT::i1";
   case MVT::i8:       return "MVT::i8";
   case MVT::i16:      return "MVT::i16";
   case MVT::i32:      return "MVT::i32";
   case MVT::i64:      return "MVT::i64";
   case MVT::i128:     return "MVT::i128";
   case MVT::Any:      return "MVT::Any";
+  case MVT::bAny:     return "MVT::bAny";
   case MVT::iAny:     return "MVT::iAny";
   case MVT::fAny:     return "MVT::fAny";
   case MVT::vAny:     return "MVT::vAny";
   case MVT::f16:      return "MVT::f16";
   case MVT::f32:      return "MVT::f32";
   case MVT::f64:      return "MVT::f64";
   case MVT::f80:      return "MVT::f80";
   case MVT::f128:     return "MVT::f128";
if (TyEl->isSubClassOf("LLVMMatchType")) {
         PrintFatalError(Twine("ParamTypes is ") + TypeList->getAsString());
       }
       VT = OverloadedVTs[MatchTy];
       // It only makes sense to use the extended and truncated vector element
       // variants with iAny types; otherwise, if the intrinsic is not
       // overloaded, all the types can be specified directly.
       assert(((!TyEl->isSubClassOf("LLVMExtendedType") &&
                !TyEl->isSubClassOf("LLVMTruncatedType") &&
-               !TyEl->isSubClassOf("LLVMVectorSameWidth")) ||
-              VT == MVT::iAny || VT == MVT::vAny) &&
+               !TyEl->isSubClassOf("LLVMVectorSameWidth") &&
+               !TyEl->isSubClassOf("LLVMVectorSameLengthOrScalar")) ||
+              VT == MVT::bAny || VT == MVT::iAny || VT == MVT::vAny) &&
              "Expected iAny or vAny type");
     } else
       VT = getValueType(TyEl->getValueAsDef("VT"));
 
     if (MVT(VT).isOverloaded()) {
       OverloadedVTs.push_back(VT);
       isOverloaded = true;
     }

utils/TableGen/IntrinsicEmitter.cpp

enum IIT_Info {
   IIT_PTR_TO_ELT = 33,
   IIT_VEC_OF_ANYPTRS_TO_ELT = 34,
   IIT_I128 = 35,
   IIT_V512 = 36,
   IIT_V1024 = 37,
   IIT_STRUCT6 = 38,
   IIT_STRUCT7 = 39,
   IIT_STRUCT8 = 40,
-  IIT_F128 = 41
+  IIT_F128 = 41,
+  IIT_SAME_VEC_LENGTH_OR_SCALAR_ARG = 42
 };
 
 static void EncodeFixedValueType(MVT::SimpleValueType VT,
                                  std::vector<unsigned char> &Sig) {
   if (MVT(VT).isInteger()) {
     unsigned BitWidth = MVT(VT).getSizeInBits();
     switch (BitWidth) {
     default: PrintFatalError("unhandled integer type width in intrinsic!");
if (R->isSubClassOf("LLVMMatchType")) {
     unsigned Number = R->getValueAsInt("Number");
     assert(Number < ArgCodes.size() && "Invalid matching number!");
     if (R->isSubClassOf("LLVMExtendedType"))
       Sig.push_back(IIT_EXTEND_ARG);
     else if (R->isSubClassOf("LLVMTruncatedType"))
       Sig.push_back(IIT_TRUNC_ARG);
     else if (R->isSubClassOf("LLVMHalfElementsVectorType"))
       Sig.push_back(IIT_HALF_VEC_ARG);
+    else if (R->isSubClassOf("LLVMVectorSameLengthOrScalar")) {
+      Sig.push_back(IIT_SAME_VEC_LENGTH_OR_SCALAR_ARG);
+      unsigned ArgNo = ArgCodes.size();
+      ArgCodes.push_back(3 /*vAny*/);
+      // Encode overloaded ArgNo
+      Sig.push_back(ArgNo);
+      // Encode LLVMMatchType<Number> ArgNo
+      Sig.push_back(Number);
+      return;
+    }
     else if (R->isSubClassOf("LLVMVectorSameWidth")) {
       Sig.push_back(IIT_SAME_VEC_WIDTH_ARG);
       Sig.push_back((Number << 3) | ArgCodes[Number]);
       MVT::SimpleValueType VT = getValueType(R->getValueAsDef("ElTy"));
       EncodeFixedValueType(VT, Sig);
       return;
     }
     else if (R->isSubClassOf("LLVMPointerTo"))
if (R->isSubClassOf("LLVMMatchType")) {
     return Sig.push_back((Number << 3) | ArgCodes[Number]);
   }
 
   MVT::SimpleValueType VT = getValueType(R->getValueAsDef("VT"));
 
   unsigned Tmp = 0;
   switch (VT) {
   default: break;
+  case MVT::bAny: ++Tmp; LLVM_FALLTHROUGH;
   case MVT::iPTRAny: ++Tmp; LLVM_FALLTHROUGH;
   case MVT::vAny: ++Tmp;    LLVM_FALLTHROUGH;
   case MVT::fAny: ++Tmp;    LLVM_FALLTHROUGH;
   case MVT::iAny: ++Tmp;    LLVM_FALLTHROUGH;
   case MVT::Any: {
     // If this is an "any" valuetype, then the type is the type of the next
     // type in the list specified to getIntrinsic().
     Sig.push_back(IIT_ARG);