[SDAG] Preserve unpredictable metadata, teach X86CmovConversion to respect this metadata

[xbolva00]

Summary

Sometimes an developer would like to have more control over cmov vs branch. We have unpredictable metadata in LLVM IR, but currently it is ignored by X86 backend. Propagate this metadata and avoid cmov->branch conversion in X86CmovConversion for cmov with this metadata.

Example:

int MaxIndex(int n, int *a) {
    int t = 0;
    for (int i = 1; i < n; i++) {
        // cmov is converted to branch by X86CmovConversion
        if (a[i] > a[t]) t = i;
    }
    return t;
}

int MaxIndex2(int n, int *a) {
    int t = 0;
    for (int i = 1; i < n; i++) {
        // cmov is preserved
        if (__builtin_unpredictable(a[i] > a[t])) t = i;
    }
    return t;
}

[xbolva00]

Not super familiar in this area, to be noted.

I also plan to update release notes if the patch is accepted.

[nikic]

llvm/include/llvm/CodeGen/MachineInstr.h

124

This will increase MI size by 8 bytes.

[xbolva00]

llvm/include/llvm/CodeGen/MachineInstr.h

124

Yes, but what we can do?

Is it a big problem in practise?

[xbolva00]

llvm/include/llvm/CodeGen/MachineInstr.h

123–124

Do we need *uint32* for NumOperands?

[RKSimon]

@xbolva00 Are you specifically aiming to get this in for 14.x? TBH this close to the branch date, I'd prefer to delay this work until the branch is made and initial cherrypicks are out of the way, and we aim for 15.x

We've tried similar things to this in the past and its taken a long time to get it stable (usually by disabling it a lot of it).

[xbolva00]

In D118118#3269263, @RKSimon wrote:

@xbolva00 Are you specifically aiming to get this in for 14.x? TBH this close to the branch date, I'd prefer to delay this work until the branch is made and initial cherrypicks are out of the way, and we aim for 15.x

We've tried similar things to this in the past and its taken a long time to get it stable (usually by disabling it a lot of it).

No problem, not aiming for 14.x. We can delay it.

[RKSimon]

@xbolva00 Are you still working on this atm or are you waiting to see what happens with https://discourse.llvm.org/t/rfc-cmov-vs-branch-optimization/59628 ?

llvm/include/llvm/CodeGen/MachineInstr.h

84

I don't suppose there's any existing MIFlag bits that we can get rid of?

[xbolva00]

In D118118#3299513, @RKSimon wrote:

@xbolva00 Are you still working on this atm or are you waiting to see what happens with https://discourse.llvm.org/t/rfc-cmov-vs-branch-optimization/59628 ?

Yes, I am still interested in this patch.

That RFC would benefit from this work as well, especially if they go for midend LLVM IR pass - but I think that pass could be super tricky to do it right and actually do not regress the performance for "important workloads" (I remember some huge regression around LLVM 9 related to cmov vs branch).

[craig.topper]

llvm/include/llvm/CodeGen/MachineInstr.h

84

I wondered if we could have separate X86 CMOV opcodes for unpredictable? Since we're only trying to convey information to this one X86 specific pass. But that might require too many updates to the CMOV related functions in X86InstrInfo.

[xbolva00]

llvm/include/llvm/CodeGen/MachineInstr.h

123–124

@RKSimon @craig.topper ?

[RKSimon]

llvm/include/llvm/CodeGen/MachineInstr.h

123–124

That might be possible - maybe investigate with a suitable check inside MachineInstr::addOperand?

[RKSimon]

llvm/include/llvm/CodeGen/MachineInstr.h

1893

Have you been able to ensure none of the callers of these functions implicitly truncate back to u16?

llvm/include/llvm/CodeGen/SelectionDAGNodes.h

398

Missing comment description?

[xbolva00]

This comment was removed by xbolva00.

[xbolva00]

llvm/include/llvm/CodeGen/MachineInstr.h

123–124

Now done.

124

Last time I tried I was not able to observe ani significant memory usage regressions.
I reduced MI size by switching NumOperands to 16bits so I can get use these saved bits for Flags.

[xbolva00]

Rebased and updated callsites.

[nikic]

llvm/lib/Target/X86/X86CmovConversion.cpp

312

Is it safe to just skip it entirely, or do we need to treat it as a non-cmov instruction?

llvm/lib/Target/X86/X86ISelLowering.cpp

26486

Is it possible to directly pass the flags here instead? Seems odd to use FlagInserter for a single instruction.

[xbolva00]

llvm/lib/Target/X86/X86CmovConversion.cpp

312

Good point, I am not sure, probably we should threat it that way.

@RKSimon / @craig.topper any suggestions?

llvm/lib/Target/X86/X86ISelLowering.cpp

26486

Right, I will fix it.

[craig.topper]

llvm/include/llvm/CodeGen/MachineInstr.h

115

I kind of think this description is overly short. Unpredictable for an instruction could mean many things.

124

Use uint32_t since we've very carefully counted bits in all of the fields.

[craig.topper]

llvm/lib/Target/X86/X86CmovConversion.cpp

309

unpredictable*

312

What instructions can this flag appear on? There's a check later for instructions that define EFLAGS we can't skip that check.

[xbolva00]

Addressed review feedback

[xbolva00]

llvm/include/llvm/CodeGen/MachineInstr.h

115

Changed, feel free to suggest better description.

llvm/lib/Target/X86/X86CmovConversion.cpp

312

Select -> CMOV.

Okay, fixed and EFLAGS check is not skipped.

[craig.topper]

llvm/lib/CodeGen/MIRPrinter.cpp

805

This needs to be added to MIRParser.cpp, MIRLexer.h and MIRLexer.cpp

[xbolva00]

Addressed review comments.

[xbolva00]

llvm/lib/CodeGen/MIRPrinter.cpp

805

Thanks

[nikic]

LGTM

llvm/include/llvm/CodeGen/SelectionDAGNodes.h

398

Not done.

[xbolva00]

Added missing comment.

llvm/include/llvm/CodeGen/MachineInstr.h

public:
   ///
   enum CommentFlag {
     ReloadReuse = 0x1,    // higher bits are reserved for target dep comments.
     NoSchedComment = 0x2,
     TAsmComments = 0x4    // Target Asm comments should start from this value.
   };
 
   enum MIFlag {
-    NoFlags      = 0,
-    FrameSetup   = 1 << 0,              // Instruction is used as a part of
-                                        // function frame setup code.
-    FrameDestroy = 1 << 1,              // Instruction is used as a part of
-                                        // function frame destruction code.
-    BundledPred  = 1 << 2,              // Instruction has bundled predecessors.
-    BundledSucc  = 1 << 3,              // Instruction has bundled successors.
-    FmNoNans     = 1 << 4,              // Instruction does not support Fast
-                                        // math nan values.
-    FmNoInfs     = 1 << 5,              // Instruction does not support Fast
-                                        // math infinity values.
-    FmNsz        = 1 << 6,              // Instruction is not required to retain
-                                        // signed zero values.
-    FmArcp       = 1 << 7,              // Instruction supports Fast math
-                                        // reciprocal approximations.
-    FmContract   = 1 << 8,              // Instruction supports Fast math
-                                        // contraction operations like fma.
-    FmAfn        = 1 << 9,              // Instruction may map to Fast math
-                                        // intrinsic approximation.
-    FmReassoc    = 1 << 10,             // Instruction supports Fast math
-                                        // reassociation of operand order.
-    NoUWrap      = 1 << 11,             // Instruction supports binary operator
-                                        // no unsigned wrap.
-    NoSWrap      = 1 << 12,             // Instruction supports binary operator
-                                        // no signed wrap.
-    IsExact      = 1 << 13,             // Instruction supports division is
-                                        // known to be exact.
-    NoFPExcept   = 1 << 14,             // Instruction does not raise
-                                        // floatint-point exceptions.
-    NoMerge      = 1 << 15,             // Passes that drop source location info
-                                        // (e.g. branch folding) should skip
-                                        // this instruction.
+    NoFlags = 0,
+    FrameSetup = 1 << 0,     // Instruction is used as a part of
+    FrameDestroy = 1 << 1,   // Instruction is used as a part of
+    BundledPred = 1 << 2,    // Instruction has bundled predecessors.
+    BundledSucc = 1 << 3,    // Instruction has bundled successors.
+    FmNoNans = 1 << 4,       // Instruction does not support Fast
+    FmNoInfs = 1 << 5,       // Instruction does not support Fast
+    FmNsz = 1 << 6,          // Instruction is not required to retain
+    FmArcp = 1 << 7,         // Instruction supports Fast math
+    FmContract = 1 << 8,     // Instruction supports Fast math
+    FmAfn = 1 << 9,          // Instruction may map to Fast math
+    FmReassoc = 1 << 10,     // Instruction supports Fast math
+    NoUWrap = 1 << 11,       // Instruction supports binary operator
+    NoSWrap = 1 << 12,       // Instruction supports binary operator
+    IsExact = 1 << 13,       // Instruction supports division is
+    NoFPExcept = 1 << 14,    // Instruction does not raise
+    NoMerge = 1 << 15,       // Passes that drop source location info
+    Unpredictable = 1 << 16, // Instruction with unpredictable condition.
   };
 
 private:
   const MCInstrDesc *MCID;              // Instruction descriptor.
   MachineBasicBlock *Parent = nullptr;  // Pointer to the owning basic block.
 
   // Operands are allocated by an ArrayRecycler.
   MachineOperand *Operands = nullptr;   // Pointer to the first operand.
-  uint16_t NumOperands = 0;             // Number of operands on instruction.
-
-  uint16_t Flags = 0;                   // Various bits of additional
+  uint32_t Flags = 0;                   // Various bits of additional
                                         // information about machine
                                         // instruction.
-
+  uint16_t NumOperands = 0;             // Number of operands on instruction.
   uint8_t AsmPrinterFlags = 0;          // Various bits of information used by
                                         // the AsmPrinter to emit helpful
                                         // comments.  This is *not* semantic
                                         // information.  Do not use this for
                                         // anything other than to convey comment
                                         // information to AsmPrinter.
 
   // OperandCapacity has uint8_t size, so it should be next to AsmPrinterFlags
public:
   }
 
   /// Clear specific AsmPrinter flags.
   void clearAsmPrinterFlag(CommentFlag Flag) {
     AsmPrinterFlags &= ~Flag;
   }
 
   /// Return the MI flags bitvector.
-  uint16_t getFlags() const {
+  uint32_t getFlags() const {
     return Flags;
   }
 
   /// Return whether an MI flag is set.
   bool getFlag(MIFlag Flag) const {
     return Flags & Flag;
   }
 
   /// Set a MI flag.
   void setFlag(MIFlag Flag) {
-    Flags |= (uint16_t)Flag;
+    Flags |= (uint32_t)Flag;
   }
 
   void setFlags(unsigned flags) {
     // Filter out the automatically maintained flags.
     unsigned Mask = BundledPred | BundledSucc;
     Flags = (Flags & Mask) | (flags & ~Mask);
   }
 
   /// clearFlag - Clear a MI flag.
   void clearFlag(MIFlag Flag) {
-    Flags &= ~((uint16_t)Flag);
+    Flags &= ~((uint32_t)Flag);
   }
 
   /// Return true if MI is in a bundle (but not the first MI in a bundle).
   ///
   /// A bundle looks like this before it's finalized:
   ///   ----------------
   ///   |      MI      |
   ///   ----------------

   void setPCSections(MachineFunction &MF, MDNode *MD);
 
   /// Set the CFI type for the instruction.
   void setCFIType(MachineFunction &MF, uint32_t Type);
 
   /// Return the MIFlags which represent both MachineInstrs. This
   /// should be used when merging two MachineInstrs into one. This routine does
   /// not modify the MIFlags of this MachineInstr.
-  uint16_t mergeFlagsWith(const MachineInstr& Other) const;
+  uint32_t mergeFlagsWith(const MachineInstr& Other) const;
 
-  static uint16_t copyFlagsFromInstruction(const Instruction &I);
+  static uint32_t copyFlagsFromInstruction(const Instruction &I);
 
   /// Copy all flags to MachineInst MIFlags
   void copyIRFlags(const Instruction &I);
 
   /// Break any tie involving OpIdx.
   void untieRegOperand(unsigned OpIdx) {
     MachineOperand &MO = getOperand(OpIdx);
     if (MO.isReg() && MO.isTied()) {

llvm/include/llvm/CodeGen/SelectionDAGNodes.h

private:
   bool AllowReassociation : 1;
 
   // We assume instructions do not raise floating-point exceptions by default,
   // and only those marked explicitly may do so.  We could choose to represent
   // this via a positive "FPExcept" flags like on the MI level, but having a
   // negative "NoFPExcept" flag here (that defaults to true) makes the flag
   // intersection logic more straightforward.
   bool NoFPExcept : 1;
+  // Instructions with attached 'unpredictable' metadata on IR level.
+  bool Unpredictable : 1;
 
 public:
   /// Default constructor turns off all optimization flags.
   SDNodeFlags()
       : NoUnsignedWrap(false), NoSignedWrap(false), Exact(false), NoNaNs(false),
         NoInfs(false), NoSignedZeros(false), AllowReciprocal(false),
         AllowContract(false), ApproximateFuncs(false),
-        AllowReassociation(false), NoFPExcept(false) {}
+        AllowReassociation(false), NoFPExcept(false), Unpredictable(false) {}
 
   /// Propagate the fast-math-flags from an IR FPMathOperator.
   void copyFMF(const FPMathOperator &FPMO) {
     setNoNaNs(FPMO.hasNoNaNs());
     setNoInfs(FPMO.hasNoInfs());
     setNoSignedZeros(FPMO.hasNoSignedZeros());
     setAllowReciprocal(FPMO.hasAllowReciprocal());
     setAllowContract(FPMO.hasAllowContract());
     setApproximateFuncs(FPMO.hasApproxFunc());
     setAllowReassociation(FPMO.hasAllowReassoc());
   }
 
   // These are mutators for each flag.
   void setNoUnsignedWrap(bool b) { NoUnsignedWrap = b; }
   void setNoSignedWrap(bool b) { NoSignedWrap = b; }
   void setExact(bool b) { Exact = b; }
   void setNoNaNs(bool b) { NoNaNs = b; }
   void setNoInfs(bool b) { NoInfs = b; }
   void setNoSignedZeros(bool b) { NoSignedZeros = b; }
   void setAllowReciprocal(bool b) { AllowReciprocal = b; }
   void setAllowContract(bool b) { AllowContract = b; }
   void setApproximateFuncs(bool b) { ApproximateFuncs = b; }
   void setAllowReassociation(bool b) { AllowReassociation = b; }
   void setNoFPExcept(bool b) { NoFPExcept = b; }
+  void setUnpredictable(bool b) { Unpredictable = b; }
 
   // These are accessors for each flag.
   bool hasNoUnsignedWrap() const { return NoUnsignedWrap; }
   bool hasNoSignedWrap() const { return NoSignedWrap; }
   bool hasExact() const { return Exact; }
   bool hasNoNaNs() const { return NoNaNs; }
   bool hasNoInfs() const { return NoInfs; }
   bool hasNoSignedZeros() const { return NoSignedZeros; }
   bool hasAllowReciprocal() const { return AllowReciprocal; }
   bool hasAllowContract() const { return AllowContract; }
   bool hasApproximateFuncs() const { return ApproximateFuncs; }
   bool hasAllowReassociation() const { return AllowReassociation; }
   bool hasNoFPExcept() const { return NoFPExcept; }
+  bool hasUnpredictable() const { return Unpredictable; }
 
   /// Clear any flags in this flag set that aren't also set in Flags. All
   /// flags will be cleared if Flags are undefined.
   void intersectWith(const SDNodeFlags Flags) {
     NoUnsignedWrap &= Flags.NoUnsignedWrap;
     NoSignedWrap &= Flags.NoSignedWrap;
     Exact &= Flags.Exact;
     NoNaNs &= Flags.NoNaNs;
     NoInfs &= Flags.NoInfs;
     NoSignedZeros &= Flags.NoSignedZeros;
     AllowReciprocal &= Flags.AllowReciprocal;
     AllowContract &= Flags.AllowContract;
     ApproximateFuncs &= Flags.ApproximateFuncs;
     AllowReassociation &= Flags.AllowReassociation;
     NoFPExcept &= Flags.NoFPExcept;
+    Unpredictable &= Flags.Unpredictable;
   }
 };
 
 /// Represents one node in the SelectionDAG.
 ///
 class SDNode : public FoldingSetNode, public ilist_node<SDNode> {
 private:
   /// The operation that this node performs.

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

bool IRTranslator::translateBinaryOp(unsigned Opcode, const User &U,
                                      MachineIRBuilder &MIRBuilder) {
   // Get or create a virtual register for each value.
   // Unless the value is a Constant => loadimm cst?
   // or inline constant each time?
   // Creation of a virtual register needs to have a size.
   Register Op0 = getOrCreateVReg(*U.getOperand(0));
   Register Op1 = getOrCreateVReg(*U.getOperand(1));
   Register Res = getOrCreateVReg(U);
-  uint16_t Flags = 0;
+  uint32_t Flags = 0;
   if (isa<Instruction>(U)) {
     const Instruction &I = cast<Instruction>(U);
     Flags = MachineInstr::copyFlagsFromInstruction(I);
   }
 
   MIRBuilder.buildInstr(Opcode, {Res}, {Op0, Op1}, Flags);
   return true;
 }
 
 bool IRTranslator::translateUnaryOp(unsigned Opcode, const User &U,
                                     MachineIRBuilder &MIRBuilder) {
   Register Op0 = getOrCreateVReg(*U.getOperand(0));
   Register Res = getOrCreateVReg(U);
-  uint16_t Flags = 0;
+  uint32_t Flags = 0;
   if (isa<Instruction>(U)) {
     const Instruction &I = cast<Instruction>(U);
     Flags = MachineInstr::copyFlagsFromInstruction(I);
   }
   MIRBuilder.buildInstr(Opcode, {Res}, {Op0}, Flags);
   return true;
 }
 
if (CmpInst::isIntPredicate(Pred))
     MIRBuilder.buildICmp(Pred, Res, Op0, Op1);
   else if (Pred == CmpInst::FCMP_FALSE)
     MIRBuilder.buildCopy(
         Res, getOrCreateVReg(*Constant::getNullValue(U.getType())));
   else if (Pred == CmpInst::FCMP_TRUE)
     MIRBuilder.buildCopy(
         Res, getOrCreateVReg(*Constant::getAllOnesValue(U.getType())));
   else {
-    uint16_t Flags = 0;
+    uint32_t Flags = 0;
     if (CI)
       Flags = MachineInstr::copyFlagsFromInstruction(*CI);
     MIRBuilder.buildFCmp(Pred, Res, Op0, Op1, Flags);
   }
 
   return true;
 }
 

 
 bool IRTranslator::translateSelect(const User &U,
                                    MachineIRBuilder &MIRBuilder) {
   Register Tst = getOrCreateVReg(*U.getOperand(0));
   ArrayRef<Register> ResRegs = getOrCreateVRegs(U);
   ArrayRef<Register> Op0Regs = getOrCreateVRegs(*U.getOperand(1));
   ArrayRef<Register> Op1Regs = getOrCreateVRegs(*U.getOperand(2));
 
-  uint16_t Flags = 0;
+  uint32_t Flags = 0;
   if (const SelectInst *SI = dyn_cast<SelectInst>(&U))
     Flags = MachineInstr::copyFlagsFromInstruction(*SI);
 
   for (unsigned i = 0; i < ResRegs.size(); ++i) {
     MIRBuilder.buildSelect(ResRegs[i], Tst, Op0Regs[i], Op1Regs[i], Flags);
   }
 
   return true;

 bool IRTranslator::translateConstrainedFPIntrinsic(
   const ConstrainedFPIntrinsic &FPI, MachineIRBuilder &MIRBuilder) {
   fp::ExceptionBehavior EB = *FPI.getExceptionBehavior();
 
   unsigned Opcode = getConstrainedOpcode(FPI.getIntrinsicID());
   if (!Opcode)
     return false;
 
-  unsigned Flags = MachineInstr::copyFlagsFromInstruction(FPI);
+  uint32_t Flags = MachineInstr::copyFlagsFromInstruction(FPI);
   if (EB == fp::ExceptionBehavior::ebIgnore)
     Flags |= MachineInstr::NoFPExcept;
 
   SmallVector<llvm::SrcOp, 4> VRegs;
   VRegs.push_back(getOrCreateVReg(*FPI.getArgOperand(0)));
   if (!FPI.isUnaryOp())
     VRegs.push_back(getOrCreateVReg(*FPI.getArgOperand(1)));
   if (FPI.isTernaryOp())
if (ID == Intrinsic::ubsantrap) {
                                CI.getArgOperand(0)->getType(), 0});
     }
     Info.Callee = MachineOperand::CreateES(TrapFuncName.data());
     Info.CB = &CI;
     Info.OrigRet = {Register(), Type::getVoidTy(CI.getContext()), 0};
     return CLI->lowerCall(MIRBuilder, Info);
   }
   case Intrinsic::fptrunc_round: {
-    unsigned Flags = MachineInstr::copyFlagsFromInstruction(CI);
+    uint32_t Flags = MachineInstr::copyFlagsFromInstruction(CI);
 
     // Convert the metadata argument to a constant integer
     Metadata *MD = cast<MetadataAsValue>(CI.getArgOperand(1))->getMetadata();
     std::optional<RoundingMode> RoundMode =
         convertStrToRoundingMode(cast<MDString>(MD)->getString());
 
     // Add the Rounding mode as an integer
     MIRBuilder

llvm/lib/CodeGen/MIRParser/MILexer.h

enum TokenKind {
     kw_arcp,
     kw_contract,
     kw_afn,
     kw_reassoc,
     kw_nuw,
     kw_nsw,
     kw_exact,
     kw_nofpexcept,
+    kw_unpredictable,
     kw_debug_location,
     kw_debug_instr_number,
     kw_dbg_instr_ref,
     kw_cfi_same_value,
     kw_cfi_offset,
     kw_cfi_rel_offset,
     kw_cfi_def_cfa_register,
     kw_cfi_def_cfa_offset,

llvm/lib/CodeGen/MIRParser/MILexer.cpp

return StringSwitch<MIToken::TokenKind>(Identifier)
       .Case("arcp", MIToken::kw_arcp)
       .Case("contract", MIToken::kw_contract)
       .Case("afn", MIToken::kw_afn)
       .Case("reassoc", MIToken::kw_reassoc)
       .Case("nuw", MIToken::kw_nuw)
       .Case("nsw", MIToken::kw_nsw)
       .Case("exact", MIToken::kw_exact)
       .Case("nofpexcept", MIToken::kw_nofpexcept)
+      .Case("unpredictable", MIToken::kw_unpredictable)
       .Case("debug-location", MIToken::kw_debug_location)
       .Case("debug-instr-number", MIToken::kw_debug_instr_number)
       .Case("dbg-instr-ref", MIToken::kw_dbg_instr_ref)
       .Case("same_value", MIToken::kw_cfi_same_value)
       .Case("offset", MIToken::kw_cfi_offset)
       .Case("rel_offset", MIToken::kw_cfi_rel_offset)
       .Case("def_cfa_register", MIToken::kw_cfi_def_cfa_register)
       .Case("def_cfa_offset", MIToken::kw_cfi_def_cfa_offset)

llvm/lib/CodeGen/MIRParser/MIParser.cpp

while (Token.is(MIToken::kw_frame_setup) ||
          Token.is(MIToken::kw_nsz) ||
          Token.is(MIToken::kw_arcp) ||
          Token.is(MIToken::kw_contract) ||
          Token.is(MIToken::kw_afn) ||
          Token.is(MIToken::kw_reassoc) ||
          Token.is(MIToken::kw_nuw) ||
          Token.is(MIToken::kw_nsw) ||
          Token.is(MIToken::kw_exact) ||
-         Token.is(MIToken::kw_nofpexcept)) {
+         Token.is(MIToken::kw_nofpexcept) ||
+         Token.is(MIToken::kw_unpredictable)) {
     // Mine frame and fast math flags
     if (Token.is(MIToken::kw_frame_setup))
       Flags |= MachineInstr::FrameSetup;
     if (Token.is(MIToken::kw_frame_destroy))
       Flags |= MachineInstr::FrameDestroy;
     if (Token.is(MIToken::kw_nnan))
       Flags |= MachineInstr::FmNoNans;
     if (Token.is(MIToken::kw_ninf))
while (Token.is(MIToken::kw_frame_setup) ||
     if (Token.is(MIToken::kw_nuw))
       Flags |= MachineInstr::NoUWrap;
     if (Token.is(MIToken::kw_nsw))
       Flags |= MachineInstr::NoSWrap;
     if (Token.is(MIToken::kw_exact))
       Flags |= MachineInstr::IsExact;
     if (Token.is(MIToken::kw_nofpexcept))
       Flags |= MachineInstr::NoFPExcept;
+    if (Token.is(MIToken::kw_unpredictable))
+      Flags |= MachineInstr::Unpredictable;
 
     lex();
   }
   if (Token.isNot(MIToken::Identifier))
     return error("expected a machine instruction");
   StringRef InstrName = Token.stringValue();
   if (PFS.Target.parseInstrName(InstrName, OpCode))
     return error(Twine("unknown machine instruction name '") + InstrName + "'");

llvm/lib/CodeGen/MIRPrinter.cpp

void MIPrinter::print(const MachineInstr &MI) {
   if (MI.getFlag(MachineInstr::NoSWrap))
     OS << "nsw ";
   if (MI.getFlag(MachineInstr::IsExact))
     OS << "exact ";
   if (MI.getFlag(MachineInstr::NoFPExcept))
     OS << "nofpexcept ";
   if (MI.getFlag(MachineInstr::NoMerge))
     OS << "nomerge ";
+  if (MI.getFlag(MachineInstr::Unpredictable))
+    OS << "unpredictable ";
 
   OS << TII->getName(MI.getOpcode());
   if (I < E)
     OS << ' ';
 
   bool NeedComma = false;
   for (; I < E; ++I) {
     if (NeedComma)

llvm/lib/CodeGen/MachineInstr.cpp

assert(&MF == MI.getMF() &&
          "Invalid machine functions when cloning instruction symbols!");
 
   setPreInstrSymbol(MF, MI.getPreInstrSymbol());
   setPostInstrSymbol(MF, MI.getPostInstrSymbol());
   setHeapAllocMarker(MF, MI.getHeapAllocMarker());
   setPCSections(MF, MI.getPCSections());
 }
 
-uint16_t MachineInstr::mergeFlagsWith(const MachineInstr &Other) const {
+uint32_t MachineInstr::mergeFlagsWith(const MachineInstr &Other) const {
   // For now, the just return the union of the flags. If the flags get more
   // complicated over time, we might need more logic here.
   return getFlags() | Other.getFlags();
 }
 
-uint16_t MachineInstr::copyFlagsFromInstruction(const Instruction &I) {
-  uint16_t MIFlags = 0;
+uint32_t MachineInstr::copyFlagsFromInstruction(const Instruction &I) {
+  uint32_t MIFlags = 0;
   // Copy the wrapping flags.
   if (const OverflowingBinaryOperator *OB =
           dyn_cast<OverflowingBinaryOperator>(&I)) {
     if (OB->hasNoSignedWrap())
       MIFlags |= MachineInstr::MIFlag::NoSWrap;
     if (OB->hasNoUnsignedWrap())
       MIFlags |= MachineInstr::MIFlag::NoUWrap;
   }
if (const FPMathOperator *FP = dyn_cast<FPMathOperator>(&I)) {
     if (Flags.allowContract())
       MIFlags |= MachineInstr::MIFlag::FmContract;
     if (Flags.approxFunc())
       MIFlags |= MachineInstr::MIFlag::FmAfn;
     if (Flags.allowReassoc())
       MIFlags |= MachineInstr::MIFlag::FmReassoc;
   }
 
+  if (I.getMetadata(LLVMContext::MD_unpredictable))
+    MIFlags |= MachineInstr::MIFlag::Unpredictable;
+
   return MIFlags;
 }
 
 void MachineInstr::copyIRFlags(const Instruction &I) {
   Flags = copyFlagsFromInstruction(I);
 }
 
 bool MachineInstr::hasPropertyInBundle(uint64_t Mask, QueryType Type) const {

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

if (NumResults) {
     if (Flags.hasNoSignedWrap())
       MI->setFlag(MachineInstr::MIFlag::NoSWrap);
 
     if (Flags.hasExact())
       MI->setFlag(MachineInstr::MIFlag::IsExact);
 
     if (Flags.hasNoFPExcept())
       MI->setFlag(MachineInstr::MIFlag::NoFPExcept);
+
+    if (Flags.hasUnpredictable())
+      MI->setFlag(MachineInstr::MIFlag::Unpredictable);
   }
 
   // Emit all of the actual operands of this instruction, adding them to the
   // instruction as appropriate.
   bool HasOptPRefs = NumDefs > NumResults;
   assert((!HasOptPRefs || !HasPhysRegOuts) &&
          "Unable to cope with optional defs and phys regs defs!");
   unsigned NumSkip = HasOptPRefs ? NumDefs - NumResults : 0;

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

void SelectionDAGBuilder::visitSelect(const User &I) {
 
   bool IsUnaryAbs = false;
   bool Negate = false;
 
   SDNodeFlags Flags;
   if (auto *FPOp = dyn_cast<FPMathOperator>(&I))
     Flags.copyFMF(*FPOp);
 
+  Flags.setUnpredictable(
+      cast<SelectInst>(I).getMetadata(LLVMContext::MD_unpredictable));
+
   // Min/max matching is only viable if all output VTs are the same.
   if (all_equal(ValueVTs)) {
     EVT VT = ValueVTs[0];
     LLVMContext &Ctx = *DAG.getContext();
     auto &TLI = DAG.getTargetLoweringInfo();
 
     // We care about the legality of the operation after it has been type
     // legalized.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

void AArch64InstrInfo::genAlternativeCodeSequence(
   } // end switch (Pattern)
   // Record MUL and ADD/SUB for deletion
   if (MUL)
     DelInstrs.push_back(MUL);
   DelInstrs.push_back(&Root);
 
   // Set the flags on the inserted instructions to be the merged flags of the
   // instructions that we have combined.
-  uint16_t Flags = Root.getFlags();
+  uint32_t Flags = Root.getFlags();
   if (MUL)
     Flags = Root.mergeFlagsWith(*MUL);
   for (auto *MI : InsInstrs)
     MI->setFlags(Flags);
 }
 
 /// Replace csincr-branch sequence by simple conditional branch
 ///

llvm/lib/Target/PowerPC/PPCInstrInfo.h

#endif
   int getExtendResourceLenLimit() const override { return 1; }
 
   void setSpecialOperandAttr(MachineInstr &OldMI1, MachineInstr &OldMI2,
                              MachineInstr &NewMI1,
                              MachineInstr &NewMI2) const override;
 
   // PowerPC specific version of setSpecialOperandAttr that copies Flags to MI
   // and clears nuw, nsw, and exact flags.
-  void setSpecialOperandAttr(MachineInstr &MI, uint16_t Flags) const;
+  void setSpecialOperandAttr(MachineInstr &MI, uint32_t Flags) const;
 
   bool isCoalescableExtInstr(const MachineInstr &MI,
                              Register &SrcReg, Register &DstReg,
                              unsigned &SubIdx) const override;
   unsigned isLoadFromStackSlot(const MachineInstr &MI,
                                int &FrameIndex) const override;
   bool isReallyTriviallyReMaterializable(const MachineInstr &MI) const override;
   unsigned isStoreToStackSlot(const MachineInstr &MI,

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

 /// This is an architecture-specific helper function of reassociateOps.
 /// Set special operand attributes for new instructions after reassociation.
 void PPCInstrInfo::setSpecialOperandAttr(MachineInstr &OldMI1,
                                          MachineInstr &OldMI2,
                                          MachineInstr &NewMI1,
                                          MachineInstr &NewMI2) const {
   // Propagate FP flags from the original instructions.
   // But clear poison-generating flags because those may not be valid now.
-  uint16_t IntersectedFlags = OldMI1.getFlags() & OldMI2.getFlags();
+  uint32_t IntersectedFlags = OldMI1.getFlags() & OldMI2.getFlags();
   NewMI1.setFlags(IntersectedFlags);
   NewMI1.clearFlag(MachineInstr::MIFlag::NoSWrap);
   NewMI1.clearFlag(MachineInstr::MIFlag::NoUWrap);
   NewMI1.clearFlag(MachineInstr::MIFlag::IsExact);
 
   NewMI2.setFlags(IntersectedFlags);
   NewMI2.clearFlag(MachineInstr::MIFlag::NoSWrap);
   NewMI2.clearFlag(MachineInstr::MIFlag::NoUWrap);
   NewMI2.clearFlag(MachineInstr::MIFlag::IsExact);
 }
 
 void PPCInstrInfo::setSpecialOperandAttr(MachineInstr &MI,
-                                         uint16_t Flags) const {
+                                         uint32_t Flags) const {
   MI.setFlags(Flags);
   MI.clearFlag(MachineInstr::MIFlag::NoSWrap);
   MI.clearFlag(MachineInstr::MIFlag::NoUWrap);
   MI.clearFlag(MachineInstr::MIFlag::IsExact);
 }
 
 // This function does not list all associative and commutative operations, but
 // only those worth feeding through the machine combiner in an attempt to
void PPCInstrInfo::reassociateFMA(
   case MachineCombinerPattern::REASSOC_XY_BCA: {
     Register MULReg = TRI->lookThruCopyLike(
         Root.getOperand(FirstMulOpIdx + 1).getReg(), &MRI);
     Leaf = MRI.getVRegDef(MULReg);
     break;
   }
   }
 
-  uint16_t IntersectedFlags = 0;
+  uint32_t IntersectedFlags = 0;
   if (IsILPReassociate)
     IntersectedFlags = Root.getFlags() & Prev->getFlags() & Leaf->getFlags();
   else
     IntersectedFlags = Root.getFlags() & Leaf->getFlags();
 
   auto GetOperandInfo = [&](const MachineOperand &Operand, Register &Reg,
                             bool &KillFlag) {
     Reg = Operand.getReg();

llvm/lib/Target/RISCV/RISCVInstrInfo.cpp

MachineTraceStrategy RISCVInstrInfo::getMachineCombinerTraceStrategy() const {
   // The strategy was forced by the option.
   return ForceMachineCombinerStrategy;
 }
 
 void RISCVInstrInfo::setSpecialOperandAttr(MachineInstr &OldMI1,
                                            MachineInstr &OldMI2,
                                            MachineInstr &NewMI1,
                                            MachineInstr &NewMI2) const {
-  uint16_t IntersectedFlags = OldMI1.getFlags() & OldMI2.getFlags();
+  uint32_t IntersectedFlags = OldMI1.getFlags() & OldMI2.getFlags();
   NewMI1.setFlags(IntersectedFlags);
   NewMI2.setFlags(IntersectedFlags);
 }
 
 void RISCVInstrInfo::finalizeInsInstrs(
     MachineInstr &Root, MachineCombinerPattern &P,
     SmallVectorImpl<MachineInstr *> &InsInstrs) const {
   int16_t FrmOpIdx =
static void combineFPFusedMultiply(MachineInstr &Root, MachineInstr &Prev,
 
   MachineOperand &Mul1 = Prev.getOperand(1);
   MachineOperand &Mul2 = Prev.getOperand(2);
   MachineOperand &Dst = Root.getOperand(0);
   MachineOperand &Addend = Root.getOperand(getAddendOperandIdx(Pattern));
 
   Register DstReg = Dst.getReg();
   unsigned FusedOpc = getFPFusedMultiplyOpcode(Root.getOpcode(), Pattern);
-  auto IntersectedFlags = Root.getFlags() & Prev.getFlags();
+  uint32_t IntersectedFlags = Root.getFlags() & Prev.getFlags();
   DebugLoc MergedLoc =
       DILocation::getMergedLocation(Root.getDebugLoc(), Prev.getDebugLoc());
 
   MachineInstrBuilder MIB =
       BuildMI(*MF, MergedLoc, TII->get(FusedOpc), DstReg)
           .addReg(Mul1.getReg(), getKillRegState(Mul1.isKill()))
           .addReg(Mul2.getReg(), getKillRegState(Mul2.isKill()))
           .addReg(Addend.getReg(), getKillRegState(Addend.isKill()))

llvm/lib/Target/SPIRV/SPIRVPreLegalizer.cpp

Register insertAssignInstr(Register Reg, Type *Ty, SPIRVType *SpirvTy,
   }
   SpirvTy = SpirvTy ? SpirvTy : GR->getOrCreateSPIRVType(Ty, MIB);
   GR->assignSPIRVTypeToVReg(SpirvTy, Reg, MIB.getMF());
   // This is to make it convenient for Legalizer to get the SPIRVType
   // when processing the actual MI (i.e. not pseudo one).
   GR->assignSPIRVTypeToVReg(SpirvTy, NewReg, MIB.getMF());
   // Copy MIFlags from Def to ASSIGN_TYPE instruction. It's required to keep
   // the flags after instruction selection.
-  const uint16_t Flags = Def->getFlags();
+  const uint32_t Flags = Def->getFlags();
   MIB.buildInstr(SPIRV::ASSIGN_TYPE)
       .addDef(Reg)
       .addUse(NewReg)
       .addUse(GR->getSPIRVTypeID(SpirvTy))
       .setMIFlags(Flags);
   Def->getOperand(0).setReg(NewReg);
   return NewReg;
 }

llvm/lib/Target/X86/X86CmovConversion.cpp

for (auto *MBB : Blocks) {
     bool FoundNonCMOVInst = false;
     // Indicator for current processed CMOV-group if it should be skipped.
     bool SkipGroup = false;
 
     for (auto &I : *MBB) {
       // Skip debug instructions.
       if (I.isDebugInstr())
         continue;
+
       X86::CondCode CC = X86::getCondFromCMov(I);
-      // Check if we found a X86::CMOVrr instruction.
-      if (CC != X86::COND_INVALID && (IncludeLoads || !I.mayLoad())) {
+      // Check if we found a X86::CMOVrr instruction. If it is marked as
+      // unpredictable, skip it and do not convert it to branch.
+      if (CC != X86::COND_INVALID &&
+          !I.getFlag(MachineInstr::MIFlag::Unpredictable) &&
+          (IncludeLoads || !I.mayLoad())) {
         if (Group.empty()) {
           // We found first CMOV in the range, reset flags.
           FirstCC = CC;
           FirstOppCC = X86::GetOppositeBranchCondition(CC);
           // Clear out the prior group's memory operand CC.
           MemOpCC = X86::COND_INVALID;
           FoundNonCMOVInst = false;
           SkipGroup = false;

llvm/lib/Target/X86/X86ISelLowering.cpp

SDValue X86TargetLowering::LowerSELECT(SDValue Op, SelectionDAG &DAG) const {
   if (Op.getValueType() == MVT::i8 &&
       Op1.getOpcode() == ISD::TRUNCATE && Op2.getOpcode() == ISD::TRUNCATE) {
     SDValue T1 = Op1.getOperand(0), T2 = Op2.getOperand(0);
     if (T1.getValueType() == T2.getValueType() &&
         // Exclude CopyFromReg to avoid partial register stalls.
         T1.getOpcode() != ISD::CopyFromReg && T2.getOpcode()!=ISD::CopyFromReg){
       SDValue Cmov = DAG.getNode(X86ISD::CMOV, DL, T1.getValueType(), T2, T1,
                                  CC, Cond);
       return DAG.getNode(ISD::TRUNCATE, DL, Op.getValueType(), Cmov);
     }
   }
 
   // Or finally, promote i8 cmovs if we have CMOV,
   //                 or i16 cmovs if it won't prevent folding a load.
   // FIXME: we should not limit promotion of i8 case to only when the CMOV is
   //        legal, but EmitLoweredSelect() can not deal with these extensions
   //        being inserted between two CMOV's. (in i16 case too TBN)
   //        https://bugs.llvm.org/show_bug.cgi?id=40974
   if ((Op.getValueType() == MVT::i8 && Subtarget.canUseCMOV()) ||
       (Op.getValueType() == MVT::i16 && !X86::mayFoldLoad(Op1, Subtarget) &&
        !X86::mayFoldLoad(Op2, Subtarget))) {
     Op1 = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i32, Op1);
     Op2 = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i32, Op2);
     SDValue Ops[] = { Op2, Op1, CC, Cond };
     SDValue Cmov = DAG.getNode(X86ISD::CMOV, DL, MVT::i32, Ops);
     return DAG.getNode(ISD::TRUNCATE, DL, Op.getValueType(), Cmov);
   }
 
   // X86ISD::CMOV means set the result (which is operand 1) to the RHS if
   // condition is true.
   SDValue Ops[] = { Op2, Op1, CC, Cond };
-  return DAG.getNode(X86ISD::CMOV, DL, Op.getValueType(), Ops);
+  return DAG.getNode(X86ISD::CMOV, DL, Op.getValueType(), Ops, Op->getFlags());
 }
 
 static SDValue LowerSIGN_EXTEND_Mask(SDValue Op,
                                      const X86Subtarget &Subtarget,
                                      SelectionDAG &DAG) {
   MVT VT = Op->getSimpleValueType(0);
   SDValue In = Op->getOperand(0);
   MVT InVT = In.getSimpleValueType();

llvm/lib/Target/X86/X86InstrInfo.cpp

 /// Set special operand attributes for new instructions after reassociation.
 void X86InstrInfo::setSpecialOperandAttr(MachineInstr &OldMI1,
                                          MachineInstr &OldMI2,
                                          MachineInstr &NewMI1,
                                          MachineInstr &NewMI2) const {
   // Propagate FP flags from the original instructions.
   // But clear poison-generating flags because those may not be valid now.
   // TODO: There should be a helper function for copying only fast-math-flags.
-  uint16_t IntersectedFlags = OldMI1.getFlags() & OldMI2.getFlags();
+  uint32_t IntersectedFlags = OldMI1.getFlags() & OldMI2.getFlags();
   NewMI1.setFlags(IntersectedFlags);
   NewMI1.clearFlag(MachineInstr::MIFlag::NoSWrap);
   NewMI1.clearFlag(MachineInstr::MIFlag::NoUWrap);
   NewMI1.clearFlag(MachineInstr::MIFlag::IsExact);
 
   NewMI2.setFlags(IntersectedFlags);
   NewMI2.clearFlag(MachineInstr::MIFlag::NoSWrap);
   NewMI2.clearFlag(MachineInstr::MIFlag::NoUWrap);

llvm/test/CodeGen/X86/x86-cmov-converter.ll

for.body:                                         ; preds = %for.body.preheader, %for.body
   %cmp3 = icmp sgt i32 %0, %1
   %2 = trunc i64 %indvars.iv to i32
   %i.0.t.0 = select i1 %cmp3, i32 %2, i32 %t.015
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
   %exitcond = icmp eq i64 %indvars.iv.next, %wide.trip.count
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
-; TODO: If cmov instruction is marked as unpredicatable, do not convert it to branch.
+; If cmov instruction is marked as unpredictable, do not convert it to branch.
 define i32 @MaxIndex_unpredictable(i32 %n, ptr nocapture readonly %a) #0 {
 ; CHECK-LABEL: MaxIndex_unpredictable:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    cmpl $2, %edi
-; CHECK-NEXT:    jl .LBB3_5
+; CHECK-NEXT:    jl .LBB3_3
 ; CHECK-NEXT:  # %bb.1: # %for.body.preheader
 ; CHECK-NEXT:    movl %edi, %ecx
-; CHECK-NEXT:    xorl %edi, %edi
+; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    movl $1, %edx
 ; CHECK-NEXT:  .LBB3_2: # %for.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    movl (%rsi,%rdx,4), %r8d
-; CHECK-NEXT:    movslq %edi, %r9
-; CHECK-NEXT:    movl %edx, %eax
-; CHECK-NEXT:    cmpl (%rsi,%r9,4), %r8d
-; CHECK-NEXT:    jg .LBB3_4
-; CHECK-NEXT:  # %bb.3: # %for.body
-; CHECK-NEXT:    # in Loop: Header=BB3_2 Depth=1
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:  .LBB3_4: # %for.body
-; CHECK-NEXT:    # in Loop: Header=BB3_2 Depth=1
+; CHECK-NEXT:    movl (%rsi,%rdx,4), %edi
+; CHECK-NEXT:    cltq
+; CHECK-NEXT:    cmpl (%rsi,%rax,4), %edi
+; CHECK-NEXT:    cmovgl %edx, %eax
 ; CHECK-NEXT:    addq $1, %rdx
-; CHECK-NEXT:    movl %eax, %edi
 ; CHECK-NEXT:    cmpq %rdx, %rcx
 ; CHECK-NEXT:    jne .LBB3_2
-; CHECK-NEXT:  .LBB3_5: # %for.cond.cleanup
+; CHECK-NEXT:  .LBB3_3: # %for.cond.cleanup
+; CHECK-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-NEXT:    retq
 ;
 ; CHECK-FORCEALL-LABEL: MaxIndex_unpredictable:
 ; CHECK-FORCEALL:       # %bb.0: # %entry
 ; CHECK-FORCEALL-NEXT:    xorl %eax, %eax
 ; CHECK-FORCEALL-NEXT:    cmpl $2, %edi
-; CHECK-FORCEALL-NEXT:    jl .LBB3_5
+; CHECK-FORCEALL-NEXT:    jl .LBB3_3
 ; CHECK-FORCEALL-NEXT:  # %bb.1: # %for.body.preheader
 ; CHECK-FORCEALL-NEXT:    movl %edi, %ecx
-; CHECK-FORCEALL-NEXT:    xorl %edi, %edi
+; CHECK-FORCEALL-NEXT:    xorl %eax, %eax
 ; CHECK-FORCEALL-NEXT:    movl $1, %edx
 ; CHECK-FORCEALL-NEXT:  .LBB3_2: # %for.body
 ; CHECK-FORCEALL-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-FORCEALL-NEXT:    movl (%rsi,%rdx,4), %r8d
-; CHECK-FORCEALL-NEXT:    movslq %edi, %r9
-; CHECK-FORCEALL-NEXT:    movl %edx, %eax
-; CHECK-FORCEALL-NEXT:    cmpl (%rsi,%r9,4), %r8d
-; CHECK-FORCEALL-NEXT:    jg .LBB3_4
-; CHECK-FORCEALL-NEXT:  # %bb.3: # %for.body
-; CHECK-FORCEALL-NEXT:    # in Loop: Header=BB3_2 Depth=1
-; CHECK-FORCEALL-NEXT:    movl %edi, %eax
-; CHECK-FORCEALL-NEXT:  .LBB3_4: # %for.body
-; CHECK-FORCEALL-NEXT:    # in Loop: Header=BB3_2 Depth=1
+; CHECK-FORCEALL-NEXT:    movl (%rsi,%rdx,4), %edi
+; CHECK-FORCEALL-NEXT:    cltq
+; CHECK-FORCEALL-NEXT:    cmpl (%rsi,%rax,4), %edi
+; CHECK-FORCEALL-NEXT:    cmovgl %edx, %eax
 ; CHECK-FORCEALL-NEXT:    addq $1, %rdx
-; CHECK-FORCEALL-NEXT:    movl %eax, %edi
 ; CHECK-FORCEALL-NEXT:    cmpq %rdx, %rcx
 ; CHECK-FORCEALL-NEXT:    jne .LBB3_2
-; CHECK-FORCEALL-NEXT:  .LBB3_5: # %for.cond.cleanup
+; CHECK-FORCEALL-NEXT:  .LBB3_3: # %for.cond.cleanup
+; CHECK-FORCEALL-NEXT:    # kill: def $eax killed $eax killed $rax
 ; CHECK-FORCEALL-NEXT:    retq
 entry:
   %cmp14 = icmp sgt i32 %n, 1
   br i1 %cmp14, label %for.body.preheader, label %for.cond.cleanup
 
 for.body.preheader:                               ; preds = %entry
   %wide.trip.count = zext i32 %n to i64
   br label %for.body

 ; CHECK-FORCEALL-NEXT:    retq
 entry:
   %cond = icmp ugt i32 %a, %b
   %load = load i32, ptr %y
   %z = select i1 %cond, i32 %x, i32 %load
   ret i32 %z
 }
 
-; TODO: If cmov instruction is marked as unpredicatable, do not convert it to branch.
+; If cmov instruction is marked as unpredictable, do not convert it to branch.
 define i32 @test_cmov_memoperand_unpredictable(i32 %a, i32 %b, i32 %x, ptr %y) #0 {
 ; CHECK-LABEL: test_cmov_memoperand_unpredictable:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movl %edx, %eax
 ; CHECK-NEXT:    cmpl %esi, %edi
-; CHECK-NEXT:    ja .LBB8_2
-; CHECK-NEXT:  # %bb.1: # %entry
-; CHECK-NEXT:    movl (%rcx), %eax
-; CHECK-NEXT:  .LBB8_2: # %entry
+; CHECK-NEXT:    cmovbel (%rcx), %eax
 ; CHECK-NEXT:    retq
 ;
 ; CHECK-FORCEALL-LABEL: test_cmov_memoperand_unpredictable:
 ; CHECK-FORCEALL:       # %bb.0: # %entry
 ; CHECK-FORCEALL-NEXT:    movl %edx, %eax
 ; CHECK-FORCEALL-NEXT:    cmpl %esi, %edi
-; CHECK-FORCEALL-NEXT:    ja .LBB8_2
-; CHECK-FORCEALL-NEXT:  # %bb.1: # %entry
-; CHECK-FORCEALL-NEXT:    movl (%rcx), %eax
-; CHECK-FORCEALL-NEXT:  .LBB8_2: # %entry
+; CHECK-FORCEALL-NEXT:    cmovbel (%rcx), %eax
 ; CHECK-FORCEALL-NEXT:    retq
 entry:
   %cond = icmp ugt i32 %a, %b
   %load = load i32, ptr %y
   %z = select i1 %cond, i32 %x, i32 %load, !unpredictable !0
   ret i32 %z
 }