[X86] Added support for nocf_check attribute for indirect Branch Tracking

[oren_ben_simhon]

Summary

Jump Oriented Programming attacks rely on tampering addresses used by indirect call / jmp, e.g. redirect control-flow to non-programmer intended bytes in binary.
X86 Supports Indirect Branch Tracking (IBT) as part of Control-Flow Enforcement Technology (CET).
IBT instruments ENDBR instructions used to specify valid targets of indirect call / jmp.

The `nocf_check` attribute has two roles in the context of X86 IBT technology:

1. Appertains to a function - do not add ENDBR instruction at the beginning of the function.
2. Appertains to a function pointer - do not track the target function of this pointer by adding nocf_check prefix to the indirect-call instruction.

When the CPU decodes `nocf_check` prefix, it will not update IBT state machine, hence, the target addresses of the following indirect jump will not be tracked.
So in that case there is no need for ENDBR instructions.

The patch implements `nocf_check` context for Indirect Branch Tracking.
It also auto generates `nocf_check` prefixes before indirect branchs to jump tables that are guarded by range checks.
Those cases are common in switch-case statements and it is safe to optimize them.
Meaning instead of adding many ENDBR instructions for each target address of a case statement, we add a single`nocf_check` prefix before the indirect jump of the switch statement.

[craig.topper]

lib/Target/X86/X86InstrSystem.td

564 ↗

(On Diff #129146)

I'm not sure I like feel very safe emitting the DS_PREFIX as a separate instruction. The glue won't exist after the SelectionDAG is converted to MIR so I'm not sure we would be guaranteed to keep it together.

I think I'd rather see special flavors of CALL*r/JMP*r that have a DS_PREFIX bit set in the TSFlags for their opcode and then the code emitter would know to emit the prefix. Basically what we do with LOCK and REP today. That way the instruction is one machine IR instruction that can never be separated.

[oren_ben_simhon]

lib/Target/X86/X86InstrSystem.td

564 ↗

(On Diff #129146)

I share your concern. That is why I added the glue (for SelectionDAG) and X86nocf_check_calljmp pattern (for MI), which will make sure that only DS_PREFIX that precedes indirect jmp/call will be lowered.

Your solution means that i need to double the number of JMP/CALL MIs and DAG nodes. correct?
I am not sure that the cost is fair for esoteric feature like nocf_check prefix.

If you are still worried about future passes that change the order, what do you think about creating a bundle instead (DS_PREFIX<->JMP/CALL)?

[craig.topper]

Should we be printing the DS_PREFIX as "notrack" like gcc?

lib/Target/X86/X86InstrSystem.td

564 ↗

(On Diff #129146)

I don't think we've ever used a bundle in x86 before.

Is there any kind of flag we could add to the MachineInstruction that we could expand in X86AsmPrinter when we're converting to MCInst.

[oren_ben_simhon]

In D41879#978046, @craig.topper wrote:

Should we be printing the DS_PREFIX as "notrack" like gcc?

In LLVM you can't print two different instructions (notrack and ds) for the same encoding. The only solution to that is creating separate call/jmp instructions (just like lock/rep).
You can't use flags (TSFlags or MIFlags) to change the instruction encoding, you must use different call/jmp instructions.

So we are down to 2 options:

1. Remain as is and benefit from call/jmp optimization with the risk of MI optimization that will change the order after instruction selection (unless we try using bundle).
2. Use same mechanism as lock/rep, multiple the number of call/jmp instructions and ISEL nodes but have the flexibility to emit notrack instead of ds.

You suggest to use the second option. are we in sync?

[craig.topper]

You can have two instructions use the same encoding if you mark one with isAsmParserOnly = 1. See XACQUIRE_PREFIX and XRELEASE_PREFIX. isAsmParserOnly will hide from the disassembler that checks encoding collisions.

For sure you can't emit two instructions based on TSFlags.

I thought maybe with MIFlags you could catch the opcodes in X86AsmPrinter.cpp just before they are emitted and emit two instructions instead of one based on the MIFlag. It would be target specific code.

But I think separate opcodes is probably the easiest solution.

@RKSimon do you have any thoughts on this?

[oren_ben_simhon]

Thanks Craig for your comments, i made the following changes:

• I print notrack prefix instead of ds prefix
• I used same mechanism as lock/rep and created unified SDNode for the prefix and the jmp/call

[craig.topper]

I think you need to add support to the assembly parser to read assembly with notrack in it as well.

lib/Target/X86/X86ISelLowering.cpp

37874 ↗

(On Diff #131074)

createds is mispelled.

lib/Target/X86/X86ISelLowering.h

1110 ↗

(On Diff #131074)

SDLoc should be passed by const reference I think.

lib/Target/X86/X86IndirectBranchTracking.cpp

146 ↗

(On Diff #131074)

Is this TODO implemented by this patch? I didn't see anywhere that checks for the number of destination.

test/CodeGen/X86/indirect-branch-tracking.ll

8 ↗

(On Diff #131074)

DS_PREFIX should be NOTRACK

[oren_ben_simhon]

I will add "notrack" to X86 ASM Parser (and also add symbolic test)

lib/Target/X86/X86IndirectBranchTracking.cpp

146 ↗

(On Diff #131074)

I decided to follow a better approach. I don't add ENDBR to guarded switch-case. Instead adds NOTRACK prefix when the switch-case is safe.

A safe/guarded switch-case is a an indirect branch that uses a jump table and does a range check before the branch. It usually happens in large switch-case statement.

So now, I change ISD::BRIND to ISD::NT_BRIND every time the branch originates from a jump table (ISD::BR_JT).
ISD::NT_BRIND will have no track prefix before it.

In case of jump tables, the target addresses of an indirect branch are not flagged as "address taken". So no ENDBR will be instrumented.

[oren_ben_simhon]

Implemented comments posted until 01/28 (Thanks Craig)

[craig.topper]

lib/Target/X86/X86InstrControl.td

245 ↗

(On Diff #131806)

Can you name these as CALL16r_NT, etc? I think then the regular expressions in the scheduler models will pick them up and group them with their counterparts without the prefix.

[craig.topper]

The new attribute needs to be added to the LangRef.

[craig.topper]

Also I think new attributes are usually sent for RFC on llvm-dev. Was that done here?

[oren_ben_simhon]

Implemented comments posted until 01/30 (Thanks Craig)

[oren_ben_simhon]

ping

[craig.topper]

LGTM

llvm/trunk/bindings/go/llvm/ir_test.go

attrTests := []string{
     "sspreq",
     "sspstrong",
     "sret",
     "sanitize_thread",
     "sanitize_memory",
     "uwtable",
     "zeroext",
     "cold",
+    "nocf_check"
   }
 
   for _, name := range attrTests {
     testAttribute(t, name)
   }
 }
 
 func TestDebugLoc(t *testing.T) {

llvm/trunk/docs/BitCodeFormat.rst

 * code 48: ``norecurse``
 * code 49: ``inaccessiblememonly``
 * code 50: ``inaccessiblememonly_or_argmemonly``
 * code 51: ``allocsize(<EltSizeParam>[, <NumEltsParam>])``
 * code 52: ``writeonly``
 * code 53: ``speculatable``
 * code 54: ``strictfp``
 * code 55: ``sanitize_hwaddress``
+* code 56: ``nocf_check``
 
 .. note::
   The ``allocsize`` attribute has a special encoding for its arguments. Its two
   arguments, which are 32-bit integers, are packed into one 64-bit integer value
   (i.e. ``(EltSizeParam << 32) | NumEltsParam``), with ``NumEltsParam`` taking on
   the sentinel value -1 if it is not specified.
 
 .. _TYPE_BLOCK:

llvm/trunk/docs/LangRef.rst

``"thunk"``
     optimization purposes. The caller is expected to cast the thunk prototype to
     match the thunk target prototype.
 ``uwtable``
     This attribute indicates that the ABI being targeted requires that
     an unwind table entry be produced for this function even if we can
     show that no exceptions passes by it. This is normally the case for
     the ELF x86-64 abi, but it can be disabled for some compilation
     units.
+``nocf_check``
+    This attribute indicates that no control-flow check will be perfomed on
+    the attributed entity. It disables -fcf-protection=<> for a specific
+    entity to fine grain the HW control flow protection mechanism. The flag
+    is target independant and currently appertains to a function or function
+    pointer.
 
 .. _glattrs:
 
 Global Attributes
 -----------------
 
 Attributes may be set to communicate additional information about a global variable.
 Unlike :ref:`function attributes <fnattrs>`, attributes on a global variable

llvm/trunk/include/llvm/Bitcode/LLVMBitCodes.h

enum AttributeKindCodes {
   ATTR_KIND_NO_RECURSE = 48,
   ATTR_KIND_INACCESSIBLEMEM_ONLY = 49,
   ATTR_KIND_INACCESSIBLEMEM_OR_ARGMEMONLY = 50,
   ATTR_KIND_ALLOC_SIZE = 51,
   ATTR_KIND_WRITEONLY = 52,
   ATTR_KIND_SPECULATABLE = 53,
   ATTR_KIND_STRICT_FP = 54,
   ATTR_KIND_SANITIZE_HWADDRESS = 55,
+  ATTR_KIND_NOCF_CHECK = 56,
 };
 
 enum ComdatSelectionKindCodes {
   COMDAT_SELECTION_KIND_ANY = 1,
   COMDAT_SELECTION_KIND_EXACT_MATCH = 2,
   COMDAT_SELECTION_KIND_LARGEST = 3,
   COMDAT_SELECTION_KIND_NO_DUPLICATES = 4,
   COMDAT_SELECTION_KIND_SAME_SIZE = 5,

llvm/trunk/include/llvm/CodeGen/TargetLowering.h

virtual SDValue emitStackGuardXorFP(SelectionDAG &DAG, SDValue Val,
                                       const SDLoc &DL) const {
     llvm_unreachable("not implemented for this target");
   }
 
   /// Lower TLS global address SDNode for target independent emulated TLS model.
   virtual SDValue LowerToTLSEmulatedModel(const GlobalAddressSDNode *GA,
                                           SelectionDAG &DAG) const;
 
+  /// Expands target specific indirect branch for the case of JumpTable
+  /// expanasion.
+  virtual SDValue expandIndirectJTBranch(const SDLoc& dl, SDValue Value, SDValue Addr,
+                                         SelectionDAG &DAG) const {
+    return DAG.getNode(ISD::BRIND, dl, MVT::Other, Value, Addr);
+  }
+
   // seteq(x, 0) -> truncate(srl(ctlz(zext(x)), log2(#bits)))
   // If we're comparing for equality to zero and isCtlzFast is true, expose the
   // fact that this can be implemented as a ctlz/srl pair, so that the dag
   // combiner can fold the new nodes.
   SDValue lowerCmpEqZeroToCtlzSrl(SDValue Op, SelectionDAG &DAG) const;
 
 private:
   SDValue simplifySetCCWithAnd(EVT VT, SDValue N0, SDValue N1,

llvm/trunk/include/llvm/IR/Attributes.td

 def NoRecurse : EnumAttr<"norecurse">;
 
 /// Disable redzone.
 def NoRedZone : EnumAttr<"noredzone">;
 
 /// Mark the function as not returning.
 def NoReturn : EnumAttr<"noreturn">;
 
+/// Disable Indirect Branch Tracking.
+def NoCfCheck : EnumAttr<"nocf_check">;
+
 /// Function doesn't unwind stack.
 def NoUnwind : EnumAttr<"nounwind">;
 
 /// opt_size.
 def OptimizeForSize : EnumAttr<"optsize">;
 
 /// Function must not be optimized.
 def OptimizeNone : EnumAttr<"optnone">;

llvm/trunk/include/llvm/IR/Function.h

public:
   /// @brief Determine if the function cannot return.
   bool doesNotReturn() const {
     return hasFnAttribute(Attribute::NoReturn);
   }
   void setDoesNotReturn() {
     addFnAttr(Attribute::NoReturn);
   }
 
+  /// Determine if the function should not perform indirect branch tracking.
+  bool doesNoCfCheck() const { return hasFnAttribute(Attribute::NoCfCheck); }
+
   /// @brief Determine if the function cannot unwind.
   bool doesNotThrow() const {
     return hasFnAttribute(Attribute::NoUnwind);
   }
   void setDoesNotThrow() {
     addFnAttr(Attribute::NoUnwind);
   }
 

llvm/trunk/include/llvm/IR/Instructions.h

void setOnlyAccessesInaccessibleMemOrArgMem() {
     addAttribute(AttributeList::FunctionIndex, Attribute::InaccessibleMemOrArgMemOnly);
   }
   /// Determine if the call cannot return.
   bool doesNotReturn() const { return hasFnAttr(Attribute::NoReturn); }
   void setDoesNotReturn() {
     addAttribute(AttributeList::FunctionIndex, Attribute::NoReturn);
   }
 
+  /// Determine if the call should not perform indirect branch tracking.
+  bool doesNoCfCheck() const { return hasFnAttr(Attribute::NoCfCheck); }
+
   /// Determine if the call cannot unwind.
   bool doesNotThrow() const { return hasFnAttr(Attribute::NoUnwind); }
   void setDoesNotThrow() {
     addAttribute(AttributeList::FunctionIndex, Attribute::NoUnwind);
   }
 
   /// Determine if the invoke cannot be duplicated.
   bool cannotDuplicate() const {return hasFnAttr(Attribute::NoDuplicate); }
public:
   /// insert it before \p InsertPt.
   ///
   /// The returned invoke instruction is identical to \p II in every way except
   /// that the operand bundles for the new instruction are set to the operand
   /// bundles in \p Bundles.
   static InvokeInst *Create(InvokeInst *II, ArrayRef<OperandBundleDef> Bundles,
                             Instruction *InsertPt = nullptr);
 
+  /// Determine if the call should not perform indirect branch tracking.
+  bool doesNoCfCheck() const { return hasFnAttr(Attribute::NoCfCheck); }
 
+  /// Determine if the call cannot unwind.
+  bool doesNotThrow() const { return hasFnAttr(Attribute::NoUnwind); }
+  void setDoesNotThrow() {
+    addAttribute(AttributeList::FunctionIndex, Attribute::NoUnwind);
+  }
   
   /// Return the function called, or null if this is an
   /// indirect function invocation.
   ///
   Function *getCalledFunction() const {
     return dyn_cast<Function>(Op<-3>());
   }
 
   /// Get a pointer to the function that is invoked by this

llvm/trunk/lib/AsmParser/LLLexer.cpp

#define KEYWORD(STR)                                                           \
   KEYWORD(noduplicate);
   KEYWORD(noimplicitfloat);
   KEYWORD(noinline);
   KEYWORD(norecurse);
   KEYWORD(nonlazybind);
   KEYWORD(nonnull);
   KEYWORD(noredzone);
   KEYWORD(noreturn);
+  KEYWORD(nocf_check);
   KEYWORD(nounwind);
   KEYWORD(optnone);
   KEYWORD(optsize);
   KEYWORD(readnone);
   KEYWORD(readonly);
   KEYWORD(returned);
   KEYWORD(returns_twice);
   KEYWORD(signext);

llvm/trunk/lib/AsmParser/LLParser.cpp

while (true) {
     case lltok::kw_nobuiltin: B.addAttribute(Attribute::NoBuiltin); break;
     case lltok::kw_noduplicate: B.addAttribute(Attribute::NoDuplicate); break;
     case lltok::kw_noimplicitfloat:
       B.addAttribute(Attribute::NoImplicitFloat); break;
     case lltok::kw_noinline: B.addAttribute(Attribute::NoInline); break;
     case lltok::kw_nonlazybind: B.addAttribute(Attribute::NonLazyBind); break;
     case lltok::kw_noredzone: B.addAttribute(Attribute::NoRedZone); break;
     case lltok::kw_noreturn: B.addAttribute(Attribute::NoReturn); break;
+    case lltok::kw_nocf_check: B.addAttribute(Attribute::NoCfCheck); break;
     case lltok::kw_norecurse: B.addAttribute(Attribute::NoRecurse); break;
     case lltok::kw_nounwind: B.addAttribute(Attribute::NoUnwind); break;
     case lltok::kw_optnone: B.addAttribute(Attribute::OptimizeNone); break;
     case lltok::kw_optsize: B.addAttribute(Attribute::OptimizeForSize); break;
     case lltok::kw_readnone: B.addAttribute(Attribute::ReadNone); break;
     case lltok::kw_readonly: B.addAttribute(Attribute::ReadOnly); break;
     case lltok::kw_returns_twice:
       B.addAttribute(Attribute::ReturnsTwice); break;
while (true) {
     case lltok::kw_naked:
     case lltok::kw_nobuiltin:
     case lltok::kw_noduplicate:
     case lltok::kw_noimplicitfloat:
     case lltok::kw_noinline:
     case lltok::kw_nonlazybind:
     case lltok::kw_noredzone:
     case lltok::kw_noreturn:
+    case lltok::kw_nocf_check:
     case lltok::kw_nounwind:
     case lltok::kw_optnone:
     case lltok::kw_optsize:
     case lltok::kw_returns_twice:
     case lltok::kw_sanitize_address:
     case lltok::kw_sanitize_hwaddress:
     case lltok::kw_sanitize_memory:
     case lltok::kw_sanitize_thread:
while (true) {
     case lltok::kw_naked:
     case lltok::kw_nobuiltin:
     case lltok::kw_noduplicate:
     case lltok::kw_noimplicitfloat:
     case lltok::kw_noinline:
     case lltok::kw_nonlazybind:
     case lltok::kw_noredzone:
     case lltok::kw_noreturn:
+    case lltok::kw_nocf_check:
     case lltok::kw_nounwind:
     case lltok::kw_optnone:
     case lltok::kw_optsize:
     case lltok::kw_returns_twice:
     case lltok::kw_sanitize_address:
     case lltok::kw_sanitize_hwaddress:
     case lltok::kw_sanitize_memory:
     case lltok::kw_sanitize_thread:

llvm/trunk/lib/AsmParser/LLToken.h

enum Kind {
   kw_noduplicate,
   kw_noimplicitfloat,
   kw_noinline,
   kw_norecurse,
   kw_nonlazybind,
   kw_nonnull,
   kw_noredzone,
   kw_noreturn,
+  kw_nocf_check,
   kw_nounwind,
   kw_optnone,
   kw_optsize,
   kw_readnone,
   kw_readonly,
   kw_returned,
   kw_returns_twice,
   kw_signext,

llvm/trunk/lib/Bitcode/Reader/BitcodeReader.cpp

static uint64_t getRawAttributeMask(Attribute::AttrKind Val) {
   case Attribute::InaccessibleMemOnly:         return 1ULL << 49;
   case Attribute::InaccessibleMemOrArgMemOnly: return 1ULL << 50;
   case Attribute::SwiftSelf:       return 1ULL << 51;
   case Attribute::SwiftError:      return 1ULL << 52;
   case Attribute::WriteOnly:       return 1ULL << 53;
   case Attribute::Speculatable:    return 1ULL << 54;
   case Attribute::StrictFP:        return 1ULL << 55;
   case Attribute::SanitizeHWAddress: return 1ULL << 56;
+  case Attribute::NoCfCheck:       return 1ULL << 57;
   case Attribute::Dereferenceable:
     llvm_unreachable("dereferenceable attribute not supported in raw format");
     break;
   case Attribute::DereferenceableOrNull:
     llvm_unreachable("dereferenceable_or_null attribute not supported in raw "
                      "format");
     break;
   case Attribute::ArgMemOnly:
static Attribute::AttrKind getAttrFromCode(uint64_t Code) {
   case bitc::ATTR_KIND_DEREFERENCEABLE_OR_NULL:
     return Attribute::DereferenceableOrNull;
   case bitc::ATTR_KIND_ALLOC_SIZE:
     return Attribute::AllocSize;
   case bitc::ATTR_KIND_NO_RED_ZONE:
     return Attribute::NoRedZone;
   case bitc::ATTR_KIND_NO_RETURN:
     return Attribute::NoReturn;
+  case bitc::ATTR_KIND_NOCF_CHECK:
+    return Attribute::NoCfCheck;
   case bitc::ATTR_KIND_NO_UNWIND:
     return Attribute::NoUnwind;
   case bitc::ATTR_KIND_OPTIMIZE_FOR_SIZE:
     return Attribute::OptimizeForSize;
   case bitc::ATTR_KIND_OPTIMIZE_NONE:
     return Attribute::OptimizeNone;
   case bitc::ATTR_KIND_READ_NONE:
     return Attribute::ReadNone;

llvm/trunk/lib/Bitcode/Writer/BitcodeWriter.cpp

static uint64_t getAttrKindEncoding(Attribute::AttrKind Kind) {
   case Attribute::Dereferenceable:
     return bitc::ATTR_KIND_DEREFERENCEABLE;
   case Attribute::DereferenceableOrNull:
     return bitc::ATTR_KIND_DEREFERENCEABLE_OR_NULL;
   case Attribute::NoRedZone:
     return bitc::ATTR_KIND_NO_RED_ZONE;
   case Attribute::NoReturn:
     return bitc::ATTR_KIND_NO_RETURN;
+  case Attribute::NoCfCheck:
+    return bitc::ATTR_KIND_NOCF_CHECK;
   case Attribute::NoUnwind:
     return bitc::ATTR_KIND_NO_UNWIND;
   case Attribute::OptimizeForSize:
     return bitc::ATTR_KIND_OPTIMIZE_FOR_SIZE;
   case Attribute::OptimizeNone:
     return bitc::ATTR_KIND_OPTIMIZE_NONE;
   case Attribute::ReadNone:
     return bitc::ATTR_KIND_READ_NONE;

llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

case ISD::BR_JT: {
     Addr = LD;
     if (TLI.isJumpTableRelative()) {
       // For PIC, the sequence is:
       // BRIND(load(Jumptable + index) + RelocBase)
       // RelocBase can be JumpTable, GOT or some sort of global base.
       Addr = DAG.getNode(ISD::ADD, dl, PTy, Addr,
                           TLI.getPICJumpTableRelocBase(Table, DAG));
     }
-    Tmp1 = DAG.getNode(ISD::BRIND, dl, MVT::Other, LD.getValue(1), Addr);
+
+    Tmp1 = TLI.expandIndirectJTBranch(dl, LD.getValue(1), Addr, DAG);
     Results.push_back(Tmp1);
     break;
   }
   case ISD::BRCOND:
     // Expand brcond's setcc into its constituent parts and create a BR_CC
     // Node.
     Tmp1 = Node->getOperand(0);
     Tmp2 = Node->getOperand(1);

llvm/trunk/lib/IR/Attributes.cpp

std::string Attribute::getAsString(bool InAttrGrp) const {
   if (hasAttribute(Attribute::NonLazyBind))
     return "nonlazybind";
   if (hasAttribute(Attribute::NonNull))
     return "nonnull";
   if (hasAttribute(Attribute::NoRedZone))
     return "noredzone";
   if (hasAttribute(Attribute::NoReturn))
     return "noreturn";
+  if (hasAttribute(Attribute::NoCfCheck))
+    return "nocf_check";
   if (hasAttribute(Attribute::NoRecurse))
     return "norecurse";
   if (hasAttribute(Attribute::NoUnwind))
     return "nounwind";
   if (hasAttribute(Attribute::OptimizeNone))
     return "optnone";
   if (hasAttribute(Attribute::OptimizeForSize))
     return "optsize";

llvm/trunk/lib/IR/Verifier.cpp

Assert(M.getNamedMetadata("llvm.linker.options"),
            "'Linker Options' named metadata no longer supported");
   }
 }
 
 /// Return true if this attribute kind only applies to functions.
 static bool isFuncOnlyAttr(Attribute::AttrKind Kind) {
   switch (Kind) {
   case Attribute::NoReturn:
+  case Attribute::NoCfCheck:
   case Attribute::NoUnwind:
   case Attribute::NoInline:
   case Attribute::AlwaysInline:
   case Attribute::OptimizeForSize:
   case Attribute::StackProtect:
   case Attribute::StackProtectReq:
   case Attribute::StackProtectStrong:
   case Attribute::SafeStack:

llvm/trunk/lib/Target/X86/AsmParser/X86AsmParser.cpp

bool X86AsmParser::ParseInstruction(ParseInstructionInfo &Info, StringRef Name,
   // lock addq %rax, %rbx ; Destination operand must be of memory type
   // xacquire <insn>      ; xacquire must be accompanied by 'lock'
   bool isPrefix = StringSwitch<bool>(Name)
                       .Cases("rex64", "data32", "data16", true)
                       .Cases("xacquire", "xrelease", true)
                       .Cases("acquire", "release", isParsingIntelSyntax())
                       .Default(false);
 
-  auto isLockRepeatPrefix = [](StringRef N) {
+  auto isLockRepeatNtPrefix = [](StringRef N) {
     return StringSwitch<bool>(N)
-        .Cases("lock", "rep", "repe", "repz", "repne", "repnz", true)
+        .Cases("lock", "rep", "repe", "repz", "repne", "repnz", "notrack", true)
         .Default(false);
   };
 
   bool CurlyAsEndOfStatement = false;
 
   unsigned Flags = X86::IP_NO_PREFIX;
-  while (isLockRepeatPrefix(Name.lower())) {
+  while (isLockRepeatNtPrefix(Name.lower())) {
     unsigned Prefix =
         StringSwitch<unsigned>(Name)
             .Cases("lock", "lock", X86::IP_HAS_LOCK)
             .Cases("rep", "repe", "repz", X86::IP_HAS_REPEAT)
             .Cases("repne", "repnz", X86::IP_HAS_REPEAT_NE)
+            .Cases("notrack", "notrack", X86::IP_HAS_NOTRACK)
             .Default(X86::IP_NO_PREFIX); // Invalid prefix (impossible)
     Flags |= Prefix;
     if (getLexer().is(AsmToken::EndOfStatement)) {
       // We don't have real instr with the given prefix
       //  let's use the prefix as the instr.
       // TODO: there could be several prefixes one after another
       Flags = X86::IP_NO_PREFIX;
       break;

llvm/trunk/lib/Target/X86/InstPrinter/X86ATTInstPrinter.cpp

void X86ATTInstPrinter::printInst(const MCInst *MI, raw_ostream &OS,
   if (CommentStream)
     HasCustomInstComment =
         EmitAnyX86InstComments(MI, *CommentStream, MII, getRegisterName);
 
   unsigned Flags = MI->getFlags();
   if ((TSFlags & X86II::LOCK) || (Flags & X86::IP_HAS_LOCK))
     OS << "\tlock\t";
 
+  if ((TSFlags & X86II::NOTRACK) || (Flags & X86::IP_HAS_NOTRACK))
+    OS << "\tnotrack\t";
+
   if (Flags & X86::IP_HAS_REPEAT_NE)
     OS << "\trepne\t";
   else if (Flags & X86::IP_HAS_REPEAT)
     OS << "\trep\t";
 
   // Output CALLpcrel32 as "callq" in 64-bit mode.
   // In Intel annotation it's always emitted as "call".
   //

llvm/trunk/lib/Target/X86/InstPrinter/X86IntelInstPrinter.cpp

void X86IntelInstPrinter::printInst(const MCInst *MI, raw_ostream &OS,
   if ((TSFlags & X86II::LOCK) || (Flags & X86::IP_HAS_LOCK))
     OS << "\tlock\t";
 
   if (Flags & X86::IP_HAS_REPEAT_NE)
     OS << "\trepne\t";
   else if (Flags & X86::IP_HAS_REPEAT)
     OS << "\trep\t";
 
+  if ((TSFlags & X86II::NOTRACK) || (Flags & X86::IP_HAS_NOTRACK))
+    OS << "\tnotrack\t";
+
   printInstruction(MI, OS);
 
   // Next always print the annotation.
   printAnnotation(OS, Annot);
 
   // If verbose assembly is enabled, we can print some informative comments.
   if (CommentStream)
     EmitAnyX86InstComments(MI, *CommentStream, MII, getRegisterName);

llvm/trunk/lib/Target/X86/MCTargetDesc/X86BaseInfo.h

namespace X86 {
   /// The constants to describe instr prefixes if there are
   enum IPREFIXES {
     IP_NO_PREFIX = 0,
     IP_HAS_OP_SIZE = 1,
     IP_HAS_AD_SIZE = 2,
     IP_HAS_REPEAT_NE = 4,
     IP_HAS_REPEAT = 8,
     IP_HAS_LOCK = 16,
-    NO_SCHED_INFO = 32 // Don't add sched comment to the current instr because
+    NO_SCHED_INFO = 32, // Don't add sched comment to the current instr because
                         // it was already added
+    IP_HAS_NOTRACK = 64
   };
 } // end namespace X86;
 
 /// X86II - This namespace holds all of the target specific flags that
 /// instruction info tracks.
 ///
 namespace X86II {
   /// Target Operand Flag enum.
enum : uint64_t {
     /// storing a classifier in the imm8 field.  To simplify our implementation,
     /// we handle this by storeing the classifier in the opcode field and using
     /// this flag to indicate that the encoder should do the wacky 3DNow! thing.
     Has3DNow0F0FOpcodeShift = CD8_Scale_Shift + 7,
     Has3DNow0F0FOpcode = 1ULL << Has3DNow0F0FOpcodeShift,
 
     /// Explicitly specified rounding control
     EVEX_RCShift = Has3DNow0F0FOpcodeShift + 1,
-    EVEX_RC = 1ULL << EVEX_RCShift
+    EVEX_RC = 1ULL << EVEX_RCShift,
+
+    // NOTRACK prefix
+    NoTrackShift = EVEX_RCShift + 1,
+    NOTRACK = 1ULL << NoTrackShift
   };
 
   // getBaseOpcodeFor - This function returns the "base" X86 opcode for the
   // specified machine instruction.
   //
   inline unsigned char getBaseOpcodeFor(uint64_t TSFlags) {
     return TSFlags >> X86II::OpcodeShift;
   }

llvm/trunk/lib/Target/X86/MCTargetDesc/X86MCCodeEmitter.cpp

bool X86MCCodeEmitter::emitOpcodePrefix(uint64_t TSFlags, unsigned &CurByte,
   if ((TSFlags & X86II::OpSizeMask) == (is16BitMode(STI) ? X86II::OpSize32
                                                          : X86II::OpSize16))
     EmitByte(0x66, CurByte, OS);
 
   // Emit the LOCK opcode prefix.
   if (TSFlags & X86II::LOCK || MI.getFlags() & X86::IP_HAS_LOCK)
     EmitByte(0xF0, CurByte, OS);
 
+  // Emit the NOTRACK opcode prefix.
+  if (TSFlags & X86II::NOTRACK || MI.getFlags() & X86::IP_HAS_NOTRACK)
+    EmitByte(0x3E, CurByte, OS);
+
   switch (TSFlags & X86II::OpPrefixMask) {
   case X86II::PD:   // 66
     EmitByte(0x66, CurByte, OS);
     break;
   case X86II::XS:   // F3
     EmitByte(0xF3, CurByte, OS);
     break;
   case X86II::XD:   // F2

llvm/trunk/lib/Target/X86/X86FastISel.cpp

bool X86FastISel::fastLowerCall(CallLoweringInfo &CLI) {
 
   bool Is64Bit        = Subtarget->is64Bit();
   bool IsWin64        = Subtarget->isCallingConvWin64(CC);
 
   const CallInst *CI =
       CLI.CS ? dyn_cast<CallInst>(CLI.CS->getInstruction()) : nullptr;
   const Function *CalledFn = CI ? CI->getCalledFunction() : nullptr;
 
+  // Call / invoke instructions with NoCfCheck attribute require special
+  // handling.
+  const auto *II =
+      CLI.CS ? dyn_cast<InvokeInst>(CLI.CS->getInstruction()) : nullptr;
+  if ((CI && CI->doesNoCfCheck()) || (II && II->doesNoCfCheck()))
+    return false;
+
   // Functions with no_caller_saved_registers that need special handling.
   if ((CI && CI->hasFnAttr("no_caller_saved_registers")) ||
       (CalledFn && CalledFn->hasFnAttribute("no_caller_saved_registers")))
     return false;
 
   // Functions using retpoline should use SDISel for calls.
   if (Subtarget->useRetpoline())
     return false;

llvm/trunk/lib/Target/X86/X86ISelLowering.h

enum NodeType : unsigned {
       /// The result values of these nodes are:
       ///
       ///     #0 - The outgoing token chain
       ///     #1 - The first register result value (optional)
       ///     #2 - The second register result value (optional)
       ///
       CALL,
 
+      /// Same as call except it adds the NoTrack prefix.
+      NT_CALL,
+
       /// This operation implements the lowering for readcyclecounter.
       RDTSC_DAG,
 
       /// X86 Read Time-Stamp Counter and Processor ID.
       RDTSCP_DAG,
 
       /// X86 Read Performance Monitoring Counters.
       RDPMC_DAG,
enum NodeType : unsigned {
       CMOV,
 
       /// X86 conditional branches. Operand 0 is the chain operand, operand 1
       /// is the block to branch if condition is true, operand 2 is the
       /// condition code, and operand 3 is the flag operand produced by a CMP
       /// or TEST instruction.
       BRCOND,
 
+      /// BRIND node with NoTrack prefix. Operand 0 is the chain operand and
+      /// operand 1 is the target address.
+      NT_BRIND,
+
       /// Return with a flag operand. Operand 0 is the chain operand, operand
       /// 1 is the number of bytes of stack to pop.
       RET_FLAG,
 
       /// Return from interrupt. Operand 0 is the number of bytes to pop.
       IRET,
 
       /// Repeat fill, corresponds to X86::REP_STOSx.
bool lowerInterleavedLoad(LoadInst *LI,
                               ArrayRef<unsigned> Indices,
                               unsigned Factor) const override;
 
     /// \brief Lower interleaved store(s) into target specific
     /// instructions/intrinsics.
     bool lowerInterleavedStore(StoreInst *SI, ShuffleVectorInst *SVI,
                                unsigned Factor) const override;
 
-
     void finalizeLowering(MachineFunction &MF) const override;
 
+    SDValue expandIndirectJTBranch(const SDLoc& dl, SDValue Value, 
+                                   SDValue Addr, SelectionDAG &DAG) 
+                                   const override;
+
   protected:
     std::pair<const TargetRegisterClass *, uint8_t>
     findRepresentativeClass(const TargetRegisterInfo *TRI,
                             MVT VT) const override;
 
   private:
     /// Keep a reference to the X86Subtarget around so that we can
     /// make the right decision when generating code for different targets.

llvm/trunk/lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI,
   StructReturnType SR = callIsStructReturn(Outs, Subtarget.isTargetMCU());
   bool IsSibcall      = false;
   X86MachineFunctionInfo *X86Info = MF.getInfo<X86MachineFunctionInfo>();
   auto Attr = MF.getFunction().getFnAttribute("disable-tail-calls");
   const auto *CI = dyn_cast_or_null<CallInst>(CLI.CS.getInstruction());
   const Function *Fn = CI ? CI->getCalledFunction() : nullptr;
   bool HasNCSR = (CI && CI->hasFnAttr("no_caller_saved_registers")) ||
                  (Fn && Fn->hasFnAttribute("no_caller_saved_registers"));
+  const auto *II = dyn_cast_or_null<InvokeInst>(CLI.CS.getInstruction());
+  bool HasNoCfCheck =
+      (CI && CI->doesNoCfCheck()) || (II && II->doesNoCfCheck());
+  const Module *M = MF.getMMI().getModule();
+  Metadata *IsCFProtectionSupported = M->getModuleFlag("cf-protection-branch");
 
   if (CallConv == CallingConv::X86_INTR)
     report_fatal_error("X86 interrupts may not be called directly");
 
   if (Attr.getValueAsString() == "true")
     isTailCall = false;
 
   if (Subtarget.isPICStyleGOT() &&
if (isTailCall) {
     //// to the liveout set for the function.
     // This isn't right, although it's probably harmless on x86; liveouts
     // should be computed from returns not tail calls.  Consider a void
     // function making a tail call to a function returning int.
     MF.getFrameInfo().setHasTailCall();
     return DAG.getNode(X86ISD::TC_RETURN, dl, NodeTys, Ops);
   }
 
+  if (HasNoCfCheck && IsCFProtectionSupported) {
+    Chain = DAG.getNode(X86ISD::NT_CALL, dl, NodeTys, Ops);
+  } else {
     Chain = DAG.getNode(X86ISD::CALL, dl, NodeTys, Ops);
+  }
   InFlag = Chain.getValue(1);
 
   // Create the CALLSEQ_END node.
   unsigned NumBytesForCalleeToPop;
   if (X86::isCalleePop(CallConv, Is64Bit, isVarArg,
                        DAG.getTarget().Options.GuaranteedTailCallOpt))
     NumBytesForCalleeToPop = NumBytes;    // Callee pops everything
   else if (!Is64Bit && !canGuaranteeTCO(CallConv) &&
const char *X86TargetLowering::getTargetNodeName(unsigned Opcode) const {
   case X86ISD::VPDPBUSD:           return "X86ISD::VPDPBUSD";
   case X86ISD::VPDPBUSDS:          return "X86ISD::VPDPBUSDS";
   case X86ISD::VPDPWSSD:           return "X86ISD::VPDPWSSD";
   case X86ISD::VPDPWSSDS:          return "X86ISD::VPDPWSSDS";
   case X86ISD::VPSHUFBITQMB:       return "X86ISD::VPSHUFBITQMB";
   case X86ISD::GF2P8MULB:          return "X86ISD::GF2P8MULB";
   case X86ISD::GF2P8AFFINEQB:      return "X86ISD::GF2P8AFFINEQB";
   case X86ISD::GF2P8AFFINEINVQB:   return "X86ISD::GF2P8AFFINEINVQB";
+  case X86ISD::NT_CALL:            return "X86ISD::NT_CALL";
+  case X86ISD::NT_BRIND:           return "X86ISD::NT_BRIND";
   }
   return nullptr;
 }
 
 /// Return true if the addressing mode represented by AM is legal for this
 /// target, for a load/store of the specified type.
 bool X86TargetLowering::isLegalAddressingMode(const DataLayout &DL,
                                               const AddrMode &AM, Type *Ty,
void X86TargetLowering::finalizeLowering(MachineFunction &MF) const {
   if (hasCopyImplyingStackAdjustment(MF)) {
     MachineFrameInfo &MFI = MF.getFrameInfo();
     MFI.setHasCopyImplyingStackAdjustment(true);
   }
 
   TargetLoweringBase::finalizeLowering(MF);
 }
 
+SDValue X86TargetLowering::expandIndirectJTBranch(const SDLoc& dl, 
+                                                  SDValue Value, SDValue Addr,
+                                                  SelectionDAG &DAG) const {
+  const Module *M = DAG.getMachineFunction().getMMI().getModule();
+  Metadata *IsCFProtectionSupported = M->getModuleFlag("cf-protection-branch");
+  if (IsCFProtectionSupported) {
+    // In case control-flow branch protection is enabled, we need to add
+    // notrack prefix to the indirect branch.
+    // In order to do that we create NT_BRIND SDNode.
+    // Upon ISEL, the pattern will convert it to jmp with NoTrack prefix.
+    return DAG.getNode(X86ISD::NT_BRIND, dl, MVT::Other, Value, Addr);
+  }
+
+  return TargetLowering::expandIndirectJTBranch(dl, Value, Addr, DAG);
+}
+
 /// This method query the target whether it is beneficial for dag combiner to
 /// promote the specified node. If true, it should return the desired promotion
 /// type by reference.
 bool X86TargetLowering::IsDesirableToPromoteOp(SDValue Op, EVT &PVT) const {
   EVT VT = Op.getValueType();
   if (VT != MVT::i16)
     return false;
 

llvm/trunk/lib/Target/X86/X86IndirectBranchTracking.cpp

 //===----------------------------------------------------------------------===//
 
 #include "X86.h"
 #include "X86InstrInfo.h"
 #include "X86Subtarget.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
-#include "llvm/CodeGen/MachineJumpTableInfo.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "x86-indirect-branch-tracking"
 
 static cl::opt<bool> IndirectBranchTracking(
     "x86-indirect-branch-tracking", cl::init(false), cl::Hidden,
private:
   const X86InstrInfo *TII;
 
   /// Endbr opcode for the current machine function.
   unsigned int EndbrOpcode;
 
   /// Adds a new ENDBR instruction to the begining of the MBB.
   /// The function will not add it if already exists.
   /// It will add ENDBR32 or ENDBR64 opcode, depending on the target.
-  void addENDBR(MachineBasicBlock &MBB) const;
+  /// \returns true if the ENDBR was added and false otherwise.
+  bool addENDBR(MachineBasicBlock &MBB) const;
 };
 
 } // end anonymous namespace
 
 char X86IndirectBranchTrackingPass::ID = 0;
 
 FunctionPass *llvm::createX86IndirectBranchTrackingPass() {
   return new X86IndirectBranchTrackingPass();
 }
 
-void X86IndirectBranchTrackingPass::addENDBR(MachineBasicBlock &MBB) const {
+bool X86IndirectBranchTrackingPass::addENDBR(MachineBasicBlock &MBB) const {
   assert(TII && "Target instruction info was not initialized");
   assert((X86::ENDBR64 == EndbrOpcode || X86::ENDBR32 == EndbrOpcode) &&
          "Unexpected Endbr opcode");
 
   auto MI = MBB.begin();
   // If the MBB is empty or the first instruction is not ENDBR,
   // add the ENDBR instruction to the beginning of the MBB.
   if (MI == MBB.end() || EndbrOpcode != MI->getOpcode()) {
     BuildMI(MBB, MI, MBB.findDebugLoc(MI), TII->get(EndbrOpcode));
     NumEndBranchAdded++;
+    return true;
   }
+
+  return false;
 }
 
 bool X86IndirectBranchTrackingPass::runOnMachineFunction(MachineFunction &MF) {
   const X86Subtarget &SubTarget = MF.getSubtarget<X86Subtarget>();
 
-  // Make sure that the target supports ENDBR instruction.
+  // Make sure that the target supports IBT instruction.
   if (!SubTarget.hasIBT())
     return false;
 
   // Check that the cf-protection-branch is enabled.
   Metadata *isCFProtectionSupported =
       MF.getMMI().getModule()->getModuleFlag("cf-protection-branch");
   if (!isCFProtectionSupported && !IndirectBranchTracking)
     return false;
 
   // True if the current MF was changed and false otherwise.
   bool Changed = false;
 
   TII = SubTarget.getInstrInfo();
   EndbrOpcode = SubTarget.is64Bit() ? X86::ENDBR64 : X86::ENDBR32;
 
   // Non-internal function or function whose address was taken, can be
-  // invoked through indirect calls. Mark the first BB with ENDBR instruction.
-  // TODO: Do not add ENDBR instruction in case notrack attribute is used.
-  if (MF.getFunction().hasAddressTaken() ||
-      !MF.getFunction().hasLocalLinkage()) {
+  // accessed through indirect calls. Mark the first BB with ENDBR instruction
+  // unless nocf_check attribute is used.
+  if ((MF.getFunction().hasAddressTaken() ||
+       !MF.getFunction().hasLocalLinkage()) &&
+      !MF.getFunction().doesNoCfCheck()) {
     auto MBB = MF.begin();
-    addENDBR(*MBB);
-    Changed = true;
+    Changed |= addENDBR(*MBB);
   }
 
-  for (auto &MBB : MF) {
-    // Find all basic blocks that thier address was taken (for example
+  for (auto &MBB : MF)
+    // Find all basic blocks that their address was taken (for example
     // in the case of indirect jump) and add ENDBR instruction.
-    if (MBB.hasAddressTaken()) {
-      addENDBR(MBB);
-      Changed = true;
-    }
-  }
-
-  // Adds ENDBR instructions to MBB destinations of the jump table.
-  // TODO: In case of more than 50 destinations, do not add ENDBR and
-  // instead add DS_PREFIX.
-  if (MachineJumpTableInfo *JTI = MF.getJumpTableInfo()) {
-    for (const auto &JT : JTI->getJumpTables()) {
-      for (auto *MBB : JT.MBBs) {
-  // This assert verifies the assumption that this MBB has an indirect
-  // jump terminator in one of its predecessor.
-  // Jump tables are generated when lowering switch-case statements or
-  // setjmp/longjump functions. As a result only indirect jumps use jump
-  // tables.
-        #ifndef NDEBUG
-        bool hasIndirectJumpTerm = false;
-        for (auto &PredMBB : MBB->predecessors())
-          for (auto &TermI : PredMBB->terminators())
-            if (TermI.isIndirectBranch())
-              hasIndirectJumpTerm = true;
-        assert(hasIndirectJumpTerm &&
-               "The MBB is not the destination of an indirect jump");
-  (void)hasIndirectJumpTerm;
-  #endif
-        addENDBR(*MBB);
-        Changed = true;
-      }
-    }
-  }
+    if (MBB.hasAddressTaken())
+      Changed |= addENDBR(MBB);
 
   return Changed;
 }

llvm/trunk/lib/Target/X86/X86InstrControl.td

let isBranch = 1, isTerminator = 1, isBarrier = 1, isIndirectBranch = 1 in {
 
   def JMP64r     : I<0xFF, MRM4r, (outs), (ins GR64:$dst), "jmp{q}\t{*}$dst",
                      [(brind GR64:$dst)], IIC_JMP_REG>, Requires<[In64BitMode]>,
                    Sched<[WriteJump]>;
   def JMP64m     : I<0xFF, MRM4m, (outs), (ins i64mem:$dst), "jmp{q}\t{*}$dst",
                      [(brind (loadi64 addr:$dst))], IIC_JMP_MEM>,
                    Requires<[In64BitMode]>, Sched<[WriteJumpLd]>;
 
+  let isCodeGenOnly = 1, Predicates = [HasIBT] in {
+    def JMP16r_NT : I<0xFF, MRM4r, (outs), (ins GR16 : $dst), "jmp{w}\t{*}$dst",
+                   [(X86NoTrackBrind GR16 : $dst)], IIC_JMP_REG>, Requires<[Not64BitMode]>,
+                 OpSize16, Sched<[WriteJump]>, NOTRACK;
+
+    def JMP16m_NT : I<0xFF, MRM4m, (outs), (ins i16mem : $dst), "jmp{w}\t{*}$dst",
+                   [(X86NoTrackBrind (loadi16 addr : $dst))], IIC_JMP_MEM>,
+                 Requires<[Not64BitMode]>, OpSize16, Sched<[WriteJumpLd]>, NOTRACK;
+
+    def JMP32r_NT : I<0xFF, MRM4r, (outs), (ins GR32 : $dst), "jmp{l}\t{*}$dst",
+                   [(X86NoTrackBrind GR32 : $dst)], IIC_JMP_REG>, Requires<[Not64BitMode]>,
+                 OpSize32, Sched<[WriteJump]>, NOTRACK;
+    def JMP32m_NT : I<0xFF, MRM4m, (outs), (ins i32mem : $dst), "jmp{l}\t{*}$dst",
+                   [(X86NoTrackBrind (loadi32 addr : $dst))], IIC_JMP_MEM>,
+                 Requires<[Not64BitMode]>, OpSize32, Sched<[WriteJumpLd]>, NOTRACK;
+
+    def JMP64r_NT : I<0xFF, MRM4r, (outs), (ins GR64 : $dst), "jmp{q}\t{*}$dst",
+                      [(X86NoTrackBrind GR64 : $dst)], IIC_JMP_REG>, Requires<[In64BitMode]>,
+                    Sched<[WriteJump]>, NOTRACK;
+    def JMP64m_NT : I<0xFF, MRM4m, (outs), (ins i64mem : $dst), "jmp{q}\t{*}$dst",
+                      [(X86NoTrackBrind(loadi64 addr : $dst))], IIC_JMP_MEM>,
+                    Requires<[In64BitMode]>, Sched<[WriteJumpLd]>, NOTRACK;
+  }
+
   let Predicates = [Not64BitMode] in {
     def FARJMP16i  : Iseg16<0xEA, RawFrmImm16, (outs),
                             (ins i16imm:$off, i16imm:$seg),
                             "ljmp{w}\t$seg, $off", [],
                             IIC_JMP_FAR_PTR>, OpSize16, Sched<[WriteJump]>;
     def FARJMP32i  : Iseg32<0xEA, RawFrmImm16, (outs),
                             (ins i32imm:$off, i16imm:$seg),
                             "ljmp{l}\t$seg, $off", [],
                             IIC_JMP_FAR_PTR>, OpSize32, Sched<[WriteJump]>;
   }
   def FARJMP64   : RI<0xFF, MRM5m, (outs), (ins opaque80mem:$dst),
                       "ljmp{q}\t{*}$dst", [], IIC_JMP_FAR_MEM>,
                    Sched<[WriteJump]>;
 
   def FARJMP16m  : I<0xFF, MRM5m, (outs), (ins opaque32mem:$dst),
                      "ljmp{w}\t{*}$dst", [], IIC_JMP_FAR_MEM>, OpSize16,
                    Sched<[WriteJumpLd]>;
   def FARJMP32m  : I<0xFF, MRM5m, (outs), (ins opaque48mem:$dst),
                      "{l}jmp{l}\t{*}$dst", [], IIC_JMP_FAR_MEM>, OpSize32,
                    Sched<[WriteJumpLd]>;
 }
 
-
 // Loop instructions
 let SchedRW = [WriteJump] in {
 def LOOP   : Ii8PCRel<0xE2, RawFrm, (outs), (ins brtarget8:$dst), "loop\t$dst", [], IIC_LOOP>;
 def LOOPE  : Ii8PCRel<0xE1, RawFrm, (outs), (ins brtarget8:$dst), "loope\t$dst", [], IIC_LOOPE>;
 def LOOPNE : Ii8PCRel<0xE0, RawFrm, (outs), (ins brtarget8:$dst), "loopne\t$dst", [], IIC_LOOPNE>;
 }
 
 //===----------------------------------------------------------------------===//
def CALL32r     : I<0xFF, MRM2r, (outs), (ins GR32:$dst),
                       OpSize32, Requires<[Not64BitMode,NotUseRetpoline]>,
                       Sched<[WriteJump]>;
     def CALL32m     : I<0xFF, MRM2m, (outs), (ins i32mem:$dst),
                         "call{l}\t{*}$dst", [(X86call (loadi32 addr:$dst))],
                         IIC_CALL_MEM>, OpSize32,
                       Requires<[Not64BitMode,FavorMemIndirectCall,NotUseRetpoline]>,
                       Sched<[WriteJumpLd]>;
 
+    let isCodeGenOnly = 1, Predicates = [HasIBT] in {
+      def CALL16r_NT : I<0xFF, MRM2r, (outs), (ins GR16 : $dst),
+                        "call{w}\t{*}$dst",[(X86NoTrackCall GR16 : $dst)], IIC_CALL_RI>,
+                        OpSize16, Requires<[Not64BitMode]>, Sched<[WriteJump]>, NOTRACK;
+      def CALL16m_NT : I<0xFF, MRM2m, (outs), (ins i16mem : $dst),
+                        "call{w}\t{*}$dst",[(X86NoTrackCall(loadi16 addr : $dst))],
+                        IIC_CALL_MEM>, OpSize16,
+                        Requires<[Not64BitMode,FavorMemIndirectCall]>,
+                        Sched<[WriteJumpLd]>, NOTRACK;
+      def CALL32r_NT : I<0xFF, MRM2r, (outs), (ins GR32 : $dst),
+                        "call{l}\t{*}$dst",[(X86NoTrackCall GR32 : $dst)], IIC_CALL_RI>,
+                        OpSize32, Requires<[Not64BitMode]>, Sched<[WriteJump]>, NOTRACK;
+      def CALL32m_NT : I<0xFF, MRM2m, (outs), (ins i32mem : $dst),
+                        "call{l}\t{*}$dst",[(X86NoTrackCall(loadi32 addr : $dst))],
+                        IIC_CALL_MEM>, OpSize32,
+                        Requires<[Not64BitMode,FavorMemIndirectCall]>,
+                        Sched<[WriteJumpLd]>, NOTRACK;
+    }
+
     let Predicates = [Not64BitMode] in {
       def FARCALL16i  : Iseg16<0x9A, RawFrmImm16, (outs),
                                (ins i16imm:$off, i16imm:$seg),
                                "lcall{w}\t$seg, $off", [],
                                IIC_CALL_FAR_PTR>, OpSize16, Sched<[WriteJump]>;
       def FARCALL32i  : Iseg32<0x9A, RawFrmImm16, (outs),
                                (ins i32imm:$off, i16imm:$seg),
                                "lcall{l}\t$seg, $off", [],
def CALL64r       : I<0xFF, MRM2r, (outs), (ins GR64:$dst),
                         IIC_CALL_RI>,
                       Requires<[In64BitMode,NotUseRetpoline]>;
   def CALL64m       : I<0xFF, MRM2m, (outs), (ins i64mem:$dst),
                         "call{q}\t{*}$dst", [(X86call (loadi64 addr:$dst))],
                         IIC_CALL_MEM>,
                       Requires<[In64BitMode,FavorMemIndirectCall,
                                 NotUseRetpoline]>;
 
+  let isCodeGenOnly = 1, Predicates = [HasIBT] in{
+    def CALL64r_NT : I<0xFF, MRM2r, (outs), (ins GR64 : $dst),
+                      "call{q}\t{*}$dst",[(X86NoTrackCall GR64 : $dst)],
+                      IIC_CALL_RI>,
+                      Requires<[In64BitMode]>, NOTRACK;
+    def CALL64m_NT : I<0xFF, MRM2m, (outs), (ins i64mem : $dst),
+                       "call{q}\t{*}$dst",[(X86NoTrackCall(loadi64 addr : $dst))],
+                       IIC_CALL_MEM>,
+                      Requires<[In64BitMode,FavorMemIndirectCall]>, NOTRACK;
+  }
+
   def FARCALL64   : RI<0xFF, MRM3m, (outs), (ins opaque80mem:$dst),
                        "lcall{q}\t{*}$dst", [], IIC_CALL_FAR_MEM>;
 }
 
 let isCall = 1, isTerminator = 1, isReturn = 1, isBarrier = 1,
     isCodeGenOnly = 1, Uses = [RSP, SSP], SchedRW = [WriteJump] in {
   def TCRETURNdi64   : PseudoI<(outs),
                         (ins i64i32imm_pcrel:$dst, i32imm:$offset),

llvm/trunk/lib/Target/X86/X86InstrFormats.td

 class EVEX_4V : VEX_4V { Encoding OpEnc = EncEVEX; }
 class EVEX_K { bit hasEVEX_K = 1; }
 class EVEX_KZ : EVEX_K { bit hasEVEX_Z = 1; }
 class EVEX_B { bit hasEVEX_B = 1; }
 class EVEX_RC { bit hasEVEX_RC = 1; }
 class EVEX_V512 { bit hasEVEX_L2 = 1; bit hasVEX_L = 0; }
 class EVEX_V256 { bit hasEVEX_L2 = 0; bit hasVEX_L = 1; }
 class EVEX_V128 { bit hasEVEX_L2 = 0; bit hasVEX_L = 0; }
+class NOTRACK { bit hasNoTrackPrefix = 1; }
 
 // Specify AVX512 8-bit compressed displacement encoding based on the vector
 // element size in bits (8, 16, 32, 64) and the CDisp8 form.
 class EVEX_CD8<int esize, CD8VForm form> {
   int CD8_EltSize = !srl(esize, 3);
   bits<3> CD8_Form = form.Value;
 }
 
class X86Inst<bits<8> opcod, Format f, ImmType i, dag outs, dag ins,
   bit hasEVEX_L2 = 0;       // Does this inst set the EVEX_L2 field?
   bit hasEVEX_B = 0;        // Does this inst set the EVEX_B field?
   bits<3> CD8_Form = 0;     // Compressed disp8 form - vector-width.
   // Declare it int rather than bits<4> so that all bits are defined when
   // assigning to bits<7>.
   int CD8_EltSize = 0;      // Compressed disp8 form - element-size in bytes.
   bit has3DNow0F0FOpcode =0;// Wacky 3dNow! encoding?
   bit hasEVEX_RC = 0;       // Explicitly specified rounding control in FP instruction.
+  bit hasNoTrackPrefix = 0; // Does this inst has 0x3E (NoTrack) prefix?
 
   bits<2> EVEX_LL;
   let EVEX_LL{0} = hasVEX_L;
   let EVEX_LL{1} = hasEVEX_L2;
   // Vector size in bytes.
   bits<7> VectSize = !shl(16, EVEX_LL);
 
   // The scaling factor for AVX512's compressed displacement is either
class X86Inst<bits<8> opcod, Format f, ImmType i, dag outs, dag ins,
   let TSFlags{42}    = hasEVEX_K;
   let TSFlags{43}    = hasEVEX_Z;
   let TSFlags{44}    = hasEVEX_L2;
   let TSFlags{45}    = hasEVEX_B;
   // If we run out of TSFlags bits, it's possible to encode this in 3 bits.
   let TSFlags{52-46} = CD8_Scale;
   let TSFlags{53}    = has3DNow0F0FOpcode;
   let TSFlags{54}    = hasEVEX_RC;
+  let TSFlags{55}    = hasNoTrackPrefix;
 }
 
 class PseudoI<dag oops, dag iops, list<dag> pattern,
               InstrItinClass itin = NoItinerary>
   : X86Inst<0, Pseudo, NoImm, oops, iops, "", itin> {
   let Pattern = pattern;
 }
 

llvm/trunk/lib/Target/X86/X86InstrInfo.td

 
 def SDT_X86CallSeqStart : SDCallSeqStart<[SDTCisVT<0, i32>,
                                           SDTCisVT<1, i32>]>;
 def SDT_X86CallSeqEnd   : SDCallSeqEnd<[SDTCisVT<0, i32>,
                                         SDTCisVT<1, i32>]>;
 
 def SDT_X86Call   : SDTypeProfile<0, -1, [SDTCisVT<0, iPTR>]>;
 
+def SDT_X86NtBrind : SDTypeProfile<0, -1, [SDTCisVT<0, iPTR>]>;
+
 def SDT_X86VASTART_SAVE_XMM_REGS : SDTypeProfile<0, -1, [SDTCisVT<0, i8>,
                                                          SDTCisVT<1, iPTR>,
                                                          SDTCisVT<2, iPTR>]>;
 
 def SDT_X86VAARG_64 : SDTypeProfile<1, -1, [SDTCisPtrTy<0>,
                                             SDTCisPtrTy<1>,
                                             SDTCisVT<2, i32>,
                                             SDTCisVT<3, i8>,

 def X86callseq_end :
                  SDNode<"ISD::CALLSEQ_END",   SDT_X86CallSeqEnd,
                         [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue]>;
 
 def X86call    : SDNode<"X86ISD::CALL",     SDT_X86Call,
                         [SDNPHasChain, SDNPOutGlue, SDNPOptInGlue,
                          SDNPVariadic]>;
 
+def X86NoTrackCall : SDNode<"X86ISD::NT_CALL", SDT_X86Call,
+                            [SDNPHasChain, SDNPOutGlue, SDNPOptInGlue, 
+                             SDNPVariadic]>;
+def X86NoTrackBrind : SDNode<"X86ISD::NT_BRIND", SDT_X86NtBrind, 
+                             [SDNPHasChain]>;
+
 def X86rep_stos: SDNode<"X86ISD::REP_STOS", SDTX86RepStr,
                         [SDNPHasChain, SDNPInGlue, SDNPOutGlue, SDNPMayStore]>;
 def X86rep_movs: SDNode<"X86ISD::REP_MOVS", SDTX86RepStr,
                         [SDNPHasChain, SDNPInGlue, SDNPOutGlue, SDNPMayStore,
                          SDNPMayLoad]>;
 
 def X86rdtsc   : SDNode<"X86ISD::RDTSC_DAG", SDTX86Void,
                         [SDNPHasChain, SDNPOutGlue, SDNPSideEffect]>;

llvm/trunk/lib/Transforms/IPO/ForceFunctionAttrs.cpp

return StringSwitch<Attribute::AttrKind>(Kind)
       .Case("naked", Attribute::Naked)
       .Case("nobuiltin", Attribute::NoBuiltin)
       .Case("noduplicate", Attribute::NoDuplicate)
       .Case("noimplicitfloat", Attribute::NoImplicitFloat)
       .Case("noinline", Attribute::NoInline)
       .Case("nonlazybind", Attribute::NonLazyBind)
       .Case("noredzone", Attribute::NoRedZone)
       .Case("noreturn", Attribute::NoReturn)
+      .Case("nocf_check", Attribute::NoCfCheck)
       .Case("norecurse", Attribute::NoRecurse)
       .Case("nounwind", Attribute::NoUnwind)
       .Case("optnone", Attribute::OptimizeNone)
       .Case("optsize", Attribute::OptimizeForSize)
       .Case("readnone", Attribute::ReadNone)
       .Case("readonly", Attribute::ReadOnly)
       .Case("argmemonly", Attribute::ArgMemOnly)
       .Case("returns_twice", Attribute::ReturnsTwice)

llvm/trunk/lib/Transforms/Utils/CodeExtractor.cpp

if (Attr.isStringAttribute()) {
       case Attribute::SanitizeMemory:
       case Attribute::SanitizeThread:
       case Attribute::SanitizeHWAddress:
       case Attribute::StackProtect:
       case Attribute::StackProtectReq:
       case Attribute::StackProtectStrong:
       case Attribute::StrictFP:
       case Attribute::UWTable:
+      case Attribute::NoCfCheck:
         break;
       }
 
     newFunction->addFnAttr(Attr);
   }
   newFunction->getBasicBlockList().push_back(newRootNode);
 
   // Create an iterator to name all of the arguments we inserted.

llvm/trunk/test/CodeGen/X86/indirect-branch-tracking.ll

-; RUN: llc -mtriple=x86_64-unknown-unknown -mattr=+ibt -x86-indirect-branch-tracking < %s | FileCheck %s --check-prefix=ALL --check-prefix=X86_64
-; RUN: llc -mtriple=i386-unknown-unknown -mattr=+ibt -x86-indirect-branch-tracking < %s | FileCheck %s --check-prefix=ALL --check-prefix=X86
+; RUN: llc -mtriple=x86_64-unknown-unknown -mattr=+ibt < %s | FileCheck %s --check-prefix=ALL --check-prefix=X86_64
+; RUN: llc -mtriple=i386-unknown-unknown -mattr=+ibt < %s | FileCheck %s --check-prefix=ALL --check-prefix=X86
+; RUN: llc -mtriple i386-windows-gnu -mattr=+ibt -exception-model sjlj < %s | FileCheck %s --check-prefix=SJLJ
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; Test1
 ;; -----
-;; Checks ENDBR insertion in case of indirect branch IR instruction.
+;; Checks ENDBR insertion in case of switch case statement.
 ;; Also since the function is not internal, make sure that endbr32/64 was 
 ;; added at the beginning of the function.
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
 define i8 @test1(){
 ; ALL-LABEL:   test1
 ; X86_64:      endbr64
 ; X86:         endbr32

 
 bb6:                                              ; preds = %entry
   ret i8 2
 }
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; Test2
 ;; -----
-;; Checks ENDBR insertion in case of switch case statement.
+;; Checks NOTRACK insertion in case of switch case statement.
+;; Check that there is no ENDBR insertion in the following case statements.
 ;; Also since the function is not internal, ENDBR instruction should be
 ;; added to its first basic block.
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
 define i32 @test2(i32 %a) {
 ; ALL-LABEL:   test2
 ; X86_64:      endbr64
 ; X86:         endbr32
-; ALL:         jmp{{q|l}} *
-; ALL:         .LBB1_2:
-; X86_64-NEXT: endbr64
-; X86-NEXT:    endbr32
-; ALL:         .LBB1_7:
+; ALL:         notrack jmp{{q|l}} *
 ; X86_64-NOT:      endbr64
 ; X86-NOT:         endbr32
-; ALL:         .LBB1_3:
-; X86_64-NEXT: endbr64
-; X86-NEXT:    endbr32
-; ALL:         .LBB1_4:
-; X86_64-NEXT: endbr64
-; X86-NEXT:    endbr32
-; ALL:         .LBB1_5:
-; X86_64-NEXT: endbr64
-; X86-NEXT:    endbr32
-; ALL:         .LBB1_6:
-; X86_64-NEXT: endbr64
-; X86-NEXT:    endbr32
 entry:
   %retval = alloca i32, align 4
   %a.addr = alloca i32, align 4
   store i32 %a, i32* %a.addr, align 4
   %0 = load i32, i32* %a.addr, align 4
   switch i32 %0, label %sw.default [
     i32 0, label %sw.bb
     i32 1, label %sw.bb1

 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
 define i32 @test7() {
 ; ALL-LABEL:   test7
 ; X86_64:      endbr64
 ; X86:         endbr32
   ret i32 1
 }
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; Test8
+;; -----
+;; Checks that NO TRACK prefix is not added for indirect jumps to a jump-
+;; table that was created for SJLJ dispatch.
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+declare void @_Z20function_that_throwsv()
+declare i32 @__gxx_personality_sj0(...)
+declare i8* @__cxa_begin_catch(i8*)
+declare void @__cxa_end_catch()
+
+define void @test8() personality i8* bitcast (i32 (...)* @__gxx_personality_sj0 to i8*) {
+;SJLJ-LABEL:    test8
+;SJLJ-NOT:      ds
+entry:
+  invoke void @_Z20function_that_throwsv()
+          to label %try.cont unwind label %lpad
+
+lpad:
+  %0 = landingpad { i8*, i32 }
+          catch i8* null
+  %1 = extractvalue { i8*, i32 } %0, 0
+  %2 = tail call i8* @__cxa_begin_catch(i8* %1)
+  tail call void @__cxa_end_catch()
+  br label %try.cont
+
+try.cont:
+  ret void
+}
+
+!llvm.module.flags = !{!0}
+
+!0 = !{i32 4, !"cf-protection-branch", i32 1}
 No newline at end of file

llvm/trunk/test/CodeGen/X86/nocf_check.ll

+; RUN: llc -mtriple=x86_64-unknown-unknown -mattr=+ibt -x86-indirect-branch-tracking < %s | FileCheck %s
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; This test verify the handling of ''nocf_check'' attribute by the backend. ;;
+;; The file was generated using the following C code:                        ;;
+;;                                                                           ;;
+;; void __attribute__((nocf_check)) NoCfCheckFunc(void) {}                   ;;
+;;                                                                           ;;
+;; typedef void(*FuncPointer)(void);                                         ;;
+;; void NoCfCheckCall(FuncPointer f) {                                       ;;
+;;   __attribute__((nocf_check)) FuncPointer p = f;                          ;;
+;;   (*p)();                                                                 ;;
+;; }                                                                         ;;
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+; Make sure that a function with ''nocf_check'' attribute is not instrumented
+; with endbr instruction at the beginning.
+define void @NoCfCheckFunc() #0 {
+; CHECK-LABEL: NoCfCheckFunc
+; CHECK-NOT:   endbr64
+; CHECK:       retq
+entry:
+  ret void
+}
+
+; Make sure that notrack prefix is added before a call with ''nocf_check'' attribute.
+define void @NoCfCheckCall(void ()* %f) {
+; CHECK-LABEL: NoCfCheckCall
+; CHECK:       notrack call
+entry:
+  %f.addr = alloca void ()*, align 4
+  %p = alloca void ()*, align 4
+  store void ()* %f, void ()** %f.addr, align 4
+  %0 = load void ()*, void ()** %f.addr, align 4
+  store void ()* %0, void ()** %p, align 4
+  %1 = load void ()*, void ()** %p, align 4
+  call void %1() #1
+  ret void
+}
+
+attributes #0 = { noinline nocf_check nounwind optnone "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "no-frame-pointer-elim"="false" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-features"="+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
+attributes #1 = { nocf_check }
+
+!llvm.module.flags = !{!0}
+
+!0 = !{i32 4, !"cf-protection-branch", i32 1}

llvm/trunk/test/MC/X86/x86-32-coverage.s

 // CHECK: sarl  305419896
 // CHECK:  encoding: [0xd1,0x3d,0x78,0x56,0x34,0x12]
           sarl  0x12345678
 
 // CHECK: calll *%ecx
 // CHECK:  encoding: [0xff,0xd1]
           call  *%ecx
 
+// CHECK: notrack calll *%ecx
+// CHECK:  encoding: [0x3e,0xff,0xd1]
+            notrack call  *%ecx
+
 // CHECK: calll *3735928559(%ebx,%ecx,8)
 // CHECK:  encoding: [0xff,0x94,0xcb,0xef,0xbe,0xad,0xde]
           call  *0xdeadbeef(%ebx,%ecx,8)
 
+// CHECK: notrack calll *3735928559(%ebx,%ecx,8)
+// CHECK:  encoding: [0x3e,0xff,0x94,0xcb,0xef,0xbe,0xad,0xde]
+            notrack call  *0xdeadbeef(%ebx,%ecx,8)
+
 // CHECK: calll *3135175374
 // CHECK:  encoding: [0xff,0x15,0xce,0xfa,0xde,0xba]
           call  *0xbadeface
 
 // CHECK: calll *3735928559(%ebx,%ecx,8)
 // CHECK:  encoding: [0xff,0x94,0xcb,0xef,0xbe,0xad,0xde]
           call  *0xdeadbeef(%ebx,%ecx,8)
 

 // CHECK: jmpl  *3135175374
 // CHECK:  encoding: [0xff,0x25,0xce,0xfa,0xde,0xba]
           jmp *0xbadeface
 
 // CHECK: jmpl  *3735928559(%ebx,%ecx,8)
 // CHECK:  encoding: [0xff,0xa4,0xcb,0xef,0xbe,0xad,0xde]
           jmp *0xdeadbeef(%ebx,%ecx,8)
 
+// CHECK: notrack jmpl  *3735928559(%ebx,%ecx,8)
+// CHECK:  encoding: [0x3e,0xff,0xa4,0xcb,0xef,0xbe,0xad,0xde]
+             notrack  jmp *0xdeadbeef(%ebx,%ecx,8)
+
 // CHECK: jmpl  *3135175374
 // CHECK:  encoding: [0xff,0x25,0xce,0xfa,0xde,0xba]
           jmp *0xbadeface
 
 // CHECK: ljmpl *3735928559(%ebx,%ecx,8)
 // CHECK:  encoding: [0xff,0xac,0xcb,0xef,0xbe,0xad,0xde]
           ljmpl *0xdeadbeef(%ebx,%ecx,8)
 

llvm/trunk/utils/TableGen/X86FoldTablesEmitter.cpp

if (RegRec->getValueAsDef("OpEnc") != MemRec->getValueAsDef("OpEnc") ||
         RegRec->getValueAsBit("hasEVEX_B") != 0 ||
         MemRec->getValueAsBit("hasEVEX_B") != 0 ||
         RegRec->getValueAsBit("hasEVEX_RC") !=
             MemRec->getValueAsBit("hasEVEX_RC") ||
         RegRec->getValueAsBit("hasREX_WPrefix") !=
             MemRec->getValueAsBit("hasREX_WPrefix") ||
         RegRec->getValueAsBit("hasLockPrefix") !=
             MemRec->getValueAsBit("hasLockPrefix") ||
+        RegRec->getValueAsBit("hasNoTrackPrefix") !=
+            MemRec->getValueAsBit("hasNoTrackPrefix") ||
         !equalBitsInits(RegRec->getValueAsBitsInit("EVEX_LL"),
                         MemRec->getValueAsBitsInit("EVEX_LL")) ||
         !equalBitsInits(RegRec->getValueAsBitsInit("VEX_WPrefix"),
                         MemRec->getValueAsBitsInit("VEX_WPrefix")) ||
         // Instruction's format - The register form's "Form" field should be
         // the opposite of the memory form's "Form" field.
         !areOppositeForms(RegRec->getValueAsBitsInit("FormBits"),
                           MemRec->getValueAsBitsInit("FormBits")) ||