[XRay] Typed event logging intrinsic

[kpw]

Summary

Add an LLVM intrinsic for type discriminated event logging with XRay.
Similar to the existing intrinsic for custom events, but also accepts
a type tag argument to allow plugins to be aware of different types
and semantically interpret logged events they know about without
choking on those they don't.

Relies on a symbol defined in compiler-rt patch D43668. I may wait
to submit before I can see demo everything working together including
a still to come clang patch.

[kpw]

Adding some comments for questions I had. Please chime in if you can shed light on these.

include/llvm/IR/Intrinsics.td

900 ↗

(On Diff #142446)

I confess I copied IntrWriteMem from the custom event intrinsic without understanding why it is the right choice.

lib/Target/X86/X86MCInstLower.cpp

1098–1100 ↗

(On Diff #142446)

The reason I didn't do this now is that I wasn't sure whether it should be done with a sled version change.

While it is obviously a change, it doesn't require a change in the patching/unpatching logic.

test/CodeGen/X86/xray-typed-event-log.ll

13 ↗

(On Diff #142446)

I would have liked to have an additional test for when the registers match up and the sled uses noops, but I couldn't figure out how to force that in IR.

I tried a variant with "call cc 78 void ..." to try and force SystemV, but I think at the ir level, but it didn't have the desired effect. Any ideas?

[dberris]

LGTM

include/llvm/IR/Intrinsics.td

900 ↗

(On Diff #142446)

This is so that the optimisation passes cannot assume that the call to the intrinsic will not touch any global memory.

lib/Target/X86/X86MCInstLower.cpp

1098–1100 ↗

(On Diff #142446)

I think we can change the lowering now, as long as the number of bytes doesn't change. Also we don't need to change the sled version just for the re-ordering.

1188 ↗

(On Diff #142446)

Yes, mostly because we don't want to make the stack-based and/or constant global value handling special. It could be possible to support those too, but it's not as simple (and it's unclear whether we can do that with the same amount of bytes in the sled).

test/CodeGen/X86/xray-typed-event-log.ll

13 ↗

(On Diff #142446)

There's a lot of randomness in how the register allocators decide which registers to use. Unfortunately I don't think we can force that at the IR level. If you'd like to see how this works at the MIR level, that's an option but I don't know enough how to do this yet. Maybe look around at existing MIR tests to see whether that might be something you can try?

[kpw]

include/llvm/IR/Intrinsics.td

900 ↗

(On Diff #142446)

What about IntrReadMem as well? IntrWriteMem is commented as

This intrinsic writes to unspecified memory, but does not
read from memory, and has no other side effects. This means dead stores
before calls to this intrinsics may be removed.

lib/Target/X86/X86MCInstLower.cpp

1098–1100 ↗

(On Diff #142446)

That sounds fine, I'll add a change to this patch.

1188 ↗

(On Diff #142446)

Is there a better way we can support the contract that the intrinsic is invoked with register arguments than runtime assert?

test/CodeGen/X86/xray-typed-event-log.ll

13 ↗

(On Diff #142446)

I'll take a look at some MIR tests. Thanks.

[dberris]

include/llvm/IR/Intrinsics.td

900 ↗

(On Diff #142446)

Technically, the ReadOnly<1> attribute preserves the data pointed to, which is the minimal requirement for this intrinsic.

IntrReadMem signals that globals may be read, but the intrinsic itself really doesn't need this. This is so that we minimise the interference of the intrinsic when it's emitted in a function that may be in-lined. We want to say that this function acts like some sort of write barrier, but not a read barrier.

lib/Target/X86/X86MCInstLower.cpp

1188 ↗

(On Diff #142446)

The way we've forced the arguments to be in registers when lowering the pseudo-instruction is how we're telling the register allocators to do this for us. Until we come up with a way to force specific registers to be used at the MachineInstr level (which up until now I'm not sure is possible or desirable) we're going to be at the mercy of the default calling convention and therefore have to handle re-ordering the inputs per call-site.

The assertion here is meant to signal that we (us working on XRay) and the intervening optimisations on the machine instructions, have not messed up the fact that the arguments to the intrinsic call have to be in registers. Now there might be a way (similar to how the stack maps do it) to have much less interference with both typed events and custom events if we're able to record where the data is stored (could be global, could be in the stack already, could be a constant in a register already, etc), and emit the appropriate instrumentation point (with versioning and maybe some additional metadata).

We could do the more complicated thing which will reduce the register pressure from calling the intrinsic functions. But we'd need to measure the effects of that before committing to that course of action. :)

That's not to say we're not going to do it or that it will not be worth it -- I think we should do it, it's just a matter of "not now". But "not now" is closer to "in the next couple of months" rather than "never". :D

llvm/trunk/include/llvm/CodeGen/AsmPrinter.h

 public:
   // This describes the kind of sled we're storing in the XRay table.
   enum class SledKind : uint8_t {
     FUNCTION_ENTER = 0,
     FUNCTION_EXIT = 1,
     TAIL_CALL = 2,
     LOG_ARGS_ENTER = 3,
     CUSTOM_EVENT = 4,
+    TYPED_EVENT = 5,
   };
 
   // The table will contain these structs that point to the sled, the function
   // containing the sled, and what kind of sled (and whether they should always
   // be instrumented). We also use a version identifier that the runtime can use
   // to decide what to do with the sled, depending on the version of the sled.
   struct XRayFunctionEntry {
     const MCSymbol *Sled;
private:
   void EmitLLVMUsedList(const ConstantArray *InitList);
   /// Emit llvm.ident metadata in an '.ident' directive.
   void EmitModuleIdents(Module &M);
   void EmitXXStructorList(const DataLayout &DL, const Constant *List,
                           bool isCtor);
 
   GCMetadataPrinter *GetOrCreateGCPrinter(GCStrategy &C);
   /// Emit GlobalAlias or GlobalIFunc.
-  void emitGlobalIndirectSymbol(Module &M,
-                                const GlobalIndirectSymbol& GIS);
+  void emitGlobalIndirectSymbol(Module &M, const GlobalIndirectSymbol &GIS);
   void setupCodePaddingContext(const MachineBasicBlock &MBB,
                                MCCodePaddingContext &Context) const;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_ASMPRINTER_H

llvm/trunk/include/llvm/CodeGen/FastISel.h

protected:
   bool selectPatchpoint(const CallInst *I);
   bool selectCall(const User *Call);
   bool selectIntrinsicCall(const IntrinsicInst *II);
   bool selectBitCast(const User *I);
   bool selectCast(const User *I, unsigned Opcode);
   bool selectExtractValue(const User *I);
   bool selectInsertValue(const User *I);
   bool selectXRayCustomEvent(const CallInst *II);
+  bool selectXRayTypedEvent(const CallInst *II);
 
 private:
   /// \brief Handle PHI nodes in successor blocks.
   ///
   /// Emit code to ensure constants are copied into registers when needed.
   /// Remember the virtual registers that need to be added to the Machine PHI
   /// nodes as input.  We cannot just directly add them, because expansion might
   /// result in multiple MBB's for one BB.  As such, the start of the BB might

llvm/trunk/include/llvm/CodeGen/TargetLowering.h

protected:
   /// sequence of memory operands that is recognized by PrologEpilogInserter.
   MachineBasicBlock *emitPatchPoint(MachineInstr &MI,
                                     MachineBasicBlock *MBB) const;
 
   /// Replace/modify the XRay custom event operands with target-dependent
   /// details.
   MachineBasicBlock *emitXRayCustomEvent(MachineInstr &MI,
                                          MachineBasicBlock *MBB) const;
+
+  /// Replace/modify the XRay typed event operands with target-dependent
+  /// details.
+  MachineBasicBlock *emitXRayTypedEvent(MachineInstr &MI,
+                                        MachineBasicBlock *MBB) const;
 };
 
 /// This class defines information used to lower LLVM code to legal SelectionDAG
 /// operators that the target instruction selector can accept natively.
 ///
 /// This class also defines callbacks that targets must implement to lower
 /// target-specific constructs to SelectionDAG operators.
 class TargetLowering : public TargetLoweringBase {

llvm/trunk/include/llvm/IR/Intrinsics.td

def int_load_relative: Intrinsic<[llvm_ptr_ty], [llvm_ptr_ty, llvm_anyint_ty],
                                  [IntrReadMem, IntrArgMemOnly]>;
 
 // Xray intrinsics
 //===----------------------------------------------------------------------===//
 // Custom event logging for x-ray.
 // Takes a pointer to a string and the length of the string.
 def int_xray_customevent : Intrinsic<[], [llvm_ptr_ty, llvm_i32_ty],
                                      [NoCapture<0>, ReadOnly<0>, IntrWriteMem]>;
+// Typed event logging for x-ray.
+// Takes a numeric type tag, a pointer to a string and the length of the string.
+def int_xray_typedevent : Intrinsic<[], [llvm_i16_ty, llvm_ptr_ty, llvm_i32_ty],
+                                        [NoCapture<1>, ReadOnly<1>, IntrWriteMem]>;
 //===----------------------------------------------------------------------===//
 
 //===------ Memory intrinsics with element-wise atomicity guarantees ------===//
 //
 
 // @llvm.memcpy.element.unordered.atomic.*(dest, src, length, elementsize)
 def int_memcpy_element_unordered_atomic
     : Intrinsic<[],

llvm/trunk/include/llvm/Support/TargetOpcodes.def

 /// call it, it does the tracing, preserves the stack and returns.
 HANDLE_TARGET_OPCODE(PATCHABLE_FUNCTION_EXIT)
 
 /// Wraps a tail call instruction and its operands to enable adding nop sleds
 /// either before or after the tail exit. We use this as a disambiguation from
 /// PATCHABLE_RET which specifically only works for return instructions.
 HANDLE_TARGET_OPCODE(PATCHABLE_TAIL_CALL)
 
-/// Wraps a logging call and its arguments with nop sleds. At runtime, this can be
-/// patched to insert instrumentation instructions.
+/// Wraps a logging call and its arguments with nop sleds. At runtime, this can
+/// be patched to insert instrumentation instructions.
 HANDLE_TARGET_OPCODE(PATCHABLE_EVENT_CALL)
 
+/// Wraps a typed logging call and its argument with nop sleds. At runtime, this
+/// can be patched to insert instrumentation instructions.
+HANDLE_TARGET_OPCODE(PATCHABLE_TYPED_EVENT_CALL)
+
 HANDLE_TARGET_OPCODE(ICALL_BRANCH_FUNNEL)
 
 /// The following generic opcodes are not supposed to appear after ISel.
 /// This is something we might want to relax, but for now, this is convenient
 /// to produce diagnostics.
 
 /// Generic ADD instruction. This is an integer add.
 HANDLE_TARGET_OPCODE(G_ADD)

llvm/trunk/include/llvm/Target/Target.td

def PATCHABLE_EVENT_CALL : StandardPseudoInstruction {
   let InOperandList = (ins ptr_rc:$event, i8imm:$size);
   let AsmString = "# XRay Custom Event Log.";
   let usesCustomInserter = 1;
   let isCall = 1;
   let mayLoad = 1;
   let mayStore = 1;
   let hasSideEffects = 1;
 }
+def PATCHABLE_TYPED_EVENT_CALL : StandardPseudoInstruction {
+  let OutOperandList = (outs);
+  let InOperandList = (ins i16imm:$type, ptr_rc:$event, i32imm:$size);
+  let AsmString = "# XRay Typed Event Log.";
+  let usesCustomInserter = 1;
+  let isCall = 1;
+  let mayLoad = 1;
+  let mayStore = 1;
+  let hasSideEffects = 1;
+}
 def FENTRY_CALL : StandardPseudoInstruction {
   let OutOperandList = (outs unknown:$dst);
   let InOperandList = (ins variable_ops);
   let AsmString = "# FEntry call";
   let usesCustomInserter = 1;
   let mayLoad = 1;
   let mayStore = 1;
   let hasSideEffects = 1;

llvm/trunk/lib/CodeGen/SelectionDAG/FastISel.cpp

MachineInstrBuilder MIB =
               TII.get(TargetOpcode::PATCHABLE_EVENT_CALL));
   for (auto &MO : Ops)
     MIB.add(MO);
 
   // Insert the Patchable Event Call instruction, that gets lowered properly.
   return true;
 }
 
+bool FastISel::selectXRayTypedEvent(const CallInst *I) {
+  const auto &Triple = TM.getTargetTriple();
+  if (Triple.getArch() != Triple::x86_64 || !Triple.isOSLinux())
+    return true; // don't do anything to this instruction.
+  SmallVector<MachineOperand, 8> Ops;
+  Ops.push_back(MachineOperand::CreateReg(getRegForValue(I->getArgOperand(0)),
+                                          /*IsDef=*/false));
+  Ops.push_back(MachineOperand::CreateReg(getRegForValue(I->getArgOperand(1)),
+                                          /*IsDef=*/false));
+  Ops.push_back(MachineOperand::CreateReg(getRegForValue(I->getArgOperand(2)),
+                                          /*IsDef=*/false));
+  MachineInstrBuilder MIB =
+      BuildMI(*FuncInfo.MBB, FuncInfo.InsertPt, DbgLoc,
+              TII.get(TargetOpcode::PATCHABLE_TYPED_EVENT_CALL));
+  for (auto &MO : Ops)
+    MIB.add(MO);
+
+  // Insert the Patchable Typed Event Call instruction, that gets lowered properly.
+  return true;
+}
 
 /// Returns an AttributeList representing the attributes applied to the return
 /// value of the given call.
 static AttributeList getReturnAttrs(FastISel::CallLoweringInfo &CLI) {
   SmallVector<Attribute::AttrKind, 2> Attrs;
   if (CLI.RetSExt)
     Attrs.push_back(Attribute::SExt);
   if (CLI.RetZExt)
bool FastISel::selectIntrinsicCall(const IntrinsicInst *II) {
   case Intrinsic::experimental_stackmap:
     return selectStackmap(II);
   case Intrinsic::experimental_patchpoint_void:
   case Intrinsic::experimental_patchpoint_i64:
     return selectPatchpoint(II);
 
   case Intrinsic::xray_customevent:
     return selectXRayCustomEvent(II);
+  case Intrinsic::xray_typedevent:
+    return selectXRayTypedEvent(II);
   }
 
   return fastLowerIntrinsicCall(II);
 }
 
 bool FastISel::selectCast(const User *I, unsigned Opcode) {
   EVT SrcVT = TLI.getValueType(DL, I->getOperand(0)->getType());
   EVT DstVT = TLI.getValueType(DL, I->getType());

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

case Intrinsic::xray_customevent: {
     // them across calls to the intrinsic.
     MachineSDNode *MN = DAG.getMachineNode(TargetOpcode::PATCHABLE_EVENT_CALL,
                                            DL, NodeTys, Ops);
     SDValue patchableNode = SDValue(MN, 0);
     DAG.setRoot(patchableNode);
     setValue(&I, patchableNode);
     return nullptr;
   }
+  case Intrinsic::xray_typedevent: {
+    // Here we want to make sure that the intrinsic behaves as if it has a
+    // specific calling convention, and only for x86_64.
+    // FIXME: Support other platforms later.
+    const auto &Triple = DAG.getTarget().getTargetTriple();
+    if (Triple.getArch() != Triple::x86_64 || !Triple.isOSLinux())
+      return nullptr;
+
+    SDLoc DL = getCurSDLoc();
+    SmallVector<SDValue, 8> Ops;
+
+    // We want to say that we always want the arguments in registers.
+    // It's unclear to me how manipulating the selection DAG here forces callers
+    // to provide arguments in registers instead of on the stack.
+    SDValue LogTypeId = getValue(I.getArgOperand(0));
+    SDValue LogEntryVal = getValue(I.getArgOperand(1));
+    SDValue StrSizeVal = getValue(I.getArgOperand(2));
+    SDVTList NodeTys = DAG.getVTList(MVT::Other, MVT::Glue);
+    SDValue Chain = getRoot();
+    Ops.push_back(LogTypeId);
+    Ops.push_back(LogEntryVal);
+    Ops.push_back(StrSizeVal);
+    Ops.push_back(Chain);
+
+    // We need to enforce the calling convention for the callsite, so that
+    // argument ordering is enforced correctly, and that register allocation can
+    // see that some registers may be assumed clobbered and have to preserve
+    // them across calls to the intrinsic.
+    MachineSDNode *MN = DAG.getMachineNode(
+        TargetOpcode::PATCHABLE_TYPED_EVENT_CALL, DL, NodeTys, Ops);
+    SDValue patchableNode = SDValue(MN, 0);
+    DAG.setRoot(patchableNode);
+    setValue(&I, patchableNode);
+    return nullptr;
+  }
   case Intrinsic::experimental_deoptimize:
     LowerDeoptimizeCall(&I);
     return nullptr;
 
   case Intrinsic::experimental_vector_reduce_fadd:
   case Intrinsic::experimental_vector_reduce_fmul:
   case Intrinsic::experimental_vector_reduce_add:
   case Intrinsic::experimental_vector_reduce_mul:

llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp

TargetLoweringBase::emitXRayCustomEvent(MachineInstr &MI,
   for (unsigned OpIdx = 0; OpIdx != MI.getNumOperands(); ++OpIdx)
     MIB.add(MI.getOperand(OpIdx));
 
   MBB->insert(MachineBasicBlock::iterator(MI), MIB);
   MI.eraseFromParent();
   return MBB;
 }
 
+MachineBasicBlock *
+TargetLoweringBase::emitXRayTypedEvent(MachineInstr &MI,
+                                       MachineBasicBlock *MBB) const {
+  assert(MI.getOpcode() == TargetOpcode::PATCHABLE_TYPED_EVENT_CALL &&
+         "Called emitXRayTypedEvent on the wrong MI!");
+  auto &MF = *MI.getMF();
+  auto MIB = BuildMI(MF, MI.getDebugLoc(), MI.getDesc());
+  for (unsigned OpIdx = 0; OpIdx != MI.getNumOperands(); ++OpIdx)
+    MIB.add(MI.getOperand(OpIdx));
+
+  MBB->insert(MachineBasicBlock::iterator(MI), MIB);
+  MI.eraseFromParent();
+  return MBB;
+}
+
 /// findRepresentativeClass - Return the largest legal super-reg register class
 /// of the register class for the specified type and its associated "cost".
 // This function is in TargetLowering because it uses RegClassForVT which would
 // need to be moved to TargetRegisterInfo and would necessitate moving
 // isTypeLegal over as well - a massive change that would just require
 // TargetLowering having a TargetRegisterInfo class member that it would use.
 std::pair<const TargetRegisterClass *, uint8_t>
 TargetLoweringBase::findRepresentativeClass(const TargetRegisterInfo *TRI,

llvm/trunk/lib/Target/X86/X86AsmPrinter.h

class LLVM_LIBRARY_VISIBILITY X86AsmPrinter : public AsmPrinter {
   void LowerTlsAddr(X86MCInstLower &MCInstLowering, const MachineInstr &MI);
 
   // XRay-specific lowering for X86.
   void LowerPATCHABLE_FUNCTION_ENTER(const MachineInstr &MI,
                                      X86MCInstLower &MCIL);
   void LowerPATCHABLE_RET(const MachineInstr &MI, X86MCInstLower &MCIL);
   void LowerPATCHABLE_TAIL_CALL(const MachineInstr &MI, X86MCInstLower &MCIL);
   void LowerPATCHABLE_EVENT_CALL(const MachineInstr &MI, X86MCInstLower &MCIL);
+  void LowerPATCHABLE_TYPED_EVENT_CALL(const MachineInstr &MI,
+                                       X86MCInstLower &MCIL);
 
   void LowerFENTRY_CALL(const MachineInstr &MI, X86MCInstLower &MCIL);
 
   // Choose between emitting .seh_ directives and .cv_fpo_ directives.
   void EmitSEHInstruction(const MachineInstr *MI);
 
 public:
   X86AsmPrinter(TargetMachine &TM, std::unique_ptr<MCStreamer> Streamer);

llvm/trunk/lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
 
   case TargetOpcode::STACKMAP:
   case TargetOpcode::PATCHPOINT:
     return emitPatchPoint(MI, BB);
 
   case TargetOpcode::PATCHABLE_EVENT_CALL:
     return emitXRayCustomEvent(MI, BB);
 
+  case TargetOpcode::PATCHABLE_TYPED_EVENT_CALL:
+    return emitXRayTypedEvent(MI, BB);
+
   case X86::LCMPXCHG8B: {
     const X86RegisterInfo *TRI = Subtarget.getRegisterInfo();
     // In addition to 4 E[ABCD] registers implied by encoding, CMPXCHG8B
     // requires a memory operand. If it happens that current architecture is
     // i686 and for current function we need a base pointer
     // - which is ESI for i686 - register allocator would not be able to
     // allocate registers for an address in form of X(%reg, %reg, Y)
     // - there never would be enough unreserved registers during regalloc

llvm/trunk/lib/Target/X86/X86MCInstLower.cpp

 
 /// X86MCInstLower - This class is used to lower an MachineInstr into an MCInst.
 class X86MCInstLower {
   MCContext &Ctx;
   const MachineFunction &MF;
   const TargetMachine &TM;
   const MCAsmInfo &MAI;
   X86AsmPrinter &AsmPrinter;
+
 public:
   X86MCInstLower(const MachineFunction &MF, X86AsmPrinter &asmprinter);
 
   Optional<MCOperand> LowerMachineOperand(const MachineInstr *MI,
                                           const MachineOperand &MO) const;
   void Lower(const MachineInstr *MI, MCInst &OutMI) const;
 
   MCSymbol *GetSymbolFromOperand(const MachineOperand &MO) const;
X86MCInstLower::X86MCInstLower(const MachineFunction &mf,
                                X86AsmPrinter &asmprinter)
     : Ctx(mf.getContext()), MF(mf), TM(mf.getTarget()), MAI(*TM.getMCAsmInfo()),
       AsmPrinter(asmprinter) {}
 
 MachineModuleInfoMachO &X86MCInstLower::getMachOMMI() const {
   return MF.getMMI().getObjFileInfo<MachineModuleInfoMachO>();
 }
 
-
 /// GetSymbolFromOperand - Lower an MO_GlobalAddress or MO_ExternalSymbol
 /// operand to an MCSymbol.
-MCSymbol *X86MCInstLower::
-GetSymbolFromOperand(const MachineOperand &MO) const {
+MCSymbol *X86MCInstLower::GetSymbolFromOperand(const MachineOperand &MO) const {
   const DataLayout &DL = MF.getDataLayout();
-  assert((MO.isGlobal() || MO.isSymbol() || MO.isMBB()) && "Isn't a symbol reference");
+  assert((MO.isGlobal() || MO.isSymbol() || MO.isMBB()) &&
+         "Isn't a symbol reference");
 
   MCSymbol *Sym = nullptr;
   SmallString<128> Name;
   StringRef Suffix;
 
   switch (MO.getTargetFlags()) {
   case X86II::MO_DLLIMPORT:
     // Handle dllimport linkage.
MCSymbol *X86MCInstLower::GetSymbolFromOperand(const MachineOperand &MO) const {
 
   Name += Suffix;
   if (!Sym)
     Sym = Ctx.getOrCreateSymbol(Name);
 
   // If the target flags on the operand changes the name of the symbol, do that
   // before we return the symbol.
   switch (MO.getTargetFlags()) {
-  default: break;
+  default:
+    break;
   case X86II::MO_DARWIN_NONLAZY:
   case X86II::MO_DARWIN_NONLAZY_PIC_BASE: {
     MachineModuleInfoImpl::StubValueTy &StubSym =
         getMachOMMI().getGVStubEntry(Sym);
     if (!StubSym.getPointer()) {
       assert(MO.isGlobal() && "Extern symbol not handled yet");
-      StubSym =
-        MachineModuleInfoImpl::
-        StubValueTy(AsmPrinter.getSymbol(MO.getGlobal()),
-                    !MO.getGlobal()->hasInternalLinkage());
+      StubSym = MachineModuleInfoImpl::StubValueTy(
+          AsmPrinter.getSymbol(MO.getGlobal()),
     }
     break;
   }
   }
 
   return Sym;
 }
 
 MCOperand X86MCInstLower::LowerSymbolOperand(const MachineOperand &MO,
                                              MCSymbol *Sym) const {
   // FIXME: We would like an efficient form for this, so we don't have to do a
   // lot of extra uniquing.
   const MCExpr *Expr = nullptr;
   MCSymbolRefExpr::VariantKind RefKind = MCSymbolRefExpr::VK_None;
 
   switch (MO.getTargetFlags()) {
-  default: llvm_unreachable("Unknown target flag on GV operand");
-  case X86II::MO_NO_FLAG:    // No flag.
+  default:
+    llvm_unreachable("Unknown target flag on GV operand");
+  case X86II::MO_NO_FLAG: // No flag.
   // These affect the name of the symbol, not any suffix.
   case X86II::MO_DARWIN_NONLAZY:
   case X86II::MO_DLLIMPORT:
     break;
 
-  case X86II::MO_TLVP:      RefKind = MCSymbolRefExpr::VK_TLVP; break;
+  case X86II::MO_TLVP:
+    RefKind = MCSymbolRefExpr::VK_TLVP;
+    break;
   case X86II::MO_TLVP_PIC_BASE:
     Expr = MCSymbolRefExpr::create(Sym, MCSymbolRefExpr::VK_TLVP, Ctx);
     // Subtract the pic base.
-    Expr = MCBinaryExpr::createSub(Expr,
-                                  MCSymbolRefExpr::create(MF.getPICBaseSymbol(),
-                                                           Ctx),
-                                   Ctx);
-    break;
-  case X86II::MO_SECREL:    RefKind = MCSymbolRefExpr::VK_SECREL; break;
-  case X86II::MO_TLSGD:     RefKind = MCSymbolRefExpr::VK_TLSGD; break;
-  case X86II::MO_TLSLD:     RefKind = MCSymbolRefExpr::VK_TLSLD; break;
-  case X86II::MO_TLSLDM:    RefKind = MCSymbolRefExpr::VK_TLSLDM; break;
-  case X86II::MO_GOTTPOFF:  RefKind = MCSymbolRefExpr::VK_GOTTPOFF; break;
-  case X86II::MO_INDNTPOFF: RefKind = MCSymbolRefExpr::VK_INDNTPOFF; break;
-  case X86II::MO_TPOFF:     RefKind = MCSymbolRefExpr::VK_TPOFF; break;
-  case X86II::MO_DTPOFF:    RefKind = MCSymbolRefExpr::VK_DTPOFF; break;
-  case X86II::MO_NTPOFF:    RefKind = MCSymbolRefExpr::VK_NTPOFF; break;
-  case X86II::MO_GOTNTPOFF: RefKind = MCSymbolRefExpr::VK_GOTNTPOFF; break;
-  case X86II::MO_GOTPCREL:  RefKind = MCSymbolRefExpr::VK_GOTPCREL; break;
-  case X86II::MO_GOT:       RefKind = MCSymbolRefExpr::VK_GOT; break;
-  case X86II::MO_GOTOFF:    RefKind = MCSymbolRefExpr::VK_GOTOFF; break;
-  case X86II::MO_PLT:       RefKind = MCSymbolRefExpr::VK_PLT; break;
-  case X86II::MO_ABS8:      RefKind = MCSymbolRefExpr::VK_X86_ABS8; break;
+    Expr = MCBinaryExpr::createSub(
+        Expr, MCSymbolRefExpr::create(MF.getPICBaseSymbol(), Ctx), Ctx);
+    break;
+  case X86II::MO_SECREL:
+    RefKind = MCSymbolRefExpr::VK_SECREL;
+    break;
+  case X86II::MO_TLSGD:
+    RefKind = MCSymbolRefExpr::VK_TLSGD;
+    break;
+  case X86II::MO_TLSLD:
+    RefKind = MCSymbolRefExpr::VK_TLSLD;
+    break;
+  case X86II::MO_TLSLDM:
+    RefKind = MCSymbolRefExpr::VK_TLSLDM;
+    break;
+  case X86II::MO_GOTTPOFF:
+    RefKind = MCSymbolRefExpr::VK_GOTTPOFF;
+    break;
+  case X86II::MO_INDNTPOFF:
+    RefKind = MCSymbolRefExpr::VK_INDNTPOFF;
+    break;
+  case X86II::MO_TPOFF:
+    RefKind = MCSymbolRefExpr::VK_TPOFF;
+    break;
+  case X86II::MO_DTPOFF:
+    RefKind = MCSymbolRefExpr::VK_DTPOFF;
+    break;
+  case X86II::MO_NTPOFF:
+    RefKind = MCSymbolRefExpr::VK_NTPOFF;
+    break;
+  case X86II::MO_GOTNTPOFF:
+    RefKind = MCSymbolRefExpr::VK_GOTNTPOFF;
+    break;
+  case X86II::MO_GOTPCREL:
+    RefKind = MCSymbolRefExpr::VK_GOTPCREL;
+    break;
+  case X86II::MO_GOT:
+    RefKind = MCSymbolRefExpr::VK_GOT;
+    break;
+  case X86II::MO_GOTOFF:
+    RefKind = MCSymbolRefExpr::VK_GOTOFF;
+    break;
+  case X86II::MO_PLT:
+    RefKind = MCSymbolRefExpr::VK_PLT;
+    break;
+  case X86II::MO_ABS8:
+    RefKind = MCSymbolRefExpr::VK_X86_ABS8;
+    break;
   case X86II::MO_PIC_BASE_OFFSET:
   case X86II::MO_DARWIN_NONLAZY_PIC_BASE:
     Expr = MCSymbolRefExpr::create(Sym, Ctx);
     // Subtract the pic base.
-    Expr = MCBinaryExpr::createSub(Expr,
-                            MCSymbolRefExpr::create(MF.getPICBaseSymbol(), Ctx),
-                                   Ctx);
+    Expr = MCBinaryExpr::createSub(
+        Expr, MCSymbolRefExpr::create(MF.getPICBaseSymbol(), Ctx), Ctx);
     if (MO.isJTI()) {
       assert(MAI.doesSetDirectiveSuppressReloc());
       // If .set directive is supported, use it to reduce the number of
       // relocations the assembler will generate for differences between
       // local labels. This is only safe when the symbols are in the same
       // section so we are restricting it to jumptable references.
       MCSymbol *Label = Ctx.createTempSymbol();
       AsmPrinter.OutStreamer->EmitAssignment(Label, Expr);
       Expr = MCSymbolRefExpr::create(Label, Ctx);
     }
     break;
   }
 
   if (!Expr)
     Expr = MCSymbolRefExpr::create(Sym, RefKind, Ctx);
 
   if (!MO.isJTI() && !MO.isMBB() && MO.getOffset())
-    Expr = MCBinaryExpr::createAdd(Expr,
-                                   MCConstantExpr::create(MO.getOffset(), Ctx),
-                                   Ctx);
+    Expr = MCBinaryExpr::createAdd(
+        Expr, MCConstantExpr::create(MO.getOffset(), Ctx), Ctx);
   return MCOperand::createExpr(Expr);
 }
 
-
 /// \brief Simplify FOO $imm, %{al,ax,eax,rax} to FOO $imm, for instruction with
 /// a short fixed-register form.
 static void SimplifyShortImmForm(MCInst &Inst, unsigned Opcode) {
   unsigned ImmOp = Inst.getNumOperands() - 1;
   assert(Inst.getOperand(0).isReg() &&
          (Inst.getOperand(ImmOp).isImm() || Inst.getOperand(ImmOp).isExpr()) &&
          ((Inst.getNumOperands() == 3 && Inst.getOperand(1).isReg() &&
            Inst.getOperand(0).getReg() == Inst.getOperand(1).getReg()) ||
-          Inst.getNumOperands() == 2) && "Unexpected instruction!");
+          Inst.getNumOperands() == 2) &&
+         "Unexpected instruction!");
 
   // Check whether the destination register can be fixed.
   unsigned Reg = Inst.getOperand(0).getReg();
   if (Reg != X86::AL && Reg != X86::AX && Reg != X86::EAX && Reg != X86::RAX)
     return;
 
   // If so, rewrite the instruction.
   MCOperand Saved = Inst.getOperand(ImmOp);
   Inst = MCInst();
   Inst.setOpcode(Opcode);
   Inst.addOperand(Saved);
 }
 
 /// \brief If a movsx instruction has a shorter encoding for the used register
 /// simplify the instruction to use it instead.
 static void SimplifyMOVSX(MCInst &Inst) {
   unsigned NewOpcode = 0;
   unsigned Op0 = Inst.getOperand(0).getReg(), Op1 = Inst.getOperand(1).getReg();
   switch (Inst.getOpcode()) {
   default:
     llvm_unreachable("Unexpected instruction!");
-  case X86::MOVSX16rr8:  // movsbw %al, %ax   --> cbtw
+  case X86::MOVSX16rr8: // movsbw %al, %ax   --> cbtw
     if (Op0 == X86::AX && Op1 == X86::AL)
       NewOpcode = X86::CBW;
     break;
   case X86::MOVSX32rr16: // movswl %ax, %eax  --> cwtl
     if (Op0 == X86::EAX && Op1 == X86::AX)
       NewOpcode = X86::CWDE;
     break;
   case X86::MOVSX64rr32: // movslq %eax, %rax --> cltq
static void SimplifyShortMoveForm(X86AsmPrinter &Printer, MCInst &Inst,
   // perform them because they make the code larger.
   if (Printer.getSubtarget().is64Bit())
     return;
 
   bool IsStore = Inst.getOperand(0).isReg() && Inst.getOperand(1).isReg();
   unsigned AddrBase = IsStore;
   unsigned RegOp = IsStore ? 0 : 5;
   unsigned AddrOp = AddrBase + 3;
-  assert(Inst.getNumOperands() == 6 && Inst.getOperand(RegOp).isReg() &&
-         Inst.getOperand(AddrBase + X86::AddrBaseReg).isReg() &&
-         Inst.getOperand(AddrBase + X86::AddrScaleAmt).isImm() &&
-         Inst.getOperand(AddrBase + X86::AddrIndexReg).isReg() &&
-         Inst.getOperand(AddrBase + X86::AddrSegmentReg).isReg() &&
-         (Inst.getOperand(AddrOp).isExpr() ||
-          Inst.getOperand(AddrOp).isImm()) &&
-         "Unexpected instruction!");
+  assert(
+      Inst.getNumOperands() == 6 && Inst.getOperand(RegOp).isReg() &&
+      (Inst.getOperand(AddrOp).isExpr() || Inst.getOperand(AddrOp).isImm()) &&
 
   // Check whether the destination register can be fixed.
   unsigned Reg = Inst.getOperand(RegOp).getReg();
   if (Reg != X86::AL && Reg != X86::AX && Reg != X86::EAX && Reg != X86::RAX)
     return;
 
   // Check whether this is an absolute address.
   // FIXME: We know TLVP symbol refs aren't, but there should be a better way
void X86MCInstLower::Lower(const MachineInstr *MI, MCInst &OutMI) const {
   // Handle a few special cases to eliminate operand modifiers.
 ReSimplify:
   switch (OutMI.getOpcode()) {
   case X86::LEA64_32r:
   case X86::LEA64r:
   case X86::LEA16r:
   case X86::LEA32r:
     // LEA should have a segment register, but it must be empty.
-    assert(OutMI.getNumOperands() == 1+X86::AddrNumOperands &&
+    assert(OutMI.getNumOperands() == 1 + X86::AddrNumOperands &&
            "Unexpected # of LEA operands");
-    assert(OutMI.getOperand(1+X86::AddrSegmentReg).getReg() == 0 &&
+    assert(OutMI.getOperand(1 + X86::AddrSegmentReg).getReg() == 0 &&
            "LEA has segment specified!");
     break;
 
   // Commute operands to get a smaller encoding by using VEX.R instead of VEX.B
   // if one of the registers is extended, but other isn't.
   case X86::VMOVZPQILo2PQIrr:
   case X86::VMOVAPDrr:
   case X86::VMOVAPDYrr:
   case X86::VMOVAPSrr:
   case X86::VMOVAPSYrr:
   case X86::VMOVDQArr:
   case X86::VMOVDQAYrr:
   case X86::VMOVDQUrr:
   case X86::VMOVDQUYrr:
   case X86::VMOVUPDrr:
   case X86::VMOVUPDYrr:
   case X86::VMOVUPSrr:
   case X86::VMOVUPSYrr: {
     if (!X86II::isX86_64ExtendedReg(OutMI.getOperand(0).getReg()) &&
         X86II::isX86_64ExtendedReg(OutMI.getOperand(1).getReg())) {
       unsigned NewOpc;
       switch (OutMI.getOpcode()) {
-      default: llvm_unreachable("Invalid opcode");
-      case X86::VMOVZPQILo2PQIrr: NewOpc = X86::VMOVPQI2QIrr;   break;
-      case X86::VMOVAPDrr:        NewOpc = X86::VMOVAPDrr_REV;  break;
-      case X86::VMOVAPDYrr:       NewOpc = X86::VMOVAPDYrr_REV; break;
-      case X86::VMOVAPSrr:        NewOpc = X86::VMOVAPSrr_REV;  break;
-      case X86::VMOVAPSYrr:       NewOpc = X86::VMOVAPSYrr_REV; break;
-      case X86::VMOVDQArr:        NewOpc = X86::VMOVDQArr_REV;  break;
-      case X86::VMOVDQAYrr:       NewOpc = X86::VMOVDQAYrr_REV; break;
-      case X86::VMOVDQUrr:        NewOpc = X86::VMOVDQUrr_REV;  break;
-      case X86::VMOVDQUYrr:       NewOpc = X86::VMOVDQUYrr_REV; break;
-      case X86::VMOVUPDrr:        NewOpc = X86::VMOVUPDrr_REV;  break;
-      case X86::VMOVUPDYrr:       NewOpc = X86::VMOVUPDYrr_REV; break;
-      case X86::VMOVUPSrr:        NewOpc = X86::VMOVUPSrr_REV;  break;
-      case X86::VMOVUPSYrr:       NewOpc = X86::VMOVUPSYrr_REV; break;
+      default:
+        llvm_unreachable("Invalid opcode");
+      case X86::VMOVZPQILo2PQIrr:
+        NewOpc = X86::VMOVPQI2QIrr;
+        break;
+      case X86::VMOVAPDrr:
+        NewOpc = X86::VMOVAPDrr_REV;
+        break;
+      case X86::VMOVAPDYrr:
+        NewOpc = X86::VMOVAPDYrr_REV;
+        break;
+      case X86::VMOVAPSrr:
+        NewOpc = X86::VMOVAPSrr_REV;
+        break;
+      case X86::VMOVAPSYrr:
+        NewOpc = X86::VMOVAPSYrr_REV;
+        break;
+      case X86::VMOVDQArr:
+        NewOpc = X86::VMOVDQArr_REV;
+        break;
+      case X86::VMOVDQAYrr:
+        NewOpc = X86::VMOVDQAYrr_REV;
+        break;
+      case X86::VMOVDQUrr:
+        NewOpc = X86::VMOVDQUrr_REV;
+        break;
+      case X86::VMOVDQUYrr:
+        NewOpc = X86::VMOVDQUYrr_REV;
+        break;
+      case X86::VMOVUPDrr:
+        NewOpc = X86::VMOVUPDrr_REV;
+        break;
+      case X86::VMOVUPDYrr:
+        NewOpc = X86::VMOVUPDYrr_REV;
+        break;
+      case X86::VMOVUPSrr:
+        NewOpc = X86::VMOVUPSrr_REV;
+        break;
+      case X86::VMOVUPSYrr:
+        NewOpc = X86::VMOVUPSYrr_REV;
+        break;
       }
       OutMI.setOpcode(NewOpc);
     }
     break;
   }
   case X86::VMOVSDrr:
   case X86::VMOVSSrr: {
     if (!X86II::isX86_64ExtendedReg(OutMI.getOperand(0).getReg()) &&
         X86II::isX86_64ExtendedReg(OutMI.getOperand(2).getReg())) {
       unsigned NewOpc;
       switch (OutMI.getOpcode()) {
-      default: llvm_unreachable("Invalid opcode");
-      case X86::VMOVSDrr:   NewOpc = X86::VMOVSDrr_REV;   break;
-      case X86::VMOVSSrr:   NewOpc = X86::VMOVSSrr_REV;   break;
+      default:
+        llvm_unreachable("Invalid opcode");
+      case X86::VMOVSDrr:
+        NewOpc = X86::VMOVSDrr_REV;
+        break;
+      case X86::VMOVSSrr:
+        NewOpc = X86::VMOVSSrr_REV;
+        break;
       }
       OutMI.setOpcode(NewOpc);
     }
     break;
   }
 
   // TAILJMPr64, CALL64r, CALL64pcrel32 - These instructions have register
   // inputs modeled as normal uses instead of implicit uses.  As such, truncate
case X86::CATCHRET: {
     const X86Subtarget &Subtarget = AsmPrinter.getSubtarget();
     unsigned ReturnReg = Subtarget.is64Bit() ? X86::RAX : X86::EAX;
     OutMI = MCInst();
     OutMI.setOpcode(getRetOpcode(Subtarget));
     OutMI.addOperand(MCOperand::createReg(ReturnReg));
     break;
   }
 
-  // TAILJMPd, TAILJMPd64, TailJMPd_cc - Lower to the correct jump instruction.
-  { unsigned Opcode;
-  case X86::TAILJMPr:   Opcode = X86::JMP32r; goto SetTailJmpOpcode;
+    // TAILJMPd, TAILJMPd64, TailJMPd_cc - Lower to the correct jump
+    // instruction.
+    {
+      unsigned Opcode;
+    case X86::TAILJMPr:
+      Opcode = X86::JMP32r;
+      goto SetTailJmpOpcode;
     case X86::TAILJMPd:
-  case X86::TAILJMPd64: Opcode = X86::JMP_1;  goto SetTailJmpOpcode;
+    case X86::TAILJMPd64:
+      Opcode = X86::JMP_1;
+      goto SetTailJmpOpcode;
     case X86::TAILJMPd_CC:
     case X86::TAILJMPd64_CC:
       Opcode = X86::GetCondBranchFromCond(
           static_cast<X86::CondCode>(MI->getOperand(1).getImm()));
       goto SetTailJmpOpcode;
 
     SetTailJmpOpcode:
       MCOperand Saved = OutMI.getOperand(0);
       OutMI = MCInst();
       OutMI.setOpcode(Opcode);
       OutMI.addOperand(Saved);
       break;
     }
 
   case X86::DEC16r:
   case X86::DEC32r:
   case X86::INC16r:
   case X86::INC32r:
     // If we aren't in 64-bit mode we can use the 1-byte inc/dec instructions.
     if (!AsmPrinter.getSubtarget().is64Bit()) {
       unsigned Opcode;
       switch (OutMI.getOpcode()) {
-      default: llvm_unreachable("Invalid opcode");
-      case X86::DEC16r: Opcode = X86::DEC16r_alt; break;
-      case X86::DEC32r: Opcode = X86::DEC32r_alt; break;
-      case X86::INC16r: Opcode = X86::INC16r_alt; break;
-      case X86::INC32r: Opcode = X86::INC32r_alt; break;
+      default:
+        llvm_unreachable("Invalid opcode");
+      case X86::DEC16r:
+        Opcode = X86::DEC16r_alt;
+        break;
+      case X86::DEC32r:
+        Opcode = X86::DEC32r_alt;
+        break;
+      case X86::INC16r:
+        Opcode = X86::INC16r_alt;
+        break;
+      case X86::INC32r:
+        Opcode = X86::INC32r_alt;
+        break;
       }
       OutMI.setOpcode(Opcode);
     }
     break;
 
   // These are pseudo-ops for OR to help with the OR->ADD transformation.  We do
   // this with an ugly goto in case the resultant OR uses EAX and needs the
   // short form.
-  case X86::ADD16rr_DB:   OutMI.setOpcode(X86::OR16rr); goto ReSimplify;
-  case X86::ADD32rr_DB:   OutMI.setOpcode(X86::OR32rr); goto ReSimplify;
-  case X86::ADD64rr_DB:   OutMI.setOpcode(X86::OR64rr); goto ReSimplify;
-  case X86::ADD16ri_DB:   OutMI.setOpcode(X86::OR16ri); goto ReSimplify;
-  case X86::ADD32ri_DB:   OutMI.setOpcode(X86::OR32ri); goto ReSimplify;
-  case X86::ADD64ri32_DB: OutMI.setOpcode(X86::OR64ri32); goto ReSimplify;
-  case X86::ADD16ri8_DB:  OutMI.setOpcode(X86::OR16ri8); goto ReSimplify;
-  case X86::ADD32ri8_DB:  OutMI.setOpcode(X86::OR32ri8); goto ReSimplify;
-  case X86::ADD64ri8_DB:  OutMI.setOpcode(X86::OR64ri8); goto ReSimplify;
+  case X86::ADD16rr_DB:
+    OutMI.setOpcode(X86::OR16rr);
+    goto ReSimplify;
+  case X86::ADD32rr_DB:
+    OutMI.setOpcode(X86::OR32rr);
+    goto ReSimplify;
+  case X86::ADD64rr_DB:
+    OutMI.setOpcode(X86::OR64rr);
+    goto ReSimplify;
+  case X86::ADD16ri_DB:
+    OutMI.setOpcode(X86::OR16ri);
+    goto ReSimplify;
+  case X86::ADD32ri_DB:
+    OutMI.setOpcode(X86::OR32ri);
+    goto ReSimplify;
+  case X86::ADD64ri32_DB:
+    OutMI.setOpcode(X86::OR64ri32);
+    goto ReSimplify;
+  case X86::ADD16ri8_DB:
+    OutMI.setOpcode(X86::OR16ri8);
+    goto ReSimplify;
+  case X86::ADD32ri8_DB:
+    OutMI.setOpcode(X86::OR32ri8);
+    goto ReSimplify;
+  case X86::ADD64ri8_DB:
+    OutMI.setOpcode(X86::OR64ri8);
+    goto ReSimplify;
 
   // Atomic load and store require a separate pseudo-inst because Acquire
   // implies mayStore and Release implies mayLoad; fix these to regular MOV
   // instructions here
-  case X86::ACQUIRE_MOV8rm:    OutMI.setOpcode(X86::MOV8rm); goto ReSimplify;
-  case X86::ACQUIRE_MOV16rm:   OutMI.setOpcode(X86::MOV16rm); goto ReSimplify;
-  case X86::ACQUIRE_MOV32rm:   OutMI.setOpcode(X86::MOV32rm); goto ReSimplify;
-  case X86::ACQUIRE_MOV64rm:   OutMI.setOpcode(X86::MOV64rm); goto ReSimplify;
-  case X86::RELEASE_MOV8mr:    OutMI.setOpcode(X86::MOV8mr); goto ReSimplify;
-  case X86::RELEASE_MOV16mr:   OutMI.setOpcode(X86::MOV16mr); goto ReSimplify;
-  case X86::RELEASE_MOV32mr:   OutMI.setOpcode(X86::MOV32mr); goto ReSimplify;
-  case X86::RELEASE_MOV64mr:   OutMI.setOpcode(X86::MOV64mr); goto ReSimplify;
-  case X86::RELEASE_MOV8mi:    OutMI.setOpcode(X86::MOV8mi); goto ReSimplify;
-  case X86::RELEASE_MOV16mi:   OutMI.setOpcode(X86::MOV16mi); goto ReSimplify;
-  case X86::RELEASE_MOV32mi:   OutMI.setOpcode(X86::MOV32mi); goto ReSimplify;
-  case X86::RELEASE_MOV64mi32: OutMI.setOpcode(X86::MOV64mi32); goto ReSimplify;
-  case X86::RELEASE_ADD8mi:    OutMI.setOpcode(X86::ADD8mi); goto ReSimplify;
-  case X86::RELEASE_ADD8mr:    OutMI.setOpcode(X86::ADD8mr); goto ReSimplify;
-  case X86::RELEASE_ADD32mi:   OutMI.setOpcode(X86::ADD32mi); goto ReSimplify;
-  case X86::RELEASE_ADD32mr:   OutMI.setOpcode(X86::ADD32mr); goto ReSimplify;
-  case X86::RELEASE_ADD64mi32: OutMI.setOpcode(X86::ADD64mi32); goto ReSimplify;
-  case X86::RELEASE_ADD64mr:   OutMI.setOpcode(X86::ADD64mr); goto ReSimplify;
-  case X86::RELEASE_AND8mi:    OutMI.setOpcode(X86::AND8mi); goto ReSimplify;
-  case X86::RELEASE_AND8mr:    OutMI.setOpcode(X86::AND8mr); goto ReSimplify;
-  case X86::RELEASE_AND32mi:   OutMI.setOpcode(X86::AND32mi); goto ReSimplify;
-  case X86::RELEASE_AND32mr:   OutMI.setOpcode(X86::AND32mr); goto ReSimplify;
-  case X86::RELEASE_AND64mi32: OutMI.setOpcode(X86::AND64mi32); goto ReSimplify;
-  case X86::RELEASE_AND64mr:   OutMI.setOpcode(X86::AND64mr); goto ReSimplify;
-  case X86::RELEASE_OR8mi:     OutMI.setOpcode(X86::OR8mi); goto ReSimplify;
-  case X86::RELEASE_OR8mr:     OutMI.setOpcode(X86::OR8mr); goto ReSimplify;
-  case X86::RELEASE_OR32mi:    OutMI.setOpcode(X86::OR32mi); goto ReSimplify;
-  case X86::RELEASE_OR32mr:    OutMI.setOpcode(X86::OR32mr); goto ReSimplify;
-  case X86::RELEASE_OR64mi32:  OutMI.setOpcode(X86::OR64mi32); goto ReSimplify;
-  case X86::RELEASE_OR64mr:    OutMI.setOpcode(X86::OR64mr); goto ReSimplify;
-  case X86::RELEASE_XOR8mi:    OutMI.setOpcode(X86::XOR8mi); goto ReSimplify;
-  case X86::RELEASE_XOR8mr:    OutMI.setOpcode(X86::XOR8mr); goto ReSimplify;
-  case X86::RELEASE_XOR32mi:   OutMI.setOpcode(X86::XOR32mi); goto ReSimplify;
-  case X86::RELEASE_XOR32mr:   OutMI.setOpcode(X86::XOR32mr); goto ReSimplify;
-  case X86::RELEASE_XOR64mi32: OutMI.setOpcode(X86::XOR64mi32); goto ReSimplify;
-  case X86::RELEASE_XOR64mr:   OutMI.setOpcode(X86::XOR64mr); goto ReSimplify;
-  case X86::RELEASE_INC8m:     OutMI.setOpcode(X86::INC8m); goto ReSimplify;
-  case X86::RELEASE_INC16m:    OutMI.setOpcode(X86::INC16m); goto ReSimplify;
-  case X86::RELEASE_INC32m:    OutMI.setOpcode(X86::INC32m); goto ReSimplify;
-  case X86::RELEASE_INC64m:    OutMI.setOpcode(X86::INC64m); goto ReSimplify;
-  case X86::RELEASE_DEC8m:     OutMI.setOpcode(X86::DEC8m); goto ReSimplify;
-  case X86::RELEASE_DEC16m:    OutMI.setOpcode(X86::DEC16m); goto ReSimplify;
-  case X86::RELEASE_DEC32m:    OutMI.setOpcode(X86::DEC32m); goto ReSimplify;
-  case X86::RELEASE_DEC64m:    OutMI.setOpcode(X86::DEC64m); goto ReSimplify;
+  case X86::ACQUIRE_MOV8rm:
+    OutMI.setOpcode(X86::MOV8rm);
+    goto ReSimplify;
+  case X86::ACQUIRE_MOV16rm:
+    OutMI.setOpcode(X86::MOV16rm);
+    goto ReSimplify;
+  case X86::ACQUIRE_MOV32rm:
+    OutMI.setOpcode(X86::MOV32rm);
+    goto ReSimplify;
+  case X86::ACQUIRE_MOV64rm:
+    OutMI.setOpcode(X86::MOV64rm);
+    goto ReSimplify;
+  case X86::RELEASE_MOV8mr:
+    OutMI.setOpcode(X86::MOV8mr);
+    goto ReSimplify;
+  case X86::RELEASE_MOV16mr:
+    OutMI.setOpcode(X86::MOV16mr);
+    goto ReSimplify;
+  case X86::RELEASE_MOV32mr:
+    OutMI.setOpcode(X86::MOV32mr);
+    goto ReSimplify;
+  case X86::RELEASE_MOV64mr:
+    OutMI.setOpcode(X86::MOV64mr);
+    goto ReSimplify;
+  case X86::RELEASE_MOV8mi:
+    OutMI.setOpcode(X86::MOV8mi);
+    goto ReSimplify;
+  case X86::RELEASE_MOV16mi:
+    OutMI.setOpcode(X86::MOV16mi);
+    goto ReSimplify;
+  case X86::RELEASE_MOV32mi:
+    OutMI.setOpcode(X86::MOV32mi);
+    goto ReSimplify;
+  case X86::RELEASE_MOV64mi32:
+    OutMI.setOpcode(X86::MOV64mi32);
+    goto ReSimplify;
+  case X86::RELEASE_ADD8mi:
+    OutMI.setOpcode(X86::ADD8mi);
+    goto ReSimplify;
+  case X86::RELEASE_ADD8mr:
+    OutMI.setOpcode(X86::ADD8mr);
+    goto ReSimplify;
+  case X86::RELEASE_ADD32mi:
+    OutMI.setOpcode(X86::ADD32mi);
+    goto ReSimplify;
+  case X86::RELEASE_ADD32mr:
+    OutMI.setOpcode(X86::ADD32mr);
+    goto ReSimplify;
+  case X86::RELEASE_ADD64mi32:
+    OutMI.setOpcode(X86::ADD64mi32);
+    goto ReSimplify;
+  case X86::RELEASE_ADD64mr:
+    OutMI.setOpcode(X86::ADD64mr);
+    goto ReSimplify;
+  case X86::RELEASE_AND8mi:
+    OutMI.setOpcode(X86::AND8mi);
+    goto ReSimplify;
+  case X86::RELEASE_AND8mr:
+    OutMI.setOpcode(X86::AND8mr);
+    goto ReSimplify;
+  case X86::RELEASE_AND32mi:
+    OutMI.setOpcode(X86::AND32mi);
+    goto ReSimplify;
+  case X86::RELEASE_AND32mr:
+    OutMI.setOpcode(X86::AND32mr);
+    goto ReSimplify;
+  case X86::RELEASE_AND64mi32:
+    OutMI.setOpcode(X86::AND64mi32);
+    goto ReSimplify;
+  case X86::RELEASE_AND64mr:
+    OutMI.setOpcode(X86::AND64mr);
+    goto ReSimplify;
+  case X86::RELEASE_OR8mi:
+    OutMI.setOpcode(X86::OR8mi);
+    goto ReSimplify;
+  case X86::RELEASE_OR8mr:
+    OutMI.setOpcode(X86::OR8mr);
+    goto ReSimplify;
+  case X86::RELEASE_OR32mi:
+    OutMI.setOpcode(X86::OR32mi);
+    goto ReSimplify;
+  case X86::RELEASE_OR32mr:
+    OutMI.setOpcode(X86::OR32mr);
+    goto ReSimplify;
+  case X86::RELEASE_OR64mi32:
+    OutMI.setOpcode(X86::OR64mi32);
+    goto ReSimplify;
+  case X86::RELEASE_OR64mr:
+    OutMI.setOpcode(X86::OR64mr);
+    goto ReSimplify;
+  case X86::RELEASE_XOR8mi:
+    OutMI.setOpcode(X86::XOR8mi);
+    goto ReSimplify;
+  case X86::RELEASE_XOR8mr:
+    OutMI.setOpcode(X86::XOR8mr);
+    goto ReSimplify;
+  case X86::RELEASE_XOR32mi:
+    OutMI.setOpcode(X86::XOR32mi);
+    goto ReSimplify;
+  case X86::RELEASE_XOR32mr:
+    OutMI.setOpcode(X86::XOR32mr);
+    goto ReSimplify;
+  case X86::RELEASE_XOR64mi32:
+    OutMI.setOpcode(X86::XOR64mi32);
+    goto ReSimplify;
+  case X86::RELEASE_XOR64mr:
+    OutMI.setOpcode(X86::XOR64mr);
+    goto ReSimplify;
+  case X86::RELEASE_INC8m:
+    OutMI.setOpcode(X86::INC8m);
+    goto ReSimplify;
+  case X86::RELEASE_INC16m:
+    OutMI.setOpcode(X86::INC16m);
+    goto ReSimplify;
+  case X86::RELEASE_INC32m:
+    OutMI.setOpcode(X86::INC32m);
+    goto ReSimplify;
+  case X86::RELEASE_INC64m:
+    OutMI.setOpcode(X86::INC64m);
+    goto ReSimplify;
+  case X86::RELEASE_DEC8m:
+    OutMI.setOpcode(X86::DEC8m);
+    goto ReSimplify;
+  case X86::RELEASE_DEC16m:
+    OutMI.setOpcode(X86::DEC16m);
+    goto ReSimplify;
+  case X86::RELEASE_DEC32m:
+    OutMI.setOpcode(X86::DEC32m);
+    goto ReSimplify;
+  case X86::RELEASE_DEC64m:
+    OutMI.setOpcode(X86::DEC64m);
+    goto ReSimplify;
 
   // We don't currently select the correct instruction form for instructions
   // which have a short %eax, etc. form. Handle this by custom lowering, for
   // now.
   //
   // Note, we are currently not handling the following instructions:
   // MOV64ao8, MOV64o8a
   // XCHG16ar, XCHG32ar, XCHG64ar
   case X86::MOV8mr_NOREX:
   case X86::MOV8mr:
   case X86::MOV8rm_NOREX:
   case X86::MOV8rm:
   case X86::MOV16mr:
   case X86::MOV16rm:
   case X86::MOV32mr:
   case X86::MOV32rm: {
     unsigned NewOpc;
     switch (OutMI.getOpcode()) {
-    default: llvm_unreachable("Invalid opcode");
+    default:
+      llvm_unreachable("Invalid opcode");
     case X86::MOV8mr_NOREX:
-    case X86::MOV8mr:     NewOpc = X86::MOV8o32a; break;
+    case X86::MOV8mr:
+      NewOpc = X86::MOV8o32a;
+      break;
     case X86::MOV8rm_NOREX:
-    case X86::MOV8rm:     NewOpc = X86::MOV8ao32; break;
-    case X86::MOV16mr:    NewOpc = X86::MOV16o32a; break;
-    case X86::MOV16rm:    NewOpc = X86::MOV16ao32; break;
-    case X86::MOV32mr:    NewOpc = X86::MOV32o32a; break;
-    case X86::MOV32rm:    NewOpc = X86::MOV32ao32; break;
+    case X86::MOV8rm:
+      NewOpc = X86::MOV8ao32;
+      break;
+    case X86::MOV16mr:
+      NewOpc = X86::MOV16o32a;
+      break;
+    case X86::MOV16rm:
+      NewOpc = X86::MOV16ao32;
+      break;
+    case X86::MOV32mr:
+      NewOpc = X86::MOV32o32a;
+      break;
+    case X86::MOV32rm:
+      NewOpc = X86::MOV32ao32;
+      break;
     }
     SimplifyShortMoveForm(AsmPrinter, OutMI, NewOpc);
     break;
   }
 
-  case X86::ADC8ri: case X86::ADC16ri: case X86::ADC32ri: case X86::ADC64ri32:
-  case X86::ADD8ri: case X86::ADD16ri: case X86::ADD32ri: case X86::ADD64ri32:
-  case X86::AND8ri: case X86::AND16ri: case X86::AND32ri: case X86::AND64ri32:
-  case X86::CMP8ri: case X86::CMP16ri: case X86::CMP32ri: case X86::CMP64ri32:
-  case X86::OR8ri:  case X86::OR16ri:  case X86::OR32ri:  case X86::OR64ri32:
-  case X86::SBB8ri: case X86::SBB16ri: case X86::SBB32ri: case X86::SBB64ri32:
-  case X86::SUB8ri: case X86::SUB16ri: case X86::SUB32ri: case X86::SUB64ri32:
-  case X86::TEST8ri:case X86::TEST16ri:case X86::TEST32ri:case X86::TEST64ri32:
-  case X86::XOR8ri: case X86::XOR16ri: case X86::XOR32ri: case X86::XOR64ri32: {
+  case X86::ADC8ri:
+  case X86::ADC16ri:
+  case X86::ADC32ri:
+  case X86::ADC64ri32:
+  case X86::ADD8ri:
+  case X86::ADD16ri:
+  case X86::ADD32ri:
+  case X86::ADD64ri32:
+  case X86::AND8ri:
+  case X86::AND16ri:
+  case X86::AND32ri:
+  case X86::AND64ri32:
+  case X86::CMP8ri:
+  case X86::CMP16ri:
+  case X86::CMP32ri:
+  case X86::CMP64ri32:
+  case X86::OR8ri:
+  case X86::OR16ri:
+  case X86::OR32ri:
+  case X86::OR64ri32:
+  case X86::SBB8ri:
+  case X86::SBB16ri:
+  case X86::SBB32ri:
+  case X86::SBB64ri32:
+  case X86::SUB8ri:
+  case X86::SUB16ri:
+  case X86::SUB32ri:
+  case X86::SUB64ri32:
+  case X86::TEST8ri:
+  case X86::TEST16ri:
+  case X86::TEST32ri:
+  case X86::TEST64ri32:
+  case X86::XOR8ri:
+  case X86::XOR16ri:
+  case X86::XOR32ri:
+  case X86::XOR64ri32: {
     unsigned NewOpc;
     switch (OutMI.getOpcode()) {
-    default: llvm_unreachable("Invalid opcode");
-    case X86::ADC8ri:     NewOpc = X86::ADC8i8;    break;
-    case X86::ADC16ri:    NewOpc = X86::ADC16i16;  break;
-    case X86::ADC32ri:    NewOpc = X86::ADC32i32;  break;
-    case X86::ADC64ri32:  NewOpc = X86::ADC64i32;  break;
-    case X86::ADD8ri:     NewOpc = X86::ADD8i8;    break;
-    case X86::ADD16ri:    NewOpc = X86::ADD16i16;  break;
-    case X86::ADD32ri:    NewOpc = X86::ADD32i32;  break;
-    case X86::ADD64ri32:  NewOpc = X86::ADD64i32;  break;
-    case X86::AND8ri:     NewOpc = X86::AND8i8;    break;
-    case X86::AND16ri:    NewOpc = X86::AND16i16;  break;
-    case X86::AND32ri:    NewOpc = X86::AND32i32;  break;
-    case X86::AND64ri32:  NewOpc = X86::AND64i32;  break;
-    case X86::CMP8ri:     NewOpc = X86::CMP8i8;    break;
-    case X86::CMP16ri:    NewOpc = X86::CMP16i16;  break;
-    case X86::CMP32ri:    NewOpc = X86::CMP32i32;  break;
-    case X86::CMP64ri32:  NewOpc = X86::CMP64i32;  break;
-    case X86::OR8ri:      NewOpc = X86::OR8i8;     break;
-    case X86::OR16ri:     NewOpc = X86::OR16i16;   break;
-    case X86::OR32ri:     NewOpc = X86::OR32i32;   break;
-    case X86::OR64ri32:   NewOpc = X86::OR64i32;   break;
-    case X86::SBB8ri:     NewOpc = X86::SBB8i8;    break;
-    case X86::SBB16ri:    NewOpc = X86::SBB16i16;  break;
-    case X86::SBB32ri:    NewOpc = X86::SBB32i32;  break;
-    case X86::SBB64ri32:  NewOpc = X86::SBB64i32;  break;
-    case X86::SUB8ri:     NewOpc = X86::SUB8i8;    break;
-    case X86::SUB16ri:    NewOpc = X86::SUB16i16;  break;
-    case X86::SUB32ri:    NewOpc = X86::SUB32i32;  break;
-    case X86::SUB64ri32:  NewOpc = X86::SUB64i32;  break;
-    case X86::TEST8ri:    NewOpc = X86::TEST8i8;   break;
-    case X86::TEST16ri:   NewOpc = X86::TEST16i16; break;
-    case X86::TEST32ri:   NewOpc = X86::TEST32i32; break;
-    case X86::TEST64ri32: NewOpc = X86::TEST64i32; break;
-    case X86::XOR8ri:     NewOpc = X86::XOR8i8;    break;
-    case X86::XOR16ri:    NewOpc = X86::XOR16i16;  break;
-    case X86::XOR32ri:    NewOpc = X86::XOR32i32;  break;
-    case X86::XOR64ri32:  NewOpc = X86::XOR64i32;  break;
+    default:
+      llvm_unreachable("Invalid opcode");
+    case X86::ADC8ri:
+      NewOpc = X86::ADC8i8;
+      break;
+    case X86::ADC16ri:
+      NewOpc = X86::ADC16i16;
+      break;
+    case X86::ADC32ri:
+      NewOpc = X86::ADC32i32;
+      break;
+    case X86::ADC64ri32:
+      NewOpc = X86::ADC64i32;
+      break;
+    case X86::ADD8ri:
+      NewOpc = X86::ADD8i8;
+      break;
+    case X86::ADD16ri:
+      NewOpc = X86::ADD16i16;
+      break;
+    case X86::ADD32ri:
+      NewOpc = X86::ADD32i32;
+      break;
+    case X86::ADD64ri32:
+      NewOpc = X86::ADD64i32;
+      break;
+    case X86::AND8ri:
+      NewOpc = X86::AND8i8;
+      break;
+    case X86::AND16ri:
+      NewOpc = X86::AND16i16;
+      break;
+    case X86::AND32ri:
+      NewOpc = X86::AND32i32;
+      break;
+    case X86::AND64ri32:
+      NewOpc = X86::AND64i32;
+      break;
+    case X86::CMP8ri:
+      NewOpc = X86::CMP8i8;
+      break;
+    case X86::CMP16ri:
+      NewOpc = X86::CMP16i16;
+      break;
+    case X86::CMP32ri:
+      NewOpc = X86::CMP32i32;
+      break;
+    case X86::CMP64ri32:
+      NewOpc = X86::CMP64i32;
+      break;
+    case X86::OR8ri:
+      NewOpc = X86::OR8i8;
+      break;
+    case X86::OR16ri:
+      NewOpc = X86::OR16i16;
+      break;
+    case X86::OR32ri:
+      NewOpc = X86::OR32i32;
+      break;
+    case X86::OR64ri32:
+      NewOpc = X86::OR64i32;
+      break;
+    case X86::SBB8ri:
+      NewOpc = X86::SBB8i8;
+      break;
+    case X86::SBB16ri:
+      NewOpc = X86::SBB16i16;
+      break;
+    case X86::SBB32ri:
+      NewOpc = X86::SBB32i32;
+      break;
+    case X86::SBB64ri32:
+      NewOpc = X86::SBB64i32;
+      break;
+    case X86::SUB8ri:
+      NewOpc = X86::SUB8i8;
+      break;
+    case X86::SUB16ri:
+      NewOpc = X86::SUB16i16;
+      break;
+    case X86::SUB32ri:
+      NewOpc = X86::SUB32i32;
+      break;
+    case X86::SUB64ri32:
+      NewOpc = X86::SUB64i32;
+      break;
+    case X86::TEST8ri:
+      NewOpc = X86::TEST8i8;
+      break;
+    case X86::TEST16ri:
+      NewOpc = X86::TEST16i16;
+      break;
+    case X86::TEST32ri:
+      NewOpc = X86::TEST32i32;
+      break;
+    case X86::TEST64ri32:
+      NewOpc = X86::TEST64i32;
+      break;
+    case X86::XOR8ri:
+      NewOpc = X86::XOR8i8;
+      break;
+    case X86::XOR16ri:
+      NewOpc = X86::XOR16i16;
+      break;
+    case X86::XOR32ri:
+      NewOpc = X86::XOR32i32;
+      break;
+    case X86::XOR64ri32:
+      NewOpc = X86::XOR64i32;
+      break;
     }
     SimplifyShortImmForm(OutMI, NewOpc);
     break;
   }
 
   // Try to shrink some forms of movsx.
   case X86::MOVSX16rr8:
   case X86::MOVSX32rr16:
void X86AsmPrinter::LowerTlsAddr(X86MCInstLower &MCInstLowering,
 
   MCContext &context = OutStreamer->getContext();
 
   if (needsPadding)
     EmitAndCountInstruction(MCInstBuilder(X86::DATA16_PREFIX));
 
   MCSymbolRefExpr::VariantKind SRVK;
   switch (MI.getOpcode()) {
-    case X86::TLS_addr32:
-    case X86::TLS_addr64:
-      SRVK = MCSymbolRefExpr::VK_TLSGD;
-      break;
-    case X86::TLS_base_addr32:
-      SRVK = MCSymbolRefExpr::VK_TLSLDM;
-      break;
-    case X86::TLS_base_addr64:
-      SRVK = MCSymbolRefExpr::VK_TLSLD;
-      break;
-    default:
-      llvm_unreachable("unexpected opcode");
   }
 
   MCSymbol *sym = MCInstLowering.GetSymbolFromOperand(MI.getOperand(3));
   const MCSymbolRefExpr *symRef = MCSymbolRefExpr::create(sym, SRVK, context);
 
   MCInst LEA;
   if (is64Bits) {
     LEA.setOpcode(X86::LEA64r);
if (needsPadding) {
     EmitAndCountInstruction(MCInstBuilder(X86::DATA16_PREFIX));
     EmitAndCountInstruction(MCInstBuilder(X86::DATA16_PREFIX));
     EmitAndCountInstruction(MCInstBuilder(X86::REX64_PREFIX));
   }
 
   StringRef name = is64Bits ? "__tls_get_addr" : "___tls_get_addr";
   MCSymbol *tlsGetAddr = context.getOrCreateSymbol(name);
   const MCSymbolRefExpr *tlsRef =
-    MCSymbolRefExpr::create(tlsGetAddr,
-                            MCSymbolRefExpr::VK_PLT,
-                            context);
+      MCSymbolRefExpr::create(tlsGetAddr, MCSymbolRefExpr::VK_PLT, context);
 
-  EmitAndCountInstruction(MCInstBuilder(is64Bits ? X86::CALL64pcrel32
-                                                 : X86::CALLpcrel32)
-                            .addExpr(tlsRef));
+  EmitAndCountInstruction(
+      MCInstBuilder(is64Bits ? X86::CALL64pcrel32 : X86::CALLpcrel32)
 }
 
 /// \brief Emit the largest nop instruction smaller than or equal to \p NumBytes
 /// bytes.  Return the size of nop emitted.
 static unsigned EmitNop(MCStreamer &OS, unsigned NumBytes, bool Is64Bit,
                         const MCSubtargetInfo &STI) {
   // This works only for 64bit. For 32bit we have to do additional checking if
   // the CPU supports multi-byte nops.
   assert(Is64Bit && "EmitNops only supports X86-64");
 
   unsigned NopSize;
   unsigned Opc, BaseReg, ScaleVal, IndexReg, Displacement, SegmentReg;
   Opc = IndexReg = Displacement = SegmentReg = 0;
   BaseReg = X86::RAX;
   ScaleVal = 1;
   switch (NumBytes) {
-  case  0: llvm_unreachable("Zero nops?"); break;
-  case  1: NopSize = 1; Opc = X86::NOOP; break;
-  case  2: NopSize = 2; Opc = X86::XCHG16ar; break;
-  case  3: NopSize = 3; Opc = X86::NOOPL; break;
-  case  4: NopSize = 4; Opc = X86::NOOPL; Displacement = 8; break;
-  case  5: NopSize = 5; Opc = X86::NOOPL; Displacement = 8;
-           IndexReg = X86::RAX; break;
-  case  6: NopSize = 6; Opc = X86::NOOPW; Displacement = 8;
-           IndexReg = X86::RAX; break;
-  case  7: NopSize = 7; Opc = X86::NOOPL; Displacement = 512; break;
-  case  8: NopSize = 8; Opc = X86::NOOPL; Displacement = 512;
-           IndexReg = X86::RAX; break;
-  case  9: NopSize = 9; Opc = X86::NOOPW; Displacement = 512;
-           IndexReg = X86::RAX; break;
-  default: NopSize = 10; Opc = X86::NOOPW; Displacement = 512;
-           IndexReg = X86::RAX; SegmentReg = X86::CS; break;
+  case 0:
+    llvm_unreachable("Zero nops?");
+    break;
+  case 1:
+    NopSize = 1;
+    Opc = X86::NOOP;
+    break;
+  case 2:
+    NopSize = 2;
+    Opc = X86::XCHG16ar;
+    break;
+  case 3:
+    NopSize = 3;
+    Opc = X86::NOOPL;
+    break;
+  case 4:
+    NopSize = 4;
+    Opc = X86::NOOPL;
+    Displacement = 8;
+    break;
+  case 5:
+    NopSize = 5;
+    Opc = X86::NOOPL;
+    Displacement = 8;
+    IndexReg = X86::RAX;
+    break;
+  case 6:
+    NopSize = 6;
+    Opc = X86::NOOPW;
+    Displacement = 8;
+    IndexReg = X86::RAX;
+    break;
+  case 7:
+    NopSize = 7;
+    Opc = X86::NOOPL;
+    Displacement = 512;
+    break;
+  case 8:
+    NopSize = 8;
+    Opc = X86::NOOPL;
+    Displacement = 512;
+    IndexReg = X86::RAX;
+    break;
+  case 9:
+    NopSize = 9;
+    Opc = X86::NOOPW;
+    Displacement = 512;
+    IndexReg = X86::RAX;
+    break;
+  default:
+    NopSize = 10;
+    Opc = X86::NOOPW;
+    Displacement = 512;
+    IndexReg = X86::RAX;
+    SegmentReg = X86::CS;
+    break;
   }
 
   unsigned NumPrefixes = std::min(NumBytes - NopSize, 5U);
   NopSize += NumPrefixes;
   for (unsigned i = 0; i != NumPrefixes; ++i)
     OS.EmitBytes("\x66");
 
   switch (Opc) {
if (MinSize == 2 && Opcode == X86::PUSH64r) {
       //
       // NB! In some cases the encoding for PUSH64r (e.g. PUSH64r %r9) takes two
       // bytes too, so the check on MinSize is important.
       MCI.setOpcode(X86::PUSH64rmr);
     } else {
       unsigned NopSize = EmitNop(*OutStreamer, MinSize, Subtarget->is64Bit(),
                                  getSubtargetInfo());
       assert(NopSize == MinSize && "Could not implement MinSize!");
-      (void) NopSize;
+      (void)NopSize;
     }
   }
 
   OutStreamer->EmitInstruction(MCI, getSubtargetInfo());
 }
 
 // Lower a stackmap of the form:
 // <id>, <shadowBytes>, ...
default:
       /// FIXME: Add a verifier check for bad callee types.
       llvm_unreachable("Unrecognized callee operand type.");
     case MachineOperand::MO_Immediate:
       if (CalleeMO.getImm())
         CalleeMCOp = MCOperand::createImm(CalleeMO.getImm());
       break;
     case MachineOperand::MO_ExternalSymbol:
     case MachineOperand::MO_GlobalAddress:
-      CalleeMCOp =
-        MCIL.LowerSymbolOperand(CalleeMO,
+      CalleeMCOp = MCIL.LowerSymbolOperand(CalleeMO,
                                            MCIL.GetSymbolFromOperand(CalleeMO));
       break;
     }
 
     // Emit MOV to materialize the target address and the CALL to target.
     // This is encoded with 12-13 bytes, depending on which register is used.
     unsigned ScratchReg = MI.getOperand(ScratchIdx).getReg();
     if (X86II::isX86_64ExtendedReg(ScratchReg))
       EncodedBytes = 13;
void X86AsmPrinter::LowerPATCHABLE_EVENT_CALL(const MachineInstr &MI,
 
   // Use a two-byte `jmp`. This version of JMP takes an 8-bit relative offset as
   // an operand (computed as an offset from the jmp instruction).
   // FIXME: Find another less hacky way do force the relative jump.
   OutStreamer->EmitBinaryData("\xeb\x0f");
 
   // The default C calling convention will place two arguments into %rcx and
   // %rdx -- so we only work with those.
-  unsigned UsedRegs[] = {X86::RDI, X86::RSI};
+  unsigned DestRegs[] = {X86::RDI, X86::RSI};
   bool UsedMask[] = {false, false};
+  // Filled out in loop.
+  unsigned SrcRegs[] = {0, 0};
 
   // Then we put the operands in the %rdi and %rsi registers. We spill the
   // values in the register before we clobber them, and mark them as used in
   // UsedMask. In case the arguments are already in the correct register, we use
   // emit nops appropriately sized to keep the sled the same size in every
   // situation.
   for (unsigned I = 0; I < MI.getNumOperands(); ++I)
     if (auto Op = MCIL.LowerMachineOperand(&MI, MI.getOperand(I))) {
       assert(Op->isReg() && "Only support arguments in registers");
-      if (Op->getReg() != UsedRegs[I]) {
+      SrcRegs[I] = Op->getReg();
+      if (SrcRegs[I] != DestRegs[I]) {
         UsedMask[I] = true;
         EmitAndCountInstruction(
-            MCInstBuilder(X86::PUSH64r).addReg(UsedRegs[I]));
-        EmitAndCountInstruction(MCInstBuilder(X86::MOV64rr)
-                                    .addReg(UsedRegs[I])
-                                    .addReg(Op->getReg()));
+            MCInstBuilder(X86::PUSH64r).addReg(DestRegs[I]));
       } else {
         EmitNops(*OutStreamer, 4, Subtarget->is64Bit(), getSubtargetInfo());
       }
     }
 
+  // Now that the register values are stashed, mov arguments into place.
+  for (unsigned I = 0; I < MI.getNumOperands(); ++I)
+    if (SrcRegs[I] != DestRegs[I])
+      EmitAndCountInstruction(
+          MCInstBuilder(X86::MOV64rr).addReg(DestRegs[I]).addReg(SrcRegs[I]));
+
   // We emit a hard dependency on the __xray_CustomEvent symbol, which is the
   // name of the trampoline to be implemented by the XRay runtime.
   auto TSym = OutContext.getOrCreateSymbol("__xray_CustomEvent");
   MachineOperand TOp = MachineOperand::CreateMCSymbol(TSym);
   if (isPositionIndependent())
     TOp.setTargetFlags(X86II::MO_PLT);
 
   // Emit the call instruction.
   EmitAndCountInstruction(MCInstBuilder(X86::CALL64pcrel32)
                               .addOperand(MCIL.LowerSymbolOperand(TOp, TSym)));
 
   // Restore caller-saved and used registers.
   for (unsigned I = sizeof UsedMask; I-- > 0;)
     if (UsedMask[I])
-      EmitAndCountInstruction(MCInstBuilder(X86::POP64r).addReg(UsedRegs[I]));
+      EmitAndCountInstruction(MCInstBuilder(X86::POP64r).addReg(DestRegs[I]));
     else
       EmitNops(*OutStreamer, 1, Subtarget->is64Bit(), getSubtargetInfo());
 
   OutStreamer->AddComment("xray custom event end.");
 
   // Record the sled version. Older versions of this sled were spelled
   // differently, so we let the runtime handle the different offsets we're
   // using.
   recordSled(CurSled, MI, SledKind::CUSTOM_EVENT, 1);
 }
 
+void X86AsmPrinter::LowerPATCHABLE_TYPED_EVENT_CALL(const MachineInstr &MI,
+                                                    X86MCInstLower &MCIL) {
+  assert(Subtarget->is64Bit() && "XRay typed events only supports X86-64");
+
+  // We want to emit the following pattern, which follows the x86 calling
+  // convention to prepare for the trampoline call to be patched in.
+  //
+  //   .p2align 1, ...
+  // .Lxray_event_sled_N:
+  //   jmp +N                        // jump across the instrumentation sled
+  //   ...                           // set up arguments in register
+  //   callq __xray_TypedEvent@plt  // force dependency to symbol
+  //   ...
+  //   <jump here>
+  //
+  // After patching, it would look something like:
+  //
+  //   nopw (2-byte nop)
+  //   ...
+  //   callq __xrayTypedEvent  // already lowered
+  //   ...
+  //
+  // ---
+  // First we emit the label and the jump.
+  auto CurSled = OutContext.createTempSymbol("xray_typed_event_sled_", true);
+  OutStreamer->AddComment("# XRay Typed Event Log");
+  OutStreamer->EmitCodeAlignment(2);
+  OutStreamer->EmitLabel(CurSled);
+
+  // Use a two-byte `jmp`. This version of JMP takes an 8-bit relative offset as
+  // an operand (computed as an offset from the jmp instruction).
+  // FIXME: Find another less hacky way do force the relative jump.
+  OutStreamer->EmitBinaryData("\xeb\x14");
+
+  // An x86-64 convention may place three arguments into %rcx, %rdx, and R8,
+  // so we'll work with those. Or we may be called via SystemV, in which case
+  // we don't have to do any translation.
+  unsigned DestRegs[] = {X86::RDI, X86::RSI, X86::RDX};
+  bool UsedMask[] = {false, false, false};
+
+  // Will fill out src regs in the loop.
+  unsigned SrcRegs[] = {0, 0, 0};
+
+  // Then we put the operands in the SystemV registers. We spill the values in
+  // the registers before we clobber them, and mark them as used in UsedMask.
+  // In case the arguments are already in the correct register, we emit nops
+  // appropriately sized to keep the sled the same size in every situation.
+  for (unsigned I = 0; I < MI.getNumOperands(); ++I)
+    if (auto Op = MCIL.LowerMachineOperand(&MI, MI.getOperand(I))) {
+      // TODO: Is register only support adequate?
+      assert(Op->isReg() && "Only supports arguments in registers");
+      SrcRegs[I] = Op->getReg();
+      if (SrcRegs[I] != DestRegs[I]) {
+        UsedMask[I] = true;
+        EmitAndCountInstruction(
+            MCInstBuilder(X86::PUSH64r).addReg(DestRegs[I]));
+      } else {
+        EmitNops(*OutStreamer, 4, Subtarget->is64Bit(), getSubtargetInfo());
+      }
+    }
+
+  // In the above loop we only stash all of the destination registers or emit
+  // nops if the arguments are already in the right place. Doing the actually
+  // moving is postponed until after all the registers are stashed so nothing
+  // is clobbers. We've already added nops to account for the size of mov and
+  // push if the register is in the right place, so we only have to worry about
+  // emitting movs.
+  for (unsigned I = 0; I < MI.getNumOperands(); ++I)
+    if (UsedMask[I])
+      EmitAndCountInstruction(
+          MCInstBuilder(X86::MOV64rr).addReg(DestRegs[I]).addReg(SrcRegs[I]));
+
+  // We emit a hard dependency on the __xray_TypedEvent symbol, which is the
+  // name of the trampoline to be implemented by the XRay runtime.
+  auto TSym = OutContext.getOrCreateSymbol("__xray_TypedEvent");
+  MachineOperand TOp = MachineOperand::CreateMCSymbol(TSym);
+  if (isPositionIndependent())
+    TOp.setTargetFlags(X86II::MO_PLT);
+
+  // Emit the call instruction.
+  EmitAndCountInstruction(MCInstBuilder(X86::CALL64pcrel32)
+                              .addOperand(MCIL.LowerSymbolOperand(TOp, TSym)));
+
+  // Restore caller-saved and used registers.
+  for (unsigned I = sizeof UsedMask; I-- > 0;)
+    if (UsedMask[I])
+      EmitAndCountInstruction(MCInstBuilder(X86::POP64r).addReg(DestRegs[I]));
+    else
+      EmitNops(*OutStreamer, 1, Subtarget->is64Bit(), getSubtargetInfo());
+
+  OutStreamer->AddComment("xray typed event end.");
+
+  // Record the sled version.
+  recordSled(CurSled, MI, SledKind::TYPED_EVENT, 0);
+}
+
 void X86AsmPrinter::LowerPATCHABLE_FUNCTION_ENTER(const MachineInstr &MI,
                                                   X86MCInstLower &MCIL) {
   // We want to emit the following pattern:
   //
   //   .p2align 1, ...
   // .Lxray_sled_N:
   //   jmp .tmpN
   //   # 9 bytes worth of noops
void X86AsmPrinter::LowerPATCHABLE_RET(const MachineInstr &MI,
   for (auto &MO : make_range(MI.operands_begin() + 1, MI.operands_end()))
     if (auto MaybeOperand = MCIL.LowerMachineOperand(&MI, MO))
       Ret.addOperand(MaybeOperand.getValue());
   OutStreamer->EmitInstruction(Ret, getSubtargetInfo());
   EmitNops(*OutStreamer, 10, Subtarget->is64Bit(), getSubtargetInfo());
   recordSled(CurSled, MI, SledKind::FUNCTION_EXIT);
 }
 
-void X86AsmPrinter::LowerPATCHABLE_TAIL_CALL(const MachineInstr &MI, X86MCInstLower &MCIL) {
+void X86AsmPrinter::LowerPATCHABLE_TAIL_CALL(const MachineInstr &MI,
+                                             X86MCInstLower &MCIL) {
   // Like PATCHABLE_RET, we have the actual instruction in the operands to this
   // instruction so we lower that particular instruction and its operands.
   // Unlike PATCHABLE_RET though, we put the sled before the JMP, much like how
   // we do it for PATCHABLE_FUNCTION_ENTER. The sled should be very similar to
   // the PATCHABLE_FUNCTION_ENTER case, followed by the lowering of the actual
   // tail call much like how we have it in PATCHABLE_RET.
   auto CurSled = OutContext.createTempSymbol("xray_sled_", true);
   OutStreamer->EmitCodeAlignment(2);

 
 static const Constant *getConstantFromPool(const MachineInstr &MI,
                                            const MachineOperand &Op) {
   if (!Op.isCPI())
     return nullptr;
 
   ArrayRef<MachineConstantPoolEntry> Constants =
       MI.getParent()->getParent()->getConstantPool()->getConstants();
-  const MachineConstantPoolEntry &ConstantEntry =
-      Constants[Op.getIndex()];
+  const MachineConstantPoolEntry &ConstantEntry = Constants[Op.getIndex()];
 
   // Bail if this is a machine constant pool entry, we won't be able to dig out
   // anything useful.
   if (ConstantEntry.isMachineConstantPoolEntry())
     return nullptr;
 
   auto *C = dyn_cast<Constant>(ConstantEntry.Val.ConstVal);
   assert((!C || ConstantEntry.getType() == C->getType()) &&
          "Expected a constant of the same type!");
   return C;
 }
 
-static std::string getShuffleComment(const MachineInstr *MI,
-                                     unsigned SrcOp1Idx,
-                                     unsigned SrcOp2Idx,
-                                     ArrayRef<int> Mask) {
+static std::string getShuffleComment(const MachineInstr *MI, unsigned SrcOp1Idx,
+                                     unsigned SrcOp2Idx, ArrayRef<int> Mask) {
   std::string Comment;
 
   // Compute the name for a register. This is really goofy because we have
   // multiple instruction printers that could (in theory) use different
   // names. Fortunately most people use the ATT style (outside of Windows)
   // and they actually agree on register naming here. Ultimately, this is
   // a comment, and so its OK if it isn't perfect.
   auto GetRegisterName = [](unsigned RegNum) -> StringRef {
void X86AsmPrinter::EmitSEHInstruction(const MachineInstr *MI) {
 
   default:
     llvm_unreachable("expected SEH_ instruction");
   }
 }
 
 void X86AsmPrinter::EmitInstruction(const MachineInstr *MI) {
   X86MCInstLower MCInstLowering(*MF, *this);
-  const X86RegisterInfo *RI = MF->getSubtarget<X86Subtarget>().getRegisterInfo();
+  const X86RegisterInfo *RI =
+      MF->getSubtarget<X86Subtarget>().getRegisterInfo();
 
   // Add a comment about EVEX-2-VEX compression for AVX-512 instrs that
   // are compressed from EVEX encoding to VEX encoding.
   if (TM.Options.MCOptions.ShowMCEncoding) {
     if (MI->getAsmPrinterFlags() & X86::AC_EVEX_2_VEX)
       OutStreamer->AddComment("EVEX TO VEX Compression ", false);
   }
 
   switch (MI->getOpcode()) {
   case TargetOpcode::DBG_VALUE:
     llvm_unreachable("Should be handled target independently");
 
   // Emit nothing here but a comment if we can.
   case X86::Int_MemBarrier:
     OutStreamer->emitRawComment("MEMBARRIER");
     return;
 
-
   case X86::EH_RETURN:
   case X86::EH_RETURN64: {
     // Lower these as normal, but add some comments.
     unsigned Reg = MI->getOperand(0).getReg();
     OutStreamer->AddComment(StringRef("eh_return, addr: %") +
                             X86ATTInstPrinter::getRegisterName(Reg));
     break;
   }
case X86::MOVPC32r: {
     //     call "L1$pb"
     // "L1$pb":
     //     popl %esi
 
     // Emit the call.
     MCSymbol *PICBase = MF->getPICBaseSymbol();
     // FIXME: We would like an efficient form for this, so we don't have to do a
     // lot of extra uniquing.
-    EmitAndCountInstruction(MCInstBuilder(X86::CALLpcrel32)
+    EmitAndCountInstruction(
+        MCInstBuilder(X86::CALLpcrel32)
             .addExpr(MCSymbolRefExpr::create(PICBase, OutContext)));
 
-    const X86FrameLowering* FrameLowering =
+    const X86FrameLowering *FrameLowering =
         MF->getSubtarget<X86Subtarget>().getFrameLowering();
     bool hasFP = FrameLowering->hasFP(*MF);
-    
     // TODO: This is needed only if we require precise CFA.
     bool HasActiveDwarfFrame = OutStreamer->getNumFrameInfos() &&
                                !OutStreamer->getDwarfFrameInfos().back().End;
 
     int stackGrowth = -RI->getSlotSize();
 
     if (HasActiveDwarfFrame && !hasFP) {
       OutStreamer->EmitCFIAdjustCfaOffset(-stackGrowth);
     }
 
     // Emit the label.
     OutStreamer->EmitLabel(PICBase);
 
     // popl $reg
-    EmitAndCountInstruction(MCInstBuilder(X86::POP32r)
-                            .addReg(MI->getOperand(0).getReg()));
+    EmitAndCountInstruction(
+        MCInstBuilder(X86::POP32r).addReg(MI->getOperand(0).getReg()));
 
     if (HasActiveDwarfFrame && !hasFP) {
       OutStreamer->EmitCFIAdjustCfaOffset(stackGrowth);
     }
     return;
   }
 
   case X86::ADD32ri: {
case X86::ADD32ri: {
     MCSymbol *DotSym = OutContext.createTempSymbol();
     OutStreamer->EmitLabel(DotSym);
 
     // Now that we have emitted the label, lower the complex operand expression.
     MCSymbol *OpSym = MCInstLowering.GetSymbolFromOperand(MI->getOperand(2));
 
     const MCExpr *DotExpr = MCSymbolRefExpr::create(DotSym, OutContext);
     const MCExpr *PICBase =
         MCSymbolRefExpr::create(MF->getPICBaseSymbol(), OutContext);
     DotExpr = MCBinaryExpr::createSub(DotExpr, PICBase, OutContext);
 
-    DotExpr = MCBinaryExpr::createAdd(MCSymbolRefExpr::create(OpSym,OutContext),
-                                      DotExpr, OutContext);
+    DotExpr = MCBinaryExpr::createAdd(
+        MCSymbolRefExpr::create(OpSym, OutContext), DotExpr, OutContext);
 
     EmitAndCountInstruction(MCInstBuilder(X86::ADD32ri)
                                 .addReg(MI->getOperand(0).getReg())
                                 .addReg(MI->getOperand(1).getReg())
                                 .addExpr(DotExpr));
     return;
   }
   case TargetOpcode::STATEPOINT:
     return LowerSTATEPOINT(*MI, MCInstLowering);
 
   case TargetOpcode::FAULTING_OP:
     return LowerFAULTING_OP(*MI, MCInstLowering);
 
void X86AsmPrinter::EmitInstruction(const MachineInstr *MI) {
   case TargetOpcode::PATCHABLE_FUNCTION_ENTER:
     return LowerPATCHABLE_FUNCTION_ENTER(*MI, MCInstLowering);
 
   case TargetOpcode::PATCHABLE_RET:
     return LowerPATCHABLE_RET(*MI, MCInstLowering);
 
   case TargetOpcode::PATCHABLE_TAIL_CALL:
     return LowerPATCHABLE_TAIL_CALL(*MI, MCInstLowering);
-    
   case TargetOpcode::PATCHABLE_EVENT_CALL:
     return LowerPATCHABLE_EVENT_CALL(*MI, MCInstLowering);
 
+  case TargetOpcode::PATCHABLE_TYPED_EVENT_CALL:
+    return LowerPATCHABLE_TYPED_EVENT_CALL(*MI, MCInstLowering);
+
   case X86::MORESTACK_RET:
     EmitAndCountInstruction(MCInstBuilder(getRetOpcode(*Subtarget)));
     return;
 
   case X86::MORESTACK_RET_RESTORE_R10:
     // Return, then restore R10.
     EmitAndCountInstruction(MCInstBuilder(getRetOpcode(*Subtarget)));
-    EmitAndCountInstruction(MCInstBuilder(X86::MOV64rr)
-                            .addReg(X86::R10)
-                            .addReg(X86::RAX));
+    EmitAndCountInstruction(
+        MCInstBuilder(X86::MOV64rr).addReg(X86::R10).addReg(X86::RAX));
     return;
 
   case X86::SEH_PushReg:
   case X86::SEH_SaveReg:
   case X86::SEH_SaveXMM:
   case X86::SEH_StackAlloc:
   case X86::SEH_SetFrame:
   case X86::SEH_PushFrame:
void X86AsmPrinter::EmitInstruction(const MachineInstr *MI) {
   case X86::VPSHUFBZ256rmkz:
   case X86::VPSHUFBZrm:
   case X86::VPSHUFBZrmk:
   case X86::VPSHUFBZrmkz: {
     if (!OutStreamer->isVerboseAsm())
       break;
     unsigned SrcIdx, MaskIdx;
     switch (MI->getOpcode()) {
-    default: llvm_unreachable("Invalid opcode");
+    default:
+      llvm_unreachable("Invalid opcode");
     case X86::PSHUFBrm:
     case X86::VPSHUFBrm:
     case X86::VPSHUFBYrm:
     case X86::VPSHUFBZ128rm:
     case X86::VPSHUFBZ256rm:
     case X86::VPSHUFBZrm:
-      SrcIdx = 1; MaskIdx = 5; break;
+      SrcIdx = 1;
+      MaskIdx = 5;
+      break;
     case X86::VPSHUFBZ128rmkz:
     case X86::VPSHUFBZ256rmkz:
     case X86::VPSHUFBZrmkz:
-      SrcIdx = 2; MaskIdx = 6; break;
+      SrcIdx = 2;
+      MaskIdx = 6;
+      break;
     case X86::VPSHUFBZ128rmk:
     case X86::VPSHUFBZ256rmk:
     case X86::VPSHUFBZrmk:
-      SrcIdx = 3; MaskIdx = 7; break;
+      SrcIdx = 3;
+      MaskIdx = 7;
+      break;
     }
 
     assert(MI->getNumOperands() >= 6 &&
            "We should always have at least 6 operands!");
 
     const MachineOperand &MaskOp = MI->getOperand(MaskIdx);
     if (auto *C = getConstantFromPool(*MI, MaskOp)) {
       SmallVector<int, 64> Mask;
void X86AsmPrinter::EmitInstruction(const MachineInstr *MI) {
   case X86::VPERMILPDZrm:
   case X86::VPERMILPDZrmk:
   case X86::VPERMILPDZrmkz: {
     if (!OutStreamer->isVerboseAsm())
       break;
     unsigned SrcIdx, MaskIdx;
     unsigned ElSize;
     switch (MI->getOpcode()) {
-    default: llvm_unreachable("Invalid opcode");
+    default:
+      llvm_unreachable("Invalid opcode");
     case X86::VPERMILPSrm:
     case X86::VPERMILPSYrm:
     case X86::VPERMILPSZ128rm:
     case X86::VPERMILPSZ256rm:
     case X86::VPERMILPSZrm:
-      SrcIdx = 1; MaskIdx = 5; ElSize = 32; break;
+      SrcIdx = 1;
+      MaskIdx = 5;
+      ElSize = 32;
+      break;
     case X86::VPERMILPSZ128rmkz:
     case X86::VPERMILPSZ256rmkz:
     case X86::VPERMILPSZrmkz:
-      SrcIdx = 2; MaskIdx = 6; ElSize = 32; break;
+      SrcIdx = 2;
+      MaskIdx = 6;
+      ElSize = 32;
+      break;
     case X86::VPERMILPSZ128rmk:
     case X86::VPERMILPSZ256rmk:
     case X86::VPERMILPSZrmk:
-      SrcIdx = 3; MaskIdx = 7; ElSize = 32; break;
+      SrcIdx = 3;
+      MaskIdx = 7;
+      ElSize = 32;
+      break;
     case X86::VPERMILPDrm:
     case X86::VPERMILPDYrm:
     case X86::VPERMILPDZ128rm:
     case X86::VPERMILPDZ256rm:
     case X86::VPERMILPDZrm:
-      SrcIdx = 1; MaskIdx = 5; ElSize = 64; break;
+      SrcIdx = 1;
+      MaskIdx = 5;
+      ElSize = 64;
+      break;
     case X86::VPERMILPDZ128rmkz:
     case X86::VPERMILPDZ256rmkz:
     case X86::VPERMILPDZrmkz:
-      SrcIdx = 2; MaskIdx = 6; ElSize = 64; break;
+      SrcIdx = 2;
+      MaskIdx = 6;
+      ElSize = 64;
+      break;
     case X86::VPERMILPDZ128rmk:
     case X86::VPERMILPDZ256rmk:
     case X86::VPERMILPDZrmk:
-      SrcIdx = 3; MaskIdx = 7; ElSize = 64; break;
+      SrcIdx = 3;
+      MaskIdx = 7;
+      ElSize = 64;
+      break;
     }
 
     assert(MI->getNumOperands() >= 6 &&
            "We should always have at least 6 operands!");
 
     const MachineOperand &MaskOp = MI->getOperand(MaskIdx);
     if (auto *C = getConstantFromPool(*MI, MaskOp)) {
       SmallVector<int, 16> Mask;
assert(MI->getNumOperands() >= 8 &&
            "We should always have at least 8 operands!");
 
     const MachineOperand &CtrlOp = MI->getOperand(MI->getNumOperands() - 1);
     if (!CtrlOp.isImm())
       break;
 
     unsigned ElSize;
     switch (MI->getOpcode()) {
-    default: llvm_unreachable("Invalid opcode");
-    case X86::VPERMIL2PSrm: case X86::VPERMIL2PSYrm: ElSize = 32; break;
-    case X86::VPERMIL2PDrm: case X86::VPERMIL2PDYrm: ElSize = 64; break;
+    default:
+      llvm_unreachable("Invalid opcode");
+    case X86::VPERMIL2PSrm:
+    case X86::VPERMIL2PSYrm:
+      ElSize = 32;
+      break;
+    case X86::VPERMIL2PDrm:
+    case X86::VPERMIL2PDYrm:
+      ElSize = 64;
+      break;
     }
 
     const MachineOperand &MaskOp = MI->getOperand(6);
     if (auto *C = getConstantFromPool(*MI, MaskOp)) {
       SmallVector<int, 16> Mask;
       DecodeVPERMIL2PMask(C, (unsigned)CtrlOp.getImm(), ElSize, Mask);
       if (!Mask.empty())
         OutStreamer->AddComment(getShuffleComment(MI, 1, 2, Mask),
if (auto *C = getConstantFromPool(*MI, MI->getOperand(4))) {
       if (auto *CF = dyn_cast<ConstantFP>(C)) {
         CS << "0x" << CF->getValueAPF().bitcastToAPInt().toString(16, false);
         OutStreamer->AddComment(CS.str(), !EnablePrintSchedInfo);
       }
     }
     break;
   }
 
-#define MOV_CASE(Prefix, Suffix)        \
-  case X86::Prefix##MOVAPD##Suffix##rm: \
-  case X86::Prefix##MOVAPS##Suffix##rm: \
-  case X86::Prefix##MOVUPD##Suffix##rm: \
-  case X86::Prefix##MOVUPS##Suffix##rm: \
-  case X86::Prefix##MOVDQA##Suffix##rm: \
+#define MOV_CASE(Prefix, Suffix)                                               \
+  case X86::Prefix##MOVAPD##Suffix##rm:                                        \
+  case X86::Prefix##MOVAPS##Suffix##rm:                                        \
+  case X86::Prefix##MOVUPD##Suffix##rm:                                        \
+  case X86::Prefix##MOVUPS##Suffix##rm:                                        \
+  case X86::Prefix##MOVDQA##Suffix##rm:                                        \
   case X86::Prefix##MOVDQU##Suffix##rm:
 
-#define MOV_AVX512_CASE(Suffix)         \
-  case X86::VMOVDQA64##Suffix##rm:      \
-  case X86::VMOVDQA32##Suffix##rm:      \
-  case X86::VMOVDQU64##Suffix##rm:      \
-  case X86::VMOVDQU32##Suffix##rm:      \
-  case X86::VMOVDQU16##Suffix##rm:      \
-  case X86::VMOVDQU8##Suffix##rm:       \
-  case X86::VMOVAPS##Suffix##rm:        \
-  case X86::VMOVAPD##Suffix##rm:        \
-  case X86::VMOVUPS##Suffix##rm:        \
+#define MOV_AVX512_CASE(Suffix)                                                \
+  case X86::VMOVDQA64##Suffix##rm:                                             \
+  case X86::VMOVDQA32##Suffix##rm:                                             \
+  case X86::VMOVDQU64##Suffix##rm:                                             \
+  case X86::VMOVDQU32##Suffix##rm:                                             \
+  case X86::VMOVDQU16##Suffix##rm:                                             \
+  case X86::VMOVDQU8##Suffix##rm:                                              \
+  case X86::VMOVAPS##Suffix##rm:                                               \
+  case X86::VMOVAPD##Suffix##rm:                                               \
+  case X86::VMOVUPS##Suffix##rm:                                               \
   case X86::VMOVUPD##Suffix##rm:
 
-#define CASE_ALL_MOV_RM()               \
-  MOV_CASE(, )   /* SSE */              \
-  MOV_CASE(V, )  /* AVX-128 */          \
-  MOV_CASE(V, Y) /* AVX-256 */          \
-  MOV_AVX512_CASE(Z)                    \
-  MOV_AVX512_CASE(Z256)                 \
+#define CASE_ALL_MOV_RM()                                                      \
+  MOV_CASE(, )   /* SSE */                                                     \
+  MOV_CASE(V, )  /* AVX-128 */                                                 \
+  MOV_CASE(V, Y) /* AVX-256 */                                                 \
+  MOV_AVX512_CASE(Z)                                                           \
+  MOV_AVX512_CASE(Z256)                                                        \
   MOV_AVX512_CASE(Z128)
 
     // For loads from a constant pool to a vector register, print the constant
     // loaded.
     CASE_ALL_MOV_RM()
   case X86::VBROADCASTF128:
   case X86::VBROADCASTI128:
   case X86::VBROADCASTF32X4Z256rm:
   case X86::VBROADCASTF32X4rm:
   case X86::VBROADCASTF32X8rm:
   case X86::VBROADCASTF64X2Z128rm:
   case X86::VBROADCASTF64X2rm:
   case X86::VBROADCASTF64X4rm:
   case X86::VBROADCASTI32X4Z256rm:
   case X86::VBROADCASTI32X4rm:
   case X86::VBROADCASTI32X8rm:
   case X86::VBROADCASTI64X2Z128rm:
   case X86::VBROADCASTI64X2rm:
   case X86::VBROADCASTI64X4rm:
     if (!OutStreamer->isVerboseAsm())
       break;
     if (MI->getNumOperands() <= 4)
       break;
     if (auto *C = getConstantFromPool(*MI, MI->getOperand(4))) {
       int NumLanes = 1;
       // Override NumLanes for the broadcast instructions.
       switch (MI->getOpcode()) {
-      case X86::VBROADCASTF128:         NumLanes = 2;  break;
-      case X86::VBROADCASTI128:         NumLanes = 2;  break;
-      case X86::VBROADCASTF32X4Z256rm:  NumLanes = 2;  break;
-      case X86::VBROADCASTF32X4rm:      NumLanes = 4;  break;
-      case X86::VBROADCASTF32X8rm:      NumLanes = 2;  break;
-      case X86::VBROADCASTF64X2Z128rm:  NumLanes = 2;  break;
-      case X86::VBROADCASTF64X2rm:      NumLanes = 4;  break;
-      case X86::VBROADCASTF64X4rm:      NumLanes = 2;  break;
-      case X86::VBROADCASTI32X4Z256rm:  NumLanes = 2;  break;
-      case X86::VBROADCASTI32X4rm:      NumLanes = 4;  break;
-      case X86::VBROADCASTI32X8rm:      NumLanes = 2;  break;
-      case X86::VBROADCASTI64X2Z128rm:  NumLanes = 2;  break;
-      case X86::VBROADCASTI64X2rm:      NumLanes = 4;  break;
-      case X86::VBROADCASTI64X4rm:      NumLanes = 2;  break;
+      case X86::VBROADCASTF128:
+        NumLanes = 2;
+        break;
+      case X86::VBROADCASTI128:
+        NumLanes = 2;
+        break;
+      case X86::VBROADCASTF32X4Z256rm:
+        NumLanes = 2;
+        break;
+      case X86::VBROADCASTF32X4rm:
+        NumLanes = 4;
+        break;
+      case X86::VBROADCASTF32X8rm:
+        NumLanes = 2;
+        break;
+      case X86::VBROADCASTF64X2Z128rm:
+        NumLanes = 2;
+        break;
+      case X86::VBROADCASTF64X2rm:
+        NumLanes = 4;
+        break;
+      case X86::VBROADCASTF64X4rm:
+        NumLanes = 2;
+        break;
+      case X86::VBROADCASTI32X4Z256rm:
+        NumLanes = 2;
+        break;
+      case X86::VBROADCASTI32X4rm:
+        NumLanes = 4;
+        break;
+      case X86::VBROADCASTI32X8rm:
+        NumLanes = 2;
+        break;
+      case X86::VBROADCASTI64X2Z128rm:
+        NumLanes = 2;
+        break;
+      case X86::VBROADCASTI64X2rm:
+        NumLanes = 4;
+        break;
+      case X86::VBROADCASTI64X4rm:
+        NumLanes = 2;
+        break;
       }
 
       std::string Comment;
       raw_string_ostream CS(Comment);
       const MachineOperand &DstOp = MI->getOperand(0);
       CS << X86ATTInstPrinter::getRegisterName(DstOp.getReg()) << " = ";
       if (auto *CDS = dyn_cast<ConstantDataSequential>(C)) {
         CS << "[";
         for (int l = 0; l != NumLanes; ++l) {
-          for (int i = 0, NumElements = CDS->getNumElements(); i < NumElements; ++i) {
+          for (int i = 0, NumElements = CDS->getNumElements(); i < NumElements;
+               ++i) {
             if (i != 0 || l != 0)
               CS << ",";
             if (CDS->getElementType()->isIntegerTy())
               CS << CDS->getElementAsInteger(i);
             else if (CDS->getElementType()->isFloatTy())
               CS << CDS->getElementAsFloat(i);
             else if (CDS->getElementType()->isDoubleTy())
               CS << CDS->getElementAsDouble(i);
             else
               CS << "?";
           }
         }
         CS << "]";
         OutStreamer->AddComment(CS.str(), !EnablePrintSchedInfo);
       } else if (auto *CV = dyn_cast<ConstantVector>(C)) {
         CS << "<";
         for (int l = 0; l != NumLanes; ++l) {
-          for (int i = 0, NumOperands = CV->getNumOperands(); i < NumOperands; ++i) {
+          for (int i = 0, NumOperands = CV->getNumOperands(); i < NumOperands;
+               ++i) {
             if (i != 0 || l != 0)
               CS << ",";
             printConstant(CV->getOperand(i), CS);
           }
         }
         CS << ">";
         OutStreamer->AddComment(CS.str(), !EnablePrintSchedInfo);
       }
#define CASE_ALL_MOV_RM()                                                      \
   case X86::VPBROADCASTWZm:
     if (!OutStreamer->isVerboseAsm())
       break;
     if (MI->getNumOperands() <= 4)
       break;
     if (auto *C = getConstantFromPool(*MI, MI->getOperand(4))) {
       int NumElts;
       switch (MI->getOpcode()) {
-      default: llvm_unreachable("Invalid opcode");
-      case X86::VBROADCASTSSrm:    NumElts = 4;  break;
-      case X86::VBROADCASTSSYrm:   NumElts = 8;  break;
-      case X86::VBROADCASTSSZ128m: NumElts = 4;  break;
-      case X86::VBROADCASTSSZ256m: NumElts = 8;  break;
-      case X86::VBROADCASTSSZm:    NumElts = 16; break;
-      case X86::VBROADCASTSDYrm:   NumElts = 4;  break;
-      case X86::VBROADCASTSDZ256m: NumElts = 4;  break;
-      case X86::VBROADCASTSDZm:    NumElts = 8;  break;
-      case X86::VPBROADCASTBrm:    NumElts = 16; break;
-      case X86::VPBROADCASTBYrm:   NumElts = 32; break;
-      case X86::VPBROADCASTBZ128m: NumElts = 16; break;
-      case X86::VPBROADCASTBZ256m: NumElts = 32; break;
-      case X86::VPBROADCASTBZm:    NumElts = 64; break;
-      case X86::VPBROADCASTDrm:    NumElts = 4;  break;
-      case X86::VPBROADCASTDYrm:   NumElts = 8;  break;
-      case X86::VPBROADCASTDZ128m: NumElts = 4;  break;
-      case X86::VPBROADCASTDZ256m: NumElts = 8;  break;
-      case X86::VPBROADCASTDZm:    NumElts = 16; break;
-      case X86::VPBROADCASTQrm:    NumElts = 2;  break;
-      case X86::VPBROADCASTQYrm:   NumElts = 4;  break;
-      case X86::VPBROADCASTQZ128m: NumElts = 2;  break;
-      case X86::VPBROADCASTQZ256m: NumElts = 4;  break;
-      case X86::VPBROADCASTQZm:    NumElts = 8;  break;
-      case X86::VPBROADCASTWrm:    NumElts = 8;  break;
-      case X86::VPBROADCASTWYrm:   NumElts = 16; break;
-      case X86::VPBROADCASTWZ128m: NumElts = 8;  break;
-      case X86::VPBROADCASTWZ256m: NumElts = 16; break;
-      case X86::VPBROADCASTWZm:    NumElts = 32; break;
+      default:
+        llvm_unreachable("Invalid opcode");
+      case X86::VBROADCASTSSrm:
+        NumElts = 4;
+        break;
+      case X86::VBROADCASTSSYrm:
+        NumElts = 8;
+        break;
+      case X86::VBROADCASTSSZ128m:
+        NumElts = 4;
+        break;
+      case X86::VBROADCASTSSZ256m:
+        NumElts = 8;
+        break;
+      case X86::VBROADCASTSSZm:
+        NumElts = 16;
+        break;
+      case X86::VBROADCASTSDYrm:
+        NumElts = 4;
+        break;
+      case X86::VBROADCASTSDZ256m:
+        NumElts = 4;
+        break;
+      case X86::VBROADCASTSDZm:
+        NumElts = 8;
+        break;
+      case X86::VPBROADCASTBrm:
+        NumElts = 16;
+        break;
+      case X86::VPBROADCASTBYrm:
+        NumElts = 32;
+        break;
+      case X86::VPBROADCASTBZ128m:
+        NumElts = 16;
+        break;
+      case X86::VPBROADCASTBZ256m:
+        NumElts = 32;
+        break;
+      case X86::VPBROADCASTBZm:
+        NumElts = 64;
+        break;
+      case X86::VPBROADCASTDrm:
+        NumElts = 4;
+        break;
+      case X86::VPBROADCASTDYrm:
+        NumElts = 8;
+        break;
+      case X86::VPBROADCASTDZ128m:
+        NumElts = 4;
+        break;
+      case X86::VPBROADCASTDZ256m:
+        NumElts = 8;
+        break;
+      case X86::VPBROADCASTDZm:
+        NumElts = 16;
+        break;
+      case X86::VPBROADCASTQrm:
+        NumElts = 2;
+        break;
+      case X86::VPBROADCASTQYrm:
+        NumElts = 4;
+        break;
+      case X86::VPBROADCASTQZ128m:
+        NumElts = 2;
+        break;
+      case X86::VPBROADCASTQZ256m:
+        NumElts = 4;
+        break;
+      case X86::VPBROADCASTQZm:
+        NumElts = 8;
+        break;
+      case X86::VPBROADCASTWrm:
+        NumElts = 8;
+        break;
+      case X86::VPBROADCASTWYrm:
+        NumElts = 16;
+        break;
+      case X86::VPBROADCASTWZ128m:
+        NumElts = 8;
+        break;
+      case X86::VPBROADCASTWZ256m:
+        NumElts = 16;
+        break;
+      case X86::VPBROADCASTWZm:
+        NumElts = 32;
+        break;
       }
 
       std::string Comment;
       raw_string_ostream CS(Comment);
       const MachineOperand &DstOp = MI->getOperand(0);
       CS << X86ATTInstPrinter::getRegisterName(DstOp.getReg()) << " = ";
       CS << "[";
       for (int i = 0; i != NumElts; ++i) {

llvm/trunk/test/CodeGen/X86/xray-custom-log.ll

 ; RUN: llc -filetype=asm -o - -mtriple=x86_64-unknown-linux-gnu < %s | FileCheck %s
 ; RUN: llc -filetype=asm -o - -mtriple=x86_64-unknown-linux-gnu \
 ; RUN:    -relocation-model=pic < %s | FileCheck %s -check-prefix=PIC
 
 define i32 @fn() nounwind noinline uwtable "function-instrument"="xray-always" {
     %eventptr = alloca i8
     %eventsize = alloca i32
     store i32 3, i32* %eventsize
     %val = load i32, i32* %eventsize
     call void @llvm.xray.customevent(i8* %eventptr, i32 %val)
     ; CHECK-LABEL: Lxray_event_sled_0:
     ; CHECK:       .byte 0xeb, 0x0f
     ; CHECK-NEXT:  pushq %rdi
-    ; CHECK-NEXT:  movq {{.*}}, %rdi
     ; CHECK-NEXT:  pushq %rsi
+    ; CHECK-NEXT:  movq {{.*}}, %rdi
     ; CHECK-NEXT:  movq {{.*}}, %rsi
     ; CHECK-NEXT:  callq __xray_CustomEvent
     ; CHECK-NEXT:  popq %rsi
     ; CHECK-NEXT:  popq %rdi
 
     ; PIC-LABEL: Lxray_event_sled_0:
     ; PIC:       .byte 0xeb, 0x0f
     ; PIC-NEXT:  pushq %rdi
-    ; PIC-NEXT:  movq {{.*}}, %rdi
     ; PIC-NEXT:  pushq %rsi
+    ; PIC-NEXT:  movq {{.*}}, %rdi
     ; PIC-NEXT:  movq {{.*}}, %rsi
     ; PIC-NEXT:  callq __xray_CustomEvent@PLT
     ; PIC-NEXT:  popq %rsi
     ; PIC-NEXT:  popq %rdi
     ret i32 0
 }
 ; CHECK-LABEL: xray_instr_map
 ; CHECK-LABEL: Lxray_sleds_start0:
 ; CHECK:       .quad {{.*}}xray_event_sled_0
 
 declare void @llvm.xray.customevent(i8*, i32)

llvm/trunk/test/CodeGen/X86/xray-typed-event-log.ll

+; RUN: llc -filetype=asm -o - -mtriple=x86_64-unknown-linux-gnu < %s | FileCheck %s
+; RUN: llc -filetype=asm -o - -mtriple=x86_64-unknown-linux-gnu \
+; RUN:    -relocation-model=pic < %s | FileCheck %s -check-prefix=PIC
+
+define i32 @fn() nounwind noinline uwtable "function-instrument"="xray-always" {
+    %eventptr = alloca i8
+    %eventsize = alloca i32
+    %eventtype = alloca i16
+    store i16 6, i16* %eventtype
+    %type = load i16, i16* %eventtype
+    store i32 3, i32* %eventsize
+    %val = load i32, i32* %eventsize
+    call void @llvm.xray.typedevent(i16 %type, i8* %eventptr, i32 %val)
+    ; CHECK-LABEL: Lxray_typed_event_sled_0:
+    ; CHECK:       .byte 0xeb, 0x14
+    ; CHECK-NEXT:  pushq %rdi
+    ; CHECK-NEXT:  pushq %rsi
+    ; CHECK-NEXT:  pushq %rdx
+    ; CHECK-NEXT:  movq {{.*}}, %rdi
+    ; CHECK-NEXT:  movq {{.*}}, %rsi
+    ; CHECK-NEXT:  movq {{.*}}, %rdx
+    ; CHECK-NEXT:  callq __xray_TypedEvent
+    ; CHECK-NEXT:  popq %rdx
+    ; CHECK-NEXT:  popq %rsi
+    ; CHECK-NEXT:  popq %rdi
+
+    ; PIC-LABEL: Lxray_typed_event_sled_0:
+    ; PIC:       .byte 0xeb, 0x14
+    ; PIC-NEXT:  pushq %rdi
+    ; PIC-NEXT:  pushq %rsi
+    ; PIC-NEXT:  pushq %rdx
+    ; PIC-NEXT:  movq {{.*}}, %rdi
+    ; PIC-NEXT:  movq {{.*}}, %rsi
+    ; PIC-NEXT:  movq {{.*}}, %rdx
+    ; PIC-NEXT:  callq __xray_TypedEvent@PLT
+    ; PIC-NEXT:  popq %rdx
+    ; PIC-NEXT:  popq %rsi
+    ; PIC-NEXT:  popq %rdi
+    ret i32 0
+}
+; CHECK-LABEL: xray_instr_map
+; CHECK-LABEL: Lxray_sleds_start0:
+; CHECK:       .quad {{.*}}xray_typed_event_sled_0
+
+declare void @llvm.xray.typedevent(i16, i8*, i32)