[XRay] [compiler-rt] Implement trampoline and handler for typed xray event tracing.

[kpw]

Summary

Compiler-rt support first before defining the __xray_typedevent() lowering in
llvm. I'm looking for some early feedback before I touch much more code.

[kpw]

• Jump 20

[dberris]

Great start, @kpw! Thanks. Please see comments below.

include/xray/xray_interface.h

88

I don't think you need the additional const for the declaration.

Also, I suspect 'noexcept' in the extern "C" { ... } section wouldn't make any sense, and would be superfluous.

lib/xray/xray_fdr_logging.cc

328

Add a comment why we need to have this at 3 bytes?

Or, if we encounter that the EventType is > unsigned 24 bits, then we should say something.

Alternatively, we can limit the size of the payload to 16 bits (64Kb) instead of going the full 32 bit length. WDYT?

lib/xray/xray_interface.cc

389

You might want to make sure that the descriptor count doesn't exceed the 24 bits maximum, and provide for returning an invalid number. We can make either 0 or numeric_limits<uint32_t>::max() represent an error condition when that happens. Either we do it here, or do it in the FDR implementation somehow.

lib/xray/xray_x86_64.cc

293–301

Sled versioning is per-sled, so we can use Version 0 to be this implementation. Note that the XRaySledEntry type has a version field that's associated with each sled, so when lowering we fill it out by default with 0s -- in the custom event sleds, we've had to change the version but still be backwards compatible (hence the comment from which this came).

[kpw]

Thanks for the feedback Dean.

include/xray/xray_interface.h

88

Fair enough about the const.

I wasn't sure if extern "C" only specified linkage. Digging was inconclusive.

https://stackoverflow.com/questions/24362616/does-the-c-standard-mandate-that-c-linkage-functions-are-noexcept

lib/xray/xray_fdr_logging.cc

328

32 bit length payloads seem excessive for our use cases, as do 32 bits worth of differing event types. Constraining the event types seemed more natural because we control the id registration.

There's probably better error handling to do here.

lib/xray/xray_interface.cc

389

Ack. Doing it here seems reasonable.

[dberris]

include/xray/xray_interface.h

88

The noexcept doesn't really add much here, and could be distracting, and is not something we can guarantee either. Is there something in particular you're guarding for with putting the noexcept specifier(s) on these functions?

[dberris]

Adding just one comment, after thinking about it a little more.

lib/xray/xray_fdr_logging.cc

328

I agree. Can we live with unsigned 16 bits worth of types and payloads? That would make a lot of this stuff more conservative.

[kpw]

[XRay] [compiler-rt] Implement trampoline and handler for typed xray event tracing.

Compiler-rt support first before defining the __xray_typedevent() lowering in
llvm.

[kpw]

These comments were dangling for a while, so I'm posting an update. Currently compiling llvm with a new intrinsic to match. Fingers crossed!

include/xray/xray_interface.h

88

I've removed them. I'm not sure the stack unwinder even works for functions that use the C calling convention. Since I'm not sure we can support exceptions properly, my thinking was that it might be better to put noexcept so that this behavior just becomes std::terminate if anything is thrown. I don't know what you mean that we can't guarantee no exceptions? We've written the implementations without throwing. These are just handler registration, not handler execution.

Sure there could be a std::bad_alloc, but that's common in noexcept functions and noexcept just means exceptions will be transformed into std::terminate.

I don't want to make the file inconsistent and confusing, by adding the specifier selectively, but it may be more clear to add to all the registration functions since we haven't written implementations that can throw.

I don't think compiler optimizations are very relevant here. They mostly come down to

1. std::move_if_no_except - Not relevant here.
2. Should the compiler generate code for stack unwinding? - Registering is rare compared to invocation anyway.

[kpw]

include/xray/xray_interface.h

87

I should change the return type here as well. Whoops.

[dberris]

LGTM

include/xray/xray_interface.h

88

Thanks -- I think you raise interesting points here that are worth chasing down.

In particular, we should definitely think more about the exception safety of the XRay implementation in general. Especially in the parts where we're calling the event handlers, which internally could be throwing exceptions -- the assembler parts ought to either handle that properly by having some exception-handling built-in.

That might be too hard, or not worth it (depending on the usage scenario for XRay) but we certainly should document the requirements for event handlers (function entry/exit, typed events, custom events, etc.).

Let's keep thinking about this, and incrementally address the exception safety aspect as we go along. I definitely think that noexcept is appropriate in a lot of the XRay implementation (if not all of them) but we should do it in a disciplined, thoughtful manner. :)

[kpw]

[XRay] [compiler-rt] Implement trampoline and handler for typed xray event tracing.

Compiler-rt support first before defining the __xray_typedevent() lowering in llvm.

[kpw]

I'm going to land this and the llvm/clang changes and in a separate CL make some improvements as discussed offline for event type registration (handling deduping event descriptors).

include/xray/xray_interface.h

 
 /// Synchronize this with AsmPrinter::SledKind in LLVM.
 enum XRayEntryType {
   ENTRY = 0,
   EXIT = 1,
   TAIL = 2,
   LOG_ARGS_ENTRY = 3,
   CUSTOM_EVENT = 4,
+  TYPED_EVENT = 5,
 };
 
 /// Provide a function to invoke for when instrumentation points are hit. This
 /// is a user-visible control surface that overrides the default implementation.
 /// The function provided should take the following arguments:
 ///
 ///   - function id: an identifier that indicates the id of a function; this id
 ///                  is generated by xray; the mapping between the function id

 extern int __xray_set_handler_arg1(void (*entry)(int32_t, XRayEntryType,
                                                  uint64_t));
 
 /// Disables the XRay handler used to log first arguments of function calls.
 /// Returns 1 on success, 0 on error.
 extern int __xray_remove_handler_arg1();
 
 /// Provide a function to invoke when XRay encounters a custom event.
-extern int __xray_set_customevent_handler(void (*entry)(void*, std::size_t));
+extern int __xray_set_customevent_handler(void (*entry)(void *, std::size_t));
 
 /// This removes whatever the currently provided custom event handler is.
 /// Returns 1 on success, 0 on error.
 extern int __xray_remove_customevent_handler();
 
+/// Set a handler for xray typed event logging. The first parameter is a type
+/// identifier, the second is a payload, and the third is the payload size.
+extern int __xray_set_typedevent_handler(void (*entry)(uint16_t, const void *,
+                                                       std::size_t));
+
+/// Removes the currently set typed event handler.
+/// Returns 1 on success, 0 on error.
+extern int __xray_remove_typedevent_handler();
+
+extern uint16_t __xray_register_event_type(const char *event_type);
+
 enum XRayPatchingStatus {
   NOT_INITIALIZED = 0,
   SUCCESS = 1,
   ONGOING = 2,
   FAILED = 3,
 };
 
 /// This tells XRay to patch the instrumentation points. See XRayPatchingStatus

lib/xray/xray_fdr_log_records.h

enum class RecordKinds : uint8_t {
     NewBuffer,
     EndOfBuffer,
     NewCPUId,
     TSCWrap,
     WalltimeMarker,
     CustomEventMarker,
     CallArgument,
     BufferExtents,
+    TypedEventMarker,
   };
 
   // Use 7 bits to identify this record type.
   /* RecordKinds */ uint8_t RecordKind : 7;
   char Data[15];
 } __attribute__((packed));
 
 static_assert(sizeof(MetadataRecord) == 16, "Wrong size for MetadataRecord.");

lib/xray/xray_fdr_logging.cc

void fdrLoggingHandleCustomEvent(void *Event,
   std::memcpy(&CustomEvent.Data[sizeof(int32_t)], &TSC, TSCSize);
   std::memcpy(TLD.RecordPtr, &CustomEvent, sizeof(CustomEvent));
   TLD.RecordPtr += sizeof(CustomEvent);
   std::memcpy(TLD.RecordPtr, Event, ReducedEventSize);
   incrementExtents(MetadataRecSize + EventSize);
   endBufferIfFull();
 }
 
+void fdrLoggingHandleTypedEvent(
+    uint16_t EventType, const void *Event,
+    std::size_t EventSize) noexcept XRAY_NEVER_INSTRUMENT {
+  using namespace __xray_fdr_internal;
+  auto TC = getTimestamp();
+  auto &TSC = TC.TSC;
+  auto &CPU = TC.CPU;
+  RecursionGuard Guard{Running};
+  if (!Guard)
+    return;
+  if (EventSize > std::numeric_limits<int32_t>::max()) {
+    using Empty = struct {};
+    static Empty Once = [&] {
+      Report("Event size too large = %zu ; > max = %d\n", EventSize,
+             std::numeric_limits<int32_t>::max());
+      return Empty();
+    }();
+    (void)Once;
+  }
+  int32_t ReducedEventSize = static_cast<int32_t>(EventSize);
+  auto &TLD = getThreadLocalData();
+  if (!isLogInitializedAndReady(TLD.BQ, TSC, CPU, clock_gettime))
+    return;
+
+  // Here we need to prepare the log to handle:
+  //   - The metadata record we're going to write. (16 bytes)
+  //   - The additional data we're going to write. Currently, that's the size of
+  //   the event we're going to dump into the log as free-form bytes.
+  if (!prepareBuffer(TSC, CPU, clock_gettime, MetadataRecSize + EventSize)) {
+    TLD.BQ = nullptr;
+    return;
+  }
+  // Write the custom event metadata record, which consists of the following
+  // information:
+  //   - 8 bytes (64-bits) for the full TSC when the event started.
+  //   - 4 bytes (32-bits) for the length of the data.
+  //   - 2 bytes (16-bits) for the event type. 3 bytes remain since one of the
+  //       bytes has the record type (Metadata Record) and kind (TypedEvent).
+  //       We'll log the error if the event type is greater than 2 bytes.
+  //       Event types are generated sequentially, so 2^16 is enough.
+  MetadataRecord TypedEvent;
+  TypedEvent.Type = uint8_t(RecordType::Metadata);
+  TypedEvent.RecordKind =
+      uint8_t(MetadataRecord::RecordKinds::TypedEventMarker);
+  constexpr auto TSCSize = sizeof(TC.TSC);
+  std::memcpy(&TypedEvent.Data, &ReducedEventSize, sizeof(int32_t));
+  std::memcpy(&TypedEvent.Data[sizeof(int32_t)], &TSC, TSCSize);
+  std::memcpy(&TypedEvent.Data[sizeof(int32_t) + TSCSize], &EventType,
+              sizeof(EventType));
+  std::memcpy(TLD.RecordPtr, &TypedEvent, sizeof(TypedEvent));
+
+  TLD.RecordPtr += sizeof(TypedEvent);
+  std::memcpy(TLD.RecordPtr, Event, ReducedEventSize);
+  incrementExtents(MetadataRecSize + EventSize);
+  endBufferIfFull();
+}
+
 XRayLogInitStatus fdrLoggingInit(std::size_t BufferSize, std::size_t BufferMax,
                                  void *Options,
                                  size_t OptionsSize) XRAY_NEVER_INSTRUMENT {
   if (OptionsSize != sizeof(FDRLoggingOptions)) {
     if (__sanitizer::Verbosity())
       Report("Cannot initialize FDR logging; wrong size for options: %d\n",
              OptionsSize);
     return static_cast<XRayLogInitStatus>(__sanitizer::atomic_load(
XRayLogInitStatus fdrLoggingInit(std::size_t BufferSize, std::size_t BufferMax,
   }();
 
   // Arg1 handler should go in first to avoid concurrent code accidentally
   // falling back to arg0 when it should have ran arg1.
   __xray_set_handler_arg1(fdrLoggingHandleArg1);
   // Install the actual handleArg0 handler after initialising the buffers.
   __xray_set_handler(fdrLoggingHandleArg0);
   __xray_set_customevent_handler(fdrLoggingHandleCustomEvent);
+  __xray_set_typedevent_handler(fdrLoggingHandleTypedEvent);
 
   __sanitizer::atomic_store(&LoggingStatus,
                             XRayLogInitStatus::XRAY_LOG_INITIALIZED,
                             __sanitizer::memory_order_release);
 
   if (__sanitizer::Verbosity())
     Report("XRay FDR init successful.\n");
   return XRayLogInitStatus::XRAY_LOG_INITIALIZED;

lib/xray/xray_fdr_logging_impl.h

static bool Once = [&] {
       Report("Internal error: patched an XRay custom event call as a function; "
              "func id = %d\n",
              FuncId);
       return true;
     }();
     (void)Once;
     return;
   }
+  case XRayEntryType::TYPED_EVENT: {
+    static bool Once = [&] {
+      Report("Internal error: patched an XRay typed event call as a function; "
+             "func id = %d\n",
+             FuncId);
+      return true;
+    }();
+    (void)Once;
+    return;
+  }
   }
 
   std::memcpy(TLD.RecordPtr, &FuncRecord, sizeof(FunctionRecord));
   TLD.RecordPtr += sizeof(FunctionRecord);
   incrementExtents(sizeof(FunctionRecord));
 }
 
 static uint64_t thresholdTicks() {
static bool Once = [&] {
       Report("Internal error: patched an XRay custom event call as a function; "
              "func id = %d",
              FuncId);
       return true;
     }();
     (void)Once;
     return;
   }
+  case XRayEntryType::TYPED_EVENT: {
+    static bool Once = [&] {
+      Report("Internal error: patched an XRay typed event call as a function; "
+             "func id = %d\n",
+             FuncId);
+      return true;
+    }();
+    (void)Once;
+    return;
+  }
   }
 
   writeFunctionRecord(FuncId, RecordTSCDelta, Entry);
   if (Entry == XRayEntryType::LOG_ARGS_ENTRY)
     writeCallArgumentMetadata(Arg1);
 
   // If we've exhausted the buffer by this time, we then release the buffer to
   // make sure that other threads may start using this buffer.
   endBufferIfFull();
 }
 
 } // namespace __xray_fdr_internal
 } // namespace __xray
 
 #endif // XRAY_XRAY_FDR_LOGGING_IMPL_H

lib/xray/xray_inmemory_log.cc

Report("WARNING: Required CPU features missing for XRay instrumentation, "
            "using emulation instead.\n");
 
   GlobalOptions = *reinterpret_cast<BasicLoggingOptions *>(Options);
   __xray_set_handler_arg1(UseRealTSC ? basicLoggingHandleArg1RealTSC
                                      : basicLoggingHandleArg1EmulateTSC);
   __xray_set_handler(UseRealTSC ? basicLoggingHandleArg0RealTSC
                                 : basicLoggingHandleArg0EmulateTSC);
   __xray_remove_customevent_handler();
+  __xray_remove_typedevent_handler();
 
   return XRayLogInitStatus::XRAY_LOG_INITIALIZED;
 }
 
 XRayLogInitStatus basicLoggingFinalize() XRAY_NEVER_INSTRUMENT {
   uint8_t Expected = 0;
   if (!__sanitizer::atomic_compare_exchange_strong(
           &BasicInitialized, &Expected, 0, __sanitizer::memory_order_acq_rel) &&

lib/xray/xray_interface.cc

 //===----------------------------------------------------------------------===//
 
 #include "xray_interface_internal.h"
 
 #include <cstdint>
 #include <cstdio>
 #include <errno.h>
 #include <limits>
+#include <string.h>
 #include <sys/mman.h>
 
+#include "sanitizer_common/sanitizer_addrhashmap.h"
 #include "sanitizer_common/sanitizer_common.h"
+
 #include "xray_defs.h"
 #include "xray_flags.h"
 
 extern __sanitizer::SpinMutex XRayInstrMapMutex;
 extern __sanitizer::atomic_uint8_t XRayInitialized;
 extern __xray::XRaySledMap XRayInstrMap;
 
 namespace __xray {

 __sanitizer::atomic_uintptr_t XRayPatchedFunction{0};
 
 // This is the function to call from the arg1-enabled sleds/trampolines.
 __sanitizer::atomic_uintptr_t XRayArgLogger{0};
 
 // This is the function to call when we encounter a custom event log call.
 __sanitizer::atomic_uintptr_t XRayPatchedCustomEvent{0};
 
+// This is the function to call when we encounter a typed event log call.
+__sanitizer::atomic_uintptr_t XRayPatchedTypedEvent{0};
+
 // This is the global status to determine whether we are currently
 // patching/unpatching.
 __sanitizer::atomic_uint8_t XRayPatching{0};
 
-// MProtectHelper is an RAII wrapper for calls to mprotect(...) that will undo
-// any successful mprotect(...) changes. This is used to make a page writeable
-// and executable, and upon destruction if it was successful in doing so returns
-// the page into a read-only and executable page.
+struct TypeDescription {
+  uint32_t type_id;
+  std::size_t description_string_length;
+};
+
+using TypeDescriptorMapType = __sanitizer::AddrHashMap<TypeDescription, 11>;
+// An address map from immutable descriptors to type ids.
+TypeDescriptorMapType TypeDescriptorAddressMap{};
+
+__sanitizer::atomic_uint32_t TypeEventDescriptorCounter{0};
+
+// MProtectHelper is an RAII wrapper for calls to mprotect(...) that will
+// undo any successful mprotect(...) changes. This is used to make a page
+// writeable and executable, and upon destruction if it was successful in
+// doing so returns the page into a read-only and executable page.
 //
 // This is only used specifically for runtime-patching of the XRay
-// instrumentation points. This assumes that the executable pages are originally
-// read-and-execute only.
+// instrumentation points. This assumes that the executable pages are
+// originally read-and-execute only.
 class MProtectHelper {
   void *PageAlignedAddr;
   std::size_t MProtectLen;
   bool MustCleanup;
 
 public:
   explicit MProtectHelper(void *PageAlignedAddr,
                           std::size_t MProtectLen) XRAY_NEVER_INSTRUMENT
case XRayEntryType::TAIL:
     Success = patchFunctionTailExit(Enable, FuncId, Sled);
     break;
   case XRayEntryType::LOG_ARGS_ENTRY:
     Success = patchFunctionEntry(Enable, FuncId, Sled, __xray_ArgLoggerEntry);
     break;
   case XRayEntryType::CUSTOM_EVENT:
     Success = patchCustomEvent(Enable, FuncId, Sled);
     break;
+  case XRayEntryType::TYPED_EVENT:
+    Success = patchTypedEvent(Enable, FuncId, Sled);
+    break;
   default:
     Report("Unsupported sled kind '%d' @%04x\n", Sled.Address, int(Sled.Kind));
     return false;
   }
   return Success;
 }
 
 XRayPatchingStatus patchFunction(int32_t FuncId,
if (__sanitizer::atomic_load(&XRayInitialized,
     __sanitizer::atomic_store(&__xray::XRayPatchedCustomEvent,
                               reinterpret_cast<uintptr_t>(entry),
                               __sanitizer::memory_order_release);
     return 1;
   }
   return 0;
 }
 
+int __xray_set_typedevent_handler(void (*entry)(
+    uint16_t, const void *, size_t)) noexcept XRAY_NEVER_INSTRUMENT {
+  if (__sanitizer::atomic_load(&XRayInitialized,
+                               __sanitizer::memory_order_acquire)) {
+    __sanitizer::atomic_store(&__xray::XRayPatchedTypedEvent,
+                              reinterpret_cast<uintptr_t>(entry),
+                              __sanitizer::memory_order_release);
+    return 1;
+  }
+  return 0;
+}
+
 int __xray_remove_handler() XRAY_NEVER_INSTRUMENT {
   return __xray_set_handler(nullptr);
 }
 
 int __xray_remove_customevent_handler() XRAY_NEVER_INSTRUMENT {
   return __xray_set_customevent_handler(nullptr);
 }
 
+int __xray_remove_typedevent_handler() noexcept XRAY_NEVER_INSTRUMENT {
+  return __xray_set_typedevent_handler(nullptr);
+}
+
+uint16_t __xray_register_event_type(
+    const char *const event_type) noexcept XRAY_NEVER_INSTRUMENT {
+  TypeDescriptorMapType::Handle h(&TypeDescriptorAddressMap, (uptr)event_type);
+  if (h.created()) {
+    h->type_id = __sanitizer::atomic_fetch_add(
+        &TypeEventDescriptorCounter, 1, __sanitizer::memory_order_acq_rel);
+    h->description_string_length = strnlen(event_type, 1024);
+  }
+  return h->type_id;
+}
+
 XRayPatchingStatus __xray_patch() XRAY_NEVER_INSTRUMENT {
   return controlPatching(true);
 }
 
 XRayPatchingStatus __xray_unpatch() XRAY_NEVER_INSTRUMENT {
   return controlPatching(false);
 }
 

lib/xray/xray_interface_internal.h

#elif SANITIZER_WORDSIZE == 32
   unsigned char Version;
   unsigned char Padding[5]; // Need 16 bytes
 #else
 #error "Unsupported word size."
 #endif
 };
 
 struct XRayFunctionSledIndex {
-  const XRaySledEntry* Begin;
-  const XRaySledEntry* End;
+  const XRaySledEntry *Begin;
+  const XRaySledEntry *End;
 };
 }
 
 namespace __xray {
 
 struct XRaySledMap {
   const XRaySledEntry *Sleds;
   size_t Entries;
   const XRayFunctionSledIndex *SledsIndex;
   size_t Functions;
 };
 
-bool patchFunctionEntry(bool Enable, uint32_t FuncId,
-                        const XRaySledEntry &Sled, void (*Trampoline)());
+bool patchFunctionEntry(bool Enable, uint32_t FuncId, const XRaySledEntry &Sled,
+                        void (*Trampoline)());
 bool patchFunctionExit(bool Enable, uint32_t FuncId, const XRaySledEntry &Sled);
 bool patchFunctionTailExit(bool Enable, uint32_t FuncId,
                            const XRaySledEntry &Sled);
 bool patchCustomEvent(bool Enable, uint32_t FuncId, const XRaySledEntry &Sled);
+bool patchTypedEvent(bool Enable, uint32_t FuncId, const XRaySledEntry &Sled);
 
 } // namespace __xray
 
 extern "C" {
 // The following functions have to be defined in assembler, on a per-platform
 // basis. See xray_trampoline_*.S files for implementations.
 extern void __xray_FunctionEntry();
 extern void __xray_FunctionExit();
 extern void __xray_FunctionTailExit();
 extern void __xray_ArgLoggerEntry();
 extern void __xray_CustomEvent();
+extern void __xray_TypedEvent();
 }
 
 #endif

lib/xray/xray_trampoline_x86_64.S

ASM_SYMBOL(__xray_CustomEvent):
   ALIGNED_CALL_RAX
 
 .LcustomEventCleanup:
   RESTORE_REGISTERS
   retq
   ASM_SIZE(__xray_CustomEvent)
   CFI_ENDPROC
 
+//===----------------------------------------------------------------------===//
+
+  .global ASM_SYMBOL(__xray_TypedEvent)
+  .align 16, 0x90
+  ASM_TYPE_FUNCTION(__xray_TypedEvent)
+ASM_SYMBOL(__xray_TypedEvent):
+  CFI_STARTPROC
+  SAVE_REGISTERS
+
+  // We pass three arguments to this trampoline, which should be in rdi, rsi
+  // and rdx without our intervention.
+  movq ASM_SYMBOL(_ZN6__xray21XRayPatchedTypedEventE)(%rip), %rax
+  testq %rax,%rax
+  je .LtypedEventCleanup
+
+  ALIGNED_CALL_RAX
+
+.LtypedEventCleanup:
+  RESTORE_REGISTERS
+  retq
+  ASM_SIZE(__xray_TypedEvent)
+  CFI_ENDPROC
+
+//===----------------------------------------------------------------------===//
+
 NO_EXEC_STACK_DIRECTIVE

lib/xray/xray_x86_64.cc

if (sysctlbyname("machdep.tsc_freq", &TSCFrequency, &tscfreqsz,
         NULL, 0) != -1) {
 #endif
         return static_cast<uint64_t>(TSCFrequency);
     } else {
       Report("Unable to determine CPU frequency for TSC accounting.\n");
     }
 
     return 0;
-    
 }
 #else
 uint64_t getTSCFrequency() XRAY_NEVER_INSTRUMENT {
     /* Not supported */
     return 0;
 }
 #endif
 
default:
           reinterpret_cast<std::atomic<uint16_t> *>(Sled.Address), Jmp20Seq,
           std::memory_order_release);
       break;
     }
     }
   return false;
 }
 
+bool patchTypedEvent(const bool Enable, const uint32_t FuncId,
+                      const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT {
+  // Here we do the dance of replacing the following sled:
+  //
+  // xray_sled_n:
+  //   jmp +20          // 2 byte instruction
+  //   ...
+  //
+  // With the following:
+  //
+  //   nopw             // 2 bytes
+  //   ...
+  //
+  //
+  // The "unpatch" should just turn the 'nopw' back to a 'jmp +20'.
+  // The 20 byte sled stashes three argument registers, calls the trampoline,
+  // unstashes the registers and returns. If the arguments are already in
+  // the correct registers, the stashing and unstashing become equivalently
+  // sized nops.
+  if (Enable) {
+    std::atomic_store_explicit(
+        reinterpret_cast<std::atomic<uint16_t> *>(Sled.Address), NopwSeq,
+        std::memory_order_release);
+  } else {
+      std::atomic_store_explicit(
+          reinterpret_cast<std::atomic<uint16_t> *>(Sled.Address), Jmp20Seq,
+          std::memory_order_release);
+  }
+  return false;
+}
+
 // We determine whether the CPU we're running on has the correct features we
 // need. In x86_64 this will be rdtscp support.
 bool probeRequiredCPUFeatures() XRAY_NEVER_INSTRUMENT {
   unsigned int EAX, EBX, ECX, EDX;
 
   // We check whether rdtscp support is enabled. According to the x86_64 manual,
   // level should be set at 0x80000001, and we should have a look at bit 27 in
   // EDX. That's 0x8000000 (or 1u << 27).