[compiler-rt] [XRay] Basic initialization and flag definition for XRay runtime

[dberris]

Summary

This patch implements the initialisation and patching routines for the XRay runtime, along with the necessary trampolines for function entry/exit handling. For now we only define the basic hooks for allowing an implementation to define a handler that gets run on function entry/exit. We expose a minimal API for controlling the behaviour of the runtime (patching, cleanup, and setting the handler to invoke when instrumenting).

Depends on D19904

[dberris]

• Add support for function exits

[dberris]

• Use COMDAT merged section names instead of adhoc globals for instrumentation map.

[dberris]

• Use rdtscp more correctly to get timestamp and cpuid

[dberris]

• Add automatic patching in init for demo

[dberris]

Rebase

[rnk]

include/CMakeLists.txt

1

Maybe rename this to COMPILER_RT_HEADERS, since it's not just sanitizers?

include/xray/xray_interface.h

18–20

Are you sure you want to use a C++ scoped enum in this interface? The whole thing is extern "C", it might be nicer to make it a C style enum, so you can use it in the handler prototype instead of unsigned short.

46–48

Maybe this should be another enum? XRayPatchingStatus?

lib/xray/xray_interface.cc

34–53

This inline asm is too fragile. There's nothing to say the compiler won't use R10 to load Fn. I think this should write a standalone .s file like xray_hooks_x86_64.s, and then we can add _arm.s, _aarch64.s, etc as needed. There's existing cmake for adding .s files in lib/builtins.

38

RBP is already a CSR, why do we need to save it?

lib/xray/xray_interface_internal.h

30–33

We aren't allowed to include xray/xray_interface.h for these?

46

ditto, it'd be nice to get this from the public interface

[dberris]

Thanks for the review Reid, I'm going to clean this up a bit more later with your suggestions from the mailing list on configuration and other things. In the meantime, please have another look at what's now here.

Cheers

include/CMakeLists.txt

1

Good point. Unfortunately the install path is still .../include/sanitizer -- I've added a new variable instead, COMPILER_RT_HEADERS which aggregates SANITIZER_HEADERS and XRAY_HEADERS, and another install rule for xray headers to go into .../include/xray.

Does this make sense?

include/xray/xray_interface.h

18–20

Good point, switched to using a C-style enum instead.

lib/xray/xray_interface.cc

38

That's because this function isn't typically directly called -- this is jumped into unconditionally. Without stashing the RBP, it gets lost when the function call later on returns. Experimentation and hours of debugging led to this state of having to stash the RBP properly.

lib/xray/xray_interface_internal.h

30–33

I've had to add an include directory directive to include the public header, but it works now. It's a little different from how we're doing it in the other sanitisers.

[dberris]

• Address review comments

[dberris]

lib/xray/xray_interface.cc

35–54

Sorry, I forgot to address this one.

I'll need to have a look at how this is done in other places, but did you mean using the assembler to also generate the call to the function pointer? Or having this whole routine be implemented in assembly?

[rnk]

lib/xray/xray_interface.cc

35–54

Yeah, I would recommend calling the function pointer in with assembly. I guess it's a little crappy that the assembly has to know the numeric value of XRayEntryType::EXIT, but I can't think of a good way to solve that.

[dberris]

• Rebase
• Remove demo parts, define some flags

[dberris]

I've added one flag, and the beginnings of having more flags to be supported going forward.

This is now ready for another look.

lib/xray/xray_interface.cc

36–55

I thought about this a little more, and I think it's going to be a bit harder to implement a few of the things being done here in pure assembler. I'm sure I can do it, but I'd rather do it in a separate change, after this one lands, if that's alright with you?

[dberris]

• Actually initialize the flags

[rnk]

lib/xray/xray_interface.cc

37–56

I actually think this is important enough to get right the first time. Inline asm has scarred me enough times already.

39

OK, but the prologue will push RBP if it is clobbered anywhere in this function, and if it isn't, the prologue of 'Fn' will preserve it.

63

What about saving and restoring XMM registers? Those are used for parameters and return values, and are usually considered to be scratch. Is it assumed that XRay authors will carefully audit their code to avoid SSE instructions? Even memcpy uses SSE these days, though.

70–71

I guess you only save and restore RAX/RDX since those are use for return values. I'm sure other calling conventions can break this assumption, but we can leave it this way for now if you want.

[echristo]

lib/xray/xray_interface.cc

38

Meta comment: Typically inline asm can be a single block rather than N statements of 1 per instruction.

[dberris]

• Implement the trampolines in assembler
• Tweak dependencies and implementation to rely on hand-written assembler

lib/xray/xray_interface.cc

37–56

Good call. It didn't turn out to be as bad as I thought, though I did need to reverse-engineer what the compiler does for a bit to get it "right". :)

38

Thanks -- I tried that but I didn't know how to properly write out the operands when I tried, so I split them out. :D

Now this is all in its own file, so it shouldn't be that big of a problem anymore.

63

Yeah, that's a tricky one. I've been thinking about this too and I think it's worth saving/restoring those just to be safe. Unfortunately that might mean there's a bit more overhead as far as runtime and space requirements is concerned.

70–71

Added a FIXME in the assembly file to address at a later point.

[dberris]

• Typo on filename

[dberris]

• Fix flag parsing to support verbosity

[dberris]

• Put the function id on the stack for the argument

[dberris]

Rebase

[dberris]

This is now ready for another look -- thanks in advance!

[rnk]

Thanks for messing around with the asm parts, it's a pain. lgtm with nits

lib/xray/xray_trampoline_x86.S

27

This is the unwind info you should have after the sub:

.cfi_def_cfa_offset 88
.cfi_offset %rbp, -16

43–44

Can this be movl %r10d, %edi? The spill seems unnecessary.

[dberris]

• movl %r10d directly into %edi instead of spilling

[dberris]

Thanks Reid! Yeah, it's fun though hacking on low-level things like this (for me) so I take it as a learning opportunity. :)

[rnk]

This doesn't build on Windows:
http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/14021/steps/build%20stage%201/logs/stdio

I also don't think it's tested on Mac, so we should find a way to avoid building it. There should be some XRAY_SUPPORTED_OS cmake variable or something else we can use to avoid building it.

[rnk]

I tried to fix this in r276124, but you might want to make sure that does the right thing everywhere, I only tested that it locally doesn't build xray on Windows anymore.

[dberris]

Thanks Reid -- I was a little hesitant to lump XRay along with the other sanitizers, but I suspect it's easier to do that right now. I'll check to see whether it's possible to make this a little smarter based on the OS as well.

[rSerge]

It looks like there are 2 places where the second mprotect (returning the executable code block permissions to no-modify) can leak in compiler-rt/trunk/lib/xray/xray_interface.cc . Consider using RAII idiom. See my comments inline.

[rSerge]

compiler-rt/trunk/lib/xray/xray_interface.cc

126 ↗

(On Diff #64686)

mprotect() leaks due to this (so that the executable code block stays modifiable).

162 ↗

(On Diff #64686)

mprotect() leaks due to this (so that the executable code block stays modifiable).

[dberris]

Thanks rSerge, I've created D22757 which makes the changes.

[dberris]

This has been submitted already, BTW -- the changes in D22757 apply to trunk. Please review that instead.

[rSerge]

Is xray_trampoline_x86.S for x86_64 only or is it supposed to work for both x86 and x86_64? If the latter, then there seems to be a bug with always using RETQ, rather than RETL for x86.

[rSerge]

compiler-rt/trunk/lib/xray/xray_trampoline_x86.S

90 ↗

(On Diff #64686)

Shouldn't it be RETL for x86 and RETQ for x86_64 ? The original instruction in the compile-time sled could be RETL (then we should do RETL too) or RETQ (then we should do RETQ too), I think.

[rnk]

compiler-rt/trunk/lib/xray/xray_trampoline_x86.S

90 ↗

(On Diff #64686)

This is definitely x86_64 only code. Perhaps this file should be renamed xray_trampoline_x86_64.S

[rSerge]

compiler-rt/trunk/lib/xray/xray_trampoline_x86.S

90 ↗

(On Diff #64686)

Please, see llvm/trunk/lib/Target/X86/X86InstrInfo.cpp at https://reviews.llvm.org/D19904 : the instrumentation seems to be performed for both x86 and x86_64 (at least, both RETL and RETQ are marked for replacement), so I'm afraid that on x86 the code from the current file (which is x86_64 only) may try to execute.

[rnk]

compiler-rt/trunk/lib/xray/xray_trampoline_x86.S

90 ↗

(On Diff #64686)

Sure, but this whole assembly file won't assemble for 32-bit. It uses RSP, RDX, R10, pushq, etc, and those are 64-bit only. We'd need a separate .s file to really support 32-bit.

[rSerge]

compiler-rt/trunk/lib/xray/xray_trampoline_x86.S

90 ↗

(On Diff #64686)

Ok, understood, thanks.

[dberris]

compiler-rt/trunk/lib/xray/xray_trampoline_x86.S

90 ↗

(On Diff #64686)

Sure, but this whole assembly file won't assemble for 32-bit. It uses RSP, RDX, R10, pushq, etc, and those are 64-bit only. We'd need a separate .s file to really support 32-bit.

Yes, along with different patching routines (we assume certain instruction widths), and probably different sleds too.

I'll rename the file in D21982 as well to make it unambiguous that this is x86_64.

[eugenis]

compiler-rt/trunk/lib/xray/CMakeLists.txt

45 ↗

(On Diff #64686)

Remove this chunk, it can be added later with the tests commit.

compiler-rt/trunk/lib/xray/xray_init.cc

26 ↗

(On Diff #64686)

I think this extern is unnecessary

36 ↗

(On Diff #64686)

Is it unimplemented in this revision? /proc/xx/maps may be inaccessible under sandbox, would dl_iterate_phdr work for this?

compiler-rt/trunk/lib/xray/xray_interface.cc

65 ↗

(On Diff #64686)

Does it need to be atomic at all? It looks like you are already synchronized on XRayPatching.

67 ↗

(On Diff #64686)

Do you need to reset XRayPatching to false? Also in multiple places below.

74 ↗

(On Diff #64686)

out of curiosity, why std:: instead of just "size_t" ?

[dberris]

Thanks for the review Evgeniy -- I'll abandon this revision (so that no further reviews happen here, as this has already been submitted upstream) but I'll prepare another change that addresses some of the suggestions here.

[rSerge]

According to https://www.lri.fr/~filliatr/ens/compil/x86-64.pdf :

Floating arguments (up to 8) are passed in SSE registers %xmm0, %xmm1, ..., %xmm7. Additional arguments, if needed, are passed in stack slots. When calling a function that takes a variable number of arguments (notably printf) or lacks a prototype, byte register %al must be set before the call to indicate how many of the %xmm registers are used. A floating point return value is returned in %xmm0. All the %xmm registers are caller-save.

compiler-rt/trunk/lib/xray/xray_trampoline_x86.S

45 ↗

(On Diff #64686)

I think that if the tracing callback (stored in xray::XRayPatchedFunction ) performs floating-point operations, then the floating-point parameters passed to the instrumented function in SSE registers (%xmm0 to %xmm7) can get corrupted so that the instrumented function gets garbage on input. Shouldn't xray_FunctionEntry push %xmm0...%xmm7 to the stack upon entering and pop them upon exit?

83 ↗

(On Diff #64686)

The floating-point return value of the instrumented function may be in %xmm0 SSE register. So if xray::XRayPatchedFunction (or its callees) performs floating-point operation, it may clobber the register thus the instrumented function would return garbage. Shouldn't xray_FunctionExit push %xmm0 to the stack on entry and pop it on exit?

[dberris]

Yes Serge, they will get clobbered. We can change this to save the XMM registers too quite easily, but in a different patch. Let me put that in the list of things to do for me. :)

[rSerge]

Because "jmp +9" and "ret" instructions (as I understood) are not 2-byte aligned, I doublt atomicity of replacing them with 2-byte instructions of the runtime patches. Could you take a look?

compiler-rt/trunk/lib/xray/xray_interface.cc

131–133 ↗

(On Diff #64686)

Can it happen that the 2 machine code bytes of "jmp +9" instruction belong to different 8-byte blocks of memory? I think that in this case the operation will not be atomic: the CPU will have to write different 8-byte blocks of memory in different cycles. Please, consider aligning "jmp +9" instruction at least at a multiple of 2 bytes.

167–169 ↗

(On Diff #64686)

Can it happen that the 2 machine code bytes of "mov r10d, <function id>" instruction belong to different 8-byte blocks of memory? I think that in this case the operation will not be atomic: the CPU will have to write different 8-byte blocks of memory in different cycles. Please, consider aligning "RET" instruction at least at a multiple of 2 bytes. Otherwise the CPU executing the function on a different core may get the first byte of "mov r10d, <function id>" and the second byte (and consequtive bytes) of whatever was earlier behing the "RET" instruction.

[dberris]

Thanks Serge -- would you consider perhaps not sending comments onto this patch anymore? We can continue more of this on either one of the dependent changes or on the llvm-dev mailing list. I'd very much appreciate that.

PS. These sleds were implemented in D19904, you can see the changes in lowering at http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86MCInstLower.cpp?r1=275272&r2=275367&pathrev=275367&diff_format=h.

compiler-rt/trunk/lib/xray/xray_interface.cc

131–133 ↗

(On Diff #64686)

We already align the function start sled to 2-byte alignment.

167–169 ↗

(On Diff #64686)

This we don't do yet. It should be easy to do. I'll send a patch to address exactly this concern.

CMakeLists.txt

list(INSERT CMAKE_MODULE_PATH 0
   )
 
 include(base-config-ix)
 
 option(COMPILER_RT_BUILD_BUILTINS "Build builtins" ON)
 mark_as_advanced(COMPILER_RT_BUILD_BUILTINS)
 option(COMPILER_RT_BUILD_SANITIZERS "Build sanitizers" ON)
 mark_as_advanced(COMPILER_RT_BUILD_SANITIZERS)
+option(COMPILER_RT_BUILD_XRAY "Build xray" ON)
+mark_as_advanced(COMPILER_RT_BUILD_XRAY)
 
 if (COMPILER_RT_STANDALONE_BUILD)
   if (NOT LLVM_CONFIG_PATH)
     find_program(LLVM_CONFIG_PATH "llvm-config"
                  DOC "Path to llvm-config binary")
     if (NOT LLVM_CONFIG_PATH)
       message(FATAL_ERROR "llvm-config not found: specify LLVM_CONFIG_PATH")
     endif()

cmake/config-ix.cmake

set(ALL_PROFILE_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64} ${PPC64}
     ${MIPS32} ${MIPS64})
 set(ALL_TSAN_SUPPORTED_ARCH ${X86_64} ${MIPS64} ${ARM64} ${PPC64})
 set(ALL_UBSAN_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64}
     ${MIPS32} ${MIPS64} ${PPC64} ${S390X})
 set(ALL_SAFESTACK_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM64} ${MIPS32} ${MIPS64})
 set(ALL_CFI_SUPPORTED_ARCH ${X86} ${X86_64} ${MIPS64})
 set(ALL_ESAN_SUPPORTED_ARCH ${X86_64})
 set(ALL_SCUDO_SUPPORTED_ARCH ${X86_64})
+set(ALL_XRAY_SUPPORTED_ARCH ${X86_64})
 
 if(APPLE)
   include(CompilerRTDarwinUtils)
 
   find_darwin_sdk_dir(DARWIN_osx_SYSROOT macosx)
   find_darwin_sdk_dir(DARWIN_iossim_SYSROOT iphonesimulator)
   find_darwin_sdk_dir(DARWIN_ios_SYSROOT iphoneos)
   find_darwin_sdk_dir(DARWIN_watchossim_SYSROOT watchsimulator)
list_intersect(CFI_SUPPORTED_ARCH
     ALL_CFI_SUPPORTED_ARCH
     SANITIZER_COMMON_SUPPORTED_ARCH)
   list_intersect(ESAN_SUPPORTED_ARCH
     ALL_ESAN_SUPPORTED_ARCH
     SANITIZER_COMMON_SUPPORTED_ARCH)
   list_intersect(SCUDO_SUPPORTED_ARCH
     ALL_SCUDO_SUPPORTED_ARCH
     SANITIZER_COMMON_SUPPORTED_ARCH)
+  list_intersect(XRAY_SUPPORTED_ARCH
+    ALL_XRAY_SUPPORTED_ARCH
+    SANITIZER_COMMON_SUPPORTED_ARCH)
 else()
   # Architectures supported by compiler-rt libraries.
   filter_available_targets(SANITIZER_COMMON_SUPPORTED_ARCH
     ${ALL_SANITIZER_COMMON_SUPPORTED_ARCH})
   # LSan and UBSan common files should be available on all architectures
   # supported by other sanitizers (even if they build into dummy object files).
   filter_available_targets(LSAN_COMMON_SUPPORTED_ARCH
     ${SANITIZER_COMMON_SUPPORTED_ARCH})
   filter_available_targets(UBSAN_COMMON_SUPPORTED_ARCH
     ${SANITIZER_COMMON_SUPPORTED_ARCH})
   filter_available_targets(ASAN_SUPPORTED_ARCH ${ALL_ASAN_SUPPORTED_ARCH})
   filter_available_targets(DFSAN_SUPPORTED_ARCH ${ALL_DFSAN_SUPPORTED_ARCH})
   filter_available_targets(LSAN_SUPPORTED_ARCH ${ALL_LSAN_SUPPORTED_ARCH})
   filter_available_targets(MSAN_SUPPORTED_ARCH ${ALL_MSAN_SUPPORTED_ARCH})
   filter_available_targets(PROFILE_SUPPORTED_ARCH ${ALL_PROFILE_SUPPORTED_ARCH})
   filter_available_targets(TSAN_SUPPORTED_ARCH ${ALL_TSAN_SUPPORTED_ARCH})
   filter_available_targets(UBSAN_SUPPORTED_ARCH ${ALL_UBSAN_SUPPORTED_ARCH})
   filter_available_targets(SAFESTACK_SUPPORTED_ARCH
     ${ALL_SAFESTACK_SUPPORTED_ARCH})
   filter_available_targets(CFI_SUPPORTED_ARCH ${ALL_CFI_SUPPORTED_ARCH})
   filter_available_targets(ESAN_SUPPORTED_ARCH ${ALL_ESAN_SUPPORTED_ARCH})
   filter_available_targets(SCUDO_SUPPORTED_ARCH
     ${ALL_SCUDO_SUPPORTED_ARCH})
+  filter_available_targets(XRAY_SUPPORTED_ARCH ${ALL_XRAY_SUPPORTED_ARCH})
 endif()
 
 if (MSVC)
   # See if the DIA SDK is available and usable.
   set(MSVC_DIA_SDK_DIR "$ENV{VSINSTALLDIR}DIA SDK")
   if (IS_DIRECTORY ${MSVC_DIA_SDK_DIR})
     set(CAN_SYMBOLIZE 1)
   else()

include/CMakeLists.txt

 set(SANITIZER_HEADERS
   sanitizer/allocator_interface.h
   sanitizer/asan_interface.h
   sanitizer/common_interface_defs.h
   sanitizer/coverage_interface.h
   sanitizer/dfsan_interface.h
   sanitizer/esan_interface.h
   sanitizer/linux_syscall_hooks.h
   sanitizer/lsan_interface.h
   sanitizer/msan_interface.h
   sanitizer/tsan_interface_atomic.h)
 
+set(XRAY_HEADERS
+  xray/xray_interface.h)
+
+set(COMPILER_RT_HEADERS
+  ${SANITIZER_HEADERS}
+  ${XRAY_HEADERS})
+
 set(output_dir ${COMPILER_RT_OUTPUT_DIR}/include)
 
 # Copy compiler-rt headers to the build tree.
 set(out_files)
-foreach( f ${SANITIZER_HEADERS} )
+foreach( f ${COMPILER_RT_HEADERS} )
   set( src ${CMAKE_CURRENT_SOURCE_DIR}/${f} )
   set( dst ${output_dir}/${f} )
   add_custom_command(OUTPUT ${dst}
     DEPENDS ${src}
     COMMAND ${CMAKE_COMMAND} -E copy_if_different ${src} ${dst}
     COMMENT "Copying compiler-rt's ${f}...")
   list(APPEND out_files ${dst})
 endforeach( f )
 
 add_custom_target(compiler-rt-headers ALL DEPENDS ${out_files})
 add_dependencies(compiler-rt compiler-rt-headers)
 set_target_properties(compiler-rt-headers PROPERTIES FOLDER "Compiler-RT Misc")
 
 # Install sanitizer headers.
 install(FILES ${SANITIZER_HEADERS}
   PERMISSIONS OWNER_READ OWNER_WRITE GROUP_READ WORLD_READ
   DESTINATION ${COMPILER_RT_INSTALL_PATH}/include/sanitizer)
+# Install xray headers.
+install(FILES ${XRAY_HEADERS}
+  PERMISSIONS OWNER_READ OWNER_WRITE GROUP_READ WORLD_READ
+  DESTINATION ${COMPILER_RT_INSTALL_PATH}/include/xray)

include/xray/xray_interface.h

+//===-- xray_interface.h ----------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+// APIs for controlling XRay functionality explicitly.
+//===----------------------------------------------------------------------===//
+#ifndef XRAY_XRAY_INTERFACE_H
+#define XRAY_XRAY_INTERFACE_H
+
+#include <cstdint>
+
+extern "C" {
+
+enum XRayEntryType { ENTRY = 0, EXIT = 1 };
+
+// Provide a function to invoke for when instrumentation points are hit. This is
+// a user-visible control surface that overrides the default implementation. The
+// function provided should take the following arguments:
+//
+//   - function id: an identifier that indicates the id of a function; this id
+//                  is generated by xray; the mapping between the function id
+//                  and the actual function pointer is available through
+//                  __xray_table.
+//   - entry type: identifies what kind of instrumentation point was encountered
+//                 (function entry, function exit, etc.). See the enum
+//                 XRayEntryType for more details.
+//
+// Returns 1 on success, 0 on error.
+extern int __xray_set_handler(void (*entry)(int32_t, XRayEntryType));
+
+// This removes whatever the currently provided handler is. Returns 1 on
+// success, 0 on error.
+extern int __xray_remove_handler();
+
+enum XRayPatchingStatus {
+  NOT_INITIALIZED = 0,
+  NOTIFIED = 1,
+  ONGOING = 2,
+  FAILED = 3
+};
+
+// This tells XRay to patch the instrumentation points. This is an asynchronous
+// process, and returns the following status in specific cases:
+//
+//   - 0 : XRay is not initialized.
+//   - 1 : We've done the notification.
+//   - 2 : Patching / un-patching is on-going.
+extern XRayPatchingStatus __xray_patch();
+
+// Reverses the effect of __xray_patch(). This is an asynchronous process, and
+// returns the following status in specific cases.
+//
+//   - 0 : XRay is not initialized.
+//   - 1 : We've done the notification.
+//   - 2 : Patching / un-patching is on-going.
+extern int __xray_unpatch();
+}
+
+#endif

lib/CMakeLists.txt

if(COMPILER_RT_BUILD_SANITIZERS)
   if(COMPILER_RT_HAS_ESAN)
     add_subdirectory(esan)
   endif()
 
   if(COMPILER_RT_HAS_SCUDO)
     add_subdirectory(scudo)
   endif()
 endif()
+
+if(COMPILER_RT_BUILD_XRAY)
+  add_subdirectory(xray)
+endif()

lib/xray/CMakeLists.txt

+# Build for the XRay runtime support library.
+
+set(XRAY_SOURCES
+  xray_init.cc
+  xray_interface.cc
+  xray_flags.cc
+)
+
+set(x86_64_SOURCES
+    xray_trampoline_x86.S
+    ${XRAY_SOURCES})
+
+include_directories(..)
+include_directories(../../include)
+
+set(XRAY_CFLAGS ${SANITIZER_COMMON_CFLAGS})
+
+set(XRAY_COMMON_DEFINITIONS XRAY_HAS_EXCEPTIONS=1)
+
+add_compiler_rt_object_libraries(RTXray
+    ARCHS ${XRAY_SUPPORTED_ARCH}
+    SOURCES ${XRAY_SOURCES} CFLAGS ${XRAY_CFLAGS}
+    DEFS ${XRAY_COMMON_DEFINITIONS})
+
+add_custom_target(xray)
+set(XRAY_COMMON_RUNTIME_OBJECT_LIBS
+    RTSanitizerCommon
+    RTSanitizerCommonLibc)
+
+foreach (arch ${XRAY_SUPPORTED_ARCH})
+    if (CAN_TARGET_${arch})
+        add_compiler_rt_runtime(clang_rt.xray
+            STATIC
+            ARCHS ${arch}
+            SOURCES ${${arch}_SOURCES}
+            CFLAGS ${XRAY_CFLAGS}
+            DEFS ${XRAY_COMMON_DEFINITIONS}
+            OBJECT_LIBS ${XRAY_COMMON_RUNTIME_OBJECT_LIBS}
+            PARENT_TARGET xray)
+    endif ()
+endforeach()
+
+add_dependencies(compiler-rt xray)
+
+# if(COMPILER_RT_INCLUDE_TESTS)
+#   add_subdirectory(tests)
+# endif()

lib/xray/xray_flags.h

+//===-- xray_flags.h -------------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instruementation system.
+//
+// XRay runtime flags.
+//===----------------------------------------------------------------------===//
+
+#ifndef XRAY_FLAGS_H
+#define XRAY_FLAGS_H
+
+#include "sanitizer_common/sanitizer_flag_parser.h"
+
+namespace __xray {
+
+struct Flags {
+#define XRAY_FLAG(Type, Name, DefaultValue, Description) Type Name;
+#include "xray_flags.inc"
+#undef XRAY_FLAG
+
+  void SetDefaults();
+};
+
+extern Flags xray_flags_dont_use_directly;
+inline Flags *flags() { return &xray_flags_dont_use_directly; }
+
+void InitializeFlags();
+
+} // namespace __xray
+
+#endif // XRAY_FLAGS_H

lib/xray/xray_flags.cc

+//===-- xray_flags.cc -------------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+// XRay flag parsing logic.
+//===----------------------------------------------------------------------===//
+
+#include "xray_flags.h"
+#include "sanitizer_common/sanitizer_common.h"
+#include "sanitizer_common/sanitizer_flag_parser.h"
+#include "sanitizer_common/sanitizer_libc.h"
+
+using namespace __sanitizer;
+
+namespace __xray {
+
+Flags xray_flags_dont_use_directly; // use via flags().
+
+void Flags::SetDefaults() {
+#define XRAY_FLAG(Type, Name, DefaultValue, Description) Name = DefaultValue;
+#include "xray_flags.inc"
+#undef XRAY_FLAG
+}
+
+static void RegisterXRayFlags(FlagParser *P, Flags *F) {
+#define XRAY_FLAG(Type, Name, DefaultValue, Description)                       \
+  RegisterFlag(P, #Name, Description, &F->Name);
+#include "xray_flags.inc"
+#undef XRAY_FLAG
+}
+
+void InitializeFlags() {
+  SetCommonFlagsDefaults();
+  auto *F = flags();
+  F->SetDefaults();
+
+  FlagParser XRayParser;
+  RegisterXRayFlags(&XRayParser, F);
+  RegisterCommonFlags(&XRayParser);
+
+  // Override from command line.
+  XRayParser.ParseString(GetEnv("XRAY_OPTIONS"));
+
+  InitializeCommonFlags();
+
+  if (Verbosity())
+    ReportUnrecognizedFlags();
+
+  if (common_flags()->help) {
+    XRayParser.PrintFlagDescriptions();
+  }
+}
+
+} // namespace __xray

lib/xray/xray_flags.inc

+//===-- xray_flags.inc ------------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// XRay runtime flags.
+//
+//===----------------------------------------------------------------------===//
+#ifndef XRAY_FLAG
+#error "Define XRAY_FLAG prior to including this file!"
+#endif
+
+XRAY_FLAG(bool, patch_premain, true,
+          "Whether to patch instrumentation points before main.")

lib/xray/xray_init.cc

+//===-- xray_init.cc --------------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+// XRay initialisation logic.
+//===----------------------------------------------------------------------===//
+
+#include <atomic>
+#include <fcntl.h>
+#include <strings.h>
+#include <unistd.h>
+
+#include "sanitizer_common/sanitizer_common.h"
+#include "xray_flags.h"
+#include "xray_interface_internal.h"
+#include "llvm/Support/ELF.h"
+
+extern "C" {
+extern void __xray_init();
+extern const XRaySledEntry __start_xray_instr_map[] __attribute__((weak));
+extern const XRaySledEntry __stop_xray_instr_map[] __attribute__((weak));
+}
+
+using namespace __xray;
+
+// We initialize some global variables that pertain to specific sections of XRay
+// data structures in the binary. We do this for the current process using
+// /proc/curproc/map and make sure that we're able to get it. We signal failure
+// via a global atomic boolean to indicate whether we've initialized properly.
+//
+std::atomic<bool> XRayInitialized{false};
+
+// This should always be updated before XRayInitialized is updated.
+std::atomic<__xray::XRaySledMap> XRayInstrMap{};
+
+// __xray_init() will do the actual loading of the current process' memory map
+// and then proceed to look for the .xray_instr_map section/segment.
+void __xray_init() {
+  InitializeFlags();
+  if (__start_xray_instr_map == nullptr) {
+    Report("XRay instrumentation map missing. Not initializing XRay.\n");
+    return;
+  }
+
+  // Now initialize the XRayInstrMap global struct with the address of the
+  // entries, reinterpreted as an array of XRaySledEntry objects. We use the
+  // virtual pointer we have from the section to provide us the correct
+  // information.
+  __xray::XRaySledMap SledMap{};
+  SledMap.Sleds = __start_xray_instr_map;
+  SledMap.Entries = __stop_xray_instr_map - __start_xray_instr_map;
+  XRayInstrMap.store(SledMap, std::memory_order_release);
+  XRayInitialized.store(true, std::memory_order_release);
+
+  // FIXME: Check the flag/environment before patching.
+  if (flags()->patch_premain)
+    __xray_patch();
+}
+
+__attribute__((section(".preinit_array"),
+               used)) void (*__local_xray_preinit)(void) = __xray_init;

lib/xray/xray_interface.cc

+//===-- xray_interface.cpp --------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+// Implementation of the API functions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "xray_interface_internal.h"
+#include <atomic>
+#include <cstdint>
+#include <cstdio>
+#include <errno.h>
+#include <limits>
+#include <sys/mman.h>
+
+namespace __xray {
+
+// This is the function to call when we encounter the entry or exit sleds.
+std::atomic<void (*)(int32_t, XRayEntryType)> XRayPatchedFunction{nullptr};
+
+} // namespace __xray
+
+extern "C" {
+// The following functions have to be defined in assembler, on a per-platform
+// basis. See xray_trampoline_*.s files for implementations.
+extern void __xray_FunctionEntry();
+extern void __xray_FunctionExit();
+}
+
+extern std::atomic<bool> XRayInitialized;
+extern std::atomic<__xray::XRaySledMap> XRayInstrMap;
+
+int __xray_set_handler(void (*entry)(int32_t, XRayEntryType)) {
+  if (XRayInitialized.load(std::memory_order_acquire)) {
+    __xray::XRayPatchedFunction.store(entry, std::memory_order_release);
+    return 1;
+  }
+  return 0;
+}
+
+std::atomic<bool> XRayPatching{false};
+
+XRayPatchingStatus __xray_patch() {
+  // FIXME: Make this happen asynchronously. For now just do this sequentially.
+  if (!XRayInitialized.load(std::memory_order_acquire))
+    return XRayPatchingStatus::NOT_INITIALIZED; // Not initialized.
+
+  static bool NotPatching = false;
+  if (!XRayPatching.compare_exchange_strong(NotPatching, true,
+                                            std::memory_order_acq_rel,
+                                            std::memory_order_acquire)) {
+    return XRayPatchingStatus::ONGOING; // Already patching.
+  }
+
+  // Step 1: Compute the function id, as a unique identifier per function in the
+  // instrumentation map.
+  __xray::XRaySledMap InstrMap = XRayInstrMap.load(std::memory_order_acquire);
+  if (InstrMap.Entries == 0)
+    return XRayPatchingStatus::NOT_INITIALIZED;
+
+  int32_t FuncId = 1;
+  static constexpr uint8_t CallOpCode = 0xe8;
+  static constexpr uint16_t MovR10Seq = 0xba41;
+  static constexpr uint8_t JmpOpCode = 0xe9;
+  uint64_t CurFun = 0;
+  for (std::size_t I = 0; I < InstrMap.Entries; I++) {
+    auto Sled = InstrMap.Sleds[I];
+    auto F = Sled.Function;
+    if (CurFun == 0)
+      CurFun = F;
+    if (F != CurFun) {
+      ++FuncId;
+      CurFun = F;
+    }
+
+    // While we're here, we should patch the nop sled. To do that we mprotect
+    // the page containing the function to be writeable.
+    void *PageAlignedAddr =
+        reinterpret_cast<void *>(Sled.Address & ~((2 << 16) - 1));
+    std::size_t MProtectLen =
+        (Sled.Address + 12) - reinterpret_cast<uint64_t>(PageAlignedAddr);
+    if (mprotect(PageAlignedAddr, MProtectLen,
+                 PROT_READ | PROT_WRITE | PROT_EXEC) == -1) {
+      printf("Failed mprotect: %d\n", errno);
+      return XRayPatchingStatus::FAILED;
+    }
+
+    static constexpr int64_t MinOffset{std::numeric_limits<int32_t>::min()};
+    static constexpr int64_t MaxOffset{std::numeric_limits<int32_t>::max()};
+    if (Sled.Kind == XRayEntryType::ENTRY) {
+      // Here we do the dance of replacing the following sled:
+      //
+      // xray_sled_n:
+      //   jmp +9
+      //   <9 byte nop>
+      //
+      // With the following:
+      //
+      //   mov r10d, <function id>
+      //   call <relative 32bit offset to entry trampoline>
+      //
+      // We need to do this in the following order:
+      //
+      // 1. Put the function id first, 2 bytes from the start of the sled (just
+      // after the 2-byte jmp instruction).
+      // 2. Put the call opcode 6 bytes from the start of the sled.
+      // 3. Put the relative offset 7 bytes from the start of the sled.
+      // 4. Do an atomic write over the jmp instruction for the "mov r10d"
+      // opcode and first operand.
+      //
+      // Prerequisite is to compute the relative offset to the
+      // __xray_FunctionEntry function's address.
+      int64_t TrampolineOffset =
+          reinterpret_cast<int64_t>(__xray_FunctionEntry) -
+          (static_cast<int64_t>(Sled.Address) + 11);
+      if (TrampolineOffset < MinOffset || TrampolineOffset > MaxOffset) {
+        // FIXME: Print out an error here.
+        continue;
+      }
+      *reinterpret_cast<uint32_t *>(Sled.Address + 2) = FuncId;
+      *reinterpret_cast<uint8_t *>(Sled.Address + 6) = CallOpCode;
+      *reinterpret_cast<uint32_t *>(Sled.Address + 7) = TrampolineOffset;
+      std::atomic_store_explicit(
+          reinterpret_cast<std::atomic<uint16_t> *>(Sled.Address), MovR10Seq,
+          std::memory_order_release);
+    }
+
+    if (Sled.Kind == XRayEntryType::EXIT) {
+      // Here we do the dance of replacing the following sled:
+      //
+      // xray_sled_n:
+      //   ret
+      //   <10 byte nop>
+      //
+      // With the following:
+      //
+      //   mov r10d, <function id>
+      //   jmp <relative 32bit offset to exit trampoline>
+      //
+      // 1. Put the function id first, 2 bytes from the start of the sled (just
+      // after the 1-byte ret instruction).
+      // 2. Put the jmp opcode 6 bytes from the start of the sled.
+      // 3. Put the relative offset 7 bytes from the start of the sled.
+      // 4. Do an atomic write over the jmp instruction for the "mov r10d"
+      // opcode and first operand.
+      //
+      // Prerequisite is to compute the relative offset fo the
+      // __xray_FunctionExit function's address.
+      int64_t TrampolineOffset =
+          reinterpret_cast<int64_t>(__xray_FunctionExit) -
+          (static_cast<int64_t>(Sled.Address) + 11);
+      if (TrampolineOffset < MinOffset || TrampolineOffset > MaxOffset) {
+        // FIXME: Print out an error here.
+        continue;
+      }
+      *reinterpret_cast<uint32_t *>(Sled.Address + 2) = FuncId;
+      *reinterpret_cast<uint8_t *>(Sled.Address + 6) = JmpOpCode;
+      *reinterpret_cast<uint32_t *>(Sled.Address + 7) = TrampolineOffset;
+      std::atomic_store_explicit(
+          reinterpret_cast<std::atomic<uint16_t> *>(Sled.Address), MovR10Seq,
+          std::memory_order_release);
+    }
+
+    if (mprotect(PageAlignedAddr, MProtectLen, PROT_READ | PROT_EXEC) == -1) {
+      printf("Failed mprotect: %d\n", errno);
+      return XRayPatchingStatus::FAILED;
+    }
+  }
+  XRayPatching.store(false, std::memory_order_release);
+  return XRayPatchingStatus::NOTIFIED;
+}

lib/xray/xray_interface_internal.h

+//===-- xray_interface_internal.h -------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+// Implementation of the API functions. See also include/xray/xray_interface.h.
+//
+//===----------------------------------------------------------------------===//
+#ifndef XRAY_INTERFACE_INTERNAL_H
+#define XRAY_INTERFACE_INTERNAL_H
+
+#include "xray/xray_interface.h"
+#include <cstdint>
+
+extern "C" {
+
+struct XRaySledEntry {
+  uint64_t Address;
+  uint64_t Function;
+  unsigned char Kind;
+  unsigned char AlwaysInstrument;
+  unsigned char Padding[14]; // Need 32 bytes
+};
+}
+
+namespace __xray {
+
+struct XRaySledMap {
+  const XRaySledEntry *Sleds;
+  std::size_t Entries;
+};
+}
+
+#endif

lib/xray/xray_trampoline_x86.S

+//===-- xray_trampoline_x86.s -----------------------------------*- ASM -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+// This implements the X86-specific assembler for the trampolines.
+//
+//===----------------------------------------------------------------------===//
+
+  .text
+  .file "xray_trampoline_x86.S"
+  .globl __xray_FunctionEntry
+  .align 16, 0x90
+  .type __xray_FunctionEntry,@function
+
+__xray_FunctionEntry:
+  .cfi_startproc
+  // Save caller provided registers before doing any actual work.
+  pushq %rbp
+  .cfi_def_cfa_offset 16
+  subq $72, %rsp
+  movq  %rdi, 64(%rsp)
+  movq  %rax, 56(%rsp)
+  movq  %rdx, 48(%rsp)
+  movq  %rsi, 40(%rsp)
+  movq  %rcx, 32(%rsp)
+  movq  %r8, 24(%rsp)
+  movq  %r9, 16(%rsp)
+
+  // de-mangled, that's __xray::XRayPatchedFunction, and we're doing an acquire
+  // load (on x86 is a normal mov instruction).
+  movq  _ZN6__xray19XRayPatchedFunctionE(%rip), %rax
+  testq %rax, %rax
+  je  .Ltmp0
+
+  // assume that %r10d has the function id.
+  movl  %r10d, 4(%rsp)
+  movl  4(%rsp), %edi
+  xor %esi,%esi
+  callq *%rax
+.Ltmp0:
+  // restore the registers
+  movq  64(%rsp), %rdi
+  movq  56(%rsp), %rax
+  movq  48(%rsp), %rdx
+  movq  40(%rsp), %rsi
+  movq  32(%rsp), %rcx
+  movq  24(%rsp), %r8
+  movq  16(%rsp), %r9
+  addq  $72, %rsp
+  popq  %rbp
+  retq
+.Ltmp1:
+  .size __xray_FunctionEntry, .Ltmp1-__xray_FunctionEntry
+  .cfi_endproc
+
+  .globl __xray_FunctionExit
+  .align 16, 0x90
+  .type __xray_FunctionExit,@function
+__xray_FunctionExit:
+  .cfi_startproc
+  // Save the important registers first. Since we're assuming that this
+  // function is only jumped into, we only preserve the registers for
+  // returning.
+  // FIXME: Figure out whether this is sufficient.
+  pushq %rbp
+  .cfi_def_cfa_offset 16
+  subq  $24, %rsp
+  .cfi_def_cfa_offset 32
+  movq  %rax, 16(%rsp)
+  movq  %rdx, 8(%rsp)
+  movq  _ZN6__xray19XRayPatchedFunctionE(%rip), %rax
+  testq %rax,%rax
+  je  .Ltmp2
+
+  movl  %r10d, 4(%rsp)
+  movl  4(%rsp), %edi
+  movl  $1, %esi
+  callq *%rax
+.Ltmp2:
+  // Restore the important registers.
+  movq  16(%rsp), %rax
+  movq  8(%rsp), %rdx
+  addq  $24, %rsp
+  popq  %rbp
+  retq
+.Ltmp3:
+  .size __xray_FunctionExit, .Ltmp3-__xray_FunctionExit
+  .cfi_endproc