[XRay][compiler-rt] Support TSC emulation even for x86_64

[dberris]

Summary

Use TSC emulation in cases where RDTSCP isn't available on the host
running an XRay instrumented binary. We can then fall back into
emulation instead of not even installing XRay's runtime functionality.
We only do this for now in the naive/basic logging implementation, but
should be useful in even FDR mode.

Should fix http://llvm.org/PR32148.

[pelikan]

lib/xray/xray_inmemory_log.cc

141–150 ↗

(On Diff #90790)

This lambda is already in xray_tsc.h as ALWAYS_INLINE. I remember the discussion about how important it is to have it inlined, and here it's a copy+paste lambda with unclear inliner results. Why can't we just use the readTSC function pointer here?

In that case we'll have to rename it to something like emulateTSC and move out of the #elif, as (in this case) readTSC is a real thing reading the real TSC. And then even the other architectures' code would look clearer, because they *emulate* TSC, not *read* it. Intel and Power would call "read" and others would call "emulate".

[dberris]

lib/xray/xray_inmemory_log.cc

141–150 ↗

(On Diff #90790)

Considered it, and came to the conclusion that the alternative is a much more invasive change. I also don't want to preclude the implementation currently used in arm and mips from being able to use platform-specific features if/when we find out how to do them later (thereby imposing the burden on refactoring that to whoever is making that change later). The repetition here is fine and correct for x86_64 only. I'm attempting to localise the change here with a potentially more invasive refactoring to follow later.

The inlining question is easy to answer here -- __xray::readTSC is always inlined, so no function pointers (and indirect calls) were gotten in the making of this change. The template ensures that since all the definitions are inline and visible, there' no reason for the compiler to ignore those and not inline. It's the same rationale the standard library uses function templates for stuff that relies on inlining for performance purposes.

[bcain]

I tested this patch. It was effective in addressing this bug for me. I have an ivy bridge i7 CPU, SR0PK (E1).

[dberris]

@pelikan -- can we do the suggested refactoring in a subsequent change?

[pelikan]

OK then. Let's proceed and fix it up later.

compiler-rt/trunk/lib/xray/xray_inmemory_log.cc

 };
 
 } // namespace __xray
 
 using namespace __xray;
 
 static int __xray_OpenLogFile() XRAY_NEVER_INSTRUMENT {
   int F = getLogFD();
-  auto TSCFrequency = getTSCFrequency();
   if (F == -1)
     return -1;
   // Since we're here, we get to write the header. We set it up so that the
   // header will only be written once, at the start, and let the threads
   // logging do writes which just append.
   XRayFileHeader Header;
   Header.Version = 1;
   Header.Type = FileTypes::NAIVE_LOG;
-  Header.CycleFrequency = TSCFrequency;
+  Header.CycleFrequency = probeRequiredCPUFeatures()
+                              ? getTSCFrequency()
+                              : __xray::NanosecondsPerSecond;
 
   // FIXME: Actually check whether we have 'constant_tsc' and 'nonstop_tsc'
   // before setting the values in the header.
   Header.ConstantTSC = 1;
   Header.NonstopTSC = 1;
   retryingWriteAll(F, reinterpret_cast<char *>(&Header),
                    reinterpret_cast<char *>(&Header) + sizeof(Header));
   return F;
 }
 
-void __xray_InMemoryRawLog(int32_t FuncId,
-                           XRayEntryType Type) XRAY_NEVER_INSTRUMENT {
+template <class RDTSC>
+void __xray_InMemoryRawLog(int32_t FuncId, XRayEntryType Type,
+                           RDTSC ReadTSC) XRAY_NEVER_INSTRUMENT {
   using Buffer =
       std::aligned_storage<sizeof(XRayRecord), alignof(XRayRecord)>::type;
   static constexpr size_t BuffLen = 1024;
   thread_local static Buffer InMemoryBuffer[BuffLen] = {};
   thread_local static size_t Offset = 0;
   static int Fd = __xray_OpenLogFile();
   if (Fd == -1)
     return;
   thread_local __xray::ThreadExitFlusher Flusher(
       Fd, reinterpret_cast<__xray::XRayRecord *>(InMemoryBuffer), Offset);
   thread_local pid_t TId = syscall(SYS_gettid);
 
   // First we get the useful data, and stuff it into the already aligned buffer
   // through a pointer offset.
   auto &R = reinterpret_cast<__xray::XRayRecord *>(InMemoryBuffer)[Offset];
   R.RecordType = RecordTypes::NORMAL;
-  R.TSC = __xray::readTSC(R.CPU);
+  R.TSC = ReadTSC(R.CPU);
   R.TId = TId;
   R.Type = Type;
   R.FuncId = FuncId;
   ++Offset;
   if (Offset == BuffLen) {
     std::lock_guard<std::mutex> L(LogMutex);
     auto RecordBuffer = reinterpret_cast<__xray::XRayRecord *>(InMemoryBuffer);
     retryingWriteAll(Fd, reinterpret_cast<char *>(RecordBuffer),
                      reinterpret_cast<char *>(RecordBuffer + Offset));
     Offset = 0;
   }
 }
 
-static auto UNUSED Unused = [] {
-  if (!probeRequiredCPUFeatures()) {
-    Report("Required CPU features missing for XRay instrumentation, not "
-           "installing instrumentation hooks.\n");
-    return false;
-  }
+void __xray_InMemoryRawLogRealTSC(int32_t FuncId,
+                                  XRayEntryType Type) XRAY_NEVER_INSTRUMENT {
+  __xray_InMemoryRawLog(FuncId, Type, __xray::readTSC);
+
+void __xray_InMemoryEmulateTSC(int32_t FuncId,
+                               XRayEntryType Type) XRAY_NEVER_INSTRUMENT {
+  __xray_InMemoryRawLog(FuncId, Type, [](uint8_t &CPU) XRAY_NEVER_INSTRUMENT {
+    timespec TS;
+    int result = clock_gettime(CLOCK_REALTIME, &TS);
+    if (result != 0) {
+      Report("clock_gettimg(2) return %d, errno=%d.", result, int(errno));
+      TS = {0, 0};
+    }
+    CPU = 0;
+    return TS.tv_sec * __xray::NanosecondsPerSecond + TS.tv_nsec;
+  });
+}
+
+static auto UNUSED Unused = [] {
+  auto UseRealTSC = probeRequiredCPUFeatures();
+  if (!UseRealTSC)
+    Report("WARNING: Required CPU features missing for XRay instrumentation, "
+           "using emulation instead.\n");
   if (flags()->xray_naive_log)
-    __xray_set_handler(__xray_InMemoryRawLog);
+    __xray_set_handler(UseRealTSC ? __xray_InMemoryRawLogRealTSC
+                                  : __xray_InMemoryEmulateTSC);
   return true;
 }();

compiler-rt/trunk/lib/xray/xray_tsc.h

 //===-- xray_tsc.h ----------------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // This file is a part of XRay, a dynamic runtime instrumentation system.
 //
 //===----------------------------------------------------------------------===//
 #ifndef XRAY_EMULATE_TSC_H
 #define XRAY_EMULATE_TSC_H
 
+namespace __xray {
+static constexpr uint64_t NanosecondsPerSecond = 1000ULL * 1000 * 1000;
+}
+
 #if defined(__x86_64__)
 #include "xray_x86_64.inc"
 #elif defined(__powerpc64__)
 #include "xray_powerpc64.inc"
 #elif defined(__arm__) || defined(__aarch64__) || defined(__mips__)
 // Emulated TSC.
 // There is no instruction like RDTSCP in user mode on ARM. ARM's CP15 does
 //   not have a constant frequency like TSC on x86(_64), it may go faster
 //   or slower depending on CPU turbo or power saving mode. Furthermore,
 //   to read from CP15 on ARM a kernel modification or a driver is needed.
 //   We can not require this from users of compiler-rt.
 // So on ARM we use clock_gettime() which gives the result in nanoseconds.
 //   To get the measurements per second, we scale this by the number of
 //   nanoseconds per second, pretending that the TSC frequency is 1GHz and
 //   one TSC tick is 1 nanosecond.
 #include "sanitizer_common/sanitizer_common.h"
 #include "sanitizer_common/sanitizer_internal_defs.h"
 #include "xray_defs.h"
 #include <cerrno>
 #include <cstdint>
 #include <time.h>
 
 namespace __xray {
 
-static constexpr uint64_t NanosecondsPerSecond = 1000ULL * 1000 * 1000;
-
 inline bool probeRequiredCPUFeatures() XRAY_NEVER_INSTRUMENT { return true; }
 
 ALWAYS_INLINE uint64_t readTSC(uint8_t &CPU) XRAY_NEVER_INSTRUMENT {
   timespec TS;
   int result = clock_gettime(CLOCK_REALTIME, &TS);
   if (result != 0) {
     Report("clock_gettime(2) returned %d, errno=%d.", result, int(errno));
     TS.tv_sec = 0;
     TS.tv_nsec = 0;
   }
   CPU = 0;
   return TS.tv_sec * NanosecondsPerSecond + TS.tv_nsec;
 }
 
 inline uint64_t getTSCFrequency() XRAY_NEVER_INSTRUMENT {
   return NanosecondsPerSecond;
 }
 
 } // namespace __xray
 
 #else
-"Unsupported CPU Architecture"
+#error Target architecture is not supported.
 #endif // CPU architecture
 
 #endif // XRAY_EMULATE_TSC_H