[XRay] Implement powerpc64le xray.

[timshen]

Summary

powerpc64 bigendian is not supported, but I believe that most logic can
be shared, except for xray_powerpc64.cc.

Also add a function InvalidateInstructionCache to xray_util.h, which is
copied from llvm/Support/Memory.cpp. I'm not sure if I need to add a unittest,
and I don't know how.

[dberris]

compiler-rt/lib/xray/xray_powerpc64.cc

34 ↗

(On Diff #87741)

Just curious -- does ppc not have an equivalent for rdtscp instruction in userspace?

If it does, we should try and use that as well as get the actual cycle frequency in Linux through the various possible means.

50–52 ↗

(On Diff #87741)

Can we do this check once somewhere at the top of the file? Something like:

#if !defined(__LITTLE_ENDIAN__)
#error ...
#endif

compiler-rt/lib/xray/xray_utils.cc

175 ↗

(On Diff #87741)

As far as I can tell, we don't build XRay for Mac OS (unless this is being done for other reasons). Will this work just fine without #if defined(__APPLE__) here?

[timshen]

Update on comments.

[timshen]

compiler-rt/lib/xray/xray_powerpc64.cc

34 ↗

(On Diff #87741)

It's a good question.

It turns out, Power CPU keeps track of the ticks as well as x86 does, but in a different frequency, not necessarily the CPU clock frequency. The frequency is configurable.

Luckily, glibc exposes two functions to get the frequency as well as the current tick: https://www.gnu.org/software/libc/manual/html_node/PowerPC.html. I think this is exactly what we need.

However, currently the xray_utils.h interfaces are designed in a way to assume the TSC frequency is the same as CPU clock frequency, and every logging module is asking for CPU frequency.

I think a more proper change is to generalize CPU frequency to a "timer frequency" or something, but I'd better do that in a succeeding patch, leaving this patch a bit cleaner. I can check in the frequency change first.

compiler-rt/lib/xray/xray_utils.cc

175 ↗

(On Diff #87741)

Done. I realized that compiler_rt, as well as libgcc, has this __clear_cache function, but they seem to do poorly on the job. I moved the clearCache function into xray_powerpc64.cc and kept powerpc64 case only.

[timshen]

Use emulated TSC for now.

[timshen]

Use emulated TSC until the frequency interface design is clear.

[timshen]

Rebase onto D29796.

[dberris]

I defer to @echristo on the ppc-specific bits. I suspect you want to stage committing this by doing the LLVM side changes first, and the clang changes next, then finally the compiler-rt implementation.

llvm/lib/Target/PowerPC/PPCAsmPrinter.cpp

1136 ↗

(On Diff #87909)

Probably want a friendly/informative message saying why this ought to be unreachable -- does the codegen not disambiguate tail calls and patchable returns?

[echristo]

Rough request for more elaborate comments and a couple of inline comments, but nothing major. Feel free to commit after.

Thanks!

-eric

llvm/lib/Target/PowerPC/PPCAsmPrinter.cpp

1136 ↗

(On Diff #87909)

+1

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

68 ↗

(On Diff #87909)

-1? Can you make this a bit more descriptive? :)

[timshen]

Updated as commented.

[timshen]

llvm/lib/Target/PowerPC/PPCAsmPrinter.cpp

1136 ↗

(On Diff #87909)

Sorry, I didn't mean to leave it empty... sometimes when I turn a poc into a real patch I miss things. :P

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

68 ↗

(On Diff #87909)

Added a block comment indicating the parameter. It's default-initialized to -1 if I don't manually initialize it. I have to, however, manually initialize it, because of our lovely C++ default-argument rules.

[vitalybuka]

reverted to fix http://lab.llvm.org:8011/builders/sanitizer-ppc64be-linux/builds/1486/steps/test%20standalone%20compiler-rt/logs/stdio

[timshen]

This patch gets reverted, because the sanitizer-ppc64le bots aren't happy about:

http://lab.llvm.org:8011/builders/sanitizer-ppc64le-linux/builds/1414/steps/test%20standalone%20compiler-rt/logs/stdio

kbarton@ can you take a look?

[timshen]

Dean found out that we actually compare different types (std::errc against std::error_code) in the tests, and the implicit std::errc -> std::error_code may never happen. Checked in r295248 to fix that. Landing the xray patch again to see how the bots react. :P

compiler-rt/trunk/cmake/config-ix.cmake

set(ALL_PROFILE_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64} ${PPC64}
     ${MIPS32} ${MIPS64} ${S390X})
 set(ALL_TSAN_SUPPORTED_ARCH ${X86_64} ${MIPS64} ${ARM64} ${PPC64})
 set(ALL_UBSAN_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64}
     ${MIPS32} ${MIPS64} ${PPC64} ${S390X})
 set(ALL_SAFESTACK_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM64} ${MIPS32} ${MIPS64})
 set(ALL_CFI_SUPPORTED_ARCH ${X86} ${X86_64} ${MIPS64})
 set(ALL_ESAN_SUPPORTED_ARCH ${X86_64} ${MIPS64})
 set(ALL_SCUDO_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64})
-set(ALL_XRAY_SUPPORTED_ARCH ${X86_64} ${ARM32} ${ARM64} ${MIPS32} ${MIPS64})
+set(ALL_XRAY_SUPPORTED_ARCH ${X86_64} ${ARM32} ${ARM64} ${MIPS32} ${MIPS64} powerpc64le)
 
 if(APPLE)
   include(CompilerRTDarwinUtils)
 
   find_darwin_sdk_dir(DARWIN_osx_SYSROOT macosx)
   find_darwin_sdk_dir(DARWIN_iossim_SYSROOT iphonesimulator)
   find_darwin_sdk_dir(DARWIN_ios_SYSROOT iphoneos)
   find_darwin_sdk_dir(DARWIN_watchossim_SYSROOT watchsimulator)

compiler-rt/trunk/lib/xray/CMakeLists.txt

set(mips64_SOURCES
     xray_trampoline_mips64.S
     ${XRAY_SOURCES})
 
 set(mips64el_SOURCES
     xray_mips64.cc
     xray_trampoline_mips64.S
     ${XRAY_SOURCES})
 
+set(powerpc64le_SOURCES
+        xray_powerpc64.cc
+        xray_trampoline_powerpc64.cc
+        xray_trampoline_powerpc64.S
+        ${XRAY_SOURCES})
+
 include_directories(..)
 include_directories(../../include)
 
 set(XRAY_CFLAGS ${SANITIZER_COMMON_CFLAGS})
 set(XRAY_COMMON_DEFINITIONS XRAY_HAS_EXCEPTIONS=1)
 append_list_if(
   COMPILER_RT_HAS_XRAY_COMPILER_FLAG XRAY_SUPPORTED=1 XRAY_COMMON_DEFINITIONS)
 

compiler-rt/trunk/lib/xray/xray_interface.cc

 #elif defined(__aarch64__)
 static const int16_t cSledLength = 32;
 #elif defined(__arm__)
 static const int16_t cSledLength = 28;
 #elif SANITIZER_MIPS32
 static const int16_t cSledLength = 48;
 #elif SANITIZER_MIPS64
 static const int16_t cSledLength = 64;
+#elif defined(__powerpc64__)
+static const int16_t cSledLength = 8;
 #else
 #error "Unsupported CPU Architecture"
 #endif /* CPU architecture */
 
 // This is the function to call when we encounter the entry or exit sleds.
 std::atomic<void (*)(int32_t, XRayEntryType)> XRayPatchedFunction{nullptr};
 
 // MProtectHelper is an RAII wrapper for calls to mprotect(...) that will undo

compiler-rt/trunk/lib/xray/xray_powerpc64.cc

+//===-- xray_powerpc64.cc ---------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+// Implementation of powerpc64 and powerpc64le routines.
+//
+//===----------------------------------------------------------------------===//
+#include "sanitizer_common/sanitizer_common.h"
+#include "xray_defs.h"
+#include "xray_interface_internal.h"
+#include "xray_utils.h"
+#include <atomic>
+#include <cassert>
+#include <cstring>
+
+#ifndef __LITTLE_ENDIAN__
+#error powerpc64 big endian is not supported for now.
+#endif
+
+namespace {
+
+constexpr unsigned long long JumpOverInstNum = 7;
+
+void clearCache(void *Addr, size_t Len) {
+  const size_t LineSize = 32;
+
+  const intptr_t Mask = ~(LineSize - 1);
+  const intptr_t StartLine = ((intptr_t)Addr) & Mask;
+  const intptr_t EndLine = ((intptr_t)Addr + Len + LineSize - 1) & Mask;
+
+  for (intptr_t Line = StartLine; Line < EndLine; Line += LineSize)
+    asm volatile("dcbf 0, %0" : : "r"(Line));
+  asm volatile("sync");
+
+  for (intptr_t Line = StartLine; Line < EndLine; Line += LineSize)
+    asm volatile("icbi 0, %0" : : "r"(Line));
+  asm volatile("isync");
+}
+
+} // namespace
+
+extern "C" void __clear_cache(void *start, void *end);
+
+namespace __xray {
+
+bool patchFunctionEntry(const bool Enable, uint32_t FuncId,
+                        const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT {
+  if (Enable) {
+    // lis 0, FuncId[16..32]
+    // li 0, FuncId[0..15]
+    *reinterpret_cast<uint64_t *>(Sled.Address) =
+        (0x3c000000ull + (FuncId >> 16)) +
+        ((0x60000000ull + (FuncId & 0xffff)) << 32);
+  } else {
+    // b +JumpOverInstNum instructions.
+    *reinterpret_cast<uint32_t *>(Sled.Address) =
+        0x48000000ull + (JumpOverInstNum << 2);
+  }
+  clearCache(reinterpret_cast<void *>(Sled.Address), 8);
+  return true;
+}
+
+bool patchFunctionExit(const bool Enable, uint32_t FuncId,
+                       const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT {
+  if (Enable) {
+    // lis 0, FuncId[16..32]
+    // li 0, FuncId[0..15]
+    *reinterpret_cast<uint64_t *>(Sled.Address) =
+        (0x3c000000ull + (FuncId >> 16)) +
+        ((0x60000000ull + (FuncId & 0xffff)) << 32);
+  } else {
+    // Copy the blr/b instruction after JumpOverInstNum instructions.
+    *reinterpret_cast<uint32_t *>(Sled.Address) =
+        *(reinterpret_cast<uint32_t *>(Sled.Address) + JumpOverInstNum);
+  }
+  clearCache(reinterpret_cast<void *>(Sled.Address), 8);
+  return true;
+}
+
+bool patchFunctionTailExit(const bool Enable, const uint32_t FuncId,
+                           const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT {
+  return patchFunctionExit(Enable, FuncId, Sled);
+}
+
+// FIXME: Maybe implement this better?
+bool probeRequiredCPUFeatures() XRAY_NEVER_INSTRUMENT { return true; }
+
+} // namespace __xray

compiler-rt/trunk/lib/xray/xray_powerpc64.inc

+//===-- xray_powerpc64.inc --------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+//===----------------------------------------------------------------------===//
+
+#include <cstdint>
+#include <mutex>
+#include <sys/platform/ppc.h>
+
+#include "xray_defs.h"
+
+namespace __xray {
+
+ALWAYS_INLINE uint64_t readTSC(uint8_t &CPU) XRAY_NEVER_INSTRUMENT {
+  CPU = 0;
+  return __ppc_get_timebase();
+}
+
+inline uint64_t getTSCFrequency() XRAY_NEVER_INSTRUMENT {
+  static std::mutex M;
+  std::lock_guard<std::mutex> Guard(M);
+  return __ppc_get_timebase_freq();
+}
+
+inline bool probeRequiredCPUFeatures() XRAY_NEVER_INSTRUMENT {
+  return true;
+}
+
+} // namespace __xray

compiler-rt/trunk/lib/xray/xray_trampoline_powerpc64.S

+  .text
+  .abiversion 2
+  .globl  __xray_FunctionEntry
+  .p2align  4
+__xray_FunctionEntry:
+  std 0, 16(1)
+  stdu 1, -408(1)
+# Spill r3-r10, f1-f13, and vsr34-vsr45, which are parameter registers.
+# If this appears to be slow, the caller needs to pass in number of generic,
+# floating point, and vector parameters, so that we only spill those live ones.
+  std 3, 32(1)
+  ld 3, 400(1) # FuncId
+  std 4, 40(1)
+  std 5, 48(1)
+  std 6, 56(1)
+  std 7, 64(1)
+  std 8, 72(1)
+  std 9, 80(1)
+  std 10, 88(1)
+  addi 4, 1, 96
+  stxsdx 1, 0, 4
+  addi 4, 1, 104
+  stxsdx 2, 0, 4
+  addi 4, 1, 112
+  stxsdx 3, 0, 4
+  addi 4, 1, 120
+  stxsdx 4, 0, 4
+  addi 4, 1, 128
+  stxsdx 5, 0, 4
+  addi 4, 1, 136
+  stxsdx 6, 0, 4
+  addi 4, 1, 144
+  stxsdx 7, 0, 4
+  addi 4, 1, 152
+  stxsdx 8, 0, 4
+  addi 4, 1, 160
+  stxsdx 9, 0, 4
+  addi 4, 1, 168
+  stxsdx 10, 0, 4
+  addi 4, 1, 176
+  stxsdx 11, 0, 4
+  addi 4, 1, 184
+  stxsdx 12, 0, 4
+  addi 4, 1, 192
+  stxsdx 13, 0, 4
+  addi 4, 1, 200
+  stxvd2x 34, 0, 4
+  addi 4, 1, 216
+  stxvd2x 35, 0, 4
+  addi 4, 1, 232
+  stxvd2x 36, 0, 4
+  addi 4, 1, 248
+  stxvd2x 37, 0, 4
+  addi 4, 1, 264
+  stxvd2x 38, 0, 4
+  addi 4, 1, 280
+  stxvd2x 39, 0, 4
+  addi 4, 1, 296
+  stxvd2x 40, 0, 4
+  addi 4, 1, 312
+  stxvd2x 41, 0, 4
+  addi 4, 1, 328
+  stxvd2x 42, 0, 4
+  addi 4, 1, 344
+  stxvd2x 43, 0, 4
+  addi 4, 1, 360
+  stxvd2x 44, 0, 4
+  addi 4, 1, 376
+  stxvd2x 45, 0, 4
+  std 2, 392(1)
+  mflr 0
+  std 0, 400(1)
+
+  li 4, 0
+  bl _ZN6__xray23CallXRayPatchedFunctionEi13XRayEntryType
+  nop
+
+  addi 4, 1, 96
+  lxsdx 1, 0, 4
+  addi 4, 1, 104
+  lxsdx 2, 0, 4
+  addi 4, 1, 112
+  lxsdx 3, 0, 4
+  addi 4, 1, 120
+  lxsdx 4, 0, 4
+  addi 4, 1, 128
+  lxsdx 5, 0, 4
+  addi 4, 1, 136
+  lxsdx 6, 0, 4
+  addi 4, 1, 144
+  lxsdx 7, 0, 4
+  addi 4, 1, 152
+  lxsdx 8, 0, 4
+  addi 4, 1, 160
+  lxsdx 9, 0, 4
+  addi 4, 1, 168
+  lxsdx 10, 0, 4
+  addi 4, 1, 176
+  lxsdx 11, 0, 4
+  addi 4, 1, 184
+  lxsdx 12, 0, 4
+  addi 4, 1, 192
+  lxsdx 13, 0, 4
+  addi 4, 1, 200
+  lxvd2x 34, 0, 4
+  addi 4, 1, 216
+  lxvd2x 35, 0, 4
+  addi 4, 1, 232
+  lxvd2x 36, 0, 4
+  addi 4, 1, 248
+  lxvd2x 37, 0, 4
+  addi 4, 1, 264
+  lxvd2x 38, 0, 4
+  addi 4, 1, 280
+  lxvd2x 39, 0, 4
+  addi 4, 1, 296
+  lxvd2x 40, 0, 4
+  addi 4, 1, 312
+  lxvd2x 41, 0, 4
+  addi 4, 1, 328
+  lxvd2x 42, 0, 4
+  addi 4, 1, 344
+  lxvd2x 43, 0, 4
+  addi 4, 1, 360
+  lxvd2x 44, 0, 4
+  addi 4, 1, 376
+  lxvd2x 45, 0, 4
+  ld 0, 400(1)
+  mtlr 0
+  ld 2, 392(1)
+  ld 3, 32(1)
+  ld 4, 40(1)
+  ld 5, 48(1)
+  ld 6, 56(1)
+  ld 7, 64(1)
+  ld 8, 72(1)
+  ld 9, 80(1)
+  ld 10, 88(1)
+
+  addi 1, 1, 408
+  ld 0, 16(1)
+  blr
+
+  .globl  __xray_FunctionExit
+  .p2align  4
+__xray_FunctionExit:
+  std 0, 16(1)
+  ld 0, -8(1) # FuncId
+  stdu 1, -72(1)
+# Spill r3, f1, and vsr34, the return value registers.
+  std 3, 32(1)
+  mr 3, 0
+  addi 4, 1, 40
+  stxsdx 1, 0, 4
+  addi 4, 1, 48
+  stxvd2x 34, 0, 4
+  mflr 0
+  std 0, 64(1)
+  li 4, 1
+  bl _ZN6__xray23CallXRayPatchedFunctionEi13XRayEntryType
+  nop
+  ld 0, 64(1)
+  mtlr 0
+  ld 3, 32(1)
+  addi 4, 1, 40
+  lxsdx 1, 0, 4
+  addi 4, 1, 48
+  lxvd2x 34, 0, 4
+  addi 1, 1, 72
+  ld 0, 16(1)
+  blr

compiler-rt/trunk/lib/xray/xray_trampoline_powerpc64.cc

+#include <atomic>
+#include <xray/xray_interface.h>
+
+namespace __xray {
+
+extern std::atomic<void (*)(int32_t, XRayEntryType)> XRayPatchedFunction;
+
+// Implement this in C++ instead of assembly, to avoid dealing with ToC by hand.
+void CallXRayPatchedFunction(int32_t FuncId, XRayEntryType Type) {
+  auto fptr = __xray::XRayPatchedFunction.load();
+  if (fptr != nullptr)
+    (*fptr)(FuncId, Type);
+}
+
+} // namespace __xray

compiler-rt/trunk/lib/xray/xray_tsc.h

 // This file is a part of XRay, a dynamic runtime instrumentation system.
 //
 //===----------------------------------------------------------------------===//
 #ifndef XRAY_EMULATE_TSC_H
 #define XRAY_EMULATE_TSC_H
 
 #if defined(__x86_64__)
 #include "xray_x86_64.inc"
+#elif defined(__powerpc64__)
+#include "xray_powerpc64.inc"
 #elif defined(__arm__) || defined(__aarch64__) || defined(__mips__)
 // Emulated TSC.
 // There is no instruction like RDTSCP in user mode on ARM. ARM's CP15 does
 //   not have a constant frequency like TSC on x86(_64), it may go faster
 //   or slower depending on CPU turbo or power saving mode. Furthermore,
 //   to read from CP15 on ARM a kernel modification or a driver is needed.
 //   We can not require this from users of compiler-rt.
 // So on ARM we use clock_gettime() which gives the result in nanoseconds.