[tsan] Add TSan debugger APIs

[kubamracek]

Summary

Currently, TSan only reports everything in a formatted textual form. The idea behind this patch is to provide a consistent API that can be used to query information contained in a TSan-produced report. User can use these APIs either in a debugger (via a script or directly), or they can use it directly from the process (e.g. in the __tsan_on_report callback). ASan already has a similar API, see http://reviews.llvm.org/D4466.

Some details about the patch:

• The API is in C (and not C++), because we want it simple and stable.
• The callback __tsan_on_report is called just after the report is printed to the console.
• Added a test case which uses the callback __tsan_on_report to query and verify various data from the report.

[dvyukov]

What do you want to achieve?
__tsan::OnReport already exposes ReportDesc. And it is enough to do the checks.
Long time ago we had lots of tests that did exactly this:

assert(tid == 0);
assert(addr == &my_global);
assert(size == 8);
assert(write == 1);
assert(atomic == 0);
assert(trace[0] > (void *)&main);
assert(trace[0] < (void *)&__tsan_on_report);
assert(trace[1] == 0);

They proved to be very painful to write and to maintain. And in the end they don't actually verify what uses sees (e..g all checks can pass but we print nothing at all). I've dropped them in favor of output_tests. I've spent half an hour trying to find them, but unfortunately it seems that we've dropped our history and masked it with file copies without using svn cp.

[kubamracek]

What do you want to achieve?
__tsan::OnReport already exposes ReportDesc. And it is enough to do the checks.

Exposing ReportDesc is hard, because its definition requires a lot of header files, the structure is complex, and it's not stable. I want the user to be able to use this API in a binary distribution of Clang, where we don't have TSan internal headers. I want to provide API so the user can do something like:

__tsan_on_report(void *report) {
  void *addr = ...; // retrieve the address on which a race was detected
  my_logging(..., addr); // do whatever I want with the report data
}
int main() {
  ... the program I'm testing with TSan ...
}

Long time ago we had lots of tests that did exactly this:

assert(tid == 0);
assert(addr == &my_global);
assert(size == 8);
assert(write == 1);
assert(atomic == 0);
assert(trace[0] > (void *)&main);
assert(trace[0] < (void *)&__tsan_on_report);
assert(trace[1] == 0);

They proved to be very painful to write and to maintain. And in the end they don't actually verify what uses sees (e..g all checks can pass but we print nothing at all). I've dropped them in favor of output_tests. I've spent half an hour trying to find them, but unfortunately it seems that we've dropped our history and masked it with file copies without using svn cp.

Are you against asserts or against the API? If the first, I can print all the stuff in the test and verify it with // CHECK:.

[dvyukov]

I want to provide API so the user can do something like:

Why? What's the value and end goal of this?

[kubamracek]

In D16191#327652, @dvyukov wrote:

I want to provide API so the user can do something like:

Why? What's the value and end goal of this?

Debugger integration and programmatic access to structured data (to avoid parsing of the text output) for various purposes (report post-processing, tracking, aggregation, ...).

[dvyukov]

Debugger integration and programmatic access to structured data (to avoid parsing of the text output) for various purposes (report post-processing, tracking, aggregation, ...).

[dvyukov]

Debugger integration and programmatic access to structured data (to avoid parsing of the text output) for various purposes (report post-processing, tracking, aggregation, ...).

Who and how is going to use this?
Without knowing the answer, I don't know whether we export the right info in the right form. E.g. I don't see how mutex count is useful for report post-processing, tracking, aggregation.

[kubamracek]

Debugger integration and programmatic access to structured data (to avoid parsing of the text output) for various purposes (report post-processing, tracking, aggregation, ...).

Who and how is going to use this?

I'd like to use this in the same way AddressSanitizer reports are integrated into the debugger (LLDB), see http://reviews.llvm.org/D4466. The idea is to provide all the information in a TSan report via a structured API, so that LLDB can query it. The end goal might be to simply present this information in a GUI debugger (that uses LLDB) and/or to show all TSan-produced warnings in some user-friendly way - let's say a table with filtering options. I don't want to limit the options here, if someone wants to generate XML reports of TSan warnings, this should be possible. That's why I'm proposing a very generic API that provides almost all the report information.

[kubamracek]

Changing the lit test to use // CHECK: instead of asserts.

[dvyukov]

So this is for lldb.
I though that you want to write tsan tests in such style using __tsan_on_report callback.
lldb is fine.

lib/tsan/rtl/tsan_debugging.cc

57

Do we need this function?
User already gets report pointer as argument to __tsan_on_report. So this function looks unnecessary and subject to calls at a wrong time, and provoking dangling cur_thread()->current_report (as I can see below).
I would remove it.

63

Argument set for this function looks somewhat strange.
I would either move sleep_trace to a separate function (sleep trace is very auxiliary). Or make this function returns as much assorted data as possible (e.g. also return number of stacks, number of locations, etc).

lib/tsan/rtl/tsan_rtl_report.cc

51

I afraid this will break some of out internal setups which define TSAN_EXTERNAL_HOOKS and define OnReport function. They don't define __tsan_on_report, so they will break on next llvm update.

Define it as:

SANITIZER_WEAK_DEFAULT_IMPL
void __tsan_on_report(const ReportDesc *rep) {

(void)rep;

}

outside of #ifdef/else/endif TSAN_EXTERNAL_HOOKS.

522

Forget to reset thr->current_report here.

test/tsan/debugging.cc

44

I guess you wanted to print result of __tsan_get_current_report() here.

[kubamracek]

Addressing review comments.

Do we need this function? (__tsan_get_current_report)

It's very convenient to use it from LLDB. Even when we break inside __tsan_on_report, the function argument is not easily accessible (it might be optimized out).

[dvyukov]

LGTM with nits

lib/tsan/rtl/tsan_debugging.cc

58

Please add a comment that this is meant to be manually called from a debugger.

lib/tsan/rtl/tsan_rtl_report.cc

500

I think it makes sense to do:

CHECK_EQ(thr->current_report, nullptr);

here. That would catch the bug below and any future similar bugs.

lib/tsan/CMakeLists.txt

 append_list_if(COMPILER_RT_HAS_MSSE3_FLAG -msse3 TSAN_RTL_CFLAGS)
 append_list_if(SANITIZER_LIMIT_FRAME_SIZE -Wframe-larger-than=512
                TSAN_RTL_CFLAGS)
 append_list_if(COMPILER_RT_HAS_WGLOBAL_CONSTRUCTORS_FLAG -Wglobal-constructors
                TSAN_RTL_CFLAGS)
 
 set(TSAN_SOURCES
   rtl/tsan_clock.cc
+  rtl/tsan_debugging.cc
   rtl/tsan_flags.cc
   rtl/tsan_fd.cc
   rtl/tsan_ignoreset.cc
   rtl/tsan_interceptors.cc
   rtl/tsan_interface_ann.cc
   rtl/tsan_interface_atomic.cc
   rtl/tsan_interface.cc
   rtl/tsan_interface_java.cc

lib/tsan/rtl/tsan_debugging.cc

+//===-- tsan_debugging.cc -------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of ThreadSanitizer (TSan), a race detector.
+//
+// TSan debugging API implementation.
+//===----------------------------------------------------------------------===//
+#include "tsan_interface.h"
+#include "tsan_report.h"
+#include "tsan_rtl.h"
+
+using namespace __tsan;
+
+static const char *ReportTypeDescription(ReportType typ) {
+  if (typ == ReportTypeRace) return "data-race";
+  if (typ == ReportTypeVptrRace) return "data-race-vptr";
+  if (typ == ReportTypeUseAfterFree) return "heap-use-after-free";
+  if (typ == ReportTypeVptrUseAfterFree) return "heap-use-after-free-vptr";
+  if (typ == ReportTypeThreadLeak) return "thread-leak";
+  if (typ == ReportTypeMutexDestroyLocked) return "locked-mutex-destroy";
+  if (typ == ReportTypeMutexDoubleLock) return "mutex-double-lock";
+  if (typ == ReportTypeMutexBadUnlock) return "mutex-bad-unlock";
+  if (typ == ReportTypeMutexBadReadLock) return "mutex-bad-read-lock";
+  if (typ == ReportTypeMutexBadReadUnlock) return "mutex-bad-read-unlock";
+  if (typ == ReportTypeSignalUnsafe) return "signal-unsafe-call";
+  if (typ == ReportTypeErrnoInSignal) return "errno-in-signal-handler";
+  if (typ == ReportTypeDeadlock) return "lock-order-inversion";
+  return "";
+}
+
+static const char *ReportLocationTypeDescription(ReportLocationType typ) {
+  if (typ == ReportLocationGlobal) return "global";
+  if (typ == ReportLocationHeap) return "heap";
+  if (typ == ReportLocationStack) return "stack";
+  if (typ == ReportLocationTLS) return "tls";
+  if (typ == ReportLocationFD) return "fd";
+  return "";
+}
+
+static void CopyTrace(SymbolizedStack *first_frame, void **trace,
+                      uptr trace_size) {
+  uptr i = 0;
+  for (SymbolizedStack *frame = first_frame; frame != nullptr;
+       frame = frame->next) {
+    trace[i++] = (void *)frame->info.address;
+    if (i >= trace_size) break;
+  }
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+void *__tsan_get_current_report() {
+  const ReportDesc *rep = cur_thread()->current_report;
+  return (void *)rep;
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_data(void *report, const char **description, int *count,
+                           void **sleep_trace, uptr trace_size) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  *description = ReportTypeDescription(rep->typ);
+  *count = rep->count;
+  if (rep->sleep) CopyTrace(rep->sleep->frames, sleep_trace, trace_size);
+  return 1;
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_stack_count(void *report) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  return rep->stacks.Size();
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_stack(void *report, uptr idx, void **trace,
+                            uptr trace_size) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  CHECK_LT(idx, rep->stacks.Size());
+  ReportStack *stack = rep->stacks[idx];
+  CopyTrace(stack->frames, trace, trace_size);
+  return 1;
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_mop_count(void *report) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  return rep->mops.Size();
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_mop(void *report, uptr idx, int *tid, void **addr,
+                          int *size, int *write, int *atomic, void **trace,
+                          uptr trace_size) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  CHECK_LT(idx, rep->mops.Size());
+  ReportMop *mop = rep->mops[idx];
+  *tid = mop->tid;
+  *addr = (void *)mop->addr;
+  *size = mop->size;
+  *write = mop->write ? 1 : 0;
+  *atomic = mop->atomic ? 1 : 0;
+  CopyTrace(mop->stack->frames, trace, trace_size);
+  return 1;
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_loc_count(void *report) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  return rep->locs.Size();
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_loc(void *report, uptr idx, const char **type,
+                          void **addr, uptr *start, uptr *size, int *tid,
+                          int *fd, int *suppressable, void **trace,
+                          uptr trace_size) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  CHECK_LT(idx, rep->locs.Size());
+  ReportLocation *loc = rep->locs[idx];
+  *type = ReportLocationTypeDescription(loc->type);
+  *addr = (void *)loc->global.start;
+  *start = loc->heap_chunk_start;
+  *size = loc->heap_chunk_size;
+  *tid = loc->tid;
+  *fd = loc->fd;
+  *suppressable = loc->suppressable;
+  if (loc->stack) CopyTrace(loc->stack->frames, trace, trace_size);
+  return 1;
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_mutex_count(void *report) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  return rep->mutexes.Size();
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_mutex(void *report, uptr idx, uptr *mutex_id, void **addr,
+                            int *destroyed, void **trace, uptr trace_size) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  CHECK_LT(idx, rep->mutexes.Size());
+  ReportMutex *mutex = rep->mutexes[idx];
+  *mutex_id = mutex->id;
+  *addr = (void *)mutex->addr;
+  *destroyed = mutex->destroyed;
+  CopyTrace(mutex->stack->frames, trace, trace_size);
+  return 1;
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_thread_count(void *report) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  return rep->threads.Size();
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_thread(void *report, uptr idx, int *tid, uptr *pid,
+                             int *running, const char **name, int *parent_tid,
+                             void **trace, uptr trace_size) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  CHECK_LT(idx, rep->threads.Size());
+  ReportThread *thread = rep->threads[idx];
+  *tid = thread->id;
+  *pid = thread->pid;
+  *running = thread->running;
+  *name = thread->name;
+  *parent_tid = thread->parent_tid;
+  CopyTrace(thread->stack->frames, trace, trace_size);
+  return 1;
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_unique_tid_count(void *report) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  return rep->unique_tids.Size();
+}
+
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_unique_tid(void *report, uptr idx, int *tid) {
+  const ReportDesc *rep = (ReportDesc *)report;
+  CHECK_LT(idx, rep->unique_tids.Size());
+  *tid = rep->unique_tids[idx];
+  return 1;
+}

lib/tsan/rtl/tsan_interface.h

 SANITIZER_INTERFACE_ATTRIBUTE void __tsan_func_entry(void *call_pc);
 SANITIZER_INTERFACE_ATTRIBUTE void __tsan_func_exit();
 
 SANITIZER_INTERFACE_ATTRIBUTE
 void __tsan_read_range(void *addr, unsigned long size);  // NOLINT
 SANITIZER_INTERFACE_ATTRIBUTE
 void __tsan_write_range(void *addr, unsigned long size);  // NOLINT
 
+// User may provide function that would be called right when TSan detects
+// an error. The argument 'report' is an opaque pointer that can be used to
+// gather additional information using other TSan report API functions.
+SANITIZER_INTERFACE_ATTRIBUTE
+void __tsan_on_report(void *report);
+
+// If TSan is currently reporting a detected issue on the current thread,
+// returns an opaque pointer to the current report. Otherwise returns NULL.
+SANITIZER_INTERFACE_ATTRIBUTE
+void *__tsan_get_current_report();
+
+// Returns a report's description (issue type), number of duplicate issues
+// found, and a stack trace of a sleep() call (if one was involved in the
+// issue).
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_data(void *report, const char **description, int *count,
+                           void **sleep_trace, uptr trace_size);
+
+// Returns count and information about stack traces included in the report.
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_stack_count(void *report);
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_stack(void *report, uptr idx, void **trace,
+                            uptr trace_size);
+
+// Returns count and information about memory operations included in the report.
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_mop_count(void *report);
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_mop(void *report, uptr idx, int *tid, void **addr,
+                          int *size, int *write, int *atomic, void **trace,
+                          uptr trace_size);
+
+// Returns count and information about locations included in the report.
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_loc_count(void *report);
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_loc(void *report, uptr idx, const char **type,
+                          void **addr, uptr *start, uptr *size, int *tid,
+                          int *fd, int *suppressable, void **trace,
+                          uptr trace_size);
+
+// Returns count and information about mutexes included in the report.
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_mutex_count(void *report);
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_mutex(void *report, uptr idx, uptr *mutex_id, void **addr,
+                            int *destroyed, void **trace, uptr trace_size);
+
+// Returns count and information about threads included in the report.
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_thread_count(void *report);
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_thread(void *report, uptr idx, int *tid, uptr *pid,
+                             int *running, const char **name, int *parent_tid,
+                             void **trace, uptr trace_size);
+
+// Returns count and information about unique thread IDs included in the report.
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_unique_tid_count(void *report);
+SANITIZER_INTERFACE_ATTRIBUTE
+int __tsan_get_report_unique_tid(void *report, uptr idx, int *tid);
+
 #ifdef __cplusplus
 }  // extern "C"
 #endif
 
 #endif  // TSAN_INTERFACE_H

lib/tsan/rtl/tsan_rtl.h

#ifndef SANITIZER_GO
   u32 last_sleep_stack_id;
   ThreadClock last_sleep_clock;
 #endif
 
   // Set in regions of runtime that must be signal-safe and fork-safe.
   // If set, malloc must not be called.
   int nomalloc;
 
+  const ReportDesc *current_report;
+
   explicit ThreadState(Context *ctx, int tid, int unique_id, u64 epoch,
                        unsigned reuse_count,
                        uptr stk_addr, uptr stk_size,
                        uptr tls_addr, uptr tls_size);
 };
 
 #ifndef SANITIZER_GO
 #if SANITIZER_MAC || SANITIZER_ANDROID

lib/tsan/rtl/tsan_rtl_report.cc

Printf("FATAL: ThreadSanitizer CHECK failed: "
          file, line, cond, (uptr)v1, (uptr)v2);
   PrintCurrentStackSlow(StackTrace::GetCurrentPc());
   Die();
 }
 
 // Can be overriden by an application/test to intercept reports.
 #ifdef TSAN_EXTERNAL_HOOKS
 bool OnReport(const ReportDesc *rep, bool suppressed);
+extern "C" void __tsan_on_report(const ReportDesc *rep);
 #else
 SANITIZER_WEAK_CXX_DEFAULT_IMPL
 bool OnReport(const ReportDesc *rep, bool suppressed) {
   (void)rep;
   return suppressed;
 }
+SANITIZER_WEAK_DEFAULT_IMPL
+void __tsan_on_report(const ReportDesc *rep) {
+  (void)rep;
+}
 #endif
 
 static void StackStripMain(SymbolizedStack *frames) {
   SymbolizedStack *last_frame = nullptr;
   SymbolizedStack *last_frame2 = nullptr;
   for (SymbolizedStack *cur = frames; cur; cur = cur->next) {
     last_frame2 = last_frame;
     last_frame = cur;
static void AddRacyStacks(ThreadState *thr, VarSizeStackTrace traces[2],
   }
 }
 
 bool OutputReport(ThreadState *thr, const ScopedReport &srep) {
   if (!flags()->report_bugs)
     return false;
   atomic_store_relaxed(&ctx->last_symbolize_time_ns, NanoTime());
   const ReportDesc *rep = srep.GetReport();
+  thr->current_report = rep;
   Suppression *supp = 0;
   uptr pc_or_addr = 0;
   for (uptr i = 0; pc_or_addr == 0 && i < rep->mops.Size(); i++)
     pc_or_addr = IsSuppressed(rep->typ, rep->mops[i]->stack, &supp);
   for (uptr i = 0; pc_or_addr == 0 && i < rep->stacks.Size(); i++)
     pc_or_addr = IsSuppressed(rep->typ, rep->stacks[i], &supp);
   for (uptr i = 0; pc_or_addr == 0 && i < rep->threads.Size(); i++)
     pc_or_addr = IsSuppressed(rep->typ, rep->threads[i]->stack, &supp);
   for (uptr i = 0; pc_or_addr == 0 && i < rep->locs.Size(); i++)
     pc_or_addr = IsSuppressed(rep->typ, rep->locs[i], &supp);
   if (pc_or_addr != 0) {
     Lock lock(&ctx->fired_suppressions_mtx);
     FiredSuppression s = {srep.GetReport()->typ, pc_or_addr, supp};
     ctx->fired_suppressions.push_back(s);
   }
   {
     bool old_is_freeing = thr->is_freeing;
     thr->is_freeing = false;
     bool suppressed = OnReport(rep, pc_or_addr != 0);
     thr->is_freeing = old_is_freeing;
     if (suppressed)
       return false;
   }
   PrintReport(rep);
+  __tsan_on_report(rep);
   ctx->nreported++;
   if (flags()->halt_on_error)
     Die();
+  thr->current_report = nullptr;
   return true;
 }
 
 bool IsFiredSuppression(Context *ctx, ReportType type, StackTrace trace) {
   ReadLock lock(&ctx->fired_suppressions_mtx);
   for (uptr k = 0; k < ctx->fired_suppressions.size(); k++) {
     if (ctx->fired_suppressions[k].type != type)
       continue;

test/tsan/debugging.cc

+// RUN: %clangxx_tsan -O1 %s -o %t
+// RUN: %deflake %run %t 2>&1 | FileCheck %s
+
+#include <pthread.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "test.h"
+
+extern "C" {
+void __tsan_on_report(void *report);
+void *__tsan_get_current_report();
+int __tsan_get_report_data(void *report, const char **description, int *count,
+                           void **sleep_trace, unsigned long trace_size);
+int __tsan_get_report_mop_count(void *report);
+int __tsan_get_report_mop(void *report, unsigned long idx, int *tid,
+                          void **addr, int *size, int *write, int *atomic,
+                          void **trace, unsigned long trace_size);
+}
+
+long my_global;
+
+void *Thread(void *a) {
+  barrier_wait(&barrier);
+  my_global = 42;
+  return NULL;
+}
+
+int main() {
+  barrier_init(&barrier, 2);
+  fprintf(stderr, "&my_global = %p\n", &my_global);
+  // CHECK: &my_global = [[GLOBAL:0x[0-9a-f]+]]
+  pthread_t t;
+  pthread_create(&t, 0, Thread, 0);
+  my_global = 41;
+  barrier_wait(&barrier);
+  pthread_join(t, 0);
+  fprintf(stderr, "Done.\n");
+}
+
+void __tsan_on_report(void *report) {
+  fprintf(stderr, "__tsan_on_report(%p)\n", report);
+  fprintf(stderr, "__tsan_get_current_report() = %p\n", report);
+  // CHECK: __tsan_on_report([[REPORT:0x[0-9a-f]+]])
+  // CHECK: __tsan_get_current_report() = [[REPORT]]
+
+  const char *description;
+  int count;
+  void *sleep_trace[16] = {0};
+  __tsan_get_report_data(report, &description, &count, sleep_trace, 16);
+  fprintf(stderr, "report type = '%s', count = %d\n", description, count);
+  // CHECK: report type = 'data-race', count = 0
+
+  int mop_count = __tsan_get_report_mop_count(report);
+  fprintf(stderr, "mop_count = %d\n", mop_count);
+  // CHECK: mop_count = 2
+
+  int tid;
+  void *addr;
+  int size, write, atomic;
+  void *trace[16] = {0};
+
+  __tsan_get_report_mop(report, 0, &tid, &addr, &size, &write, &atomic, trace,
+                        16);
+  fprintf(stderr, "tid = %d, addr = %p, size = %d, write = %d, atomic = %d\n",
+          tid, addr, size, write, atomic);
+  // CHECK: tid = 1, addr = [[GLOBAL]], size = 8, write = 1, atomic = 0
+  fprintf(stderr, "trace[0] = %p, trace[1] = %p\n", trace[0], trace[1]);
+  // CHECK: trace[0] = 0x{{[0-9a-f]+}}, trace[1] = 0x0
+
+  __tsan_get_report_mop(report, 1, &tid, &addr, &size, &write, &atomic, trace,
+                        16);
+  fprintf(stderr, "tid = %d, addr = %p, size = %d, write = %d, atomic = %d\n",
+          tid, addr, size, write, atomic);
+  // CHECK: tid = 0, addr = [[GLOBAL]], size = 8, write = 1, atomic = 0
+  fprintf(stderr, "trace[0] = %p, trace[1] = %p\n", trace[0], trace[1]);
+  // CHECK: trace[0] = 0x{{[0-9a-f]+}}, trace[1] = 0x0
+}
+
+// CHECK: Done.
+// CHECK: ThreadSanitizer: reported 1 warnings