[OMPT] Add implementation and tests of Archer tool

[protze.joachim]

Summary

The tool provides TSAN annotations for OpenMP synchronization. Enabled for
application execution by setting OMP_TOOL_LIBRARIES=libarcher.so, this
library should get installed in the same order as libomp, so the linker
should find the file when the application is linked with OpenMP.

The tool detects whether the application was built with TSan and rejects
activation according to the OMPT protocol if there is no TSan-rt.

Further it is recommended to set
TSAN_OPTIONS="ignore_noninstrumented_modules=1"
This avoids some false positive errors from the OpenMP runtime.

[Hahnfeld]

Do we really want to call this tool Archer? I understand that it was written for that purpose but IMO this can serve a much more general purpose, even without the Archer "framework" around it.

[protze.joachim]

Reasons to keep the name archer (or archer-rt):

• We already have that name, we don't need to come up with a new name :)
• You can easily find related publications under that name, which explain the reasoning behind the library
• The programmer should never see the name, once the tool is integrated into the runtime workflow

[simoatze]

This looks great, thanks guys!
I agree, the name is mostly a "marketing" thing at this point, people know it in that way, changing it would be confusing.

[protze.joachim]

This will not compile with the current version of the OMPT interface, needs updates first

[protze.joachim]

The code is updated to be compatible with OMPT as in OpenMP 5.0.

The OpenMP runtime code is modified to automatically load Archer, which will itself only make active if the application is compiled with TSan.

We are currently in the process to relicense the code under "Apache License v2.0 with LLVM Exceptions", but this should have no influence to the code review.

[protze.joachim]

Relicensed Archer under Apache license, updated the files.

[Hahnfeld]

Some high-level comments inline

runtime/src/ompt-general.cpp

233–235 ↗

(On Diff #200309)

These preprocessor guards are redundant.

240–245 ↗

(On Diff #200309)

If it's not tested, it should be removed.

tools/CMakeLists.txt

8

Why do we need to handle this dummy?

tools/archer/CMakeLists.txt

2–5

I don't think this should be here...

tools/archer/LICENSE.txt

1–3 ↗

(On Diff #200309)

there is a LICENSE.txt in the openmp folder, I don't think we should have a copy in this subdirectory.

tools/archer/counter.h

3

counter.h

tools/archer/ftsan.c

3

Why do we need this in here? This should surely be handled in the compiler?

[protze.joachim]

Addressing Jonas' comments

tools/archer/ftsan.c

3

This allows to use those basic annotations of synchronization in Fortran code.
The annotation interface is also intended for manual instrumenting application code.

Like for SPEC OMP: 371.applu331/src/syncs.F90

@@ -82,6 +85,8 @@
 !$omp flush(isync)
          end do
+       CALL AnnotateHappensAfter(__FILE__, __LINE__, isync(omp_get_thread_num()))
+       CALL AnnotateHappensBefore(__FILE__, __LINE__, isync(iam))
          isync(iam) = 1
 !$omp flush(isync)
       endif

[protze.joachim]

Removed a file which was mistakenly added

[Hahnfeld]

More comments

tools/CMakeLists.txt

3

s/project/tool/

tools/archer/CMakeLists.txt

5

I think this should now say See https://llvm.org/LICENSE.txt for license information., at least that's what other files have.

18–19

Please use OPENMP_INSTALL_LIBDIR to handle libdir suffix.

tools/archer/ftsan.c

3

It's not required for Archer and belongs into the sanitizer runtime.

tools/archer/ompt-tsan.cpp

204

This should be standard now.

545

Indention looks odd...

721–722

?

748

?

tools/archer/tests/CMakeLists.txt

6

Do they?

34–35

At least these two don't exist here.

tools/archer/tests/lit.cfg

106–109

Is this needed?

118–131

Is this needed?

tools/archer/tests/lit.site.cfg.in

14–16

This should not be needed

tools/archer/tests/races/critical-unrelated.c

38

This depends on the compiler used, right? Can the tests be executed by GCC?

[protze.joachim]

tools/archer/ftsan.c

3

It doesn't harm to have this duplicately. I'm not sure how this would fit into the LLVM/sanitizer as long as there is no Fortran frontend supporting this sanitizer.

I moved the code to the other declarations in ompt-tsan.cpp.

[protze.joachim]

Addressed comments by Jonas.
The tests also support gcc now.

[Hahnfeld]

tools/archer/ompt-tsan.cpp

176–193

Moving the code doesn't solve the underlying issues:

1. It's not required for Archer, or at least for the basic tests to pass.
2. It's not related to OpenMP per se.

Besides, with the current usage to dynamically load the library during runtime, it will not resolve symbols during linkage.

384–385

This comment doesn't sound right.

924–925

This functionality is not used in the tests and could be a separate patch.

[Hahnfeld]

tools/archer/tests/lit.site.cfg.in

14–15

Still not needed

30–52

Not needed

[protze.joachim]

Removed the Fortran annotation functions :/
Fixed also the other comments

tools/archer/ompt-tsan.cpp

924–925

This functionality is useful for debugging errors resulting from errors in the OMPT implementation.

[Hahnfeld]

tools/archer/ompt-tsan.cpp

924–925

I agree, and this sounds like a good motivation for a second tool. What's the advantage of having this in ompt-tsan.cpp?

[protze.joachim]

Removed the counters as requested

[Hahnfeld]

The code itself looks good to me (it should probably be clang-formated). If others have comments, please raise them now.

Apart from that, I still found many unused lines in the testing framework...

tools/archer/README.md

168–177 ↗

(On Diff #219468)

Do we still want to speak of clang-archer?

tools/archer/tests/CMakeLists.txt

19–20

These variables are not defined, so I guess they can be safely removed

30–31

Likewise

tools/archer/tests/lit.cfg

54–63

I think these lines can be removed, see comments in CMakeLists.txt and implications

93–97

This can probably be removed

tools/archer/tests/lit.site.cfg.in

14–15

Ping, there's no definition of OPENMP_TEST_ARCHER_FLAGS and no use of test_archer_flags

19–25

All of these are either not used or are probably not needed, see comment in CMakeLists.txt.

tools/archer/tests/ompt/ompt-signal.h

3

We should probably work on deduplicating this from the OMPT tests. Not a strict requirement for now, the file is probably small enough...

[twoh]

openmp/tools/archer/tests/lit.site.cfg.in

13 ↗

(On Diff #229820)

@protze.joachim Shouldn't this be @LIBARCHER_HAVE_LIBATOMIC@ , based on openmp/tools/archer/tests/CMakeLists.txt? For our local unit test runs, they fail with

llvm-lit: /home/engshare/third-party2/llvm/src/llvm/utils/lit/lit/TestingConfig.py:102: fatal: unable to parse config file '/home/engshare/third-party2/llvm/src/build-centos7-native/build/projects/openmp/tools/archer/tests/lit.site.cfg', traceback: Traceback (most recent call last):
  File "/home/engshare/third-party2/llvm/src/llvm/utils/lit/lit/TestingConfig.py", line 89, in load_from_path
    exec(compile(data, path, 'exec'), cfg_globals, None)
  File "/home/engshare/third-party2/llvm/src/build-centos7-native/build/projects/openmp/tools/archer/tests/lit.site.cfg", line 14
    config.has_libatomic =
                          ^
SyntaxError: invalid syntax

[gribozavr2]

The newly added tests fail on buildbots: http://lab.llvm.org:8011/builders/openmp-gcc-x86_64-linux-debian/builds/236

Could you take a look? And please consider reverting in the meanwhile.

[protze.joachim]

In D45890#1756465, @gribozavr2 wrote:

The newly added tests fail on buildbots: http://lab.llvm.org:8011/builders/openmp-gcc-x86_64-linux-debian/builds/236

Could you take a look? And please consider reverting in the meanwhile.

Several things:

• the tests rely on the availability of thread-sanitizer
• I disabled the tests for standalone builds (as in the specific build-bot)
• in a follow-up review, I'll enable the tests based on llvm-config information

• MOST important: we should make the sanitizers available in the build-bot

[gribozavr2]

In D45890#1756747, @protze.joachim wrote:

• MOST important: we should make the sanitizers available in the build-bot

May I ask you to make the necessary changes to the bot configuration? It is here: https://github.com/llvm/llvm-zorg/blob/master/zorg/buildbot/builders/OpenMPBuilder.py

[gribozavr2]

In D45890#1756747, @protze.joachim wrote:

• the tests rely on the availability of thread-sanitizer

The tests should be annotated as REQUIRES: tsan or whatever the appropriate tag is.

[protze.joachim]

In D45890#1756796, @gribozavr2 wrote:

In D45890#1756747, @protze.joachim wrote:

• the tests rely on the availability of thread-sanitizer

The tests should be annotated as REQUIRES: tsan or whatever the appropriate tag is.

I pushed this change, so that the tests are off currently.
I'll create a review to check and announce this capability. (@Hahnfeld I think, you have experience in testing the test-compiler for features?)

Thanks for the pointer to the buildbot configuration, I'll have a look!

[gribozavr2]

Thank you very much for the quick workaround!

[aaronpuchert]

openmp/tools/archer/CMakeLists.txt

13–18 ↗

(On Diff #229820)

The commit message talks about setting OMP_TOOL_LIBRARIES=libarcher.so, so why do we build and install this as static library?

If there are uses for the static library, could we decide based on LIBOMP_ENABLE_SHARED or BUILD_SHARED_LIBS whether to build it?

CMakeLists.txt

if (APPLE OR WIN32)
     message(FATAL_ERROR "libomptarget cannot be built on Windows and MacOS X!")
   elseif (NOT OPENMP_HAVE_STD_CPP11_FLAG)
     message(FATAL_ERROR "Host compiler must support C++11 to build libomptarget!")
   endif()
 
   add_subdirectory(libomptarget)
 endif()
 
+set(ENABLE_OMPT_TOOLS ON)
+# Currently tools are not tested well on Windows or MacOS X.
+if (APPLE OR WIN32)
+  set(ENABLE_OMPT_TOOLS OFF)
+endif()
+
+option(OPENMP_ENABLE_OMPT_TOOLS "Enable building ompt based tools for OpenMP."
+       ${ENABLE_OMPT_TOOLS})
+if (OPENMP_ENABLE_OMPT_TOOLS)
+  add_subdirectory(tools)
+endif()
+
 # Now that we have seen all testuites, create the check-openmp target.
 construct_check_openmp_target()

runtime/CMakeLists.txt

if(${OPENMP_STANDALONE_BUILD})
   libomp_say("Use Adaptive locks   -- ${LIBOMP_USE_ADAPTIVE_LOCKS}")
   libomp_say("Use quad precision   -- ${LIBOMP_USE_QUAD_PRECISION}")
   libomp_say("Use TSAN-support     -- ${LIBOMP_TSAN_SUPPORT}")
   libomp_say("Use Hwloc library    -- ${LIBOMP_USE_HWLOC}")
 endif()
 
 add_subdirectory(src)
 add_subdirectory(test)
+
+# make these variables available for tools:
+set(LIBOMP_LIBRARY_DIR ${LIBOMP_LIBRARY_DIR} PARENT_SCOPE)
+set(LIBOMP_INCLUDE_DIR ${LIBOMP_INCLUDE_DIR} PARENT_SCOPE)

runtime/src/CMakeLists.txt

else()
   get_target_property(LIBOMP_LIBRARY_DIR omp RUNTIME_OUTPUT_DIRECTORY)
 endif()
 if(NOT LIBOMP_LIBRARY_DIR)
   set(LIBOMP_LIBRARY_DIR ${CMAKE_CURRENT_BINARY_DIR})
   set(LIBOMP_LIBRARY_DIR ${CMAKE_CURRENT_BINARY_DIR} PARENT_SCOPE)
 else()
   set(LIBOMP_LIBRARY_DIR ${LIBOMP_LIBRARY_DIR} PARENT_SCOPE)
 endif()
+set(LIBOMP_INCLUDE_DIR ${CMAKE_CURRENT_BINARY_DIR} PARENT_SCOPE)
 
 # Add symbolic links to libomp
 if(NOT WIN32)
   add_custom_command(TARGET omp POST_BUILD
     COMMAND ${CMAKE_COMMAND} -E create_symlink ${LIBOMP_LIB_FILE}
       libgomp${LIBOMP_LIBRARY_SUFFIX}
     COMMAND ${CMAKE_COMMAND} -E create_symlink ${LIBOMP_LIB_FILE}
       libiomp5${LIBOMP_LIBRARY_SUFFIX}

tools/CMakeLists.txt

+# Discover the tools that use CMake in the subdirectories.
+# Note that explicit cmake invocation is required every time a new project
+# is added or removed.
+file(GLOB entries *)
+foreach(entry ${entries})
+  if(IS_DIRECTORY ${entry} AND EXISTS ${entry}/CMakeLists.txt)
+    if((NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/dummy))
+      add_subdirectory(${entry})
+    endif()
+  endif()
+endforeach(entry)

tools/archer/CMakeLists.txt

+#
+# ompt-tsan.cpp -- Archer runtime library, TSan annotations for Archer
+#
+#
+# //===----------------------------------------------------------------------===//
+# //
+# //                     The LLVM Compiler Infrastructure
+# //
+# // This file is licensed under the LLNS/LLNL "BSD-Like" license.
+# // See tools/archer/LICENSE.txt for details.
+# //
+# //===----------------------------------------------------------------------===//
+  
+  
+
+include_directories(${LIBOMP_INCLUDE_DIR})
+
+add_library(archer SHARED ompt-tsan.cpp counter.cpp ftsan.c)
+add_library(archer_static STATIC ompt-tsan.cpp counter.cpp ftsan.c)
+
+install(TARGETS archer archer_static
+  LIBRARY DESTINATION lib
+  ARCHIVE DESTINATION lib)
+
+add_subdirectory(tests)

tools/archer/counter.h

+/*
+ * counter.cpp -- Archer runtime library, counter header file
+ */
+  
+  //===----------------------------------------------------------------------===//
+  //
+  //                     The LLVM Compiler Infrastructure
+  //
+  // This file is licensed under the LLNS/LLNL "BSD-Like" license.
+  // See tools/archer/LICENSE.txt for details.
+  //
+  //===----------------------------------------------------------------------===//
+  
+  
+
+#include <stdio.h>
+#include <inttypes.h>
+#include <omp.h>
+#if !defined(__powerpc64__)
+#include <ompt.h>
+#endif
+
+// 240 should be enough for xeon-phi
+#define MAX_THREADS 240
+
+#define CACHE_LINE 128
+
+#define COUNT_EVENT1(name)                                                     \
+  if (this_event_counter)                                                      \
+  this_event_counter->name++
+#define COUNT_EVENT2(name, scope)                                              \
+  if (this_event_counter)                                                      \
+  this_event_counter->name##_##scope++
+#define COUNT_EVENT3(name, scope, kind)                                        \
+  if (this_event_counter)                                                      \
+  this_event_counter->name##_##scope##_##kind++
+
+typedef struct alignas(128) {
+  int thread_begin; // (1)  thread_begin
+  int thread_end; // (2)  thread_end
+  int parallel_begin; // (3)  parallel_begin
+  int parallel_end; // (4)  parallel_end
+  int task_create_initial; // (5) task_create:  task_initial
+  int task_create_explicit; //      task_explicit
+  int task_create_target; //      task_target
+  int task_create_included; //      task_included
+  int task_create_untied; //      task_untied
+  int task_schedule; // (6) task_schedule
+  int implicit_task_scope_begin; // (7) implicit task:  scope_begin
+  int implicit_task_scope_end; //     scope_end
+  int mutex_released_lock; // (15) mutex_released:  mutex_lock
+  int mutex_released_nest_lock; //      mutex_nest_lock
+  int mutex_released_critical; //     mutex_critical
+  int mutex_released_atomic; //     mutex_atomic
+  int mutex_released_ordered; //      mutex_ordered
+  int mutex_released_default; //      default
+  int task_dependences; // (16) task_dependences
+  int task_dependence; // (17) task_dependence
+  int sync_region_scope_begin_barrier; // (21) sync_region: scope_begin:
+                                       // sync_region_barrier
+  int sync_region_scope_begin_taskwait; //                                      sync_region_taskwait
+  int sync_region_scope_begin_taskgroup; //                                      sync_region_taskgroup
+  int sync_region_scope_end_barrier; //                   scope_end:
+                                     //                  sync_region_barrier
+  int sync_region_scope_end_taskwait; //                                      sync_region_taskwait
+  int sync_region_scope_end_taskgroup; //                                      sync_region_taskgroup
+  int lock_init_lock; // (22) lock_init:  mutex_lock
+  int lock_init_nest_lock; //       mutex_nest_lock
+  int lock_init_default; //     default
+  int lock_destroy_lock; // (23) lock_destroy mutex_lock
+  int lock_destroy_nest_lock; //      mutex_nest_lock
+  int lock_destroy_default; //      default
+  int mutex_acquire_lock; // (24) mutex_acquire:  mutex_lock
+  int mutex_acquire_nest_lock; //       mutex_nest_lock
+  int mutex_acquire_critical; //      mutex_critical
+  int mutex_acquire_atomic; //      mutex_atomic
+  int mutex_acquire_ordered; //     mutex_ordered
+  int mutex_acquire_default; //     default
+  int mutex_acquired_lock; // (25) mutex_acquired: mutex_lock
+  int mutex_acquired_nest_lock; //                      mutex_nest_lock
+  int mutex_acquired_critical; //                      mutex_critical
+  int mutex_acquired_atomic; //                      mutex_atomic
+  int mutex_acquired_ordered; //                      mutex_ordered
+  int mutex_acquired_default; //      default
+  int nest_lock_scope_begin; // (26) nest_lock: scope_begin
+  int nest_lock_scope_end; //     scope_end
+  int flush; // (27) flush
+} callback_counter_t;
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+void print_callbacks(callback_counter_t *counter);
+
+#ifdef __cplusplus
+}
+#endif

tools/archer/counter.cpp

+/*
+ * counter.cpp -- Archer runtime library, counting callbacks
+ */
+  
+  //===----------------------------------------------------------------------===//
+  //
+  //                     The LLVM Compiler Infrastructure
+  //
+  // This file is licensed under the LLNS/LLNL "BSD-Like" license.
+  // See tools/archer/LICENSE.txt for details.
+  //
+  //===----------------------------------------------------------------------===//
+  
+  
+
+#include "counter.h"
+
+#define OUTPUT_IF_NOT_NULL(format, value)                                      \
+  if (value)                                                                   \
+  printf(format, value)
+
+void print_callbacks(callback_counter_t *counter) {
+  int *basecounter = (int *)counter;
+  int total_callbacks = 0;
+  for (int i = 1; i < MAX_THREADS; i++) {
+    // if we have at least i threads, this thread should have a thread_begin
+    // event:
+    if (counter[i].thread_begin == 0)
+      break;
+    int *threadcounter = (int *)(counter + i);
+    for (int j = 0; j < (int)(sizeof(callback_counter_t) / sizeof(int)); j++)
+      basecounter[j] += threadcounter[j];
+  }
+
+  for (int j = 0; j < (int)(sizeof(callback_counter_t) / sizeof(int)); j++)
+    total_callbacks += basecounter[j];
+
+  printf("Total callbacks: %d\n", total_callbacks);
+  printf("--------------------------------------\n");
+  OUTPUT_IF_NOT_NULL("%5d thread_begin\n", counter[0].thread_begin);
+  OUTPUT_IF_NOT_NULL("%5d thread_end\n", counter[0].thread_end);
+  OUTPUT_IF_NOT_NULL("%5d parallel_begin\n", counter[0].parallel_begin);
+  OUTPUT_IF_NOT_NULL("%5d parallel_end\n", counter[0].parallel_end);
+  OUTPUT_IF_NOT_NULL("%5d task_create : initial\n",
+                     counter[0].task_create_initial);
+  OUTPUT_IF_NOT_NULL("%5d task_create : explicit\n",
+                     counter[0].task_create_explicit);
+  OUTPUT_IF_NOT_NULL("%5d task_create : target\n",
+                     counter[0].task_create_target);
+  OUTPUT_IF_NOT_NULL("%5d task_create : included\n",
+                     counter[0].task_create_included);
+  OUTPUT_IF_NOT_NULL("%5d task_create : untied\n",
+                     counter[0].task_create_untied);
+  OUTPUT_IF_NOT_NULL("%5d task_schedule\n", counter[0].task_schedule);
+  OUTPUT_IF_NOT_NULL("%5d implicit_task : scope_begin\n",
+                     counter[0].implicit_task_scope_begin);
+  OUTPUT_IF_NOT_NULL("%5d implicit_task : scope_end\n",
+                     counter[0].implicit_task_scope_end);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_lock\n",
+                     counter[0].mutex_released_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_nest_lock\n",
+                     counter[0].mutex_released_nest_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_critical\n",
+                     counter[0].mutex_released_critical);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_atomic\n",
+                     counter[0].mutex_released_atomic);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_ordered\n",
+                     counter[0].mutex_released_ordered);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_default\n",
+                     counter[0].mutex_released_default);
+  OUTPUT_IF_NOT_NULL("%5d task_dependences\n", counter[0].task_dependences);
+  OUTPUT_IF_NOT_NULL("%5d task_dependence\n", counter[0].task_dependence);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_begin : barrier\n",
+                     counter[0].sync_region_scope_begin_barrier);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_begin : taskwait\n",
+                     counter[0].sync_region_scope_begin_taskwait);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_begin : taskgroup\n",
+                     counter[0].sync_region_scope_begin_taskgroup);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_end : barrier\n",
+                     counter[0].sync_region_scope_end_barrier);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_end : taskwait\n",
+                     counter[0].sync_region_scope_end_taskwait);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_end : taskgroup\n",
+                     counter[0].sync_region_scope_end_taskgroup);
+  OUTPUT_IF_NOT_NULL("%5d lock_init_lock\n", counter[0].lock_init_lock);
+  OUTPUT_IF_NOT_NULL("%5d lock_init_nest_lock\n",
+                     counter[0].lock_init_nest_lock);
+  OUTPUT_IF_NOT_NULL("%5d lock_init_default\n", counter[0].lock_init_default);
+  OUTPUT_IF_NOT_NULL("%5d lock_destroy_lock\n", counter[0].lock_destroy_lock);
+  OUTPUT_IF_NOT_NULL("%5d lock_destroy_nest_lock\n",
+                     counter[0].lock_destroy_nest_lock);
+  OUTPUT_IF_NOT_NULL("%5d lock_destroy_default\n",
+                     counter[0].lock_destroy_default);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_lock\n", counter[0].mutex_acquire_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_nest_lock\n",
+                     counter[0].mutex_acquire_nest_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_critical\n",
+                     counter[0].mutex_acquire_critical);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_atomic\n",
+                     counter[0].mutex_acquire_atomic);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_ordered\n",
+                     counter[0].mutex_acquire_ordered);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_default\n",
+                     counter[0].mutex_acquire_default);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_lock\n",
+                     counter[0].mutex_acquired_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_nest_lock\n",
+                     counter[0].mutex_acquired_nest_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_critical\n",
+                     counter[0].mutex_acquired_critical);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_atomic\n",
+                     counter[0].mutex_acquired_atomic);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_ordered\n",
+                     counter[0].mutex_acquired_ordered);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_default\n",
+                     counter[0].mutex_acquired_default);
+  OUTPUT_IF_NOT_NULL("%5d nest_lock_scope_begin\n",
+                     counter[0].nest_lock_scope_begin);
+  OUTPUT_IF_NOT_NULL("%5d nest_lock_scope_end\n",
+                     counter[0].nest_lock_scope_end);
+  OUTPUT_IF_NOT_NULL("%5d flush\n", counter[0].flush);
+
+  return;
+}

tools/archer/ftsan.c

+/*
+ * ftsan.c -- Archer runtime library, TSan annotations for Fortran
+ */
+  
+  //===----------------------------------------------------------------------===//
+  //
+  //                     The LLVM Compiler Infrastructure
+  //
+  // This file is licensed under the LLNS/LLNL "BSD-Like" license.
+  // See tools/archer/LICENSE.txt for details.
+  //
+  //===----------------------------------------------------------------------===//
+  
+  
+
+void __attribute__((weak))
+AnnotateHappensAfter(const char *file, int line, const volatile void *cv) {}
+void __attribute__((weak))
+AnnotateHappensBefore(const char *file, int line, const volatile void *cv) {}
+void __attribute__((weak))
+AnnotateIgnoreWritesBegin(const char *file, int line) {}
+void __attribute__((weak)) AnnotateIgnoreWritesEnd(const char *file, int line) {
+}
+
+// This is a Fortran wrapper for TSan Annotation functions.
+
+void annotatehappensafter_(const char *file, int *line,
+                           const volatile void *cv) {
+  AnnotateHappensAfter(file, *line, cv);
+}
+void annotatehappensbefore_(const char *file, int *line,
+                            const volatile void *cv) {
+  AnnotateHappensBefore(file, *line, cv);
+}
+void annotateignorewritesbegin_(const char *file, int line) {
+  AnnotateIgnoreWritesBegin(file, line);
+}
+void annotateignorewritesend_(const char *file, int line) {
+  AnnotateIgnoreWritesEnd(file, line);
+}

tools/archer/ompt-tsan.cpp

+/*
+ * ompt-tsan.cpp -- Archer runtime library, TSan annotations for Archer
+ */
+  
+  //===----------------------------------------------------------------------===//
+  //
+  //                     The LLVM Compiler Infrastructure
+  //
+  // This file is licensed under the LLNS/LLNL "BSD-Like" license.
+  // See tools/archer/LICENSE.txt for details.
+  //
+  //===----------------------------------------------------------------------===//
+
+
+#include "counter.h"
+
+#ifndef __STDC_FORMAT_MACROS
+#define __STDC_FORMAT_MACROS
+#endif
+
+#include <atomic>
+#include <cassert>
+#include <cstdlib>
+#include <cstring>
+#include <inttypes.h>
+#include <iostream>
+#include <mutex>
+#include <sstream>
+#include <stack>
+#include <list>
+#include <string>
+#include <iostream>
+#include <unordered_map>
+#include <vector>
+
+#if (defined __APPLE__ && defined __MACH__)
+#include <dlfcn.h>
+#endif
+
+#include <sys/resource.h>
+#define _OPENMP
+#include "omp.h"
+#if !defined(__powerpc64__)
+#include <ompt.h>
+#endif
+
+callback_counter_t *all_counter;
+__thread callback_counter_t *this_event_counter;
+static int runOnTsan = 1;
+
+class ArcherFlags {
+public:
+#if (LLVM_VERSION) >= 40
+  int flush_shadow;
+#endif
+  int print_ompt_counters;
+  int print_max_rss;
+
+  ArcherFlags(const char *env)
+      :
+#if (LLVM_VERSION) >= 40
+        flush_shadow(0),
+#endif
+        print_ompt_counters(0), print_max_rss(0) {
+    if (env) {
+      std::vector<std::string> tokens;
+      std::string token;
+      std::string str(env);
+      std::istringstream iss(str);
+      while (std::getline(iss, token, ' '))
+        tokens.push_back(token);
+
+      int ret;
+      for (std::vector<std::string>::iterator it = tokens.begin();
+           it != tokens.end(); ++it) {
+#if (LLVM_VERSION) >= 40
+        ret = sscanf(it->c_str(), "flush_shadow=%d", &flush_shadow);
+#endif
+        ret =
+            sscanf(it->c_str(), "print_ompt_counters=%d", &print_ompt_counters);
+        ret = sscanf(it->c_str(), "print_max_rss=%d", &print_max_rss);
+        if (ret) {
+          std::cerr << "Illegal values for ARCHER_OPTIONS variable: " << token
+                    << std::endl;
+        }
+      }
+    }
+  }
+};
+
+#if (LLVM_VERSION) >= 40
+extern "C" {
+int __attribute__((weak)) __archer_get_omp_status();
+void __attribute__((weak)) __tsan_flush_memory() {}
+}
+#endif
+ArcherFlags *archer_flags;
+
+// The following definitions are pasted from "llvm/Support/Compiler.h" to allow
+// the code
+// to be compiled with other compilers like gcc:
+
+#ifndef TsanHappensBefore
+// Thread Sanitizer is a tool that finds races in code.
+// See http://code.google.com/p/data-race-test/wiki/DynamicAnnotations .
+// tsan detects these exact functions by name.
+extern "C" {
+#if (defined __APPLE__ && defined __MACH__)
+static void AnnotateHappensAfter(const char *file, int line,
+                                 const volatile void *cv) {
+  void (*fptr)(const char *, int, const volatile void *);
+
+  fptr = (void (*)(const char *, int, const volatile void *))dlsym(
+      RTLD_DEFAULT, "AnnotateHappensAfter");
+  (*fptr)(file, line, cv);
+}
+static void AnnotateHappensBefore(const char *file, int line,
+                                  const volatile void *cv) {
+  void (*fptr)(const char *, int, const volatile void *);
+
+  fptr = (void (*)(const char *, int, const volatile void *))dlsym(
+      RTLD_DEFAULT, "AnnotateHappensBefore");
+  (*fptr)(file, line, cv);
+}
+static void AnnotateIgnoreWritesBegin(const char *file, int line) {
+  void (*fptr)(const char *, int);
+
+  fptr = (void (*)(const char *, int))dlsym(RTLD_DEFAULT,
+                                            "AnnotateIgnoreWritesBegin");
+  (*fptr)(file, line);
+}
+static void AnnotateIgnoreWritesEnd(const char *file, int line) {
+  void (*fptr)(const char *, int);
+
+  fptr = (void (*)(const char *, int))dlsym(RTLD_DEFAULT,
+                                            "AnnotateIgnoreWritesEnd");
+  (*fptr)(file, line);
+}
+static void AnnotateNewMemory(const char *file, int line,
+                              const volatile void *cv, size_t size) {
+  void (*fptr)(const char *, int, const volatile void *, size_t);
+
+  fptr = (void (*)(const char *, int, const volatile void *, size_t))dlsym(
+      RTLD_DEFAULT, "AnnotateNewMemory");
+  (*fptr)(file, line, cv, size);
+}
+static int RunningOnValgrind() {
+  int (*fptr)();
+
+  fptr = (int (*)())dlsym(RTLD_DEFAULT, "RunningOnValgrind");
+  if (fptr && fptr != RunningOnValgrind)
+    runOnTsan = 0;
+  return 0;
+}
+#else
+void __attribute__((weak))
+AnnotateHappensAfter(const char *file, int line, const volatile void *cv) {}
+void __attribute__((weak))
+AnnotateHappensBefore(const char *file, int line, const volatile void *cv) {}
+void __attribute__((weak))
+AnnotateIgnoreWritesBegin(const char *file, int line) {}
+void __attribute__((weak)) AnnotateIgnoreWritesEnd(const char *file, int line) {
+}
+void __attribute__((weak))
+AnnotateNewMemory(const char *file, int line, const volatile void *cv,
+                  size_t size) {}
+int __attribute__((weak)) RunningOnValgrind() {
+  runOnTsan = 0;
+  return 0;
+}
+#endif
+}
+
+// This marker is used to define a happens-before arc. The race detector will
+// infer an arc from the begin to the end when they share the same pointer
+// argument.
+#define TsanHappensBefore(cv) AnnotateHappensBefore(__FILE__, __LINE__, cv)
+
+// This marker defines the destination of a happens-before arc.
+#define TsanHappensAfter(cv) AnnotateHappensAfter(__FILE__, __LINE__, cv)
+
+// Ignore any races on writes between here and the next TsanIgnoreWritesEnd.
+#define TsanIgnoreWritesBegin() AnnotateIgnoreWritesBegin(__FILE__, __LINE__)
+
+// Resume checking for racy writes.
+#define TsanIgnoreWritesEnd() AnnotateIgnoreWritesEnd(__FILE__, __LINE__)
+
+// We don't really delete the clock for now
+#define TsanDeleteClock(cv)
+
+// newMemory
+#define TsanNewMemory(addr, size)                                              \
+  AnnotateNewMemory(__FILE__, __LINE__, addr, size)
+#define TsanFreeMemory(addr, size)                                             \
+  AnnotateNewMemory(__FILE__, __LINE__, addr, size)
+#endif
+
+/// Required OMPT inquiry functions.
+static ompt_get_parallel_info_t ompt_get_parallel_info;
+static ompt_get_thread_data_t ompt_get_thread_data;
+
+typedef int (*ompt_get_task_memory_t)(void **addr, size_t *size, int blocknum);
+
+typedef uint64_t ompt_tsan_clockid;
+
+static uint64_t my_next_id() {
+  static uint64_t ID = 0;
+  uint64_t ret = __sync_fetch_and_add(&ID, 1);
+  return ret;
+}
+
+// Data structure to provide a threadsafe pool of reusable objects.
+// DataPool<Type of objects, Size of blockalloc>
+template <typename T, int N> struct DataPool {
+  std::mutex DPMutex;
+  std::stack<T *> DataPointer;
+  std::list<void *> memory;
+  int total;
+
+  void newDatas() {
+    // prefix the Data with a pointer to 'this', allows to return memory to
+    // 'this',
+    // without explicitly knowing the source.
+    //
+    // To reduce lock contention, we use thread local DataPools, but Data
+    // objects move to other threads.
+    // The strategy is to get objects from local pool. Only if the object moved
+    // to another
+    // thread, we might see a penalty on release (returnData).
+    // For "single producer" pattern, a single thread creates tasks, these are
+    // executed by other threads.
+    // The master will have a high demand on TaskData, so return after use.
+    struct pooldata {
+      DataPool<T, N> *dp;
+      T data;
+    };
+    // We alloc without initialize the memory. We cannot call constructors.
+    // Therfore use malloc!
+    pooldata *datas = (pooldata *)malloc(sizeof(pooldata) * N);
+    memory.push_back(datas);
+    for (int i = 0; i < N; i++) {
+      datas[i].dp = this;
+      DataPointer.push(&(datas[i].data));
+    }
+    total += N;
+  }
+
+  T *getData() {
+    T *ret;
+    DPMutex.lock();
+    if (DataPointer.empty())
+      newDatas();
+    ret = DataPointer.top();
+    DataPointer.pop();
+    DPMutex.unlock();
+    return ret;
+  }
+
+  void returnData(T *data) {
+    DPMutex.lock();
+    DataPointer.push(data);
+    DPMutex.unlock();
+  }
+
+  void getDatas(int n, T **datas) {
+    DPMutex.lock();
+    for (int i = 0; i < n; i++) {
+      if (DataPointer.empty())
+        newDatas();
+      datas[i] = DataPointer.top();
+      DataPointer.pop();
+    }
+    DPMutex.unlock();
+  }
+
+  void returnDatas(int n, T **datas) {
+    DPMutex.lock();
+    for (int i = 0; i < n; i++) {
+      DataPointer.push(datas[i]);
+    }
+    DPMutex.unlock();
+  }
+
+  DataPool() : DPMutex(), DataPointer(), total(0) {}
+
+  ~DataPool() {
+    // we assume all memory is returned when the thread finished / destructor is
+    // called
+    for (auto i : memory)
+      if (i)
+        free(i);
+  }
+};
+
+// This function takes care to return the data to the originating DataPool
+// A pointer to the originating DataPool is stored just before the actual data.
+template <typename T, int N> static void retData(void *data) {
+  ((DataPool<T, N> **)data)[-1]->returnData((T *)data);
+}
+
+struct ParallelData;
+__thread DataPool<ParallelData, 4> *pdp;
+
+/// Data structure to store additional information for parallel regions.
+struct ParallelData {
+
+  // Parallel fork is just another barrier, use Barrier[1]
+
+  /// Two addresses for relationships with barriers.
+  ompt_tsan_clockid Barrier[2];
+
+  void *GetParallelPtr() { return &(Barrier[1]); }
+
+  void *GetBarrierPtr(unsigned Index) { return &(Barrier[Index]); }
+
+  ~ParallelData() {
+    TsanDeleteClock(&(Barrier[0]));
+    TsanDeleteClock(&(Barrier[1]));
+  }
+  // overload new/delete to use DataPool for memory management.
+  void *operator new(size_t size) { return pdp->getData(); }
+  void operator delete(void *p, size_t) { retData<ParallelData, 4>(p); }
+};
+
+static inline ParallelData *ToParallelData(ompt_data_t *parallel_data) {
+  return reinterpret_cast<ParallelData *>(parallel_data->ptr);
+}
+
+struct Taskgroup;
+__thread DataPool<Taskgroup, 4> *tgp;
+
+/// Data structure to support stacking of taskgroups and allow synchronization.
+struct Taskgroup {
+  /// Its address is used for relationships of the taskgroup's task set.
+  ompt_tsan_clockid Ptr;
+
+  /// Reference to the parent taskgroup.
+  Taskgroup *Parent;
+
+  Taskgroup(Taskgroup *Parent) : Parent(Parent) {}
+  ~Taskgroup() { TsanDeleteClock(&Ptr); }
+
+  void *GetPtr() { return &Ptr; }
+  // overload new/delete to use DataPool for memory management.
+  void *operator new(size_t size) { return tgp->getData(); }
+  void operator delete(void *p, size_t) { retData<Taskgroup, 4>(p); }
+};
+
+struct TaskData;
+__thread DataPool<TaskData, 4> *tdp;
+
+/// Data structure to store additional information for tasks.
+struct TaskData {
+  /// Its address is used for relationships of this task.
+  ompt_tsan_clockid Task;
+
+  /// Child tasks use its address to declare a relationship to a taskwait in
+  /// this task.
+  ompt_tsan_clockid Taskwait;
+
+  /// Whether this task is currently executing a barrier.
+  bool InBarrier;
+
+  /// Whether this task is currently executing a barrier.
+  bool Included;
+
+  /// Index of which barrier to use next.
+  char BarrierIndex;
+
+  /// Count how often this structure has been put into child tasks + 1.
+  std::atomic_int RefCount;
+
+  /// Reference to the parent that created this task.
+  TaskData *Parent;
+
+  /// Reference to the implicit task in the stack above this task.
+  TaskData *ImplicitTask;
+
+  /// Reference to the team of this task.
+  ParallelData *Team;
+
+  /// Reference to the current taskgroup that this task either belongs to or
+  /// that it just created.
+  Taskgroup *TaskGroup;
+
+  /// Dependency information for this task.
+  ompt_task_dependence_t *Dependencies;
+
+  /// Number of dependency entries.
+  unsigned DependencyCount;
+
+  void *PrivateData;
+  size_t PrivateDataSize;
+
+  int execution;
+  int freed;
+
+  TaskData(TaskData *Parent)
+      : InBarrier(false), Included(false), BarrierIndex(0), RefCount(1),
+        Parent(Parent), ImplicitTask(nullptr), Team(Parent->Team),
+        TaskGroup(nullptr), DependencyCount(0), execution(0), freed(0) {
+    if (Parent != nullptr) {
+      Parent->RefCount++;
+      // Copy over pointer to taskgroup. This task may set up its own stack
+      // but for now belongs to its parent's taskgroup.
+      TaskGroup = Parent->TaskGroup;
+    }
+  }
+
+  TaskData(ParallelData *Team = nullptr)
+      : InBarrier(false), Included(false), BarrierIndex(0), RefCount(1),
+        Parent(nullptr), ImplicitTask(this), Team(Team), TaskGroup(nullptr),
+        DependencyCount(0), execution(1), freed(0) {}
+
+  ~TaskData() {
+    TsanDeleteClock(&Task);
+    TsanDeleteClock(&Taskwait);
+  }
+
+  void *GetTaskPtr() { return &Task; }
+
+  void *GetTaskwaitPtr() { return &Taskwait; }
+  // overload new/delete to use DataPool for memory management.
+  void *operator new(size_t size) { return tdp->getData(); }
+  void operator delete(void *p, size_t) { retData<TaskData, 4>(p); }
+};
+
+static inline TaskData *ToTaskData(ompt_data_t *task_data) {
+  return reinterpret_cast<TaskData *>(task_data->ptr);
+}
+
+static inline void *ToInAddr(void *OutAddr) {
+  // FIXME: This will give false negatives when a second variable lays directly
+  //        behind a variable that only has a width of 1 byte.
+  //        Another approach would be to "negate" the address or to flip the
+  //        first bit...
+  return reinterpret_cast<char *>(OutAddr) + 1;
+}
+
+/// Store a mutex for each wait_id to resolve race condition with callbacks.
+std::unordered_map<ompt_wait_id_t, std::mutex> Locks;
+std::mutex LocksMutex;
+
+static void ompt_tsan_thread_begin(ompt_thread_type_t thread_type,
+                                   ompt_data_t *thread_data) {
+  pdp = new DataPool<ParallelData, 4>;
+  TsanNewMemory(pdp, sizeof(pdp));
+  tgp = new DataPool<Taskgroup, 4>;
+  TsanNewMemory(tgp, sizeof(tgp));
+  tdp = new DataPool<TaskData, 4>;
+  TsanNewMemory(tdp, sizeof(tdp));
+  thread_data->value = my_next_id();
+  if (archer_flags->print_ompt_counters && thread_data->value < MAX_THREADS)
+    this_event_counter = &(all_counter[thread_data->value]);
+  else
+    this_event_counter = NULL;
+  COUNT_EVENT1(thread_begin);
+}
+
+static void ompt_tsan_thread_end(ompt_data_t *thread_data) {
+  delete pdp;
+  delete tgp;
+  delete tdp;
+  COUNT_EVENT1(thread_end);
+}
+
+/// OMPT event callbacks for handling parallel regions.
+
+static void ompt_tsan_parallel_begin(ompt_data_t *parent_task_data,
+                                     const ompt_frame_t *parent_task_frame,
+                                     ompt_data_t *parallel_data,
+                                     uint32_t requested_team_size,
+                                     //  uint32_t actual_team_size,
+                                     ompt_invoker_t invoker,
+                                     const void *codeptr_ra) {
+  ParallelData *Data = new ParallelData;
+  parallel_data->ptr = Data;
+
+  TsanHappensBefore(Data->GetParallelPtr());
+  COUNT_EVENT1(parallel_begin);
+}
+
+static void ompt_tsan_parallel_end(ompt_data_t *parallel_data,
+                                   ompt_data_t *task_data,
+                                   ompt_invoker_t invoker,
+                                   const void *codeptr_ra) {
+  ParallelData *Data = ToParallelData(parallel_data);
+  TsanHappensAfter(Data->GetBarrierPtr(0));
+  TsanHappensAfter(Data->GetBarrierPtr(1));
+
+  delete Data;
+
+#if (LLVM_VERSION >= 40)
+  if (&__archer_get_omp_status) {
+    if (__archer_get_omp_status() == 0 && archer_flags->flush_shadow)
+      __tsan_flush_memory();
+  }
+#endif
+
+  COUNT_EVENT1(parallel_end);
+}
+
+static void ompt_tsan_implicit_task(ompt_scope_endpoint_t endpoint,
+                                    ompt_data_t *parallel_data,
+                                    ompt_data_t *task_data,
+                                    unsigned int team_size,
+                                    unsigned int thread_num) {
+  switch (endpoint) {
+  case ompt_scope_begin:
+    task_data->ptr = new TaskData(ToParallelData(parallel_data));
+    TsanHappensAfter(ToParallelData(parallel_data)->GetParallelPtr());
+    COUNT_EVENT2(implicit_task, scope_begin);
+    break;
+  case ompt_scope_end:
+    TaskData *Data = ToTaskData(task_data);
+    assert(Data->freed == 0 && "Implicit task end should only be called once!");
+    Data->freed = 1;
+    assert(Data->RefCount == 1 &&
+           "All tasks should have finished at the implicit barrier!");
+    delete Data;
+    COUNT_EVENT2(implicit_task, scope_end);
+    break;
+  }
+}
+
+static void ompt_tsan_sync_region(ompt_sync_region_kind_t kind,
+                                  ompt_scope_endpoint_t endpoint,
+                                  ompt_data_t *parallel_data,
+                                  ompt_data_t *task_data,
+                                  const void *codeptr_ra) {
+  TaskData *Data = ToTaskData(task_data);
+  switch (endpoint) {
+  case ompt_scope_begin:
+    switch (kind) {
+    case ompt_sync_region_barrier: {
+      char BarrierIndex = Data->BarrierIndex;
+      TsanHappensBefore(Data->Team->GetBarrierPtr(BarrierIndex));
+
+      // We ignore writes inside the barrier. These would either occur during
+      // 1. reductions performed by the runtime which are guaranteed to be
+      // race-free.
+      // 2. execution of another task.
+      // For the latter case we will re-enable tracking in task_switch.
+      Data->InBarrier = true;
+      TsanIgnoreWritesBegin();
+      COUNT_EVENT3(sync_region, scope_begin, barrier);
+      break;
+    }
+    case ompt_sync_region_taskwait:
+
+      COUNT_EVENT3(sync_region, scope_begin, taskwait);
+      break;
+    case ompt_sync_region_taskgroup:
+      Data->TaskGroup = new Taskgroup(Data->TaskGroup);
+      COUNT_EVENT3(sync_region, scope_begin, taskgroup);
+      break;
+    }
+    break;
+  case ompt_scope_end:
+    switch (kind) {
+    case ompt_sync_region_barrier: {
+      // We want to track writes after the barrier again.
+      Data->InBarrier = false;
+      TsanIgnoreWritesEnd();
+
+      char BarrierIndex = Data->BarrierIndex;
+      // Barrier will end after it has been entered by all threads.
+      if (parallel_data)
+        TsanHappensAfter(Data->Team->GetBarrierPtr(BarrierIndex));
+
+      // It is not guaranteed that all threads have exited this barrier before
+      // we enter the next one. So we will use a different address.
+      // We are however guaranteed that this current barrier is finished
+      // by the time we exit the next one. So we can then reuse the first
+      // address.
+      Data->BarrierIndex = (BarrierIndex + 1) % 2;
+      COUNT_EVENT3(sync_region, scope_end, barrier);
+      break;
+    }
+    case ompt_sync_region_taskwait: {
+      COUNT_EVENT3(sync_region, scope_end, taskwait);
+      if (Data->execution > 1)
+        TsanHappensAfter(Data->GetTaskwaitPtr());
+      break;
+    }
+    case ompt_sync_region_taskgroup: {
+      assert(Data->TaskGroup != nullptr &&
+             "Should have at least one taskgroup!");
+
+      TsanHappensAfter(Data->TaskGroup->GetPtr());
+
+      // Delete this allocated taskgroup, all descendent task are finished by
+      // now.
+      Taskgroup *Parent = Data->TaskGroup->Parent;
+      delete Data->TaskGroup;
+      Data->TaskGroup = Parent;
+      COUNT_EVENT3(sync_region, scope_end, taskgroup);
+      break;
+    }
+    }
+    break;
+  }
+}
+
+/// OMPT event callbacks for handling tasks.
+
+static void ompt_tsan_task_create(
+    ompt_data_t *parent_task_data, /* id of parent task            */
+    const ompt_frame_t *parent_frame, /* frame data for parent task   */
+    ompt_data_t *new_task_data, /* id of created task           */
+    int type, int has_dependences,
+    const void *codeptr_ra) /* pointer to outlined function */
+{
+  TaskData *Data;
+  assert(new_task_data->ptr == NULL &&
+         "Task data should be initialized to NULL");
+  if (type & ompt_task_initial) {
+    ompt_data_t *parallel_data;
+    int team_size = 1;
+    ompt_get_parallel_info(0, &parallel_data, &team_size);
+    ParallelData *PData = new ParallelData;
+    parallel_data->ptr = PData;
+
+    Data = new TaskData(PData);
+    new_task_data->ptr = Data;
+    COUNT_EVENT2(task_create, initial);
+  } else if (type & ompt_task_undeferred) {
+    Data = new TaskData(ToTaskData(parent_task_data));
+    new_task_data->ptr = Data;
+    Data->Included = true;
+    COUNT_EVENT2(task_create, included);
+  } else if (type & ompt_task_explicit || type & ompt_task_target) {
+    Data = new TaskData(ToTaskData(parent_task_data));
+    new_task_data->ptr = Data;
+
+    // Use the newly created address. We cannot use a single address from the
+    // parent because that would declare wrong relationships with other
+    // sibling tasks that may be created before this task is started!
+    TsanHappensBefore(Data->GetTaskPtr());
+    ToTaskData(parent_task_data)->execution++;
+    COUNT_EVENT2(task_create, explicit);
+  }
+}
+
+static void ompt_tsan_task_schedule(ompt_data_t *first_task_data,
+                                    ompt_task_status_t prior_task_status,
+                                    ompt_data_t *second_task_data) {
+  COUNT_EVENT1(task_schedule);
+  TaskData *FromTask = ToTaskData(first_task_data);
+  TaskData *ToTask = ToTaskData(second_task_data);
+
+  if (ToTask->Included && prior_task_status != ompt_task_complete)
+    return; // No further synchronization for begin included tasks
+  if (FromTask->Included && prior_task_status == ompt_task_complete) {
+    // Just delete the task:
+    while (FromTask != nullptr && --FromTask->RefCount == 0) {
+      TaskData *Parent = FromTask->Parent;
+      if (FromTask->DependencyCount > 0) {
+        delete[] FromTask->Dependencies;
+      }
+      delete FromTask;
+      FromTask = Parent;
+    }
+    return;
+  }
+
+  // we will use new stack, assume growing down
+  // TsanNewMemory((char*)&FromTask-1024, 1024);
+  if (ToTask->execution == 0) {
+    ToTask->execution++;
+    // 1. Task will begin execution after it has been created.
+    TsanHappensAfter(ToTask->GetTaskPtr());
+    for (unsigned i = 0; i < ToTask->DependencyCount; i++) {
+      ompt_task_dependence_t *Dependency = &ToTask->Dependencies[i];
+
+      TsanHappensAfter(Dependency->variable_addr);
+      // in and inout dependencies are also blocked by prior in dependencies!
+      if (Dependency->dependence_flags & ompt_task_dependence_type_out) {
+        TsanHappensAfter(ToInAddr(Dependency->variable_addr));
+      }
+    }
+  } else {
+    // 2. Task will resume after it has been switched away.
+    TsanHappensAfter(ToTask->GetTaskPtr());
+  }
+
+  if (prior_task_status != ompt_task_complete) {
+    ToTask->ImplicitTask = FromTask->ImplicitTask;
+    assert(ToTask->ImplicitTask != NULL &&
+           "A task belongs to a team and has an implicit task on the stack");
+  }
+
+  // Task may be resumed at a later point in time.
+  // TsanHappensBeforeUC(FromTask->GetTaskPtr());
+  TsanHappensBefore(FromTask->GetTaskPtr());
+
+  if (FromTask->InBarrier) {
+    // We want to ignore writes in the runtime code during barriers,
+    // but not when executing tasks with user code!
+    TsanIgnoreWritesEnd();
+  }
+
+  if (prior_task_status == ompt_task_complete) { // task finished
+
+    // Task will finish before a barrier in the surrounding parallel region ...
+    ParallelData *PData = FromTask->Team;
+    TsanHappensBefore(
+        PData->GetBarrierPtr(FromTask->ImplicitTask->BarrierIndex));
+
+    // ... and before an eventual taskwait by the parent thread.
+    TsanHappensBefore(FromTask->Parent->GetTaskwaitPtr());
+
+    if (FromTask->TaskGroup != nullptr) {
+      // This task is part of a taskgroup, so it will finish before the
+      // corresponding taskgroup_end.
+      TsanHappensBefore(FromTask->TaskGroup->GetPtr());
+    }
+    for (unsigned i = 0; i < FromTask->DependencyCount; i++) {
+      ompt_task_dependence_t *Dependency = &FromTask->Dependencies[i];
+
+      // in dependencies block following inout and out dependencies!
+      TsanHappensBefore(ToInAddr(Dependency->variable_addr));
+      if (Dependency->dependence_flags & ompt_task_dependence_type_out) {
+        TsanHappensBefore(Dependency->variable_addr);
+      }
+    }
+    while (FromTask != nullptr && --FromTask->RefCount == 0) {
+      TaskData *Parent = FromTask->Parent;
+      if (FromTask->DependencyCount > 0) {
+        delete[] FromTask->Dependencies;
+      }
+      delete FromTask;
+      FromTask = Parent;
+    }
+  }
+  if (ToTask->InBarrier) {
+    // We re-enter runtime code which currently performs a barrier.
+    TsanIgnoreWritesBegin();
+  }
+}
+
+static void ompt_tsan_task_dependences(ompt_data_t *task_data,
+                                       const ompt_task_dependence_t *deps,
+                                       int ndeps) {
+  COUNT_EVENT1(task_dependences);
+  if (ndeps > 0) {
+    // Copy the data to use it in task_switch and task_end.
+    TaskData *Data = ToTaskData(task_data);
+    Data->Dependencies = new ompt_task_dependence_t[ndeps];
+    std::memcpy(Data->Dependencies, deps,
+                sizeof(ompt_task_dependence_t) * ndeps);
+    Data->DependencyCount = ndeps;
+
+    // This callback is executed before this task is first started.
+    TsanHappensBefore(Data->GetTaskPtr());
+  }
+}
+
+/// OMPT event callbacks for handling locking.
+static void ompt_tsan_mutex_acquired(ompt_mutex_kind_t kind,
+                                     ompt_wait_id_t wait_id,
+                                     const void *codeptr_ra) {
+  if (archer_flags->print_ompt_counters)
+    switch (kind) {
+    case ompt_mutex_lock:
+      COUNT_EVENT2(mutex_acquired, lock);
+      break;
+    case ompt_mutex_nest_lock:
+      COUNT_EVENT2(mutex_acquired, nest_lock);
+      break;
+    case ompt_mutex_critical:
+      COUNT_EVENT2(mutex_acquired, critical);
+      break;
+    case ompt_mutex_atomic:
+      COUNT_EVENT2(mutex_acquired, atomic);
+      break;
+    case ompt_mutex_ordered:
+      COUNT_EVENT2(mutex_acquired, ordered);
+      break;
+    default:
+      COUNT_EVENT2(mutex_acquired, default);
+      break;
+    }
+
+  // Acquire our own lock to make sure that
+  // 1. the previous release has finished.
+  // 2. the next acquire doesn't start before we have finished our release.
+  {
+    LocksMutex.lock();
+    std::mutex &Lock = Locks[wait_id];
+    LocksMutex.unlock();
+
+    Lock.lock();
+    TsanHappensAfter(&Lock);
+  }
+}
+
+static void ompt_tsan_mutex_released(ompt_mutex_kind_t kind,
+                                     ompt_wait_id_t wait_id,
+                                     const void *codeptr_ra) {
+  if (archer_flags->print_ompt_counters)
+    switch (kind) {
+    case ompt_mutex_lock:
+      COUNT_EVENT2(mutex_released, lock);
+      break;
+    case ompt_mutex_nest_lock:
+      COUNT_EVENT2(mutex_released, nest_lock);
+      break;
+    case ompt_mutex_critical:
+      COUNT_EVENT2(mutex_released, critical);
+      break;
+    case ompt_mutex_atomic:
+      COUNT_EVENT2(mutex_released, atomic);
+      break;
+    case ompt_mutex_ordered:
+      COUNT_EVENT2(mutex_released, ordered);
+      break;
+    default:
+      COUNT_EVENT2(mutex_released, default);
+      break;
+    }
+
+  {
+    LocksMutex.lock();
+    std::mutex &Lock = Locks[wait_id];
+    LocksMutex.unlock();
+    TsanHappensBefore(&Lock);
+
+    Lock.unlock();
+  }
+}
+
+#define SET_CALLBACK_T(event, type)                                            \
+  do {                                                                         \
+    ompt_callback_##type##_t tsan_##event = &ompt_tsan_##event;                \
+    int ret = ompt_set_callback(ompt_callback_##event,                         \
+                                (ompt_callback_t)tsan_##event);                \
+    if (ret != ompt_set_always)                                                \
+      printf("Registered callback '" #event "' is not always invoked (%i)\n",  \
+             ret);                                                             \
+  } while (0)
+
+#define SET_CALLBACK(event) SET_CALLBACK_T(event, event)
+
+static int ompt_tsan_initialize(ompt_function_lookup_t lookup,
+                                ompt_data_t *tool_data) {
+  const char *options = getenv("ARCHER_OPTIONS");
+  archer_flags = new ArcherFlags(options);
+
+  if (archer_flags->print_ompt_counters)
+    all_counter = new callback_counter_t[MAX_THREADS];
+
+  ompt_set_callback_t ompt_set_callback =
+      (ompt_set_callback_t)lookup("ompt_set_callback");
+  if (ompt_set_callback == NULL) {
+    std::cerr << "Could not set callback, exiting..." << std::endl;
+    std::exit(1);
+  }
+  ompt_get_parallel_info =
+      (ompt_get_parallel_info_t)lookup("ompt_get_parallel_info");
+  ompt_get_thread_data = (ompt_get_thread_data_t)lookup("ompt_get_thread_data");
+
+  if (ompt_get_parallel_info == NULL) {
+    fprintf(stderr, "Could not get inquiry function 'ompt_get_parallel_info', "
+                    "exiting...\n");
+    exit(1);
+  }
+
+  SET_CALLBACK(thread_begin);
+  SET_CALLBACK(thread_end);
+  SET_CALLBACK(parallel_begin);
+  SET_CALLBACK(implicit_task);
+  SET_CALLBACK(sync_region);
+  SET_CALLBACK(parallel_end);
+
+  SET_CALLBACK(task_create);
+  SET_CALLBACK(task_schedule);
+  SET_CALLBACK(task_dependences);
+
+  SET_CALLBACK_T(mutex_acquired, mutex);
+  SET_CALLBACK_T(mutex_released, mutex);
+  return 1; // success
+}
+
+static void ompt_tsan_finalize(ompt_data_t *tool_data) {
+  if (archer_flags->print_ompt_counters) {
+    print_callbacks(all_counter);
+    delete[] all_counter;
+  }
+
+  if (archer_flags->print_max_rss) {
+    struct rusage end;
+    getrusage(RUSAGE_SELF, &end);
+    printf("MAX RSS[KBytes] during execution: %ld\n", end.ru_maxrss);
+  }
+
+  if (archer_flags)
+    delete archer_flags;
+}
+
+ompt_start_tool_result_t *ompt_start_tool(unsigned int omp_version,
+                                          const char *runtime_version) {
+  static ompt_start_tool_result_t ompt_start_tool_result = {
+      &ompt_tsan_initialize, &ompt_tsan_finalize, {0}};
+  RunningOnValgrind();
+  if (!runOnTsan) // if we are not running on TSAN, give a different tool the
+                  // chance to be loaded
+    return NULL;
+
+  return &ompt_start_tool_result;
+}

tools/archer/tests/CMakeLists.txt

+# CMakeLists.txt file for unit testing Archer runtime library.
+include(CheckFunctionExists)
+include(CheckLibraryExists)
+
+# Some tests use math functions
+check_library_exists(m sqrt "" LIBARCHER_HAVE_LIBM)
+# When using libgcc, -latomic may be needed for atomics
+# (but when using compiler-rt, the atomics will be built-in)
+# Note: we can not check for __atomic_load because clang treats it
+# as special built-in and that breaks CMake checks
+check_function_exists(__atomic_load_1 LIBARCHER_HAVE_BUILTIN_ATOMIC)
+if(NOT LIBARCHER_HAVE_BUILTIN_ATOMIC)
+  check_library_exists(atomic __atomic_load_1 "" LIBARCHER_HAVE_LIBATOMIC)
+else()
+  # not needed
+  set(LIBARCHER_HAVE_LIBATOMIC 0)
+endif()
+
+set(LIBARCHER_TEST_PATH ${CMAKE_CURRENT_SOURCE_DIR})
+set(LIBARCHER_HAVE_ARCHER_LIBRARY ${LIBARCHER_LIB_PATH})
+set(LIBARCHER_HAVE_ARCHER_RUNTIME ${LIBARCHER_RUNTIME_PATH})
+
+macro(pythonize_bool var)
+  if (${var})
+    set(${var} True)
+  else()
+    set(${var} False)
+  endif()
+endmacro()
+
+pythonize_bool(LIBARCHER_HAVE_ARCHER_LIBRARY)
+pythonize_bool(LIBARCHER_HAVE_ARCHER_RUNTIME)
+pythonize_bool(LIBOMP_OMPT_SUPPORT)
+pythonize_bool(LIBOMP_OMPT_OPTIONAL)
+pythonize_bool(LIBOMP_HAVE_LIBM)
+pythonize_bool(LIBOMP_HAVE_LIBATOMIC)
+pythonize_bool(LIBOMP_USE_HWLOC)
+
+add_openmp_testsuite(check-libarcher "Running libarcher tests" ${CMAKE_CURRENT_BINARY_DIR} DEPENDS archer)
+
+# Configure the lit.site.cfg.in file
+set(AUTO_GEN_COMMENT "## Autogenerated by libarcher configuration.\n# Do not edit!")
+configure_file(lit.site.cfg.in lit.site.cfg @ONLY)

tools/archer/tests/barrier/barrier.c

+/*
+ * barrier.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    if (omp_get_thread_num() == 0) {
+      var++;
+    }
+
+#pragma omp barrier
+
+    if (omp_get_thread_num() == 1) {
+      var++;
+    }
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/critical/critical.c

+/*
+ * critical.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(8) shared(var)
+  {
+#pragma omp critical
+    { var++; }
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 8);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/critical/lock-nested.c

+/*
+ * lock-nested.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+  omp_nest_lock_t lock;
+  omp_init_nest_lock(&lock);
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    omp_set_nest_lock(&lock);
+    omp_set_nest_lock(&lock);
+    var++;
+    omp_unset_nest_lock(&lock);
+    omp_unset_nest_lock(&lock);
+  }
+
+  omp_destroy_nest_lock(&lock);
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/critical/lock.c

+/*
+ * lock.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+  omp_lock_t lock;
+  omp_init_lock(&lock);
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    omp_set_lock(&lock);
+    var++;
+    omp_unset_lock(&lock);
+  }
+
+  omp_destroy_lock(&lock);
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/deflake.bash

+#!/usr/bin/env bash
+# This script is used to deflake inherently flaky archer tests.
+# It is invoked from lit tests as:
+# %deflake mybinary
+# which is then substituted by lit to:
+# $(dirname %s)/deflake.bash mybinary
+# The script runs the target program up to 10 times,
+# until it fails (i.e. produces a race report).
+
+for i in $(seq 1 10); do
+    OUT=`$@ 2>&1`
+    if [[ $? != 0 ]]; then
+  echo "$OUT"
+  exit 0
+    fi
+done
+exit 1

tools/archer/tests/lit.cfg

+# -*- Python -*- vim: set ft=python ts=4 sw=4 expandtab tw=79:
+# Configuration file for the 'lit' test runner.
+
+import os
+import re
+import subprocess
+import lit.formats
+
+# Tell pylint that we know config and lit_config exist somewhere.
+if 'PYLINT_IMPORT' in os.environ:
+    config = object()
+    lit_config = object()
+
+def append_dynamic_library_path(path):
+    if config.operating_system == 'Windows':
+        name = 'PATH'
+        sep = ';'
+    elif config.operating_system == 'Darwin':
+        name = 'DYLD_LIBRARY_PATH'
+        sep = ':'
+    else:
+        name = 'LD_LIBRARY_PATH'
+        sep = ':'
+    if name in config.environment:
+        config.environment[name] = path + sep + config.environment[name]
+    else:
+        config.environment[name] = path
+
+# name: The name of this test suite.
+config.name = 'libarcher'
+
+# suffixes: A list of file extensions to treat as test files.
+config.suffixes = ['.c', '.cpp']
+
+# test_source_root: The root path where tests are located.
+config.test_source_root = os.path.dirname(__file__)
+
+# test_exec_root: The root object directory where output is placed
+config.test_exec_root = config.libarcher_obj_root
+
+# test format
+config.test_format = lit.formats.ShTest()
+
+# compiler flags
+config.test_flags = " -I " + config.test_source_root + \
+    " -I " + config.omp_header_dir + \
+    " -L " + config.omp_library_dir + \
+    " -Wl,-rpath," + config.omp_library_dir + \
+    " " + config.test_extra_flags
+
+config.archer_flags = "-g -O1 -fsanitize=thread"
+
+libs_archer = ""
+if config.has_archer_runtime:
+    libs_archer += " -L" + config.archer_runtime_dir + " -l" + \
+    config.archer_runtime.replace("lib", "").replace(".so", "").replace(".dy", "") + \
+    " -Wl,-rpath," + config.archer_runtime_dir
+
+# if config.has_archer_library:
+#   config.test_compiler += " -Xclang -load -Xclang " + \
+#     config.archer_library_dir + "/" + config.archer_library + \
+#     " -Wl,-rpath," + config.archer_library_dir
+
+
+# extra libraries
+libs = ""
+if config.has_libm:
+    libs += " -lm"
+if config.has_libatomic:
+    libs += " -latomic"
+
+# Allow XFAIL to work
+config.target_triple = [ ]
+for feature in config.test_compiler_features:
+    config.available_features.add(feature)
+
+# Setup environment to find dynamic library at runtime
+append_dynamic_library_path(config.omp_library_dir)
+
+# Rpath modifications for Darwin
+if config.operating_system == 'Darwin':
+    config.test_flags += " -Wl,-rpath," + config.omp_library_dir
+
+# Find the SDK on Darwin
+if config.operating_system == 'Darwin':
+  cmd = subprocess.Popen(['xcrun', '--show-sdk-path'],
+                         stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+  out, err = cmd.communicate()
+  out = out.strip()
+  res = cmd.wait()
+  if res == 0 and out:
+    config.test_flags += " -isysroot " + out
+
+config.suppression = ""
+# need to deal with the static analysis with another execution line in the test
+# if not config.has_archer_library and config.has_archer_runtime:
+if config.has_archer_runtime:
+  config.suppression = "env TSAN_OPTIONS=\"ignore_noninstrumented_modules=1\""
+
+# Disable OMPT tests if FileCheck was not found
+if config.has_ompt and config.test_filecheck == "":
+    lit_config.note("Not testing OMPT because FileCheck was not found")
+    config.has_ompt = False
+
+if config.has_ompt:
+    config.available_features.add("ompt")
+    # for callback.h
+    config.test_flags += " -I " + config.test_source_root + "/ompt"
+
+if 'Linux' in config.operating_system:
+    config.available_features.add("linux")
+
+# to run with icc INTEL_LICENSE_FILE must be set
+if 'INTEL_LICENSE_FILE' in os.environ:
+    config.environment['INTEL_LICENSE_FILE'] = os.environ['INTEL_LICENSE_FILE']
+
+# OMPT Tests
+config.substitutions.append(("%libomp-compile-and-run", \
+    "%libomp-compile && %libomp-run"))
+config.substitutions.append(("%libomp-cxx-compile-and-run", \
+    "%libomp-cxx-compile && %libomp-run"))
+config.substitutions.append(("%libomp-cxx-compile", \
+    "%clangXX %openmp_flags %flags -std=c++11 %s -o %t" + libs))
+config.substitutions.append(("%libomp-compile", \
+    "%clang %openmp_flags %flags %s -o %t" + libs))
+config.substitutions.append(("%libomp-run", "%t | tee %t.log"))
+config.substitutions.append(("%clangXX", config.test_cxx_compiler))
+config.substitutions.append(("%clang", config.test_c_compiler))
+config.substitutions.append(("%openmp_flags", config.test_openmp_flags))
+config.substitutions.append(("%flags", config.test_flags))
+
+# Race Tests
+config.substitutions.append(("%libarcher-compile-and-run-race", \
+    "%libarcher-compile && %libarcher-run-race"))
+config.substitutions.append(("%libarcher-compile-and-run", \
+                             "%libarcher-compile && %libarcher-run"))
+config.substitutions.append(("%libarcher-cxx-compile-and-run", \
+    "%libarcher-cxx-compile && %libarcher-run"))
+config.substitutions.append(("%libarcher-cxx-compile", \
+    "%clang-archerXX %openmp_flags %archer_flags %flags -std=c++11 %s -o %t" + libs))
+config.substitutions.append(("%libarcher-compile", \
+                             "%clang-archer %openmp_flags %archer_flags %flags %s -o %t" + libs + libs_archer))
+config.substitutions.append(("%libarcher-run-race", "%suppression %deflake %t 2>&1 | tee %t.log"))
+config.substitutions.append(("%libarcher-run", "env OMP_TOOL_LIBRARIES=libarcher.so %suppression %t 2>&1 | tee %t.log"))
+config.substitutions.append(("%clang-archerXX", config.test_cxx_compiler))
+config.substitutions.append(("%clang-archer", config.test_c_compiler))
+config.substitutions.append(("%openmp_flags", config.test_openmp_flags))
+config.substitutions.append(("%archer_flags", config.archer_flags))
+config.substitutions.append(("%flags", config.test_flags))
+config.substitutions.append(("%suppression", "env TSAN_OPTIONS='ignore_noninstrumented_modules=1'"))
+config.substitutions.append(("%deflake", os.path.join(os.path.dirname(__file__), "deflake.bash")))
+
+config.substitutions.append(("FileCheck", config.test_filecheck))
+config.substitutions.append(("%sort-threads", "sort --numeric-sort --stable"))
+if config.operating_system == 'Windows':
+    # No such environment variable on Windows.
+    config.substitutions.append(("%preload-tool", "true ||"))
+elif config.operating_system == 'Darwin':
+    config.substitutions.append(("%preload-tool", "env DYLD_INSERT_LIBRARIES=%T/tool.so"))
+else:
+    config.substitutions.append(("%preload-tool", "env LD_PRELOAD=%T/tool.so"))

tools/archer/tests/lit.site.cfg.in

+@AUTO_GEN_COMMENT@
+
+config.test_c_compiler = "@OPENMP_TEST_C_COMPILER@"
+config.test_cxx_compiler = "@OPENMP_TEST_CXX_COMPILER@"
+config.test_compiler_features = @OPENMP_TEST_COMPILER_FEATURES@
+config.test_filecheck = "@OPENMP_FILECHECK_EXECUTABLE@"
+config.test_openmp_flags = "@OPENMP_TEST_OPENMP_FLAGS@"
+config.test_extra_flags = "@OPENMP_TEST_FLAGS@"
+config.libomp_obj_root = "@CMAKE_CURRENT_BINARY_DIR@"
+config.omp_library_dir = "@LIBOMP_LIBRARY_DIR@"
+config.omp_header_dir = "@LIBOMP_INCLUDE_DIR@"
+config.operating_system = "@CMAKE_SYSTEM_NAME@"
+config.hwloc_library_dir = "@LIBOMP_HWLOC_LIBRARY_DIR@"
+config.using_hwloc = @LIBOMP_USE_HWLOC@
+config.has_ompt = @LIBOMP_OMPT_SUPPORT@ and @LIBOMP_OMPT_OPTIONAL@
+config.has_libm = @LIBOMP_HAVE_LIBM@
+config.has_libatomic = @LIBOMP_HAVE_LIBATOMIC@
+
+config.test_archer_flags = "@OPENMP_TEST_ARCHER_FLAGS@"
+config.libarcher_obj_root = "@CMAKE_CURRENT_BINARY_DIR@"
+
+config.archer_tools_dir = "@LIBARCHER_TOOLS_DIR@"
+config.archer_library_dir = "@LIBARCHER_LIB_PATH@"
+config.archer_runtime_dir = "@LIBARCHER_RUNTIME_PATH@"
+config.archer_library = "@LIBARCHER_LIB@"
+config.archer_runtime = "@LIBARCHER_RTL@"
+config.has_archer_library = @LIBARCHER_HAVE_ARCHER_LIBRARY@
+config.has_archer_runtime = @LIBARCHER_HAVE_ARCHER_RUNTIME@
+
+# print("\nStart")
+# print("config.test_c_compiler: " + str(config.test_c_compiler))
+# print("config.test_cxx_compiler: " + str(config.test_cxx_compiler))
+# print("config.test_compiler_features: " + str(config.test_compiler_features))
+# print("config.test_filecheck: " + str(config.test_filecheck))
+# print("config.test_archer_flags: " + str(config.test_archer_flags))
+# print("config.test_extra_flags: " + str(config.test_extra_flags))
+# print("config.libarcher_obj_root: " + str(config.libarcher_obj_root))
+# print("config.omp_library_dir: " + str(config.omp_library_dir))
+# print("config.omp_header_dir: " + str(config.omp_header_dir))
+# print("config.operating_system: " + str(config.operating_system))
+# print("config.has_ompt:" + str(config.has_ompt))
+# print("config.has_libm:" + str(config.has_libm))
+# print("config.has_libatomic: " + str(config.has_libatomic))
+# print("config.archer_tools_dir" + str(config.archer_tools_dir))
+# print("config.archer_library_dir: " + str(config.archer_library_dir))
+# print("config.archer_runtime_dir: " + str(config.archer_runtime_dir))
+# print("config.archer_library: " + str(config.archer_library))
+# print("config.archer_runtime: " + str(config.archer_runtime))
+# print("config.has_archer_library: " + str(config.has_archer_library))
+# print("config.has_archer_runtime: " + str(config.has_archer_runtime))
+# print("config.suppressions_archer_runtime_file: " + str(config.suppressions_archer_runtime_file))
+# print("End\n")
+
+# Let the main config do the real work.
+lit_config.load_config(config, "@CMAKE_CURRENT_SOURCE_DIR@/lit.cfg")

tools/archer/tests/ompt/ompt-signal.h

+/*
+ * ompt-signal.h -- Header providing low-level synchronization for tests
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is a copy from runtime/test/ompt/
+//
+//===----------------------------------------------------------------------===//
+
+#if defined(WIN32) || defined(_WIN32)
+#include <windows.h>
+#define delay() Sleep(1);
+#else
+#include <unistd.h>
+#define delay(t) usleep(t);
+#endif
+
+// These functions are used to provide a signal-wait mechanism to enforce
+// expected scheduling for the test cases.
+// Conditional variable (s) needs to be shared! Initialize to 0
+
+#define OMPT_SIGNAL(s) ompt_signal(&s)
+// inline
+void ompt_signal(int *s) {
+#pragma omp atomic
+  (*s)++;
+}
+
+#define OMPT_WAIT(s, v) ompt_wait(&s, v)
+// wait for s >= v
+// inline
+void ompt_wait(int *s, int v) {
+  int wait = 0;
+  do {
+    delay(10);
+#pragma omp atomic read
+    wait = (*s);
+  } while (wait < v);
+}

tools/archer/tests/parallel/parallel-firstprivate.c

+/*
+ * parallel-firstprivate.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(2) firstprivate(var)
+  { var = 1; }
+
+  fprintf(stderr, "DONE\n");
+  // var should still be 0!
+  return var;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/parallel/parallel-simple.c

+/*
+ * parallel-simple.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    if (omp_get_thread_num() == 1) {
+      var++;
+    }
+  } // implicit barrier
+
+  var++;
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/parallel/parallel-simple2.c

+/*
+ * parallel-simple2.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run  | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+// Create team of threads so that there is no implicit happens before
+// when creating the thread.
+#pragma omp parallel num_threads(2)
+  {}
+
+  var++;
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    if (omp_get_thread_num() == 1) {
+      var++;
+    }
+  } // implicit barrier
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/races/critical-unrelated.c

+/*
+ * critical-unrelated.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+#pragma omp critical
+    {
+      // Dummy region.
+    }
+
+    var++;
+  }
+
+  fprintf(stderr, "DONE\n");
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/lock-nested-unrelated.c

+/*
+ * lock-nested-unrelated.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+  omp_nest_lock_t lock;
+  omp_init_nest_lock(&lock);
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    omp_set_nest_lock(&lock);
+    omp_set_nest_lock(&lock);
+    // Dummy locking.
+    omp_unset_nest_lock(&lock);
+    omp_unset_nest_lock(&lock);
+
+    var++;
+  }
+
+  omp_destroy_nest_lock(&lock);
+
+  fprintf(stderr, "DONE\n");
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/lock-unrelated.c

+/*
+ * lock-unrelated.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+  omp_lock_t lock;
+  omp_init_lock(&lock);
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    omp_set_lock(&lock);
+    // Dummy locking.
+    omp_unset_lock(&lock);
+
+    var++;
+  }
+
+  omp_destroy_lock(&lock);
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/parallel-simple.c

+/*
+ * parallel-simple.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(2) shared(var)
+  { var++; }
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/task-dependency.c

+/*
+ * task-deoendency.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt/ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task shared(var, a) depend(out : var)
+    {
+      OMPT_SIGNAL(a);
+      var++;
+    }
+
+#pragma omp task shared(a) depend(in : var)
+    {
+      OMPT_SIGNAL(a);
+      OMPT_WAIT(a, 3);
+    }
+
+#pragma omp task shared(var) // depend(in: var) is missing here!
+    {
+      var++;
+      OMPT_SIGNAL(a);
+    }
+
+    // Give other thread time to steal the task.
+    OMPT_WAIT(a, 2);
+  }
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: #1 .omp_task_entry.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: #1 .omp_task_entry.
+// CHECK: DONE

tools/archer/tests/races/task-taskgroup-unrelated.c

+/*
+ * task-taskgroup-unrelated.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task shared(var, a)
+    {
+      var++;
+      OMPT_SIGNAL(a);
+      // Give master thread time to execute the task in the taskgroup.
+      OMPT_WAIT(a, 2);
+    }
+
+#pragma omp taskgroup
+    {
+#pragma omp task if (0)
+      {
+        // Dummy task.
+      }
+
+      // Give other threads time to steal the tasks.
+      OMPT_WAIT(a, 1);
+      OMPT_SIGNAL(a);
+    }
+
+    var++;
+  }
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/task-taskwait-nested.c

+/*
+ * task-taskwait-nested.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task shared(var, a)
+    {
+#pragma omp task shared(var, a)
+      {
+        // wait for master to pass the taskwait
+        OMPT_SIGNAL(a);
+        OMPT_WAIT(a, 2);
+        var++;
+      }
+    }
+
+    // Give other thread time to steal the task and execute its child.
+    OMPT_WAIT(a, 1);
+
+// Only directly generated children are guaranteed to be executed.
+#pragma omp taskwait
+    OMPT_SIGNAL(a);
+    var++;
+  }
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/task-two.c

+/*
+ * task-two.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#define NUM_THREADS 2
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+  int i;
+
+#pragma omp parallel for num_threads(NUM_THREADS) shared(var) schedule(static, \
+                                                                       1)
+  {
+    for (i = 0; i < NUM_THREADS; i++) {
+#pragma omp task shared(var) if (0) // the task is inlined an executed locally
+      { var++; }
+    }
+  }
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK:     #0 .omp_outlined.
+// CHECK:     #1 .omp_task_entry.
+// CHECK:   Previous write of size 4
+// CHECK:     #0 .omp_outlined.
+// CHECK:     #1 .omp_task_entry.
+// CHECK: DONE

tools/archer/tests/reduction/parallel-reduction-nowait.c

+/*
+ * parallel-reduction-nowait.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0, i;
+  int sum1 = 0;
+  int sum2 = 0;
+
+// Number of threads is empirical: We need enough threads so that
+// the reduction is really performed hierarchically in the barrier!
+#pragma omp parallel num_threads(5) reduction(+ : var)
+  {
+#pragma omp for schedule(static) nowait reduction(+ : sum1)
+    for (i = 0; i < 5; i++)
+      sum1 += i;
+#pragma omp for schedule(static) reduction(+ : sum2)
+    for (i = 0; i < 5; i++)
+      sum2 += i;
+
+    var = sum1 + sum2;
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 100);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/reduction/parallel-reduction.c

+/*
+ * parallel-reduction.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run| FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+// Number of threads is empirical: We need enough threads so that
+// the reduction is really performed hierarchically in the barrier!
+#pragma omp parallel num_threads(5) reduction(+ : var)
+  { var = 1; }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 5);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-barrier.c

+/*
+ * task-barrier.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt/ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+  {
+#pragma omp master
+    {
+#pragma omp task shared(var)
+      {
+        OMPT_SIGNAL(a);
+        var++;
+      }
+
+      // Give other thread time to steal the task.
+      OMPT_WAIT(a, 1);
+    }
+
+#pragma omp barrier
+
+#pragma omp master
+    { var++; }
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-create.c

+/*
+ * task-create.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+    var++;
+#pragma omp task shared(var, a)
+    {
+      var++;
+      OMPT_SIGNAL(a);
+    }
+
+    // Give other thread time to steal the task.
+    OMPT_WAIT(a, 1);
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-dependency.c

+/*
+ * task-dependency.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task shared(var, a) depend(out : var)
+    {
+      var++;
+      OMPT_SIGNAL(a);
+    }
+
+#pragma omp task shared(var, a) depend(in : var)
+    { OMPT_WAIT(a, 2); }
+
+#pragma omp task shared(var, a) depend(in : var)
+    {
+      OMPT_SIGNAL(a);
+      var++;
+    }
+
+    // Give other thread time to steal the task.
+    OMPT_WAIT(a, 1);
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-taskgroup-nested.c

+/*
+ * task-taskgroup-nested.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp taskgroup
+    {
+#pragma omp task
+      {
+#pragma omp task shared(var, a)
+        {
+          var++;
+          OMPT_SIGNAL(a);
+        }
+      }
+
+      // Give other thread time to steal the task and execute its child.
+      OMPT_WAIT(a, 1);
+    }
+
+    var++;
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-taskgroup.c

+/*
+ * task-taskgroup.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp taskgroup
+    {
+#pragma omp task shared(var, a)
+      {
+        var++;
+        OMPT_SIGNAL(a);
+      }
+
+      // Give other thread time to steal the task.
+      OMPT_WAIT(a, 1);
+    }
+
+    var++;
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-taskwait-nested.c

+/*
+ * task-taskwait-nested.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task
+    {
+#pragma omp task shared(var, a)
+      {
+        OMPT_SIGNAL(a);
+        delay(100);
+        var++;
+      }
+#pragma omp taskwait
+    }
+
+    // Give other thread time to steal the task and execute its child.
+    OMPT_WAIT(a, 1);
+
+#pragma omp taskwait
+    var++;
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-taskwait.c

+/*
+ * task-taskwait.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task shared(var, a)
+    {
+      OMPT_SIGNAL(a);
+      OMPT_WAIT(a, 2);
+      delay(100);
+      var++;
+    }
+
+    // Give other thread time to steal the task.
+    OMPT_WAIT(a, 1);
+    OMPT_SIGNAL(a);
+#pragma omp taskwait
+    var++;
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/worksharing/ordered.c

+/*
+ * ordered.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is licensed under the LLNS/LLNL "BSD-Like" license.
+// See tools/archer/LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+#define NUM_THREADS 2
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+  int i;
+
+#pragma omp parallel for ordered num_threads(NUM_THREADS) shared(var)
+  for (i = 0; i < NUM_THREADS; i++) {
+#pragma omp ordered
+    { var++; }
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE