[OpenMP] Enable ThreadSanitizer to check OpenMP programs

[simoatze]

Summary

This patch allows ThreadSanitizer (Tsan) to verify OpenMP programs. It means that no false positive will be reported by Tsan when verifying an OpenMP programs.
This patch introduces annotations within the OpenMP runtime module to provide information about thread synchronization to the Tsan runtime.

In order to enable the Tsan support when building the runtime, you must enable the TSAN_SUPPORT option with the following environment variable:

-DLIBOMP_TSAN_SUPPORT=TRUE

The building will generate a library called libomp_tsan.so.
I guess this is not something that we want, so probably it should be removed, but I will leave it for the first review.

[Hahnfeld]

I have only marked one of the whitespace issues...

runtime/CMakeLists.txt

324

Could this be moved to kmp_config.h.cmake?

runtime/src/kmp_barrier.cpp

78

tab instead of 4 spaces

[simoatze]

Fixed tabs to spaces

[hfinkel]

Thanks for working on this; I'm very excited to see this functionality!

As a general comment, can you always define ANNOTATE_* macros, and just make them empty if TSAN_SUPPORT is not true. What would make things less verbose.

The building will generate a library called libomp_tsan.so.
I guess this is not something that we want, so probably it should be removed, but I will leave it for the first review.

Why would this not be desirable? What's the alternative?

[simoatze]

Activated by default ANNOTATIONS (switch on/off only with TSAN_SUPPORT).

[simoatze]

In D13072#252137, @hfinkel wrote:

Thanks for working on this; I'm very excited to see this functionality!

As a general comment, can you always define ANNOTATE_* macros, and just make them empty if TSAN_SUPPORT is not true. What would make things less verbose.

The dynamic annotations have a DYNAMIC_ANNOTATIONS_ENABLED macro to define or not the annotations.
I introduced the TSAN_SUPPORT macro to give a better idea that they are related for supporting ThreadSanitizer.
I just modified the patch setting by default DYNAMIC_ANNOTATIONS_ENABLED=1 so the annotations will be enabled by setting TSAN_SUPPORT to TRUE.
Is that what you meant?

The building will generate a library called libomp_tsan.so.
I guess this is not something that we want, so probably it should be removed, but I will leave it for the first review.

Why would this not be desirable? What's the alternative?

Generating the libomp_tsan.so wouldn't require to modify clang to accept a different library name (besides libomp, libgomp and libiomp5)?
I noticed that if I compile with -fopenmp=libomp_tsan it complains because clang does not know libomp_tsan.so.
Maybe the right thing is just to have one library "libomp.so" and the user would decide to activate Tsan support or not at compile time.

[simoatze]

In D13072#252137, @hfinkel wrote:

Thanks for working on this; I'm very excited to see this functionality!

As a general comment, can you always define ANNOTATE_* macros, and just make them empty if TSAN_SUPPORT is not true. What would make things less verbose.

Oh I guess you mean to remove all the TSAN_SUPPORT and just leave the ANNOTATIONS, if TSAN support is not defined it will make them empty, right?

[hfinkel]

In D13072#252176, @simoatze wrote:

In D13072#252137, @hfinkel wrote:

Thanks for working on this; I'm very excited to see this functionality!

As a general comment, can you always define ANNOTATE_* macros, and just make them empty if TSAN_SUPPORT is not true. What would make things less verbose.

Oh I guess you mean to remove all the TSAN_SUPPORT and just leave the ANNOTATIONS, if TSAN support is not defined it will make them empty, right?

Yes, exactly. That way each annotation is one line instead of three ;)

[simoatze]

Removed TSAN_SUPPORT macro.

[jcownie]

It generally looks fine to me.

My one concern is over the licence in the header file. It looks like a BSD-ish licence, but it's not the same as either of the licences which apply to the rest of the code ( http://llvm.org/svn/llvm-project/openmp/trunk/LICENSE.txt ). (Of course, IANAL).

[simoatze]

Adding Joachim, who is the main creator of the patch.
He can give more details if needed.

[protze.joachim]

In D13072#252137, @hfinkel wrote:

The building will generate a library called libomp_tsan.so.
I guess this is not something that we want, so probably it should be removed, but I will leave it for the first review.

Why would this not be desirable? What's the alternative?

I see two ways to work around the duplicate build for the two libraries:
We might add weak, empty copies of the annotation functions to the library which would be overwritten by the tsan-rt functions if these are loaded. This would be at the cost of additional function calls for each annotation.

Keeping the separate instrumented library, we might define the library as an additional build target, so we build both libraries with a single make step. But I found the latest version of the CMake file very confusing for adding a second library target with only slightly different options.

[jlpeyton]

Keeping the separate instrumented library, we might define the library as an additional build target, so we build both libraries with a single make step. But I found the latest version of the CMake file very confusing for adding a second library target with only slightly different options.

It's probably confusing because it isn't structured to do that. It is somewhat modeled after libcxx's CMake which does something similar (compile just a single library) although ours is always shared. i.e., there is a single, hard-coded add_library(omp SHARED ...) call. Hence, only one library can be built. I know compiler-rt has custom CMake AddClangRuntimeLibrary(${libname} ...) functionality where several slightly different libraries are generated for the compiler. For instance, when I build llvm/clang on x86 about 40 libclang_rt*.a's are created. If we need this functionality it will have to be added to the CMake logic.

[• dberlin]

In D13072#252598, @jcownie wrote:

It generally looks fine to me.

My one concern is over the licence in the header file. It looks like a BSD-ish licence, but it's not the same as either of the licences which apply to the rest of the code ( http://llvm.org/svn/llvm-project/openmp/trunk/LICENSE.txt ). (Of course, IANAL).

Urf. Speaking as a lawyer, in practice, this doesn't matter.

However, we should fix all of these, not just the Google one.

The BSD license in that project should not be the license we use. We have generally not given credit to specific groups *in the license*, as that LICENSE does, but instead elsewhere. The license should be the same license we use for runtime libraries elsewhere (which is not even BSD, but MIT).

I'll start a thread about this.

[protze.joachim]

In D13072#256044, @dberlin wrote:

I'll start a thread about this.

is this related to http://lists.llvm.org/pipermail/llvm-dev/2015-October/091536.html?

The intention of using this annotation header file was, to make the annotations available not only to TSan, but also to other tools like valgrind/helgrind.

Do you expect us to align the license to the licenses used by LLVM/OpenMP or will someone from your team take care of this?

[jcownie]

Do you expect us to align the license to the licenses used by LLVM/OpenMP or
will someone from your team take care of this?

I do not believe that we (the non-license holders) can arbitrarily change the license.
(Otherwise what was the point of a license in the first place!?)
Therefore the license changes have to be made by people submitting the code.

• Jim

James Cownie <[e-mail]>
SSG/DPD/TCAR (Technical Computing, Analyzers and Runtimes)
Tel: [phone]

[hfinkel]

In D13072#256043, @dberlin wrote:

In D13072#252598, @jcownie wrote:

It generally looks fine to me.

My one concern is over the licence in the header file. It looks like a BSD-ish licence, but it's not the same as either of the licences which apply to the rest of the code ( http://llvm.org/svn/llvm-project/openmp/trunk/LICENSE.txt ). (Of course, IANAL).

Urf. Speaking as a lawyer, in practice, this doesn't matter.

However, we should fix all of these, not just the Google one.

The BSD license in that project should not be the license we use. We have generally not given credit to specific groups *in the license*, as that LICENSE does, but instead elsewhere. The license should be the same license we use for runtime libraries elsewhere (which is not even BSD, but MIT).

I'll start a thread about this.

@dberlin , how do we move forward here? Having the OpenMP runtime library work well with TSan is important.

[hfinkel]

To summarize the e-mail thread which phabricator did not pick up: @dberlin has authorized a change of the license of dynamic_annotations.h file to comply with LLVM's standard licencing policy.

@jlpeyton, @AndreyChurbanov , et al., is this patch otherwise okay to go in?

I agree that we should make the CMake setup more compiler-rt-like, so we can have multiple targets. Generalizing that logic, or otherwise reusing it, seems like something we can/should do as a separate change.

[simoatze]

We rewrote the annotations file to deal with the license issue.

If configured defining the following variable

-D LIBOMP_TSAN_SUPPORT=TRUE

the annotations will be enabled in the main shared library (same mechanism of OMPT).

[jlpeyton]

runtime/CMakeLists.txt

322

Unused?

323–326

It seems TSAN is only supported on Linux X86-64 so inside cmake/config-ix.cmake this should be appended:

if("${CMAKE_SYSTEM_NAME}" MATCHES "Linux" AND ${INTEL64})
  set(LIBOMP_HAVE_TSAN_SUPPORT TRUE)
else()
  set(LIBOMP_HAVE_TSAN_SUPPORT FALSE)
endif()

and in the main CMakeLists.txt file (You'll see this pattern a lot in this file):

set(LIBOMP_TSAN_SUPPORT FALSE CACHE BOOL
  "TSAN-support?")
if(LIBOMP_TSAN_SUPPORT AND (NOT LIBOMP_HAVE_TSAN_SUPPORT))
  libomp_error_say("TSAN functionality requested but not available")
endif()

Also, as Jonas pointed out, we usually use the kmp_config.h.cmake file instead of sending in definitions. So in kmp_config.h.cmake there should be:

#cmakedefine01 LIBOMP_TSAN_SUPPORT
#if LIBOMP_TSAN_SUPPORT
#define TSAN_SUPPORT
#endif

runtime/src/kmp_barrier.cpp

310

I'm not familiar with the thread sanitizer or what these functions do. Can you explain a little bit about BARRIER_BEFORE and AFTER and what they do with their argument?

runtime/src/tsan_annotations.h

20

#include "kmp_config.h"

[simoatze]

I updated the patch according to the last comments.
I have tried to compile the runtime on PPC64LE and I was getting several compiling errors, have anybody compiled the runtime on power recently?

[simoatze]

runtime/CMakeLists.txt

323–326

Tsan is also supported for other architectures (I personally ported it on PPC64), however I wasn't able to compile the OpenMP runtime on PPC64, I'll leave only x86_64 for now.

runtime/src/kmp_barrier.cpp

310

Those annotations basically update the vector clock for a certain thread that Tsan keeps to compute the happens-before relation.

[jlpeyton]

LGTM

[Hahnfeld]

There are also some indention errors (especially when annotating the reductions)...

runtime/src/kmp_barrier.cpp

809–815

Is that dead code?

runtime/src/kmp_tasking.c

1179–1184

You are missing braces here, this won't work as expected!

1279–1282

You are missing braces here, this won't work as expected!

runtime/src/tsan_annotations.h

86–101

What's the difference between the new and the old barrier annotations? Can the old ones be removed?

[simoatze]

Fixed indentation errors, removed dead code and old barrier annotations.

[Hahnfeld]

LGTM now

[jwhowarth]

Are there any benchmarks yet on potential performance penalties from building libomp with -D LIBOMP_TSAN_SUPPORT=TRUE as the release version of the library?

[Hahnfeld]

In D13072#589299, @jwhowarth wrote:

Are there any benchmarks yet on potential performance penalties from building libomp with -D LIBOMP_TSAN_SUPPORT=TRUE as the release version of the library?

Hi Jack,

in the past we used the EPCC microbenchmarks to measure the difference when OMPT is disabled, enabled and enabled with a tool attached. I think this should also work here...

Regards,
Jonas

runtime/CMakeLists.txt

 set(LIBOMP_OMPT_BLAME TRUE CACHE BOOL
   "OMPT-blame?")
 set(LIBOMP_OMPT_TRACE TRUE CACHE BOOL
   "OMPT-trace?")
 if(LIBOMP_OMPT_SUPPORT AND (NOT LIBOMP_HAVE_OMPT_SUPPORT))
   libomp_error_say("OpenMP Tools Interface requested but not available")
 endif()
 
+# TSAN-support
+set(LIBOMP_TSAN_SUPPORT FALSE CACHE BOOL
+  "TSAN-support?")
+if(LIBOMP_TSAN_SUPPORT AND (NOT LIBOMP_HAVE_TSAN_SUPPORT))
+  libomp_error_say("TSAN functionality requested but not available")
+endif()
+
 # Error check hwloc support after config-ix has run
 if(LIBOMP_USE_HWLOC AND (NOT LIBOMP_HAVE_HWLOC))
   libomp_error_say("Hwloc requested but not available")
 endif()
 
 # Setting final library name
 set(LIBOMP_DEFAULT_LIB_NAME libomp)
 if(${PROFILE_LIBRARY})
   set(LIBOMP_DEFAULT_LIB_NAME ${LIBOMP_DEFAULT_LIB_NAME}prof)
 endif()
if(${LIBOMP_STANDALONE_BUILD})
   libomp_say("Use ITT notify       -- ${LIBOMP_USE_ITT_NOTIFY}")
   libomp_say("Use OMPT-support     -- ${LIBOMP_OMPT_SUPPORT}")
   if(${LIBOMP_OMPT_SUPPORT})
     libomp_say("Use OMPT-blame       -- ${LIBOMP_OMPT_BLAME}")
     libomp_say("Use OMPT-trace       -- ${LIBOMP_OMPT_TRACE}")
   endif()
   libomp_say("Use Adaptive locks   -- ${LIBOMP_USE_ADAPTIVE_LOCKS}")
   libomp_say("Use quad precision   -- ${LIBOMP_USE_QUAD_PRECISION}")
+  libomp_say("Use TSAN-support     -- ${LIBOMP_TSAN_SUPPORT}")
   libomp_say("Use Hwloc library    -- ${LIBOMP_USE_HWLOC}")
 endif()
 
 add_subdirectory(src)
 add_subdirectory(test)
-

runtime/cmake/config-ix.cmake

if(${LIBOMP_USE_HWLOC})
   if(LIBOMP_HAVE_HWLOC_H AND LIBOMP_HAVE_LIBHWLOC AND LIBOMP_HWLOC_LIBRARY)
     set(LIBOMP_HAVE_HWLOC TRUE)
   else()
     set(LIBOMP_HAVE_HWLOC FALSE)
     libomp_say("Could not find hwloc")
   endif()
 endif()
 
+# Check if ThreadSanitizer support is available
+if("${CMAKE_SYSTEM_NAME}" MATCHES "Linux" AND ${INTEL64})
+  set(LIBOMP_HAVE_TSAN_SUPPORT TRUE)
+else()
+  set(LIBOMP_HAVE_TSAN_SUPPORT FALSE)
+endif()

runtime/src/CMakeLists.txt

else()
   endif()
 endif()
 # Files common to stubs and normal library
 libomp_append(LIBOMP_CFILES kmp_ftn_cdecl.c)
 libomp_append(LIBOMP_CFILES kmp_ftn_extra.c)
 libomp_append(LIBOMP_CFILES kmp_version.c)
 libomp_append(LIBOMP_CFILES ompt-general.c IF_TRUE LIBOMP_OMPT_SUPPORT)
 
+libomp_append(LIBOMP_CFILES tsan_annotations.c IF_TRUE LIBOMP_TSAN_SUPPORT)
+
 set(LIBOMP_SOURCE_FILES ${LIBOMP_CFILES} ${LIBOMP_CXXFILES} ${LIBOMP_ASMFILES})
 # For Windows, there is a resource file (.rc -> .res) that is also compiled
 libomp_append(LIBOMP_SOURCE_FILES libomp.rc WIN32)
 
 # Get compiler and assembler flags
 libomp_get_cflags(LIBOMP_CONFIGURED_CFLAGS)
 libomp_get_cxxflags(LIBOMP_CONFIGURED_CXXFLAGS)
 libomp_get_asmflags(LIBOMP_CONFIGURED_ASMFLAGS)

runtime/src/exports_so.txt

global: # Exported symbols.
         omp_*;     # Standard OpenMP functions.
 
         #
         # OMPT API
         #
         ompt_tool;           # OMPT initialization interface
         ompt_control;        # OMPT control interface
 
+        # icc drops weak attribute at linking step without the following line:
+        Annotate*;           # TSAN annotation
+
         #
         # OMPT state placeholders
         #
         ompt_idle;
         ompt_overhead;
         ompt_barrier_wait;
         ompt_task_wait;
         ompt_mutex_wait;

runtime/src/kmp_barrier.cpp

 #include "kmp_os.h"
 
 
 #if KMP_MIC
 #include <immintrin.h>
 #define USE_NGO_STORES 1
 #endif // KMP_MIC
 
+#include "tsan_annotations.h"
+
 #if KMP_MIC && USE_NGO_STORES
 // ICV copying
 #define ngo_load(src)            __m512d Vt = _mm512_load_pd((void *)(src))
 #define ngo_store_icvs(dst, src) _mm512_storenrngo_pd((void *)(dst), Vt)
 #define ngo_store_go(dst, src)   _mm512_storenrngo_pd((void *)(dst), Vt)
 #define ngo_sync()               __asm__ volatile ("lock; addl $0,0(%%rsp)" ::: "memory")
 #else
 #define ngo_load(src)            ((void)0)
#endif
     if (!KMP_MASTER_TID(tid)) {
         KA_TRACE(20, ("__kmp_linear_barrier_gather: T#%d(%d:%d) releasing T#%d(%d:%d)"
                       "arrived(%p): %llu => %llu\n", gtid, team->t.t_id, tid,
                       __kmp_gtid_from_tid(0, team), team->t.t_id, 0, &thr_bar->b_arrived,
                       thr_bar->b_arrived, thr_bar->b_arrived + KMP_BARRIER_STATE_BUMP));
         // Mark arrival to master thread
         /* After performing this write, a worker thread may not assume that the team is valid
            any more - it could be deallocated by the master thread at any time. */
+        ANNOTATE_BARRIER_BEFORE(this_thr);
         kmp_flag_64 flag(&thr_bar->b_arrived, other_threads[0]);
         flag.release();
     } else {
         register kmp_balign_team_t *team_bar = &team->t.t_bar[bt];
         register int nproc = this_thr->th.th_team_nproc;
         register int i;
         // Don't have to worry about sleep bit here or atomic since team setting
         register kmp_uint64 new_state = team_bar->b_arrived + KMP_BARRIER_STATE_BUMP;
 
         // Collect all the worker team member threads.
         for (i=1; i<nproc; ++i) {
 #if KMP_CACHE_MANAGE
             // Prefetch next thread's arrived count
             if (i+1 < nproc)
                 KMP_CACHE_PREFETCH(&other_threads[i+1]->th.th_bar[bt].bb.b_arrived);
 #endif /* KMP_CACHE_MANAGE */
             KA_TRACE(20, ("__kmp_linear_barrier_gather: T#%d(%d:%d) wait T#%d(%d:%d) "
                           "arrived(%p) == %llu\n", gtid, team->t.t_id, tid,
                             __kmp_gtid_from_tid(i, team), team->t.t_id, i,
                             &other_threads[i]->th.th_bar[bt].bb.b_arrived, new_state));
 
             // Wait for worker thread to arrive
             kmp_flag_64 flag(&other_threads[i]->th.th_bar[bt].bb.b_arrived, new_state);
             flag.wait(this_thr, FALSE
                       USE_ITT_BUILD_ARG(itt_sync_obj) );
+            ANNOTATE_BARRIER_AFTER(other_threads[ i ]);
 #if USE_ITT_BUILD && USE_ITT_NOTIFY
             // Barrier imbalance - write min of the thread time and the other thread time to the thread.
             if (__kmp_forkjoin_frames_mode == 2) {
                 this_thr->th.th_bar_min_time = KMP_MIN(this_thr->th.th_bar_min_time,
                                                           other_threads[i]->th.th_bar_min_time);
             }
 #endif
             if (reduce) {
                 KA_TRACE(100, ("__kmp_linear_barrier_gather: T#%d(%d:%d) += T#%d(%d:%d)\n", gtid,
                                team->t.t_id, tid, __kmp_gtid_from_tid(i, team), team->t.t_id, i));
+              ANNOTATE_REDUCE_AFTER(reduce);
                 (*reduce)(this_thr->th.th_local.reduce_data,
                           other_threads[i]->th.th_local.reduce_data);
+              ANNOTATE_REDUCE_BEFORE(reduce);
+              ANNOTATE_REDUCE_BEFORE(&team->t.t_bar);
             }
         }
         // Don't have to worry about sleep bit here or atomic since team setting
         team_bar->b_arrived = new_state;
         KA_TRACE(20, ("__kmp_linear_barrier_gather: T#%d(%d:%d) set team %d arrived(%p) = %llu\n",
                       gtid, team->t.t_id, tid, team->t.t_id, &team_bar->b_arrived, new_state));
     }
     KA_TRACE(20, ("__kmp_linear_barrier_gather: T#%d(%d:%d) exit for barrier type %d\n",
#if KMP_CACHE_MANAGE
                     KMP_CACHE_PREFETCH(&other_threads[i+1]->th.th_bar[bt].bb.b_go);
 #endif /* KMP_CACHE_MANAGE */
                 KA_TRACE(20, ("__kmp_linear_barrier_release: T#%d(%d:%d) releasing T#%d(%d:%d) "
                               "go(%p): %u => %u\n", gtid, team->t.t_id, tid,
                               other_threads[i]->th.th_info.ds.ds_gtid, team->t.t_id, i,
                               &other_threads[i]->th.th_bar[bt].bb.b_go,
                               other_threads[i]->th.th_bar[bt].bb.b_go,
                               other_threads[i]->th.th_bar[bt].bb.b_go + KMP_BARRIER_STATE_BUMP));
+                ANNOTATE_BARRIER_BEFORE(other_threads[i]);
                 kmp_flag_64 flag(&other_threads[i]->th.th_bar[bt].bb.b_go, other_threads[i]);
                 flag.release();
             }
         }
     } else { // Wait for the MASTER thread to release us
         KA_TRACE(20, ("__kmp_linear_barrier_release: T#%d wait go(%p) == %u\n",
                       gtid, &thr_bar->b_go, KMP_BARRIER_STATE_BUMP));
         kmp_flag_64 flag(&thr_bar->b_go, KMP_BARRIER_STATE_BUMP);
         flag.wait(this_thr, TRUE
                   USE_ITT_BUILD_ARG(itt_sync_obj) );
+        ANNOTATE_BARRIER_AFTER(this_thr);
 #if USE_ITT_BUILD && USE_ITT_NOTIFY
         if ((__itt_sync_create_ptr && itt_sync_obj == NULL) || KMP_ITT_DEBUG) {
             // In a fork barrier; cannot get the object reliably (or ITTNOTIFY is disabled)
             itt_sync_obj = __kmp_itt_barrier_object(gtid, bs_forkjoin_barrier, 0, -1);
             // Cancel wait on previous parallel region...
             __kmp_itt_task_starting(itt_sync_obj);
 
             if (bt == bs_forkjoin_barrier && TCR_4(__kmp_global.g.g_done))
#endif /* KMP_CACHE_MANAGE */
             KA_TRACE(20, ("__kmp_tree_barrier_gather: T#%d(%d:%d) wait T#%d(%d:%u) "
                           "arrived(%p) == %llu\n", gtid, team->t.t_id, tid,
                             __kmp_gtid_from_tid(child_tid, team), team->t.t_id, child_tid,
                             &child_bar->b_arrived, new_state));
             // Wait for child to arrive
             kmp_flag_64 flag(&child_bar->b_arrived, new_state);
             flag.wait(this_thr, FALSE
                       USE_ITT_BUILD_ARG(itt_sync_obj) );
+            ANNOTATE_BARRIER_AFTER(child_thr);
 #if USE_ITT_BUILD && USE_ITT_NOTIFY
             // Barrier imbalance - write min of the thread time and a child time to the thread.
             if (__kmp_forkjoin_frames_mode == 2) {
                 this_thr->th.th_bar_min_time = KMP_MIN(this_thr->th.th_bar_min_time,
                                                           child_thr->th.th_bar_min_time);
             }
 #endif
             if (reduce) {
                 KA_TRACE(100, ("__kmp_tree_barrier_gather: T#%d(%d:%d) += T#%d(%d:%u)\n",
                                gtid, team->t.t_id, tid, __kmp_gtid_from_tid(child_tid, team),
                                team->t.t_id, child_tid));
+              ANNOTATE_REDUCE_AFTER(reduce);
                 (*reduce)(this_thr->th.th_local.reduce_data, child_thr->th.th_local.reduce_data);
+              ANNOTATE_REDUCE_BEFORE(reduce);
+              ANNOTATE_REDUCE_BEFORE(&team->t.t_bar);
             }
             child++;
             child_tid++;
         }
         while (child <= branch_factor && child_tid < nproc);
     }
 
     if (!KMP_MASTER_TID(tid)) { // Worker threads
         register kmp_int32 parent_tid = (tid - 1) >> branch_bits;
 
         KA_TRACE(20, ("__kmp_tree_barrier_gather: T#%d(%d:%d) releasing T#%d(%d:%d) "
                       "arrived(%p): %llu => %llu\n", gtid, team->t.t_id, tid,
                       __kmp_gtid_from_tid(parent_tid, team), team->t.t_id, parent_tid,
                       &thr_bar->b_arrived, thr_bar->b_arrived,
                       thr_bar->b_arrived + KMP_BARRIER_STATE_BUMP));
 
         // Mark arrival to parent thread
         /* After performing this write, a worker thread may not assume that the team is valid
            any more - it could be deallocated by the master thread at any time.  */
+        ANNOTATE_BARRIER_BEFORE(this_thr);
         kmp_flag_64 flag(&thr_bar->b_arrived, other_threads[parent_tid]);
         flag.release();
     } else {
         // Need to update the team arrived pointer if we are the master thread
         if (nproc > 1) // New value was already computed above
             team->t.t_bar[bt].b_arrived = new_state;
         else
             team->t.t_bar[bt].b_arrived += KMP_BARRIER_STATE_BUMP;
__kmp_tree_barrier_release(enum barrier_type bt, kmp_info_t *this_thr, int gtid, int tid,
     // Perform a tree release for all of the threads that have been gathered
     if (!KMP_MASTER_TID(tid)) { // Handle fork barrier workers who aren't part of a team yet
         KA_TRACE(20, ("__kmp_tree_barrier_release: T#%d wait go(%p) == %u\n",
                       gtid, &thr_bar->b_go, KMP_BARRIER_STATE_BUMP));
         // Wait for parent thread to release us
         kmp_flag_64 flag(&thr_bar->b_go, KMP_BARRIER_STATE_BUMP);
         flag.wait(this_thr, TRUE
                   USE_ITT_BUILD_ARG(itt_sync_obj) );
+        ANNOTATE_BARRIER_AFTER(this_thr);
 #if USE_ITT_BUILD && USE_ITT_NOTIFY
         if ((__itt_sync_create_ptr && itt_sync_obj == NULL) || KMP_ITT_DEBUG) {
             // In fork barrier where we could not get the object reliably (or ITTNOTIFY is disabled)
             itt_sync_obj = __kmp_itt_barrier_object(gtid, bs_forkjoin_barrier, 0, -1);
             // Cancel wait on previous parallel region...
             __kmp_itt_task_starting(itt_sync_obj);
 
             if (bt == bs_forkjoin_barrier && TCR_4(__kmp_global.g.g_done))
#if KMP_BARRIER_ICV_PUSH
             }
 #endif // KMP_BARRIER_ICV_PUSH
             KA_TRACE(20, ("__kmp_tree_barrier_release: T#%d(%d:%d) releasing T#%d(%d:%u)"
                           "go(%p): %u => %u\n", gtid, team->t.t_id, tid,
                           __kmp_gtid_from_tid(child_tid, team), team->t.t_id,
                           child_tid, &child_bar->b_go, child_bar->b_go,
                           child_bar->b_go + KMP_BARRIER_STATE_BUMP));
             // Release child from barrier
+            ANNOTATE_BARRIER_BEFORE(child_thr);
             kmp_flag_64 flag(&child_bar->b_go, child_thr);
             flag.release();
             child++;
             child_tid++;
         }
         while (child <= branch_factor && child_tid < nproc);
     }
     KA_TRACE(20, ("__kmp_tree_barrier_release: T#%d(%d:%d) exit for barrier type %d\n",
for (level=0, offset=1; offset<num_threads; level+=branch_bits, offset<<=branch_bits)
                           "arrived(%p): %llu => %llu\n", gtid, team->t.t_id, tid,
                           __kmp_gtid_from_tid(parent_tid, team), team->t.t_id, parent_tid,
                           &thr_bar->b_arrived, thr_bar->b_arrived,
                           thr_bar->b_arrived + KMP_BARRIER_STATE_BUMP));
             // Mark arrival to parent thread
             /* After performing this write (in the last iteration of the enclosing for loop),
                a worker thread may not assume that the team is valid any more - it could be
                deallocated by the master thread at any time.  */
+            ANNOTATE_BARRIER_BEFORE(this_thr);
             p_flag.set_waiter(other_threads[parent_tid]);
             p_flag.release();
             break;
         }
 
         // Parent threads wait for children to arrive
         if (new_state == KMP_BARRIER_UNUSED_STATE)
             new_state = team->t.t_bar[bt].b_arrived + KMP_BARRIER_STATE_BUMP;
#endif /* KMP_CACHE_MANAGE */
             KA_TRACE(20, ("__kmp_hyper_barrier_gather: T#%d(%d:%d) wait T#%d(%d:%u) "
                           "arrived(%p) == %llu\n", gtid, team->t.t_id, tid,
                           __kmp_gtid_from_tid(child_tid, team), team->t.t_id, child_tid,
                           &child_bar->b_arrived, new_state));
             // Wait for child to arrive
             kmp_flag_64 c_flag(&child_bar->b_arrived, new_state);
             c_flag.wait(this_thr, FALSE
                         USE_ITT_BUILD_ARG(itt_sync_obj) );
+            ANNOTATE_BARRIER_AFTER(child_thr);
 #if USE_ITT_BUILD && USE_ITT_NOTIFY
             // Barrier imbalance - write min of the thread time and a child time to the thread.
             if (__kmp_forkjoin_frames_mode == 2) {
                 this_thr->th.th_bar_min_time = KMP_MIN(this_thr->th.th_bar_min_time,
                                                           child_thr->th.th_bar_min_time);
             }
 #endif
             if (reduce) {
                 KA_TRACE(100, ("__kmp_hyper_barrier_gather: T#%d(%d:%d) += T#%d(%d:%u)\n",
                                gtid, team->t.t_id, tid, __kmp_gtid_from_tid(child_tid, team),
                                team->t.t_id, child_tid));
+              ANNOTATE_REDUCE_AFTER(reduce);
                 (*reduce)(this_thr->th.th_local.reduce_data, child_thr->th.th_local.reduce_data);
+              ANNOTATE_REDUCE_BEFORE(reduce);
+              ANNOTATE_REDUCE_BEFORE(&team->t.t_bar);
             }
         }
     }
 
     if (KMP_MASTER_TID(tid)) {
         // Need to update the team arrived pointer if we are the master thread
         if (new_state == KMP_BARRIER_UNUSED_STATE)
             team->t.t_bar[bt].b_arrived += KMP_BARRIER_STATE_BUMP;
#endif
     }
     else  { // Handle fork barrier workers who aren't part of a team yet
         KA_TRACE(20, ("__kmp_hyper_barrier_release: T#%d wait go(%p) == %u\n",
                       gtid, &thr_bar->b_go, KMP_BARRIER_STATE_BUMP));
         // Wait for parent thread to release us
         kmp_flag_64 flag(&thr_bar->b_go, KMP_BARRIER_STATE_BUMP);
         flag.wait(this_thr, TRUE
                   USE_ITT_BUILD_ARG(itt_sync_obj) );
+        ANNOTATE_BARRIER_AFTER(this_thr);
 #if USE_ITT_BUILD && USE_ITT_NOTIFY
         if ((__itt_sync_create_ptr && itt_sync_obj == NULL) || KMP_ITT_DEBUG) {
             // In fork barrier where we could not get the object reliably
             itt_sync_obj = __kmp_itt_barrier_object(gtid, bs_forkjoin_barrier, 0, -1);
             // Cancel wait on previous parallel region...
             __kmp_itt_task_starting(itt_sync_obj);
 
             if (bt == bs_forkjoin_barrier && TCR_4(__kmp_global.g.g_done))

 #endif // KMP_BARRIER_ICV_PUSH
 
                 KA_TRACE(20, ("__kmp_hyper_barrier_release: T#%d(%d:%d) releasing T#%d(%d:%u)"
                               "go(%p): %u => %u\n", gtid, team->t.t_id, tid,
                               __kmp_gtid_from_tid(child_tid, team), team->t.t_id,
                               child_tid, &child_bar->b_go, child_bar->b_go,
                               child_bar->b_go + KMP_BARRIER_STATE_BUMP));
                 // Release child from barrier
+                ANNOTATE_BARRIER_BEFORE(child_thr);
                 kmp_flag_64 flag(&child_bar->b_go, child_thr);
                 flag.release();
             }
         }
     }
 #if KMP_BARRIER_ICV_PUSH
     if (propagate_icvs && !KMP_MASTER_TID(tid)) { // copy ICVs locally to final dest
         __kmp_init_implicit_task(team->t.t_ident, team->t.t_threads[tid], team, tid, FALSE);
if (thr_bar->my_level) { // not a leaf (my_level==0 means leaf)
             if (thr_bar->leaf_kids) { // First, wait for leaf children to check-in on my b_arrived flag
                 kmp_uint64 leaf_state = KMP_MASTER_TID(tid) ? thr_bar->b_arrived | thr_bar->leaf_state : team->t.t_bar[bt].b_arrived | thr_bar->leaf_state;
                 KA_TRACE(20, ("__kmp_hierarchical_barrier_gather: T#%d(%d:%d) waiting for leaf kids\n",
                               gtid, team->t.t_id, tid));
                 kmp_flag_64 flag(&thr_bar->b_arrived, leaf_state);
                 flag.wait(this_thr, FALSE
                           USE_ITT_BUILD_ARG(itt_sync_obj) );
                 if (reduce) {
+                    ANNOTATE_REDUCE_AFTER(reduce);
                     for (child_tid=tid+1; child_tid<=tid+thr_bar->leaf_kids; ++child_tid) {
                         KA_TRACE(100, ("__kmp_hierarchical_barrier_gather: T#%d(%d:%d) += T#%d(%d:%d)\n",
                                        gtid, team->t.t_id, tid, __kmp_gtid_from_tid(child_tid, team),
                                        team->t.t_id, child_tid));
+                        ANNOTATE_BARRIER_AFTER(other_threads[child_tid]);
                         (*reduce)(this_thr->th.th_local.reduce_data, other_threads[child_tid]->th.th_local.reduce_data);
                     }
+                    ANNOTATE_REDUCE_BEFORE(reduce);
+                    ANNOTATE_REDUCE_BEFORE(&team->t.t_bar);
+                }
+#ifdef DYN
+                else{
+                  for (child_tid=tid+1; child_tid<=tid+thr_bar->leaf_kids; ++child_tid) {
+                    ANNOTATE_BARRIER_AFTER(other_threads[child_tid]);
+                  }
                 }
+#endif
                 (void) KMP_TEST_THEN_AND64((volatile kmp_int64 *)&thr_bar->b_arrived, ~(thr_bar->leaf_state)); // clear leaf_state bits
             }
             // Next, wait for higher level children on each child's b_arrived flag
             for (kmp_uint32 d=1; d<thr_bar->my_level; ++d) { // gather lowest level threads first, but skip 0
                 kmp_uint32 last = tid+thr_bar->skip_per_level[d+1], skip = thr_bar->skip_per_level[d];
                 if (last > nproc) last = nproc;
                 for (child_tid=tid+skip; child_tid<(int)last; child_tid+=skip) {
                     register kmp_info_t *child_thr = other_threads[child_tid];
                     register kmp_bstate_t *child_bar = &child_thr->th.th_bar[bt].bb;
                     KA_TRACE(20, ("__kmp_hierarchical_barrier_gather: T#%d(%d:%d) wait T#%d(%d:%d) "
                                   "arrived(%p) == %llu\n",
                                   gtid, team->t.t_id, tid, __kmp_gtid_from_tid(child_tid, team),
                                   team->t.t_id, child_tid, &child_bar->b_arrived, new_state));
                     kmp_flag_64 flag(&child_bar->b_arrived, new_state);
                     flag.wait(this_thr, FALSE
                               USE_ITT_BUILD_ARG(itt_sync_obj) );
+                    ANNOTATE_BARRIER_AFTER(child_thr);
                     if (reduce) {
                         KA_TRACE(100, ("__kmp_hierarchical_barrier_gather: T#%d(%d:%d) += T#%d(%d:%d)\n",
                                        gtid, team->t.t_id, tid, __kmp_gtid_from_tid(child_tid, team),
                                        team->t.t_id, child_tid));
+                      ANNOTATE_REDUCE_AFTER(reduce);
                         (*reduce)(this_thr->th.th_local.reduce_data, child_thr->th.th_local.reduce_data);
+                      ANNOTATE_REDUCE_BEFORE(reduce);
+                      ANNOTATE_REDUCE_BEFORE(&team->t.t_bar);
                     }
                 }
             }
         }
         else { // Blocktime is not infinite
             for (kmp_uint32 d=0; d<thr_bar->my_level; ++d) { // Gather lowest level threads first
                 kmp_uint32 last = tid+thr_bar->skip_per_level[d+1], skip = thr_bar->skip_per_level[d];
                 if (last > nproc) last = nproc;
                 for (child_tid=tid+skip; child_tid<(int)last; child_tid+=skip) {
                     register kmp_info_t *child_thr = other_threads[child_tid];
                     register kmp_bstate_t *child_bar = &child_thr->th.th_bar[bt].bb;
                     KA_TRACE(20, ("__kmp_hierarchical_barrier_gather: T#%d(%d:%d) wait T#%d(%d:%d) "
                                   "arrived(%p) == %llu\n",
                                   gtid, team->t.t_id, tid, __kmp_gtid_from_tid(child_tid, team),
                                   team->t.t_id, child_tid, &child_bar->b_arrived, new_state));
                     kmp_flag_64 flag(&child_bar->b_arrived, new_state);
                     flag.wait(this_thr, FALSE
                               USE_ITT_BUILD_ARG(itt_sync_obj) );
+                    ANNOTATE_BARRIER_AFTER(child_thr);
                     if (reduce) {
                         KA_TRACE(100, ("__kmp_hierarchical_barrier_gather: T#%d(%d:%d) += T#%d(%d:%d)\n",
                                        gtid, team->t.t_id, tid, __kmp_gtid_from_tid(child_tid, team),
                                        team->t.t_id, child_tid));
+                      ANNOTATE_REDUCE_AFTER(reduce);
                         (*reduce)(this_thr->th.th_local.reduce_data, child_thr->th.th_local.reduce_data);
+                      ANNOTATE_REDUCE_BEFORE(reduce);
+                      ANNOTATE_REDUCE_BEFORE(&team->t.t_bar);
                     }
                 }
             }
         }
     }
     // All subordinates are gathered; now release parent if not master thread
 
     if (!KMP_MASTER_TID(tid)) { // worker threads release parent in hierarchy
         KA_TRACE(20, ("__kmp_hierarchical_barrier_gather: T#%d(%d:%d) releasing T#%d(%d:%d) "
                       "arrived(%p): %llu => %llu\n", gtid, team->t.t_id, tid,
                       __kmp_gtid_from_tid(thr_bar->parent_tid, team), team->t.t_id, thr_bar->parent_tid,
                       &thr_bar->b_arrived, thr_bar->b_arrived, thr_bar->b_arrived+KMP_BARRIER_STATE_BUMP));
         /* Mark arrival to parent: After performing this write, a worker thread may not assume that
            the team is valid any more - it could be deallocated by the master thread at any time. */
         if (thr_bar->my_level || __kmp_dflt_blocktime != KMP_MAX_BLOCKTIME
             || !thr_bar->use_oncore_barrier) { // Parent is waiting on my b_arrived flag; release it
+          ANNOTATE_BARRIER_BEFORE(this_thr);
             kmp_flag_64 flag(&thr_bar->b_arrived, other_threads[thr_bar->parent_tid]);
             flag.release();
         }
         else { // Leaf does special release on the "offset" bits of parent's b_arrived flag
             thr_bar->b_arrived = team->t.t_bar[bt].b_arrived + KMP_BARRIER_STATE_BUMP;
             kmp_flag_oncore flag(&thr_bar->parent_bar->b_arrived, thr_bar->offset);
             flag.set_waiter(other_threads[thr_bar->parent_tid]);
             flag.release();
else { // Worker threads
         // Wait for parent thread to release me
         if (!thr_bar->use_oncore_barrier || __kmp_dflt_blocktime != KMP_MAX_BLOCKTIME
             || thr_bar->my_level != 0 || thr_bar->team == NULL) {
             // Use traditional method of waiting on my own b_go flag
             thr_bar->wait_flag = KMP_BARRIER_OWN_FLAG;
             kmp_flag_64 flag(&thr_bar->b_go, KMP_BARRIER_STATE_BUMP);
             flag.wait(this_thr, TRUE
                       USE_ITT_BUILD_ARG(itt_sync_obj) );
+            ANNOTATE_BARRIER_AFTER(this_thr);
             TCW_8(thr_bar->b_go, KMP_INIT_BARRIER_STATE); // Reset my b_go flag for next time
         }
         else { // Thread barrier data is initialized, this is a leaf, blocktime is infinite, not nested
             // Wait on my "offset" bits on parent's b_go flag
             thr_bar->wait_flag = KMP_BARRIER_PARENT_FLAG;
             kmp_flag_oncore flag(&thr_bar->parent_bar->b_go, KMP_BARRIER_STATE_BUMP, thr_bar->offset,
                                  bt, this_thr
                                  USE_ITT_BUILD_ARG(itt_sync_obj) );
if (thr_bar->my_level) { // not a leaf
                         register kmp_info_t   *child_thr = team->t.t_threads[child_tid];
                         register kmp_bstate_t *child_bar = &child_thr->th.th_bar[bt].bb;
                         KA_TRACE(20, ("__kmp_hierarchical_barrier_release: T#%d(%d:%d) releasing"
                                       " T#%d(%d:%d) go(%p): %u => %u\n",
                                       gtid, team->t.t_id, tid, __kmp_gtid_from_tid(child_tid, team),
                                       team->t.t_id, child_tid, &child_bar->b_go, child_bar->b_go,
                                       child_bar->b_go + KMP_BARRIER_STATE_BUMP));
                         // Release child using child's b_go flag
+                        ANNOTATE_BARRIER_BEFORE(child_thr);
                         kmp_flag_64 flag(&child_bar->b_go, child_thr);
                         flag.release();
                     }
                 }
                 else { // Release all children at once with leaf_state bits on my own b_go flag
                     thr_bar->b_go |= thr_bar->leaf_state;
                 }
             }
         }
         else { // Blocktime is not infinite; do a simple hierarchical release
             for (int d=thr_bar->my_level-1; d>=0; --d) { // Release highest level threads first
                 last = tid+thr_bar->skip_per_level[d+1];
                 kmp_uint32 skip = thr_bar->skip_per_level[d];
                 if (last > nproc) last = nproc;
                 for (child_tid=tid+skip; child_tid<(int)last; child_tid+=skip) {
                     register kmp_info_t   *child_thr = team->t.t_threads[child_tid];
                     register kmp_bstate_t *child_bar = &child_thr->th.th_bar[bt].bb;
                     KA_TRACE(20, ("__kmp_hierarchical_barrier_release: T#%d(%d:%d) releasing T#%d(%d:%d)"
                                   " go(%p): %u => %u\n",
                                   gtid, team->t.t_id, tid, __kmp_gtid_from_tid(child_tid, team),
                                   team->t.t_id, child_tid, &child_bar->b_go, child_bar->b_go,
                                   child_bar->b_go + KMP_BARRIER_STATE_BUMP));
                     // Release child using child's b_go flag
+                    ANNOTATE_BARRIER_BEFORE(child_thr);
                     kmp_flag_64 flag(&child_bar->b_go, child_thr);
                     flag.release();
                 }
             }
         }
 #if KMP_BARRIER_ICV_PUSH
         if (propagate_icvs && !KMP_MASTER_TID(tid)) // non-leaves copy ICVs from fixed ICVs to local dest
             copy_icvs(&team->t.t_implicit_task_taskdata[tid].td_icvs, &thr_bar->th_fixed_icvs);

 #if OMPT_SUPPORT
     ompt_task_id_t my_task_id;
     ompt_parallel_id_t my_parallel_id;
 #endif
 
     KA_TRACE(15, ("__kmp_barrier: T#%d(%d:%d) has arrived\n",
                   gtid, __kmp_team_from_gtid(gtid)->t.t_id, __kmp_tid_from_gtid(gtid)));
 
+    ANNOTATE_NEW_BARRIER_BEGIN(&team->t.t_bar);
 #if OMPT_SUPPORT
     if (ompt_enabled) {
 #if OMPT_BLAME
         my_task_id = team->t.t_implicit_task_taskdata[tid].ompt_task_info.task_id;
         my_parallel_id = team->t.ompt_team_info.parallel_id;
 
 #if OMPT_TRACE
         if (this_thr->th.ompt_thread_info.state == ompt_state_wait_single) {
#if OMPT_BLAME
         if (ompt_callbacks.ompt_callback(ompt_event_barrier_end)) {
             ompt_callbacks.ompt_callback(ompt_event_barrier_end)(
                 my_parallel_id, my_task_id);
         }
 #endif
         this_thr->th.ompt_thread_info.state = ompt_state_work_parallel;
     }
 #endif
+    ANNOTATE_NEW_BARRIER_END(&team->t.t_bar);
 
     return status;
 }
 
 
 void
 __kmp_end_split_barrier(enum barrier_type bt, int gtid)
 {
     KMP_TIME_DEVELOPER_BLOCK(KMP_end_split_barrier);
     int tid = __kmp_tid_from_gtid(gtid);
     kmp_info_t *this_thr = __kmp_threads[gtid];
     kmp_team_t *team = this_thr->th.th_team;
 
+    ANNOTATE_NEW_BARRIER_BEGIN(&team->t.t_bar);
     if (!team->t.t_serialized) {
         if (KMP_MASTER_GTID(gtid)) {
             switch (__kmp_barrier_release_pattern[bt]) {
             case bp_hyper_bar: {
                 KMP_ASSERT(__kmp_barrier_release_branch_bits[bt]);
                 __kmp_hyper_barrier_release(bt, this_thr, gtid, tid, FALSE
                                             USE_ITT_BUILD_ARG(NULL) );
                 break;
if (!team->t.t_serialized) {
                                              USE_ITT_BUILD_ARG(NULL) );
             }
             }
             if (__kmp_tasking_mode != tskm_immediate_exec) {
                 __kmp_task_team_sync(this_thr, team);
             } // if
         }
     }
+    ANNOTATE_NEW_BARRIER_END(&team->t.t_bar);
 }
 
 
 void
 __kmp_join_barrier(int gtid)
 {
     KMP_TIME_PARTITIONED_BLOCK(OMP_fork_join_barrier);
     KMP_SET_THREAD_STATE_BLOCK(FORK_JOIN_BARRIER);
#endif /* KMP_DEBUG */
 
     // Verify state
     KMP_DEBUG_ASSERT(__kmp_threads && __kmp_threads[gtid]);
     KMP_DEBUG_ASSERT(TCR_PTR(this_thr->th.th_team));
     KMP_DEBUG_ASSERT(TCR_PTR(this_thr->th.th_root));
     KMP_DEBUG_ASSERT(this_thr == team->t.t_threads[tid]);
     KA_TRACE(10, ("__kmp_join_barrier: T#%d(%d:%d) arrived at join barrier\n", gtid, team_id, tid));
 
+    ANNOTATE_NEW_BARRIER_BEGIN(&team->t.t_bar);
 #if OMPT_SUPPORT
 #if OMPT_TRACE
     if (ompt_enabled &&
         ompt_callbacks.ompt_callback(ompt_event_barrier_begin)) {
         ompt_callbacks.ompt_callback(ompt_event_barrier_begin)(
             team->t.ompt_team_info.parallel_id,
             team->t.t_implicit_task_taskdata[tid].ompt_task_info.task_id);
     }
#if OMPT_BLAME
                 team->t.t_implicit_task_taskdata[tid].ompt_task_info.task_id);
         }
 #endif
 
         // return to default state
         this_thr->th.ompt_thread_info.state = ompt_state_overhead;
     }
 #endif
+    ANNOTATE_NEW_BARRIER_END(&team->t.t_bar);
 }
 
 
 // TODO release worker threads' fork barriers as we are ready instead of all at once
 void
 __kmp_fork_barrier(int gtid, int tid)
 {
     KMP_TIME_PARTITIONED_BLOCK(OMP_fork_join_barrier);
     KMP_SET_THREAD_STATE_BLOCK(FORK_JOIN_BARRIER);
     KMP_TIME_DEVELOPER_BLOCK(KMP_fork_barrier);
     kmp_info_t *this_thr = __kmp_threads[gtid];
     kmp_team_t *team = (tid == 0) ? this_thr->th.th_team : NULL;
 #if USE_ITT_BUILD
     void * itt_sync_obj = NULL;
 #endif /* USE_ITT_BUILD */
+    if (team)
+      ANNOTATE_NEW_BARRIER_END(&team->t.t_bar);
 
     KA_TRACE(10, ("__kmp_fork_barrier: T#%d(%d:%d) has arrived\n",
                   gtid, (team != NULL) ? team->t.t_id : -1, tid));
 
     // th_team pointer only valid for master thread here
     if (KMP_MASTER_TID(tid)) {
 #if USE_ITT_BUILD && USE_ITT_NOTIFY
         if (__itt_sync_create_ptr || KMP_ITT_DEBUG) {
#if USE_ITT_BUILD && USE_ITT_NOTIFY
     if (__itt_sync_create_ptr || KMP_ITT_DEBUG) {
         if (!KMP_MASTER_TID(tid)) {
             // Get correct barrier object
             itt_sync_obj = __kmp_itt_barrier_object(gtid, bs_forkjoin_barrier);
             __kmp_itt_barrier_finished(gtid, itt_sync_obj);  // Workers call acquired
         } // (prepare called inside barrier_release)
     }
 #endif /* USE_ITT_BUILD && USE_ITT_NOTIFY */
+    ANNOTATE_NEW_BARRIER_END(&team->t.t_bar);
     KA_TRACE(10, ("__kmp_fork_barrier: T#%d(%d:%d) is leaving\n", gtid, team->t.t_id, tid));
 }
 
 
 void
 __kmp_setup_icv_copy(kmp_team_t *team, int new_nproc, kmp_internal_control_t *new_icvs, ident_t *loc )
 {
     KMP_TIME_DEVELOPER_BLOCK(KMP_setup_icv_copy);

runtime/src/kmp_config.h.cmake

 #define KMP_LIBRARY_FILE "@LIBOMP_LIB_FILE@"
 #define KMP_VERSION_MAJOR @LIBOMP_VERSION_MAJOR@
 #define KMP_VERSION_MINOR @LIBOMP_VERSION_MINOR@
 #define LIBOMP_OMP_VERSION @LIBOMP_OMP_VERSION@
 #define OMP_50_ENABLED (LIBOMP_OMP_VERSION >= 50)
 #define OMP_45_ENABLED (LIBOMP_OMP_VERSION >= 45)
 #define OMP_40_ENABLED (LIBOMP_OMP_VERSION >= 40)
 #define OMP_30_ENABLED (LIBOMP_OMP_VERSION >= 30)
+#cmakedefine01 LIBOMP_TSAN_SUPPORT
+#if LIBOMP_TSAN_SUPPORT
+#define TSAN_SUPPORT
+#endif
 
 // Configured cache line based on architecture
 #if KMP_ARCH_PPC64
 # define CACHE_LINE 128
 #else
 # define CACHE_LINE 64
 #endif
 

runtime/src/kmp_lock.cpp

 #include <atomic>
 
 #include "kmp.h"
 #include "kmp_itt.h"
 #include "kmp_i18n.h"
 #include "kmp_lock.h"
 #include "kmp_io.h"
 
+#include "tsan_annotations.h"
+
 #if KMP_USE_FUTEX
 # include <unistd.h>
 # include <sys/syscall.h>
 // We should really include <futex.h>, but that causes compatibility problems on different
 // Linux* OS distributions that either require that you include (or break when you try to include)
 // <pci/types.h>.
 // Since all we need is the two macros below (which are part of the kernel ABI, so can't change)
 // we just define the constants here and don't include <futex.h>
#endif /* USE_LOCK_PROFILE */
     }
     KMP_FSYNC_ACQUIRED( lck );
     return KMP_LOCK_ACQUIRED_FIRST;
 }
 
 int
 __kmp_acquire_tas_lock( kmp_tas_lock_t *lck, kmp_int32 gtid )
 {
-    return __kmp_acquire_tas_lock_timed_template( lck, gtid );
+  int retval = __kmp_acquire_tas_lock_timed_template( lck, gtid );
+  ANNOTATE_TAS_ACQUIRED(lck);
+  return retval;
 }
 
 static int
 __kmp_acquire_tas_lock_with_checks( kmp_tas_lock_t *lck, kmp_int32 gtid )
 {
     char const * const func = "omp_set_lock";
     if ( ( sizeof ( kmp_tas_lock_t ) <= OMP_LOCK_T_SIZE )
       && __kmp_is_tas_lock_nestable( lck ) ) {

 }
 
 int
 __kmp_release_tas_lock( kmp_tas_lock_t *lck, kmp_int32 gtid )
 {
     KMP_MB();       /* Flush all pending memory write invalidates.  */
 
     KMP_FSYNC_RELEASING(lck);
+    ANNOTATE_TAS_RELEASED(lck);
     KMP_ST_REL32( &(lck->lk.poll), KMP_LOCK_FREE(tas) );
     KMP_MB();       /* Flush all pending memory write invalidates.  */
 
     KMP_YIELD( TCR_4( __kmp_nth ) > ( __kmp_avail_proc ? __kmp_avail_proc :
       __kmp_xproc ) );
     return KMP_LOCK_RELEASED;
 }
 
__kmp_acquire_nested_tas_lock( kmp_tas_lock_t *lck, kmp_int32 gtid )
     KMP_DEBUG_ASSERT( gtid >= 0 );
 
     if ( __kmp_get_tas_lock_owner( lck ) == gtid ) {
         lck->lk.depth_locked += 1;
         return KMP_LOCK_ACQUIRED_NEXT;
     }
     else {
         __kmp_acquire_tas_lock_timed_template( lck, gtid );
+        ANNOTATE_TAS_ACQUIRED(lck);
         lck->lk.depth_locked = 1;
         return KMP_LOCK_ACQUIRED_FIRST;
     }
 }
 
 static int
 __kmp_acquire_nested_tas_lock_with_checks( kmp_tas_lock_t *lck, kmp_int32 gtid )
 {
#endif /* USE_LOCK_PROFILE */
     KA_TRACE( 1000, ("__kmp_acquire_futex_lock: lck:%p(0x%x), T#%d exiting\n",
       lck, lck->lk.poll, gtid ) );
     return KMP_LOCK_ACQUIRED_FIRST;
 }
 
 int
 __kmp_acquire_futex_lock( kmp_futex_lock_t *lck, kmp_int32 gtid )
 {
-   return __kmp_acquire_futex_lock_timed_template( lck, gtid );
+  int retval = __kmp_acquire_futex_lock_timed_template( lck, gtid );
+  ANNOTATE_FUTEX_ACQUIRED(lck);
+  return retval;
 }
 
 static int
 __kmp_acquire_futex_lock_with_checks( kmp_futex_lock_t *lck, kmp_int32 gtid )
 {
     char const * const func = "omp_set_lock";
     if ( ( sizeof ( kmp_futex_lock_t ) <= OMP_LOCK_T_SIZE )
       && __kmp_is_futex_lock_nestable( lck ) ) {

 __kmp_release_futex_lock( kmp_futex_lock_t *lck, kmp_int32 gtid )
 {
     KMP_MB();       /* Flush all pending memory write invalidates.  */
 
     KA_TRACE( 1000, ("__kmp_release_futex_lock: lck:%p(0x%x), T#%d entering\n",
       lck, lck->lk.poll, gtid ) );
 
     KMP_FSYNC_RELEASING(lck);
+    ANNOTATE_FUTEX_RELEASED(lck);
 
     kmp_int32 poll_val = KMP_XCHG_FIXED32( & ( lck->lk.poll ), KMP_LOCK_FREE(futex) );
 
     KA_TRACE( 1000, ("__kmp_release_futex_lock: lck:%p, T#%d released poll_val = 0x%x\n",
        lck, gtid, poll_val ) );
 
     if ( KMP_LOCK_STRIP(poll_val) & 1 ) {
         KA_TRACE( 1000, ("__kmp_release_futex_lock: lck:%p, T#%d futex_wake 1 thread\n",
__kmp_acquire_nested_futex_lock( kmp_futex_lock_t *lck, kmp_int32 gtid )
     KMP_DEBUG_ASSERT( gtid >= 0 );
 
     if ( __kmp_get_futex_lock_owner( lck ) == gtid ) {
         lck->lk.depth_locked += 1;
         return KMP_LOCK_ACQUIRED_NEXT;
     }
     else {
         __kmp_acquire_futex_lock_timed_template( lck, gtid );
+        ANNOTATE_FUTEX_ACQUIRED(lck);
         lck->lk.depth_locked = 1;
         return KMP_LOCK_ACQUIRED_FIRST;
     }
 }
 
 static int
 __kmp_acquire_nested_futex_lock_with_checks( kmp_futex_lock_t *lck, kmp_int32 gtid )
 {
#endif /* USE_LOCK_PROFILE */
     }
     KMP_WAIT_YIELD_PTR( &lck->lk.now_serving, my_ticket, __kmp_bakery_check, lck );
     return KMP_LOCK_ACQUIRED_FIRST;
 }
 
 int
 __kmp_acquire_ticket_lock( kmp_ticket_lock_t *lck, kmp_int32 gtid )
 {
-    return __kmp_acquire_ticket_lock_timed_template( lck, gtid );
+  int retval = __kmp_acquire_ticket_lock_timed_template( lck, gtid );
+  ANNOTATE_TICKET_ACQUIRED(lck);
+  return retval;
 }
 
 static int
 __kmp_acquire_ticket_lock_with_checks( kmp_ticket_lock_t *lck, kmp_int32 gtid )
 {
     char const * const func = "omp_set_lock";
 
     if ( ! std::atomic_load_explicit( &lck->lk.initialized, std::memory_order_relaxed ) ) {
__kmp_test_ticket_lock_with_checks( kmp_ticket_lock_t *lck, kmp_int32 gtid )
     return retval;
 }
 
 int
 __kmp_release_ticket_lock( kmp_ticket_lock_t *lck, kmp_int32 gtid )
 {
     kmp_uint32 distance = std::atomic_load_explicit( &lck->lk.next_ticket, std::memory_order_relaxed ) - std::atomic_load_explicit( &lck->lk.now_serving, std::memory_order_relaxed );
 
+    ANNOTATE_TICKET_RELEASED(lck);
     std::atomic_fetch_add_explicit( &lck->lk.now_serving, 1U, std::memory_order_release );
 
     KMP_YIELD( distance
       > (kmp_uint32) (__kmp_avail_proc ? __kmp_avail_proc : __kmp_xproc) );
     return KMP_LOCK_RELEASED;
 }
 
 static int
__kmp_acquire_nested_ticket_lock( kmp_ticket_lock_t *lck, kmp_int32 gtid )
     KMP_DEBUG_ASSERT( gtid >= 0 );
 
     if ( __kmp_get_ticket_lock_owner( lck ) == gtid ) {
         std::atomic_fetch_add_explicit( &lck->lk.depth_locked, 1, std::memory_order_relaxed );
         return KMP_LOCK_ACQUIRED_NEXT;
     }
     else {
         __kmp_acquire_ticket_lock_timed_template( lck, gtid );
+        ANNOTATE_TICKET_ACQUIRED(lck);
         std::atomic_store_explicit( &lck->lk.depth_locked, 1, std::memory_order_relaxed );
         std::atomic_store_explicit( &lck->lk.owner_id, gtid + 1, std::memory_order_relaxed );
         return KMP_LOCK_ACQUIRED_FIRST;
     }
 }
 
 static int
 __kmp_acquire_nested_ticket_lock_with_checks( kmp_ticket_lock_t *lck, kmp_int32 gtid )
#endif
     return KMP_LOCK_ACQUIRED_FIRST;
 }
 
 int
 __kmp_acquire_queuing_lock( kmp_queuing_lock_t *lck, kmp_int32 gtid )
 {
     KMP_DEBUG_ASSERT( gtid >= 0 );
 
-    return __kmp_acquire_queuing_lock_timed_template<false>( lck, gtid );
+    int retval = __kmp_acquire_queuing_lock_timed_template<false>( lck, gtid );
+    ANNOTATE_QUEUING_ACQUIRED(lck);
+    return retval;
 }
 
 static int
 __kmp_acquire_queuing_lock_with_checks( kmp_queuing_lock_t *lck,
   kmp_int32 gtid )
 {
     char const * const func = "omp_set_lock";
     if ( lck->lk.initialized != lck ) {
#endif
 
     if ( head == 0 ) { /* nobody on queue, nobody holding */
 
         /* try (0,0)->(-1,0) */
 
         if ( KMP_COMPARE_AND_STORE_ACQ32( head_id_p, 0, -1 ) ) {
             KA_TRACE( 1000, ("__kmp_test_queuing_lock: T#%d exiting: holding lock\n", gtid ));
             KMP_FSYNC_ACQUIRED(lck);
+            ANNOTATE_QUEUING_ACQUIRED(lck);
             return TRUE;
         }
     }
 
     KA_TRACE( 1000, ("__kmp_test_queuing_lock: T#%d exiting: without lock\n", gtid ));
     return FALSE;
 }
 
if ( this_thr->th.th_spin_here )
         __kmp_dump_queuing_lock( this_thr, gtid, lck, *head_id_p, *tail_id_p );
     if ( this_thr->th.th_next_waiting != 0 )
         __kmp_dump_queuing_lock( this_thr, gtid, lck, *head_id_p, *tail_id_p );
 #endif
     KMP_DEBUG_ASSERT( !this_thr->th.th_spin_here );
     KMP_DEBUG_ASSERT( this_thr->th.th_next_waiting == 0 );
 
     KMP_FSYNC_RELEASING(lck);
+    ANNOTATE_QUEUING_RELEASED(lck);
 
     while( 1 ) {
         kmp_int32 dequeued;
         kmp_int32 head;
         kmp_int32 tail;
 
         head = *head_id_p;
 
__kmp_acquire_nested_queuing_lock( kmp_queuing_lock_t *lck, kmp_int32 gtid )
     KMP_DEBUG_ASSERT( gtid >= 0 );
 
     if ( __kmp_get_queuing_lock_owner( lck ) == gtid ) {
         lck->lk.depth_locked += 1;
         return KMP_LOCK_ACQUIRED_NEXT;
     }
     else {
         __kmp_acquire_queuing_lock_timed_template<false>( lck, gtid );
+        ANNOTATE_QUEUING_ACQUIRED(lck);
         KMP_MB();
         lck->lk.depth_locked = 1;
         KMP_MB();
         lck->lk.owner_id = gtid + 1;
         return KMP_LOCK_ACQUIRED_FIRST;
     }
 }
 
__kmp_acquire_adaptive_lock( kmp_adaptive_lock_t * lck, kmp_int32 gtid )
 
     // Speculative acquisition failed, so acquire it non-speculatively.
     // Count the non-speculative acquire attempt
     lck->lk.adaptive.acquire_attempts++;
 
     __kmp_acquire_queuing_lock_timed_template<FALSE>( GET_QLK_PTR(lck), gtid );
     // We have acquired the base lock, so count that.
     KMP_INC_STAT(lck,nonSpeculativeAcquires );
+    ANNOTATE_QUEUING_ACQUIRED(lck);
 }
 
 static void
 __kmp_acquire_adaptive_lock_with_checks( kmp_adaptive_lock_t *lck, kmp_int32 gtid )
 {
     char const * const func = "omp_set_lock";
     if ( lck->lk.qlk.initialized != GET_QLK_PTR(lck) ) {
         KMP_FATAL( LockIsUninitialized, func );
if (lck->lk.old_polls == NULL) {
         }
     }
     return KMP_LOCK_ACQUIRED_FIRST;
 }
 
 int
 __kmp_acquire_drdpa_lock( kmp_drdpa_lock_t *lck, kmp_int32 gtid )
 {
-    return __kmp_acquire_drdpa_lock_timed_template( lck, gtid );
+    int retval = __kmp_acquire_drdpa_lock_timed_template( lck, gtid );
+    ANNOTATE_DRDPA_ACQUIRED(lck);
+    return retval;
 }
 
 static int
 __kmp_acquire_drdpa_lock_with_checks( kmp_drdpa_lock_t *lck, kmp_int32 gtid )
 {
     char const * const func = "omp_set_lock";
     if ( lck->lk.initialized != lck ) {
         KMP_FATAL( LockIsUninitialized, func );
__kmp_release_drdpa_lock( kmp_drdpa_lock_t *lck, kmp_int32 gtid )
     kmp_uint64 ticket = lck->lk.now_serving + 1;        // non-volatile load
     volatile struct kmp_base_drdpa_lock::kmp_lock_poll *polls
       = (volatile struct kmp_base_drdpa_lock::kmp_lock_poll *)
       TCR_PTR(lck->lk.polls);                           // volatile load
     kmp_uint64 mask = TCR_8(lck->lk.mask);              // volatile load
     KA_TRACE(1000, ("__kmp_release_drdpa_lock: ticket #%lld released lock %p\n",
        ticket - 1, lck));
     KMP_FSYNC_RELEASING(lck);
+    ANNOTATE_DRDPA_RELEASED(lck);
     KMP_ST_REL64(&(polls[ticket & mask].poll), ticket); // volatile store
     return KMP_LOCK_RELEASED;
 }
 
 static int
 __kmp_release_drdpa_lock_with_checks( kmp_drdpa_lock_t *lck, kmp_int32 gtid )
 {
     char const * const func = "omp_unset_lock";
__kmp_acquire_nested_drdpa_lock( kmp_drdpa_lock_t *lck, kmp_int32 gtid )
     KMP_DEBUG_ASSERT( gtid >= 0 );
 
     if ( __kmp_get_drdpa_lock_owner( lck ) == gtid ) {
         lck->lk.depth_locked += 1;
         return KMP_LOCK_ACQUIRED_NEXT;
     }
     else {
         __kmp_acquire_drdpa_lock_timed_template( lck, gtid );
+        ANNOTATE_DRDPA_ACQUIRED(lck);
         KMP_MB();
         lck->lk.depth_locked = 1;
         KMP_MB();
         lck->lk.owner_id = gtid + 1;
         return KMP_LOCK_ACQUIRED_FIRST;
     }
 }
 
kmp_lock_flags_t flags )
     kmp_user_lock_p lck;
     kmp_lock_index_t index;
     KMP_DEBUG_ASSERT( user_lock );
 
     __kmp_acquire_lock( &__kmp_global_lock, gtid );
 
     if ( __kmp_lock_pool == NULL ) {
         // Lock pool is empty. Allocate new memory.
+
+        // ANNOTATION: Found no good way to express the syncronisation
+        // between allocation and usage, so ignore the allocation
+        ANNOTATE_IGNORE_WRITES_BEGIN();
         if ( __kmp_num_locks_in_block <= 1 ) { // Tune this cutoff point.
             lck = (kmp_user_lock_p) __kmp_allocate( __kmp_user_lock_size );
         }
         else {
             lck = __kmp_lock_block_allocate();
         }
+        ANNOTATE_IGNORE_WRITES_END();
 
         // Insert lock in the table so that it can be freed in __kmp_cleanup,
         // and debugger has info on all allocated locks.
         index = __kmp_lock_table_insert( lck );
     }
     else {
         // Pick up lock from pool.
         lck = __kmp_lock_pool;

runtime/src/kmp_runtime.c

 
 /* these are temporary issues to be dealt with */
 #define KMP_USE_PRCTL 0
 
 #if KMP_OS_WINDOWS
 #include <process.h>
 #endif
 
+#include "tsan_annotations.h"
 
 #if defined(KMP_GOMP_COMPAT)
 char const __kmp_version_alt_comp[] = KMP_VERSION_PREFIX "alternative compiler support: yes";
 #endif /* defined(KMP_GOMP_COMPAT) */
 
 char const __kmp_version_omp_api[] = KMP_VERSION_PREFIX "API version: "
 #if OMP_45_ENABLED
     "4.5 (201511)";
) {
     gtid = thread->th.th_info.ds.ds_gtid;
 
     if ( ! is_root ) {
 
         if ( __kmp_dflt_blocktime != KMP_MAX_BLOCKTIME ) {
             /* Assume the threads are at the fork barrier here */
             KA_TRACE( 20, ("__kmp_reap_thread: releasing T#%d from fork barrier for reap\n", gtid ) );
             /* Need release fence here to prevent seg faults for tree forkjoin barrier (GEH) */
+            ANNOTATE_HAPPENS_BEFORE(thread);
             kmp_flag_64 flag(&thread->th.th_bar[ bs_forkjoin_barrier ].bb.b_go, thread);
             __kmp_release_64(&flag);
         }; // if
 
         // Terminate OS thread.
         __kmp_reap_worker( thread );
 
         //

runtime/src/kmp_tasking.c

 #include "kmp_itt.h"
 #include "kmp_wait_release.h"
 #include "kmp_stats.h"
 
 #if OMPT_SUPPORT
 #include "ompt-specific.h"
 #endif
 
+#include "tsan_annotations.h"
+
 /* ------------------------------------------------------------------------ */
 /* ------------------------------------------------------------------------ */
 
 
 /* forward declaration */
 static void __kmp_enable_tasking( kmp_task_team_t *task_team, kmp_info_t *this_thr );
 static void __kmp_alloc_task_deque( kmp_info_t *thread, kmp_thread_data_t *thread_data );
 static int  __kmp_realloc_task_threads_data( kmp_info_t *thread, kmp_task_team_t *task_team );
__kmp_free_task( kmp_int32 gtid, kmp_taskdata_t * taskdata, kmp_info_t * thread )
     KMP_DEBUG_ASSERT( taskdata->td_flags.tasktype == TASK_EXPLICIT );
     KMP_DEBUG_ASSERT( taskdata->td_flags.executing == 0 );
     KMP_DEBUG_ASSERT( taskdata->td_flags.complete == 1 );
     KMP_DEBUG_ASSERT( taskdata->td_flags.freed == 0 );
     KMP_DEBUG_ASSERT( TCR_4(taskdata->td_allocated_child_tasks) == 0  || taskdata->td_flags.task_serial == 1);
     KMP_DEBUG_ASSERT( TCR_4(taskdata->td_incomplete_child_tasks) == 0 );
 
     taskdata->td_flags.freed = 1;
+    ANNOTATE_HAPPENS_BEFORE(taskdata);
     // deallocate the taskdata and shared variable blocks associated with this task
     #if USE_FAST_MEMORY
         __kmp_fast_free( thread, taskdata );
     #else /* ! USE_FAST_MEMORY */
         __kmp_thread_free( thread, taskdata );
     #endif
 
     KA_TRACE(20, ("__kmp_free_task: T#%d freed task %p\n",
KA_TRACE(30, ("__kmp_task_alloc: T#%d Second malloc size: %ld\n",
                   gtid, sizeof_shareds) );
 
     // Avoid double allocation here by combining shareds with taskdata
     #if USE_FAST_MEMORY
     taskdata = (kmp_taskdata_t *) __kmp_fast_allocate( thread, shareds_offset + sizeof_shareds );
     #else /* ! USE_FAST_MEMORY */
     taskdata = (kmp_taskdata_t *) __kmp_thread_malloc( thread, shareds_offset + sizeof_shareds );
     #endif /* USE_FAST_MEMORY */
+    ANNOTATE_HAPPENS_AFTER(taskdata);
 
     task                      = KMP_TASKDATA_TO_TASK(taskdata);
 
     // Make sure task & taskdata are aligned appropriately
 #if KMP_ARCH_X86 || KMP_ARCH_PPC64 || !KMP_HAVE_QUAD
     KMP_DEBUG_ASSERT( ( ((kmp_uintptr_t)taskdata) & (sizeof(double)-1) ) == 0 );
     KMP_DEBUG_ASSERT( ( ((kmp_uintptr_t)task) & (sizeof(double)-1) ) == 0 );
 #else
#endif
         // Only need to keep track of allocated child tasks for explicit tasks since implicit not deallocated
         if ( taskdata->td_parent->td_flags.tasktype == TASK_EXPLICIT ) {
             KMP_TEST_THEN_INC32( (kmp_int32 *)(& taskdata->td_parent->td_allocated_child_tasks) );
         }
     }
 
     KA_TRACE(20, ("__kmp_task_alloc(exit): T#%d created task %p parent=%p\n",
                   gtid, taskdata, taskdata->td_parent) );
+    ANNOTATE_HAPPENS_BEFORE(task);
 
 #if OMPT_SUPPORT
     __kmp_task_init_ompt(taskdata, gtid, (void*) task_entry);
 #endif
 
     return task;
 }
 

 
 #if USE_ITT_BUILD && USE_ITT_NOTIFY
     if(__kmp_forkjoin_frames_mode == 3) {
         // Get the current time stamp to measure task execution time to correct barrier imbalance time
         cur_time = __itt_get_timestamp();
     }
 #endif
 
 #if OMP_45_ENABLED
     // Proxy tasks are not handled by the runtime
     if ( taskdata->td_flags.proxy != TASK_PROXY )
 #endif
+    ANNOTATE_HAPPENS_AFTER(task);
     __kmp_task_start( gtid, task, current_task );
 
 #if OMPT_SUPPORT
     ompt_thread_info_t oldInfo;
     kmp_info_t * thread;
     if (ompt_enabled) {
         // Store the threads states and restore them after the task
         thread = __kmp_threads[ gtid ];
         oldInfo = thread->th.ompt_thread_info;
#if OMPT_SUPPORT
     if (ompt_enabled) {
         thread->th.ompt_thread_info = oldInfo;
         taskdata->ompt_task_info.frame.exit_runtime_frame = NULL;
     }
 #endif
 
 #if OMP_45_ENABLED
     // Proxy tasks are not handled by the runtime
     if ( taskdata->td_flags.proxy != TASK_PROXY )
 #endif
+      ANNOTATE_HAPPENS_BEFORE(taskdata->td_parent);
        __kmp_task_finish( gtid, task, current_task );
 
 #if USE_ITT_BUILD && USE_ITT_NOTIFY
     // Barrier imbalance - correct arrive time after the task finished
     if(__kmp_forkjoin_frames_mode == 3) {
         kmp_info_t *this_thr = __kmp_threads [ gtid ];
         if(this_thr->th.th_bar_arrive_time) {
             this_thr->th.th_bar_arrive_time += (__itt_get_timestamp() - cur_time);
         }
if ( __kmp_push_task( gtid, new_task ) == TASK_NOT_PUSHED ) // if cannot defer
         new_taskdata->td_flags.task_serial = 1;
         __kmp_invoke_task( gtid, new_task, current_task );
     }
 
     KA_TRACE(10, ("__kmpc_omp_task_parts(exit): T#%d returning TASK_CURRENT_NOT_QUEUED: "
                   "loc=%p task=%p, return: TASK_CURRENT_NOT_QUEUED\n", gtid, loc_ref,
                   new_taskdata ) );
 
+    ANNOTATE_HAPPENS_BEFORE(new_task);
     return TASK_CURRENT_NOT_QUEUED;
 }
 
 //---------------------------------------------------------------------
 // __kmp_omp_task: Schedule a non-thread-switchable task for execution
 // gtid: Global Thread ID of encountering thread
 // new_task: non-thread-switchable task thunk allocated by __kmp_omp_task_alloc()
 // serialize_immediate: if TRUE then if the task is executed immediately its execution will be serialized
#endif
     }
 
 #if OMPT_SUPPORT
     if (ompt_enabled) {
         new_taskdata->ompt_task_info.frame.reenter_runtime_frame = NULL;
     }
 #endif
 
+    ANNOTATE_HAPPENS_BEFORE(new_task);
     return TASK_CURRENT_NOT_QUEUED;
 }
 
 //---------------------------------------------------------------------
 // __kmpc_omp_task: Wrapper around __kmp_omp_task to schedule a non-thread-switchable task from
 // the parent thread only!
 // loc_ref: location of original task pragma (ignored)
 // gtid: Global Thread ID of encountering thread
#if OMPT_SUPPORT && OMPT_TRACE
         if (ompt_enabled) {
             if (ompt_callbacks.ompt_callback(ompt_event_taskwait_end)) {
                 ompt_callbacks.ompt_callback(ompt_event_taskwait_end)(
                                 my_parallel_id, my_task_id);
             }
             taskdata->ompt_task_info.frame.reenter_runtime_frame = NULL;
         }
 #endif
+        ANNOTATE_HAPPENS_AFTER(taskdata);
     }
 
     KA_TRACE(10, ("__kmpc_omp_taskwait(exit): T#%d task %p finished waiting, "
                   "returning TASK_CURRENT_NOT_QUEUED\n", gtid, taskdata) );
 
     return TASK_CURRENT_NOT_QUEUED;
 }
 
#endif /* USE_ITT_BUILD */
     }
     KMP_DEBUG_ASSERT( taskgroup->count == 0 );
 
     // Restore parent taskgroup for the current task
     taskdata->td_taskgroup = taskgroup->parent;
     __kmp_thread_free( thread, taskgroup );
 
     KA_TRACE(10, ("__kmpc_end_taskgroup(exit): T#%d task %p finished waiting\n", gtid, taskdata) );
+    ANNOTATE_HAPPENS_AFTER(taskdata);
 }
 #endif
 
 
 //------------------------------------------------------
 // __kmp_remove_my_task: remove a task from my own deque
 
 static kmp_task_t *
#endif // BUILD_TIED_TASK_STACK
             }
             else {
                 KE_TRACE( 10, ( "__kmp_realloc_task_threads_data: T#%d allocating "
                                "threads data for task_team %p, size = %d\n",
                                __kmp_gtid_from_thread( thread ), task_team, nthreads ) );
                 // Make the initial allocate for threads_data array, and zero entries
                 // Cannot use __kmp_thread_calloc() because threads not around for
                 // kmp_reap_task_team( ).
+                ANNOTATE_IGNORE_WRITES_BEGIN();
                 *threads_data_p = (kmp_thread_data_t *)
                                   __kmp_allocate( nthreads * sizeof(kmp_thread_data_t) );
+                ANNOTATE_IGNORE_WRITES_END();
 #ifdef BUILD_TIED_TASK_STACK
                 // GEH: Figure out if this is the right thing to do
                 for (i = 0; i < nthreads; i++) {
                     kmp_thread_data_t *thread_data = & (*threads_data_p)[i];
                     __kmp_init_task_stack( __kmp_gtid_from_thread( thread ), thread_data );
                 }
 #endif // BUILD_TIED_TASK_STACK
             }

runtime/src/tsan_annotations.h

+/*! \file */
+/*
+ * tsan_annotations.h -- ThreadSanitizer annotations to support data
+ * race detection in OpenMP programs.
+ */
+
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef TSAN_ANNOTATIONS_H
+#define TSAN_ANNOTATIONS_H
+
+#include "kmp_config.h"
+
+/* types as used in tsan/rtl/tsan_interface_ann.cc */
+typedef unsigned long uptr;
+typedef signed   long sptr;
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Declaration of all annotation functions in tsan/rtl/tsan_interface_ann.cc */
+void AnnotateHappensBefore(const char *f, int l, uptr addr);
+void AnnotateHappensAfter(const char *f, int l, uptr addr);
+void AnnotateCondVarSignal(const char *f, int l, uptr cv);
+void AnnotateCondVarSignalAll(const char *f, int l, uptr cv);
+void AnnotateMutexIsNotPHB(const char *f, int l, uptr mu);
+void AnnotateCondVarWait(const char *f, int l, uptr cv, uptr lock);
+void AnnotateRWLockCreate(const char *f, int l, uptr m);
+void AnnotateRWLockCreateStatic(const char *f, int l, uptr m);
+void AnnotateRWLockDestroy(const char *f, int l, uptr m);
+void AnnotateRWLockAcquired(const char *f, int l, uptr m, uptr is_w);
+void AnnotateRWLockReleased(const char *f, int l, uptr m, uptr is_w);
+void AnnotateTraceMemory(const char *f, int l, uptr mem);
+void AnnotateFlushState(const char *f, int l);
+void AnnotateNewMemory(const char *f, int l, uptr mem, uptr size);
+void AnnotateNoOp(const char *f, int l, uptr mem);
+void AnnotateFlushExpectedRaces(const char *f, int l);
+void AnnotateEnableRaceDetection( const char *f, int l, int enable);
+void AnnotateMutexIsUsedAsCondVar( const char *f, int l, uptr mu);
+void AnnotatePCQGet( const char *f, int l, uptr pcq);
+void AnnotatePCQPut( const char *f, int l, uptr pcq);
+void AnnotatePCQDestroy( const char *f, int l, uptr pcq);
+void AnnotatePCQCreate( const char *f, int l, uptr pcq);
+void AnnotateExpectRace( const char *f, int l, uptr mem, char *desc);
+void AnnotateBenignRaceSized( const char *f, int l, uptr mem, uptr size, char *desc);
+void AnnotateBenignRace( const char *f, int l, uptr mem, char *desc);
+void AnnotateIgnoreReadsBegin(const char *f, int l);
+void AnnotateIgnoreReadsEnd(const char *f, int l);
+void AnnotateIgnoreWritesBegin(const char *f, int l);
+void AnnotateIgnoreWritesEnd(const char *f, int l);
+void AnnotateIgnoreSyncBegin(const char *f, int l);
+void AnnotateIgnoreSyncEnd(const char *f, int l);
+void AnnotatePublishMemoryRange( const char *f, int l, uptr addr, uptr size);
+void AnnotateUnpublishMemoryRange( const char *f, int l, uptr addr, uptr size);
+void AnnotateThreadName( const char *f, int l, char *name);
+void WTFAnnotateHappensBefore(const char *f, int l, uptr addr);
+void WTFAnnotateHappensAfter(const char *f, int l, uptr addr);
+void WTFAnnotateBenignRaceSized( const char *f, int l, uptr mem, uptr sz, char *desc);
+int RunningOnValgrind();
+double ValgrindSlowdown(void);
+const char * ThreadSanitizerQuery(const char *query);
+void AnnotateMemoryIsInitialized(const char *f, int l, uptr mem, uptr sz);
+
+#ifdef __cplusplus
+}
+#endif
+
+#ifdef TSAN_SUPPORT
+#define ANNOTATE_HAPPENS_AFTER(addr) AnnotateHappensAfter(__FILE__, __LINE__, (uptr)addr)
+#define ANNOTATE_HAPPENS_BEFORE(addr) AnnotateHappensBefore(__FILE__, __LINE__, (uptr)addr)
+#define ANNOTATE_IGNORE_WRITES_BEGIN() AnnotateIgnoreWritesBegin(__FILE__, __LINE__)
+#define ANNOTATE_IGNORE_WRITES_END() AnnotateIgnoreWritesEnd(__FILE__, __LINE__)
+#define ANNOTATE_RWLOCK_CREATE(lck) AnnotateRWLockCreate(__FILE__, __LINE__, (uptr)lck)
+#define ANNOTATE_RWLOCK_RELEASED(lck) AnnotateRWLockAcquired(__FILE__, __LINE__, (uptr)lck, 1)
+#define ANNOTATE_RWLOCK_ACQUIRED(lck) AnnotateRWLockReleased(__FILE__, __LINE__, (uptr)lck, 1)
+
+/* new higher level barrier annotations */
+#define ANNOTATE_NEW_BARRIER_BEGIN(addr) AnnotateHappensBefore(__FILE__, __LINE__, (uptr)addr)
+#define ANNOTATE_NEW_BARRIER_END(addr) AnnotateHappensAfter(__FILE__, __LINE__, (uptr)addr)
+// #define ANNOTATE_NEW_BARRIER_BEGIN(addr)
+// #define ANNOTATE_NEW_BARRIER_END(addr)
+
+/* old fine-grain barrier annotations; are replaced by higher level annotation */
+#define ANNOTATE_BARRIER_AFTER(addr) AnnotateHappensAfter(__FILE__, __LINE__, (uptr)addr)
+#define ANNOTATE_BARRIER_BEFORE(addr) AnnotateHappensBefore(__FILE__, __LINE__, (uptr)addr)
+// #define ANNOTATE_BARRIER_AFTER(addr)
+// #define ANNOTATE_BARRIER_BEFORE(addr)
+
+#define ANNOTATE_REDUCE_AFTER(addr) AnnotateHappensAfter(__FILE__, __LINE__, (uptr)addr)
+#define ANNOTATE_REDUCE_BEFORE(addr) AnnotateHappensBefore(__FILE__, __LINE__, (uptr)addr)
+// #define ANNOTATE_REDUCE_AFTER(addr)
+// #define ANNOTATE_REDUCE_BEFORE(addr)
+
+#else
+#define ANNOTATE_HAPPENS_AFTER(addr)
+#define ANNOTATE_HAPPENS_BEFORE(addr)
+#define ANNOTATE_IGNORE_WRITES_BEGIN()
+#define ANNOTATE_IGNORE_WRITES_END()
+#define ANNOTATE_RWLOCK_CREATE(lck)
+#define ANNOTATE_RWLOCK_RELEASED(lck)
+#define ANNOTATE_RWLOCK_ACQUIRED(lck)
+#define ANNOTATE_NEW_BARRIER_BEGIN(addr)
+#define ANNOTATE_NEW_BARRIER_END(addr)
+#define ANNOTATE_BARRIER_AFTER(addr)
+#define ANNOTATE_BARRIER_BEFORE(addr)
+#define ANNOTATE_REDUCE_AFTER(addr)
+#define ANNOTATE_REDUCE_BEFORE(addr)
+#endif
+
+#define ANNOTATE_QUEUING
+#define ANNOTATE_TICKET
+#define ANNOTATE_FUTEX
+#define ANNOTATE_TAS
+#define ANNOTATE_DRDPA
+
+#ifdef ANNOTATE_QUEUING
+#define ANNOTATE_QUEUING_CREATE(lck)
+#define ANNOTATE_QUEUING_RELEASED(lck) ANNOTATE_HAPPENS_BEFORE(lck)
+#define ANNOTATE_QUEUING_ACQUIRED(lck) ANNOTATE_HAPPENS_AFTER(lck)
+#else
+#define ANNOTATE_QUEUING_CREATE(lck)
+#define ANNOTATE_QUEUING_RELEASED(lck)
+#define ANNOTATE_QUEUING_ACQUIRED(lck)
+#endif
+
+#ifdef ANNOTATE_TICKET
+#define ANNOTATE_TICKET_CREATE(lck)
+#define ANNOTATE_TICKET_RELEASED(lck) ANNOTATE_HAPPENS_BEFORE(lck)
+#define ANNOTATE_TICKET_ACQUIRED(lck) ANNOTATE_HAPPENS_AFTER(lck)
+#else
+#define ANNOTATE_TICKET_CREATE(lck)
+#define ANNOTATE_TICKET_RELEASED(lck)
+#define ANNOTATE_TICKET_ACQUIRED(lck)
+#endif
+
+#ifdef ANNOTATE_FUTEX
+#define ANNOTATE_FUTEX_CREATE(lck)
+#define ANNOTATE_FUTEX_RELEASED(lck) ANNOTATE_HAPPENS_BEFORE(lck)
+#define ANNOTATE_FUTEX_ACQUIRED(lck) ANNOTATE_HAPPENS_AFTER(lck)
+#else
+#define ANNOTATE_FUTEX_CREATE(lck)
+#define ANNOTATE_FUTEX_RELEASED(lck)
+#define ANNOTATE_FUTEX_ACQUIRED(lck)
+#endif
+
+#ifdef ANNOTATE_TAS
+#define ANNOTATE_TAS_CREATE(lck)
+#define ANNOTATE_TAS_RELEASED(lck) ANNOTATE_HAPPENS_BEFORE(lck)
+#define ANNOTATE_TAS_ACQUIRED(lck) ANNOTATE_HAPPENS_AFTER(lck)
+#else
+#define ANNOTATE_TAS_CREATE(lck)
+#define ANNOTATE_TAS_RELEASED(lck)
+#define ANNOTATE_TAS_ACQUIRED(lck)
+#endif
+
+#ifdef ANNOTATE_DRDPA
+#define ANNOTATE_DRDPA_CREATE(lck)
+#define ANNOTATE_DRDPA_RELEASED(lck) ANNOTATE_HAPPENS_BEFORE(lck)
+#define ANNOTATE_DRDPA_ACQUIRED(lck) ANNOTATE_HAPPENS_AFTER(lck)
+#else
+#define ANNOTATE_DRDPA_CREATE(lck)
+#define ANNOTATE_DRDPA_RELEASED(lck)
+#define ANNOTATE_DRDPA_ACQUIRED(lck)
+#endif
+
+#endif

runtime/src/tsan_annotations.c

+/*
+ * tsan_annotations.c -- ThreadSanitizer annotations to support data
+ * race detection in OpenMP programs.
+ */
+
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.txt for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "tsan_annotations.h"
+
+#include <stdio.h>
+
+typedef unsigned long uptr;
+typedef signed   long sptr;
+
+extern "C" __attribute__((weak)) void AnnotateHappensBefore(const char *f, int l, uptr addr) {}
+extern "C" __attribute__((weak)) void AnnotateHappensAfter(const char *f, int l, uptr addr) {}
+extern "C" __attribute__((weak)) void AnnotateCondVarSignal(const char *f, int l, uptr cv) {}
+extern "C" __attribute__((weak)) void AnnotateCondVarSignalAll(const char *f, int l, uptr cv) {}
+extern "C" __attribute__((weak)) void AnnotateMutexIsNotPHB(const char *f, int l, uptr mu) {}
+extern "C" __attribute__((weak)) void AnnotateCondVarWait(const char *f, int l, uptr cv, uptr lock) {}
+extern "C" __attribute__((weak)) void AnnotateRWLockCreate(const char *f, int l, uptr m) {}
+extern "C" __attribute__((weak)) void AnnotateRWLockCreateStatic(const char *f, int l, uptr m) {}
+extern "C" __attribute__((weak)) void AnnotateRWLockDestroy(const char *f, int l, uptr m) {}
+extern "C" __attribute__((weak)) void AnnotateRWLockAcquired(const char *f, int l, uptr m, uptr is_w) {}
+extern "C" __attribute__((weak)) void AnnotateRWLockReleased(const char *f, int l, uptr m, uptr is_w) {}
+extern "C" __attribute__((weak)) void AnnotateTraceMemory(const char *f, int l, uptr mem) {}
+extern "C" __attribute__((weak)) void AnnotateFlushState(const char *f, int l) {}
+extern "C" __attribute__((weak)) void AnnotateNewMemory(const char *f, int l, uptr mem, uptr size) {}
+extern "C" __attribute__((weak)) void AnnotateNoOp(const char *f, int l, uptr mem) {}
+extern "C" __attribute__((weak)) void AnnotateFlushExpectedRaces(const char *f, int l) {}
+extern "C" __attribute__((weak)) void AnnotateEnableRaceDetection( const char *f, int l, int enable) {}
+extern "C" __attribute__((weak)) void AnnotateMutexIsUsedAsCondVar( const char *f, int l, uptr mu) {}
+extern "C" __attribute__((weak)) void AnnotatePCQGet( const char *f, int l, uptr pcq) {}
+extern "C" __attribute__((weak)) void AnnotatePCQPut( const char *f, int l, uptr pcq) {}
+extern "C" __attribute__((weak)) void AnnotatePCQDestroy( const char *f, int l, uptr pcq) {}
+extern "C" __attribute__((weak)) void AnnotatePCQCreate( const char *f, int l, uptr pcq) {}
+extern "C" __attribute__((weak)) void AnnotateExpectRace( const char *f, int l, uptr mem, char *desc) {}
+extern "C" __attribute__((weak)) void AnnotateBenignRaceSized( const char *f, int l, uptr mem, uptr size, char *desc) {}
+extern "C" __attribute__((weak)) void AnnotateBenignRace( const char *f, int l, uptr mem, char *desc) {}
+extern "C" __attribute__((weak)) void AnnotateIgnoreReadsBegin(const char *f, int l) {}
+extern "C" __attribute__((weak)) void AnnotateIgnoreReadsEnd(const char *f, int l) {}
+extern "C" __attribute__((weak)) void AnnotateIgnoreWritesBegin(const char *f, int l) {}
+extern "C" __attribute__((weak)) void AnnotateIgnoreWritesEnd(const char *f, int l) {}
+extern "C" __attribute__((weak)) void AnnotateIgnoreSyncBegin(const char *f, int l) {}
+extern "C" __attribute__((weak)) void AnnotateIgnoreSyncEnd(const char *f, int l) {}
+extern "C" __attribute__((weak)) void AnnotatePublishMemoryRange( const char *f, int l, uptr addr, uptr size) {}
+extern "C" __attribute__((weak)) void AnnotateUnpublishMemoryRange( const char *f, int l, uptr addr, uptr size) {}
+extern "C" __attribute__((weak)) void AnnotateThreadName( const char *f, int l, char *name) {}
+extern "C" __attribute__((weak)) void WTFAnnotateHappensBefore(const char *f, int l, uptr addr) {}
+extern "C" __attribute__((weak)) void WTFAnnotateHappensAfter(const char *f, int l, uptr addr) {}
+extern "C" __attribute__((weak)) void WTFAnnotateBenignRaceSized( const char *f, int l, uptr mem, uptr sz, char *desc) {}
+extern "C" __attribute__((weak)) int RunningOnValgrind() {return 0;}
+extern "C" __attribute__((weak)) double ValgrindSlowdown(void) {return 0;}
+extern "C" __attribute__((weak)) const char __attribute__((weak))* ThreadSanitizerQuery(const char *query) {return 0;}
+extern "C" __attribute__((weak)) void AnnotateMemoryIsInitialized(const char *f, int l, uptr mem, uptr sz) {}

runtime/src/z_Linux_util.c

 #elif KMP_OS_FREEBSD
 # include <pthread_np.h>
 #endif
 
 #include <dirent.h>
 #include <ctype.h>
 #include <fcntl.h>
 
+#include "tsan_annotations.h"
+
 /* ------------------------------------------------------------------------ */
 /* ------------------------------------------------------------------------ */
 
 struct kmp_sys_timer {
     struct timespec     start;
 };
 
 // Convert timespec to nanoseconds.
__kmp_suspend_initialize( void )
     KMP_CHECK_SYSFAIL( "pthread_mutexattr_init", status );
     status = pthread_condattr_init( &__kmp_suspend_cond_attr );
     KMP_CHECK_SYSFAIL( "pthread_condattr_init", status );
 }
 
 static void
 __kmp_suspend_initialize_thread( kmp_info_t *th )
 {
+    ANNOTATE_HAPPENS_AFTER(&th->th.th_suspend_init_count);
     if ( th->th.th_suspend_init_count <= __kmp_fork_count ) {
         /* this means we haven't initialized the suspension pthread objects for this thread
            in this instance of the process */
         int     status;
         status = pthread_cond_init( &th->th.th_suspend_cv.c_cond, &__kmp_suspend_cond_attr );
         KMP_CHECK_SYSFAIL( "pthread_cond_init", status );
         status = pthread_mutex_init( &th->th.th_suspend_mx.m_mutex, & __kmp_suspend_mutex_attr );
         KMP_CHECK_SYSFAIL( "pthread_mutex_init", status );
         *(volatile int*)&th->th.th_suspend_init_count = __kmp_fork_count + 1;
+        ANNOTATE_HAPPENS_BEFORE(&th->th.th_suspend_init_count);
     };
 }
 
 void
 __kmp_suspend_uninitialize_thread( kmp_info_t *th )
 {
     if(th->th.th_suspend_init_count > __kmp_fork_count) {
         /* this means we have initialize the suspension pthread objects for this thread