[OMPT] Add implementation and tests of Archer tool

[protze.joachim]

Summary

The tool provides TSAN annotations for OpenMP synchronization. Enabled for
application execution by setting OMP_TOOL_LIBRARIES=libarcher.so, this
library should get installed in the same order as libomp, so the linker
should find the file when the application is linked with OpenMP.

The tool detects whether the application was built with TSan and rejects
activation according to the OMPT protocol if there is no TSan-rt.

Further it is recommended to set
TSAN_OPTIONS="ignore_noninstrumented_modules=1"
This avoids some false positive errors from the OpenMP runtime.

[Hahnfeld]

Do we really want to call this tool Archer? I understand that it was written for that purpose but IMO this can serve a much more general purpose, even without the Archer "framework" around it.

[protze.joachim]

Reasons to keep the name archer (or archer-rt):

• We already have that name, we don't need to come up with a new name :)
• You can easily find related publications under that name, which explain the reasoning behind the library
• The programmer should never see the name, once the tool is integrated into the runtime workflow

[simoatze]

This looks great, thanks guys!
I agree, the name is mostly a "marketing" thing at this point, people know it in that way, changing it would be confusing.

[protze.joachim]

This will not compile with the current version of the OMPT interface, needs updates first

[protze.joachim]

The code is updated to be compatible with OMPT as in OpenMP 5.0.

The OpenMP runtime code is modified to automatically load Archer, which will itself only make active if the application is compiled with TSan.

We are currently in the process to relicense the code under "Apache License v2.0 with LLVM Exceptions", but this should have no influence to the code review.

[protze.joachim]

Relicensed Archer under Apache license, updated the files.

[Hahnfeld]

Some high-level comments inline

runtime/src/ompt-general.cpp

233–235

These preprocessor guards are redundant.

240–245

If it's not tested, it should be removed.

tools/CMakeLists.txt

7

Why do we need to handle this dummy?

tools/archer/CMakeLists.txt

1–4

I don't think this should be here...

tools/archer/LICENSE.txt

1–3

there is a LICENSE.txt in the openmp folder, I don't think we should have a copy in this subdirectory.

tools/archer/counter.h

2

counter.h

tools/archer/ftsan.c

2

Why do we need this in here? This should surely be handled in the compiler?

[protze.joachim]

Addressing Jonas' comments

tools/archer/ftsan.c

2

This allows to use those basic annotations of synchronization in Fortran code.
The annotation interface is also intended for manual instrumenting application code.

Like for SPEC OMP: 371.applu331/src/syncs.F90

@@ -82,6 +85,8 @@
 !$omp flush(isync)
          end do
+       CALL AnnotateHappensAfter(__FILE__, __LINE__, isync(omp_get_thread_num()))
+       CALL AnnotateHappensBefore(__FILE__, __LINE__, isync(iam))
          isync(iam) = 1
 !$omp flush(isync)
       endif

[protze.joachim]

Removed a file which was mistakenly added

[Hahnfeld]

More comments

tools/CMakeLists.txt

3

s/project/tool/

tools/archer/CMakeLists.txt

5

I think this should now say See https://llvm.org/LICENSE.txt for license information., at least that's what other files have.

18–19

Please use OPENMP_INSTALL_LIBDIR to handle libdir suffix.

tools/archer/ftsan.c

2

It's not required for Archer and belongs into the sanitizer runtime.

tools/archer/ompt-tsan.cpp

204

This should be standard now.

545

Indention looks odd...

721–722

?

748

?

tools/archer/tests/CMakeLists.txt

6

Do they?

34–35

At least these two don't exist here.

tools/archer/tests/lit.cfg

106–109

Is this needed?

118–131

Is this needed?

tools/archer/tests/lit.site.cfg.in

14–16

This should not be needed

tools/archer/tests/races/critical-unrelated.c

38

This depends on the compiler used, right? Can the tests be executed by GCC?

[protze.joachim]

tools/archer/ftsan.c

2

It doesn't harm to have this duplicately. I'm not sure how this would fit into the LLVM/sanitizer as long as there is no Fortran frontend supporting this sanitizer.

I moved the code to the other declarations in ompt-tsan.cpp.

[protze.joachim]

Addressed comments by Jonas.
The tests also support gcc now.

[Hahnfeld]

tools/archer/ompt-tsan.cpp

176–193

Moving the code doesn't solve the underlying issues:

1. It's not required for Archer, or at least for the basic tests to pass.
2. It's not related to OpenMP per se.

Besides, with the current usage to dynamically load the library during runtime, it will not resolve symbols during linkage.

384–385

This comment doesn't sound right.

924–925

This functionality is not used in the tests and could be a separate patch.

[Hahnfeld]

tools/archer/tests/lit.site.cfg.in

14–15

Still not needed

30–52

Not needed

[protze.joachim]

Removed the Fortran annotation functions :/
Fixed also the other comments

tools/archer/ompt-tsan.cpp

924–925

This functionality is useful for debugging errors resulting from errors in the OMPT implementation.

[Hahnfeld]

tools/archer/ompt-tsan.cpp

924–925

I agree, and this sounds like a good motivation for a second tool. What's the advantage of having this in ompt-tsan.cpp?

[protze.joachim]

Removed the counters as requested

[Hahnfeld]

The code itself looks good to me (it should probably be clang-formated). If others have comments, please raise them now.

Apart from that, I still found many unused lines in the testing framework...

tools/archer/README.md

169–178

Do we still want to speak of clang-archer?

tools/archer/tests/CMakeLists.txt

19–20

These variables are not defined, so I guess they can be safely removed

30–31

Likewise

tools/archer/tests/lit.cfg

54–63

I think these lines can be removed, see comments in CMakeLists.txt and implications

93–97

This can probably be removed

tools/archer/tests/lit.site.cfg.in

14–15

Ping, there's no definition of OPENMP_TEST_ARCHER_FLAGS and no use of test_archer_flags

19–25

All of these are either not used or are probably not needed, see comment in CMakeLists.txt.

tools/archer/tests/ompt/ompt-signal.h

3

We should probably work on deduplicating this from the OMPT tests. Not a strict requirement for now, the file is probably small enough...

[twoh]

openmp/tools/archer/tests/lit.site.cfg.in

13 ↗

(On Diff #229820)

@protze.joachim Shouldn't this be @LIBARCHER_HAVE_LIBATOMIC@ , based on openmp/tools/archer/tests/CMakeLists.txt? For our local unit test runs, they fail with

llvm-lit: /home/engshare/third-party2/llvm/src/llvm/utils/lit/lit/TestingConfig.py:102: fatal: unable to parse config file '/home/engshare/third-party2/llvm/src/build-centos7-native/build/projects/openmp/tools/archer/tests/lit.site.cfg', traceback: Traceback (most recent call last):
  File "/home/engshare/third-party2/llvm/src/llvm/utils/lit/lit/TestingConfig.py", line 89, in load_from_path
    exec(compile(data, path, 'exec'), cfg_globals, None)
  File "/home/engshare/third-party2/llvm/src/build-centos7-native/build/projects/openmp/tools/archer/tests/lit.site.cfg", line 14
    config.has_libatomic =
                          ^
SyntaxError: invalid syntax

[gribozavr2]

The newly added tests fail on buildbots: http://lab.llvm.org:8011/builders/openmp-gcc-x86_64-linux-debian/builds/236

Could you take a look? And please consider reverting in the meanwhile.

[protze.joachim]

In D45890#1756465, @gribozavr2 wrote:

The newly added tests fail on buildbots: http://lab.llvm.org:8011/builders/openmp-gcc-x86_64-linux-debian/builds/236

Could you take a look? And please consider reverting in the meanwhile.

Several things:

• the tests rely on the availability of thread-sanitizer
• I disabled the tests for standalone builds (as in the specific build-bot)
• in a follow-up review, I'll enable the tests based on llvm-config information

• MOST important: we should make the sanitizers available in the build-bot

[gribozavr2]

In D45890#1756747, @protze.joachim wrote:

• MOST important: we should make the sanitizers available in the build-bot

May I ask you to make the necessary changes to the bot configuration? It is here: https://github.com/llvm/llvm-zorg/blob/master/zorg/buildbot/builders/OpenMPBuilder.py

[gribozavr2]

In D45890#1756747, @protze.joachim wrote:

• the tests rely on the availability of thread-sanitizer

The tests should be annotated as REQUIRES: tsan or whatever the appropriate tag is.

[protze.joachim]

In D45890#1756796, @gribozavr2 wrote:

In D45890#1756747, @protze.joachim wrote:

• the tests rely on the availability of thread-sanitizer

The tests should be annotated as REQUIRES: tsan or whatever the appropriate tag is.

I pushed this change, so that the tests are off currently.
I'll create a review to check and announce this capability. (@Hahnfeld I think, you have experience in testing the test-compiler for features?)

Thanks for the pointer to the buildbot configuration, I'll have a look!

[gribozavr2]

Thank you very much for the quick workaround!

[aaronpuchert]

openmp/tools/archer/CMakeLists.txt

13–18 ↗

(On Diff #229820)

The commit message talks about setting OMP_TOOL_LIBRARIES=libarcher.so, so why do we build and install this as static library?

If there are uses for the static library, could we decide based on LIBOMP_ENABLE_SHARED or BUILD_SHARED_LIBS whether to build it?

CMakeLists.txt

if (APPLE OR WIN32)
     message(FATAL_ERROR "libomptarget cannot be built on Windows and MacOS X!")
   elseif (NOT OPENMP_HAVE_STD_CPP11_FLAG)
     message(FATAL_ERROR "Host compiler must support C++11 to build libomptarget!")
   endif()
 
   add_subdirectory(libomptarget)
 endif()
 
+set(ENABLE_OMPT_TOOLS ON)
+# Currently tools are not tested well on Windows or MacOS X.
+if (APPLE OR WIN32)
+  set(ENABLE_OMPT_TOOLS OFF)
+endif()
+
+option(OPENMP_ENABLE_OMPT_TOOLS "Enable building ompt based tools for OpenMP."
+       ${ENABLE_OMPT_TOOLS})
+if (OPENMP_ENABLE_OMPT_TOOLS)
+  add_subdirectory(tools)
+endif()
+
 # Now that we have seen all testuites, create the check-openmp target.
 construct_check_openmp_target()

runtime/CMakeLists.txt

if(${OPENMP_STANDALONE_BUILD})
   libomp_say("Use Adaptive locks   -- ${LIBOMP_USE_ADAPTIVE_LOCKS}")
   libomp_say("Use quad precision   -- ${LIBOMP_USE_QUAD_PRECISION}")
   libomp_say("Use TSAN-support     -- ${LIBOMP_TSAN_SUPPORT}")
   libomp_say("Use Hwloc library    -- ${LIBOMP_USE_HWLOC}")
 endif()
 
 add_subdirectory(src)
 add_subdirectory(test)
+
+# make these variables available for tools:
+set(LIBOMP_LIBRARY_DIR ${LIBOMP_LIBRARY_DIR} PARENT_SCOPE)
+set(LIBOMP_INCLUDE_DIR ${LIBOMP_INCLUDE_DIR} PARENT_SCOPE)

runtime/src/CMakeLists.txt

else()
   get_target_property(LIBOMP_LIBRARY_DIR omp RUNTIME_OUTPUT_DIRECTORY)
 endif()
 if(NOT LIBOMP_LIBRARY_DIR)
   set(LIBOMP_LIBRARY_DIR ${CMAKE_CURRENT_BINARY_DIR})
   set(LIBOMP_LIBRARY_DIR ${CMAKE_CURRENT_BINARY_DIR} PARENT_SCOPE)
 else()
   set(LIBOMP_LIBRARY_DIR ${LIBOMP_LIBRARY_DIR} PARENT_SCOPE)
 endif()
+set(LIBOMP_INCLUDE_DIR ${CMAKE_CURRENT_BINARY_DIR} PARENT_SCOPE)
 
 # Add symbolic links to libomp
 if(NOT WIN32)
   add_custom_command(TARGET omp POST_BUILD
     COMMAND ${CMAKE_COMMAND} -E create_symlink ${LIBOMP_LIB_FILE}
       libgomp${LIBOMP_LIBRARY_SUFFIX}
     COMMAND ${CMAKE_COMMAND} -E create_symlink ${LIBOMP_LIB_FILE}
       libiomp5${LIBOMP_LIBRARY_SUFFIX}

runtime/src/ompt-general.cpp

ompt_try_start_tool(unsigned int omp_version, const char *runtime_version) {
   ompt_start_tool_t start_tool = NULL;
 #if KMP_OS_WINDOWS
   // Cannot use colon to describe a list of absolute paths on Windows
   const char *sep = ";";
 #else
   const char *sep = ":";
 #endif
 
+#if KMP_OS_UNIX
+  { // Non-standard: load archer tool if application is built with TSan
+#if KMP_OS_UNIX
+    const char *fname = "libarcher.so";
+    void *h = dlopen(fname, RTLD_LAZY);
+    if (h) {
+      start_tool = (ompt_start_tool_t)dlsym(h, "ompt_start_tool");
+#elif 0 && KMP_OS_WINDOWS // Not yet tested. How is the fname of libarcher on Windows?
+    char *fname = "archer.dll";
+    HMODULE h = LoadLibrary(fname);
+    if (h) {
+      start_tool = (ompt_start_tool_t)GetProcAddress(h, "ompt_start_tool");
+#endif
+      if (start_tool)
+        ret = (*start_tool)(omp_version, runtime_version);
+      if (ret)
+        return ret;
+    }
+  }
+#endif
+
 #if KMP_OS_DARWIN
   // Try in the current address space
   ret = ompt_tool_darwin(omp_version, runtime_version);
 #elif OMPT_HAVE_WEAK_ATTRIBUTE
   ret = ompt_start_tool(omp_version, runtime_version);
 #elif OMPT_HAVE_PSAPI
   ret = ompt_tool_windows(omp_version, runtime_version);
 #else

tools/CMakeLists.txt

+# Discover the tools that use CMake in the subdirectories.
+# Note that explicit cmake invocation is required every time a new project
+# is added or removed.
+file(GLOB entries *)
+foreach(entry ${entries})
+  if(IS_DIRECTORY ${entry} AND EXISTS ${entry}/CMakeLists.txt)
+    if((NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/dummy))
+      add_subdirectory(${entry})
+    endif()
+  endif()
+endforeach(entry)

tools/archer/CMakeLists.txt

+#
+# ompt-tsan.cpp -- Archer runtime library, TSan annotations for Archer
+#
+#
+# //===----------------------------------------------------------------------===//
+# //
+# // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# // See tools/archer/LICENSE.txt for details.
+# // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+# //
+# //===----------------------------------------------------------------------===//
+  
+  
+
+include_directories(${LIBOMP_INCLUDE_DIR})
+
+add_library(archer SHARED ompt-tsan.cpp counter.cpp ftsan.c)
+add_library(archer_static STATIC ompt-tsan.cpp counter.cpp ftsan.c)
+
+install(TARGETS archer archer_static
+  LIBRARY DESTINATION lib
+  ARCHIVE DESTINATION lib)
+
+add_subdirectory(tests)

tools/archer/LICENSE.txt

+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction,
+and distribution as defined by Sections 1 through 9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by
+the copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all
+other entities that control, are controlled by, or are under common
+control with that entity. For the purposes of this definition,
+"control" means (i) the power, direct or indirect, to cause the
+direction or management of such entity, whether by contract or
+otherwise, or (ii) ownership of fifty percent (50%) or more of the
+outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity
+exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications,
+including but not limited to software source code, documentation
+source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical
+transformation or translation of a Source form, including but
+not limited to compiled object code, generated documentation,
+and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or
+Object form, made available under the License, as indicated by a
+copyright notice that is included in or attached to the work
+(an example is provided in the Appendix below).
+
+"Derivative Works" shall mean any work, whether in Source or Object
+form, that is based on (or derived from) the Work and for which the
+editorial revisions, annotations, elaborations, or other modifications
+represent, as a whole, an original work of authorship. For the purposes
+of this License, Derivative Works shall not include works that remain
+separable from, or merely link (or bind by name) to the interfaces of,
+the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including
+the original version of the Work and any modifications or additions
+to that Work or Derivative Works thereof, that is intentionally
+submitted to Licensor for inclusion in the Work by the copyright owner
+or by an individual or Legal Entity authorized to submit on behalf of
+the copyright owner. For the purposes of this definition, "submitted"
+means any form of electronic, verbal, or written communication sent
+to the Licensor or its representatives, including but not limited to
+communication on electronic mailing lists, source code control systems,
+and issue tracking systems that are managed by, or on behalf of, the
+Licensor for the purpose of discussing and improving the Work, but
+excluding communication that is conspicuously marked or otherwise
+designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity
+on behalf of whom a Contribution has been received by Licensor and
+subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+this License, each Contributor hereby grants to You a perpetual,
+worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+copyright license to reproduce, prepare Derivative Works of,
+publicly display, publicly perform, sublicense, and distribute the
+Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+this License, each Contributor hereby grants to You a perpetual,
+worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+(except as stated in this section) patent license to make, have made,
+use, offer to sell, sell, import, and otherwise transfer the Work,
+where such license applies only to those patent claims licensable
+by such Contributor that are necessarily infringed by their
+Contribution(s) alone or by combination of their Contribution(s)
+with the Work to which such Contribution(s) was submitted. If You
+institute patent litigation against any entity (including a
+cross-claim or counterclaim in a lawsuit) alleging that the Work
+or a Contribution incorporated within the Work constitutes direct
+or contributory patent infringement, then any patent licenses
+granted to You under this License for that Work shall terminate
+as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+Work or Derivative Works thereof in any medium, with or without
+modifications, and in Source or Object form, provided that You
+meet the following conditions:
+
+(a) You must give any other recipients of the Work or
+Derivative Works a copy of this License; and
+
+(b) You must cause any modified files to carry prominent notices
+stating that You changed the files; and
+
+(c) You must retain, in the Source form of any Derivative Works
+that You distribute, all copyright, patent, trademark, and
+attribution notices from the Source form of the Work,
+excluding those notices that do not pertain to any part of
+the Derivative Works; and
+
+(d) If the Work includes a "NOTICE" text file as part of its
+distribution, then any Derivative Works that You distribute must
+include a readable copy of the attribution notices contained
+within such NOTICE file, excluding those notices that do not
+pertain to any part of the Derivative Works, in at least one
+of the following places: within a NOTICE text file distributed
+as part of the Derivative Works; within the Source form or
+documentation, if provided along with the Derivative Works; or,
+within a display generated by the Derivative Works, if and
+wherever such third-party notices normally appear. The contents
+of the NOTICE file are for informational purposes only and
+do not modify the License. You may add Your own attribution
+notices within Derivative Works that You distribute, alongside
+or as an addendum to the NOTICE text from the Work, provided
+that such additional attribution notices cannot be construed
+as modifying the License.
+
+You may add Your own copyright statement to Your modifications and
+may provide additional or different license terms and conditions
+for use, reproduction, or distribution of Your modifications, or
+for any such Derivative Works as a whole, provided Your use,
+reproduction, and distribution of the Work otherwise complies with
+the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+any Contribution intentionally submitted for inclusion in the Work
+by You to the Licensor shall be under the terms and conditions of
+this License, without any additional terms or conditions.
+Notwithstanding the above, nothing herein shall supersede or modify
+the terms of any separate license agreement you may have executed
+with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+names, trademarks, service marks, or product names of the Licensor,
+except as required for reasonable and customary use in describing the
+origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+agreed to in writing, Licensor provides the Work (and each
+Contributor provides its Contributions) on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+implied, including, without limitation, any warranties or conditions
+of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+PARTICULAR PURPOSE. You are solely responsible for determining the
+appropriateness of using or redistributing the Work and assume any
+risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+whether in tort (including negligence), contract, or otherwise,
+unless required by applicable law (such as deliberate and grossly
+negligent acts) or agreed to in writing, shall any Contributor be
+liable to You for damages, including any direct, indirect, special,
+incidental, or consequential damages of any character arising as a
+result of this License or out of the use or inability to use the
+Work (including but not limited to damages for loss of goodwill,
+work stoppage, computer failure or malfunction, or any and all
+other commercial damages or losses), even if such Contributor
+has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+the Work or Derivative Works thereof, You may choose to offer,
+and charge a fee for, acceptance of support, warranty, indemnity,
+or other liability obligations and/or rights consistent with this
+License. However, in accepting such obligations, You may act only
+on Your own behalf and on Your sole responsibility, not on behalf
+of any other Contributor, and only if You agree to indemnify,
+defend, and hold each Contributor harmless for any liability
+incurred by, or claims asserted against, such Contributor by reason
+of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+To apply the Apache License to your work, attach the following
+boilerplate notice, with the fields enclosed by brackets "[]"
+replaced with your own identifying information. (Don't include
+the brackets!)  The text should be enclosed in the appropriate
+comment syntax for the file format. We also recommend that a
+file or class name and description of purpose be included on the
+same "printed page" as the copyright notice for easier
+identification within third-party archives.
+
+Copyright 2019, Lawrence Livermore National Security, LLC
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

tools/archer/README.md

+<div id="table-of-contents">
+<h2>Table of Contents</h2>
+<div id="text-table-of-contents">
+<ul>
+<li><a href="#org8ca70b5">1. License</a></li>
+<li><a href="#orgc6a2b10">2. Introduction</a></li>
+<li><a href="#org9a459f1">3. Installation</a></li>
+<li><a href="#orgb820ad0">4. Usage</a>
+<ul>
+<li><a href="#org213ff1a">4.1. How to compile</a></li>
+<li><a href="#org110062c">4.2. Runtime Flags</a></li>
+</ul>
+</li>
+<li><a href="#org73e58a9">5. Example</a></li>
+<li><a href="#orgcc38a36">6. Contacts and Support</a></li>
+</ul>
+</div>
+</div>
+
+
+<a id="org8ca70b5"></a>
+
+# License
+
+Archer is distributed under the terms of the Apache License.
+
+Please see LICENSE.txt for usage terms.
+
+LLNL-CODE-773957
+
+<a id="orgc6a2b10"></a>
+
+# Introduction
+
+**Archer** is an OMPT tool which annotates OpenMP synchronization semantics for data race
+detection.
+This avoids false alerts in data race detection.
+Archer is automatically loaded for OpenMP applications which are compiled
+with ThreadSanitizer option.
+
+<a id="org9a459f1"></a>
+
+# Build Archer within Clang/LLVM
+
+This distribution of Archer is automatically built with the OpenMP runtime
+and automatically loaded by the OpenMP runtime.
+
+<a id="orgb820ad0"></a>
+
+# Usage
+
+
+<a id="org213ff1a"></a>
+
+## How to compile
+
+To use archer, compile the application with the extra flag
+`-fsanitize=thread`:
+
+    clang -O3 -g -fopenmp -fsanitize=thread app.c
+    clang++ -O3 -g -fopenmp -fsanitize=thread app.cpp
+
+To compile Fortran applications, compile with gfortran, link with clang:
+
+    gfortran -g -c -fopenmp -fsanitize=thread app.f
+    clang -fopenmp -fsanitize=thread app.o -lgfortran
+
+
+<a id="org110062c"></a>
+
+## Runtime Flags
+
+TSan runtime flags are passed via **TSAN&#95;OPTIONS** environment variable,
+we highly recommend the following option to aviod false alerts for the
+OpenMP or MPI runtime implementation:
+
+    export TSAN_OPTIONS="ignore_noninstrumented_modules=1"
+
+
+Runtime flags are passed via **ARCHER&#95;OPTIONS** environment variable,
+different flags are separated by spaces, e.g.:
+
+    ARCHER_OPTIONS="flush_shadow=1" ./myprogram
+
+<table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
+
+
+<colgroup>
+<col  class="org-left" />
+
+<col  class="org-right" />
+
+<col  class="org-left" />
+</colgroup>
+<thead>
+<tr>
+<th scope="col" class="org-left">Flag Name</th>
+<th scope="col" class="org-right">Default value</th>
+<th scope="col" class="org-left">Description</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td class="org-left">flush&#95;shadow</td>
+<td class="org-right">0</td>
+<td class="org-left">Flush shadow memory at the end of an outer OpenMP parallel region. Our experiments show that this can reduce memory overhead by ~30% and runtime overhead by ~10%. This flag is useful for large OpenMP applications that typically require large amounts of memory, causing out-of-memory exceptions when checked by Archer.</td>
+</tr>
+</tbody>
+
+<tbody>
+<tr>
+<td class="org-left">print&#95;ompt&#95;counters</td>
+<td class="org-right">0</td>
+<td class="org-left">Print the number of triggered OMPT events at the end of the execution.</td>
+</tr>
+</tbody>
+
+<tbody>
+<tr>
+<td class="org-left">print&#95;max&#95;rss</td>
+<td class="org-right">0</td>
+<td class="org-left">Print the RSS memory peak at the end of the execution.</td>
+</tr>
+</tbody>
+
+<tbody>
+<tr>
+<td class="org-left">verbose</td>
+<td class="org-right">0</td>
+<td class="org-left">Print startup information.</td>
+</tr>
+</tbody>
+
+<tbody>
+<tr>
+<td class="org-left">enable</td>
+<td class="org-right">1</td>
+<td class="org-left">Use Archer runtime library during execution.</td>
+</tr>
+</tbody>
+</table>
+
+
+<a id="org73e58a9"></a>
+
+# Example
+
+Let us take the program below and follow the steps to compile and
+check the program for data races.
+
+Suppose our program is called *myprogram.c*:
+
+     1  #include <stdio.h>
+     2
+     3  #define N 1000
+     4
+     5  int main (int argc, char **argv)
+     6  {
+     7    int a[N];
+     8
+     9  #pragma omp parallel for
+    10    for (int i = 0; i < N - 1; i++) {
+    11      a[i] = a[i + 1];
+    12    }
+    13  }
+
+In case we installed Archer with the official LLVM OpenMP runtime and
+ThreadSanitizer support, we compile the program as follow:
+
+    clang-archer myprogram.c -o myprogram
+
+otherwise, if we installed Archer with the LLVM OpenMP runtime and
+ThreadSanitizer OMPT support our compile command will look like:
+
+    clang-archer myprogram.c -o myprogram -larcher
+
+Now we can run the program with the following commands:
+
+    export OMP_NUM_THREADS=2
+    ./myprogram
+
+Archer will output a report in case it finds data races. In our case
+the report will look as follow:
+
+    ==================
+    WARNING: ThreadSanitizer: data race (pid=13641)
+      Read of size 4 at 0x7fff79a01170 by main thread:
+        #0 .omp_outlined. myprogram.c:11:12 (myprogram+0x00000049b5a2)
+        #1 __kmp_invoke_microtask <null> (libomp.so+0x000000077842)
+        #2 __libc_start_main /build/glibc-t3gR2i/glibc-2.23/csu/../csu/libc-start.c:291 (libc.so.6+0x00000002082f)
+
+      Previous write of size 4 at 0x7fff79a01170 by thread T1:
+        #0 .omp_outlined. myprogram.c:11:10 (myprogram+0x00000049b5d6)
+        #1 __kmp_invoke_microtask <null> (libomp.so+0x000000077842)
+
+      Location is stack of main thread.
+
+      Thread T1 (tid=13643, running) created by main thread at:
+        #0 pthread_create tsan_interceptors.cc:902:3 (myprogram+0x00000043db75)
+        #1 __kmp_create_worker <null> (libomp.so+0x00000006c364)
+        #2 __libc_start_main /build/glibc-t3gR2i/glibc-2.23/csu/../csu/libc-start.c:291 (libc.so.6+0x00000002082f)
+
+    SUMMARY: ThreadSanitizer: data race myprogram.c:11:12 in .omp_outlined.
+    ==================
+    ThreadSanitizer: reported 1 warnings
+
+
+<a id="orgcc38a36"></a>
+
+# Contacts and Support
+
+-   [Google group](https://groups.google.com/forum/#!forum/archer-pruner)
+-   [Slack Channel](https://pruners.slack.com)
+
+    <ul style="list-style-type:circle"> <li> For an invitation please write an email to <a href="mailto:[e-mail]?Subject=[archer-slack] Slack Invitation" target="_top">Simone Atzeni</a> with a reason why you want to be part of the PRUNERS Slack Team. </li> </ul>
+-   E-Mail Contacts:
+
+    <ul style="list-style-type:circle"> <li> <a href="mailto:[e-mail]?Subject=[archer-dev]%20" target="_top">Simone Atzeni</a> </li> <li> <a href="mailto:[e-mail]?Subject=[archer-dev]%20" target="_top">Joachim Protze</a> </li> </ul>
+
+

tools/archer/counter.h

+/*
+ * counter.cpp -- Archer runtime library, counter header file
+ */
+  
+  //===----------------------------------------------------------------------===//
+  //
+  // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+  // See tools/archer/LICENSE.txt for details.
+  // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+  //
+  //===----------------------------------------------------------------------===//
+  
+  
+
+#include <stdio.h>
+#include <inttypes.h>
+#include <omp-tools.h>
+
+// 240 should be enough for xeon-phi
+#define MAX_THREADS 240
+
+#define CACHE_LINE 128
+
+#define COUNT_EVENT1(name)                                                     \
+  if (this_event_counter)                                                      \
+  this_event_counter->name++
+#define COUNT_EVENT2(name, scope)                                              \
+  if (this_event_counter)                                                      \
+  this_event_counter->name##_##scope++
+#define COUNT_EVENT3(name, scope, kind)                                        \
+  if (this_event_counter)                                                      \
+  this_event_counter->name##_##scope##_##kind++
+
+typedef struct alignas(128) {
+  int thread_begin; // (1)  thread_begin
+  int thread_end; // (2)  thread_end
+  int parallel_begin; // (3)  parallel_begin
+  int parallel_end; // (4)  parallel_end
+  int task_create_initial; // (5) task_create:  task_initial
+  int task_create_explicit; //      task_explicit
+  int task_create_target; //      task_target
+  int task_create_included; //      task_included
+  int task_create_untied; //      task_untied
+  int task_schedule; // (6) task_schedule
+  int implicit_task_scope_begin; // (7) implicit task:  scope_begin
+  int implicit_task_scope_end; //     scope_end
+  int mutex_released_lock; // (15) mutex_released:  mutex_lock
+  int mutex_released_nest_lock; //      mutex_nest_lock
+  int mutex_released_critical; //     mutex_critical
+  int mutex_released_atomic; //     mutex_atomic
+  int mutex_released_ordered; //      mutex_ordered
+  int mutex_released_default; //      default
+  int task_dependences; // (16) task_dependences
+  int task_dependence; // (17) task_dependence
+  int sync_region_scope_begin_barrier; // (21) sync_region: scope_begin:
+                                       //                sync_region_barrier
+  int sync_region_scope_begin_reduction; //              sync_region_reduction
+  int sync_region_scope_begin_taskwait; //               sync_region_taskwait
+  int sync_region_scope_begin_taskgroup; //              sync_region_taskgroup
+  int sync_region_scope_end_barrier; //                   scope_end:
+                                     //                  sync_region_barrier
+  int sync_region_scope_end_reduction; //                sync_region_reduction
+  int sync_region_scope_end_taskwait; //                 sync_region_taskwait
+  int sync_region_scope_end_taskgroup; //                sync_region_taskgroup
+  int lock_init_lock; // (22) lock_init:  mutex_lock
+  int lock_init_nest_lock; //       mutex_nest_lock
+  int lock_init_default; //     default
+  int lock_destroy_lock; // (23) lock_destroy mutex_lock
+  int lock_destroy_nest_lock; //      mutex_nest_lock
+  int lock_destroy_default; //      default
+  int mutex_acquire_lock; // (24) mutex_acquire:  mutex_lock
+  int mutex_acquire_nest_lock; //       mutex_nest_lock
+  int mutex_acquire_critical; //      mutex_critical
+  int mutex_acquire_atomic; //      mutex_atomic
+  int mutex_acquire_ordered; //     mutex_ordered
+  int mutex_acquire_default; //     default
+  int mutex_acquired_lock; // (25) mutex_acquired: mutex_lock
+  int mutex_acquired_nest_lock; //                      mutex_nest_lock
+  int mutex_acquired_critical; //                      mutex_critical
+  int mutex_acquired_atomic; //                      mutex_atomic
+  int mutex_acquired_ordered; //                      mutex_ordered
+  int mutex_acquired_default; //      default
+  int nest_lock_scope_begin; // (26) nest_lock: scope_begin
+  int nest_lock_scope_end; //     scope_end
+  int flush; // (27) flush
+} callback_counter_t;
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+void print_callbacks(callback_counter_t *counter);
+
+#ifdef __cplusplus
+}
+#endif

tools/archer/counter.cpp

+/*
+ * counter.cpp -- Archer runtime library, counting callbacks
+ */
+  
+  //===----------------------------------------------------------------------===//
+  //
+  // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+  // See tools/archer/LICENSE.txt for details.
+  // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+  //
+  //===----------------------------------------------------------------------===//
+  
+  
+
+#include "counter.h"
+
+#define OUTPUT_IF_NOT_NULL(format, value)                                      \
+  if (value)                                                                   \
+  printf(format, value)
+
+void print_callbacks(callback_counter_t *counter) {
+  int *basecounter = (int *)counter;
+  int total_callbacks = 0;
+  for (int i = 1; i < MAX_THREADS; i++) {
+    // if we have at least i threads, this thread should have a thread_begin
+    // event:
+    if (counter[i].thread_begin == 0)
+      break;
+    int *threadcounter = (int *)(counter + i);
+    for (int j = 0; j < (int)(sizeof(callback_counter_t) / sizeof(int)); j++)
+      basecounter[j] += threadcounter[j];
+  }
+
+  for (int j = 0; j < (int)(sizeof(callback_counter_t) / sizeof(int)); j++)
+    total_callbacks += basecounter[j];
+
+  printf("Total callbacks: %d\n", total_callbacks);
+  printf("--------------------------------------\n");
+  OUTPUT_IF_NOT_NULL("%5d thread_begin\n", counter[0].thread_begin);
+  OUTPUT_IF_NOT_NULL("%5d thread_end\n", counter[0].thread_end);
+  OUTPUT_IF_NOT_NULL("%5d parallel_begin\n", counter[0].parallel_begin);
+  OUTPUT_IF_NOT_NULL("%5d parallel_end\n", counter[0].parallel_end);
+  OUTPUT_IF_NOT_NULL("%5d task_create : initial\n",
+                     counter[0].task_create_initial);
+  OUTPUT_IF_NOT_NULL("%5d task_create : explicit\n",
+                     counter[0].task_create_explicit);
+  OUTPUT_IF_NOT_NULL("%5d task_create : target\n",
+                     counter[0].task_create_target);
+  OUTPUT_IF_NOT_NULL("%5d task_create : included\n",
+                     counter[0].task_create_included);
+  OUTPUT_IF_NOT_NULL("%5d task_create : untied\n",
+                     counter[0].task_create_untied);
+  OUTPUT_IF_NOT_NULL("%5d task_schedule\n", counter[0].task_schedule);
+  OUTPUT_IF_NOT_NULL("%5d implicit_task : scope_begin\n",
+                     counter[0].implicit_task_scope_begin);
+  OUTPUT_IF_NOT_NULL("%5d implicit_task : scope_end\n",
+                     counter[0].implicit_task_scope_end);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_lock\n",
+                     counter[0].mutex_released_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_nest_lock\n",
+                     counter[0].mutex_released_nest_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_critical\n",
+                     counter[0].mutex_released_critical);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_atomic\n",
+                     counter[0].mutex_released_atomic);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_ordered\n",
+                     counter[0].mutex_released_ordered);
+  OUTPUT_IF_NOT_NULL("%5d mutex_released_default\n",
+                     counter[0].mutex_released_default);
+  OUTPUT_IF_NOT_NULL("%5d task_dependences\n", counter[0].task_dependences);
+  OUTPUT_IF_NOT_NULL("%5d task_dependence\n", counter[0].task_dependence);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_begin : barrier\n",
+                     counter[0].sync_region_scope_begin_barrier);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_begin : taskwait\n",
+                     counter[0].sync_region_scope_begin_taskwait);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_begin : taskgroup\n",
+                     counter[0].sync_region_scope_begin_taskgroup);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_end : barrier\n",
+                     counter[0].sync_region_scope_end_barrier);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_end : taskwait\n",
+                     counter[0].sync_region_scope_end_taskwait);
+  OUTPUT_IF_NOT_NULL("%5d sync_region : scope_end : taskgroup\n",
+                     counter[0].sync_region_scope_end_taskgroup);
+  OUTPUT_IF_NOT_NULL("%5d lock_init_lock\n", counter[0].lock_init_lock);
+  OUTPUT_IF_NOT_NULL("%5d lock_init_nest_lock\n",
+                     counter[0].lock_init_nest_lock);
+  OUTPUT_IF_NOT_NULL("%5d lock_init_default\n", counter[0].lock_init_default);
+  OUTPUT_IF_NOT_NULL("%5d lock_destroy_lock\n", counter[0].lock_destroy_lock);
+  OUTPUT_IF_NOT_NULL("%5d lock_destroy_nest_lock\n",
+                     counter[0].lock_destroy_nest_lock);
+  OUTPUT_IF_NOT_NULL("%5d lock_destroy_default\n",
+                     counter[0].lock_destroy_default);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_lock\n", counter[0].mutex_acquire_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_nest_lock\n",
+                     counter[0].mutex_acquire_nest_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_critical\n",
+                     counter[0].mutex_acquire_critical);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_atomic\n",
+                     counter[0].mutex_acquire_atomic);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_ordered\n",
+                     counter[0].mutex_acquire_ordered);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquire_default\n",
+                     counter[0].mutex_acquire_default);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_lock\n",
+                     counter[0].mutex_acquired_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_nest_lock\n",
+                     counter[0].mutex_acquired_nest_lock);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_critical\n",
+                     counter[0].mutex_acquired_critical);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_atomic\n",
+                     counter[0].mutex_acquired_atomic);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_ordered\n",
+                     counter[0].mutex_acquired_ordered);
+  OUTPUT_IF_NOT_NULL("%5d mutex_acquired_default\n",
+                     counter[0].mutex_acquired_default);
+  OUTPUT_IF_NOT_NULL("%5d nest_lock_scope_begin\n",
+                     counter[0].nest_lock_scope_begin);
+  OUTPUT_IF_NOT_NULL("%5d nest_lock_scope_end\n",
+                     counter[0].nest_lock_scope_end);
+  OUTPUT_IF_NOT_NULL("%5d flush\n", counter[0].flush);
+
+  return;
+}

tools/archer/ftsan.c

+/*
+ * ftsan.c -- Archer runtime library, TSan annotations for Fortran
+ */
+  
+  //===----------------------------------------------------------------------===//
+  //
+  // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+  // See tools/archer/LICENSE.txt for details.
+  // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+  //
+  //===----------------------------------------------------------------------===//
+  
+  
+
+void __attribute__((weak))
+AnnotateHappensAfter(const char *file, int line, const volatile void *cv) {}
+void __attribute__((weak))
+AnnotateHappensBefore(const char *file, int line, const volatile void *cv) {}
+void __attribute__((weak))
+AnnotateIgnoreWritesBegin(const char *file, int line) {}
+void __attribute__((weak)) AnnotateIgnoreWritesEnd(const char *file, int line) {
+}
+
+// This is a Fortran wrapper for TSan Annotation functions.
+
+void annotatehappensafter_(const char *file, int *line,
+                           const volatile void *cv) {
+  AnnotateHappensAfter(file, *line, cv);
+}
+void annotatehappensbefore_(const char *file, int *line,
+                            const volatile void *cv) {
+  AnnotateHappensBefore(file, *line, cv);
+}
+void annotateignorewritesbegin_(const char *file, int line) {
+  AnnotateIgnoreWritesBegin(file, line);
+}
+void annotateignorewritesend_(const char *file, int line) {
+  AnnotateIgnoreWritesEnd(file, line);
+}

tools/archer/ompt-tsan.cpp

+/*
+ * ompt-tsan.cpp -- Archer runtime library, TSan annotations for Archer
+ */
+  
+  //===----------------------------------------------------------------------===//
+  //
+  // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+  // See tools/archer/LICENSE.txt for details.
+  // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+  //
+  //===----------------------------------------------------------------------===//
+
+
+#include "counter.h"
+
+#ifndef __STDC_FORMAT_MACROS
+#define __STDC_FORMAT_MACROS
+#endif
+
+#include <atomic>
+#include <cassert>
+#include <cstdlib>
+#include <cstring>
+#include <inttypes.h>
+#include <iostream>
+#include <mutex>
+#include <sstream>
+#include <stack>
+#include <list>
+#include <string>
+#include <iostream>
+#include <unordered_map>
+#include <vector>
+
+#if (defined __APPLE__ && defined __MACH__)
+#include <dlfcn.h>
+#endif
+
+#include <sys/resource.h>
+#include "omp-tools.h"
+
+callback_counter_t *all_counter;
+__thread callback_counter_t *this_event_counter;
+static int runOnTsan;
+static int hasReductionCallback;
+
+class ArcherFlags {
+public:
+#if (LLVM_VERSION) >= 40
+  int flush_shadow;
+#endif
+  int print_ompt_counters;
+  int print_max_rss;
+  int verbose;
+  int enabled;
+
+  ArcherFlags(const char *env)
+      :
+#if (LLVM_VERSION) >= 40
+        flush_shadow(0),
+#endif
+        print_ompt_counters(0), print_max_rss(0), verbose(0), enabled(1) {
+    if (env) {
+      std::vector<std::string> tokens;
+      std::string token;
+      std::string str(env);
+      std::istringstream iss(str);
+      while (std::getline(iss, token, ' '))
+        tokens.push_back(token);
+
+      for (std::vector<std::string>::iterator it = tokens.begin();
+           it != tokens.end(); ++it) {
+#if (LLVM_VERSION) >= 40
+        if (sscanf(it->c_str(), "flush_shadow=%d", &flush_shadow))
+          continue;
+#endif
+        if (sscanf(it->c_str(), "print_ompt_counters=%d", &print_ompt_counters))
+          continue;
+        if (sscanf(it->c_str(), "print_max_rss=%d", &print_max_rss))
+          continue;
+        if (sscanf(it->c_str(), "verbose=%d", &verbose))
+          continue;
+        if (sscanf(it->c_str(), "enable=%d", &enabled))
+          continue;
+        std::cerr << "Illegal values for ARCHER_OPTIONS variable: " << token
+                  << std::endl;
+      }
+    }
+  }
+};
+
+#if (LLVM_VERSION) >= 40
+extern "C" {
+int __attribute__((weak)) __archer_get_omp_status();
+void __attribute__((weak)) __tsan_flush_memory() {}
+}
+#endif
+ArcherFlags *archer_flags;
+
+// The following definitions are pasted from "llvm/Support/Compiler.h" to allow
+// the code
+// to be compiled with other compilers like gcc:
+
+#ifndef TsanHappensBefore
+// Thread Sanitizer is a tool that finds races in code.
+// See http://code.google.com/p/data-race-test/wiki/DynamicAnnotations .
+// tsan detects these exact functions by name.
+extern "C" {
+#if (defined __APPLE__ && defined __MACH__)
+static void AnnotateHappensAfter(const char *file, int line,
+                                 const volatile void *cv) {
+  void (*fptr)(const char *, int, const volatile void *);
+
+  fptr = (void (*)(const char *, int, const volatile void *))dlsym(
+      RTLD_DEFAULT, "AnnotateHappensAfter");
+  (*fptr)(file, line, cv);
+}
+static void AnnotateHappensBefore(const char *file, int line,
+                                  const volatile void *cv) {
+  void (*fptr)(const char *, int, const volatile void *);
+
+  fptr = (void (*)(const char *, int, const volatile void *))dlsym(
+      RTLD_DEFAULT, "AnnotateHappensBefore");
+  (*fptr)(file, line, cv);
+}
+static void AnnotateIgnoreWritesBegin(const char *file, int line) {
+  void (*fptr)(const char *, int);
+
+  fptr = (void (*)(const char *, int))dlsym(RTLD_DEFAULT,
+                                            "AnnotateIgnoreWritesBegin");
+  (*fptr)(file, line);
+}
+static void AnnotateIgnoreWritesEnd(const char *file, int line) {
+  void (*fptr)(const char *, int);
+
+  fptr = (void (*)(const char *, int))dlsym(RTLD_DEFAULT,
+                                            "AnnotateIgnoreWritesEnd");
+  (*fptr)(file, line);
+}
+static void AnnotateNewMemory(const char *file, int line,
+                              const volatile void *cv, size_t size) {
+  void (*fptr)(const char *, int, const volatile void *, size_t);
+
+  fptr = (void (*)(const char *, int, const volatile void *, size_t))dlsym(
+      RTLD_DEFAULT, "AnnotateNewMemory");
+  (*fptr)(file, line, cv, size);
+}
+static int RunningOnValgrind() {
+  int (*fptr)();
+
+  fptr = (int (*)())dlsym(RTLD_DEFAULT, "RunningOnValgrind");
+  if (fptr && fptr != RunningOnValgrind)
+    runOnTsan = 0;
+  return 0;
+}
+#else
+void __attribute__((weak))
+AnnotateHappensAfter(const char *file, int line, const volatile void *cv) {}
+void __attribute__((weak))
+AnnotateHappensBefore(const char *file, int line, const volatile void *cv) {}
+void __attribute__((weak))
+AnnotateIgnoreWritesBegin(const char *file, int line) {}
+void __attribute__((weak)) AnnotateIgnoreWritesEnd(const char *file, int line) {
+}
+void __attribute__((weak))
+AnnotateNewMemory(const char *file, int line, const volatile void *cv,
+                  size_t size) {}
+int __attribute__((weak)) RunningOnValgrind() {
+  runOnTsan = 0;
+  return 0;
+}
+#endif
+}
+
+// This marker is used to define a happens-before arc. The race detector will
+// infer an arc from the begin to the end when they share the same pointer
+// argument.
+#define TsanHappensBefore(cv) AnnotateHappensBefore(__FILE__, __LINE__, cv)
+
+// This marker defines the destination of a happens-before arc.
+#define TsanHappensAfter(cv) AnnotateHappensAfter(__FILE__, __LINE__, cv)
+
+// Ignore any races on writes between here and the next TsanIgnoreWritesEnd.
+#define TsanIgnoreWritesBegin() AnnotateIgnoreWritesBegin(__FILE__, __LINE__)
+
+// Resume checking for racy writes.
+#define TsanIgnoreWritesEnd() AnnotateIgnoreWritesEnd(__FILE__, __LINE__)
+
+// We don't really delete the clock for now
+#define TsanDeleteClock(cv)
+
+// newMemory
+#define TsanNewMemory(addr, size)                                              \
+  AnnotateNewMemory(__FILE__, __LINE__, addr, size)
+#define TsanFreeMemory(addr, size)                                             \
+  AnnotateNewMemory(__FILE__, __LINE__, addr, size)
+#endif
+
+/// Required OMPT inquiry functions.
+static ompt_get_parallel_info_t ompt_get_parallel_info;
+static ompt_get_thread_data_t ompt_get_thread_data;
+
+typedef int (*ompt_get_task_memory_t)(void **addr, size_t *size, int blocknum);
+
+typedef uint64_t ompt_tsan_clockid;
+
+static uint64_t my_next_id() {
+  static uint64_t ID = 0;
+  uint64_t ret = __sync_fetch_and_add(&ID, 1);
+  return ret;
+}
+
+// Data structure to provide a threadsafe pool of reusable objects.
+// DataPool<Type of objects, Size of blockalloc>
+template <typename T, int N> struct DataPool {
+  std::mutex DPMutex;
+  std::stack<T *> DataPointer;
+  std::list<void *> memory;
+  int total;
+
+  void newDatas() {
+    // prefix the Data with a pointer to 'this', allows to return memory to
+    // 'this',
+    // without explicitly knowing the source.
+    //
+    // To reduce lock contention, we use thread local DataPools, but Data
+    // objects move to other threads.
+    // The strategy is to get objects from local pool. Only if the object moved
+    // to another
+    // thread, we might see a penalty on release (returnData).
+    // For "single producer" pattern, a single thread creates tasks, these are
+    // executed by other threads.
+    // The master will have a high demand on TaskData, so return after use.
+    struct pooldata {
+      DataPool<T, N> *dp;
+      T data;
+    };
+    // We alloc without initialize the memory. We cannot call constructors.
+    // Therfore use malloc!
+    pooldata *datas = (pooldata *)malloc(sizeof(pooldata) * N);
+    memory.push_back(datas);
+    for (int i = 0; i < N; i++) {
+      datas[i].dp = this;
+      DataPointer.push(&(datas[i].data));
+    }
+    total += N;
+  }
+
+  T *getData() {
+    T *ret;
+    DPMutex.lock();
+    if (DataPointer.empty())
+      newDatas();
+    ret = DataPointer.top();
+    DataPointer.pop();
+    DPMutex.unlock();
+    return ret;
+  }
+
+  void returnData(T *data) {
+    DPMutex.lock();
+    DataPointer.push(data);
+    DPMutex.unlock();
+  }
+
+  void getDatas(int n, T **datas) {
+    DPMutex.lock();
+    for (int i = 0; i < n; i++) {
+      if (DataPointer.empty())
+        newDatas();
+      datas[i] = DataPointer.top();
+      DataPointer.pop();
+    }
+    DPMutex.unlock();
+  }
+
+  void returnDatas(int n, T **datas) {
+    DPMutex.lock();
+    for (int i = 0; i < n; i++) {
+      DataPointer.push(datas[i]);
+    }
+    DPMutex.unlock();
+  }
+
+  DataPool() : DPMutex(), DataPointer(), total(0) {}
+
+  ~DataPool() {
+    // we assume all memory is returned when the thread finished / destructor is
+    // called
+    for (auto i : memory)
+      if (i)
+        free(i);
+  }
+};
+
+// This function takes care to return the data to the originating DataPool
+// A pointer to the originating DataPool is stored just before the actual data.
+template <typename T, int N> static void retData(void *data) {
+  ((DataPool<T, N> **)data)[-1]->returnData((T *)data);
+}
+
+struct ParallelData;
+__thread DataPool<ParallelData, 4> *pdp;
+
+/// Data structure to store additional information for parallel regions.
+struct ParallelData {
+
+  // Parallel fork is just another barrier, use Barrier[1]
+
+  /// Two addresses for relationships with barriers.
+  ompt_tsan_clockid Barrier[2];
+
+  void *GetParallelPtr() { return &(Barrier[1]); }
+
+  void *GetBarrierPtr(unsigned Index) { return &(Barrier[Index]); }
+
+  ~ParallelData() {
+    TsanDeleteClock(&(Barrier[0]));
+    TsanDeleteClock(&(Barrier[1]));
+  }
+  // overload new/delete to use DataPool for memory management.
+  void *operator new(size_t size) { return pdp->getData(); }
+  void operator delete(void *p, size_t) { retData<ParallelData, 4>(p); }
+};
+
+static inline ParallelData *ToParallelData(ompt_data_t *parallel_data) {
+  return reinterpret_cast<ParallelData *>(parallel_data->ptr);
+}
+
+struct Taskgroup;
+__thread DataPool<Taskgroup, 4> *tgp;
+
+/// Data structure to support stacking of taskgroups and allow synchronization.
+struct Taskgroup {
+  /// Its address is used for relationships of the taskgroup's task set.
+  ompt_tsan_clockid Ptr;
+
+  /// Reference to the parent taskgroup.
+  Taskgroup *Parent;
+
+  Taskgroup(Taskgroup *Parent) : Parent(Parent) {}
+  ~Taskgroup() { TsanDeleteClock(&Ptr); }
+
+  void *GetPtr() { return &Ptr; }
+  // overload new/delete to use DataPool for memory management.
+  void *operator new(size_t size) { return tgp->getData(); }
+  void operator delete(void *p, size_t) { retData<Taskgroup, 4>(p); }
+};
+
+struct TaskData;
+__thread DataPool<TaskData, 4> *tdp;
+
+/// Data structure to store additional information for tasks.
+struct TaskData {
+  /// Its address is used for relationships of this task.
+  ompt_tsan_clockid Task;
+
+  /// Child tasks use its address to declare a relationship to a taskwait in
+  /// this task.
+  ompt_tsan_clockid Taskwait;
+
+  /// Whether this task is currently executing a barrier.
+  bool InBarrier;
+
+  /// Whether this task is currently executing a barrier.
+  bool Included;
+
+  /// Index of which barrier to use next.
+  char BarrierIndex;
+
+  /// Count how often this structure has been put into child tasks + 1.
+  std::atomic_int RefCount;
+
+  /// Reference to the parent that created this task.
+  TaskData *Parent;
+
+  /// Reference to the implicit task in the stack above this task.
+  TaskData *ImplicitTask;
+
+  /// Reference to the team of this task.
+  ParallelData *Team;
+
+  /// Reference to the current taskgroup that this task either belongs to or
+  /// that it just created.
+  Taskgroup *TaskGroup;
+
+  /// Dependency information for this task.
+  ompt_dependence_t *Dependencies;
+
+  /// Number of dependency entries.
+  unsigned DependencyCount;
+
+  void *PrivateData;
+  size_t PrivateDataSize;
+
+  int execution;
+  int freed;
+
+  TaskData(TaskData *Parent)
+      : InBarrier(false), Included(false), BarrierIndex(0), RefCount(1),
+        Parent(Parent), ImplicitTask(nullptr), Team(Parent->Team),
+        TaskGroup(nullptr), DependencyCount(0), execution(0), freed(0) {
+    if (Parent != nullptr) {
+      Parent->RefCount++;
+      // Copy over pointer to taskgroup. This task may set up its own stack
+      // but for now belongs to its parent's taskgroup.
+      TaskGroup = Parent->TaskGroup;
+    }
+  }
+
+  TaskData(ParallelData *Team = nullptr)
+      : InBarrier(false), Included(false), BarrierIndex(0), RefCount(1),
+        Parent(nullptr), ImplicitTask(this), Team(Team), TaskGroup(nullptr),
+        DependencyCount(0), execution(1), freed(0) {}
+
+  ~TaskData() {
+    TsanDeleteClock(&Task);
+    TsanDeleteClock(&Taskwait);
+  }
+
+  void *GetTaskPtr() { return &Task; }
+
+  void *GetTaskwaitPtr() { return &Taskwait; }
+  // overload new/delete to use DataPool for memory management.
+  void *operator new(size_t size) { return tdp->getData(); }
+  void operator delete(void *p, size_t) { retData<TaskData, 4>(p); }
+};
+
+static inline TaskData *ToTaskData(ompt_data_t *task_data) {
+  return reinterpret_cast<TaskData *>(task_data->ptr);
+}
+
+static inline void *ToInAddr(void *OutAddr) {
+  // FIXME: This will give false negatives when a second variable lays directly
+  //        behind a variable that only has a width of 1 byte.
+  //        Another approach would be to "negate" the address or to flip the
+  //        first bit...
+  return reinterpret_cast<char *>(OutAddr) + 1;
+}
+
+/// Store a mutex for each wait_id to resolve race condition with callbacks.
+std::unordered_map<ompt_wait_id_t, std::mutex> Locks;
+std::mutex LocksMutex;
+
+static void ompt_tsan_thread_begin(ompt_thread_t thread_type,
+                                   ompt_data_t *thread_data) {
+  pdp = new DataPool<ParallelData, 4>;
+  TsanNewMemory(pdp, sizeof(pdp));
+  tgp = new DataPool<Taskgroup, 4>;
+  TsanNewMemory(tgp, sizeof(tgp));
+  tdp = new DataPool<TaskData, 4>;
+  TsanNewMemory(tdp, sizeof(tdp));
+  thread_data->value = my_next_id();
+  if (archer_flags->print_ompt_counters && thread_data->value < MAX_THREADS)
+    this_event_counter = &(all_counter[thread_data->value]);
+  else
+    this_event_counter = NULL;
+  COUNT_EVENT1(thread_begin);
+}
+
+static void ompt_tsan_thread_end(ompt_data_t *thread_data) {
+  delete pdp;
+  delete tgp;
+  delete tdp;
+  COUNT_EVENT1(thread_end);
+}
+
+/// OMPT event callbacks for handling parallel regions.
+
+static void ompt_tsan_parallel_begin(ompt_data_t *parent_task_data,
+                                     const ompt_frame_t *parent_task_frame,
+                                     ompt_data_t *parallel_data,
+                                     uint32_t requested_team_size,
+                                     int flag,
+                                     const void *codeptr_ra) {
+  ParallelData *Data = new ParallelData;
+  parallel_data->ptr = Data;
+
+  TsanHappensBefore(Data->GetParallelPtr());
+  COUNT_EVENT1(parallel_begin);
+}
+
+static void ompt_tsan_parallel_end(ompt_data_t *parallel_data,
+                                   ompt_data_t *task_data,
+                                   int flag,
+                                   const void *codeptr_ra) {
+  ParallelData *Data = ToParallelData(parallel_data);
+  TsanHappensAfter(Data->GetBarrierPtr(0));
+  TsanHappensAfter(Data->GetBarrierPtr(1));
+
+  delete Data;
+
+#if (LLVM_VERSION >= 40)
+  if (&__archer_get_omp_status) {
+    if (__archer_get_omp_status() == 0 && archer_flags->flush_shadow)
+      __tsan_flush_memory();
+  }
+#endif
+
+  COUNT_EVENT1(parallel_end);
+}
+
+static void ompt_tsan_implicit_task(ompt_scope_endpoint_t endpoint,
+                                    ompt_data_t *parallel_data,
+                                    ompt_data_t *task_data,
+                                    unsigned int team_size,
+                                    unsigned int thread_num,
+                                    int type) {
+  switch (endpoint) {
+  case ompt_scope_begin:
+    task_data->ptr = new TaskData(ToParallelData(parallel_data));
+    TsanHappensAfter(ToParallelData(parallel_data)->GetParallelPtr());
+    COUNT_EVENT2(implicit_task, scope_begin);
+    break;
+  case ompt_scope_end:
+    TaskData *Data = ToTaskData(task_data);
+    assert(Data->freed == 0 && "Implicit task end should only be called once!");
+    Data->freed = 1;
+    assert(Data->RefCount == 1 &&
+           "All tasks should have finished at the implicit barrier!");
+    delete Data;
+    COUNT_EVENT2(implicit_task, scope_end);
+    break;
+  }
+}
+
+static void ompt_tsan_sync_region(ompt_sync_region_t kind,
+                                  ompt_scope_endpoint_t endpoint,
+                                  ompt_data_t *parallel_data,
+                                  ompt_data_t *task_data,
+                                  const void *codeptr_ra) {
+  TaskData *Data = ToTaskData(task_data);
+  switch (endpoint) {
+  case ompt_scope_begin:
+    switch (kind) {
+      case ompt_sync_region_barrier_implementation:
+        break;
+      case ompt_sync_region_barrier_implicit:
+      case ompt_sync_region_barrier_explicit:
+      case ompt_sync_region_barrier: {
+        char BarrierIndex = Data->BarrierIndex;
+        TsanHappensBefore(Data->Team->GetBarrierPtr(BarrierIndex));
+
+  if (hasReductionCallback < ompt_set_always) {
+          // We ignore writes inside the barrier. These would either occur during
+          // 1. reductions performed by the runtime which are guaranteed to be
+          // race-free.
+          // 2. execution of another task.
+          // For the latter case we will re-enable tracking in task_switch.
+          Data->InBarrier = true;
+          TsanIgnoreWritesBegin();
+        }
+
+        COUNT_EVENT3(sync_region, scope_begin, barrier);
+        break;
+      }
+
+      case ompt_sync_region_taskwait:
+        COUNT_EVENT3(sync_region, scope_begin, taskwait);
+        break;
+
+      case ompt_sync_region_taskgroup:
+        Data->TaskGroup = new Taskgroup(Data->TaskGroup);
+        COUNT_EVENT3(sync_region, scope_begin, taskgroup);
+        break;
+
+      default:
+        break;
+    }
+    break;
+  case ompt_scope_end:
+    switch (kind) {
+      case ompt_sync_region_barrier_implementation:
+        break;
+
+      case ompt_sync_region_barrier_implicit:
+      case ompt_sync_region_barrier_explicit:
+      case ompt_sync_region_barrier: {
+  if (hasReductionCallback < ompt_set_always) {
+          // We want to track writes after the barrier again.
+          Data->InBarrier = false;
+          TsanIgnoreWritesEnd();
+        }
+
+        char BarrierIndex = Data->BarrierIndex;
+        // Barrier will end after it has been entered by all threads.
+        if (parallel_data)
+          TsanHappensAfter(Data->Team->GetBarrierPtr(BarrierIndex));
+
+        // It is not guaranteed that all threads have exited this barrier before
+        // we enter the next one. So we will use a different address.
+        // We are however guaranteed that this current barrier is finished
+        // by the time we exit the next one. So we can then reuse the first
+        // address.
+        Data->BarrierIndex = (BarrierIndex + 1) % 2;
+        COUNT_EVENT3(sync_region, scope_end, barrier);
+        break;
+      }
+
+      case ompt_sync_region_taskwait: {
+        COUNT_EVENT3(sync_region, scope_end, taskwait);
+        if (Data->execution > 1)
+          TsanHappensAfter(Data->GetTaskwaitPtr());
+        break;
+      }
+
+      case ompt_sync_region_taskgroup: {
+        assert(Data->TaskGroup != nullptr &&
+               "Should have at least one taskgroup!");
+
+        TsanHappensAfter(Data->TaskGroup->GetPtr());
+
+        // Delete this allocated taskgroup, all descendent task are finished by
+        // now.
+        Taskgroup *Parent = Data->TaskGroup->Parent;
+        delete Data->TaskGroup;
+        Data->TaskGroup = Parent;
+        COUNT_EVENT3(sync_region, scope_end, taskgroup);
+        break;
+      }
+
+      default:
+        break;
+    }
+    break;
+  }
+}
+
+static void ompt_tsan_reduction(ompt_sync_region_t kind,
+                                ompt_scope_endpoint_t endpoint,
+                                ompt_data_t *parallel_data,
+                                ompt_data_t *task_data,
+                                const void *codeptr_ra) {
+  switch (endpoint) {
+  case ompt_scope_begin:
+    switch (kind) {
+      case ompt_sync_region_reduction:
+        TsanIgnoreWritesBegin();
+        COUNT_EVENT3(sync_region, scope_begin, reduction);
+        break;
+      default:
+        break;
+    }
+    break;
+  case ompt_scope_end:
+    switch (kind) {
+      case ompt_sync_region_reduction:
+        TsanIgnoreWritesEnd();
+        COUNT_EVENT3(sync_region, scope_begin, reduction);
+        break;
+      default:
+        break;
+    }
+    break;
+  }
+}
+
+/// OMPT event callbacks for handling tasks.
+
+static void ompt_tsan_task_create(
+    ompt_data_t *parent_task_data, /* id of parent task            */
+    const ompt_frame_t *parent_frame, /* frame data for parent task   */
+    ompt_data_t *new_task_data, /* id of created task           */
+    int type, int has_dependences,
+    const void *codeptr_ra) /* pointer to outlined function */
+{
+  TaskData *Data;
+  assert(new_task_data->ptr == NULL &&
+         "Task data should be initialized to NULL");
+  if (type & ompt_task_initial) {
+    ompt_data_t *parallel_data;
+    int team_size = 1;
+    ompt_get_parallel_info(0, &parallel_data, &team_size);
+    ParallelData *PData = new ParallelData;
+    parallel_data->ptr = PData;
+
+    Data = new TaskData(PData);
+    new_task_data->ptr = Data;
+    COUNT_EVENT2(task_create, initial);
+  } else if (type & ompt_task_undeferred) {
+    Data = new TaskData(ToTaskData(parent_task_data));
+    new_task_data->ptr = Data;
+    Data->Included = true;
+    COUNT_EVENT2(task_create, included);
+  } else if (type & ompt_task_explicit || type & ompt_task_target) {
+    Data = new TaskData(ToTaskData(parent_task_data));
+    new_task_data->ptr = Data;
+
+    // Use the newly created address. We cannot use a single address from the
+    // parent because that would declare wrong relationships with other
+    // sibling tasks that may be created before this task is started!
+    TsanHappensBefore(Data->GetTaskPtr());
+    ToTaskData(parent_task_data)->execution++;
+    COUNT_EVENT2(task_create, explicit);
+  }
+}
+
+static void ompt_tsan_task_schedule(ompt_data_t *first_task_data,
+                                    ompt_task_status_t prior_task_status,
+                                    ompt_data_t *second_task_data) {
+  COUNT_EVENT1(task_schedule);
+  TaskData *FromTask = ToTaskData(first_task_data);
+  TaskData *ToTask = ToTaskData(second_task_data);
+
+  if (ToTask->Included && prior_task_status != ompt_task_complete)
+    return; // No further synchronization for begin included tasks
+  if (FromTask->Included && prior_task_status == ompt_task_complete) {
+    // Just delete the task:
+    while (FromTask != nullptr && --FromTask->RefCount == 0) {
+      TaskData *Parent = FromTask->Parent;
+      if (FromTask->DependencyCount > 0) {
+        delete[] FromTask->Dependencies;
+      }
+      delete FromTask;
+      FromTask = Parent;
+    }
+    return;
+  }
+
+  // we will use new stack, assume growing down
+  // TsanNewMemory((char*)&FromTask-1024, 1024);
+  if (ToTask->execution == 0) {
+    ToTask->execution++;
+    // 1. Task will begin execution after it has been created.
+    TsanHappensAfter(ToTask->GetTaskPtr());
+    for (unsigned i = 0; i < ToTask->DependencyCount; i++) {
+      ompt_dependence_t *Dependency = &ToTask->Dependencies[i];
+
+      TsanHappensAfter(Dependency->variable.ptr);
+      // in and inout dependencies are also blocked by prior in dependencies!
+      if (Dependency->dependence_type == ompt_dependence_type_out || Dependency->dependence_type == ompt_dependence_type_inout) {
+        TsanHappensAfter(ToInAddr(Dependency->variable.ptr));
+      }
+    }
+  } else {
+    // 2. Task will resume after it has been switched away.
+    TsanHappensAfter(ToTask->GetTaskPtr());
+  }
+
+  if (prior_task_status != ompt_task_complete) {
+    ToTask->ImplicitTask = FromTask->ImplicitTask;
+    assert(ToTask->ImplicitTask != NULL &&
+           "A task belongs to a team and has an implicit task on the stack");
+  }
+
+  // Task may be resumed at a later point in time.
+  // TsanHappensBeforeUC(FromTask->GetTaskPtr());
+  TsanHappensBefore(FromTask->GetTaskPtr());
+
+  if (hasReductionCallback < ompt_set_always && FromTask->InBarrier) {
+    // We want to ignore writes in the runtime code during barriers,
+    // but not when executing tasks with user code!
+    TsanIgnoreWritesEnd();
+  }
+
+  if (prior_task_status == ompt_task_complete) { // task finished
+
+    // Task will finish before a barrier in the surrounding parallel region ...
+    ParallelData *PData = FromTask->Team;
+    TsanHappensBefore(
+        PData->GetBarrierPtr(FromTask->ImplicitTask->BarrierIndex));
+
+    // ... and before an eventual taskwait by the parent thread.
+    TsanHappensBefore(FromTask->Parent->GetTaskwaitPtr());
+
+    if (FromTask->TaskGroup != nullptr) {
+      // This task is part of a taskgroup, so it will finish before the
+      // corresponding taskgroup_end.
+      TsanHappensBefore(FromTask->TaskGroup->GetPtr());
+    }
+    for (unsigned i = 0; i < FromTask->DependencyCount; i++) {
+      ompt_dependence_t *Dependency = &FromTask->Dependencies[i];
+
+      // in dependencies block following inout and out dependencies!
+      TsanHappensBefore(ToInAddr(Dependency->variable.ptr));
+      if (Dependency->dependence_type == ompt_dependence_type_out || Dependency->dependence_type == ompt_dependence_type_inout) {
+        TsanHappensBefore(Dependency->variable.ptr);
+      }
+    }
+    while (FromTask != nullptr && --FromTask->RefCount == 0) {
+      TaskData *Parent = FromTask->Parent;
+      if (FromTask->DependencyCount > 0) {
+        delete[] FromTask->Dependencies;
+      }
+      delete FromTask;
+      FromTask = Parent;
+    }
+  }
+  if (hasReductionCallback < ompt_set_always && ToTask->InBarrier) {
+    // We re-enter runtime code which currently performs a barrier.
+    TsanIgnoreWritesBegin();
+  }
+}
+
+static void ompt_tsan_dependences(ompt_data_t *task_data,
+                                  const ompt_dependence_t *deps,
+                                  int ndeps) {
+  COUNT_EVENT1(task_dependences);
+  if (ndeps > 0) {
+    // Copy the data to use it in task_switch and task_end.
+    TaskData *Data = ToTaskData(task_data);
+    Data->Dependencies = new ompt_dependence_t[ndeps];
+    std::memcpy(Data->Dependencies, deps,
+                sizeof(ompt_dependence_t) * ndeps);
+    Data->DependencyCount = ndeps;
+
+    // This callback is executed before this task is first started.
+    TsanHappensBefore(Data->GetTaskPtr());
+  }
+}
+
+/// OMPT event callbacks for handling locking.
+static void ompt_tsan_mutex_acquired(ompt_mutex_t kind,
+                                     ompt_wait_id_t wait_id,
+                                     const void *codeptr_ra) {
+  if (archer_flags->print_ompt_counters)
+    switch (kind) {
+    case ompt_mutex_lock:
+      COUNT_EVENT2(mutex_acquired, lock);
+      break;
+    case ompt_mutex_nest_lock:
+      COUNT_EVENT2(mutex_acquired, nest_lock);
+      break;
+    case ompt_mutex_critical:
+      COUNT_EVENT2(mutex_acquired, critical);
+      break;
+    case ompt_mutex_atomic:
+      COUNT_EVENT2(mutex_acquired, atomic);
+      break;
+    case ompt_mutex_ordered:
+      COUNT_EVENT2(mutex_acquired, ordered);
+      break;
+    default:
+      COUNT_EVENT2(mutex_acquired, default);
+      break;
+    }
+
+  // Acquire our own lock to make sure that
+  // 1. the previous release has finished.
+  // 2. the next acquire doesn't start before we have finished our release.
+  {
+    LocksMutex.lock();
+    std::mutex &Lock = Locks[wait_id];
+    LocksMutex.unlock();
+
+    Lock.lock();
+    TsanHappensAfter(&Lock);
+  }
+}
+
+static void ompt_tsan_mutex_released(ompt_mutex_t kind,
+                                     ompt_wait_id_t wait_id,
+                                     const void *codeptr_ra) {
+  if (archer_flags->print_ompt_counters)
+    switch (kind) {
+    case ompt_mutex_lock:
+      COUNT_EVENT2(mutex_released, lock);
+      break;
+    case ompt_mutex_nest_lock:
+      COUNT_EVENT2(mutex_released, nest_lock);
+      break;
+    case ompt_mutex_critical:
+      COUNT_EVENT2(mutex_released, critical);
+      break;
+    case ompt_mutex_atomic:
+      COUNT_EVENT2(mutex_released, atomic);
+      break;
+    case ompt_mutex_ordered:
+      COUNT_EVENT2(mutex_released, ordered);
+      break;
+    default:
+      COUNT_EVENT2(mutex_released, default);
+      break;
+    }
+
+  {
+    LocksMutex.lock();
+    std::mutex &Lock = Locks[wait_id];
+    LocksMutex.unlock();
+    TsanHappensBefore(&Lock);
+
+    Lock.unlock();
+  }
+}
+
+// callback , signature , variable to store result , required support level
+#define SET_OPTIONAL_CALLBACK_T(event, type, result, level)                             \
+  do {                                                                                  \
+    ompt_callback_##type##_t tsan_##event = &ompt_tsan_##event;                         \
+    result = ompt_set_callback(ompt_callback_##event,                                   \
+                                (ompt_callback_t)tsan_##event);                         \
+    if (result < level)                                                                 \
+      printf("Registered callback '" #event "' is not supported at " #level " (%i)\n",  \
+             result);                                                                   \
+  } while (0)
+
+#define SET_CALLBACK_T(event, type)                              \
+  do {                                                           \
+    int res;                                                     \
+    SET_OPTIONAL_CALLBACK_T(event, type, res, ompt_set_always);  \
+  } while (0)
+
+#define SET_CALLBACK(event) SET_CALLBACK_T(event, event)
+
+static int ompt_tsan_initialize(ompt_function_lookup_t lookup,
+                                int device_num,
+                                ompt_data_t *tool_data) {
+  const char *options = getenv("ARCHER_OPTIONS");
+  archer_flags = new ArcherFlags(options);
+
+  if (archer_flags->print_ompt_counters)
+    all_counter = new callback_counter_t[MAX_THREADS];
+
+  ompt_set_callback_t ompt_set_callback =
+      (ompt_set_callback_t)lookup("ompt_set_callback");
+  if (ompt_set_callback == NULL) {
+    std::cerr << "Could not set callback, exiting..." << std::endl;
+    std::exit(1);
+  }
+  ompt_get_parallel_info =
+      (ompt_get_parallel_info_t)lookup("ompt_get_parallel_info");
+  ompt_get_thread_data = (ompt_get_thread_data_t)lookup("ompt_get_thread_data");
+
+  if (ompt_get_parallel_info == NULL) {
+    fprintf(stderr, "Could not get inquiry function 'ompt_get_parallel_info', "
+                    "exiting...\n");
+    exit(1);
+  }
+
+  SET_CALLBACK(thread_begin);
+  SET_CALLBACK(thread_end);
+  SET_CALLBACK(parallel_begin);
+  SET_CALLBACK(implicit_task);
+  SET_CALLBACK(sync_region);
+  SET_CALLBACK(parallel_end);
+
+  SET_CALLBACK(task_create);
+  SET_CALLBACK(task_schedule);
+  SET_CALLBACK(dependences);
+
+  SET_CALLBACK_T(mutex_acquired, mutex);
+  SET_CALLBACK_T(mutex_released, mutex);
+  SET_OPTIONAL_CALLBACK_T(reduction, sync_region, hasReductionCallback, ompt_set_never);
+  return 1; // success
+}
+
+static void ompt_tsan_finalize(ompt_data_t *tool_data) {
+  if (archer_flags->print_ompt_counters) {
+    print_callbacks(all_counter);
+    delete[] all_counter;
+  }
+
+  if (archer_flags->print_max_rss) {
+    struct rusage end;
+    getrusage(RUSAGE_SELF, &end);
+    printf("MAX RSS[KBytes] during execution: %ld\n", end.ru_maxrss);
+  }
+
+  if (archer_flags)
+    delete archer_flags;
+}
+
+extern "C"
+ompt_start_tool_result_t *ompt_start_tool(unsigned int omp_version,
+                                          const char *runtime_version) {
+  const char *options = getenv("ARCHER_OPTIONS");
+  archer_flags = new ArcherFlags(options);
+  if (!archer_flags->enabled)
+  {
+    if (archer_flags->verbose)
+      std::cout << "Archer disabled, stopping operation"
+                << std::endl;
+    delete archer_flags;
+    return NULL;
+  }
+  
+  static ompt_start_tool_result_t ompt_start_tool_result = {
+      &ompt_tsan_initialize, &ompt_tsan_finalize, {0}};
+  runOnTsan=1;
+  RunningOnValgrind();
+  if (!runOnTsan) // if we are not running on TSAN, give a different tool the
+    // chance to be loaded
+  {
+    if (archer_flags->verbose)
+      std::cout << "Archer detected OpenMP application without TSan "
+                   "stopping operation"
+                << std::endl;
+    delete archer_flags;
+    return NULL;
+  }
+
+  if (archer_flags->verbose)
+    std::cout << "Archer detected OpenMP application with TSan, supplying "
+                 "OpenMP synchronization semantics"
+              << std::endl;
+  return &ompt_start_tool_result;
+}

tools/archer/tests/CMakeLists.txt

+# CMakeLists.txt file for unit testing Archer runtime library.
+include(CheckFunctionExists)
+include(CheckLibraryExists)
+
+# Some tests use math functions
+check_library_exists(m sqrt "" LIBARCHER_HAVE_LIBM)
+# When using libgcc, -latomic may be needed for atomics
+# (but when using compiler-rt, the atomics will be built-in)
+# Note: we can not check for __atomic_load because clang treats it
+# as special built-in and that breaks CMake checks
+check_function_exists(__atomic_load_1 LIBARCHER_HAVE_BUILTIN_ATOMIC)
+if(NOT LIBARCHER_HAVE_BUILTIN_ATOMIC)
+  check_library_exists(atomic __atomic_load_1 "" LIBARCHER_HAVE_LIBATOMIC)
+else()
+  # not needed
+  set(LIBARCHER_HAVE_LIBATOMIC 0)
+endif()
+
+set(LIBARCHER_TEST_PATH ${CMAKE_CURRENT_SOURCE_DIR})
+set(LIBARCHER_HAVE_ARCHER_LIBRARY ${LIBARCHER_LIB_PATH})
+set(LIBARCHER_HAVE_ARCHER_RUNTIME ${LIBARCHER_RUNTIME_PATH})
+
+macro(pythonize_bool var)
+  if (${var})
+    set(${var} True)
+  else()
+    set(${var} False)
+  endif()
+endmacro()
+
+pythonize_bool(LIBARCHER_HAVE_ARCHER_LIBRARY)
+pythonize_bool(LIBARCHER_HAVE_ARCHER_RUNTIME)
+pythonize_bool(LIBOMP_OMPT_SUPPORT)
+pythonize_bool(LIBOMP_OMPT_OPTIONAL)
+pythonize_bool(LIBOMP_HAVE_LIBM)
+pythonize_bool(LIBOMP_HAVE_LIBATOMIC)
+pythonize_bool(LIBOMP_USE_HWLOC)
+
+add_openmp_testsuite(check-libarcher "Running libarcher tests" ${CMAKE_CURRENT_BINARY_DIR} DEPENDS archer)
+
+# Configure the lit.site.cfg.in file
+set(AUTO_GEN_COMMENT "## Autogenerated by libarcher configuration.\n# Do not edit!")
+configure_file(lit.site.cfg.in lit.site.cfg @ONLY)

tools/archer/tests/barrier/barrier.c

+/*
+ * barrier.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    if (omp_get_thread_num() == 0) {
+      var++;
+    }
+
+#pragma omp barrier
+
+    if (omp_get_thread_num() == 1) {
+      var++;
+    }
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/critical/critical.c

+/*
+ * critical.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(8) shared(var)
+  {
+#pragma omp critical
+    { var++; }
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 8);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/critical/lock-nested.c

+/*
+ * lock-nested.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+  omp_nest_lock_t lock;
+  omp_init_nest_lock(&lock);
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    omp_set_nest_lock(&lock);
+    omp_set_nest_lock(&lock);
+    var++;
+    omp_unset_nest_lock(&lock);
+    omp_unset_nest_lock(&lock);
+  }
+
+  omp_destroy_nest_lock(&lock);
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/critical/lock.c

+/*
+ * lock.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+  omp_lock_t lock;
+  omp_init_lock(&lock);
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    omp_set_lock(&lock);
+    var++;
+    omp_unset_lock(&lock);
+  }
+
+  omp_destroy_lock(&lock);
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/deflake.bash

+#!/usr/bin/env bash
+# This script is used to deflake inherently flaky archer tests.
+# It is invoked from lit tests as:
+# %deflake mybinary
+# which is then substituted by lit to:
+# $(dirname %s)/deflake.bash mybinary
+# The script runs the target program up to 10 times,
+# until it fails (i.e. produces a race report).
+
+for i in $(seq 1 10); do
+    OUT=`$@ 2>&1`
+    if [[ $? != 0 ]]; then
+  echo "$OUT"
+  exit 0
+    fi
+done
+exit 1

tools/archer/tests/lit.cfg

+# -*- Python -*- vim: set ft=python ts=4 sw=4 expandtab tw=79:
+# Configuration file for the 'lit' test runner.
+
+import os
+import re
+import subprocess
+import lit.formats
+
+# Tell pylint that we know config and lit_config exist somewhere.
+if 'PYLINT_IMPORT' in os.environ:
+    config = object()
+    lit_config = object()
+
+def append_dynamic_library_path(path):
+    if config.operating_system == 'Windows':
+        name = 'PATH'
+        sep = ';'
+    elif config.operating_system == 'Darwin':
+        name = 'DYLD_LIBRARY_PATH'
+        sep = ':'
+    else:
+        name = 'LD_LIBRARY_PATH'
+        sep = ':'
+    if name in config.environment:
+        config.environment[name] = path + sep + config.environment[name]
+    else:
+        config.environment[name] = path
+
+# name: The name of this test suite.
+config.name = 'libarcher'
+
+# suffixes: A list of file extensions to treat as test files.
+config.suffixes = ['.c', '.cpp']
+
+# test_source_root: The root path where tests are located.
+config.test_source_root = os.path.dirname(__file__)
+
+# test_exec_root: The root object directory where output is placed
+config.test_exec_root = config.libarcher_obj_root
+
+# test format
+config.test_format = lit.formats.ShTest()
+
+# compiler flags
+config.test_flags = " -I " + config.test_source_root + \
+    " -I " + config.omp_header_dir + \
+    " -L " + config.omp_library_dir + \
+    " -Wl,-rpath," + config.omp_library_dir + \
+    " " + config.test_extra_flags
+
+config.archer_flags = "-g -O1 -fsanitize=thread"
+
+libs_archer = ""
+if config.has_archer_runtime:
+    libs_archer += " -L" + config.archer_runtime_dir + " -l" + \
+    config.archer_runtime.replace("lib", "").replace(".so", "").replace(".dy", "") + \
+    " -Wl,-rpath," + config.archer_runtime_dir
+
+# if config.has_archer_library:
+#   config.test_compiler += " -Xclang -load -Xclang " + \
+#     config.archer_library_dir + "/" + config.archer_library + \
+#     " -Wl,-rpath," + config.archer_library_dir
+
+
+# extra libraries
+libs = ""
+if config.has_libm:
+    libs += " -lm"
+if config.has_libatomic:
+    libs += " -latomic"
+
+# Allow XFAIL to work
+config.target_triple = [ ]
+for feature in config.test_compiler_features:
+    config.available_features.add(feature)
+
+# Setup environment to find dynamic library at runtime
+append_dynamic_library_path(config.omp_library_dir)
+
+# Rpath modifications for Darwin
+if config.operating_system == 'Darwin':
+    config.test_flags += " -Wl,-rpath," + config.omp_library_dir
+
+# Find the SDK on Darwin
+if config.operating_system == 'Darwin':
+  cmd = subprocess.Popen(['xcrun', '--show-sdk-path'],
+                         stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+  out, err = cmd.communicate()
+  out = out.strip()
+  res = cmd.wait()
+  if res == 0 and out:
+    config.test_flags += " -isysroot " + out
+
+config.suppression = ""
+# need to deal with the static analysis with another execution line in the test
+# if not config.has_archer_library and config.has_archer_runtime:
+if config.has_archer_runtime:
+  config.suppression = "env TSAN_OPTIONS=\"ignore_noninstrumented_modules=1\""
+
+# Disable OMPT tests if FileCheck was not found
+if config.has_ompt and config.test_filecheck == "":
+    lit_config.note("Not testing OMPT because FileCheck was not found")
+    config.has_ompt = False
+
+if config.has_ompt:
+    config.available_features.add("ompt")
+    # for callback.h
+    config.test_flags += " -I " + config.test_source_root + "/ompt"
+
+if 'Linux' in config.operating_system:
+    config.available_features.add("linux")
+
+# to run with icc INTEL_LICENSE_FILE must be set
+if 'INTEL_LICENSE_FILE' in os.environ:
+    config.environment['INTEL_LICENSE_FILE'] = os.environ['INTEL_LICENSE_FILE']
+
+# OMPT Tests
+config.substitutions.append(("%libomp-compile-and-run", \
+    "%libomp-compile && %libomp-run"))
+config.substitutions.append(("%libomp-cxx-compile-and-run", \
+    "%libomp-cxx-compile && %libomp-run"))
+config.substitutions.append(("%libomp-cxx-compile", \
+    "%clangXX %openmp_flags %flags -std=c++11 %s -o %t" + libs))
+config.substitutions.append(("%libomp-compile", \
+    "%clang %openmp_flags %flags %s -o %t" + libs))
+config.substitutions.append(("%libomp-run", "%t | tee %t.log"))
+config.substitutions.append(("%clangXX", config.test_cxx_compiler))
+config.substitutions.append(("%clang", config.test_c_compiler))
+config.substitutions.append(("%openmp_flags", config.test_openmp_flags))
+config.substitutions.append(("%flags", config.test_flags))
+
+# Race Tests
+config.substitutions.append(("%libarcher-compile-and-run-race", \
+    "%libarcher-compile && %libarcher-run-race"))
+config.substitutions.append(("%libarcher-compile-and-run", \
+                             "%libarcher-compile && %libarcher-run"))
+config.substitutions.append(("%libarcher-cxx-compile-and-run", \
+    "%libarcher-cxx-compile && %libarcher-run"))
+config.substitutions.append(("%libarcher-cxx-compile", \
+    "%clang-archerXX %openmp_flags %archer_flags %flags -std=c++11 %s -o %t" + libs))
+config.substitutions.append(("%libarcher-compile", \
+                             "%clang-archer %openmp_flags %archer_flags %flags %s -o %t" + libs + libs_archer))
+config.substitutions.append(("%libarcher-run-race", "%suppression %deflake %t 2>&1 | tee %t.log"))
+config.substitutions.append(("%libarcher-run", "%suppression %t 2>&1 | tee %t.log"))
+config.substitutions.append(("%clang-archerXX", config.test_cxx_compiler))
+config.substitutions.append(("%clang-archer", config.test_c_compiler))
+config.substitutions.append(("%openmp_flags", config.test_openmp_flags))
+config.substitutions.append(("%archer_flags", config.archer_flags))
+config.substitutions.append(("%flags", config.test_flags))
+config.substitutions.append(("%suppression", "env OMP_TOOL_LIBRARIES=" + config.libomp_obj_root + \
+           "/../libarcher.so TSAN_OPTIONS='ignore_noninstrumented_modules=1'"))
+config.substitutions.append(("%deflake", os.path.join(os.path.dirname(__file__), "deflake.bash")))
+
+config.substitutions.append(("FileCheck", config.test_filecheck))
+config.substitutions.append(("%sort-threads", "sort --numeric-sort --stable"))
+if config.operating_system == 'Windows':
+    # No such environment variable on Windows.
+    config.substitutions.append(("%preload-tool", "true ||"))
+elif config.operating_system == 'Darwin':
+    config.substitutions.append(("%preload-tool", "env DYLD_INSERT_LIBRARIES=%T/tool.so"))
+else:
+    config.substitutions.append(("%preload-tool", "env LD_PRELOAD=%T/tool.so"))

tools/archer/tests/lit.site.cfg.in

+@AUTO_GEN_COMMENT@
+
+config.test_c_compiler = "@OPENMP_TEST_C_COMPILER@"
+config.test_cxx_compiler = "@OPENMP_TEST_CXX_COMPILER@"
+config.test_compiler_features = @OPENMP_TEST_COMPILER_FEATURES@
+config.test_filecheck = "@OPENMP_FILECHECK_EXECUTABLE@"
+config.test_openmp_flags = "@OPENMP_TEST_OPENMP_FLAGS@"
+config.test_extra_flags = "@OPENMP_TEST_FLAGS@"
+config.libomp_obj_root = "@CMAKE_CURRENT_BINARY_DIR@"
+config.omp_library_dir = "@LIBOMP_LIBRARY_DIR@"
+config.omp_header_dir = "@LIBOMP_INCLUDE_DIR@"
+config.operating_system = "@CMAKE_SYSTEM_NAME@"
+config.hwloc_library_dir = "@LIBOMP_HWLOC_LIBRARY_DIR@"
+config.using_hwloc = @LIBOMP_USE_HWLOC@
+config.has_ompt = @LIBOMP_OMPT_SUPPORT@ and @LIBOMP_OMPT_OPTIONAL@
+config.has_libm = @LIBOMP_HAVE_LIBM@
+config.has_libatomic = @LIBOMP_HAVE_LIBATOMIC@
+
+config.test_archer_flags = "@OPENMP_TEST_ARCHER_FLAGS@"
+config.libarcher_obj_root = "@CMAKE_CURRENT_BINARY_DIR@"
+
+config.archer_tools_dir = "@LIBARCHER_TOOLS_DIR@"
+config.archer_library_dir = "@LIBARCHER_LIB_PATH@"
+config.archer_runtime_dir = "@LIBARCHER_RUNTIME_PATH@"
+config.archer_library = "@LIBARCHER_LIB@"
+config.archer_runtime = "@LIBARCHER_RTL@"
+config.has_archer_library = @LIBARCHER_HAVE_ARCHER_LIBRARY@
+config.has_archer_runtime = @LIBARCHER_HAVE_ARCHER_RUNTIME@
+
+# print("\nStart")
+# print("config.test_c_compiler: " + str(config.test_c_compiler))
+# print("config.test_cxx_compiler: " + str(config.test_cxx_compiler))
+# print("config.test_compiler_features: " + str(config.test_compiler_features))
+# print("config.test_filecheck: " + str(config.test_filecheck))
+# print("config.test_archer_flags: " + str(config.test_archer_flags))
+# print("config.test_extra_flags: " + str(config.test_extra_flags))
+# print("config.libarcher_obj_root: " + str(config.libarcher_obj_root))
+# print("config.omp_library_dir: " + str(config.omp_library_dir))
+# print("config.omp_header_dir: " + str(config.omp_header_dir))
+# print("config.operating_system: " + str(config.operating_system))
+# print("config.has_ompt:" + str(config.has_ompt))
+# print("config.has_libm:" + str(config.has_libm))
+# print("config.has_libatomic: " + str(config.has_libatomic))
+# print("config.archer_tools_dir" + str(config.archer_tools_dir))
+# print("config.archer_library_dir: " + str(config.archer_library_dir))
+# print("config.archer_runtime_dir: " + str(config.archer_runtime_dir))
+# print("config.archer_library: " + str(config.archer_library))
+# print("config.archer_runtime: " + str(config.archer_runtime))
+# print("config.has_archer_library: " + str(config.has_archer_library))
+# print("config.has_archer_runtime: " + str(config.has_archer_runtime))
+# print("config.suppressions_archer_runtime_file: " + str(config.suppressions_archer_runtime_file))
+# print("End\n")
+
+# Let the main config do the real work.
+lit_config.load_config(config, "@CMAKE_CURRENT_SOURCE_DIR@/lit.cfg")

tools/archer/tests/ompt/ompt-signal.h

+/*
+ * ompt-signal.h -- Header providing low-level synchronization for tests
+ */
+
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is a copy from runtime/test/ompt/
+//
+//===----------------------------------------------------------------------===//
+
+#if defined(WIN32) || defined(_WIN32)
+#include <windows.h>
+#define delay() Sleep(1);
+#else
+#include <unistd.h>
+#define delay(t) usleep(t);
+#endif
+
+// These functions are used to provide a signal-wait mechanism to enforce
+// expected scheduling for the test cases.
+// Conditional variable (s) needs to be shared! Initialize to 0
+
+#define OMPT_SIGNAL(s) ompt_signal(&s)
+// inline
+void ompt_signal(int *s) {
+#pragma omp atomic
+  (*s)++;
+}
+
+#define OMPT_WAIT(s, v) ompt_wait(&s, v)
+// wait for s >= v
+// inline
+void ompt_wait(int *s, int v) {
+  int wait = 0;
+  do {
+    delay(10);
+#pragma omp atomic read
+    wait = (*s);
+  } while (wait < v);
+}

tools/archer/tests/parallel/parallel-firstprivate.c

+/*
+ * parallel-firstprivate.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(2) firstprivate(var)
+  { var = 1; }
+
+  fprintf(stderr, "DONE\n");
+  // var should still be 0!
+  return var;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/parallel/parallel-simple.c

+/*
+ * parallel-simple.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    if (omp_get_thread_num() == 1) {
+      var++;
+    }
+  } // implicit barrier
+
+  var++;
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/parallel/parallel-simple2.c

+/*
+ * parallel-simple2.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run  | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+// Create team of threads so that there is no implicit happens before
+// when creating the thread.
+#pragma omp parallel num_threads(2)
+  {}
+
+  var++;
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    if (omp_get_thread_num() == 1) {
+      var++;
+    }
+  } // implicit barrier
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/races/critical-unrelated.c

+/*
+ * critical-unrelated.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+#pragma omp critical
+    {
+      // Dummy region.
+    }
+
+    var++;
+  }
+
+  fprintf(stderr, "DONE\n");
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/lock-nested-unrelated.c

+/*
+ * lock-nested-unrelated.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+  omp_nest_lock_t lock;
+  omp_init_nest_lock(&lock);
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    omp_set_nest_lock(&lock);
+    omp_set_nest_lock(&lock);
+    // Dummy locking.
+    omp_unset_nest_lock(&lock);
+    omp_unset_nest_lock(&lock);
+
+    var++;
+  }
+
+  omp_destroy_nest_lock(&lock);
+
+  fprintf(stderr, "DONE\n");
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/lock-unrelated.c

+/*
+ * lock-unrelated.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+  omp_lock_t lock;
+  omp_init_lock(&lock);
+
+#pragma omp parallel num_threads(2) shared(var)
+  {
+    omp_set_lock(&lock);
+    // Dummy locking.
+    omp_unset_lock(&lock);
+
+    var++;
+  }
+
+  omp_destroy_lock(&lock);
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/parallel-simple.c

+/*
+ * parallel-simple.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+#pragma omp parallel num_threads(2) shared(var)
+  { var++; }
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/task-dependency.c

+/*
+ * task-deoendency.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt/ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task shared(var, a) depend(out : var)
+    {
+      OMPT_SIGNAL(a);
+      var++;
+    }
+
+#pragma omp task shared(a) depend(in : var)
+    {
+      OMPT_SIGNAL(a);
+      OMPT_WAIT(a, 3);
+    }
+
+#pragma omp task shared(var) // depend(in: var) is missing here!
+    {
+      var++;
+      OMPT_SIGNAL(a);
+    }
+
+    // Give other thread time to steal the task.
+    OMPT_WAIT(a, 2);
+  }
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: #1 .omp_task_entry.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: #1 .omp_task_entry.
+// CHECK: DONE

tools/archer/tests/races/task-taskgroup-unrelated.c

+/*
+ * task-taskgroup-unrelated.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task shared(var, a)
+    {
+      var++;
+      OMPT_SIGNAL(a);
+      // Give master thread time to execute the task in the taskgroup.
+      OMPT_WAIT(a, 2);
+    }
+
+#pragma omp taskgroup
+    {
+#pragma omp task if (0)
+      {
+        // Dummy task.
+      }
+
+      // Give other threads time to steal the tasks.
+      OMPT_WAIT(a, 1);
+      OMPT_SIGNAL(a);
+    }
+
+    var++;
+  }
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/task-taskwait-nested.c

+/*
+ * task-taskwait-nested.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task shared(var, a)
+    {
+#pragma omp task shared(var, a)
+      {
+        // wait for master to pass the taskwait
+        OMPT_SIGNAL(a);
+        OMPT_WAIT(a, 2);
+        var++;
+      }
+    }
+
+    // Give other thread time to steal the task and execute its child.
+    OMPT_WAIT(a, 1);
+
+// Only directly generated children are guaranteed to be executed.
+#pragma omp taskwait
+    OMPT_SIGNAL(a);
+    var++;
+  }
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK:   Previous write of size 4
+// CHECK: #0 .omp_outlined.
+// CHECK: DONE

tools/archer/tests/races/task-two.c

+/*
+ * task-two.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run-race | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#define NUM_THREADS 2
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+  int i;
+
+#pragma omp parallel for num_threads(NUM_THREADS) shared(var) schedule(static, \
+                                                                       1)
+  {
+    for (i = 0; i < NUM_THREADS; i++) {
+#pragma omp task shared(var) if (0) // the task is inlined an executed locally
+      { var++; }
+    }
+  }
+
+  int error = (var != 2);
+  fprintf(stderr, "DONE\n");
+  return error;
+}
+
+// CHECK: WARNING: ThreadSanitizer: data race
+// CHECK:   Write of size 4
+// CHECK:     #0 .omp_outlined.
+// CHECK:     #1 .omp_task_entry.
+// CHECK:   Previous write of size 4
+// CHECK:     #0 .omp_outlined.
+// CHECK:     #1 .omp_task_entry.
+// CHECK: DONE

tools/archer/tests/reduction/parallel-reduction-nowait.c

+/*
+ * parallel-reduction-nowait.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0, i;
+  int sum1 = 0;
+  int sum2 = 0;
+
+// Number of threads is empirical: We need enough threads so that
+// the reduction is really performed hierarchically in the barrier!
+#pragma omp parallel num_threads(5) reduction(+ : var)
+  {
+#pragma omp for schedule(static) nowait reduction(+ : sum1)
+    for (i = 0; i < 5; i++)
+      sum1 += i;
+#pragma omp for schedule(static) reduction(+ : sum2)
+    for (i = 0; i < 5; i++)
+      sum2 += i;
+
+    var = sum1 + sum2;
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 100);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/reduction/parallel-reduction.c

+/*
+ * parallel-reduction.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run| FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+
+// Number of threads is empirical: We need enough threads so that
+// the reduction is really performed hierarchically in the barrier!
+#pragma omp parallel num_threads(5) reduction(+ : var)
+  { var = 1; }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 5);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-barrier.c

+/*
+ * task-barrier.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt/ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+  {
+#pragma omp master
+    {
+#pragma omp task shared(var)
+      {
+        OMPT_SIGNAL(a);
+        var++;
+      }
+
+      // Give other thread time to steal the task.
+      OMPT_WAIT(a, 1);
+    }
+
+#pragma omp barrier
+
+#pragma omp master
+    { var++; }
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-create.c

+/*
+ * task-create.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+    var++;
+#pragma omp task shared(var, a)
+    {
+      var++;
+      OMPT_SIGNAL(a);
+    }
+
+    // Give other thread time to steal the task.
+    OMPT_WAIT(a, 1);
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-dependency.c

+/*
+ * task-dependency.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task shared(var, a) depend(out : var)
+    {
+      var++;
+      OMPT_SIGNAL(a);
+    }
+
+#pragma omp task shared(var, a) depend(in : var)
+    { OMPT_WAIT(a, 2); }
+
+#pragma omp task shared(var, a) depend(in : var)
+    {
+      OMPT_SIGNAL(a);
+      var++;
+    }
+
+    // Give other thread time to steal the task.
+    OMPT_WAIT(a, 1);
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-taskgroup-nested.c

+/*
+ * task-taskgroup-nested.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp taskgroup
+    {
+#pragma omp task
+      {
+#pragma omp task shared(var, a)
+        {
+          var++;
+          OMPT_SIGNAL(a);
+        }
+      }
+
+      // Give other thread time to steal the task and execute its child.
+      OMPT_WAIT(a, 1);
+    }
+
+    var++;
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-taskgroup.c

+/*
+ * task-taskgroup.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp taskgroup
+    {
+#pragma omp task shared(var, a)
+      {
+        var++;
+        OMPT_SIGNAL(a);
+      }
+
+      // Give other thread time to steal the task.
+      OMPT_WAIT(a, 1);
+    }
+
+    var++;
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-taskwait-nested.c

+/*
+ * task-taskwait-nested.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task
+    {
+#pragma omp task shared(var, a)
+      {
+        OMPT_SIGNAL(a);
+        delay(100);
+        var++;
+      }
+#pragma omp taskwait
+    }
+
+    // Give other thread time to steal the task and execute its child.
+    OMPT_WAIT(a, 1);
+
+#pragma omp taskwait
+    var++;
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/task/task-taskwait.c

+/*
+ * task-taskwait.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+#include <unistd.h>
+#include "ompt-signal.h"
+
+int main(int argc, char *argv[]) {
+  int var = 0, a = 0;
+
+#pragma omp parallel num_threads(2) shared(var, a)
+#pragma omp master
+  {
+#pragma omp task shared(var, a)
+    {
+      OMPT_SIGNAL(a);
+      OMPT_WAIT(a, 2);
+      delay(100);
+      var++;
+    }
+
+    // Give other thread time to steal the task.
+    OMPT_WAIT(a, 1);
+    OMPT_SIGNAL(a);
+#pragma omp taskwait
+    var++;
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE

tools/archer/tests/worksharing/ordered.c

+/*
+ * ordered.c -- Archer testcase
+ */
+
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//
+// See tools/archer/LICENSE.txt for details.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+
+// RUN: %libarcher-compile-and-run | FileCheck %s
+#include <omp.h>
+#include <stdio.h>
+
+#define NUM_THREADS 2
+
+int main(int argc, char *argv[]) {
+  int var = 0;
+  int i;
+
+#pragma omp parallel for ordered num_threads(NUM_THREADS) shared(var)
+  for (i = 0; i < NUM_THREADS; i++) {
+#pragma omp ordered
+    { var++; }
+  }
+
+  fprintf(stderr, "DONE\n");
+  int error = (var != 2);
+  return error;
+}
+
+// CHECK-NOT: ThreadSanitizer: data race
+// CHECK-NOT: ThreadSanitizer: reported
+// CHECK: DONE