[OpenMP][OMPT] Implement verbose tool loading

[protze.joachim]

Summary

OpenMP 5.1 introduces the new env variable OMP_TOOL_VERBOSE_INIT=(disabled|stdout|stderr|<filename>) to enable verbose loading and initialization of OMPT tools.
This env variable helps to understand the cause when loading of a tool fails (e.g., undefined symbols or dependency not in LD_LIBRARY_PATH)

Tests for this patch are integrated into the different existing tool loading tests, making these tests more verbose.

Patch prepared by: Isabel Thärigen

[hbae]

Can we also add print routine for OMP_TOOL_VERBOSE_INIT in kmp_settings.cpp (for OMP_DISPLAY_ENV) ?

openmp/runtime/src/ompt-general.cpp

248

Can we put the common pattern if (verbose_init) fprintf(verbose_file, into a macro? e.g., OMPT_VERBOSE(<message>). We may be able to add some common prefix to the macro if desired.

[protze.joachim]

Use macros for the conditional verbose output.
We also added a test for the archer-specific output.

[protze.joachim]

Also include OMP_TOOL_VERBOSE_INIT in the output for OMP_DISPLAY_ENV as requested.

[hbae]

Looks good to me.

[tianshilei1992]

The verbose information used in this patch is limited to Linux only. On macOS it prints completely different information, which causes regression on macOS (https://github.com/llvm/llvm-project/issues/56833). Can you please fix it?

[tianshilei1992]

FYI, the following is the output on macOS. I'm not sure if the output makes sense or not.

➜ env OMP_TOOL_LIBRARIES=$PWD/non_existing_file.so:$PWD/first_tool.so:$PWD/second_tool.so:$PWD/third_tool.so OMP_TOOL_VERBOSE_INIT=stdout ./tool_available_search.c.tmp
----- START LOGGING OF TOOL REGISTRATION -----
Search for OMP tool in current address space... Failed.
Searching tool libraries...
OMP_TOOL_LIBRARIES = /Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/non_existing_file.so:/Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/first_tool.so:/Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/second_tool.so:/Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/third_tool.so
Opening /Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/non_existing_file.so... Failed: dlopen(/Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/non_existing_file.so, 0x0001): tried: '/Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/non_existing_file.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/non_existing_file.so' (no such file), '/Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/non_existing_file.so' (no such file)
Opening /Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/first_tool.so... Success.
Searching for ompt_start_tool in /Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/first_tool.so... Failed: dlsym(0x7ff90127b170, ompt_start_tool): symbol not found
Opening /Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/second_tool.so... Success.
Searching for ompt_start_tool in /Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/second_tool.so... 0: Do not initialize tool
Found but not using the OMPT interface.
Continuing search...
Opening /Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/third_tool.so... Success.
Searching for ompt_start_tool in /Users/shiltian/Documents/code/build/openmp/release/runtime/test/ompt/loadtool/tool_available_search/Output/third_tool.so... 0: Do initialize tool
Success.
Tool was started and is using the OMPT interface.
----- END LOGGING OF TOOL REGISTRATION -----
0: Tool initialized
0: ompt_event_thread_begin
0: control_tool()=-1
0: ompt_event_thread_begin
0: Tool finalized

openmp/runtime/src/ompt-general.cpp

 #define no_tool_present 0
 
 #define OMPT_API_ROUTINE static
 
 #ifndef OMPT_STR_MATCH
 #define OMPT_STR_MATCH(haystack, needle) (!strcasecmp(haystack, needle))
 #endif
 
+// prints for an enabled OMP_TOOL_VERBOSE_INIT.
+// In the future a prefix could be added in the first define, the second define
+// omits the prefix to allow for continued lines. Example: "PREFIX: Start
+// tool... Success." instead of "PREFIX: Start tool... PREFIX: Success."
+#define OMPT_VERBOSE_INIT_PRINT(...)                                           \
+  if (verbose_init)                                                            \
+  fprintf(verbose_file, __VA_ARGS__)
+#define OMPT_VERBOSE_INIT_CONTINUED_PRINT(...)                                 \
+  if (verbose_init)                                                            \
+  fprintf(verbose_file, __VA_ARGS__)
+
+static FILE *verbose_file;
+static int verbose_init;
+
 /*****************************************************************************
  * types
  ****************************************************************************/
 
 typedef struct {
   const char *state_name;
   ompt_state_t state_id;
 } ompt_state_info_t;
ompt_try_start_tool(unsigned int omp_version, const char *runtime_version) {
   ompt_start_tool_t start_tool = NULL;
 #if KMP_OS_WINDOWS
   // Cannot use colon to describe a list of absolute paths on Windows
   const char *sep = ";";
 #else
   const char *sep = ":";
 #endif
 
+  OMPT_VERBOSE_INIT_PRINT("----- START LOGGING OF TOOL REGISTRATION -----\n");
+  OMPT_VERBOSE_INIT_PRINT("Search for OMP tool in current address space... ");
+
 #if KMP_OS_DARWIN
   // Try in the current address space
   ret = ompt_tool_darwin(omp_version, runtime_version);
 #elif OMPT_HAVE_WEAK_ATTRIBUTE
   ret = ompt_start_tool(omp_version, runtime_version);
 #elif OMPT_HAVE_PSAPI
   ret = ompt_tool_windows(omp_version, runtime_version);
 #else
 #error Activation of OMPT is not supported on this platform.
 #endif
-  if (ret)
+  if (ret) {
+    OMPT_VERBOSE_INIT_CONTINUED_PRINT("Sucess.\n");
+    OMPT_VERBOSE_INIT_PRINT(
+        "Tool was started and is using the OMPT interface.\n");
+    OMPT_VERBOSE_INIT_PRINT("----- END LOGGING OF TOOL REGISTRATION -----\n");
     return ret;
+  }
 
   // Try tool-libraries-var ICV
+  OMPT_VERBOSE_INIT_CONTINUED_PRINT("Failed.\n");
+  OMPT_VERBOSE_INIT_PRINT("Searching tool libraries...\n");
   const char *tool_libs = getenv("OMP_TOOL_LIBRARIES");
   if (tool_libs) {
     char *libs = __kmp_str_format("%s", tool_libs);
     char *buf;
     char *fname = __kmp_str_token(libs, sep, &buf);
+    // Reset dl-error
+    dlerror();
+
     while (fname) {
 #if KMP_OS_UNIX
+      OMPT_VERBOSE_INIT_PRINT("Opening %s... ", fname);
       void *h = dlopen(fname, RTLD_LAZY);
-      if (h) {
+      if (!h) {
+        OMPT_VERBOSE_INIT_CONTINUED_PRINT("Failed: %s\n", dlerror());
+      } else {
+        OMPT_VERBOSE_INIT_CONTINUED_PRINT("Success. \n");
+        OMPT_VERBOSE_INIT_PRINT("Searching for ompt_start_tool in %s... ",
+                                fname);
         start_tool = (ompt_start_tool_t)dlsym(h, "ompt_start_tool");
+        if (!start_tool) {
+          OMPT_VERBOSE_INIT_CONTINUED_PRINT("Failed: %s\n", dlerror());
+        } else
 #elif KMP_OS_WINDOWS
+      OMPT_VERBOSE_INIT_PRINT("Opening %s... ", fname);
       HMODULE h = LoadLibrary(fname);
-      if (h) {
+      if (!h) {
+        OMPT_VERBOSE_INIT_CONTINUED_PRINT("Failed: Error %u\n", GetLastError());
+      } else {
+        OMPT_VERBOSE_INIT_CONTINUED_PRINT("Success. \n");
+        OMPT_VERBOSE_INIT_PRINT("Searching for ompt_start_tool in %s... ",
+                                fname);
         start_tool = (ompt_start_tool_t)GetProcAddress(h, "ompt_start_tool");
+        if (!start_tool) {
+          OMPT_VERBOSE_INIT_CONTINUED_PRINT("Failed: Error %s\n",
+                                            GetLastError());
+        } else
 #else
 #error Activation of OMPT is not supported on this platform.
 #endif
-        if (start_tool && (ret = (*start_tool)(omp_version, runtime_version)))
+        {// if (start_tool)
+          ret = (*start_tool)(omp_version, runtime_version);
+          if (ret) {
+            OMPT_VERBOSE_INIT_CONTINUED_PRINT("Success.\n");
+            OMPT_VERBOSE_INIT_PRINT(
+                "Tool was started and is using the OMPT interface.\n");
             break;
           }
+          OMPT_VERBOSE_INIT_CONTINUED_PRINT(
+              "Found but not using the OMPT interface.\n");
+          OMPT_VERBOSE_INIT_PRINT("Continuing search...\n");
+        }
+      }
       fname = __kmp_str_token(NULL, sep, &buf);
     }
     __kmp_str_free(&libs);
   }
-  if (ret)
+
+  // usable tool found in tool-libraries
+  if (ret) {
+    OMPT_VERBOSE_INIT_PRINT("----- END LOGGING OF TOOL REGISTRATION -----\n");
     return ret;
+  }
 
 #if KMP_OS_UNIX
   { // Non-standard: load archer tool if application is built with TSan
     const char *fname = "libarcher.so";
+    OMPT_VERBOSE_INIT_PRINT(
+        "...searching tool libraries failed. Using archer tool.\n");
+    OMPT_VERBOSE_INIT_PRINT("Opening %s... ", fname);
     void *h = dlopen(fname, RTLD_LAZY);
     if (h) {
+      OMPT_VERBOSE_INIT_CONTINUED_PRINT("Success.\n");
+      OMPT_VERBOSE_INIT_PRINT("Searching for ompt_start_tool in %s... ", fname);
       start_tool = (ompt_start_tool_t)dlsym(h, "ompt_start_tool");
-      if (start_tool)
+      if (start_tool) {
         ret = (*start_tool)(omp_version, runtime_version);
-      if (ret)
+        if (ret) {
+          OMPT_VERBOSE_INIT_CONTINUED_PRINT("Success.\n");
+          OMPT_VERBOSE_INIT_PRINT(
+              "Tool was started and is using the OMPT interface.\n");
+          OMPT_VERBOSE_INIT_PRINT(
+              "----- END LOGGING OF TOOL REGISTRATION -----\n");
           return ret;
         }
+        OMPT_VERBOSE_INIT_CONTINUED_PRINT(
+            "Found but not using the OMPT interface.\n");
+      } else {
+        OMPT_VERBOSE_INIT_CONTINUED_PRINT("Failed: %s\n", dlerror());
+      }
+    }
   }
 #endif
+  OMPT_VERBOSE_INIT_PRINT("No OMP tool loaded.\n");
+  OMPT_VERBOSE_INIT_PRINT("----- END LOGGING OF TOOL REGISTRATION -----\n");
   return ret;
 }
 
 void ompt_pre_init() {
   //--------------------------------------------------
   // Execute the pre-initialization logic only once.
   //--------------------------------------------------
   static int ompt_pre_initialized = 0;
void ompt_pre_init() {
 
   if (!ompt_env_var || !strcmp(ompt_env_var, ""))
     tool_setting = omp_tool_unset;
   else if (OMPT_STR_MATCH(ompt_env_var, "disabled"))
     tool_setting = omp_tool_disabled;
   else if (OMPT_STR_MATCH(ompt_env_var, "enabled"))
     tool_setting = omp_tool_enabled;
 
+  const char *ompt_env_verbose_init = getenv("OMP_TOOL_VERBOSE_INIT");
+  // possible options: disabled | stdout | stderr | <filename>
+  // if set, not empty and not disabled -> prepare for logging
+  if (ompt_env_verbose_init && strcmp(ompt_env_verbose_init, "") &&
+      !OMPT_STR_MATCH(ompt_env_verbose_init, "disabled")) {
+    verbose_init = 1;
+    if (OMPT_STR_MATCH(ompt_env_verbose_init, "STDERR"))
+      verbose_file = stderr;
+    else if (OMPT_STR_MATCH(ompt_env_verbose_init, "STDOUT"))
+      verbose_file = stdout;
+    else
+      verbose_file = fopen(ompt_env_verbose_init, "w");
+  } else
+    verbose_init = 0;
+
 #if OMPT_DEBUG
   printf("ompt_pre_init(): tool_setting = %d\n", tool_setting);
 #endif
   switch (tool_setting) {
   case omp_tool_disabled:
+    OMPT_VERBOSE_INIT_PRINT("OMP tool disabled. \n");
     break;
 
   case omp_tool_unset:
   case omp_tool_enabled:
 
     //--------------------------------------------------
     // Load tool iff specified in environment variable
     //--------------------------------------------------
     ompt_start_tool_result =
         ompt_try_start_tool(__kmp_openmp_version, ompt_get_runtime_version());
 
     memset(&ompt_enabled, 0, sizeof(ompt_enabled));
     break;
 
   case omp_tool_error:
     fprintf(stderr, "Warning: OMP_TOOL has invalid value \"%s\".\n"
                     "  legal values are (NULL,\"\",\"disabled\","
                     "\"enabled\").\n",
             ompt_env_var);
     break;
   }
+  if (verbose_init && verbose_file != stderr && verbose_file != stdout)
+    fclose(verbose_file);
 #if OMPT_DEBUG
   printf("ompt_pre_init(): ompt_enabled = %d\n", ompt_enabled);
 #endif
 }
 
 extern "C" int omp_get_initial_device(void);
 
 void ompt_post_init() {

openmp/runtime/test/ompt/loadtool/tool_available/tool_available.c

 // The OpenMP standard defines 3 ways of providing ompt_start_tool:
-// 1. "statically-linking the tool’s definition of ompt_start_tool into an OpenMP application"
-// RUN: %libomp-compile -DCODE -DTOOL && %libomp-run | FileCheck %s
+
+// 1. "statically-linking the tool’s definition of ompt_start_tool into an 
+//     OpenMP application"
+
+// RUN: %libomp-compile -DCODE -DTOOL && env OMP_TOOL_VERBOSE_INIT=stdout \
+// RUN:    %libomp-run | FileCheck %s --check-prefixes CHECK,ADDRSPACE
 
 // Note: We should compile the tool without -fopenmp as other tools developer
-//       would do. Otherwise this test may pass for the wrong reasons on Darwin.
+//      would do. Otherwise this test may pass for the wrong reasons on Darwin.
+
 // RUN: %clang %flags -DTOOL -shared -fPIC %s -o %T/tool.so
-// 2. "introducing a dynamically-linked library that includes the tool’s definition of ompt_start_tool into the application’s address space"
+
+// 2. "introducing a dynamically-linked library that includes the tool’s 
+//    definition of ompt_start_tool into the application’s address space"
+
 // 2.1 Link with tool during compilation
-// RUN: %libomp-compile -DCODE %no-as-needed-flag %T/tool.so && %libomp-run | FileCheck %s
+
+// RUN: %libomp-compile -DCODE %no-as-needed-flag %T/tool.so && \
+// RUN:    env OMP_TOOL_VERBOSE_INIT=stdout %libomp-run | FileCheck %s \
+// RUN:    --check-prefixes CHECK,ADDRSPACE 
+
 // 2.2 Link with tool during compilation, but AFTER the runtime
-// RUN: %libomp-compile -DCODE -lomp %no-as-needed-flag %T/tool.so && %libomp-run | FileCheck %s
+
+// RUN: %libomp-compile -DCODE -lomp %no-as-needed-flag %T/tool.so && \
+// RUN:    env OMP_TOOL_VERBOSE_INIT=stdout %libomp-run | FileCheck %s \
+// RUN:    --check-prefixes CHECK,ADDRSPACE
+
 // 2.3 Inject tool via the dynamic loader
-// RUN: %libomp-compile -DCODE && %preload-tool %libomp-run | FileCheck %s
 
-// 3. "providing the name of a dynamically-linked library appropriate for the architecture and operating system used by the application in the tool-libraries-var ICV"
-// RUN: %libomp-compile -DCODE && env OMP_TOOL_LIBRARIES=%T/tool.so %libomp-run | FileCheck %s
+// RUN: %libomp-compile -DCODE && env OMP_TOOL_VERBOSE_INIT=stdout \
+// RUN:    %preload-tool %libomp-run | FileCheck %s \
+// RUN:    --check-prefixes CHECK,ADDRSPACE
+
+// 3. "providing the name of a dynamically-linked library appropriate for the
+//    architecture and operating system used by the application in the 
+//    tool-libraries-var ICV"
+
+// 3.1 OMP_TOOL_VERBOSE_INIT not set 
+
+// RUN: %libomp-compile -DCODE && \
+// RUN:    env OMP_TOOL_LIBRARIES=%T/tool.so %libomp-run | FileCheck %s
+
+// 3.2 OMP_TOOL_VERBOSE_INIT disabled
+
+// RUN: env OMP_TOOL_LIBRARIES=%T/tool.so OMP_TOOL_VERBOSE_INIT=disabled \
+// RUN:    %libomp-run | FileCheck %s
+
+// 3.3 OMP_TOOL_VERBOSE_INIT to stdout
+
+// RUN: %libomp-compile -DCODE && env OMP_TOOL_LIBRARIES=%T/tool.so \
+// RUN:    OMP_TOOL_VERBOSE_INIT=stdout %libomp-run | \
+// RUN:    FileCheck %s -DPARENTPATH=%T --check-prefixes CHECK,TOOLLIB
+
+// 3.4 OMP_TOOL_VERBOSE_INIT to stderr, check merged stdout and stderr
+
+// RUN: env OMP_TOOL_LIBRARIES=%T/tool.so OMP_TOOL_VERBOSE_INIT=stderr \
+// RUN:    %libomp-run 2>&1 | \
+// RUN:    FileCheck %s -DPARENTPATH=%T --check-prefixes CHECK,TOOLLIB
+
+// 3.5 OMP_TOOL_VERBOSE_INIT to stderr, check just stderr
+
+// RUN: env OMP_TOOL_LIBRARIES=%T/tool.so OMP_TOOL_VERBOSE_INIT=stderr \
+// RUN:    %libomp-run 2>&1 >/dev/null | \
+// RUN:    FileCheck %s -DPARENTPATH=%T --check-prefixes TOOLLIB
+
+// 3.6 OMP_TOOL_VERBOSE_INIT to file "init.log"
+
+// RUN: env OMP_TOOL_LIBRARIES=%T/tool.so OMP_TOOL_VERBOSE_INIT=%T/init.log \
+// RUN:    %libomp-run | FileCheck %s && cat %T/init.log | \
+// RUN:    FileCheck %s -DPARENTPATH=%T --check-prefixes TOOLLIB
+
 
 // REQUIRES: ompt
 
 /*
  *  This file contains code for an OMPT shared library tool to be
  *  loaded and the code for the OpenMP executable.
  *  -DTOOL enables the code for the tool during compilation
  *  -DCODE enables the code for the executable during compilation
  */
 
+// Check if libomp supports the callbacks for this test.
+// CHECK-NOT: {{^}}0: Could not register callback
+
+// ADDRSPACE: ----- START LOGGING OF TOOL REGISTRATION -----
+// ADDRSPACE-NEXT: Search for OMP tool in current address space... Sucess.
+// ADDRSPACE-NEXT: Tool was started and is using the OMPT interface.
+// ADDRSPACE-NEXT: ----- END LOGGING OF TOOL REGISTRATION -----
+
+// TOOLLIB: ----- START LOGGING OF TOOL REGISTRATION -----
+// TOOLLIB-NEXT: Search for OMP tool in current address space... Failed.
+// TOOLLIB-NEXT: Searching tool libraries...
+// TOOLLIB-NEXT: Opening [[PARENTPATH]]/tool.so... Success.
+// TOOLLIB-NEXT: Searching for ompt_start_tool in
+// TOOLLIB-SAME: [[PARENTPATH]]/tool.so... Success.
+// TOOLLIB-NEXT: Tool was started and is using the OMPT interface.
+// TOOLLIB-NEXT: ----- END LOGGING OF TOOL REGISTRATION -----
+
 #ifdef CODE
 #include "omp.h"
 
 int main()
 {
   #pragma omp parallel num_threads(2)
   {
   }
 
-
-  // Check if libomp supports the callbacks for this test.
-  // CHECK-NOT: {{^}}0: Could not register callback
+  // CHECK-NOT: ----- START LOGGING OF TOOL REGISTRATION -----
+  // CHECK-NOT: ----- END LOGGING OF TOOL REGISTRATION -----
 
   // CHECK: {{^}}0: NULL_POINTER=[[NULL:.*$]]
   // CHECK: {{^}}0: ompt_event_runtime_shutdown
 
   return 0;
 }
 
 #endif /* CODE */

openmp/runtime/test/ompt/loadtool/tool_available_search/tool_available_search.c

 // RUN: %clang %flags -shared -fPIC %s -o %T/first_tool.so
 // RUN: %clang %flags -DTOOL -DSECOND_TOOL -shared -fPIC %s -o %T/second_tool.so
 // RUN: %clang %flags -DTOOL -DTHIRD_TOOL -shared -fPIC %s -o %T/third_tool.so
-// RUN: %libomp-compile -DCODE && env OMP_TOOL_LIBRARIES=%T/non_existing_file.so:%T/first_tool.so:%T/second_tool.so:%T/third_tool.so %libomp-run | FileCheck %s
+// RUN: %libomp-compile -DCODE
+// RUN: env OMP_TOOL_LIBRARIES=%T/non_existing_file.so:%T/first_tool.so:%T/second_tool.so:%T/third_tool.so \
+// RUN: OMP_TOOL_VERBOSE_INIT=stdout %libomp-run | FileCheck %s -DPARENTPATH=%T
 
 // REQUIRES: ompt
 
 /*
  *  This file contains code for three OMPT shared library tool to be 
  *  loaded and the code for the OpenMP executable. 
  *  No option enables code for the first shared library 
  *  (without an implementation of ompt_start_tool) during compilation
  *  -DTOOL -DSECOND_TOOL enables the code for the second tool during compilation
  *  -DTOOL -DTHIRD_TOOL enables the code for the third tool during compilation
  *  -DCODE enables the code for the executable during compilation
  */
 
+// CHECK: ----- START LOGGING OF TOOL REGISTRATION -----
+// CHECK-NEXT: Search for OMP tool in current address space... Failed.
+// CHECK-NEXT: Searching tool libraries...
+// CHECK-NEXT: Opening [[PARENTPATH]]/non_existing_file.so... Failed:
+// CHECK-SAME: [[PARENTPATH]]/non_existing_file.so: cannot open shared object
+// CHECK-SAME: file: No such file or directory
+// CHECK-NEXT: Opening [[PARENTPATH]]/first_tool.so... Success.
+// CHECK-NEXT: Searching for ompt_start_tool in
+// CHECK-SAME: [[PARENTPATH]]/first_tool.so... Failed:
+// CHECK-SAME: [[PARENTPATH]]/first_tool.so: undefined symbol: ompt_start_tool
+// CHECK-NEXT: Opening [[PARENTPATH]]/second_tool.so... Success.
+// CHECK-NEXT: Searching for ompt_start_tool in
+// CHECK-SAME: [[PARENTPATH]]/second_tool.so... 0: Do not initialize tool
+// CHECK-NEXT: Found but not using the OMPT interface.
+// CHECK-NEXT: Continuing search...
+// CHECK-NEXT: Opening [[PARENTPATH]]/third_tool.so... Success.
+// CHECK-NEXT: Searching for ompt_start_tool in
+// CHECK-SAME: [[PARENTPATH]]/third_tool.so... 0: Do initialize tool
+// CHECK-NEXT: Success.
+// CHECK-NEXT: Tool was started and is using the OMPT interface.
+// CHECK-NEXT: ----- END LOGGING OF TOOL REGISTRATION -----
+
+// Check if libomp supports the callbacks for this test.
+
+// CHECK-NOT: {{^}}0: Could not register callback 
+// CHECK: {{^}}0: Tool initialized
+// CHECK: {{^}}0: ompt_event_thread_begin
+// CHECK-DAG: {{^}}0: ompt_event_thread_begin
+// CHECK-DAG: {{^}}0: control_tool()=-1
+// CHECK: {{^}}0: Tool finalized
+
+
 #ifdef CODE
 #include "stdio.h"
 #include "omp.h"
 #include "omp-tools.h"
 
 int main()
 {
   #pragma omp parallel num_threads(2)
   {
     #pragma omp master
     {
       int result = omp_control_tool(omp_control_tool_start, 0, NULL);
       printf("0: control_tool()=%d\n", result);
     }
   }
 
 
-  // Check if libomp supports the callbacks for this test.
-  // CHECK-NOT: {{^}}0: Could not register callback 
-  
-  // CHECK: {{^}}0: Do not initialize tool
-
-  // CHECK: {{^}}0: Do initialize tool
-  // CHECK: {{^}}0: Tool initialized
-  // CHECK: {{^}}0: ompt_event_thread_begin
-  // CHECK-DAG: {{^}}0: ompt_event_thread_begin
-  // CHECK-DAG: {{^}}0: control_tool()=-1
-  // CHECK: {{^}}0: Tool finalized
-  
-
   return 0;
 }
 
 #endif /* CODE */
 
 #ifdef TOOL
 
 #include <omp-tools.h>

openmp/runtime/test/ompt/loadtool/tool_not_available/tool_not_available.c

 // The OpenMP standard defines 3 ways of providing ompt_start_tool:
-// 1. "statically-linking the tool’s definition of ompt_start_tool into an OpenMP application"
-// RUN: %libomp-compile -DCODE -DTOOL && %libomp-run | FileCheck %s
+
+// 1. "statically-linking the tool’s definition of ompt_start_tool into an
+// OpenMP application"
+
+// RUN: %libomp-compile -DCODE -DTOOL && \
+// RUN:    env OMP_TOOL_VERBOSE_INIT=stdout %libomp-run | \
+// RUN:    FileCheck %s --check-prefixes CHECK,ADDRSPACE 
 
 // Note: We should compile the tool without -fopenmp as other tools developer
-//       would do. Otherwise this test may pass for the wrong reasons on Darwin.
+//      would do. Otherwise this test may pass for the wrong reasons on Darwin.
+
 // RUN: %clang %flags -DTOOL -shared -fPIC %s -o %T/tool.so
-// 2. "introducing a dynamically-linked library that includes the tool’s definition of ompt_start_tool into the application’s address space"
+
+// 2. "introducing a dynamically-linked library that includes the tool’s 
+//    definition of ompt_start_tool into the application’s address space"
+
 // 2.1 Link with tool during compilation
-// RUN: %libomp-compile -DCODE %no-as-needed-flag %T/tool.so && %libomp-run | FileCheck %s
+
+// RUN: %libomp-compile -DCODE %no-as-needed-flag %T/tool.so && \
+// RUN:    env OMP_TOOL_VERBOSE_INIT=stdout %libomp-run | \
+// RUN:    FileCheck %s --check-prefixes CHECK,ADDRSPACE
+
 // 2.2 Link with tool during compilation, but AFTER the runtime
-// RUN: %libomp-compile -DCODE -lomp %no-as-needed-flag %T/tool.so && %libomp-run | FileCheck %s
+
+// RUN: %libomp-compile -DCODE -lomp %no-as-needed-flag %T/tool.so && \
+// RUN:    env OMP_TOOL_VERBOSE_INIT=stdout %libomp-run | \
+// RUN:    FileCheck %s --check-prefixes CHECK,ADDRSPACE 
+
 // 2.3 Inject tool via the dynamic loader
-// RUN: %libomp-compile -DCODE && %preload-tool %libomp-run | FileCheck %s
 
-// 3. "providing the name of a dynamically-linked library appropriate for the architecture and operating system used by the application in the tool-libraries-var ICV"
-// RUN: %libomp-compile -DCODE && env OMP_TOOL_LIBRARIES=%T/tool.so %libomp-run | FileCheck %s
+// RUN: %libomp-compile -DCODE && \
+// RUN:    env OMP_TOOL_VERBOSE_INIT=stdout %preload-tool %libomp-run | \
+// RUN:    FileCheck %s --check-prefixes CHECK,ADDRSPACE 
+
+// 3. "providing the name of a dynamically-linked library appropriate for the
+//    architecture and operating system used by the application in the 
+//    tool-libraries-var ICV"
+
+// RUN: %libomp-compile -DCODE && env OMP_TOOL_LIBRARIES=%T/tool.so \
+// RUN:    OMP_TOOL_VERBOSE_INIT=stdout %libomp-run | \
+// RUN:    FileCheck %s -DPARENTPATH=%T --check-prefixes CHECK,TOOLLIB
 
 // REQUIRES: ompt
 
 /*
  *  This file contains code for an OMPT shared library tool to be 
  *  loaded and the code for the OpenMP executable. 
  *  -DTOOL enables the code for the tool during compilation
  *  -DCODE enables the code for the executable during compilation
#pragma omp parallel num_threads(2)
     {
       int result = omp_control_tool(omp_control_tool_start, 0, NULL);
       printf("0: control_tool()=%d\n", result);
     }
   }
 
 
   // Check if libomp supports the callbacks for this test.
-  // CHECK-NOT: {{^}}0: Could not register callback 
-  
-  // CHECK: {{^}}0: Do not initialize tool
+  // CHECK-NOT: {{^}}0: Could not register callback
+  // ADDRSPACE: ----- START LOGGING OF TOOL REGISTRATION -----
+  // ADDRSPACE-NEXT: Search for OMP tool in current address space...
+
+  // TOOLLIB: ----- START LOGGING OF TOOL REGISTRATION -----
+  // TOOLLIB-NEXT: Search for OMP tool in current address space... Failed.
+  // TOOLLIB-NEXT: Searching tool libraries...
+  // TOOLLIB-NEXT: Opening [[PARENTPATH]]/tool.so... Success.
+  // TOOLLIB-NEXT: Searching for ompt_start_tool in
+  // TOOLLIB-SAME: [[PARENTPATH]]/tool.so...
+
+  // CHECK: 0: Do not initialize tool
+
+  // ADDRSPACE-NEXT: Failed.
+  // ADDRSPACE-NEXT: Searching tool libraries...
+  // ADDRSPACE-NEXT: ...searching tool libraries failed.
+  // ADDRSPACE: No OMP tool loaded.
+  // ADDRSPACE-NEXT: ----- END LOGGING OF TOOL REGISTRATION -----
+
+  // TOOLLIB-NEXT: Found but not using the OMPT interface.
+  // TOOLLIB-NEXT: Continuing search...
+  // TOOLLIB-NEXT: ...searching tool libraries failed.
+  // TOOLLIB: No OMP tool loaded.
+  // TOOLLIB-NEXT: ----- END LOGGING OF TOOL REGISTRATION -----
+
   // CHECK: {{^}}0: control_tool()=-2
   
 
   return 0;
 }
 
 #endif /* CODE */
 

openmp/tools/archer/tests/parallel/parallel-simple.c

 /*
  * parallel-simple.c -- Archer testcase
  */
 
 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 //
 // See tools/archer/LICENSE.txt for details.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 
-// RUN: %libarcher-compile-and-run | FileCheck %s
+// RUN: %libarcher-compile && env OMP_TOOL_VERBOSE_INIT=stderr %libarcher-run 2>&1 | FileCheck %s --check-prefixes CHECK,OMPT_ON
+// RUN: %clang-archer %openmp_flags %flags %s -o %t && env OMP_TOOL_VERBOSE_INIT=stderr %t 2>&1 | FileCheck %s --check-prefixes CHECK,OMPT_OFF
 // REQUIRES: tsan
 #include <omp.h>
 #include <stdio.h>
 
+// OMPT_ON: ----- START LOGGING OF TOOL REGISTRATION -----
+// OMPT_ON-NEXT: Search for OMP tool in current address space... Failed.
+// OMPT_ON-NEXT: Searching tool libraries...
+// OMPT_ON-NEXT: ...searching tool libraries failed. Using archer tool.
+// OMPT_ON-NEXT: Opening libarcher.so... Success.
+// OMPT_ON-NEXT: Searching for ompt_start_tool in libarcher.so... Success.
+// OMPT_ON-NEXT: Tool was started and is using the OMPT interface.
+// OMPT_ON-NEXT: ----- END LOGGING OF TOOL REGISTRATION -----
+
+// OMPT_OFF: ----- START LOGGING OF TOOL REGISTRATION -----
+// OMPT_OFF-NEXT: Search for OMP tool in current address space... Failed.
+// OMPT_OFF-NEXT: Searching tool libraries...
+// OMPT_OFF-NEXT: ...searching tool libraries failed. Using archer tool.
+// OMPT_OFF-NEXT: Opening libarcher.so... Success.
+// OMPT_OFF-NEXT: Searching for ompt_start_tool in libarcher.so... Found but not using the OMPT interface.
+// OMPT_OFF-NEXT: No OMP tool loaded.
+// OMPT_OFF-NEXT: ----- END LOGGING OF TOOL REGISTRATION -----
+
+
 int main(int argc, char *argv[]) {
   int var = 0;
 
 #pragma omp parallel num_threads(2) shared(var)
   {
     if (omp_get_thread_num() == 1) {
       var++;
     }