[OMPT] Provide initialization for Mac OS X

[Hahnfeld]

Summary

Traditionally, the library had a weak symbol for ompt_start_tool()
that served as fallback and disabled OMPT if called. Tools could
provide their own version and replace the default implementation
to register callbacks and lookup functions. This mechanism has
worked reasonably well on Linux systems where this interface was
initially developed.

On Darwin / Mac OS X the situation is a bit more complicated and
the weak symbol doesn't work out-of-the-box. In my tests, the
library with the tool needed to link against the OpenMP runtime
to make the process work. This would effectively mean that a tool
needed to choose a runtime library whereas one design goal of the
interface was to allow tools that are agnostic of the runtime.

The solution is to use dlsym() with the argument RTLD_DEFAULT so
that static implementations of ompt_start_tool() are found in the
main executable. This works because the linker on Mac OS X includes
all symbols of an executable in the global symbol table by default.
To use the same code path on Linux, the application would need to
be built with -Wl,--export-dynamic. To avoid this restriction, we
continue to use weak symbols on Linux systems as before.

Finally this patch extends the existing test to cover all possible
ways of initializing the tool as described by the standard. It
also fixes ompt_finalize() to not call omp_get_thread_num() when
the library is shut down which resulted in hangs on Darwin.
The changes have been tested on Linux to make sure that it passes
the current tests as well as the newly extended one.

[omalyshe]

runtime/cmake/config-ix.cmake

241

WIN32 AND LIBOMP_HAVE_PSAPI

[Hahnfeld]

Fix typo in config-ix.cmake

runtime/cmake/config-ix.cmake

241

Oops, right you are. Thanks CMake for telling me...

[hbae]

runtime/src/ompt-general.cpp

124

I think this code now requires LD_DYNAMIC_WEAK to be set in order to override weak symbols (e.g., when libomp.so is linked before tool library).
We used RTLD_NEXT to avoid this, by making the fallback ompt_start_tool search for the next available symbol.

[Hahnfeld]

Readd dlsym() to cover cases where the runtime is linked before the tool.

runtime/src/ompt-general.cpp

124

Thanks, that was the test case and explanation that I was missing. I have readded the code with a comment describing the situation and added a test case for this.

Note that KMP_DYNAMIC_LIB is always true and we require OMPT_HAVE_WEAK_ATTRIBUTE for Unix systems.

[hbae]

LGTM.

[protze.joachim]

LGTM

We might want to add another RUN-line for 3, where the first tool returns NULL for ompt_start_tool and only the second tool in the list is initialized.
We can add this test in a later patch

[Hahnfeld]

In D39801#921884, @protze.joachim wrote:

We might want to add another RUN-line for 3, where the first tool returns NULL for ompt_start_tool and only the second tool in the list is initialized.
We can add this test in a later patch

And another tool that doesn't have ompt_start_tool at all. I will do that (in another patch) because I think that these libraries should be dlclosed if not found suitable.

runtime/cmake/config-ix.cmake

if(LIBOMP_HAVE___THREAD AND (LIBOMP_HAVE___RDTSC OR LIBOMP_HAVE___BUILTIN_READCYCLECOUNTER))
     set(LIBOMP_HAVE_STATS TRUE)
   else()
     set(LIBOMP_HAVE_STATS FALSE)
   endif()
 endif()
 
 # Check if OMPT support is available
 # Currently, __builtin_frame_address() is required for OMPT
-# Weak attribute is required for Unices, LIBPSAPI is used for Windows
+# Weak attribute is required for Unices (except Darwin), LIBPSAPI is used for Windows
 check_c_source_compiles("int main(int argc, char** argv) {
   void* p = __builtin_frame_address(0);
   return 0;}" LIBOMP_HAVE___BUILTIN_FRAME_ADDRESS)
 check_c_source_compiles("__attribute__ ((weak)) int foo(int a) { return a*a; }
   int main(int argc, char** argv) {
   return foo(argc);}" LIBOMP_HAVE_WEAK_ATTRIBUTE)
 check_include_files("windows.h;psapi.h" LIBOMP_HAVE_PSAPI_H)
 check_library_exists(psapi EnumProcessModules "" LIBOMP_HAVE_LIBPSAPI)
 if(LIBOMP_HAVE_PSAPI_H AND LIBOMP_HAVE_LIBPSAPI)
   set(LIBOMP_HAVE_PSAPI TRUE)
 endif()
 if(NOT LIBOMP_HAVE___BUILTIN_FRAME_ADDRESS)
   set(LIBOMP_HAVE_OMPT_SUPPORT FALSE)
 else()
-  if(LIBOMP_HAVE_WEAK_ATTRIBUTE OR LIBOMP_HAVE_PSAPI)
+  if((WIN32 AND LIBOMP_HAVE_PSAPI) OR APPLE OR (NOT WIN32 AND LIBOMP_HAVE_WEAK_ATTRIBUTE))
     set(LIBOMP_HAVE_OMPT_SUPPORT TRUE)
   else()
     set(LIBOMP_HAVE_OMPT_SUPPORT FALSE)
   endif()
 endif()
 
 # Check if HWLOC support is available
 if(${LIBOMP_USE_HWLOC})

runtime/src/ompt-general.cpp

 static ompt_interface_fn_t ompt_fn_lookup(const char *s);
 
 OMPT_API_ROUTINE ompt_data_t *ompt_get_thread_data(void);
 
 /*****************************************************************************
  * initialization and finalization (private operations)
  ****************************************************************************/
 
-/* On Unix-like systems that support weak symbols the following implementation
- * of ompt_start_tool() will be used in case no tool-supplied implementation of
- * this function is present in the address space of a process.
- *
- * On Windows, the ompt_tool_windows function is used to find the
- * ompt_tool symbol across all modules loaded by a process. If ompt_tool is
- * found, ompt_tool's return value is used to initialize the tool. Otherwise,
- * NULL is returned and OMPT won't be enabled */
-
 typedef ompt_start_tool_result_t *(*ompt_start_tool_t)(unsigned int,
                                                        const char *);
 
-#if KMP_OS_UNIX
+#if KMP_OS_DARWIN
 
-#if OMPT_HAVE_WEAK_ATTRIBUTE
-_OMP_EXTERN OMPT_WEAK_ATTRIBUTE
-#elif defined KMP_DYNAMIC_LIB
-_OMP_EXTERN
-#warning Activation of OMPT is might fail for tools statically linked into the application.
-#else
-#error Activation of OMPT is not supported on this platform.
-#endif
-ompt_start_tool_result_t *
-ompt_start_tool(unsigned int omp_version, const char *runtime_version) {
-#ifdef KMP_DYNAMIC_LIB
+// While Darwin supports weak symbols, the library that wishes to provide a new
+// implementation has to link against this runtime which defeats the purpose
+// of having tools that are agnostic of the underlying runtime implementation.
+//
+// Fortunately, the linker includes all symbols of an executable in the global
+// symbol table by default so dlsym() even finds static implementations of
+// ompt_start_tool. For this to work on Linux, -Wl,--export-dynamic needs to be
+// passed when building the application which we don't want to rely on.
+
+static ompt_start_tool_result_t *ompt_tool_darwin(unsigned int omp_version,
+                                                  const char *runtime_version) {
   ompt_start_tool_result_t *ret = NULL;
-  // Try next symbol in the address space
-  ompt_start_tool_t next_tool =
-      (ompt_start_tool_t)dlsym(RTLD_NEXT, "ompt_start_tool");
-  if (next_tool)
-    ret = (next_tool)(omp_version, runtime_version);
+  // Search symbol in the current address space.
+  ompt_start_tool_t start_tool =
+      (ompt_start_tool_t)dlsym(RTLD_DEFAULT, "ompt_start_tool");
+  if (start_tool) {
+    ret = start_tool(omp_version, runtime_version);
+  }
   return ret;
-#else
+}
+
+#elif OMPT_HAVE_WEAK_ATTRIBUTE
+
+// On Unix-like systems that support weak symbols the following implementation
+// of ompt_start_tool() will be used in case no tool-supplied implementation of
+// this function is present in the address space of a process.
+
+_OMP_EXTERN OMPT_WEAK_ATTRIBUTE ompt_start_tool_result_t *
+ompt_start_tool(unsigned int omp_version, const char *runtime_version) {
 #if OMPT_DEBUG
   printf("ompt_start_tool() is called from the RTL\n");
 #endif
   return NULL;
-#endif
 }
 
 #elif OMPT_HAVE_PSAPI
 
+// On Windows, the ompt_tool_windows function is used to find the
+// ompt_start_tool symbol across all modules loaded by a process. If
+// ompt_start_tool is found, ompt_start_tool's return value is used to
+// initialize the tool. Otherwise, NULL is returned and OMPT won't be enabled.
+
 #include <psapi.h>
 #pragma comment(lib, "psapi.lib")
-#define ompt_start_tool ompt_tool_windows
 
 // The number of loaded modules to start enumeration with EnumProcessModules()
 #define NUM_MODULES 128
 
 static ompt_start_tool_result_t *
 ompt_tool_windows(unsigned int omp_version, const char *runtime_version) {
   int i;
   DWORD needed, new_size;
else {
                modName);
     }
 #endif
   }
   free(modules);
   return NULL;
 }
 #else
-#error Either __attribute__((weak)) or psapi.dll are required for OMPT support
-#endif // OMPT_HAVE_WEAK_ATTRIBUTE
+#error Activation of OMPT is not supported on this platform.
+#endif
 
 static ompt_start_tool_result_t *
 ompt_try_start_tool(unsigned int omp_version, const char *runtime_version) {
   ompt_start_tool_result_t *ret = NULL;
   ompt_start_tool_t start_tool = NULL;
 #if KMP_OS_WINDOWS
   // Cannot use colon to describe a list of absolute paths on Windows
   const char *sep = ";";
 #else
   const char *sep = ":";
 #endif
 
   // Try in the current address space
-  if ((ret = ompt_start_tool(omp_version, runtime_version)))
+#if KMP_OS_DARWIN
+  ret = ompt_tool_darwin(omp_version, runtime_version);
+#elif OMPT_HAVE_WEAK_ATTRIBUTE
+  ret = ompt_start_tool(omp_version, runtime_version);
+#elif OMPT_HAVE_PSAPI
+  ret = ompt_tool_windows(omp_version, runtime_version);
+#else
+#error Activation of OMPT is not supported on this platform.
+#endif
+  if (ret)
     return ret;
 
   // Try tool-libraries-var ICV
   const char *tool_libs = getenv("OMP_TOOL_LIBRARIES");
   if (tool_libs) {
     char *libs = __kmp_str_format("%s", tool_libs);
     char *buf;
     char *fname = __kmp_str_token(libs, sep, &buf);

runtime/test/lit.cfg

 
 # test_exec_root: The root object directory where output is placed
 config.test_exec_root = config.libomp_obj_root
 
 # test format
 config.test_format = lit.formats.ShTest()
 
 # compiler flags
-config.test_cflags = config.test_openmp_flag + \
-    " -I " + config.test_source_root + \
+config.test_cflags = " -I " + config.test_source_root + \
     " -I " + config.omp_header_directory + \
     " -L " + config.library_dir + \
     " " + config.test_extra_cflags
 
 # extra libraries
 libs = ""
 if config.has_libm:
     libs += " -lm"

 
 
 # substitutions
 config.substitutions.append(("%libomp-compile-and-run", \
     "%libomp-compile && %libomp-run"))
 config.substitutions.append(("%libomp-cxx-compile-and-run", \
     "%libomp-cxx-compile && %libomp-run"))
 config.substitutions.append(("%libomp-cxx-compile", \
-    "%clangXX %cflags -std=c++11 %s -o %t" + libs))
+    "%clangXX %openmp_flag %cflags -std=c++11 %s -o %t" + libs))
 config.substitutions.append(("%libomp-compile", \
-    "%clang %cflags %s -o %t" + libs))
-config.substitutions.append(("%libomp-tool", \
-    "%clang %cflags -shared -fPIC -o %T/tool.so" + libs))
+    "%clang %openmp_flag %cflags %s -o %t" + libs))
 config.substitutions.append(("%libomp-run", "%t"))
 config.substitutions.append(("%clangXX", config.test_cxx_compiler))
 config.substitutions.append(("%clang", config.test_compiler))
 config.substitutions.append(("%openmp_flag", config.test_openmp_flag))
 config.substitutions.append(("%cflags", config.test_cflags))
 
 if config.has_ompt:
     config.substitutions.append(("FileCheck", config.test_filecheck))
     config.substitutions.append(("%sort-threads", "sort --numeric-sort --stable"))
+    if config.operating_system == 'Windows':
+        # No such environment variable on Windows.
+        config.substitutions.append(("%preload-tool", "true ||"))
+    elif config.operating_system == 'Darwin':
+        config.substitutions.append(("%preload-tool", "env DYLD_INSERT_LIBRARIES=%T/tool.so"))
+    else:
+        config.substitutions.append(("%preload-tool", "env LD_PRELOAD=%T/tool.so"))

runtime/test/ompt/loadtool/tool_available.c

-// RUN: %libomp-compile -DCODE && %libomp-compile -DTOOL -o%T/tool.so -shared -fPIC && env OMP_TOOL_LIBRARIES=%T/tool.so %libomp-run | FileCheck %s
+// The OpenMP standard defines 3 ways of providing ompt_start_tool:
+// 1. "statically-linking the tool’s definition of ompt_start_tool into an OpenMP application"
+// RUN: %libomp-compile -DCODE -DTOOL && %libomp-run | FileCheck %s
+
+// Note: We should compile the tool without -fopenmp as other tools developer
+//       would do. Otherwise this test may pass for the wrong reasons on Darwin.
+// RUN: %clang %cflags -DTOOL -shared -fPIC %s -o %T/tool.so
+// 2. "introducing a dynamically-linked library that includes the tool’s definition of ompt_start_tool into the application’s address space"
+// 2.1 Link with tool during compilation
+// RUN: %libomp-compile -DCODE %T/tool.so && %libomp-run | FileCheck %s
+// 2.2 Inject tool via the dynamic loader
+// RUN: %libomp-compile -DCODE && %preload-tool %libomp-run | FileCheck %s
+
+// 3. "providing the name of a dynamically-linked library appropriate for the architecture and operating system used by the application in the tool-libraries-var ICV"
+// RUN: %libomp-compile -DCODE && env OMP_TOOL_LIBRARIES=%T/tool.so %libomp-run | FileCheck %s
+
 // REQUIRES: ompt
 
 /*
- *  This file contains code for an OMPT shared library tool to be 
- *  loaded and the code for the OpenMP executable. 
+ *  This file contains code for an OMPT shared library tool to be
+ *  loaded and the code for the OpenMP executable.
  *  -DTOOL enables the code for the tool during compilation
  *  -DCODE enables the code for the executable during compilation
- *  The RUN line compiles the two binaries and then tries to load
- *  the tool using the OMP_TOOL_LIBRARIES environmental variable.
  */
 
 #ifdef CODE
 #include "omp.h"
 
 int main()
 {
   #pragma omp parallel num_threads(2)
   {
   }
 
 
   // Check if libomp supports the callbacks for this test.
-  // CHECK-NOT: {{^}}0: Could not register callback 
-  
+  // CHECK-NOT: {{^}}0: Could not register callback
   // CHECK: {{^}}0: NULL_POINTER=[[NULL:.*$]]
   // CHECK: {{^}}0: ompt_event_runtime_shutdown
 
   return 0;
 }
 
 #endif /* CODE */
 
 #ifdef TOOL
 
 #include <stdio.h>
 #include <ompt.h>
 
 int ompt_initialize(
   ompt_function_lookup_t lookup,
   ompt_data_t* tool_data)
 {
   printf("0: NULL_POINTER=%p\n", (void*)NULL);
   return 1; //success
 }
 
 void ompt_finalize(ompt_data_t* tool_data)
 {
-  printf("%d: ompt_event_runtime_shutdown\n", omp_get_thread_num());
+  printf("0: ompt_event_runtime_shutdown\n");
 }
 
 ompt_start_tool_result_t* ompt_start_tool(
   unsigned int omp_version,
   const char *runtime_version)
 {
   static ompt_start_tool_result_t ompt_start_tool_result = {&ompt_initialize,&ompt_finalize, 0};
   return &ompt_start_tool_result;
 }
 #endif /* TOOL */