[OpenMP] Consolidate error handling and debug messages in Libomptarget

[jhuber6]

Summary

This patch consolidates the error handling and messaging routines to a single file Debug.h. The goal is to simplify the error handling interface prior to adding more error handling support.

[jhuber6]

First iteration. There's a few bugs I need to work out and I want to make sure I'm doing the correct approach.

openmp/libomptarget/include/omptargetmessage.h

1 ↗

(On Diff #288670)

Is this the right location for this file? As I understand this will be visible to the rest of Clang.

openmp/libomptarget/plugins/amdgpu/src/rtl.cpp

105

I elected to just have a definition that's used for the debug message.

openmp/libomptarget/plugins/cuda/src/rtl.cpp

25–26

This is the source of a really bizarre error. This used to be static but since the symbol DebugLevel is extern in omptargetmessage.h it cannot be static here. If you print out debug messages, after Libomptarget has been unregistered you'll get some errors from CUDA saying that an error returned from cuCtxSetCurrent about three times. If DebugLevel is declared static the cuda error string will be printed out as (null). If it's not static like here it'll cause a segmentation fault. Any ideas? I feel like these errors shouldn't be emitted in the first place because they occur after Libomptarget has been unregistered.

Target CUDA RTL --> Error returned from cuCtxSetCurrent
Target CUDA RTL --> Segmentation fault (core dumped)

[jdoerfert]

I have a lot of comments that do not directly relate to your changes but basically the code before. Anyway, while we are moving stuff we can clean it up as well.

openmp/libomptarget/include/omptargetmessage.h

1 ↗

(On Diff #288670)

I dislike the omptarget prefix on headers. This is in the libomptarget subfolder, so that is redundant. I'm aware the other file has such a name, that one is bad too ;)

I would suggest something like:

• Debug.h
• Output.h
• DebugInfo.h

but anything like this is better IMHO.

18 ↗

(On Diff #288670)

We should add more documentation here. Also, what is "each RTL"?

24 ↗

(On Diff #288670)

I would image the above macro is only needed for this include.
undef it after and introduce some newlines to make that clear.
If it is not only for the include, never mind.

51 ↗

(On Diff #288670)

I'd remove this comment, move the one above into the doxygen file comment at the top of the file.

113 ↗

(On Diff #288670)

Already above.

[jhuber6]

Making changes

[JonChesterfield]

Nice! Thank you, there is way too much copy and paste between these files.

The global DebugLevel/InfoLevel are nasty, but it'll be easy to introduce bugs while changing this. Keeping the non-functional code changes separate from making things better at runtime is a good strategy.

The amdgpu/impl directory contains more ad-hoc debug printing, but that should probably be incrementally deleted instead of updated to match this new scheme.

[jhuber6]

openmp/libomptarget/include/Debug.h

44

Having these as global variables is irritating. I could replace it with a function called getDebugLevel that returns the value of the environment variable and use that in the debug macros so it wouldn't need to be called manually in the files. Could potentially store a cached result so it doesn't call getenv every time.

openmp/libomptarget/plugins/cuda/src/rtl.cpp

38–39

I'm still not sure why having DebugLevel not static causes cuGetErrorString to segfault instead of return (null).

openmp/libomptarget/plugins/ve/src/rtl.cpp

33–35

Is this supposed to be called VE?

[jhuber6]

openmp/libomptarget/include/Debug.h

44

Is something like this an acceptable solution? It fixes the weird problems I was getting in the CUDA RTL without a static variable. The variables are still global unless I want to add a source file.

static int DebugLevel = -1; 
static int InfoLevel = -1; 

static int getInfoLevel() {
  if (InfoLevel >= 0)
    return InfoLevel;

  if (char *EnvStr = getenv("LIBOMPTARGET_INFO"))
    InfoLevel = std::stoi(EnvStr);

  return InfoLevel;
}

static int getDebugLevel() {
  if (DebugLevel >= 0)
    return DebugLevel;

  if (char *EnvStr = getenv("LIBOMPTARGET_DEBUG"))
    DebugLevel = std::stoi(EnvStr);

  return DebugLevel;
}

[jdoerfert]

openmp/libomptarget/include/Debug.h

44

if u do this, move the static declaration into the functions to make them private to the function.

[jhuber6]

Changed global debugLevel variable to a function that returns the debugLevel. Initializes once and removed the need for the RTL to check the environment variable manually as well as keeps the variables out of global scope.

[ye-luo]

I don't feel right having Debug.h shared by libomptarget and plugins especially when Debug.h is not just macro but also functions.

[jdoerfert]

In D86804#2247599, @ye-luo wrote:

I don't feel right having Debug.h shared by libomptarget and plugins especially when Debug.h is not just macro but also functions.

Could you elaborate on this please. What issues are better solved by code duplication?

[jhuber6]

In D86804#2247599, @ye-luo wrote:

I don't feel right having Debug.h shared by libomptarget and plugins especially when Debug.h is not just macro but also functions.

omptarget.h already shared the DEBUGP with libomptarget and the plugins, this just moved the macros redefined in each plugin to a common header. The functions are static and inline so I would think they wouldn't cause any issues because they don't extend beyond the file.

[ye-luo]

It seems that functions are marked static so they should be OK. However, including the whole Debug.h in a plugin cpp makes it feel OK to use any function/macro from the header file. But actually only part of the macros are for the plugin. some are only for the libomptarget.

[jdoerfert]

In D86804#2247697, @ye-luo wrote:

It seems that functions are marked static so they should be OK. However, including the whole Debug.h in a plugin cpp makes it feel OK to use any function/macro from the header file. But actually only part of the macros are for the plugin. some are only for the libomptarget.

I'm not sure we want to make a distinction, the point is to move to a unified debug/message model. You can choose not only the level of information but also the kind of output (text, json, ...). The messages will then be tied to the webpage via enums, that allows all plugins to emit the same message for the same thing with the same link to more information. There will certainly things that are only used in libomptarget or the plugins, but I don't see how that is any worse than duplicating the parts that are used by both.

[ye-luo]

In D86804#2247785, @jdoerfert wrote:

In D86804#2247697, @ye-luo wrote:

It seems that functions are marked static so they should be OK. However, including the whole Debug.h in a plugin cpp makes it feel OK to use any function/macro from the header file. But actually only part of the macros are for the plugin. some are only for the libomptarget.

I'm not sure we want to make a distinction, the point is to move to a unified debug/message model. You can choose not only the level of information but also the kind of output (text, json, ...). The messages will then be tied to the webpage via enums, that allows all plugins to emit the same message for the same thing with the same link to more information. There will certainly things that are only used in libomptarget or the plugins, but I don't see how that is any worse than duplicating the parts that are used by both.

I didn't mean to duplicate anything. Instead you need multiple header files. One for common parts, one for libomptarget and one for plugins. The latter two both include the first one. Later you expand OFFLOAD_XXX signals, they can be added to the common file. The return signal is generated by the plugins and captured by the libomptarget. Some users may want to see only the messages captured universally by libomptarget. Some users still want to see the native error message. So the libomptarget and plugin side error handling still needs to be separated.

[jdoerfert]

In D86804#2248201, @ye-luo wrote:

In D86804#2247785, @jdoerfert wrote:

In D86804#2247697, @ye-luo wrote:

It seems that functions are marked static so they should be OK. However, including the whole Debug.h in a plugin cpp makes it feel OK to use any function/macro from the header file. But actually only part of the macros are for the plugin. some are only for the libomptarget.

I'm not sure we want to make a distinction, the point is to move to a unified debug/message model. You can choose not only the level of information but also the kind of output (text, json, ...). The messages will then be tied to the webpage via enums, that allows all plugins to emit the same message for the same thing with the same link to more information. There will certainly things that are only used in libomptarget or the plugins, but I don't see how that is any worse than duplicating the parts that are used by both.

I didn't mean to duplicate anything. Instead you need multiple header files. One for common parts, one for libomptarget and one for plugins. The latter two both include the first one. Later you expand OFFLOAD_XXX signals, they can be added to the common file. The return signal is generated by the plugins and captured by the libomptarget. Some users may want to see only the messages captured universally by libomptarget. Some users still want to see the native error message. So the libomptarget and plugin side error handling still needs to be separated.

I fail to see why this machinery is necessary to emit only messages from one place and not the other. I am not against a hierarchy of headers per se, but right now, and maybe also later, there seems to be little point.
I mean, we need to introduce a new env variable, actually two, that allow separate control. Once we have that we can argue about separation.
Alternatively, I would have suggested to define the "location" prior to including the debug header, e.g.,:

#define DEBUG_LOCATION "PLUGIN"
#include "Debug.h"

which we verify like:

#if DEBUG_LOCATION != "PLUGIN" and DEBUG_LOCATION != "OMPTARGET"
#error ...
#endif

At the end of the day I want to simplify things. A single location for all our debug needs sounds simpler than 1 + #plugins to me, even if we don't use all functionality at each location. If separation does not allow anything we cannot reasonably do in a single location, I doubt it provides a benefit.

[ye-luo]

This comment was removed by ye-luo.

[ye-luo]

In D86804#2248214, @jdoerfert wrote:

In D86804#2248201, @ye-luo wrote:

In D86804#2247785, @jdoerfert wrote:

In D86804#2247697, @ye-luo wrote:

It seems that functions are marked static so they should be OK. However, including the whole Debug.h in a plugin cpp makes it feel OK to use any function/macro from the header file. But actually only part of the macros are for the plugin. some are only for the libomptarget.

I'm not sure we want to make a distinction, the point is to move to a unified debug/message model. You can choose not only the level of information but also the kind of output (text, json, ...). The messages will then be tied to the webpage via enums, that allows all plugins to emit the same message for the same thing with the same link to more information. There will certainly things that are only used in libomptarget or the plugins, but I don't see how that is any worse than duplicating the parts that are used by both.

I didn't mean to duplicate anything. Instead you need multiple header files. One for common parts, one for libomptarget and one for plugins. The latter two both include the first one. Later you expand OFFLOAD_XXX signals, they can be added to the common file. The return signal is generated by the plugins and captured by the libomptarget. Some users may want to see only the messages captured universally by libomptarget. Some users still want to see the native error message. So the libomptarget and plugin side error handling still needs to be separated.

I fail to see why this machinery is necessary to emit only messages from one place and not the other. I am not against a hierarchy of headers per se, but right now, and maybe also later, there seems to be little point.
I mean, we need to introduce a new env variable, actually two, that allow separate control. Once we have that we can argue about separation.
Alternatively, I would have suggested to define the "location" prior to including the debug header, e.g.,:

#define DEBUG_LOCATION "PLUGIN"
#include "Debug.h"

which we verify like:

#if DEBUG_LOCATION != "PLUGIN" and DEBUG_LOCATION != "OMPTARGET"
#error ...
#endif

At the end of the day I want to simplify things. A single location for all our debug needs sounds simpler than 1 + #plugins to me, even if we don't use all functionality at each location. If separation does not allow anything we cannot reasonably do in a single location, I doubt it provides a benefit.

OK. it seems that macros are unified by DEBUG_PREFIX. So it is good for the purpose of this patch.
Long term, I think the plugin side info should be controlled separately from libomptarget side.

[grokos]

openmp/libomptarget/include/Debug.h

13

libomp target --> libomptarget

14

themessage --> the message

openmp/libomptarget/plugins/common/elf_common.c

18

This message is not applicable anymore. The DP(...) macro used to be defined in each plugin, but now with this patch this is no longer the case. The macro in defined in Debug.h, so the right message here would be "... and Debug.h has been included".

[jhuber6]

Fixed typos.

[jdoerfert]

LGTM, one nit.

openmp/libomptarget/plugins/common/elf_common.c

18

remove "and macro DP(...) has been defined." as it always is (I think)

[jhuber6]

Screwed up the commit message, whoops.

[jdoerfert]

pushed in ae95ceeb8f98

openmp/libomptarget/include/Debug.h

+//===------- Debug.h - Target independent OpenMP target RTL -- C++ --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Routines used to provide debug messages and information from libomptarget
+// and plugin RTLs to the user.
+//
+// Each plugin RTL and libomptarget define TARGET_NAME and DEBUG_PREFIX for use
+// when sending messages to the user. These indicate which RTL sent the message
+//
+// Debug and information messages are controlled by the environment variables
+// LIBOMPTARGET_DEBUG and LIBOMPTARGET_INFO which is set upon initialization
+// of libomptarget or the plugin RTL. 
+//
+// To printf a pointer in hex with a fixed width of 16 digits and a leading 0x,
+// use printf("ptr=" DPxMOD "...\n", DPxPTR(ptr));
+// 
+// DPxMOD expands to:
+//   "0x%0*" PRIxPTR
+// where PRIxPTR expands to an appropriate modifier for the type uintptr_t on a
+// specific platform, e.g. "lu" if uintptr_t is typedef'd as unsigned long:
+//   "0x%0*lu"
+// 
+// Ultimately, the whole statement expands to:
+//   printf("ptr=0x%0*lu...\n",  // the 0* modifier expects an extra argument
+//                               // specifying the width of the output
+//   (int)(2*sizeof(uintptr_t)), // the extra argument specifying the width
+//                               // 8 digits for 32bit systems
+//                               // 16 digits for 64bit
+//   (uintptr_t) ptr);
+//
+//===----------------------------------------------------------------------===//
+#ifndef _OMPTARGET_DEBUG_H
+#define _OMPTARGET_DEBUG_H
+
+static inline int getInfoLevel() {
+  static int InfoLevel = -1;
+  if (InfoLevel >= 0)
+    return InfoLevel;
+
+  if (char *EnvStr = getenv("LIBOMPTARGET_INFO"))
+    InfoLevel = std::stoi(EnvStr);
+
+  return InfoLevel;
+}
+
+static inline int getDebugLevel() {
+  static int DebugLevel = -1;
+  if (DebugLevel >= 0)
+    return DebugLevel;
+
+  if (char *EnvStr = getenv("LIBOMPTARGET_DEBUG"))
+    DebugLevel = std::stoi(EnvStr);
+
+  return DebugLevel;
+}
+
+#ifndef __STDC_FORMAT_MACROS
+#define __STDC_FORMAT_MACROS
+#endif
+#include <inttypes.h>
+#undef __STDC_FORMAT_MACROS
+
+#define DPxMOD "0x%0*" PRIxPTR
+#define DPxPTR(ptr) ((int)(2 * sizeof(uintptr_t))), ((uintptr_t)(ptr))
+#define GETNAME2(name) #name
+#define GETNAME(name) GETNAME2(name)
+
+// Messaging interface
+#define MESSAGE0(_str)                                                         \
+  do {                                                                         \
+    fprintf(stderr, GETNAME(TARGET_NAME) " message: %s\n", _str);              \
+  } while (0)
+
+#define MESSAGE(_str, ...)                                                     \
+  do {                                                                         \
+    fprintf(stderr, GETNAME(TARGET_NAME) " message: " _str "\n", __VA_ARGS__); \
+  } while (0)
+
+#define FATAL_MESSAGE0(_num, _str)                                             \
+  do {                                                                         \
+    fprintf(stderr, GETNAME(TARGET_NAME) " fatal error %d: %s\n", _num, _str); \
+    abort();                                                                   \
+  } while (0)
+
+#define FATAL_MESSAGE(_num, _str, ...)                                         \
+  do {                                                                         \
+    fprintf(stderr, GETNAME(TARGET_NAME) " fatal error %d:" _str "\n", _num,   \
+            __VA_ARGS__);                                                      \
+    abort();                                                                   \
+  } while (0)
+
+#define FAILURE_MESSAGE(...)                                                   \
+  do {                                                                         \
+    fprintf(stderr, GETNAME(TARGET_NAME) " error: ");                          \
+    fprintf(stderr, __VA_ARGS__);                                              \
+  } while (0)
+
+// Debugging messages
+#ifdef OMPTARGET_DEBUG
+#include <stdio.h>
+
+#define DEBUGP(prefix, ...)                                                    \
+  {                                                                            \
+    fprintf(stderr, "%s --> ", prefix);                                        \
+    fprintf(stderr, __VA_ARGS__);                                              \
+  }
+
+#define DP(...)                                                                \
+  do {                                                                         \
+    if (getDebugLevel() > 0) {                                                 \
+      DEBUGP(DEBUG_PREFIX, __VA_ARGS__);                                       \
+    }                                                                          \
+  } while (false)
+
+#define REPORT(...)                                                            \
+  do {                                                                         \
+    if (getDebugLevel() > 0) {                                                 \
+      DP(__VA_ARGS__);                                                         \
+    } else {                                                                   \
+      FAILURE_MESSAGE(__VA_ARGS__);                                            \
+    }                                                                          \
+  } while (false)
+#else
+#define DEBUGP(prefix, ...)                                                    \
+  {}
+#define DP(...)                                                                \
+  {}
+#define REPORT(...) FAILURE_MESSAGE(__VA_ARGS__);
+#endif // OMPTARGET_DEBUG
+
+#endif // _OMPTARGET_DEBUG_H

openmp/libomptarget/include/omptarget.h

int __tgt_target_teams_nowait_mapper(
     int32_t noAliasDepNum, void *noAliasDepList);
 
 void __kmpc_push_target_tripcount(int64_t device_id, uint64_t loop_tripcount);
 
 #ifdef __cplusplus
 }
 #endif
 
-#ifndef __STDC_FORMAT_MACROS
-#define __STDC_FORMAT_MACROS
-#endif
-
-#include <inttypes.h>
-#define DPxMOD "0x%0*" PRIxPTR
-#define DPxPTR(ptr) ((int)(2*sizeof(uintptr_t))), ((uintptr_t) (ptr))
-
-/*
- * To printf a pointer in hex with a fixed width of 16 digits and a leading 0x,
- * use printf("ptr=" DPxMOD "...\n", DPxPTR(ptr));
- *
- * DPxMOD expands to:
- *   "0x%0*" PRIxPTR
- * where PRIxPTR expands to an appropriate modifier for the type uintptr_t on a
- * specific platform, e.g. "lu" if uintptr_t is typedef'd as unsigned long:
- *   "0x%0*lu"
- *
- * Ultimately, the whole statement expands to:
- *   printf("ptr=0x%0*lu...\n",  // the 0* modifier expects an extra argument
- *                               // specifying the width of the output
- *   (int)(2*sizeof(uintptr_t)), // the extra argument specifying the width
- *                               // 8 digits for 32bit systems
- *                               // 16 digits for 64bit
- *   (uintptr_t) ptr);
- */
-
-#ifdef OMPTARGET_DEBUG
-#include <stdio.h>
-#define DEBUGP(prefix, ...)                                                    \
-  {                                                                            \
-    fprintf(stderr, "%s --> ", prefix);                                        \
-    fprintf(stderr, __VA_ARGS__);                                              \
-  }
-#else
-#define DEBUGP(prefix, ...)                                                    \
-  {}
-#endif
-
 #ifdef __cplusplus
 #define EXTERN extern "C"
 #else
 #define EXTERN extern
 #endif
 
 #endif // _OMPTARGET_H_

openmp/libomptarget/plugins/amdgpu/src/rtl.cpp

 #include <cstdio>
 #include <cstdlib>
 #include <cstring>
 #include <dlfcn.h>
 #include <elf.h>
 #include <ffi.h>
 #include <fstream>
 #include <iostream>
 #include <libelf.h>
 #include <list>
 #include <memory>
 #include <mutex>
 #include <shared_mutex>
 #include <thread>
 #include <unordered_map>
 #include <vector>
 
 // Header from ATMI interface
 #include "atmi_interop_hsa.h"
 #include "atmi_runtime.h"
 
 #include "internal.h"
 
+#include "Debug.h"
 #include "omptargetplugin.h"
 
 // Get static gpu grid values from clang target-specific constants managed
 // in the header file llvm/Frontend/OpenMP/OMPGridValues.h
 // Copied verbatim to meet the requirement that libomptarget builds without
 // a copy of llvm checked out nearby
 namespace llvm {
 namespace omp {
static constexpr unsigned AMDGPUGpuGridValues[] = {
     63         // GV_Warp_Size_Log2_MaskL
 };
 } // namespace omp
 } // namespace llvm
 
 #ifndef TARGET_NAME
 #define TARGET_NAME AMDHSA
 #endif
+#define DEBUG_PREFIX "Target " GETNAME(TARGET_NAME) " RTL"
 
 int print_kernel_trace;
 
 // Size of the target call stack struture
 uint32_t TgtStackItemSize = 0;
 
-#ifdef OMPTARGET_DEBUG
-static int DebugLevel = 0;
-
-#define GETNAME2(name) #name
-#define GETNAME(name) GETNAME2(name)
-#define DP(...)                                                                \
-  do {                                                                         \
-    if (DebugLevel > 0) {                                                      \
-      DEBUGP("Target " GETNAME(TARGET_NAME) " RTL", __VA_ARGS__);              \
-    }                                                                          \
-  } while (false)
-#else // OMPTARGET_DEBUG
-#define DP(...)                                                                \
-  {}
-#endif // OMPTARGET_DEBUG
-
 #undef check // Drop definition from internal.h
 #ifdef OMPTARGET_DEBUG
 #define check(msg, status)                                                     \
   if (status != ATMI_STATUS_SUCCESS) {                                         \
     /* fprintf(stderr, "[%s:%d] %s failed.\n", __FILE__, __LINE__, #msg);*/    \
     DP(#msg " failed\n");                                                      \
     /*assert(0);*/                                                             \
   } else {                                                                     \
void clearOffloadEntriesTable(int device_id) {
     FuncGblEntries[device_id].emplace_back();
     FuncOrGblEntryTy &E = FuncGblEntries[device_id].back();
     // KernelArgPoolMap.clear();
     E.Entries.clear();
     E.Table.EntriesBegin = E.Table.EntriesEnd = 0;
   }
 
   RTLDeviceInfoTy() {
-#ifdef OMPTARGET_DEBUG
-    if (char *envStr = getenv("LIBOMPTARGET_DEBUG"))
-      DebugLevel = std::stoi(envStr);
-#endif // OMPTARGET_DEBUG
-
     // LIBOMPTARGET_KERNEL_TRACE provides a kernel launch trace to stderr
     // anytime. You do not need a debug library build.
     //  0 => no tracing
     //  1 => tracing dispatch only
     // >1 => verbosity increase
     if (char *envStr = getenv("LIBOMPTARGET_KERNEL_TRACE"))
       print_kernel_trace = atoi(envStr);
     else

openmp/libomptarget/plugins/common/elf_common.c

 //===-- elf_common.c - Common ELF functionality -------------------*- C -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Common ELF functionality for target plugins.
 // Must be included in the plugin source file AFTER omptarget.h has been
 // included and macro DP(...) has been defined.
 // .
 //
 //===----------------------------------------------------------------------===//
 
-#if !(defined(_OMPTARGET_H_) && defined(DP))
-#error Include elf_common.c in the plugin source AFTER omptarget.h has been\
- included and macro DP(...) has been defined.
+#if !(defined(_OMPTARGET_DEBUG_H) && defined(DP))
+#error Include elf_common.c in the plugin source AFTER Debug.h has\
+ been included and macro DP(...) has been defined.
 #endif
 
 #include <elf.h>
 #include <libelf.h>
 
 // Check whether an image is valid for execution on target_id
 static inline int32_t elf_check_machine(__tgt_device_image *image,
     uint16_t target_id) {
 
   // Is the library version incompatible with the header file?
   if (elf_version(EV_CURRENT) == EV_NONE) {
     DP("Incompatible ELF library!\n");

openmp/libomptarget/plugins/cuda/src/rtl.cpp

 //===----RTLs/cuda/src/rtl.cpp - Target RTLs Implementation ------- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // RTL for CUDA machine
 //
 //===----------------------------------------------------------------------===//
 
 #include <cassert>
 #include <cstddef>
 #include <cuda.h>
 #include <list>
 #include <memory>
 #include <mutex>
 #include <string>
 #include <vector>
 
+#include "Debug.h"
 #include "omptargetplugin.h"
 
-#ifndef TARGET_NAME
 #define TARGET_NAME CUDA
-#endif
-
-#ifdef OMPTARGET_DEBUG
-static int DebugLevel = 0;
-
-#define GETNAME2(name) #name
-#define GETNAME(name) GETNAME2(name)
-#define DP(...) \
-  do { \
-    if (DebugLevel > 0) { \
-      DEBUGP("Target " GETNAME(TARGET_NAME) " RTL", __VA_ARGS__); \
-    } \
-  } while (false)
+#define DEBUG_PREFIX "Target " GETNAME(TARGET_NAME) " RTL"
 
 // Utility for retrieving and printing CUDA error string.
-#define CUDA_ERR_STRING(err) \
-  do { \
-    if (DebugLevel > 0) { \
-      const char *errStr; \
-      cuGetErrorString(err, &errStr); \
-      DEBUGP("Target " GETNAME(TARGET_NAME) " RTL", "CUDA error is: %s\n", errStr); \
-    } \
+#ifdef OMPTARGET_DEBUG
+#define CUDA_ERR_STRING(err)                                                   \
+  do {                                                                         \
+    if (getDebugLevel() > 0) {                                                      \
+      const char *errStr;                                                      \
+      cuGetErrorString(err, &errStr);                                          \
+      DP("CUDA error is: %s\n", errStr);                                       \
+    }                                                                          \
   } while (false)
 #else // OMPTARGET_DEBUG
-#define DP(...) {}
 #define CUDA_ERR_STRING(err) {}
 #endif // OMPTARGET_DEBUG
 
 #include "../../common/elf_common.c"
 
 /// Keep entries table per device.
 struct FuncOrGblEntryTy {
   __tgt_target_table Table;
   std::vector<__tgt_offload_entry> Entries;

 public:
   // This class should not be copied
   DeviceRTLTy(const DeviceRTLTy &) = delete;
   DeviceRTLTy(DeviceRTLTy &&) = delete;
 
   DeviceRTLTy()
       : NumberOfDevices(0), EnvNumTeams(-1), EnvTeamLimit(-1),
         RequiresFlags(OMP_REQ_UNDEFINED) {
-#ifdef OMPTARGET_DEBUG
-    if (const char *EnvStr = getenv("LIBOMPTARGET_DEBUG"))
-      DebugLevel = std::stoi(EnvStr);
-#endif // OMPTARGET_DEBUG
 
     DP("Start initializing CUDA\n");
 
     CUresult Err = cuInit(0);
     if (!checkResult(Err, "Error returned from cuInit\n")) {
       return;
     }
 

openmp/libomptarget/plugins/generic-elf-64bit/src/rtl.cpp

 //===----------------------------------------------------------------------===//
 
 #include <cassert>
 #include <cstdio>
 #include <cstring>
 #include <cstdlib>
 #include <dlfcn.h>
 #include <ffi.h>
 #include <gelf.h>
 #include <link.h>
 #include <list>
 #include <string>
 #include <vector>
 
+#include "Debug.h"
 #include "omptargetplugin.h"
 
 #ifndef TARGET_NAME
 #define TARGET_NAME Generic ELF - 64bit
 #endif
+#define DEBUG_PREFIX "TARGET " GETNAME(TARGET_NAME) " RTL"
 
 #ifndef TARGET_ELF_ID
 #define TARGET_ELF_ID 0
 #endif
 
-#ifdef OMPTARGET_DEBUG
-static int DebugLevel = 0;
-
-#define GETNAME2(name) #name
-#define GETNAME(name) GETNAME2(name)
-#define DP(...) \
-  do { \
-    if (DebugLevel > 0) { \
-      DEBUGP("Target " GETNAME(TARGET_NAME) " RTL", __VA_ARGS__); \
-    } \
-  } while (false)
-#else // OMPTARGET_DEBUG
-#define DP(...) {}
-#endif // OMPTARGET_DEBUG
-
 #include "../../common/elf_common.c"
 
 #define NUMBER_OF_DEVICES 4
 #define OFFLOADSECTIONNAME "omp_offloading_entries"
 
 /// Array of Dynamic libraries loaded for this target.
 struct DynLibTy {
   char *FileName;
__tgt_target_table *getOffloadEntriesTable(int32_t device_id) {
     assert(device_id < (int32_t)FuncGblEntries.size() &&
            "Unexpected device id!");
     FuncOrGblEntryTy &E = FuncGblEntries[device_id].back();
 
     return &E.Table;
   }
 
   RTLDeviceInfoTy(int32_t num_devices) {
-#ifdef OMPTARGET_DEBUG
-    if (char *envStr = getenv("LIBOMPTARGET_DEBUG")) {
-      DebugLevel = std::stoi(envStr);
-    }
-#endif // OMPTARGET_DEBUG
 
     FuncGblEntries.resize(num_devices);
   }
 
   ~RTLDeviceInfoTy() {
     // Close dynamic libraries
     for (auto &lib : DynLibs) {
       if (lib.Handle) {

openmp/libomptarget/plugins/ve/src/rtl.cpp

 //===-RTLs/nec-aurora/src/rtl.cpp - Target RTLs Implementation - C++ -*-======//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.txt for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // RTL for NEC Aurora TSUBASA machines
 //
 //===----------------------------------------------------------------------===//
 
+#include "Debug.h"
 #include "omptargetplugin.h"
 
 #include <algorithm>
 #include <cassert>
 #include <cerrno>
 #include <cstring>
 #include <list>
 #include <stdlib.h>
 #include <string>
 #include <sys/stat.h>
 #include <ve_offload.h>
 #include <vector>
 #include <veosinfo/veosinfo.h>
 
 #ifndef TARGET_ELF_ID
 #define TARGET_ELF_ID 0
 #endif
 
-#ifdef OMPTARGET_DEBUG
-static int DebugLevel = 0;
+#define TARGET_NAME VE
 
-#define GETNAME2(name) #name
-#define GETNAME(name) GETNAME2(name)
-#define DP(...)                                                                \
-  do {                                                                         \
-    if (DebugLevel > 0) {                                                      \
-      DEBUGP("Target " GETNAME(TARGET_NAME) " RTL", __VA_ARGS__);              \
-    }                                                                          \
-  } while (false)
-#else // OMPTARGET_DEBUG
-#define DP(...)                                                                \
-  {}
-#endif // OMPTARGET_DEBUG
+#define DEBUG_PREFIX "Target " GETNAME(TARGET_NAME) " RTL"
 
 #include "../../common/elf_common.c"
 
 struct DynLibTy {
   char *FileName;
   uint64_t VeoLibHandle;
 };
 
void buildOffloadTableFromHost(int32_t device_id, uint64_t VeoLibHandle,
     FuncOrGblEntry[device_id].back().Table.EntriesEnd = &T.back() + 1;
   }
 
   __tgt_target_table *getOffloadTable(int32_t device_id) {
     return &FuncOrGblEntry[device_id].back().Table;
   }
 
   RTLDeviceInfoTy() {
-#ifdef OMPTARGET_DEBUG
-    if (char *envStr = getenv("LIBOMPTARGET_DEBUG")) {
-      DebugLevel = std::stoi(envStr);
-    }
-#endif // OMPTARGET_DEBUG
 
     struct ve_nodeinfo node_info;
     ve_node_info(&node_info);
 
     // Build a predictable mapping between VE node ids and OpenMP device ids.
     // This is necessary, because nodes can be missing or offline and (active)
     // node ids are thus not consecutive. The entries in ve_nodeinfo may also
     // not be in the order of their node ids.

openmp/libomptarget/src/api.cpp

 //===----------- api.cpp - Target independent OpenMP target RTL -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Implementation of OpenMP API interface functions.
 //
 //===----------------------------------------------------------------------===//
 
-#include <omptarget.h>
-
 #include "device.h"
 #include "private.h"
 #include "rtl.h"
 
 #include <climits>
 #include <cstring>
 #include <cstdlib>
 

openmp/libomptarget/src/interface.cpp

 //===-------- interface.cpp - Target independent OpenMP target RTL --------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Implementation of the interface to be used by Clang during the codegen of a
 // target region.
 //
 //===----------------------------------------------------------------------===//
 
-#include <omptarget.h>
-
 #include "device.h"
 #include "private.h"
 #include "rtl.h"
 
 #include <cassert>
 #include <cstdlib>
 #include <mutex>
 
case tgt_disabled:
       }
       break;
     case tgt_default:
       FATAL_MESSAGE0(1, "default offloading policy must be switched to "
                         "mandatory or disabled");
       break;
     case tgt_mandatory:
       if (!success) {
-        if (InfoLevel > 0)
+        if (getInfoLevel() > 0)
           MESSAGE0("LIBOMPTARGET_INFO is not supported yet");
         FATAL_MESSAGE0(1, "failure of target construct while offloading is mandatory");
       }
       break;
   }
 }
 
 ////////////////////////////////////////////////////////////////////////////////

openmp/libomptarget/src/omptarget.cpp

 //===------ omptarget.cpp - Target independent OpenMP target RTL -- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Implementation of the interface to be used by Clang during the codegen of a
 // target region.
 //
 //===----------------------------------------------------------------------===//
 
-#include <omptarget.h>
-
 #include "device.h"
 #include "private.h"
 #include "rtl.h"
 
 #include <cassert>
 #include <vector>
 
-#ifdef OMPTARGET_DEBUG
-int DebugLevel = 0;
-#endif // OMPTARGET_DEBUG
-int InfoLevel = 0;
-
 /* All begin addresses for partially mapped structs must be 8-aligned in order
  * to ensure proper alignment of members. E.g.
  *
  * struct S {
  *   int a;   // 4-aligned
  *   int b;   // 4-aligned
  *   int *p;  // 8-aligned
  * } s1;

openmp/libomptarget/src/private.h

 //===---------- private.h - Target independent OpenMP target RTL ----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Private function declarations and helper macros for debugging output.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef _OMPTARGET_PRIVATE_H
 #define _OMPTARGET_PRIVATE_H
 
 #include <omptarget.h>
+#include <Debug.h>
 
 #include <cstdint>
 
 extern int targetDataBegin(DeviceTy &Device, int32_t arg_num, void **args_base,
                            void **args, int64_t *arg_sizes, int64_t *arg_types,
                            void **arg_mappers,
                            __tgt_async_info *async_info_ptr);
 
 extern int targetDataEnd(DeviceTy &Device, int32_t ArgNum, void **ArgBases,
                          void **Args, int64_t *ArgSizes, int64_t *ArgTypes,
                          void **ArgMappers, __tgt_async_info *AsyncInfo);
 
 extern int target_data_update(DeviceTy &Device, int32_t arg_num,
                               void **args_base, void **args,
                               int64_t *arg_sizes, int64_t *arg_types,
                               void **arg_mappers,
                               __tgt_async_info *async_info_ptr = nullptr);
 
 extern int target(int64_t DeviceId, void *HostPtr, int32_t ArgNum,
                   void **ArgBases, void **Args, int64_t *ArgSizes,
                   int64_t *ArgTypes, void **ArgMappers, int32_t TeamNum,
struct MapComponentInfoTy {
   MapComponentInfoTy(void *Base, void *Begin, int64_t Size, int64_t Type)
       : Base(Base), Begin(Begin), Size(Size), Type(Type) {}
 };
 
 // This structure stores all components of a user-defined mapper. The number of
 // components are dynamically decided, so we utilize C++ STL vector
 // implementation here.
 struct MapperComponentsTy {
   std::vector<MapComponentInfoTy> Components;
   int32_t size() { return Components.size(); }
 };
 
 // The mapper function pointer type. It follows the signature below:
 // void .omp_mapper.<type_name>.<mapper_id>.(void *rt_mapper_handle,
 //                                           void *base, void *begin,
 //                                           size_t size, int64_t type);
 typedef void (*MapperFuncPtrTy)(void *, void *, void *, int64_t, int64_t);
 
 // Function pointer type for target_data_* functions (targetDataBegin,
 // targetDataEnd and target_data_update).
 typedef int (*TargetDataFuncPtrTy)(DeviceTy &, int32_t, void **, void **,
                                    int64_t *, int64_t *, void **,
                                    __tgt_async_info *);
 
-////////////////////////////////////////////////////////////////////////////////
-// implementation for messages
-////////////////////////////////////////////////////////////////////////////////
-
-#define MESSAGE0(_str)                                                         \
-  do {                                                                         \
-    fprintf(stderr, "Libomptarget message: %s\n", _str);                       \
-  } while (0)
-
-#define MESSAGE(_str, ...)                                                     \
-  do {                                                                         \
-    fprintf(stderr, "Libomptarget message: " _str "\n", __VA_ARGS__);          \
-  } while (0)
-
-#define FATAL_MESSAGE0(_num, _str)                                             \
-  do {                                                                         \
-    fprintf(stderr, "Libomptarget fatal error %d: %s\n", _num, _str);          \
-    abort();                                                                   \
-  } while (0)
-
-#define FATAL_MESSAGE(_num, _str, ...)                                         \
-  do {                                                                         \
-    fprintf(stderr, "Libomptarget fatal error %d:" _str "\n", _num,            \
-            __VA_ARGS__);                                                      \
-    abort();                                                                   \
-  } while (0)
-
-#define FAILURE_MESSAGE(...)                                                   \
-  do {                                                                         \
-    fprintf(stderr, "Libomptarget error: ");                                   \
-    fprintf(stderr, __VA_ARGS__);                                              \
-  } while (0)
-
 // Implemented in libomp, they are called from within __tgt_* functions.
 #ifdef __cplusplus
 extern "C" {
 #endif
 // functions that extract info from libomp; keep in sync
 int omp_get_default_device(void) __attribute__((weak));
 int32_t __kmpc_omp_taskwait(void *loc_ref, int32_t gtid) __attribute__((weak));
 int32_t __kmpc_global_thread_num(void *) __attribute__((weak));
 int __kmpc_get_target_offload(void) __attribute__((weak));
 #ifdef __cplusplus
 }
 #endif
 
-extern int InfoLevel;
-#ifdef OMPTARGET_DEBUG
-extern int DebugLevel;
-
-#define DP(...) \
-  do { \
-    if (DebugLevel > 0) { \
-      DEBUGP("Libomptarget", __VA_ARGS__); \
-    } \
-  } while (false)
-#else // OMPTARGET_DEBUG
-#define DP(...) {}
-#endif // OMPTARGET_DEBUG
-
-// Report debug messages that result in offload failure always
-#ifdef OMPTARGET_DEBUG
-#define REPORT(...)                                                            \
-  do {                                                                         \
-    if (DebugLevel > 0) {                                                      \
-      DP(__VA_ARGS__);                                                         \
-    } else {                                                                   \
-      FAILURE_MESSAGE(__VA_ARGS__);                                            \
-    }                                                                          \
-  } while (false)
-#else
-#define REPORT(...) FAILURE_MESSAGE(__VA_ARGS__);
-#endif
+#define TARGET_NAME Libomptarget
+#define DEBUG_PREFIX GETNAME(TARGET_NAME)
 
 #endif

openmp/libomptarget/src/rtl.cpp

__attribute__((destructor(101))) void deinit() {
   delete RTLsMtx;
   delete HostEntriesBeginToTransTable;
   delete TrlTblMtx;
   delete HostPtrToTableMap;
   delete TblMapMtx;
 }
 
 void RTLsTy::LoadRTLs() {
-
-  if (char *envStr = getenv("LIBOMPTARGET_INFO")) {
-    InfoLevel = std::stoi(envStr);
-  }
-#ifdef OMPTARGET_DEBUG
-  if (char *envStr = getenv("LIBOMPTARGET_DEBUG")) {
-    DebugLevel = std::stoi(envStr);
-  }
-#endif // OMPTARGET_DEBUG
-
   // Parse environment variable OMP_TARGET_OFFLOAD (if set)
   TargetOffloadPolicy = (kmp_target_offload_kind_t) __kmpc_get_target_offload();
   if (TargetOffloadPolicy == tgt_disabled) {
     return;
   }
 
   DP("Loading RTLs...\n");