[OpenMP] Add time profiling for libomptarget

[JonChesterfield]

Interesting. Not sure about the dependency on an llvm library, though maybe guarding it makes that acceptable.

The DIY equivalent we've been using is functionally similar. Implemented at trace.h in as unintrusive a fashion as I could manage.

In addition to the time taken by calls, maintaining a log of the arguments to the functions (and the return value) has been very helpful as a debugging hack. The proper thing to do is probably to write said data to an in memory ring buffer. The above header just splats it to stdout.

Does this clang-format nicely? I'd expect to have to spell it TIMESCOPE(); or similar.

[jdoerfert]

I left two comments that we should address now, a third can be future work and the rest has already nice TODOs in the code. Thanks for working on this so quickly.

openmp/libomptarget/src/CMakeLists.txt

40

Add a CMAKE flag to disable this, by default it can be enabled though.

openmp/libomptarget/src/api.cpp

21

Maybe we could move this in one of the header, private or rtl? We might even later use it for internal functions.

openmp/libomptarget/src/interface.cpp

130

We should pass loc to TIMESCOPE.
If it is not null, we can pass reinterpret_cast<const char *>(loc->psource) for now as second argument to the TimeScope constructor.

[jdoerfert]

openmp/libomptarget/src/api.cpp

21

After looking at the output in a flamegraph, I'm not sure if we should not go with __FUNCTION__ instead. Unclear if the types help in any way.

[protze.joachim]

openmp/libomptarget/src/api.cpp

16–21

Then use it like:

TIMESCOPE();

?

[ggeorgakoudis]

Update for comments

[ggeorgakoudis]

In D93055#2446636, @JonChesterfield wrote:

Interesting. Not sure about the dependency on an llvm library, though maybe guarding it makes that acceptable.

The DIY equivalent we've been using is functionally similar. Implemented at trace.h in as unintrusive a fashion as I could manage.

In addition to the time taken by calls, maintaining a log of the arguments to the functions (and the return value) has been very helpful as a debugging hack. The proper thing to do is probably to write said data to an in memory ring buffer. The above header just splats it to stdout.

Does this clang-format nicely? I'd expect to have to spell it TIMESCOPE(); or similar.

Thanks for the comment Jon. It's interesting to include debugging information. I have changed the macro to the format you suggest.

[ggeorgakoudis]

openmp/libomptarget/src/interface.cpp

130

Left that for future work

[ggeorgakoudis]

openmp/libomptarget/src/api.cpp

16–21

Thanks! That's indeed much nicer.

[ggeorgakoudis]

Add env variable to enable profiling, better naming

[jdoerfert]

LGTM, I left two minor change requests below. Thanks for getting this up and running so quickly!

openmp/libomptarget/src/CMakeLists.txt

41

Move the conditional if(OPENMP_ENABLE_LIBOMPTARGET_PROFILING) to include all of the stuff, including the linking of things. So if it is disabled we build as we did before.

openmp/libomptarget/src/rtl.cpp

37

Make this a char* and set it to the value of LIBOMPTARGET_PROFILE. We will assume that is a path for the resulting file.

[sylvestre.ledru]

I think there is something wrong in this patch
It creates a new library called:
usr/lib/llvm-12/lib/libomptarget.so.12

When current files are:
/usr/lib/llvm-12/lib/libomp.so.5

So, at least for consistency, the lib should be called libomptarget.so.12

@ggeorgakoudis should I report a new bug? thanks

[jdoerfert]

In D93055#2450692, @sylvestre.ledru wrote:

I think there is something wrong in this patch
It creates a new library called:
usr/lib/llvm-12/lib/libomptarget.so.12

When current files are:
/usr/lib/llvm-12/lib/libomp.so.5

So, at least for consistency, the lib should be called libomptarget.so.12

@ggeorgakoudis should I report a new bug? thanks

I don't understand this comment

[JonChesterfield]

In D93055#2450692, @sylvestre.ledru wrote:

I think there is something wrong in this patch
It creates a new library called:
...
@ggeorgakoudis should I report a new bug? thanks

There's no change to the cmake files that control the output file name in this patch so I don't think this can be the cause of the change you're seeing. There may be an existing bug about how the libraries should be named.

[tianshilei1992]

openmp/CMakeLists.txt

86

So is it intentional to enable profiling as long as libomptarget is enabled?

[jdoerfert]

openmp/CMakeLists.txt

86

Yes, profiling is by default a single thread local memory access per entry point. If it turns out to be a problem we can revisit this.

[ayermolo]

@ggeorgakoudis In our local build environment I am now observing error below:

FAILED: lib/libomptarget.so.12git
..
/home/engshare/third-party2/binutils/2.32/centos7-native/da39a3e/bin/ld.gold: error: lib/libLLVMSupport.a(TimeProfiler.cpp.o): requires dynamic R_X86_64_32 reloc which may overflow at runtime; recompile with -fPIC

We are building llvm with -DLLVM_ENABLE_PIC=OFF.

So with ENABLE_LIBOMPTARGET enabling OPENMP_ENABLE_LIBOMPTARGET_PROFILING it goes to else case and includes Support library, which leads to link failure.

Maybe add explicit flag NOT_LINK_SUPPORT or something like that for this case, or break up the ENABLE_LIBOMPTARGET/OPENMP_ENABLE_LIBOMPTARGET_PROFILING dependency?

[JonChesterfield]

Ah, yes. That makes sense. Building llvmsupport without PIC is a reasonable thing to do, and will cause problems when linked into libomptarget. One solution is to link libomptarget against a dynamically built llvmsupport, which might mean building llvmsupport twice. I don't know whether the existing cmake is wired up to do that.

I'd prefer we default to profiling disabled (and the llvm libs not linked in) for now.

(actually my preferred solution is to link libomptarget statically, but that feels like a longer term goal)

[ayermolo]

In D93055#2461829, @JonChesterfield wrote:

Ah, yes. That makes sense. Building llvmsupport without PIC is a reasonable thing to do, and will cause problems when linked into libomptarget. One solution is to link libomptarget against a dynamically built llvmsupport, which might mean building llvmsupport twice. I don't know whether the existing cmake is wired up to do that.

I'd prefer we default to profiling disabled (and the llvm libs not linked in) for now.

(actually my preferred solution is to link libomptarget statically, but that feels like a longer term goal)

I would prefer that also. :)

[jdoerfert]

In D93055#2461668, @ayermolo wrote:

Maybe add explicit flag NOT_LINK_SUPPORT or something like that for this case, or break up the ENABLE_LIBOMPTARGET/OPENMP_ENABLE_LIBOMPTARGET_PROFILING dependency?

Why add a new flag if you can disable profiling with the one added? I mean, OPENMP_ENABLE_LIBOMPTARGET_PROFILING=OFF should unbreak your build, correct?

In D93055#2461829, @JonChesterfield wrote:

I'd prefer we default to profiling disabled (and the llvm libs not linked in) for now.

(actually my preferred solution is to link libomptarget statically, but that feels like a longer term goal)

So far there was no convincing reason why it should be off. Having it on by default has the distinct advantage that most people get the feature and can use profiling with their local build or release of LLVM, that is arguably useful. If you start with some fancy build you might need to disable it, or upstream support of your fancy build, neither is unprecedented.

In D93055#2461842, @ayermolo wrote:

I'd prefer we default to profiling disabled (and the llvm libs not linked in) for now.

(actually my preferred solution is to link libomptarget statically, but that feels like a longer term goal)

I would prefer that also. :)

See above.

[vzakhari]

FYI, libomptarget LIT tests fail with profiling enabled in shared-libs builds, because libLLVMSupport.so.12git dependency cannot be resolved. Apparently, LD_LIBRARY_PATH is only set to <build>/llvm/projects/openmp/libomptarget, which obviously does not contain the support library.

[kkwli0]

In D93055#2516351, @vzakhari wrote:

FYI, libomptarget LIT tests fail with profiling enabled in shared-libs builds, because libLLVMSupport.so.12git dependency cannot be resolved. Apparently, LD_LIBRARY_PATH is only set to <build>/llvm/projects/openmp/libomptarget, which obviously does not contain the support library.

@vzakhari Is it an out-of-tree build? If it is, you need to set LIBOMPTARGET_OPENMP_HEADER_FOLDER and DLIBOMPTARGET_OPENMP_HOST_RTL_FOLDER (https://github.com/llvm/llvm-project/tree/main/openmp#id11).

[ggeorgakoudis]

In D93055#2516351, @vzakhari wrote:

FYI, libomptarget LIT tests fail with profiling enabled in shared-libs builds, because libLLVMSupport.so.12git dependency cannot be resolved. Apparently, LD_LIBRARY_PATH is only set to <build>/llvm/projects/openmp/libomptarget, which obviously does not contain the support library.

Thanks for reporting that @vzakhari. Will fix soon.

[vzakhari]

In D93055#2516823, @kkwli0 wrote:

In D93055#2516351, @vzakhari wrote:

FYI, libomptarget LIT tests fail with profiling enabled in shared-libs builds, because libLLVMSupport.so.12git dependency cannot be resolved. Apparently, LD_LIBRARY_PATH is only set to <build>/llvm/projects/openmp/libomptarget, which obviously does not contain the support library.

@vzakhari Is it an out-of-tree build? If it is, you need to set LIBOMPTARGET_OPENMP_HEADER_FOLDER and DLIBOMPTARGET_OPENMP_HOST_RTL_FOLDER (https://github.com/llvm/llvm-project/tree/main/openmp#id11).

No, it is an in-tree build. I see libomptarget's output location (LIBOMPTARGET_LIBRARY_DIR) does not align well with the default rpath setting done inside add_llvm_library.

[vzakhari]

In D93055#2516898, @ggeorgakoudis wrote:

In D93055#2516351, @vzakhari wrote:

FYI, libomptarget LIT tests fail with profiling enabled in shared-libs builds, because libLLVMSupport.so.12git dependency cannot be resolved. Apparently, LD_LIBRARY_PATH is only set to <build>/llvm/projects/openmp/libomptarget, which obviously does not contain the support library.

Thanks for reporting that @vzakhari. Will fix soon.

Thank you, @ggeorgakoudis!

[ggeorgakoudis]

In D93055#2516901, @vzakhari wrote:

In D93055#2516898, @ggeorgakoudis wrote:

In D93055#2516351, @vzakhari wrote:

FYI, libomptarget LIT tests fail with profiling enabled in shared-libs builds, because libLLVMSupport.so.12git dependency cannot be resolved. Apparently, LD_LIBRARY_PATH is only set to <build>/llvm/projects/openmp/libomptarget, which obviously does not contain the support library.

Thanks for reporting that @vzakhari. Will fix soon.

Thank you, @ggeorgakoudis!

@vzakhari Could you please share your cmake run to reproduce?

[vzakhari]

Hi @ggeorgakoudis, please try something along these lines:

cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;openmp" -DCMAKE_COLOR_MAKEFILE=0 -DBUILD_SHARED_LIBS=1 -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="X86" -DCMAKE_C_COMPILER=".../bin/gcc" -DCMAKE_CXX_COMPILER=".../bin/g++" -DLLVM_BINUTILS_INCDIR=".../lib/gcc/x86_64-linux-gnu/7.3.0/plugin/include" -DCMAKE_INSTALL_PREFIX="..." ".../llvm"
make install
make check-libomptarget

[ggeorgakoudis]

In D93055#2518783, @vzakhari wrote:

Hi @ggeorgakoudis, please try something along these lines:

cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;openmp" -DCMAKE_COLOR_MAKEFILE=0 -DBUILD_SHARED_LIBS=1 -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="X86" -DCMAKE_C_COMPILER=".../bin/gcc" -DCMAKE_CXX_COMPILER=".../bin/g++" -DLLVM_BINUTILS_INCDIR=".../lib/gcc/x86_64-linux-gnu/7.3.0/plugin/include" -DCMAKE_INSTALL_PREFIX="..." ".../llvm"
make install
make check-libomptarget

Thank you @vzakhari. Patch D95376 should be fixing this issue.

[vzakhari]

Thank you, @ggeorgakoudis !

[vzakhari]

openmp/libomptarget/src/CMakeLists.txt

35

Hello. This function adds a LLVM_VERSION_MAJOR suffix to libomptarget library name, and libomptarget.so is just a link now. This actually causes dynamic dependency in the executables to be set to libomptarget.so.##git. When LLVM changes its major version, I guess all the executables built with the previous version will stop working. Is this expected or I am missing something?

[jdoerfert]

openmp/libomptarget/src/CMakeLists.txt

35

The way I understand you this seems desirable.

[vzakhari]

openmp/libomptarget/src/CMakeLists.txt

35

I am not sure why we want to make libomptarget versioning dependent on llvm versioning. We used to have backward compatible libomptarget, i.e. an executable built with llvm12 libomptarget could work with llvm13 libomptarget even if there was a breaking LLVM IR change between these two versions of llvm. Now we disallow this, but it does not seem like this was the main goal of this commit.

I bthink if we want to introduce versioning for libomptarget it should be independent of llvm versioning.

openmp/CMakeLists.txt

 
 if (NOT LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR)
   message(STATUS "Missing definition for LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR, disabling libomptarget")
   set(ENABLE_LIBOMPTARGET OFF)
 endif()
 
 option(OPENMP_ENABLE_LIBOMPTARGET "Enable building libomptarget for offloading."
        ${ENABLE_LIBOMPTARGET})
+option(OPENMP_ENABLE_LIBOMPTARGET_PROFILING "Enable time profiling for libomptarget."
+       ${ENABLE_LIBOMPTARGET})
 if (OPENMP_ENABLE_LIBOMPTARGET)
   # Check that the library can actually be built.
   if (APPLE OR WIN32)
     message(FATAL_ERROR "libomptarget cannot be built on Windows and MacOS X!")
   elseif (NOT OPENMP_HAVE_STD_CPP14_FLAG)
     message(FATAL_ERROR "Host compiler must support C++14 to build libomptarget!")
   endif()
 

openmp/libomptarget/src/CMakeLists.txt

set(src_files
   api.cpp
   device.cpp
   interface.cpp
   MemoryManager.cpp
   rtl.cpp
   omptarget.cpp
 )
 
-# Build libomptarget library with libdl dependency.
+# Build libomptarget library with libdl dependency. Add LLVMSupport
+# dependency if building in-tree with profiling enabled.
+if(OPENMP_STANDALONE_BUILD OR (NOT OPENMP_ENABLE_LIBOMPTARGET_PROFILING))
   add_library(omptarget SHARED ${src_files})
   target_link_libraries(omptarget
     ${CMAKE_DL_LIBS}
     "-Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/exports")
+else()
+  set(LLVM_LINK_COMPONENTS
+    Support
+    )
+  add_llvm_library(omptarget SHARED ${src_files}
+      LINK_LIBS ${CMAKE_DL_LIBS}
+      "-Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/exports"
+      )
+  target_compile_definitions(omptarget PUBLIC OMPTARGET_PROFILE_ENABLED)
+endif()
 
 # Install libomptarget under the lib destination folder.
 install(TARGETS omptarget LIBRARY COMPONENT omptarget
   DESTINATION "${OPENMP_INSTALL_LIBDIR}")

openmp/libomptarget/src/api.cpp

 //===----------- api.cpp - Target independent OpenMP target RTL -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Implementation of OpenMP API interface functions.
 //
 //===----------------------------------------------------------------------===//
 
 #include "device.h"
 #include "private.h"
 #include "rtl.h"
 
 #include <climits>
 #include <cstring>
 #include <cstdlib>
 
 EXTERN int omp_get_num_devices(void) {
+  TIMESCOPE();
   PM->RTLsMtx.lock();
   size_t DevicesSize = PM->Devices.size();
   PM->RTLsMtx.unlock();
 
   DP("Call to omp_get_num_devices returning %zd\n", DevicesSize);
 
   return DevicesSize;
 }
 
 EXTERN int omp_get_initial_device(void) {
+  TIMESCOPE();
   int hostDevice = omp_get_num_devices();
   DP("Call to omp_get_initial_device returning %d\n", hostDevice);
   return hostDevice;
 }
 
 EXTERN void *omp_target_alloc(size_t size, int device_num) {
+  TIMESCOPE();
   DP("Call to omp_target_alloc for device %d requesting %zu bytes\n",
       device_num, size);
 
   if (size <= 0) {
     DP("Call to omp_target_alloc with non-positive length\n");
     return NULL;
   }
 
EXTERN void *omp_target_alloc(size_t size, int device_num) {
   }
 
   rc = PM->Devices[device_num].allocData(size);
   DP("omp_target_alloc returns device ptr " DPxMOD "\n", DPxPTR(rc));
   return rc;
 }
 
 EXTERN void omp_target_free(void *device_ptr, int device_num) {
+  TIMESCOPE();
   DP("Call to omp_target_free for device %d and address " DPxMOD "\n",
       device_num, DPxPTR(device_ptr));
 
   if (!device_ptr) {
     DP("Call to omp_target_free with NULL ptr\n");
     return;
   }
 
   if (device_num == omp_get_initial_device()) {
     free(device_ptr);
     DP("omp_target_free deallocated host ptr\n");
     return;
   }
 
   if (!device_is_ready(device_num)) {
     DP("omp_target_free returns, nothing to do\n");
     return;
   }
 
   PM->Devices[device_num].deleteData(device_ptr);
   DP("omp_target_free deallocated device ptr\n");
 }
 
 EXTERN int omp_target_is_present(void *ptr, int device_num) {
+  TIMESCOPE();
   DP("Call to omp_target_is_present for device %d and address " DPxMOD "\n",
       device_num, DPxPTR(ptr));
 
   if (!ptr) {
     DP("Call to omp_target_is_present with NULL ptr, returning false\n");
     return false;
   }
 
EXTERN int omp_target_is_present(void *ptr, int device_num) {
   if (PM->RTLs.RequiresFlags & OMP_REQ_UNIFIED_SHARED_MEMORY)
     rc = !IsHostPtr;
   DP("Call to omp_target_is_present returns %d\n", rc);
   return rc;
 }
 
 EXTERN int omp_target_memcpy(void *dst, void *src, size_t length,
     size_t dst_offset, size_t src_offset, int dst_device, int src_device) {
+  TIMESCOPE();
   DP("Call to omp_target_memcpy, dst device %d, src device %d, "
       "dst addr " DPxMOD ", src addr " DPxMOD ", dst offset %zu, "
       "src offset %zu, length %zu\n", dst_device, src_device, DPxPTR(dst),
       DPxPTR(src), dst_offset, src_offset, length);
 
   if (!dst || !src || length <= 0) {
     REPORT("Call to omp_target_memcpy with invalid arguments\n");
     return OFFLOAD_FAIL;
EXTERN int omp_target_memcpy(void *dst, void *src, size_t length,
   DP("omp_target_memcpy returns %d\n", rc);
   return rc;
 }
 
 EXTERN int omp_target_memcpy_rect(void *dst, void *src, size_t element_size,
     int num_dims, const size_t *volume, const size_t *dst_offsets,
     const size_t *src_offsets, const size_t *dst_dimensions,
     const size_t *src_dimensions, int dst_device, int src_device) {
+  TIMESCOPE();
   DP("Call to omp_target_memcpy_rect, dst device %d, src device %d, "
       "dst addr " DPxMOD ", src addr " DPxMOD ", dst offsets " DPxMOD ", "
       "src offsets " DPxMOD ", dst dims " DPxMOD ", src dims " DPxMOD ", "
       "volume " DPxMOD ", element size %zu, num_dims %d\n", dst_device,
       src_device, DPxPTR(dst), DPxPTR(src), DPxPTR(dst_offsets),
       DPxPTR(src_offsets), DPxPTR(dst_dimensions), DPxPTR(src_dimensions),
       DPxPTR(volume), element_size, num_dims);
 
EXTERN int omp_target_memcpy_rect(void *dst, void *src, size_t element_size,
   }
 
   DP("omp_target_memcpy_rect returns %d\n", rc);
   return rc;
 }
 
 EXTERN int omp_target_associate_ptr(void *host_ptr, void *device_ptr,
     size_t size, size_t device_offset, int device_num) {
+  TIMESCOPE();
   DP("Call to omp_target_associate_ptr with host_ptr " DPxMOD ", "
       "device_ptr " DPxMOD ", size %zu, device_offset %zu, device_num %d\n",
       DPxPTR(host_ptr), DPxPTR(device_ptr), size, device_offset, device_num);
 
   if (!host_ptr || !device_ptr || size <= 0) {
     REPORT("Call to omp_target_associate_ptr with invalid arguments\n");
     return OFFLOAD_FAIL;
   }
EXTERN int omp_target_associate_ptr(void *host_ptr, void *device_ptr,
   DeviceTy &Device = PM->Devices[device_num];
   void *device_addr = (void *)((uint64_t)device_ptr + (uint64_t)device_offset);
   int rc = Device.associatePtr(host_ptr, device_addr, size);
   DP("omp_target_associate_ptr returns %d\n", rc);
   return rc;
 }
 
 EXTERN int omp_target_disassociate_ptr(void *host_ptr, int device_num) {
+  TIMESCOPE();
   DP("Call to omp_target_disassociate_ptr with host_ptr " DPxMOD ", "
       "device_num %d\n", DPxPTR(host_ptr), device_num);
 
   if (!host_ptr) {
     REPORT("Call to omp_target_associate_ptr with invalid host_ptr\n");
     return OFFLOAD_FAIL;
   }
 

openmp/libomptarget/src/interface.cpp

case tgt_mandatory:
     }
     break;
   }
 }
 
 ////////////////////////////////////////////////////////////////////////////////
 /// adds requires flags
 EXTERN void __tgt_register_requires(int64_t flags) {
+  TIMESCOPE();
   PM->RTLs.RegisterRequires(flags);
 }
 
 ////////////////////////////////////////////////////////////////////////////////
 /// adds a target shared library to the target execution image
 EXTERN void __tgt_register_lib(__tgt_bin_desc *desc) {
+  TIMESCOPE();
   PM->RTLs.RegisterLib(desc);
 }
 
 ////////////////////////////////////////////////////////////////////////////////
 /// unloads a target shared library
 EXTERN void __tgt_unregister_lib(__tgt_bin_desc *desc) {
+  TIMESCOPE();
   PM->RTLs.UnregisterLib(desc);
 }
 
 /// creates host-to-target data mapping, stores it in the
 /// libomptarget.so internal structure (an entry in a stack of data maps)
 /// and passes the data to the device.
 EXTERN void __tgt_target_data_begin(int64_t device_id, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types) {
+  TIMESCOPE();
   __tgt_target_data_begin_mapper(nullptr, device_id, arg_num, args_base, args,
                                  arg_sizes, arg_types, nullptr, nullptr);
 }
 
 EXTERN void __tgt_target_data_begin_nowait(int64_t device_id, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types,
     int32_t depNum, void *depList, int32_t noAliasDepNum,
     void *noAliasDepList) {
+  TIMESCOPE();
   if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(NULL, __kmpc_global_thread_num(NULL));
 
   __tgt_target_data_begin_mapper(nullptr, device_id, arg_num, args_base, args,
                                  arg_sizes, arg_types, nullptr, nullptr);
 }
 
 EXTERN void __tgt_target_data_begin_mapper(ident_t *loc, int64_t device_id,
                                            int32_t arg_num, void **args_base,
                                            void **args, int64_t *arg_sizes,
                                            int64_t *arg_types,
                                            map_var_info_t *arg_names,
                                            void **arg_mappers) {
+  TIMESCOPE();
   if (IsOffloadDisabled()) return;
 
   DP("Entering data begin region for device %" PRId64 " with %d mappings\n",
       device_id, arg_num);
 
   // No devices available?
   if (device_id == OFFLOAD_DEVICE_DEFAULT) {
     device_id = omp_get_default_device();
#endif
   HandleTargetOutcome(rc == OFFLOAD_SUCCESS, loc);
 }
 
 EXTERN void __tgt_target_data_begin_nowait_mapper(
     ident_t *loc, int64_t device_id, int32_t arg_num, void **args_base,
     void **args, int64_t *arg_sizes, int64_t *arg_types,
     map_var_info_t *arg_names, void **arg_mappers, int32_t depNum,
     void *depList, int32_t noAliasDepNum, void *noAliasDepList) {
+  TIMESCOPE();
   if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(loc, __kmpc_global_thread_num(loc));
 
   __tgt_target_data_begin_mapper(loc, device_id, arg_num, args_base, args,
                                  arg_sizes, arg_types, arg_names, arg_mappers);
 }
 
 /// passes data from the target, releases target memory and destroys
 /// the host-target mapping (top entry from the stack of data maps)
 /// created by the last __tgt_target_data_begin.
 EXTERN void __tgt_target_data_end(int64_t device_id, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types) {
+  TIMESCOPE();
   __tgt_target_data_end_mapper(nullptr, device_id, arg_num, args_base, args,
                                arg_sizes, arg_types, nullptr, nullptr);
 }
 
 EXTERN void __tgt_target_data_end_nowait(int64_t device_id, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types,
     int32_t depNum, void *depList, int32_t noAliasDepNum,
     void *noAliasDepList) {
+  TIMESCOPE();
   if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(NULL, __kmpc_global_thread_num(NULL));
 
   __tgt_target_data_end_mapper(nullptr, device_id, arg_num, args_base, args,
                                arg_sizes, arg_types, nullptr, nullptr);
 }
 
 EXTERN void __tgt_target_data_end_mapper(ident_t *loc, int64_t device_id,
                                          int32_t arg_num, void **args_base,
                                          void **args, int64_t *arg_sizes,
                                          int64_t *arg_types,
                                          map_var_info_t *arg_names,
                                          void **arg_mappers) {
+  TIMESCOPE();
   if (IsOffloadDisabled()) return;
   DP("Entering data end region with %d mappings\n", arg_num);
 
   // No devices available?
   if (device_id == OFFLOAD_DEVICE_DEFAULT) {
     device_id = omp_get_default_device();
   }
 
#endif
   HandleTargetOutcome(rc == OFFLOAD_SUCCESS, loc);
 }
 
 EXTERN void __tgt_target_data_end_nowait_mapper(
     ident_t *loc, int64_t device_id, int32_t arg_num, void **args_base,
     void **args, int64_t *arg_sizes, int64_t *arg_types,
     map_var_info_t *arg_names, void **arg_mappers, int32_t depNum,
     void *depList, int32_t noAliasDepNum, void *noAliasDepList) {
+  TIMESCOPE();
   if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(loc, __kmpc_global_thread_num(loc));
 
   __tgt_target_data_end_mapper(loc, device_id, arg_num, args_base, args,
                                arg_sizes, arg_types, arg_names, arg_mappers);
 }
 
 EXTERN void __tgt_target_data_update(int64_t device_id, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types) {
+  TIMESCOPE();
   __tgt_target_data_update_mapper(nullptr, device_id, arg_num, args_base, args,
                                   arg_sizes, arg_types, nullptr, nullptr);
 }
 
 EXTERN void __tgt_target_data_update_nowait(int64_t device_id, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types,
     int32_t depNum, void *depList, int32_t noAliasDepNum,
     void *noAliasDepList) {
+  TIMESCOPE();
   if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(NULL, __kmpc_global_thread_num(NULL));
 
   __tgt_target_data_update_mapper(nullptr, device_id, arg_num, args_base, args,
                                   arg_sizes, arg_types, nullptr, nullptr);
 }
 
 EXTERN void __tgt_target_data_update_mapper(ident_t *loc, int64_t device_id,
                                             int32_t arg_num, void **args_base,
                                             void **args, int64_t *arg_sizes,
                                             int64_t *arg_types,
                                             map_var_info_t *arg_names,
                                             void **arg_mappers) {
+  TIMESCOPE();
   if (IsOffloadDisabled()) return;
   DP("Entering data update with %d mappings\n", arg_num);
 
   // No devices available?
   if (device_id == OFFLOAD_DEVICE_DEFAULT) {
     device_id = omp_get_default_device();
   }
 
EXTERN void __tgt_target_data_update_mapper(ident_t *loc, int64_t device_id,
   HandleTargetOutcome(rc == OFFLOAD_SUCCESS, loc);
 }
 
 EXTERN void __tgt_target_data_update_nowait_mapper(
     ident_t *loc, int64_t device_id, int32_t arg_num, void **args_base,
     void **args, int64_t *arg_sizes, int64_t *arg_types,
     map_var_info_t *arg_names, void **arg_mappers, int32_t depNum,
     void *depList, int32_t noAliasDepNum, void *noAliasDepList) {
+  TIMESCOPE();
   if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(loc, __kmpc_global_thread_num(loc));
 
   __tgt_target_data_update_mapper(loc, device_id, arg_num, args_base, args,
                                   arg_sizes, arg_types, arg_names, arg_mappers);
 }
 
 EXTERN int __tgt_target(int64_t device_id, void *host_ptr, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types) {
+  TIMESCOPE();
   return __tgt_target_mapper(nullptr, device_id, host_ptr, arg_num, args_base,
                              args, arg_sizes, arg_types, nullptr, nullptr);
 }
 
 EXTERN int __tgt_target_nowait(int64_t device_id, void *host_ptr,
     int32_t arg_num, void **args_base, void **args, int64_t *arg_sizes,
     int64_t *arg_types, int32_t depNum, void *depList, int32_t noAliasDepNum,
     void *noAliasDepList) {
+  TIMESCOPE();
   if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(NULL, __kmpc_global_thread_num(NULL));
 
   return __tgt_target_mapper(nullptr, device_id, host_ptr, arg_num, args_base,
                              args, arg_sizes, arg_types, nullptr, nullptr);
 }
 
 EXTERN int __tgt_target_mapper(ident_t *loc, int64_t device_id, void *host_ptr,
                                int32_t arg_num, void **args_base, void **args,
                                int64_t *arg_sizes, int64_t *arg_types,
                                map_var_info_t *arg_names, void **arg_mappers) {
+  TIMESCOPE();
   if (IsOffloadDisabled()) return OFFLOAD_FAIL;
   DP("Entering target region with entry point " DPxMOD " and device Id %"
       PRId64 "\n", DPxPTR(host_ptr), device_id);
 
   if (device_id == OFFLOAD_DEVICE_DEFAULT) {
     device_id = omp_get_default_device();
   }
 
#endif
   return rc;
 }
 
 EXTERN int __tgt_target_nowait_mapper(
     ident_t *loc, int64_t device_id, void *host_ptr, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types,
     map_var_info_t *arg_names, void **arg_mappers, int32_t depNum,
     void *depList, int32_t noAliasDepNum, void *noAliasDepList) {
+  TIMESCOPE();
   if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(loc, __kmpc_global_thread_num(loc));
 
   return __tgt_target_mapper(loc, device_id, host_ptr, arg_num, args_base, args,
                              arg_sizes, arg_types, arg_names, arg_mappers);
 }
 
 EXTERN int __tgt_target_teams(int64_t device_id, void *host_ptr,
     int32_t arg_num, void **args_base, void **args, int64_t *arg_sizes,
     int64_t *arg_types, int32_t team_num, int32_t thread_limit) {
+  TIMESCOPE();
   return __tgt_target_teams_mapper(nullptr, device_id, host_ptr, arg_num,
                                    args_base, args, arg_sizes, arg_types,
                                    nullptr, nullptr, team_num, thread_limit);
 }
 
 EXTERN int __tgt_target_teams_nowait(int64_t device_id, void *host_ptr,
     int32_t arg_num, void **args_base, void **args, int64_t *arg_sizes,
     int64_t *arg_types, int32_t team_num, int32_t thread_limit, int32_t depNum,
     void *depList, int32_t noAliasDepNum, void *noAliasDepList) {
+  TIMESCOPE();
   if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(NULL, __kmpc_global_thread_num(NULL));
 
   return __tgt_target_teams_mapper(nullptr, device_id, host_ptr, arg_num,
                                    args_base, args, arg_sizes, arg_types,
                                    nullptr, nullptr, team_num, thread_limit);
 }
 
 EXTERN int __tgt_target_teams_mapper(ident_t *loc, int64_t device_id,
                                      void *host_ptr, int32_t arg_num,
                                      void **args_base, void **args,
                                      int64_t *arg_sizes, int64_t *arg_types,
                                      map_var_info_t *arg_names,
                                      void **arg_mappers, int32_t team_num,
                                      int32_t thread_limit) {
+  TIMESCOPE();
   if (IsOffloadDisabled()) return OFFLOAD_FAIL;
   DP("Entering target region with entry point " DPxMOD " and device Id %"
       PRId64 "\n", DPxPTR(host_ptr), device_id);
 
   if (device_id == OFFLOAD_DEVICE_DEFAULT) {
     device_id = omp_get_default_device();
   }
 

 }
 
 EXTERN int __tgt_target_teams_nowait_mapper(
     ident_t *loc, int64_t device_id, void *host_ptr, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types,
     map_var_info_t *arg_names, void **arg_mappers, int32_t team_num,
     int32_t thread_limit, int32_t depNum, void *depList, int32_t noAliasDepNum,
     void *noAliasDepList) {
+  TIMESCOPE();
   if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(loc, __kmpc_global_thread_num(loc));
 
   return __tgt_target_teams_mapper(loc, device_id, host_ptr, arg_num, args_base,
                                    args, arg_sizes, arg_types, arg_names,
                                    arg_mappers, team_num, thread_limit);
 }
 
 // Get the current number of components for a user-defined mapper.
 EXTERN int64_t __tgt_mapper_num_components(void *rt_mapper_handle) {
+  TIMESCOPE();
   auto *MapperComponentsPtr = (struct MapperComponentsTy *)rt_mapper_handle;
   int64_t size = MapperComponentsPtr->Components.size();
   DP("__tgt_mapper_num_components(Handle=" DPxMOD ") returns %" PRId64 "\n",
      DPxPTR(rt_mapper_handle), size);
   return size;
 }
 
 // Push back one component for a user-defined mapper.
 EXTERN void __tgt_push_mapper_component(void *rt_mapper_handle, void *base,
                                         void *begin, int64_t size,
                                         int64_t type) {
+  TIMESCOPE();
   DP("__tgt_push_mapper_component(Handle=" DPxMOD
      ") adds an entry (Base=" DPxMOD ", Begin=" DPxMOD ", Size=%" PRId64
      ", Type=0x%" PRIx64 ").\n",
      DPxPTR(rt_mapper_handle), DPxPTR(base), DPxPTR(begin), size, type);
   auto *MapperComponentsPtr = (struct MapperComponentsTy *)rt_mapper_handle;
   MapperComponentsPtr->Components.push_back(
       MapComponentInfoTy(base, begin, size, type));
 }
 
 EXTERN void __kmpc_push_target_tripcount(ident_t *loc, int64_t device_id,
                                          uint64_t loop_tripcount) {
+  TIMESCOPE();
   if (IsOffloadDisabled())
     return;
 
   if (device_id == OFFLOAD_DEVICE_DEFAULT) {
     device_id = omp_get_default_device();
   }
 
   if (CheckDeviceAndCtors(device_id) != OFFLOAD_SUCCESS) {

openmp/libomptarget/src/private.h

for (const auto &HostTargetMap : Device.HostDataToTargetMap) {
     fprintf(stderr, DPxMOD " " DPxMOD " %-8lu %s at %s:%d:%d\n",
             DPxPTR(HostTargetMap.HstPtrBegin),
             DPxPTR(HostTargetMap.TgtPtrBegin),
             HostTargetMap.HstPtrEnd - HostTargetMap.HstPtrBegin, info.getName(),
             info.getFilename(), info.getLine(), info.getColumn());
   }
 }
 
+#ifdef OMPTARGET_PROFILE_ENABLED
+#include "llvm/Support/TimeProfiler.h"
+#define TIMESCOPE() llvm::TimeTraceScope TimeScope(__FUNCTION__)
+#else
+#define TIMESCOPE()
+#endif
+
 #endif

openmp/libomptarget/src/rtl.cpp

static const char *RTLNames[] = {
     /* CUDA target          */ "libomptarget.rtl.cuda.so",
     /* AArch64 target       */ "libomptarget.rtl.aarch64.so",
     /* SX-Aurora VE target  */ "libomptarget.rtl.ve.so",
     /* AMDGPU target        */ "libomptarget.rtl.amdgpu.so",
 };
 
 PluginManager *PM;
 
+#if OMPTARGET_PROFILE_ENABLED
+static char *ProfileTraceFile = nullptr;
+#endif
+
 __attribute__((constructor(101))) void init() {
   DP("Init target library!\n");
   PM = new PluginManager();
+
+#ifdef OMPTARGET_PROFILE_ENABLED
+  ProfileTraceFile = getenv("LIBOMPTARGET_PROFILE");
+  // TODO: add a configuration option for time granularity
+  if (ProfileTraceFile)
+    llvm::timeTraceProfilerInitialize(500 /* us */, "libomptarget");
+#endif
 }
 
 __attribute__((destructor(101))) void deinit() {
   DP("Deinit target library!\n");
   delete PM;
+
+#ifdef OMPTARGET_PROFILE_ENABLED
+  if (ProfileTraceFile) {
+    // TODO: add env var for file output
+    if (auto E = llvm::timeTraceProfilerWrite(ProfileTraceFile, "-"))
+      fprintf(stderr, "Error writing out the time trace\n");
+
+    llvm::timeTraceProfilerCleanup();
+  }
+#endif
 }
 
 void RTLsTy::LoadRTLs() {
   // Parse environment variable OMP_TARGET_OFFLOAD (if set)
   PM->TargetOffloadPolicy =
       (kmp_target_offload_kind_t)__kmpc_get_target_offload();
   if (PM->TargetOffloadPolicy == tgt_disabled) {
     return;