This is an archive of the discontinued LLVM Phabricator instance.

[OpenMP] [OMPT] [amdgpu] [4/8] Implemented callback registration in nextgen plugins
ClosedPublic

Authored by mhalk on Apr 19 2022, 11:46 PM.

Details

Summary

The purpose of this patch is to Implement registration of callback functions in the generic plugin by looking up corresponding callbacks in libomptarget. The overall design document is https://rice.app.box.com/s/pf3gix2hs4d4o1aatwir1set05xmjljc

Defined an object of type OmptDeviceCallbacksTy in the amdgpu plugin for holding the tool-provided callback functions. Implemented a global constructor in the plugin that creates a connector object to connect with libomptarget. The callbacks that are already registered with libomptarget are looked up and registered with the plugin.

Combined with an internal patch from Dhruva Chakrabarti, which fixes the OMPT initialization ordering.
Achieved through removal of the constructor attribute from ompt_init.

Patch from John Mellor-Crummey <johnmc@rice.edu>
With contributions from:
Dhruva Chakrabarti <Dhruva.Chakrabarti@amd.com>
Michael Halkenhaeuser <MichaelGerald.Halkenhauser@amd.com>

Diff Detail

Event Timeline

dhruvachak created this revision.Apr 19 2022, 11:46 PM
Herald added a project: Restricted Project. · View Herald TranscriptApr 19 2022, 11:46 PM
Herald added subscribers: kerbowa, t-tye, tpr and 5 others. · View Herald Transcript
dhruvachak requested review of this revision.Apr 19 2022, 11:46 PM
Herald added a project: Restricted Project. · View Herald TranscriptApr 19 2022, 11:46 PM
dhruvachak retitled this revision from [OMPT] [amdgpu] Implemented callback registration in amdgpu plugin to [OpenMP] [OMPT] [amdgpu] [4/8] Implemented callback registration in amdgpu plugin.Jun 20 2022, 7:58 PM
dhruvachak edited the summary of this revision. (Show Details)
jplehr added a subscriber: jplehr.Dec 1 2022, 8:04 AM
jplehr added inline comments.Dec 1 2022, 8:21 AM
openmp/libomptarget/plugins/amdgpu/src/ompt_callback.cpp
57 ↗(On Diff #438531)

Minor change to be consistent with other functions printing the function name in these debug messages.

dhruvachak edited the summary of this revision. (Show Details)Feb 9 2023, 9:11 AM
jplehr updated this revision to Diff 496363.Feb 10 2023, 12:52 AM

Updating D124070: [OpenMP] [OMPT] [amdgpu] [4/8] Implemented callback registration in amdgpu plugin

Rebase and follow naming scheme of already landed patches.

dhruvachak added inline comments.Feb 10 2023, 9:31 AM
openmp/libomptarget/plugins/amdgpu/src/ompt_callback.cpp
22 ↗(On Diff #496363)

We use OmptDeviceCallbacks variable name in libomptarget. Let's keep it consistent.

35 ↗(On Diff #496363)

We should be able to move the ifdef to the top. Let's keep it consistent with libomptarget.

jplehr updated this revision to Diff 496580.Feb 10 2023, 12:18 PM

Addressing feedback from Dhruva

jplehr updated this revision to Diff 497062.Feb 13 2023, 11:55 AM

Address compile error in debug build.

mhalk updated this revision to Diff 499474.Feb 22 2023, 6:34 AM
mhalk added a subscriber: mhalk.

Update

Fix OMPT init ordering and library typo.

mhalk updated this revision to Diff 499481.Feb 22 2023, 6:46 AM

Fixed wrong diff update.

jhuber6 added a subscriber: jhuber6.Mar 3 2023, 6:59 AM

These need to be updated for the next-gen plugins.

mhalk updated this revision to Diff 506965.Mar 21 2023, 7:07 AM

Refactored patch for use with nextgen plugins.

mhalk edited the summary of this revision. (Show Details)Mar 21 2023, 8:24 AM
mhalk added reviewers: jplehr, dhruvachak.
jplehr added inline comments.Mar 21 2023, 8:39 AM
openmp/libomptarget/plugins-nextgen/common/OMPT/OmptCallback.cpp
43

Functions and methods should start with a lower case letter and read as active voice, e.g., initializeOmptDevice.

mhalk commandeered this revision.Mar 21 2023, 8:39 AM

As discussed with Dhruva.

dhruvachak added inline comments.Mar 21 2023, 10:25 PM
openmp/libomptarget/plugins-nextgen/common/OMPT/OmptCallback.cpp
68

The comment says it is a constructor but I don't see the attribute. Are you calling it directly from elsewhere in a subsequent patch?

mhalk added inline comments.Mar 22 2023, 2:26 AM
openmp/libomptarget/plugins-nextgen/common/OMPT/OmptCallback.cpp
68

Yes, it will be called from PluginInterface (or GenericPluginTy::init() to be more precise) in patch 5/8 (https://reviews.llvm.org/D124652).
I'll remove the constructor comment.

jdoerfert added inline comments.Mar 22 2023, 5:07 PM
openmp/libomptarget/plugins-nextgen/common/OMPT/OmptCallback.cpp
23

Can we put this stuff into a namespace (ompt::) rather than prefixing it with Ompt?

72

Multiple plugins can be active, are you sure you want static variables here?

openmp/libomptarget/src/rtl.cpp
75

Is this patch run through clang-format (only the patch part)?

jdoerfert retitled this revision from [OpenMP] [OMPT] [amdgpu] [4/8] Implemented callback registration in amdgpu plugin to [OpenMP] [OMPT] [amdgpu] [4/8] Implemented callback registration in nextgen plugins.Mar 23 2023, 10:03 AM
mhalk updated this revision to Diff 517237.Apr 26 2023, 10:37 AM

Corrected commit (apologies)

dhruvachak added inline comments.Apr 27 2023, 9:34 AM
openmp/libomptarget/plugins-nextgen/common/OMPT/OmptCallback.cpp
68

I think LibomptargetConnector and OmptResult need not be statics, locals will suffice. The connector object is used to copy the callback function pointers from libomptarget and once that is done, the object is not required any more. The original intent of the static was to guard against accidental repeat connect attempts. If we add the assert to the finalizer like I suggested, that will still be guarded against. Regarding OmptResult, it is used to communicate some pointers to the other library which caches the pointers, so no need to keep the wrapper object around.

We could make similar changes to the connect logic in libomptarget and add an assert during finalizer registration in libomp but let's keep that for a separate commit, if at all.

openmp/libomptarget/src/OmptCallback.cpp
39

In registerRtl, before assigning to FiniFn, add an assert that it is nullptr.

mhalk updated this revision to Diff 517649.EditedApr 27 2023, 11:06 AM

Implemented further feedback (removed statics & added corresponding assert)
edit: Please note that the namespace feedback has been addressed in the "next" patch in this stack: D12465

dhruvachak accepted this revision.Apr 27 2023, 11:12 AM

LGTM. I understand that the namespace changes are already made in https://reviews.llvm.org/D124652 I think all other comments have been resolved.

@jdoerfert Okay to land this patch?

This revision is now accepted and ready to land.Apr 27 2023, 11:12 AM
mhalk updated this revision to Diff 518769.May 2 2023, 9:24 AM

Small fix w.r.t. debug builds.

I encountered a compile error when building openmp.

[4/143] Building CXX object libomptarget/src/CMakeFiles/omptarget.dir/OmptCallback.cpp.o
FAILED: libomptarget/src/CMakeFiles/omptarget.dir/OmptCallback.cpp.o
/gpfs/jlse-fs0/users/ac.shilei.tian/build/llvm/release/bin/clang++ -DGTEST_HAS_RTTI=0 -DOMPTARGET_DEBUG -I/home/ac.shilei.tian/Documents/vscode/llvm-project/llvm/include -I/gpfs/jlse-fs0/users/ac.shilei.tian/build/llvm/release/include -I/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-enum-constexpr-conversion -Wno-extra -Wno-pedantic -std=c++17 -g -fPIC  -fno-exceptions -funwind-tables -fno-rtti -MD -MT libomptarget/src/CMakeFiles/omptarget.dir/OmptCallback.cpp.o -MF libomptarget/src/CMakeFiles/omptarget.dir/OmptCallback.cpp.o.d -o libomptarget/src/CMakeFiles/omptarget.dir/OmptCallback.cpp.o -c /home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:124:32: error: unknown type name 'ompt_start_tool_result_t'
void ompt_libomptarget_connect(ompt_start_tool_result_t *result) {
                               ^
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:125:3: error: use of undeclared identifier 'DP'
  DP("OMPT: Enter ompt_libomptarget_connect\n");
  ^
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:126:7: error: use of undeclared identifier 'OmptEnabled'
  if (OmptEnabled && result) {
      ^
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:128:5: error: use of undeclared identifier 'LibraryFinalizer'
    LibraryFinalizer.registerRtl(result->finalize);
    ^
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:132:24: error: use of undeclared identifier 'OmptDeviceCallbacksTy'
    result->initialize(OmptDeviceCallbacksTy::doLookup,
                       ^
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:135:3: error: use of undeclared identifier 'DP'
  DP("OMPT: Leave ompt_libomptarget_connect\n");
  ^
6 errors generated.

Do we want to revert it or a quick fix will be landed soon?

tianshilei1992 added a comment.EditedMay 2 2023, 11:31 AM

Okay, I see why it fails here. ompt_start_tool_result_t is in openmp/runtime/src/include/omp-tools.h.var, which will be generated when libomp is built. This makes it dependent on libomp, but the dependences are not set. I'm gonna revert the patch for now. I have another patch (https://reviews.llvm.org/D149617) that will make libomptarget depend on libomp, which can solve the issue automatically. It is reverted because it broke something but it will be landed soon.
Another thing is, I disabled OMPT on my side, and OmptCallback.cpp is added to the source file without any guard. That has to be fixed as well.

[AMD Official Use Only - General]

Revert sounds good, thx

Like that you have a related patch queued up that will help fix OMPT-4 patch.

mhalk added a comment.May 2 2023, 12:47 PM

@tianshilei1992 Thank you very much!
Sorry that I was unavailable for the last hour, just got at the PC.

I'll have a look at the errors and see how I can reproduce them, but I'm honestly surprised (since I built this on multiple systems without problems).

Another thing is, I disabled OMPT on my side, and OmptCallback.cpp is added to the source file without any guard. That has to be fixed as well.

Just to make sure, you're talking about a guard that e.g. prevents OmptCallback.cpp to produce an object file when OMPT is not enabled (e.g. in the CMakeLists), correct?

dhruvachak added inline comments.May 2 2023, 1:03 PM
openmp/libomptarget/src/OmptCallback.cpp
141

This should be at the end of this file since omp-tools.h is included depending on whether OMPT is enabled. Missed this earlier.

Either only adding the source file when OMPT is enabled, or as suggested by @dhruvachak, guarding the entire file with the OMPT_SUPPORT.

mhalk added a comment.May 2 2023, 1:31 PM

Either only adding the source file when OMPT is enabled, or as suggested by @dhruvachak, guarding the entire file with the OMPT_SUPPORT.

Thanks for getting back so fast -- I'll look into the latter approach of moving #endif // OMPT_SUPPORT to the bottom of OmptCallback.cpp and re-check with different configurations (esp. disabled OMPT).

mhalk reopened this revision.May 3 2023, 6:58 AM
This revision is now accepted and ready to land.May 3 2023, 6:58 AM
mhalk updated this revision to Diff 519062.May 3 2023, 7:10 AM

Made #ifdef span OmptCallback.cpp to avoid build errors when OMPT is disabled.
Added dummy definition for exported symbol ompt_libomptarget_connect.

tianshilei1992 added inline comments.May 3 2023, 7:21 AM
openmp/libomptarget/plugins-nextgen/amdgpu/CMakeLists.txt
85

A better practice is to link OMPT if it is enabled.

if (OMPT_ENABLED)
  target_link_libraries(omptarget.rtl.amdgpu.nextgen PRIVATE OMPT)
endif()
mhalk added a comment.EditedMay 3 2023, 7:27 AM

If there are no further problems or requests, I would like to re-land this revision.

openmp/libomptarget/plugins-nextgen/amdgpu/CMakeLists.txt
85

Thank you, will be addressed!

tianshilei1992 added inline comments.May 3 2023, 8:38 AM
openmp/libomptarget/src/OmptCallback.cpp
137

Will it cause some link error if ompt_libomptarget_connect is not defined but mentioned in exports?

mhalk added inline comments.May 3 2023, 8:58 AM
openmp/libomptarget/src/OmptCallback.cpp
137

Yes, also with the guarded target_link_libraries(...) you suggested -- I just tried it.
If there's another approach or better practice, I'd be very interested.

dhruvachak added inline comments.May 3 2023, 9:08 AM
openmp/libomptarget/plugins-nextgen/amdgpu/CMakeLists.txt
85

For this to work, you probably want to check
if ((OMPT_TARGET_DEFAULT) AND (LIBOMPTARGET_OMPT_SUPPORT))
the same way libomptarget cmake does.

The former needs to be exported to global scope first in libomptarget. The latter should already be available to the plugin.

mhalk updated this revision to Diff 519132.May 3 2023, 9:42 AM

Added conditional linking of OMPT to PluginInterface as well as amdgpu and cuda plugins`.

mhalk added inline comments.May 3 2023, 9:48 AM
openmp/libomptarget/plugins-nextgen/amdgpu/CMakeLists.txt
85

Fhe correct values were available (without previous export of OMPT_TARGET_DEFAULT to global scope) when I checked against the different configurations -DLIBOMP[TARGET]_OMPT_SUPPORT=ON|OFF within CMakeLists.txt of PluginInterface and the amdgpu plugin.

mhalk added a comment.May 4 2023, 1:01 PM

@tianshilei1992 If it is not too much to ask for: Could you please build the current patch with your configuration and let me know if there were issues?

So far, I have not encountered any problems and would like to re-land this revision.

tianshilei1992 accepted this revision.May 4 2023, 2:12 PM

I think the patch looks good, though I didn't test it yet. I don't want to block here. Go for it. :-)

mhalk added a comment.May 5 2023, 4:20 AM

I think the patch looks good, though I didn't test it yet. I don't want to block here. Go for it. :-)

Much appreciated -- thank you!
However, I wanted to be available if there should be issues.