Add support for ompt_event_task_dependences and ompt_event_task_dependence_pair

[harald.servat]

Summary

The attached patch adds support for ompt_event_task_dependences and ompt_event_task_dependence_pair events from the OMPT specification [1]. These events only apply to OpenMP 4.0 and 4.1 (aka 4.5) because task dependencies were introduced in 4.0.

With respect to the changes:

ompt_event_task_dependences
According to the specification, this event is raised after the task has been created, thefore this event needs to be raised after ompt_event_task_begin (in _kmp_task_start). However, the dependencies are known at _kmpc_omp_task_with_deps which occurs before _kmp_task_start. My modifications extend the ompt_task_info_t struct in order to store the dependencies of the task when _kmpc_omp_task_with_deps occurs and then they are emitted in _kmp_task_start just after raising the ompt_event_task_begin. The deps field is allocated and valid until the event is raised and it is freed and set to null afterwards.

ompt_event_task_dependence_pair
The processing of the dependences (i.e. checking whenever a dependence is already satisfied) is done within _kmp_process_deps. That function checks every dependence and calls the _kmp_track_dependence routine which gives some support for graphical output. I used that routine to emit the dependence pair but I also needed to know the sink_task. Despite the fact that the code within KMP_SUPPORT_GRAPH_OUTPUT refers to task_sink it may be null because sink->dn.task (there's a comment regarding this) and in fact it does not point to a proper pointer value because the value is set in node->dn.task = task; after the _kmp_process_deps calls in _kmp_check_deps. I have extended the _kmp_process_deps and _kmp_track_dependence parameter list to receive the sink_task.

PS: I have removed the extra leading underscore from the kmp calls because they appeared as underlined.

[1] https://github.com/OpenMPToolsInterface/OMPT-Technical-Report/blob/target/ompt-tr.pdf

[jmellorcrummey]

Overall, the patch looks good. Sorry that it has taken me so long to review it.

A few minor quibbles:

The # if USE_FAST_MEMORY is used to in two places to select between allocation and free routines. I think it would be better to use it in only one place and say

#if USE_FAST_MEMORY
#define KMP_OMPT_DEPS_ALLOC kmp_fast_allocate
#define KMP_OMPT_DEPS_FREE kmp_fast_free
#else
#define KMP_OMPT_DEPS_ALLOC kmp_thread_malloc
#define KMP_OMPT_DEPS_FREE kmp_thread_free
#endif

and then use the macros defined above in the OMPT code. Note: in the above definitions, I omitted the leading underscores, which created underlined phrases in the formatting.

The comments in the ompt.h.var files shouldn't say "new task dependences" and "new task dependence pair". It would be better to say "report task dependences" and "report task dependence pair".

[harald.servat]

I have addressed John M. Crummey's comments.

I have added the macros KMP_OMPT_DEPS_ALLOC/KMP_OMPT_DEPS_FREE into ompt-internal.h. I also have modified the comments regarding the added events into the ompt.h.var files.

[jmellorcrummey]

LGTM.

[Hahnfeld]

Changes LGTM as well but compilation fails with LIBOMP_OMPT_SUPPORT=ON and LIBOMP_OMP_VERSION=30 (rather theoretical combination).

(Note: kmp_taskdeps.cpp is only compiled if LIBOMP_OMP_VERSION >= 40 so you don't have to guard this)

runtime/src/kmp_tasking.c

466–481

Please guard with #if OMP_40_ENABLED

781–782

see above

runtime/src/ompt-event-specific.h

144–145

see above

runtime/src/ompt-internal.h

32–33

see above

69–77

see above

[harald.servat]

I have addressed Jonas Hahnfeld by guarding the code with OMP_40_ENABLED.

My only concern is regarding runtime/src/ompt-event-specific.h. In that file, I have added the guard but I also provided the #else condition to mark that these events cannot be provided if OMP40 is not enabled.

[Hahnfeld]

LGTM. Shall I commit for you?

In D14746#338148, @harald.servat wrote:

I have addressed Jonas Hahnfeld by guarding the code with OMP_40_ENABLED.

My only concern is regarding runtime/src/ompt-event-specific.h. In that file, I have added the guard but I also provided the #else condition to mark that these events cannot be provided if OMP40 is not enabled.

Good idea, this makes it explicitely clear that this is not implemented for versions prior to 4.0 (although the defines will probably never be used in this case...)

[harald.servat]

In D14746#338164, @Hahnfeld wrote:

LGTM. Shall I commit for you?

Yes, please. Thank you very much!

In D14746#338148, @harald.servat wrote:

I have addressed Jonas Hahnfeld by guarding the code with OMP_40_ENABLED.

My only concern is regarding runtime/src/ompt-event-specific.h. In that file, I have added the guard but I also provided the #else condition to mark that these events cannot be provided if OMP40 is not enabled.

Good idea, this makes it explicitely clear that this is not implemented for versions prior to 4.0 (although the defines will probably never be used in this case...)

runtime/src/include/40/ompt.h.var

#define FOREACH_OMPT_EVENT(macro)                                                                               \
     macro (ompt_event_acquired_ordered,         ompt_wait_callback_t,          56) /* ordered acquired */       \
                                                                                                                 \
     macro (ompt_event_init_lock,                ompt_wait_callback_t,          57) /* lock init */              \
     macro (ompt_event_init_nest_lock,           ompt_wait_callback_t,          58) /* nest lock init */         \
                                                                                                                 \
     macro (ompt_event_destroy_lock,             ompt_wait_callback_t,          59) /* lock destruction */       \
     macro (ompt_event_destroy_nest_lock,        ompt_wait_callback_t,          60) /* nest lock destruction */  \
                                                                                                                 \
-    macro (ompt_event_flush,                    ompt_callback_t,               61) /* after executing flush */
+    macro (ompt_event_flush,                    ompt_callback_t,               61) /* after executing flush */  \
+                                                                                                                \
+    macro (ompt_event_task_dependences,         ompt_task_dependences_callback_t, 69) /* report task dependences  */\
+    macro (ompt_event_task_dependence_pair,     ompt_task_pair_callback_t,     70) /* report task dependence pair */
 
 
 
 /*****************************************************************************
  * data types
  *****************************************************************************/
 
 /*---------------------

  *---------------------*/
 
 typedef struct ompt_frame_s {
     void *exit_runtime_frame;    /* next frame is user code     */
     void *reenter_runtime_frame; /* previous frame is user code */
 } ompt_frame_t;
 
 
+/*---------------------
+ * dependences types
+ *---------------------*/
+
+typedef enum ompt_task_dependence_flag_e {
+    // a two bit field for the dependence type
+    ompt_task_dependence_type_out   = 1,
+    ompt_task_dependence_type_in    = 2,
+    ompt_task_dependence_type_inout = 3,
+} ompt_task_dependence_flag_t;
+
+typedef struct ompt_task_dependence_s {
+    void *variable_addr;
+    uint32_t  dependence_flags;
+} ompt_task_dependence_t;
+
+
 /*****************************************************************************
  * enumerations for thread states and runtime events
  *****************************************************************************/
 
 /*---------------------
  * runtime states
  *---------------------*/
 

 
 typedef void (*ompt_new_task_callback_t) (
     ompt_task_id_t parent_task_id,    /* id of parent task            */
     ompt_frame_t *parent_task_frame,  /* frame data for parent task   */
     ompt_task_id_t  new_task_id,      /* id of created task           */
     void *task_function               /* pointer to outlined function */
 );
 
+/* task dependences */
+typedef void (*ompt_task_dependences_callback_t) (
+    ompt_task_id_t task_id,            /* ID of task with dependences */
+    const ompt_task_dependence_t *deps,/* vector of task dependences  */
+    int ndeps                          /* number of dependences       */
+);
+
 /* program */
 typedef void (*ompt_control_callback_t) (
     uint64_t command,                 /* command of control call      */
     uint64_t modifier                 /* modifier of control call     */
 );
 
 typedef void (*ompt_callback_t)(void);
 

runtime/src/include/41/ompt.h.var

#define FOREACH_OMPT_EVENT(macro)                                                                               \
     macro (ompt_event_acquired_ordered,         ompt_wait_callback_t,          56) /* ordered acquired */       \
                                                                                                                 \
     macro (ompt_event_init_lock,                ompt_wait_callback_t,          57) /* lock init */              \
     macro (ompt_event_init_nest_lock,           ompt_wait_callback_t,          58) /* nest lock init */         \
                                                                                                                 \
     macro (ompt_event_destroy_lock,             ompt_wait_callback_t,          59) /* lock destruction */       \
     macro (ompt_event_destroy_nest_lock,        ompt_wait_callback_t,          60) /* nest lock destruction */  \
                                                                                                                 \
-    macro (ompt_event_flush,                    ompt_callback_t,               61) /* after executing flush */
+    macro (ompt_event_flush,                    ompt_callback_t,               61) /* after executing flush */  \
+                                                                                                                \
+    macro (ompt_event_task_dependences,         ompt_task_dependences_callback_t, 69) /* report task dependences  */\
+    macro (ompt_event_task_dependence_pair,     ompt_task_pair_callback_t,     70) /* report task dependence pair */
 
 
 
 /*****************************************************************************
  * data types
  *****************************************************************************/
 
 /*---------------------

  *---------------------*/
 
 typedef struct ompt_frame_s {
     void *exit_runtime_frame;    /* next frame is user code     */
     void *reenter_runtime_frame; /* previous frame is user code */
 } ompt_frame_t;
 
 
+/*---------------------
+ * dependences types
+ *---------------------*/
+
+typedef enum ompt_task_dependence_flag_e {
+    // a two bit field for the dependence type
+    ompt_task_dependence_type_out   = 1,
+    ompt_task_dependence_type_in    = 2,
+    ompt_task_dependence_type_inout = 3,
+} ompt_task_dependence_flag_t;
+
+typedef struct ompt_task_dependence_s {
+    void *variable_addr;
+    uint32_t  dependence_flags;
+} ompt_task_dependence_t;
+
+
 /*****************************************************************************
  * enumerations for thread states and runtime events
  *****************************************************************************/
 
 /*---------------------
  * runtime states
  *---------------------*/
 

 
 typedef void (*ompt_new_task_callback_t) (
     ompt_task_id_t parent_task_id,    /* id of parent task            */
     ompt_frame_t *parent_task_frame,  /* frame data for parent task   */
     ompt_task_id_t  new_task_id,      /* id of created task           */
     void *task_function               /* pointer to outlined function */
 );
 
+/* task dependences */
+typedef void (*ompt_task_dependences_callback_t) (
+    ompt_task_id_t task_id,            /* ID of task with dependences */
+    const ompt_task_dependence_t *deps,/* vector of task dependences  */
+    int ndeps                          /* number of dependences       */
+);
+
 /* program */
 typedef void (*ompt_control_callback_t) (
     uint64_t command,                 /* command of control call      */
     uint64_t modifier                 /* modifier of control call     */
 );
 
 typedef void (*ompt_callback_t)(void);
 

runtime/src/kmp_taskdeps.cpp

#if USE_FAST_MEMORY
         __kmp_fast_free(thread,list);
 #else
         __kmp_thread_free(thread,list);
 #endif
     }
 }
 
 static inline void
-__kmp_track_dependence ( kmp_depnode_t *source, kmp_depnode_t *sink )
+__kmp_track_dependence ( kmp_depnode_t *source, kmp_depnode_t *sink,
+                         kmp_task_t *sink_task )
 {
 #ifdef KMP_SUPPORT_GRAPH_OUTPUT
     kmp_taskdata_t * task_source = KMP_TASK_TO_TASKDATA(source->dn.task);
     kmp_taskdata_t * task_sink = KMP_TASK_TO_TASKDATA(sink->dn.task);    // this can be NULL when if(0) ...
 
     __kmp_printf("%d(%s) -> %d(%s)\n", source->dn.id, task_source->td_ident->psource, sink->dn.id, task_sink->td_ident->psource);
 #endif
+#if OMPT_SUPPORT && OMPT_TRACE
+    /* OMPT tracks dependences between task (a=source, b=sink) in which
+       task a blocks the execution of b through the ompt_new_dependence_callback */
+    if (ompt_enabled &&
+        ompt_callbacks.ompt_callback(ompt_event_task_dependence_pair))
+    {
+        kmp_taskdata_t * task_source = KMP_TASK_TO_TASKDATA(source->dn.task);
+        kmp_taskdata_t * task_sink = KMP_TASK_TO_TASKDATA(sink_task);
+
+        ompt_callbacks.ompt_callback(ompt_event_task_dependence_pair)(
+          task_source->ompt_task_info.task_id,
+          task_sink->ompt_task_info.task_id);
+    }
+#endif /* OMPT_SUPPORT && OMPT_TRACE */
 }
 
 template< bool filter >
 static inline kmp_int32
 __kmp_process_deps ( kmp_int32 gtid, kmp_depnode_t *node, kmp_dephash_t *hash,
-                     bool dep_barrier,kmp_int32 ndeps, kmp_depend_info_t *dep_list)
+                     bool dep_barrier,kmp_int32 ndeps, kmp_depend_info_t *dep_list,
+                     kmp_task_t *task )
 {
     KA_TRACE(30, ("__kmp_process_deps<%d>: T#%d processing %d depencies : dep_barrier = %d\n", filter, gtid, ndeps, dep_barrier ) );
     
     kmp_info_t *thread = __kmp_threads[ gtid ];
     kmp_int32 npredecessors=0;
     for ( kmp_int32 i = 0; i < ndeps ; i++ ) {
         const kmp_depend_info_t * dep = &dep_list[i];
 
         KMP_DEBUG_ASSERT(dep->flags.in);
 
         if ( filter && dep->base_addr == 0 ) continue; // skip filtered entries
 
         kmp_dephash_entry_t *info = __kmp_dephash_find(thread,hash,dep->base_addr);
         kmp_depnode_t *last_out = info->last_out;
 
         if ( dep->flags.out && info->last_ins ) {
             for ( kmp_depnode_list_t * p = info->last_ins; p; p = p->next ) {
                 kmp_depnode_t * indep = p->node;
                 if ( indep->dn.task ) {
                     KMP_ACQUIRE_DEPNODE(gtid,indep);
                     if ( indep->dn.task ) {
-                        __kmp_track_dependence(indep,node);
+                        __kmp_track_dependence(indep,node,task);
                         indep->dn.successors = __kmp_add_node(thread, indep->dn.successors, node);
                         KA_TRACE(40,("__kmp_process_deps<%d>: T#%d adding dependence from %p to %p",
                                  filter,gtid, KMP_TASK_TO_TASKDATA(indep->dn.task), KMP_TASK_TO_TASKDATA(node->dn.task)));
                         npredecessors++;
                     }
                     KMP_RELEASE_DEPNODE(gtid,indep);
                 }
             }
 
             __kmp_depnode_list_free(thread,info->last_ins);
             info->last_ins = NULL;
 
         } else if ( last_out && last_out->dn.task ) {
             KMP_ACQUIRE_DEPNODE(gtid,last_out);
             if ( last_out->dn.task ) {
-                __kmp_track_dependence(last_out,node);
+                __kmp_track_dependence(last_out,node,task);
                 last_out->dn.successors = __kmp_add_node(thread, last_out->dn.successors, node);
                 KA_TRACE(40,("__kmp_process_deps<%d>: T#%d adding dependence from %p to %p", 
                              filter,gtid, KMP_TASK_TO_TASKDATA(last_out->dn.task), KMP_TASK_TO_TASKDATA(node->dn.task)));
                 
                 npredecessors++;
             }
             KMP_RELEASE_DEPNODE(gtid,last_out);
         }
#endif
 
     // doesn't need to be atomic as no other thread is going to be accessing this node just yet
     // npredecessors is set -1 to ensure that none of the releasing tasks queues this task before we have finished processing all the dependencies
     node->dn.npredecessors = -1;
 
     // used to pack all npredecessors additions into a single atomic operation at the end
     int npredecessors;
 
-    npredecessors = __kmp_process_deps<true>(gtid, node, hash, dep_barrier, ndeps, dep_list);
-    npredecessors += __kmp_process_deps<false>(gtid, node, hash, dep_barrier, ndeps_noalias, noalias_dep_list);
+    npredecessors = __kmp_process_deps<true>(gtid, node, hash, dep_barrier,
+      ndeps, dep_list, task);
+    npredecessors += __kmp_process_deps<false>(gtid, node, hash, dep_barrier,
+      ndeps_noalias, noalias_dep_list, task);
 
     node->dn.task = task;
     KMP_MB();
 
     // Account for our initial fake value
     npredecessors++;
 
     // Update predecessors and obtain current value to check if there are still any outstandig dependences (some tasks may have finished while we processed the dependences)
__kmpc_omp_task_with_deps( ident_t *loc_ref, kmp_int32 gtid, kmp_task_t * new_task,
 
     kmp_taskdata_t * new_taskdata = KMP_TASK_TO_TASKDATA(new_task);
     KA_TRACE(10, ("__kmpc_omp_task_with_deps(enter): T#%d loc=%p task=%p\n",
                   gtid, loc_ref, new_taskdata ) );
 
     kmp_info_t *thread = __kmp_threads[ gtid ];
     kmp_taskdata_t * current_task = thread->th.th_current_task;
 
+#if OMPT_SUPPORT && OMPT_TRACE
+    /* OMPT grab all dependences if requested by the tool */
+    if (ompt_enabled && ndeps+ndeps_noalias > 0 &&
+        ompt_callbacks.ompt_callback(ompt_event_task_dependences))
+  {
+        kmp_int32 i;
+
+        new_taskdata->ompt_task_info.ndeps = ndeps+ndeps_noalias;
+        new_taskdata->ompt_task_info.deps = (ompt_task_dependence_t *)
+          KMP_OMPT_DEPS_ALLOC(thread,
+             (ndeps+ndeps_noalias)*sizeof(ompt_task_dependence_t));
+
+        KMP_ASSERT(new_taskdata->ompt_task_info.deps != NULL);
+
+        for (i = 0; i < ndeps; i++)
+        {
+            new_taskdata->ompt_task_info.deps[i].variable_addr =
+              (void*) dep_list[i].base_addr;
+            if (dep_list[i].flags.in && dep_list[i].flags.out)
+                new_taskdata->ompt_task_info.deps[i].dependence_flags =
+                  ompt_task_dependence_type_inout;
+            else if (dep_list[i].flags.out)
+                new_taskdata->ompt_task_info.deps[i].dependence_flags =
+                  ompt_task_dependence_type_out;
+            else if (dep_list[i].flags.in)
+                new_taskdata->ompt_task_info.deps[i].dependence_flags =
+                  ompt_task_dependence_type_in;
+        }
+        for (i = 0; i < ndeps_noalias; i++)
+        {
+            new_taskdata->ompt_task_info.deps[ndeps+i].variable_addr =
+              (void*) noalias_dep_list[i].base_addr;
+            if (noalias_dep_list[i].flags.in && noalias_dep_list[i].flags.out)
+                new_taskdata->ompt_task_info.deps[ndeps+i].dependence_flags =
+                  ompt_task_dependence_type_inout;
+            else if (noalias_dep_list[i].flags.out)
+                new_taskdata->ompt_task_info.deps[ndeps+i].dependence_flags =
+                  ompt_task_dependence_type_out;
+            else if (noalias_dep_list[i].flags.in)
+                new_taskdata->ompt_task_info.deps[ndeps+i].dependence_flags =
+                  ompt_task_dependence_type_in;
+        }
+    }
+#endif /* OMPT_SUPPORT && OMPT_TRACE */
+
     bool serial = current_task->td_flags.team_serial || current_task->td_flags.tasking_ser || current_task->td_flags.final;
 #if OMP_41_ENABLED
     serial = serial && !(new_taskdata->td_flags.proxy == TASK_PROXY);
 #endif
 
     if ( !serial && ( ndeps > 0 || ndeps_noalias > 0 )) {
         /* if no dependencies have been tracked yet, create the dependence hash */
         if ( current_task->td_dephash == NULL )

runtime/src/kmp_tasking.c

if (ompt_enabled &&
         kmp_taskdata_t *parent = taskdata->td_parent;
         ompt_callbacks.ompt_callback(ompt_event_task_begin)(
             parent ? parent->ompt_task_info.task_id : ompt_task_id_none,
             parent ? &(parent->ompt_task_info.frame) : NULL,
             taskdata->ompt_task_info.task_id,
             taskdata->ompt_task_info.function);
     }
 #endif
+#if OMP_40_ENABLED && OMPT_SUPPORT && OMPT_TRACE
+    /* OMPT emit all dependences if requested by the tool */
+    if (ompt_enabled && taskdata->ompt_task_info.ndeps > 0 &&
+        ompt_callbacks.ompt_callback(ompt_event_task_dependences))
+  {
+        ompt_callbacks.ompt_callback(ompt_event_task_dependences)(
+            taskdata->ompt_task_info.task_id,
+            taskdata->ompt_task_info.deps,
+            taskdata->ompt_task_info.ndeps
+        );
+    /* We can now free the allocated memory for the dependencies */
+    KMP_OMPT_DEPS_FREE (thread, taskdata->ompt_task_info.deps);
+        taskdata->ompt_task_info.deps = NULL;
+        taskdata->ompt_task_info.ndeps = 0;
+    }
+#endif /* OMP_40_ENABLED && OMPT_SUPPORT && OMPT_TRACE */
 
     return;
 }
 
 
 //----------------------------------------------------------------------
 // __kmpc_omp_task_begin_if0: report that a given serialized task has started execution
 // loc_ref: source location information; points to beginning of task block.

 static inline void
 __kmp_task_init_ompt( kmp_taskdata_t * task, int tid, void * function )
 {
     if (ompt_enabled) {
         task->ompt_task_info.task_id = __ompt_task_id_new(tid);
         task->ompt_task_info.function = function;
         task->ompt_task_info.frame.exit_runtime_frame = NULL;
         task->ompt_task_info.frame.reenter_runtime_frame = NULL;
+#if OMP_40_ENABLED
+        task->ompt_task_info.ndeps = 0;
+        task->ompt_task_info.deps = NULL;
+#endif /* OMP_40_ENABLED */
     }
 }
 #endif
 
 
 //----------------------------------------------------------------------------------------------------
 // __kmp_init_implicit_task: Initialize the appropriate fields in the implicit task for a given thread
 //

runtime/src/ompt-event-specific.h

 #define ompt_event_init_lock_implemented                ompt_event_MAY_ALWAYS_TRACE
 #define ompt_event_init_nest_lock_implemented           ompt_event_MAY_ALWAYS_TRACE
 
 #define ompt_event_destroy_lock_implemented             ompt_event_MAY_ALWAYS_TRACE
 #define ompt_event_destroy_nest_lock_implemented        ompt_event_MAY_ALWAYS_TRACE
 
 #define ompt_event_flush_implemented                    ompt_event_UNIMPLEMENTED
 
+#if OMP_40_ENABLED
+# define ompt_event_task_dependences_implemented         ompt_event_MAY_ALWAYS_TRACE
+# define ompt_event_task_dependence_pair_implemented     ompt_event_MAY_ALWAYS_TRACE
+#else
+# define ompt_event_task_dependences_implemented         ompt_event_UNIMPLEMENTED
+# define ompt_event_task_dependence_pair_implemented     ompt_event_UNIMPLEMENTED
+#endif /* OMP_40_ENABLED */
+
 #endif

runtime/src/ompt-internal.h

#define ompt_event_macro(event, callback, eventid) callback ompt_callback(event);
     FOREACH_OMPT_EVENT(ompt_event_macro)
 
 #undef ompt_event_macro
 } ompt_callbacks_t;
 
 
 
 typedef struct {
-    ompt_frame_t        frame;
-    void*               function;
-    ompt_task_id_t      task_id;
+    ompt_frame_t            frame;
+    void*                   function;
+    ompt_task_id_t          task_id;
+#if OMP_40_ENABLED
+    int                     ndeps;
+    ompt_task_dependence_t  *deps;
+#endif /* OMP_40_ENABLED */
 } ompt_task_info_t;
 
 
 typedef struct {
     ompt_parallel_id_t  parallel_id;
     void                *microtask;
 } ompt_team_info_t;
 
typedef struct {
     ompt_state_t        state;
     ompt_wait_id_t      wait_id;
     void                *idle_frame;
 } ompt_thread_info_t;
 
 
 extern ompt_callbacks_t ompt_callbacks;
 
+#if OMP_40_ENABLED && OMPT_SUPPORT && OMPT_TRACE
+#if USE_FAST_MEMORY
+#  define KMP_OMPT_DEPS_ALLOC __kmp_fast_allocate
+#  define KMP_OMPT_DEPS_FREE __kmp_fast_free
+# else
+#  define KMP_OMPT_DEPS_ALLOC __kmp_thread_malloc
+#  define KMP_OMPT_DEPS_FREE __kmp_thread_free
+# endif
+#endif /* OMP_40_ENABLED && OMPT_SUPPORT && OMPT_TRACE */
+
 #ifdef __cplusplus
 extern "C" {
 #endif
 
 void ompt_pre_init(void);
 void ompt_post_init(void);
 void ompt_fini(void);
 
 extern int ompt_enabled;
 
 #ifdef __cplusplus
 };
 #endif
 
 #endif