This is an archive of the discontinued LLVM Phabricator instance.

Differential D49857

[OpenMP] Implement GOMP doacross compatibility
ClosedPublic

Authored by jlpeyton on Jul 26 2018, 8:22 AM.

Details

Reviewers
tlwilmar
AndreyChurbanov
hbae
omalyshe
Commits
rG369d72db116a: [OpenMP] Implement GOMP doacross compatibility
rL338280: [OpenMP] Implement GOMP doacross compatibility
rOMP338280: [OpenMP] Implement GOMP doacross compatibility
Summary

This change introduces GOMP doacross compatibility. There are 12 new interface
functions 6 for long type and 6 for unsigned long long type:
GOMP_doacross_post, GOMP_doacross_wait, GOMP_loop_doacross_[schedule]_start
where schedule can be static, dynamic, guided, or runtime.

These functions just translate the parameters if necessary and send them
to the corresponding kmp function.
E.g., GOMP_doacross_post() -> __kmpc_doacross_post()

For the GOMP_doacross_post function, there is template specialization to
account for when long is a four byte vs an eight byte type. If it is a
four byte type, then a temporary array has to be created to convert the
four byte integers into eight byte integers and then sending that into
__kmpc_doacross_post(). Because GOMP_doacross_wait uses varargs, it
always needs a temporary array and does not need template specialization.

Diff Detail

Repository
rOMP OpenMP

Event Timeline

jlpeyton created this revision.Jul 26 2018, 8:22 AM
Herald added a subscriber: guansong. · View Herald TranscriptJul 26 2018, 8:22 AM
tlwilmar accepted this revision.Jul 26 2018, 11:56 AM

LGTM

This revision is now accepted and ready to land.Jul 26 2018, 11:56 AM
Closed by commit rOMP338280: [OpenMP] Implement GOMP doacross compatibility (authored by jlpeyton). · Explain WhyJul 30 2018, 10:49 AM
This revision was automatically updated to reflect the committed changes.
Hahnfeld added a subscriber: Hahnfeld.Jul 30 2018, 12:05 PM

I'm not sure why but this seems to have broken worksharing/for/kmp_doacross_check.c with GCC. That's weird because this test calls the __kmpc_doacross_* functions directly, investigating...

Hahnfeld added inline comments.Jul 30 2018, 12:50 PM
runtime/src/kmp_gsupport.cpp
615

This statement seems to cause the harm. I think this makes all threads free their doacross buffer after processing their last iteration, right?

That's breaking runtime/test/worksharing/for/kmp_doacross_check.c which has its own call to __kmpc_doacross_fini at line 46 :-(

jlpeyton added a comment.Jul 30 2018, 1:14 PM

I can create a typical 2D wavefront testcase that uses the actual doacross clauses, and we can omit gcc from kmp_doacross_check.c. Does this suffice?

Hahnfeld added a comment.Jul 31 2018, 12:40 AM
In D49857#1180955, @jlpeyton wrote:

I can create a typical 2D wavefront testcase that uses the actual doacross clauses, and we can omit gcc from kmp_doacross_check.c. Does this suffice?

I think it's good to have an end-to-end test now that the library has interfaces for all compilers.

For kmp_doacross_check.c another possibility would be to guard the call to __kmpc_doacross_fini in #ifndef __GNUC__. I'm not sure if this is elegant though...

AndreyChurbanov added a comment.Jul 31 2018, 2:19 AM
In D49857#1181735, @Hahnfeld wrote:
In D49857#1180955, @jlpeyton wrote:

I can create a typical 2D wavefront testcase that uses the actual doacross clauses, and we can omit gcc from kmp_doacross_check.c. Does this suffice?

I think it's good to have an end-to-end test now that the library has interfaces for all compilers.

For kmp_doacross_check.c another possibility would be to guard the call to __kmpc_doacross_fini in #ifndef __GNUC__. I'm not sure if this is elegant though...

Intel compiler defines GNUC to express compatibility I guess, so it is safer to disable the whole test.

Hahnfeld mentioned this in D50014: [OpenMP] Fix doacross testing for gcc.Jul 31 2018, 7:47 AM

Revision Contents

PathSize
runtime/
src/
7 lines
7 lines
20 lines
447 lines
5 lines

Diff 158003

runtime/src/kmp.h

Show First 20 Lines • Show All 3,801 Lines • ▼ Show 20 Lines
#endif #endif
#if OMP_45_ENABLED #if OMP_45_ENABLED
struct kmp_dim { // loop bounds info casted to kmp_int64 struct kmp_dim { // loop bounds info casted to kmp_int64
kmp_int64 lo; // lower kmp_int64 lo; // lower
kmp_int64 up; // upper kmp_int64 up; // upper
kmp_int64 st; // stride kmp_int64 st; // stride
}; };
KMP_EXPORT void __kmpc_doacross_init(ident_t *loc, kmp_int32 gtid, KMP_EXPORT void __kmpc_doacross_init(ident_t *loc, kmp_int32 gtid,
kmp_int32 num_dims, struct kmp_dim *dims); kmp_int32 num_dims,
const struct kmp_dim *dims);
KMP_EXPORT void __kmpc_doacross_wait(ident_t *loc, kmp_int32 gtid, KMP_EXPORT void __kmpc_doacross_wait(ident_t *loc, kmp_int32 gtid,
kmp_int64 *vec); const kmp_int64 *vec);
KMP_EXPORT void __kmpc_doacross_post(ident_t *loc, kmp_int32 gtid, KMP_EXPORT void __kmpc_doacross_post(ident_t *loc, kmp_int32 gtid,
kmp_int64 *vec); const kmp_int64 *vec);
KMP_EXPORT void __kmpc_doacross_fini(ident_t *loc, kmp_int32 gtid); KMP_EXPORT void __kmpc_doacross_fini(ident_t *loc, kmp_int32 gtid);
#endif #endif
KMP_EXPORT void *__kmpc_threadprivate_cached(ident_t *loc, kmp_int32 global_tid, KMP_EXPORT void *__kmpc_threadprivate_cached(ident_t *loc, kmp_int32 global_tid,
void *data, size_t size, void *data, size_t size,
void ***cache); void ***cache);
// Symbols for MS mutual detection. // Symbols for MS mutual detection.
▲ Show 20 Lines • Show All 67 Lines • Show Last 20 Lines

runtime/src/kmp_csupport.cpp

Show First 20 Lines • Show All 3,776 Lines • ▼ Show 20 Lines
@param num_dims number of associated doacross loops. @param num_dims number of associated doacross loops.
@param dims info on loops bounds. @param dims info on loops bounds.
Initialize doacross loop information. Initialize doacross loop information.
Expect compiler send us inclusive bounds, Expect compiler send us inclusive bounds,
e.g. for(i=2;i<9;i+=2) lo=2, up=8, st=2. e.g. for(i=2;i<9;i+=2) lo=2, up=8, st=2.
*/ */
void __kmpc_doacross_init(ident_t *loc, int gtid, int num_dims, void __kmpc_doacross_init(ident_t *loc, int gtid, int num_dims,
struct kmp_dim *dims) { const struct kmp_dim *dims) {
int j, idx; int j, idx;
kmp_int64 last, trace_count; kmp_int64 last, trace_count;
kmp_info_t *th = __kmp_threads[gtid]; kmp_info_t *th = __kmp_threads[gtid];
kmp_team_t *team = th->th.th_team; kmp_team_t *team = th->th.th_team;
kmp_uint32 *flags; kmp_uint32 *flags;
kmp_disp_t *pr_buf = th->th.th_dispatch; kmp_disp_t *pr_buf = th->th.th_dispatch;
dispatch_shared_info_t *sh_buf; dispatch_shared_info_t *sh_buf;
▲ Show 20 Lines • Show All 102 Lines • ▼ Show 20 Lines#endif
} }
KMP_DEBUG_ASSERT(sh_buf->doacross_flags > (kmp_uint32 *)1); // check ptr value KMP_DEBUG_ASSERT(sh_buf->doacross_flags > (kmp_uint32 *)1); // check ptr value
pr_buf->th_doacross_flags = pr_buf->th_doacross_flags =
sh_buf->doacross_flags; // save private copy in order to not sh_buf->doacross_flags; // save private copy in order to not
// touch shared buffer on each iteration // touch shared buffer on each iteration
KA_TRACE(20, ("__kmpc_doacross_init() exit: T#%d\n", gtid)); KA_TRACE(20, ("__kmpc_doacross_init() exit: T#%d\n", gtid));
} }
void __kmpc_doacross_wait(ident_t *loc, int gtid, long long *vec) { void __kmpc_doacross_wait(ident_t *loc, int gtid, const kmp_int64 *vec) {
kmp_int32 shft, num_dims, i; kmp_int32 shft, num_dims, i;
kmp_uint32 flag; kmp_uint32 flag;
kmp_int64 iter_number; // iteration number of "collapsed" loop nest kmp_int64 iter_number; // iteration number of "collapsed" loop nest
kmp_info_t *th = __kmp_threads[gtid]; kmp_info_t *th = __kmp_threads[gtid];
kmp_team_t *team = th->th.th_team; kmp_team_t *team = th->th.th_team;
kmp_disp_t *pr_buf; kmp_disp_t *pr_buf;
kmp_int64 lo, up, st; kmp_int64 lo, up, st;
▲ Show 20 Lines • Show All 76 Lines • ▼ Show 20 Lines while ((flag & pr_buf->th_doacross_flags[iter_number]) == 0) {
KMP_YIELD(TRUE); KMP_YIELD(TRUE);
} }
KMP_MB(); KMP_MB();
KA_TRACE(20, KA_TRACE(20,
("__kmpc_doacross_wait() exit: T#%d wait for iter %lld completed\n", ("__kmpc_doacross_wait() exit: T#%d wait for iter %lld completed\n",
gtid, (iter_number << 5) + shft)); gtid, (iter_number << 5) + shft));
} }
void __kmpc_doacross_post(ident_t *loc, int gtid, long long *vec) { void __kmpc_doacross_post(ident_t *loc, int gtid, const kmp_int64 *vec) {
kmp_int32 shft, num_dims, i; kmp_int32 shft, num_dims, i;
kmp_uint32 flag; kmp_uint32 flag;
kmp_int64 iter_number; // iteration number of "collapsed" loop nest kmp_int64 iter_number; // iteration number of "collapsed" loop nest
kmp_info_t *th = __kmp_threads[gtid]; kmp_info_t *th = __kmp_threads[gtid];
kmp_team_t *team = th->th.th_team; kmp_team_t *team = th->th.th_team;
kmp_disp_t *pr_buf; kmp_disp_t *pr_buf;
kmp_int64 lo, st; kmp_int64 lo, st;
▲ Show 20 Lines • Show All 65 Lines • ▼ Show 20 Lines if (num_done == th->th.th_team_nproc) {
KMP_DEBUG_ASSERT(idx == sh_buf->doacross_buf_idx); KMP_DEBUG_ASSERT(idx == sh_buf->doacross_buf_idx);
__kmp_thread_free(th, CCAST(kmp_uint32 *, sh_buf->doacross_flags)); __kmp_thread_free(th, CCAST(kmp_uint32 *, sh_buf->doacross_flags));
sh_buf->doacross_flags = NULL; sh_buf->doacross_flags = NULL;
sh_buf->doacross_num_done = 0; sh_buf->doacross_num_done = 0;
sh_buf->doacross_buf_idx += sh_buf->doacross_buf_idx +=
__kmp_dispatch_num_buffers; // free buffer for future re-use __kmp_dispatch_num_buffers; // free buffer for future re-use
} }
// free private resources (need to keep buffer index forever) // free private resources (need to keep buffer index forever)
pr_buf->th_doacross_flags = NULL;
__kmp_thread_free(th, (void *)pr_buf->th_doacross_info); __kmp_thread_free(th, (void *)pr_buf->th_doacross_info);
pr_buf->th_doacross_info = NULL; pr_buf->th_doacross_info = NULL;
KA_TRACE(20, ("__kmpc_doacross_fini() exit: T#%d\n", gtid)); KA_TRACE(20, ("__kmpc_doacross_fini() exit: T#%d\n", gtid));
} }
#endif #endif
#if OMP_50_ENABLED #if OMP_50_ENABLED
int __kmpc_get_target_offload(void) { int __kmpc_get_target_offload(void) {
if (!__kmp_init_serial) { if (!__kmp_init_serial) {
__kmp_serial_initialize(); __kmp_serial_initialize();
} }
return __kmp_target_offload; return __kmp_target_offload;
} }
#endif // OMP_50_ENABLED #endif // OMP_50_ENABLED
// end of file // // end of file //

runtime/src/kmp_ftn_os.h

Show First 20 Lines • Show All 610 Lines • ▼ Show 20 Lines
#define KMP_API_NAME_GOMP_TARGET_DATA GOMP_target_data #define KMP_API_NAME_GOMP_TARGET_DATA GOMP_target_data
#define KMP_API_NAME_GOMP_TARGET_END_DATA GOMP_target_end_data #define KMP_API_NAME_GOMP_TARGET_END_DATA GOMP_target_end_data
#define KMP_API_NAME_GOMP_TARGET_UPDATE GOMP_target_update #define KMP_API_NAME_GOMP_TARGET_UPDATE GOMP_target_update
#define KMP_API_NAME_GOMP_TEAMS GOMP_teams #define KMP_API_NAME_GOMP_TEAMS GOMP_teams
// All GOMP_4.5 symbols // All GOMP_4.5 symbols
#define KMP_API_NAME_GOMP_TASKLOOP GOMP_taskloop #define KMP_API_NAME_GOMP_TASKLOOP GOMP_taskloop
#define KMP_API_NAME_GOMP_TASKLOOP_ULL GOMP_taskloop_ull #define KMP_API_NAME_GOMP_TASKLOOP_ULL GOMP_taskloop_ull
#define KMP_API_NAME_GOMP_DOACROSS_POST GOMP_doacross_post
#define KMP_API_NAME_GOMP_DOACROSS_WAIT GOMP_doacross_wait
#define KMP_API_NAME_GOMP_LOOP_DOACROSS_STATIC_START \
GOMP_loop_doacross_static_start
#define KMP_API_NAME_GOMP_LOOP_DOACROSS_DYNAMIC_START \
GOMP_loop_doacross_dynamic_start
#define KMP_API_NAME_GOMP_LOOP_DOACROSS_GUIDED_START \
GOMP_loop_doacross_guided_start
#define KMP_API_NAME_GOMP_LOOP_DOACROSS_RUNTIME_START \
GOMP_loop_doacross_runtime_start
#define KMP_API_NAME_GOMP_DOACROSS_ULL_POST GOMP_doacross_ull_post
#define KMP_API_NAME_GOMP_DOACROSS_ULL_WAIT GOMP_doacross_ull_wait
#define KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_STATIC_START \
GOMP_loop_ull_doacross_static_start
#define KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_DYNAMIC_START \
GOMP_loop_ull_doacross_dynamic_start
#define KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_GUIDED_START \
GOMP_loop_ull_doacross_guided_start
#define KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_RUNTIME_START \
GOMP_loop_ull_doacross_runtime_start
#endif /* KMP_FTN_OS_H */ #endif /* KMP_FTN_OS_H */

runtime/src/kmp_gsupport.cpp

Show First 20 Lines • Show All 517 Lines • ▼ Show 20 Lines
#endif #endif
#define LOOP_START(func, schedule) \ #define LOOP_START(func, schedule) \
int func(long lb, long ub, long str, long chunk_sz, long *p_lb, \ int func(long lb, long ub, long str, long chunk_sz, long *p_lb, \
long *p_ub) { \ long *p_ub) { \
int status; \ int status; \
long stride; \ long stride; \
int gtid = __kmp_entry_gtid(); \ int gtid = __kmp_entry_gtid(); \
MKLOC(loc, #func); \ MKLOC(loc, KMP_STR(func)); \
KA_TRACE(20, \ KA_TRACE( \
(#func ": T#%d, lb 0x%lx, ub 0x%lx, str 0x%lx, chunk_sz 0x%lx\n", \ 20, \
(KMP_STR( \
func) ": T#%d, lb 0x%lx, ub 0x%lx, str 0x%lx, chunk_sz 0x%lx\n", \
gtid, lb, ub, str, chunk_sz)); \ gtid, lb, ub, str, chunk_sz)); \
\ \
if ((str > 0) ? (lb < ub) : (lb > ub)) { \ if ((str > 0) ? (lb < ub) : (lb > ub)) { \
IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \ IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \
KMP_DISPATCH_INIT(&loc, gtid, (schedule), lb, \ KMP_DISPATCH_INIT(&loc, gtid, (schedule), lb, \
(str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, \ (str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, \
(schedule) != kmp_sch_static); \ (schedule) != kmp_sch_static); \
IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \ IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \
status = KMP_DISPATCH_NEXT(&loc, gtid, NULL, (kmp_int *)p_lb, \ status = KMP_DISPATCH_NEXT(&loc, gtid, NULL, (kmp_int *)p_lb, \
(kmp_int *)p_ub, (kmp_int *)&stride); \ (kmp_int *)p_ub, (kmp_int *)&stride); \
if (status) { \ if (status) { \
KMP_DEBUG_ASSERT(stride == str); \ KMP_DEBUG_ASSERT(stride == str); \
*p_ub += (str > 0) ? 1 : -1; \ *p_ub += (str > 0) ? 1 : -1; \
} \ } \
} else { \ } else { \
status = 0; \ status = 0; \
} \ } \
\ \
KA_TRACE(20, \ KA_TRACE( \
(#func " exit: T#%d, *p_lb 0x%lx, *p_ub 0x%lx, returning %d\n", \ 20, \
(KMP_STR( \
func) " exit: T#%d, *p_lb 0x%lx, *p_ub 0x%lx, returning %d\n", \
gtid, *p_lb, *p_ub, status)); \ gtid, *p_lb, *p_ub, status)); \
return status; \ return status; \
} }
#define LOOP_RUNTIME_START(func, schedule) \ #define LOOP_RUNTIME_START(func, schedule) \
int func(long lb, long ub, long str, long *p_lb, long *p_ub) { \ int func(long lb, long ub, long str, long *p_lb, long *p_ub) { \
int status; \ int status; \
long stride; \ long stride; \
long chunk_sz = 0; \ long chunk_sz = 0; \
int gtid = __kmp_entry_gtid(); \ int gtid = __kmp_entry_gtid(); \
MKLOC(loc, #func); \ MKLOC(loc, KMP_STR(func)); \
KA_TRACE(20, \ KA_TRACE( \
(#func ": T#%d, lb 0x%lx, ub 0x%lx, str 0x%lx, chunk_sz %d\n", \ 20, \
(KMP_STR(func) ": T#%d, lb 0x%lx, ub 0x%lx, str 0x%lx, chunk_sz %d\n", \
gtid, lb, ub, str, chunk_sz)); \ gtid, lb, ub, str, chunk_sz)); \
\ \
if ((str > 0) ? (lb < ub) : (lb > ub)) { \ if ((str > 0) ? (lb < ub) : (lb > ub)) { \
IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \ IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \
KMP_DISPATCH_INIT(&loc, gtid, (schedule), lb, \ KMP_DISPATCH_INIT(&loc, gtid, (schedule), lb, \
(str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, TRUE); \ (str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, TRUE); \
IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \ IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \
status = KMP_DISPATCH_NEXT(&loc, gtid, NULL, (kmp_int *)p_lb, \ status = KMP_DISPATCH_NEXT(&loc, gtid, NULL, (kmp_int *)p_lb, \
(kmp_int *)p_ub, (kmp_int *)&stride); \ (kmp_int *)p_ub, (kmp_int *)&stride); \
if (status) { \ if (status) { \
KMP_DEBUG_ASSERT(stride == str); \ KMP_DEBUG_ASSERT(stride == str); \
*p_ub += (str > 0) ? 1 : -1; \ *p_ub += (str > 0) ? 1 : -1; \
} \ } \
} else { \ } else { \
status = 0; \ status = 0; \
} \ } \
\ \
KA_TRACE(20, \ KA_TRACE( \
(#func " exit: T#%d, *p_lb 0x%lx, *p_ub 0x%lx, returning %d\n", \ 20, \
(KMP_STR( \
func) " exit: T#%d, *p_lb 0x%lx, *p_ub 0x%lx, returning %d\n", \
gtid, *p_lb, *p_ub, status)); \ gtid, *p_lb, *p_ub, status)); \
return status; \ return status; \
} }
#if OMP_45_ENABLED
#define KMP_DOACROSS_FINI(status, gtid) \
if (!status && __kmp_threads[gtid]->th.th_dispatch->th_doacross_flags) { \
__kmpc_doacross_fini(NULL, gtid); \
}
#else
#define KMP_DOACROSS_FINI(status, gtid) /* Nothing */
#endif
#define LOOP_NEXT(func, fini_code) \ #define LOOP_NEXT(func, fini_code) \
int func(long *p_lb, long *p_ub) { \ int func(long *p_lb, long *p_ub) { \
int status; \ int status; \
long stride; \ long stride; \
int gtid = __kmp_get_gtid(); \ int gtid = __kmp_get_gtid(); \
MKLOC(loc, #func); \ MKLOC(loc, KMP_STR(func)); \
KA_TRACE(20, (#func ": T#%d\n", gtid)); \ KA_TRACE(20, (KMP_STR(func) ": T#%d\n", gtid)); \
\ \
IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \ IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \
fini_code status = KMP_DISPATCH_NEXT(&loc, gtid, NULL, (kmp_int *)p_lb, \ fini_code status = KMP_DISPATCH_NEXT(&loc, gtid, NULL, (kmp_int *)p_lb, \
(kmp_int *)p_ub, (kmp_int *)&stride); \ (kmp_int *)p_ub, (kmp_int *)&stride); \
if (status) { \ if (status) { \
*p_ub += (stride > 0) ? 1 : -1; \ *p_ub += (stride > 0) ? 1 : -1; \
} \ } \
KMP_DOACROSS_FINI(status, gtid) \
HahnfeldUnsubmitted
Not Done
ReplyInline Actions

This statement seems to cause the harm. I think this makes all threads free their doacross buffer after processing their last iteration, right?

That's breaking runtime/test/worksharing/for/kmp_doacross_check.c which has its own call to __kmpc_doacross_fini at line 46 :-(

Hahnfeld: This statement seems to cause the harm. I think this makes all threads free their doacross…
\ \
KA_TRACE(20, \ KA_TRACE( \
(#func " exit: T#%d, *p_lb 0x%lx, *p_ub 0x%lx, stride 0x%lx, " \ 20, \
(KMP_STR(func) " exit: T#%d, *p_lb 0x%lx, *p_ub 0x%lx, stride 0x%lx, " \
"returning %d\n", \ "returning %d\n", \
gtid, *p_lb, *p_ub, stride, status)); \ gtid, *p_lb, *p_ub, stride, status)); \
return status; \ return status; \
} }
LOOP_START(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_STATIC_START), kmp_sch_static) LOOP_START(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_STATIC_START), kmp_sch_static)
LOOP_NEXT(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_STATIC_NEXT), {}) LOOP_NEXT(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_STATIC_NEXT), {})
LOOP_START(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_DYNAMIC_START), LOOP_START(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_DYNAMIC_START),
kmp_sch_dynamic_chunked) kmp_sch_dynamic_chunked)
LOOP_NEXT(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_DYNAMIC_NEXT), {}) LOOP_NEXT(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_DYNAMIC_NEXT), {})
Show All 17 Lines
LOOP_NEXT(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ORDERED_GUIDED_NEXT), LOOP_NEXT(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ORDERED_GUIDED_NEXT),
{ KMP_DISPATCH_FINI_CHUNK(&loc, gtid); }) { KMP_DISPATCH_FINI_CHUNK(&loc, gtid); })
LOOP_RUNTIME_START( LOOP_RUNTIME_START(
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ORDERED_RUNTIME_START), KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ORDERED_RUNTIME_START),
kmp_ord_runtime) kmp_ord_runtime)
LOOP_NEXT(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ORDERED_RUNTIME_NEXT), LOOP_NEXT(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ORDERED_RUNTIME_NEXT),
{ KMP_DISPATCH_FINI_CHUNK(&loc, gtid); }) { KMP_DISPATCH_FINI_CHUNK(&loc, gtid); })
#if OMP_45_ENABLED
#define LOOP_DOACROSS_START(func, schedule) \
bool func(unsigned ncounts, long *counts, long chunk_sz, long *p_lb, \
long *p_ub) { \
int status; \
long stride, lb, ub, str; \
int gtid = __kmp_entry_gtid(); \
struct kmp_dim *dims = \
(struct kmp_dim *)__kmp_allocate(sizeof(struct kmp_dim) * ncounts); \
MKLOC(loc, KMP_STR(func)); \
for (unsigned i = 0; i < ncounts; ++i) { \
dims[i].lo = 0; \
dims[i].up = counts[i] - 1; \
dims[i].st = 1; \
} \
__kmpc_doacross_init(&loc, gtid, (int)ncounts, dims); \
lb = 0; \
ub = counts[0]; \
str = 1; \
KA_TRACE(20, (KMP_STR(func) ": T#%d, ncounts %u, lb 0x%lx, ub 0x%lx, str " \
"0x%lx, chunk_sz " \
"0x%lx\n", \
gtid, ncounts, lb, ub, str, chunk_sz)); \
\
if ((str > 0) ? (lb < ub) : (lb > ub)) { \
KMP_DISPATCH_INIT(&loc, gtid, (schedule), lb, \
(str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, \
(schedule) != kmp_sch_static); \
status = KMP_DISPATCH_NEXT(&loc, gtid, NULL, (kmp_int *)p_lb, \
(kmp_int *)p_ub, (kmp_int *)&stride); \
if (status) { \
KMP_DEBUG_ASSERT(stride == str); \
*p_ub += (str > 0) ? 1 : -1; \
} \
} else { \
status = 0; \
} \
KMP_DOACROSS_FINI(status, gtid); \
\
KA_TRACE( \
20, \
(KMP_STR( \
func) " exit: T#%d, *p_lb 0x%lx, *p_ub 0x%lx, returning %d\n", \
gtid, *p_lb, *p_ub, status)); \
__kmp_free(dims); \
return status; \
}
#define LOOP_DOACROSS_RUNTIME_START(func, schedule) \
int func(unsigned ncounts, long *counts, long *p_lb, long *p_ub) { \
int status; \
long stride, lb, ub, str; \
long chunk_sz = 0; \
int gtid = __kmp_entry_gtid(); \
struct kmp_dim *dims = \
(struct kmp_dim *)__kmp_allocate(sizeof(struct kmp_dim) * ncounts); \
MKLOC(loc, KMP_STR(func)); \
for (unsigned i = 0; i < ncounts; ++i) { \
dims[i].lo = 0; \
dims[i].up = counts[i] - 1; \
dims[i].st = 1; \
} \
__kmpc_doacross_init(&loc, gtid, (int)ncounts, dims); \
lb = 0; \
ub = counts[0]; \
str = 1; \
KA_TRACE( \
20, \
(KMP_STR(func) ": T#%d, lb 0x%lx, ub 0x%lx, str 0x%lx, chunk_sz %d\n", \
gtid, lb, ub, str, chunk_sz)); \
\
if ((str > 0) ? (lb < ub) : (lb > ub)) { \
KMP_DISPATCH_INIT(&loc, gtid, (schedule), lb, \
(str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, TRUE); \
status = KMP_DISPATCH_NEXT(&loc, gtid, NULL, (kmp_int *)p_lb, \
(kmp_int *)p_ub, (kmp_int *)&stride); \
if (status) { \
KMP_DEBUG_ASSERT(stride == str); \
*p_ub += (str > 0) ? 1 : -1; \
} \
} else { \
status = 0; \
} \
KMP_DOACROSS_FINI(status, gtid); \
\
KA_TRACE( \
20, \
(KMP_STR( \
func) " exit: T#%d, *p_lb 0x%lx, *p_ub 0x%lx, returning %d\n", \
gtid, *p_lb, *p_ub, status)); \
__kmp_free(dims); \
return status; \
}
LOOP_DOACROSS_START(
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_DOACROSS_STATIC_START),
kmp_sch_static)
LOOP_DOACROSS_START(
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_DOACROSS_DYNAMIC_START),
kmp_sch_dynamic_chunked)
LOOP_DOACROSS_START(
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_DOACROSS_GUIDED_START),
kmp_sch_guided_chunked)
LOOP_DOACROSS_RUNTIME_START(
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_DOACROSS_RUNTIME_START),
kmp_sch_runtime)
#endif // OMP_45_ENABLED
void KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_END)(void) { void KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_END)(void) {
int gtid = __kmp_get_gtid(); int gtid = __kmp_get_gtid();
KA_TRACE(20, ("GOMP_loop_end: T#%d\n", gtid)) KA_TRACE(20, ("GOMP_loop_end: T#%d\n", gtid))
#if OMPT_SUPPORT && OMPT_OPTIONAL #if OMPT_SUPPORT && OMPT_OPTIONAL
omp_frame_t *ompt_frame; omp_frame_t *ompt_frame;
if (ompt_enabled.enabled) { if (ompt_enabled.enabled) {
__ompt_get_task_info_internal(0, NULL, NULL, &ompt_frame, NULL, NULL); __ompt_get_task_info_internal(0, NULL, NULL, &ompt_frame, NULL, NULL);
Show All 22 Lines
#define LOOP_START_ULL(func, schedule) \ #define LOOP_START_ULL(func, schedule) \
int func(int up, unsigned long long lb, unsigned long long ub, \ int func(int up, unsigned long long lb, unsigned long long ub, \
unsigned long long str, unsigned long long chunk_sz, \ unsigned long long str, unsigned long long chunk_sz, \
unsigned long long *p_lb, unsigned long long *p_ub) { \ unsigned long long *p_lb, unsigned long long *p_ub) { \
int status; \ int status; \
long long str2 = up ? ((long long)str) : -((long long)str); \ long long str2 = up ? ((long long)str) : -((long long)str); \
long long stride; \ long long stride; \
int gtid = __kmp_entry_gtid(); \ int gtid = __kmp_entry_gtid(); \
MKLOC(loc, #func); \ MKLOC(loc, KMP_STR(func)); \
\ \
KA_TRACE( \ KA_TRACE(20, (KMP_STR(func) ": T#%d, up %d, lb 0x%llx, ub 0x%llx, str " \
20, \ "0x%llx, chunk_sz 0x%llx\n", \
(#func \
": T#%d, up %d, lb 0x%llx, ub 0x%llx, str 0x%llx, chunk_sz 0x%llx\n", \
gtid, up, lb, ub, str, chunk_sz)); \ gtid, up, lb, ub, str, chunk_sz)); \
\ \
if ((str > 0) ? (lb < ub) : (lb > ub)) { \ if ((str > 0) ? (lb < ub) : (lb > ub)) { \
KMP_DISPATCH_INIT_ULL(&loc, gtid, (schedule), lb, \ KMP_DISPATCH_INIT_ULL(&loc, gtid, (schedule), lb, \
(str2 > 0) ? (ub - 1) : (ub + 1), str2, chunk_sz, \ (str2 > 0) ? (ub - 1) : (ub + 1), str2, chunk_sz, \
(schedule) != kmp_sch_static); \ (schedule) != kmp_sch_static); \
status = \ status = \
KMP_DISPATCH_NEXT_ULL(&loc, gtid, NULL, (kmp_uint64 *)p_lb, \ KMP_DISPATCH_NEXT_ULL(&loc, gtid, NULL, (kmp_uint64 *)p_lb, \
(kmp_uint64 *)p_ub, (kmp_int64 *)&stride); \ (kmp_uint64 *)p_ub, (kmp_int64 *)&stride); \
if (status) { \ if (status) { \
KMP_DEBUG_ASSERT(stride == str2); \ KMP_DEBUG_ASSERT(stride == str2); \
*p_ub += (str > 0) ? 1 : -1; \ *p_ub += (str > 0) ? 1 : -1; \
} \ } \
} else { \ } else { \
status = 0; \ status = 0; \
} \ } \
\ \
KA_TRACE(20, \ KA_TRACE( \
(#func " exit: T#%d, *p_lb 0x%llx, *p_ub 0x%llx, returning %d\n", \ 20, \
(KMP_STR( \
func) " exit: T#%d, *p_lb 0x%llx, *p_ub 0x%llx, returning %d\n", \
gtid, *p_lb, *p_ub, status)); \ gtid, *p_lb, *p_ub, status)); \
return status; \ return status; \
} }
#define LOOP_RUNTIME_START_ULL(func, schedule) \ #define LOOP_RUNTIME_START_ULL(func, schedule) \
int func(int up, unsigned long long lb, unsigned long long ub, \ int func(int up, unsigned long long lb, unsigned long long ub, \
unsigned long long str, unsigned long long *p_lb, \ unsigned long long str, unsigned long long *p_lb, \
unsigned long long *p_ub) { \ unsigned long long *p_ub) { \
int status; \ int status; \
long long str2 = up ? ((long long)str) : -((long long)str); \ long long str2 = up ? ((long long)str) : -((long long)str); \
unsigned long long stride; \ unsigned long long stride; \
unsigned long long chunk_sz = 0; \ unsigned long long chunk_sz = 0; \
int gtid = __kmp_entry_gtid(); \ int gtid = __kmp_entry_gtid(); \
MKLOC(loc, #func); \ MKLOC(loc, KMP_STR(func)); \
\ \
KA_TRACE( \ KA_TRACE(20, (KMP_STR(func) ": T#%d, up %d, lb 0x%llx, ub 0x%llx, str " \
20, \ "0x%llx, chunk_sz 0x%llx\n", \
(#func \
": T#%d, up %d, lb 0x%llx, ub 0x%llx, str 0x%llx, chunk_sz 0x%llx\n", \
gtid, up, lb, ub, str, chunk_sz)); \ gtid, up, lb, ub, str, chunk_sz)); \
\ \
if ((str > 0) ? (lb < ub) : (lb > ub)) { \ if ((str > 0) ? (lb < ub) : (lb > ub)) { \
KMP_DISPATCH_INIT_ULL(&loc, gtid, (schedule), lb, \ KMP_DISPATCH_INIT_ULL(&loc, gtid, (schedule), lb, \
(str2 > 0) ? (ub - 1) : (ub + 1), str2, chunk_sz, \ (str2 > 0) ? (ub - 1) : (ub + 1), str2, chunk_sz, \
TRUE); \ TRUE); \
status = \ status = \
KMP_DISPATCH_NEXT_ULL(&loc, gtid, NULL, (kmp_uint64 *)p_lb, \ KMP_DISPATCH_NEXT_ULL(&loc, gtid, NULL, (kmp_uint64 *)p_lb, \
(kmp_uint64 *)p_ub, (kmp_int64 *)&stride); \ (kmp_uint64 *)p_ub, (kmp_int64 *)&stride); \
if (status) { \ if (status) { \
KMP_DEBUG_ASSERT((long long)stride == str2); \ KMP_DEBUG_ASSERT((long long)stride == str2); \
*p_ub += (str > 0) ? 1 : -1; \ *p_ub += (str > 0) ? 1 : -1; \
} \ } \
} else { \ } else { \
status = 0; \ status = 0; \
} \ } \
\ \
KA_TRACE(20, \ KA_TRACE( \
(#func " exit: T#%d, *p_lb 0x%llx, *p_ub 0x%llx, returning %d\n", \ 20, \
(KMP_STR( \
func) " exit: T#%d, *p_lb 0x%llx, *p_ub 0x%llx, returning %d\n", \
gtid, *p_lb, *p_ub, status)); \ gtid, *p_lb, *p_ub, status)); \
return status; \ return status; \
} }
#define LOOP_NEXT_ULL(func, fini_code) \ #define LOOP_NEXT_ULL(func, fini_code) \
int func(unsigned long long *p_lb, unsigned long long *p_ub) { \ int func(unsigned long long *p_lb, unsigned long long *p_ub) { \
int status; \ int status; \
long long stride; \ long long stride; \
int gtid = __kmp_get_gtid(); \ int gtid = __kmp_get_gtid(); \
MKLOC(loc, #func); \ MKLOC(loc, KMP_STR(func)); \
KA_TRACE(20, (#func ": T#%d\n", gtid)); \ KA_TRACE(20, (KMP_STR(func) ": T#%d\n", gtid)); \
\ \
fini_code status = \ fini_code status = \
KMP_DISPATCH_NEXT_ULL(&loc, gtid, NULL, (kmp_uint64 *)p_lb, \ KMP_DISPATCH_NEXT_ULL(&loc, gtid, NULL, (kmp_uint64 *)p_lb, \
(kmp_uint64 *)p_ub, (kmp_int64 *)&stride); \ (kmp_uint64 *)p_ub, (kmp_int64 *)&stride); \
if (status) { \ if (status) { \
*p_ub += (stride > 0) ? 1 : -1; \ *p_ub += (stride > 0) ? 1 : -1; \
} \ } \
\ \
KA_TRACE(20, \ KA_TRACE( \
(#func " exit: T#%d, *p_lb 0x%llx, *p_ub 0x%llx, stride 0x%llx, " \ 20, \
(KMP_STR( \
func) " exit: T#%d, *p_lb 0x%llx, *p_ub 0x%llx, stride 0x%llx, " \
"returning %d\n", \ "returning %d\n", \
gtid, *p_lb, *p_ub, stride, status)); \ gtid, *p_lb, *p_ub, stride, status)); \
return status; \ return status; \
} }
LOOP_START_ULL(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_STATIC_START), LOOP_START_ULL(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_STATIC_START),
kmp_sch_static) kmp_sch_static)
LOOP_NEXT_ULL(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_STATIC_NEXT), {}) LOOP_NEXT_ULL(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_STATIC_NEXT), {})
LOOP_START_ULL(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_DYNAMIC_START), LOOP_START_ULL(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_DYNAMIC_START),
kmp_sch_dynamic_chunked) kmp_sch_dynamic_chunked)
Show All 19 Lines
LOOP_NEXT_ULL(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_ORDERED_GUIDED_NEXT), LOOP_NEXT_ULL(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_ORDERED_GUIDED_NEXT),
{ KMP_DISPATCH_FINI_CHUNK_ULL(&loc, gtid); }) { KMP_DISPATCH_FINI_CHUNK_ULL(&loc, gtid); })
LOOP_RUNTIME_START_ULL( LOOP_RUNTIME_START_ULL(
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_ORDERED_RUNTIME_START), KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_ORDERED_RUNTIME_START),
kmp_ord_runtime) kmp_ord_runtime)
LOOP_NEXT_ULL(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_ORDERED_RUNTIME_NEXT), LOOP_NEXT_ULL(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_ORDERED_RUNTIME_NEXT),
{ KMP_DISPATCH_FINI_CHUNK_ULL(&loc, gtid); }) { KMP_DISPATCH_FINI_CHUNK_ULL(&loc, gtid); })
#if OMP_45_ENABLED
#define LOOP_DOACROSS_START_ULL(func, schedule) \
int func(unsigned ncounts, unsigned long long *counts, \
unsigned long long chunk_sz, unsigned long long *p_lb, \
unsigned long long *p_ub) { \
int status; \
long long stride, str, lb, ub; \
int gtid = __kmp_entry_gtid(); \
struct kmp_dim *dims = \
(struct kmp_dim *)__kmp_allocate(sizeof(struct kmp_dim) * ncounts); \
MKLOC(loc, KMP_STR(func)); \
for (unsigned i = 0; i < ncounts; ++i) { \
dims[i].lo = 0; \
dims[i].up = counts[i] - 1; \
dims[i].st = 1; \
} \
__kmpc_doacross_init(&loc, gtid, (int)ncounts, dims); \
lb = 0; \
ub = counts[0]; \
str = 1; \
\
KA_TRACE(20, (KMP_STR(func) ": T#%d, lb 0x%llx, ub 0x%llx, str " \
"0x%llx, chunk_sz 0x%llx\n", \
gtid, lb, ub, str, chunk_sz)); \
\
if ((str > 0) ? (lb < ub) : (lb > ub)) { \
KMP_DISPATCH_INIT_ULL(&loc, gtid, (schedule), lb, \
(str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, \
(schedule) != kmp_sch_static); \
status = \
KMP_DISPATCH_NEXT_ULL(&loc, gtid, NULL, (kmp_uint64 *)p_lb, \
(kmp_uint64 *)p_ub, (kmp_int64 *)&stride); \
if (status) { \
KMP_DEBUG_ASSERT(stride == str); \
*p_ub += (str > 0) ? 1 : -1; \
} \
} else { \
status = 0; \
} \
KMP_DOACROSS_FINI(status, gtid); \
\
KA_TRACE( \
20, \
(KMP_STR( \
func) " exit: T#%d, *p_lb 0x%llx, *p_ub 0x%llx, returning %d\n", \
gtid, *p_lb, *p_ub, status)); \
__kmp_free(dims); \
return status; \
}
#define LOOP_DOACROSS_RUNTIME_START_ULL(func, schedule) \
int func(unsigned ncounts, unsigned long long *counts, \
unsigned long long *p_lb, unsigned long long *p_ub) { \
int status; \
unsigned long long stride, str, lb, ub; \
unsigned long long chunk_sz = 0; \
int gtid = __kmp_entry_gtid(); \
struct kmp_dim *dims = \
(struct kmp_dim *)__kmp_allocate(sizeof(struct kmp_dim) * ncounts); \
MKLOC(loc, KMP_STR(func)); \
for (unsigned i = 0; i < ncounts; ++i) { \
dims[i].lo = 0; \
dims[i].up = counts[i] - 1; \
dims[i].st = 1; \
} \
__kmpc_doacross_init(&loc, gtid, (int)ncounts, dims); \
lb = 0; \
ub = counts[0]; \
str = 1; \
KA_TRACE(20, (KMP_STR(func) ": T#%d, lb 0x%llx, ub 0x%llx, str " \
"0x%llx, chunk_sz 0x%llx\n", \
gtid, lb, ub, str, chunk_sz)); \
\
if ((str > 0) ? (lb < ub) : (lb > ub)) { \
KMP_DISPATCH_INIT_ULL(&loc, gtid, (schedule), lb, \
(str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, \
TRUE); \
status = \
KMP_DISPATCH_NEXT_ULL(&loc, gtid, NULL, (kmp_uint64 *)p_lb, \
(kmp_uint64 *)p_ub, (kmp_int64 *)&stride); \
if (status) { \
KMP_DEBUG_ASSERT((long long)stride == str); \
*p_ub += (str > 0) ? 1 : -1; \
} \
} else { \
status = 0; \
} \
KMP_DOACROSS_FINI(status, gtid); \
\
KA_TRACE( \
20, \
(KMP_STR( \
func) " exit: T#%d, *p_lb 0x%llx, *p_ub 0x%llx, returning %d\n", \
gtid, *p_lb, *p_ub, status)); \
__kmp_free(dims); \
return status; \
}
LOOP_DOACROSS_START_ULL(
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_STATIC_START),
kmp_sch_static)
LOOP_DOACROSS_START_ULL(
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_DYNAMIC_START),
kmp_sch_dynamic_chunked)
LOOP_DOACROSS_START_ULL(
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_GUIDED_START),
kmp_sch_guided_chunked)
LOOP_DOACROSS_RUNTIME_START_ULL(
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_RUNTIME_START),
kmp_sch_runtime)
#endif
// Combined parallel / loop worksharing constructs // Combined parallel / loop worksharing constructs
// //
// There are no ull versions (yet). // There are no ull versions (yet).
#define PARALLEL_LOOP_START(func, schedule, ompt_pre, ompt_post) \ #define PARALLEL_LOOP_START(func, schedule, ompt_pre, ompt_post) \
void func(void (*task)(void *), void *data, unsigned num_threads, long lb, \ void func(void (*task)(void *), void *data, unsigned num_threads, long lb, \
long ub, long str, long chunk_sz) { \ long ub, long str, long chunk_sz) { \
int gtid = __kmp_entry_gtid(); \ int gtid = __kmp_entry_gtid(); \
MKLOC(loc, #func); \ MKLOC(loc, KMP_STR(func)); \
KA_TRACE(20, \ KA_TRACE( \
(#func ": T#%d, lb 0x%lx, ub 0x%lx, str 0x%lx, chunk_sz 0x%lx\n", \ 20, \
(KMP_STR( \
func) ": T#%d, lb 0x%lx, ub 0x%lx, str 0x%lx, chunk_sz 0x%lx\n", \
gtid, lb, ub, str, chunk_sz)); \ gtid, lb, ub, str, chunk_sz)); \
\ \
ompt_pre(); \ ompt_pre(); \
\ \
if (__kmpc_ok_to_fork(&loc) && (num_threads != 1)) { \ if (__kmpc_ok_to_fork(&loc) && (num_threads != 1)) { \
if (num_threads != 0) { \ if (num_threads != 0) { \
__kmp_push_num_threads(&loc, gtid, num_threads); \ __kmp_push_num_threads(&loc, gtid, num_threads); \
} \ } \
__kmp_GOMP_fork_call(&loc, gtid, task, \ __kmp_GOMP_fork_call(&loc, gtid, task, \
(microtask_t)__kmp_GOMP_parallel_microtask_wrapper, \ (microtask_t)__kmp_GOMP_parallel_microtask_wrapper, \
9, task, data, num_threads, &loc, (schedule), lb, \ 9, task, data, num_threads, &loc, (schedule), lb, \
(str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz); \ (str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz); \
IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid)); \ IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid)); \
} else { \ } else { \
__kmp_GOMP_serialized_parallel(&loc, gtid, task); \ __kmp_GOMP_serialized_parallel(&loc, gtid, task); \
IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid)); \ IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid)); \
} \ } \
\ \
KMP_DISPATCH_INIT(&loc, gtid, (schedule), lb, \ KMP_DISPATCH_INIT(&loc, gtid, (schedule), lb, \
(str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, \ (str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, \
(schedule) != kmp_sch_static); \ (schedule) != kmp_sch_static); \
\ \
ompt_post(); \ ompt_post(); \
\ \
KA_TRACE(20, (#func " exit: T#%d\n", gtid)); \ KA_TRACE(20, (KMP_STR(func) " exit: T#%d\n", gtid)); \
} }
#if OMPT_SUPPORT && OMPT_OPTIONAL #if OMPT_SUPPORT && OMPT_OPTIONAL
#define OMPT_LOOP_PRE() \ #define OMPT_LOOP_PRE() \
omp_frame_t *parent_frame; \ omp_frame_t *parent_frame; \
if (ompt_enabled.enabled) { \ if (ompt_enabled.enabled) { \
__ompt_get_task_info_internal(0, NULL, NULL, &parent_frame, NULL, NULL); \ __ompt_get_task_info_internal(0, NULL, NULL, &parent_frame, NULL, NULL); \
▲ Show 20 Lines • Show All 379 Lines • ▼ Show 20 Lines#endif
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_PARALLEL_END)(); KMP_EXPAND_NAME(KMP_API_NAME_GOMP_PARALLEL_END)();
KA_TRACE(20, ("GOMP_parallel_sections exit: T#%d\n", gtid)); KA_TRACE(20, ("GOMP_parallel_sections exit: T#%d\n", gtid));
} }
#define PARALLEL_LOOP(func, schedule, ompt_pre, ompt_post) \ #define PARALLEL_LOOP(func, schedule, ompt_pre, ompt_post) \
void func(void (*task)(void *), void *data, unsigned num_threads, long lb, \ void func(void (*task)(void *), void *data, unsigned num_threads, long lb, \
long ub, long str, long chunk_sz, unsigned flags) { \ long ub, long str, long chunk_sz, unsigned flags) { \
int gtid = __kmp_entry_gtid(); \ int gtid = __kmp_entry_gtid(); \
MKLOC(loc, #func); \ MKLOC(loc, KMP_STR(func)); \
KA_TRACE(20, \ KA_TRACE( \
(#func ": T#%d, lb 0x%lx, ub 0x%lx, str 0x%lx, chunk_sz 0x%lx\n", \ 20, \
(KMP_STR( \
func) ": T#%d, lb 0x%lx, ub 0x%lx, str 0x%lx, chunk_sz 0x%lx\n", \
gtid, lb, ub, str, chunk_sz)); \ gtid, lb, ub, str, chunk_sz)); \
\ \
ompt_pre(); \ ompt_pre(); \
if (__kmpc_ok_to_fork(&loc) && (num_threads != 1)) { \ if (__kmpc_ok_to_fork(&loc) && (num_threads != 1)) { \
if (num_threads != 0) { \ if (num_threads != 0) { \
__kmp_push_num_threads(&loc, gtid, num_threads); \ __kmp_push_num_threads(&loc, gtid, num_threads); \
} \ } \
if (flags != 0) { \ if (flags != 0) { \
__kmp_push_proc_bind(&loc, gtid, (kmp_proc_bind_t)flags); \ __kmp_push_proc_bind(&loc, gtid, (kmp_proc_bind_t)flags); \
Show All 9 Lines void func(void (*task)(void *), void *data, unsigned num_threads, long lb, \
IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \ IF_OMPT_SUPPORT(OMPT_STORE_RETURN_ADDRESS(gtid);) \
KMP_DISPATCH_INIT(&loc, gtid, (schedule), lb, \ KMP_DISPATCH_INIT(&loc, gtid, (schedule), lb, \
(str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, \ (str > 0) ? (ub - 1) : (ub + 1), str, chunk_sz, \
(schedule) != kmp_sch_static); \ (schedule) != kmp_sch_static); \
task(data); \ task(data); \
KMP_EXPAND_NAME(KMP_API_NAME_GOMP_PARALLEL_END)(); \ KMP_EXPAND_NAME(KMP_API_NAME_GOMP_PARALLEL_END)(); \
ompt_post(); \ ompt_post(); \
\ \
KA_TRACE(20, (#func " exit: T#%d\n", gtid)); \ KA_TRACE(20, (KMP_STR(func) " exit: T#%d\n", gtid)); \
} }
PARALLEL_LOOP(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_PARALLEL_LOOP_STATIC), PARALLEL_LOOP(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_PARALLEL_LOOP_STATIC),
kmp_sch_static, OMPT_LOOP_PRE, OMPT_LOOP_POST) kmp_sch_static, OMPT_LOOP_PRE, OMPT_LOOP_POST)
PARALLEL_LOOP(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_PARALLEL_LOOP_DYNAMIC), PARALLEL_LOOP(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_PARALLEL_LOOP_DYNAMIC),
kmp_sch_dynamic_chunked, OMPT_LOOP_PRE, OMPT_LOOP_POST) kmp_sch_dynamic_chunked, OMPT_LOOP_PRE, OMPT_LOOP_POST)
PARALLEL_LOOP(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_PARALLEL_LOOP_GUIDED), PARALLEL_LOOP(KMP_EXPAND_NAME(KMP_API_NAME_GOMP_PARALLEL_LOOP_GUIDED),
kmp_sch_guided_chunked, OMPT_LOOP_PRE, OMPT_LOOP_POST) kmp_sch_guided_chunked, OMPT_LOOP_PRE, OMPT_LOOP_POST)
▲ Show 20 Lines • Show All 251 Lines • ▼ Show 20 Lines#endif
loop_bounds = (T *)task->shareds; loop_bounds = (T *)task->shareds;
loop_bounds[0] = start; loop_bounds[0] = start;
loop_bounds[1] = end + (up ? -1 : 1); loop_bounds[1] = end + (up ? -1 : 1);
__kmpc_taskloop(&loc, gtid, task, if_val, (kmp_uint64 *)&(loop_bounds[0]), __kmpc_taskloop(&loc, gtid, task, if_val, (kmp_uint64 *)&(loop_bounds[0]),
(kmp_uint64 *)&(loop_bounds[1]), (kmp_int64)step, nogroup, (kmp_uint64 *)&(loop_bounds[1]), (kmp_int64)step, nogroup,
sched, (kmp_uint64)num_tasks, (void *)task_dup); sched, (kmp_uint64)num_tasks, (void *)task_dup);
} }
// 4 byte version of GOMP_doacross_post
// This verison needs to create a temporary array which converts 4 byte
// integers into 8 byte integeres
template <typename T, bool need_conversion = (sizeof(long) == 4)>
void __kmp_GOMP_doacross_post(T *count);
template <> void __kmp_GOMP_doacross_post<long, true>(long *count) {
int gtid = __kmp_entry_gtid();
kmp_info_t *th = __kmp_threads[gtid];
MKLOC(loc, "GOMP_doacross_post");
kmp_int64 num_dims = th->th.th_dispatch->th_doacross_info[0];
kmp_int64 *vec =
(kmp_int64 *)__kmp_thread_malloc(th, sizeof(kmp_int64) * num_dims);
for (kmp_int64 i = 0; i < num_dims; ++i) {
vec[i] = (kmp_int64)count[i];
}
__kmpc_doacross_post(&loc, gtid, vec);
__kmp_thread_free(th, vec);
}
// 8 byte versions of GOMP_doacross_post
// This version can just pass in the count array directly instead of creating
// a temporary array
template <> void __kmp_GOMP_doacross_post<long, false>(long *count) {
int gtid = __kmp_entry_gtid();
MKLOC(loc, "GOMP_doacross_post");
__kmpc_doacross_post(&loc, gtid, RCAST(kmp_int64 *, count));
}
template <typename T> void __kmp_GOMP_doacross_wait(T first, va_list args) {
int gtid = __kmp_entry_gtid();
kmp_info_t *th = __kmp_threads[gtid];
MKLOC(loc, "GOMP_doacross_wait");
kmp_int64 num_dims = th->th.th_dispatch->th_doacross_info[0];
kmp_int64 *vec =
(kmp_int64 *)__kmp_thread_malloc(th, sizeof(kmp_int64) * num_dims);
vec[0] = (kmp_int64)first;
for (kmp_int64 i = 1; i < num_dims; ++i) {
T item = va_arg(args, T);
vec[i] = (kmp_int64)item;
}
__kmpc_doacross_wait(&loc, gtid, vec);
__kmp_thread_free(th, vec);
return;
}
#ifdef __cplusplus #ifdef __cplusplus
extern "C" { extern "C" {
#endif // __cplusplus #endif // __cplusplus
void KMP_EXPAND_NAME(KMP_API_NAME_GOMP_TASKLOOP)( void KMP_EXPAND_NAME(KMP_API_NAME_GOMP_TASKLOOP)(
void (*func)(void *), void *data, void (*copy_func)(void *, void *), void (*func)(void *), void *data, void (*copy_func)(void *, void *),
long arg_size, long arg_align, unsigned gomp_flags, unsigned long num_tasks, long arg_size, long arg_align, unsigned gomp_flags, unsigned long num_tasks,
int priority, long start, long end, long step) { int priority, long start, long end, long step) {
__GOMP_taskloop<long>(func, data, copy_func, arg_size, arg_align, gomp_flags, __GOMP_taskloop<long>(func, data, copy_func, arg_size, arg_align, gomp_flags,
num_tasks, priority, start, end, step); num_tasks, priority, start, end, step);
} }
void KMP_EXPAND_NAME(KMP_API_NAME_GOMP_TASKLOOP_ULL)( void KMP_EXPAND_NAME(KMP_API_NAME_GOMP_TASKLOOP_ULL)(
void (*func)(void *), void *data, void (*copy_func)(void *, void *), void (*func)(void *), void *data, void (*copy_func)(void *, void *),
long arg_size, long arg_align, unsigned gomp_flags, unsigned long num_tasks, long arg_size, long arg_align, unsigned gomp_flags, unsigned long num_tasks,
int priority, unsigned long long start, unsigned long long end, int priority, unsigned long long start, unsigned long long end,
unsigned long long step) { unsigned long long step) {
__GOMP_taskloop<unsigned long long>(func, data, copy_func, arg_size, __GOMP_taskloop<unsigned long long>(func, data, copy_func, arg_size,
arg_align, gomp_flags, num_tasks, arg_align, gomp_flags, num_tasks,
priority, start, end, step); priority, start, end, step);
} }
#endif void KMP_EXPAND_NAME(KMP_API_NAME_GOMP_DOACROSS_POST)(long *count) {
__kmp_GOMP_doacross_post(count);
}
void KMP_EXPAND_NAME(KMP_API_NAME_GOMP_DOACROSS_WAIT)(long first, ...) {
va_list args;
va_start(args, first);
__kmp_GOMP_doacross_wait<long>(first, args);
va_end(args);
}
void KMP_EXPAND_NAME(KMP_API_NAME_GOMP_DOACROSS_ULL_POST)(
unsigned long long *count) {
int gtid = __kmp_entry_gtid();
MKLOC(loc, "GOMP_doacross_ull_post");
__kmpc_doacross_post(&loc, gtid, RCAST(kmp_int64 *, count));
}
void KMP_EXPAND_NAME(KMP_API_NAME_GOMP_DOACROSS_ULL_WAIT)(
unsigned long long first, ...) {
va_list args;
va_start(args, first);
__kmp_GOMP_doacross_wait<unsigned long long>(first, args);
va_end(args);
}
#endif // OMP_45_ENABLED
/* The following sections of code create aliases for the GOMP_* functions, then /* The following sections of code create aliases for the GOMP_* functions, then
create versioned symbols using the assembler directive .symver. This is only create versioned symbols using the assembler directive .symver. This is only
pertinent for ELF .so library. The KMP_VERSION_SYMBOL macro is defined in pertinent for ELF .so library. The KMP_VERSION_SYMBOL macro is defined in
kmp_os.h */ kmp_os.h */
#ifdef KMP_USE_VERSION_SYMBOLS #ifdef KMP_USE_VERSION_SYMBOLS
// GOMP_1.0 versioned symbols // GOMP_1.0 versioned symbols
▲ Show 20 Lines • Show All 93 Lines • ▼ Show 20 Lines
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_SECTIONS_END_CANCEL, 40, "GOMP_4.0"); KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_SECTIONS_END_CANCEL, 40, "GOMP_4.0");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TARGET, 40, "GOMP_4.0"); KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TARGET, 40, "GOMP_4.0");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TARGET_DATA, 40, "GOMP_4.0"); KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TARGET_DATA, 40, "GOMP_4.0");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TARGET_END_DATA, 40, "GOMP_4.0"); KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TARGET_END_DATA, 40, "GOMP_4.0");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TARGET_UPDATE, 40, "GOMP_4.0"); KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TARGET_UPDATE, 40, "GOMP_4.0");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TEAMS, 40, "GOMP_4.0"); KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TEAMS, 40, "GOMP_4.0");
#endif #endif
// GOMP_4.5 versioned symbols
#if OMP_45_ENABLED #if OMP_45_ENABLED
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TASKLOOP, 45, "GOMP_4.5"); KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TASKLOOP, 45, "GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TASKLOOP_ULL, 45, "GOMP_4.5"); KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_TASKLOOP_ULL, 45, "GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_DOACROSS_POST, 45, "GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_DOACROSS_WAIT, 45, "GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_LOOP_DOACROSS_STATIC_START, 45,
"GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_LOOP_DOACROSS_DYNAMIC_START, 45,
"GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_LOOP_DOACROSS_GUIDED_START, 45,
"GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_LOOP_DOACROSS_RUNTIME_START, 45,
"GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_DOACROSS_ULL_POST, 45, "GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_DOACROSS_ULL_WAIT, 45, "GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_STATIC_START, 45,
"GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_DYNAMIC_START, 45,
"GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_GUIDED_START, 45,
"GOMP_4.5");
KMP_VERSION_SYMBOL(KMP_API_NAME_GOMP_LOOP_ULL_DOACROSS_RUNTIME_START, 45,
"GOMP_4.5");
#endif #endif
#endif // KMP_USE_VERSION_SYMBOLS #endif // KMP_USE_VERSION_SYMBOLS
#ifdef __cplusplus #ifdef __cplusplus
} // extern "C" } // extern "C"
#endif // __cplusplus #endif // __cplusplus

runtime/src/kmp_os.h

Show First 20 Lines • Show All 307 Lines • ▼ Show 20 Lines
#if KMP_HAVE_WEAK_ATTRIBUTE #if KMP_HAVE_WEAK_ATTRIBUTE
#define KMP_WEAK_ATTRIBUTE __attribute__((weak)) #define KMP_WEAK_ATTRIBUTE __attribute__((weak))
#else #else
#define KMP_WEAK_ATTRIBUTE /* Nothing */ #define KMP_WEAK_ATTRIBUTE /* Nothing */
#endif #endif
// Define KMP_VERSION_SYMBOL and KMP_EXPAND_NAME // Define KMP_VERSION_SYMBOL and KMP_EXPAND_NAME
#ifdef KMP_USE_VERSION_SYMBOLS #ifndef KMP_STR
#define KMP_STR(x) _KMP_STR(x) #define KMP_STR(x) _KMP_STR(x)
#define _KMP_STR(x) #x #define _KMP_STR(x) #x
#endif
#ifdef KMP_USE_VERSION_SYMBOLS
// If using versioned symbols, KMP_EXPAND_NAME prepends // If using versioned symbols, KMP_EXPAND_NAME prepends
// __kmp_api_ to the real API name // __kmp_api_ to the real API name
#define KMP_EXPAND_NAME(api_name) _KMP_EXPAND_NAME(api_name) #define KMP_EXPAND_NAME(api_name) _KMP_EXPAND_NAME(api_name)
#define _KMP_EXPAND_NAME(api_name) __kmp_api_##api_name #define _KMP_EXPAND_NAME(api_name) __kmp_api_##api_name
#define KMP_VERSION_SYMBOL(api_name, ver_num, ver_str) \ #define KMP_VERSION_SYMBOL(api_name, ver_num, ver_str) \
_KMP_VERSION_SYMBOL(api_name, ver_num, ver_str, "VERSION") _KMP_VERSION_SYMBOL(api_name, ver_num, ver_str, "VERSION")
#define _KMP_VERSION_SYMBOL(api_name, ver_num, ver_str, default_ver) \ #define _KMP_VERSION_SYMBOL(api_name, ver_num, ver_str, default_ver) \
__typeof__(__kmp_api_##api_name) __kmp_api_##api_name##_##ver_num##_alias \ __typeof__(__kmp_api_##api_name) __kmp_api_##api_name##_##ver_num##_alias \
▲ Show 20 Lines • Show All 624 Lines • Show Last 20 Lines