[OpenMP] Support for wasm32 architecture

[arsnyder16]

Summary

These are changes i made to build a static library of libomp to run on wasm32

using the emscripten tool chain

export CFLAGS="-pthread"
export CXXFLAGS="-pthread"
emcmake cmake \
  -DOPENMP_STANDALONE_BUILD=ON \
  -DOPENMP_ENABLE_OMPT_TOOLS=OFF \
  -DOPENMP_ENABLE_LIBOMPTARGET=OFF \
  -DLIBOMP_HAVE_OMPT_SUPPORT=OFF \
  -DLIBOMP_OMPT_SUPPORT=OFF \
  -DLIBOMP_OMPD_SUPPORT=OFF \
  -DLIBOMP_USE_DEBUGGER=OFF \
  -DLIBOMP_FORTRAN_MODULES=OFF \
  -DLIBOMP_ENABLE_SHARED=OFF \
  -DLIBOMP_ARCH=wasm32 \
  -DOPENMP_ENABLE_LIBOMPTARGET_PROFILING=OFF \
  .

emmake make -j

downstream libaries need to compile with
-pthread -fopenmp=libomp`
then link to libomp.a that was built above

[tianshilei1992]

Could you please rebase your patch and add context based on https://llvm.org/docs/Phabricator.html#requesting-a-review-via-the-web-interface? Thanks.

[penzn]

This was first shared as a patch here: https://github.com/llvm/llvm-project/issues/52714

For those who were added as reviewers, but were not part of the prior conversation, the patch wasn't presented as "it already works", though it should compile.

[arsnyder16]

In D142593#4081696, @penzn wrote:

This was first shared as a patch here: https://github.com/llvm/llvm-project/issues/52714

For those who were added as reviewers, but were not part of the prior conversation, the patch wasn't presented as "it already works", though it should compile.

This does work for my application, we don’t fully exhaust all of the omp feature so I doubt its a full solution, but does support the basics.

[arsnyder16]

-U999999 patch

[arsnyder16]

update patch for main

[penzn]

openmp/runtime/src/kmp_os.h

1289–1290

Emscripten has dlopen and dlsym emulation under a flag, though other ways to use Clang won't provide that.

openmp/runtime/src/kmp_platform.h

42

Maybe it should be __wasm32__ instead of __EMSCRIPTEN__, so that it is possible to build with non-Emscripten Clang (WASI-SDK, or even bare).

openmp/runtime/src/z_Linux_util.cpp

2450

Do you have to add those definitions because of how Wasm handles varargs?

[arsnyder16]

openmp/runtime/src/kmp_os.h

1289–1290

Emulation is only supported if you are also compiling with dynamic linking, which isn't always the case. I am not sure how critical these code paths are to keep.

https://github.com/emscripten-core/emscripten/blob/33a9cf1bceaaf52229285b6e77bf7f5747b18dba/src/library_dylink.js#L9

openmp/runtime/src/kmp_platform.h

42

Possibly, but the Emscripten tool chain is the one supplying LINUX/POSIX (via musl). Using just the clang compiler or other tool chains does not guarantee it will mimic a POSIX compatible system.

Another option that could be explored is getting this working without setting KMP_OS_LINUX at all. My guess would the main sticking point the code expecting to use pthreads if KMP_OS_LINUX is defined. I didn't actually try that i was looking for the path of least resistance, To make an official patch it might make sense to explore this further.

openmp/runtime/src/z_Linux_util.cpp

2450

Yea so this was the main runtime problem i encountered. I am little blurry on the exact details at it was about a year ago that i got this working.

For starters function pointer can be problematic with emscripten https://emscripten.org/docs/porting/guidelines/function_pointer_issues.html

What i do recall is that when the address of the function was looked up in the function table it was always the same signature, so my assumption would be similar to yours its related to using a function pointer to function using varargs.

We do have vargs in other areas of the the code base using libomp and i have yet to see any problem with those function. So it must be a combination of the function pointer and vargs.

Another interesting thing is that even recently i had to add even more microtasks signatures with more arguments (20-25). I wonder if the binaryen optimizer is adjusting call signatures in ways that causes these extra overloads. When I hit that problem it worked fine when using a debug build which would not have run through the optimizer.

As a side, i find it interesting that the limit is 15 arguments. Is that a documented restriction or are there other things at a play that should never allow it beyond 15?

[tianshilei1992]

Is it feasible to test it somehow? I doubt the regular way to invoke tests can work with WASM.

[arsnyder16]

In D142593#4101532, @tianshilei1992 wrote:

Is it feasible to test it somehow? I doubt the regular way to invoke tests can work with WASM.

Yes, the compiled wasm can be run through node. In our product we use gtest and we only ship our wasm in the browser so we build the gtest projects using emrun so we test within a headless chrome browser.

I think node would make sense here. Which tests specifically would you like to get working? runtime/test? Can you point me to any docs or examples of running the tests. As i get time i can poke at working through that.

[tianshilei1992]

In D142593#4102328, @arsnyder16 wrote:

In D142593#4101532, @tianshilei1992 wrote:

Is it feasible to test it somehow? I doubt the regular way to invoke tests can work with WASM.

Yes, the compiled wasm can be run through node. In our product we use gtest and we only ship our wasm in the browser so we build the gtest projects using emrun so we test within a headless chrome browser.

I think node would make sense here. Which tests specifically would you like to get working? runtime/test? Can you point me to any docs or examples of running the tests. As i get time i can poke at working through that.

The whole LLVM project depends on LIT test (https://llvm.org/docs/CommandGuide/lit.html). The LIT configuration for libomp is in openmp/runtime/test/lit.cfg and openmp/runtime/test/lit.site.cfg.in.

I think for the testing infrastructure, you might need CMake to check the existence of node (silently), and then replace those RUN lines accordingly (by those LIT configuration).

openmp/runtime/CMakeLists.txt

 if(LIBOMP_ARCH STREQUAL "aarch64")
   libomp_is_aarch64_a64fx(LIBOMP_DETECT_AARCH64_A64FX)
   if (LIBOMP_DETECT_AARCH64_A64FX)
     set(LIBOMP_ARCH "aarch64_a64fx")
     set(LIBOMP_ARCH_AARCH64_A64FX TRUE)
   endif()
 endif()
 
-libomp_check_variable(LIBOMP_ARCH 32e x86_64 32 i386 arm ppc64 ppc64le aarch64 aarch64_a64fx mic mips mips64 riscv64 loongarch64)
+libomp_check_variable(LIBOMP_ARCH 32e x86_64 32 i386 arm ppc64 ppc64le aarch64 aarch64_a64fx mic mips mips64 riscv64 loongarch64 wasm32)
 
 set(LIBOMP_LIB_TYPE normal CACHE STRING
   "Performance,Profiling,Stubs library (normal/profile/stubs)")
 libomp_check_variable(LIBOMP_LIB_TYPE normal profile stubs)
 # Set the OpenMP Year and Month associated with version
 set(LIBOMP_OMP_YEAR_MONTH 201611)
 set(LIBOMP_MIC_ARCH knc CACHE STRING
   "Intel(R) Many Integrated Core Architecture (Intel(R) MIC Architecture) (knf/knc).  Ignored if not Intel(R) MIC Architecture build.")

openmp/runtime/src/kmp_gsupport.cpp

#endif
   __kmpc_end_ordered(&loc, gtid);
 }
 
 // Dispatch macro defs
 //
 // They come in two flavors: 64-bit unsigned, and either 32-bit signed
 // (IA-32 architecture) or 64-bit signed (Intel(R) 64).
 
-#if KMP_ARCH_X86 || KMP_ARCH_ARM || KMP_ARCH_MIPS
+#if KMP_ARCH_X86 || KMP_ARCH_ARM || KMP_ARCH_MIPS || KMP_ARCH_WASM
 #define KMP_DISPATCH_INIT __kmp_aux_dispatch_init_4
 #define KMP_DISPATCH_FINI_CHUNK __kmp_aux_dispatch_fini_chunk_4
 #define KMP_DISPATCH_NEXT __kmpc_dispatch_next_4
 #else
 #define KMP_DISPATCH_INIT __kmp_aux_dispatch_init_8
 #define KMP_DISPATCH_FINI_CHUNK __kmp_aux_dispatch_fini_chunk_8
 #define KMP_DISPATCH_NEXT __kmpc_dispatch_next_8
 #endif /* KMP_ARCH_X86 */

openmp/runtime/src/kmp_os.h

 #define KMP_COMPILER_GCC 1
 #elif defined(_MSC_VER)
 #undef KMP_COMPILER_MSVC
 #define KMP_COMPILER_MSVC 1
 #else
 #error Unknown compiler
 #endif
 
-#if (KMP_OS_LINUX || KMP_OS_WINDOWS || KMP_OS_FREEBSD)
+#if (KMP_OS_LINUX || KMP_OS_WINDOWS || KMP_OS_FREEBSD) && !KMP_ARCH_WASM
 #define KMP_AFFINITY_SUPPORTED 1
 #if KMP_OS_WINDOWS && KMP_ARCH_X86_64
 #define KMP_GROUP_AFFINITY 1
 #else
 #define KMP_GROUP_AFFINITY 0
 #endif
 #else
 #define KMP_AFFINITY_SUPPORTED 0

 typedef long long kmp_int64;
 typedef unsigned long long kmp_uint64;
 #define KMP_INT32_SPEC "d"
 #define KMP_UINT32_SPEC "u"
 #define KMP_INT64_SPEC "lld"
 #define KMP_UINT64_SPEC "llu"
 #endif /* KMP_OS_UNIX */
 
-#if KMP_ARCH_X86 || KMP_ARCH_ARM || KMP_ARCH_MIPS
+#if KMP_ARCH_X86 || KMP_ARCH_ARM || KMP_ARCH_MIPS || KMP_ARCH_WASM
 #define KMP_SIZE_T_SPEC KMP_UINT32_SPEC
 #elif KMP_ARCH_X86_64 || KMP_ARCH_PPC64 || KMP_ARCH_AARCH64 ||                 \
     KMP_ARCH_MIPS64 || KMP_ARCH_RISCV64 || KMP_ARCH_LOONGARCH64
 #define KMP_SIZE_T_SPEC KMP_UINT64_SPEC
 #else
 #error "Can't determine size_t printf format specifier."
 #endif
 
-#if KMP_ARCH_X86 || KMP_ARCH_ARM
+#if KMP_ARCH_X86 || KMP_ARCH_ARM || KMP_ARCH_WASM
 #define KMP_SIZE_T_MAX (0xFFFFFFFF)
 #else
 #define KMP_SIZE_T_MAX (0xFFFFFFFFFFFFFFFF)
 #endif
 
 typedef size_t kmp_size_t;
 typedef float kmp_real32;
 typedef double kmp_real64;
#define TCX_SYNC_4(a, b, c)                                                    \
   KMP_COMPARE_AND_STORE_REL32((volatile kmp_int32 *)(volatile void *)&(a),     \
                               (kmp_int32)(b), (kmp_int32)(c))
 #define TCR_SYNC_8(a) (a)
 #define TCW_SYNC_8(a, b) (a) = (b)
 #define TCX_SYNC_8(a, b, c)                                                    \
   KMP_COMPARE_AND_STORE_REL64((volatile kmp_int64 *)(volatile void *)&(a),     \
                               (kmp_int64)(b), (kmp_int64)(c))
 
-#if KMP_ARCH_X86 || KMP_ARCH_MIPS
+#if KMP_ARCH_X86 || KMP_ARCH_MIPS || KMP_ARCH_WASM
 // What about ARM?
 #define TCR_PTR(a) ((void *)TCR_4(a))
 #define TCW_PTR(a, b) TCW_4((a), (b))
 #define TCR_SYNC_PTR(a) ((void *)TCR_SYNC_4(a))
 #define TCW_SYNC_PTR(a, b) TCW_SYNC_4((a), (b))
 #define TCX_SYNC_PTR(a, b, c) ((void *)TCX_SYNC_4((a), (b), (c)))
 
 #else /* 64 bit pointers */
return p->compare_exchange_strong(
       expected, desired, std::memory_order_release, std::memory_order_relaxed);
 }
 
 // Symbol lookup on Linux/Windows
 #if KMP_OS_WINDOWS
 extern void *__kmp_lookup_symbol(const char *name);
 #define KMP_DLSYM(name) __kmp_lookup_symbol(name)
 #define KMP_DLSYM_NEXT(name) nullptr
+#elif KMP_ARCH_WASM
+#define KMP_DLSYM(name) nullptr
+#define KMP_DLSYM_NEXT(name) nullptr
 #else
 #define KMP_DLSYM(name) dlsym(RTLD_DEFAULT, name)
 #define KMP_DLSYM_NEXT(name) dlsym(RTLD_NEXT, name)
 #endif
 
 #endif /* KMP_OS_H */

openmp/runtime/src/kmp_platform.h

 #undef KMP_OS_DARWIN
 #define KMP_OS_DARWIN 1
 #endif
 
 // in some ppc64 linux installations, only the second condition is met
 #if (defined __linux)
 #undef KMP_OS_LINUX
 #define KMP_OS_LINUX 1
-#elif (defined __linux__)
+#elif (defined __linux__) || defined(__EMSCRIPTEN__)
 #undef KMP_OS_LINUX
 #define KMP_OS_LINUX 1
 #else
 #endif
 
 #if (defined __DragonFly__)
 #undef KMP_OS_DRAGONFLY
 #define KMP_OS_DRAGONFLY 1

 #if defined(KMP_ARCH_ARMV3) || defined(__ARM_ARCH_2__)
 #define KMP_ARCH_ARMV2 1
 #endif
 
 #if defined(KMP_ARCH_ARMV2)
 #define KMP_ARCH_ARM 1
 #endif
 
+#if defined(__wasm32__)
+#define KMP_ARCH_WASM 1
+#endif
+
 #if defined(__MIC__) || defined(__MIC2__)
 #define KMP_MIC 1
 #if __MIC2__ || __KNC__
 #define KMP_MIC1 0
 #define KMP_MIC2 1
 #else
 #define KMP_MIC1 1
 #define KMP_MIC2 0
 #endif
 #else
 #define KMP_MIC 0
 #define KMP_MIC1 0
 #define KMP_MIC2 0
 #endif
 
 /* Specify 32 bit architectures here */
-#define KMP_32_BIT_ARCH (KMP_ARCH_X86 || KMP_ARCH_ARM || KMP_ARCH_MIPS)
+#define KMP_32_BIT_ARCH (KMP_ARCH_X86 || KMP_ARCH_ARM || KMP_ARCH_MIPS || KMP_ARCH_WASM)
 
 // Platforms which support Intel(R) Many Integrated Core Architecture
 #define KMP_MIC_SUPPORTED                                                      \
   ((KMP_ARCH_X86 || KMP_ARCH_X86_64) && (KMP_OS_LINUX || KMP_OS_WINDOWS))
 
 // TODO: Fixme - This is clever, but really fugly
 #if (1 != KMP_ARCH_X86 + KMP_ARCH_X86_64 + KMP_ARCH_ARM + KMP_ARCH_PPC64 +     \
               KMP_ARCH_AARCH64 + KMP_ARCH_MIPS + KMP_ARCH_MIPS64 +             \
-              KMP_ARCH_RISCV64 + KMP_ARCH_LOONGARCH64)
+              KMP_ARCH_RISCV64 + KMP_ARCH_LOONGARCH64 + KMP_ARCH_WASM)
 #error Unknown or unsupported architecture
 #endif
 
 #endif // KMP_PLATFORM_H

openmp/runtime/src/kmp_runtime.cpp

#endif
     } else if (atomic_available) {
       retval = atomic_reduce_block;
     }
 #else
 #error "Unknown or unsupported OS"
 #endif // KMP_OS_LINUX || KMP_OS_DRAGONFLY || KMP_OS_FREEBSD || KMP_OS_NETBSD ||
        // KMP_OS_OPENBSD || KMP_OS_WINDOWS || KMP_OS_DARWIN || KMP_OS_HURD
 
-#elif KMP_ARCH_X86 || KMP_ARCH_ARM || KMP_ARCH_AARCH || KMP_ARCH_MIPS
+#elif KMP_ARCH_X86 || KMP_ARCH_ARM || KMP_ARCH_AARCH || KMP_ARCH_MIPS || KMP_ARCH_WASM
 
 #if KMP_OS_LINUX || KMP_OS_FREEBSD || KMP_OS_WINDOWS || KMP_OS_HURD
 
     // basic tuning
 
     if (atomic_available) {
       if (num_vars <= 2) { // && ( team_size <= 8 ) due to false-sharing ???
         retval = atomic_reduce_block;

openmp/runtime/src/z_Linux_asm.S

#ifdef __ELF__
     .size KMP_PREFIX_UNDERSCORE(__kmp_unnamed_critical_addr),8
 #endif
 #endif /* KMP_ARCH_PPC64 || KMP_ARCH_AARCH64 || KMP_ARCH_MIPS64 ||
           KMP_ARCH_RISCV64 || KMP_ARCH_LOONGARCH64 */
 
 #if KMP_OS_LINUX
 # if KMP_ARCH_ARM
 .section .note.GNU-stack,"",%progbits
-# else
+# elif !KMP_ARCH_WASM
 .section .note.GNU-stack,"",@progbits
 # endif
 #endif
+
+#if KMP_ARCH_WASM
+.data
+.global .gomp_critical_user_
+.global .gomp_critical_user_.var
+.global .gomp_critical_user_.reduction.var
+.global __kmp_unnamed_critical_addr
+.gomp_critical_user_:
+.zero 4
+.size .gomp_critical_user_, 4
+.gomp_critical_user_.var:
+.zero 4
+.size .gomp_critical_user_.var, 4
+.gomp_critical_user_.reduction.var:
+.zero 4
+.size .gomp_critical_user_.reduction.var, 4
+__kmp_unnamed_critical_addr:
+    .4byte .gomp_critical_user_
+    .size __kmp_unnamed_critical_addr, 4
+#endif

openmp/runtime/src/z_Linux_util.cpp

int __kmp_futex_determine_capable() {
   KA_TRACE(10, ("__kmp_futex_determine_capable: futex syscall%s supported\n",
                 retval ? "" : " not"));
 
   return retval;
 }
 
 #endif // KMP_USE_FUTEX
 
-#if (KMP_ARCH_X86 || KMP_ARCH_X86_64) && (!KMP_ASM_INTRINS)
+#if (KMP_ARCH_X86 || KMP_ARCH_X86_64 || KMP_ARCH_WASM) && (!KMP_ASM_INTRINS)
 /* Only 32-bit "add-exchange" instruction on IA-32 architecture causes us to
    use compare_and_store for these routines */
 
 kmp_int8 __kmp_test_then_or8(volatile kmp_int8 *p, kmp_int8 d) {
   kmp_int8 old_value, new_value;
 
   old_value = TCR_1(*p);
   new_value = old_value | d;
kmp_uint32 __kmp_test_then_and32(volatile kmp_uint32 *p, kmp_uint32 d) {
   while (!KMP_COMPARE_AND_STORE_REL32(p, old_value, new_value)) {
     KMP_CPU_PAUSE();
     old_value = TCR_4(*p);
     new_value = old_value & d;
   }
   return old_value;
 }
 
-#if KMP_ARCH_X86
+#if KMP_ARCH_X86 || KMP_ARCH_WASM
 kmp_int8 __kmp_test_then_add8(volatile kmp_int8 *p, kmp_int8 d) {
   kmp_int8 old_value, new_value;
 
   old_value = TCR_1(*p);
   new_value = old_value + d;
 
   while (!KMP_COMPARE_AND_STORE_REL8(p, old_value, new_value)) {
     KMP_CPU_PAUSE();
finish: // Clean up and exit.
   return running_threads;
 
 } // __kmp_get_load_balance
 
 #endif // KMP_OS_DARWIN
 
 #endif // USE_LOAD_BALANCE
 
+
+typedef void (*microtask_t0)(int *, int *);
+typedef void (*microtask_t1)(int *, int *, void*);
+typedef void (*microtask_t2)(int *, int *, void*, void*);
+typedef void (*microtask_t3)(int *, int *, void*, void*, void*);
+typedef void (*microtask_t4)(int *, int *, void*, void*, void*, void*);
+typedef void (*microtask_t5)(int *, int *, void*, void*, void*, void*, void*);
+typedef void (*microtask_t6)(int *, int *, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t7)(int *, int *, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t8)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t9)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t10)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t11)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t12)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t13)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t14)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t15)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t16)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t17)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t18)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t19)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t20)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t21)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t22)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t23)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t24)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+typedef void (*microtask_t25)(int *, int *, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*, void*);
+
 #if !(KMP_ARCH_X86 || KMP_ARCH_X86_64 || KMP_MIC ||                            \
       ((KMP_OS_LINUX || KMP_OS_DARWIN) && KMP_ARCH_AARCH64) ||                 \
       KMP_ARCH_PPC64 || KMP_ARCH_RISCV64 || KMP_ARCH_LOONGARCH64 ||            \
       KMP_ARCH_ARM)
 
 // we really only need the case with 1 argument, because CLANG always build
 // a struct of pointers to shared variables referenced in the outlined function
 int __kmp_invoke_microtask(microtask_t pkfn, int gtid, int tid, int argc,
                            void *p_argv[]
 #if OMPT_SUPPORT
                            ,
                            void **exit_frame_ptr
 #endif
 ) {
 #if OMPT_SUPPORT
   *exit_frame_ptr = OMPT_GET_FRAME_ADDRESS(0);
 #endif
 
   switch (argc) {
   default:
     fprintf(stderr, "Too many args to microtask: %d!\n", argc);
     fflush(stderr);
     exit(-1);
   case 0:
-    (*pkfn)(&gtid, &tid);
+    (*(microtask_t0)pkfn)(&gtid, &tid);
     break;
   case 1:
-    (*pkfn)(&gtid, &tid, p_argv[0]);
+    (*(microtask_t1)pkfn)(&gtid, &tid, p_argv[0]);
     break;
   case 2:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1]);
+    (*(microtask_t2)pkfn)(&gtid, &tid, p_argv[0], p_argv[1]);
     break;
   case 3:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2]);
+    (*(microtask_t3)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2]);
     break;
   case 4:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3]);
+    (*(microtask_t4)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3]);
     break;
   case 5:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4]);
+    (*(microtask_t5)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4]);
     break;
   case 6:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+    (*(microtask_t6)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
             p_argv[5]);
     break;
   case 7:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+    (*(microtask_t7)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
             p_argv[5], p_argv[6]);
     break;
   case 8:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+    (*(microtask_t8)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
             p_argv[5], p_argv[6], p_argv[7]);
     break;
   case 9:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+    (*(microtask_t9)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
             p_argv[5], p_argv[6], p_argv[7], p_argv[8]);
     break;
   case 10:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+    (*(microtask_t10)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
             p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9]);
     break;
   case 11:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+    (*(microtask_t11)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
             p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10]);
     break;
   case 12:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+    (*(microtask_t12)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
             p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
             p_argv[11]);
     break;
   case 13:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+    (*(microtask_t13)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
             p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
             p_argv[11], p_argv[12]);
     break;
   case 14:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+    (*(microtask_t14)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
             p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
             p_argv[11], p_argv[12], p_argv[13]);
     break;
   case 15:
-    (*pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+    (*(microtask_t15)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
             p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
             p_argv[11], p_argv[12], p_argv[13], p_argv[14]);
     break;
+  case 16:
+    (*(microtask_t16)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+            p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
+            p_argv[11], p_argv[12], p_argv[13], p_argv[14], p_argv[15]);
+    break;
+  case 17:
+    (*(microtask_t17)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+            p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
+            p_argv[11], p_argv[12], p_argv[13], p_argv[14], p_argv[15], p_argv[16]);
+    break;
+  case 18:
+    (*(microtask_t18)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+            p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
+            p_argv[11], p_argv[12], p_argv[13], p_argv[14], p_argv[15], p_argv[16], p_argv[17]);
+    break;
+  case 19:
+    (*(microtask_t19)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+            p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
+            p_argv[11], p_argv[12], p_argv[13], p_argv[14], p_argv[15], p_argv[16], p_argv[17], p_argv[18]);
+    break;
+  case 20:
+    (*(microtask_t20)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+            p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
+            p_argv[11], p_argv[12], p_argv[13], p_argv[14], p_argv[15], p_argv[16], p_argv[17], p_argv[18], p_argv[19]);
+    break;
+  case 21:
+    (*(microtask_t21)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+            p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
+            p_argv[11], p_argv[12], p_argv[13], p_argv[14], p_argv[15], p_argv[16], p_argv[17], p_argv[18], p_argv[19], p_argv[20]);
+    break;
+  case 22:
+    (*(microtask_t22)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+            p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
+            p_argv[11], p_argv[12], p_argv[13], p_argv[14], p_argv[15], p_argv[16], p_argv[17], p_argv[18], p_argv[19], p_argv[20], p_argv[21]);
+    break;
+  case 23:
+    (*(microtask_t23)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+            p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
+            p_argv[11], p_argv[12], p_argv[13], p_argv[14], p_argv[15], p_argv[16], p_argv[17], p_argv[18], p_argv[19], p_argv[20], p_argv[21], p_argv[22]);
+    break;
+  case 24:
+    (*(microtask_t24)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+            p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
+            p_argv[11], p_argv[12], p_argv[13], p_argv[14], p_argv[15], p_argv[16], p_argv[17], p_argv[18], p_argv[19], p_argv[20], p_argv[21], p_argv[22], p_argv[23]);
+    break;
+  case 25:
+    (*(microtask_t25)pkfn)(&gtid, &tid, p_argv[0], p_argv[1], p_argv[2], p_argv[3], p_argv[4],
+            p_argv[5], p_argv[6], p_argv[7], p_argv[8], p_argv[9], p_argv[10],
+            p_argv[11], p_argv[12], p_argv[13], p_argv[14], p_argv[15], p_argv[16], p_argv[17], p_argv[18], p_argv[19], p_argv[20], p_argv[21], p_argv[22], p_argv[23], p_argv[24]);
+    break;
   }
 
   return 1;
 }
 
 #endif
 
 #if KMP_OS_LINUX