Solve 'Too many args to microtask' problem

[pawosm01]

Summary

This patch solves 'Too many args to microtask' problem which occurs
while executing lulesh2.0.3 benchmark on AArch64.

To solve this I had to wrtite AArch64 assembly version of
__kmp_invoke_microtask() function, similar to x86 and x86_64
implementations.

[jcownie]

On the microtask stuff: I have no objection to this, but I'm very surprised that it is needed, since I was under the impression that Clang/OpenMP only ever emits an outlined function for the parallel region that takes a single pointer argument, and then generates code that uses offsets from that to find all the actual arguments.

It also seems very odd that most code is OK, but LULESH fails. (I klnow some SpecOMP codes have a lot of references to shared variables...)

Have you tried test cases which simply pass large numbers of arguments?
(So, my concern here is that you may be fixing the wrong bug!)

[pawosm01]

This needs to be round up

[pawosm01]

In D19879#420879, @jcownie wrote:

On the microtask stuff: I have no objection to this, but I'm very surprised that it is needed, since I was under the impression that Clang/OpenMP only ever emits an outlined function for the parallel region that takes a single pointer argument, and then generates code that uses offsets from that to find all the actual arguments.

It also seems very odd that most code is OK, but LULESH fails. (I klnow some SpecOMP codes have a lot of references to shared variables...)

Have you tried test cases which simply pass large numbers of arguments?
(So, my concern here is that you may be fixing the wrong bug!)

Hi James,

This LULESH benchmark doesn't do anything unusual. It doesn't do any explicit tasking, only implicit tasks due to plain old 'parallel for' loops with 'firstprivate' clause.
I added raise(11); in place where "Too many args to microtask" is printed and stared LULESH (built with -g) along with 'ulimit -c unlimited' and 'OMP_NUM_THREADS=1'. As it crashed on raised signal, I could analyse dumped core file in gdb:
(gdb) bt
#0 0x000003ff8f981450 in raise () from /lib64/libpthread.so.0
#1 0x000003ff8fa298d8 in ..kmp_invoke_microtask () from /home/pawosm01/llvm/lib/libomp.so
#2 0x000003ff8fa097f4 in ..kmp_fork_call () from /home/pawosm01/llvm/lib/libomp.so
#3 0x000003ff8fa00048 in ..kmpc_fork_call () from /home/pawosm01/llvm/lib/libomp.so
#4 0x00000000004036a0 in CalcEnergyForElems (p_new=0x3ef85120, e_new=<optimized out>, q_new=<optimized out>, bvc=<optimized out>, pbvc=<optimized out>, p_old=<optimized out>, e_old=<optimized out>, q_old=<optimized out>, compHalfStep=<optimized out>, vnewc=0x3ed53de0, work=<optimized out>, delvc=<optimized out>,

e_cut=<optimized out>, q_cut=<optimized out>, emin=<optimized out>, qq_old=<optimized out>, ql_old=<optimized out>, rho0=<optimized out>, length=1056463064, regElemList=<optimized out>, compression=<optimized out>, pmin=<optimized out>, p_cut=<optimized out>, eosvmax=<optimized out>) at lulesh.cc:2145

#5 EvalEOSForElems (domain=..., vnewc=<optimized out>, numElemReg=<optimized out>, regElemList=<optimized out>, rep=<optimized out>) at lulesh.cc:2318
#6 ApplyMaterialPropertiesForElems (domain=..., vnew=<optimized out>) at lulesh.cc:2424
#7 LagrangeElements (domain=..., numElem=<optimized out>) at lulesh.cc:2463
#8 LagrangeLeapFrog (domain=...) at lulesh.cc:2656
#9 main (argc=<optimized out>, argv=<optimized out>) at lulesh.cc:2774

lulesh.cc line 2145 is a start of parallel region:

#pragma omp parallel for firstprivate(length, rho0, emin, e_cut)

Inside of it, 13 arrays from the outside of this region are accessed plus we got 4 firstprivate variables in the clause, also defined outside the region. This gives 17 variables and this is reflected by error message:

Running problem size 30^3 per domain until completion
Num processors: 1
Num threads: 6
Total number of elements: 27000

To run other sizes, use -s <integer>.
To run a fixed number of iterations, use -i <integer>.
To run a more or less balanced region set, use -b <integer>.
To change the relative costs of regions, use -c <integer>.
To print out progress, use -p
To write an output file for VisIt, use -v
See help (-h) for more options

Too many args to microtask: 17!
Too many args to microtask: 17!
Too many args to microtask: 17!
Too many args to microtask: 17!

I removed use of arbitrarily selected (the least used) two of them in the loop body (as the switch in ..kmp_invoke_kicrotask() handles up to 15 params) and - as expected - this error message stopped to appear.

[pawosm01]

Simple test case:

$ cat case16.c
#include <stdio.h>

int main()
{

int i1 = 0;
int i2 = 1;
int i3 = 2;
int i4 = 3;
int i5 = 4;
int i6 = 6;
int i7 = 7;
int i8 = 8;
int i9 = 9;
int i10 = 10;
int i11 = 11;
int i12 = 12;
int i13 = 13;
int i14 = 14;
int i15 = 15;
int i16 = 16;

#pragma omp parallel for firstprivate(i1, i2, i3, i4, i5, i6, i7, i8, i9, i10, i11, i12, i13, i14, i15, i16)
for (int i = 0; i < i16; i++) {
  printf("%d\n", i + i1 + i2 + i3 + i4 + i5 + i6 + i7 + i8 + i9 + i10 + i11 + i12 + i13 + i14 + i15 + i16);
}

return 0;

}

$ $HOME/llvm/bin/clang -Wall -O3 -fopenmp -std=c11 -o case16 case16.c -Wl,-rpath=$HOME/llvm/lib
$ OMP_NUM_THREADS=1 ./case16
Too many args to microtask: 16!
$

[AndreyChurbanov]

LGTM.

Still unclear though how this problem corresponds with the comment just above the error print "Too many args to microtask" in z_Linux_util.c line 2579:

#if !(KMP_ARCH_X86 || KMP_ARCH_X86_64 || KMP_MIC)
we really only need the case with 1 argument, because CLANG always build
a struct of pointers to shared variables referenced in the outlined function

Probably something is wrong with compiler code generation for AArch64...

[hfinkel]

Alexey,

Could you please comment on this? Is the comment inaccurate?

-Hal

• Original Message -----

From: "Andrey Churbanov via Openmp-commits" <[e-mail]>
To: "pawel osmialowski" <[e-mail]>, "jonathan l peyton" <[e-mail]>, "andrey
churbanov" <[e-mail]>
Cc: [e-mail], "amara emerson" <[e-mail]>
Sent: Thursday, May 12, 2016 1:43:41 PM
Subject: Re: [Openmp-commits] [PATCH] D19879: Solve 'Too many args to microtask' problem

AndreyChurbanov accepted this revision.
AndreyChurbanov added a comment.
This revision is now accepted and ready to land.

LGTM.

Still unclear though how this problem corresponds with the comment
just above the error print "Too many args to microtask" in
z_Linux_util.c line 2579:

#if !(KMP_ARCH_X86 || KMP_ARCH_X86_64 || KMP_MIC)
we really only need the case with 1 argument, because CLANG always
build
a struct of pointers to shared variables referenced in the
outlined function

Probably something is wrong with compiler code generation for
AArch64...

Repository:

rL LLVM

http://reviews.llvm.org/D19879

---

Openmp-commits mailing list
[e-mail]
http://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-commits

openmp/trunk/runtime/src/z_Linux_asm.s

.macro KMP_CFI_REGISTER reg
   .cfi_def_cfa_register \reg
 .endm
 .macro KMP_CFI_DEF reg, sz
   .cfi_def_cfa  \reg,\sz
 .endm
 # endif // KMP_OS_DARWIN
 #endif // KMP_ARCH_X86 || KMP_ARCH_x86_64
 
+#if KMP_OS_LINUX && KMP_ARCH_AARCH64
+
+#  define KMP_PREFIX_UNDERSCORE(x) x  // no extra underscore for Linux* OS symbols
+// Format labels so that they don't override function names in gdb's backtraces
+#  define KMP_LABEL(x) .L_##x         // local label hidden from backtraces
+
+.macro ALIGN size
+  .align 1<<(\size)
+.endm
+
+.macro DEBUG_INFO proc
+  .cfi_endproc
+// Not sure why we need .type and .size for the functions
+  ALIGN 2
+  .type  \proc,@function
+  .size  \proc,.-\proc
+.endm
+
+.macro PROC proc
+  ALIGN 2
+  .globl KMP_PREFIX_UNDERSCORE(\proc)
+KMP_PREFIX_UNDERSCORE(\proc):
+  .cfi_startproc
+.endm
+
+#endif // KMP_OS_LINUX && KMP_ARCH_AARCH64
 
 // -----------------------------------------------------------------------
 // data
 // -----------------------------------------------------------------------
 
 #ifdef KMP_GOMP_COMPAT
 
 //
//
         ret
 
         DEBUG_INFO __kmp_bsr32
 
   
 // -----------------------------------------------------------------------
 #endif /* KMP_ARCH_X86_64 */
 
+// '
+#if KMP_OS_LINUX && KMP_ARCH_AARCH64
+
+//------------------------------------------------------------------------
+//
+// typedef void (*microtask_t)( int *gtid, int *tid, ... );
+//
+// int
+// __kmp_invoke_microtask( void (*pkfn) (int gtid, int tid, ...),
+//               int gtid, int tid,
+//                         int argc, void *p_argv[] ) {
+//    (*pkfn)( & gtid, & tid, argv[0], ... );
+//    return 1;
+// }
+//
+// parameters:
+//  x0: pkfn
+//  w1: gtid
+//  w2: tid
+//  w3: argc
+//  x4: p_argv
+//  x5: &exit_frame
+//
+// locals:
+//  __gtid: gtid parm pushed on stack so can pass &gtid to pkfn
+//  __tid:  tid parm pushed on stack so can pass &tid to pkfn
+//
+// reg temps:
+//   x8:  used to hold pkfn address
+//   w9:  used as temporary for number of pkfn parms
+//  x10:  used to traverse p_argv array
+//  x11:  used as temporary for stack placement calculation
+//  x12:  used as temporary for stack parameters
+//  x19:  used to preserve exit_frame_ptr, callee-save
+//
+// return:  w0  (always 1/TRUE)
+//
+
+__gtid = 4
+__tid = 8
+
+// -- Begin __kmp_invoke_microtask
+// mark_begin;
+  .text
+  PROC __kmp_invoke_microtask
+
+  stp  x29, x30, [sp, #-16]!
+# if OMPT_SUPPORT
+  stp  x19, x20, [sp, #-16]!
+# endif
+  mov x29, sp
+
+  orr w9, wzr, #1
+  add w9, w9, w3, lsr #1
+  sub sp, sp, w9, lsl #4
+  mov x11, sp
+
+  mov x8, x0
+  str w1, [x29, #-__gtid]
+  str w2, [x29, #-__tid]
+  mov w9, w3
+  mov x10, x4
+# if OMPT_SUPPORT
+  mov x19, x5
+  str x29, [x19]
+# endif
+
+  sub x0, x29, #__gtid
+  sub x1, x29, #__tid
+
+  cbz w9, KMP_LABEL(kmp_1)
+  ldr x2, [x10]
+
+  sub w9, w9, #1
+  cbz w9, KMP_LABEL(kmp_1)
+  ldr x3, [x10, #8]!
+
+  sub w9, w9, #1
+  cbz w9, KMP_LABEL(kmp_1)
+  ldr x4, [x10, #8]!
+
+  sub w9, w9, #1
+  cbz w9, KMP_LABEL(kmp_1)
+  ldr x5, [x10, #8]!
+
+  sub w9, w9, #1
+  cbz w9, KMP_LABEL(kmp_1)
+  ldr x6, [x10, #8]!
+
+  sub w9, w9, #1
+  cbz w9, KMP_LABEL(kmp_1)
+  ldr x7, [x10, #8]!
+
+KMP_LABEL(kmp_0):
+  sub w9, w9, #1
+  cbz w9, KMP_LABEL(kmp_1)
+  ldr x12, [x10, #8]!
+  str x12, [x11], #8
+  b KMP_LABEL(kmp_0)
+KMP_LABEL(kmp_1):
+  blr x8
+  orr w0, wzr, #1
+  mov sp, x29
+# if OMPT_SUPPORT
+  str xzr, [x19]
+  ldp x19, x20, [sp], #16
+# endif
+  ldp x29, x30, [sp], #16
+  ret
+
+  DEBUG_INFO __kmp_invoke_microtask
+// -- End  __kmp_invoke_microtask
+
+#endif /* KMP_OS_LINUX && KMP_ARCH_AARCH64 */
+
 #if KMP_ARCH_ARM
     .data
     .comm .gomp_critical_user_,32,8
     .data
     .align 4
     .global __kmp_unnamed_critical_addr
 __kmp_unnamed_critical_addr:
     .4byte .gomp_critical_user_

openmp/trunk/runtime/src/z_Linux_util.c

__kmp_test_then_and32( volatile kmp_int32 *p, kmp_int32 d )
     {
         KMP_CPU_PAUSE();
         old_value = TCR_4( *p );
         new_value = old_value & d;
     }
     return old_value;
 }
 
-# if KMP_ARCH_X86 || KMP_ARCH_PPC64 || KMP_ARCH_AARCH64
+# if KMP_ARCH_X86 || KMP_ARCH_PPC64 || (KMP_OS_LINUX && KMP_ARCH_AARCH64)
 kmp_int8
 __kmp_test_then_add8( volatile kmp_int8 *p, kmp_int8 d )
 {
     kmp_int8 old_value, new_value;
 
     old_value = TCR_1( *p );
     new_value = old_value + d;
 
__kmp_test_then_add64( volatile kmp_int64 *p, kmp_int64 d )
     while ( ! KMP_COMPARE_AND_STORE_REL64 ( p, old_value, new_value ) )
     {
         KMP_CPU_PAUSE();
         old_value = TCR_8( *p );
         new_value = old_value + d;
     }
     return old_value;
 }
-# endif /* KMP_ARCH_X86 */
+# endif /* KMP_ARCH_X86 || KMP_ARCH_PPC64 || (KMP_OS_LINUX && KMP_ARCH_AARCH64) */
 
 kmp_int64
 __kmp_test_then_or64( volatile kmp_int64 *p, kmp_int64 d )
 {
     kmp_int64 old_value, new_value;
 
     old_value = TCR_8( *p );
     new_value = old_value | d;
__kmp_get_load_balance( int max )
     return running_threads;
 
 } // __kmp_get_load_balance
 
 # endif // KMP_OS_DARWIN
 
 #endif // USE_LOAD_BALANCE
 
-#if !(KMP_ARCH_X86 || KMP_ARCH_X86_64 || KMP_MIC)
+#if !(KMP_ARCH_X86 || KMP_ARCH_X86_64 || KMP_MIC || (KMP_OS_LINUX && KMP_ARCH_AARCH64))
 
 // we really only need the case with 1 argument, because CLANG always build
 // a struct of pointers to shared variables referenced in the outlined function
 int
 __kmp_invoke_microtask( microtask_t pkfn,
                         int gtid, int tid,
                         int argc, void *p_argv[] 
 #if OMPT_SUPPORT