Fix for third case reported in https://bugs.llvm.org/show_bug.cgi?id=41584

[AndreyChurbanov]

Summary

The code fragments of recruiting a thread from pool (kmp_runtime.cpp:4297):

new_thr->th.th_in_pool = FALSE;
__kmp_lock_suspend_mx(new_thr);
if (new_thr->th.th_active_in_pool == TRUE) {
  ...
else
  KMP_DEBUG_ASSERT(new_thr->th.th_active == FALSE);

and its awakening, executed under the mutex (z_Linux_util.cpp:1534):

th->th.th_active = TRUE;
if (th->th.th_in_pool) {
  th->th.th_active_in_pool = TRUE;
}

can be executed in parallel. Since the "th_in_pool=0" assignment is out of critical section, it is valid to have "th_active==1" and "th_active_in_pool==0" for the worker thread been recruited from common pool (e.g. in case of spurious wakeup). So the assertion triggered looks wrong. The patch deletes it.

I actually failed to reproduce the assertion on my server with RHEL7 (probably because it is hard to force spurious wakeup of a thread, and without it I failed to find code path of simultaneous execution of code fragments I referenced), but the patch looks logically correct.

[jdenny]

I tried at r361335, where the original reproducer failed this assert consistently quickly for me today. After applying your patch there and running each reproducer from my bug report in a shell while loop for about an hour, I saw no failures. Thanks!

[hbae]

LGTM.

openmp/trunk/runtime/src/kmp_runtime.cpp

if (__kmp_thread_pool) {
     TCW_4(new_thr->th.th_in_pool, FALSE);
     __kmp_suspend_initialize_thread(new_thr);
     __kmp_lock_suspend_mx(new_thr);
     if (new_thr->th.th_active_in_pool == TRUE) {
       KMP_DEBUG_ASSERT(new_thr->th.th_active == TRUE);
       KMP_ATOMIC_DEC(&__kmp_thread_pool_active_nth);
       new_thr->th.th_active_in_pool = FALSE;
     }
-#if KMP_DEBUG
-    else {
-      KMP_DEBUG_ASSERT(new_thr->th.th_active == FALSE);
-    }
-#endif
     __kmp_unlock_suspend_mx(new_thr);
 
     KA_TRACE(20, ("__kmp_allocate_thread: T#%d using thread T#%d\n",
                   __kmp_get_gtid(), new_thr->th.th_info.ds.ds_gtid));
     KMP_ASSERT(!new_thr->th.th_team);
     KMP_DEBUG_ASSERT(__kmp_nth < __kmp_threads_capacity);
 
     /* setup the thread structure */