Force runtime to honor nthreads-var and thread-limit-var inside the teams construct on host.

[AndreyChurbanov]

Summary

This patch fixes https://bugs.llvm.org/show_bug.cgi?id=42906, via adding adjustment of number of threads on enter to the teams construct on host according to user settings. This allows to pass checks and avoid assertions at time of team of threads creation.

[hbae]

LGTM.

[simoatze]

Will this fix make it to the 9.0 release? Thank you!

[AndreyChurbanov]

In D66351#1637753, @simoatze wrote:

Will this fix make it to the 9.0 release? Thank you!

Sorry for late reply, I was on vacation w/o internet access.
I guess no. We are probably missing 9.0 release. Do you think it is important enough to request its pull at this (late) stage of the release process?

[simoatze]

I think this can wait at this point. I noticed the problem on Power when I was doing performance tests. On Power using all the available threads (including HT) was causing slow downs with SPEC Accel OMP, so I had to set the OMP_THREAD_LIMIT and noticed the seg faults. I can patch my own runtime and wait for the next release that will integrate this fix. Thanks!

[AndreyChurbanov]

BTW, there is also workaround without runtime patching, - set KMP_TEAMS_THREAD_LIMIT to the same value as OMP_THREAD_LIMIT. This avoids runtime segfault with old library.

runtime/src/kmp_runtime.cpp

if (num_teams > __kmp_teams_max_nth) { // if too many teams requested?
     }
     num_teams = __kmp_teams_max_nth;
   }
   // Set number of teams (number of threads in the outer "parallel" of the
   // teams)
   thr->th.th_set_nproc = thr->th.th_teams_size.nteams = num_teams;
 
   // Remember the number of threads for inner parallel regions
-  if (num_threads == 0) {
-    if (!TCR_4(__kmp_init_middle))
-      __kmp_middle_initialize(); // get __kmp_avail_proc calculated
+    __kmp_middle_initialize(); // get internal globals calculated
+  KMP_DEBUG_ASSERT(__kmp_avail_proc);
+  KMP_DEBUG_ASSERT(__kmp_dflt_team_nth);
+  if (num_threads == 0) {
     num_threads = __kmp_avail_proc / num_teams;
-    if (num_teams * num_threads > __kmp_teams_max_nth) {
-      // adjust num_threads w/o warning as it is not user setting
+    // num_threads = min(num_threads, nthreads-var, thread-limit-var)
+    // no thread_limit clause specified -  do not change thread-limit-var ICV
+    if (num_threads > __kmp_dflt_team_nth) {
+      num_threads = __kmp_dflt_team_nth; // honor nthreads-var ICV
+    }
+    if (num_threads > thr->th.th_current_task->td_icvs.thread_limit) {
+      num_threads = thr->th.th_current_task->td_icvs.thread_limit;
+    } // prevent team size to exceed thread-limit-var
+    if (num_teams * num_threads > __kmp_teams_max_nth) {
       num_threads = __kmp_teams_max_nth / num_teams;
     }
   } else {
     // This thread will be the master of the league masters
     // Store new thread limit; old limit is saved in th_cg_roots list
     thr->th.th_current_task->td_icvs.thread_limit = num_threads;
-
+    // num_threads = min(num_threads, nthreads-var)
+    if (num_threads > __kmp_dflt_team_nth) {
+      num_threads = __kmp_dflt_team_nth; // honor nthreads-var ICV
+    }
     if (num_teams * num_threads > __kmp_teams_max_nth) {
       int new_threads = __kmp_teams_max_nth / num_teams;
       if (!__kmp_reserve_warn) { // user asked for too many threads
         __kmp_reserve_warn = 1; // conflicts with KMP_TEAMS_THREAD_LIMIT
         __kmp_msg(kmp_ms_warning,
                   KMP_MSG(CantFormThrTeam, num_threads, new_threads),
                   KMP_HNT(Unset_ALL_THREADS), __kmp_msg_null);
       }