[OpenMP][libomp] Add hidden helper affinity

[jlpeyton]

Summary

Add new hidden helper affinity via the environment variable,
KMP_HIDDEN_HELPER_AFFINITY, which allows users to assign thread
affinity to hidden helper threads using the same syntax as
KMP_AFFINITY. OMP_PLACES/OMP_PROC_BIND have no interaction with
KMP_HIDDEN_HELPER_AFFINITY.

[jdoerfert]

LG, let's see if someone else wants to chime in, otherwise this can be merged. (@tianshilei1992)

[jlpeyton]

ping. This is the third patch set in a series of three to introduce hidden helper affinity.

[jdoerfert]

LG

[tianshilei1992]

Sorry for the late response. This is a long-wanted feature. Thanks for it!

[mgorny]

I'm seeing test failures on Gentoo due to this change (both on amd64 and i386):

FAIL: libomp :: env/kmp_aff_disable_hwloc.c (54 of 301)
******************** TEST 'libomp :: env/kmp_aff_disable_hwloc.c' FAILED ********************
Script:
--
: 'RUN: at line 1';   /usr/lib/ccache/bin/x86_64-pc-linux-gnu-clang -fno-experimental-isel -fopenmp   -I /tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp_build-abi_x86_64.amd64/runtime/src -I /tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp/runtime/test -L /tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp_build-abi_x86_64.amd64/runtime/src  -fno-omit-frame-pointer -I /tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp/runtime/test/ompt /tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp/runtime/test/env/kmp_aff_disable_hwloc.c -o /tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp_build-abi_x86_64.amd64/runtime/test/env/Output/kmp_aff_disable_hwloc.c.tmp -lm -latomic && env KMP_AFFINITY=disabled KMP_TOPOLOGY_METHOD=hwloc /tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp_build-abi_x86_64.amd64/runtime/test/env/Output/kmp_aff_disable_hwloc.c.tmp
--
Exit Code: -6

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "/usr/lib/ccache/bin/x86_64-pc-linux-gnu-clang" "-fno-experimental-isel" "-fopenmp" "-I" "/tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp_build-abi_x86_64.amd64/runtime/src" "-I" "/tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp/runtime/test" "-L" "/tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp_build-abi_x86_64.amd64/runtime/src" "-fno-omit-frame-pointer" "-I" "/tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp/runtime/test/ompt" "/tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp/runtime/test/env/kmp_aff_disable_hwloc.c" "-o" "/tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp_build-abi_x86_64.amd64/runtime/test/env/Output/kmp_aff_disable_hwloc.c.tmp" "-lm" "-latomic"
$ "env" "KMP_AFFINITY=disabled" "KMP_TOPOLOGY_METHOD=hwloc" "/tmp/portage/sys-libs/libomp-16.0.0_pre20221029/work/openmp_build-abi_x86_64.amd64/runtime/test/env/Output/kmp_aff_disable_hwloc.c.tmp"
# command stderr:
OMP: Warning #264: KMP_AFFINITY: Ignoring hwloc mechanism.
Assertion failure at kmp_settings.cpp(5926): affinity->type != affinity_default.
OMP: Error #13: Assertion failure at kmp_settings.cpp(5926).
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://bugs.llvm.org/.

error: command failed with exit status: -6

--

********************

If that's of any relevance, I'm testing in a systemd-nspawn container.

openmp/docs/design/Runtimes.rst

.. note::
     (which is default on Windows). Otherwise, the respect affinity modifier is used.
 
 .. note::
     On Windows with multiple processor groups, if the granularity is too coarse, it
     will be set to ``granularity=group``. For example, if two processor groups exist
     across one socket, and ``granularity=socket`` the runtime will shift the
     granularity down to group since that is the largest granularity allowed by the OS.
 
+KMP_HIDDEN_HELPER_AFFINITY (Windows, Linux)
+"""""""""""""""""""""""""""""
+
+Enables run-time library to bind hidden helper threads to physical processing units.
+This environment variable has the same syntax and semantics as ``KMP_AFFINIY`` but only
+applies to the hidden helper team.
+
+You must set this environment variable before the first parallel region, or
+certain API calls including ``omp_get_max_threads()``, ``omp_get_num_procs()``
+and any affinity API calls.
+
+**Syntax:** Same as ``KMP_AFFINITY``
+
+The following ``modifiers`` are ignored in ``KMP_HIDDEN_HELPER_AFFINITY`` and are only valid
+for ``KMP_AFFINITY``:
+* ``respect`` and ``norespect``
+* ``reset`` and ``noreset``
+
 KMP_ALL_THREADS
 """""""""""""""
 
 Limits the number of simultaneously-executing threads in an OpenMP program.
 If this limit is reached and another native operating system thread encounters
 OpenMP API calls or constructs, then the program may abort with an error
 message. If this limit is reached at the time an OpenMP parallel region begins,
 a one-time warning message may be generated indicating that the number of

openmp/runtime/src/kmp.h

#define KMP_AFFINITY_INIT(env)                                                 \
   {                                                                            \
     nullptr, affinity_default, KMP_HW_UNKNOWN, -1, 0, 0,                       \
         {TRUE, FALSE, TRUE, affinity_respect_mask_default, FALSE, FALSE}, 0,   \
         nullptr, 0, nullptr, env                                               \
   }
 
 extern enum affinity_top_method __kmp_affinity_top_method;
 extern kmp_affinity_t __kmp_affinity;
+extern kmp_affinity_t __kmp_hh_affinity;
+extern kmp_affinity_t *__kmp_affinities[2];
 
 extern void __kmp_affinity_bind_thread(int which);
 
 extern kmp_affin_mask_t *__kmp_affin_fullMask;
 extern kmp_affin_mask_t *__kmp_affin_origMask;
 extern char *__kmp_cpuinfo_file;
 
 #endif /* KMP_AFFINITY_SUPPORTED */

 
 // Check whether a given thread is a hidden helper thread
 #define KMP_HIDDEN_HELPER_THREAD(gtid)                                         \
   ((gtid) >= 1 && (gtid) <= __kmp_hidden_helper_threads_num)
 
 #define KMP_HIDDEN_HELPER_WORKER_THREAD(gtid)                                  \
   ((gtid) > 1 && (gtid) <= __kmp_hidden_helper_threads_num)
 
+#define KMP_HIDDEN_HELPER_MAIN_THREAD(gtid)                                    \
+  ((gtid) == 1 && (gtid) <= __kmp_hidden_helper_threads_num)
+
 #define KMP_HIDDEN_HELPER_TEAM(team)                                           \
   (team->t.t_threads[0] == __kmp_hidden_helper_main_thread)
 
 // Map a gtid to a hidden helper thread. The first hidden helper thread, a.k.a
 // main thread, is skipped.
 #define KMP_GTID_TO_SHADOW_GTID(gtid)                                          \
   ((gtid) % (__kmp_hidden_helper_threads_num - 1) + 2)
 

openmp/runtime/src/kmp_affinity.h

class kmp_topology_t {
   // and the value is the equivalent hardware topology type in the
   // types[] array, if the value is KMP_HW_UNKNOWN, then there is no
   // known equivalence for the topology type
   kmp_hw_t equivalent[KMP_HW_LAST];
 
   // Flags describing the topology
   flags_t flags;
 
+  // Compact value used during sort_compact()
+  int compact;
+
   // Insert a new topology layer after allocation
   void _insert_layer(kmp_hw_t type, const int *ids);
 
 #if KMP_GROUP_AFFINITY
   // Insert topology information about Windows Processor groups
   void _insert_windows_proc_groups();
 #endif
 
#endif
   }
   // Return the number of cores with attribute
   // 'attr' per topology level 'above'
   int get_ncores_with_attr_per(const kmp_hw_attr_t &attr, int above) const {
     return _get_ncores_with_attr(attr, above, false);
   }
 
 #if KMP_AFFINITY_SUPPORTED
-  void sort_compact() {
+  friend int kmp_hw_thread_t::compare_compact(const void *a, const void *b);
+  void sort_compact(kmp_affinity_t &affinity) {
+    compact = affinity.compact;
     qsort(hw_threads, num_hw_threads, sizeof(kmp_hw_thread_t),
           kmp_hw_thread_t::compare_compact);
   }
 #endif
   void print(const char *env_var = "KMP_AFFINITY") const;
   void dump() const;
 };
 extern kmp_topology_t *__kmp_topology;

openmp/runtime/src/kmp_affinity.cpp

 }
 
 #if KMP_AFFINITY_SUPPORTED
 int kmp_hw_thread_t::compare_compact(const void *a, const void *b) {
   int i;
   const kmp_hw_thread_t *aa = (const kmp_hw_thread_t *)a;
   const kmp_hw_thread_t *bb = (const kmp_hw_thread_t *)b;
   int depth = __kmp_topology->get_depth();
-  KMP_DEBUG_ASSERT(__kmp_affinity.compact >= 0);
-  KMP_DEBUG_ASSERT(__kmp_affinity.compact <= depth);
-  for (i = 0; i < __kmp_affinity.compact; i++) {
+  int compact = __kmp_topology->compact;
+  KMP_DEBUG_ASSERT(compact >= 0);
+  KMP_DEBUG_ASSERT(compact <= depth);
+  for (i = 0; i < compact; i++) {
     int j = depth - i - 1;
     if (aa->sub_ids[j] < bb->sub_ids[j])
       return -1;
     if (aa->sub_ids[j] > bb->sub_ids[j])
       return 1;
   }
   for (; i < depth; i++) {
-    int j = i - __kmp_affinity.compact;
+    int j = i - compact;
     if (aa->sub_ids[j] < bb->sub_ids[j])
       return -1;
     if (aa->sub_ids[j] > bb->sub_ids[j])
       return 1;
   }
   return 0;
 }
 #endif
kmp_topology_t *kmp_topology_t::allocate(int nproc, int ndepth,
   retval->depth = ndepth;
   int *arr =
       (int *)(bytes + sizeof(kmp_topology_t) + sizeof(kmp_hw_thread_t) * nproc);
   retval->types = (kmp_hw_t *)arr;
   retval->ratio = arr + (size_t)KMP_HW_LAST;
   retval->count = arr + 2 * (size_t)KMP_HW_LAST;
   retval->num_core_efficiencies = 0;
   retval->num_core_types = 0;
+  retval->compact = 0;
   for (int i = 0; i < KMP_HW_MAX_NUM_CORE_TYPES; ++i)
     retval->core_types[i] = KMP_HW_CORE_TYPE_UNKNOWN;
   KMP_FOREACH_HW_TYPE(type) { retval->equivalent[type] = KMP_HW_UNKNOWN; }
   for (int i = 0; i < ndepth; ++i) {
     retval->types[i] = types[i];
     retval->equivalent[types[i]] = types[i];
   }
   return retval;
#endif
   }
   if (filtered && verbose)
     __kmp_topology->print("KMP_HW_SUBSET");
   return success;
 }
 
 static void __kmp_aux_affinity_initialize(kmp_affinity_t &affinity) {
   bool is_regular_affinity = (&affinity == &__kmp_affinity);
+  bool is_hidden_helper_affinity = (&affinity == &__kmp_hh_affinity);
   const char *env_var = affinity.env_var;
 
   if (affinity.flags.initialized) {
     KMP_ASSERT(__kmp_affin_fullMask != NULL);
     return;
   }
 
   if (is_regular_affinity && (!__kmp_affin_fullMask || !__kmp_affin_origMask))
#endif
   if (affinity.gran_levels == 0) {
     KMP_DEBUG_ASSERT((int)numUnique == __kmp_avail_proc);
   }
 
   switch (affinity.type) {
 
   case affinity_explicit:
     KMP_DEBUG_ASSERT(affinity.proclist != NULL);
-    if (__kmp_nested_proc_bind.bind_types[0] == proc_bind_intel) {
+    if (is_hidden_helper_affinity ||
+        __kmp_nested_proc_bind.bind_types[0] == proc_bind_intel) {
       __kmp_affinity_process_proclist(affinity);
     } else {
       __kmp_affinity_process_placelist(affinity);
     }
     if (affinity.num_masks == 0) {
       KMP_AFF_WARNING(affinity, AffNoValidProcID);
       affinity.type = affinity_none;
       __kmp_create_affinity_none_places(affinity);
#endif
 
   case affinity_compact:
     if (affinity.compact >= depth) {
       affinity.compact = depth - 1;
     }
     goto sortTopology;
 
   case affinity_balanced:
-    if (depth <= 1) {
+    if (depth <= 1 || is_hidden_helper_affinity) {
       KMP_AFF_WARNING(affinity, AffBalancedNotAvail, env_var);
       affinity.type = affinity_none;
       __kmp_create_affinity_none_places(affinity);
       affinity.flags.initialized = TRUE;
       return;
     } else if (!__kmp_topology->is_uniform()) {
       // Save the depth for further usage
       __kmp_aff_depth = depth;
sortTopology:
     if (affinity.flags.dups) {
       affinity.num_masks = __kmp_avail_proc;
     } else {
       affinity.num_masks = numUnique;
     }
 
     if ((__kmp_nested_proc_bind.bind_types[0] != proc_bind_intel) &&
         (__kmp_affinity_num_places > 0) &&
-        ((unsigned)__kmp_affinity_num_places < affinity.num_masks)) {
+        ((unsigned)__kmp_affinity_num_places < affinity.num_masks) &&
+        !is_hidden_helper_affinity) {
       affinity.num_masks = __kmp_affinity_num_places;
     }
 
     KMP_CPU_ALLOC_ARRAY(affinity.masks, affinity.num_masks);
 
     // Sort the topology table according to the current setting of
     // affinity.compact, then fill out affinity.masks.
-    __kmp_topology->sort_compact();
+    __kmp_topology->sort_compact(affinity);
     {
       int i;
       unsigned j;
       int num_hw_threads = __kmp_topology->get_num_hw_threads();
       for (i = 0, j = 0; i < num_hw_threads; i++) {
         if ((!affinity.flags.dups) && (!__kmp_topology->at(i).leader)) {
           continue;
         }
void __kmp_affinity_initialize(kmp_affinity_t &affinity) {
   if (disabled)
     affinity.type = affinity_none;
   __kmp_aux_affinity_initialize(affinity);
   if (disabled)
     affinity.type = affinity_disabled;
 }
 
 void __kmp_affinity_uninitialize(void) {
-  {
-    kmp_affinity_t *affinity = &__kmp_affinity;
+  for (kmp_affinity_t *affinity : __kmp_affinities) {
     if (affinity->masks != NULL)
       KMP_CPU_FREE_ARRAY(affinity->masks, affinity->num_masks);
     if (affinity->os_id_masks != NULL)
       KMP_CPU_FREE_ARRAY(affinity->os_id_masks, affinity->num_os_id_masks);
     if (affinity->proclist != NULL)
       __kmp_free(affinity->proclist);
     *affinity = KMP_AFFINITY_INIT(affinity->env_var);
   }
#endif
   }
   if (__kmp_topology) {
     kmp_topology_t::deallocate(__kmp_topology);
     __kmp_topology = nullptr;
   }
   KMPAffinity::destroy_api();
 }
 
+static void __kmp_select_mask_by_gtid(int gtid, const kmp_affinity_t *affinity,
+                                      int *place, kmp_affin_mask_t **mask) {
+  int mask_idx;
+  bool is_hidden_helper = KMP_HIDDEN_HELPER_THREAD(gtid);
+  if (is_hidden_helper)
+    // The first gtid is the regular primary thread, the second gtid is the main
+    // thread of hidden team which does not participate in task execution.
+    mask_idx = gtid - 2;
+  else
+    mask_idx = __kmp_adjust_gtid_for_hidden_helpers(gtid);
+  KMP_DEBUG_ASSERT(affinity->num_masks > 0);
+  *place = (mask_idx + affinity->offset) % affinity->num_masks;
+  *mask = KMP_CPU_INDEX(affinity->masks, *place);
+}
+
 void __kmp_affinity_set_init_mask(int gtid, int isa_root) {
   if (!KMP_AFFINITY_CAPABLE()) {
     return;
   }
 
   kmp_info_t *th = (kmp_info_t *)TCR_SYNC_PTR(__kmp_threads[gtid]);
   if (th->th.th_affin_mask == NULL) {
     KMP_CPU_ALLOC(th->th.th_affin_mask);
   } else {
     KMP_CPU_ZERO(th->th.th_affin_mask);
   }
 
   // Copy the thread mask to the kmp_info_t structure. If
   // __kmp_affinity.type == affinity_none, copy the "full" mask, i.e.
   // one that has all of the OS proc ids set, or if
   // __kmp_affinity.flags.respect is set, then the full mask is the
   // same as the mask of the initialization thread.
   kmp_affin_mask_t *mask;
   int i;
-  const kmp_affinity_t *affinity = &__kmp_affinity;
-  const char *env_var = affinity->env_var;
+  const kmp_affinity_t *affinity;
+  const char *env_var;
+  bool is_hidden_helper = KMP_HIDDEN_HELPER_THREAD(gtid);
+
+  if (is_hidden_helper)
+    affinity = &__kmp_hh_affinity;
+  else
+    affinity = &__kmp_affinity;
+  env_var = affinity->env_var;
 
-  if (KMP_AFFINITY_NON_PROC_BIND) {
+  if (KMP_AFFINITY_NON_PROC_BIND || is_hidden_helper) {
     if ((affinity->type == affinity_none) ||
         (affinity->type == affinity_balanced) ||
-        KMP_HIDDEN_HELPER_THREAD(gtid)) {
+        KMP_HIDDEN_HELPER_MAIN_THREAD(gtid)) {
 #if KMP_GROUP_AFFINITY
       if (__kmp_num_proc_groups > 1) {
         return;
       }
 #endif
       KMP_ASSERT(__kmp_affin_fullMask != NULL);
       i = 0;
       mask = __kmp_affin_fullMask;
     } else {
-      int mask_idx = __kmp_adjust_gtid_for_hidden_helpers(gtid);
-      KMP_DEBUG_ASSERT(affinity->num_masks > 0);
-      i = (mask_idx + affinity->offset) % affinity->num_masks;
-      mask = KMP_CPU_INDEX(affinity->masks, i);
+      __kmp_select_mask_by_gtid(gtid, affinity, &i, &mask);
     }
   } else {
-    if ((!isa_root) || KMP_HIDDEN_HELPER_THREAD(gtid) ||
-        (__kmp_nested_proc_bind.bind_types[0] == proc_bind_false)) {
+    if (!isa_root || __kmp_nested_proc_bind.bind_types[0] == proc_bind_false) {
 #if KMP_GROUP_AFFINITY
       if (__kmp_num_proc_groups > 1) {
         return;
       }
 #endif
       KMP_ASSERT(__kmp_affin_fullMask != NULL);
       i = KMP_PLACE_ALL;
       mask = __kmp_affin_fullMask;
     } else {
-      // int i = some hash function or just a counter that doesn't
-      // always start at 0.  Use adjusted gtid for now.
-      int mask_idx = __kmp_adjust_gtid_for_hidden_helpers(gtid);
-      KMP_DEBUG_ASSERT(affinity->num_masks > 0);
-      i = (mask_idx + affinity->offset) % affinity->num_masks;
-      mask = KMP_CPU_INDEX(affinity->masks, i);
+      __kmp_select_mask_by_gtid(gtid, affinity, &i, &mask);
     }
   }
 
   th->th.th_current_place = i;
-  if (isa_root || KMP_HIDDEN_HELPER_THREAD(gtid)) {
+  if (isa_root && !is_hidden_helper) {
     th->th.th_new_place = i;
     th->th.th_first_place = 0;
     th->th.th_last_place = affinity->num_masks - 1;
   } else if (KMP_AFFINITY_NON_PROC_BIND) {
     // When using a Non-OMP_PROC_BIND affinity method,
     // set all threads' place-partition-var to the entire place list
     th->th.th_first_place = 0;
     th->th.th_last_place = affinity->num_masks - 1;
   }
 
   if (i == KMP_PLACE_ALL) {
     KA_TRACE(100, ("__kmp_affinity_set_init_mask: binding T#%d to all places\n",
                    gtid));
   } else {
     KA_TRACE(100, ("__kmp_affinity_set_init_mask: binding T#%d to place %d\n",
                    gtid, i));
   }
 
   KMP_CPU_COPY(th->th.th_affin_mask, mask);
 
-  if (affinity->flags.verbose && !KMP_HIDDEN_HELPER_THREAD(gtid)
-      /* to avoid duplicate printing (will be correctly printed on barrier) */
-      && (affinity->type == affinity_none ||
-          (i != KMP_PLACE_ALL && affinity->type != affinity_balanced))) {
+  if (affinity->flags.verbose &&
+      (affinity->type == affinity_none ||
+       (i != KMP_PLACE_ALL && affinity->type != affinity_balanced)) &&
+      !KMP_HIDDEN_HELPER_MAIN_THREAD(gtid)) {
     char buf[KMP_AFFIN_MASK_PRINT_LEN];
     __kmp_affinity_print_mask(buf, KMP_AFFIN_MASK_PRINT_LEN,
                               th->th.th_affin_mask);
     KMP_INFORM(BoundToOSProcSet, env_var, (kmp_int32)getpid(), __kmp_gettid(),
                gtid, buf);
   }
 
-#if KMP_DEBUG
-  // Hidden helper thread affinity only printed for debug builds
-  if (affinity->flags.verbose && KMP_HIDDEN_HELPER_THREAD(gtid)) {
-    char buf[KMP_AFFIN_MASK_PRINT_LEN];
-    __kmp_affinity_print_mask(buf, KMP_AFFIN_MASK_PRINT_LEN,
-                              th->th.th_affin_mask);
-    KMP_INFORM(BoundToOSProcSet, "KMP_AFFINITY (hidden helper thread)",
-               (kmp_int32)getpid(), __kmp_gettid(), gtid, buf);
-  }
-#endif
-
 #if KMP_OS_WINDOWS
   // On Windows* OS, the process affinity mask might have changed. If the user
   // didn't request affinity and this call fails, just continue silently.
   // See CQ171393.
   if (affinity->type == affinity_none) {
     __kmp_set_system_affinity(th->th.th_affin_mask, FALSE);
   } else
 #endif
     __kmp_set_system_affinity(th->th.th_affin_mask, TRUE);
 }
 
 void __kmp_affinity_set_place(int gtid) {
-  if (!KMP_AFFINITY_CAPABLE()) {
+  // Hidden helper threads should not be affected by OMP_PLACES/OMP_PROC_BIND
+  if (!KMP_AFFINITY_CAPABLE() || KMP_HIDDEN_HELPER_THREAD(gtid)) {
     return;
   }
 
   kmp_info_t *th = (kmp_info_t *)TCR_SYNC_PTR(__kmp_threads[gtid]);
 
   KA_TRACE(100, ("__kmp_affinity_set_place: binding T#%d to place %d (current "
                  "place = %d)\n",
                  gtid, th->th.th_new_place, th->th.th_current_place));

openmp/runtime/src/kmp_global.cpp

 #endif /* KMP_OS_WINDOWS */
 
 size_t __kmp_affin_mask_size = 0;
 enum affinity_top_method __kmp_affinity_top_method =
     affinity_top_method_default;
 
 // Regular thread affinity settings from KMP_AFFINITY
 kmp_affinity_t __kmp_affinity = KMP_AFFINITY_INIT("KMP_AFFINITY");
+// Hidden helper thread affinity settings from KMP_HIDDEN_HELPER_AFFINITY
+kmp_affinity_t __kmp_hh_affinity =
+    KMP_AFFINITY_INIT("KMP_HIDDEN_HELPER_AFFINITY");
+kmp_affinity_t *__kmp_affinities[] = {&__kmp_affinity, &__kmp_hh_affinity};
 
 char *__kmp_cpuinfo_file = NULL;
 
 #endif /* KMP_AFFINITY_SUPPORTED */
 
 kmp_nested_proc_bind_t __kmp_nested_proc_bind = {NULL, 0, 0};
 kmp_proc_bind_t __kmp_teams_proc_bind = proc_bind_spread;
 int __kmp_affinity_num_places = 0;

openmp/runtime/src/kmp_runtime.cpp

void __kmp_hidden_helper_initialize() {
   // Double check. Note that this double check should not be placed before
   // __kmp_parallel_initialize as it will cause dead lock.
   __kmp_acquire_bootstrap_lock(&__kmp_initz_lock);
   if (TCR_4(__kmp_init_hidden_helper)) {
     __kmp_release_bootstrap_lock(&__kmp_initz_lock);
     return;
   }
 
+#if KMP_AFFINITY_SUPPORTED
+  // Initialize hidden helper affinity settings.
+  // The above __kmp_parallel_initialize() will initialize
+  // regular affinity (and topology) if not already done.
+  if (!__kmp_hh_affinity.flags.initialized)
+    __kmp_affinity_initialize(__kmp_hh_affinity);
+#endif
+
   // Set the count of hidden helper tasks to be executed to zero
   KMP_ATOMIC_ST_REL(&__kmp_unexecuted_hidden_helper_tasks, 0);
 
   // Set the global variable indicating that we're initializing hidden helper
   // team/threads
   TCW_SYNC_4(__kmp_init_hidden_helper_threads, TRUE);
 
   // Platform independent initialization

openmp/runtime/src/kmp_settings.cpp

static void __kmp_stg_parse_num_hidden_helper_threads(char const *name,
                                                       char const *value,
                                                       void *data) {
   __kmp_stg_parse_int(name, value, 0, 16, &__kmp_hidden_helper_threads_num);
   // If the number of hidden helper threads is zero, we disable hidden helper
   // task
   if (__kmp_hidden_helper_threads_num == 0) {
     __kmp_enable_hidden_helper = FALSE;
   } else {
-    // Since the main thread of hidden helper team dooes not participate
+    // Since the main thread of hidden helper team does not participate
     // in tasks execution let's increment the number of threads by one
     // so that requested number of threads do actual job.
     __kmp_hidden_helper_threads_num++;
   }
 } // __kmp_stg_parse_num_hidden_helper_threads
 
 static void __kmp_stg_print_num_hidden_helper_threads(kmp_str_buf_t *buffer,
                                                       char const *name,
static void __kmp_stg_parse_affinity(char const *name, char const *value,
   rc = __kmp_stg_check_rivals(name, value, rivals);
   if (rc) {
     return;
   }
 
   __kmp_parse_affinity_env(name, value, &__kmp_affinity);
 
 } // __kmp_stg_parse_affinity
+static void __kmp_stg_parse_hh_affinity(char const *name, char const *value,
+                                        void *data) {
+  __kmp_parse_affinity_env(name, value, &__kmp_hh_affinity);
+  // Warn about unused parts of hidden helper affinity settings if specified.
+  if (__kmp_hh_affinity.flags.reset) {
+    KMP_WARNING(AffInvalidParam, name, "reset");
+  }
+  if (__kmp_hh_affinity.flags.respect != affinity_respect_mask_default) {
+    KMP_WARNING(AffInvalidParam, name, "respect");
+  }
+}
 
 static void __kmp_print_affinity_env(kmp_str_buf_t *buffer, char const *name,
                                      const kmp_affinity_t &affinity) {
+  bool is_hh_affinity = (&affinity == &__kmp_hh_affinity);
   if (__kmp_env_format) {
     KMP_STR_BUF_PRINT_NAME_EX(name);
   } else {
     __kmp_str_buf_print(buffer, "   %s='", name);
   }
   if (affinity.flags.verbose) {
     __kmp_str_buf_print(buffer, "%s,", "verbose");
   } else {
     __kmp_str_buf_print(buffer, "%s,", "noverbose");
   }
   if (affinity.flags.warnings) {
     __kmp_str_buf_print(buffer, "%s,", "warnings");
   } else {
     __kmp_str_buf_print(buffer, "%s,", "nowarnings");
   }
   if (KMP_AFFINITY_CAPABLE()) {
+    // Hidden helper affinity does not affect global reset
+    // or respect flags. That is still solely controlled by KMP_AFFINITY.
+    if (!is_hh_affinity) {
       if (affinity.flags.respect) {
         __kmp_str_buf_print(buffer, "%s,", "respect");
       } else {
         __kmp_str_buf_print(buffer, "%s,", "norespect");
       }
       if (affinity.flags.reset) {
         __kmp_str_buf_print(buffer, "%s,", "reset");
       } else {
         __kmp_str_buf_print(buffer, "%s,", "noreset");
       }
+    }
     __kmp_str_buf_print(buffer, "granularity=%s,",
                         __kmp_hw_get_keyword(affinity.gran, false));
   }
   if (!KMP_AFFINITY_CAPABLE()) {
     __kmp_str_buf_print(buffer, "%s", "disabled");
   } else {
     int compact = affinity.compact;
     int offset = affinity.offset;
static void __kmp_print_affinity_env(kmp_str_buf_t *buffer, char const *name,
   }
   __kmp_str_buf_print(buffer, "'\n");
 } //__kmp_stg_print_affinity
 
 static void __kmp_stg_print_affinity(kmp_str_buf_t *buffer, char const *name,
                                      void *data) {
   __kmp_print_affinity_env(buffer, name, __kmp_affinity);
 }
+static void __kmp_stg_print_hh_affinity(kmp_str_buf_t *buffer, char const *name,
+                                        void *data) {
+  __kmp_print_affinity_env(buffer, name, __kmp_hh_affinity);
+}
 
 #ifdef KMP_GOMP_COMPAT
 
 static void __kmp_stg_parse_gomp_cpu_affinity(char const *name,
                                               char const *value, void *data) {
   const char *next = NULL;
   char *temp_proclist;
   kmp_setting_t **rivals = (kmp_setting_t **)data;
#endif
      __kmp_stg_print_all_threadprivate, NULL, 0, 0},
     {"KMP_FOREIGN_THREADS_THREADPRIVATE",
      __kmp_stg_parse_foreign_threads_threadprivate,
      __kmp_stg_print_foreign_threads_threadprivate, NULL, 0, 0},
 
 #if KMP_AFFINITY_SUPPORTED
     {"KMP_AFFINITY", __kmp_stg_parse_affinity, __kmp_stg_print_affinity, NULL,
      0, 0},
+    {"KMP_HIDDEN_HELPER_AFFINITY", __kmp_stg_parse_hh_affinity,
+     __kmp_stg_print_hh_affinity, NULL, 0, 0},
 #ifdef KMP_GOMP_COMPAT
     {"GOMP_CPU_AFFINITY", __kmp_stg_parse_gomp_cpu_affinity, NULL,
      /* no print */ NULL, 0, 0},
 #endif /* KMP_GOMP_COMPAT */
     {"OMP_PROC_BIND", __kmp_stg_parse_proc_bind, __kmp_stg_print_proc_bind,
      NULL, 0, 0},
     {"KMP_TEAMS_PROC_BIND", __kmp_stg_parse_teams_proc_bind,
      __kmp_stg_print_teams_proc_bind, NULL, 0, 0},
#if KMP_GROUP_AFFINITY
           __kmp_affinity.flags.respect = FALSE;
         }
         // Use compact affinity with anticipation of pinning to at least the
         // group granularity since threads can only be bound to one group.
         if (__kmp_affinity.type == affinity_default) {
           __kmp_affinity.type = affinity_compact;
           __kmp_nested_proc_bind.bind_types[0] = proc_bind_intel;
         }
+        if (__kmp_hh_affinity.type == affinity_default)
+          __kmp_hh_affinity.type = affinity_compact;
         if (__kmp_affinity_top_method == affinity_top_method_default)
           __kmp_affinity_top_method = affinity_top_method_all;
         if (__kmp_affinity.gran == KMP_HW_UNKNOWN)
           __kmp_affinity.gran = KMP_HW_PROC_GROUP;
+        if (__kmp_hh_affinity.gran == KMP_HW_UNKNOWN)
+          __kmp_hh_affinity.gran = KMP_HW_PROC_GROUP;
       } else
 
 #endif /* KMP_GROUP_AFFINITY */
 
       {
         if (__kmp_affinity.flags.respect == affinity_respect_mask_default) {
 #if KMP_GROUP_AFFINITY
           if (__kmp_num_proc_groups > 1 && exactly_one_group) {
#if KMP_MIC_SUPPORTED
           if (__kmp_mic_type != non_mic) {
             __kmp_affinity.type = affinity_scatter;
           } else
 #endif
           {
             __kmp_affinity.type = affinity_none;
           }
         }
+        if (__kmp_hh_affinity.type == affinity_default)
+          __kmp_hh_affinity.type = affinity_none;
         if ((__kmp_affinity.gran == KMP_HW_UNKNOWN) &&
             (__kmp_affinity.gran_levels < 0)) {
 #if KMP_MIC_SUPPORTED
           if (__kmp_mic_type != non_mic) {
             __kmp_affinity.gran = KMP_HW_THREAD;
           } else
 #endif
           {
             __kmp_affinity.gran = KMP_HW_CORE;
           }
         }
+        if ((__kmp_hh_affinity.gran == KMP_HW_UNKNOWN) &&
+            (__kmp_hh_affinity.gran_levels < 0)) {
+#if KMP_MIC_SUPPORTED
+          if (__kmp_mic_type != non_mic) {
+            __kmp_hh_affinity.gran = KMP_HW_THREAD;
+          } else
+#endif
+          {
+            __kmp_hh_affinity.gran = KMP_HW_CORE;
+          }
+        }
         if (__kmp_affinity_top_method == affinity_top_method_default) {
           __kmp_affinity_top_method = affinity_top_method_all;
         }
       }
     }
 
 #ifdef KMP_DEBUG
-    __kmp_print_affinity_settings(&__kmp_affinity);
+    for (const kmp_affinity_t *affinity : __kmp_affinities)
+      __kmp_print_affinity_settings(affinity);
     KMP_DEBUG_ASSERT(__kmp_nested_proc_bind.bind_types[0] != proc_bind_default);
     K_DIAG(1, ("__kmp_nested_proc_bind.bind_types[0] == %d\n",
                __kmp_nested_proc_bind.bind_types[0]));
 #endif
   }
 
 #endif /* KMP_AFFINITY_SUPPORTED */
 

openmp/runtime/src/z_Linux_util.cpp

#if KMP_OS_LINUX || KMP_OS_FREEBSD
   kmp_set_thread_affinity_mask_initial();
 #endif
   // Set default not to bind threads tightly in the child (we're expecting
   // over-subscription after the fork and this can improve things for
   // scripting languages that use OpenMP inside process-parallel code).
   if (__kmp_nested_proc_bind.bind_types != NULL) {
     __kmp_nested_proc_bind.bind_types[0] = proc_bind_false;
   }
-  __kmp_affinity = KMP_AFFINITY_INIT("KMP_AFFINITY");
+  for (kmp_affinity_t *affinity : __kmp_affinities)
+    *affinity = KMP_AFFINITY_INIT(affinity->env_var);
   __kmp_affin_fullMask = nullptr;
   __kmp_affin_origMask = nullptr;
 #endif // KMP_AFFINITY_SUPPORTED
 
 #if KMP_USE_MONITOR
   __kmp_init_monitor = 0;
 #endif
   __kmp_init_parallel = FALSE;

openmp/runtime/test/tasking/hidden_helper_task/affinity.cpp

+// RUN: %libomp-cxx-compile
+// RUN: env LIBOMP_USE_HIDDEN_HELPER_TASK=1 LIBOMP_NUM_HIDDEN_HELPER_THREADS=8 \
+// RUN:     KMP_HIDDEN_HELPER_AFFINITY=verbose,granularity=socket,compact %libomp-run 2>&1 | FileCheck --check-prefix=SOCKET %s
+// RUN: env LIBOMP_USE_HIDDEN_HELPER_TASK=1 LIBOMP_NUM_HIDDEN_HELPER_THREADS=8 \
+// RUN:     KMP_HIDDEN_HELPER_AFFINITY=verbose,granularity=core,scatter %libomp-run 2>&1 | FileCheck --check-prefix=CORE %s
+// RUN: env LIBOMP_USE_HIDDEN_HELPER_TASK=1 LIBOMP_NUM_HIDDEN_HELPER_THREADS=8 \
+// RUN:     KMP_HIDDEN_HELPER_AFFINITY='verbose,granularity=fine,explicit,proclist=[0,1]' %libomp-run 2>&1 | FileCheck --check-prefix=FINE %s
+// RUN: env LIBOMP_USE_HIDDEN_HELPER_TASK=1 LIBOMP_NUM_HIDDEN_HELPER_THREADS=8 \
+// RUN:     KMP_HIDDEN_HELPER_AFFINITY=verbose,granularity=socket,compact KMP_AFFINITY=compact,granularity=fine %libomp-run 2>&1 | FileCheck --check-prefix=SOCKET %s
+// RUN: env LIBOMP_USE_HIDDEN_HELPER_TASK=1 LIBOMP_NUM_HIDDEN_HELPER_THREADS=8 \
+// RUN:     KMP_HIDDEN_HELPER_AFFINITY=verbose,granularity=core,scatter KMP_AFFINITY=compact,granularity=socket %libomp-run 2>&1 | FileCheck --check-prefix=CORE %s
+// RUN: env LIBOMP_USE_HIDDEN_HELPER_TASK=1 LIBOMP_NUM_HIDDEN_HELPER_THREADS=8 \
+// RUN:     KMP_HIDDEN_HELPER_AFFINITY='verbose,granularity=fine,explicit,proclist=[0,1]' KMP_AFFINITY=compact,granularity=core %libomp-run 2>&1 | FileCheck --check-prefix=FINE %s
+// RUN: env LIBOMP_USE_HIDDEN_HELPER_TASK=1 LIBOMP_NUM_HIDDEN_HELPER_THREADS=8 \
+// RUN:     KMP_HIDDEN_HELPER_AFFINITY=verbose,granularity=socket,compact OMP_PROC_BIND=close OMP_PLACES=threads %libomp-run 2>&1 | FileCheck --check-prefix=SOCKET %s
+// RUN: env LIBOMP_USE_HIDDEN_HELPER_TASK=1 LIBOMP_NUM_HIDDEN_HELPER_THREADS=8 \
+// RUN:     KMP_HIDDEN_HELPER_AFFINITY=verbose,granularity=core,scatter OMP_PROC_BIND=close OMP_PLACES=sockets %libomp-run 2>&1 | FileCheck --check-prefix=CORE %s
+// RUN: env LIBOMP_USE_HIDDEN_HELPER_TASK=1 LIBOMP_NUM_HIDDEN_HELPER_THREADS=8 \
+// RUN:     KMP_HIDDEN_HELPER_AFFINITY='verbose,granularity=fine,explicit,proclist=[0,1]' OMP_PROC_BIND=cores OMP_PLACES=cores %libomp-run 2>&1 | FileCheck --check-prefix=FINE %s
+
+/*
+ * This test aims to check hidden helper affinity
+ *
+ * #pragma omp parallel for
+ * for (int i = 0; i < N; ++i) {
+ *   int data1 = 0, data2 = 0;
+ * #pragma omp taskgroup
+ *   {
+ * #pragma omp hidden helper task shared(data1)
+ *    {
+ *      data1 = 1;
+ *    }
+ * #pragma omp hidden helper task shared(data2)
+ *    {
+ *      data2 = 2;
+ *    }
+ *   }
+ *   assert(data1 == 1);
+ *   assert(data2 == 2);
+ * }
+ */
+
+#include "common.h"
+
+extern "C" {
+struct kmp_task_t_with_privates {
+  kmp_task_t task;
+};
+
+struct anon {
+  int32_t *data;
+};
+}
+
+template <int I>
+kmp_int32 omp_task_entry(kmp_int32 gtid, kmp_task_t_with_privates *task) {
+  auto shareds = reinterpret_cast<anon *>(task->task.shareds);
+  auto p = shareds->data;
+  *p = I;
+  return 0;
+}
+
+int main(int argc, char *argv[]) {
+  constexpr const int N = 16;
+#pragma omp parallel for
+  for (int i = 0; i < N; ++i) {
+    int32_t gtid = __kmpc_global_thread_num(nullptr);
+    int32_t data1 = 0;
+    __kmpc_taskgroup(nullptr, gtid);
+
+    auto task1 = __kmpc_omp_target_task_alloc(
+        nullptr, gtid, 1, sizeof(kmp_task_t_with_privates), sizeof(anon),
+        reinterpret_cast<kmp_routine_entry_t>(omp_task_entry<1>), -1);
+    auto shareds = reinterpret_cast<anon *>(task1->shareds);
+    shareds->data = &data1;
+    __kmpc_omp_task(nullptr, gtid, task1);
+
+    __kmpc_end_taskgroup(nullptr, gtid);
+
+    assert(data1 == 1);
+  }
+
+  std::cout << "PASS\n";
+  return 0;
+}
+
+// SOCKET: OMP: Info #{{[0-9]+}}: KMP_HIDDEN_HELPER_AFFINITY: Threads may migrate across
+// SOCKET-NOT: OMP: Info #{{[0-9]+}}: KMP_HIDDEN_HELPER_AFFINITY: pid {{[0-9]+}} tid {{[0-9]+}} thread 1 bound to OS proc set
+// SOCKET-DAG: OMP: Info #[[NUM:[0-9]+]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID:[0-9]+]] tid {{[0-9]+}} thread 2 bound to OS proc set
+// SOCKET-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 3 bound to OS proc set
+// SOCKET-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 4 bound to OS proc set
+// SOCKET-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 5 bound to OS proc set
+// SOCKET-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 6 bound to OS proc set
+// SOCKET-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 7 bound to OS proc set
+// SOCKET-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 8 bound to OS proc set
+// SOCKET-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 9 bound to OS proc set
+
+// CORE: OMP: Info #{{[0-9]+}}: KMP_HIDDEN_HELPER_AFFINITY: Threads may migrate across
+// CORE-NOT: OMP: Info #{{[0-9]+}}: KMP_HIDDEN_HELPER_AFFINITY: pid {{[0-9]+}} tid {{[0-9]+}} thread 1 bound to OS proc set
+// CORE-DAG: OMP: Info #[[NUM:[0-9]+]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID:[0-9]+]] tid {{[0-9]+}} thread 2 bound to OS proc set
+// CORE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 3 bound to OS proc set
+// CORE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 4 bound to OS proc set
+// CORE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 5 bound to OS proc set
+// CORE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 6 bound to OS proc set
+// CORE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 7 bound to OS proc set
+// CORE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 8 bound to OS proc set
+// CORE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 9 bound to OS proc set
+
+// FINE-NOT: OMP: Info #{{[0-9]+}}: KMP_HIDDEN_HELPER_AFFINITY: Threads may migrate across
+// FINE-NOT: OMP: Info #{{[0-9]+}}: KMP_HIDDEN_HELPER_AFFINITY: pid {{[0-9]+}} tid {{[0-9]+}} thread 1 bound to OS proc set
+// FINE-DAG: OMP: Info #[[NUM:[0-9]+]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID:[0-9]+]] tid {{[0-9]+}} thread 2 bound to OS proc set
+// FINE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 3 bound to OS proc set
+// FINE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 4 bound to OS proc set
+// FINE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 5 bound to OS proc set
+// FINE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 6 bound to OS proc set
+// FINE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 7 bound to OS proc set
+// FINE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 8 bound to OS proc set
+// FINE-DAG: OMP: Info #[[NUM]]: KMP_HIDDEN_HELPER_AFFINITY: pid [[PID]] tid {{[0-9]+}} thread 9 bound to OS proc set
+
+// End of file