Fix affinity mask usage on big-endian CPUs

[tstellar]

Summary

This fixes almost all the lit tests on ppc64 big-endian. Prior to this patch,
the affinity masks were printed like this on a 8-core system:

OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {56,57,58,59,60,61,62,63}

[jlpeyton]

To avoid the glibc version issue, can we just switch the mask_t from unsigned char to unsigned long? This is the actual underlying type for cpuset according to the man page. This would require different changes, particularly __kmp_affin_mask_size. I think that in z_Linux_util.cpp:__kmp_determine_affinity_capable() (where __kmp_affin_mask_size is set), you could change __kmp_affin_mask_size to be the closest multiple of sizeof(unsigned long), then the Mask() constructor would perform __kmp_allocate(sizeof(unsigned long)*__kmp_affin_mask_size);. Most of the mask API stays the same, but in set|get_system_affinity(), have the size argument be sizeof(unsigned long)*__kmp_affin_mask_size for the system calls.

Does this make sense?

[tstellar]

Is this more what you had in mind?

[jlpeyton]

Almost, if KMP_AFFINITY_ENABLE() is reverted back and if the KMPNativeAffinity::Mask::mask_t type was made public (I don't see any problem doing that), then along with the rest of changes you made, I think it should look something like this:

diff --git a/runtime/src/z_Linux_util.cpp b/runtime/src/z_Linux_util.cpp
index 8c59b43..7506380 100644
--- a/runtime/src/z_Linux_util.cpp
+++ b/runtime/src/z_Linux_util.cpp
@@ -179,7 +179,10 @@ void __kmp_affinity_determine_capable(const char *env_var) {
         KMP_INTERNAL_FREE(buf);
       }
       if (errno == EFAULT) {
-        KMP_AFFINITY_ENABLE(gCode);
+        int mask_size = ((gCode + sizeof(KMPNativeAffinity::Mask::mask_t) - 1) &
+                         ~(sizeof(KMPNativeAffinity::Mask::mask_t))) /
+                        sizeof(KMPNativeAffinity::Mask::mask_t);
+        KMP_AFFINITY_ENABLE(mask_size);
         KA_TRACE(10, ("__kmp_affinity_determine_capable: "
                       "affinity supported (mask size %d)\n",
                       (int)__kmp_affin_mask_size));
@@ -256,7 +259,10 @@ void __kmp_affinity_determine_capable(const char *env_var) {
         return;
       }
       if (errno == EFAULT) {
-        KMP_AFFINITY_ENABLE(gCode);
+        int mask_size = ((gCode + sizeof(KMPNativeAffinity::Mask::mask_t) - 1) &
+                         ~(sizeof(KMPNativeAffinity::Mask::mask_t))) /
+                        sizeof(KMPNativeAffinity::Mask::mask_t);
+        KMP_AFFINITY_ENABLE(mask_size);
         KA_TRACE(10, ("__kmp_affinity_determine_capable: "
                       "affinity supported (mask size %d)\n",
                       (int)__kmp_affin_mask_size));

Sorry, I didn't explain it all the way in my previous comment. __kmp_affin_mask_size should be the number of mask_ts to allocate (instead of just the raw size in bytes) which requires further dividing by sizeof(mask_t). So if we take the code you have in the macro and apply it to the two places in z_Linux_util.cpp (plus the extra division), then it would look like above. This also allows us to change mask_t to whatever type we please without any further modifications to the code. Also, this won't disturb the Windows/Hwloc code if we leave the KMP_AFFINITY_ENABLE() macro alone.

So

1. Revert KMP_AFFINITY_ENABLE()
2. Keep the rest of your current changes
3. Make mask_t public
4. Add the changes to z_Linux_util.cpp

[jlpeyton]

Changing state to request changes.

runtime/src/kmp_affinity.h

class KMPNativeAffinity : public KMPAffinity {
   public:
     mask_t *mask;
     Mask() { mask = (mask_t *)__kmp_allocate(__kmp_affin_mask_size); }
     ~Mask() {
       if (mask)
         __kmp_free(mask);
     }
     void set(int i) override {
-      mask[i / BITS_PER_MASK_T] |= ((mask_t)1 << (i % BITS_PER_MASK_T));
+      CPU_SET(i, (cpu_set_t*)mask);
     }
     bool is_set(int i) const override {
-      return (mask[i / BITS_PER_MASK_T] & ((mask_t)1 << (i % BITS_PER_MASK_T)));
+      return CPU_ISSET(i, (cpu_set_t*)mask);
     }
     void clear(int i) override {
-      mask[i / BITS_PER_MASK_T] &= ~((mask_t)1 << (i % BITS_PER_MASK_T));
+      CPU_CLR(i, (cpu_set_t*)mask);
     }
     void zero() override {
-      for (size_t i = 0; i < __kmp_affin_mask_size; ++i)
-        mask[i] = 0;
+      CPU_ZERO((cpu_set_t*)mask);
     }
     void copy(const KMPAffinity::Mask *src) override {
       const Mask *convert = static_cast<const Mask *>(src);
       for (size_t i = 0; i < __kmp_affin_mask_size; ++i)
         mask[i] = convert->mask[i];
     }
     void bitwise_and(const KMPAffinity::Mask *rhs) override {
       const Mask *convert = static_cast<const Mask *>(rhs);
-      for (size_t i = 0; i < __kmp_affin_mask_size; ++i)
-        mask[i] &= convert->mask[i];
+      CPU_AND((cpu_set_t*)mask, (cpu_set_t*)mask, (cpu_set_t*)(convert->mask));
     }
     void bitwise_or(const KMPAffinity::Mask *rhs) override {
       const Mask *convert = static_cast<const Mask *>(rhs);
-      for (size_t i = 0; i < __kmp_affin_mask_size; ++i)
-        mask[i] |= convert->mask[i];
+      CPU_OR((cpu_set_t*)mask, (cpu_set_t*)mask, (cpu_set_t*)(convert->mask));
     }
     void bitwise_not() override {
       for (size_t i = 0; i < __kmp_affin_mask_size; ++i)
         mask[i] = ~(mask[i]);
     }
     int begin() const override {
       int retval = 0;
       while (retval < end() && !is_set(retval))