This is an archive of the discontinued LLVM Phabricator instance.

Adding Hwloc library option for affinity mechanism.
ClosedPublic

Authored by jlpeyton on Oct 22 2015, 12:03 PM.

Details

Summary

These changes allow libhwloc to be used as the topology discovery/affinity mechanism for libomp.
It is supported on Unices. The code additions:

  1. Canonicalize KMP_CPU_* interface macros so bitmask operations are implementation independent and work with both hwloc bitmaps and libomp bitmaps. So there are new KMP_CPU_ALLOC_* and KMP_CPU_ITERATE() macros and the like. These are all in kmp.h and appropriately placed.
  2. Hwloc topology discovery code in kmp_affinity.cpp. This uses the hwloc interface to create a libomp address2os object which the rest of libomp knows how to handle already.
  3. To build, use -DLIBOMP_USE_HWLOC=on and -DLIBOMP_HWLOC_INSTALL_DIR=/path/to/install/dir [default /usr/local]. If CMake can't find the library or hwloc.h, then it will tell you and exit.

Diff Detail

Repository
rL LLVM

Event Timeline

jlpeyton updated this revision to Diff 38152.Oct 22 2015, 12:03 PM
jlpeyton retitled this revision from to Adding Hwloc library option for affinity mechanism..
jlpeyton updated this object.
jlpeyton set the repository for this revision to rL LLVM.
jlpeyton added a subscriber: openmp-commits.
hfinkel edited edge metadata.Oct 27 2015, 12:40 PM

This seems like a good idea. Are you thinking about eventually requiring hwloc for affinity support and eliminating the builtin code?

runtime/Build_With_CMake.txt
163 ↗(On Diff #38152)

We should add a link here to the hwloc project page (http://www.open-mpi.org/projects/hwloc/).

There are some very subtle details with the way that the cpuid HW topology detection mechanism works that I've spent years hammering out, and hwloc has the same bugs as the Intel runtime affinity mechanism had in 2006. We aren't ready to switch to hwloc as a default, and I don't want to see the LLVM version of libiomp5 diverge from the Intel version WRT thread affinity...

FYI

-bb

There are some very subtle details with the way that the cpuid HW topology detection mechanism works that I've spent years hammering out, and hwloc has the same bugs as the Intel runtime affinity mechanism had in 2006. We aren't ready to switch to hwloc as a default, and I don't want to see the LLVM version of libiomp5 diverge from the Intel version WRT thread affinity...

FYI

In that case, it seems like this option should carry a very stern *EXPERIMENTAL* warning.

Will we be able to enable affinity support by default for all targets when hwloc is in use?

-bb

jlpeyton updated this revision to Diff 38603.Oct 27 2015, 3:21 PM
jlpeyton edited edge metadata.

Added hwloc website to both Build_With_CMake.txt and CMakeLists.txt.

It doesn't work on Windows, but Unices are fine. I've added a check that says Hwloc support isn't supported on Windows if a user tries to use it there.

tlwilmar edited edge metadata.Nov 4 2015, 2:25 PM

In that case, it seems like this option should carry a very stern *EXPERIMENTAL* warning.

Will we be able to enable affinity support by default for all targets when hwloc is in use?

Actually, after Brian's recent examination of the hwloc source, there aren't any outstanding bugs in hwloc that are cause for concern. So we do plan to migrate in the direction of using it. We have other logistical issues to work out before we use it internally. so we don't plan to propose a switch over to it here for some time yet, and it should be considered experimental for now (but perhaps stern warnings aren't needed).

I haven't gone over the Windows side yet. The Linux side looks reasonable.

-bb

AndreyChurbanov accepted this revision.Nov 20 2015, 5:13 AM
AndreyChurbanov edited edge metadata.

LGTM

This revision is now accepted and ready to land.Nov 20 2015, 5:13 AM
This revision was automatically updated to reflect the committed changes.