[libomptarget] Require LLVM source tree to build libomptarget

[JonChesterfield]

Summary

[libomptarget] Require LLVM source tree to build libomptarget

This is to permit reliably #including files from the LLVM tree in libomptarget,
as an improvement on the copy and paste that is currently in use. See D87841
for the first example of removing duplication given this new requirement.

The weekly openmp dev call reached consensus on this approach. See also D87841
for some alternatives that were considered. In the future, we may want to
introduce a new top level repo for shared constants, or start using the ADT
library within openmp.

This will break sufficiently exotic build systems, trivial fixes as below.

Building libomptarget as part of the monorepo will continue to work.
If openmp is built separately, it now requires a cmake macro indicating
where to find the LLVM source tree.

If openmp is built separately, without the llvm source tree already on disk,
the build machine will need a copy of a subset of the llvm source tree and
the cmake macro indicating where it is.

[protze.joachim]

Will it still be possible to build the OpenMP runtime without LLVM headers/sources? Can we disable libomptarget if LLVM headers are not available?

[JonChesterfield]

This test is in libomptarget, so that should work. I don't know if there's an existing hook in libomp to disable libomptarget, or whether the former actually works without the latter. Can take a look.

[tianshilei1992]

It is a FATAL_ERROR so when CMake decides to include libomptarget, this would break the CMake generation.

[protze.joachim]

In openmp/CMakeLists.txt there is ENABLE_LIBOMPTARGET to decide about the default value for the CMake option OPENMP_ENABLE_LIBOMPTARGET. This way libomptarget gets disabled on apple and win32 by default.

if (OPENMP_STANDALONE_BUILD and NOT LLVM_MAIN_INCLUDE_DIR)
  set(ENABLE_LIBOMPTARGET OFF)
endif()

cmake -DOPENMP_ENABLE_LIBOMPTARGET=ON would allow to overwrite this default.

[JonChesterfield]

I'm unclear on what that cmake is doing. Copied here:

set(ENABLE_LIBOMPTARGET ON)
# Currently libomptarget cannot be compiled on Windows or MacOS X.                                                                                                                                                 
# Since the device plugins are only supported on Linux anyway,                                                                                                                                                     
# there is no point in trying to compile libomptarget on other OSes.                                                                                                                                               
if (APPLE OR WIN32 OR NOT OPENMP_HAVE_STD_CPP14_FLAG)
  set(ENABLE_LIBOMPTARGET OFF)
endif()

option(OPENMP_ENABLE_LIBOMPTARGET "Enable building libomptarget for offloading."
       ${ENABLE_LIBOMPTARGET})
if (OPENMP_ENABLE_LIBOMPTARGET)
  # Check that the library can actually be built.                                                                                                                                                                  
  if (APPLE OR WIN32)
    message(FATAL_ERROR "libomptarget cannot be built on Windows and MacOS X!")
  elseif (NOT OPENMP_HAVE_STD_CPP14_FLAG)
    message(FATAL_ERROR "Host compiler must support C++14 to build libomptarget!")
  endif()

  add_subdirectory(libomptarget)
endif()

Is this using ENABLE_LIBOMPTARGET as the default value for OPENMP_ENABLE_LIBOMPTARGET? If so, does setting OPENMP_ENABLE_LIBOMPTARGET win out?

A reasonable guess seems to be:

if (NOT LLVM_MAIN_INCLUDE_DIR)
message(...) #not fatal
set(ENABLE_LIBOMPTARGET OFF)
endif()

Probably combined with the above for anyone who builds libomptarget without the containing libomp.

[jdoerfert]

In D89426#2334516, @protze.joachim wrote:

Will it still be possible to build the OpenMP runtime without LLVM headers/sources? Can we disable libomptarget if LLVM headers are not available?

FWIW, that was (my) intend.

[JonChesterfield]

• review comments

[JonChesterfield]

Answering my own questions about cmake semantics by trial and error, the above revision handles the missing definition by not building libomptarget.

Tested the various permutations of LLVM_MAIN_INCLUDE_DIR present/absent in each file locally and all behave as one would hope.

Passing -DOPENMP_ENABLE_LIBOMPTARGET wins out over this default. Picked 'status' as that is used with message elsewhere in openmp's cmake.

[protze.joachim]

ENABLE_LIBOMPTARGET determines the initial default value for OPENMP_ENABLE_LIBOMPTARGET. Therefore the variable is only relevant in the first cmake run, but has no effect in successive cmake or ccmake configures. It will never overwrite the value of OPENMP_ENABLE_LIBOMPTARGET.

I'm not sure, whether the mono-repo structure is considered somewhere to find LLVM_MAIN_INCLUDE_DIR, but we should set the variable here to allow building openmp stand-alone in a mono-repo clone.

openmp/CMakeLists.txt

69–72

This change would allow to build openmp (with libomptarget) standalone, while checked out as mono-repo:

[JonChesterfield]

set(LLVM_MAIN_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../llvm/include) is interesting.

I'm not sure it's reasonable for openmp to guess at a plausible definition for LLVM_MAIN_INCLUDE_DIR and overwrite it. That composes poorly with (possibly imaginary) cmake that do their own testing based on this variable.

How about we introduce a whole new one,
LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR,
which is set to LLVM_MAIN_INCLUDE_DIR if available, otherwise to that guess, otherwise skip libomptarget?

[JonChesterfield]

• review comments
• Try harder to find LLVM on disk

[JonChesterfield]

New spin. Attempts to handle LLVM_MAIN_INCLUDE_DIR missing by looking for a plausible source directory, and avoids setting a variable which isn't LIBOMPTARGET_ prefixed.

[protze.joachim]

lgtm

openmp/CMakeLists.txt

 set(ENABLE_LIBOMPTARGET ON)
 # Currently libomptarget cannot be compiled on Windows or MacOS X.
 # Since the device plugins are only supported on Linux anyway,
 # there is no point in trying to compile libomptarget on other OSes.
 if (APPLE OR WIN32 OR NOT OPENMP_HAVE_STD_CPP14_FLAG)
   set(ENABLE_LIBOMPTARGET OFF)
 endif()
 
+# Attempt to locate LLVM source, required by libomptarget
+if (NOT LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR)
+  if (LLVM_MAIN_INCLUDE_DIR)
+    set(LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR ${LLVM_MAIN_INCLUDE_DIR})
+  elseif (EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/../llvm/include)
+    set(LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../llvm/include)
+  endif()
+endif()
+
+if (NOT LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR)
+  message(STATUS "Missing definition for LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR, disabling libomptarget")
+  set(ENABLE_LIBOMPTARGET OFF)
+endif()
+
 option(OPENMP_ENABLE_LIBOMPTARGET "Enable building libomptarget for offloading."
        ${ENABLE_LIBOMPTARGET})
 if (OPENMP_ENABLE_LIBOMPTARGET)
   # Check that the library can actually be built.
   if (APPLE OR WIN32)
     message(FATAL_ERROR "libomptarget cannot be built on Windows and MacOS X!")
   elseif (NOT OPENMP_HAVE_STD_CPP14_FLAG)
     message(FATAL_ERROR "Host compiler must support C++14 to build libomptarget!")

openmp/libomptarget/CMakeLists.txt

 endif()
 
 # Message utilities.
 include(LibomptargetUtils)
 
 # Get dependencies for the different components of the project.
 include(LibomptargetGetDependencies)
 
+# LLVM source tree is required at build time for libomptarget
+if (NOT LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR)
+  message(FATAL_ERROR "Missing definition for LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR")
+endif()
+
 # This is a list of all the targets that are supported/tested right now.
 set (LIBOMPTARGET_ALL_TARGETS "${LIBOMPTARGET_ALL_TARGETS} aarch64-unknown-linux-gnu")
 set (LIBOMPTARGET_ALL_TARGETS "${LIBOMPTARGET_ALL_TARGETS} powerpc64le-ibm-linux-gnu")
 set (LIBOMPTARGET_ALL_TARGETS "${LIBOMPTARGET_ALL_TARGETS} powerpc64-ibm-linux-gnu")
 set (LIBOMPTARGET_ALL_TARGETS "${LIBOMPTARGET_ALL_TARGETS} x86_64-pc-linux-gnu")
 set (LIBOMPTARGET_ALL_TARGETS "${LIBOMPTARGET_ALL_TARGETS} nvptx64-nvidia-cuda")
 
 # Once the plugins for the different targets are validated, they will be added to

openmp/libomptarget/plugins/amdgpu/CMakeLists.txt

if(NOT LIBOMPTARGET_DEP_LIBELF_FOUND)
   return()
 endif()
 
 if(NOT CMAKE_SYSTEM_PROCESSOR MATCHES "(x86_64)|(ppc64le)|(aarch64)$" AND CMAKE_SYSTEM_NAME MATCHES "Linux")
   libomptarget_say("Not building AMDGPU plugin: only support AMDGPU in Linux x86_64, ppc64le, or aarch64 hosts.")
   return()
 endif()
 
-if (NOT LLVM_MAIN_INCLUDE_DIR)
-  libomptarget_say("Not building AMDGPU plugin: Missing definition for LLVM_MAIN_INCLUDE_DIR")
+if (NOT LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR)
+  libomptarget_say("Not building AMDGPU plugin: Missing definition for LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR")
   return()
 endif()
 
 libomptarget_say("Building amdgpu offloading plugin")
 
 ################################################################################
 # Define the suffix for the runtime messaging dumps.
 add_definitions(-DTARGET_NAME=AMDGPU)
 if(CMAKE_SYSTEM_PROCESSOR MATCHES "(ppc64le)|(aarch64)$")
    add_definitions(-DLITTLEENDIAN_CPU=1)
 endif()
 
 if(CMAKE_BUILD_TYPE MATCHES Debug)
   add_definitions(-DDEBUG)
 endif()
 
 include_directories(
   ${CMAKE_CURRENT_SOURCE_DIR}/impl
-  ${LLVM_MAIN_INCLUDE_DIR}
+  ${LIBOMPTARGET_LLVM_MAIN_INCLUDE_DIR}
 )
 
 add_library(omptarget.rtl.amdgpu SHARED
       impl/atmi.cpp
       impl/atmi_interop_hsa.cpp
       impl/data.cpp
       impl/machine.cpp
       impl/system.cpp