Add CMake bits necessary for standalone build

[krytarowski]

Summary

Developed on NetBSD with pkgsrc.

[krytarowski]

@labath feel free to disagree with the regex files. It was copied from LLVM. If you disagree, help me please to export it in LLVM build for everybody or find a better solution.

pkgsrc-wip commit affirming that it works:
https://wip.pkgsrc.org/cgi-bin/gitweb.cgi?p=pkgsrc-wip.git;a=commit;h=85621c794d26ed60844215ae16d3b4a4309d969e

Without this patch I'm falling asleep in front of the computer waiting for build to be finished.

[krytarowski]

Upload the proper latest patch.

[labath]

It seems to follow what clang does here, so I think it should be ok.

However, I'm not actually using the standalone build... I've added some people who I think might be using it. Let's give them a bit of time to respond. If there are no objections, then I'll commit this..

[zturner]

@labath, can you point out where clang is doing the same thing? I searched for files named regcomp.c for example, and only found the one copy in llvm. Duplicating code seems like a blocking issue to me, this will make the THIRD regex implementation in LLDB if we do this.

Standalone build does not mean you don't need to link against LLVM libraries.

[labath]

Hmm.. you are right, of course.. I was only looking at the second version of the patch, which seems to omit the regex files. However that seems to by accidental, as they are still references in the CMake files.

I agree that we should not copy the files over.

@krytarowski, I assume that the problem is that the regex headers are in llvm/lib (instead of include/llvm), which means they are not installed, and you cannot include them later. Is that correct?

It seems that is being done on purpose, so I guess it's a question whether lldb-mi should have been using them in the first place. We'll need to think about this more.

[zturner]

lldb-mi is including from the source tree, it should be including from the incldue tree.

In other words, the incldue should be changed to:

#include "llvm/Support/Regex.h"

and the code should be updated to use llvm::Regex instead of the raw c functions.

[labath]

OK, llvm seems to export the regex interface via include/llvm/Support/Regex.h. I think we should fix lldb-mi to use this interface instead, and then our problem will go away. The MI wrapper seems to be very simple, so it should not be a big problem to port it over.

[Eugene.Zelenko]

I build LLVM/Clang with LLVM_ENABLE_WARNINGS=ON, so I'll need to use llvm-config from build directory. I'm not sure how well find_program will work if path to <build directory>/bin will be not added to PATH, so probably will be good idea to allow LLVM_CONFIG to be set in Cmake command line and use find_program only if LLVM_CONFIG is not set.

[krytarowski]

In D15067#298701, @Eugene.Zelenko wrote:

I build LLVM/Clang with LLVM_ENABLE_WARNINGS=ON, so I'll need to use llvm-config from build directory. I'm not sure how well find_program will work if path to <build directory>/bin will be not added to PATH, so probably will be good idea to allow LLVM_CONFIG to be set in Cmake command line and use find_program only if LLVM_CONFIG is not set.

This should be already handled by CMake:
https://cmake.org/cmake/help/v3.0/command/find_program.html

[Eugene.Zelenko]

In D15067#298830, @krytarowski wrote:

In D15067#298701, @Eugene.Zelenko wrote:

I build LLVM/Clang with LLVM_ENABLE_WARNINGS=ON, so I'll need to use llvm-config from build directory. I'm not sure how well find_program will work if path to <build directory>/bin will be not added to PATH, so probably will be good idea to allow LLVM_CONFIG to be set in Cmake command line and use find_program only if LLVM_CONFIG is not set.

This should be already handled by CMake:
https://cmake.org/cmake/help/v3.0/command/find_program.html

But it's necessary to pass path from CMake command line argument to find_program after PATHS. Another problem if default OS location (if Clang installed and of old version) will be returned instead of build one.

[labath]

In D15067#298902, @Eugene.Zelenko wrote:

But it's necessary to pass path from CMake command line argument to find_program after PATHS. Another problem if default OS location (if Clang installed and of old version) will be returned instead of build one.

I don't know what would be the idiomatic cmake-fu to achieve that, but I'd just pass the the full path to the llvm-config binary, including the name. This way, we could also handle cases when the binary has a version suffix (e.g., debian likes to ship the binary as llvm-config-3.x).

[Eugene.Zelenko]

In D15067#299220, @labath wrote:

In D15067#298902, @Eugene.Zelenko wrote:

But it's necessary to pass path from CMake command line argument to find_program after PATHS. Another problem if default OS location (if Clang installed and of old version) will be returned instead of build one.

I don't know what would be the idiomatic cmake-fu to achieve that, but I'd just pass the the full path to the llvm-config binary, including the name. This way, we could also handle cases when the binary has a version suffix (e.g., debian likes to ship the binary as llvm-config-3.x).

I think find_program() should be called if LLVM_CONFIG is empty or not exist.

[krytarowski]

For the reference, I hold on with pushing this forward, till I will get significantly larger coverage with tests on my platform.

I keep this patch locally for local development.

[krytarowski]

Just in case someone will be quicker, new version against revamped CMake dirs is in pkgsrc-wip/lldb-git

[labath]

I am cleaning out my review queue. Feel free to add me back when you want to get this moving again...

[krytarowski]

Revamp the patch and reduce it only to the CMake part.

Leave regex code for later.

[krytarowski]

please take a look @zturner @labath

[krytarowski]

cmake/modules/LLDBStandalone.cmake

85 ↗

(On Diff #55742)

When I am including conditionally include(FindPythonInterp) with this PYTHON_EXECUTABLE STREQUAL "" conditional I'm facing the said issue with:

CMake Error at scripts/cmake_install.cmake:31 (file):
  file INSTALL cannot find
  "/tmp/pkgsrc-tmp/wip/lldb-git/work/build/lib/python.".
Call Stack (most recent call first):
  cmake_install.cmake:37 (include)
  


*** Error code 1

If I put here include(FindPythonInterp) unconditionally, there is no error in cmake_install.cmake:37.

[krytarowski]

cmake/modules/LLDBStandalone.cmake

85 ↗

(On Diff #55742)

My solution to this issue was to include include(FindPythonInterp) in scripts/CMakeLists.txt as noted in D19772.

[labath]

Looks good.

[krytarowski]

In D15067#419632, @labath wrote:

Looks good.

Thanks!

Are you accepting it with the patch in scripts/CMakeLists.txt as noted in the comment?

[labath]

Good question. I don't really have an opinion on that... Is it just that single line? (this would be simpler if you uploaded the final version :) ). If it just that single line, then I think it's fine. If you also need to play with the ADDITIONAL_VERSIONS and such, then we should think about making a macro or doing something else to avoid code duplication (there is already a FIXME in scripts/Python/modules/readline mentioning the additional versions).

[krytarowski]

In D15067#419964, @labath wrote:

Good question. I don't really have an opinion on that... Is it just that single line? (this would be simpler if you uploaded the final version :) ). If it just that single line, then I think it's fine. If you also need to play with the ADDITIONAL_VERSIONS and such, then we should think about making a macro or doing something else to avoid code duplication (there is already a FIXME in scripts/Python/modules/readline mentioning the additional versions).

Sorry for delay. I'm going to update the diff.

[krytarowski]

Updated. Can I commit it this way as it is?

[Eugene.Zelenko]

I would like to see implemented my comment about LLVM_CONFIG and find_program().

[krytarowski]

In D15067#427814, @Eugene.Zelenko wrote:

I would like to see implemented my comment about LLVM_CONFIG and find_program().

Could you first push this to clang? I don't want to drift from clang myself.

[Eugene.Zelenko]

In D15067#427834, @krytarowski wrote:

In D15067#427814, @Eugene.Zelenko wrote:

I would like to see implemented my comment about LLVM_CONFIG and find_program().

Could you first push this to clang? I don't want to drift from clang myself.

I'm not sure about Clang make files, but I built Clang with LLVM. LLDB is more problematic, so I build it standalone with previously build Clang. This why I'd like to specify path in CMake parameters, not relying in find_program().

[labath]

In D15067#427848, @Eugene.Zelenko wrote:

In D15067#427834, @krytarowski wrote:

In D15067#427814, @Eugene.Zelenko wrote:

I would like to see implemented my comment about LLVM_CONFIG and find_program().

Could you first push this to clang? I don't want to drift from clang myself.

I'm not sure about Clang make files, but I built Clang with LLVM. LLDB is more problematic, so I build it standalone with previously build Clang. This why I'd like to specify path in CMake parameters, not relying in find_program().

From cmake documentation on find program:

This command is used to find a program. A cache entry named by <VAR> is created to store the result of this command. If the program is found the result is stored in the variable and the search will not be repeated unless the variable is cleared.

So, I think the functionality you request is already baked into find_program and we should not duplicate it. I made a quick test and setting the variable via -DLLVM_CONFIG on the cmake command line does indeed prevent find-program from doing anything. (which is not surprising, as it is a fairly common use case for people to want to specify their own paths to things...)

I think this is good to go know. I don't think we need to wait for @zturner, as his objections were about the regular expression thing, which has been removed now.

[krytarowski]

@Eugene.Zelenko do you agree with this patch and what @labath wrote?

[Eugene.Zelenko]

In D15067#428146, @krytarowski wrote:

@Eugene.Zelenko do you agree with this patch and what @labath wrote?

If CMake works that way, it's what I need :-)

lldb/trunk/cmake/modules/LLDBStandalone.cmake

 # If we are not building as a part of LLVM, build LLDB as an
 # standalone project, using LLVM as an external library:
 if (CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
   project(lldb)
   cmake_minimum_required(VERSION 2.8.12.2)
 
   option(LLVM_INSTALL_TOOLCHAIN_ONLY "Only include toolchain files in the 'install' target." OFF)
 
-  set(LLDB_PATH_TO_LLVM_SOURCE "" CACHE PATH
-    "Path to LLVM source code. Not necessary if using an installed LLVM.")
-  set(LLDB_PATH_TO_LLVM_BUILD "" CACHE PATH
-    "Path to the directory where LLVM was built or installed.")
-
-  set(LLDB_PATH_TO_CLANG_SOURCE "" CACHE PATH
-    "Path to Clang source code. Not necessary if using an installed Clang.")
-  set(LLDB_PATH_TO_CLANG_BUILD "" CACHE PATH
-    "Path to the directory where Clang was built or installed.")
-
-  if (LLDB_PATH_TO_LLVM_SOURCE)
-    if (NOT EXISTS "${LLDB_PATH_TO_LLVM_SOURCE}/cmake/config-ix.cmake")
-      message(FATAL_ERROR "Please set LLDB_PATH_TO_LLVM_SOURCE to the root "
-              "directory of LLVM source code.")
-    else()
-      get_filename_component(LLVM_MAIN_SRC_DIR ${LLDB_PATH_TO_LLVM_SOURCE}
-                             ABSOLUTE)
-      set(LLVM_MAIN_INCLUDE_DIR "${LLVM_MAIN_SRC_DIR}/include")
-      list(APPEND CMAKE_MODULE_PATH "${LLVM_MAIN_SRC_DIR}/cmake/modules")
-    endif()
-  endif()
+  # Rely on llvm-config.
+  set(CONFIG_OUTPUT)
+  find_program(LLVM_CONFIG "llvm-config")
+  if(LLVM_CONFIG)
+    message(STATUS "Found LLVM_CONFIG as ${LLVM_CONFIG}")
+    set(CONFIG_COMMAND ${LLVM_CONFIG}
+      "--assertion-mode"
+      "--bindir"
+      "--libdir"
+      "--includedir"
+      "--prefix"
+      "--src-root")
+    execute_process(
+      COMMAND ${CONFIG_COMMAND}
+      RESULT_VARIABLE HAD_ERROR
+      OUTPUT_VARIABLE CONFIG_OUTPUT
+    )
+    if(NOT HAD_ERROR)
+      string(REGEX REPLACE
+        "[ \t]*[\r\n]+[ \t]*" ";"
+        CONFIG_OUTPUT ${CONFIG_OUTPUT})
 
-  if (LLDB_PATH_TO_CLANG_SOURCE)
-      get_filename_component(CLANG_MAIN_SRC_DIR ${LLDB_PATH_TO_CLANG_SOURCE}
-                             ABSOLUTE)
-      set(CLANG_MAIN_INCLUDE_DIR "${CLANG_MAIN_SRC_DIR}/include")
+    else()
+      string(REPLACE ";" " " CONFIG_COMMAND_STR "${CONFIG_COMMAND}")
+      message(STATUS "${CONFIG_COMMAND_STR}")
+      message(FATAL_ERROR "llvm-config failed with status ${HAD_ERROR}")
     endif()
-
-  list(APPEND CMAKE_MODULE_PATH "${LLDB_PATH_TO_LLVM_BUILD}/lib${LLVM_LIBDIR_SUFFIX}/cmake/llvm")
-
-  if (LLDB_PATH_TO_LLVM_BUILD)
-    get_filename_component(PATH_TO_LLVM_BUILD ${LLDB_PATH_TO_LLVM_BUILD}
-                           ABSOLUTE)
   else()
-    message(FATAL_ERROR "Please set LLDB_PATH_TO_LLVM_BUILD to the root "
-            "directory of LLVM build or install site.")
+    message(FATAL_ERROR "llvm-config not found -- ${LLVM_CONFIG}")
   endif()
 
-  if (LLDB_PATH_TO_CLANG_BUILD)
-    get_filename_component(PATH_TO_CLANG_BUILD ${LLDB_PATH_TO_CLANG_BUILD}
-                           ABSOLUTE)
+  list(GET CONFIG_OUTPUT 0 ENABLE_ASSERTIONS)
+  list(GET CONFIG_OUTPUT 1 TOOLS_BINARY_DIR)
+  list(GET CONFIG_OUTPUT 2 LIBRARY_DIR)
+  list(GET CONFIG_OUTPUT 3 INCLUDE_DIR)
+  list(GET CONFIG_OUTPUT 4 LLVM_OBJ_ROOT)
+  list(GET CONFIG_OUTPUT 5 MAIN_SRC_DIR)
+
+  if(NOT MSVC_IDE)
+    set(LLVM_ENABLE_ASSERTIONS ${ENABLE_ASSERTIONS}
+      CACHE BOOL "Enable assertions")
+    # Assertions should follow llvm-config's.
+    mark_as_advanced(LLVM_ENABLE_ASSERTIONS)
+  endif()
+
+  set(LLVM_TOOLS_BINARY_DIR ${TOOLS_BINARY_DIR} CACHE PATH "Path to llvm/bin")
+  set(LLVM_LIBRARY_DIR ${LIBRARY_DIR} CACHE PATH "Path to llvm/lib")
+  set(LLVM_MAIN_INCLUDE_DIR ${INCLUDE_DIR} CACHE PATH "Path to llvm/include")
+  set(LLVM_BINARY_DIR ${LLVM_OBJ_ROOT} CACHE PATH "Path to LLVM build tree")
+  set(LLVM_MAIN_SRC_DIR ${MAIN_SRC_DIR} CACHE PATH "Path to LLVM source tree")
+
+  find_program(LLVM_TABLEGEN_EXE "llvm-tblgen" ${LLVM_TOOLS_BINARY_DIR}
+    NO_DEFAULT_PATH)
+
+  set(LLVM_CMAKE_PATH "${LLVM_BINARY_DIR}/lib${LLVM_LIBDIR_SUFFIX}/cmake/llvm")
+  set(LLVMCONFIG_FILE "${LLVM_CMAKE_PATH}/LLVMConfig.cmake")
+  if(EXISTS ${LLVMCONFIG_FILE})
+    list(APPEND CMAKE_MODULE_PATH "${LLVM_CMAKE_PATH}")
+    include(${LLVMCONFIG_FILE})
   else()
-    message(FATAL_ERROR "Please set LLDB_PATH_TO_CLANG_BUILD to the root "
-            "directory of Clang build or install site.")
+    message(FATAL_ERROR "Not found: ${LLVMCONFIG_FILE}")
   endif()
 
-
-  # These variables are used by add_llvm_library.
+  # They are used as destination of target generators.
   set(LLVM_RUNTIME_OUTPUT_INTDIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/bin)
   set(LLVM_LIBRARY_OUTPUT_INTDIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/lib${LLVM_LIBDIR_SUFFIX})
-  set(LLVM_LIBRARY_DIR ${LLVM_LIBRARY_OUTPUT_INTDIR})
+  if(WIN32 OR CYGWIN)
+    # DLL platform -- put DLLs into bin.
+    set(LLVM_SHLIB_OUTPUT_INTDIR ${LLVM_RUNTIME_OUTPUT_INTDIR})
+  else()
+    set(LLVM_SHLIB_OUTPUT_INTDIR ${LLVM_LIBRARY_OUTPUT_INTDIR})
+  endif()
 
   include(AddLLVM)
   include(HandleLLVMOptions)
 
   if (PYTHON_EXECUTABLE STREQUAL "")
     set(Python_ADDITIONAL_VERSIONS 3.5 3.4 3.3 3.2 3.1 3.0 2.7 2.6 2.5)
     include(FindPythonInterp)
     if( NOT PYTHONINTERP_FOUND )
       message(FATAL_ERROR
               "Unable to find Python interpreter, required for builds and testing.
                Please install Python or specify the PYTHON_EXECUTABLE CMake variable.")
     endif()
   else()
     message("-- Found PythonInterp: ${PYTHON_EXECUTABLE}")
   endif()
+
   # Import CMake library targets from LLVM and Clang.
-  include("${LLDB_PATH_TO_LLVM_BUILD}/lib${LLVM_LIBDIR_SUFFIX}/cmake/llvm/LLVMConfig.cmake")
-  if (EXISTS "${LLDB_PATH_TO_CLANG_BUILD}/lib${LLVM_LIBDIR_SUFFIX}/cmake/clang/ClangConfig.cmake")
-      include("${LLDB_PATH_TO_CLANG_BUILD}/lib${LLVM_LIBDIR_SUFFIX}/cmake/clang/ClangConfig.cmake")
-  endif()
+  include("${LLVM_OBJ_ROOT}/lib${LLVM_LIBDIR_SUFFIX}/cmake/llvm/LLVMConfig.cmake")
+  include("${LLVM_OBJ_ROOT}/lib${LLVM_LIBDIR_SUFFIX}/cmake/clang/ClangConfig.cmake")
 
   set(PACKAGE_VERSION "${LLVM_PACKAGE_VERSION}")
 
   set(LLVM_BINARY_DIR ${CMAKE_BINARY_DIR})
 
   set(CMAKE_INCLUDE_CURRENT_DIR ON)
-  include_directories("${PATH_TO_LLVM_BUILD}/include"
-                      "${LLVM_MAIN_INCLUDE_DIR}"
-                      "${PATH_TO_CLANG_BUILD}/include"
-                      "${CLANG_MAIN_INCLUDE_DIR}"
-                      "${CMAKE_CURRENT_SOURCE_DIR}/source")
-  link_directories("${PATH_TO_LLVM_BUILD}/lib${LLVM_LIBDIR_SUFFIX}"
-                   "${PATH_TO_CLANG_BUILD}/lib${LLVM_LIBDIR_SUFFIX}")
+  include_directories("${LLVM_BINARY_DIR}/include" "${LLVM_MAIN_INCLUDE_DIR}")
+  link_directories("${LLVM_LIBRARY_DIR}")
 
   set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)
   set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib${LLVM_LIBDIR_SUFFIX})
   set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib${LLVM_LIBDIR_SUFFIX})
 
   set(LLDB_BUILT_STANDALONE 1)
 endif()

lldb/trunk/scripts/CMakeLists.txt

 file(GLOB SWIG_INTERFACES interface/*.i)
 file(GLOB_RECURSE SWIG_SOURCES *.swig)
 set(SWIG_HEADERS
   ${LLDB_SOURCE_DIR}/include/lldb/API/SBDefines.h
   ${LLDB_SOURCE_DIR}/include/lldb/lldb-defines.h
   ${LLDB_SOURCE_DIR}/include/lldb/lldb-enumerations.h
   ${LLDB_SOURCE_DIR}/include/lldb/lldb-forward.h
   ${LLDB_SOURCE_DIR}/include/lldb/lldb-types.h
   ${LLDB_SOURCE_DIR}/include/lldb/lldb-versioning.h
 )
 
+include(FindPythonInterp)
+
 find_package(SWIG REQUIRED)
 add_custom_command(
   OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/LLDBWrapPython.cpp
   OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/lldb.py
   DEPENDS ${SWIG_SOURCES}
   DEPENDS ${SWIG_INTERFACES}
   DEPENDS ${SWIG_HEADERS}
   DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/Python/prepare_binding_Python.py