Make standalone-build-x86_64 a standalone builder

[kwk]

Summary

I've written an annotated bash script that will be used when running
builds of the standalone-build-x86_64 builder.

The bash script builds LLVM and installs it into a directory for later
consumption when building Clang independently.

[tstellar]

zorg/buildbot/builders/annotated/standalone-build.sh

46

We should wait until this patch is no longer needed before we commit this change.

71

Here are the minimal set of cmake args that should work:

-DCMAKE_BUILD_TYPE=Release \
-DLLVM_BUILD_LLVM_DYLIB=ON \
-DLLVM_LINK_LLVM_DYLIB=ON \
-DLLVM_INCLUDE_BENCHMARKS=OFF \
-DLLVM_INSTALL_UTILS=ON \
-DCMAKE_INSTALL_PREFIX=$install_prefix

157

Minimal CMake args for clang:

-DCMAKE_BUILD_TYPE=Release \
-DCLANG_LINK_CLANG_DYLIB=ON \
-DCLANG_INCLUDE_TESTS=ON \
-DLLVM_EXTERNAL_LIT=/usr/bin/lit \
-DCMAKE_INSTALL_PREFIX=$install_prefix \
-DLLVM_ROOT=$install_prefix \

[kwk]

@tstellar I'm trying your minimal config now on my build machine. I thought you wanted more details which is why I've taken the settings from Copr builds.

zorg/buildbot/builders/annotated/standalone-build.sh

46

We should wait until this patch is no longer needed before we commit this change.

No, I disagree with waiting because the patch is from Feb 22nd. How long do you want to wait? Maybe by showing that we need it we can have it land more quickly.

[kwk]

zorg/buildbot/builders/annotated/standalone-build.sh

157

Don't you think we should use the just installed lit as in -DLLVM_EXTERNAL_LIT=${LLVM_INSTALL_DIR}/bin/lit?

[kwk]

• Minimal config as suggested by @tstellar

[tstellar]

zorg/buildbot/builders/annotated/standalone-build.sh

46

Is clang-tools-extra even being built?

157

@kwk Yes, that's fine.

[kwk]

• Remove no longer needed export

[kwk]

zorg/buildbot/builders/annotated/standalone-build.sh

46

Well, without the patch things won't build:

CMake Error at /home/fedora/llvm-build-standalone/llvm-install/lib64/cmake/llvm/AddLLVM.cmake:1820 (add_dependencies):
  The dependency target "LLVMHello" of target "check-all" does not exist.
Call Stack (most recent call first):
  CMakeLists.txt:596 (add_lit_target)

[nikic]

zorg/buildbot/builders/annotated/standalone-build.sh

46

Maybe specify LLVM_MAIN_SRC_DIR until this is resolved?

[tstellar]

zorg/buildbot/builders/annotated/standalone-build.sh

46

Is it automatically building clang-tools-extra, because the source directory is there? I think we should be removing directories that we don't need so that we can catch when new dependencies are introduced between directories. We can use sparse-checkouts for this to make it easier.

[kwk]

• Patch not needed with minimal config and LLVM_ROOT

[kwk]

@nikic @tstellar The patch is no longer needed because of the changes to the minimal config suggested by @tstellar . I hope we can get it in now.

zorg/buildbot/builders/annotated/standalone-build.sh

46

No please no. I've rebuilt everything and the minimal config you suggested no longer produced the error anymore. This can go in without any patch whatsoever ;)

[tstellar]

zorg/buildbot/builders/annotated/standalone-build.sh

46

I still think we should use sparse-checkouts to only checkout the directories we need, but if you want to do that as a follow on patch, that's fine.

[serge-sans-paille]

zorg/buildbot/builders/annotated/standalone-build.sh

58

Do we want -DLLVM_INCLUDE_TOOLS:BOOL=ON -DLLVM_BUILD_TOOLS:BOOL=ON ?

92

Do we want to cleanup all build / install dir here to decrease pressure on filesystem or is it fine?

[kwk]

zorg/buildbot/builders/annotated/standalone-build.sh

46

I still think we should use sparse-checkouts to only checkout the directories we need, but if you want to do that as a follow on patch, that's fine.

I can put the sparse checkout in this patch, that's fine.

92

Do we want to cleanup all build / install dir here to decrease pressure on filesystem or is it fine?

Do you ask for why we clean up or why we don't clean up after the install?

We clean up before the install to make sure that whatever is installed contains no relics.

[kwk]

• Sparse checkout only the directories needed

[kwk]

@tstellar sparse checkout is implemented

[serge-sans-paille]

Looks good to me. Please wait for confirmation by @tstellar though.

[kwk]

• Organize build script with functions

[nikic]

zorg/buildbot/builders/annotated/standalone-build.sh

108

What does the LLVM_ROOT variable do? I couldn't find any uses of it in cmake files.

buildbot/osuosl/master/config/builders.py

'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
          })},
 
     # Standalone builder
     {'name' : "standalone-build-x86_64",
     'tags'  : ["clang"],
     'collapseRequests': False,
     'workernames':["standalone-build-x86_64"],
     'builddir':"standalone-build-x86_64",
-    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
-                    llvm_srcdir="llvm.src",
-                    obj_dir="llvm.obj",
-                    clean=True,
-                    depends_on_projects=['llvm','clang'],
-                    extra_configure_args=[
-                        "-DCOMPILER_RT_BUILD_BUILTINS:BOOL=OFF",
-                        "-DCOMPILER_RT_BUILD_ORC:BOOL=OFF",
-                        "-DCOMPILER_RT_BUILD_SANITIZERS:BOOL=OFF",
-                        "-DCOMPILER_RT_BUILD_XRAY:BOOL=OFF",
-                        "-DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF",
-                        "-DCMAKE_C_FLAGS=-Wdocumentation -Wno-documentation-deprecated-sync",
-                        "-DCMAKE_CXX_FLAGS=-std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync",
-                    ],
-                    )},
+    'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
+                    script="standalone-build.sh",
+                    checkout_llvm_sources=False,
+                    script_interpreter=None)},
 
 ]

zorg/buildbot/builders/annotated/standalone-build.patch

+From 6ef3bbdc10086760ff02e9bc63427de9120aaa4f Mon Sep 17 00:00:00 2001
+From: Konrad Kleine <[e-mail]>
+Date: Thu, 5 May 2022 11:44:15 +0000
+Subject: D120301 rebased on main to support standalone builds
+
+---
+ clang-tools-extra/test/CMakeLists.txt                  | 10 +++++++++-
+ clang-tools-extra/test/clang-tidy/CTTestTidyModule.cpp |  2 +-
+ clang-tools-extra/test/lit.cfg.py                      |  3 +++
+ clang-tools-extra/test/lit.site.cfg.py.in              |  1 +
+ 4 files changed, 14 insertions(+), 2 deletions(-)
+
+diff --git a/clang-tools-extra/test/CMakeLists.txt b/clang-tools-extra/test/CMakeLists.txt
+index 2cdf186081ef..b03d99c8abb3 100644
+--- a/clang-tools-extra/test/CMakeLists.txt
++++ b/clang-tools-extra/test/CMakeLists.txt
+@@ -7,10 +7,15 @@
+ set(CLANG_TOOLS_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/..")
+ set(CLANG_TOOLS_BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/..")
+ 
++if (TARGET LLVMHello)
++  set (LLVM_HAS_LLVM_HELLO 1)
++endif()
++
+ llvm_canonicalize_cmake_booleans(
+   CLANG_TIDY_ENABLE_STATIC_ANALYZER
+   CLANG_PLUGIN_SUPPORT
+   LLVM_INSTALL_TOOLCHAIN_ONLY
++  LLVM_HAS_LLVM_HELLO
+   )
+ 
+ configure_lit_site_cfg(
+@@ -91,7 +96,10 @@ if (NOT LLVM_INSTALL_TOOLCHAIN_ONLY)
+   endif()
+ 
+   if(TARGET CTTestTidyModule)
+-      list(APPEND CLANG_TOOLS_TEST_DEPS CTTestTidyModule LLVMHello)
++      list(APPEND CLANG_TOOLS_TEST_DEPS CTTestTidyModule)
++      if (TARGET  LLVMHello)
++        list(APPEND CLANG_TOOLS_TEST_DEPS CTTestTidyModule)
++      endif()
+       target_include_directories(CTTestTidyModule PUBLIC BEFORE "${CLANG_TOOLS_SOURCE_DIR}")
+       if(CLANG_PLUGIN_SUPPORT AND (WIN32 OR CYGWIN))
+         set(LLVM_LINK_COMPONENTS
+diff --git a/clang-tools-extra/test/clang-tidy/CTTestTidyModule.cpp b/clang-tools-extra/test/clang-tidy/CTTestTidyModule.cpp
+index c66a94f458cf..b4e7a5d691e5 100644
+--- a/clang-tools-extra/test/clang-tidy/CTTestTidyModule.cpp
++++ b/clang-tools-extra/test/clang-tidy/CTTestTidyModule.cpp
+@@ -1,4 +1,4 @@
+-// REQUIRES: plugins
++// REQUIRES: plugins, llvm-hello
+ // RUN: clang-tidy -checks='-*,mytest*' --list-checks -load %llvmshlibdir/CTTestTidyModule%pluginext -load %llvmshlibdir/LLVMHello%pluginext | FileCheck --check-prefix=CHECK-LIST %s
+ // CHECK-LIST: Enabled checks:
+ // CHECK-LIST-NEXT:    mytest1
+diff --git a/clang-tools-extra/test/lit.cfg.py b/clang-tools-extra/test/lit.cfg.py
+index f8fdfbd0a920..947220134555 100644
+--- a/clang-tools-extra/test/lit.cfg.py
++++ b/clang-tools-extra/test/lit.cfg.py
+@@ -55,3 +55,6 @@ config.substitutions.append(
+ # Plugins (loadable modules)
+ if config.has_plugins and config.llvm_plugin_ext:
+     config.available_features.add('plugins')
++
++if config.has_llvm_hello:
++    config.available_features.add("llvm-hello")
+diff --git a/clang-tools-extra/test/lit.site.cfg.py.in b/clang-tools-extra/test/lit.site.cfg.py.in
+index 4eb830a1baf1..4f5228226292 100644
+--- a/clang-tools-extra/test/lit.site.cfg.py.in
++++ b/clang-tools-extra/test/lit.site.cfg.py.in
+@@ -16,6 +16,7 @@ config.has_plugins = @CLANG_PLUGIN_SUPPORT@ & ~@LLVM_INSTALL_TOOLCHAIN_ONLY@
+ config.llvm_tools_dir = lit_config.substitute("@LLVM_TOOLS_DIR@")
+ config.llvm_libs_dir = lit_config.substitute("@LLVM_LIBS_DIR@")
+ config.clang_tools_dir = lit_config.substitute("@CURRENT_TOOLS_DIR@")
++config.has_llvm_hello = @LLVM_HAS_LLVM_HELLO@
+ 
+ import lit.llvm
+ lit.llvm.initialize(lit_config, config)
+-- 
+2.31.1
+

zorg/buildbot/builders/annotated/standalone-build.sh

+#!/bin/bash
+
+# This script builds LLVM and Clang in standalone mode that means it first
+# builds LLVM and installs it into a specific directory. That directory is then
+# used when building Clang which depends on it. The configration is currently
+# very similar to the one being used to generate LLVM daily Fedora snapshots.
+# (See: https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/llvm-snapshots/)
+
+# Enable Error tracing
+set -o errtrace
+
+# Print trace for all commands ran before execution
+set -x
+
+# Include the Buildbot helper functions
+HERE="$(realpath $(dirname $0))"
+. ${HERE}/buildbot-helper.sh
+
+# Ensure all commands pass, and not dereferencing unset variables.
+set -eu
+halt_on_failure
+
+BUILDBOT_ROOT=${BUILDBOT_ROOT:-${HERE}}
+REVISION=${BUILDBOT_REVISION:-origin/main}
+LLVM_ROOT="${BUILDBOT_ROOT}/llvm-project"
+
+LLVM_INSTALL_DIR=${BUILDBOT_ROOT}/llvm-install/
+LLVM_BUILD_DIR=${BUILDBOT_ROOT}/llvm-build/
+CLANG_INSTALL_DIR=${BUILDBOT_ROOT}/clang-install/
+CLANG_BUILD_DIR=${BUILDBOT_ROOT}/clang-build/
+BUILD_TYPE=RelWithDebInfo
+
+# Set-up llvm-project
+if [ ! -d "${LLVM_ROOT}" ]; then
+  build_step "Cloning llvm-project repo"
+  git clone --progress https://github.com/llvm/llvm-project.git ${LLVM_ROOT}
+fi
+
+build_step "Updating llvm-project repo"
+git -C "${LLVM_ROOT}" fetch origin
+git -C "${LLVM_ROOT}" reset --hard ${REVISION}
+
+# Clang standalone builds are currently not posible unless this patch is applied
+build_step "Patch clang and clang-extra-tools for standalone builds (https://reviews.llvm.org/D120301)"
+git -C ${LLVM_ROOT} reset --hard origin/main
+git -C ${LLVM_ROOT} am ${HERE}/standalone-build.patch
+
+build_step "Pre-build cleanup"
+rm -rf ${LLVM_BUILD_DIR}
+rm -rf ${LLVM_INSTALL_DIR}
+rm -rf ${CLANG_BUILD_DIR}
+rm -rf ${CLANG_INSTALL_DIR}
+
+# See https://docs.fedoraproject.org/en-US/packaging-guidelines/RPMMacros/#macros_installation
+# for these variable names.
+
+build_step "Configuring llvm"
+
+_lib=lib64
+llvm_triple=x86_64-redhat-linux-gnu
+
+_prefix=${LLVM_INSTALL_DIR}
+_exec_prefix=${_prefix}
+_includedir=${_prefix}/include
+_bindir=${_exec_prefix}/bin
+_libdir=${_prefix}/${_lib}
+_sysconfdir=${_prefix}/etc
+_datarootdir=${_prefix}/share
+_pkgdocdir=${_prefix}/share/doc/llvm
+
+cmake \
+    -S ${LLVM_ROOT}/llvm \
+    -B ${LLVM_BUILD_DIR} \
+    -G Ninja \
+    -DCMAKE_C_FLAGS_RELEASE:STRING=-DNDEBUG \
+    -DCMAKE_CXX_FLAGS_RELEASE:STRING=-DNDEBUG \
+    -DCMAKE_Fortran_FLAGS_RELEASE:STRING=-DNDEBUG \
+    -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
+    -DCMAKE_INSTALL_DO_STRIP:BOOL=OFF \
+    -DCMAKE_INSTALL_PREFIX:PATH=${_prefix} \
+    -DINCLUDE_INSTALL_DIR:PATH=${_includedir} \
+    -DLIB_INSTALL_DIR:PATH=${_libdir} \
+    -DSYSCONF_INSTALL_DIR:PATH=${_sysconfdir} \
+    -DSHARE_INSTALL_PREFIX:PATH=${_datarootdir} \
+    -DLIB_SUFFIX=64 \
+    -DBUILD_SHARED_LIBS:BOOL=ON \
+    -DBUILD_SHARED_LIBS:BOOL=OFF \
+    -DLLVM_PARALLEL_LINK_JOBS=1 \
+    -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
+    -DCMAKE_SKIP_RPATH:BOOL=ON \
+    -DLLVM_LIBDIR_SUFFIX=64 \
+    -DLLVM_TARGETS_TO_BUILD=all \
+    -DLLVM_ENABLE_LIBCXX:BOOL=OFF \
+    -DLLVM_ENABLE_ZLIB:BOOL=ON \
+    -DLLVM_ENABLE_FFI:BOOL=ON \
+    -DLLVM_ENABLE_RTTI:BOOL=ON \
+    -DLLVM_USE_PERF:BOOL=ON \
+    -DLLVM_BINUTILS_INCDIR=${_includedir} \
+    -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=AVR \
+    -DLLVM_BUILD_RUNTIME:BOOL=ON \
+    -DLLVM_INCLUDE_TOOLS:BOOL=ON \
+    -DLLVM_BUILD_TOOLS:BOOL=ON \
+    -DLLVM_INCLUDE_TESTS:BOOL=ON \
+    -DLLVM_BUILD_TESTS:BOOL=ON \
+    -DLLVM_LIT_EXTRA_ARGS=-v \
+    -DLLVM_INCLUDE_EXAMPLES:BOOL=ON \
+    -DLLVM_BUILD_EXAMPLES:BOOL=OFF \
+    -DLLVM_INCLUDE_UTILS:BOOL=ON \
+    -DLLVM_INSTALL_UTILS:BOOL=ON \
+    -DLLVM_UTILS_INSTALL_DIR:PATH=${_bindir} \
+    -DLLVM_TOOLS_INSTALL_DIR:PATH=bin \
+    -DLLVM_INCLUDE_DOCS:BOOL=ON \
+    -DLLVM_BUILD_DOCS:BOOL=ON \
+    -DLLVM_ENABLE_SPHINX:BOOL=ON \
+    -DLLVM_ENABLE_DOXYGEN:BOOL=OFF \
+    -DLLVM_BUILD_LLVM_DYLIB:BOOL=ON \
+    -DLLVM_LINK_LLVM_DYLIB:BOOL=ON \
+    -DLLVM_BUILD_EXTERNAL_COMPILER_RT:BOOL=ON \
+    -DLLVM_INSTALL_TOOLCHAIN_ONLY:BOOL=OFF \
+    -DLLVM_DEFAULT_TARGET_TRIPLE=${llvm_triple} \
+    -DSPHINX_WARNINGS_AS_ERRORS=OFF \
+    -DLLVM_INSTALL_SPHINX_HTML_DIR=${_pkgdocdir}/html \
+    -DSPHINX_EXECUTABLE=/usr/bin/sphinx-build-3 \
+
+build_step "Building llvm"
+cmake --build ${LLVM_BUILD_DIR}
+
+build_step "Installing llvm"
+rm -rf "${LLVM_INSTALL_DIR}"
+cmake --install ${LLVM_BUILD_DIR}
+
+# This is meant to extinguish any dependency on files being taken
+# from the llvm build dir when building clang.
+build_step "Removing llvm build directory"
+rm -rf "${LLVM_BUILD_DIR}"
+
+build_step "Configuring clang"
+
+# TODO: Is this really needed?
+export LLVM_DIR=${LLVM_INSTALL_DIR}/lib64/cmake/llvm
+
+_prefix=${CLANG_INSTALL_DIR}
+_exec_prefix=${_prefix}
+_includedir=${_prefix}/include
+_bindir=${_exec_prefix}/bin
+_libdir=${_prefix}/${_lib}
+_sysconfdir=${_prefix}/etc
+_datarootdir=${_prefix}/share
+_pkgdocdir=${_prefix}/share/doc/llvm
+
+# help find llvm-config that we just build with build-llvm.sh
+# PATH="${LLVM_INSTALL_DIR}/bin;$PATH"
+
+cmake \
+    -S ${LLVM_ROOT}/clang \
+    -B ${CLANG_BUILD_DIR} \
+    -G Ninja \
+    -DLLVM_CMAKE_DIR=${LLVM_INSTALL_DIR}/lib64/cmake/llvm/ \
+    -DCMAKE_C_FLAGS_RELEASE:STRING=-DNDEBUG \
+    -DCMAKE_CXX_FLAGS_RELEASE:STRING=-DNDEBUG \
+    -DCMAKE_Fortran_FLAGS_RELEASE:STRING=-DNDEBUG \
+    -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
+    -DCMAKE_INSTALL_DO_STRIP:BOOL=OFF \
+    -DCMAKE_INSTALL_PREFIX:PATH=${_prefix} \
+    -DINCLUDE_INSTALL_DIR:PATH=${_includedir} \
+    -DLIB_INSTALL_DIR:PATH=${_libdir} \
+    -DSYSCONF_INSTALL_DIR:PATH=${_sysconfdir} \
+    -DSHARE_INSTALL_PREFIX:PATH=${_datarootdir} \
+    -DLIB_SUFFIX=64 \
+    -DBUILD_SHARED_LIBS:BOOL=ON \
+    -DLLVM_PARALLEL_LINK_JOBS=1 \
+    -DLLVM_LINK_LLVM_DYLIB:BOOL=ON \
+    -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
+    -DPYTHON_EXECUTABLE=/usr/bin/python3 \
+    -DCMAKE_SKIP_RPATH:BOOL=ON \
+    -DLLVM_ENABLE_PLUGINS:BOOL=ON \
+    -DCLANG_ENABLE_PLUGINS:BOOL=ON \
+    -DCLANG_INCLUDE_TESTS:BOOL=ON \
+    -DLLVM_EXTERNAL_CLANG_TOOLS_EXTRA_SOURCE_DIR=${LLVM_ROOT}/clang-tools-extra \
+    -DLLVM_EXTERNAL_LIT=${LLVM_INSTALL_DIR}/bin/lit \
+    -DLLVM_MAIN_SRC_DIR=${_datarootdir}/llvm/src \
+    -DLLVM_LIBDIR_SUFFIX=64 \
+    -DLLVM_TABLEGEN_EXE:FILEPATH=${LLVM_INSTALL_DIR}/bin/llvm-tblgen \
+    -DCLANG_ENABLE_ARCMT:BOOL=ON \
+    -DCLANG_ENABLE_STATIC_ANALYZER:BOOL=ON \
+    -DCLANG_INCLUDE_DOCS:BOOL=ON \
+    -DCLANG_PLUGIN_SUPPORT:BOOL=ON \
+    -DENABLE_LINKER_BUILD_ID:BOOL=ON \
+    -DLLVM_ENABLE_EH=ON \
+    -DLLVM_ENABLE_RTTI=ON \
+    -DLLVM_BUILD_DOCS=ON \
+    -DLLVM_ENABLE_NEW_PASS_MANAGER=ON \
+    -DLLVM_ENABLE_SPHINX=ON \
+    -DCLANG_LINK_CLANG_DYLIB=ON \
+    -DSPHINX_WARNINGS_AS_ERRORS=OFF \
+    -DCLANG_BUILD_EXAMPLES:BOOL=OFF \
+    -DBUILD_SHARED_LIBS=OFF \
+    -DCLANG_DEFAULT_UNWINDLIB=libgcc \
+    -DCMAKE_INSTALL_LIBDIR=${_libdir}
+
+build_step "Building clang"
+LD_LIBRARY_PATH="${LLVM_INSTALL_DIR}/lib64" cmake --build ${CLANG_BUILD_DIR}
+
+build_step "Installing clang"
+rm -rf ${CLANG_INSTALL_DIR}
+cmake --install ${CLANG_INSTALL_DIR}
+
+exit 0
 No newline at end of file