Make standalone-build-x86_64 a standalone builder

[kwk]

Summary

I've written an annotated bash script that will be used when running
builds of the standalone-build-x86_64 builder.

The bash script builds LLVM and installs it into a directory for later
consumption when building Clang independently.

[tstellar]

zorg/buildbot/builders/annotated/standalone-build.sh

47

We should wait until this patch is no longer needed before we commit this change.

72

Here are the minimal set of cmake args that should work:

-DCMAKE_BUILD_TYPE=Release \
-DLLVM_BUILD_LLVM_DYLIB=ON \
-DLLVM_LINK_LLVM_DYLIB=ON \
-DLLVM_INCLUDE_BENCHMARKS=OFF \
-DLLVM_INSTALL_UTILS=ON \
-DCMAKE_INSTALL_PREFIX=$install_prefix

158

Minimal CMake args for clang:

-DCMAKE_BUILD_TYPE=Release \
-DCLANG_LINK_CLANG_DYLIB=ON \
-DCLANG_INCLUDE_TESTS=ON \
-DLLVM_EXTERNAL_LIT=/usr/bin/lit \
-DCMAKE_INSTALL_PREFIX=$install_prefix \
-DLLVM_ROOT=$install_prefix \

[kwk]

@tstellar I'm trying your minimal config now on my build machine. I thought you wanted more details which is why I've taken the settings from Copr builds.

zorg/buildbot/builders/annotated/standalone-build.sh

47

We should wait until this patch is no longer needed before we commit this change.

No, I disagree with waiting because the patch is from Feb 22nd. How long do you want to wait? Maybe by showing that we need it we can have it land more quickly.

[kwk]

zorg/buildbot/builders/annotated/standalone-build.sh

158

Don't you think we should use the just installed lit as in -DLLVM_EXTERNAL_LIT=${LLVM_INSTALL_DIR}/bin/lit?

[kwk]

• Minimal config as suggested by @tstellar

[tstellar]

zorg/buildbot/builders/annotated/standalone-build.sh

47

Is clang-tools-extra even being built?

158

@kwk Yes, that's fine.

[kwk]

• Remove no longer needed export

[kwk]

zorg/buildbot/builders/annotated/standalone-build.sh

47

Well, without the patch things won't build:

CMake Error at /home/fedora/llvm-build-standalone/llvm-install/lib64/cmake/llvm/AddLLVM.cmake:1820 (add_dependencies):
  The dependency target "LLVMHello" of target "check-all" does not exist.
Call Stack (most recent call first):
  CMakeLists.txt:596 (add_lit_target)

[nikic]

zorg/buildbot/builders/annotated/standalone-build.sh

47

Maybe specify LLVM_MAIN_SRC_DIR until this is resolved?

[tstellar]

zorg/buildbot/builders/annotated/standalone-build.sh

47

Is it automatically building clang-tools-extra, because the source directory is there? I think we should be removing directories that we don't need so that we can catch when new dependencies are introduced between directories. We can use sparse-checkouts for this to make it easier.

[kwk]

• Patch not needed with minimal config and LLVM_ROOT

[kwk]

@nikic @tstellar The patch is no longer needed because of the changes to the minimal config suggested by @tstellar . I hope we can get it in now.

zorg/buildbot/builders/annotated/standalone-build.sh

47

No please no. I've rebuilt everything and the minimal config you suggested no longer produced the error anymore. This can go in without any patch whatsoever ;)

[tstellar]

zorg/buildbot/builders/annotated/standalone-build.sh

47

I still think we should use sparse-checkouts to only checkout the directories we need, but if you want to do that as a follow on patch, that's fine.

[serge-sans-paille]

zorg/buildbot/builders/annotated/standalone-build.sh

58

Do we want -DLLVM_INCLUDE_TOOLS:BOOL=ON -DLLVM_BUILD_TOOLS:BOOL=ON ?

92

Do we want to cleanup all build / install dir here to decrease pressure on filesystem or is it fine?

[kwk]

zorg/buildbot/builders/annotated/standalone-build.sh

47

I still think we should use sparse-checkouts to only checkout the directories we need, but if you want to do that as a follow on patch, that's fine.

I can put the sparse checkout in this patch, that's fine.

92

Do we want to cleanup all build / install dir here to decrease pressure on filesystem or is it fine?

Do you ask for why we clean up or why we don't clean up after the install?

We clean up before the install to make sure that whatever is installed contains no relics.

[kwk]

• Sparse checkout only the directories needed

[kwk]

@tstellar sparse checkout is implemented

[serge-sans-paille]

Looks good to me. Please wait for confirmation by @tstellar though.

[kwk]

• Organize build script with functions

[nikic]

zorg/buildbot/builders/annotated/standalone-build.sh

107

What does the LLVM_ROOT variable do? I couldn't find any uses of it in cmake files.

buildbot/osuosl/master/config/builders.py

'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
          })},
 
     # Standalone builder
     {'name' : "standalone-build-x86_64",
     'tags'  : ["clang"],
     'collapseRequests': False,
     'workernames':["standalone-build-x86_64"],
     'builddir':"standalone-build-x86_64",
-    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
-                    llvm_srcdir="llvm.src",
-                    obj_dir="llvm.obj",
-                    clean=True,
-                    depends_on_projects=['llvm','clang'],
-                    extra_configure_args=[
-                        "-DCOMPILER_RT_BUILD_BUILTINS:BOOL=OFF",
-                        "-DCOMPILER_RT_BUILD_ORC:BOOL=OFF",
-                        "-DCOMPILER_RT_BUILD_SANITIZERS:BOOL=OFF",
-                        "-DCOMPILER_RT_BUILD_XRAY:BOOL=OFF",
-                        "-DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF",
-                        "-DCMAKE_C_FLAGS=-Wdocumentation -Wno-documentation-deprecated-sync",
-                        "-DCMAKE_CXX_FLAGS=-std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync",
-                    ],
-                    )},
+    'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
+                    script="standalone-build.sh",
+                    checkout_llvm_sources=False,
+                    script_interpreter=None)},
 
 ]

zorg/buildbot/builders/annotated/standalone-build.sh

+#!/bin/bash
+
+# This script builds LLVM and Clang in standalone mode that means it first
+# builds LLVM and installs it into a specific directory. That directory is then
+# used when building Clang which depends on it.
+
+# Enable Error tracing
+set -o errtrace
+
+# Print trace for all commands ran before execution
+set -x
+
+# Include the Buildbot helper functions
+HERE="$(realpath $(dirname $0))"
+. ${HERE}/buildbot-helper.sh
+
+# Ensure all commands pass, and not dereferencing unset variables.
+set -eu
+halt_on_failure
+
+BUILDBOT_ROOT=${BUILDBOT_ROOT:-${HERE}}
+REVISION=${BUILDBOT_REVISION:-origin/main}
+LLVM_ROOT="${BUILDBOT_ROOT}/llvm-project"
+BUILD_TYPE=Release
+INSTALL_ROOT_DIR=${BUILDBOT_ROOT}/install
+BUILD_ROOT_DIR=${BUILDBOT_ROOT}/build
+
+install_dir() {
+    echo ${INSTALL_ROOT_DIR}/$1
+}
+
+build_dir() {
+    echo ${BUILD_ROOT_DIR}/$1
+}
+
+setup_llvm_project() {
+    build_step "Setup llvm-project"
+
+    if [ ! -d "${LLVM_ROOT}" ]; then
+      build_step "Cloning llvm-project repo"
+      git clone --progress https://github.com/llvm/llvm-project.git ${LLVM_ROOT}
+    fi
+
+    build_step "Updating llvm-project repo"
+    git -C "${LLVM_ROOT}" fetch origin
+    git -C "${LLVM_ROOT}" reset --hard ${REVISION}
+    git -C "${LLVM_ROOT}" sparse-checkout init --cone
+}
+
+pre_build_cleanup() {
+    build_step "Pre-build cleanup"
+    rm -rf ${INSTALL_ROOT_DIR}
+    rm -rf ${BUILD_ROOT_DIR}
+}
+
+build_llvm() {
+    local LLVM_BUILD_DIR=$(build_dir llvm)
+    local LLVM_INSTALL_DIR=$(install_dir llvm)
+
+    build_step "Sparse checkout out llvm"
+    git -C "${LLVM_ROOT}" sparse-checkout set llvm cmake
+
+    build_step "Configuring llvm"
+
+    cmake \
+        -S ${LLVM_ROOT}/llvm \
+        -B ${LLVM_BUILD_DIR} \
+        -G Ninja \
+        -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
+        -DLLVM_BUILD_LLVM_DYLIB=ON \
+        -DLLVM_LINK_LLVM_DYLIB=ON \
+        -DLLVM_INCLUDE_BENCHMARKS=OFF \
+        -DLLVM_INSTALL_UTILS=ON \
+        -DCMAKE_INSTALL_PREFIX=${LLVM_INSTALL_DIR}
+
+    build_step "Building llvm"
+    cmake --build ${LLVM_BUILD_DIR}
+
+    build_step "Installing llvm"
+    rm -rf ${LLVM_INSTALL_DIR}
+    cmake --install ${LLVM_BUILD_DIR}
+
+    # This is meant to extinguish any dependency on files being taken
+    # from the llvm build dir when building clang.
+    build_step "Removing llvm build directory"
+    rm -rf "${LLVM_BUILD_DIR}"
+}
+
+build_clang() {
+    local LLVM_INSTALL_DIR=$(install_dir llvm)
+    local CLANG_BUILD_DIR=$(build_dir clang)
+    local CLANG_INSTALL_DIR=$(install_dir clang)
+
+    build_step "Sparse checkout out clang"
+    git -C "${LLVM_ROOT}" sparse-checkout set clang cmake
+
+    build_step "Configuring clang"
+    cmake \
+        -S ${LLVM_ROOT}/clang \
+        -B ${CLANG_BUILD_DIR} \
+        -G Ninja \
+        -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
+        -DCLANG_LINK_CLANG_DYLIB=ON \
+        -DCLANG_INCLUDE_TESTS=ON \
+        -DLLVM_EXTERNAL_LIT=${LLVM_INSTALL_DIR}/bin/lit \
+        -DCMAKE_INSTALL_PREFIX=${CLANG_INSTALL_DIR} \
+        -DLLVM_ROOT=${LLVM_INSTALL_DIR}
+
+    build_step "Building clang"
+    LD_LIBRARY_PATH="${LLVM_INSTALL_DIR}/lib64" cmake --build ${CLANG_BUILD_DIR}
+
+    build_step "Installing clang"
+    rm -rf ${CLANG_INSTALL_DIR}
+    cmake --install ${CLANG_BUILD_DIR}
+}
+
+setup_llvm_project
+pre_build_cleanup
+
+build_llvm
+build_clang
+
+exit 0