[libc++/abi/unwind][AIX] Add scripts and cache file for building libc++/libc++abi/libunwind

[daltenty]

Summary

The script follows the pattern of the existing apple-install-libcxx.sh, and allows creating the combined 32/64-bit library archives
required for the C++ runtime on AIX.

Sample invocation

$./libcxx/utils/ci/aix-install-libcxx.sh --llvm-root  $PWD --build-dir ~/build-libcxx --install-dir ~/install-libcxx --cache ./libcxx/cmake/caches/AIX.cmake

[jasonliu]

It would help if we could put the sample invocation in the summary.

libcxx/utils/ci/aix-install-libcxx.sh

105–106

Could we check if these two settings are needed? i.e. are they on by default already in the current cmake config?

108

If a build did not complete successfully, for the ease of find out where it fails, do we want to have an early exit whenever there is an error on setting up cmake and during building?

[jasonliu]

libcxx/utils/ci/aix-install-libcxx.sh

108

For developers who invokes this script: do we want to detect if we already have a config build so that we don't need to run the cmake config every time? And provide a --clobber option if people wants to reconfig an existing config?

[lei]

libcxx/utils/ci/aix-install-libcxx.sh

50

Missing doc of this option in usage()

95–96

113–114

115

118–125

All these should be quoted.

[ldionne]

Hmm, this one hurts. The thing here is that I completely understand why you followed the lead of apple-install-libcxx.sh, however the truth is that that script is a hack for not being able to do what we want from CMake. It would be awesome if we could find a less ad-hoc way of doing this (both for you folks and for the Apple version).

@phosek, using the Runtimes build, would it be possible to build libc++ for 64 and 32 bits, and merge those? And is it possible to use the runtimes build without bootstrapping Clang first, i.e. with the system compiler?

[phosek]

In D101959#2745022, @ldionne wrote:

Hmm, this one hurts. The thing here is that I completely understand why you followed the lead of apple-install-libcxx.sh, however the truth is that that script is a hack for not being able to do what we want from CMake. It would be awesome if we could find a less ad-hoc way of doing this (both for you folks and for the Apple version).

@phosek, using the Runtimes build, would it be possible to build libc++ for 64 and 32 bits, and merge those? And is it possible to use the runtimes build without bootstrapping Clang first, i.e. with the system compiler?

Yes, it's something I'm actively looking into right now for Apple platforms since we want to ship libc++ as universal library in our toolchain. @beanz suggested using CMAKE_OSX_ARCHITECTURES and after some experimentation, I'm convinced that it's possible with a bit of CMake refactoring. Right now the blocker is some of the post-processing like the support for merging libc++abi and libc++ archives since utils/merge_archives.py doesn't work for universal archives, but that can be solved by using object libraries.

Unfortunately, CMake as far as I'm aware doesn't have any support for XCOFF or AIX so it's unlikely to help in this case. We could consider implementing a more general support in the runtimes build where you could run multiple target builds and then combine their results, but it's still likely going to require some platform-specific logic since there's no standard way of creating universal binaries.

To your second question, yes it's possible. If you build ${LLVM_MONOREPO}/runtimes as your CMake project, you can use arbitrary host compiler.

[daltenty]

In D101959#2745480, @phosek wrote:

In D101959#2745022, @ldionne wrote:

Hmm, this one hurts. The thing here is that I completely understand why you followed the lead of apple-install-libcxx.sh, however the truth is that that script is a hack for not being able to do what we want from CMake. It would be awesome if we could find a less ad-hoc way of doing this (both for you folks and for the Apple version).

@phosek, using the Runtimes build, would it be possible to build libc++ for 64 and 32 bits, and merge those? And is it possible to use the runtimes build without bootstrapping Clang first, i.e. with the system compiler?

Yes, it's something I'm actively looking into right now for Apple platforms since we want to ship libc++ as universal library in our toolchain. @beanz suggested using CMAKE_OSX_ARCHITECTURES and after some experimentation, I'm convinced that it's possible with a bit of CMake refactoring. Right now the blocker is some of the post-processing like the support for merging libc++abi and libc++ archives since utils/merge_archives.py doesn't work for universal archives, but that can be solved by using object libraries.

When reviews open on this refactoring, we'd definitely be interested reviewing. Hopefully we'd be able to re-use some of that setup to solve this problem as well.

Unfortunately, CMake as far as I'm aware doesn't have any support for XCOFF or AIX so it's unlikely to help in this case. We could consider implementing a more general support in the runtimes build where you could run multiple target builds and then combine their results, but it's still likely going to require some platform-specific logic since there's no standard way of creating universal binaries.

Yes, this is something we always wish there was better native CMake support for 🙂 Maybe getting a CMake property setup similar to CMAKE_OSX_ARCHITECTURES is worth investigating. Refactoring the runtimes build is possible but IIUC if Apple targets end up using CMAKE_OSX_ARCHITECTURES instead seems like AIX would be the only interested target?

I think a side question is, there is definitely a desire to do something better here, but both of the above options seem like they are going to require a larger refactoring. Should this script still proceed so there is something available within the LLVM 13 timeframe?

[daltenty]

libcxx/utils/ci/aix-install-libcxx.sh

50

Ah, this got moved to the cache file to be consistent with the existing setup, so I'll drop the option.

[daltenty]

• Address review comments

[daltenty]

libcxx/utils/ci/aix-install-libcxx.sh

108

If a build did not complete successfully, for the ease of find out where it fails, do we want to have an early exit whenever there is an error on setting up cmake and during building?

The set -e should hopefully take care of that for us.

[daltenty]

• Address comments #2

[daltenty]

• Add clobber option

[daltenty]

• Cleanup script style

[daltenty]

• Adjust cxxabi header install location to be consistent with previous install path on AIX
• Reindent file and do other minor cleanups

[daltenty]

Abandoned in favour of D111244

libcxx/cmake/caches/AIX.cmake

+set(CMAKE_BUILD_TYPE Release CACHE STRING "")
+set(CMAKE_BUILD_WITH_INSTALL_RPATH ON CACHE BOOL "")
+set(CMAKE_C_FLAGS "-D_ALL_SOURCE -mignore-xcoff-visibility" CACHE STRING "")
+set(CMAKE_CXX_FLAGS "-D_ALL_SOURCE -mignore-xcoff-visibility" CACHE STRING "")
+set(CMAKE_SHARED_LINKER_FLAGS "-Wl,-G -Wl,-bcdtors:all:-2147483548:s" CACHE STRING "")
+
+set(LIBCXX_ABI_VERSION "1" CACHE STRING "")
+set(LIBCXX_ENABLE_ASSERTIONS OFF CACHE BOOL "")
+set(LIBCXX_ENABLE_ABI_LINKER_SCRIPT OFF CACHE BOOL "")
+set(LIBCXX_ENABLE_EXPERIMENTAL_LIBRARY OFF CACHE BOOL "")
+set(LIBCXX_ENABLE_SHARED ON CACHE BOOL "")
+set(LIBCXX_ENABLE_STATIC OFF CACHE BOOL "")
+set(LIBCXX_CXX_ABI libcxxabi CACHE STRING "")
+set(LIBCXX_USE_COMPILER_RT ON CACHE BOOL "")
+
+set(LIBCXXABI_ENABLE_SHARED ON CACHE BOOL "")
+set(LIBCXXABI_ENABLE_STATIC OFF CACHE BOOL "")
+set(LIBCXXABI_USE_LLVM_UNWINDER ON CACHE BOOL "")
+
+set(LIBUNWIND_ENABLE_SHARED ON CACHE BOOL "")
+set(LIBUNWIND_ENABLE_STATIC OFF CACHE BOOL "")

libcxx/utils/ci/aix-install-libcxx.sh

+#!/usr/bin/env bash
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+set -e
+
+PROGNAME="$(basename "${0}")"
+
+function error() {
+  printf "error: %s\n" "$*"
+  exit 1
+}
+
+function usage() {
+cat <<EOF
+Usage:
+${PROGNAME} [options]
+
+[-h|--help]                  Display this help and exit.
+
+--llvm-root <DIR>            Path to the root of the LLVM monorepo. Only the libcxx,
+                             libcxxabi and libunwind directories are required.
+
+--build-dir <DIR>            Path to the directory to use for building. This will
+                             contain intermediate build products.
+
+--clobber                    Remove the build and install directories before building.
+
+--install-dir <DIR>          Path to the directory to install the library to.
+
+--cache <PATH>               The CMake cache to use to control how the library gets built.
+EOF
+}
+
+function realpath() {
+  if [[ $1 = /* ]]; then echo "$1"; else echo "$(pwd)/${1#./}"; fi
+}
+
+function parse_args() {
+  while [[ $# -gt 0 ]]; do
+    case ${1} in
+      -h|--help)
+        usage
+        exit 0
+        ;;
+      --llvm-root)
+        llvm_root="${2}"
+        shift
+        ;;
+      --build-dir)
+        build_dir="${2}"
+        shift
+        ;;
+      --clobber)
+        clobber=1
+        ;;
+      --install-dir)
+        install_dir="${2}"
+        shift
+        ;;
+      --cache)
+        cache="${2}"
+        shift
+        ;;
+      *)
+        error "Unknown argument '${1}'"
+        ;;
+    esac
+    shift
+  done
+
+  for arg in llvm_root build_dir install_dir cache; do
+    if [ -z ${!arg+x} ]; then
+      error "Missing required argument '--${arg//_/-}'"
+    elif [ "${!arg}" == "" ]; then
+      error "Argument to --${arg//_/-} must not be empty"
+    fi
+  done
+
+  # Allow using relative paths
+  for arg in llvm_root build_dir install_dir cache; do
+    path="$(realpath "${!arg}")"
+    eval "${arg}=\"${path}\""
+  done
+}
+
+function step() {
+  separator="$(printf "%0.s-" $(seq 1 ${#1}))"
+  echo
+  echo "${separator}"
+  echo "${1}"
+  echo "${separator}"
+}
+
+function configure_and_build() {
+  for bitmode in 32 64; do
+    step "Building libc++.a, libc++abi.a and libunwind.a for ${bitmode}-bit"
+    [[ ! -d "${build_dir}/${bitmode}" ]] && mkdir -p "${build_dir}/${bitmode}"
+    [[ ! -e "${build_dir}/${bitmode}/build.ninja" ]] && \
+      (cd "${build_dir}/${bitmode}" &&
+        env OBJECT_MODE="${bitmode}" cmake "${llvm_root}/llvm" \
+          -C "${cache}" \
+          -GNinja \
+          -DLLVM_ENABLE_PROJECTS="libcxx;libcxxabi;libunwind" \
+          -DCMAKE_INSTALL_PREFIX="${build_dir}/${bitmode}-install"
+      )
+    env OBJECT_MODE="${bitmode}" cmake --build "${build_dir}/${bitmode}" --target install-cxx install-cxxabi install-unwind
+  done
+}
+
+function merge_libraries() {
+  local libs="libc++ libc++abi libunwind"
+
+  step "Merging the resulting 32-bit and 64-bit libc++.a, libc++abi.a and libunwind.a into single libraries"
+  mkdir -p "${install_dir}/lib"
+
+
+  cd "${install_dir}/lib"
+  for lib in ${libs}; do
+    cp "${build_dir}/32/lib/${lib}.a" .
+    ar -X64 -x "${build_dir}/64/lib/${lib}.a"
+    ar -X64 r "${lib}.a" "${lib}.so.1"
+    rm "${lib}.so.1"
+  done
+}
+
+function install() {
+  # Install the headers by copying the headers from one of the builds into the
+  # install directory. Headers from both bitmodes should be the same.
+  local libs="libcxx libcxxabi libunwind"
+
+  step "Installing the libc++, libc++abi and libunwind headers to ${install_dir}/include"
+  mkdir -p "${install_dir}"
+  cp -r "${build_dir}/32-install/include" "${install_dir}"
+  [[ ! -d "${install_dir}/include/c++/v1" ]] && error "${install_dir}/include/c++/v1 was not created during install"
+  cp -r "${llvm_root}/libcxxabi/include/"* "${install_dir}/include/c++/v1" # TODO: libcxxabi should install its headers in CMake
+
+  step "Installing the libc++, libc++abi and libunwind licenses"
+  mkdir -p "${install_dir}/notices"
+  for lib in ${libs}; do
+    cp "${llvm_root}/${lib}/LICENSE.TXT" "${install_dir}/notices/${lib}.txt"
+  done
+}
+
+function main() {
+  parse_args "$@"
+
+  [[ -n "${clobber}" ]] && rm -rf "${build_dir}" && rm -rf "${install_dir}"
+
+  configure_and_build
+  merge_libraries
+  install
+}
+
+main "$@"