[libc++][Android] Add Android support to CI Docker image

[rprichard]

Summary

Add an Android compiler and sysroot to support cross-compilation to
Android.

Add the Android platform-tools (e.g. adb) so that an Android buildbot
can run programs on an Android device. At container startup, copy
these platform tools to an "android-platform-tools" Docker volume to
share them with an emulator container. This copying ensures that the
emulator and libcxx-builder containers avoid mismatched adb versions.

Install Docker into libcxx-builder so that an Android buildbot can
manage a sibling Docker container that runs the Android emulator.

Stop configuring the BuildKite tags in buildkite-agent.cfg and provide
the tags using BUILDKITE_AGENT_TAGS instead. Using the environment
variable makes it possible to run separate VMs for non-Android versus
Android.

[rprichard]

libcxx/utils/ci/Dockerfile

131

I removed this setting in favor of setting an environment variable, e.g.:

BUILDKITE_AGENT_TAGS=queue=libcxx-builders,arch=x86_64,os=linux

However, if we want to use the same GCE VMs for Android and non-Android Linux, then we could keep setting the tags in the Dockerfile.

[rprichard]

This change was copied from the larger D139147 change.

[Mordante]

I would like @ldionne to have a look too since this changes a bit of the Buildkite infrastructure.

libcxx/utils/ci/run-buildbot-container

24

Why do we need this change?

[rprichard]

libcxx/utils/ci/run-buildbot-container

24

I needed it for testing the updated Docker image. i.e. I would build ldionne/libcxx-builder locally with the Android stuff added to it, and then I'd enter the buildbot container to run run-buildbot android-ndk. If I were to run docker pull ldionne/libcxx-builder, though, my locally-modified image would be replaced with the published one.

This isn't a new problem, though, so I'm wondering how the Docker image is ordinarily updated. e.g. When updating ldionne/libcxx-builder, is someone (e.g. ldionne or a bot) running tests with the new image before pushing it? The script was simpler before, so maybe the old process was to run docker run ... by hand, and not use this run-buildbot-container script.

[Mordante]

libcxx/utils/ci/run-buildbot-container

24

I just comment out that line when testing with a locally built version. I typically test a few build locally before pushing an update to Dockerhub.

[rprichard]

libcxx/utils/ci/run-buildbot-container

24

I kinda dislike the practice of keeping local edits around because it makes other version control work harder. e.g. I've been rebasing/merging/splitting/amending the Android commits for a while. I think I can put a docker wrapper script in my PATH that disables docker pull ldionne/libcxx-builder. I might do that instead of a local edit.

I'm thinking --no-pull or --local-image might be better names for the option if it were kept.

[rprichard]

Add the Android team's current Clang build to the libcxx-builder Dockerfile.

[rprichard]

Restore the buildkite-agent.cfg setting. I confirmed that the tags from this config file can be overridden using the BUILDKITE_AGENT_TAGS environment variable.

[rprichard]

Rebase.

[rprichard]

Rebase.

[rprichard]

Moved to GitHub, https://github.com/llvm/llvm-project/pull/69273.

libcxx/utils/ci/Dockerfile

 RUN rm /tmp/install-cmake.sh
 
 # Install a newer CMake for modules
 # TODO Remove the duplicated installation when all runtimes can be build with CMake 3.27.
 RUN wget https://github.com/Kitware/CMake/releases/download/v3.27.0-rc2/cmake-3.27.0-rc2-linux-x86_64.sh -O /tmp/install-cmake.sh
 RUN bash /tmp/install-cmake.sh --prefix=/opt --exclude-subdir --skip-license
 RUN rm /tmp/install-cmake.sh
 
+# Install the Android platform tools (e.g. adb) into /opt/android/sdk.
+RUN apt-get update && apt-get install -y unzip
+RUN mkdir -p /opt/android/sdk && cd /opt/android/sdk && \
+    curl -LO https://dl.google.com/android/repository/platform-tools-latest-linux.zip && \
+    unzip platform-tools-latest-linux.zip && \
+    rm platform-tools-latest-linux.zip
+
+# Install the current Android compiler.
+ENV ANDROID_CLANG_VERSION=r498229
+RUN git clone --filter=blob:none --sparse \
+        https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86 \
+        /opt/android/clang && \
+    git -C /opt/android/clang sparse-checkout add clang-${ANDROID_CLANG_VERSION} && \
+    rm -fr /opt/android/clang/.git && \
+    ln -sf /opt/android/clang/clang-${ANDROID_CLANG_VERSION} /opt/android/clang/clang-current
+
+# Install an Android sysroot. New AOSP sysroots are available at
+# https://ci.android.com/builds/branches/aosp-master/grid, the "ndk" target. The
+# NDK also makes its sysroot prebuilt available at
+# https://android.googlesource.com/platform/prebuilts/ndk/+/refs/heads/dev/platform/sysroot.
+ENV ANDROID_SYSROOT_BID=10375398
+RUN cd /opt/android && \
+    curl -L -o ndk_platform.tar.bz2 \
+        https://androidbuildinternal.googleapis.com/android/internal/build/v3/builds/${ANDROID_SYSROOT_BID}/ndk/attempts/latest/artifacts/ndk_platform.tar.bz2/url && \
+    tar xf ndk_platform.tar.bz2 && \
+    rm ndk_platform.tar.bz2
+
+# Install Docker. Mark the binary setuid so it can be run without prefixing it
+# with sudo. Adding the container user to the docker group doesn't work because
+# /var/run/docker.sock is owned by the host's docker GID, not the container's
+# docker GID.
+RUN apt-get update && apt-get install -y gpg && \
+    install -m 0755 -d /etc/apt/keyrings && \
+    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
+    chmod a+r /etc/apt/keyrings/docker.gpg && \
+    echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
+         "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
+         tee /etc/apt/sources.list.d/docker.list > /dev/null && \
+    apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin && \
+    chmod u+s /usr/bin/docker
+
+COPY ./container-setup.sh /opt/bin/container-setup.sh
+
 # Change the user to a non-root user, since some of the libc++ tests
 # (e.g. filesystem) require running as non-root. Also setup passwordless sudo.
 RUN apt-get update && apt-get install -y sudo
 RUN echo "ALL ALL = (ALL) NOPASSWD: ALL" >> /etc/sudoers
 RUN useradd --create-home libcxx-builder
 USER libcxx-builder
 WORKDIR /home/libcxx-builder
 
+# Add Android platform-tools to the PATH.
+ENV PATH="/opt/android/sdk/platform-tools:${PATH}"
+
 # Install the Buildkite agent and dependencies. This must be done as non-root
 # for the Buildkite agent to be installed in a path where we can find it.
 RUN bash -c "$(curl -sL https://raw.githubusercontent.com/buildkite/agent/main/install.sh)"
 ENV PATH="${PATH}:/home/libcxx-builder/.buildkite-agent/bin"
-RUN echo "tags=\"queue=libcxx-builders,arch=$(uname -m),os=linux\"" >> "/home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg"
 
 # By default, start the Buildkite agent (this requires a token).
-CMD buildkite-agent start
+CMD /opt/bin/container-setup.sh && buildkite-agent start

libcxx/utils/ci/container-setup.sh

+#!/usr/bin/env bash
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+set -e
+
+# Different versions of adb can sometimes be incompatible (i.e. "adb server
+# version (nn) doesn't match this client (mm); killing..."). Ensure that the adb
+# in the main builder image matches that in the emulator by sharing the
+# platform-tools from the main image.
+if [ -d /mnt/android-platform-tools ]; then
+    sudo rm -fr /mnt/android-platform-tools/platform-tools
+    sudo cp -r /opt/android/sdk/platform-tools /mnt/android-platform-tools
+fi

libcxx/utils/ci/run-buildbot-container

 # If you are on Linux you will likely not be able to write to the mount because
 # the user in the container doesn't have permissions to do so.
 # If you need to do this, give that user permission to do so after running
 # the container or add this flag to run the container as your local user IDs:
 # --user $(id -u):$(id -g)
 
 set -e
 
+SKIP_PULL=0
+while [[ ${#} -gt 0 ]]; do
+    case "${1}" in
+        --skip-pull) SKIP_PULL=1; shift ;;
+        *) echo "error: unrecognized argument: ${1}" >&2 ; exit 1 ;;
+    esac
+done
+
 MONOREPO_ROOT="$(git rev-parse --show-toplevel)"
 if [[ ! -d "${MONOREPO_ROOT}/libcxx/utils/ci" ]]; then
     echo "Was unable to find the root of the LLVM monorepo; are you running from within the monorepo?"
     exit 1
 fi
+
+if [ ${SKIP_PULL} -eq 0 ]; then
     docker pull ldionne/libcxx-builder
-docker run -it --volume "${MONOREPO_ROOT}:/llvm" --workdir "/llvm" --cap-add=SYS_PTRACE ldionne/libcxx-builder \
-    bash -c 'git config --global --add safe.directory /llvm ; exec bash'
+fi
+
+DOCKER_OPTIONS=(-it)
+DOCKER_OPTIONS+=(--volume "${MONOREPO_ROOT}:/llvm")
+DOCKER_OPTIONS+=(--workdir "/llvm")
+DOCKER_OPTIONS+=(--cap-add=SYS_PTRACE)
+
+# Mount this volume to allow the main image to share its copy of the Android
+# platform tools with the emulator image, ensuring that the adb versions match.
+# This argument will create a new volume if it doesn't already exist.
+DOCKER_OPTIONS+=(--volume android-platform-tools:/mnt/android-platform-tools)
+
+# Pass through the Docker socket so that the buildbot can start a sibling
+# container running an Android emulator.
+if [ -S /var/run/docker.sock ]; then
+    DOCKER_OPTIONS+=(--volume /var/run/docker.sock:/var/run/docker.sock)
+fi
+
+docker run "${DOCKER_OPTIONS[@]}" ldionne/libcxx-builder \
+    bash -c 'git config --global --add safe.directory /llvm; (/opt/bin/container-setup.sh && exec bash)'