[libc++][Android] Add Android support to CI Docker image

[rprichard]

Summary

Add an Android compiler and sysroot to support cross-compilation to
Android.

Add the Android platform-tools (e.g. adb) so that an Android buildbot
can run programs on an Android device. At container startup, copy
these platform tools to an "android-platform-tools" Docker volume to
share them with an emulator container. This copying ensures that the
emulator and libcxx-builder containers avoid mismatched adb versions.

Install Docker into libcxx-builder so that an Android buildbot can
manage a sibling Docker container that runs the Android emulator.

Stop configuring the BuildKite tags in buildkite-agent.cfg and provide
the tags using BUILDKITE_AGENT_TAGS instead. Using the environment
variable makes it possible to run separate VMs for non-Android versus
Android.

[rprichard]

libcxx/utils/ci/Dockerfile

131 ↗

(On Diff #540314)

I removed this setting in favor of setting an environment variable, e.g.:

BUILDKITE_AGENT_TAGS=queue=libcxx-builders,arch=x86_64,os=linux

However, if we want to use the same GCE VMs for Android and non-Android Linux, then we could keep setting the tags in the Dockerfile.

[rprichard]

This change was copied from the larger D139147 change.

[Mordante]

I would like @ldionne to have a look too since this changes a bit of the Buildkite infrastructure.

libcxx/utils/ci/run-buildbot-container

24 ↗

(On Diff #540314)

Why do we need this change?

[rprichard]

libcxx/utils/ci/run-buildbot-container

24 ↗

(On Diff #540314)

I needed it for testing the updated Docker image. i.e. I would build ldionne/libcxx-builder locally with the Android stuff added to it, and then I'd enter the buildbot container to run run-buildbot android-ndk. If I were to run docker pull ldionne/libcxx-builder, though, my locally-modified image would be replaced with the published one.

This isn't a new problem, though, so I'm wondering how the Docker image is ordinarily updated. e.g. When updating ldionne/libcxx-builder, is someone (e.g. ldionne or a bot) running tests with the new image before pushing it? The script was simpler before, so maybe the old process was to run docker run ... by hand, and not use this run-buildbot-container script.

[Mordante]

libcxx/utils/ci/run-buildbot-container

24 ↗

(On Diff #540314)

I just comment out that line when testing with a locally built version. I typically test a few build locally before pushing an update to Dockerhub.

[rprichard]

libcxx/utils/ci/run-buildbot-container

24 ↗

(On Diff #540314)

I kinda dislike the practice of keeping local edits around because it makes other version control work harder. e.g. I've been rebasing/merging/splitting/amending the Android commits for a while. I think I can put a docker wrapper script in my PATH that disables docker pull ldionne/libcxx-builder. I might do that instead of a local edit.

I'm thinking --no-pull or --local-image might be better names for the option if it were kept.

[rprichard]

Add the Android team's current Clang build to the libcxx-builder Dockerfile.

[rprichard]

Restore the buildkite-agent.cfg setting. I confirmed that the tags from this config file can be overridden using the BUILDKITE_AGENT_TAGS environment variable.

[rprichard]

Rebase.

[rprichard]

Rebase.

[rprichard]

Moved to GitHub, https://github.com/llvm/llvm-project/pull/69273.

libcxx/utils/ci/vendor/android/Dockerfile

+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+#
+# This Dockerfile extends ldionne/libcxx-builder with Android support, including
+# Android Clang and sysroot, Android platform-tools, and the Docker client.
+#
+#   $ docker build -t libcxx-builder-android libcxx/utils/ci/vendor/android
+#
+
+FROM ldionne/libcxx-builder
+
+# Switch back to the root user to install things into /opt and /usr.
+USER root
+WORKDIR /
+
+# Install the Android platform tools (e.g. adb) into /opt/android/sdk.
+RUN apt-get update && apt-get install -y unzip
+RUN mkdir -p /opt/android/sdk && cd /opt/android/sdk && \
+    curl -LO https://dl.google.com/android/repository/platform-tools-latest-linux.zip && \
+    unzip platform-tools-latest-linux.zip && \
+    rm platform-tools-latest-linux.zip
+
+# Install the current Android compiler. Specify the prebuilts commit to retrieve
+# this compiler version even after it's removed from HEAD.
+ENV ANDROID_CLANG_VERSION=r498229b
+ENV ANDROID_CLANG_PREBUILTS_COMMIT=5186d132c99aa75dc25207c392e3ea5b93d0107e
+RUN git clone --filter=blob:none --sparse \
+        https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86 \
+        /opt/android/clang && \
+    git -C /opt/android/clang checkout ${ANDROID_CLANG_PREBUILTS_COMMIT} && \
+    git -C /opt/android/clang sparse-checkout add clang-${ANDROID_CLANG_VERSION} && \
+    rm -fr /opt/android/clang/.git && \
+    ln -sf /opt/android/clang/clang-${ANDROID_CLANG_VERSION} /opt/android/clang/clang-current && \
+    # The "git sparse-checkout" and "ln" commands succeed even if nothing was
+    # checked out, so use this "ls" command to fix that.
+    ls /opt/android/clang/clang-current/bin/clang
+
+# Install an Android sysroot. New AOSP sysroots are available at
+# https://ci.android.com/builds/branches/aosp-main/grid, the "ndk" target. The
+# NDK also makes its sysroot prebuilt available at
+# https://android.googlesource.com/platform/prebuilts/ndk/+/refs/heads/dev/platform/sysroot.
+ENV ANDROID_SYSROOT_BID=10957860
+RUN cd /opt/android && \
+    curl -L -o ndk_platform.tar.bz2 \
+        https://androidbuildinternal.googleapis.com/android/internal/build/v3/builds/${ANDROID_SYSROOT_BID}/ndk/attempts/latest/artifacts/ndk_platform.tar.bz2/url && \
+    tar xf ndk_platform.tar.bz2 && \
+    rm ndk_platform.tar.bz2
+
+# Install Docker. Mark the binary setuid so it can be run without prefixing it
+# with sudo. Adding the container user to the docker group doesn't work because
+# /var/run/docker.sock is owned by the host's docker GID, not the container's
+# docker GID.
+RUN apt-get update && apt-get install -y gpg && \
+    install -m 0755 -d /etc/apt/keyrings && \
+    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
+    chmod a+r /etc/apt/keyrings/docker.gpg && \
+    echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
+         "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
+         tee /etc/apt/sources.list.d/docker.list > /dev/null && \
+    apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin && \
+    chmod u+s /usr/bin/docker
+
+COPY ./container-setup.sh /opt/android/container-setup.sh
+
+USER libcxx-builder
+WORKDIR /home/libcxx-builder
+
+# Add Android platform-tools to the PATH.
+ENV PATH="/opt/android/sdk/platform-tools:${PATH}"
+
+# Reset the buildkite-agent.cfg file. The tags are provided by an environment
+# variable instead.
+RUN cp /home/libcxx-builder/.buildkite-agent/buildkite-agent.dist.cfg \
+       /home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg
+
+# Modify the Buildkite agent cmdline to do Android setup stuff first.
+CMD /opt/android/container-setup.sh && buildkite-agent start

libcxx/utils/ci/vendor/android/container-setup.sh

+#!/usr/bin/env bash
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+set -e
+
+# Different versions of adb can sometimes be incompatible (i.e. "adb server
+# version (nn) doesn't match this client (mm); killing..."). Ensure that the adb
+# in the main builder image matches that in the emulator by sharing the
+# platform-tools from the main image.
+if [ -d /mnt/android-platform-tools ]; then
+    sudo rm -fr /mnt/android-platform-tools/platform-tools
+    sudo cp -r /opt/android/sdk/platform-tools /mnt/android-platform-tools
+fi

libcxx/utils/ci/vendor/android/run-buildbot-container

+#!/usr/bin/env bash
+
+# Similar to libcxx/utils/ci/run-buildbot-container, but adds additional options
+# needed for running Android tests.
+
+set -e
+
+MONOREPO_ROOT="$(git rev-parse --show-toplevel)"
+if [[ ! -d "${MONOREPO_ROOT}/libcxx/utils/ci/vendor/android" ]]; then
+    echo "Was unable to find the root of the LLVM monorepo; are you running from within the monorepo?"
+    exit 1
+fi
+
+DOCKER_OPTIONS=(-it)
+DOCKER_OPTIONS+=(--volume "${MONOREPO_ROOT}:/llvm")
+DOCKER_OPTIONS+=(--workdir "/llvm")
+DOCKER_OPTIONS+=(--cap-add=SYS_PTRACE)
+
+# Mount this volume to allow the main image to share its copy of the Android
+# platform tools with the emulator image, ensuring that the adb versions match.
+# This argument will create a new volume if it doesn't already exist.
+DOCKER_OPTIONS+=(--volume android-platform-tools:/mnt/android-platform-tools)
+
+# Pass through the Docker socket so that the buildbot can start a sibling
+# container running an Android emulator.
+if [ -S /var/run/docker.sock ]; then
+    DOCKER_OPTIONS+=(--volume /var/run/docker.sock:/var/run/docker.sock)
+fi
+
+docker run "${DOCKER_OPTIONS[@]}" libcxx-builder-android \
+    bash -c 'git config --global --add safe.directory /llvm; (/opt/android/container-setup.sh && exec bash)'

libcxx/utils/data/ignore_format.txt

 libcxx/test/support/user_defined_integral.h
 libcxx/test/support/uses_alloc_types.h
 libcxx/test/support/variant_test_helpers.h
 libcxx/test/support/wide_temp_file.h
 libcxx/utils/ci/Dockerfile
 libcxx/utils/ci/macos-ci-setup
 libcxx/utils/ci/run-buildbot
 libcxx/utils/ci/run-buildbot-container
+libcxx/utils/ci/vendor/android/Dockerfile
+libcxx/utils/ci/vendor/android/run-buildbot-container
 libcxx/utils/libcxx-lit