Index: llvm/trunk/docs/Docker.rst =================================================================== --- llvm/trunk/docs/Docker.rst +++ llvm/trunk/docs/Docker.rst @@ -53,24 +53,15 @@ LLVM components, compiled from sources. The sources are checked out from the upstream svn repository when building the image. -Inside each subfolder we host Dockerfiles for two images: +The resulting image contains only the requested LLVM components and a few extra +packages to make the image minimally useful for C++ development, e.g. libstdc++ +and binutils. -- ``build/`` image is used to compile LLVM, it installs a system compiler and all - build dependencies of LLVM. After the build process is finished, the build - image will have an archive with compiled components at ``/tmp/clang.tar.gz``. -- ``release/`` image usually only contains LLVM components, compiled by the - ``build/`` image, and also libstdc++ and binutils to make image minimally - useful for C++ development. The assumption is that you usually want clang to - be one of the provided components. - -To build both of those images, use ``build_docker_image.sh`` script. -It will checkout LLVM sources and build clang in the ``build`` container, copy results -of the build to the local filesystem and then build the ``release`` container using -those. The ``build_docker_image.sh`` accepts a list of LLVM repositories to -checkout, and arguments for CMake invocation. +The interface to run the build is ``build_docker_image.sh`` script. It accepts a +list of LLVM repositories to checkout and arguments for CMake invocation. If you want to write your own docker image, start with an ``example/`` subfolder. -It provides incomplete Dockerfiles with (very few) FIXMEs explaining the steps +It provides an incomplete Dockerfile with (very few) FIXMEs explaining the steps you need to take in order to make your Dockerfiles functional. Usage @@ -110,10 +101,10 @@ -DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \ -DCLANG_ENABLE_BOOTSTRAP=ON -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-headers" -This will produce two images, a release image ``clang-debian8:staging`` and a -build image ``clang-debian8-build:staging`` from the latest upstream revision. -After the image is built you can run bash inside a container based on your -image like this: +This will produce a new image ``clang-debian8:staging`` from the latest +upstream revision. +After the image is built you can run bash inside a container based on your image +like this: .. code-block:: bash @@ -181,19 +172,14 @@ Minimizing docker image size ============================ -Due to Docker restrictions we use two images (i.e., build and release folders) -for the release image to be as small as possible. It's much easier to achieve -that using two images, because Docker would store a filesystem layer for each -command in the Dockerfile, i.e. if you install some packages in one command, -then remove those in a separate command, the size of the resulting image will -still be proportinal to the size of an image with installed packages. -Therefore, we strive to provide a very simple release image which only copies -compiled clang and does not do anything else. - -Docker 1.13 added a ``--squash`` flag that allows to flatten the layers of the -image, i.e. remove the parts that were actually deleted. That is an easier way -to produce the smallest images possible by using just a single image. We do not -use it because as of today the flag is in experimental stage and not everyone -may have the latest docker version available. When the flag is out of -experimental stage, we should investigate replacing two images approach with -just a single image, built using ``--squash`` flag. +Due to how Docker's filesystem works, all intermediate writes are persisted in +the resulting image, even if they are removed in the following commands. +To minimize the resulting image size we use `multi-stage Docker builds +`_. +Internally Docker builds two images. The first image does all the work: installs +build dependencies, checks out LLVM source code, compiles LLVM, etc. +The first image is only used during build and does not have a descriptive name, +i.e. it is only accessible via the hash value after the build is finished. +The second image is our resulting image. It contains only the built binaries +and not any build dependencies. It is also accessible via a descriptive name +(specified by -d and -t flags). Index: llvm/trunk/utils/docker/build_docker_image.sh =================================================================== --- llvm/trunk/utils/docker/build_docker_image.sh +++ llvm/trunk/utils/docker/build_docker_image.sh @@ -163,19 +163,9 @@ DOCKER_TAG=":$DOCKER_TAG" fi -echo "Building from $IMAGE_SOURCE" -echo "Building $DOCKER_REPOSITORY-build$DOCKER_TAG" -docker build -t "$DOCKER_REPOSITORY-build$DOCKER_TAG" \ +echo "Building ${DOCKER_REPOSITORY}${DOCKER_TAG} from $IMAGE_SOURCE" +docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \ --build-arg "buildscript_args=$BUILDSCRIPT_ARGS" \ - -f "$BUILD_DIR/$IMAGE_SOURCE/build/Dockerfile" \ + -f "$BUILD_DIR/$IMAGE_SOURCE/Dockerfile" \ "$BUILD_DIR" - -echo "Copying clang installation to release image sources" -docker run -v "$BUILD_DIR/$IMAGE_SOURCE:/workspace" "$DOCKER_REPOSITORY-build$DOCKER_TAG" \ - cp /tmp/clang.tar.gz /workspace/release - -echo "Building release image" -docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \ - "$BUILD_DIR/$IMAGE_SOURCE/release" - echo "Done" Index: llvm/trunk/utils/docker/debian8/Dockerfile =================================================================== --- llvm/trunk/utils/docker/debian8/Dockerfile +++ llvm/trunk/utils/docker/debian8/Dockerfile @@ -0,0 +1,58 @@ +#===- llvm/utils/docker/debian8/build/Dockerfile -------------------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// +# Stage 1. Check out LLVM source code and run the build. +FROM launcher.gcr.io/google/debian8:latest as builder +LABEL maintainer "LLVM Developers" +# Install build dependencies of llvm. +# First, Update the apt's source list and include the sources of the packages. +RUN grep deb /etc/apt/sources.list | \ + sed 's/^deb/deb-src /g' >> /etc/apt/sources.list +# Install compiler, python and subversion. +RUN apt-get update && \ + apt-get install -y --no-install-recommends ca-certificates gnupg \ + build-essential python wget subversion unzip && \ + rm -rf /var/lib/apt/lists/* +# Install a newer ninja release. It seems the older version in the debian repos +# randomly crashes when compiling llvm. +RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \ + echo "d2fea9ff33b3ef353161ed906f260d565ca55b8ca0568fa07b1d2cab90a84a07 ninja-linux.zip" \ + | sha256sum -c && \ + unzip ninja-linux.zip -d /usr/local/bin && \ + rm ninja-linux.zip +# Import public key required for verifying signature of cmake download. +RUN gpg --keyserver hkp://pgp.mit.edu --recv 0x2D2CEF1034921684 +# Download, verify and install cmake version that can compile clang into /usr/local. +# (Version in debian8 repos is is too old) +RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \ + wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt.asc" && \ + wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt" && \ + gpg --verify cmake-3.7.2-SHA-256.txt.asc cmake-3.7.2-SHA-256.txt && \ + wget "https://cmake.org/files/v3.7/cmake-3.7.2-Linux-x86_64.tar.gz" && \ + ( grep "cmake-3.7.2-Linux-x86_64.tar.gz" cmake-3.7.2-SHA-256.txt | \ + sha256sum -c - ) && \ + tar xzf cmake-3.7.2-Linux-x86_64.tar.gz -C /usr/local --strip-components=1 && \ + cd / && rm -rf /tmp/cmake-install + +ADD checksums /tmp/checksums +ADD scripts /tmp/scripts +# Arguments passed to build_install_clang.sh. +ARG buildscript_args +# Run the build. Results of the build will be available at /tmp/clang-install/. +RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args} + + +# Stage 2. Produce a minimal release image with build results. +FROM launcher.gcr.io/google/debian8:latest +LABEL maintainer "LLVM Developers" +# Install packages for minimal useful image. +RUN apt-get update && \ + apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \ + rm -rf /var/lib/apt/lists/* +# Copy build results of stage 1 to /usr/local. +COPY --from=builder /tmp/clang-install/ /usr/local/ Index: llvm/trunk/utils/docker/debian8/build/Dockerfile =================================================================== --- llvm/trunk/utils/docker/debian8/build/Dockerfile +++ llvm/trunk/utils/docker/debian8/build/Dockerfile @@ -1,55 +0,0 @@ -#===- llvm/utils/docker/debian8/build/Dockerfile -------------------------===// -# -# The LLVM Compiler Infrastructure -# -# This file is distributed under the University of Illinois Open Source -# License. See LICENSE.TXT for details. -# -#===----------------------------------------------------------------------===// -# Produces an image that compiles and archives clang, based on debian8. -FROM launcher.gcr.io/google/debian8:latest - -LABEL maintainer "LLVM Developers" - -# Install build dependencies of llvm. -# First, Update the apt's source list and include the sources of the packages. -RUN grep deb /etc/apt/sources.list | \ - sed 's/^deb/deb-src /g' >> /etc/apt/sources.list - -# Install compiler, python and subversion. -RUN apt-get update && \ - apt-get install -y --no-install-recommends ca-certificates gnupg \ - build-essential python wget subversion unzip && \ - rm -rf /var/lib/apt/lists/* - -# Install a newer ninja release. It seems the older version in the debian repos -# randomly crashes when compiling llvm. -RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \ - echo "d2fea9ff33b3ef353161ed906f260d565ca55b8ca0568fa07b1d2cab90a84a07 ninja-linux.zip" \ - | sha256sum -c && \ - unzip ninja-linux.zip -d /usr/local/bin && \ - rm ninja-linux.zip - -# Import public key required for verifying signature of cmake download. -RUN gpg --keyserver hkp://pgp.mit.edu --recv 0x2D2CEF1034921684 - -# Download, verify and install cmake version that can compile clang into /usr/local. -# (Version in debian8 repos is is too old) -RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \ - wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt.asc" && \ - wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt" && \ - gpg --verify cmake-3.7.2-SHA-256.txt.asc cmake-3.7.2-SHA-256.txt && \ - wget "https://cmake.org/files/v3.7/cmake-3.7.2-Linux-x86_64.tar.gz" && \ - ( grep "cmake-3.7.2-Linux-x86_64.tar.gz" cmake-3.7.2-SHA-256.txt | \ - sha256sum -c - ) && \ - tar xzf cmake-3.7.2-Linux-x86_64.tar.gz -C /usr/local --strip-components=1 && \ - cd / && rm -rf /tmp/cmake-install - -ADD checksums /tmp/checksums -ADD scripts /tmp/scripts - -# Arguments passed to build_install_clang.sh. -ARG buildscript_args - -# Run the build. Results of the build will be available as /tmp/clang.tar.gz. -RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args} Index: llvm/trunk/utils/docker/debian8/release/Dockerfile =================================================================== --- llvm/trunk/utils/docker/debian8/release/Dockerfile +++ llvm/trunk/utils/docker/debian8/release/Dockerfile @@ -1,21 +0,0 @@ -#===- llvm/utils/docker/debian8/release/Dockerfile -----------------------===// -# -# The LLVM Compiler Infrastructure -# -# This file is distributed under the University of Illinois Open Source -# License. See LICENSE.TXT for details. -# -#===----------------------------------------------------------------------===// -# A release image, containing clang installation, produced by the 'build/' image -# and adding libstdc++ and binutils. -FROM launcher.gcr.io/google/debian8:latest - -LABEL maintainer "LLVM Developers" - -# Install packages for minimal useful image. -RUN apt-get update && \ - apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \ - rm -rf /var/lib/apt/lists/* - -# Unpack clang installation into this image. -ADD clang.tar.gz /usr/local/ Index: llvm/trunk/utils/docker/example/Dockerfile =================================================================== --- llvm/trunk/utils/docker/example/Dockerfile +++ llvm/trunk/utils/docker/example/Dockerfile @@ -0,0 +1,37 @@ +#===- llvm/utils/docker/example/build/Dockerfile -------------------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// +# This is an example Dockerfile to build an image that compiles clang. +# Replace FIXMEs to prepare your own image. + +# Stage 1. Check out LLVM source code and run the build. +# FIXME: Replace 'ubuntu' with your base image +FROM ubuntu as builder +# FIXME: Change maintainer name +LABEL maintainer "Maintainer " +# FIXME: Install llvm/clang build dependencies here. Including compiler to +# build stage1, cmake, subversion, ninja, etc. + +ADD checksums /tmp/checksums +ADD scripts /tmp/scripts +# Arguments passed to build_install_clang.sh. +ARG buildscript_args +# Run the build. Results of the build will be available as /tmp/clang-install. +RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args} + + +# Stage 2. Produce a minimal release image with build results. +# FIXME: Replace 'ubuntu' with your base image. +FROM ubuntu +# FIXME: Change maintainer name. +LABEL maintainer "Maintainer " +# FIXME: Install all packages you want to have in your release container. +# A minimal useful installation should include at least libstdc++ and binutils. + +# Copy build results of stage 1 to /usr/local. +COPY --from=builder /tmp/clang-install/ /usr/local/ Index: llvm/trunk/utils/docker/example/build/Dockerfile =================================================================== --- llvm/trunk/utils/docker/example/build/Dockerfile +++ llvm/trunk/utils/docker/example/build/Dockerfile @@ -1,28 +0,0 @@ -#===- llvm/utils/docker/example/build/Dockerfile -------------------------===// -# -# The LLVM Compiler Infrastructure -# -# This file is distributed under the University of Illinois Open Source -# License. See LICENSE.TXT for details. -# -#===----------------------------------------------------------------------===// -# This is an example Dockerfile to build an image that compiles clang. -# Replace FIXMEs to prepare your own image. - -# FIXME: Replace 'ubuntu' with your base image -FROM ubuntu - -# FIXME: Change maintainer name -LABEL maintainer "Maintainer " - -# FIXME: Install llvm/clang build dependencies. Including compiler to -# build stage1, cmake, subversion, ninja, etc. - -ADD checksums /tmp/checksums -ADD scripts /tmp/scripts - -# Arguments passed to build_install_clang.sh. -ARG buildscript_args - -# Run the build. Results of the build will be available as /tmp/clang.tar.gz. -RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args} Index: llvm/trunk/utils/docker/example/release/Dockerfile =================================================================== --- llvm/trunk/utils/docker/example/release/Dockerfile +++ llvm/trunk/utils/docker/example/release/Dockerfile @@ -1,24 +0,0 @@ -#===- llvm/utils/docker/example/release/Dockerfile -----------------------===// -# -# The LLVM Compiler Infrastructure -# -# This file is distributed under the University of Illinois Open Source -# License. See LICENSE.TXT for details. -# -#===----------------------------------------------------------------------===// -# An image that unpacks a clang installation, compiled by the 'build/' -# container. -# Replace FIXMEs to prepare your own image. - -# FIXME: Replace 'ubuntu' with your base image. -FROM ubuntu - -# FIXME: Change maintainer name. -LABEL maintainer "Maintainer " - -# FIXME: Install all packages you want to have in your release container. -# A minimal useful installation must include libstdc++ and binutils. - -# Unpack clang installation into this container. -# It is copied to this directory by build_docker_image.sh script. -ADD clang.tar.gz /usr/local/ Index: llvm/trunk/utils/docker/nvidia-cuda/Dockerfile =================================================================== --- llvm/trunk/utils/docker/nvidia-cuda/Dockerfile +++ llvm/trunk/utils/docker/nvidia-cuda/Dockerfile @@ -0,0 +1,31 @@ +#===- llvm/utils/docker/nvidia-cuda/build/Dockerfile ---------------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// +# Stage 1. Check out LLVM source code and run the build. +FROM nvidia/cuda:8.0-devel as builder +LABEL maintainer "LLVM Developers" +# Install llvm build dependencies. +RUN apt-get update && \ + apt-get install -y --no-install-recommends ca-certificates cmake python \ + subversion ninja-build && \ + rm -rf /var/lib/apt/lists/* + +ADD checksums /tmp/checksums +ADD scripts /tmp/scripts +# Arguments passed to build_install_clang.sh. +ARG buildscript_args +# Run the build. Results of the build will be available at /tmp/clang-install/. +RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args} + + +# Stage 2. Produce a minimal release image with build results. +FROM nvidia/cuda:8.0-devel +LABEL maintainer "LLVM Developers" +# Copy clang installation into this container. +COPY --from=builder /tmp/clang-install/ /usr/local/ +# C++ standard library and binutils are already included in the base package. Index: llvm/trunk/utils/docker/nvidia-cuda/build/Dockerfile =================================================================== --- llvm/trunk/utils/docker/nvidia-cuda/build/Dockerfile +++ llvm/trunk/utils/docker/nvidia-cuda/build/Dockerfile @@ -1,31 +0,0 @@ -#===- llvm/utils/docker/nvidia-cuda/build/Dockerfile ---------------------===// -# -# The LLVM Compiler Infrastructure -# -# This file is distributed under the University of Illinois Open Source -# License. See LICENSE.TXT for details. -# -#===----------------------------------------------------------------------===// -# Produces an image that compiles and archives clang, based on nvidia/cuda -# image. -FROM nvidia/cuda:8.0-devel - -LABEL maintainer "LLVM Developers" - -# Arguments to pass to build_install_clang.sh. -ARG buildscript_args - -# Install llvm build dependencies. -RUN apt-get update && \ - apt-get install -y --no-install-recommends ca-certificates cmake python \ - subversion ninja-build && \ - rm -rf /var/lib/apt/lists/* - -ADD checksums /tmp/checksums -ADD scripts /tmp/scripts - -# Arguments passed to build_install_clang.sh. -ARG buildscript_args - -# Run the build. Results of the build will be available as /tmp/clang.tar.gz. -RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args} Index: llvm/trunk/utils/docker/nvidia-cuda/release/Dockerfile =================================================================== --- llvm/trunk/utils/docker/nvidia-cuda/release/Dockerfile +++ llvm/trunk/utils/docker/nvidia-cuda/release/Dockerfile @@ -1,23 +0,0 @@ -#===- llvm/utils/docker/nvidia-cuda/release/Dockerfile -------------------===// -# -# The LLVM Compiler Infrastructure -# -# This file is distributed under the University of Illinois Open Source -# License. See LICENSE.TXT for details. -# -#===----------------------------------------------------------------------===// -# This is an example Dockerfile that copies a clang installation, compiled -# by the 'build/' container into a fresh docker image to get a container of -# minimal size. -# Replace FIXMEs to prepare a new Dockerfile. - -# FIXME: Replace 'ubuntu' with your base image. -FROM nvidia/cuda:8.0-devel - -# FIXME: Change maintainer name. -LABEL maintainer "LLVM Developers" - -# Unpack clang installation into this container. -ADD clang.tar.gz /usr/local/ - -# C++ standard library and binutils are already included in the base package. Index: llvm/trunk/utils/docker/scripts/build_install_llvm.sh =================================================================== --- llvm/trunk/utils/docker/scripts/build_install_llvm.sh +++ llvm/trunk/utils/docker/scripts/build_install_llvm.sh @@ -16,8 +16,8 @@ Checkout svn sources and run cmake with the specified arguments. Used inside docker container. -Passes additional -DCMAKE_INSTALL_PREFIX and archives the contents of -the directory to /tmp/clang.tar.gz. +Passes additional -DCMAKE_INSTALL_PREFIX and puts the build results into +/tmp/clang-install/ directory. Available options: -h|--help show this help message @@ -244,12 +244,7 @@ popd -# Pack the installed clang into an archive. -echo "Archiving clang installation to /tmp/clang.tar.gz" -cd "$CLANG_INSTALL_DIR" -tar -czf /tmp/clang.tar.gz * - # Cleanup. -rm -rf "$CLANG_BUILD_DIR" "$CLANG_INSTALL_DIR" +rm -rf "$CLANG_BUILD_DIR" echo "Done"