Index: llvm/trunk/docs/Docker.rst
===================================================================
--- llvm/trunk/docs/Docker.rst
+++ llvm/trunk/docs/Docker.rst
@@ -53,24 +53,15 @@
LLVM components, compiled from sources. The sources are checked out from the
upstream svn repository when building the image.
-Inside each subfolder we host Dockerfiles for two images:
+The resulting image contains only the requested LLVM components and a few extra
+packages to make the image minimally useful for C++ development, e.g. libstdc++
+and binutils.
-- ``build/`` image is used to compile LLVM, it installs a system compiler and all
- build dependencies of LLVM. After the build process is finished, the build
- image will have an archive with compiled components at ``/tmp/clang.tar.gz``.
-- ``release/`` image usually only contains LLVM components, compiled by the
- ``build/`` image, and also libstdc++ and binutils to make image minimally
- useful for C++ development. The assumption is that you usually want clang to
- be one of the provided components.
-
-To build both of those images, use ``build_docker_image.sh`` script.
-It will checkout LLVM sources and build clang in the ``build`` container, copy results
-of the build to the local filesystem and then build the ``release`` container using
-those. The ``build_docker_image.sh`` accepts a list of LLVM repositories to
-checkout, and arguments for CMake invocation.
+The interface to run the build is ``build_docker_image.sh`` script. It accepts a
+list of LLVM repositories to checkout and arguments for CMake invocation.
If you want to write your own docker image, start with an ``example/`` subfolder.
-It provides incomplete Dockerfiles with (very few) FIXMEs explaining the steps
+It provides an incomplete Dockerfile with (very few) FIXMEs explaining the steps
you need to take in order to make your Dockerfiles functional.
Usage
@@ -110,10 +101,10 @@
-DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \
-DCLANG_ENABLE_BOOTSTRAP=ON -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-headers"
-This will produce two images, a release image ``clang-debian8:staging`` and a
-build image ``clang-debian8-build:staging`` from the latest upstream revision.
-After the image is built you can run bash inside a container based on your
-image like this:
+This will produce a new image ``clang-debian8:staging`` from the latest
+upstream revision.
+After the image is built you can run bash inside a container based on your image
+like this:
.. code-block:: bash
@@ -181,19 +172,14 @@
Minimizing docker image size
============================
-Due to Docker restrictions we use two images (i.e., build and release folders)
-for the release image to be as small as possible. It's much easier to achieve
-that using two images, because Docker would store a filesystem layer for each
-command in the Dockerfile, i.e. if you install some packages in one command,
-then remove those in a separate command, the size of the resulting image will
-still be proportinal to the size of an image with installed packages.
-Therefore, we strive to provide a very simple release image which only copies
-compiled clang and does not do anything else.
-
-Docker 1.13 added a ``--squash`` flag that allows to flatten the layers of the
-image, i.e. remove the parts that were actually deleted. That is an easier way
-to produce the smallest images possible by using just a single image. We do not
-use it because as of today the flag is in experimental stage and not everyone
-may have the latest docker version available. When the flag is out of
-experimental stage, we should investigate replacing two images approach with
-just a single image, built using ``--squash`` flag.
+Due to how Docker's filesystem works, all intermediate writes are persisted in
+the resulting image, even if they are removed in the following commands.
+To minimize the resulting image size we use `multi-stage Docker builds
+`_.
+Internally Docker builds two images. The first image does all the work: installs
+build dependencies, checks out LLVM source code, compiles LLVM, etc.
+The first image is only used during build and does not have a descriptive name,
+i.e. it is only accessible via the hash value after the build is finished.
+The second image is our resulting image. It contains only the built binaries
+and not any build dependencies. It is also accessible via a descriptive name
+(specified by -d and -t flags).
Index: llvm/trunk/utils/docker/build_docker_image.sh
===================================================================
--- llvm/trunk/utils/docker/build_docker_image.sh
+++ llvm/trunk/utils/docker/build_docker_image.sh
@@ -163,19 +163,9 @@
DOCKER_TAG=":$DOCKER_TAG"
fi
-echo "Building from $IMAGE_SOURCE"
-echo "Building $DOCKER_REPOSITORY-build$DOCKER_TAG"
-docker build -t "$DOCKER_REPOSITORY-build$DOCKER_TAG" \
+echo "Building ${DOCKER_REPOSITORY}${DOCKER_TAG} from $IMAGE_SOURCE"
+docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \
--build-arg "buildscript_args=$BUILDSCRIPT_ARGS" \
- -f "$BUILD_DIR/$IMAGE_SOURCE/build/Dockerfile" \
+ -f "$BUILD_DIR/$IMAGE_SOURCE/Dockerfile" \
"$BUILD_DIR"
-
-echo "Copying clang installation to release image sources"
-docker run -v "$BUILD_DIR/$IMAGE_SOURCE:/workspace" "$DOCKER_REPOSITORY-build$DOCKER_TAG" \
- cp /tmp/clang.tar.gz /workspace/release
-
-echo "Building release image"
-docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \
- "$BUILD_DIR/$IMAGE_SOURCE/release"
-
echo "Done"
Index: llvm/trunk/utils/docker/debian8/Dockerfile
===================================================================
--- llvm/trunk/utils/docker/debian8/Dockerfile
+++ llvm/trunk/utils/docker/debian8/Dockerfile
@@ -0,0 +1,58 @@
+#===- llvm/utils/docker/debian8/build/Dockerfile -------------------------===//
+#
+# The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License. See LICENSE.TXT for details.
+#
+#===----------------------------------------------------------------------===//
+# Stage 1. Check out LLVM source code and run the build.
+FROM launcher.gcr.io/google/debian8:latest as builder
+LABEL maintainer "LLVM Developers"
+# Install build dependencies of llvm.
+# First, Update the apt's source list and include the sources of the packages.
+RUN grep deb /etc/apt/sources.list | \
+ sed 's/^deb/deb-src /g' >> /etc/apt/sources.list
+# Install compiler, python and subversion.
+RUN apt-get update && \
+ apt-get install -y --no-install-recommends ca-certificates gnupg \
+ build-essential python wget subversion unzip && \
+ rm -rf /var/lib/apt/lists/*
+# Install a newer ninja release. It seems the older version in the debian repos
+# randomly crashes when compiling llvm.
+RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \
+ echo "d2fea9ff33b3ef353161ed906f260d565ca55b8ca0568fa07b1d2cab90a84a07 ninja-linux.zip" \
+ | sha256sum -c && \
+ unzip ninja-linux.zip -d /usr/local/bin && \
+ rm ninja-linux.zip
+# Import public key required for verifying signature of cmake download.
+RUN gpg --keyserver hkp://pgp.mit.edu --recv 0x2D2CEF1034921684
+# Download, verify and install cmake version that can compile clang into /usr/local.
+# (Version in debian8 repos is is too old)
+RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \
+ wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt.asc" && \
+ wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt" && \
+ gpg --verify cmake-3.7.2-SHA-256.txt.asc cmake-3.7.2-SHA-256.txt && \
+ wget "https://cmake.org/files/v3.7/cmake-3.7.2-Linux-x86_64.tar.gz" && \
+ ( grep "cmake-3.7.2-Linux-x86_64.tar.gz" cmake-3.7.2-SHA-256.txt | \
+ sha256sum -c - ) && \
+ tar xzf cmake-3.7.2-Linux-x86_64.tar.gz -C /usr/local --strip-components=1 && \
+ cd / && rm -rf /tmp/cmake-install
+
+ADD checksums /tmp/checksums
+ADD scripts /tmp/scripts
+# Arguments passed to build_install_clang.sh.
+ARG buildscript_args
+# Run the build. Results of the build will be available at /tmp/clang-install/.
+RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
+
+
+# Stage 2. Produce a minimal release image with build results.
+FROM launcher.gcr.io/google/debian8:latest
+LABEL maintainer "LLVM Developers"
+# Install packages for minimal useful image.
+RUN apt-get update && \
+ apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \
+ rm -rf /var/lib/apt/lists/*
+# Copy build results of stage 1 to /usr/local.
+COPY --from=builder /tmp/clang-install/ /usr/local/
Index: llvm/trunk/utils/docker/debian8/build/Dockerfile
===================================================================
--- llvm/trunk/utils/docker/debian8/build/Dockerfile
+++ llvm/trunk/utils/docker/debian8/build/Dockerfile
@@ -1,55 +0,0 @@
-#===- llvm/utils/docker/debian8/build/Dockerfile -------------------------===//
-#
-# The LLVM Compiler Infrastructure
-#
-# This file is distributed under the University of Illinois Open Source
-# License. See LICENSE.TXT for details.
-#
-#===----------------------------------------------------------------------===//
-# Produces an image that compiles and archives clang, based on debian8.
-FROM launcher.gcr.io/google/debian8:latest
-
-LABEL maintainer "LLVM Developers"
-
-# Install build dependencies of llvm.
-# First, Update the apt's source list and include the sources of the packages.
-RUN grep deb /etc/apt/sources.list | \
- sed 's/^deb/deb-src /g' >> /etc/apt/sources.list
-
-# Install compiler, python and subversion.
-RUN apt-get update && \
- apt-get install -y --no-install-recommends ca-certificates gnupg \
- build-essential python wget subversion unzip && \
- rm -rf /var/lib/apt/lists/*
-
-# Install a newer ninja release. It seems the older version in the debian repos
-# randomly crashes when compiling llvm.
-RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \
- echo "d2fea9ff33b3ef353161ed906f260d565ca55b8ca0568fa07b1d2cab90a84a07 ninja-linux.zip" \
- | sha256sum -c && \
- unzip ninja-linux.zip -d /usr/local/bin && \
- rm ninja-linux.zip
-
-# Import public key required for verifying signature of cmake download.
-RUN gpg --keyserver hkp://pgp.mit.edu --recv 0x2D2CEF1034921684
-
-# Download, verify and install cmake version that can compile clang into /usr/local.
-# (Version in debian8 repos is is too old)
-RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \
- wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt.asc" && \
- wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt" && \
- gpg --verify cmake-3.7.2-SHA-256.txt.asc cmake-3.7.2-SHA-256.txt && \
- wget "https://cmake.org/files/v3.7/cmake-3.7.2-Linux-x86_64.tar.gz" && \
- ( grep "cmake-3.7.2-Linux-x86_64.tar.gz" cmake-3.7.2-SHA-256.txt | \
- sha256sum -c - ) && \
- tar xzf cmake-3.7.2-Linux-x86_64.tar.gz -C /usr/local --strip-components=1 && \
- cd / && rm -rf /tmp/cmake-install
-
-ADD checksums /tmp/checksums
-ADD scripts /tmp/scripts
-
-# Arguments passed to build_install_clang.sh.
-ARG buildscript_args
-
-# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
-RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
Index: llvm/trunk/utils/docker/debian8/release/Dockerfile
===================================================================
--- llvm/trunk/utils/docker/debian8/release/Dockerfile
+++ llvm/trunk/utils/docker/debian8/release/Dockerfile
@@ -1,21 +0,0 @@
-#===- llvm/utils/docker/debian8/release/Dockerfile -----------------------===//
-#
-# The LLVM Compiler Infrastructure
-#
-# This file is distributed under the University of Illinois Open Source
-# License. See LICENSE.TXT for details.
-#
-#===----------------------------------------------------------------------===//
-# A release image, containing clang installation, produced by the 'build/' image
-# and adding libstdc++ and binutils.
-FROM launcher.gcr.io/google/debian8:latest
-
-LABEL maintainer "LLVM Developers"
-
-# Install packages for minimal useful image.
-RUN apt-get update && \
- apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \
- rm -rf /var/lib/apt/lists/*
-
-# Unpack clang installation into this image.
-ADD clang.tar.gz /usr/local/
Index: llvm/trunk/utils/docker/example/Dockerfile
===================================================================
--- llvm/trunk/utils/docker/example/Dockerfile
+++ llvm/trunk/utils/docker/example/Dockerfile
@@ -0,0 +1,37 @@
+#===- llvm/utils/docker/example/build/Dockerfile -------------------------===//
+#
+# The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License. See LICENSE.TXT for details.
+#
+#===----------------------------------------------------------------------===//
+# This is an example Dockerfile to build an image that compiles clang.
+# Replace FIXMEs to prepare your own image.
+
+# Stage 1. Check out LLVM source code and run the build.
+# FIXME: Replace 'ubuntu' with your base image
+FROM ubuntu as builder
+# FIXME: Change maintainer name
+LABEL maintainer "Maintainer "
+# FIXME: Install llvm/clang build dependencies here. Including compiler to
+# build stage1, cmake, subversion, ninja, etc.
+
+ADD checksums /tmp/checksums
+ADD scripts /tmp/scripts
+# Arguments passed to build_install_clang.sh.
+ARG buildscript_args
+# Run the build. Results of the build will be available as /tmp/clang-install.
+RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
+
+
+# Stage 2. Produce a minimal release image with build results.
+# FIXME: Replace 'ubuntu' with your base image.
+FROM ubuntu
+# FIXME: Change maintainer name.
+LABEL maintainer "Maintainer "
+# FIXME: Install all packages you want to have in your release container.
+# A minimal useful installation should include at least libstdc++ and binutils.
+
+# Copy build results of stage 1 to /usr/local.
+COPY --from=builder /tmp/clang-install/ /usr/local/
Index: llvm/trunk/utils/docker/example/build/Dockerfile
===================================================================
--- llvm/trunk/utils/docker/example/build/Dockerfile
+++ llvm/trunk/utils/docker/example/build/Dockerfile
@@ -1,28 +0,0 @@
-#===- llvm/utils/docker/example/build/Dockerfile -------------------------===//
-#
-# The LLVM Compiler Infrastructure
-#
-# This file is distributed under the University of Illinois Open Source
-# License. See LICENSE.TXT for details.
-#
-#===----------------------------------------------------------------------===//
-# This is an example Dockerfile to build an image that compiles clang.
-# Replace FIXMEs to prepare your own image.
-
-# FIXME: Replace 'ubuntu' with your base image
-FROM ubuntu
-
-# FIXME: Change maintainer name
-LABEL maintainer "Maintainer "
-
-# FIXME: Install llvm/clang build dependencies. Including compiler to
-# build stage1, cmake, subversion, ninja, etc.
-
-ADD checksums /tmp/checksums
-ADD scripts /tmp/scripts
-
-# Arguments passed to build_install_clang.sh.
-ARG buildscript_args
-
-# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
-RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
Index: llvm/trunk/utils/docker/example/release/Dockerfile
===================================================================
--- llvm/trunk/utils/docker/example/release/Dockerfile
+++ llvm/trunk/utils/docker/example/release/Dockerfile
@@ -1,24 +0,0 @@
-#===- llvm/utils/docker/example/release/Dockerfile -----------------------===//
-#
-# The LLVM Compiler Infrastructure
-#
-# This file is distributed under the University of Illinois Open Source
-# License. See LICENSE.TXT for details.
-#
-#===----------------------------------------------------------------------===//
-# An image that unpacks a clang installation, compiled by the 'build/'
-# container.
-# Replace FIXMEs to prepare your own image.
-
-# FIXME: Replace 'ubuntu' with your base image.
-FROM ubuntu
-
-# FIXME: Change maintainer name.
-LABEL maintainer "Maintainer "
-
-# FIXME: Install all packages you want to have in your release container.
-# A minimal useful installation must include libstdc++ and binutils.
-
-# Unpack clang installation into this container.
-# It is copied to this directory by build_docker_image.sh script.
-ADD clang.tar.gz /usr/local/
Index: llvm/trunk/utils/docker/nvidia-cuda/Dockerfile
===================================================================
--- llvm/trunk/utils/docker/nvidia-cuda/Dockerfile
+++ llvm/trunk/utils/docker/nvidia-cuda/Dockerfile
@@ -0,0 +1,31 @@
+#===- llvm/utils/docker/nvidia-cuda/build/Dockerfile ---------------------===//
+#
+# The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License. See LICENSE.TXT for details.
+#
+#===----------------------------------------------------------------------===//
+# Stage 1. Check out LLVM source code and run the build.
+FROM nvidia/cuda:8.0-devel as builder
+LABEL maintainer "LLVM Developers"
+# Install llvm build dependencies.
+RUN apt-get update && \
+ apt-get install -y --no-install-recommends ca-certificates cmake python \
+ subversion ninja-build && \
+ rm -rf /var/lib/apt/lists/*
+
+ADD checksums /tmp/checksums
+ADD scripts /tmp/scripts
+# Arguments passed to build_install_clang.sh.
+ARG buildscript_args
+# Run the build. Results of the build will be available at /tmp/clang-install/.
+RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
+
+
+# Stage 2. Produce a minimal release image with build results.
+FROM nvidia/cuda:8.0-devel
+LABEL maintainer "LLVM Developers"
+# Copy clang installation into this container.
+COPY --from=builder /tmp/clang-install/ /usr/local/
+# C++ standard library and binutils are already included in the base package.
Index: llvm/trunk/utils/docker/nvidia-cuda/build/Dockerfile
===================================================================
--- llvm/trunk/utils/docker/nvidia-cuda/build/Dockerfile
+++ llvm/trunk/utils/docker/nvidia-cuda/build/Dockerfile
@@ -1,31 +0,0 @@
-#===- llvm/utils/docker/nvidia-cuda/build/Dockerfile ---------------------===//
-#
-# The LLVM Compiler Infrastructure
-#
-# This file is distributed under the University of Illinois Open Source
-# License. See LICENSE.TXT for details.
-#
-#===----------------------------------------------------------------------===//
-# Produces an image that compiles and archives clang, based on nvidia/cuda
-# image.
-FROM nvidia/cuda:8.0-devel
-
-LABEL maintainer "LLVM Developers"
-
-# Arguments to pass to build_install_clang.sh.
-ARG buildscript_args
-
-# Install llvm build dependencies.
-RUN apt-get update && \
- apt-get install -y --no-install-recommends ca-certificates cmake python \
- subversion ninja-build && \
- rm -rf /var/lib/apt/lists/*
-
-ADD checksums /tmp/checksums
-ADD scripts /tmp/scripts
-
-# Arguments passed to build_install_clang.sh.
-ARG buildscript_args
-
-# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
-RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
Index: llvm/trunk/utils/docker/nvidia-cuda/release/Dockerfile
===================================================================
--- llvm/trunk/utils/docker/nvidia-cuda/release/Dockerfile
+++ llvm/trunk/utils/docker/nvidia-cuda/release/Dockerfile
@@ -1,23 +0,0 @@
-#===- llvm/utils/docker/nvidia-cuda/release/Dockerfile -------------------===//
-#
-# The LLVM Compiler Infrastructure
-#
-# This file is distributed under the University of Illinois Open Source
-# License. See LICENSE.TXT for details.
-#
-#===----------------------------------------------------------------------===//
-# This is an example Dockerfile that copies a clang installation, compiled
-# by the 'build/' container into a fresh docker image to get a container of
-# minimal size.
-# Replace FIXMEs to prepare a new Dockerfile.
-
-# FIXME: Replace 'ubuntu' with your base image.
-FROM nvidia/cuda:8.0-devel
-
-# FIXME: Change maintainer name.
-LABEL maintainer "LLVM Developers"
-
-# Unpack clang installation into this container.
-ADD clang.tar.gz /usr/local/
-
-# C++ standard library and binutils are already included in the base package.
Index: llvm/trunk/utils/docker/scripts/build_install_llvm.sh
===================================================================
--- llvm/trunk/utils/docker/scripts/build_install_llvm.sh
+++ llvm/trunk/utils/docker/scripts/build_install_llvm.sh
@@ -16,8 +16,8 @@
Checkout svn sources and run cmake with the specified arguments. Used
inside docker container.
-Passes additional -DCMAKE_INSTALL_PREFIX and archives the contents of
-the directory to /tmp/clang.tar.gz.
+Passes additional -DCMAKE_INSTALL_PREFIX and puts the build results into
+/tmp/clang-install/ directory.
Available options:
-h|--help show this help message
@@ -244,12 +244,7 @@
popd
-# Pack the installed clang into an archive.
-echo "Archiving clang installation to /tmp/clang.tar.gz"
-cd "$CLANG_INSTALL_DIR"
-tar -czf /tmp/clang.tar.gz *
-
# Cleanup.
-rm -rf "$CLANG_BUILD_DIR" "$CLANG_INSTALL_DIR"
+rm -rf "$CLANG_BUILD_DIR"
echo "Done"