Index: docs/Docker.rst =================================================================== --- /dev/null +++ docs/Docker.rst @@ -0,0 +1,112 @@ +========================================= +A guide to Dockerfiles for building clang +========================================= + +Introduction +============ +You can find a number of sources to build docker images with clang in +`llvm/utils/docker`. They can be used by anyone who wants to build the docker +images for their own use, or as a starting point for someone who wants to write +their own Dockerfiles. + +We currently provide Dockerfiles with `debian8` and `nvidia-cuda` base images. +We also provide an `example` image, which contains placeholders that one would need +to fill out in order to produce Dockerfiles for a new docker image. + +Why? +---- +Docker images provide a way to produce binary distributions of +software inside a controlled environment. Having Dockerfiles to builds docker images +inside LLVM repo makes them much more discoverable than putting them into any other +place. + +Docker basics +------------- +If you've never heard about Docker before, you might find this section helpful +to get a very basic explanation of it. +`Docker `_ is a popular solution for running programs in +an isolated and reproducible environment, especially to maintain releases for +software deployed to large distributed fleets. +It uses linux kernel namespaces and cgroups to provide a lightweight isolation +inside currently running linux kernel. +A single active instance of dockerized environment is called a *docker +container*. +A snapshot of a docker container filesystem is called a *docker image*. +One can start a container from a prebuilt docker image. + +Docker images are built from a so-called *Dockerfile*, a source file written in +a specialized language that defines instructions to be used when build +the docker image (see `official +documentation `_ for more +details). A minimal Dockerfile typically contains a base image and a number +of RUN commands that have to be executed to build the image. When building a new +image, docker will first download your base image, mount its filesystem as +read-only and then add a writable overlay on top of it to keep track of all +filesystem modifications, performed while building your image. When the build +process is finished, a diff between your image's final filesystem state and the +base image's filesystem is stored in the resulting image. + +Overview +======== +The `llvm/utils/docker` folder contains Dockerfiles and simple bash scripts to +serve as a basis for anyone who wants to create their own Docker image with +clang, compiled from sources. The sources are checked out from the upstream svn +repository when building the image. + +Inside each subfolder we host Dockerfiles for two images: + +- `build/` image is used to compile clang, it installs gcc and all build + dependencies of clang. After the build process is done, the build image will + have an archived clang at `/tmp/clang.tar.gz` +- `release/` image usually only contains clang, compiled by the `build/` image, + libstdc++ and binutils to make image minimally useful. + +To build both of those images, use `build_docker_image.sh` script. +It will checkout LLVM sources and build clang in the `build` container, copy results +of the build to the local filesystem and then build the `release` container using +those. + +If you want to write your own docker image, start with an `example/` subfolder. +It provides incomplete Dockerfiles with (very few) FIXMEs explaining the steps +you need to take in order to make your Dockerfiles functional. + +Usage +===== +Use `llvm/utils/build_docker_image.sh` to build docker images: + +.. code-block:: bash + + ./llvm/utils/docker/build_docker_image.sh llvm/utils/docker/debian8 \ + clang-debian8 "staging" + +This will produce two images, a release image `clang-debian8:staging` and a +build image `clang-debian8-build:staging` from the latest upstream revision. + +Stable build +============ +If you want a somewhat recent and somewhat stable build, use the +`branches/google/stable` branch, i.e. the following command will produce a +debian8-based image using the latest `google/stable` sources for you: + +.. code-block:: bash + + ./llvm/utils/docker/build_docker_image.sh llvm/utils/docker/debian8 \ + clang-debian8 "stable" -b branches/google/stable + + +Design explanataion +=================== +Due to Docker restrictions we had to make some non-obvious decisions: + +- The `build_install_clang.sh` is copied between different folders, instead of + being symlinked. This is due to Docker not allowing symlinked files to be + added into resulting images. + +- We use two images (i.e., build and release folders) for the release image to + be as small as possible. It's much easier to achieve that using two images, + because Docker would store a filesystem layer for each command in the + Dockerfile. I.e., if you install some packages in one command, then remove + those in a separate command, the size of the resulting image will still be + proportinal to the size of an image with installed packages. + Therefore, we strive to provide a very simple release image which only copies + compiled clang and does not do anything else. Index: docs/index.rst =================================================================== --- docs/index.rst +++ docs/index.rst @@ -91,6 +91,7 @@ CompileCudaWithLLVM ReportingGuide Benchmarking + Docker :doc:`GettingStarted` Discusses how to get up and running quickly with the LLVM infrastructure. @@ -161,6 +162,9 @@ A collection of tips for frontend authors on how to generate IR which LLVM is able to effectively optimize. +:doc:`Docker` + A reference for using Dockerfiles provided with llvm. + Programming Documentation ========================= Index: utils/docker/README =================================================================== --- /dev/null +++ utils/docker/README @@ -0,0 +1 @@ +See llvm/docs/Docker.rst for details Index: utils/docker/build_docker_image.sh =================================================================== --- /dev/null +++ utils/docker/build_docker_image.sh @@ -0,0 +1,58 @@ +#!/bin/bash +#===- llvm/utils/docker/build_docker_image.sh ----------------------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// +set -e + +if [[ $# -lt 3 ]]; then + echo \ +"Usage $0 [buildscript_arg]..." +echo " For example, running" +echo " $0 debian8 mydocker/debian8-clang latest" +echo " will produce two docker images:" +echo " mydocker/debian8-clang-build:latest - Intermediate image used that" +echo " was used to compile clang." +echo " mydocker/clang-debian8:latest - A small image with preinstalled" +echo " clang." + exit 1 +fi + +command -v docker >/dev/null || + { + echo "Docker binary cannot be found. Please install Docker to use this script." + exit 1 + } + +REPO_PATH="$1" +DOCKER_REPOSITORY="$2" +DOCKER_TAG="$3" +shift 3 + +if [ "$DOCKER_TAG" != "" ]; then + DOCKER_TAG=":$DOCKER_TAG" +fi + +BUILDSCRIPT_ARGS="$@" + +echo "Building from $REPO_PATH" +cd "$REPO_PATH" + +echo "Building $DOCKER_REPOSITORY-build$DOCKER_TAG" +docker build -t "$DOCKER_REPOSITORY-build$DOCKER_TAG" \ + --build-arg "buildscript_args=$*" \ + build + +echo "Copying clang installation to release image sources" +docker run -v "$PWD:/workspace" "$DOCKER_REPOSITORY-build$DOCKER_TAG" \ + cp /tmp/clang.tar.gz /workspace/release +trap "rm -f $PWD/release/clang.tar.gz" EXIT + +echo "Building release image" +docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" release + +echo "Done" Index: utils/docker/debian8/build/Dockerfile =================================================================== --- /dev/null +++ utils/docker/debian8/build/Dockerfile @@ -0,0 +1,35 @@ +#===- llvm/utils/docker/debian8/build/Dockerfile -------------------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// +# Produces an image that compiles and archives clang, based on debian8. +FROM launcher.gcr.io/google/debian8:latest + +LABEL maintainer "LLVM Developers" + +# Install build dependencies of llvm. +# First, Update the apt's source list and include the sources of the packages. +RUN grep deb /etc/apt/sources.list | \ + sed 's/^deb/deb-src /g' >> /etc/apt/sources.list + +# Install compiler, python and subversion. +RUN apt-get update && \ + apt-get install -y --no-install-recommends build-essential python2.7 wget \ + subversion && \ + rm -rf /var/lib/apt/lists/* + +# Install cmake version that can compile clang into /usr/local. +# (Version in debian8 repos is is too old) +RUN wget -O - "https://cmake.org/files/v3.7/cmake-3.7.2-Linux-x86_64.tar.gz" | \ + tar xzf - -C /usr/local --strip-components=1 + +# Arguments passed to build_install_clang.sh. +ARG buildscript_args + +# Run the build. Results of the build will be available as /tmp/clang.tar.gz. +ADD build_install_clang.sh /tmp +RUN /tmp/build_install_clang.sh ${buildscript_args} Index: utils/docker/debian8/build/build_install_clang.sh =================================================================== --- /dev/null +++ utils/docker/debian8/build/build_install_clang.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +#===- llvm/utils/docker/debian8/build/build_install_clang.sh -------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// + +set -e + +function show_usage() { + echo "Usage: $0 [-r ] [-b ] [-h]" + echo "Checkout and build clang for docker container. Results of the build are" + echo "in /tmp/clang.tar.gz." + echo "Available arguments:" + echo " -h show this help message" + echo " -r specific svn revision to checkout" + echo " -b branch to checkout; i.e. 'trunk', 'branches/release_40'" + echo " (default: 'trunk')" + exit 1 +} + +while getopts "rbh" opt; do + case $opt in + r) + LLVM_SVN_REVISION="$OPTARG" + ;; + b) + LLVM_BRANCH="$OPTARG" + ;; + h) + show_usage + exit 0 + esac +done + +if [ "$LLVM_BRANCH" == "" ]; then + LLVM_BRANCH="trunk" +fi + +if [ "$LLVM_SVN_REVISION" != "" ]; then + SVN_REV_ARG="-r$LLVM_SVN_REVISION" +else + SVN_REV_ARG="" +fi + + +CLANG_BUILD_DIR=/tmp/clang-build +CLANG_INSTALL_DIR=/tmp/clang-install + +mkdir "$CLANG_BUILD_DIR" + +pushd "$CLANG_BUILD_DIR" + +# Get the sources from svn +echo "Checking out sources from svn" +svn co -q $SVN_REV_ARG "http://llvm.org/svn/llvm-project/llvm/$LLVM_BRANCH" \ + "$CLANG_BUILD_DIR/llvm" +svn co -q $SVN_REV_ARG "http://llvm.org/svn/llvm-project/cfe/$LLVM_BRANCH" \ + "$CLANG_BUILD_DIR/llvm/tools/clang" + +# Build 1st stage. Compile clang with system compiler. +echo "Running stage 1" +mkdir "$CLANG_BUILD_DIR/build-1" +cd "$CLANG_BUILD_DIR/build-1" +cmake -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release "$CLANG_BUILD_DIR/llvm" +make -j `nproc` clang clang-headers + +# Build 2nd stage. Compile clang with clang built in stage 1. +echo "Running stage 2" +mkdir "$CLANG_BUILD_DIR/build-2" +cd "$CLANG_BUILD_DIR/build-2" + +echo "Installing stage 2" +CC="$CLANG_BUILD_DIR/build-1/bin/clang" \ +CXX="$CLANG_BUILD_DIR/build-1/bin/clang++" \ +cmake -G"Unix Makefiles" \ + -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="$CLANG_INSTALL_DIR" \ + "$CLANG_BUILD_DIR/llvm" + +make -j `nproc` install-clang install-clang-headers + +popd + +# Pack the installed clang into an archive +echo "Archiving clang installation to /tmp/clang.tar.gz" +cd "$CLANG_INSTALL_DIR" +tar -czf /tmp/clang.tar.gz * + +# Cleanup. +rm -rf "$CLANG_BUILD_DIR" "$CLANG_INSTALL_DIR" + +echo "Done" Index: utils/docker/debian8/release/Dockerfile =================================================================== --- /dev/null +++ utils/docker/debian8/release/Dockerfile @@ -0,0 +1,21 @@ +#===- llvm/utils/docker/debian8/release/Dockerfile -----------------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// +# A release image, containing clang installation, produced by the 'build/' image +# and adding libstdc++ and binutils. +FROM launcher.gcr.io/google/debian8:latest + +LABEL maintainer "LLVM Developers" + +# Unpack clang installation into this image. +ADD clang.tar.gz / + +# Install packages for minimal usefull image. +RUN apt-get update && \ + apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \ + rm -rf /var/lib/apt/lists/* Index: utils/docker/example/build/Dockerfile =================================================================== --- /dev/null +++ utils/docker/example/build/Dockerfile @@ -0,0 +1,26 @@ +#===- llvm/docker/example/build/Dockerfile -------------------------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// +# This is an example Dockerfile to build an image that compiles clang. +# Replace FIXMEs to prepare your own image. + +# FIXME: Replace 'ubuntu' with your base image +FROM ubuntu + +# FIXME: Change maintainer name +LABEL maintainer "Maintainer " + +# FIXME: Install llvm/clang build dependencies. Including compiler to +# build stage1, cmake, subversion, etc. + +# Arguments to pass to build_install_clang.sh. +ARG buildscript_args + +# Run the build. Results of the build will be available as /tmp/clang.tar.gz. +ADD build_install_clang.sh /tmp +RUN /tmp/build_install_clang.sh ${buildscript_args} Index: utils/docker/example/build/build_install_clang.sh =================================================================== --- /dev/null +++ utils/docker/example/build/build_install_clang.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +#===- llvm/utils/docker/example/build/build_install_clang.sh -------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// + +set -e + +function show_usage() { + echo "Usage: $0 [-r ] [-b ] [-h]" + echo "Checkout and build clang for docker container. Results of the build are" + echo "in /tmp/clang.tar.gz." + echo "Available arguments:" + echo " -h show this help message" + echo " -r specific svn revision to checkout" + echo " -b branch to checkout; i.e. 'trunk', 'branches/release_40'" + echo " (default: 'trunk')" + exit 1 +} + +while getopts "rbh" opt; do + case $opt in + r) + LLVM_SVN_REVISION="$OPTARG" + ;; + b) + LLVM_BRANCH="$OPTARG" + ;; + h) + show_usage + exit 0 + esac +done + +if [ "$LLVM_BRANCH" == "" ]; then + LLVM_BRANCH="trunk" +fi + +if [ "$LLVM_SVN_REVISION" != "" ]; then + SVN_REV_ARG="-r$LLVM_SVN_REVISION" +else + SVN_REV_ARG="" +fi + + +CLANG_BUILD_DIR=/tmp/clang-build +CLANG_INSTALL_DIR=/tmp/clang-install + +mkdir "$CLANG_BUILD_DIR" + +pushd "$CLANG_BUILD_DIR" + +# Get the sources from svn +echo "Checking out sources from svn" +svn co -q $SVN_REV_ARG "http://llvm.org/svn/llvm-project/llvm/$LLVM_BRANCH" \ + "$CLANG_BUILD_DIR/llvm" +svn co -q $SVN_REV_ARG "http://llvm.org/svn/llvm-project/cfe/$LLVM_BRANCH" \ + "$CLANG_BUILD_DIR/llvm/tools/clang" + +# Build 1st stage. Compile clang with system compiler. +echo "Running stage 1" +mkdir "$CLANG_BUILD_DIR/build-1" +cd "$CLANG_BUILD_DIR/build-1" +cmake -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release "$CLANG_BUILD_DIR/llvm" +make -j `nproc` clang clang-headers + +# Build 2nd stage. Compile clang with clang built in stage 1. +echo "Running stage 2" +mkdir "$CLANG_BUILD_DIR/build-2" +cd "$CLANG_BUILD_DIR/build-2" + +echo "Installing stage 2" +CC="$CLANG_BUILD_DIR/build-1/bin/clang" \ +CXX="$CLANG_BUILD_DIR/build-1/bin/clang++" \ +cmake -G"Unix Makefiles" \ + -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="$CLANG_INSTALL_DIR" \ + "$CLANG_BUILD_DIR/llvm" + +make -j `nproc` install-clang install-clang-headers + +popd + +# Pack the installed clang into an archive +echo "Archiving clang installation to /tmp/clang.tar.gz" +cd "$CLANG_INSTALL_DIR" +tar -czf /tmp/clang.tar.gz * + +# Cleanup. +rm -rf "$CLANG_BUILD_DIR" "$CLANG_INSTALL_DIR" + +echo "Done" Index: utils/docker/example/release/Dockerfile =================================================================== --- /dev/null +++ utils/docker/example/release/Dockerfile @@ -0,0 +1,24 @@ +#===- llvm/docker/example/release/Dockerfile -----------------------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// +# An image that unpacks a clang installation, compiled by the 'build/' +# container. +# Replace FIXMEs to prepare your own image. + +# FIXME: Replace 'ubuntu' with your base image. +FROM ubuntu + +# FIXME: Change maintainer name. +LABEL maintainer "Maintainer " + +# Unpack clang installation into this container. +# It is copied to this directory by build_docker_image.sh script. +ADD clang.tar.gz / + +# FIXME: Install all other packages you want to have in your release container. +# A minimal usefull installation must include libstdc++ and binutils. Index: utils/docker/nvidia-cuda/build/Dockerfile =================================================================== --- /dev/null +++ utils/docker/nvidia-cuda/build/Dockerfile @@ -0,0 +1,26 @@ +#===- llvm/utils/docker/nvidia-cuda/build/Dockerfile ---------------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// +# Produces an image that compiles and archives clang, based on nvidia/cuda +# image. +FROM nvidia/cuda:8.0-devel + +# FIXME: Change maintainer name +LABEL maintainer "LLVM Developers" + +# Arguments to pass to build_install_clang.sh. +ARG buildscript_args + +# Install llvm build dependencies. +RUN apt-get update && \ + apt-get install -y --no-install-recommends cmake python2.7 subversion && \ + rm -rf /var/lib/apt/lists/* + +# Run the build. Results of the build will be available as /tmp/clang.tar.gz. +ADD build_install_clang.sh /tmp +RUN /tmp/build_install_clang.sh ${buildscript_args} Index: utils/docker/nvidia-cuda/build/build_install_clang.sh =================================================================== --- /dev/null +++ utils/docker/nvidia-cuda/build/build_install_clang.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +#===- llvm/utils/docker/nvidia-cuda/build/build_install_clang.sh ---------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// + +set -e + +function show_usage() { + echo "Usage: $0 [-r ] [-b ] [-h]" + echo "Checkout and build clang for docker container. Results of the build are" + echo "in /tmp/clang.tar.gz." + echo "Available arguments:" + echo " -h show this help message" + echo " -r specific svn revision to checkout" + echo " -b branch to checkout; i.e. 'trunk', 'branches/release_40'" + echo " (default: 'trunk')" + exit 1 +} + +while getopts "rbh" opt; do + case $opt in + r) + LLVM_SVN_REVISION="$OPTARG" + ;; + b) + LLVM_BRANCH="$OPTARG" + ;; + h) + show_usage + exit 0 + esac +done + +if [ "$LLVM_BRANCH" == "" ]; then + LLVM_BRANCH="trunk" +fi + +if [ "$LLVM_SVN_REVISION" != "" ]; then + SVN_REV_ARG="-r$LLVM_SVN_REVISION" +else + SVN_REV_ARG="" +fi + + +CLANG_BUILD_DIR=/tmp/clang-build +CLANG_INSTALL_DIR=/tmp/clang-install + +mkdir "$CLANG_BUILD_DIR" + +pushd "$CLANG_BUILD_DIR" + +# Get the sources from svn +echo "Checking out sources from svn" +svn co -q $SVN_REV_ARG "http://llvm.org/svn/llvm-project/llvm/$LLVM_BRANCH" \ + "$CLANG_BUILD_DIR/llvm" +svn co -q $SVN_REV_ARG "http://llvm.org/svn/llvm-project/cfe/$LLVM_BRANCH" \ + "$CLANG_BUILD_DIR/llvm/tools/clang" + +# Build 1st stage. Compile clang with system compiler. +echo "Running stage 1" +mkdir "$CLANG_BUILD_DIR/build-1" +cd "$CLANG_BUILD_DIR/build-1" +cmake -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release "$CLANG_BUILD_DIR/llvm" +make -j `nproc` clang clang-headers + +# Build 2nd stage. Compile clang with clang built in stage 1. +echo "Running stage 2" +mkdir "$CLANG_BUILD_DIR/build-2" +cd "$CLANG_BUILD_DIR/build-2" + +echo "Installing stage 2" +CC="$CLANG_BUILD_DIR/build-1/bin/clang" \ +CXX="$CLANG_BUILD_DIR/build-1/bin/clang++" \ +cmake -G"Unix Makefiles" \ + -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="$CLANG_INSTALL_DIR" \ + "$CLANG_BUILD_DIR/llvm" + +make -j `nproc` install-clang install-clang-headers + +popd + +# Pack the installed clang into an archive +echo "Archiving clang installation to /tmp/clang.tar.gz" +cd "$CLANG_INSTALL_DIR" +tar -czf /tmp/clang.tar.gz * + +# Cleanup. +rm -rf "$CLANG_BUILD_DIR" "$CLANG_INSTALL_DIR" + +echo "Done" Index: utils/docker/nvidia-cuda/release/Dockerfile =================================================================== --- /dev/null +++ utils/docker/nvidia-cuda/release/Dockerfile @@ -0,0 +1,23 @@ +#===- llvm/utils/docker/nvidia-cuda/release/Dockerfile -------------------===// +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===----------------------------------------------------------------------===// +# This is an example Dockerfile that copies a clang installation, compiled +# by the 'build/' container into a fresh docker image to get a container of +# minimal size. +# Replace FIXMEs to prepare a new Dockerfile. + +# FIXME: Replace 'ubuntu' with your base image. +FROM nvidia/cuda:8.0-devel + +# FIXME: Change maintainer name. +LABEL maintainer "LLVM Developers" + +# Unpack clang installation into this container. +ADD clang.tar.gz / + +# C++ standard library and binutils are already included in the base package.