[Dockerfile] Upgrade debian base image to version 10

[XiaodongLoong]

Summary

Debian 8 is too old to build LLVM project, the version
of GCC, CMake and python are lower than the requirements:
https://llvm.org/docs/GettingStarted.html#software

Debian 10 is the earliest release that have software
packages that compiling LLVM requires.

[XiaodongLoong]

@MaskRay Thanks!

[XiaodongLoong]

A gentle ping.

[thieta]

I have no problem with this - but I am not sure if these images are used by anyone for something specific. I would hope that anyone using these images in production have their own copy of them instead.

I wonder if the more pragmatic course of action would be to keep the debian8 dockerfile and just add a warning somewhere that notes that it's not really going to work and people should migrate to the debian11 one instead.

[sammccall]

Thanks!

I know there were google-internal users of this, I'm trying to find out if moving from 8 to 11 is okay for them.

There might be a reason to keep the minimal version that works (e.g. 9 or 10 as opposed to the "current" debian stable version). An older build environment generally produces more portable binaries.
I think we shouldn't keep multiple versions around if the docker8 version isn't useful/used, it just creates maintenance burden.

[sammccall]

Do you happen to know which tools specifically are too old in debian8, and/or if 11 is the earliest release that satisfies them?

[sammccall]

In D120826#3389122, @sammccall wrote:

Do you happen to know which tools specifically are too old in debian8, and/or if 11 is the earliest release that satisfies them?

Sorry, I wasn't reading carefully enough.

Requirements are currently CMake 3.13.4, GCC 5.1.0
debian8 provides CMake 3.6.2 (backports), GCC 4.9.2
debian9 provides CMake 3.16.3 (backports), GCC 6.3.0

So I think switching to debian9 should work and may be preferable to 11 for producing portable binaries.
Do you have a reason to prefer the newer version? (I'm still going to find out if 11 is acceptable)

[XiaodongLoong]

@sammccall @thieta
The python version of debian9 is 3.5, which can not meet the current (>=3.6) requirements.

Originally I chose debain11, because I thought it might meet the requirements of compiling LLVM for a long time.

The earliest release that satisfies the requirements is debian10. I can upgrade the base image to debian10, too.

Thanks!

[sammccall]

In D120826#3392144, @XiaodongLoong wrote:

@sammccall @thieta
The python version of debian9 is 3.5, which can not meet the current (>=3.6) requirements.

3.16 for debian9 is available in stretch-backports, will that work for this purpose?
If not debian10 does sound like the best we can do.

Originally I chose debain11, because I thought it might meet the requirements of compiling LLVM for a long time.

I think it's worthwhile to upgrade to the minimum required version, at the cost of upgrading in more frequent smaller steps.

[XiaodongLoong]

In D120826#3399374, @sammccall wrote:

3.16 for debian9 is available in stretch-backports, will that work for this purpose?
If not debian10 does sound like the best we can do.

I add the following item in sources.list.

deb http://deb.debian.org/debian stretch-backports main

Then I run the command blow and get the python version.

$ apt-get update && apt list python3
...
python3/oldoldstable 3.5.3-1 amd64

So I think python 3.6 for debian9 is not available in stretch-backports.

In the view of this, debian10 is the best version we can do. Then I will update the patch.

[XiaodongLoong]

Update patch that upgrade debian to version 10.

[sammccall]

Thank you!

In D120826#3402209, @XiaodongLoong wrote:

In D120826#3399374, @sammccall wrote:

3.16 for debian9 is available in stretch-backports, will that work for this purpose?
If not debian10 does sound like the best we can do.

I add the following item in sources.list.

deb http://deb.debian.org/debian stretch-backports main

Then I run the command blow and get the python version.

$ apt-get update && apt list python3
...
python3/oldoldstable 3.5.3-1 amd64

So I think python 3.6 for debian9 is not available in stretch-backports.

You're absolutely right, I'm sorry. I was confusing python and cmake.
Debian 10 sounds great, thank you.

llvm/docs/Docker.rst

135

nit: Debian10

[benshi001]

llvm/docs/Docker.rst

135

I will help him commit, and correct it to Debian10.

llvm/docs/Docker.rst

 =========================================
 A guide to Dockerfiles for building LLVM
 =========================================
 
 Introduction
 ============
 You can find a number of sources to build docker images with LLVM components in
 ``llvm/utils/docker``. They can be used by anyone who wants to build the docker
 images for their own use, or as a starting point for someone who wants to write
 their own Dockerfiles.
 
-We currently provide Dockerfiles with ``debian8`` and ``nvidia-cuda`` base images.
+We currently provide Dockerfiles with ``debian10`` and ``nvidia-cuda`` base images.
 We also provide an ``example`` image, which contains placeholders that one would need
 to fill out in order to produce Dockerfiles for a new docker image.
 
 Why?
 ----
 Docker images provide a way to produce binary distributions of
 software inside a controlled environment. Having Dockerfiles to builds docker images
 inside LLVM repo makes them much more discoverable than putting them into any other

 Usage
 =====
 The ``llvm/utils/build_docker_image.sh`` script provides a rather high degree of
 control on how to run the build. It allows you to specify the projects to
 checkout from git and provide a list of CMake arguments to use during when
 building LLVM inside docker container.
 
 Here's a very simple example of getting a docker image with clang binary,
-compiled by the system compiler in the debian8 image:
+compiled by the system compiler in the debian10 image:
 
 .. code-block:: bash
 
     ./llvm/utils/docker/build_docker_image.sh \
-  --source debian8 \
-  --docker-repository clang-debian8 --docker-tag "staging" \
+  --source debian10 \
+  --docker-repository clang-debian10 --docker-tag "staging" \
   -p clang -i install-clang -i install-clang-resource-headers \
   -- \
   -DCMAKE_BUILD_TYPE=Release
 
 Note that a build like that doesn't use a 2-stage build process that
 you probably want for clang. Running a 2-stage build is a little more intricate,
 this command will do that:
 
 .. code-block:: bash
 
     # Run a 2-stage build.
     #   LLVM_TARGETS_TO_BUILD=Native is to reduce stage1 compile time.
     #   Options, starting with BOOTSTRAP_* are passed to stage2 cmake invocation.
     ./build_docker_image.sh \
-  --source debian8 \
-  --docker-repository clang-debian8 --docker-tag "staging" \
+  --source debian10 \
+  --docker-repository clang-debian10 --docker-tag "staging" \
   -p clang -i stage2-install-clang -i stage2-install-clang-resource-headers \
   -- \
   -DLLVM_TARGETS_TO_BUILD=Native -DCMAKE_BUILD_TYPE=Release \
   -DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \
   -DCLANG_ENABLE_BOOTSTRAP=ON -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-resource-headers"
   
-This will produce a new image ``clang-debian8:staging`` from the latest
+This will produce a new image ``clang-debian10:staging`` from the latest
 upstream revision.
 After the image is built you can run bash inside a container based on your image
 like this:
 
 .. code-block:: bash
 
-    docker run -ti clang-debian8:staging bash
+    docker run -ti clang-debian10:staging bash
 
 Now you can run bash commands as you normally would:
 
 .. code-block:: bash
 
     root@80f351b51825:/# clang -v
     clang version 5.0.0 (trunk 305064)
     Target: x86_64-unknown-linux-gnu
     Thread model: posix
     InstalledDir: /bin
     Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
     Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4
     Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
     Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.2
     Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
     Candidate multilib: .;@m64
     Selected multilib: .;@m64
 
 
 Which image should I choose?
 ============================
-We currently provide two images: debian8-based and nvidia-cuda-based. They
+We currently provide two images: Debian10-based and nvidia-cuda-based. They
 differ in the base image that they use, i.e. they have a different set of
 preinstalled binaries. Debian8 is very minimal, nvidia-cuda is larger, but has
 preinstalled CUDA libraries and allows to access a GPU, installed on your
 machine.
 
 If you need a minimal linux distribution with only clang and libstdc++ included,
-you should try debian8-based image.
+you should try Debian10-based image.
 
 If you want to use CUDA libraries and have access to a GPU on your machine,
 you should choose nvidia-cuda-based image and use `nvidia-docker
 <https://github.com/NVIDIA/nvidia-docker>`_ to run your docker containers. Note
 that you don't need nvidia-docker to build the images, but you need it in order
 to have an access to GPU from a docker container that is running the built
 image.
 
 If you have a different use-case, you could create your own image based on
 ``example/`` folder.
 
 Any docker image can be built and run using only the docker binary, i.e. you can
-run debian8 build on Fedora or any other Linux distribution. You don't need to
+run debian10 build on Fedora or any other Linux distribution. You don't need to
 install CMake, compilers or any other clang dependencies. It is all handled
 during the build process inside Docker's isolated environment.
 
 Stable build
 ============
 If you want a somewhat recent and somewhat stable build, use the
 ``branches/google/stable`` branch, i.e. the following command will produce a
-debian8-based image using the latest ``google/stable`` sources for you:
+Debian10-based image using the latest ``google/stable`` sources for you:
 
 .. code-block:: bash
 
     ./llvm/utils/docker/build_docker_image.sh \
-  -s debian8 --d clang-debian8 -t "staging" \
+  -s debian10 --d clang-debian10 -t "staging" \
   --branch branches/google/stable \
   -p clang -i install-clang -i install-clang-resource-headers \
   -- \
   -DCMAKE_BUILD_TYPE=Release
 
 
 Minimizing docker image size
 ============================

llvm/utils/docker/build_docker_image.sh

 function show_usage() {
   cat << EOF
 Usage: build_docker_image.sh [options] [-- [cmake_args]...]
 
 Available options:
   General:
     -h|--help               show this help message
   Docker-specific:
-    -s|--source             image source dir (i.e. debian8, nvidia-cuda, etc)
+    -s|--source             image source dir (i.e. debian10, nvidia-cuda, etc)
     -d|--docker-repository  docker repository for the image
     -t|--docker-tag         docker tag for the image
   Checkout arguments:
     -b|--branch         svn branch to checkout, i.e. 'trunk',
                         'branches/release_40'
                         (default: 'trunk')
     -r|--revision       svn revision to checkout
     -c|--cherrypick     revision to cherry-pick. Can be specified multiple times.
-i|--install-target name of a cmake install target to build and include in
                         the resulting archive. Can be specified multiple times.
 
 Required options: --source and --docker-repository, at least one
   --install-target.
 
 All options after '--' are passed to CMake invocation.
 
 For example, running:
-$ build_docker_image.sh -s debian8 -d mydocker/debian8-clang -t latest \ 
+$ build_docker_image.sh -s debian10 -d mydocker/debian10-clang -t latest \
   -p clang -i install-clang -i install-clang-resource-headers
 will produce two docker images:
-    mydocker/debian8-clang-build:latest - an intermediate image used to compile
+    mydocker/debian10-clang-build:latest - an intermediate image used to compile
       clang.
-    mydocker/clang-debian8:latest       - a small image with preinstalled clang.
+    mydocker/clang-debian10:latest       - a small image with preinstalled clang.
 Please note that this example produces a not very useful installation, since it
 doesn't override CMake defaults, which produces a Debug and non-boostrapped
 version of clang.
 
 To get a 2-stage clang build, you could use this command:
-$ ./build_docker_image.sh -s debian8 -d mydocker/clang-debian8 -t "latest" \ 
+$ ./build_docker_image.sh -s debian10 -d mydocker/clang-debian10 -t "latest" \
     -p clang -i stage2-install-clang -i stage2-install-clang-resource-headers \ 
     -- \ 
     -DLLVM_TARGETS_TO_BUILD=Native -DCMAKE_BUILD_TYPE=Release \ 
     -DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \ 
     -DCLANG_ENABLE_BOOTSTRAP=ON \ 
     -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-resource-headers"
 EOF
 }

llvm/utils/docker/debian10/Dockerfile

-#===- llvm/utils/docker/debian8/build/Dockerfile -------------------------===//
+#===- llvm/utils/docker/debian10/build/Dockerfile -------------------------===//
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===//
 # Stage 1. Check out LLVM source code and run the build.
-FROM launcher.gcr.io/google/debian8:latest as builder
+FROM launcher.gcr.io/google/debian10:latest as builder
 LABEL maintainer "LLVM Developers"
 # Install build dependencies of llvm.
 # First, Update the apt's source list and include the sources of the packages.
 RUN grep deb /etc/apt/sources.list | \
     sed 's/^deb/deb-src /g' >> /etc/apt/sources.list
 # Install compiler, python and subversion.
 RUN apt-get update && \
     apt-get install -y --no-install-recommends ca-certificates gnupg \
-           build-essential python wget subversion unzip && \
+           build-essential cmake make python3 zlib1g wget subversion unzip && \
     rm -rf /var/lib/apt/lists/*
 # Install a newer ninja release. It seems the older version in the debian repos
 # randomly crashes when compiling llvm.
 RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \
     echo "d2fea9ff33b3ef353161ed906f260d565ca55b8ca0568fa07b1d2cab90a84a07 ninja-linux.zip" \
         | sha256sum -c  && \
     unzip ninja-linux.zip -d /usr/local/bin && \
     rm ninja-linux.zip
-# Import public key required for verifying signature of cmake download.
-RUN gpg --keyserver hkp://pgp.mit.edu --recv 0x2D2CEF1034921684
-# Download, verify and install cmake version that can compile clang into /usr/local.
-# (Version in debian8 repos is is too old)
-RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \
-    wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt.asc" && \
-    wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt" && \
-    gpg --verify cmake-3.7.2-SHA-256.txt.asc cmake-3.7.2-SHA-256.txt && \
-    wget "https://cmake.org/files/v3.7/cmake-3.7.2-Linux-x86_64.tar.gz" && \
-    ( grep "cmake-3.7.2-Linux-x86_64.tar.gz" cmake-3.7.2-SHA-256.txt | \
-      sha256sum -c - ) && \
-    tar xzf cmake-3.7.2-Linux-x86_64.tar.gz -C /usr/local --strip-components=1 && \
-    cd / && rm -rf /tmp/cmake-install
 
 ADD checksums /tmp/checksums
 ADD scripts /tmp/scripts
 
 # Checkout the source code.
 ARG checkout_args
 RUN /tmp/scripts/checkout.sh ${checkout_args}
 # Run the build. Results of the build will be available at /tmp/clang-install/.
 ARG buildscript_args
 RUN /tmp/scripts/build_install_llvm.sh --to /tmp/clang-install ${buildscript_args}
 
 
 # Stage 2. Produce a minimal release image with build results.
-FROM launcher.gcr.io/google/debian8:latest
+FROM launcher.gcr.io/google/debian10:latest
 LABEL maintainer "LLVM Developers"
 # Install packages for minimal useful image.
 RUN apt-get update && \
-    apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \
+    apt-get install -y --no-install-recommends libstdc++-7-dev binutils && \
     rm -rf /var/lib/apt/lists/*
 # Copy build results of stage 1 to /usr/local.
 COPY --from=builder /tmp/clang-install/ /usr/local/