Allow to apply cherry-picks when building Docker images.

[ioeric]

utils/docker/scripts/build_install_llvm.sh

84

Do we want to sort the "cherries" in case they are not ordered? Does the order matter?

[ilya-biryukov]

• Sort cherrypicks before applying them.

[ilya-biryukov]

utils/docker/scripts/build_install_llvm.sh

84

Good point. Applying them in the sorted order is the most natural choice.
Now sorting cherrypicks and removing duplicates before applying them.

[ioeric]

utils/docker/scripts/build_install_llvm.sh

167

I'd do the sorting in apply_cherrypicks.

208

IIUC,$CHERRYPICKS is a list of cherries for all llvm sub-projects, and we try applying them on each project, even if some of them might not apply for certain projects. Is this safe with svn? Might worth a comment.

[ilya-biryukov]

utils/docker/scripts/build_install_llvm.sh

167

Why not here? apply_cherrypicks is called multiple times, sorting outside of it seems natural.
Maybe we should define apply_cherrypicks after we do the sorting or add a comment to make it clear that it always applies cherrypicks in a sorted order instead?

208

Fortunately we're good here, svn diff -c provides empty patches for revisions that don't touch specific repositories so we end up with a no-op.
Will add a comment.

[ilya-biryukov]

• Added a comment on applying cherrypicks to all repos
• Restructured the code to make it clear apply_cherrypicks works on a sorted list of cherrypicks

[ilya-biryukov]

• s/cherry-pick/cherrypick in a comment

[ioeric]

lg after code comment is added.

Do we want to mention this change in the doc?

utils/docker/scripts/build_install_llvm.sh

167

Sorting inside apply_cherrypicks, you wouldn't need to worry about the state of CHERRYPICKS. The sorting here should also be really cheap. But up to you. This is shell script after all :)

A comment in apply_cherrypicks requiring CHERRYPICKS to be sorted would still be nice.

utils/docker/build_docker_image.sh

Docker-specific:
     -s|--source             image source dir (i.e. debian8, nvidia-cuda, etc)
     -d|--docker-repository  docker repository for the image
     -t|--docker-tag         docker tag for the image
   LLVM-specific:
     -b|--branch         svn branch to checkout, i.e. 'trunk',
                         'branches/release_40'
                         (default: 'trunk')
     -r|--revision       svn revision to checkout
+    -c|--cherrypick     revision to cherry-pick. Can be specified multiple times.
+                        Cherry-picks are performed in the sorted order using the
+                        following command:
+                        'svn patch <(svn diff -c \$rev)'.
     -p|--llvm-project   name of an svn project to checkout. Will also add the
                         project to a list LLVM_ENABLE_PROJECTS, passed to CMake.
                         For clang, please use 'clang', not 'cfe'.
                         Project 'llvm' is always included and ignored, if
                         specified.
                         Can be specified multiple times.
     -i|--install-target name of a cmake install target to build and include in
                         the resulting archive. Can be specified multiple times.
-d|--docker-repository)
       DOCKER_REPOSITORY="$1"
       shift
       ;;
     -t|--docker-tag)
       shift
       DOCKER_TAG="$1"
       shift
       ;;
-    -i|--install-target|-r|--revision|-b|--branch|-p|--llvm-project)
+    -i|--install-target|-r|--revision|-c|-cherrypick|-b|--branch|-p|--llvm-project)
       if [ "$1" == "-i" ] || [ "$1" == "--install-target" ]; then
         SEEN_INSTALL_TARGET=1
       fi
       BUILDSCRIPT_ARGS="$BUILDSCRIPT_ARGS $1 $2"
       shift 2
       ;;
     -c|--checksums)
       shift

utils/docker/scripts/build_install_llvm.sh

 the directory to /tmp/clang.tar.gz.
 
 Available options:
   -h|--help           show this help message
   -b|--branch         svn branch to checkout, i.e. 'trunk',
                       'branches/release_40'
                       (default: 'trunk')
   -r|--revision       svn revision to checkout
+  -c|--cherrypick     revision to cherry-pick. Can be specified multiple times.
+                      Cherry-picks are performed in the sorted order using the
+                      following command:
+                      'svn patch <(svn diff -c \$rev)'.
   -p|--llvm-project   name of an svn project to checkout. Will also add the
                       project to a list LLVM_ENABLE_PROJECTS, passed to CMake.
                       For clang, please use 'clang', not 'cfe'.
                       Project 'llvm' is always included and ignored, if
                       specified.
                       Can be specified multiple times.
   -i|--install-target name of a cmake install target to build and include in
                       the resulting archive. Can be specified multiple times.
 Required options: At least one --install-target.
 
 All options after '--' are passed to CMake invocation.
 EOF
 }
 
 LLVM_SVN_REV=""
+CHERRYPICKS=""
 LLVM_BRANCH=""
 CMAKE_ARGS=""
 CMAKE_INSTALL_TARGETS=""
 # We always checkout llvm
 LLVM_PROJECTS="llvm"
 CMAKE_LLVM_ENABLE_PROJECTS=""
 CLANG_TOOLS_EXTRA_ENABLED=0
 

 }
 
 while [[ $# -gt 0 ]]; do
   case "$1" in
     -r|--revision)
       shift
       LLVM_SVN_REV="$1"
       shift
       ;;
+    -c|--cherrypick)
+      shift
+      CHERRYPICKS="$CHERRYPICKS $1"
+      shift
+      ;;
     -b|--branch)
       shift
       LLVM_BRANCH="$1"
       shift
       ;;
     -p|--llvm-project)
       shift
       PROJ="$1"

 if [ "$LLVM_SVN_REV" != "" ]; then
   SVN_REV_ARG="-r$LLVM_SVN_REV"
   echo "Checking out svn revision r$LLVM_SVN_REV."
 else
   SVN_REV_ARG=""
   echo "Checking out latest svn revision."
 fi
 
+# Sort cherrypicks and remove duplicates.
+CHERRYPICKS="$(echo "$CHERRYPICKS" | xargs -n1 | sort | uniq | xargs)"
+
+function apply_cherrypicks() {
+  local CHECKOUT_DIR="$1"
+
+  [ "$CHERRYPICKS" == "" ] || echo "Applying cherrypicks"
+  pushd "$CHECKOUT_DIR"
+
+  # This function is always called on a sorted list of cherrypicks.
+  for CHERRY_REV in $CHERRYPICKS; do
+    echo "Cherry-picking r$CHERRY_REV into $CHECKOUT_DIR"
+
+    local PATCH_FILE="$(mktemp)"
+    svn diff -c $CHERRY_REV > "$PATCH_FILE"
+    svn patch "$PATCH_FILE"
+    rm "$PATCH_FILE"
+  done
+
+  popd
+}
+
 CLANG_BUILD_DIR=/tmp/clang-build
 CLANG_INSTALL_DIR=/tmp/clang-install
 
 mkdir "$CLANG_BUILD_DIR"
 
 # Get the sources from svn.
 echo "Checking out sources from svn"
 mkdir "$CLANG_BUILD_DIR/src"
 for LLVM_PROJECT in $LLVM_PROJECTS; do
   if [ "$LLVM_PROJECT" == "clang" ]; then
     SVN_PROJECT="cfe"
   else
     SVN_PROJECT="$LLVM_PROJECT"
   fi
 
   echo "Checking out https://llvm.org/svn/llvm-project/$SVN_PROJECT to $CLANG_BUILD_DIR/src/$LLVM_PROJECT"
   svn co -q $SVN_REV_ARG \
     "https://llvm.org/svn/llvm-project/$SVN_PROJECT/$LLVM_BRANCH" \
     "$CLANG_BUILD_DIR/src/$LLVM_PROJECT"
+
+  # We apply cherrypicks to all repositories regardless of whether the revision
+  # changes this repository or not. For repositories not affected by the
+  # cherrypick, applying the cherrypick is a no-op.
+  apply_cherrypicks "$CLANG_BUILD_DIR/src/$LLVM_PROJECT"
 done
 
 if [ $CLANG_TOOLS_EXTRA_ENABLED -ne 0 ]; then
   echo "Checking out https://llvm.org/svn/llvm-project/clang-tools-extra to $CLANG_BUILD_DIR/src/clang/tools/extra"
   svn co -q $SVN_REV_ARG \
     "https://llvm.org/svn/llvm-project/clang-tools-extra/$LLVM_BRANCH" \
     "$CLANG_BUILD_DIR/src/clang/tools/extra"
+
+  apply_cherrypicks "$CLANG_BUILD_DIR/src/clang/tools/extra"
 fi
 
 CHECKSUMS_FILE="/tmp/checksums/checksums.txt"
 
 if [ -f "$CHECKSUMS_FILE" ]; then
   echo "Validating checksums for LLVM checkout..."
   python "$(dirname $0)/llvm_checksum/llvm_checksum.py" -c "$CHECKSUMS_FILE" \
     --partial --multi_dir "$CLANG_BUILD_DIR/src"