Build reproducible tarballs for releases

[aaronpuchert]

Summary

Currently the tarballs contain superfluous metadata, like the user name
of the packager and via Pax headers even the PID of the tar process that
packaged the files. We build the monorepo projects directly from the git
repo using "git archive" and for the test-suite we add some flags as
recommended by https://reproducible-builds.org/docs/archives/. We don't
use numeric owners though to be compatible with "git archive".

The advantage of "git archive" is that the releaser doesn't have to
download the tar ball and extract it, rather the archive is built
directly from the repository. This is probably what GitHub uses
internally to produce the tarballs, so I wouldn't expect a difference.

[hans]

Making the tarballs more reproducible sounds great.

I kind of liked that the script always downloads from github and doesn't depend on the state of my local repro. I don't feel strongly about this though, so I'll defer to Tom.

[aaronpuchert]

In D91494#2397260, @hans wrote:

I kind of liked that the script always downloads from github and doesn't depend on the state of my local repro.

It requires the tag to be on the same commit as on GitHub, but since you're setting the tag I'd assume that. Otherwise it doesn't depend on either your working tree or the index, or whether you're currently rebasing or whatever. The archive is generated from the snapshot defined by the git tag, and since git addresses everything by hashes this should reproducibly be the same wherever you do it.

[tstellar]

This is fine with me.

llvm/utils/release/export.sh

 #!/bin/sh
 #===-- tag.sh - Tag the LLVM release candidates ----------------------------===#
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===------------------------------------------------------------------------===#
 #
 # Create branches and release candidates for the LLVM release.
 #
 #===------------------------------------------------------------------------===#
 
 set -e
 
-projects="llvm clang test-suite compiler-rt libcxx libcxxabi libclc clang-tools-extra polly lldb lld openmp libunwind flang"
+projects="llvm clang compiler-rt libcxx libcxxabi libclc clang-tools-extra polly lldb lld openmp libunwind flang"
 
 release=""
 rc=""
 
 usage() {
     echo "Export the Git sources and build tarballs from them"
     echo "usage: `basename $0`"
     echo " "
     echo "  -release <num> The version number of the release"
     echo "  -rc <num>      The release candidate number"
     echo "  -final         The final tag"
 }
 
 export_sources() {
     release_no_dot=`echo $release | sed -e 's,\.,,g'`
     tag="llvmorg-$release"
 
     if [ "$rc" = "final" ]; then
         rc=""
     else
         tag="$tag-$rc"
     fi
 
-    llvm_src_dir=llvm-project-$release$rc
-    mkdir -p $llvm_src_dir
+    llvm_src_dir=$(readlink -f $(dirname "$(readlink -f "$0")")/../../..)
+    [ -d $llvm_src_dir/.git ] || ( echo "No git repository at $llvm_src_dir" ; exit 1 )
 
     echo $tag
-    echo "Fetching LLVM project source ..."
-    curl -L https://github.com/llvm/llvm-project/archive/$tag.tar.gz | \
-        tar -C $llvm_src_dir --strip-components=1 -xzf -
+    target_dir=$(pwd)
 
     echo "Creating tarball for llvm-project ..."
-    tar -cJf llvm-project-$release$rc.tar.xz $llvm_src_dir
+    pushd $llvm_src_dir/
+    git archive --prefix=llvm-project-$release$rc.src/ $tag . | xz >$target_dir/llvm-project-$release$rc.src.tar.xz
+    popd
 
+    if [ ! -d test-suite-$release$rc.src ]
+    then
       echo "Fetching LLVM test-suite source ..."
-    mkdir -p $llvm_src_dir/test-suite
+      mkdir -p test-suite-$release$rc.src
       curl -L https://github.com/llvm/test-suite/archive/$tag.tar.gz | \
-        tar -C $llvm_src_dir/test-suite --strip-components=1 -xzf -
+          tar -C test-suite-$release$rc.src --strip-components=1 -xzf -
+    fi
+    echo "Creating tarball for test-suite ..."
+    tar --sort=name --owner=0 --group=0 \
+        --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
+        -cJf test-suite-$release$rc.src.tar.xz test-suite-$release$rc.src
 
     for proj in $projects; do
         echo "Creating tarball for $proj ..."
-        mv $llvm_src_dir/$proj $llvm_src_dir/$proj-$release$rc.src
-        tar -C $llvm_src_dir -cJf $proj-$release$rc.src.tar.xz $proj-$release$rc.src
+        pushd $llvm_src_dir/$proj
+        git archive --prefix=$proj-$release$rc.src/ $tag . | xz >$target_dir/$proj-$release$rc.src.tar.xz
+        popd
     done
 }
 
 while [ $# -gt 0 ]; do
     case $1 in
         -release | --release )
             shift
             release=$1