Add cmake/ to release tarballs via concatenation

[aaronpuchert]

Summary

The solution using append was reported not to work, but additionally it
would use the contents of the checked-out source tree instead of the git
tag or commit. This uses git archive, so it will use the right commit,
and at least for me (with GNU tar) it seems to work as intended.

Should fix #53281.

[tstellar]

Would it be easier just to package the cmake directory in a separate tarball?

[kwk]

In D121972#3391268, @tstellar wrote:

Would it be easier just to package the cmake directory in a separate tarball?

IMHO it would be simpler, yes. But this patch looks good for now and it fixes the issue of potentially bundling the wrong version of the cmake directory. I've tested this patch and fixed a small issue:

commit f61e4194da963adf618e9ddfc319e2938ab0c958 (HEAD -> arcpatch-D121972)
Author: Konrad Kleine <[e-mail]>
Date:   Fri Mar 18 14:33:54 2022 +0100

    Properly delete temporary files with trap
    
    I didn't notice before but deleting multiple files by calling a bash
    `trap` on the same signal (`EXIT`) will actually override the previous
    `trap` handler, thus leading to left-overs.

diff --git a/llvm/utils/release/export.sh b/llvm/utils/release/export.sh
index a3586dec782a..bd4fc8a59bd1 100755
--- a/llvm/utils/release/export.sh
+++ b/llvm/utils/release/export.sh
@@ -130,14 +130,15 @@ export_sources() {
 
     # Package up top-level cmake directory so that we can append it to all projects.
     cmake_archive_file=$target_dir/$(template_file cmake)
-    trap "rm -fv $cmake_archive_file.tmp" EXIT
+    trap_files_to_delete="$cmake_archive_file.tmp"
+    trap 'rm -fv $trap_files_to_delete' EXIT
     git archive -o $cmake_archive_file.tmp $tree_id cmake/
 
     for proj in $projects; do
         echo "Creating tarball for $proj ..."
         pushd $llvm_src_dir/$proj
         target_archive_file=$target_dir/$(template_file $proj)
-        trap "rm -fv $target_archive_file.tmp" EXIT
+        trap_files_to_delete="$trap_files_to_delete $target_archive_file.tmp"
         git archive --prefix=$proj-$release$rc.src/ -o $target_archive_file.tmp $tree_id .
         tar -Af $target_archive_file.tmp $cmake_archive_file.tmp
         cat $target_archive_file.tmp | xz > $target_archive_file

[aaronpuchert]

Ah, the trap is a singleton and the second invocation overwrites the earlier trap.

Perhaps we could also do a globbed rm *.tmp? Not sure if that would be too dangerous, or if we could choose a prefix that makes it more acceptable.

[aaronpuchert]

Use mktemp to create a directory for all intermediate files, which puts them in /tmp by default. This way we can easily clean them all up on exit.

[aaronpuchert]

Also forgot the pushd, without that it doesn't work in a subdirectory.

[Ericson2314]

git archive --help says:

<path>
    Without an optional path parameter, all files and subdirectories of the
    current working directory are included in the archive. If one or more paths
    are specified, only these are included.

so perhaps we should consider just doing something like that? if a two directory tarbomb is no good, we can add another layer of nesting?

[aaronpuchert]

In D121972#3393048, @Ericson2314 wrote:

if a two directory tarbomb is no good, we can add another layer of nesting?

I suggested a separate tarball for cmake/, but @kwk had the idea to pack it along with the existing projects (D118481) which would make it work transparently.

That is, if you have a pre-14 build script that unpacks the tarball, then switches into e.g. llvm-$version.src/ and starts CMake there, it should also work with this, because the top-level CMake folder is in ../cmake. Even if you add subprojects into e.g. llvm/tools, they should also find these files, because a copy comes with their own tarball.

If we nest this, existing build scripts will certainly have to be changed. Then I think we better just provide a separate copy.

[kwk]

I've tested this patch and it works for me:

Here's the relevant part of the transcript when enabling set -x and just packaging up compiler-rt:

++ mktemp -d
+ tmp_dir=/tmp/tmp.GYsvF1Vg58
+ cmake_archive_file=/tmp/tmp.GYsvF1Vg58/cmake.tar
+ trap 'rm -rv /tmp/tmp.GYsvF1Vg58' EXIT
+ pushd /home/kkleine/llvm-project
~/llvm-project ~/llvm-project/llvm/utils/release
+ git archive -o /tmp/tmp.GYsvF1Vg58/cmake.tar upstream/main cmake/
+ popd
~/llvm-project/llvm/utils/release
+ for proj in $projects
+ echo 'Creating tarball for compiler-rt ...'
Creating tarball for compiler-rt ...
+ pushd /home/kkleine/llvm-project/compiler-rt
~/llvm-project/compiler-rt ~/llvm-project/llvm/utils/release
+ tmp_archive_file=/tmp/tmp.GYsvF1Vg58/compiler-rt.tar
+ git archive --prefix=compiler-rt-15.0.0.src/ -o /tmp/tmp.GYsvF1Vg58/compiler-rt.tar upstream/main .
+ tar -Af /tmp/tmp.GYsvF1Vg58/compiler-rt.tar /tmp/tmp.GYsvF1Vg58/cmake.tar
+ xz
++ template_file compiler-rt
++ export PROJECT=compiler-rt YYYYMMDD=20220321 RC= RELEASE=15.0.0 GIT_REF=5be5d0f56e25b8f1669df517018f43621f9b7e0f
++ PROJECT=compiler-rt
++ YYYYMMDD=20220321
++ RC=
++ RELEASE=15.0.0
++ GIT_REF=5be5d0f56e25b8f1669df517018f43621f9b7e0f
+++ echo '${PROJECT}-${YYYYMMDD}.src.tar.xz'
+++ envsubst '$PROJECT $RELEASE $RC $YYYYMMDD $GIT_REF'
++ basename compiler-rt-20220321.src.tar.xz
++ unset PROJECT YYYYMMDD RC RELEASE GIT_REF
+ popd
~/llvm-project/llvm/utils/release
+ exit 0
+ rm -rv /tmp/tmp.GYsvF1Vg58
removed '/tmp/tmp.GYsvF1Vg58/compiler-rt.tar'
removed '/tmp/tmp.GYsvF1Vg58/cmake.tar'
removed directory '/tmp/tmp.GYsvF1Vg58'

The resulting .tar.xz file contains everything we wanted.

Here's the diff I applied to speed up the testing:

diff --git a/llvm/utils/release/export.sh b/llvm/utils/release/export.sh
index 55e38ab12bc7..b67fc69ac7a2 100755
--- a/llvm/utils/release/export.sh
+++ b/llvm/utils/release/export.sh
@@ -11,9 +11,10 @@
 #
 #===------------------------------------------------------------------------===#
 
-set -e
+set -ex
 
 projects="llvm clang compiler-rt libcxx libcxxabi libclc clang-tools-extra polly lldb lld openmp libunwind flang"
+projects="compiler-rt"
 
 release=""
 rc=""
@@ -112,7 +113,7 @@ export_sources() {
     echo "$rc" > $target_dir/llvm-rc-$yyyymmdd.txt
     echo "$git_rev" > $target_dir/llvm-git-revision-$yyyymmdd.txt
     
-    git archive --prefix=llvm-project-$release$rc.src/ $tree_id . | xz >$target_dir/$(template_file llvm-project)
+    # git archive --prefix=llvm-project-$release$rc.src/ $tree_id . | xz >$target_dir/$(template_file llvm-project)
     popd
 
     if [ -z "$snapshot" ]; then

llvm/utils/release/export.sh

134

I'm unsure if this needs to be verbose.

[kwk]

In D121972#3393048, @Ericson2314 wrote:

git archive --help says:

<path>
    Without an optional path parameter, all files and subdirectories of the
    current working directory are included in the archive. If one or more paths
    are specified, only these are included.

so perhaps we should consider just doing something like that? if a two directory tarbomb is no good, we can add another layer of nesting?

This won't work because it will pack the cmake next to the content of compiler-rt. This can cause unexpected overwrites because there already is a cmake directory inside of compiler-rt.

[aaronpuchert]

llvm/utils/release/export.sh

134

Yes, it's likely not interesting. I just decided to copy over the -v for now.

llvm/utils/release/export.sh

if [ -z "$snapshot" ]; then
                 tar -C test-suite-$release$rc.src --strip-components=1 -xzf -
         fi
         echo "Creating tarball for test-suite ..."
         tar --sort=name --owner=0 --group=0 \
             --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
             -cJf test-suite-$release$rc.src.tar.xz test-suite-$release$rc.src
     fi
 
+    # Package up top-level cmake directory so that we can append it to all projects.
+    tmp_dir=$(mktemp -d)
+    cmake_archive_file=$tmp_dir/cmake.tar
+    trap "rm -rv $tmp_dir" EXIT
+    pushd $llvm_src_dir
+    git archive -o $cmake_archive_file $tree_id cmake/
+    popd
+
     for proj in $projects; do
         echo "Creating tarball for $proj ..."
         pushd $llvm_src_dir/$proj
-        target_archive_file=$target_dir/$(template_file $proj)
-        trap "rm -fv $target_archive_file.tmp" EXIT
-        git archive --prefix=$proj-$release$rc.src/ -o $target_archive_file.tmp $tree_id .
-        # Get relative path to top-level cmake directory to be packaged
-        # alongside the project. Append that path to the tarball.
-        cmake_rel_path=$(realpath --relative-to=. $llvm_src_dir/cmake)
-        tar --append -f $target_archive_file.tmp $cmake_rel_path
-        cat $target_archive_file.tmp | xz > $target_archive_file
+        tmp_archive_file=$tmp_dir/$proj.tar
+        git archive --prefix=$proj-$release$rc.src/ -o $tmp_archive_file $tree_id .
+        tar -Af $tmp_archive_file $cmake_archive_file
+        xz < $tmp_archive_file > $target_dir/$(template_file $proj)
         popd
     done
 }
 
 while [ $# -gt 0 ]; do
     case $1 in
         -release | --release )
             shift