[llvm] Improve export.sh with help and snapshot

[kwk]

Summary

This change adds the ability to create source tarballs for unreleased or untagged code by providing the --git-ref <GIT_REF> flag to the llvm/utils/release/export.sh script. This is useful for creating daily snapshot tarballs that can easily be consumed by packagers who want to build a daily snapshot.

The default behavior of export.sh hasn't changed.

You may also provide a --template argument to say how the artifacts
are supposed to be named (as suggested by @hans).

The -help output of export.sh was changed quite significantly to look like this:

Export the Git sources and build tarballs from them.

Usage: export.sh [-release|--release <major>.<minor>.<patch>]
                      [-rc|--rc <num>]
                      [-final|--final]
                      [-git-ref|--git-ref <git-ref>]
                      [-template|--template <template>]

Flags:

  -release  | --release <major>.<minor>.<patch>    The version number of the release
  -rc       | --rc <num>                           The release candidate number
  -final    | --final                              When provided, this option will disable the rc flag
  -git-ref  | --git-ref <git-ref>                  (optional) Use <git-ref> to determine the release and don't export the test-suite files
  -template | --template <template>                (optional) Possible placeholders: $PROJECT $YYYYMMDD $GIT_REF $RELEASE $RC.
                                                   Defaults to '${PROJECT}-${RELEASE}${RC}.src.tar.xz'.

The following list shows the filenames (with <placeholders>) for the artifacts
that are being generated (given that you don't touch --template).

  * llvm-<RELEASE><RC>.src.tar.xz
  * clang-<RELEASE><RC>.src.tar.xz
  * compiler-rt-<RELEASE><RC>.src.tar.xz
  * libcxx-<RELEASE><RC>.src.tar.xz
  * libcxxabi-<RELEASE><RC>.src.tar.xz
  * libclc-<RELEASE><RC>.src.tar.xz
  * clang-tools-extra-<RELEASE><RC>.src.tar.xz
  * polly-<RELEASE><RC>.src.tar.xz
  * lldb-<RELEASE><RC>.src.tar.xz
  * lld-<RELEASE><RC>.src.tar.xz
  * openmp-<RELEASE><RC>.src.tar.xz
  * libunwind-<RELEASE><RC>.src.tar.xz
  * flang-<RELEASE><RC>.src.tar.xz

Additional files being generated:

  * llvm-project-<RELEASE><RC>.src.tar.xz    (the complete LLVM source project)
  * test-suite-<RELEASE><RC>.src.tar.xz      (only when not using --git-ref)

To ease the creation of snapshot builds, we also provide these files

  * llvm-release-<YYYYMMDD>.txt        (contains the <RELEASE> as a text)
  * llvm-rc-<YYYYMMDD>.txt             (contains the rc version passed to the invocation of export.sh)
  * llvm-git-revision-<YYYYMMDD>.txt   (contains the current git revision sha1)

Example values for the placeholders:

  * <RELEASE>  -> 13.0.0
  * <YYYYMMDD> -> 20210414
  * <RC>       -> rc4        (will be empty when using --git-ref)

In order to generate snapshots of the upstream main branch you could do this for example:

  export.sh --git-ref upstream/main --template '${PROJECT}-${YYYYMMDD}.src.tar.xz'

[kwk]

• Fixup

[kwk]

• Fixup

[kwk]

llvm/utils/release/export.sh

31

This wasn't used anywhere.

[hans]

Is having <yyyymmdd> in the filename really the most useful thing. I'd imagine having the git ref directly might be more useful, or perhaps the output of "git-describe" on it. Or maybe it should have both the git-ref and date in the filename?
And when using the date, perhaps it should use the date of the commit (commitdate, not authordate i guess) rather than the date when the script is run?
Or maybe the script should take a template argument for the filenames?

llvm/utils/release/export.sh

31

The test-release.sh script (and maybe others?) have a "-git-ref" option. Maybe that's a better name than "-snapshot", for consistency?

51

Additional to what?

[aaronpuchert]

This is useful for creating daily snapshot tarballs that can easily be consumed by packagers who want to build a daily snapshot.

My understanding (derived from the containing directory's name) is that this script is used for creating the official release tarballs to be published on releases.llvm.org or GitHub. Are there plans to publish snapshots?

Short of that it's just a "create tarball for a given git hash", which isn't really LLVM-specific. Packagers will often have their own preferred way of creating snapshots, e.g. in Open Build Service I'd use a source service to create the snapshots automatically.

But that's not up to me. If @hans and @tstellar are fine with this, it's good to go.

In D101446#2738274, @hans wrote:

Is having <yyyymmdd> in the filename really the most useful thing. I'd imagine having the git ref directly might be more useful, or perhaps the output of "git-describe" on it. Or maybe it should have both the git-ref and date in the filename?

Makes sense to me, after all the result should ideally be reproducible. (Prior to compression at least.)

llvm/utils/release/export.sh

100–106

Seems like this duplicates 77-83, maybe remove those lines?

[kwk]

• Rename outward facing option --snapshot to --git-ref

[kwk]

• Remove duplicate code

[kwk]

In D101446#2738274, @hans wrote:

Is having <yyyymmdd> in the filename really the most useful thing. I'd imagine having the git ref directly might be more useful, or perhaps the output of "git-describe" on it. Or maybe it should have both the git-ref and date in the filename?
And when using the date, perhaps it should use the date of the commit (commitdate, not authordate i guess) rather than the date when the script is run?
Or maybe the script should take a template argument for the filenames?

Okay, the question is valid but I have a perfect reason for why you want the date of the script being run. As a package maintainer I already generate daily source tarballs and keep the last three days in this pre-release of my fork: https://github.com/kwk/llvm-project/releases/tag/source-snapshot.

Now, when I want to build a package today, I can create the download URL simply by putting the current date into this URL: https://github.com/kwk/llvm-project/releases/download/source-snapshot/llvm-<YYYYYMMDD>.src.tar.xz. And if I want to know which git revision or LLVM version this was, I can query these URLs respectively:

• https://github.com/kwk/llvm-project/releases/download/source-snapshot/llvm-git-revision-<YYYYYMMDD>.txt
• https://github.com/kwk/llvm-project/releases/download/source-snapshot/llvm-version-<YYYYYMMDD>.txt

If the date in here was anything else than the current date, I cannot construct the download URL. Having the date seems a bit old-school, yes, but it most certainly is easier to consume than any date of a commit which you cannot know upfront.

In D101446#2740503, @aaronpuchert wrote:

This is useful for creating daily snapshot tarballs that can easily be consumed by packagers who want to build a daily snapshot.

My understanding (derived from the containing directory's name) is that this script is used for creating the official release tarballs to be published on releases.llvm.org or GitHub. Are there plans to publish snapshots?

Like I mentioned to @hans above, I think we could utilize github to generate daily source tarballs for us and stored them in a pre-release. The advantage of a pre-release in this case is that it doesn't show up on the repo's front-page. I'm using my own fork's pre-release: https://github.com/kwk/llvm-project/releases/tag/source-snapshot and I'm using my own github workflow to generate this on a schedule: https://github.com/kwk/llvm-project/blob/snapshot/.github/workflows/re-build-source-snapshot.yml. Here you can see the individual runs of this action: https://github.com/kwk/llvm-project/actions/workflows/re-build-source-snapshot.yml.

Short of that it's just a "create tarball for a given git hash", which isn't really LLVM-specific. Packagers will often have their own preferred way of creating snapshots, e.g. in Open Build Service I'd use a source service to create the snapshots automatically.

But that's not up to me. If @hans and @tstellar are fine with this, it's good to go.

In D101446#2738274, @hans wrote:

Is having <yyyymmdd> in the filename really the most useful thing. I'd imagine having the git ref directly might be more useful, or perhaps the output of "git-describe" on it. Or maybe it should have both the git-ref and date in the filename?

Makes sense to me, after all the result should ideally be reproducible. (Prior to compression at least.)

I agree that it's a bit of a sacrifice. When you run the export script twice a day you get two different source tarballs for the same day that carry the same name. But I'd say that by the extra or additional files, you can always know which revision is being tar-ed: https://github.com/kwk/llvm-project/releases/download/source-snapshot/llvm-git-revision-<YYYYYMMDD>.txt.

I've played with both solutions, i.e. having the git checkout and tarball creation in my snapshot pipeline or outsourcing it. I must say that I liked the outsourced version better, simply because it makes packaging a bliss. All I have to do is take our regular RPM spec files and prefix the content with something like this, which is easily generated, given the llvm-git-revision-<YYYYMMDD>.txt and llvm-version-<YYYYYMMDD>.txt files:

################################################################################
# BEGIN SNAPSHOT PREFIX
################################################################################

%global _with_snapshot_build 1
%bcond_with snapshot_build

%if %{with snapshot_build}
%global llvm_snapshot_yyyymmdd 20210506
%global llvm_snapshot_version 13.0.0
%global llvm_snapshot_git_revision 207b08a9130bd8167851f9e053f5d67bfd1969c8

# Split version
%global llvm_snapshot_version_major %{lua: print(string.match(rpm.expand("%{llvm_snapshot_version}"), "[0-9]+"));}
%global llvm_snapshot_version_minor %{lua: print(string.match(rpm.expand("%{llvm_snapshot_version}"), "%p([0-9]+)%p"));}
%global llvm_snapshot_version_patch %{lua: print(string.match(rpm.expand("%{llvm_snapshot_version}"), "%p([0-9]+)$"));}

# Shorten git revision
%global llvm_snapshot_git_revision_short %{lua: print(string.sub(rpm.expand("%llvm_snapshot_git_revision"), 0, 14));}
%endif

################################################################################
# END SNAPSHOT PREFIX
################################################################################

All we needed to do was to have special snapshot treatment fenced with %if %{with snapshot_build}:

%if %{with snapshot_build}
Source0:        https://github.com/kwk/llvm-project/releases/download/source-snapshot/llvm-%{llvm_snapshot_yyyymmdd}.src.tar.xz
%else
Source0:        https://github.com/llvm/llvm-project/releases/download/llvmorg-%{version}%{?rc_ver:-rc%{rc_ver}}/%{llvm_srcdir}.tar.xz
Source1:        https://github.com/llvm/llvm-project/releases/download/llvmorg-%{version}%{?rc_ver:-rc%{rc_ver}}/%{llvm_srcdir}.tar.xz.sig
Source2:        tstellar-gpg-key.asc
%endif

Sorry to bore you with these RPM stuff. But I think it proofs a lot of benefit.

llvm/utils/release/export.sh

31

Agreed. Renamed.

51

Additional files to the source tarballs.

100–106

Oh, sure. I must have missed that. Done.

[kwk]

• Fixup renaming --snapshot to --git-ref

[aaronpuchert]

Somehow the patch got messed up, I think. The changes that I see look like the changes since the previous patch, not the new patch.

In D101446#2743089, @kwk wrote:

Like I mentioned to @hans above, I think we could utilize github to generate daily source tarballs for us and stored them in a pre-release.

So you want to build daily snapshots as RPM, and for that need the sources to be on GitHub to have them official?

Sorry to bore you with these RPM stuff. But I think it proofs a lot of benefit.

Not boring me, openSUSE works with RPM as well. Though our build service can generate tarballs from git repositories automatically, so if I wanted to build snapshots I would probably just use that.

[aaronpuchert]

(I think you have to squash the changes, Arcanist seems to take only the last commit.)

[kwk]

• Restoring patch

[kwk]

In D101446#2745415, @aaronpuchert wrote:

Somehow the patch got messed up, I think. The changes that I see look like the changes since the previous patch, not the new patch.

This should be fixed now. Sorry for the mess.

In D101446#2743089, @kwk wrote:

Like I mentioned to @hans above, I think we could utilize github to generate daily source tarballs for us and stored them in a pre-release.

So you want to build daily snapshots as RPM, and for that need the sources to be on GitHub to have them official?

In short, yes.

Sorry to bore you with these RPM stuff. But I think it proofs a lot of benefit.

Not boring me, openSUSE works with RPM as well. Though our build service can generate tarballs from git repositories automatically, so if I wanted to build snapshots I would probably just use that.

Generating the tarballs takes twelve minutes or so. For myself I wanted to have this outsourced of the build pipeline for the RPMs. From my short past experience with building snapshots, it wasn't LLVM in particular that needed to be adjusted in order for a build to go through. It was more about fixing paths or removing patches that already landed upstream. The feedback time for building an RPM that eventually fails was drastically reduced for me by relying on pre-built source tarballs. One other great benefit is that I can re-use the same RPM spec file for building snapshots and regular releases. This makes it easy to track changes that go into the next release more easy.

In D101446#2745416, @aaronpuchert wrote:

(I think you have to squash the changes, Arcanist seems to take only the last commit.)

Please let me know if this is fixed now.

[hans]

I don't feel strongly about this one way or the other.

Tom is the main user of this script, so I think it's up to him to decide what to do here.

[tstellar]

I'm fine with the concept of having an option for generating a snapshot tarball. I understand that it may not be useful to everyone, but we already ship tarballs as part of our releases and if we were to do official snapshots for the project, we would want to generate the tarballs in the same way. I do agree with Hans that it would be nice if there was some way to uniquely identify the tarball (e.g. with the git hash) otherwise if you do multiple tarballs a day, they would all have the same name.

[kwk]

Introducing --template <template> to get what reviewers want.

[kwk]

Fixup YYYYMMD -> YYYYMMDD

[kwk]

@hans @tstellar @aaronpuchert Can you please give this some final review? I've addressed your comments from before about a template for the filename for example.

[hans]

Looks okay to me, but I think Tom still needs to sign off on this since he's the main user of the script.

[tstellar]

llvm/utils/release/export.sh

85

This is getting the release version from the LLVM source that contains export.sh, but it should be getting the version from the source that will be packaged.

[kwk]

llvm/utils/release/export.sh

85

Oh boy, that's a very good catch!

[kwk]

@tstellar What do you think of fetching the CMakeLists.txt file from github?

llvm/utils/release/export.sh

85

I wonder if we can live with a solution that is backed by the fact that LLVM source lives on github. Then it is a matter of fetching a branch or a particular revision (NOT the local mirror/branch naming). To to a snapshot of one particular revision we could do this:

curl -s https://raw.githubusercontent.com/llvm/llvm-project/bcc29e0fcf24a74ef0ec68365afb020787ab0a88/llvm/CMakeLists.txt | grep -ioP 'set\(\s*LLVM_VERSION_(MAJOR|MINOR|PATCH)\s\K[0-9]+' | paste -sd '.'

And for the main branch we would do this:

curl -s https://raw.githubusercontent.com/llvm/llvm-project/main/llvm/CMakeLists.txt | grep -ioP 'set\(\s*LLVM_VERSION_(MAJOR|MINOR|PATCH)\s\K[0-9]+' | paste -sd '.'

Feel free to copy this to your local shell to try it out. Of course, the help text needs to be adjusted so that no one used their local remote and branch naming schemes.

The limitation of this approach is that you cannot release something that only you have locally, which is probably not a good practice anyways.

[kwk]

llvm/utils/release/export.sh

85

Or we could do this instead:

// Get CMakeListst.txt for a particular revision.

t=$(mktemp)
mv /path/to/llvm-project/llvm/CMakeLists.txt $t
git checkout <git-ref> -- /path/to/llvm-project/llvm/CMakeLists.txt

// Grep the Version from CMakeLists.txt
// ... the code exists here already.

// Restore from the backed up file:
mv $t /path/to/llvm-project/llvm/CMakeLists.txt

The downside is that we need to overwrite code in the local repo for a limited time frame.

[aaronpuchert]

You can also use git show <treeish>:<file>, e.g. git show llvmorg-12.0.1:llvm/CMakeLists.txt.

[kwk]

In D101446#2963466, @aaronpuchert wrote:

You can also use git show <treeish>:<file>, e.g. git show llvmorg-12.0.1:llvm/CMakeLists.txt.

This looks lovely. That's what I'll go for. Thank you so much @aaronpuchert !

[kwk]

Using git show to get the CMakeLists.txt file in the correct version.

[kwk]

Rebased onto upstream/main

[kwk]

Need to take care of this error: fatal: path '/home/runner/work/llvm-project/llvm-project/llvm/CMakeLists.txt' exists on disk, but not in 'upstream/main'

[kwk]

Deal with fatal: path '/home/runner/work/llvm-project/llvm-project/llvm/CMakeLists.txt' exists on disk, but not in 'upstream/main'

[kwk]

@tstellar the error you've found about the CMakeLists.txt is now fixed.

[tstellar]

LGTM, thanks.

llvm/utils/release/export.sh

-#!/bin/sh
+#!/bin/bash
 #===-- tag.sh - Tag the LLVM release candidates ----------------------------===#
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===------------------------------------------------------------------------===#
 #
 # Create branches and release candidates for the LLVM release.
 #
 #===------------------------------------------------------------------------===#
 
 set -e
 
 projects="llvm clang compiler-rt libcxx libcxxabi libclc clang-tools-extra polly lldb lld openmp libunwind flang"
 
 release=""
 rc=""
+yyyymmdd=$(date +'%Y%m%d')
+snapshot=""
+template='${PROJECT}-${RELEASE}${RC}.src.tar.xz'
 
 usage() {
-    echo "Export the Git sources and build tarballs from them"
-    echo "usage: `basename $0`"
-    echo " "
-    echo "  -release <num> The version number of the release"
-    echo "  -rc <num>      The release candidate number"
-    echo "  -final         The final tag"
+cat <<EOF
+Export the Git sources and build tarballs from them.
+
+Usage: $(basename $0) [-release|--release <major>.<minor>.<patch>]
+                      [-rc|--rc <num>]
+                      [-final|--final]
+                      [-git-ref|--git-ref <git-ref>]
+                      [-template|--template <template>]
+
+Flags:
+
+  -release  | --release <major>.<minor>.<patch>    The version number of the release
+  -rc       | --rc <num>                           The release candidate number
+  -final    | --final                              When provided, this option will disable the rc flag
+  -git-ref  | --git-ref <git-ref>                  (optional) Use <git-ref> to determine the release and don't export the test-suite files
+  -template | --template <template>                (optional) Possible placeholders: \$PROJECT \$YYYYMMDD \$GIT_REF \$RELEASE \$RC.
+                                                   Defaults to '${template}'.
+
+The following list shows the filenames (with <placeholders>) for the artifacts
+that are being generated (given that you don't touch --template).
+
+$(echo "$projects "| sed 's/\([a-z-]\+\) /  * \1-<RELEASE><RC>.src.tar.xz \n/g')
+
+Additional files being generated:
+
+  * llvm-project-<RELEASE><RC>.src.tar.xz    (the complete LLVM source project)
+  * test-suite-<RELEASE><RC>.src.tar.xz      (only when not using --git-ref)
+
+To ease the creation of snapshot builds, we also provide these files
+
+  * llvm-release-<YYYYMMDD>.txt        (contains the <RELEASE> as a text)
+  * llvm-rc-<YYYYMMDD>.txt             (contains the rc version passed to the invocation of $(basename $0))
+  * llvm-git-revision-<YYYYMMDD>.txt   (contains the current git revision sha1)
+
+Example values for the placeholders:
+
+  * <RELEASE>  -> 13.0.0
+  * <YYYYMMDD> -> 20210414   (the date when executing this script)
+  * <RC>       -> rc4        (will be empty when using --git-ref)
+
+In order to generate snapshots of the upstream main branch you could do this for example:
+
+  $(basename $0) --git-ref upstream/main --template '\${PROJECT}-\${YYYYMMDD}.src.tar.xz'
+
+EOF
+}
+
+template_file() {
+    export PROJECT=$1 YYYYMMDD=$yyyymmdd RC=$rc RELEASE=$release GIT_REF=$git_rev
+    basename $(echo $template | envsubst '$PROJECT $RELEASE $RC $YYYYMMDD $GIT_REF')
+    unset PROJECT YYYYMMDD RC RELEASE GIT_REF
 }
 
 export_sources() {
-    release_no_dot=`echo $release | sed -e 's,\.,,g'`
+    local tag="llvmorg-$release"
+
+    llvm_src_dir=$(readlink -f $(dirname "$(readlink -f "$0")")/../../..)
+    [ -d $llvm_src_dir/.git ] || ( echo "No git repository at $llvm_src_dir" ; exit 1 )
+
+    # Determine the release by fetching the version from LLVM's CMakeLists.txt
+    # in the specified git ref.
+    if [ -n "$snapshot" ]; then
+        release=$(git -C $llvm_src_dir show $snapshot:llvm/CMakeLists.txt | grep -ioP 'set\(\s*LLVM_VERSION_(MAJOR|MINOR|PATCH)\s\K[0-9]+' | paste -sd '.')
+    fi
+    
     tag="llvmorg-$release"
 
     if [ "$rc" = "final" ]; then
         rc=""
     else
         tag="$tag-$rc"
     fi
 
-    llvm_src_dir=$(readlink -f $(dirname "$(readlink -f "$0")")/../../..)
-    [ -d $llvm_src_dir/.git ] || ( echo "No git repository at $llvm_src_dir" ; exit 1 )
-
-    echo $tag
     target_dir=$(pwd)
 
     echo "Creating tarball for llvm-project ..."
     pushd $llvm_src_dir/
-    git archive --prefix=llvm-project-$release$rc.src/ $tag . | xz >$target_dir/llvm-project-$release$rc.src.tar.xz
+    tree_id=$tag
+    [ -n "$snapshot" ] && tree_id="$snapshot"
+    echo "Tree ID to archive: $tree_id"
+
+    # This might be a surprise but a package like clang or compiler-rt don't
+    # know about the LLVM version itself. That's why we also export a the
+    # llvm-version*-<YYYYMMDD> and llvm-git*-<YYYYMMDD> files.
+    git_rev=$(git rev-parse $tree_id)
+    echo "git revision: $git_rev"
+    echo "$release" > $target_dir/llvm-release-$yyyymmdd.txt
+    echo "$rc" > $target_dir/llvm-rc-$yyyymmdd.txt
+    echo "$git_rev" > $target_dir/llvm-git-revision-$yyyymmdd.txt
+    
+    git archive --prefix=llvm-project-$release$rc.src/ $tree_id . | xz >$target_dir/$(template_file llvm-project)
     popd
 
-    if [ ! -d test-suite-$release$rc.src ]
-    then
+    if [ -z "$snapshot" ]; then
+        if [ ! -d test-suite-$release$rc.src ]; then
             echo "Fetching LLVM test-suite source ..."
             mkdir -p test-suite-$release$rc.src
             curl -L https://github.com/llvm/test-suite/archive/$tag.tar.gz | \
                 tar -C test-suite-$release$rc.src --strip-components=1 -xzf -
         fi
         echo "Creating tarball for test-suite ..."
         tar --sort=name --owner=0 --group=0 \
             --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
             -cJf test-suite-$release$rc.src.tar.xz test-suite-$release$rc.src
+    fi
 
     for proj in $projects; do
         echo "Creating tarball for $proj ..."
         pushd $llvm_src_dir/$proj
-        git archive --prefix=$proj-$release$rc.src/ $tag . | xz >$target_dir/$proj-$release$rc.src.tar.xz
+        git archive --prefix=$proj-$release$rc.src/ $tree_id . | xz >$target_dir/$(template_file $proj)
         popd
     done
 }
 
 while [ $# -gt 0 ]; do
     case $1 in
         -release | --release )
             shift
             release=$1
             ;;
         -rc | --rc )
             shift
             rc="rc$1"
             ;;
         -final | --final )
             rc="final"
             ;;
+        -git-ref | --git-ref )
+            shift
+            snapshot="$1"
+            ;;
+        -template | --template )
+            shift
+            template="$1"
+            ;;
         -h | -help | --help )
             usage
             exit 0
             ;;
         * )
             echo "unknown option: $1"
             usage
             exit 1
             ;;
     esac
     shift
 done
 
-if [ "x$release" = "x" ]; then
+if [ -n "$snapshot" ]; then 
+    if [[ "$rc" != "" || "$release" != "" ]]; then
+        echo "error: must not specify -rc or -release when creating a snapshot"
+        exit 1
+    fi
+elif [ -z "$release" ]; then
     echo "error: need to specify a release version"
     exit 1
 fi
 
 # Make sure umask is not overly restrictive.
 umask 0022
 
 export_sources
 exit 0