export.sh: Fetch sources from GitHub instead of SVN

[hans]

llvm/utils/release/export.sh

56

When I was thinking about this, I figured maybe we should just ship llvm-project.tar.xz and test-suite.tar.xz, rather than split up into the old project structure?

It seems odd in a way to release in a format that doesn't match the code layout used by developers. Do you think the llvm-projects.tar.xz tarball would be prohibitively large? Or that the change is too large for users of the release?

[tstellar]

llvm/utils/release/export.sh

56

If we distributed a single tarball, that would be easy, because GitHub creates one for us and adds it to the list of release assets. The gz compressed file generated by GitHub is 108M.

However, as a distro packager, I prefer having the separate tarballs because that's what we use for building packages on Fedora and RHEL.

@mgorny, @sylvestre.ledru Do you have any thoughts about this?

[sylvestre.ledru]

llvm/utils/release/export.sh

56

For Debian & Ubuntu, I moved to a single tarball containing it all. Splitting tarballs doesn't bring much values compare to the maintenance cost.

BTW, using xz, the tarball is only 72M for 9.0.0

[hans]

llvm/utils/release/export.sh

56

I see.

I don't have strong opinions about this, but I always assumed that the non-monorepo layout build would go away at some point after the migration.

Maybe we should release both an llvm-project.tgz "monorepo tarball" and the separate ones for a while?

[mgorny]

llvm/utils/release/export.sh

56

I would prefer split tarballs. There is still a vast number of Gentoo systems whose users interface with LLVM only through Mesa dependencies, and they only need the main LLVM component (and no clang, in the more common case).

[tstellar]

llvm/utils/release/export.sh

56

If we want to release an llvm-project tarball, should we have this script generate one or just go with what GitHub gives us[1]

Pros of relying on GitHub is it's easier and then we avoid having 2 source tarballs on the GitHub releases page (you can't remove the GitHub generated one). Cons of GitHub is that it uses .tar.gz, so it's bigger than an xz compressed file. 108M vs 72M.

[1] https://github.com/llvm/llvm-project/releases/tag/llvmorg-8.0.1 (The "Source Code" links)

[mgorny]

llvm/utils/release/export.sh

56

Don't forget that git archive tarballs are not guaranteed to be stable. Their checksums occasionally change (when GitHub upgrades git, I think) and break everything relying on them.

[hans]

llvm/utils/release/export.sh

56

Non-stable tarballs doesn't sound so good.

Another con of GitHub's is that I guess we can't sign them. Or if we do, and the GitHub one changes, the signature won't match.

I would prefer us uploading .tar.xz's and .sig's both for the separate projects, and one for llvm-project.

[tstellar]

Also generate a tarball with all sub-project sources.

[merge_guards_bot]

Build result: pass - 60481 tests passed, 0 failed and 726 were skipped.

Log files: console-log.txt, CMakeCache.txt

[hans]

lgtm conceptually, just wondering how the "mv" i commented about works.

llvm/utils/release/export.sh

58

It looks like there's only one argument to the mv command. Or is there some bash magic here that makes it work?

[tstellar]

llvm/utils/release/export.sh

58

This expands to:

mv $llvm_src_dir/$pro j $llvm_src_dir/$proj -$release$rc.src

[tstellar]

llvm/utils/release/export.sh

58

I can change it if you think it makes it more readable.

[hans]

llvm/utils/release/export.sh

58

Yes please, even if the expansion is cool (and I want to learn more about these), I think the explicit version is better for readability.

llvm/utils/release/export.sh

 #!/bin/sh
 #===-- tag.sh - Tag the LLVM release candidates ----------------------------===#
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===------------------------------------------------------------------------===#
 #
 # Create branches and release candidates for the LLVM release.
 #
 #===------------------------------------------------------------------------===#
 
 set -e
 
-projects="llvm cfe test-suite compiler-rt libcxx libcxxabi clang-tools-extra polly lldb lld openmp libunwind"
-base_url="https://llvm.org/svn/llvm-project"
+projects="llvm clang test-suite compiler-rt libcxx libcxxabi clang-tools-extra polly lldb lld openmp libunwind"
 
 release=""
 rc=""
 
 usage() {
-    echo "Export the SVN sources and build tarballs from them"
+    echo "Export the Git sources and build tarballs from them"
     echo "usage: `basename $0`"
     echo " "
     echo "  -release <num> The version number of the release"
     echo "  -rc <num>      The release candidate number"
     echo "  -final         The final tag"
 }
 
 export_sources() {
     release_no_dot=`echo $release | sed -e 's,\.,,g'`
-    tag_dir="tags/RELEASE_$release_no_dot/$rc"
+    tag="llvmorg-$release"
 
     if [ "$rc" = "final" ]; then
         rc=""
+    else
+        tag="$tag-$rc"
     fi
 
-    for proj in $projects; do
-        echo "Exporting $proj ..."
-        svn export \
-            $base_url/$proj/$tag_dir \
-            $proj-$release$rc.src
+    llvm_src_dir=llvm-project-$release$rc
+    mkdir -p $llvm_src_dir
+
+    echo $tag
+    echo "Fetching LLVM project source ..."
+    curl -L https://github.com/llvm/llvm-project/archive/$tag.tar.gz | \
+        tar -C $llvm_src_dir --strip-components=1 -xzf -
+
+    echo "Creating tarball for llvm-project ..."
+    tar -cJf llvm-project-$release$rc.tar.xz $llvm_src_dir
+
+    echo "Fetching LLVM test-suite source ..."
+    mkdir -p $llvm_src_dir/test-suite
+    curl -L https://github.com/llvm/test-suite/archive/$tag.tar.gz | \
+        tar -C $llvm_src_dir/test-suite --strip-components=1 -xzf -
 
-        echo "Creating tarball ..."
-        tar cfJ $proj-$release$rc.src.tar.xz $proj-$release$rc.src
+    for proj in $projects; do
+        echo "Creating tarball for $proj ..."
+        mv $llvm_src_dir/$proj $llvm_src_dir/$proj-$release$rc.src
+        tar -C $llvm_src_dir -cJf $proj-$release$rc.src.tar.xz $proj-$release$rc.src
     done
 }
 
 while [ $# -gt 0 ]; do
     case $1 in
         -release | --release )
             shift
             release=$1