github: Add manual workflow to build and upload release binaries

[tstellar]

Rebase on main.

[tstellar]

This patch is from @str4d

.github/workflows/release-binaries.yml

4

Do we need to add:

push:
  tags:
      - 'llvmorg-*'

10

Should we add an option to specify the ref/tag ?

[thieta]

Thanks for working on this! I have a bunch of comments :)

.github/workflows/release-binaries.yml

10

+1

35–44

This block is very messy. I would l prefer if this ended up in a script or something instead. Going to be complicated to troubleshoot this logic when it lives in the action definition.

54

Is this the correct triple? I would have expected x86_64-unknown-linux-gnu or something similar instead. At least since we pass this directly as LLVM_DEFAULT_TARGET I think we need to be careful to get this right.

Probably better to have a buildname and triple separated.

88

if we pipe this to true won't we miss errors that happens here then?

[tschuett]

.github/workflows/release-binaries.yml

15

I would recommend to pin the image to, e.g. ubuntu-22.04.

[tstellar]

Addresss some review feedback and also make sure the correct ref is checked out.

[tstellar]

.github/workflows/release-binaries.yml

35–44

OK, I will move this to a script.

54

The triple is only used for naming the tarball, not for anything else.

88

Since this is being run after the tag has been created, I think it's OK to ignore failures because it's too late to make changes at this point. Also if we fail here, then the binaries won't be uploaded.

The alternative would be to drop the || true and change the 'Upload binaries' step to always run.

[tstellar]

Put setup code into a script and make sure to always do the upload.

[tstellar]

Update permissions for the action.

[joycebrum]

I've add some comments about better scoped permissions.

.github/workflows/release-binaries.yml

19

Top level permissions should always be read only.

63

Write permissions should be given on run level (jobs).

[tstellar]

Fix permissions.

[kwk]

Added some ideas. Maybe not all so good.

.github/workflows/release-binaries.yml

13

16

16

47

54

I wonder if this will work. When I today tried this I got false and true values. That's why I check differently. The string comparison is safe this way.

.github/workflows/set-release-binary-outputs.sh

26

This one is not listed in the outputs: section of of the prepare job.

[tstellar]

Address some more review comments.

[tstellar]

.github/workflows/release-binaries.yml

54

The intended logic is for upload to be true if inputs.upload is unset. Otherwise upload should equal inputs.upload.

.github/workflows/set-release-binary-outputs.sh

26

It's there now.

[tstellar]

ping.

[kwk]

@tstellar Sorry for the late reply. But I have some thoughts.

.github/workflows/release-binaries.yml

11

Why is it optional to upload the binaries?

49–54

I like the shorter syntax and the fact that we check for trimmed whitespaces. But if Github removes them automatically that's fine. These kind of problems only arise when you have the workflow inputs. Everytime I wrote one I've removed it in the end because of an insane overhead of sanity checking in awkward places like workflow files.

101–108

This way it is closer to what you can paste in a console. Also, the folded style with > is very odd IMHO. See this for example:

To get a newline using the folded style, leave a blank line by putting two newlines in. Lines with extra indentation are also not folded.

112–117

Same reason as above.

.github/workflows/set-release-binary-outputs.sh

16

This way the logic is less implicit or bashy and more easy on the eyes IMHO.

17

What's with the echo here? It doesn't do anything, except that grep will exit with sonething different than 0 and that causes the script to exit, right? That's bad style IMHO. The fact that bash exits on failing commands shouldn't be exploited here. What if this line got removed one day because someone thinks it is not needed? Or what if the -e gets remove, then we simply run over this line. Let's make it explicit and more verbose?

[tstellar]

Address some review comments. @kwk I had to make some slight modifications to some
of your suggests, so let me know how this looks.

[tstellar]

Address more review comments.

[tstellar]

.github/workflows/release-binaries.yml

11

I thought it would be useful for doing test builds that we don't want to end up on the official release page.

47

bash -e is the default shell command already.

.github/workflows/set-release-binary-outputs.sh

16

I had to change the logic slightly, but I implemented this change.

[kwk]

Thank you Tom for accepting and improving on my feedback.

.github/workflows/release-binaries.yml

11

I thought it would be useful for doing test builds that we don't want to end up on the official release page.

Yes, that makes sense.

.github/workflows/set-release-binary-outputs.sh

16

I had to change the logic slightly, but I implemented this change.

Thank you. Much better.

.github/workflows/release-binaries.yml

+name: Release Binaries
+
+on:
+  push:
+    tags:
+      - 'llvmorg-*'
+  workflow_dispatch:
+    inputs:
+      upload:
+        description: 'Upload binaries to the release page'
+        required: true
+        default: true
+        type: boolean
+      tag:
+        description: 'Tag to build'
+        required: true
+        type: string
+
+permissions:
+  contents: read # Default everything to read-only
+
+jobs:
+  prepare:
+    name: Prepare to build binaries
+    runs-on: ubuntu-22.04
+    if: github.repository == 'llvm/llvm-project'
+    outputs:
+      release-version: ${{ steps.validate-tag.outputs.release-version }}
+      release: ${{ steps.validate-tag.outputs.release }}
+      build-dir: ${{ steps.validate-tag.outputs.build-dir }}
+      rc-flags: ${{ steps.validate-tag.outputs.rc-flags }}
+      ref: ${{ steps.validate-tag.outputs.ref }}
+
+    steps:
+    - name: Checkout LLVM
+      uses: actions/checkout@v3
+
+    - name: Validate and parse tag
+      id: validate-tag
+      # In order for the test-release.sh script to run correctly, the LLVM
+      # source needs to be at the following location relative to the build dir:
+      # | X.Y.Z-rcN | ./rcN/llvm-project
+      # | X.Y.Z     | ./final/llvm-project
+      #
+      # We also need to set divergent flags based on the release version:
+      # | X.Y.Z-rcN | -rc N -test-asserts
+      # | X.Y.Z     | -final
+      run: |
+        tag="${{ github.ref_name }}"
+        trimmed=`echo ${{ inputs.tag }} | xargs`
+        [[ "$trimmed" != "" ]] && tag="$trimmed"
+        if [ -n "${{ inputs.upload }}" ]; then
+          upload="${{ inputs.upload }}"
+        else
+          upload="true"
+        fi
+        bash .github/workflows/set-release-binary-outputs.sh "${{ github.actor }}" "$tag" "$upload"
+
+  build-binaries:
+    name: ${{ matrix.target.triple }}
+    permissions:
+      contents: write # To upload assets to release.
+    needs: prepare
+    runs-on: ${{ matrix.target.runs-on }}
+    strategy:
+      fail-fast: false
+      matrix:
+        target:
+          - triple: x86_64-linux-gnu-ubuntu-22.04
+            runs-on: ubuntu-22.04-8x32
+            debian-build-deps: >
+              chrpath
+              gcc-multilib
+              ninja-build
+
+    steps:
+    - name: Checkout LLVM
+      uses: actions/checkout@v3
+      with:
+        ref: ${{ needs.prepare.outputs.ref }}
+        path: ${{ needs.prepare.outputs.build-dir }}/llvm-project
+
+    - name: Install Brew build dependencies
+      if: matrix.target.brew-build-deps != ''
+      run: brew install ${{ matrix.target.brew-build-deps }}
+
+    - name: Install Debian build dependencies
+      if: matrix.target.debian-build-deps != ''
+      run: sudo apt install ${{ matrix.target.debian-build-deps }}
+
+    - name: Set macOS build env variables
+      if: runner.os == 'macOS'
+      run: |
+        echo "MACOSX_DEPLOYMENT_TARGET=10.9" >> $GITHUB_ENV
+
+    - name: Build and test release
+      run: |
+        ${{ needs.prepare.outputs.build-dir }}/llvm-project/llvm/utils/release/test-release.sh \
+        -release ${{ needs.prepare.outputs.release }} \
+        ${{ needs.prepare.outputs.rc-flags }} \
+        -triple ${{ matrix.target.triple }} \
+        -use-ninja \
+        -no-checkout \
+        -no-test-suite
+
+    - name: Upload binaries
+      if: ${{ always() && needs.prepare.outputs.upload == 'true' }}
+      run: |
+        ${{ needs.prepare.outputs.build-dir }}/llvm-project/llvm/utils/release/github-upload-release.py \
+        --token ${{ github.token }} \
+        --release ${{ needs.prepare.outputs.release-version }} \
+        upload \
+        --files ${{ needs.prepare.outputs.build-dir }}/clang+llvm-${{ needs.prepare.outputs.release-version }}-${{ matrix.target.triple }}.tar.xz

.github/workflows/set-release-binary-outputs.sh

+# Usage: set-release-binary-outputs.sh <github_user> <tag> <upload>
+
+set -e
+
+if [ -z "$GITHUB_OUTPUT" ]; then
+  export GITHUB_OUTPUT=`mktemp`
+  echo "Warning: Environment variable GITHUB_OUTPUT is not set."
+  echo "Writing output variables to $GITHUB_OUTPUT"
+fi
+
+github_user=$1
+tag=$2
+upload=$3
+
+if [[ "$github_user" != "tstellar" && "$github_user" != "tru" ]]; then
+  echo "ERROR: User not allowed: $github_user"
+  exit 1
+fi
+pattern='^llvmorg-[0-9]\+\.[0-9]\+\.[0-9]\+\(-rc[0-9]\+\)\?$'
+echo "$tag" | grep -e $pattern
+if [ $? != 0 ]; then
+  echo "ERROR: Tag '$tag' doesn't match pattern: $pattern"
+  exit 1
+fi
+release_version=`echo "$tag" | sed 's/llvmorg-//g'`
+release=`echo "$release_version" | sed 's/-.*//g'`
+build_dir=`echo "$release_version" | sed 's,^[^-]\+,final,' | sed 's,[^-]\+-rc\(.\+\),rc\1,'`
+rc_flags=`echo "$release_version" | sed 's,^[^-]\+,-final,' | sed 's,[^-]\+-rc\(.\+\),-rc \1 -test-asserts,' | sed 's,--,-,'`
+echo "release-version=$release_version" >> $GITHUB_OUTPUT
+echo "release=$release" >> $GITHUB_OUTPUT
+echo "build-dir=$build_dir" >> $GITHUB_OUTPUT
+echo "rc-flags=$rc_flags" >> $GITHUB_OUTPUT
+echo "upload=$upload" >> $GITHUB_OUTPUT
+echo "ref=$tag" >> $GITHUB_OUTPUT