stable-merge-request.sh: Add a script for submitting merge requests via bugzilla

[tstellar]

Summary

This script will automatically create a new stable merge request bug in
bugzilla for the given svn revision and release number.

[hans]

utils/release/stable-merge-request.sh

1 ↗

(On Diff #91597)

I might have gone for bash just to avoid the worry of whether this works in other shells.

245 ↗

(On Diff #91597)

I'm a bit torn on using the Bugzilla for merge requests. On the one hand I can see the benefit of having an issue opened for each merge request to keep track of which ones are open and which ones are done. On the other hand, I don't like the lack of context and visibility. For a bug like "Please merge rXXX", I'll have to search my inbox to see what rXXX is anyway. And the author of rXXX might not be notified about the request. Also, the discussion on that bug does not get sent to any mailing list, so later searches for rXXX will not show the merge discussion.

Having said that, I think this script will help though.

A few ideas:

1. It would be awesome if the subject of the bug could include the first line of the commit message, and the body of the bug could quote the full commit message. I think this should be farily easy to get with some svn invocation.

2. Could the commit author be added as a cc on the bug or otherwise get a notification? I realize this might be tricky.. we might want to cc the release person though.

And while I've expressed some skepticism, I think one process is better than two, so maybe the script should simple be called merge-request.sh and be used for the major releases too.

[tstellar]

utils/release/stable-merge-request.sh

245 ↗

(On Diff #91597)

For me, it's much easier to stay organized when the merge requests are in bugzilla, because there is a clear record of what has been done and what needs to be done. It also allows for other people to check in and easily see which requests are outstanding, without having to look through the mailing list archives. All the bugs have a link to the commit on phabricator, so that should make it easier to track down.

1. I think adding the commit message is doable.

2. Automatically adding the author as a cc is tricky because commit emails don't always match the email people use in bugzilla. I can add an option for to add cc's manually, but I think it would be better to do auto-detection as a follow on improvement.

[tstellar]

Rename script to merge-request.sh and add commit summary to bug report.

[hans]

One thing I just realized: the bugzilla-cli package, at least for Ubuntu, seems only available in the most recent versions: http://packages.ubuntu.com/search?keywords=bugzilla-cli (similar for Debian: https://packages.debian.org/search?keywords=bugzilla-cli)

At least for my work and home machines, that makes it more impractical to use, and I'm guessing that might apply to many other developers.

There seems to be a tool called "bugz" which is more widely available: http://packages.ubuntu.com/search?keywords=bugz
Maybe the script should use that instead, or both?

utils/release/merge-request.sh

3

nit: drop/change "Test the LLVM release candidates"

36

What does "Revision project" mean?

This seems to refer to the bugzilla Product field. Do we really care about that? Maybe to simplify this, both for the script and for the user, we could just file these bugs against the "new-bugs" product.

45

Since "-user" is not spelled "-user-email", I think this could be spelled simply "-assign-to".

225

This could match other bugs too.. maybe the query should be narrower?

235

Don't you need to provide the URL to the repository? This seems to assume it's run inside a checked out repository.

237

I think if we include the svn url above, there is no need for this... it should then work even if the user doesn't have an svn repository checked out or updated past the target revision.

It does assume the user has 'svn' installed, but that's required for git-svn too.

252

Personally, I prefer this to point to the viewvc, but it's not important.

256

I assume this is just for your debugging?

[tstellar]

In D30905#709621, @hans wrote:

One thing I just realized: the bugzilla-cli package, at least for Ubuntu, seems only available in the most recent versions: http://packages.ubuntu.com/search?keywords=bugzilla-cli (similar for Debian: https://packages.debian.org/search?keywords=bugzilla-cli)

It is available on pypi, so you can install it using pip install. This is recommended anyway, since there is a bug with the older versions.

At least for my work and home machines, that makes it more impractical to use, and I'm guessing that might apply to many other developers.

There seems to be a tool called "bugz" which is more widely available: http://packages.ubuntu.com/search?keywords=bugz
Maybe the script should use that instead, or both?

I can't find the bugz package for fedora or on pypi, so it's going to be pretty hard for me to test / use that one.

utils/release/merge-request.sh

36

This is optional, and does map to bugzilla Product field. The default is to use the new-bugs product, I thought having a project option would be useful if people wanted to be able to categorize their bugs, but I don't mind dropping this and the -component option if it helps simplify the script.

252

Is there a way to get viewvc to show you the full diff with all changes on a single page?

256

Yes, but I think it could be useful for users to verify they've entered the correct options.

[tstellar]

Simplify query for revision commit message, plus some other minor cleanups.

[hans]

utils/release/merge-request.sh

36

I think we should just drop it.

241

I think the URL should just be https://llvm.org/svn/llvm-project/
That way it will work for revisions to any module, not just llvm.

Also, I think the current command will print the *whole* log going back from $revision. I think this is the way to do it:
svn log -r $revision http://llvm.org/svn/llvm-project

252

I don't think so, which does make Phabricator seem like a better alternative :-)

256

Oh right, yes that makes sense.

[tstellar]

• Drop product and component options.
• Fix svn url.

[hans]

lgtm

utils/release/merge-request.sh

40

Maybe drop this and list_valid_components now since we're just using 'new-bugs'?

utils/release/merge-request.sh

+# !/bin/bash
+#===-- merge-request.sh  ---------------------------------------------------===#
+#
+#                     The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License.
+#
+#===------------------------------------------------------------------------===#
+#
+# Submit a merge request to bugzilla.
+#
+#===------------------------------------------------------------------------===#
+
+verify_inputs=1
+dryrun=""
+stable_version=""
+revision=""
+BUGZILLA_BIN=""
+BUGZILLA_CMD=""
+release_metabug=""
+bugzilla_product="new-bugs"
+bugzilla_component=""
+bugzilla_assigned_to=""
+bugzilla_user=""
+bugzilla_version=""
+bugzilla_url="http://bugs.llvm.org/xmlrpc.cgi"
+
+function usage() {
+  echo "usage: `basename $0` -user EMAIL -stable-version X.Y -r NUM"
+  echo ""
+  echo " -user EMAIL             Your email address for logging into bugzilla."
+  echo " -stable-version X.Y     The stable release version (e.g. 4.0, 5.0)."
+  echo " -r NUM                  Revision number to merge (e.g. 1234567)."
+  echo " -project PROJECT        Revision project (e.g. clang, llvm, etc.)"
+  echo "                         use -project help for a full list of "
+  echo "                         projects (optional)."
+  echo " -component COMPONENT    Bugzilla component for the given PROJECT."
+  echo "                         Make sure to use quotes when specifying "
+  echo "                         components with spaces (optional, unless "
+  echo "                         -project is also specified).
+  echo "                         Use -component help for a list of components"
+  echo "                         for PROJECT."
+  echo " -bugzilla-bin PATH      Path to bugzilla binary (optional)."
+  echo " -assign-to EMAIL        Assign bug to user with EMAIL (optional)."
+  echo " -disable-verify         Don't verify project/component (optional)."
+  echo "                         This can be used to reduce script execution "
+  echo "                         time."
+  echo " -dry-run                Print commands instead of executing them."
+}
+
+function list_valid_projects() {
+  echo "The valid projects are..."
+  echo ""
+  $BUGZILLA_CMD info -p
+  echo "llvm"
+}
+
+function list_valid_components() {
+  bugzilla_product="$1"
+  echo "The valid components are..."
+  $BUGZILLA_CMD info -c "${bugzilla_product}"
+}
+
+while [ $# -gt 0 ]; do
+  case $1 in
+    -user)
+      shift
+      bugzilla_user="$1"
+      ;;
+    -stable-version)
+      shift
+      stable_version="$1"
+      ;;
+    -r)
+      shift
+      revision="$1"
+      ;;
+    -project)
+      shift
+      project="$1"
+      ;;
+    -component)
+      shift
+      bugzilla_component="$1"
+      ;;
+    -bugzilla-bin)
+      shift
+      BUGZILLA_BIN="$1"
+      ;;
+    -assign-to)
+      shift
+      bugzilla_assigned_to="--assigned_to=$1"
+      ;;
+    -disable-verify)
+      verify_inputs=0
+      ;;
+    -dry-run)
+      dryrun="echo"
+      ;;
+    -help | --help | -h | --h | -\? )
+      usage
+      exit 0
+      ;;
+    * )
+      echo "unknown option: $1"
+      usage
+      exit 1
+      ;;
+  esac
+  shift
+done
+
+if [ -z "$stable_version" ]; then
+  echo "error: no stable version specified"
+  exit 1
+fi
+
+case $stable_version in
+  4.0)
+    release_metabug="32061"
+    ;;
+  *)
+    echo "error: invalid stable version"
+    exit 1
+esac
+
+if [ -z "$revision" ]; then
+  echo "error: revision not specified"
+  exit 1
+fi
+
+if [ -z "$bugzilla_user" ]; then
+  echo "error: bugzilla username not specified."
+  exit 1
+fi
+
+if [ -z "$BUGZILLA_BIN" ]; then
+  BUGZILLA_BIN=`which bugzilla`
+  if [ $? -ne 0 ]; then
+    echo "error: could not find bugzilla executable."
+    echo "Make sure the bugzilla cli tool is installed on your system: "
+    echo "pip install python-bugzilla (recommended)"
+    echo ""
+    echo "Fedora: dnf install python-bugzilla"
+    echo "Ubuntu/Debian: apt-get install bugzilla-cli"
+    exit 1
+  fi
+fi
+
+BUGZILLA_MAJOR_VERSION=`$BUGZILLA_BIN --version 2>&1 | cut -d . -f 1`
+
+if [ $BUGZILLA_MAJOR_VERSION -eq 1 ]; then
+
+  echo "***************************** Warning *******************************"
+  echo "You are using an older version of the bugzilla cli tool.  You will be "
+  echo "able to create bugs, but this script will crash with the following "
+  echo "error when trying to read back information about the bug you created:"
+  echo ""
+  echo "KeyError: 'internals'"
+  echo ""
+  echo "To avoid this error, use version 2.0.0 or higher"
+  echo "https://pypi.python.org/pypi/python-bugzilla"
+  echo "*********************************************************************"
+fi
+
+BUGZILLA_CMD="$BUGZILLA_BIN --bugzilla=$bugzilla_url"
+
+case "$project" in
+  "")
+    ;;
+  llvm)
+    bugzilla_product="libraries"
+    ;;
+  OpenMP)
+    bugzilla_product="openmp"
+    ;;
+  help)
+    list_valid_projects
+    exit 0
+    ;;
+  *)
+    bugzilla_product="$project"
+    ;;
+esac
+
+# Check if this is a valid product
+if [ $verify_inputs -eq 1 ]; then
+  $BUGZILLA_CMD info -c "${bugzilla_product}" &>/dev/null
+  if [ $? -ne 0 ]; then
+    echo ""
+    echo "error: '$project' is an invalid project"
+    list_valid_projects
+    exit 1
+  fi
+fi
+
+if [ -z "$project" ]; then
+  if [ -n "$bugzilla_component" ]; then
+    echo "error: cannot specify -component without -project"
+    exit 1
+  fi
+  bugzilla_component="new bugs"
+fi
+
+if [ -n "$project" ]; then
+  if [ -z "$bugzilla_component" ]; then
+    echo "error: -component is required when you specify -project"
+    exit 1
+  fi
+
+  if [ "$bugzilla_component" = "help" ]; then
+    list_valid_components "${bugzilla_product}"
+    exit 0
+  fi
+fi
+
+# Work-around the fact that not every product has been assigned version numbers in
+# bugzilla.
+case "$bugzilla_product" in
+  lld)
+    bugzilla_version="unspecified"
+    ;;
+  *)
+    bugzilla_version=$stable_version
+    ;;
+esac
+
+bug_url="https://reviews.llvm.org/rL$revision"
+# Check for duplicate:
+check_duplicates=`$BUGZILLA_CMD query --url $bug_url`
+
+if [ -n "$check_duplicates" ]; then
+  echo "Duplicate bug found:"
+  echo $check_duplicates
+  exit 1
+fi
+
+# Get short commit summary
+commit_summary=''
+commit_msg=`svn log https://llvm.org/svn/llvm-project/llvm/trunk@$revision`
+if [ $? -ne 0 ]; then
+  echo "warning: failed to get commit message."
+  commit_msg=""
+fi
+
+if [ -n "$commit_msg" ]; then
+  commit_summary=`echo "$commit_msg" | sed '4q;d' | cut -c1-80`
+  commit_summary=" : ${commit_summary}"
+fi
+
+bug_summary="Merge r$revision into the $stable_version branch${commit_summary}"
+
+if [ -z ${dryrun} ]; then
+  set -x
+fi
+
+${dryrun} $BUGZILLA_CMD --login --user=$bugzilla_user new \
+  -p "$bugzilla_product" \
+  -c "$bugzilla_component" -u $bug_url --blocked=$release_metabug \
+  -o All --priority=P --arch All -v $bugzilla_version \
+  --summary "${bug_summary}" \
+  -l "Is this patch OK to merge to the $stable_version branch?" \
+  $bugzilla_assigned_to \
+  --oneline
+
+set +x
+
+if [ -n ${dryrun} ]; then
+  exit 0
+fi
+
+if [ $BUGZILLA_MAJOR_VERSION -eq 1 ]; then
+  success=`$BUGZILLA_CMD query --url $revision`
+  if [ -z $success ]; then
+    echo "Failed to create bug."
+    exit 1
+  fi
+
+  echo " Created new bug:"
+  echo $success
+fi