[automation] Add scripts to automate GitHub projects

[thieta]

Summary

Two new scripts that will later be called from github-actions:

• github-project.py is using GitHub GraphQL API to query and change the github projects board we use. This can show all the issues in the project and add new issues to the project.
• sync-release-repo.sh is a bash script that syncs llvm/llvm-project and llvm/llvm-project-release-prs and optionally merges the PR's that are marked as "Needs Merge".

[thieta]

Removed the auto pr merging - it was not ready

[kwk]

@thieta as much as I like the effort you've made here. I'd like to propose a standalone GraphQL library. You're adding lots of code that is hard to trust without tests. That is why I started to create a library with tests and a release process to pypi. I think much of the things you do here are so generic that we can integrate them into the library at will.

https://github.com/kwk/ghgql

So far I will try to see if we can add the queries you've written nicely into "my" library.

llvm/utils/git/github-project.py

14–36

I would love to see those queries in their own files. Then we can call them manually and verify them using tools etc. But I see that this query itself comes from here: https://docs.github.com/en/issues/planning-and-tracking-with-projects/automating-your-project/automating-projects-using-actions#example-workflow-authenticating-with-a-github-app, right?

78

154

This function is deprecated but still works and requires minimal changes. I'm doing from requests import Session and then self.__session = Session(). For example. BUT: I found out when running my tests, that the session object is never closed. That's why I've implemented my GithubGraphQL class as a context manger so that you can write:

with ghgql.GithubGraphQL() as ghapi:
    ghapi.query_from_file("foo.gql")

and the session stored in ghapi will be closed.

157–159

@thieta please take a look at the function run_graphql_query and the comments for it here: https://reviews.llvm.org/D119412. We've had some discussions on GraphQL already. And I believe we agreed to outsource the queries in the end. At least we discussed to have them clean and use the variables array which you've type here quite nicely.

Anyway I would love to see a sharing of code when it comes to GraphQL. Especially when it comes to making proper calls using an API in Python.

You're potentially raising an HttpError when calling raise_for_status. Is that on purpose or do you want to handle errors locally here?

When you run the PROJECT_FIELDS query in the GraphQL explorer (https://docs.github.com/en/graphql/overview/explorer) you're being greeted with this error:

Although you appear to have the correct authorization credentials, the `llvm` organization has enabled OAuth App access restrictions, meaning that data access to third-parties is limited. For more information on these restrictions, including how to enable this app, visit https://docs.github.com/articles/restricting-access-to-your-organization-s-data/

That is why I believe you should specify a token when making a request to the GraphQL API.

EDIT: I saw that you're setting the bearer token to the global session object directly when parsing the arguments. In the spirit of sharing code for GraphQL usage, can we please just make a class with some nice defaults? Here's an idea for such a class. Then we can get rid of global objects and alike: https://github.com/kwk/ghgql.

I think this incorporates all of the stuff from @tstellar and myself as well as your code and additional tests. I've decided to always return a ghgql.Result even in case of errors. See the tests/ folder for usage information. This way we don't have to mess with exceptions so much and you can call the method on the returned object. All that ghgql.Result does is to wrap dict. What do you think? Can we get this in as a separate patch maybe? I mean not the python package but maybe it makes sense to outsource it like I did and only keep the code in the llvm tree. @serge-sans-paille you do a lot for python-lit if I'm not mistaken. Maybe you can tell us more about the pros and cons.

162–170

I've designed the ghgql.Result class to only raise a RuntimeError if a user tries to access and element from a result but there are errors present (see https://github.com/kwk/ghgql/blob/main/src/ghgql/result.py#L36).

174–188

I like this idea. You can find this idea in my ghgql.Result class here: https://github.com/kwk/ghgql/blob/main/src/ghgql/result.py#L18

[kwk]

llvm/utils/git/github-project.py

201–204

According to https://github.blog/2021-11-16-graphql-global-id-migration-update/ you can pass an extra header 'X-Github-Next-Global-ID': '1' and you'll get the id automatically. See here for how I do it: https://github.com/kwk/ghgql/blob/main/src/ghgql/ghgql.py#L70 . I'm not sure if this helps the problem but I sense that we can get rid of some dance with the API.

[kwk]

llvm/utils/git/sync-release-repo.sh

12

You're not using SCRIPT_DIR anywhere in this script. Let's not copy snippets from other script that we don't use.

[thieta]

In D133476#3783340, @kwk wrote:

@thieta as much as I like the effort you've made here. I'd like to propose a standalone GraphQL library. You're adding lots of code that is hard to trust without tests. That is why I started to create a library with tests and a release process to pypi. I think much of the things you do here are so generic that we can integrate them into the library at will.

Hi,

No problem - I was actually not that keen on adding all that code, but the only python graphql library I found was graphene and it seemed to be insane overkill. I learned a lot about GraphQL in the process so I don't mind moving it to another library.

https://github.com/kwk/ghgql

So far I will try to see if we can add the queries you've written nicely into "my" library.

Yeah I think the only complicated part of my script was to figure out the GraphQL queries and syntax, the code to request graphql's are pretty straightforward.

The only thing I would make sure we avoid is to make this to-complicated, graphql already seems very over-engineered (to my eyes) and we just need such a small small subset that we shouldn't put to much effort into handling all kinds of queries and edge-cases.

[thieta]

Reading through your comments I think it's fair to say that we should try doing the same things with your library and see if that's working and if it's easier or more annoying. Is that something you have bandwidth for or should I try to use your library? I just want to solve the issue mentioned above so that there is less for me to do for each release, so implementation wise I am less bothered exactly how it's done.

llvm/utils/git/github-project.py

14–36

Yeah agreed.

I did most of the queries using the explorer tool by github.

[kwk]

In D133476#3783529, @thieta wrote:

[...]
The only thing I would make sure we avoid is to make this to-complicated, graphql already seems very over-engineered (to my eyes) and we just need such a small small subset that we shouldn't put to much effort into handling all kinds of queries and edge-cases.

I agree. That's why I in ghgql I focused on the simplicity of making a and accessing the result. In general I find it weird to handle exceptions in that case which is why I've *deferred* them from the call to GraphQL to the inspection of results. That said, an {"errors": ...}" result doesn't raise an exception but returns a regular Result` object that you can ask for errors or simply just access the thing you're looking for. If you call Result.get("foo.bar") but an "errors" is present, an exception is raised. I had the feeling that this makes more sense.

If possible I'd like to keep ghgql well tested so you can rely on it. It is very very slim and most of the code is for doing the releases. For me this was an experiment to try out python packaging and it works quite nice now. On each successful push to master and tests, a new version is released.

[thieta]

In D133476#3817387, @kwk wrote:

I agree. That's why I in ghgql I focused on the simplicity of making a and accessing the result. In general I find it weird to handle exceptions in that case which is why I've *deferred* them from the call to GraphQL to the inspection of results. That said, an {"errors": ...}" result doesn't raise an exception but returns a regular Result` object that you can ask for errors or simply just access the thing you're looking for. If you call Result.get("foo.bar") but an "errors" is present, an exception is raised. I had the feeling that this makes more sense.

If possible I'd like to keep ghgql well tested so you can rely on it. It is very very slim and most of the code is for doing the releases. For me this was an experiment to try out python packaging and it works quite nice now. On each successful push to master and tests, a new version is released.

I tried to port my changes over to ghgql - and it seems to work mostly. I filed two issues in the ghgql repo where this one: https://github.com/kwk/ghgql/issues/5 is currently the biggest blocker.

[kwk]

In D133476#3829413, @thieta wrote:

[...]
I tried to port my changes over to ghgql - and it seems to work mostly. I filed two issues in the ghgql repo where this one: https://github.com/kwk/ghgql/issues/5 is currently the biggest blocker.

I have a potential and very simple to solution to the problem in a PR here: https://github.com/kwk/ghgql/pull/7 . Can you look at this please?

llvm/utils/git/github-project.py

+#!/usr/bin/env python
+
+import requests
+import sys
+import json
+from pprint import pprint
+from typing import Optional, Dict, Union, Tuple, Any, List
+import argparse
+
+# GraphQL to find all the ID's we need for other operations
+# This is because things like Status in the board can't be
+# referred to by name.
+PROJECT_FIELDS = """
+query($org: String!, $number: Int!) {
+  organization(login: $org) {
+    projectV2(number: $number) {
+      id
+      fields(first:20) {
+        nodes {
+          ... on ProjectV2Field {
+            id
+            name
+          }
+          ... on ProjectV2SingleSelectField {
+            id
+            name
+            options {
+              id
+              name
+            }
+          }
+        }
+      }
+    }
+  }
+}
+"""
+
+# Map Issue Number -> Issue Id which is needed
+ISSUE_ID = """
+query($org: String!, $repo: String!, $number: Int!) {
+  organization(login: $org) {
+    repository(name: $repo) {
+      issue(number: $number) {
+        id
+      }
+    }
+  }
+}
+"""
+
+# Make a search query - this is similar to the
+# search feature on the website and can take
+# the same query.
+SEARCH_ISSUES = """
+query($searchQuery: String!) {
+  search(
+    query: $searchQuery
+    type: ISSUE
+    first: 100
+  ) {
+    edges {
+      node {
+        ... on Issue {
+          id
+          number
+          title
+          url
+        }
+      }
+    }
+  }
+}
+"""
+
+# GraphQL to add a issue to a project
+ADD_ISSUE_TO_PROJECT = """
+mutation AddToProject($project: ID!, $issue: ID!) {
+  addProjectV2ItemById(input: { projectId: $project, contentId: $issue }) {
+    item {
+      id
+    }
+  }
+}
+"""
+
+# GraphQL to query information from project
+ALL_ISSUES_IN_PROJECT = """
+query($org: String!, $number: Int!) {
+  organization(login: $org) {
+    projectV2(number: $number) {
+      items(last: 100) {
+        nodes {
+          id
+
+          fieldValues(first: 8) {
+            nodes {
+              ... on ProjectV2ItemFieldSingleSelectValue {
+                name
+                field {
+                  ... on ProjectV2FieldCommon {
+                    name
+                  }
+                }
+              }
+              ... on ProjectV2ItemFieldTextValue {
+                text
+                field {
+                  ... on ProjectV2FieldCommon {
+                    name
+                  }
+                }
+              }
+              ... on ProjectV2ItemFieldPullRequestValue {
+                pullRequests(last: 10) {
+                  nodes {
+                    url
+                    number
+                  }
+                }
+              }
+            }
+          }
+
+          content {
+            ...on Issue {
+              title
+              number
+              url
+              state
+              milestone {
+                number
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+"""
+
+# GraphQL set the status on issues in the project board
+SET_PROJECT_STATUS = """
+mutation UpdateProject($project: ID!, $item: ID!, $status_field: ID!, $status_value: String!) {
+  updateProjectV2ItemFieldValue(input: { projectId: $project, itemId: $item, fieldId: $status_field, value: { singleSelectOptionId: $status_value } }) {
+    projectV2Item {
+      id
+    }
+  }
+}
+"""
+
+session = requests.session()
+
+# Helper function to send the GraphQL request
+def graphql(query: str, variables: Dict[str, Union[int, str]]) -> Dict[Any, Any]:
+  req = session.post("https://api.github.com/graphql", json={"query": query, "variables": variables})
+  req.raise_for_status()
+  jsondata = req.json()
+
+  if "errors" in jsondata:
+    print("Something went wrong:")
+    pprint(jsondata["errors"])
+    raise RuntimeError
+
+  if not "data" in jsondata:
+    print("No data in return - instead we got:")
+    pprint(jsondata)
+    raise RuntimeError
+
+  return jsondata
+
+# Helper function to get data from a deeply nested dict
+# gv(dict, "status.item.id") -> dict["status"]["item"]["id"]
+# plus that it handles if something is not there.
+def gv(data_dict: Dict, value_key: str, def_value:Any = None) -> Any:
+  keys = value_key.split(".")
+
+  cd = data_dict
+  for key in keys:
+    if isinstance(cd, str) and cd != "null":
+      cd = json.loads(cd)
+    if key in cd:
+      cd = cd[key]
+    else:
+      return def_value
+  return cd
+
+# Get Project ID and all the fields of it.
+def project_id_and_fields(project_id: int) -> Tuple[str, Dict]:
+  data = graphql(PROJECT_FIELDS, {"org": "llvm", "number": project_id})
+  fields = {}
+  for node in gv(data, "data.organization.projectV2.fields.nodes", []):
+    fields[node["name"]] = {
+      "id": node["id"],
+      "options": {n['name']:n['id'] for n in gv(node, "options", [])}
+    }
+  return (gv(data, "data.organization.projectV2.id", 0), fields)
+
+# Get the issue id by the issue number
+def issue_id(issue_id: int) -> str:
+  data = graphql(ISSUE_ID, {"org": "llvm", "repo": "llvm-project", "number": issue_id})
+  return gv(data, "data.organization.repository.issue.id", 0)
+
+# Run a GitHub search
+def search_issues(query: str) -> List[Dict[str, Any]]:
+  data = graphql(SEARCH_ISSUES, {"searchQuery": query})
+  issues = []
+  for node in gv(data, "data.search.edges", []):
+    node = node["node"]
+    issues.append({
+      "id": node["id"],
+      "number": node["number"],
+      "title": node["title"],
+      "url": node["url"]
+    })
+  return issues
+
+# Add the issue to the project
+def add_issue_to_project(project_id:str, issue_id:str) -> str:
+  data = graphql(ADD_ISSUE_TO_PROJECT, {"project": project_id, "issue": issue_id})
+  return gv(data, "data.addProjectNextItem.projectNextItem.id", None)
+
+# Get all issues from a project with all the extra data
+def get_issues_from_project(org: str, project_id: int) -> List[Dict[str, Any]]:
+  data = graphql(ALL_ISSUES_IN_PROJECT, {"org": org, "number": project_id})
+  issues = []
+  for node in gv(data, "data.organization.projectV2.items.nodes", []):
+    idata = {
+      "milestone": gv(node, "content.milestone.number", 0),
+      "number": gv(node, "content.number", 0),
+      "title": gv(node, "content.title", ""),
+      "state": gv(node, "content.state", ""),
+      "id": gv(node, "id", None),
+      "pr_url": [],
+      "pr": [],
+      "url": gv(node, "content.url", None),
+      "status": None,
+    }
+
+    for field in gv(node, "fieldValues.nodes", []):
+      if "pullRequests" in field:
+        for pr in gv(field, "pullRequests.nodes", []):
+          idata["pr_url"].append(pr["url"])
+          idata["pr"].append(str(pr["number"]))
+
+      if "field" in field and gv(field, "field.name") == "Status":
+        idata["status"] = field["name"]
+
+    issues.append(idata)
+
+  return issues
+
+# Set the status of a item in the project
+def set_project_status(project_id:str, issue_id:str, status_id:str, new_status:str) -> str:
+  # $project: ID!, $item: ID!, $status_field: ID!, $status_value: String!
+  data = graphql(SET_PROJECT_STATUS, {"project": project_id, "item": issue_id, "status_field": status_id, "status_value": new_status})
+  return gv(data, "data.updateProjectNxetItemField.projectNextItem.id", "")
+
+def list_command(args: Any):
+  project_id = None
+  status_to_set = None
+  status_id = None
+  if args.set_status:
+    project_id, fields = project_id_and_fields(args.projectnum)
+    status_id = gv(fields, "Status.id", None)
+    possible_status = gv(fields, "Status.options")
+    if not args.set_status in possible_status:
+      print(f"Can't set status to {args.set_status} - possible values are: {', '.join(possible_status.keys())}")
+      sys.exit(1)
+
+    status_to_set = possible_status[args.set_status]
+
+  issues = get_issues_from_project(args.org, args.projectnum)
+
+  filtered_issues = issues
+  if args.milestone or args.status or args.has_pr:
+    if args.milestone:
+      filtered_issues = [iss for iss in filtered_issues if iss['milestone'] == args.milestone]
+    if args.status:
+      filtered_issues = [iss for iss in filtered_issues if iss['status'] == args.status]
+    if args.state:
+      filtered_issues = [iss for iss in filtered_issues if iss['state'] == args.state]
+    if args.has_pr:
+      filtered_issues = [iss for iss in filtered_issues if len(iss['pr']) > 0]
+
+  for iss in filtered_issues:
+    if args.output:
+      stuff = args.output.split(",")
+      output = []
+      for s in stuff:
+        val = iss[s]
+        if isinstance(val, list):
+          output.append("|".join(val))
+        else:
+          output.append(str(val))
+
+      print(";".join(output))
+    else:
+      pr = ', '.join([p for p in iss["pr_url"]])
+      if pr:
+        pr = f"  PR: {pr}\n"
+      print(f"{iss['number']}: {iss['title']}\n  State: {iss['state']}\n  URL: {iss['url']}\n  milestone: {iss['milestone']}\n{pr}  Status: {iss['status']}")
+
+    if project_id and status_to_set and status_id:
+      print(f"Setting status to {args.set_status} ...\n")
+      set_project_status(project_id, iss['id'], status_id, status_to_set)
+
+def search_command(args: Any):
+  issues = search_issues(args.query)
+  project_id = None
+  for iss in issues:
+    if args.add_to_project:
+      if not project_id:
+        project_id, _ = project_id_and_fields(args.projectnum)
+      print(f"Adding {iss['number']} to project {args.projectnum}...")
+      add_issue_to_project(project_id, iss["id"])
+    else:
+      print(f"{iss['number']}: {iss['title']}\n  URL: {iss['url']}")
+
+def search_add_command(args: Any):
+  issues = search_issues(args.query)
+  project_id, _ = project_id_and_fields(args.projectnum)
+  for iss in issues:
+    print(f"Addding {iss['number']} to {args.projectnum} ({project_id})")
+    add_issue_to_project(project_id, iss['id'])
+
+if __name__ == '__main__':
+
+  parser = argparse.ArgumentParser()
+  parser.add_argument("--token", required=True, help="GitHub token")
+  parser.add_argument("--org", default="llvm", help="Organiziation to query")
+  parser.add_argument("--projectnum", default=3, help="The GitHub project id")
+
+  commands = parser.add_subparsers(dest="command")
+
+  listparser = commands.add_parser("list", help="list issues from a project")
+  listparser.add_argument("--milestone", default=None, type=int, help="List issues in milestone")
+  listparser.add_argument("--status", default=None, type=str, help="List issues with project status")
+  listparser.add_argument("--output", default=None, type=str, help="Output these fields, can be title, url, pr, pr_url, id")
+  listparser.add_argument("--has-pr", default=False, action="store_true", help="List issues with a linked PR")
+  listparser.add_argument("--state", default=None, type=str, help="List issues that has this state, can be OPEN or CLOSED")
+  listparser.add_argument("--set-status", default=None, type=str, help="Set the project status of issues matched by this list.")
+
+  searchparser = commands.add_parser("search", help="Run a GitHub search")
+  searchparser.add_argument("--query", required=True, help="GitHub search query - uses the same syntax as the website")
+  searchparser.add_argument("--add-to-project", default=False, action="store_true", help="Add all the issues matched to the project defined with --projectnum")
+
+  args = parser.parse_args()
+
+  session.headers["Authorization"] = "Bearer " + args.token
+
+  if args.command == "list":
+    list_command(args)
+  elif args.command == "search":
+    search_command(args)

llvm/utils/git/sync-release-repo.sh

+#!/bin/bash
+
+set -e
+set -x
+
+BRANCH="release/15.x"
+REPO="https://github.com/llvm/llvm-project"
+
+git config remote.upstream.url >&- || git remote add upstream $REPO
+
+CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+# Make sure we are up to date on all our repos first
+git fetch --all
+
+# Start by resetting to the upstream release branch - this is the most important
+# branch always
+git switch -c sync-repos upstream/$BRANCH
+git merge --ff-only origin/$BRANCH
+
+if ! git diff-index --quiet origin/$BRANCH; then
+  echo "Changes in origin - pushing to upstream"
+  git push origin sync-repos:$BRANCH
+fi
+
+# Then we need to update again
+git fetch --all
+
+# And merge all the new data to the current branch
+git merge --ff-only upstream/$BRANCH
+
+# If anything changed let's merge it
+if ! git diff-index --quiet upstream/$BRANCH; then
+  echo "Changes in upstream - pushing to origin"
+  git push upstream sync-repos:$BRANCH
+fi
+
+# Done - let's clean up
+git switch $CURRENT_BRANCH
+
+if git diff-index --quiet HEAD; then
+  git reset --hard @{u}
+fi
+
+git branch -d sync-repos