Update the Bug Life Cycle docs for the switch to GitHub issues

[aaron.ballman]

Summary

This updates the Bug Life Cycle docs now that we've switched to GitHub issues. The intent is to retain the same general process we used to use for triaging bugs under Bugzilla, but with the facilities we have available in GitHub.

A few things to note: we have not yet created the confirmed label, but that is expected to be done shortly. There is already label confusion involving whether we should use beginner (89 issues) or good first issue (29 issues), so I went with good first issue in the documentation due to review feedback. I think we should strongly discourage users from creating arbitrary labels on their own because the labels will become useless pretty quickly if there's no consistency as to which are used, so I adjusted the docs accordingly.

[tstellar]

A few things to note: we have not yet created the confirmed label, but that is expected to be done shortly. There is already label confusion involving whether we should use beginner (89 issues) or good first issue (29 issues), so I went with beginner in the documentation. I think we should strongly discourage users from creating arbitrary labels on their own because the labels will become useless pretty quickly if there's no consistency as to which are used, so I adjusted the docs accordingly

'good first issue' is a default label provided by GitHub, so it seems like it would be good to use it for consistency with other projects. Is the beginner label something that was created during the github transition?

[aaron.ballman]

In D117832#3259547, @tstellar wrote:

A few things to note: we have not yet created the confirmed label, but that is expected to be done shortly. There is already label confusion involving whether we should use beginner (89 issues) or good first issue (29 issues), so I went with beginner in the documentation. I think we should strongly discourage users from creating arbitrary labels on their own because the labels will become useless pretty quickly if there's no consistency as to which are used, so I adjusted the docs accordingly

'good first issue' is a default label provided by GitHub, so it seems like it would be good to use it for consistency with other projects. Is the beginner label something that was created during the github transition?

I think what happened is that Bugzilla had a tag for beginner and it didn't get translated to good first issue when we migrated. See https://github.com/llvm/llvm-project/issues/51656 and https://bugs.llvm.org/show_bug.cgi?id=52314 as an example (note the keyword use in Bugzilla).

I can switch to using good first issue in the docs, but it'd be nice if we could migrate the beginner tag to good first issue and then remove the beginner tag to reduce confusion.

[• Quuxplusone]

I've suggested several rewordings, or additional wordings; please take whatever seems good to you. I don't want to block this, regardless.

• Please use [good first issue] instead of [beginner].
• It is probably a good idea to list the four "resolution/disposition" labels explicitly somewhere as a foursome: [duplicate], [invalid], [wontfix], [worksforme]. Btw, do we think [obsolete] is also a resolution/disposition? I don't know what its intent was, but I think it's plausibly a sort of compromise between [invalid] and [wontfix], right? If that's our intent, then we should color-code [obsolete] as a resolution (color #cccccc) and mention it in the same breath with the other four.

llvm/docs/BugLifeCycle.rst

54–56

Any open bug without the [confirmed] label is a bug that still needs to be triaged. When triage is complete, if the bug isn't simply being closed, the bug should be labeled [confirmed].
The goal is that a bug marked [confirmed] is in a good, actionable state.

I would prefer not to mention the [new issue] label at all, here, since it isn't consistently applied to new issues and also isn't consistently removed from triaged issues. Also, we should strive to limit the amount of "process" involved here; "add this label" is 50% less process than "add this label and remove that label," especially when "that label" was merely added by a bot to begin with.
https://github.com/llvm/llvm-project/issues?q=label%3A%22new+issue%22

Reductio ad absurdam: We could change the "new issue" bot so that it also removes the [new issue] label from any bug that already possesses the [confirmed] tag, and/or from any bug that is closed. (Right now there are 42 closed bugs with the [new issue] tag.) We could also change the bot to add the [new issue] tag to any bug that is open but lacks [confirmed]. Basically we can let the bot run free with that label, and leave the humans to ignore it. :)

63

If the reported behavior is not a bug, it is appropriate to close the issue with the label [invalid], and a comment explaining why you believe it's not a bug.

67

If the bug really can't be reproduced, it might be appropriate to close it with the label [worksforme], and a comment explaining your rationale. But this closure reason should be used very sparingly and judiciously, and probably only if the original submitter is no longer responding.

71–72

The fourth "resolution" label is [wontfix], which we use for things that are arguably deficiencies or bugs, but which for one reason or another (difficulty, ABI, open research questions) we know aren't going to be changed and further discussion is really, truly pointless.
...Ah, I see you cover this in "Resolving/Closing Bugs" below. Consider adding a hyperlink to "Resolving/Closing Bugs" in the bullet points above that talk about essentially [invalid] and [worksforme] (as opposed to adopting my comment-suggested-wording).

80

As Tom says, please use [good first issue] here instead. It is meaningful to GitHub: it populates https://github.com/llvm/llvm-project/contribute and several third-party websites aimed at new FOSS contributors.

[beginner], by contrast, came from a Bugzilla keyword during the migration. I don't know for sure what its original intent was, but I agree it seems like a synonym for [good first issue]. (And therefore I suggest we retag all 89 open [beginner] issues as [good first issue] and then burninate the [beginner] tag. But I won't do that unless someone tells me to. ;))

109

113–114

When an issue is fixed, please add a brief comment when you close the issue, pointing to the commit that fixed it. For example: "Fixed in 1b2c3d4."

If you are authoring the fix yourself, then your git commit message may include the phrase "Fixes #54321" on a line by itself. GitHub recognizes such commit messages and will automatically close the issue with a pointer to your commit, so that you don't have to.

122–125

I think this is saying: Any project member can create new labels, but we discourage it, because we don't want a proliferation of similar or single-use labels. If you are not a project member — or even if you are! — then when you want a new label created, please open a GitHub issue with the subject line "Create issue label [foo]", add the [infrastructure] label to that issue itself, and wait. (For what, exactly?)

My impression is that a better/faster alternative is to speak up in the #infrastructure channel on the Discord.

[• Quuxplusone]

llvm/docs/BugLifeCycle.rst

122–125

Oh, and every new label request should include a description of what the label is for, which will be used as its description on https://github.com/llvm/llvm-project/labels . If you're ever unsure of the meaning of a label, visit https://github.com/llvm/llvm-project/labels for more information.

(And post-migration corollary: if you know the intent of label [foo] but it's not actually documented on https://github.com/llvm/llvm-project/labels , then please edit its documentation there so that future readers will be able to find it!)

[• Quuxplusone]

llvm/docs/BugLifeCycle.rst

44

Argh, sorry for the multi-posting as I keep thinking of things to say here!

Let's make "labels" a hyperlink to https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels
which incidentally describes the meanings of GitHub's default labels.

The only one of them where it appears we're not using it as its "originalist" meaning is [question], which apparently "originally" meant something like "The maintainer has a question for the submitter" but in our case we're clearly using it to tag, duh, questions. https://github.com/llvm/llvm-project/labels/question I would be very cool with burninating [question] and merging it into [invalid]; is that too harsh? :)

I'm relatively cool with the idea of tagging everything as either [bug] or [enhancement], but it's going to make for a lot of tough calls in practice. Right now we also have [new-feature] and [missing-feature], which I think are arguably the same as [enhancement]?
How would you categorize "Your -std=c++20 mode doesn't support half of C++20" — bug, or request for someone to implement the missing features? Further, consider "This existing feature is good but it could be better," versus "This feature doesn't exist today but I would like it to exist." Are those both "enhancements" or is there a category difference worth reflecting in the labels? Basically all [clang:diagnostics] issues are both-bug-and-feature-request in this way.

[probinson]

llvm/docs/BugLifeCycle.rst

80

There's a similar debate on Discourse, https://llvm.discourse.group/t/good-first-issue-vs-beginner/5850

But I'd say: Do it. :)

[asl]

llvm/docs/BugLifeCycle.rst

54–56

As I already stated several times (and you consistently tend to forget / ignore this): the bot only puts "new label" in case if there are no labels at all. Any labels that are set on the issue will prevent labelling by the bot. So, the presence of any label is an indication that the issue was triaged. Note that only those who have write access can assign labels, so the presence of labels is an indication of triage already done by someone. As a result, no dedicated "confirmed" label is even necessary - labels other than "new issue" is such indication.

[aaron.ballman]

llvm/docs/BugLifeCycle.rst

54–56

As a result, no dedicated "confirmed" label is even necessary - labels other than "new issue" is such indication.

I strongly disagree that a lack of a new issue label indicates that something has been triaged properly. Users add labels to their bugs all the time when creating a bug -- a self confirmation is not the same thing as an independent confirmation. As examples:

https://github.com/llvm/llvm-project/issues/53335
https://github.com/llvm/llvm-project/issues/53329

These issues have labels, none of them have been triaged by anyone except the reporter. The latter one also demonstrates that the issue is not agreed upon as being confirmed.

Also, some folks are helpful and add labels to categorize issues after the initial report, but I don't believe this really constitutes a triage that confirms the report:

https://github.com/llvm/llvm-project/issues/53324
https://github.com/llvm/llvm-project/issues/53327

I continue to believe we need a confirmed label.

[aaron.ballman]

Updated based on review feedback.

llvm/docs/BugLifeCycle.rst

44

I added the link to labels and then removed the part about bug and enhancement.

80

My position is also: do it.

122–125

I think this is saying: Any project member can create new labels, but we discourage it, because we don't want a proliferation of similar or single-use labels.

Correct, that's what I'm going for.

and wait. (For what, exactly?)

For someone to add the label for you, at which point you can retrofit it onto your issue if you care that deeply.

I've updated the docs here, thanks for the suggestions (here and elsewhere).

[• Quuxplusone]

Awesome!

llvm/docs/BugLifeCycle.rst

19–20

32

84–86

If you think it would be uncontroversial, I'd add:

* Don't use both a general and a specific label; for example, bugs labeled ``c++17`` shouldn't also have ``c++``, and bugs labeled ``clang:codegen`` shouldn't also have ``clang``.

(but if you think that's opening a can of worms for this PR, then ignore me)

92

95

125

[aaron.ballman]

Updated again based on review feedback.

llvm/docs/BugLifeCycle.rst

84–86

I think this is reasonable, so I added the words. If others find it to be controversial, I can remove the bullet easily enough.

[aaron.ballman]

Ping. I'd mostly like to get the confirmed label added so it can be applied to previously confirmed bugs and start being used, but I'd like buy-in from more than one reviewer.

[probinson]

In D117832#3269513, @aaron.ballman wrote:

Ping. I'd mostly like to get the confirmed label added so it can be applied to previously confirmed bugs and start being used, but I'd like buy-in from more than one reviewer.

A search on bugzilla finds 974 issues with status Confirmed. While this is only 5%-6% of all open bugs (> 17000), it's enough to make me think that people do use it a fair amount.

I agree with @aaron.ballman that the existence of other labels does not count as confirmation; I certainly like to put relevant labels on issues I file, so that the right people will be notified, but that doesn't mean I've filed a meaningful bug (the last one I filed was promptly closed as Invalid).

So, I think a Confirmed label, meaning a second party has actively looked at the problem and reproduced or otherwise thinks it's valid, is a useful thing to have; and I'd recommend that someone propagate bugzilla Confirmed status to the new Confirmed GH label.

[• Quuxplusone]

Added some more reviewers who have used this "Confirmed" workflow in the past according to Bugzilla.

[tstellar]

+1 to adding the confirmed label.

[dblaikie]

All sounds pretty good to me

[aaron.ballman]

It sounds to me like we're all pretty happy with the direction of getting a confirmed label and I haven't heard any concerns with the proposed changes to the documentation. Assuming I don't hear back from anyone, I plan to land the doc changes tomorrow. @Quuxplusone, can I lean on you to coordinate doing the magic to add the label to GitHub and applying it to older reviews? (It sounds like we're clear to start that work "whenever" so it doesn't need to wait on the doc changes.)

[• Quuxplusone]

In D117832#3270143, @aaron.ballman wrote:

It sounds to me like we're all pretty happy with the direction of getting a confirmed label and I haven't heard any concerns with the proposed changes to the documentation. Assuming I don't hear back from anyone, I plan to land the doc changes tomorrow. @Quuxplusone, can I lean on you to coordinate doing the magic to add the label to GitHub and applying it to older reviews? (It sounds like we're clear to start that work "whenever" so it doesn't need to wait on the doc changes.)

Yes, I can apply the "confirmed" label; but I have the impression that @tstellar would prefer it was done via llvmbot's account rather than Quuxplusone's. @tstellar, how do you feel about running the following Python script this week? Or shall I just run it myself, as I did with the previous labeling frenzy?
https://github.com/Quuxplusone/BugzillaToGithub/blob/b6f7f49/labelmaker/gh-map-to-github.py
It will take about 20 minutes to run (at 1–2 seconds per GitHub API hit).

[aaron.ballman]

In D117832#3270175, @Quuxplusone wrote:

In D117832#3270143, @aaron.ballman wrote:

It sounds to me like we're all pretty happy with the direction of getting a confirmed label and I haven't heard any concerns with the proposed changes to the documentation. Assuming I don't hear back from anyone, I plan to land the doc changes tomorrow. @Quuxplusone, can I lean on you to coordinate doing the magic to add the label to GitHub and applying it to older reviews? (It sounds like we're clear to start that work "whenever" so it doesn't need to wait on the doc changes.)

Yes, I can apply the "confirmed" label; but I have the impression that @tstellar would prefer it was done via llvmbot's account rather than Quuxplusone's. @tstellar, how do you feel about running the following Python script this week? Or shall I just run it myself, as I did with the previous labeling frenzy?
https://github.com/Quuxplusone/BugzillaToGithub/blob/b6f7f49/labelmaker/gh-map-to-github.py
It will take about 20 minutes to run (at 1–2 seconds per GitHub API hit).

I have created the confirmed label in GitHub (https://github.com/llvm/llvm-project/labels?page=3&sort=name-asc) and commit the changes here in f3e22946e5c573eaa9b05d197d0c1036b21978fb.

Thanks, all!

llvm/docs/BugLifeCycle.rst

 ====================================================================
 
 We aim to achieve a basic level of consistency in how reported bugs evolve from
 being reported, to being worked on, and finally getting closed out. The
 consistency helps reporters, developers and others to gain a better
 understanding of what a particular bug state actually means and what to expect
 might happen next.
 
 At the same time, we aim to not over-specify the life cycle of bugs in the
-`the LLVM Bug Tracking System <https://bugs.llvm.org/enter_bug.cgi>`_, as the
-overall goal is to make it easier to work with and understand the bug reports.
+`the LLVM Bug Tracking System <https://github.com/llvm/llvm-project/issues>`_,
+as the overall goal is to make it easier to work with and understand the bug
+reports.
 
 The main parts of the life cycle documented here are:
 
 #. `Reporting`_
 #. `Triaging`_
 #. `Actively working on fixing`_
 #. `Closing`_
 
-Furthermore, some of the metadata in the bug tracker, such as who to notify on
-newly reported bugs or what the breakdown into products & components is we use,
-needs to be maintained. See the following for details:
+Furthermore, some of the metadata in the bug tracker, such as what labels we
+use needs to be maintained. See the following for details:
 
-#. `Maintenance of Bug products/component metadata`_
-#. `Maintenance of cc-by-default settings`_
+#. `Maintenance of metadata`_
 
 
 .. _Reporting:
 
 Reporting bugs
 ==============
 
 See :doc:`HowToSubmitABug` on further details on how to submit good bug reports.
 
-Make sure that you have one or more people on cc on the bug report that you
-think will react to it. We aim to automatically add specific people on cc for
-most products/components, but may not always succeed in doing so.
-
-If you know the area of LLVM code the root cause of the bug is in, good
-candidates to add as cc may be the same people you'd ask for a code review in
-that area. See :ref:`finding-potential-reviewers` for more details.
-
+You can apply labels to the bug to provide extra information to make the bug
+easier to discover, such as a label for the part of the project the bug
+pertains to. Please also add a label specifying whether the issue is a ``bug``
+or an ``enhancement``.
 
 .. _Triaging:
 
 Triaging bugs
 =============
 
-Bugs with status NEW indicate that they still need to be triaged.
-When triage is complete, the status of the bug is moved to CONFIRMED.
+Bugs with a ``new issue`` label indicate that the issue stills need to be
+triaged. When triage is complete, the ``new issue`` label should be replaced by
+a ``confirmed`` label, unless the issue is being closed.
 
 The goal of triaging a bug is to make sure a newly reported bug ends up in a
 good, actionable, state. Try to answer the following questions while triaging.
 
 * Is the reported behavior actually wrong?
 
   * E.g. does a miscompile example depend on undefined behavior?
 
 * Can you easily reproduce the bug?
 
   * If not, are there reasonable excuses why it cannot easily be reproduced?
 
 * Is it related to an already reported bug?
 
-  * Use the "See also"/"depends on"/"blocks" fields if so.
-  * Close it as a duplicate if so, pointing to the issue it duplicates.
+  * Close it as a duplicate by adding the label ``duplicate`` if so, with a
+    comment pointing to the issue it duplicates.
 
 * Are the following fields filled in correctly?
 
-  * Product
-  * Component
   * Title
+  * Description
+  * Labels
 
-* CC others not already cc’ed that you happen to know would be good to pull in.
-* Add the "beginner" keyword if you think this would be a good bug to be fixed
+* Add the ``beginner`` label if you think this would be a good bug to be fixed
   by someone new to LLVM.
 
 .. _Actively working on fixing:
 
 Actively working on fixing bugs
 ===============================
 
 Please remember to assign the bug to yourself if you're actively working on
 fixing it and to unassign it when you're no longer actively working on it.  You
-unassign a bug by setting the Assignee field to "[e-mail]".
+unassign a bug by removing the person from the the ``Assignees`` field.
 
 .. _Closing:
 
 Resolving/Closing bugs
 ======================
 
-For simplicity, we only have 1 status for all resolved or closed bugs:
-RESOLVED.
-
 Resolving bugs is good! Make sure to properly record the reason for resolving.
 Examples of reasons for resolving are:
 
 * Revision NNNNNN fixed the bug.
 * The bug cannot be reproduced with revision NNNNNN.
 * The circumstances for the bug don't apply anymore.
-* There is a sound reason for not fixing it (WONTFIX).
+* There is a sound reason for not fixing it (add the ``wontfix`` label).
 * There is a specific and plausible reason to think that a given bug is
   otherwise inapplicable or obsolete.
 
   * One example is an old open bug that doesn't contain enough information to
     clearly understand the problem being reported (e.g. not reproducible). It is
-    fine to resolve such a bug e.g. with resolution WORKSFORME and leaving a
+    fine to resolve such a bug e.g. with the ``worksforme`` label and leaving a
     comment to encourage the reporter to reopen the bug with more information
     if it's still reproducible on their end.
 
-If a bug is resolved, please fill in the revision number it was fixed in in the
-"Fixed by Commit(s)" field.
-
-
-.. _Maintenance of Bug products/component metadata:
-
-Maintenance of products/components metadata
-===========================================
-
-Please raise a bug against "Bugzilla Admin"/"Products" to request any changes
-to be made to the breakdown of products & components modeled in Bugzilla.
+If a bug is resolved, please add the revision number it was fixed in as a
+comment when closing the issue.
 
 
-.. _Maintenance of cc-by-default settings:
+.. _Maintenance of metadata:
 
-Maintenance of cc-by-default settings
-=====================================
+Maintenance of metadata
+=======================
 
-Please raise a bug against "Bugzilla Admin"/"Products" to request any changes
-to be made to the cc-by-default settings for specific components.
+Please add the ``infrastructure`` label to request any changes to be made to
+the labels used by the issue tracker instead of creating your own label. This
+increases consistency when triaging and helps reduce the proliferation of
+labels.