Introduce bug life cycle documentation.

[kristof.beyls]

Summary

Document what is expected during:

• triaging
• actively working on a bug
• closing

Also document how we maintain:

• product/component breakdown
• default-cc lists per component

[rnk]

docs/BugLifeCycle.rst

68 ↗

(On Diff #171053)

These seem to render as smart quotes. As long as it's UTF-8, this is probably fine, but they scare me. :) Up to you if you want to keep them.

103–104 ↗

(On Diff #171053)

What about this scenario:

• User reports bug with vague reproduction instructions
• Bug languishes for one year
• Developer does a pass over open bugs, tries to reproduce, but is unable to because there never was a clear, confirmed repro in the older version of llvm.

I think our policy should encourage us to close such bugs with some sort of "stale / instructions unclear / needsinfo" status along with a request for more information. The point is to put the bug into a state that says that no further action will be taken by LLVM developers unless the original reporter does something. For old bugs, chances are they won't.

I think as long as we encourage reporters to reopen if they have more info and can still reproduce the bug, this won't be discouraging to reporters. If the bug languished for a year, it's best to acknowledge that nothing happened and nothing is likely to happen without more information.

I believe this would be a better policy than asking developers to ping stale bugs for more info from the OP while leaving the bug open. That just leaves the bug open and means it will come up during the next bug scrub, requiring more developer time to close it out if there was no response.

Looking at our current resolution statuses, I guess I'd close bugs that we were never able to confirm were bugs with "WORKSFORME".

[tstellar]

docs/BugLifeCycle.rst

87–88 ↗

(On Diff #171053)

The alternative would be using the "Assigned" and "New" bug states to indicate that a bug is assigned or not assigned. The downside of this is that it's an extra step that people might forget when assigning bugs to themselves, but the advantage is that we wouldn't need the pseudo accounts for unassigned bugs which make the bugs more invisible than if they have a real assignee.

If we do want to keep using the pseudo accounts, I think we should just pick one to use ([e-mail]) and document that here.

[rnk]

docs/BugLifeCycle.rst

103–104 ↗

(On Diff #171053)

My previous comment is my opinion, and I don't think it reflects the consensus of the last llvm-dev discussion we had about this, so don't let it block the document, which I think is important. We can keep discussing policy after we have a doc.

[kristof.beyls]

docs/BugLifeCycle.rst

68 ↗

(On Diff #171053)

D'Oh - I'll fix that to use ASCII quotes.

87–88 ↗

(On Diff #171053)

I think we could only remove the pseudo accounts and introduce real assignees if we could find volunteers for every single component that exists to become the default assignee. It seems uncertain to me we'll always have volunteers for that for all components, given the state we are in today.
Therefore, I think I want to opt for continuing to use the pseudo account for now. If we start having evidence that we can assume there will always be a volunteer to be default-assigned to for every component, we can reconsider this.
I agree that we only need a single pseudo account; not two ([e-mail] is the other one I've seen. Not sure if there are more).

103–104 ↗

(On Diff #171053)

Looking back at the last llvm-dev discussion, my impression is actually that most people were in favour of something along the lines of what you're proposing. Let me have a stab at concisely writing something up that hopefully encodes consensus.

[RKSimon]

docs/BugLifeCycle.rst

96 ↗

(On Diff #171053)

Do we need to distinguish between resolving and closing a bug? We have verified/closed states but AFAICT these are barely used.

[kristof.beyls]

docs/BugLifeCycle.rst

96 ↗

(On Diff #171053)

Good point.
As far as I can see, we don't really attach much difference in meaning to the "RESOLVED", "VERIFIED" and "CLOSED" states.
Therefore, my opinion right now is that we're probably best of to just remove the "RESOLVED" and "VERIFIED" states from our bugzilla instance and only keep the "CLOSED" state, making the workflows simpler.
I do think that this change can be made mostly orthogonally to completing this documentation. If/When we remove the "VERIFIED" and "RESOLVED" states, the documentation becomes less open for interpretation.
What do others think about this?

[jhenderson]

docs/BugLifeCycle.rst

46–47 ↗

(On Diff #171053)

Maybe this should contain instructions on how to identify a good person to add as a CC? I imagine a good basis is the same as the people you'd add as reviewers for a change in the area.

96 ↗

(On Diff #171053)

This is purely a personal opinion, but I would prefer to keep RESOLVED, above VERIFIED or CLOSED, partly because that's how I'm used to using bugzilla (I've always used RESOLVED). However, if consensus is against me on this, I don't mind. If we went down this route, you should probably change "closing" to "resolving" here and elsewhere.

[kristof.beyls]

docs/BugLifeCycle.rst

46–47 ↗

(On Diff #171053)

I think we should aim to have at least one and ideally a few people by default added on CC for each component, as I expect that users that are not developers may find it hard to follow the guidelines we use for finding code reviewers (look at CODE_OWNERs; use svn/git blame on the code area affected).
Nonetheless, it may indeed be useful to say something along the lines of "If you know the area of LLVM code the root cause of the bug is in, good candidates to add as cc on the bug report may be the same people you'd ask for code review on that area. See Phabricator.html#requesting-a-review-via-the-web-interface for more details."

96 ↗

(On Diff #171053)

I have no strong preference myself on RESOLVED vs CLOSED.
Having looked at the Bugzilla documentation for a few minutes just now on how to remove possible values for field "status", it seems that the "RESOLVED" value cannot be deleted, whereas "CLOSED" and "VERIFIED" can be deleted. Therefore, I think there is a strong technical argument to just decide that "RESOLVED" is the final bug status we retain, if we decide to only have 1, not 3 final bug statuses.

[kristof.beyls]

docs/BugLifeCycle.rst

96 ↗

(On Diff #171053)

I have now updated the proposed documentation here as if we've decided to remove the VERIFIED and CLOSED states.
After looking into what it'd take to do so, it seems pretty straightforward for a bugzilla admin to remove the VERIFIED and CLOSED states.

103–104 ↗

(On Diff #171053)

I've made an attempt at concisely writing up something the conveys the intent of what I believe the consensus is.

[tstellar]

LGTM. Thanks for writting this up.

[kristof.beyls]

Add a sentence on statuses involved when triaging.

[jhenderson]

LGTM too.

llvm/trunk/docs/BugLifeCycle.rst

+===================
+LLVM Bug Life Cycle
+===================
+
+.. contents::
+   :local:
+
+
+
+Introduction - Achieving consistency in how we deal with bug reports
+====================================================================
+
+We aim to achieve a basic level of consistency in how reported bugs evolve from
+being reported, to being worked on, and finally getting closed out. The
+consistency helps reporters, developers and others to gain a better
+understanding of what a particular bug state actually means and what to expect
+might happen next.
+
+At the same time, we aim to not over-specify the life cycle of bugs in the
+`the LLVM Bug Tracking System <https://bugs.llvm.org/enter_bug.cgi>`_, as the
+overall goal is to make it easier to work with and understand the bug reports.
+
+The main parts of the life cycle documented here are:
+
+#. `Reporting`_
+#. `Triaging`_
+#. `Actively working on fixing`_
+#. `Closing`_
+
+Furthermore, some of the metadata in the bug tracker, such as who to notify on
+newly reported bugs or what the breakdown into products & components is we use,
+needs to be maintained. See the following for details:
+
+#. `Maintenance of Bug products/component metadata`_
+#. `Maintenance of cc-by-default settings`_
+
+
+.. _Reporting:
+
+Reporting bugs
+==============
+
+See :doc:`HowToSubmitABug` on further details on how to submit good bug reports.
+
+Make sure that you have one or more people on cc on the bug report that you
+think will react to it. We aim to automatically add specific people on cc for
+most products/components, but may not always succeed in doing so.
+
+If you know the area of LLVM code the root cause of the bug is in, good
+candidates to add as cc may be the same people you'd ask for a code review in
+that area. See :ref:`finding-potential-reviewers` for more details.
+
+
+.. _Triaging:
+
+Triaging bugs
+=============
+
+Bugs with status NEW indicate that they still need to be triaged.
+When triage is complete, the status of the bug is moved to CONFIRMED.
+
+The goal of triaging a bug is to make sure a newly reported bug ends up in a
+good, actionable, state. Try to answer the following questions while triaging.
+
+* Is the reported behavior actually wrong?
+
+  * E.g. does a miscompile example depend on undefined behavior?
+
+* Can you easily reproduce the bug?
+
+  * If not, are there reasonable excuses why it cannot easily be reproduced?
+
+* Is it related to an already reported bug?
+
+  * Use the "See also"/"depends on"/"blocks" fields if so.
+  * Close it as a duplicate if so, pointing to the issue it duplicates.
+
+* Are the following fields filled in correctly?
+
+  * Product
+  * Component
+  * Title
+
+* CC others not already cc’ed that you happen to know would be good to pull in.
+* Add the "beginner" keyword if you think this would be a good bug to be fixed
+  by someone new to LLVM.
+
+.. _Actively working on fixing:
+
+Actively working on fixing bugs
+===============================
+
+Please remember to assign the bug to yourself if you're actively working on
+fixing it and to unassign it when you're no longer actively working on it.  You
+unassign a bug by setting the Assignee field to "[e-mail]".
+
+.. _Closing:
+
+Resolving/Closing bugs
+======================
+
+For simplicity, we only have 1 status for all resolved or closed bugs:
+RESOLVED.
+
+Resolving bugs is good! Make sure to properly record the reason for resolving.
+Examples of reasons for resolving are:
+
+* Revision NNNNNN fixed the bug.
+* The bug cannot be reproduced with revision NNNNNN.
+* The circumstances for the bug don't apply anymore.
+* There is a sound reason for not fixing it (WONTFIX).
+* There is a specific and plausible reason to think that a given bug is
+  otherwise inapplicable or obsolete.
+
+  * One example is an old open bug that doesn't contain enough information to
+    clearly understand the problem being reported (e.g. not reproducible). It is
+    fine to resolve such a bug e.g. with resolution WORKSFORME and leaving a
+    comment to encourage the reporter to reopen the bug with more information
+    if it's still reproducable on their end.
+
+If a bug is resolved, please fill in the revision number it was fixed in in the
+"Fixed by Commit(s)" field.
+
+
+.. _Maintenance of Bug products/component metadata:
+
+Maintenance of products/components metadata
+===========================================
+
+Please raise a bug against "Bugzilla Admin"/"Products" to request any changes
+to be made to the breakdown of products & components modeled in Bugzilla.
+
+
+.. _Maintenance of cc-by-default settings:
+
+Maintenance of cc-by-default settings
+=====================================
+
+Please raise a bug against "Bugzilla Admin"/"Products" to request any changes
+to be made to the cc-by-default settings for specific components.

llvm/trunk/docs/Phabricator.rst

 * Add comments about the changes in the new diff. Click *Save*.
 
 Choosing reviewers: You typically pick one or two people as initial reviewers.
 This choice is not crucial, because you are merely suggesting and not requiring
 them to participate. Many people will see the email notification on cfe-commits
 or llvm-commits, and if the subject line suggests the patch is something they
 should look at, they will.
 
+
+.. _finding-potential-reviewers:
+
+Finding potential reviewers
+---------------------------
+
 Here are a couple of ways to pick the initial reviewer(s):
 
 * Use ``svn blame`` and the commit log to find names of people who have
   recently modified the same area of code that you are modifying.
 * Look in CODE_OWNERS.TXT to see who might be responsible for that area.
 * If you've discussed the change on a dev list, the people who participated
   might be appropriate reviewers.
 

llvm/trunk/docs/index.rst

.. toctree::
    Contributing
    DeveloperPolicy
    Projects
    LLVMBuild
    HowToReleaseLLVM
    Packaging
    ReleaseProcess
    Phabricator
+   BugLifeCycle
 
 :doc:`Contributing`
    An overview on how to contribute to LLVM.
 
 :doc:`DeveloperPolicy`
    The LLVM project's policy towards developers and their contributions.
 
 :doc:`Projects`

 
 :doc:`Packaging`
    Advice on packaging LLVM into a distribution.
 
 :doc:`Phabricator`
    Describes how to use the Phabricator code review tool hosted on
    http://reviews.llvm.org/ and its command line interface, Arcanist.
 
+:doc:`BugLifeCycle`
+   Describes how bugs are reported, triaged and closed.
+
 Community
 =========
 
 LLVM has a thriving community of friendly and helpful developers.
 The two primary communication mechanisms in the LLVM community are mailing
 lists and IRC.
 
 Mailing Lists