[docs] Adding a Support Policy

[rengolin]

Summary

As discussed in the mailing list [1-4], we need a separation of support
tiers when requiring support from the whole community versus a
sub-community. Essentially, if a sub-community is active enough and
takes maintenance into their own internal costs without affecting other
parts of the community's maintenance costs, then code that is not
immediately relevant to all parts (ie. not released, actively tested,
etc) can still find its way into the LLVM main repository without major
pain points.

The main benefit is to reduce the maintenance cost that those
sub-communities have outside of LLVM (for example, in duplicating common
code, applying the same patches on top of multiple user repositories or
downstream projects).

This document outlines the components and responsibilities of the
sub-communities with regards to maintenance costs and how they affect
the rest of the community.

It also adds an addendum on removal policies, which expand the existing
"new target removal" policy into something more generic, to encompass
any piece of code, scripts or documents in the repository.

[1] http://lists.llvm.org/pipermail/llvm-dev/2020-October/146249.html
[2] http://lists.llvm.org/pipermail/llvm-dev/2020-November/146335.html
[3] http://lists.llvm.org/pipermail/llvm-dev/2020-October/146138.html
[4] http://lists.llvm.org/pipermail/llvm-dev/2020-November/146298.html

[mehdi_amini]

llvm/docs/SupportPolicy.rst

16

I wouldn't use the word "projects" here, that is a bit confusing to me (I think of projects as "clang", "lld", "mlir", ...)

Maybe: There are, however, other components (scripts/utilities for example) within the main repository that either ...

[rengolin]

llvm/docs/SupportPolicy.rst

16

Ah, yes! I use components below. Good catch.

[rengolin]

• Adding page link to "Getting Involved" page.
• Fixing some formatting errors (Sphinx reports no warnings on my box).

[ctetreau]

I've suggested some changes, but this seems reasonable to me.

llvm/docs/SupportPolicy.rst

29

I think "peripheral" is fine. Informally, we want a "main things tier" and "side things tier", and I think "peripheral" is a suitably dignified word for "side things".

67

Wording here is a bit weird.

109

I'm not quite sure what the best way to express this is, but I feel like things in the peripheral tier must have at least one active subcommunity upstream or at least two disjoint active subcommunities downstream.

I.E. "somebody uses it in upstream" or "two separate downstreams use it"

111

The way I read this, I as a user of a peripheral tier component can open a bug against said component on bugs.llvm.org, and that this bug is valid and is expected to be addressed.

Is this correct? If so, maybe this should be explicitly codified here.

172

[GMNGeoffrey]

Overall LGTM, with some wording nits. One thing I think is missing is guidance for how a peripheral component gets added. Experimental targets already have their own documentation, and my impression is that editor bindings are just a matter of sending a patch (which seems reasonable to me). What about something like:

To add a new peripheral component not covered under an existing policy, send an RFC to the appropriate dev list proposing its addition and explaining how it will meet the support requirements listed here.

llvm/docs/SupportPolicy.rst

20

Nit: "nor are they"

20

29

+1 peripheral SGTM

67

Wording here is a bit weird.

74–75

108–109

Nit: s/remaining/residing/ since this could be used to consider something moving *into* the main repository as well, where "remaining" doesn't make much sense.

Also I don't really understand how one would count number of subcommunities per my comment above.

109

How are we measuring *number* of communities? Wouldn't we always have one sub-community that is "the sub-community that cares about thing X"?

114–117

"Negatively affect" seems broader than "break or invalidate", so I think it's important to have that included for core-tier as well. Added as a separate bullet point, since an immediate revert isn't necessarily the best course of action for such an issue. I'm thinking of the discussion we had about putting something under llvm/utils vs at the top-level utils/ and how that affects commit emails.

127–128

I think this note about the level of support required scaling with system complexity would be a nice thing to hold over from the original 3-tier policy proposal

130

"incursion" sounds like they're invading :-D

148–149

Wording is a bit weird here. I'm not quite sure what you're trying to say.

[ctetreau]

llvm/docs/SupportPolicy.rst

109

How are we measuring *number* of communities? Wouldn't we always have one sub-community that is "the sub-community that cares about thing X"?

This would be largely honor-system. In the RFC, part of the justification would be spelling out who the multiple subcommunities are. With the bazel example, the original RFC states "we use it internally at google, but tensorflow also has a parallel version of this". In this case "llvm devs at google" and "llvm devs for the tensorflow project" are the 2 subcommunities. I would think that a representative from each subcommunity should respond to the RFC stating that they agree to be maintainers (or at least that they care about it). For the bazel example, Google has clearly stated interest. A Tensorflow maintainer should respond to the RFC stating that they do in fact care (I don't know if this has happened).

The idea here being: "If only one org is using this thing in their downstream, and nobody else wants it, why would the community accept it?"

I guess perhaps the "one subcommunity upstream" requirement is kind of pointless. If it's not being used by a core-tier component, then by definition it's only used by downstreams or out of tree users of llvm. And if it were being used by a core-tier component, then I would think it should also be core-tier.

[rengolin]

Updating simple suggestions / fixups.

[rengolin]

llvm/docs/SupportPolicy.rst

130

:D

[rengolin]

llvm/docs/SupportPolicy.rst

109

I think I consider "sub-community" as a set, so the union of two sub-communities is still a "sub-community". I also don't see how we could "count" them, as there is a lot of overlap.

Worst case we'd be forcing people to create bogus splits to "increase the number of sub-communities" to pass a threshold.

I'm simplifying the second bullet to avoid confusion. In the end, the points I wanted to make are already there on the second (must not) bullet section.

111

Not just a user for component X, but developers on core tier stuff as well as developers of component Y.

It is the responsibility of the sub-communities to maintain their code so that their quality is at the very least "invisible" to other users, but optimally it should behave well not "negatively affect" others.

148–149

I'm trying to say that we're not all looking hard to destroy other people's work, and this document is mostly to allow us to do so when it's affecting the rest of the community. I have removed the weird paragraph and have simplified the last one. I hope that's clearer.

[rengolin]

Rewriting some confusing paragraphs.

[GMNGeoffrey]

llvm/docs/SupportPolicy.rst

109

I think I consider "sub-community" as a set, so the union of two sub-communities is still a "sub-community". I also don't see how we could "count" them, as there is a lot of overlap.

Yes this was my reasoning as well

148–149

I think you kept the paragraph I find confusing. Maybe my issue is with the phrasing "impose degradation" which sounds odd to me. It seems like mostly this is covered by the "must nots" above, but maybe you think it still is useful to include in this section as well? I took a swing at rewording

For code to remain in the repository, its presence must not impose an undue burden on maintaining other components (core or peripheral).

[rengolin]

In D90761#2374520, @GMNGeoffrey wrote:

Overall LGTM, with some wording nits. One thing I think is missing is guidance for how a peripheral component gets added. Experimental targets already have their own documentation, and my impression is that editor bindings are just a matter of sending a patch (which seems reasonable to me). What about something like:

To add a new peripheral component not covered under an existing policy, send an RFC to the appropriate dev list proposing its addition and explaining how it will meet the support requirements listed here.

Sorry I missed this. I have now added a whole new "Inclusion Policy" section, which starts with your phrase, but adds a bit more of the mechanics to be clear what to expect.

llvm/docs/SupportPolicy.rst

109

The idea here being: "If only one org is using this thing in their downstream, and nobody else wants it, why would the community accept it?"

I think the main idea of the policy is to say that it's ok to have side-projects in the main repo if they follow the guidelines and don't break other people's stuff. Basically "be excellent to each other".

If only one downstream user is using it and that user always fixes stuff whenever it breaks and keeps it up-to-date, then this follows the spirit of the policy in two ways:

1. It is being "excellent" to all others.
2. It allows people that haven't used that component to experiment with it without having to pay the cost of downstream maintenance (and learning), which is often prohibitive.

Point 2 above encourages sub-community growth, and if/when the sub-community is large enough, we may even make it core.

Case 1: Autoconf vs. CMake.

When I started working on LLVM, the default was autoconf and CMake support was patchy. I had rarely used CMake until then, but when I did I wonder why the heck were we still using autoconf. After enough people gathered behind CMake, we moved official support to CMake (after fixing all its bugs).

This would be a typical tier-inversion, with autoconf moving down to peripheral tier, while CMake went up to core.

Later on, when I was building Arm buildbots, I realised we were still supporting autoconf. Back then, most Arm developers were still using autoconf to build because CMake support on Arm was still patchy. So I fixed what I could and took the conscious decision to not create any autoconf Arm bot ever. Other bot maintainers have moved their existing autoconf bots to CMake and in due time, there were none.

That conscious decision by the community is what drove a lot of people to fix most CMake bugs and improve it considerably. This is when we decided to drop support for autoconf.

Case 2: GN

I had no idea it existed until George proposed to include Bazel. I still have no idea what it does or why it exists, but I still literally don't care. I don't know if the sub-community is one person or half of the developers, and that doesn't matter because I have never been affected by it.

It can continue to exist for as long as it does and I probably won't even notice. This means to me that either those files are so stable that no one ever changes and it never breaks (similar to most editor configuration) or the sub-community is extremely responsive. Either way, they're already following the policy we're trying to create.

[rengolin]

• Adding inclusion policy section
• Reword phrase in deprecation policy

[rengolin]

Sorry, s/George/Geoffrey/. :S

[GMNGeoffrey]

LGTM, but please wait for others' review, obviously :-)

llvm/docs/SupportPolicy.rst

145

I think it's worth saying that simple things like editor configs don't need to go through this process. I believe they haven't in the past and it seems like a lot of overhead to add.

156

[ctetreau]

llvm/docs/SupportPolicy.rst

109

I guess I have a personal aversion to an excess of cruft.

I suppose the proposal for inclusion will justify the "who cares", and as long as "someone does", it meets the criteria for inclusion.

[mehdi_amini]

LGTM from me on the principle, I didn't get into detailed wordsmith. Overall I would have been more concise and lay down the general principle rather than trying to prescribe this level of details (like the "Reinstatement" section for example), I believe that's also more in line with how we've been operating / documenting things in the past, but I don't really mind it either.
I suspect we can incrementally fix/improve anything as we go as well.

[rengolin]

llvm/docs/SupportPolicy.rst

109

I guess I have a personal aversion to an excess of cruft.

I suppose the proposal for inclusion will justify the "who cares", and as long as "someone does", it meets the criteria for inclusion.

That's what I hope, yes.

[rengolin]

• Include a short phrase on inclusion policy, as requested.
• Fixed some typos

[rengolin]

In D90761#2380282, @mehdi_amini wrote:

LGTM from me on the principle, I didn't get into detailed wordsmith. Overall I would have been more concise and lay down the general principle rather than trying to prescribe this level of details (like the "Reinstatement" section for example), I believe that's also more in line with how we've been operating / documenting things in the past, but I don't really mind it either.
I suspect we can incrementally fix/improve anything as we go as well.

I hoped for something leaner, too. But this policy received some opposing views and multiple requests for clarification, which led to some bloat on the text.

I tried to only write things that I know we have been doing all along, avoiding creating new rules for anything, so hopefully, it will still convey the same meaning.

I'd also be happy if someone could simplify it later. I just wanted to get something out there so that Geoffrey can continue his proposal.

I'll wait for more people to give their approvals, to make sure the document reflect the community.

@lattner @echristo @reames what do you guys think?

[lattner]

This look really really great Renato. After this lands, please also prepare a patch (~paragraph length) for the main DeveloperPolicy doc that gives the high level idea of what is going on, and links to this doc for more information.

Thank you so much for driving this -- this is a huge step forward for LLVM!

-Chris

[rengolin]

Thanks Chris! Will do.

[rengolin]

Sorry, forgot the review text on the commit message:
Closed by 25ba6b2bcd1

[echristo]

This looks fine to me for now. I really think it's too wordy and too elaborate and could be about 1/10th as long.

-eric

llvm/docs/GettingInvolved.rst

    Contributing
    DeveloperPolicy
    CodeReview
+   SupportPolicy
    SphinxQuickstartTemplate
    Phabricator
    HowToSubmitABug

 :doc:`CodeReview`
    The LLVM project's code-review process.
 
+:doc:`SupportPolicy`
+   The LLVM support policy for core and non-core components.
+
 :doc:`SphinxQuickstartTemplate`
   A template + tutorial for writing new Sphinx documentation. It is meant
   to be read in source form.

llvm/docs/SupportPolicy.rst

+=============================
+LLVM Community Support Policy
+=============================
+
+As a compilation infrastructure, LLVM has multiple types of users, both
+downstream and upstream, of many combinations of its projects, tools and
+libraries.
+
+There is a core part of it that encompass the implementation of the compiler
+(front/middle/back ends), run-time libraries (RT, C++, OpenMP, etc) and
+associated tools (debugger, linker, object file manipulation, etc). These
+components are present in the public release on our supported architectures
+and operating systems and the whole community must maintain and care about.
+
+There are, however, other components within the main repository that either
+cater to a specific sub-community of LLVM (upstream or downstream) or
+help parts of the community to integrate LLVM into their own development tools
+or external projects. Those parts of the main repository don't always have
+rigorous testing like the core parts, nor are they validated and shipped with
+our public upstream releases.
+
+Even not being a core part of the project, we have enough sub-communities
+needing those changes with enough overlap that having them in the main
+repository is beneficial to minimise the repetition of those changes in all
+the external repositories that need them.
+
+But the maintenance costs of such diverse ecosystem is non trivial, so we divide
+the level of support in two tiers: core and peripheral, with two
+different levels of impact and responsibilities. Those tiers refer only to the
+main repository (``llvm-project``) and not the other repositories in our git
+project, unless explicitly stated.
+
+Regardless of the tier, all code must follow the existing policies on quality,
+reviews, style, etc.
+
+Core Tier
+=========
+
+The core tier encompasses all of the code in the main repository that is
+in production, is actively tested and released in a regular schedule, including
+core LLVM APIs and infrastructure, front/middle/back-ends, run-time libraries,
+tools, etc.
+
+It is the responsibility of **every** LLVM developer to care for the core tier
+regardless of where their work is applied to.
+
+What is covered
+---------------
+
+The core tier is composed of:
+ * Core code (``llvm-project``) present in official releases and buildbots:
+   compiler, debugger, linker, libraries, etc, including infrastructure code
+   (table-gen, lit, file-check, unit-tests, etc).
+ * Build infrastructure that creates releases and buildbots (CMake, scripts).
+ * `Phabricator <https://github.com/llvm/phabricator>`_ and
+   `buildbot <https://github.com/llvm/llvm-zorg>`_ infrastructure.
+ * The `test-suite <https://github.com/llvm/llvm-test-suite>`_.
+
+Requirements
+------------
+
+Code in this tier must:
+ * Keep official buildbots green, with warnings on breakages being emailed to
+   all affected developers. Those must be fixed as soon as possible or patches
+   must be reverted, as per review policy.
+ * Bit-rot of a component in the core tier will result in that component being
+   downgraded to the peripheral tier or being removed. Sub-communities can
+   avoid this by fixing all raised issues in a timely manner.
+
+Peripheral Tier
+===============
+
+The peripheral tier encompass the parts of LLVM that cater to a specific
+sub-community and which don't usually affect the core components directly.
+
+This includes experimental back-ends, disabled-by-default options and
+alternative paths (work-in-progress replacements) in the same repository, as
+well as separate efforts to integrate LLVM development with local practices.
+
+It is the responsibility of each sub-community to care about their own parts
+and the intersection of that with the core tier and other peripheral parts.
+
+There are three main groups of code that fit in this category:
+ * Code that is making its way into LLVM, via the `experimental <https://llvm.org/docs/DeveloperPolicy.html#introducing-new-components-into-llvm>`_
+   roadmap or similar efforts.
+ * Code that is making its way out of LLVM, via deprecation, replacement or
+   bit-rot, and will be removed if the sub-community that cares about it
+   cannot maintain it.
+ * Code that isn't meant to be in LLVM core and can coexist with the code in
+   the core tier (and others in the peripheral tier) long term, without causing
+   breakages or disturbances.
+
+What is covered
+---------------
+
+The peripheral tier is composed of:
+ * Experimental targets and options that haven't been enable by default yet.
+ * Main repository projects that don't get released or regularly tested.
+ * Legacy tools and scripts that aren't used in upstream validation.
+ * Alternative build systems (ex. GN, Bazel) and related infrastructure.
+ * Tools support (ex. gdb scripts, editor configuration, helper scripts).
+
+Requirements
+------------
+
+Code in this tier must:
+ * Have a clear benefit for residing in the main repository, catering to an
+   active sub-community (upstream or downstream).
+ * Be actively maintained by such sub-community and have its problems addressed
+   in a timely manner.
+
+Code in this tier must **not**:
+ * Break or invalidate core tier code or infrastructure. If that happens
+   accidentally, reverting functionality and working on the issues offline
+   is the only acceptable course of action.
+ * Negatively affect development of core tier code, with the sub-community
+   involved responsible for making changes to address specific concerns.
+ * Negatively affect other peripheral tier code, with the sub-communities
+   involved tasked to resolve the issues, still making sure the solution doesn't
+   break or invalidate the core tier.
+ * Impose sub-optimal implementation strategies on core tier components as a
+   result of idiosyncrasies in the peripheral component.
+ * Have build infrastructure that spams all developers about their breakages.
+ * Fall into disrepair. This is a reflection of lack of an active sub-community
+   and will result in removal.
+
+Code in this tier should:
+ * Have infrastructure to test, whenever meaningful, with either no warnings or
+   notification contained within the sub-community.
+ * Have support and testing that scales with the complexity and resilience of
+   the component, with the bar for simple and gracefully-degrading components
+   (such as editor bindings) much lower than for complex components that must
+   remain fresh with HEAD (such as experimental back-ends or alternative build
+   systems).
+ * Have a document making clear the status of implementation, level of support
+   available, who the sub-community is and, if applicable, roadmap for inclusion
+   into the core tier.
+ * Be restricted to a specific directory or have a consistent pattern (ex.
+   unique file suffix), making it easy to remove when necessary.
+
+Inclusion Policy
+================
+
+To add a new peripheral component, send an RFC to the appropriate dev list
+proposing its addition and explaining how it will meet the support requirements
+listed above. Different types of components could require different levels of
+detail. when in doubt, ask the community what's the best approach.
+
+Inclusion must reach consensus in the RFC by the community and the approval of
+the corresponding review (by multiple members of the community) is the official
+note of acceptance.
+
+After merge, there often is a period of transition, where teething issues on
+existing buildbots are discovered and fixed. If those cannot be fixed straight
+away, the sub-community is responsible for tracking and reverting all the
+pertinent patches and retrying the inclusion review.
+
+Once the component is stable in tree, it must follow this policy and the
+deprecation rules below apply.
+
+Due to the uncertain nature of inclusion, it's advisable that new components
+are not added too close to a release branch. The time will depend on the size
+and complexity of the component, so adding release and testing managers on the
+RFC and review is strongly advisable.
+
+Deprecation Policy
+==================
+
+The LLVM code base has a number of files that aren't being actively maintained.
+But not all of those files are obstructing the development of the project and
+so it remains in the repository with the assumption that it could still be
+useful for downstream users.
+
+For code to remain in the repository, its presence must not impose an undue
+burden on maintaining other components (core or peripheral).
+
+Warnings
+--------
+
+There are multiple types of issues that might trigger a request for deprecation,
+including (but not limited to):
+
+ * Changes in a component consistently break other areas of the project.
+ * Components go broken for long periods of time (weeks or more).
+ * Clearly superior alternatives are in use and maintenance is painful.
+ * Builds and tests are harder / take longer, increasing the cost of
+   maintenance, overtaking the perceived benefits.
+
+If the maintenance cost is higher than it is acceptable by the majority of
+developers, it means that either the sub-community is too small (and the extra
+cost should be paid locally), or not active enough (and the problems won't be
+fixed any time soon). In either case, removal of such problematic component is
+justified.
+
+Steps for removal
+-----------------
+
+However clear the needs for removal are, we should take an incremental approach
+to deprecating code, especially when there's still a sub-community that cares
+about it. In that sense, code will never be removed outright without a series
+of steps are taken.
+
+A minimum set of steps should be:
+ #. A proposal for removal / deactivation should be made to the developers'
+    mailing lists (``llvm-dev``, ``cfe-dev``, ``lldb-dev``, etc), with a clear
+    statement of the maintenance costs imposed and the alternatives, if
+    applicable.
+ #. There must be enough consensus on the list that removal is warranted, and no
+    pending proposals to fix the situation from a sub-community.
+ #. An announcement for removal must be made on the same lists, with ample time
+    for downstream users to take action on their local infrastructure. The time
+    will depend on what is being removed.
+
+    #. If a script or documents are to be removed, they can always be pulled
+       from previous revision, and can be removed within days.
+    #. if a whole target is removed, we need to first announce publicly, and
+       potentially mark as deprecated in one release, only to remove on the
+       next release.
+    #. Everything else will fall in between those two extremes.
+ #. The removal is made by either the proposer or the sub-community that used to
+    maintain it, with replacements and arrangements made atomically on the same
+    commit.
+
+If a proposal for removal is delayed by the promise a sub-community will take
+care of the code affected, the sub-community will have a time to fix all the
+issues (depending on each case, as above), and if those are not fixed in time, a
+subsequent request for removal should be made and the community may elect to
+eject the component without further attempts to fix.
+
+Reinstatement
+-------------
+
+If a component is removed from LLVM, it may, at a later date, request inclusion
+of a modified version, with evidence that all of the issues were fixed and that
+there is a clear sub-community that will maintain it.
+
+By consequence, the pressure on such sub-community will be higher to keep
+overall maintenance costs to a minimum and will need to show steps to mitigate
+all of the issues that were listed as reasons for its original removal.
+
+Failing on those again, will lead to become a candidate for removal yet again.
+