[RFC] Refactoring release docs

[rengolin]

Summary

We still need a doc describing how to release and what are the requirements, who's responsible, etc.

This may not be a final take, and we may merge the two documents together, but I'm getting some early feedback on the content, not the presentation.

A few big changes:

• removal of any mention to dragonegg
• not pointing people to build by hand, but to use the release scripts
• unique list of required tools and their versions (cmake, ninja, gcc, clang, libstcd++, msvc)
• quality definition is not by arch anymore. Simple "no regression" policy on check-all+test-suite is the minimum.
• added an "official testing" section, with the names of volunteers (please, all agree that you have volunteered!)
• adding extended validation topic (*other* tests by the community, distribution validation)
• adding how to report bugs, discussed recently on the release threads

Please review the content, and lets discuss the format later. We may want to merge the two release documents, or not. Doesn't matter right now.

[rovka]

I confirm that I'm volunteering for AArch64 :)

docs/HowToReleaseLLVM.rst

244

Wasn't Hans doing Windows? At least that's what D21880 says...
In any case, if we're going to keep the 2 documents separate, we shouldn't duplicate this info.
Just an idea: maybe instead of having this list in the docs, we could have a RELEASE_TESTERS.TXT, similar to CODE_OWNERS.TXT et al? (Although I guess that's a presentation concern, so we can discuss it some other time)

[rengolin]

docs/HowToReleaseLLVM.rst

244

Hum, yeah, probably cut&paste error. I did reorder these lines quite a bit.

we could have a RELEASE_TESTERS.TXT, similar to CODE_OWNERS.TXT et al?

The thought had crossed my mind. I like the idea more than keeping this table here.

[nikola]

This looks good to me, +1 for RELEASE_TESTERS.txt as it seems more consistent.

docs/HowToReleaseLLVM.rst

244

Yes, Hans does Windows.

[rengolin]

Separating the release testers into a new file RELEASE_TESTERS.TXT.

[dim]

RELEASE_TESTERS.TXT

26

s/x85/x86/ :)

36

I test both i386 and x86_64, for the rest LGTM!

[bero]

can do a bit more -- see inline comments

RELEASE_TESTERS.TXT

31

x86_64, i586, aarch64, armv7hnl. i586 will probably be replaced with i686 soon-ish.

32

May make sense to add AOSP -- will definitely run test builds there once Minseong's stuff is ready

[rengolin]

I'm changing "x86_64" to x86 to mean the target in LLVM, not the specific architecture, so we don't start with too many variations (ex i386, i586, i686, etc), which is pointless.

RELEASE_TESTERS.TXT

26

How did that go in there?

31

This is not about what you do, but what you're responsible for, in the LLVM community.

32

We can update once that's working. I don't want to promise anything.

[hans]

RELEASE_TESTERS.TXT

3

I don't think "upstream" makes sense when referring to ourselves.

42

Can you drop Darwin from my entry? I'll still build and test it if no one else does, but it would be great if we could find someone to sign up for it.

docs/HowToReleaseLLVM.rst

91

The branches are actually created by the utils/release/tag.sh script. I'd suggest removing the commands below.

169

Isn't this already documented somewhere else? I worry that this will get out-dated, as the previous text was.

[dsanders]

RELEASE_TESTERS.TXT

54–57

Vasileios Kalintiris will be taking over my release testing starting with this release.

I also upload a x86_64 tarball for Debian at the moment because it's a by-product of some of my MIPS testing.

[rengolin]

RELEASE_TESTERS.TXT

3

ok

42

ok

54–57

done

docs/HowToReleaseLLVM.rst

91

ok

169

added a link instead.

[hans]

Looks good to me. This is a huge improvement.

[rengolin]

Thanks Hans,

I'm splitting the commit in two: doc and new file, so that we can proceed with better documents before everyone agree to have their names public. I'll be creating a new review for the RELEASE_TESTERS.TXT file.

thanks!
--renato

[rengolin]

Just the docs in r276264.

[rengolin]

FYI: Just the RELEASE_TESTERS.TXT in D22624

RELEASE_TESTERS.TXT

+This file is a list of the people responsible for ensuring that targets and
+environments get tested and validated during the release process.
+
+They will also, in conjunction with the release manager and the code owners,
+accept patches into stable release branches, tag critical bugs and release
+stoppers as well as make sure that no regressions were observed on their
+targets since the last release.
+
+N: Ben Pope
+E: [e-mail]
+T: x86
+O: Ubuntu
+
+N: Sylvestre Ledru
+E: [e-mail]
+T: x86
+O: Debian
+
+N: Nikola Smiljanic
+E: [e-mail]
+T: x86
+O: OpenSUSE, Fedora
+
+N: Brian Cain
+E: [e-mail]
+T: x86
+O: CentOS
+
+N: Bernhard Rosenkränzer
+E: [e-mail]
+T: x86
+O: OpenMandriva
+
+N: Dimitry Andric
+E: [e-mail]
+T: x86
+O: FreeBSD
+
+N: Hans Wennborg
+E: [e-mail]
+T: x86
+O: Windows
+
+N: Renato Golin
+E: [e-mail]
+T: ARM
+O: Linux
+
+N: Diana Picus
+E: [e-mail]
+T: AArch64
+O: Linux
+
+N: Vasileios Kalintiris
+E: [e-mail]
+T: MIPS
+O: Linux

docs/HowToReleaseLLVM.rst

 How To Release LLVM To The Public
 =================================
 
-.. contents::
-   :local:
-   :depth: 1
-
 Introduction
 ============
 
 This document contains information about successfully releasing LLVM ---
-including subprojects: e.g., ``clang`` and ``dragonegg`` --- to the public.  It
-is the Release Manager's responsibility to ensure that a high quality build of
-LLVM is released.
+including sub-projects: e.g., ``clang`` and ``compiler-rt`` --- to the public.
+It is the Release Manager's responsibility to ensure that a high quality build
+of LLVM is released.
 
 If you're looking for the document on how to test the release candidates and
 create the binary packages, please refer to the :doc:`ReleaseProcess` instead.

   the end of the first round of testing will be removed or disabled for the
   release.
 
-* Generate and send out the second release candidate sources.  Only *critial*
+* Generate and send out the second release candidate sources.  Only *critical*
   bugs found during this testing phase will be fixed.  Any bugs introduced by
   merged patches will be fixed.  If so a third round of testing is needed.
 

 #. Verify that the current Subversion trunk is in decent shape by
    examining nightly tester and buildbot results.
 
-#. Create the release branch for ``llvm``, ``clang``, the ``test-suite``, and
-   ``dragonegg`` from the last known good revision.  The branch's name is
+#. Create the release branch for ``llvm``, ``clang``, and other sub-projects,
+   from the last known good revision.  The branch's name is
    ``release_XY``, where ``X`` is the major and ``Y`` the minor release
-   numbers.  The branches should be created using the following commands:
-
-   ::
-
-     $ svn copy https://llvm.org/svn/llvm-project/llvm/trunk \
-                https://llvm.org/svn/llvm-project/llvm/branches/release_XY
-
-     $ svn copy https://llvm.org/svn/llvm-project/cfe/trunk \
-                https://llvm.org/svn/llvm-project/cfe/branches/release_XY
-
-     $ svn copy https://llvm.org/svn/llvm-project/dragonegg/trunk \
-                https://llvm.org/svn/llvm-project/dragonegg/branches/release_XY
-
-     $ svn copy https://llvm.org/svn/llvm-project/test-suite/trunk \
-                https://llvm.org/svn/llvm-project/test-suite/branches/release_XY
+   numbers.  Use ``utils/release/tag.sh`` to tag the release.
 
 #. Advise developers that they may now check their patches into the Subversion
    tree again.

 
      $ svn co https://llvm.org/svn/llvm-project/cfe/branches/release_XY clang-X.Y
 
-     $ svn co https://llvm.org/svn/llvm-project/dragonegg/branches/release_XY dragonegg-X.Y
-
      $ svn co https://llvm.org/svn/llvm-project/test-suite/branches/release_XY test-suite-X.Y
 
 Update LLVM Version

 This will generate source tarballs for each LLVM project being validated, which
 can be uploaded to the website for further testing.
 
-Building the Release
---------------------
-
-The builds of ``llvm``, ``clang``, and ``dragonegg`` *must* be free of
-errors and warnings in Debug, Release+Asserts, and Release builds.  If all
-builds are clean, then the release passes Build Qualification.
-
-The ``make`` options for building the different modes:
-
-+-----------------+---------------------------------------------+
-| Mode            | Options                                     |
-+=================+=============================================+
-| Debug           | ``ENABLE_OPTIMIZED=0``                      |
-+-----------------+---------------------------------------------+
-| Release+Asserts | ``ENABLE_OPTIMIZED=1``                      |
-+-----------------+---------------------------------------------+
-| Release         | ``ENABLE_OPTIMIZED=1 DISABLE_ASSERTIONS=1`` |
-+-----------------+---------------------------------------------+
-
-Build LLVM
-^^^^^^^^^^
-
-Build ``Debug``, ``Release+Asserts``, and ``Release`` versions
-of ``llvm`` on all supported platforms.  Directions to build ``llvm``
-are :doc:`here <GettingStarted>`.
-
 Build Clang Binary Distribution
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Creating the ``clang`` binary distribution (Debug/Release+Asserts/Release)
-requires performing the following steps for each supported platform:
-
-#. Build clang according to the directions `here
-   <http://clang.llvm.org/get_started.html>`__.
-
-#. Build both a Debug and Release version of clang.  The binary will be the
-   Release build.
-
-#. Package ``clang`` (details to follow).
-
-Target Specific Build Details
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The table below specifies which compilers are used for each Arch/OS combination
-when qualifying the build of ``llvm``, ``clang``, and ``dragonegg``.
-
-+--------------+---------------+----------------------+
-| Architecture | OS            | compiler             |
-+==============+===============+======================+
-| x86-32       | Mac OS 10.5   | gcc 4.0.1            |
-+--------------+---------------+----------------------+
-| x86-32       | Linux         | gcc 4.2.X, gcc 4.3.X |
-+--------------+---------------+----------------------+
-| x86-32       | FreeBSD       | gcc 4.2.X            |
-+--------------+---------------+----------------------+
-| x86-32       | mingw         | gcc 3.4.5            |
-+--------------+---------------+----------------------+
-| x86-64       | Mac OS 10.5   | gcc 4.0.1            |
-+--------------+---------------+----------------------+
-| x86-64       | Linux         | gcc 4.2.X, gcc 4.3.X |
-+--------------+---------------+----------------------+
-| x86-64       | FreeBSD       | gcc 4.2.X            |
-+--------------+---------------+----------------------+
-| ARMv7        | Linux         | gcc 4.6.X, gcc 4.7.X |
-+--------------+---------------+----------------------+
+Creating the ``clang`` binary distribution requires following the instructions
+:doc:`here <ReleaseProcess>`.
+
+That process will perform both Release+Asserts and Release builds but only
+pack the Release build for upload. You should use the Release+Asserts sysroot,
+normally under ``final/Phase3/Release+Asserts/llvmCore-3.8.1-RCn.install/``,
+for test-suite and run-time benchmarks, to make sure nothing serious has 
+passed through the net. For compile-time benchmarks, use the Release version.
+
+The minimum required version of the tools you'll need are :doc:`here <GettingStarted>`
 
 Release Qualification Criteria
 ------------------------------

 (We may tolerate some minor performance regressions if they are deemed
 necessary for the general quality of the compiler.)
 
-**Regressions are new failures in the set of tests that are used to qualify
+More specifically, Clang/LLVM is qualified when it has a clean test with all
+supported sub-projects included (``make check-all``), per target, and it has no
+regressions with the ``test-suite`` in relation to the previous release.
+
+Regressions are new failures in the set of tests that are used to qualify
 each product and only include things on the list.  Every release will have
 some bugs in it.  It is the reality of developing a complex piece of
 software.  We need a very concrete and definitive release criteria that
 ensures we have monotonically improving quality on some metric.  The metric we
 use is described below.  This doesn't mean that we don't care about other
 criteria, but these are the criteria which we found to be most important and
-which must be satisfied before a release can go out.**
-
-Qualify LLVM
-^^^^^^^^^^^^
-
-LLVM is qualified when it has a clean test run without a front-end.  And it has
-no regressions when using either ``clang`` or ``dragonegg`` with the
-``test-suite`` from the previous release.
-
-Qualify Clang
-^^^^^^^^^^^^^
-
-``Clang`` is qualified when front-end specific tests in the ``llvm`` regression
-test suite all pass, clang's own test suite passes cleanly, and there are no
-regressions in the ``test-suite``.
-
-Specific Target Qualification Details
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-+--------------+-------------+----------------+-----------------------------+
-| Architecture | OS          | clang baseline | tests                       |
-+==============+=============+================+=============================+
-| x86-32       | Linux       | last release   | llvm regression tests,      |
-|              |             |                | clang regression tests,     |
-|              |             |                | test-suite (including spec) |
-+--------------+-------------+----------------+-----------------------------+
-| x86-32       | FreeBSD     | last release   | llvm regression tests,      |
-|              |             |                | clang regression tests,     |
-|              |             |                | test-suite                  |
-+--------------+-------------+----------------+-----------------------------+
-| x86-32       | mingw       | none           | QT                          |
-+--------------+-------------+----------------+-----------------------------+
-| x86-64       | Mac OS 10.X | last release   | llvm regression tests,      |
-|              |             |                | clang regression tests,     |
-|              |             |                | test-suite (including spec) |
-+--------------+-------------+----------------+-----------------------------+
-| x86-64       | Linux       | last release   | llvm regression tests,      |
-|              |             |                | clang regression tests,     |
-|              |             |                | test-suite (including spec) |
-+--------------+-------------+----------------+-----------------------------+
-| x86-64       | FreeBSD     | last release   | llvm regression tests,      |
-|              |             |                | clang regression tests,     |
-|              |             |                | test-suite                  |
-+--------------+-------------+----------------+-----------------------------+
-| ARMv7A       | Linux       | last release   | llvm regression tests,      |
-|              |             |                | clang regression tests,     |
-|              |             |                | test-suite                  |
-+--------------+-------------+----------------+-----------------------------+
+which must be satisfied before a release can go out.
+
+Official Testing
+----------------
+
+A few developers in the community have dedicated time to validate the release
+candidates and volunteered to be the official release testers for each
+architecture.
+
+These will be the ones testing, generating and uploading the official binaries
+to the server, and will be the minimum tests *necessary* for the release to
+proceed.
+
+This will obviously not cover all OSs and distributions, so additional community
+validation is important. However, if community input is not reached before the
+release is out, all bugs reported will have to go on the next stable release.
+
+The official release managers are:
+
+* Major releases (X.0): Hans Wennborg
+* Stable releases (X.n): Tom Stellard
+
+The official testers list is in the file ``RELEASE_TESTERS.TXT``, in the ``LLVM``
+repository.
 
 Community Testing
 -----------------
 
 Once all testing has been completed and appropriate bugs filed, the release
 candidate tarballs are put on the website and the LLVM community is notified.
-Ask that all LLVM developers test the release in 2 ways:
+
+We ask that all LLVM developers test the release in any the following ways:
 
 #. Download ``llvm-X.Y``, ``llvm-test-X.Y``, and the appropriate ``clang``
    binary.  Build LLVM.  Run ``make check`` and the full LLVM test suite (``make

    everything.  Run ``make check`` and the full LLVM test suite (``make
    TEST=nightly report``).
 
-Ask LLVM developers to submit the test suite report and ``make check`` results
-to the list.  Verify that there are no regressions from the previous release.
-The results are not used to qualify a release, but to spot other potential
-problems.  For unsupported targets, verify that ``make check`` is at least
-clean.
+#. Download ``llvm-X.Y``, ``llvm-test-X.Y``, and the appropriate ``clang``
+   binary. Build whole programs with it (ex. Chromium, Firefox, Apache) for
+   your platform.
+
+#. Download ``llvm-X.Y``, ``llvm-test-X.Y``, and the appropriate ``clang``
+   binary. Build *your* programs with it and check for conformance and
+   performance regressions.
+
+#. Run the :doc:`release process <ReleaseProcess>`, if your platform is
+   *different* than that which is officially supported, and report back errors
+   only if they were not reported by the official release tester for that
+   architecture.
+
+We also ask that the OS distribution release managers test their packages with
+the first candidate of every release, and report any *new* errors in Bugzilla.
+If the bug can be reproduced with an unpatched upstream version of the release
+candidate (as opposed to the distribution's own build), the priority should be
+release blocker.
 
 During the first round of testing, all regressions must be fixed before the
 second release candidate is tagged.
 
-If this is the second round of testing, the testing is only to ensure that bug
+In the subsequent stages, the testing is only to ensure that bug
 fixes previously merged in have not created new major problems. *This is not
 the time to solve additional and unrelated bugs!* If no patches are merged in,
 the release is determined to be ready and the release manager may move onto the
 next stage.
 
+Reporting Regressions
+---------------------
+
+Every regression that is found during the tests (as per the criteria above),
+should be filled in a bug in Bugzilla with the priority *release blocker* and
+blocking a specific release.
+
+To help manage all the bugs reported and which ones are blockers or not, a new
+"[meta]" bug should be created and all regressions *blocking* that Meta. Once
+all blockers are done, the Meta can be closed.
+
+If a bug can't be reproduced, or stops being a blocker, it should be removed
+from the Meta and its priority decreased to *normal*. Debugging can continue,
+but on trunk.
+
 Release Patch Rules
 -------------------
 
 Below are the rules regarding patching the release branch:
 
 #. Patches applied to the release branch may only be applied by the release
-   manager.
+   manager, the official release testers or the code owners with approval from
+   the release manager.
 
 #. During the first round of testing, patches that fix regressions or that are
    small and relatively risk free (verified by the appropriate code owner) are

    regressions may be applied.
 
 #. For dot releases all patches must maintain both API and ABI compatibility with
-   the previous major release.  Only bugfixes will be accepted.
+   the previous major release.  Only bug-fixes will be accepted.
 
 Merging Patches
 ^^^^^^^^^^^^^^^

 
 #. Check out the ``www`` module from Subversion.
 
-#. Create a new subdirectory ``X.Y`` in the releases directory.
+#. Create a new sub-directory ``X.Y`` in the releases directory.
 
-#. Commit the ``llvm``, ``test-suite``, ``clang`` source, ``clang binaries``,
-   ``dragonegg`` source, and ``dragonegg`` binaries in this new directory.
+#. Commit the ``llvm``, ``test-suite``, ``clang`` source and binaries in this
+   new directory.
 
 #. Copy and commit the ``llvm/docs`` and ``LICENSE.txt`` files into this new
    directory.  The docs should be built with ``BUILD_FOR_WEBSITE=1``.

 Announce the Release
 ^^^^^^^^^^^^^^^^^^^^
 
-Have Chris send out the release announcement when everything is finished.
+Send an email to the list announcing the release, pointing people to all the
+relevant documentation, download pages and bugs fixed.