Index: RELEASE_TESTERS.TXT =================================================================== --- /dev/null +++ RELEASE_TESTERS.TXT @@ -0,0 +1,57 @@ +This file is a list of the people responsible for ensuring that targets and +environments get tested and validated during the release process. + +They will also, in conjunction with the release manager and the code owners, +accept patches into stable release branches, tag critical bugs and release +stoppers as well as make sure that no regressions were observed on their +targets since the last release. + +N: Ben Pope +E: benpope81@gmail.com +T: x86 +O: Ubuntu + +N: Sylvestre Ledru +E: sylvestre@debian.org +T: x86 +O: Debian + +N: Nikola Smiljanic +E: popizdeh@gmail.com +T: x86 +O: OpenSUSE, Fedora + +N: Brian Cain +E: brian.cain@gmail.com +T: x86 +O: CentOS + +N: Bernhard Rosenkränzer +E: bero@linaro.org +T: x86 +O: OpenMandriva + +N: Dimitry Andric +E: dimitry@andric.com +T: x86 +O: FreeBSD + +N: Hans Wennborg +E: hans@chromium.org +T: x86 +O: Windows + +N: Renato Golin +E: renato.golin@linaro.org +T: ARM +O: Linux + +N: Diana Picus +E: diana.picus@linaro.org +T: AArch64 +O: Linux + +N: Vasileios Kalintiris +E: Vasileios.Kalintiris@imgtec.com +T: MIPS +O: Linux Index: docs/HowToReleaseLLVM.rst =================================================================== --- docs/HowToReleaseLLVM.rst +++ docs/HowToReleaseLLVM.rst @@ -2,17 +2,13 @@ How To Release LLVM To The Public ================================= -.. contents:: - :local: - :depth: 1 - Introduction ============ This document contains information about successfully releasing LLVM --- -including subprojects: e.g., ``clang`` and ``dragonegg`` --- to the public. It -is the Release Manager's responsibility to ensure that a high quality build of -LLVM is released. +including sub-projects: e.g., ``clang`` and ``compiler-rt`` --- to the public. +It is the Release Manager's responsibility to ensure that a high quality build +of LLVM is released. If you're looking for the document on how to test the release candidates and create the binary packages, please refer to the :doc:`ReleaseProcess` instead. @@ -46,7 +42,7 @@ the end of the first round of testing will be removed or disabled for the release. -* Generate and send out the second release candidate sources. Only *critial* +* Generate and send out the second release candidate sources. Only *critical* bugs found during this testing phase will be fixed. Any bugs introduced by merged patches will be fixed. If so a third round of testing is needed. @@ -89,24 +85,10 @@ #. Verify that the current Subversion trunk is in decent shape by examining nightly tester and buildbot results. -#. Create the release branch for ``llvm``, ``clang``, the ``test-suite``, and - ``dragonegg`` from the last known good revision. The branch's name is +#. Create the release branch for ``llvm``, ``clang``, and other sub-projects, + from the last known good revision. The branch's name is ``release_XY``, where ``X`` is the major and ``Y`` the minor release - numbers. The branches should be created using the following commands: - - :: - - $ svn copy https://llvm.org/svn/llvm-project/llvm/trunk \ - https://llvm.org/svn/llvm-project/llvm/branches/release_XY - - $ svn copy https://llvm.org/svn/llvm-project/cfe/trunk \ - https://llvm.org/svn/llvm-project/cfe/branches/release_XY - - $ svn copy https://llvm.org/svn/llvm-project/dragonegg/trunk \ - https://llvm.org/svn/llvm-project/dragonegg/branches/release_XY - - $ svn copy https://llvm.org/svn/llvm-project/test-suite/trunk \ - https://llvm.org/svn/llvm-project/test-suite/branches/release_XY + numbers. Use ``utils/release/tag.sh`` to tag the release. #. Advise developers that they may now check their patches into the Subversion tree again. @@ -121,8 +103,6 @@ $ svn co https://llvm.org/svn/llvm-project/cfe/branches/release_XY clang-X.Y - $ svn co https://llvm.org/svn/llvm-project/dragonegg/branches/release_XY dragonegg-X.Y - $ svn co https://llvm.org/svn/llvm-project/test-suite/branches/release_XY test-suite-X.Y Update LLVM Version @@ -155,71 +135,19 @@ This will generate source tarballs for each LLVM project being validated, which can be uploaded to the website for further testing. -Building the Release --------------------- - -The builds of ``llvm``, ``clang``, and ``dragonegg`` *must* be free of -errors and warnings in Debug, Release+Asserts, and Release builds. If all -builds are clean, then the release passes Build Qualification. - -The ``make`` options for building the different modes: - -+-----------------+---------------------------------------------+ -| Mode | Options | -+=================+=============================================+ -| Debug | ``ENABLE_OPTIMIZED=0`` | -+-----------------+---------------------------------------------+ -| Release+Asserts | ``ENABLE_OPTIMIZED=1`` | -+-----------------+---------------------------------------------+ -| Release | ``ENABLE_OPTIMIZED=1 DISABLE_ASSERTIONS=1`` | -+-----------------+---------------------------------------------+ - -Build LLVM -^^^^^^^^^^ - -Build ``Debug``, ``Release+Asserts``, and ``Release`` versions -of ``llvm`` on all supported platforms. Directions to build ``llvm`` -are :doc:`here `. - Build Clang Binary Distribution ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Creating the ``clang`` binary distribution (Debug/Release+Asserts/Release) -requires performing the following steps for each supported platform: - -#. Build clang according to the directions `here - `__. - -#. Build both a Debug and Release version of clang. The binary will be the - Release build. - -#. Package ``clang`` (details to follow). - -Target Specific Build Details -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The table below specifies which compilers are used for each Arch/OS combination -when qualifying the build of ``llvm``, ``clang``, and ``dragonegg``. - -+--------------+---------------+----------------------+ -| Architecture | OS | compiler | -+==============+===============+======================+ -| x86-32 | Mac OS 10.5 | gcc 4.0.1 | -+--------------+---------------+----------------------+ -| x86-32 | Linux | gcc 4.2.X, gcc 4.3.X | -+--------------+---------------+----------------------+ -| x86-32 | FreeBSD | gcc 4.2.X | -+--------------+---------------+----------------------+ -| x86-32 | mingw | gcc 3.4.5 | -+--------------+---------------+----------------------+ -| x86-64 | Mac OS 10.5 | gcc 4.0.1 | -+--------------+---------------+----------------------+ -| x86-64 | Linux | gcc 4.2.X, gcc 4.3.X | -+--------------+---------------+----------------------+ -| x86-64 | FreeBSD | gcc 4.2.X | -+--------------+---------------+----------------------+ -| ARMv7 | Linux | gcc 4.6.X, gcc 4.7.X | -+--------------+---------------+----------------------+ +Creating the ``clang`` binary distribution requires following the instructions +:doc:`here `. + +That process will perform both Release+Asserts and Release builds but only +pack the Release build for upload. You should use the Release+Asserts sysroot, +normally under ``final/Phase3/Release+Asserts/llvmCore-3.8.1-RCn.install/``, +for test-suite and run-time benchmarks, to make sure nothing serious has +passed through the net. For compile-time benchmarks, use the Release version. + +The minimum required version of the tools you'll need are :doc:`here ` Release Qualification Criteria ------------------------------ @@ -229,68 +157,49 @@ (We may tolerate some minor performance regressions if they are deemed necessary for the general quality of the compiler.) -**Regressions are new failures in the set of tests that are used to qualify +More specifically, Clang/LLVM is qualified when it has a clean test with all +supported sub-projects included (``make check-all``), per target, and it has no +regressions with the ``test-suite`` in relation to the previous release. + +Regressions are new failures in the set of tests that are used to qualify each product and only include things on the list. Every release will have some bugs in it. It is the reality of developing a complex piece of software. We need a very concrete and definitive release criteria that ensures we have monotonically improving quality on some metric. The metric we use is described below. This doesn't mean that we don't care about other criteria, but these are the criteria which we found to be most important and -which must be satisfied before a release can go out.** - -Qualify LLVM -^^^^^^^^^^^^ - -LLVM is qualified when it has a clean test run without a front-end. And it has -no regressions when using either ``clang`` or ``dragonegg`` with the -``test-suite`` from the previous release. - -Qualify Clang -^^^^^^^^^^^^^ - -``Clang`` is qualified when front-end specific tests in the ``llvm`` regression -test suite all pass, clang's own test suite passes cleanly, and there are no -regressions in the ``test-suite``. - -Specific Target Qualification Details -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -+--------------+-------------+----------------+-----------------------------+ -| Architecture | OS | clang baseline | tests | -+==============+=============+================+=============================+ -| x86-32 | Linux | last release | llvm regression tests, | -| | | | clang regression tests, | -| | | | test-suite (including spec) | -+--------------+-------------+----------------+-----------------------------+ -| x86-32 | FreeBSD | last release | llvm regression tests, | -| | | | clang regression tests, | -| | | | test-suite | -+--------------+-------------+----------------+-----------------------------+ -| x86-32 | mingw | none | QT | -+--------------+-------------+----------------+-----------------------------+ -| x86-64 | Mac OS 10.X | last release | llvm regression tests, | -| | | | clang regression tests, | -| | | | test-suite (including spec) | -+--------------+-------------+----------------+-----------------------------+ -| x86-64 | Linux | last release | llvm regression tests, | -| | | | clang regression tests, | -| | | | test-suite (including spec) | -+--------------+-------------+----------------+-----------------------------+ -| x86-64 | FreeBSD | last release | llvm regression tests, | -| | | | clang regression tests, | -| | | | test-suite | -+--------------+-------------+----------------+-----------------------------+ -| ARMv7A | Linux | last release | llvm regression tests, | -| | | | clang regression tests, | -| | | | test-suite | -+--------------+-------------+----------------+-----------------------------+ +which must be satisfied before a release can go out. + +Official Testing +---------------- + +A few developers in the community have dedicated time to validate the release +candidates and volunteered to be the official release testers for each +architecture. + +These will be the ones testing, generating and uploading the official binaries +to the server, and will be the minimum tests *necessary* for the release to +proceed. + +This will obviously not cover all OSs and distributions, so additional community +validation is important. However, if community input is not reached before the +release is out, all bugs reported will have to go on the next stable release. + +The official release managers are: + +* Major releases (X.0): Hans Wennborg +* Stable releases (X.n): Tom Stellard + +The official testers list is in the file ``RELEASE_TESTERS.TXT``, in the ``LLVM`` +repository. Community Testing ----------------- Once all testing has been completed and appropriate bugs filed, the release candidate tarballs are put on the website and the LLVM community is notified. -Ask that all LLVM developers test the release in 2 ways: + +We ask that all LLVM developers test the release in any the following ways: #. Download ``llvm-X.Y``, ``llvm-test-X.Y``, and the appropriate ``clang`` binary. Build LLVM. Run ``make check`` and the full LLVM test suite (``make @@ -300,28 +209,57 @@ everything. Run ``make check`` and the full LLVM test suite (``make TEST=nightly report``). -Ask LLVM developers to submit the test suite report and ``make check`` results -to the list. Verify that there are no regressions from the previous release. -The results are not used to qualify a release, but to spot other potential -problems. For unsupported targets, verify that ``make check`` is at least -clean. +#. Download ``llvm-X.Y``, ``llvm-test-X.Y``, and the appropriate ``clang`` + binary. Build whole programs with it (ex. Chromium, Firefox, Apache) for + your platform. + +#. Download ``llvm-X.Y``, ``llvm-test-X.Y``, and the appropriate ``clang`` + binary. Build *your* programs with it and check for conformance and + performance regressions. + +#. Run the :doc:`release process `, if your platform is + *different* than that which is officially supported, and report back errors + only if they were not reported by the official release tester for that + architecture. + +We also ask that the OS distribution release managers test their packages with +the first candidate of every release, and report any *new* errors in Bugzilla. +If the bug can be reproduced with an unpatched upstream version of the release +candidate (as opposed to the distribution's own build), the priority should be +release blocker. During the first round of testing, all regressions must be fixed before the second release candidate is tagged. -If this is the second round of testing, the testing is only to ensure that bug +In the subsequent stages, the testing is only to ensure that bug fixes previously merged in have not created new major problems. *This is not the time to solve additional and unrelated bugs!* If no patches are merged in, the release is determined to be ready and the release manager may move onto the next stage. +Reporting Regressions +--------------------- + +Every regression that is found during the tests (as per the criteria above), +should be filled in a bug in Bugzilla with the priority *release blocker* and +blocking a specific release. + +To help manage all the bugs reported and which ones are blockers or not, a new +"[meta]" bug should be created and all regressions *blocking* that Meta. Once +all blockers are done, the Meta can be closed. + +If a bug can't be reproduced, or stops being a blocker, it should be removed +from the Meta and its priority decreased to *normal*. Debugging can continue, +but on trunk. + Release Patch Rules ------------------- Below are the rules regarding patching the release branch: #. Patches applied to the release branch may only be applied by the release - manager. + manager, the official release testers or the code owners with approval from + the release manager. #. During the first round of testing, patches that fix regressions or that are small and relatively risk free (verified by the appropriate code owner) are @@ -333,7 +271,7 @@ regressions may be applied. #. For dot releases all patches must maintain both API and ABI compatibility with - the previous major release. Only bugfixes will be accepted. + the previous major release. Only bug-fixes will be accepted. Merging Patches ^^^^^^^^^^^^^^^ @@ -394,10 +332,10 @@ #. Check out the ``www`` module from Subversion. -#. Create a new subdirectory ``X.Y`` in the releases directory. +#. Create a new sub-directory ``X.Y`` in the releases directory. -#. Commit the ``llvm``, ``test-suite``, ``clang`` source, ``clang binaries``, - ``dragonegg`` source, and ``dragonegg`` binaries in this new directory. +#. Commit the ``llvm``, ``test-suite``, ``clang`` source and binaries in this + new directory. #. Copy and commit the ``llvm/docs`` and ``LICENSE.txt`` files into this new directory. The docs should be built with ``BUILD_FOR_WEBSITE=1``. @@ -417,5 +355,6 @@ Announce the Release ^^^^^^^^^^^^^^^^^^^^ -Have Chris send out the release announcement when everything is finished. +Send an email to the list announcing the release, pointing people to all the +relevant documentation, download pages and bugs fixed.