Improved InformativeMailNotifier.

[ZijunZhao]

Summary

Listen for the main branch.

[buildbot] Fix worker for ThinLTO whole program devirtualization

This fixes a syntax error in the extra_configure_args provided to
getClangWithLTOBuildFactory, we shouldn't be surrounding the options
with double quotes.

Once that is fixed, I realized from testing the failing command locally
that it will still fail, because the clang-specific extra_configure_args
are also used by the initial stage1 build with the system compiler,
which is gcc. We only want these for the last stage, where LTO is
performed. Added a new parameter to getClangWithLTOBuildFactory,
extra_configure_args_lto_stage, to pass the options only to that LTO
stage.

Is there a way to test my change to ClangLTOBuilder.py?

jenkins: Do not resume the pipelines if the master restarts

It should help keep them from waiting forever to resume.

[MLGO] Corrected cmake variable

Added DoxygenDocsBuilder to build and publish doxygen documentation. This would build the doxygen documentation for all the LLVM projects which have it, and publish to proper places in llvm.org.

Added getHtmlDocsBuildFactory to build and publish LNT documentation.

This is to build and publish LNT Sphinx documentation.

Added bots to build and publish doxygen docs and LNT sphinx docs.

Fixed bad identation in ClangLTOBuilder.

Renamed HtmlSphinxDocsBuilder to HtmlDocsBuilder.

Use clang-stage2-Rthinlto host-compilers to for LLDB jobs

This should prevent that we have no regression information from LLDB because
the latest stage1 Clang we use as the host-compiler can't compile LLVM itself.

added .pylintrc

So that it can check the python code in the repo
without errors on missing imports.

mlir-nvidia: set -DMLIR_BINDINGS_PYTHON_ENABLED=ON

[jenkins] Fix exceptions raised by urllib.request

[jenkins] Fix typo in the host-compiler path used by lldb-cmake-matrix

Start Docker-izing sanitizer-windows

This defines a new docker image for sanitizer-windows based on the
existing docker image. I chose to dockerize sanitizer-windows first,
because it cycles quickly and doesn't do a two-stage build.

My plan is to have both the existing sanitizer-windows bot and the new
one run concurrently, but the new one will connect to the silent,
staging buildmaster (lab.llvm.org:9994). Once we are ready to cut over,
I will turn off the old sanitizer-windows VM and update this port in the
docker image and redeploy.

I am chosing to stick with the buildbot annotation script approach,
because it allows for much faster iteration: I can adjust paths for the
VS installation for example simply by committing to zorg. However, both
the live and staging bot will use the same script, so the same script
must work for both sanitizer-windows bots at the same time!

Attempt to fix Python 3 issues in annotated scripts

Put back missing strip in annotated script

Fix next Python 3 Unicode issue, the gift that will keep giving for the next 20 years

Try to fix Windows stage 2 build issue

[buildbot] Force lld for ThinLTO whole program devirtualization worker

Force use of lld, so that the configure step uses it. Otherwise we get
an error on the 3rd stage configure which will use the lld-only
--lto-whole-program-devirtualization option.

See: http://lab.llvm.org:8014/#/builders/137/builds/1921/steps/17/logs/stdio

Install Numpy in the MLIR Docker image, this is required for the python bindings

Also use pip to install pybind11 to ensure the most recent version is
available.

clang-ve-ninja: Build and use the LLVM dylib

jenkins/jobs: Checking in Clang jobs pipelines

Update branch names from master -> main

Fix Windows-on-Arm buildbots

... by disabling parts that don't build yet.

compiler-rt\lib\sanitizer_common\sanitizer_unwind_win.cpp assumes WIN64 is
x86_64, so, before that's fixed, disable everything that triggers its build.

Add a libc aarch64 worker and a debug builder running on it.

ClangBuilder: Enable fortran in test-suite

Automatically enable running the fortran parts of the test-suite if
flang is part of the checkout. In order to do this, we need to pass two
CMake defines when configuring the test-suite: one telling it to enable
fortran and one pointing it at flang. The latter picks the most recently
built flang (i.e. it chooses the one from stage 2 if there is one).

At the moment, this only affects clang-cmake-aarch64-full.

Deprecate build cache servers for aarch64 and aarch32 from Buildbot

• Remove two builders
  • clang-aarch64-linux-build-cache
  • clang-armv7-linux-build-cache
    
    And underlying servers:
  • packet-linux-aarch64-slave-1
  • packet-linux-armv7-slave-1
• These servers are not being used in a while, and the build processes provided by them overlap with other builders.

[libc] Make clang and clang++ as the explicit C and C++ compilers.

Explicitly use python3 for lit sandbox

Without this, virtualenv defaults to python2, which causes problems on
the ARM/AArch64 bots which no longer have python2 installed.

Removed lld-x86_64-freebsd builder.

Add worker sie-linux-builder and builder llvm-clang-x86_64-sie-ubuntu-fast.

jenkins: Update branch names for the repos from "master" to "main"

[lldb] Run check-lldb even if check-debuginfo failed

[lldb][monorepo] Run check-lldb even if check-debuginfo failed

Add Python binary path to CMake arguments for the clang-ppc64le-linux builder

LLVM now requires Python 3.6 which is not available in the official Ubuntu 16.04
repositories. This patch adds the path to a manually built Python 3.6 to the
CMake arguments of the clang-ppc64le-linux builder.

[zorg] Add Clang ppc64le cross-targeting buildbot on AIX

Support branches with the ForceSchedulers.

Update extraRecipients for the Windows LLDB box

[zorg] Remove redundant '\' in AIX buildbot config

[lldb] Don't run the debug info tests for the sanitized or matrix job

rdar://74356937

[jenkins] Fix TabError: inconsistent use of tabs and spaces in indentation

Support compatibility with buildbot 2.10.

This removes the deprecation warning and removes the
non-existing method addFactoryArguments (it was not doing
anything anyway for quite some time) which was breaking
the master.

Add flang out-of-tree new driver buildbot.

NFC. Some getSingleBranchSchedulers code cleaning.

Updated tags and removed redundant configure args for llvm-clang-x86_64-expensive-checks-win.

Updated requirements.

Fix android sanitizer tests for multiple devices

jenkins: More updates for "master" to "main" branch

Some builders.py cleaning.

NFC. Better names, more logs.

NFC. Changed getLntSchedulers() to follow the code style of the rest of the file.

And missing import log.

Added builders to build the release/* branches.

One GitHubStatusPush reporter is enough for our needs.

Have the buildbot email me when there is a failure on the sie-linux-builder builder.

Add flang debug new driver buildbot.

Add flang release buildbot.

Add flang release with assertions buildbot.

[zorg] Add buildbot for libcxx build on AIX

deployed new mlir-nvidia image

[clangd-ubuntu-clang] increased RAM

This fixes the out of memory issues when building with TSAN enabled.

fixed typo in comment

[clangd-ubuntu-clang] improved startup error logging

[zorg] Rename cross-targeting buildbot on AIX

Fixed misspelled builder name for e-mail notifications.

More master to main updates

jenkins/jobs: Add timeout to the lnt-ctmark-* jobs

This is so that these don't end up stuck and running forever in case of
any issues.

[sanitizer] Set default rtlib to libgcc

The default --rtlib for Android is about to change from libgcc to
compiler-rt (D96404), which will require a more substantial change to
this script to build the builtins and libunwind. For now, keep using
libgcc.

[sanitizer] Update x86 ABILIST filtering for x86_64 devices

An x86_64 device (e.g. Android emulator or Cuttlefish VM) might have an
ro.product.cpu.abilist like:

x86_64,x86,arm64-v8a,armeabi-v7a,armeabi
x86_64,arm64-v8a,x86,armeabi-v7a,armeabi

When the first arch starts with x86, filter out the remaining archs
that don't start with x86.

[sanitizer] Pass -e dotbytes=10m to wget

When wget writes to a log file, it defaults to 1KiB per dot, which
produces ~20000 lines of log output for the NDK. This "-e dotbytes"
syntax has existed since 1999 in wget.

[sanitizer,android] Don't check devices on error

If we already have a build error we should not check
if tests were executed. If device went offlind for some reasons
we usually show exception as it's infrastructure issue.

Howeve if we already see some compilation error we
don't want it hidden from commiters by exception.

[sanitizers] Fix checkout branch name

[sanitizers] Fix checkout branch name

[zorg] Use boolean flag to determine if building a unified standalone runtimes

For AIX libcxx build, we use ../llvm/runtimes as build source and with
-DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" option, since we are using
the new runtimes build laided out by the "A vision for building the runtimes"
RFC and D93408.

Enable mlir integration tests on the mlir-nvidia bot.

This is in preparation of landing https://reviews.llvm.org/D97463, which moves the cuda-runner tests to the integration test dir.

Fixed mistype in LibcxxAndAbiBuilder.py introduced by 1d2328ff69a8503911e85140b0786429204a2113.

Add builder and worker for XCore as default target.

Don't run testsuite on Linaro quick bots.

[VE] hpce-aurora2 worker replaces nec-arrproto41

Add check-flang option to some of the Flang builders.

Add Flang builder for latest gcc.

Add Flang builder for latest llvm and libc++.

[M68k] Add new worker debian-akiko-m68k for Linux 32-bit M68k

This adds a new builder called debian-akiko-m68k which will
build LLVM and Clang on Debian unstable on m68k.

Author: glaubitz

[libc++] Remove libc++ ARM builders

In accordance with the announcement on libcxx-dev and llvm-dev [1],
we are removing the libc++ specific buildbots in favor of running
those configurations in the pre-commit CI on Buildkite. This commit
removes the ARM-based libc++ build bots.

[1]: http://lists.llvm.org/pipermail/libcxx-dev/2020-November/001040.html

[libc builder] Temporarily disable integration tests and loader tests.

They will be enabled back on a new bot which will do a full libc build.

Added openmp doxygen; increased the doxygent step timeout, as it could take a long time without anything in TTY.

[XCore] Build fewer components. Run 4 parallel jobs.

Do not override LLVM_LIT_ARGS: use -j from the factory.

Fix flang out-of-tree with new driver builder.

[jenkins] Temporarily disable compiler-rt tests until couple of tests are fixed

Revert "[jenkins] Temporarily disable compiler-rt tests until couple of tests are fixed"

This reverts commit 33a9f3c59121b3ad1aad2e5e3c1e7653ac4f8ad1.

Add a mailer config for MLIR testing bots to Google build monitoring mailing-list

Added support for a new src_to_build_dir argument to LLVMBuildFactory to explicitly specify a directory to start a build for.

Added support for a new src_to_build_dir argument to UnifiedTreeBuilder to explicitly specify a directory to start a build for.

Use more efficient linker for flang debug builder.

Use lld linker for flang debug configuration.

Add Clang dependency for new Flang driver in out-of-tree builder.

When building out-of-tree we need to use CLANG_DIR to find
installed Clang components.

Respect jobs property for LIT tests in UnifiedTreeBuilder.

Remove 'release' and 'aarch64' tags from Flang builders.

[libc] Add a full build worker and two builders to run on it.

[libc] Enable full build builders.

[libc builder] Enable linting on full build builders.

[lldb] Move reproducers under the matrix bot

[lldb] Remove lldb-cmake-reproducers

The reproducer were successfully integrated in the matrix bot.

http://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake-matrix/

Update python deps for buildbot-mlir-nvidia.

Remove Cmake compiler arguments.

This way cmake will pick /usr/local/bin/cc or c++, which we
point to the appropriate compiler.

[lldb] Mark cmake build as unstable when one or more tests fail

Revert "[lldb] Mark cmake build as unstable when one or more tests fail"

This reverts commit 557b1768a04da0d7977cb861fc4c57198d468a78.

[lldb] Remove LLDB_CAPTURE_REPRODUCER export

[VE] use local cache

There is a need to allow SanitizerBuilder to accept extra args so cmake args can be passed in, in turn allowing us to specify -j with cmake.

By default all threads available on the machines are being used by the Buildbot. This patch limits the use of threads to 256.

[zorg] Adjust libcxx buildbot config on AIX

Enable libcxx project only

Enable build_standalone in LibcxxAndAbiBuilder

Unconditionally pre-define CMAKE_ARGS in sanitizer bot.

Fixes the following bug created by
110a39811261ab69894fe77a69813f08a8c1f1a9:

buildbot_cmake.sh: line 38: CMAKE_ARGS: unbound variable

Revert "Respect jobs property for LIT tests in UnifiedTreeBuilder."

This shall be properly handled in LitTestCommand instead.
Similar to how the jobs are handled for NinjaCommand, for example.

This reverts commit 13c3c22377e39033ff843659a8b49be278a20086.

Forgot to declare CMAKE_ARGS ahead of time to prevent unboud variable errors; I noticed a ccache: command not found error in the process and this fixes that too by checking the utility is available before running it.

Revert "Forgot to declare CMAKE_ARGS ahead of time to prevent unboud variable errors;"

This reverts commit 423a80226a9fdc789dc2ef18c4cefdfdf53d5b76.

[PowerPC] Allow the PowerPC RHEL buildbot to use cmake with test-suite

We would like to update the PowerPC RHEL buildbot (clang-ppc64le-rhel) to use
cmake instead of LNT, so that we can streamline our process whenever we update
CMakeList.txt.

[ve] fix cmake cache param

[Zorg] Add space between script and args

Add a space so that the strings are not concatenated incorrectly.

[Zorg] Change make to ninja instead for Clang RHEL buildbot

Changing clang builder's cmake_test_suite to use ninja instead
as the only buildbot that uses it builds and runs with ninja.

Deployed new mlir-nvidia

Revert "[Zorg] Change make to ninja instead for Clang RHEL buildbot"

This reverts commit f785f4216432aa0d249ee3ad1f86515caf4c7b10.

Revert "[PowerPC] Allow the PowerPC RHEL buildbot to use cmake with test-suite"

This reverts commit 4b8974b63dcc1c5eab28a6ec33328b30e63e396b.

Remove 3 Flang workers/builders

The following Flang workers/builders are removed:

• flang-aarch64-ubuntu
• flang-aarch64-ubuntu-clang
• flang-aarch64-ubuntu-gcc10

The corresponding configurations are covered by other, recently added,
Flang workers/builders from Linaro:

• flang-aarch64-latest-gcc
• flang-aarch64-ubuntu-latest-clang
• flang-aarch64-ubuntu-sharedlibs
• flang-aarch64-ubuntu-release
• flang-aarch64-ubuntu-rel-assert
• flang-aarch64-ubuntu-dylib
• flang-aarch64-ubuntu-debug

When compared to the workers/builders deleted here, the new Flang
workers from Linaro provide much better:

• granularity in terms of configurations
• build times

[zorg] Support other relative scripts in AnnotatedBuilder

AnnotatedBuilder should be able to launch both python and bash
scripts. It should launch with python for scripts that specify
the script_interpreter as python, or allow the user to specify
another interpreter. Otherwise it will execute an empty
script_interpreter script directly. Also, move the --jobs option
to be the default of extra_args, otherwise allow the user to
override extra_args.

[zorg] Add buildbot for HIP

Build HIP tests from llvm-test-suite using AMDGPU.
Will be extended to build external repo.

[sanitizer] Add scudo tests on QEMU

[sanitizer] Improve naming of bot steps

[sanitizer] Fix typo in bot script

[sanitizer] Enabled QEMU bot failures

[jenkins] Add groovy script for the llvm-cov job

[sanitizer] Cleanup build dirs hierarchy

[sanitizer] Run scudo on qemu-armhf

Adjust jobs for aix-ppc64-ppc64le worker

since the machine now has additional resources.

[zorg] Add buildbot for Synopsys ARC

Create AWS worker for LLVM Experimental Target : ARC

[zorg] Build and test project depends on project enabled in LibcxxAndAbiBuilder

Build and test project depends on project enabled in LibcxxAndAbiBuilder;
Let all steps respect env passed in;

Fixed bug introduced by D100497 - local variable jobs_flag referenced before assignment.

[zorg] Fix AnnotatedBuilder.py extra_args bug

Do not assign WithProperties("--jobs=%(jobs:-)s") to
extra_args, because extra_args_with_props will perform
WithProperties(...) on all of the args in extra_args.

Bumped urllib3 from 1.26.3 to 1.26.4.

Added enable_runtimes and enable_projects to LLVMBuildFactory.

Added support for LLVM_ENABLE_RUNTIMES to UnifiedTreeBuilder.

Documentation builds do not support LLVM_ENABLE_RUNTIMES.

Added support for LLVM_ENABLE_RUNTIMES to ClangLTOBuilder.

Changed LibcxxAndAbiBuilder to use LLVMBuildFactory enable_projects and enable_runtimes.

Revert "Support compatibility with buildbot 2.10."

Temporarily revert commit 3f44862c69dd1766b51ea408d2cba6ec5c76c30a, as we are still with 2.8.4 in the production.

Will be re-applied after migration to 2.10 or later.

Added "auto" for LLVMBuildFactory enable_runtimes arg; changed default to not use enable_runtimes.

[Zorg][OpenMP] Add CUDA offloading worker.

This worker tests OpenMP offloading for the x86_64 and NVIDIA GPU. In
addition to check-openmp, it runs the SOLLVE Validation & Verification Suite
via the LLVM test-suite External builder. The builder is configured to
only warn if the SOLLVE suite fails, as it also tests features that
have not been implemented in Clang yet.

CUDA is intentionally not installed in a default location (/opt/cuda) to
resemble setups often found in computing clusters with multiple versions
of CUDA to choose from.

[PowerPC] Change RHEL buildbot to unifiedtreebuilder

The patch switches the RHEL buildbot to utilise more jobs when building, as
well as UnifiedTreeBuilder (ie, with cmake and Ninja).

Differential: https://reviews.llvm.org/D101199

Remove -DFLANG_BUILD_NEW_DRIVER=ON from builders. NFCI

FLANG_BUILD_NEW_DRIVER defaults to ON now, so there's no need to add it
anymore.

jenkins/tasks: virtualenv: Use the Python3 interpreter

[sanitizer] Remove disabled line

[sanitizer] Fix COMPILER_RT_DEBUG assignment

Added missing comma in clang-ppc64le-rhel builder configuration.

[sanitizer][NFC] Reuse cmake setup

[sanitizer] Fix asan builder name

[sanitizer] Remove unused GN reference

[sanitizer] Add dedicated QEMU builder

[scudo] Test on mips and ppc QEMU

[sanitizer] Fix "set -e in subshell" in QEMU checkout

[scudo] Remove HWCAP2_MTE workaround

It's not needed after https://patches.linaro.org/patch/433102/

[scudo] Fix "set -e in subsell" issue

[scudo] Combine cmake/ninja steps

[scudo] Print executed tests

[scudo] Install cross compiled libs into STAGE1

We need this for LIT future lit tests.

Update configuration for Arm/Aarch64 cross Windows to Linux toolchan builders.

Those builders:

• llvm-clang-win-x-armv7l
• llvm-clang-win-x-aarch64

Run only 1 build at a time for heavily loaded workers.

Fixed undefined runtime_projects in OpenMPBuilder.

[sanitizer] Prepare for D102543

Mips requires special handling

[sanitizer] Remove QEMU from sanitizer-buildbot3

[libfuzzer] Print running tests

[libfuzzer] Remove unnecessary cmake flags

Linking job limit is unnecessary with LLD.
LIBFUZZER_ENABLE_TESTS does not exist any more.

[sanitizer] Run QEMU cmakes in parallel

[sanitizer] Remove nested buildbot_qemu call

Now it has own builder.

[PowerPC] Removed -DLLVM_ENABLE_PROJECTS=flang from PowerPC flang buildbot

With FLANG_NEW_DRIVER default to on now, it is necessary to remove this
option for the bot to be able to build again.

[clangd] Update gRPC dependency for remote index tests

Context https://github.com/clangd/clangd/pull/783

Limit the number of parallel link jobs for the full LTO builder. With the larger number of link jobs the build time grows significantly, and with the default value it randomly runs out of memory.

Limit the number of parallel link jobs for the full LTO rerlease builder.

[MLIR] Update MLIR build config to reflect cmake variable renames

Update CMake variables to reflect the migration for a consistent naming:
https://reviews.llvm.org/D102976

Additional discussion at:
https://llvm.discourse.group/t/mlir-cmake-enable-variable-naming-convention/3520

[zorg] Remove flang-aarch64-ubuntu-out-of-tree-new-driver

The new driver is on by default now, so this builder does the same thing
as flang-aarch64-ubuntu-out-of-tree.

Removed abandoned lldb-sphinx-docs builder.

[sanitizer] Switch to patched QEMU

[sanitizer] Fix QEMU checkout

[sanitizer] Test HWASan + LAM via QEMU.

Uses LAM-enabled full system QEMU emulation to test HWASan's LAM
support.

Depends on:
https://github.com/google/sanitizers/pull/1408
https://github.com/google/sanitizers/pull/1410

[buildbot-clangd-ubuntu-clang] deployed new docker image

This is just the deployment for the container updated in
cf3881f300f973e6dcc12d2c25dff626290642aa

[MLIR] Enable tensorcore integration tests

[VE] disable cmake caches until resolved

Enable retry on failed git operations

This should address issues like
fatal: unable to access 'https://github.com/llvm/llvm-project.git/': Could not resolve host: github.com
in
http://lab.llvm.org:8011/#/builders/131/builds/7

llvmbisect: Add support for monorepo artifacts

This is now compatible with both svn and git/monorepo artifacts, and
also works across them.

[libc] Add a "libc-fuzzer" step in full build mode.

[zorg] Cleanup clang-cmake-{armv7,aarch64}-quick bots

Run "ninja check" for armv7 bot and don't pretend we run
LNT testsuite for the aarch64 bot.

[zorg] Build all targets on Linaro-maintained bots

... not just AArch64 and ARM. Make an exception for the quick
bots, which build a single target.

[zorg] Add AArch64 SVE vector-length-specific (VLS) bots

We add 1-stage bot that tests SVE support by building/running testsuite,
and 2-stage bot that additionally builds 2nd stage compiler with SVE.

[zorg] Rename Linaro's AArch64 and ARM bots to use consistent names

This will simplify Linaro's buildbot maintenance scripts. We also
remove now-unused AArch64 and ARM libcxx workers, since these have
been migrated to buildkite a few months ago.

[zorg] Remove settings that match defaults from Linaro bots

Remove from arguments of ClangBuilder.getClangCMakeBuildFactory

• checkout_compiler_rt=True
• checkout_lld=True
• test=True
• useTwoStage=False
• testStage1=True

All these are defaults anyway.

[zorg] Rename Linaro-maintained 2-stage bots

... to have consistent -2stage suffix.

Update compiler on hexagon-build-0x machines

Add flang-x86_64-windows builder.

Add a buildbot to compile flang using msvc. Due to msvc doing some things differently than other compilers, the msvc flang build breaks regularly. This builder will notify committers if their change causes the build to break.

However, the tests never completed successfully on Windows, partially because many tests require a UNIX shell (see D89368). To not constantly fail, tests failures only trigger a warning, until all tests are fixed as well.

Currently running here: http://meinersbur.de:8011/#/builders/146

[M68k] Add new worker suse-gary-m68k-cross for Linux 32-bit M68k

This adds a new builder called clang-m68k-linux-cross which will
cross-build LLVM and Clang on openSUSE Leap.

[libc++] Remove build bots that are handled on BuildKite now

Those builders have been failing for months because they use a version
of Clang that we don't actively support anymore. Furthermore, all those
configurations (single-threaded, no exceptions, with LLVM's unwinder)
are being tested on BuildKite.

At this point, I think those builders are only adding noise and using
resources that could be repurposed to other tasks, so I think it is
reasonable to remove them.

[lldb] Actually let the matrix bot test something

Since https://reviews.llvm.org/D69401 landed the bot has been setting unused
CMake variables for specifying test compilers which meant this bot hasn't
been testing anything since 2019. Reenabling the tests so that we can start
fixing the failing tests.

[NFC] Reformat sanitizer workers

Reassign sanitizer builders/workers

Less builds for sanitizer-buildbot{2,8}

[sanitizer] Simplify clobbering

[sanitizer] Cleanup after build

[sanitizer] Fix compiler_rt_build cleanup

[sanitizer] Add QEMU cleanup

[NFC][sanitizer] Add newline at the end

Disable GPU in polly-arm-linux and aosp-O3-polly-before-vectorizer-unprofitable

[AIX][PowerPC] Reduce number of parallel link jobs for AIX crosstargetting buildbot

Reduce the parallel link jobs so that we reduce machine hangs and timeouts.

[sanitizers] Remove unset variables

It looks nothing else uses them.

[zorg] Reduce # targets built on Linaro flang bots

The flang bots are only running check-flang and are not really
exercising the backend in any way. At the moment most of them
are taking about 1h to run, and we should be able to reduce that
a bit by disabling targets other than the host (AArch64) on them.

[zorg] Hotfix for PPC64le MLIR build issue

The public builder for MLIR on Power currently has an
issue choosing the correct libc. This patch provides
a environment variable to guide CMake to the correct
one.

Move QEMU LAM sanitizer images to env-defined folder.

Also add some nice debug info as to how to build the images yourself.
This also makes the bot selector define the build dir and the QEMU image
dir, rather than these being hardcoded on the script side (which non-bot
users will want to execute to repro the bot).

[jenkins] Update script to use cross project lit test suite

To reflect changes announced in
https://lists.llvm.org/pipermail/llvm-dev/2021-January/148048.html,
this patch updates all occurrences of debuginfo-tests to
cross-project-debuginfo-tests.

Add chown to buildbot_qemu local repro instructions.

[sanitizer] Fix mips QEMU build

[scudo] Disable mips 32bit tests

It times out for unknown reasons.

[sanitizer] Don't tie builders with particular workers

This simplifies management of VMs. We don't need to care which
are online. If some is gone we will just see less frequent builds.

[sanitizer] Missing image is infra issues

This avoids spamming blame list

[sanitizer] Set EXE_LINKER_FLAGS same as SHARED_LINKER_FLAGS

Add cross-project-tests to SIE Linux builder.

Enable LLD tests together with the expensive checks on the expensive check builders.

nclude LLD project to build and run the tests together with LLVM expensive checks on all
appropriate builders:

llvm-clang-x86_64-expensive-checks-ubuntu
llvm-clang-x86_64-expensive-checks-win
llvm-clang-x86_64-expensive-checks-debian
llvm-clang-x86_64-expensive-checks-ubuntu-release
llvm-clang-x86_64-expensive-checks-win-release
llvm-clang-x86_64-expensive-checks-debian-release

The LLD test use the expensive check data structures and it would be good to detect
the failures when they configured with -DLLVM_ENABLE_EXPENSIVE_CHECKS=ON (ref: D105071)

Revert "[sanitizer] Don't tie builders with particular workers"

This reverts commit d37259ec73a4341700e981214b9032631adfdda0.
With some changes.

[zorg] Add HIP builder script

Simple script for HIP builder to build llvm-project incrementally,
then build and execute HIP tests from llvm-test-suite.

[ORC-RT] Disable ORC Runtime on ccache builders.

The ORC runtime contains assembly files that don't seem to build correctly
under ccache (e.g. https://lab.llvm.org/buildbot/#/builders/109/builds/18980).
This commit disables the ORC runtime on those builders, which seems to be the
same fix that was adopted for XRay and other compiler-rt libraries that use
assembly.

Remove aarch64-sve-vector-bits-min compile flag from SVE 2 stage bot

Our build is failing:
https://lab.llvm.org/buildbot/#/builders/176/builds/205

Due to a vectorisation issue being investigated as
https://bugs.llvm.org/show_bug.cgi?id=51182.

Remove the bits-min flag until this bug is fixed so we at least
get coverage of the 2nd stage build with SVE codegen.

jenkins: cmake_builder and clang_builder: Ignore error in the test stage

This allows us to proceed to generating the test result report.

jenkins: Stop uploading the compiler build artifacts to cloud storage

The storage space has been decommissioned.

https://lists.llvm.org/pipermail/llvm-dev/2021-July/151623.html

jenkins: Fix the file path pattern for the test results XML files

Include all the available test results files in the build directory.

Add missing closing quote to SVE 2 stage cmake options

Missing from 24c07902d058abcee3bcf908676bacaae1f3d448.

[Zorg] Use ccache instead incremental build for openmp-offload-cuda-runtime.

Building with LLVM_ENABLE_RUNTIMES=openmp has the disadvantage that if clang changes, the runtime that was built with the previous clang is not rebuilt. That is, if the runtime is miscompiling due to a change in clang, this will only be detected in the next clean build.

For instance, commit rG1100e4aafea233bc8bbc307c5758a7d287ad3bae caused the libomptarget device runtime to miscompile, but the openmp-offload-cuda-runtime builder shows it as green. Tests only started failing with the next clean build. In production, this would have blamed the wrong commit.

In contrast, the openmp-offload-cuda-project builder started failing with the expected commit.

Instead of building incrementally, use ccache to avoid this problem.

Revert "Remove aarch64-sve-vector-bits-min compile flag from SVE 2 stage bot"

This reverts commit 24c07902d058abcee3bcf908676bacaae1f3d448.

The issue is still present but given that it's not sending out any
new notifications I'm going to leave it as is for now.

It actively building is somewhat useful for those working on
the issue to know what the current status is.

Update config for aosp-O3-polly-before-vectorizer-unprofitable

The bot has moved to a new host with a newer AOSP codebase. Update the
buildbot config to match.

Change-Id: I474f4503078acdf4109ce755228379ae81a39be7

[sanitizer] Run two builds at the time

[NFC][sanitizer] Remove misleading comment

[sanitizer] Move cpplint to the end of the build

[sanitizers] Don't run cpplint

The script was removed.

[LLDB] Set lit arg -j4 for Arm/AArch64 Linux buildbots

We have started seeing some sporadic test failures on LLDB Arm/AArch64
Linux buildbots. This patch reduces no of parallel tests to 4 so
to provide some bandwidth for the timing critical and load dependent
test cases.

removing windows deployment

It's been unmaintained for a while and it's unclear
if we need a 32bit setup at all

Add a libc x86_64 windows worker and a debug builder running on it

Add buildbot for OpenMP on AMDGPU

Build OpenMP for AMDGPU target

Added jobs param for hip-vega20-0 worker, as getOpenMPCMakeBuildFactory cannot handle optional params.

Reverted 8ff202c1648d4dba1803aa0cade81790a8382c6c. Added jobs param for the right worker - omp-vega20-0, as getOpenMPCMakeBuildFactory cannot handle optional params.

Update openmp-offload-amdgpu builders

Disable SOLLVE testing for time being.
Clean build for openmp-offload-amdgpu-runtime.
Remove lld from depends_on_projects.
Remove LLD cmake options.

Correct openmp-offload-builders lld, openmp

Remove obsolete libc++ release builder

We should be using BuildKite instead. That build bot is broken now
anyways because it is using an unsupported compiler.

[HIP] use bootstrap to build libc++

Recent libc++ changes request clang-11 or gcc-11 to build.

Switch to a bootstrap build step to use built clang to build libc++.

Add Windows equivalent to libc-linux.py

Create an annotated Windows builder for libc buildbot.

openmp-offload-amdgpu-runtime: Disable testsuite for runtime. set ccace for runtime

disable llvm-test in openmp-offload-amdgpu-runtime
add ccache options for C and CXX

[libc] Get and set vcvars before running the steps.

Also added the CMake flag to pick up MPFR.

[libc | windows] Use the items method to iterate over dict items.

Enable flang examples on a few builders

flang-aarch64-rel-assert (which uses static libs) and
flang-aarch64-sharedlibs (which should be relevant for testing the
plugin support).

[sanitizer] Fix TMPDIR for older Android versions

[sanitizer] Move QEMU image check

[sanitizer] Move QEMU LAM steps

[sanitizer] Move more QEMU LAM steps

[sanitizer] Fix configure_hwasan_lam

[sanitizer] Name "configure hwasan" step

[sanitizer] Revert my Recent QEMU LAM changes

This reverts commit 2b6b16cbc17f5dadc621c9e96082df768be792d6.
This reverts commit b37ed8bbbce7eea797d5f684803f30d7b8532b75.
This reverts commit 75c452375a04c133c82407ecf139934efb4dd23c.
This reverts commit 71d17a80d274ffd4db8f15521d564a18c8e5c1ed.
This reverts commit e257cd977be51b29c9c3c650de4fe300a99c68f2.

[sanitizer] Delay LAM Prerequisites check

Image setup may be finalizing in background

Replace new-pass-manager-fast builder with llvm-dylib-all

Test a LLVM build with the dynamic library enabled. Remove the
clang-x86_64-debian-new-pass-manager-fast builder since the new pass
manager has been the default since Feb 21.

[sanitizer] Update Android bot for D108494

[sanitizer] Fix the previous fix for D108494

openmp-offload-amdgpu-project - disable test suite, since no SOLLVE is run

add libc scudo test to build config

move compiler-rt add before join

disable scudo integration if asan is enabled

Updated configuration for Arm/Aarch64 cross Windows to Linux toolchan release builders.

Find libc++ in the per-target runtime directory

Fixes build after "[CMake] Enable LLVM_ENABLE_PER_TARGET_RUNTIME_DIR by
default on Linux.

Use -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF

On sanitizer bots, revert to all runtime libraries in a common dir for
now.

Add openmp as dependent project , so as to trigger buildbots on openmp commits

Add openmp to dependents for openmp-runtime

add a check-tsan-shared

add a check-tsan-shared in cmake.sh

[ZijunZhao]

Sorry there is something wrong with my arc configuration

.pylintrc

+# add root folder to Pylint PYTHONPATH, so builders can be checked with zorg.*
+# packages
+[MASTER]
+init-hook='import sys; sys.path.append(".")'
+
+[FORMAT]
+max-line-length=120
 No newline at end of file

buildbot/google/docker/buildbot-clangd-ubuntu-clang/Dockerfile

 # Install build bot (server was at 2.8.5-dev at time of writing).
 RUN pip3 install buildbot-worker==2.8.4
 
 # Build and install gRPC for clangd remote index, based on instructions
 # https://github.com/llvm/llvm-project/tree/master/clang-tools-extra/clangd/index/remote#building-from-sources
 # The grpc version shipped with Ubuntu 18.04 crashes on TSan checks.
 ENV GRPC_INSTALL_PATH=/usr/local/lib/grpc
 RUN cd /tmp ; \
-    git clone -b v1.33.2 --depth 1  --recursive --shallow-submodules https://github.com/grpc/grpc ; \
+    git clone -b v1.36.3 --depth 1  --recursive --shallow-submodules https://github.com/grpc/grpc ; \
     cd grpc  ; \
     mkdir build; cd build ; \
     cmake -G Ninja -DgRPC_INSTALL=ON -DCMAKE_INSTALL_PREFIX=${GRPC_INSTALL_PATH} -DCMAKE_BUILD_TYPE=Release .. ; \
     cmake --build . --target install ; \
     # Clean up sources to reduce image size.
     cd / ; rm -rf /tmp/grpc
 
 # Workaround permissions issues when writing to named volumes

buildbot/google/docker/buildbot-clangd-ubuntu-clang/VERSION

-3
+5

buildbot/google/docker/buildbot-clangd-ubuntu-clang/run.sh

 echo "Host information:"
 cat ${WORKER_NAME}/info/host
 
 # create the folder structure
 echo "creating worker ${WORKER_NAME} at port ${BUILDBOT_PORT}..."
 buildbot-worker create-worker --keepalive=200 "${WORKER_NAME}" \
   lab.llvm.org:${BUILDBOT_PORT} "${WORKER_NAME}" "${WORKER_PASSWORD}"
 
-# start the daemon, this command return immetiately
+# start the daemon, this command returns immediately
 echo "starting worker..."
-buildbot-worker start "${WORKER_NAME}"
+set +e
+if ! buildbot-worker start "${WORKER_NAME}"; then
+  echo ERROR: starting worker failed. contents of twistd.log:
+  cat "${WORKER_NAME}/twistd.log"
+  exit 1
+fi
 
 # To keep the container running and produce log outputs: dump the worker
 # log to stdout
 echo "Following worker log..."
 tail -f ${WORKER_NAME}/twistd.log

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile

 #===-- Dockerfile --------------------------------------------------------===//
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===//
 # Docker image used for the mlir-nvidia builder
-# 
+#
 # Environment variables configurable at runtime:
 #    BUILDBOT_PORT - server port to connect to
 #===----------------------------------------------------------------------===//
 
 # Use the image from NVIDIA as base
 FROM nvidia/cuda:10.2-base-ubuntu18.04
 
 
 # install build tools
 RUN apt-get update; \
     apt-get install -y software-properties-common apt-transport-https ca-certificates \
     clang-8 lld-8 ninja-build git wget gnupg ccache \
-    python3 python3-pip python3-psutil pybind11-dev \
+    python3 python3-pip python3-psutil \
     # dumb-init recommended in
     # https://hub.docker.com/r/buildbot/buildbot-worker/dockerfile
     dumb-init;\
     update-alternatives --install /usr/bin/clang clang /usr/bin/clang-8 100 ;\
     update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-8 100 ;\
     update-alternatives --install /usr/bin/lld lld /usr/bin/lld-8 100
 
+# LTS releases often bundle obsolete pip versions that cannot access newest
+# Linux binary wheels. This pinned version is not special: it was just current
+# at the time this was added. Refer to compatibility table:
+# https://github.com/pypa/manylinux
+RUN python3 -m pip install --upgrade pip==21.0.1
+
+# Refer to mlir/lib/Bindings/Python/requirements.txt. Listed explicitly here
+# and version pinned for consistency as this is a bot.
+RUN python3 -m pip install numpy==1.19.3 pybind11==2.6.0 PyYAML==5.4.1
+
 # install cuda
 # avoid popups for keyboard configurations
 RUN DEBIAN_FRONTEND=noninteractive apt-get install -y cuda
 
 # Ubuntu ships with old cmake version, install the latest one
 # from https://apt.kitware.com/
 RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | \
        gpg --dearmor - | \

 # copy startup script
 COPY run.sh /home/buildbot/
 RUN chmod a+rx /home/buildbot/run.sh
 
 USER buildbot
 ENV WORKER_NAME="mlir-nvidia"
 
 # Allow the server port of this agent to be configurable during deployment.
-# This way we can connect the same image to production and integration. 
+# This way we can connect the same image to production and integration.
 # Ports:
 #   9990 - production
 #   9994 - integration
 ENV BUILDBOT_PORT="9994"
 
 CMD /home/buildbot/run.sh

buildbot/google/docker/buildbot-mlir-nvidia/VERSION

-15
+17

buildbot/google/docker/buildbot-sanitizer-windows/Dockerfile

+# escape=`
+FROM gcr.io/sanitizer-bots/windows-base-vscode2019:8
+
+ENV WORKER_NAME=sanitizer-windows
+
+# copy script to start the agent
+COPY run.ps1 .
+COPY fix_buildbot_tac_paths.py .
+
+# Set location for sccache cache
+ENV SCCACHE_DIR="C:\volumes\sccache"
+
+# Set the maximum sccache local cache size
+ENV SCCACHE_CACHE_SIZE="10G"
+
+# Move buildbot work dir to a volume to avoid of disc issues
+VOLUME C:\volumes\buildbot
+
+# Configure 32bit tools ("x86") instead of 64bit ("amd64")
+CMD ["powershell.exe", "-NoLogo", "-ExecutionPolicy", "Bypass", `
+     "c:\\buildbot\\run.ps1"]

buildbot/google/docker/buildbot-sanitizer-windows/VERSION

+1

buildbot/google/docker/buildbot-sanitizer-windows/admin

+Reid Kleckner <[e-mail]>

buildbot/google/docker/buildbot-sanitizer-windows/fix_buildbot_tac_paths.py

+#!/usr/bin/env python3
+# Replace backward slashes in the paths in a "buildbot.tac" file
+
+import argparse
+import re
+import shutil
+
+
+def fix_slashes(buildbot_tac_path : str):
+    shutil.copy(buildbot_tac_path, buildbot_tac_path + ".bak")
+    result = []
+    with open(buildbot_tac_path) as buildbot_tac_file:
+        buildbot_tac = buildbot_tac_file.readlines()
+        for line in buildbot_tac:
+            if line.lstrip().startswith('basedir'):
+                line = line.replace(r'\\','/')
+            result.append(line)
+    with open(buildbot_tac_path, 'w') as buildbot_tac_file:
+        buildbot_tac_file.writelines(result)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument('buildbot_tac_path', help="path of 'buildbot.tac' file")
+    args = parser.parse_args()
+    fix_slashes(args.buildbot_tac_path)

buildbot/google/docker/buildbot-sanitizer-windows/run.ps1

+# Stop the script on the first error
+$ErrorActionPreference = "Stop"
+
+# Read password from file
+$WORKER_PASSWORD=Get-Content "C:\volumes\secrets\token"
+
+#Create short drive alias "Q:" for worker dir, to keep path lengths <260 
+SUBST Q: C:\volumes\buildbot
+Get-PSDrive
+
+# chdir to the volume storing the temporary data
+# This is a mounted volume
+Set-Location Q:\
+
+# Write host and admin information
+# delete old folder if it exists
+# configure powershell output to use UTF-8, otherwiese buildbot can't read the file
+if ( Test-Path -Path 'info\' -PathType Container ) { 
+  Remove-Item -Recurse -Force info 
+}
+mkdir "info\"
+$HOST_FILE="info\host"
+$PSDefaultParameterValues['Out-File:Encoding'] = 'utf8'
+Write-Output "Windows version: $(cmd /c ver)" > $HOST_FILE
+Write-Output "VCTools version: $(ls C:\BuildTools\VC\Tools\MSVC | Select -ExpandProperty Name)" >> $HOST_FILE
+Write-Output "Cores          : $((Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors)" >> $HOST_FILE
+Write-Output "RAM            : $([int][Math]::Round((Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory / 1073741824)) GB" >> $HOST_FILE
+Write-Output "Reid Kleckner <[e-mail]>" > "info\admin"
+
+# create the worker
+Write-Output "creating worker..."
+buildslave create-slave --keepalive=200 Q: `
+  "lab.llvm.org:9994" "$env:WORKER_NAME" "$WORKER_PASSWORD"
+# Note: Powershell does NOT exit on non-zero return codes, so we need to check manually.
+if ($LASTEXITCODE -ne 0) { throw "Exit code of 'buildslave create' is $LASTEXITCODE" }
+
+# Replace backward slashes with forward slashes on buildbot.tac
+# Otherwise Cmake will go into infinite re-configuration loops
+python C:\buildbot\fix_buildbot_tac_paths.py buildbot.tac
+if ($LASTEXITCODE -ne 0) { throw "Exit code of 'fix_buildbot_tac_paths' is $LASTEXITCODE" }
+
+# start the daemon, as this does not print and logs to std out, run it in the background
+Write-Output "starting worker..."
+cmd /c Start /B buildslave start Q:
+
+# Wait a bit until the logfile exists
+Start-Sleep -s 5
+
+# To keep the container running and produce log outputs: dump the worker
+# log to stdout, this is the windows equivalent of `tail -f`
+Get-Content -Path "Q:\twistd.log" -Wait

buildbot/google/terraform/deployment-mlir-nvidia-production.yaml

strategy:
     type: RollingUpdate
   template:
     metadata:
       labels:
         app: mlir-nvidia-production
     spec:
       containers:
       # the image and version we want to run   
-      - image: gcr.io/sanitizer-bots/buildbot-mlir-nvidia:15
+      - image: gcr.io/sanitizer-bots/buildbot-mlir-nvidia:17
         name: mlir-nvidia-production
         env:
           # connect to production environment, running at port 9990 
           - name: BUILDBOT_PORT
             value: "9990"
         # reserve "<number of cores>-1" for this image, kubernetes also
         # needs <1 core for management tools
         resources:

buildbot/google/terraform/main.tf

resource "null_resource" "deployment-mlir-nvidia" {
   # Add NVIDIA daemonset and deploy mlir-nvidia.
   # Using this workaround as terraform does not support GPUs on Google Cloud.
   # https://github.com/terraform-providers/terraform-provider-kubernetes/issues/149
   provisioner "local-exec" {
       command = "kubectl apply -f ${path.module}/deployment-mlir-nvidia-production.yaml"
   }
 }
 
-# node pool for windows machines
-resource "google_container_node_pool" "windows_32core_pool_nodes" {
-  name       = "windows-32core-pool"
-  # specify a zone here (e.g. "-a") to avoid a redundant deployment
-  location   = var.gcp_config.zone_a
-  cluster    = google_container_cluster.primary.name
-
-  # use autoscaling to only create a machine when there is a deployment
-  autoscaling {
-    min_node_count = 0
-    max_node_count = 1
-  }
-  
-  node_config {
-    # use preemptible, as this saves costs
-    preemptible  = true
-    machine_type = "e2-highcpu-32"
-    # Windows deployments tend to require more disk space, so using 300GB here.
-    disk_size_gb = 300
-    # FIXME: test if SSDs are actually faster than HDDs for our use case
-    disk_type = "pd-ssd"
-
-    # Configure Windows image. As Windows is picky about the combination of
-    # host and container OS versions, this must be compatible with the version
-    # in your container. Recommondation: Use LTSC for long-term stability.
-    # For details see
-    # https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility
-    # https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster-windows#choose_your_windows_server_node_image
-    image_type = "WINDOWS_LTSC"
-
-    # set the premissions required for the deployment later
-    oauth_scopes = [
-      "https://www.googleapis.com/auth/logging.write",
-      "https://www.googleapis.com/auth/monitoring",
-      "https://www.googleapis.com/auth/devstorage.read_only",
-    ]
-
-    # add a label to all machines of this type, so we can select them 
-    # during deployment
-    labels = {
-      pool = "win-32core-pool"
-    }
-  }
-}
-
-# Deployment for the buildbot windows10_vs2019 running on Windows rather than
-# Linux. 
-# Note: Deploying this takes significantly longer (~35 min) than on Linux 
-# as the images tend to be larger (~18GB) and IO performance is lower.
-resource "kubernetes_deployment" "windows10_vs2019" {
-  metadata {
-    name = "windows10-vs2019"
-    labels = {
-      app = "windows10_vs2019"
-    }
-  }
-
-  spec {
-    # create one instance of this container
-    replicas = 1
-
-    selector {
-      match_labels = {
-        app = "windows10_vs2019"
-      }
-    }
-    strategy{
-      rolling_update{
-        # do not deploy more replicas, as the buildbot server 
-        # can't handle multiple workers with the same credentials
-        max_surge = 0
-        # Allow to have 0 replicas during updates. 
-        max_unavailable = 1
-        }
-      type = "RollingUpdate"
-    }
-    template {
-      metadata {
-        labels = {
-          app = "windows10_vs2019"
-        }
-      }
-
-      spec {
-        container {
-          image = "${var.gcp_config.gcr_prefix}/buildbot-windows10-vs2019:17"
-          name  = "windows10-vs2019"
-
-          # reserve "<number of cores>-1" for this image, kubernetes also
-          # needs <1 core for management tools
-          resources {
-            limits {
-              cpu    = "31"
-              memory = "20Gi"
-            }
-            requests {
-              cpu    = "31"
-              memory = "20Gi"
-            }
-          }
-
-          # mount the secrets into a folder  
-          volume_mount {
-            mount_path = "c:\\volumes\\secrets"
-            name = "buildbot-token"
-          }
-          volume_mount {
-            mount_path = "c:\\volumes\\sccache"
-            name = "sccache-vol"
-          }
-          volume_mount {
-            mount_path = "c:\\volumes\\buildbot"
-            name = "buildbot-vol"
-          }
-
-        }
-        # select which node pool to deploy to
-        node_selector = {
-          pool = "win-32core-pool"
-        }
-        # restart in case of any crashes
-        restart_policy = "Always"
-        
-        # select the secret to be mounted
-        volume {
-          name = "buildbot-token"
-            secret {
-              optional = false
-              secret_name = "password-windows10-vs2019"
-            }
-        }
-        volume {
-          name = "sccache-vol"
-          empty_dir {}
-        }
-        volume {
-          name = "buildbot-vol"
-          empty_dir {}
-        }
-
-        # Windows nodes from the node pool are marked with the taint 
-        # "node.kubernetes.io/os=windows". So we need to "tolerate" this to
-        # deploy to such nodes.
-        toleration {
-          effect    = "NoSchedule"
-          key       = "node.kubernetes.io/os"
-          operator  = "Equal"
-          value     = "windows"
-        }
-      }
-    }
-  }
-}
-
-
 resource "google_container_node_pool" "linux_16_core_pool" {
   name       = "linux-16-core-pool"
   # specify a zone here (e.g. "-a") to avoid a redundant deployment
   location   = var.gcp_config.zone_a
   cluster    = google_container_cluster.primary.name
   
   # use autoscaling to only create a machine when there is a deployment
   autoscaling {
     min_node_count = 0
     max_node_count = 2
   }
   
   node_config {
     # use preemptible, as this saves costs
     preemptible  = true
-    #custom machine type: 16 core, 32 GB as tsan needs more RAM
-    machine_type = "n2d-custom-16-32768"
+    #custom machine type: 16 core, 48 GB as tsan needs more RAM
+    machine_type = "n2d-custom-16-49152"
     disk_size_gb = 100
     disk_type = "pd-ssd"
 
     # set the premissions required for the deployment later
     oauth_scopes = [
       "https://www.googleapis.com/auth/logging.write",
       "https://www.googleapis.com/auth/monitoring",
       "https://www.googleapis.com/auth/devstorage.read_only",
template {
       metadata {
         labels = {
           app = "clangd-ubuntu-clang"
         }
       }
 
       spec {
         container {
-          image = "${var.gcp_config.gcr_prefix}/buildbot-clangd-ubuntu-clang:3"
+          image = "${var.gcp_config.gcr_prefix}/buildbot-clangd-ubuntu-clang:5"
           name  = "buildbot-clangd-ubuntu-clang"
 
           # reserve "<number of cores>-1" for this image, kubernetes also
           # needs <1 core for management tools
           resources {
             limits {
               cpu    = "15"
-              memory = "28G"
+              memory = "45G"
             }
             requests {
               cpu    = "15"
-              memory = "28G"
+              memory = "45G"
             }
           }
 
           # mount the secrets into a folder  
           volume_mount {
             mount_path = "/vol/secrets"
             name = "buildbot-token"
           }

buildbot/osuosl/master/config/__init__.py

 # Load local options.
 import os
 from importlib import reload
 
 from configparser import ConfigParser
 options = ConfigParser(
             interpolation=None,
             empty_lines_in_values=False,
             allow_no_value=True,
           )
 options.read(os.path.join(os.path.dirname(__file__), 'local.cfg'))
 
 import config.auth
 import config.builders
+import config.release_builders
 import config.schedulers
 import config.workers
 import config.status
 
 # Note: The following modules should be reloaded in
 # a particular order to follow the dependency chain.
 reload(config.auth)
 reload(config.workers)
 reload(config.schedulers)
 reload(config.builders)
+reload(release_builders)
 reload(config.status)

buildbot/osuosl/master/config/builders.py

-# TODO: Rename workers with "slave" as a part of the name.
+from importlib import reload
+
+from buildbot.plugins import util
 
 from buildbot.process.properties import WithProperties
 
 from zorg.buildbot.builders import ClangBuilder
 from zorg.buildbot.builders import FlangBuilder
 from zorg.buildbot.builders import PollyBuilder
 from zorg.buildbot.builders import LLDBBuilder
 from zorg.buildbot.builders import SanitizerBuilder
 from zorg.buildbot.builders import OpenMPBuilder
 from zorg.buildbot.builders import LibcxxAndAbiBuilder
 from zorg.buildbot.builders import SphinxDocsBuilder
 from zorg.buildbot.builders import ABITestsuitBuilder
 from zorg.buildbot.builders import ClangLTOBuilder
 from zorg.buildbot.builders import UnifiedTreeBuilder
 from zorg.buildbot.builders import AOSPBuilder
 from zorg.buildbot.builders import AnnotatedBuilder
 from zorg.buildbot.builders import LLDPerformanceTestsuite
 from zorg.buildbot.builders import FuchsiaBuilder
 from zorg.buildbot.builders import XToolchainBuilder
 
-from buildbot.plugins import util
+from zorg.buildbot.builders import HtmlDocsBuilder
+from zorg.buildbot.builders import DoxygenDocsBuilder
+
+reload(HtmlDocsBuilder)
+reload(DoxygenDocsBuilder)
 
 
 all = [
 
 # Clang fast builders.
 
     {'name' : "clang-x86_64-debian-fast",
     'tags'  : ["clang", "fast"],
     'collapseRequests': False,
     'workernames':["gribozavr4"],
     'builddir':"clang-x86_64-debian-fast",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     llvm_srcdir="llvm.src",
                     obj_dir="llvm.obj",
                     clean=True,
                     depends_on_projects=['llvm','clang','clang-tools-extra','compiler-rt'],
                     extra_configure_args=[
                         "-DCOMPILER_RT_BUILD_BUILTINS:BOOL=OFF",
+                        "-DCOMPILER_RT_BUILD_ORC:BOOL=OFF",
                         "-DCOMPILER_RT_BUILD_SANITIZERS:BOOL=OFF",
                         "-DCOMPILER_RT_BUILD_XRAY:BOOL=OFF",
                         "-DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF",
                         "-DCMAKE_C_FLAGS=-Wdocumentation -Wno-documentation-deprecated-sync",
                         "-DCMAKE_CXX_FLAGS=-std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync",
                     ],
                     env={
                         'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
                         'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes',
                     })},
 
-    {'name' : "clang-x86_64-debian-new-pass-manager-fast",
-    'tags'  : ["clang", "fast"],
-    'workernames': ["gribozavr4"],
-    'builddir': "clang-x86_64-debian-new-pass-manager-fast",
-    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
-                    llvm_srcdir="llvm.src",
-                    obj_dir="llvm.obj",
-                    clean=True,
-                    depends_on_projects=['llvm','clang','clang-tools-extra','compiler-rt'],
-                    extra_configure_args=[
-                        "-DCOMPILER_RT_BUILD_BUILTINS:BOOL=OFF",
-                        "-DCOMPILER_RT_BUILD_SANITIZERS:BOOL=OFF",
-                        "-DCOMPILER_RT_BUILD_XRAY:BOOL=OFF",
-                        "-DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF",
-                        "-DENABLE_EXPERIMENTAL_NEW_PASS_MANAGER=ON",
-                        "-DCMAKE_C_FLAGS=-Wdocumentation -Wno-documentation-deprecated-sync",
-                        "-DCMAKE_CXX_FLAGS=-std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync",
-                    ],
-                    env={
-                        'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
-                        'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes',
-                    })},
-
     {'name' : "llvm-clang-x86_64-win-fast",
     'tags'  : ["clang", "fast"],
     'collapseRequests': False,
     'workernames' : ["as-builder-3"],
     'builddir': "llvm-clang-x86_64-win-fast",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
                     vs="autodetect",
                     depends_on_projects=['llvm', 'clang'],
                     clean=True,
                     checks=[
                     "check-llvm-unit",
                     "check-clang-unit"],
                     extra_configure_args=[
                         "-DLLVM_ENABLE_WERROR=OFF",
                         "-DLLVM_TARGETS_TO_BUILD=ARM",
                         "-DLLVM_DEFAULT_TARGET_TRIPLE=armv7-unknown-linux-eabihf",
                         "-DLLVM_ENABLE_ASSERTIONS=OFF",
                         "-DLLVM_OPTIMIZED_TABLEGEN=OFF",
                         "-DLLVM_LIT_ARGS=-v --threads=32"])},
 
+    {'name': "llvm-clang-x86_64-sie-ubuntu-fast",
+    'tags'  : ["clang", "llvm", "clang-tools-extra", "lld", "cross-project-tests"],
+    'workernames': ["sie-linux-builder"],
+    'builddir': "llvm-clang-x86_64-sie-ubuntu-fast",
+    'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    depends_on_projects=['llvm','clang','clang-tools-extra','lld','cross-project-tests'],
+                    extra_configure_args=[
+                        "-DCMAKE_C_COMPILER=gcc",
+                        "-DCMAKE_CXX_COMPILER=g++",
+                        "-DCMAKE_BUILD_TYPE=Release",
+                        "-DCLANG_ENABLE_ARCMT=OFF",
+                        "-DCLANG_ENABLE_CLANGD=OFF",
+                        "-DLLVM_BUILD_RUNTIME=OFF",
+                        "-DLLVM_CCACHE_BUILD=ON",
+                        "-DLLVM_INCLUDE_EXAMPLES=OFF",
+                        "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-scei-ps4",
+                        "-DLLVM_ENABLE_ASSERTIONS=ON",
+                        "-DLLVM_LIT_ARGS=--verbose -j100",
+                        "-DLLVM_TARGETS_TO_BUILD=X86",
+                        "-DLLVM_USE_LINKER=gold"])},
+
 # Expensive checks builders.
 
     {'name' : "llvm-clang-x86_64-expensive-checks-ubuntu",
     'tags'  : ["llvm", "expensive-checks"],
     'workernames' : ["as-builder-4"],
     'builddir': "llvm-clang-x86_64-expensive-checks-ubuntu",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
-                    depends_on_projects=["llvm"],
+                    depends_on_projects=["llvm", "lld"],
                     clean=True,
                     extra_configure_args=[
                         "-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
                         "-DLLVM_ENABLE_WERROR=OFF",
                         "-DCMAKE_BUILD_TYPE=Debug",
                         "-DCMAKE_CXX_FLAGS=-U_GLIBCXX_DEBUG",
                         "-DLLVM_LIT_ARGS=-vv -j32"])},
 
     {'name' : "llvm-clang-x86_64-expensive-checks-win",
-    'tags'  : ["llvm", "clang", "expensive-checks"],
+    'tags'  : ["llvm", "expensive-checks"],
     'workernames' : ["as-worker-93"],
     'builddir': "llvm-clang-x86_64-expensive-checks-win",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
                     vs="autodetect",
+                    depends_on_projects=["llvm", "lld"],
                     clean=True,
                     extra_configure_args=[
                         "-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
                         "-DLLVM_ENABLE_WERROR=OFF",
-                        "-DCMAKE_BUILD_TYPE=Debug",
-                        "-DLLVM_LIT_ARGS=-vv"])},
+                        "-DCMAKE_BUILD_TYPE=Debug"])},
 
     {'name' : "llvm-clang-x86_64-expensive-checks-debian",
     'tags'  : ["llvm", "expensive-checks"],
     'collapseRequests' : False,
     'workernames' : ["gribozavr4"],
     'builddir': "llvm-clang-x86_64-expensive-checks-debian",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
-                    depends_on_projects=["llvm"],
+                    depends_on_projects=["llvm", "lld"],
                     clean=True,
                     extra_configure_args=[
                         "-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
                         "-DLLVM_ENABLE_WERROR=OFF",
                         "-DCMAKE_BUILD_TYPE=Release",
                         "-DCMAKE_CXX_FLAGS=-U_GLIBCXX_DEBUG",
                         "-DLLVM_LIT_ARGS=-v -vv -j96"],
                     env={
'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory(
                             "runtimes/runtimes-bins/libcxxabi/test"]),
                         ("libc++",
                             ['python', 'bin/llvm-lit.py',
                             '-v', '-vv', '--threads=32',
                             'runtimes/runtimes-bins/libcxx/test',
                             ])
                     ],
                     extra_configure_args=[
+                        "-DLLVM_TARGETS_TO_BUILD=ARM",
+                        "-DTARGET_TRIPLE=armv7-linux-gnueabihf",
                         "-DDEFAULT_SYSROOT=C:/buildbot/.arm-ubuntu",
                         "-DLLVM_LIT_ARGS=-v -vv --threads=32",
                         WithProperties("%(remote_test_host:+-DREMOTE_TEST_HOST=)s%(remote_test_host:-)s"),
                         WithProperties("%(remote_test_user:+-DREMOTE_TEST_USER=)s%(remote_test_user:-)s"),
                     ],
                     cmake_cache="../llvm-project/clang/cmake/caches/CrossWinToARMLinux.cmake")},
 
     {'name' : "llvm-clang-win-x-aarch64",
'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory(
                         ("libc++",
                             ['python', 'bin/llvm-lit.py',
                             '-v', '-vv', '--threads=32',
                             'runtimes/runtimes-bins/libcxx/test',
                             ])
                     ],
                     extra_configure_args=[
                         "-DLLVM_TARGETS_TO_BUILD=AArch64",
-                        "-DCMAKE_C_COMPILER_TARGET=aarch64-linux-gnu",
+                        "-DTARGET_TRIPLE=aarch64-linux-gnu",
                         "-DDEFAULT_SYSROOT=C:/buildbot/.aarch64-ubuntu",
                         "-DLLVM_LIT_ARGS=-v -vv --threads=32",
                         WithProperties("%(remote_test_host:+-DREMOTE_TEST_HOST=)s%(remote_test_host:-)s"),
                         WithProperties("%(remote_test_user:+-DREMOTE_TEST_USER=)s%(remote_test_user:-)s"),
                     ],
                     cmake_cache="../llvm-project/clang/cmake/caches/CrossWinToARMLinux.cmake")},
 
 # Clang builders.
 
     {'name': "clang-arm64-windows-msvc",
     'tags' : ["clang"],
     'workernames' : ["linaro-armv8-windows-msvc-01", "linaro-armv8-windows-msvc-02"],
     'builddir': "clang-arm64-windows-msvc",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     vs="manual",
                     test=False, # Disable testing until MCJIT failures are fixed
                     extra_cmake_args=[
                         "-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-windows-msvc",
                         "-DLLVM_HOST_TRIPLE=aarch64-windows-msvc",
                         "-DLLVM_TARGET_ARCH=AArch64",
+                        # FIXME: compiler-rt\lib\sanitizer_common\sanitizer_unwind_win.cpp assumes WIN64 is x86_64,
+                        #        so, before that's fixed, disable everything that triggers its build.
                         "-DCOMPILER_RT_BUILD_SANITIZERS=OFF",
+                        "-DCOMPILER_RT_BUILD_MEMPROF=OFF",
                         "-DCOMPILER_RT_BUILD_XRAY=OFF"])},
 
     # Cortex-A15 LNT test-suite in Benchmark mode
     {'name' : "clang-native-arm-lnt-perf",
     'tags'  : ["clang"],
     'workernames':["linaro-tk1-02"],
     'builddir':"clang-native-arm-lnt-perf",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_compiler_rt=False,
                     checkout_lld=False,
                     test=False,
-                    useTwoStage=False,
                     runTestSuite=True,
                     testsuite_flags=[
                         '--cppflags', '-O3 -mcpu=cortex-a15 -mthumb',
                         '--threads=1', '--build-threads=4',
                         '--use-perf=all',
                         '--benchmarking-only', '--exec-multisample=3',
                         '--exclude-stat-from-submission=compile'],
                     extra_cmake_args=[
                         "-DLLVM_TARGETS_TO_BUILD='ARM'",
                         "-DLLVM_PARALLEL_LINK_JOBS=2"],
                     submitURL='http://lnt.llvm.org/submitRun',
                     testerName='LNT-Thumb2v7-A15-O3')},
 
     # ARMv7 LNT test-suite in test-only mode
-    {'name' : "clang-cmake-armv7-lnt",
+    {'name' : "clang-armv7-lnt",
     'tags'  : ["clang"],
-    'workernames' : ["linaro-armv7-lnt"],
-    'builddir': "clang-cmake-armv7-lnt",
+    'workernames' : ["linaro-clang-armv7-lnt"],
+    'builddir': "clang-armv7-lnt",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_compiler_rt=False,
                     checkout_lld=False,
                     test=False,
-                    useTwoStage=False,
                     runTestSuite=True,
                     testsuite_flags=[
                         '--cppflags', '-mcpu=cortex-a15 -marm',
-                        '--threads=32', '--build-threads=32'],
-                    extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM'"])},
+                        '--threads=32', '--build-threads=32'])},
 
-    ## ARMv7 check-all self-host NEON with CMake builder
-    {'name' : "clang-cmake-armv7-selfhost-neon",
+    ## ARMv7 check-all 2-stage
+    {'name' : "clang-armv7-2stage",
     'tags'  : ["clang"],
-    'workernames': ["linaro-armv7-selfhost"],
-    'builddir':"clang-cmake-armv7-selfhost-neon",
+    'workernames': ["linaro-clang-armv7-2stage"],
+    'builddir':"clang-armv7-2stage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_compiler_rt=False,
                     checkout_lld=False,
                     useTwoStage=True,
                     testStage1=False,
                     extra_cmake_args=[
                         "-DCMAKE_C_FLAGS='-mcpu=cortex-a15 -marm'",
-                        "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a15 -marm'",
-                        "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
+                        "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a15 -marm'"])},
 
-    ## ARMv7 check-all with CMake builder
-    {'name' : "clang-cmake-armv7-quick",
+    ## ARMv7 check-all
+    {'name' : "clang-armv7-quick",
     'tags'  : ["clang"],
-    'workernames':["linaro-armv7-quick"],
-    'builddir':"clang-cmake-armv7-quick",
+    'workernames':["linaro-clang-armv7-quick"],
+    'builddir':"clang-armv7-quick",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_compiler_rt=False,
                     checkout_lld=False,
-                    extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
+                    extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM'"])},
 
-    ## ARMv7 Clang + LLVM run test-suite with GlobalISel enabled
-    {'name' : "clang-cmake-armv7-global-isel",
+    ## ARMv7 run test-suite with GlobalISel enabled
+    {'name' : "clang-armv7-global-isel",
     'tags'  : ["clang"],
-    'workernames':["linaro-armv7-global-isel"],
-    'builddir':"clang-cmake-armv7-global-isel",
+    'workernames':["linaro-clang-armv7-global-isel"],
+    'builddir':"clang-armv7-global-isel",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_compiler_rt=False,
                     checkout_lld=False,
-                    test=True,
-                    useTwoStage=False,
                     runTestSuite=True,
                     testsuite_flags=[
                         '--cppflags', '-mcpu=cortex-a15 -marm -O0 -mllvm -global-isel -mllvm -global-isel-abort=0',
-                        '--threads=32', '--build-threads=32'],
-                    extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
+                        '--threads=32', '--build-threads=32'])},
 
-    ## ARMv7 check-all self-host with CMake builder
-    {'name' : "clang-cmake-armv7-selfhost",
+    ## ARMv7 VFPv3 check-all 2-stage
+    {'name' : "clang-armv7-vfpv3-2stage",
     'tags'  : ["clang"],
-    'workernames' : ["linaro-armv7-selfhost"],
-    'builddir': "clang-cmake-armv7-selfhost",
+    'workernames' : ["linaro-clang-armv7-vfpv3-2stage"],
+    'builddir': "clang-armv7-vfpv3-2stage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_compiler_rt=False,
                     checkout_lld=False,
                     useTwoStage=True,
                     testStage1=False,
                     extra_cmake_args=[
                         "-DCMAKE_C_FLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -marm'",
-                        "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -marm'",
-                        "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
+                        "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -marm'"])},
 
-    ## AArch64 Clang+LLVM check-all + test-suite
-    {'name' : "clang-cmake-aarch64-quick",
+    ## AArch64 check-all
+    {'name' : "clang-aarch64-quick",
     'tags'  : ["clang"],
-    'workernames' : ["linaro-aarch64-quick"],
-    'builddir': "clang-cmake-aarch64-quick",
+    'workernames' : ["linaro-clang-aarch64-quick"],
+    'builddir': "clang-aarch64-quick",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_compiler_rt=False,
                     checkout_lld=False,
-                    test=True,
-                    useTwoStage=False,
-                    runTestSuite=True,
-                    testsuite_flags=[
-                        '--cppflags', '-mcpu=cortex-a57',
-                        '--threads=32', '--build-threads=32'],
-                    extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
+                    extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='AArch64'"])},
 
-    ## AArch64 Self-hosting Clang+LLVM check-all + LLD + test-suite
-    {'name' : "clang-cmake-aarch64-lld",
+    ## AArch64 check-all + LLD + test-suite 2-stage
+    {'name' : "clang-aarch64-lld-2stage",
     'tags'  : ["lld"],
-    'workernames' : ["linaro-aarch64-lld"],
-    'builddir':"clang-cmake-aarch64-lld",
+    'workernames' : ["linaro-clang-aarch64-lld-2stage"],
+    'builddir':"clang-aarch64-lld-2stage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                 clean=False,
-                checkout_compiler_rt=True,
-                checkout_lld=True,
-                test=True,
                 useTwoStage=True,
                 runTestSuite=True,
                 testsuite_flags=[
                     '--cppflags', '-mcpu=cortex-a57 -fuse-ld=lld',
                     '--threads=32', '--build-threads=32'],
                 extra_cmake_args=[
                     "-DCMAKE_C_FLAGS='-mcpu=cortex-a57'",
                     "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'",
-                    "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'",
                     "-DLLVM_ENABLE_LLD=True"])},
 
-    ## AArch64 Clang+LLVM run test-suite at -O0 (GlobalISel is now default).
-    {'name' : "clang-cmake-aarch64-global-isel",
+    ## AArch64 run test-suite at -O0 (GlobalISel is now default).
+    {'name' : "clang-aarch64-global-isel",
     'tags'  : ["clang"],
-    'workernames' : ["linaro-aarch64-global-isel"],
-    'builddir': "clang-cmake-aarch64-global-isel",
+    'workernames' : ["linaro-clang-aarch64-global-isel"],
+    'builddir': "clang-aarch64-global-isel",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_compiler_rt=False,
                     checkout_lld=False,
-                    test=True,
-                    useTwoStage=False,
                     runTestSuite=True,
                     testsuite_flags=[
                         '--cppflags', '-O0',
-                        '--threads=32', '--build-threads=32'],
-                    extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
+                        '--threads=32', '--build-threads=32'])},
 
-    ## ARMv7 without neon; check-all 2-stage full compiler-rt + testsuite
-    {'name' : "clang-cmake-armv7-full",
+    ## ARMv7 VFPv3 check-all + compiler-rt + testsuite 2-stage
+    {'name' : "clang-armv7-vfpv3-full-2stage",
     'tags'  : ["clang"],
     'workernames' : ["linaro-tk1-06", "linaro-tk1-07", "linaro-tk1-08", "linaro-tk1-09"],
-    'builddir': "clang-cmake-armv7-full",
+    'builddir': "clang-armv7-vfpv3-full-2stage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_lld=False,
-                    checkout_compiler_rt=True,
-                    testStage1=False,
                     useTwoStage=True,
+                    testStage1=False,
                     runTestSuite=True,
                     testsuite_flags=[
                         '--cppflags', '-mcpu=cortex-a15 -mfpu=vfpv3 -marm',
                         '--threads=4', '--build-threads=4'],
                     extra_cmake_args=[
                         "-DCOMPILER_RT_TEST_COMPILER_CFLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -marm'",
-                        "-DLLVM_TARGETS_TO_BUILD='ARM'",
                         "-DLLVM_PARALLEL_LINK_JOBS=2"])},
 
-    ## ARMv7 Thumb2 with neon; check-all 2-stage full compiler-rt + testsuite
-    {'name' : "clang-cmake-thumbv7-full-sh",
+    ## ARMv7 Thumb2 check-all + compiler-rt + testsuite 2-stage
+    {'name' : "clang-thumbv7-full-2stage",
     'tags'  : ["clang"],
     'workernames' : ["linaro-tk1-01", "linaro-tk1-03", "linaro-tk1-04", "linaro-tk1-05"],
-    'builddir': "clang-cmake-thumbv7-full-sh",
+    'builddir': "clang-thumbv7-full-2stage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_lld=False,
-                    checkout_compiler_rt=True,
                     testStage1=False,
                     useTwoStage=True,
                     runTestSuite=True,
                     testsuite_flags=[
                         '--cppflags', '-mcpu=cortex-a15 -mthumb',
                         '--threads=4', '--build-threads=4'],
                     extra_cmake_args=[
                         "-DCMAKE_C_FLAGS='-mcpu=cortex-a15 -mthumb'",
                         "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a15 -mthumb'",
                         "-DCOMPILER_RT_TEST_COMPILER_CFLAGS='-mcpu=cortex-a15 -mthumb'",
-                        "-DLLVM_TARGETS_TO_BUILD='ARM'",
                         "-DLLVM_PARALLEL_LINK_JOBS=2"])},
 
     ## AArch32 Self-hosting Clang+LLVM check-all + LLD + test-suite
     # Sanitizers build disabled due to PR38690
-    {'name' : "clang-cmake-armv8-lld",
+    {'name' : "clang-armv8-lld-2stage",
     'tags'  : ["lld"],
-    'workernames' : ["linaro-armv8-lld"],
-    'builddir': "clang-cmake-armv8-lld",
+    'workernames' : ["linaro-clang-armv8-lld-2stage"],
+    'builddir': "clang-armv8-lld-2stage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
-                    checkout_compiler_rt=True,
-                    checkout_lld=True,
-                    test=True,
                     useTwoStage=True,
                     runTestSuite=True,
                     testsuite_flags=[
                         '--cppflags', '-mcpu=cortex-a57 -fuse-ld=lld',
                         '--threads=32', '--build-threads=32'],
                     extra_cmake_args=[
                         "-DCMAKE_C_FLAGS='-mcpu=cortex-a57'",
                         "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'",
-                        "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'",
                         "-DCOMPILER_RT_BUILD_SANITIZERS=OFF",
                         "-DLLVM_ENABLE_LLD=True"])},
 
-    # AArch64 Clang+LLVM+RT check-all + flang + test-suite + self-hosting
-    {'name' : "clang-cmake-aarch64-full",
+    # AArch64 check-all + flang + compiler-rt + test-suite 2-stage
+    {'name' : "clang-aarch64-full-2stage",
     'tags'  : ["clang"],
-    'workernames' : ["linaro-aarch64-full"],
-    'builddir': "clang-cmake-aarch64-full",
+    'workernames' : ["linaro-clang-aarch64-full-2stage"],
+    'builddir': "clang-aarch64-full-2stage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
-                    checkout_compiler_rt=True,
                     checkout_flang=True,
                     checkout_lld=False,
-                    test=True,
                     useTwoStage=True,
                     testStage1=False,
                     runTestSuite=True,
                     testsuite_flags=[
                         '--cppflags', '-mcpu=cortex-a57',
                         '--threads=32', '--build-threads=32'],
                     extra_cmake_args=[
                         "-DCMAKE_C_FLAGS='-mcpu=cortex-a57'",
-                        "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'",
-                        "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
+                        "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'"])},
+
+    # AArch64 Clang+LLVM+RT+LLD check-all + flang + test-suite w/SVE
+    {'name' : "clang-aarch64-sve-vls",
+    'tags'  : ["clang"],
+    'workernames' : ["linaro-clang-aarch64-sve-vls"],
+    'builddir': "clang-aarch64-sve-vls",
+    'factory' : ClangBuilder.getClangCMakeBuildFactory(
+                    clean=False,
+                    checkout_flang=True,
+                    runTestSuite=True,
+                    testsuite_flags=[
+                        '--cppflags', '-mcpu=a64fx -mllvm -aarch64-sve-vector-bits-min=512',
+                        '--threads=48', '--build-threads=48'],
+                    extra_cmake_args=[
+                        "-DCMAKE_C_FLAGS='-mcpu=a64fx'",
+                        "-DCMAKE_CXX_FLAGS='-mcpu=a64fx'",
+                        "-DLLVM_ENABLE_LLD=True"])},
+
+    # AArch64 Clang+LLVM+RT+LLD check-all + flang + test-suite 2-stage w/SVE
+    {'name' : "clang-aarch64-sve-vls-2stage",
+    'tags'  : ["clang"],
+    'workernames' : ["linaro-clang-aarch64-sve-vls-2stage"],
+    'builddir': "clang-aarch64-sve-vls-2stage",
+    'factory' : ClangBuilder.getClangCMakeBuildFactory(
+                    clean=False,
+                    checkout_flang=True,
+                    useTwoStage=True,
+                    testStage1=False,
+                    runTestSuite=True,
+                    testsuite_flags=[
+                        '--cppflags', '-mcpu=a64fx -mllvm -aarch64-sve-vector-bits-min=512',
+                        '--threads=48', '--build-threads=48'],
+                    extra_cmake_args=[
+                        "-DCMAKE_C_FLAGS='-mcpu=a64fx -mllvm -aarch64-sve-vector-bits-min=512'",
+                        "-DCMAKE_CXX_FLAGS='-mcpu=a64fx -mllvm -aarch64-sve-vector-bits-min=512'",
+                        "-DLLVM_ENABLE_LLD=True"])},
 
     {'name' : "clang-arm64-windows-msvc-2stage",
     'tags'  : ["clang"],
     'workernames' : ["linaro-armv8-windows-msvc-01", "linaro-armv8-windows-msvc-02"],
     'builddir': "clang-arm64-windows-msvc-2stage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     vs="manual",
                     test=False, # Disable testing until MCJIT failures are fixed
                     useTwoStage=True,
                     testStage1=False,
                     extra_cmake_args=[
                         "-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-windows-msvc",
                         "-DLLVM_HOST_TRIPLE=aarch64-windows-msvc",
                         "-DLLVM_TARGET_ARCH=AArch64",
+                        # FIXME: compiler-rt\lib\sanitizer_common\sanitizer_unwind_win.cpp assumes WIN64 is x86_64,
+                        #        so, before that's fixed, disable everything that triggers its build.
                         "-DCOMPILER_RT_BUILD_SANITIZERS=OFF",
+                        "-DCOMPILER_RT_BUILD_MEMPROF=OFF",
                         "-DCOMPILER_RT_BUILD_XRAY=OFF"])},
 
     {'name' : 'clang-x64-windows-msvc',
     'tags'  : ["clang"],
     'workernames' : ['windows-gcebot2'],
     'builddir': 'clang-x64-windows-msvc',
     'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
                     script="clang-windows.py",
                     depends_on_projects=['llvm', 'clang', 'lld', 'debuginfo-tests'])},
 
+    {'name' : "clang-m68k-linux",
+    'tags'  : ["clang"],
+    'workernames' : ["debian-akiko-m68k"],
+    'builddir': "clang-m68k-linux",
+    'factory' : ClangBuilder.getClangCMakeBuildFactory(
+                    clean=False,
+                    checkout_lld=False,
+                    useTwoStage=False,
+                    stage1_config='Release',
+                    extra_cmake_args=[
+                        "-DLLVM_ENABLE_ASSERTIONS=ON",
+                        "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=M68k"])},
+
+    {'name' : "clang-m68k-linux-cross",
+    'tags'  : ["clang"],
+    'workernames' : ["suse-gary-m68k-cross"],
+    'builddir': "clang-m68k-linux-cross",
+    'factory' : ClangBuilder.getClangCMakeBuildFactory(
+                    clean=False,
+                    checkout_lld=False,
+                    useTwoStage=False,
+                    stage1_config='Release',
+                    extra_cmake_args=[
+                        "-DLLVM_ENABLE_ASSERTIONS=ON",
+                        "-DLLVM_TARGETS_TO_BUILD=X86",
+                        "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=M68k"])},
+
     {'name' : "clang-ppc64be-linux-lnt",
     'tags'  : ["clang", "ppc"],
     'workernames' : ["ppc64be-clang-lnt-test"],
     'builddir': "clang-ppc64be-lnt",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_lld=False,
                     useTwoStage=False,
# Clang builders.
     'tags'  : ["clang", "ppc", "ppc64le"],
     'workernames' : ["ppc64le-clang-test"],
     'builddir': "clang-ppc64le",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
                     checkout_lld=False,
                     useTwoStage=False,
                     stage1_config='Release',
-                    extra_cmake_args=["-DLLVM_ENABLE_ASSERTIONS=ON"])},
+                    extra_cmake_args=[
+                        "-DLLVM_ENABLE_ASSERTIONS=ON",
+                        "-DPYTHON_EXECUTABLE:FILEPATH=/usr/local/bin/python3"])},
 
     {'name' : "clang-ppc64le-rhel",
     'tags'  : ["clang", "ppc", "ppc64le"],
     'workernames' : ["ppc64le-clang-rhel-test"],
     'builddir': "clang-ppc64le-rhel",
-    'factory' : ClangBuilder.getClangCMakeBuildFactory(clean=False,
-                    checkout_clang_tools_extra=True,
-                    checkout_compiler_rt=False,
-                    checkout_lld=True,
-                    checkout_libcxx=False,
-                    useTwoStage=False,
-                    runTestSuite=True,
-                    stage1_config='Release',
-                    nt_flags=['--threads=16', '--build-threads=16'],
-                    extra_cmake_args=[
-                        '-DLLVM_ENABLE_PROJECTS=clang;llvm;clang-tools-extra;lld',
-                        '-DLLVM_ENABLE_RUNTIMES=compiler-rt',
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    depends_on_projects=["llvm", "clang", "clang-tools-extra", "lld", "compiler-rt"],
+                    checks=["check"],
+                    enable_runtimes="auto",
+                    extra_configure_args=[
                         "-DLLVM_ENABLE_ASSERTIONS=On", "-DCMAKE_C_COMPILER=clang",
                         "-DCMAKE_CXX_COMPILER=clang++",
                         "-DCLANG_DEFAULT_LINKER=lld",
                         "-DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN:PATH=/opt/rh/devtoolset-7/root/usr",
                         "-DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN:PATH=/opt/rh/devtoolset-7/root/usr",
                         "-DLLVM_BINUTILS_INCDIR=/usr/include", "-DBUILD_SHARED_LIBS=ON", "-DLLVM_ENABLE_WERROR=ON",
-                        '-DLLVM_LIT_ARGS=-vj 20'])},
+                        "-DCMAKE_BUILD_TYPE=Release",
+                        "-DLLVM_LIT_ARGS=-vj 20"])},
+
+    {'name' : "clang-ppc64-aix-ppc64le",
+    'tags'  : ["clang", "aix", "ppc", "ppc64le"],
+    'workernames' : ["aix-ppc64-ppc64le"],
+    'builddir': "clang-ppc64-aix-ppc64le",
+    'factory' : ClangBuilder.getClangCMakeBuildFactory(
+                    clean=False,
+                    checkout_clang_tools_extra=False,
+                    checkout_compiler_rt=False,
+                    checkout_lld=False,
+                    checkout_libcxx=False,
+                    useTwoStage=False,
+                    runTestSuite=False,
+                    stage1_config='Release',
+                    extra_cmake_args=[
+                        "-DLLVM_ENABLE_ASSERTIONS=On",
+                        "-DCMAKE_C_COMPILER=/opt/IBM/xlC/16.1.0/bin/xlclang",
+                        "-DCMAKE_CXX_COMPILER=/opt/IBM/xlC/16.1.0/bin/xlclang++",
+                        "-DCMAKE_C_FLAGS='-qmaxmem=-1 -qarch=pwr8 -qaltivec -qalias=noansi'",
+                        "-DCMAKE_CXX_FLAGS='-qmaxmem=-1 -qarch=pwr8 -qaltivec -qalias=noansi'",
+                        "-DPython3_EXECUTABLE:FILEPATH=/opt/freeware/bin/python3_64",
+                        "-DLLVM_ENABLE_ZLIB=OFF", "-DLLVM_APPEND_VC_REV=OFF",
+                        "-DLLVM_DEFAULT_TARGET_TRIPLE=powerpc64le-unknown-linux-gnu",
+                        "-DLLVM_PARALLEL_LINK_JOBS=6",
+                        "-DLLVM_ENABLE_WERROR=ON"])},
 
     {'name' : "clang-s390x-linux",
     'tags'  : ["clang"],
     'workernames' : ["systemz-1"],
     'builddir': "clang-s390x-linux",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     jobs=4,
                     clean=False,
# Clang builders.
     'tags'  : ["clang"],
     'workernames' : ["hexagon-build-02", "hexagon-build-03"],
     'builddir': "clang-hexagon-elf",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     jobs=16,
                     checkout_clang_tools_extra=False,
                     checkout_compiler_rt=False,
                     checkout_lld=False,
-                    env={'LD_LIBRARY_PATH': '/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/lib', 'PATH': ['/local/cmake-3.17.0/bin', '${PATH}']},
+                    env={'LD_LIBRARY_PATH': '/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/lib'},
                     extra_cmake_args=[
                         "-DCMAKE_BUILD_TYPE:STRING=Release",
                         "-DLLVM_TARGETS_TO_BUILD:STRING=Hexagon",
                         "-DTARGET_TRIPLE:STRING=hexagon-unknown-elf",
                         "-DLLVM_DEFAULT_TARGET_TRIPLE:STRING=hexagon-unknown-elf",
                         "-DLLVM_TARGET_ARCH:STRING=hexagon-unknown-elf",
                         "-DLLVM_BUILD_RUNTIME:BOOL=OFF",
                         "-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON",
                         "-DLLVM_ENABLE_PIC:BOOL=ON",
                         "-DLLVM_ENABLE_ASSERTIONS:BOOL=ON",
                         "-DLLVM_INCLUDE_TOOLS:BOOL=ON",
                         "-DLLVM_LIT_ARGS:STRING=-v",
                         "-DLLVM_ENABLE_LIBCXX:BOOL=ON",
                         "-DWITH_POLLY:BOOL=OFF",
                         "-DLINK_POLLY_INTO_TOOLS:BOOL=OFF",
                         "-DPOLLY_BUILD_SHARED_LIB:BOOL=OFF",
-                        "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang",
-                        "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])},
+                        "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang",
+                        "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang++"])},
 
     ## X86_64 AVX2 Clang+LLVM check-all + test-suite
     {'name' : "clang-cmake-x86_64-avx2-linux",
     'tags'  : ["clang"],
     'workernames' : ["avx2-intel64"],
     'builddir': "clang-cmake-x86_64-avx2-linux",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
'factory' : ClangBuilder.getClangCMakeBuildFactory(
                         '--make-param', 'RUNTIMELIMIT=1200'],
                     env={'PATH':'/home/ssglocal/tools/sde/latest:/usr/bin/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'},
                     extra_cmake_args=[
                         "-DLLVM_ENABLE_ASSERTIONS=ON",
                         "-DCMAKE_C_FLAGS='-march=broadwell'",
                         "-DCMAKE_CXX_FLAGS='-march=broadwell'",
                         "-DLLVM_TARGETS_TO_BUILD='X86'"])},
 
-    ## Armv7 build cache
-    {'name' : "clang-armv7-linux-build-cache",
-    'tags'  : ["clang"],
-    'workernames' : ["packet-linux-armv7-slave-1"],
-    'builddir': "clang-armv7-linux-build-cache",
-    'factory' : ClangBuilder.getClangCMakeGCSBuildFactory(
-                    stage1_config='Release',
-                    clean=True,
-                    checkout_compiler_rt=False,
-                    test=False,
-                    useTwoStage=False,
-                    runTestSuite=False,
-                    checkout_lld=True,
-                    checkout_libcxx=True,
-                    checkout_clang_tools_extra=False,
-                    use_pixz_compression=False,
-                    xz_compression_factor=0,
-                    stage1_upload_directory='clang-armv7-linux',
-                    extra_cmake_args=[
-                        "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64;X86'",
-                        "-DCMAKE_C_FLAGS='-mthumb'",
-                        "-DCMAKE_CXX_FLAGS='-mthumb'",
-                        ],
-                    env={'BUCKET': 'llvm-build-artifacts'})},
-
-    ## AArch64 build cache
-    {'name' : "clang-aarch64-linux-build-cache",
-    'tags'  : ["clang"],
-    'workernames' : ["packet-linux-aarch64-slave-1"],
-    'builddir': "clang-aarch64-linux-build-cache",
-    'factory' : ClangBuilder.getClangCMakeGCSBuildFactory(
-                    stage1_config='Release',
-                    clean=True,
-                    checkout_compiler_rt=False,
-                    test=False,
-                    useTwoStage=False,
-                    runTestSuite=False,
-                    checkout_lld=True,
-                    checkout_libcxx=True,
-                    checkout_clang_tools_extra=False,
-                    stage1_upload_directory='clang-aarch64-linux',
-                    use_pixz_compression=True,
-                    extra_cmake_args=[
-                        "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64;X86'",
-                        #"-DCMAKE_C_FLAGS=''",
-                        #"-DCMAKE_CXX_FLAGS=''",
-                        ],
-                    env={'BUCKET': 'llvm-build-artifacts'})},
-
     {'name' : "llvm-avr-linux",
     'tags'  : ["clang"],
     'workernames' : ["avr-build-01"],
     'builddir': "llvm-avr-linux",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     jobs=8,
                     clean=False,
                     checkout_lld=False,
'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     testStage1=True,
                     useTwoStage=True,
                     stage1_config='Release',
                     stage2_config='Release',
                     extra_cmake_args=[
                         '-DLLVM_ENABLE_ASSERTIONS=ON',
                         '-DLLVM_TARGETS_TO_BUILD=X86'])},
 
+    {'name' : "clang-xcore-ubuntu-20-x64",
+    'tags'  : ["clang"],
+    'workernames' : ["xcore-ubuntu20-x64"],
+    'builddir': "clang-xcore-ubuntu-20-x64",
+    'factory' : ClangBuilder.getClangCMakeBuildFactory(
+                    jobs=4,
+                    checkout_clang_tools_extra=False,
+                    checkout_compiler_rt=False,
+                    checkout_lld=False,
+                    test=True,
+                    testStage1=True,
+                    useTwoStage=False,
+                    stage1_config='Release',
+                    extra_cmake_args=[
+                        "-DLLVM_TARGETS_TO_BUILD:STRING=XCore",
+                        "-DLLVM_DEFAULT_TARGET_TRIPLE:STRING=xcore-unknown-unknown-elf",
+                        "-DLLVM_ENABLE_THREADS:BOOL=OFF"])},
+
 # Polly builders.
 
     {'name' : "polly-arm-linux",
     'tags'  : ["polly"],
     'workernames' : ["hexagon-build-02", "hexagon-build-03"],
     'builddir': "polly-arm-linux",
     'factory' : PollyBuilder.getPollyBuildFactory(
                     clean=True,
                     install=True,
                     make='ninja',
                     jobs=16,
-                    env={'LD_LIBRARY_PATH': '/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/lib', 'PATH': ['/local/cmake-3.17.0/bin', '${PATH}']},
+                    env={'LD_LIBRARY_PATH': '/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/lib'},
                     extraCmakeArgs=[
                         "-G", "Ninja",
                         "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'",
                         "-DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-gnueabi",
                         "-DLLVM_TARGET_ARCH=arm-linux-gnueabi",
                         "-DLLVM_ENABLE_ASSERTIONS=True",
                         "-DLLVM_ENABLE_LIBCXX:BOOL=ON",
-                        "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang",
-                        "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])},
+                        "-DPOLLY_ENABLE_GPGPU_CODEGEN=OFF",  # Auto-enabled in getPollyBuildFactory
+                        "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang",
+                        "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang++"])},
 
     {'name' : "polly-x86_64-linux",
     'tags'  : ["polly"],
     'workernames' : ["polly-x86_64-fdcserver", "polly-x86_64-gce1"],
     'builddir': "polly-x86_64-linux",
     'factory' : PollyBuilder.getPollyBuildFactory(
                     clean=False,
                     install=False,
'factory' : AOSPBuilder.getAOSPBuildFactory(
                     device="angler",
                     extra_cmake_args=[
                         "-G", "Ninja",
                         "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'",
                         "-DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-androideabi",
                         "-DLLVM_TARGET_ARCH=arm-linux-androideabi",
                         "-DLLVM_ENABLE_ASSERTIONS=True",
                         "-DLLVM_ENABLE_LIBCXX:BOOL=ON",
-                        "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang",
-                        "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"],
+                        "-DPOLLY_ENABLE_GPGPU_CODEGEN=OFF",  # Auto-enabled in getPollyBuildFactory (used indirectly)
+                        "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang",
+                        "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang++"],
                     timeout=240,
                     target_clang=None,
                     target_flags="-Wno-error -O3 -mllvm -polly -mllvm -polly-position=before-vectorizer -mllvm -polly-process-unprofitable -fcommon",
-                    jobs=8,
+                    jobs=32,
                     extra_make_args=None,
-                    env={'LD_LIBRARY_PATH': '/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/lib', 'PATH': ['/local/cmake-3.17.0/bin', '${PATH}']},
+                    env={'LD_LIBRARY_PATH': '/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/lib'},
                     clean=False,
                     sync=False,
                     patch=None)},
 
 # Reverse iteration builders.
 
     {'name' : "reverse-iteration",
     'tags'  : ["rev_iter"],
     'workernames' : ["hexagon-build-02", "hexagon-build-03"],
     'builddir': "reverse-iteration",
     'factory' : PollyBuilder.getPollyBuildFactory(
                     clean=True,
                     make='ninja',
                     jobs=16,
                     checkAll=True,
-                    env={'LD_LIBRARY_PATH': '/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/lib', 'PATH': ['/local/cmake-3.17.0/bin', '${PATH}']},
+                    env={'LD_LIBRARY_PATH': '/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/lib'},
                     extraCmakeArgs=[
                         "-G", "Ninja",
                         "-DLLVM_REVERSE_ITERATION:BOOL=ON",
                         "-DLLVM_ENABLE_ASSERTIONS=True",
                         "-DLLVM_ENABLE_LIBCXX:BOOL=ON",
-                        "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang",
-                        "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])},
+                        "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang",
+                        "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang++"])},
 
 # LLDB builders.
 
     {'name' : "lldb-x86_64-debian",
     'tags'  : ["lldb"],
     'workernames' : ["lldb-x86_64-debian"],
     'builddir': "lldb-x86_64-debian",
     'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
                     test=True,
                     extra_cmake_args=[
                         '-DLLVM_ENABLE_ASSERTIONS=True',
                         '-DLLVM_USE_LINKER=gold',
                         '-DLLDB_ENABLE_PYTHON=True',
                         '-DPYTHON_EXECUTABLE=/usr/bin/python3',
                         '-DCMAKE_C_COMPILER=clang',
                         '-DCMAKE_CXX_COMPILER=clang++'])},
 
     {'name' : "lldb-aarch64-ubuntu",
     'tags'  : ["lldb"],
-    'workernames' : ["linaro-aarch64-lldb"],
-    'builddir': "lldb-cmake-aarch64",
+    'workernames' : ["linaro-lldb-aarch64-ubuntu"],
+    'builddir': "lldb-aarch64-ubuntu",
     'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
                     test=True,
                     extra_cmake_args=[
                         '-DLLVM_ENABLE_ASSERTIONS=True',
-                        '-DLLVM_LIT_ARGS=-svj 8',
+                        '-DLLVM_LIT_ARGS=-svj 4',
                         '-DLLVM_USE_LINKER=gold'])},
 
     {'name' : "lldb-arm-ubuntu",
     'tags'  : ["lldb"],
-    'workernames' : ["linaro-arm-lldb"],
-    'builddir': "lldb-cmake-arm",
+    'workernames' : ["linaro-lldb-arm-ubuntu"],
+    'builddir': "lldb-arm-ubuntu",
     'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
                     test=True,
                     extra_cmake_args=[
                         '-DLLVM_ENABLE_ASSERTIONS=True',
-                        '-DLLVM_LIT_ARGS=-svj 8',
+                        '-DLLVM_LIT_ARGS=-svj 4',
                         '-DLLVM_USE_LINKER=gold'])},
 
     {'name' : "lldb-x64-windows-ninja",
     'tags'  : ["lldb"],
     'workernames' : ["win-py3-buildbot"],
     'builddir': "lldb-x64-windows-ninja",
     'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
                     clean=True,
# LLD builders.
     'workernames' : ["as-worker-93"],
     'builddir': "lld-x86_64-win",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
                     depends_on_projects=['llvm', 'lld'],
                     vs="autodetect",
                     extra_configure_args = [
                         '-DLLVM_ENABLE_WERROR=OFF'])},
 
-    {'name' : "lld-x86_64-freebsd",
-    'tags'  : ["lld"],
-    'workernames' : ["as-worker-4"],
-    'builddir': "lld-x86_64-freebsd",
-    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
-                    depends_on_projects=['llvm', 'lld'],
-                    extra_configure_args=[
-                        '-DCMAKE_EXE_LINKER_FLAGS=-lcxxrt',
-                        '-DLLVM_ENABLE_WERROR=OFF'],
-                    env={'CXXFLAGS' : "-std=c++11 -stdlib=libc++"})},
-
     {'name' : "ppc64le-lld-multistage-test",
     'tags'  : ["lld", "ppc", "ppc64le"],
     'workernames' : ["ppc64le-lld-multistage-test"],
     'builddir': "ppc64le-lld-multistage-test",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaMultistageBuildFactory(
                     extra_configure_args=[
                         '-DLLVM_ENABLE_ASSERTIONS=ON',
                         '-DLLVM_LIT_ARGS=-svj 256'],
# LTO and ThinLTO builders.
 
     {'name' : "clang-with-thin-lto-wpd-ubuntu",
     'tags'  : ["clang","lld","LTO"],
     'workernames' : ["thinlto-x86-64-bot1"],
     'builddir': "clang-with-thin-lto-wpd-ubuntu",
     'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(
                     jobs=72,
                     lto='thin',
-                    extra_configure_args=[
+                    extra_configure_args_lto_stage=[
                         '-DLLVM_CCACHE_BUILD=ON',
-                        '-DCMAKE_CXX_FLAGS="-O3 -Xclang -fwhole-program-vtables -fno-split-lto-unit"',
-                        '-DCMAKE_C_FLAGS="-O3 -Xclang -fwhole-program-vtables -fno-split-lto-unit"',
-                        '-DCMAKE_EXE_LINKER_FLAGS="-Wl,--lto-whole-program-visibility"'])},
+                        '-DCMAKE_CXX_FLAGS=-O3 -Xclang -fwhole-program-vtables -fno-split-lto-unit',
+                        '-DCMAKE_C_FLAGS=-O3 -Xclang -fwhole-program-vtables -fno-split-lto-unit',
+                        '-DCMAKE_EXE_LINKER_FLAGS=-Wl,--lto-whole-program-visibility -fuse-ld=lld'])},
 
     {'name' : "clang-with-lto-ubuntu",
     'tags'  : ["clang","lld","LTO"],
     'workernames' : ["as-worker-91"],
     'builddir': "clang-with-lto-ubuntu",
-    'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(jobs=72)},
+    'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(
+                    jobs=72,
+                    extra_configure_args_lto_stage=[
+                        '-DLLVM_PARALLEL_LINK_JOBS=14',
+                    ])},
 
 # Builders for MLIR.
 
     {'name' : "mlir-nvidia",
     'tags'  : ["mlir"],
     'workernames' : ["mlir-nvidia"],
     'builddir': "mlir-nvidia",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     llvm_srcdir="llvm.src",
                     obj_dir="llvm.obj",
                     clean=True,
                     depends_on_projects=['llvm','mlir'],
                     extra_configure_args=[
                         '-DLLVM_BUILD_EXAMPLES=ON',
                         '-DLLVM_TARGETS_TO_BUILD=host;NVPTX',
                         '-DLLVM_ENABLE_PROJECTS=mlir',
-                        '-DMLIR_CUDA_RUNNER_ENABLED=1',
+                        '-DMLIR_ENABLE_CUDA_RUNNER=1',
+                        '-DMLIR_INCLUDE_INTEGRATION_TESTS=ON',
                         '-DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc',
-                        '-DMLIR_VULKAN_RUNNER_ENABLED=1',
+                        '-DMLIR_ENABLE_VULKAN_RUNNER=1',
                         '-DBUILD_SHARED_LIBS=ON',
                         '-DLLVM_CCACHE_BUILD=ON',
+                        '-DMLIR_ENABLE_BINDINGS_PYTHON=ON',
+                        '-DMLIR_RUN_CUDA_TENSOR_CORE_TESTS=ON',
                     ],
                     env={
                         'CC':'clang',
                         'CXX': 'clang++',
                         'LD': 'lld',
                     })},
 
     {'name' : "mlir-windows",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                         '-DCMAKE_CXX_STANDARD=17',
                         '-DLLVM_ENABLE_PROJECTS=mlir',
                         '-DLLVM_LIT_ARGS=-vj 256',
                     ],
                     env={
                             'CC': 'clang',
                             'CXX': 'clang++',
                             'LD': 'lld',
+                            'LD_LIBRARY_PATH': '/usr/lib64',
                     })},
 
 # Sanitizer builders.
 
     {'name' : "sanitizer-x86_64-linux",
     'tags'  : ["sanitizer"],
-    'workernames' : ["sanitizer-buildbot1", "sanitizer-buildbot2"],
+    'workernames' : [
+        "sanitizer-buildbot1",
+        "sanitizer-buildbot2",
+    ],
     'builddir': "sanitizer-x86_64-linux",
     'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
 
     {'name' : "sanitizer-x86_64-linux-fast",
     'tags'  : ["sanitizer"],
-    'workernames' : ["sanitizer-buildbot1", "sanitizer-buildbot2"],
+    'workernames' : [
+        "sanitizer-buildbot1",
+        "sanitizer-buildbot2",
+        "sanitizer-buildbot7",
+        "sanitizer-buildbot8",
+    ],
     'builddir': "sanitizer-x86_64-linux-fast",
     'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
 
-    {'name' : "sanitizer-x86_64-linux-bootstrap",
+    {'name' : "sanitizer-x86_64-linux-bootstrap-asan",
     'tags'  : ["sanitizer"],
-    'workernames' : ["sanitizer-buildbot3", "sanitizer-buildbot4"],
-    'builddir': "sanitizer-x86_64-linux-bootstrap",
+    'workernames' : [
+        "sanitizer-buildbot3",
+        "sanitizer-buildbot4",
+    ],
+    'builddir': "sanitizer-x86_64-linux-bootstrap-asan",
     'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
 
     {'name' : "sanitizer-x86_64-linux-bootstrap-ubsan",
     'tags'  : ["sanitizer"],
-    'workernames' : ["sanitizer-buildbot3", "sanitizer-buildbot4"],
+    'workernames' : [
+        "sanitizer-buildbot3",
+        "sanitizer-buildbot4",
+    ],
     'builddir': "sanitizer-x86_64-linux-bootstrap-ubsan",
     'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
 
     {'name' : "sanitizer-x86_64-linux-bootstrap-msan",
     'tags'  : ["sanitizer"],
-    'workernames' : ["sanitizer-buildbot7", "sanitizer-buildbot8"],
+    'workernames' : [
+        "sanitizer-buildbot3",
+        "sanitizer-buildbot4",
+    ],
     'builddir': "sanitizer-x86_64-linux-bootstrap-msan",
     'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
 
     {'name' : "sanitizer-x86_64-linux-autoconf",
     'tags'  : ["sanitizer"],
-    'workernames' : ["sanitizer-buildbot7", "sanitizer-buildbot8"],
+    'workernames' : [
+        "sanitizer-buildbot7",
+        "sanitizer-buildbot8",
+    ],
     'builddir': "sanitizer-x86_64-linux-autoconf",
     'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
 
+    {'name' : "sanitizer-x86_64-linux-qemu",
+    'tags'  : ["sanitizer"],
+    'workernames' : [
+        "sanitizer-buildbot7",
+        "sanitizer-buildbot8",
+    ],
+    'builddir': "sanitizer-x86_64-linux-qemu",
+    'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
+
     {'name' : "sanitizer-x86_64-linux-fuzzer",
     'tags'  : ["sanitizer"],
-    'workernames' : ["sanitizer-buildbot7", "sanitizer-buildbot8"],
+    'workernames' : [
+        "sanitizer-buildbot7",
+        "sanitizer-buildbot8",
+    ],
     'builddir': "sanitizer-x86_64-linux-fuzzer",
     'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
 
     {'name' : "sanitizer-x86_64-linux-android",
     'tags'  : ["sanitizer"],
     'workernames' : ["sanitizer-buildbot6"],
     'builddir': "sanitizer-x86_64-linux-android",
     'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
 
     {'name' : "sanitizer-ppc64be-linux",
     'tags'  : ["sanitizer", "ppc"],
     'workernames' : ["ppc64be-sanitizer"],
     'builddir': "sanitizer-ppc64be",
     'factory' : SanitizerBuilder.getSanitizerBuildFactory(timeout=1800)},
 
     {'name' : "sanitizer-ppc64le-linux",
     'tags'  : ["sanitizer", "ppc", "ppc64le"],
     'workernames' : ["ppc64le-sanitizer"],
     'builddir': "sanitizer-ppc64le",
-    'factory' : SanitizerBuilder.getSanitizerBuildFactory(timeout=1800)},
+    'factory' : SanitizerBuilder.getSanitizerBuildFactory(
+                    timeout=1800,
+                    extra_configure_args=[
+                       "-CMAKE_ARGS='-DLLVM_LIT_ARGS=-v -j256'"])},
 
     {'name' : "sanitizer-windows",
     'tags'  : ["sanitizer"],
     'workernames' : ["sanitizer-windows"],
     'builddir': "sanitizer-windows",
     'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
                     script="sanitizer-windows.py",
                     depends_on_projects=["llvm", "clang", "lld", "compiler-rt"])},
 
+# OpenMP builders.
+
     {'name' : "openmp-gcc-x86_64-linux-debian",
     'tags'  : ["openmp"],
     'workernames' : ["gribozavr4"],
     'builddir': "openmp-gcc-x86_64-linux-debian",
     'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
                     env={
                         'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
                         'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes',
                     })},
 
     {'name' : "openmp-clang-x86_64-linux-debian",
     'tags'  : ["openmp"],
     'workernames' : ["gribozavr4"],
     'builddir': "openmp-clang-x86_64-linux-debian",
     'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
                     env={
                         'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
                         'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes',
                     })},
 
-# Libc++ builders.
-
-    {'name': 'libcxx-libcxxabi-x86_64-linux-debian',
-    'tags'  : ["libcxx"],
-    'workernames': ['gribozavr4'],
-    'builddir': 'libcxx-libcxxabi-x86_64-linux-debian',
-    'factory': LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
-                    env={'CC': 'clang', 'CXX': 'clang++'},
-                    lit_extra_args=['--shuffle'],
-                    check_libcxx_abilist=True)},
-
-    {'name' : 'libcxx-libcxxabi-x86_64-linux-debian-noexceptions',
-    'tags'  : ["libcxx"],
-    'workernames' : ['gribozavr4'],
-    'builddir': 'libcxx-libcxxabi-x86_64-linux-debian-noexceptions',
-    'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
-                    env={'CC': 'clang', 'CXX': 'clang++'},
-                    use_cache='Generic-noexceptions.cmake',
-                    lit_extra_args=['--shuffle'])},
-
-    {'name' : 'libcxx-libcxxabi-libunwind-x86_64-linux-debian',
-    'tags'  : ["libcxx"],
-    'workernames' : ['gribozavr4'],
-    'builddir': 'libcxx-libcxxabi-libunwind-x86_64-linux-debian',
-    'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
-             env={'CC': 'clang', 'CXX': 'clang++'},
-             cmake_extra_opts={'LIBCXXABI_USE_LLVM_UNWINDER': 'ON'},
-             lit_extra_args=['--shuffle'])},
-
-    {'name' : 'libcxx-libcxxabi-singlethreaded-x86_64-linux-debian',
-    'tags'  : ["libcxx"],
-    'workernames' : ['gribozavr4'],
-    'builddir': 'libcxx-libcxxabi-singlethreaded-x86_64-linux-debian',
-    'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
-                    env={'CC': 'clang', 'CXX': 'clang++'},
-                    use_cache='Generic-singlethreaded.cmake')},
+    {'name' : "openmp-offload-cuda-project",
+    'tags'  : ["openmp"],
+    'workernames' : ["minipc-1050ti-linux"],
+    'builddir': "openmp-offload-cuda-project",
+    'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
+                        clean=False,
+                        enable_runtimes=[],
+                        extraCmakeArgs=[
+                                "-DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda",
+                                "-DLIBOMPTARGET_BUILD_NVPTX_BCLIB=ON",
+                                "-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
+                                "-DCLANG_ENABLE_ARCMT=OFF",
+                                "-DCLANG_ENABLE_OBJC_REWRITER=OFF",
+                                "-DLLVM_TARGETS_TO_BUILD=X86;NVPTX",
+                                "-DLLVM_ENABLE_LLD=ON",
+                                '-DLLVM_PARALLEL_LINK_JOBS=2',
+                            ],
+                        install=True,
+                        testsuite=True,
+                        testsuite_sollvevv=True,
+                        extraTestsuiteCmakeArgs=[
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
+                        ],
+                    )},
 
-    # ARMv7 LibC++ and LibC++abi tests (require Clang+RT)
-    {'name' : 'libcxx-libcxxabi-libunwind-armv7-linux',
-    'tags'  : ["libcxx"],
-    'workernames': ['linaro-tk1-02'],
-    'builddir': 'libcxx-libcxxabi-libunwind-armv7-linux',
-    'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
-                    cmake_extra_opts={
-                        'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
-                        'CMAKE_C_FLAGS': '-mcpu=cortex-a15 -marm',
-                        'CMAKE_CXX_FLAGS': '-mcpu=cortex-a15 -marm',
-                        'LLVM_PARALLEL_LINK_JOBS': '2'})},
+    {'name' : "openmp-offload-cuda-runtime",
+    'tags'  : ["openmp"],
+    'workernames' : ["minipc-1050ti-linux"],
+    'builddir': "openmp-offload-cuda-runtime",
+    'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
+                        clean=True,
+                        enable_runtimes=['openmp'],
+                        extraCmakeArgs=[
+                                "-DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda",
+                                "-DLIBOMPTARGET_BUILD_NVPTX_BCLIB=ON",
+                                "-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
+                                "-DCLANG_ENABLE_ARCMT=OFF",
+                                "-DCLANG_ENABLE_OBJC_REWRITER=OFF",
+                                "-DLLVM_TARGETS_TO_BUILD=X86;NVPTX",
+                                "-DLLVM_ENABLE_LLD=ON",
+                                '-DLLVM_PARALLEL_LINK_JOBS=2',
+                                "-DCMAKE_C_COMPILER_LAUNCHER=ccache",
+                                "-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
+                            ],
+                        install=True,
+                        testsuite=True,
+                        testsuite_sollvevv=True,
+                        extraTestsuiteCmakeArgs=[
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
+                        ],
+                    )},
 
-    # ARMv8 LibC++ and LibC++abi tests (require Clang+RT)
-    {'name' : 'libcxx-libcxxabi-libunwind-armv8-linux',
-    'tags'  : ["libcxx"],
-    'workernames' : ['linaro-armv8-libcxx'],
-    'builddir': 'libcxx-libcxxabi-libunwind-armv8-linux',
-    'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
-                    cmake_extra_opts={
-                        'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
-                        'CMAKE_C_FLAGS': '-mcpu=cortex-a57 -marm',
-                        'CMAKE_CXX_FLAGS': '-mcpu=cortex-a57 -marm'})},
+# OpenMP AMDGPU Builders
+    {'name' : "openmp-offload-amdgpu-project",
+    'tags'  : ["openmp"],
+    'workernames' : ["omp-vega20-0"],
+    'builddir': "openmp-offload-amdgpu-project",
+    'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
+                        clean=False,
+                        enable_runtimes=[],
+                        depends_on_projects=['llvm','clang','lld','openmp'],
+                        extraCmakeArgs=[
+                            "-DCMAKE_BUILD_TYPE=Release",
+                            "-DCLANG_DEFAULT_LINKER=lld",
+                            "-DLLVM_TARGETS_TO_BUILD=X86;AMDGPU",
+                            "-DLLVM_ENABLE_ASSERTIONS=ON",
+                            "-DLLVM_ENABLE_RUNTIMES=openmp",
+                            ],
+                        install=True,
+                        testsuite=False,
+                        testsuite_sollvevv=False,
+                        extraTestsuiteCmakeArgs=[
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=amdgcn-amd-amdhsa;-Xopenmp-target=amdgcn-amd-amdhsa",
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDLAGS=-fopenmp-targets=amdgcn-amd-amdhsa;-Xopenmp-target=amdgcn-amd-amdhsa",
+                        ],
+                    )},
 
-    # ARMv7 LibC++ and LibC++abi tests w/o EH (require Clang+RT)
-    {'name' : 'libcxx-libcxxabi-libunwind-armv7-linux-noexceptions',
-    'tags'  : ["libcxx"],
-    'workernames' : ['linaro-tk1-02'],
-    'builddir': 'libcxx-libcxxabi-libunwind-armv7-linux-noexceptions',
-    'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
-                    cmake_extra_opts={
-                        'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
-                        'LIBCXX_ENABLE_EXCEPTIONS': 'OFF',
-                        'LIBCXXABI_ENABLE_EXCEPTIONS': 'OFF',
-                        'CMAKE_C_FLAGS': '-mcpu=cortex-a15 -mthumb',
-                        'CMAKE_CXX_FLAGS': '-mcpu=cortex-a15 -mthumb',
-                        'LLVM_PARALLEL_LINK_JOBS': '2'})},
+    {'name' : "openmp-offload-amdgpu-runtime",
+    'tags'  : ["openmp"],
+    'workernames' : ["omp-vega20-0"],
+    'builddir': "openmp-offload-amdgpu-runtime",
+    'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
+                        clean=True,
+                        enable_runtimes=['openmp'],
+                        depends_on_projects=['llvm','clang','lld','openmp'],
+                        extraCmakeArgs=[
+                            "-DCMAKE_BUILD_TYPE=Release",
+                            "-DCLANG_DEFAULT_LINKER=lld",
+                            "-DLLVM_TARGETS_TO_BUILD=X86;AMDGPU",
+                            "-DLLVM_ENABLE_ASSERTIONS=ON",
+                            "-DLLVM_ENABLE_RUNTIMES=openmp",
+                            "-DCMAKE_C_COMPILER_LAUNCHER=ccache",
+                            "-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
+                            ],
+                        install=True,
+                        testsuite=False,
+                        testsuite_sollvevv=False,
+                        extraTestsuiteCmakeArgs=[
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=amdgcn-amd-amdhsa;-Xopenmp-target=amdgcn-amd-amdhsa",
+                            "-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDLAGS=-fopenmp-targets=amdgcn-amd-amdhsa;-Xopenmp-target=amdgcn-amd-amdhsa",
+                        ],
+                    )},
 
-    # ARMv8 LibC++ and LibC++abi tests w/o EH (require Clang+RT)
-    {'name' : 'libcxx-libcxxabi-libunwind-armv8-linux-noexceptions',
-    'tags'  : ["libcxx"],
-    'workernames' : ['linaro-armv8-libcxx'],
-    'builddir': 'libcxx-libcxxabi-libunwind-armv8-linux-noexceptions',
-    'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
-                    cmake_extra_opts={
-                        'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
-                        'LIBCXX_ENABLE_EXCEPTIONS': 'OFF',
-                        'LIBCXXABI_ENABLE_EXCEPTIONS': 'OFF',
-                        'CMAKE_C_FLAGS': '-mcpu=cortex-a57 -mthumb',
-                        'CMAKE_CXX_FLAGS': '-mcpu=cortex-a57 -mthumb'})},
 
-    # AArch64 LibC++ and LibC++abi tests (require Clang+RT)
-    {'name' : 'libcxx-libcxxabi-libunwind-aarch64-linux',
-    'tags'  : ["libcxx"],
-    'workernames' : ['linaro-aarch64-libcxx'],
-    'builddir': 'libcxx-libcxxabi-libunwind-aarch64-linux',
-    'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
-                    cmake_extra_opts={
-                        'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
-                        'CMAKE_C_FLAGS': '-mcpu=cortex-a57',
-                        'CMAKE_CXX_FLAGS': '-mcpu=cortex-a57'})},
+# Libc++ builders.
 
-    {'name' : 'libcxx-libcxxabi-libunwind-aarch64-linux-noexceptions',
+    {'name' : 'libcxx-libcxxabi-libunwind-ppc-aix',
     'tags'  : ["libcxx"],
-    'workernames' : ['linaro-aarch64-libcxx'],
-    'builddir': 'libcxx-libcxxabi-libunwind-aarch64-linux-noexceptions',
+    'workernames' : ['aix-ppc-libcxx'],
+    'builddir': 'libcxx-libcxxabi-libunwind-ppc-aix',
     'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
+                    env={'OBJECT_MODE': '32'},
                     cmake_extra_opts={
-                        'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
-                        'LIBCXX_ENABLE_EXCEPTIONS': 'OFF',
-                        'LIBCXXABI_ENABLE_EXCEPTIONS': 'OFF',
-                        'CMAKE_C_FLAGS': '-mcpu=cortex-a57',
-                        'CMAKE_CXX_FLAGS': '-mcpu=cortex-a57'})},
+                        'CMAKE_BUILD_TYPE': 'Release',
+                        'CMAKE_C_COMPILER': 'gcc',
+                        'CMAKE_CXX_COMPILER': 'g++',
+                        'CMAKE_AR': '/usr/bin/ar',
+                        'LIBCXX_CXX_ABI': 'libstdc++',
+                        'LIBCXX_CXX_ABI_INCLUDE_PATHS': '/opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/8/include/c++;/opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/8/include/c++/powerpc-ibm-aix7.2.0.0',
+                        'CMAKE_THREAD_LIBS_INIT': '-lpthread',
+                        'LIBCXX_ENABLE_STATIC:BOOL': 'OFF',
+                        'LIBCXX_ENABLE_ABI_LINKER_SCRIPT': 'OFF'},
+                    depends_on_projects=['libcxx'])},
 
     {'name': "fuchsia-x86_64-linux",
     'tags'  : ["toolchain"],
     'workernames' :["fuchsia-debian-64-us-central1-a-1", "fuchsia-debian-64-us-central1-b-1"],
     'builddir': "fuchsia-x86_64-linux",
     'factory': FuchsiaBuilder.getFuchsiaToolchainBuildFactory()},
 
 # libc Builders.
 
+    {'name' : 'libc-x86_64-windows-dbg',
+    'tags'  : ["libc"],
+    'workernames' : ['libc-x86_64-windows'],
+    'builddir': 'libc-x86_64-windows',
+    'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
+                    script="libc-windows.py",
+                    depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
+                    extra_args=['--debug'])},
+
+    {'name' : 'libc-aarch64-ubuntu-dbg',
+    'tags'  : ["libc"],
+    'workernames' : ['libc-aarch64-ubuntu'],
+    'builddir': 'libc-aarch64-ubuntu',
+    'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
+                    script="libc-linux.py",
+                    depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
+                    extra_args=['--debug'])},
+
     {'name' : 'libc-x86_64-debian',
     'tags'  : ["libc"],
     'workernames' : ['libc-x86_64-debian'],
     'builddir': 'libc-x86_64-debian',
     'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
                     script="libc-linux.py",
                     depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'])},
 
# libc Builders.
     'tags'  : ["libc"],
     'workernames' : ["libc-x86_64-debian"],
     'builddir': "libc-x86_64-debian-dbg-asan",
     'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
                     script="libc-linux.py",
                     depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
                     extra_args=['--debug', '--asan'])},
 
-# Flang builders.
+    {'name' : "libc-x86_64-debian-fullbuild-dbg",
+    'tags'  : ["libc"],
+    'workernames' : ["libc-x86_64-debian-fullbuild"],
+    'builddir': "libc-x86_64-debian-fullbuild-dbg",
+    'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
+                    script="libc-linux.py",
+                    depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
+                    extra_args=['--debug'])},
 
-    {'name' : "flang-aarch64-ubuntu",
-    'tags'  : ["flang"],
-    'workernames' : ["flang-aarch64-ubuntu-build"],
-    'builddir': "flang-aarch64-ubuntu-build",
-    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
-                    clean=True,
-                    depends_on_projects=['llvm','mlir','clang','flang'],
-                    extra_configure_args=[
-                        "-DLLVM_TARGETS_TO_BUILD=AArch64",
-                        "-DCMAKE_C_COMPILER=/usr/bin/gcc-9",
-                        "-DCMAKE_CXX_COMPILER=/usr/bin/g++-9",
-                        "-DLLVM_INSTALL_UTILS=ON",
-                        "-DFLANG_BUILD_NEW_DRIVER=ON",
-                        "-DCMAKE_CXX_STANDARD=17",
-                    ])},
+    {'name' : "libc-x86_64-debian-fullbuild-dbg-asan",
+    'tags'  : ["libc"],
+    'workernames' : ["libc-x86_64-debian-fullbuild"],
+    'builddir': "libc-x86_64-debian-fullbuild-dbg-asan",
+    'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
+                    script="libc-linux.py",
+                    depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
+                    extra_args=['--debug', '--asan'])},
 
-    {'name' : "flang-aarch64-ubuntu-dylib",
+# Flang builders.
+
+    {'name' : "flang-aarch64-dylib",
     'tags'  : ["flang"],
-    'workernames' : ["linaro-aarch64-flang-dylib"],
-    'builddir': "flang-aarch64-ubuntu-dylib",
+    'workernames' : ["linaro-flang-aarch64-dylib"],
+    'builddir': "flang-aarch64-dylib",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     clean=True,
+                    checks=['check-flang'],
                     depends_on_projects=['llvm','mlir','clang','flang'],
                     extra_configure_args=[
                         "-DLLVM_TARGETS_TO_BUILD=AArch64",
-                        "-DFLANG_BUILD_NEW_DRIVER=ON",
                         "-DLLVM_BUILD_LLVM_DYLIB=ON",
                         "-DLLVM_LINK_LLVM_DYLIB=ON",
                         "-DCMAKE_CXX_STANDARD=17",
                     ])},
 
-    {'name' : "flang-aarch64-ubuntu-sharedlibs",
+    {'name' : "flang-aarch64-sharedlibs",
     'tags'  : ["flang"],
-    'workernames' : ["linaro-aarch64-flang-sharedlibs"],
-    'builddir': "flang-aarch64-ubuntu-sharedlibs",
+    'workernames' : ["linaro-flang-aarch64-sharedlibs"],
+    'builddir': "flang-aarch64-sharedlibs",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     clean=True,
+                    checks=['check-flang'],
                     depends_on_projects=['llvm','mlir','clang','flang'],
                     extra_configure_args=[
                         "-DLLVM_TARGETS_TO_BUILD=AArch64",
-                        "-DFLANG_BUILD_NEW_DRIVER=ON",
                         "-DBUILD_SHARED_LIBS=ON",
                         "-DLLVM_BUILD_EXAMPLES=ON",
+                        "-DFLANG_BUILD_EXAMPLES=ON",
                         "-DCMAKE_CXX_STANDARD=17",
                     ])},
 
-    {'name' : "flang-aarch64-ubuntu-out-of-tree",
+    {'name' : "flang-aarch64-out-of-tree",
     'tags'  : ["flang"],
-    'workernames' : ["linaro-aarch64-flang-oot"],
+    'workernames' : ["linaro-flang-aarch64-out-of-tree"],
     'builddir': "flang-aarch64-out-of-tree",
     'factory' : FlangBuilder.getFlangOutOfTreeBuildFactory(
+                    checks=['check-flang'],
                     llvm_extra_configure_args=[
                         "-DLLVM_TARGETS_TO_BUILD=AArch64",
                         "-DCMAKE_CXX_STANDARD=17",
                         "-DLLVM_ENABLE_WERROR=OFF",
                         "-DLLVM_ENABLE_ASSERTIONS=ON",
                         "-DCMAKE_BUILD_TYPE=Release",
                     ],
                     flang_extra_configure_args=[
                         "-DFLANG_ENABLE_WERROR=ON",
                         "-DCMAKE_BUILD_TYPE=Release",
                     ])},
 
+    {'name' : "flang-aarch64-debug",
+    'tags'  : ["flang"],
+    'workernames' : ["linaro-flang-aarch64-debug"],
+    'builddir': "flang-aarch64-debug",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    clean=True,
+                    checks=['check-flang'],
+                    depends_on_projects=['llvm','mlir','clang','flang'],
+                    extra_configure_args=[
+                        "-DLLVM_TARGETS_TO_BUILD=AArch64",
+                        "-DCMAKE_BUILD_TYPE=Debug",
+                        "-DCMAKE_CXX_STANDARD=17",
+                        "-DLLVM_USE_LINKER=lld",
+                    ])},
+
+    {'name' : "flang-aarch64-latest-clang",
+    'tags'  : ['flang'],
+    'workernames' : ["linaro-flang-aarch64-latest-clang"],
+    'builddir': "flang-aarch64-latest-clang",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    clean=True,
+                    checks=['check-flang'],
+                    depends_on_projects=['llvm','mlir','clang','flang'],
+                    extra_configure_args=[
+                        "-DLLVM_TARGETS_TO_BUILD=AArch64",
+                        "-DLLVM_INSTALL_UTILS=ON",
+                        "-DCMAKE_CXX_STANDARD=17",
+                        "-DLLVM_ENABLE_WERROR=OFF",
+                        "-DFLANG_ENABLE_WERROR=ON",
+                        "-DBUILD_SHARED_LIBS=ON",
+                        "-DLLVM_ENABLE_ASSERTIONS=ON",
+                        "-DLLVM_ENABLE_LIBCXX=On",
+                        "-DCMAKE_BUILD_TYPE=Release",
+                        ])},
+
+    {'name' : "flang-aarch64-release",
+    'tags'  : ["flang"],
+    'workernames' : ["linaro-flang-aarch64-release"],
+    'builddir': "flang-aarch64-release",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    clean=True,
+                    checks=['check-flang'],
+                    depends_on_projects=['llvm','mlir','clang','flang'],
+                    extra_configure_args=[
+                        "-DLLVM_TARGETS_TO_BUILD=AArch64",
+                        "-DCMAKE_BUILD_TYPE=Release",
+                        "-DCMAKE_CXX_STANDARD=17",
+                    ])},
+
+    {'name' : "flang-aarch64-rel-assert",
+    'tags'  : ["flang"],
+    'workernames' : ["linaro-flang-aarch64-rel-assert"],
+    'builddir': "flang-aarch64-rel-assert",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    clean=True,
+                    checks=['check-flang'],
+                    depends_on_projects=['llvm','mlir','clang','flang'],
+                    extra_configure_args=[
+                        "-DLLVM_TARGETS_TO_BUILD=AArch64",
+                        "-DLLVM_ENABLE_ASSERTIONS=ON",
+                        "-DFLANG_BUILD_EXAMPLES=ON",
+                        "-DCMAKE_BUILD_TYPE=Release",
+                        "-DCMAKE_CXX_STANDARD=17",
+                    ])},
+
+    {'name' : "flang-aarch64-latest-gcc",
+    'tags'  : ['flang'],
+    'workernames' : ["linaro-flang-aarch64-latest-gcc"],
+    'builddir': "flang-aarch64-latest-gcc",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    clean=True,
+                    checks=['check-flang'],
+                    depends_on_projects=['llvm','mlir','clang','flang'],
+                    extra_configure_args=[
+                        "-DLLVM_TARGETS_TO_BUILD=AArch64",
+                        "-DLLVM_INSTALL_UTILS=ON",
+                        "-DCMAKE_CXX_STANDARD=17",
+                        "-DLLVM_ENABLE_WERROR=OFF",
+                        "-DFLANG_ENABLE_WERROR=ON",
+                        "-DBUILD_SHARED_LIBS=ON",
+                        "-DLLVM_ENABLE_ASSERTIONS=ON",
+                        "-DCMAKE_BUILD_TYPE=Release",
+                    ])},
+
     {'name' : "flang-x86_64-linux",
     'tags'  : ["flang"],
     'workernames' : ["nersc-flang"],
     'builddir': "flang-x86_64-linux",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     depends_on_projects=['llvm','mlir','clang','flang'],
                     extra_configure_args=[
                         "-DLLVM_TARGETS_TO_BUILD=X86",
                         "-DLLVM_INSTALL_UTILS=ON",
                         "-DCMAKE_CXX_STANDARD=17",
                     ])},
 
     {'name' : "flang-x86_64-knl-linux",
     'tags'  : ["flang"],
     'workernames' : ["alcf-theta-flang"],
     'builddir': "flang-x86_64-knl-linux",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     depends_on_projects=['llvm','mlir','clang','flang'],
                     extra_configure_args=[
                         "-DLLVM_TARGETS_TO_BUILD=X86",
                         "-DCMAKE_C_COMPILER=gcc",
                         "-DCMAKE_CXX_COMPILER=g++",
                         "-DLLVM_INSTALL_UTILS=ON",
-                        "-DFLANG_BUILD_NEW_DRIVER=ON",
                         "-DCMAKE_CXX_STANDARD=17",
                     ])},
 
     {'name' : 'ppc64le-flang-rhel-clang',
     'tags'  : ["flang", "ppc", "ppc64le"],
     'collapseRequests' : False,
     'workernames' : ['ppc64le-flang-mlir-rhel-test'],
     'builddir': 'ppc64le-flang-rhel-clang-build',
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     clean=True,
                     depends_on_projects=['llvm', 'mlir', 'clang', 'flang'],
                     checks=['check-flang'],
                     extra_configure_args=[
                         '-DLLVM_TARGETS_TO_BUILD=PowerPC',
                         '-DLLVM_INSTALL_UTILS=ON',
                         '-DCMAKE_CXX_STANDARD=17',
-                        '-DLLVM_ENABLE_PROJECTS=flang',
                         '-DLLVM_LIT_ARGS=-vj 256'
                     ],
                     env={
                         'CC': 'clang',
                         'CXX': 'clang++',
                         'LD': 'lld'
                     })},
 
-# Builders responsible building Sphinix documentation.
+    {'name' : "flang-x86_64-windows",
+    'tags'  : ["flang"],
+    'workernames' : ["minipc-ryzen-win"],
+    'builddir': "flang-x86_64-windows",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    depends_on_projects=['llvm','mlir','clang','flang'],
+                    checks=['check-flang'],
+                    install_dir="flang.install",
+                    extra_configure_args=[
+                        "-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
+                        "-DCLANG_ENABLE_ARCMT=OFF",
+                        "-DCLANG_ENABLE_OBJC_REWRITER=OFF",
+                        "-DLLVM_TARGETS_TO_BUILD=X86",
+                        "-DLLVM_INSTALL_UTILS=ON",
+                        "-DCMAKE_C_COMPILER=cl",
+                        "-DCMAKE_CXX_COMPILER=cl",
+                        "-DCMAKE_CXX_STANDARD=17",
+                        '-DLLVM_PARALLEL_COMPILE_JOBS=4',
+                    ])},
+
+# Builders responsible building Sphinx documentation.
 
     {'name' : "llvm-sphinx-docs",
     'tags'  : ["llvm", "doc"],
     'workernames' : ["gribozavr3"],
     'builddir': "llvm-sphinx-docs",
     'factory': SphinxDocsBuilder.getSphinxDocsBuildFactory(llvm_html=True, llvm_man=True)},
 
     {'name' : "clang-sphinx-docs",
# Builders responsible building Sphinx documentation.
     'factory': SphinxDocsBuilder.getSphinxDocsBuildFactory(clang_tools_html=True)},
 
     {'name' : "lld-sphinx-docs",
     'tags'  : ["lld", "doc"],
     'workernames' : ["gribozavr3"],
     'builddir': "lld-sphinx-docs",
     'factory' : SphinxDocsBuilder.getSphinxDocsBuildFactory(lld_html=True)},
 
-    {'name' : "lldb-sphinx-docs",
-    'tags'  : ["lldb", "doc"],
-    'workernames' : ["gribozavr3"],
-    'builddir': "lldb-sphinx-docs",
-    'factory' : SphinxDocsBuilder.getSphinxDocsBuildFactory(lldb_html=True)},
-
     {'name':"libunwind-sphinx-docs",
     'tags'  : ["libunwind", "doc"],
     'workernames':["gribozavr3"],
     'builddir':"libunwind-sphinx-docs",
     'factory': SphinxDocsBuilder.getSphinxDocsBuildFactory(libunwind_html=True)},
 
     # Sphinx doc Publisher
     {'name' : "publish-sphinx-docs",
     'tags'  : ["doc"],
     'workernames' : ["as-worker-4"],
     'builddir': "publish-sphinx-docs",
     'factory' : SphinxDocsBuilder.getLLVMDocsBuildFactory(clean=True)},
 
+    {'name' : "publish-lnt-sphinx-docs",
+    'tags'  : ["doc"],
+    'workernames' : ["as-worker-4"],
+    'builddir': "publish-lnt-sphinx-docs",
+    'factory' : HtmlDocsBuilder.getHtmlDocsBuildFactory()},
+
+    {'name' : "publish-doxygen-docs",
+    'tags'  : ["doc"],
+    'workernames' : ["as-worker-4"],
+    'builddir': "publish-doxygen-docs",
+    'factory' : DoxygenDocsBuilder.getLLVMDocsBuildFactory()},
+
 # CUDA builders.
 
     {'name' : "clang-cuda-k80",
     'tags'  : ["clang"],
     'workernames' : ["cuda-k80-0"],
     'builddir': "clang-cuda-k80",
     'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
                     script="/buildbot/cuda-build",
# CUDA builders.
     'workernames' : ["cuda-t4-0"],
     'builddir': "clang-cuda-t4",
     'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
                     script="/buildbot/cuda-build",
                     checkout_llvm_sources=False)},
 
     {'name' : "clang-ve-ninja",
     'tags'  : ["clang"],
-    'workernames':["nec-arrproto41"],
+    'workernames':["hpce-aurora2"],
     'builddir':"clang-ve-ninja",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     clean=True,
                     depends_on_projects=['llvm','clang','openmp'],
                     extra_configure_args=[
                         "-DLLVM_TARGETS_TO_BUILD=X86",
                         "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=VE",
+                        "-DLLVM_ENABLE_RUNTIMES=compiler-rt",
+                        "-DLLVM_BUILD_LLVM_DYLIB=On",
+                        "-DBUILD_SHARED_LIBS=Off",
+                        "-DLLVM_LINK_LLVM_DYLIB=On",
+                        "-DCLANG_LINK_CLANG_DYLIB=On",
                     ])},
 
+# HIP builders.
+
+    {'name' : "clang-hip-vega20",
+    'tags'  : ["clang"],
+    'workernames' : ["hip-vega20-0"],
+    'builddir': "clang-hip-vega20",
+    'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
+                    script="hip-build.sh",
+                    checkout_llvm_sources=False,
+                    script_interpreter=None)},
+
 # Latest stable fedora running on Red Hat internal OpenShift cluster (PSI).
 
     {'name' : 'x86_64-fedora-clang',
     'tags'  : ['mlir'],
     'collapseRequests': False,
     'workernames': ['fedora-llvm-x86_64'],
     'builddir': 'x86_64-fedora-clang',
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                         '-DCMAKE_EXPORT_COMPILE_COMMANDS=1',
                         '-DLLVM_BUILD_LLVM_DYLIB=On',
                         '-DLLVM_LINK_LLVM_DYLIB=On',
                         '-DCLANG_LINK_CLANG_DYLIB=On',
                         '-DBUILD_SHARED_LIBS=Off',
                         '-DLLVM_ENABLE_LLD=ON',
                     ])},
 
+    # Build the LLVM dylib .so with all backends and link tools to it
+    {'name' : 'llvm-x86_64-debian-dylib',
+    'tags'  : ['llvm'],
+    'collapseRequests': False,
+    'workernames': ['gribozavr4'],
+    'builddir': 'llvm-x86_64-debian-dylib',
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    clean=True,
+                    depends_on_projects=['llvm'],
+                    checks=['check-all'],
+                    extra_configure_args=[
+                        '-DCMAKE_BUILD_TYPE=Release',
+                        '-DLLVM_ENABLE_ASSERTIONS=On',
+                        '-DLLVM_BUILD_EXAMPLES=Off',
+                        "-DLLVM_LIT_ARGS=-v --xunit-xml-output test-results.xml",
+                        '-DLLVM_TARGETS_TO_BUILD=All',
+                        '-DCMAKE_EXPORT_COMPILE_COMMANDS=1',
+                        '-DLLVM_BUILD_LLVM_DYLIB=On',
+                        '-DLLVM_LINK_LLVM_DYLIB=On',
+                        '-DBUILD_SHARED_LIBS=Off',
+                        '-DLLVM_ENABLE_LLD=Off',
+                    ],
+                    env={
+                        'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
+                        'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes'
+                    })},
+
     {'name' : "clang-solaris11-amd64",
     'tags' : ["clang"],
     'workernames' : ["solaris11-amd64"],
     'builddir': "clang-solaris11-amd64",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     jobs=8,
                     clean=False,
                     checkout_lld=False,
'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON',
                                         '-DLLVM_TARGETS_TO_BUILD=X86',
                                         '-DCMAKE_C_COMPILER_LAUNCHER=sccache',
                                         '-DCMAKE_CXX_COMPILER_LAUNCHER=sccache',
                                         '-DLLVM_ENABLE_ZLIB=OFF',
                                         '-DLLVM_LIT_TOOLS_DIR=C:\\Program Files\\GnuWin32\\usr\\bin',
                                         ])},
 
-# Flang builders.
-
-    {'name' : "flang-aarch64-ubuntu-clang",
-    'tags'  : ['flang'],
-    'workernames' : ["flang-aarch64-ubuntu-clang-build"],
-    'builddir': "flang-aarch64-ubuntu-clang",
-    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
-                    clean=True,
-                    depends_on_projects=['llvm','mlir','clang','flang'],
-                    checks=['check-flang'],
-                    extra_configure_args=[
-                        "-DLLVM_TARGETS_TO_BUILD=AArch64",
-                        "-DCMAKE_C_COMPILER=/usr/bin/clang-10",
-                        "-DCMAKE_CXX_COMPILER=/usr/bin/clang++-10",
-                        "-DLLVM_INSTALL_UTILS=ON",
-                        "-DCMAKE_CXX_STANDARD=17",
-                        "-DLLVM_ENABLE_WERROR=OFF",
-                        "-DFLANG_ENABLE_WERROR=ON",
-                        "-DBUILD_SHARED_LIBS=ON",
-                        "-DLLVM_ENABLE_ASSERTIONS=ON",
-                        "-DLLVM_ENABLE_LIBCXX=On",
-                        "-DFLANG_BUILD_NEW_DRIVER=ON",
-                        "-DCMAKE_BUILD_TYPE=Release",
-                        ])},
-
-    {'name' : "flang-aarch64-ubuntu-gcc10",
-    'tags'  : ['flang'],
-    'workernames' : ["flang-aarch64-ubuntu-gcc10-build"],
-    'builddir': "flang-aarch64-ubuntu-gcc10",
-    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
-                    clean=True,
-                    depends_on_projects=['llvm','mlir','clang','flang'],
-                    extra_configure_args=[
-                        "-DLLVM_TARGETS_TO_BUILD=AArch64",
-                        "-DCMAKE_C_COMPILER=/usr/bin/gcc-10",
-                        "-DCMAKE_CXX_COMPILER=/usr/bin/g++-10",
-                        "-DLLVM_INSTALL_UTILS=ON",
-                        "-DCMAKE_CXX_STANDARD=17",
-                        "-DLLVM_ENABLE_WERROR=OFF",
-                        "-DFLANG_ENABLE_WERROR=ON",
-                        "-DBUILD_SHARED_LIBS=ON",
-                        "-DLLVM_ENABLE_ASSERTIONS=ON",
-                        "-DFLANG_BUILD_NEW_DRIVER=ON",
-                        "-DCMAKE_BUILD_TYPE=Release",
-                    ])},
-
 # LLDB builders.
 
     {'name' : "lldb-x86_64-fedora",
     'tags'  : ["lldb"],
     'workernames' : ["lldb-x86_64-fedora"],
     'builddir': "lldb-x86_64-fedora",
     'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
                     clean=True,
# Builders for ML-driven compiler optimizations.
     'builddir': "ml-opt-dev-x86-64-b1",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     clean=True,
                     depends_on_projects=['llvm'],
                     extra_configure_args=[
                         "-DCMAKE_BUILD_TYPE=Release",
                         "-DLLVM_CCACHE_BUILD=ON",
                         "-DLLVM_ENABLE_ASSERTIONS=ON",
-                        "-DTENSORFLOW_API_PATH=/tmp/tensorflow",
+                        "-DTENSORFLOW_C_LIB_PATH=/tmp/tensorflow",
                         "-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON"
                     ])},
 
     # Both tensorflow C library, and the pip package, are present.
     {'name' : "ml-opt-devrel-x86-64",
     'tags'  : ["ml_opt"],
     'collapseRequests': False,
     'workernames' : ["ml-opt-devrel-x86-64-b1"],
     'builddir': "ml-opt-devrel-x86-64-b1",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     clean=True,
                     depends_on_projects=['llvm'],
                     extra_configure_args= [
                         "-DCMAKE_BUILD_TYPE=Release",
                         "-DLLVM_CCACHE_BUILD=ON",
                         "-DLLVM_ENABLE_ASSERTIONS=ON",
-                        "-DTENSORFLOW_API_PATH=/tmp/tensorflow",
+                        "-DTENSORFLOW_C_LIB_PATH=/tmp/tensorflow",
                         "-DTENSORFLOW_AOT_PATH=/var/lib/buildbot/.local/lib/python3.7/site-packages/tensorflow",
                         "-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON"
                     ])},
 
     # Release mode build bot: the model is pre-built and linked in the
     # compiler. Only the tensorflow pip package is needed, and out of it,
     # only saved_model_cli (the model compiler) and the thin C++ wrappers
     # in xla_aot_runtime_src (and include files)
'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
              "-DLLVM_CCACHE_BUILD=ON",
              "-DLLVM_USE_SANITIZER=Thread",
              "-DCMAKE_BUILD_TYPE=Release",
              "-DCLANGD_ENABLE_REMOTE=ON",
              "-DLLVM_ENABLE_ASSERTIONS=ON",
              "-DGRPC_INSTALL_PATH=/usr/local/lib/grpc"
          ])},
 
+    # Target ARC from Synopsys
+    {'name': "arc-builder",
+     'tags': ["clang", "lld"],
+     'collapseRequests': False,
+     'workernames' : ["arc-worker"],
+     'builddir': "arc-folder",
+     'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+        depends_on_projects=["llvm", "clang", "lld"],
+        extra_configure_args=[
+             "-DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON",
+             "-DLLVM_ENABLE_ASSERTIONS:BOOL=ON",
+             "-DLLVM_TOOL_CLANG_TOOLS_EXTRA_BUILD=0",
+             "-DLLVM_ENABLE_LIBPFM=OFF",
+             "-DLLVM_TARGETS_TO_BUILD=X86",
+             "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=ARC",
+         ])},
+
 
 ]

buildbot/osuosl/master/config/release_builders.py

+from importlib import reload
+
+from buildbot.process.properties import WithProperties
+
+from zorg.buildbot.builders import ClangBuilder
+from zorg.buildbot.builders import FlangBuilder
+from zorg.buildbot.builders import PollyBuilder
+from zorg.buildbot.builders import LLDBBuilder
+from zorg.buildbot.builders import SanitizerBuilder
+from zorg.buildbot.builders import OpenMPBuilder
+from zorg.buildbot.builders import SphinxDocsBuilder
+from zorg.buildbot.builders import ABITestsuitBuilder
+from zorg.buildbot.builders import ClangLTOBuilder
+from zorg.buildbot.builders import UnifiedTreeBuilder
+from zorg.buildbot.builders import AOSPBuilder
+from zorg.buildbot.builders import AnnotatedBuilder
+from zorg.buildbot.builders import LLDPerformanceTestsuite
+from zorg.buildbot.builders import FuchsiaBuilder
+from zorg.buildbot.builders import XToolchainBuilder
+
+
+# Release builders.
+
+all = [
+
+# Clang builders.
+
+    {'name' : "llvm-clang-x86_64-win-release",
+    'tags'  : ["clang", "release"],
+    'workernames' : ["as-builder-3"],
+    'builddir': "llvm-clang-x86_64-win-rel",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
+                    vs="autodetect",
+                    depends_on_projects=['llvm', 'clang'],
+                    clean=True,
+                    checks=[
+                    "check-llvm-unit",
+                    "check-clang-unit"],
+                    extra_configure_args=[
+                        "-DLLVM_ENABLE_WERROR=OFF",
+                        "-DLLVM_TARGETS_TO_BUILD=ARM",
+                        "-DLLVM_DEFAULT_TARGET_TRIPLE=armv7-unknown-linux-eabihf",
+                        "-DLLVM_ENABLE_ASSERTIONS=OFF",
+                        "-DLLVM_OPTIMIZED_TABLEGEN=OFF",
+                        "-DLLVM_LIT_ARGS=-v --threads=32"])},
+
+# Expensive checks builders.
+
+
+    {'name' : "llvm-clang-x86_64-expensive-checks-ubuntu-release",
+    'tags'  : ["llvm", "expensive-checks", "release"],
+    'workernames' : ["as-builder-4"],
+    'builddir': "llvm-clang-x86_64-expensive-checks-ubuntu-rel",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    depends_on_projects=["llvm", "lld"],
+                    clean=True,
+                    extra_configure_args=[
+                        "-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
+                        "-DLLVM_ENABLE_WERROR=OFF",
+                        "-DCMAKE_BUILD_TYPE=Debug",
+                        "-DCMAKE_CXX_FLAGS=-U_GLIBCXX_DEBUG",
+                        "-DLLVM_LIT_ARGS=-vv -j32"])},
+
+    {'name' : "llvm-clang-x86_64-expensive-checks-win-release",
+    'tags'  : ["llvm", "expensive-checks", "release"],
+    'workernames' : ["as-worker-93"],
+    'builddir': "llvm-clang-x86_64-expensive-checks-win-rel",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
+                    vs="autodetect",
+                    depends_on_projects=["llvm", "lld"],
+                    clean=True,
+                    extra_configure_args=[
+                        "-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
+                        "-DLLVM_ENABLE_WERROR=OFF",
+                        "-DCMAKE_BUILD_TYPE=Debug"])},
+
+    {'name' : "llvm-clang-x86_64-expensive-checks-debian-release",
+    'tags'  : ["llvm", "expensive-checks", "release"],
+    'workernames' : ["gribozavr4"],
+    'builddir': "llvm-clang-x86_64-expensive-checks-debian-rel",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    depends_on_projects=["llvm", "lld"],
+                    clean=True,
+                    extra_configure_args=[
+                        "-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
+                        "-DLLVM_ENABLE_WERROR=OFF",
+                        "-DCMAKE_BUILD_TYPE=Release",
+                        "-DCMAKE_CXX_FLAGS=-U_GLIBCXX_DEBUG",
+                        "-DLLVM_LIT_ARGS=-v -vv -j96"],
+                    env={
+                        'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
+                        'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes',
+                    })},
+
+# Cross builders.
+
+    {'name' : "llvm-clang-win-x-armv7l-release",
+    'tags'  : ["clang", "llvm", "compiler-rt", "cross"," armv7l", "release"],
+    'workernames' : ["as-builder-1"],
+    'builddir': "x-armv7l-rel",
+    'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory(
+                    vs="autodetect",
+                    clean=True,
+                    checks=[
+                    "check-llvm",
+                    "check-clang",
+                    "check-lld",
+                    "check-compiler-rt"
+                    ],
+                    checks_on_target = [
+                        ("libunwind",
+                            ["python", "bin/llvm-lit.py",
+                            "-v", "-vv", "--threads=32",
+                            "runtimes/runtimes-bins/libunwind/test"]),
+                        ("libc++abi",
+                            ["python", "bin/llvm-lit.py",
+                            "-v", "-vv", "--threads=32",
+                            "runtimes/runtimes-bins/libcxxabi/test"]),
+                        ("libc++",
+                            ['python', 'bin/llvm-lit.py',
+                            '-v', '-vv', '--threads=32',
+                            'runtimes/runtimes-bins/libcxx/test',
+                            ])
+                    ],
+                    extra_configure_args=[
+                        "-DLLVM_TARGETS_TO_BUILD=ARM",
+                        "-DTARGET_TRIPLE=armv7-linux-gnueabihf",
+                        "-DDEFAULT_SYSROOT=C:/buildbot/.arm-ubuntu",
+                        "-DLLVM_LIT_ARGS=-v -vv --threads=32",
+                        WithProperties("%(remote_test_host:+-DREMOTE_TEST_HOST=)s%(remote_test_host:-)s"),
+                        WithProperties("%(remote_test_user:+-DREMOTE_TEST_USER=)s%(remote_test_user:-)s"),
+                    ],
+                    cmake_cache="../llvm-project/clang/cmake/caches/CrossWinToARMLinux.cmake")},
+
+    {'name' : "llvm-clang-win-x-aarch64-release",
+    'tags'  : ["clang", "llvm", "compiler-rt", "cross", "aarch64", "release"],
+    'workernames' : ["as-builder-2"],
+    'builddir': "x-aarch64-rel",
+    'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory(
+                    vs="autodetect",
+                    clean=True,
+                    checks=[
+                    "check-llvm",
+                    "check-clang",
+                    "check-lld",
+                    "check-compiler-rt"
+                    ],
+                    checks_on_target = [
+                        ("libunwind",
+                            ["python", "bin/llvm-lit.py",
+                            "-v", "-vv", "--threads=32",
+                            "runtimes/runtimes-bins/libunwind/test"]),
+                        ("libc++abi",
+                            ["python", "bin/llvm-lit.py",
+                            "-v", "-vv", "--threads=32",
+                            "runtimes/runtimes-bins/libcxxabi/test"]),
+                        ("libc++",
+                            ['python', 'bin/llvm-lit.py',
+                            '-v', '-vv', '--threads=32',
+                            'runtimes/runtimes-bins/libcxx/test',
+                            ])
+                    ],
+                    extra_configure_args=[
+                        "-DLLVM_TARGETS_TO_BUILD=AArch64",
+                        "-DTARGET_TRIPLE=aarch64-linux-gnu",
+                        "-DDEFAULT_SYSROOT=C:/buildbot/.aarch64-ubuntu",
+                        "-DLLVM_LIT_ARGS=-v -vv --threads=32",
+                        WithProperties("%(remote_test_host:+-DREMOTE_TEST_HOST=)s%(remote_test_host:-)s"),
+                        WithProperties("%(remote_test_user:+-DREMOTE_TEST_USER=)s%(remote_test_user:-)s"),
+                    ],
+                    cmake_cache="../llvm-project/clang/cmake/caches/CrossWinToARMLinux.cmake")},
+
+# LLD builders.
+
+    {'name' : "lld-x86_64-win-release",
+    'tags'  : ["lld", "release"],
+    'workernames' : ["as-worker-93"],
+    'builddir': "lld-x86_64-win-rel",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
+                    depends_on_projects=['llvm', 'lld'],
+                    vs="autodetect",
+                    extra_configure_args = [
+                        '-DLLVM_ENABLE_WERROR=OFF'])},
+
+    {'name' : "lld-x86_64-ubuntu-release",
+    'tags'  : ["lld", "release"],
+    'workernames' : ["as-builder-4"],
+    'builddir' : "lld-x86_64-ubuntu-rel",
+    'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                            clean=True,
+                            extra_configure_args=[
+                                '-DLLVM_ENABLE_WERROR=OFF'],
+                            depends_on_projects=['llvm', 'lld'])},
+
+# LTO and ThinLTO builders.
+
+    {'name' : "clang-with-thin-lto-ubuntu-release",
+    'tags'  : ["clang", "lld", "LTO", "release"],
+    'workernames' : ["as-worker-92"],
+    'builddir': "clang-with-thin-lto-ubuntu-rel",
+    'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(jobs=72, lto='thin')},
+
+    {'name' : "clang-with-lto-ubuntu-release",
+    'tags'  : ["clang", "lld", "LTO", "release"],
+    'workernames' : ["as-worker-91"],
+    'builddir': "clang-with-lto-ubuntu-rel",
+    'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(
+                    jobs=72,
+                    extra_configure_args_lto_stage=[
+                        '-DLLVM_PARALLEL_LINK_JOBS=14',
+                    ])},
+
+# OpenMP builders.
+
+    {'name' : "openmp-clang-x86_64-linux-debian-release",
+    'tags'  : ["openmp", "release"],
+    'workernames' : ["gribozavr4"],
+    'builddir': "openmp-clang-x86_64-linux-debian-rel",
+    'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
+                    env={
+                        'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
+                        'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes',
+                    })},
+
+]

buildbot/osuosl/master/config/schedulers.py

if cp:
         changed_projects = frozenset(cp.split(','))
         if changed_projects.intersection(projects_of_interest):
             return True
     return False
 
 
 # Since we have many parametric builders, we dynamically build the minimum set
 # of schedulers, which covers all actually used combinations of dependencies.
-def getSingleBranchSchedulers(
+def getMainBranchSchedulers(
     builders,
     explicitly_set_schedulers = None,
     **kwargs):
     """
     I'm taking over all of not yet assigned builders with the
     declared source code dependencies, and automatically generate
     a minimum set of SingleBranchSchedulers to handle all the declared
     source code dependency combinations.
     """
 
     builders_with_explicit_schedulers = set()
     if explicitly_set_schedulers:
         # TODO: Make a list of builder names with already set schedulers.
         # builders_with_explicit_schedulers.add(builder)
         pass
 
     # For the builders created with LLVMBuildFactory or similar,
     # we always use automatic schedulers,
     # unless schedulers already explicitly set.
     builders_with_automatic_schedulers = [
         builder for builder in builders
         if builder.name not in builders_with_explicit_schedulers
         if getattr(builder.factory, 'depends_on_projects', None)
+        if 'release' not in getattr(builder, 'tags', [])
     ]
 
-    filter_branch = 'master'
+    filter_branch = 'main'
     treeStableTimer = kwargs.get('treeStableTimer', None)
 
     automatic_schedulers = []
 
     # Do we have any to take care of?
     if builders_with_automatic_schedulers:
         # Let's reconsile first to get a unique set of dependencies.
         # We need a set of unique sets of dependent projects.
         set_of_dependencies = set([
             frozenset(getattr(b.factory, 'depends_on_projects'))
             for b in builders_with_automatic_schedulers
         ])
 
         for projects in set_of_dependencies:
-            sch_builders = [
+            main_builders = [
                 b.name
                 for b in builders_with_automatic_schedulers
                 if frozenset(getattr(b.factory, 'depends_on_projects')) == projects
             ]
 
             automatic_scheduler_name =  filter_branch + ":" + ",".join(sorted(projects))
 
             automatic_schedulers.append(
                 schedulers.SingleBranchScheduler(
                     name=automatic_scheduler_name,
-                    treeStableTimer=kwargs.get('treeStableTimer', None),
-                    reason="Merge to github %s branch" % filter_branch,
-                    builderNames=sch_builders,
+                    treeStableTimer=treeStableTimer,
+                    reason="Merge to github {} branch".format(filter_branch),
+                    builderNames=main_builders,
                     change_filter=util.ChangeFilter(
                         project_fn= \
                             lambda c, projects_of_interest=frozenset(projects):
                                 isProjectOfInterest(c, projects_of_interest),
                         branch=filter_branch)
                 )
             )
 
             log.msg(
-                "Generated SingleBranchScheduler: { name='%s'" % automatic_scheduler_name,
-                ", builderNames=", sch_builders,
-                ", change_filter=", projects, " (branch: %s)" % filter_branch,
-                ", treeStableTimer=%s" % treeStableTimer,
+                "Generated SingleBranchScheduler: {{ name='{}'".format(automatic_scheduler_name),
+                ", builderNames=", main_builders,
+                ", change_filter=", projects, " (branch: {})".format(filter_branch),
+                ", treeStableTimer={}".format(treeStableTimer),
+                "}")
+    return automatic_schedulers
+
+def getReleaseBranchSchedulers(
+    builders,
+    explicitly_set_schedulers = None,
+    **kwargs):
+    """
+    I'm taking over all of not yet assigned builders with the
+    declared source code dependencies and the 'release' tag,
+    and automatically generate a minimum set of SingleBranchSchedulers
+    to handle all the declared source code dependency combinations.
+    """
+
+    builders_with_explicit_schedulers = set()
+    if explicitly_set_schedulers:
+        # TODO: Make a list of builder names with already set schedulers.
+        # builders_with_explicit_schedulers.add(builder)
+        pass
+
+    # For the builders created with LLVMBuildFactory or similar,
+    # we always use automatic schedulers,
+    # unless schedulers already explicitly set.
+    builders_with_automatic_schedulers = [
+        builder for builder in builders
+        if builder.name not in builders_with_explicit_schedulers
+        if getattr(builder.factory, 'depends_on_projects', None)
+        if 'release' in getattr(builder, 'tags', [])
+    ]
+
+    treeStableTimer = kwargs.get('treeStableTimer', None)
+
+    automatic_schedulers = []
+
+    # Do we have any to take care of?
+    if builders_with_automatic_schedulers:
+        # Let's reconsile first to get a unique set of dependencies.
+        # We need a set of unique sets of dependent projects.
+        set_of_dependencies = set([
+            frozenset(getattr(b.factory, 'depends_on_projects'))
+            for b in builders_with_automatic_schedulers
+        ])
+
+        for projects in set_of_dependencies:
+            release_builders = [
+                b.name
+                for b in builders_with_automatic_schedulers
+                if frozenset(getattr(b.factory, 'depends_on_projects')) == projects
+            ]
+
+            automatic_scheduler_name =  "release:" + ",".join(sorted(projects))
+
+            automatic_schedulers.append(
+                schedulers.SingleBranchScheduler(
+                    name=automatic_scheduler_name,
+                    treeStableTimer=treeStableTimer,
+                    reason="Merge to github release branch",
+                    builderNames=release_builders,
+                    change_filter=util.ChangeFilter(
+                        project_fn= \
+                            lambda c, projects_of_interest=frozenset(projects):
+                                isProjectOfInterest(c, projects_of_interest),
+                        branch_fn= \
+                            lambda branch: branch.startswith('release/'))
+                )
+            )
+
+            log.msg(
+                "Generated release SingleBranchScheduler: {{ name='{}'".format(automatic_scheduler_name),
+                ", builderNames=", release_builders,
+                ", change_filter=", projects, " (branch: {release/*})",
+                ", treeStableTimer={}".format(treeStableTimer),
                 "}")
     return automatic_schedulers
 
+def getForceSchedulers(builders):
     # TODO: Move these settings to the configuration file.
     _repourl = "https://github.com/llvm/llvm-project"
-_branch = "master"
+    _branch = "main"
 
-def getForceSchedulers(builders):
     # Walk over all builders and collect their names.
     scheduler_builders = [
         builder.name for builder in builders
     ]
 
     # Create the force schedulers.
     return [ schedulers.ForceScheduler(
                 name            = "force-build-scheduler",
return [ schedulers.ForceScheduler(
                                 "Narrow down blamelist",
                             ],
                             default     = "Build a particular revision"
                 ),
                 builderNames    = scheduler_builders,
                 codebases       = [
                     util.CodebaseParameter(
                         codebase    = "",
-                        branch          = util.FixedParameter(
+                        branch      = util.StringParameter(
                             name        = "branch",
+                            label       = "branch:",
+                            size        = 64,
                             default     = _branch
                         ),
                         revision    = util.StringParameter(
                             name        = "revision",
                             label       = "revision:",
                             size        = 45,
                             default     = ''
                         ),
return [ schedulers.ForceScheduler(
                         default     = False
                     )
                 ]
             )
         ]
 
 # TODO: Abstract this kind of scheduler better.
 def getLntSchedulers():
-    project = "lnt"
-    lnt_builders = [
+    _project = "lnt"
+    _branch = 'master'
+    _repourl = "https://github.com/llvm/llvm-lnt"
+    _lnt_builders = [
         "publish-lnt-sphinx-docs",
     ]
     return [
         schedulers.SingleBranchScheduler(
             name="lnt-scheduler",
             treeStableTimer=kwargs.get('treeStableTimer', None),
-            reason="Merge to LNT github {} branch".format(filter_branch),
-            builderNames=lnt_builders,
+            reason="Merge to LNT github {} branch".format(_branch),
+            builderNames=_lnt_builders,
             change_filter=util.ChangeFilter(
-                project_fn=project,
-                branch=filter_branch)),
+                project_fn=_project,
+                branch=_branch)),
 
         schedulers.ForceScheduler(
             name            = "force-build-scheduler",
             label           = "Force Build",
             buttonName      = "Force Build",
             reason = util.ChoiceStringParameter(
                          name        = "reason",
                          label       = "reason:",
                          required    = True,
                          choices     = [
                             "Build a particular revision",
                             "Force clean build",
                             "Narrow down blamelist",
                          ],
                          default     = "Build a particular revision"
             ),
-            builderNames    = lnt_builders,
+            builderNames    = _lnt_builders,
             codebases       = [
                 util.CodebaseParameter(
                     codebase    = "",
                     branch          = util.FixedParameter(
                         name        = "branch",
                         default     = _branch
                     ),
                     revision    = util.StringParameter(
                         name        = "revision",
                         label       = "revision:",
                         size        = 45,
                         default     = ''
                     ),
                     repository  = util.FixedParameter(
                         name        = "repository",
                         default     = _repourl
                     ),
                     project     = util.FixedParameter(
                         name        = "project",
-                        default     = project
+                        default     = _project
                     )
                 )
             ],
             properties  = [
                 util.BooleanParameter(
                     name        = "clean",
                     label       = "Clean source code and build directory",
                     default     = False

buildbot/osuosl/master/config/status.py

 from buildbot.process.properties import Interpolate
 from buildbot.plugins import reporters
 
 import config
 from zorg.buildbot.util.InformativeMailNotifier import LLVMInformativeMailNotifier
 
 # Should be a single e-mail address
 status_email = str(config.options.get('Master Options', 'status_email')).split(',')
 
 all = [
 
-    # Note: reporters.GitHubStatusPush requires txrequests package to allow
-    # interaction with GitHub REST API.
+    # Report github status for all the release builders,
+    # i.e. those with the "release" tag.
     reporters.GitHubStatusPush(
         str(config.options.get('GitHub Status', 'token')),
         context = Interpolate("%(prop:buildername)s"),
         verbose = True, # TODO: Turn off the verbosity once this is working reliably.
         builders = [
                 "llvm-clang-x86_64-expensive-checks-ubuntu",
                 "llvm-clang-x86_64-win-fast",
                 "clang-x86_64-debian-fast",
                 "llvm-clang-x86_64-expensive-checks-debian",
-        ]),
+            ] + [
+                b.get('name') for b in config.release_builders.all
+                if 'release' in b.get('tags', [])
+            ]
+        ),
 
     reporters.IRC(
         useColors = False,
         host = str(config.options.get('IRC', 'host')),
         nick = str(config.options.get('IRC', 'nick')),
         channels = str(config.options.get('IRC', 'channels')).split(','),
         #authz=... # TODO: Consider allowing "harmful" operations to authorizes users.
         useRevisions = False, # FIXME: There is a bug in the buildbot
         showBlameList = True,
         notify_events = str(config.options.get('IRC', 'notify_events')).split(','),
         ),
 
     reporters.MailNotifier(
         mode = ('problem',),
         fromaddr = "[e-mail]", # TODO: Change this to [e-mail].
         extraRecipients = status_email,
         extraHeaders = {"Reply-To": status_email[0]}, # The first from the list.
         lookup = "lab.llvm.org",
         messageFormatter = LLVMInformativeMailNotifier,
         # TODO: For debug purposes only. Remove later.
         dumpMailsToLog = True,
         ),
 
-    # In addition to that the following notifiers are defined for special
-    # cases.
-    reporters.MailNotifier(
-        fromaddr = "[e-mail]",
-        sendToInterestedUsers = False,
-        extraRecipients = ["[e-mail]"],
-        subject = "Build %(builder)s Failure",
-        mode = "failing",
-        builders = ["clang-aarch64-linux-build-cache", "clang-armv7-linux-build-cache"]),
     reporters.MailNotifier(
         fromaddr = "[e-mail]",
         sendToInterestedUsers = False,
         extraRecipients = ["[e-mail]"],
         subject = "Build %(builder)s Failure",
         mode = "failing",
         builders = ["clang-x86_64-debian-fast"]),
     reporters.MailNotifier(
reporters.MailNotifier(
         sendToInterestedUsers = False,
         extraRecipients = ["[e-mail]"],
         subject = "Build %(builder)s Failure",
         mode = "failing",
         builders = ["llvm-riscv-linux"]),
     reporters.MailNotifier(
         fromaddr = "[e-mail]",
         sendToInterestedUsers = False,
-        extraRecipients = ["[e-mail]", "[e-mail]",
-                           "[e-mail]", "[e-mail]"],
+        extraRecipients = ["[e-mail]",
+                           "[e-mail]",
+                           "[e-mail]"],
         subject = "Build %(builder)s Failure",
         mode = "failing",
         builders = ["lldb-x64-windows-ninja"]),
     reporters.MailNotifier(
         fromaddr = "[e-mail]",
         sendToInterestedUsers = False,
         extraRecipients = ["[e-mail]", "[e-mail]"],
         subject = "Build %(builder)s Failure",
reporters.MailNotifier(
         builders = ["clang-ve-ninja"]),
     reporters.MailNotifier(
         fromaddr = "[e-mail]",
         sendToInterestedUsers = False,
         extraRecipients = ["[e-mail]", "[e-mail]"],
         subject = "Build %(builder)s Failure",
         mode = "failing",
         builders = ["libc-x86_64-debian", "libc-x86_64_debian-dbg",
-                    "libc-x86_64-debian-dbg-asan"]),
+                    "libc-x86_64-debian-dbg-asan", "libc-aarch64-ubuntu-dbg",
+                    "libc-x86_64-windows-dbg",
+                    "libc-x86_64-debian-fullbuild-dbg",
+                    "libc-x86_64-debian-fullbuild-dbg-asan"]),
     reporters.MailNotifier(
         fromaddr = "[e-mail]",
         sendToInterestedUsers = False,
         extraRecipients = ["[e-mail]"],
         subject = "Sphinx build %(builder)s Failure",
         mode = "failing",
         builders = ["publish-sphinx-docs"]),
     reporters.MailNotifier(
         fromaddr="[e-mail]",
         sendToInterestedUsers = False,
         extraRecipients=[
             "[e-mail]"],
         subject = "ML Compiler Opt Failure: %(builder)s",
         mode = "failing",
         builders = [
             "ml-opt-dev-x86-64", "ml-opt-rel-x86-64", "ml-opt-devrel-x86-64"]),
     reporters.MailNotifier(
-        fromaddr = "[e-mail]",
+        fromaddr="[e-mail]",
         sendToInterestedUsers = False,
-        extraRecipients = [
-            "[e-mail]", "[e-mail]"],
+        extraRecipients=[
+            "[e-mail]"],
+        subject = "ThinLTO WPD Failure: %(builder)s",
+        mode = "failing",
+        builders = ["thinlto-x86-64-bot1"]),
+    reporters.MailNotifier(
+        fromaddr = "[e-mail]",
+        sendToInterestedUsers = False,
+        extraRecipients = ["[e-mail]"],
         subject = "Build %(builder)s Failure",
         mode = "failing",
-        builders = [
-            "flang-aarch64-ubuntu", "flang-aarch64-ubuntu-clang",
-            "flang-aarch64-ubuntu-gcc10"]),
+        builders = ["llvm-clang-x86_64-sie-ubuntu-fast"]),
     reporters.MailNotifier(
         fromaddr="[e-mail]",
         sendToInterestedUsers = False,
         extraRecipients=[
-            "[e-mail]"],
-        subject = "ThinLTO WPD Failure: %(builder)s",
+            "[e-mail]"],
+        subject = "MLIR Build Failure: %(builder)s",
         mode = "failing",
-        builders = ["thinlto-x86-64-bot1"]),
-
+        builders = ["mlir-nvidia",
+                    "mlir-windows",
+                    "ppc64le-mlir-rhel-clang"]),
+    reporters.MailNotifier(
+        fromaddr="[e-mail]",
+        sendToInterestedUsers = False,
+        extraRecipients=["[e-mail]"],
+        subject = "Build Failure: %(builder)s",
+        mode = "failing",
+        builders = ["clang-hip-vega20"]),
+    reporters.MailNotifier(
+        fromaddr="[e-mail]",
+        sendToInterestedUsers = False,
+        extraRecipients=["[e-mail]", "[e-mail]"],
+        subject = "Build Failure: %(builder)s",
+        mode = "failing",
+        builders = ["arc-builder"]),
+    reporters.MailNotifier(
+        fromaddr="[e-mail]",
+        sendToInterestedUsers = False,
+        extraRecipients=["[e-mail]"],
+        subject = "Build Failure: %(builder)s",
+        mode = "failing",
+        builders = ["openmp-offload-amdgpu-project","openmp-offload-amdgpu-runtime"]),
 ]
 
+
+from twisted.python import log
+
 # Returns a list of Status Targets. The results of each build will be
 # pushed to these targets. buildbot.plugins reporters has a variety
 # to choose from, including email senders, and IRC bots.
 def getReporters():
     if config.options.getboolean('Master Options', 'is_production'):
+        log.msg(">>> getReporters: Production mode. All reporters are registered.")
         return all
     else:
         # Staging buildbot does not report issues.
+        log.msg(">>> getReporters: Staging buildbot does not report issues.")
         return []

buildbot/osuosl/master/config/workers.py

return [
         create_worker("linaro-tk1-02", properties={'jobs' : 4}, max_builds=1),
         create_worker("linaro-tk1-03", properties={'jobs' : 4}, max_builds=1),
         create_worker("linaro-tk1-04", properties={'jobs' : 4}, max_builds=1),
         create_worker("linaro-tk1-05", properties={'jobs' : 4}, max_builds=1),
         create_worker("linaro-tk1-06", properties={'jobs' : 4}, max_builds=1),
         create_worker("linaro-tk1-07", properties={'jobs' : 4}, max_builds=1),
         create_worker("linaro-tk1-08", properties={'jobs' : 4}, max_builds=1),
         create_worker("linaro-tk1-09", properties={'jobs' : 4}, max_builds=1),
-        create_worker("linaro-armv7-lnt", properties={'jobs' : 32}, max_builds=1),
-        create_worker("linaro-armv7-selfhost", properties={'jobs' : 32}, max_builds=1),
-        create_worker("linaro-armv7-quick", properties={'jobs' : 32}, max_builds=1),
-        create_worker("linaro-armv7-global-isel", properties={'jobs' : 32}, max_builds=1),
-        create_worker("linaro-armv8-lld", properties={'jobs' : 32}, max_builds=1),
-        # Libcxx testsuite has tests with timing assumptions.  Run single-threaded to make
-        # sure we have plenty CPU cycle to satisfy timing assumptions.
-        create_worker("linaro-armv8-libcxx", properties={'jobs' : 1}, max_builds=1),
-        create_worker("linaro-arm-lldb", properties={'jobs' : 16}, max_builds=1),
+        create_worker("linaro-clang-armv7-lnt", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-clang-armv7-2stage", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-clang-armv7-quick", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-clang-armv7-global-isel", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-clang-armv7-vfpv3-2stage", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-clang-armv8-lld-2stage", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-lldb-arm-ubuntu", properties={'jobs' : 16}, max_builds=1),
 
         # AArch64 Linaro workers
-        create_worker("linaro-aarch64-quick", properties={'jobs' : 32}, max_builds=1),
-        create_worker("linaro-aarch64-full", properties={'jobs' : 32}, max_builds=1),
-        create_worker("linaro-aarch64-global-isel", properties={'jobs' : 32}, max_builds=1),
-        create_worker("linaro-aarch64-lld", properties={'jobs' : 32}, max_builds=1),
-        create_worker("linaro-aarch64-flang-oot", properties={'jobs' : 32}, max_builds=1),
-        create_worker("linaro-aarch64-flang-dylib", properties={'jobs' : 32}, max_builds=1),
-        create_worker("linaro-aarch64-flang-sharedlibs", properties={'jobs' : 32}, max_builds=1),
-        # Libcxx testsuite has tests with timing assumptions.  Run single-threaded to make
-        # sure we have plenty CPU cycle to satisfy timing assumptions.
-        create_worker("linaro-aarch64-libcxx", properties={'jobs' : 1}, max_builds=1),
-        create_worker("linaro-aarch64-lldb", properties={'jobs': 16}, max_builds=1),
+        create_worker("linaro-clang-aarch64-quick", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-clang-aarch64-lld-2stage", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-clang-aarch64-global-isel", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-clang-aarch64-full-2stage", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-clang-aarch64-sve-vls", properties={'jobs' : 48}, max_builds=1),
+        create_worker("linaro-clang-aarch64-sve-vls-2stage", properties={'jobs' : 48}, max_builds=1),
+        create_worker("linaro-lldb-aarch64-ubuntu", properties={'jobs': 16}, max_builds=1),
+        create_worker("linaro-flang-aarch64-dylib", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-flang-aarch64-sharedlibs", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-flang-aarch64-out-of-tree", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-flang-aarch64-debug", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-flang-aarch64-latest-clang", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-flang-aarch64-release", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-flang-aarch64-rel-assert", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-flang-aarch64-latest-gcc", properties={'jobs' : 32}, max_builds=1),
 
         # AArch64
         create_worker("linaro-armv8-windows-msvc-01", properties={'jobs' : 8}, max_builds=1),
         create_worker("linaro-armv8-windows-msvc-02", properties={'jobs' : 8}, max_builds=1),
 
-        # ARMv7 build cache workers
-        create_worker("packet-linux-armv7-slave-1", properties={'jobs' : 64}, max_builds=1),
-
-        # AArch64 build cache worker
-        create_worker("packet-linux-aarch64-slave-1", properties={'jobs' : 64}, max_builds=1),
-
         # Windows Server 2016 Intel Xeon(R) Quad 2.30 GHz, 56GB of RAM
         create_worker("win-py3-buildbot", properties={'jobs' : 64}, max_builds=1),
 
         # Windows Server 2016 Intel(R) Xeon(R) CPU @ 2.60GHz, 16 Core(s), 128 GB of RAM
         create_worker("win-mlir-buildbot", properties={'jobs' : 64}, max_builds=1),
 
+        # Motorola 68k 32-bit big endian (m68k)
+        create_worker("debian-akiko-m68k", properties={'jobs': 1}, max_builds=1),
+        create_worker("suse-gary-m68k-cross", properties={'jobs': 4}, max_builds=1),
+
         # POWER7 PowerPC big endian (powerpc64)
         create_worker("ppc64be-clang-test", properties={'jobs': 16}, max_builds=1),
         create_worker("ppc64be-clang-lnt-test", properties={'jobs': 16, 'vcs_protocol': 'https'}, max_builds=1),
         create_worker("ppc64be-clang-multistage-test", properties={'jobs': 16}, max_builds=1),
         create_worker("ppc64be-sanitizer", properties={'jobs': 16}, max_builds=1),
 
         # POWER 8 PowerPC little endian (powerpc64le)
         create_worker("ppc64le-clang-test", properties={'jobs': 4}, max_builds=1),
         create_worker("ppc64le-clang-lnt-test", properties={'jobs': 8}, max_builds=1),
         create_worker("ppc64le-clang-multistage-test", properties={'jobs': 8}, max_builds=1),
         create_worker("ppc64le-sanitizer", properties={'jobs': 4}, max_builds=1),
         create_worker("ppc64le-lld-multistage-test", max_builds=1),
-        create_worker("ppc64le-clang-rhel-test", properties={'jobs': 4}, max_builds=1),
+        create_worker("ppc64le-clang-rhel-test", properties={'jobs': 192}, max_builds=1),
         create_worker("ppc64le-flang-mlir-rhel-test", max_builds=1),
 
         # SPARC 64-bit big endian (sparc64)
         create_worker("debian-stadler-sparc64", properties={'jobs': 4}, max_builds=1),
 
         # Ubuntu x86-64, Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
         create_worker("hexagon-build-02", properties={'jobs': 12, 'loadaverage': 32},
             max_builds=1),
return [
         # Debian x86-64
         create_worker("gribozavr4", properties={'jobs': 96}, max_builds=2),
         create_worker("gribozavr5", properties={'jobs': 96}, max_builds=2,
                      missing_timeout=300, # notify after 5 minutes of missing
                      notify_on_missing=[
                          "[e-mail]",
                          "[e-mail]"]),
 
-        # Debian 7.7 x86_64 GCE instance
-        create_worker("sanitizer-buildbot1", properties={'jobs': 64}, max_builds=3),
-        # Debian 7.7 x86_64 GCE instance
-        create_worker("sanitizer-buildbot2", properties={'jobs': 64}, max_builds=3),
-        # Debian 7.7 x86_64 GCE instance
+        # Debian x86_64 GCE instance
+        create_worker("sanitizer-buildbot1", properties={'jobs': 64}, max_builds=2),
+        # Debian x86_64 GCE instance
+        create_worker("sanitizer-buildbot2", properties={'jobs': 64}, max_builds=2),
+        # Debian x86_64 GCE instance
         create_worker("sanitizer-buildbot3", properties={'jobs': 64}, max_builds=2),
-        # Debian 7.7 x86_64 GCE instance
+        # Debian x86_64 GCE instance
         create_worker("sanitizer-buildbot4", properties={'jobs': 64}, max_builds=2),
-        # Ubuntu 14.04 x86_64 6-core z440 workstation
+        # Ubuntu x86_64 6-core z440 workstation
         create_worker("sanitizer-buildbot6", properties={'jobs': 6}, max_builds=1),
-        # Debian 7.7 x86_64 GCE instance
-        create_worker("sanitizer-buildbot7", properties={'jobs': 64}, max_builds=3),
-        # Debian 7.7 x86_64 GCE instance
-        create_worker("sanitizer-buildbot8", properties={'jobs': 64}, max_builds=3),
+        # Debian x86_64 GCE instance
+        create_worker("sanitizer-buildbot7", properties={'jobs': 64}, max_builds=2),
+        # Debian x86_64 GCE instance
+        create_worker("sanitizer-buildbot8", properties={'jobs': 64}, max_builds=2),
+
+        # POWER 8 PowerPC AIX 7.2
+        create_worker("aix-ppc64-ppc64le", properties={'jobs': 10}, max_builds=1),
+        create_worker("aix-ppc-libcxx", properties={'jobs': 4}, max_builds=1),
 
         # IBM z13 (s390x), Ubuntu 16.04.2
         create_worker("systemz-1", properties={'jobs': 4, 'vcs_protocol': 'https'}, max_builds=4),
 
         # Windows Server 2012 x86_64 16-core GCE instance
         create_worker("sanitizer-windows", properties={'jobs': 16}, max_builds=1),
         # Windows Server 2012 x86_64 32-core GCE instance
         create_worker("windows-gcebot2", properties={'jobs': 32}, max_builds=1),
 
         # Ubuntu 14.04 x86_64-scei-ps4, 2 x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
-        create_worker("as-worker-91"),
-        create_worker("as-worker-92"),
+        create_worker("as-worker-91", max_builds=1),
+        create_worker("as-worker-92", max_builds=1),
 
         # Windows 10 Pro x86_64-scei-ps4, 2 x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
         create_worker("as-worker-93", properties={'jobs': 36}, max_builds=1),
 
         # WIP migration of the CUDA buildbot to GCE.
         create_worker("cuda-k80-0", max_builds=1),
         create_worker("cuda-p4-0", max_builds=1),
         create_worker("cuda-t4-0", max_builds=1),
 
+        # HIP on Ubuntu 18.04.5,  Intel(R) Xeon(R) Gold 5218 @ 2.30GHz, Vega20 GPU
+        create_worker("hip-vega20-0", max_builds=1),
+
         # X86_64 AVX2, Ubuntu 16.04.2, Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
         create_worker("avx2-intel64", properties={'jobs': 80}, max_builds=1),
 
         # X86_64 with SDE, Ubuntu 16.04.2, Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
         create_worker("sde-avx512-intel64", properties={'jobs': 80}, max_builds=1),
 
         # Debian 9.0 x86_64 64-core GCE instances
         create_worker("fuchsia-debian-64-us-central1-a-1", properties={'jobs': 64}, max_builds=1),
         create_worker("fuchsia-debian-64-us-central1-b-1", properties={'jobs': 64}, max_builds=1),
 
         # Fedora latest stable x86_64, Intel i5-2500, 4 cores, 12GB RAM
         create_worker("lldb-x86_64-fedora", properties={'jobs': 4}, max_builds=1),
 
         # Fedora latest stable, arch=x86_64, running on RedHat internal OpenShift PSI cluster
         create_worker("fedora-llvm-x86_64", properties={'jobs': 64}, max_builds=1),
 
         # Debian x86_64 Buster Xeon(R) Gold 6154 CPU @ 3.00GHz, 192GB RAM
         create_worker("lldb-x86_64-debian", properties={'jobs': 72}, max_builds=1),
 
+        # Windows x86_64 32 CPUs, 125 GB RAM
+        create_worker("libc-x86_64-windows", properties={'jobs': 32}, max_builds=2),
+        
+        # Ubuntu aarch64 128 CPUs, 125 GB RAM
+        create_worker("libc-aarch64-ubuntu", properties={'jobs': 32}, max_builds=2),
+
         # Debian x86_64 Intel Broadwell 32 CPUs, 120 GB RAM
         create_worker("libc-x86_64-debian", properties={'jobs': 32}, max_builds=2),
 
+        # Debian x86_64 Intel Skylake 32 CPUs, 128 GB RAM
+        create_worker("libc-x86_64-debian-fullbuild", properties={'jobs': 32}, max_builds=2),
+
         # Windows Server on Xeon Gold 6130 (2x2.1GHz), 128Gb of RAM
         create_worker("as-builder-1", properties={
                         'remote_test_host': 'jetson6.lab.llvm.org',
                         'remote_test_user': 'ubuntu'
                      },
                      max_builds=1),
 
         # Windows Server on Xeon Gold 6130 (2x2.1GHz), 128Gb of RAM
return [
         create_worker("solaris11-sparcv9", properties={'jobs' : 8}, max_builds=1),
 
         # Windows 7 Intel(R) Core(TM) CPU i7-4790K (4.00GHz), 16GB of RAM
         create_worker("windows7-buildbot", properties={'jobs': 2}, max_builds=1),
 
         # Windows 10, Visual Studio 2019, Google Cloud 16 cores
         create_worker("windows10-vs2019", properties={'jobs': 32}, max_builds=1),
 
-        # CentOS 7.5.1804 on Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz, 96GB RAM
-        create_worker("nec-arrproto41", properties={'jobs': 12}, max_builds=1),
+        # CentOS 7.8.1127 on Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz, 96GB RAM
+        create_worker("hpce-aurora2", properties={'jobs': 8}, max_builds=1),
 
         # Ubuntu 18.04 amd64 on Google Cloud, 16 core, Nvidia Tesla T4
         create_worker("mlir-nvidia", properties={'jobs': 16}, max_builds=1),
 
         # Ubuntu 18.04 on Google Cloud, for machine configuration check
         # buildbot/google/terraform/main.tf
         create_worker("clangd-ubuntu-clang", max_builds=1),
 
         # Ubuntu 18.04.LTS x86_64, GCE instance
         create_worker("polly-x86_64-gce1", properties={'jobs': 2}, max_builds=1),
         create_worker("polly-x86_64-gce2", properties={'jobs': 2}, max_builds=1),
 
         # Ubuntu 18.04.LTS x86_64, Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 32 GiB RAM
         create_worker("polly-x86_64-fdcserver", properties={'jobs': 8, 'loadaverage': 8}, max_builds=1),
 
-        create_worker("flang-aarch64-ubuntu-build"),
-        create_worker("flang-aarch64-ubuntu-clang-build", properties={'jobs': 14}),
-        create_worker("flang-aarch64-ubuntu-gcc10-build", properties={'jobs': 14}),
+        # Windows 10, AMD Ryzen 5 PRO 4650G, 16 GiB RAM
+        create_worker("minipc-ryzen-win", properties={'jobs': 12}, max_builds=1),
+
+        # Ubuntu 20.04.LTS x86_64, Intel(R) Core(TM) i5-9400F CPU @ 2.90Ghz, 16 GiB RAM, NVIDIA GeForce GTX 1050 Ti (Pascal, sm_61, 4GiB)
+        create_worker("minipc-1050ti-linux", properties={'jobs': 6}, max_builds=1),
+
         create_worker("nersc-flang"),
         create_worker("alcf-theta-flang", properties={'jobs': 12}, max_builds=1),
 
         # ML-Driven Compiler Optimizations build workers (Ubuntu x86_64)
         create_worker("ml-opt-dev-x86-64-b1",
                      properties={'jobs': 64}, max_builds=1),
         create_worker("ml-opt-rel-x86-64-b1",
                      properties={'jobs': 64}, max_builds=1),
         create_worker("ml-opt-devrel-x86-64-b1",
                      properties={'jobs': 64}, max_builds=1),
 
         # Ubuntu x86_64
         create_worker("thinlto-x86-64-bot1", properties={'jobs': 64}, max_builds=1),
+
+        # Ubuntu 20.04 on AWS, x86_64 PS4 target
+        create_worker("sie-linux-builder", properties={'jobs': 40}, max_builds=1),
+
+        # XCore target, Ubuntu 20.04 x64 host
+        create_worker("xcore-ubuntu20-x64", properties={'jobs': 4}, max_builds=1),
+
+        # ARC Worker, CentOS 7.9 x86_64 Intel Xeon Platinum 8000 @ 3.6GHz, 32GB RAM
+        create_worker("arc-worker", properties={'jobs': 16}, max_builds=1),
+
+        # OpenMP on AMDGPU, Ubuntu 18.04.5, Intel(R) Xeon(R) Gold 5218 @ 2.30GHz with 64GB Memory, 1 Vega20 GPU with 16GB Memory
+        create_worker("omp-vega20-0", properties={'jobs': 32}, max_builds=1),
         ]

buildbot/osuosl/master/master.cfg

 
 ####### BUILDERS
 
 from buildbot.plugins import util
 
 c['builders'] = builders = [
   util.BuilderConfig(**b) for b in config.builders.all
 ]
+builders.extend([
+  util.BuilderConfig(**b) for b in config.release_builders.all
+])
 
 ####### SCHEDULERS
 
-c['schedulers'] = config.schedulers.getSingleBranchSchedulers(
-                                        c['builders'])
+c['schedulers'] = config.schedulers.getMainBranchSchedulers(
+                                             builders)
+c['schedulers'].extend(config.schedulers.getReleaseBranchSchedulers(
+                                             builders,
+                                             treeStableTimer=5*60))
 c['schedulers'].extend(config.schedulers.getForceSchedulers(
-                                        c['builders']))
+                                             builders))
 
 ####### BUILDBOT SERVICES
 
 # 'services' is a list of BuildbotService items like reporter targets. The
 # status of each build will be pushed to these targets. buildbot/reporters/*.py
 # has a variety to choose from, like IRC bots.
 
 # Configure MailNotifier, IRC, buildbot.reporters.github.GitHubStatusPush, buildbot.reporters.github.GitHubCommentPush

llvmbisect/llvmlab/llvmlab.py

BUILD_NAME_REGEX = re.compile(
     r"-([A-Z][A-Za-z]+)(\.(.*))?")
 
 
 class Build(object):
     @staticmethod
     def frombasename(str, url=None):
 
         str = os.path.basename(str)
-        revision = timestamp = build = None
+        revision = revision_prefix = sha = timestamp = build = None
 
         # Check if this is a BNI style build.
         m = BUILD_NAME_REGEX.match(str)
         if m:
             name, _, major_str, _, minor_str, _, micro_str, build, \
                 _, suffix = m.groups()
             revision = [int(major_str)]
             if minor_str:
                 revision.append(int(minor_str))
                 if micro_str:
                     revision.append(int(micro_str))
-            return Build(name, tuple(revision), None, build, suffix)
+            return Build(name, None, tuple(revision), None, None, build, suffix)
 
         if '.' in str:
             str, suffix = str.split('.', 1)
         else:
             suffix = None
 
         m = re.match(r'(.*)-b([0-9]+)', str)
         if m:
             str, build = m.groups()
             build = int(build)
 
         m = re.match(r'(.*)-t([0-9-]{8,10}_[0-9-]{6,8})', str)
         if m:
             str, timestamp = m.groups()
 
+        m = re.match(r'(.*)-d([0-9]+)-(.*)', str)
+        if m:
+            str, revision, sha = m.groups()
+            revision = int(revision)
+            revision_prefix = 'd'
+
         m = re.match(r'(.*)-r([0-9]+)', str)
         if m:
             str, revision = m.groups()
             revision = int(revision)
+            revision_prefix = 'r'
 
-        return Build(str, revision, timestamp, build, suffix, url)
+        return Build(str, revision_prefix, revision, sha, timestamp,
+                     build, suffix, url)
 
     @staticmethod
     def fromdata(data):
         return Build(data['name'], data['revision'], data['timestamp'],
                      data['build'], data['suffix'])
 
     def todata(self):
         return {'name': self.name,
                 'revision': self.revision,
+                'sha': self.sha,
                 'timestamp': self.timestamp,
                 'build': self.build,
                 'suffix': self.suffix}
 
-    def __init__(self, name, revision, timestamp, build, suffix, url):
+    def __init__(self, name, revision_prefix, revision, sha, timestamp,
+                 build, suffix, url):
         self.name = name
+        self.revision_prefix = revision_prefix
         self.revision = revision
+        self.sha = sha
         self.timestamp = timestamp
         self.build = build
         self.suffix = suffix
         self.url = url
 
     def tobasename(self, include_suffix=True):
         basename = self.name
         if self.revision is not None:
             if isinstance(self.revision, (tuple, list)):
                 basename += '-' + '.'.join(str(r) for r in self.revision)
             else:
                 assert isinstance(self.revision, int)
-                basename += '-r%d' % self.revision
+                basename += '-%s%d' % (self.revision_prefix, self.revision)
+        if self.sha is not None:
+            basename += '-%s' % self.sha
         if self.timestamp is not None:
             basename += '-t%s' % self.timestamp
         if self.build is not None:
             if isinstance(self.build, str):
                 basename += '-' + self.build
             else:
                 basename += '-b%d' % self.build
         if include_suffix and self.suffix is not None:
             basename += '.%s' % self.suffix
         return basename
 
     def __repr__(self):
         return "%s%r" % (self.__class__.__name__,
-                         (self.name, self.revision, self.timestamp, self.build,
-                          self.suffix))
+                         (self.name, self.revision, self.sha, self.timestamp,
+                          self.build, self.suffix))
 
     def __cmp__(self, other):
         return cmp((self.revision, self.timestamp,
                     self.build, self.suffix, self.name),
                    ((other.revision, other.timestamp,
                     other.build, other.suffix, other.name)))
 
 

requirements.txt

-attrs==20.2.0
+attrs==20.3.0
 six==1.15.0
 Automat==20.2.0
 constantly==15.1.0
 idna==2.10
-hyperlink==20.0.1
+hyperlink==21.0.0
 incremental==17.5.0
-Jinja2==2.11.2
+Jinja2==2.11.3
 PyHamcrest==2.0.2
 Tempita==0.5.2
 decorator==4.4.2
 pbr==5.5.1
 sqlparse==0.4.1
-SQLAlchemy==1.3.20
+SQLAlchemy==1.3.23
 sqlalchemy-migrate==0.13.0
-certifi==2020.6.20
-chardet==3.0.4
-urllib3==1.25.11
-requests==2.24.0
-zope.interface==5.1.2
+certifi==2020.12.5
+chardet==4.0.0
+urllib3==1.26.4
+requests==2.25.1
+zope.interface==5.2.0
 pycparser==2.20
-pyOpenSSL==19.1.0
+pyOpenSSL==20.0.1
 pyasn1==0.4.8
 pyasn1-modules==0.2.8
 service-identity==18.1.0
-cryptography==3.2.1
-cffi==1.14.3
+cryptography==3.4.5
+cffi==1.14.5
 Twisted==20.3.0
-treq==20.9.0
+treq==21.1.0

tasks/tasktool/README.md

 This is usually done by querying the repository definitions in the `repos`
 directory (and `repos.try` in some modes). For example for an artifact
 parameter called `llvm` we could have a `repos/llvm.json` configuration file
 to query the latest revision of a git repository:
 ```json
 {
     "url": "https://github.com/my-project.git",
     "type": "git",
-    "default_rev": "refs/heads/master"
+    "default_rev": "refs/heads/main"
 }
 ```
 
 To query the latest artifact from a directory in the A2 artifact server:
 ```json
 {
     "url": "http://lab.llvm.org:8080/artifacts/clang-stage1-configure-RA",
     "type": "artifact_server"

 VARIABLE="$(build get arg --optional PARAMETER_NAME)"
 ```
 
 Other default values can be supplied with the usual shell tricks:
 ```bash
 : ${VARIABLE:='DefaultValue'}
 ```
 
-- Abstract artifacts like `refs/heads/master` in a git repository or
+- Abstract artifacts like `refs/heads/main` in a git repository or
   `clang-stage2-cmake` on an A2 server are resolved to a concrete git revision
   or a specific URL when the task is started.
 - A task with a set of artifact and parameter inputs is called a build.
 - `task try mytask.sh` resolves artifact inputs and executes a local build.
 - `task submit mytask.sh` resolves artifact inputs and submits a build to
   a queue on a jenkins server.
 - `task jenkinsrun mytask.sh` may be used to run the task as part of a
   recurring jenking project.
 
 ### Config Artifact ###
 
 The task files are part of a version control repository. When running a task
 this repository is always checked out to a subdirectory named `config`.  This
 allows task files to bundle and reference additional files in the repository
 such as helper scripts shared between multiple tasks.

tasks/tasktool/tasktool/repos/git.py

if config.get('url') is None:
         raise Exception("No 'url' specified")
 
 
 def resolve_latest(config):
     assert(config['type'] == 'git')
     rev = config.get('rev')
     default_rev = config.pop('default_rev', None)
     if default_rev is None:
-        default_rev = "refs/heads/master"
+        default_rev = "refs/heads/main"
     if rev is not None:
         return
     url = config['url']
     revs = _git_ls_remote(url, default_rev)
     # Look for an exact match
     if default_rev is not None:
         rev = revs.get(default_rev, None)
     # Otherwise hope that we only have a single match.

tasks/utils/venv.sh

 echo "@@@ LNT VirtualEnv @@@"
-/usr/local/bin/virtualenv venv
+/usr/local/bin/virtualenv --python=python3 venv
 set +u
 . venv/bin/activate
 set -u
 echo "@@@@@@@"

test/jenkins/test_monorepo_build.py

 # Testing for the Jenkins monorepo_build.py script
 #
 # RUN: export TESTING=1
 # RUN: export JOB_NAME="FOO"
 # RUN: export BUILD_NUMBER=321
-# RUN: export BRANCH=master
+# RUN: export BRANCH=main
 # Tell monorepo_build.py to just print commands instead of running.
 # RUN: mkdir -p %t.SANDBOX/host-compiler/lib %t.SANDBOX/host-compiler/bin %t.SANDBOX/llvm-project/llvm %t.SANDBOX/llvm-project/clang %t.SANDBOX/llvm-project/libcxx %t.SANDBOX/llvm-project/compiler-rt %t.SANDBOX/llvm-project/debuginfo-tests %t.SANDBOX/llvm-project/clang-tools-extra %t.SANDBOX/llvm-project/lldb
 # RUN: touch %t.SANDBOX/host-compiler/bin/clang
 # RUN: python %{src_root}/zorg/jenkins/monorepo_build.py clang all > %t.log
 # RUN: FileCheck --check-prefix CHECK-SIMPLE < %t.log %s
 # CHECK-SIMPLE: @@@ Build Directory @@@
 # CHECK-SIMPLE: cd
 # CHECK-SIMPLE: 'mkdir' '-p'

 # Test to check if timeout flag is actually being set.
 # RUN: python %{src_root}/zorg/jenkins/monorepo_build.py cmake all --timeout=900 > %t-timeout.log
 # RUN: FileCheck --check-prefix CHECK-TIMEOUT < %t-timeout.log %s
 # CHECK-TIMEOUT: --timeout=900
 
 # Test to check if default timeout is being set to 600.
 # RUN: python %{src_root}/zorg/jenkins/monorepo_build.py cmake all > %t-timeout-default.log
 # RUN: FileCheck --check-prefix CHECK-TIMEOUT-DEFAULT < %t-timeout-default.log %s
-# CHECK-TIMEOUT-DEFAULT: --timeout=600
+# CHECK-TIMEOUT-DEFAULT: --timeout=600
 

zorg/buildbot/builders/AOSPBuilder.py

 def getAOSPBuildCommand(
     device,                # Target device for AOSP build
     timeout=None,          # Maximum CPU time in seconds, umlimited if 'None'
     target_clang=None,     # Path to the Clang used for AOSP target build
                            # Set target_clang to None to use AOSP's default Clang
     target_flags=None,     # Extra C/CXX flags for AOSP target build
     jobs=None,             # Number of concurrent jobs
     extra_make_args=None): # Extra args for the make command
-    command = "source build/envsetup.sh"
+    command = "/bin/bash -c \"source build/envsetup.sh"
     command += " && lunch aosp_%s-userdebug" % device
+    command += " && export LC_ALL=C"
     command += " && make -k"
     if timeout:
         command += " TIMEOUT=%s" % timeout
     if target_clang:
         command += " TARGET_CLANG=%s" % target_clang
     if target_flags:
         command += " TARGET_FLAGS='%s'" % target_flags
     if extra_make_args:
         command += " " + extra_make_args
     if jobs:
         command += " -j" + str(jobs)
+    command += "\""
     return command
 
 
 def getAOSPBuildFactory(
     device,                # Target device for AOSP build
     extra_cmake_args=[],   # Extra args for the LLVM cmake command
                            # This flag is ignored if build_clang is False
     timeout=None,          # Maximum CPU time in seconds, umlimited if 'None'

zorg/buildbot/builders/AnnotatedBuilder.py

 from buildbot.process.properties import WithProperties
 from buildbot.steps.shell import SetProperty
 from zorg.buildbot.commands.AnnotatedCommand import AnnotatedCommand
 from zorg.buildbot.process.factory import LLVMBuildFactory
 
 
 def getAnnotatedBuildFactory(
     script,
     clean=False,
     depends_on_projects=None,
     env=None,
     extra_args=None,
     timeout=1200,
-    checkout_llvm_sources=True):
+    checkout_llvm_sources=True,
+    script_interpreter="python"):
     """
     Returns a new build factory that uses AnnotatedCommand, which
     allows the build to be run by version-controlled scripts that do
     not require a buildmaster restart to update.
 
     script: script under "builders/annotated" to be run by python
     clean: set to true for a clean build of llvm
     depends_on_projects: which subprojects to enable
         llvm must be first in the list
         (default: ["llvm", "clang", "compiler-rt", "libcxx",
                    "libcxxabi", "libunwind", "lld"])
     env: environment overrides (map; default is no overrides)
     extra_args: extra arguments to pass to the script (default: [])
     timeout: specifies the builder's timeout in seconds (default: 1200)
+    script_interpreter: specifies the interpreter to run scripts (default: "python")
     """
 
     if depends_on_projects is None:
         depends_on_projects = [
             "llvm",
             "clang",
             "compiler-rt",
             "libcxx",
             "libcxxabi",
             "libunwind",
             "lld"]
     if extra_args is None:
-        extra_args = []
+        # We used to add --jobs to all script invocations. Perserve this
+        # for cases when the user did not specify extra_args, but allow
+        # overriding it if the user did specify extra_args.
+        extra_args = ["--jobs=%(jobs:-)s"]
 
     f = LLVMBuildFactory(
         clean=clean,
         depends_on_projects=depends_on_projects)
 
     if clean:
         f.addStep(SetProperty(property='clean', command='echo 1'))
 
if checkout_llvm_sources:
       f.addGetSourcecodeSteps()
 
     extra_args_with_props = [WithProperties(arg) for arg in extra_args]
     # Explicitly use '/' as separator, because it works on *nix and Windows.
     if script.startswith('/'):
       command = [script]
     else:
       script_path = "../llvm-zorg/zorg/buildbot/builders/annotated/%s" % (script)
-      command = ["python", script_path, WithProperties("--jobs=%(jobs:-)s")]
+      # Handle scripts with script_interpreter, otherwise execute the script directly.
+      if script_interpreter:
+        command = [script_interpreter, script_path]
+      else:
+        command = [script_path]
+
     command += extra_args_with_props
 
     f.addStep(AnnotatedCommand(name="annotate",
                                description="annotate",
                                timeout=timeout,
                                haltOnFailure=True,
                                command=command,
                                env=merged_env))
     return f

zorg/buildbot/builders/ClangBuilder.py

 import copy
 from datetime import datetime
 
+from buildbot.plugins import util
 from buildbot.process.properties import WithProperties, Property
 from buildbot.steps.shell import ShellCommand, SetProperty
 from buildbot.steps.shell import WarningCountingShellCommand
 
 import zorg.buildbot.builders.Util as builders_util
 
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.conditions.FileConditions import FileDoesNotExist
if stage1_upload_directory:
                           use_pixz_compression=use_pixz_compression,
                           xz_compression_factor=xz_compression_factor)
 
     # Compute the cmake define flag to set the C and C++ compiler to clang. Use
     # clang-cl if we used MSVC for stage1.
     if not vs:
         cc = 'clang'
         cxx = 'clang++'
+        fc = 'flang'
     else:
         cc = 'clang-cl.exe'
         cxx = 'clang-cl.exe'
+        fc = 'flang.exe'
 
 
     ############# STAGE 2
     if useTwoStage:
         # We always cleanly build the stage 2. If the compiler has been
         # changed on the stage 1, we cannot trust any of the intermediate file
         # from the old compiler. And if the stage 1 compiler is the same, we
         # should not build in the first place.
if runTestSuite:
             lit = WithProperties('%(builddir)s/'+stage1_build+'/bin/llvm-lit')
             test_suite_cmd = [python, lnt, 'runtest', 'test-suite',
                               '--no-timestamp',
                               '--sandbox', sandbox,
                               '--test-suite', test_suite_dir,
                               '--cc', cc,
                               '--cxx', cxx,
                               '--use-lit', lit]
+            # Enable fortran if flang is checked out
+            if checkout_flang:
+                fortran_flags = [
+                        '--cmake-define=TEST_SUITE_FORTRAN:STRING=ON',
+                        util.Interpolate(
+                            '--cmake-define=CMAKE_Fortran_COMPILER=' +
+                            '%(prop:builddir)s/'+compiler_path+'/bin/'+fc)]
+                test_suite_cmd.extend(fortran_flags)
             # Append any option provided by the user
             test_suite_cmd.extend(testsuite_flags)
 
         # Only submit if a URL has been specified
         if submitURL is not None:
             if not isinstance(submitURL, list):
                 submitURL = [submitURL]
             for url in submitURL:
if runTestSuite:
         # Steps to prepare, build and run LNT
         f.addStep(ShellCommand(name='clean sandbox',
                                command=['rm', '-rf', 'sandbox'],
                                haltOnFailure=True,
                                description='removing sandbox directory',
                                workdir='test',
                                env=env))
         f.addStep(ShellCommand(name='recreate sandbox',
-                               command=['virtualenv', 'sandbox'],
+                               command=['virtualenv', '--python=python3', 'sandbox'],
                                haltOnFailure=True,
                                description='recreating sandbox',
                                workdir='test',
                                env=env))
         f.addStep(ShellCommand(name='setup lit',
                                command=[python, lnt_setup, 'develop'],
                                haltOnFailure=True,
                                description='setting up LNT in sandbox',

zorg/buildbot/builders/ClangLTOBuilder.py

CmakeCommand.applyDefaultOptions(cmake_args, [
         ])
 
     # Some options are required for this stage no matter what.
     CmakeCommand.applyRequiredOptions(cmake_args, [
         ('-G',                      'Ninja'),
         ('-DCMAKE_INSTALL_PREFIX=', install_dir),
         ])
 
+    if f.enable_projects:
         CmakeCommand.applyRequiredOptions(cmake_args, [
-        ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)),
+            ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),
+            ])
+    if f.enable_runtimes:
+        CmakeCommand.applyRequiredOptions(cmake_args, [
+            ('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.enable_runtimes)),
             ])
 
     # Note: On this stage we do not care of warnings, as we build with
     # a system toolchain and cannot control the environment.
     # Warnings are likely, and we ignore them.
 
     # Create configuration files with cmake
     f.addStep(CmakeCommand(name="cmake-configure-stage%s" % stage_num,
                            description=["stage%s cmake configure" % stage_num],
cmake_args.append(
         WithProperties(
             "-DCMAKE_CXX_COMPILER=%(builddir)s/" + staged_install + "/bin/clang++"
         ))
     cmake_args.append(
         WithProperties(
             "-DCMAKE_C_COMPILER=%(builddir)s/" + staged_install + "/bin/clang"
         ))
 
+    if f.enable_projects:
+        CmakeCommand.applyRequiredOptions(cmake_args, [
+            ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),
+            ])
+    if f.enable_runtimes:
         CmakeCommand.applyRequiredOptions(cmake_args, [
-        ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)),
+            ('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.enable_runtimes)),
             ])
 
     # Create configuration files with cmake
     f.addStep(CmakeCommand(name="cmake-configure-stage%s" % stage_num,
                            description=["stage%s cmake configure" % stage_num],
                            haltOnFailure=True,
                            options=cmake_args,
                            path=src_dir,
                            env=env,
f.addStep(NinjaCommand(name="install-stage%s-compiler"% stage_num,
                            ))
 
 
 def getClangWithLTOBuildFactory(
            depends_on_projects = None,
            clean = False,
            jobs  = None,
            extra_configure_args = None,
+           extra_configure_args_lto_stage = None,
            compare_last_2_stages = True,
            lto = None, # The string gets passed to -flto flag as is. Like -flto=thin.
            env = None,
            **kwargs):
 
     # Set defaults
     if depends_on_projects:
         depends_on_projects = list(depends_on_projects)
     else:
         # By default we link with LLD.
         depends_on_projects = ['llvm', 'clang', 'lld']
 
     if lto is None:
         lto = 'ON'
 
     if extra_configure_args is None:
         extra_configure_args = []
     else:
         extra_configure_args = list(extra_configure_args)
 
+    if extra_configure_args_lto_stage is None:
+        extra_configure_args_lto_stage = []
+    else:
+        extra_configure_args_lto_stage = list(extra_configure_args_lto_stage)
+
     # Make sure CMAKE_INSTALL_PREFIX and -G are not specified
     # in the extra_configure_args. We set them internally as needed.
     # TODO: assert extra_configure_args.
-    install_prefix_specified = any(a.startswith('-DCMAKE_INSTALL_PREFIX=') for a in extra_configure_args)
+    install_prefix_specified = (any(a.startswith('-DCMAKE_INSTALL_PREFIX=') for a in extra_configure_args) or
+                                any(a.startswith('-DCMAKE_INSTALL_PREFIX=') for a in extra_configure_args_lto_stage))
     assert not install_prefix_specified, "Please do not explicitly specify the install prefix for multi-stage build."
 
     # Prepare environmental variables. Set here all env we want everywhere.
     merged_env = {
         'TERM' : 'dumb' # Be cautious and disable color output from all tools.
     }
     if env is not None:
         # Overwrite pre-set items with the given ones, so user can set anything.
if 'lld' in depends_on_projects:
         CmakeCommand.applyRequiredOptions(extra_configure_args, [
             ('-DLLVM_ENABLE_LLD=', 'ON'),
             ])
 
     # The rest are test stages, which depend on the staged compiler we are ultimately after.
     s = f.staged_compiler_idx + 1
     staged_install = f.stage_installdirs[f.staged_compiler_idx]
     for i in range(s, len(f.stage_objdirs[s:]) + s):
-        configure_args = extra_configure_args[:]
+        configure_args = extra_configure_args[:] + extra_configure_args_lto_stage[:]
 
         configure_args.append(
             WithProperties(
                 "-DCMAKE_AR=%(builddir)s/" + staged_install + "/bin/llvm-ar"
             ))
         configure_args.append(
             WithProperties(
                 "-DCMAKE_RANLIB=%(builddir)s/" + staged_install + "/bin/llvm-ranlib"

zorg/buildbot/builders/DoxygenDocsBuilder.py

+from collections import OrderedDict
+from importlib import reload
+
+from buildbot.plugins import steps, util
+from buildbot.steps.shell import ShellCommand
+
+from zorg.buildbot.builders import UnifiedTreeBuilder
+from zorg.buildbot.commands import CmakeCommand
+
+from zorg.buildbot.process import factory
+reload(factory)
+reload(UnifiedTreeBuilder)
+reload(CmakeCommand)
+
+llvm_docs = OrderedDict([
+  # Project             Build target     Local path                                    Remote path
+  ("llvm",              ("doxygen",      "docs/doxygen/html/",                         "llvm")),
+  ("clang",             (None,           "tools/clang/docs/doxygen/html/",             "cfe")),
+  ("clang-tools-extra", (None,           "tools/clang/tools/extra/docs/doxygen/html/", "cfe-extra")),
+  ("flang",             (None,           "tools/flang/docs/doxygen/html/",             "flang")),
+  ("polly",             (None,           "tools/polly/docs/doxygen/html/",             "polly")),
+  ("openmp",            (None,           "projects/openmp/docs/doxygen/html/",         "openmp")),
+  ("lldb",              ("lldb-cpp-doc", "tools/lldb/docs/cpp_reference/",             "lldb/cpp_reference")),
+  # NOTE: 5/9/2020 lldb-python-doc fails to build. Disabled till be fixed.
+  #(None,   ("lldb-python-doc",         "tools/lldb/docs/python_reference/",  "lldb")),
+])
+
+def getLLVMDocsBuildFactory(
+        clean = True,
+        depends_on_projects = None,
+        extra_configure_args = None,
+        env = None,
+        **kwargs):
+
+    if depends_on_projects is None:
+        # All the projects from llvm_docs, and remove all duplicates.
+        _depends_on_projects=list(set(
+            [project for project in llvm_docs if project]))
+    else:
+        # Make a local copy of depends_on_projects, as we are going to modify
+        # that.
+        _depends_on_projects=depends_on_projects[:]
+        # Some projects are interdependent for the purpose of documentation.
+        # Enforce the dependencies.
+        # TODO: Check later the dependencies for doxygen docs and enforce them
+        # here if needed.
+
+    # Make a local copy of the configure args, as we are going to modify that.
+    if extra_configure_args:
+        cmake_args = extra_configure_args[:]
+    else:
+        cmake_args = list()
+
+    # Prepare environmental variables. Set here all env we want everywhere.
+    merged_env = {
+        'TERM' : 'dumb' # Be cautious and disable color output from all tools.
+    }
+    if env is not None:
+        # Overwrite pre-set items with the given ones, so user can set anything.
+        merged_env.update(env)
+
+    CmakeCommand.CmakeCommand.applyDefaultOptions(cmake_args, [
+        ("-G",                                "Ninja"),
+        ("-DLLVM_ENABLE_DOXYGEN=",            "ON"),
+        ("-DLLVM_BUILD_DOCS=",                "ON"),
+        ("-DCLANG_TOOLS_EXTRA_INCLUDE_DOCS=", "ON"),
+        ("-DLLVM_ENABLE_ASSERTIONS=",         "OFF"),
+        ("-DCMAKE_BUILD_TYPE=",               "Release"),
+        ])
+
+    f = UnifiedTreeBuilder.getCmakeBuildFactory(
+            clean=clean,
+            depends_on_projects=_depends_on_projects,
+            enable_runtimes=[], # Docs don't support runtimes build yet.
+            extra_configure_args=cmake_args,
+            env=merged_env,
+            **kwargs) # Pass through all the extra arguments.
+
+    # Build the documentation for all the projects.
+    for project in llvm_docs:
+        target = llvm_docs[project][0]
+
+        # Build only those with specifies targets.
+        if target:
+            UnifiedTreeBuilder.addNinjaSteps(
+                f,
+                # Doxygen builds the final result for really
+                # long time without any output.
+                # We have to have a long timeout at this step.
+                timeout=21600,
+                targets=[target],
+                checks=[],
+                env=merged_env,
+                **kwargs)
+
+    # Publish just built documentation
+    for project in llvm_docs:
+        target, local_path, remote_path = llvm_docs[project]
+
+        f.addStep(
+            ShellCommand(
+                name="Publish {}".format(project or target),
+                description=[
+                    "Publish", "just", "built", "documentation", "for",
+                    "{}".format(project or target)
+                    ],
+                command=[
+                    'rsync',
+                    '-vrl',
+                    '--delete', '--force', '--delay-updates', '--delete-delay',
+                    '--ignore-times',
+                    '--checksum',
+                    '-p', '--chmod=Du=rwx,Dg=rwx,Do=rx,Fu=rw,Fg=rw,Fo=r',
+                    "{}".format(local_path),
+                    "lists.llvm.org:web/doxygen/{}".format(remote_path),
+                    ],
+                env=merged_env,
+            )
+        )
+
+    return f

zorg/buildbot/builders/FlangBuilder.py

 from zorg.buildbot.commands.CmakeCommand import CmakeCommand
 from zorg.buildbot.process.factory import LLVMBuildFactory
 from zorg.buildbot.builders.UnifiedTreeBuilder import addNinjaSteps, getCmakeWithNinjaBuildFactory
 
 def getFlangOutOfTreeBuildFactory(
            checks = None,
            clean = False,
            llvm_extra_configure_args = None,
            flang_extra_configure_args = None,
            env = None,
            **kwargs):
 
     if env is None:
         env = dict()
 
     f = getCmakeWithNinjaBuildFactory(
-            depends_on_projects=['llvm','mlir'],
+            depends_on_projects=['llvm','clang','mlir'],
             obj_dir="build_llvm",
             checks=[],
+            install_dir="install",
             clean=clean,
             extra_configure_args=llvm_extra_configure_args,
             env=env,
             **kwargs)
 
     if checks is None:
         checks = ['check-all']
 
CmakeCommand.applyRequiredOptions(flang_cmake_args, [
         ])
 
     flang_obj_dir = "build_flang"
     flang_src_dir = "{}/flang".format(f.monorepo_dir)
 
     # Add LLVM_DIR and MLIR_DIR to the CMake invocation.
     llvm_dir = "{}/lib/cmake/llvm".format(f.obj_dir)
     mlir_dir = "{}/lib/cmake/mlir".format(f.obj_dir)
+    clang_dir = "{}/lib/cmake/clang".format(f.install_dir)
     CmakeCommand.applyRequiredOptions(flang_cmake_args, [
         # We actually need the paths to be relative to the source directory,
         # otherwise find_package can't locate the config files.
         ('-DLLVM_DIR:PATH=',
             LLVMBuildFactory.pathRelativeTo(llvm_dir, flang_src_dir)),
         ('-DMLIR_DIR:PATH=',
             LLVMBuildFactory.pathRelativeTo(mlir_dir, flang_src_dir)),
+        ('-DCLANG_DIR:PATH=',
+            LLVMBuildFactory.pathRelativeTo(clang_dir, flang_src_dir)),
         ])
 
     # We can't use addCmakeSteps as that would use the path in f.llvm_srcdir.
     f.addStep(CmakeCommand(name="cmake-configure-flang",
                            haltOnFailure=True,
                            description=["CMake", "configure", "flang"],
                            options=flang_cmake_args,
                            path=LLVMBuildFactory.pathRelativeTo(flang_src_dir,

zorg/buildbot/builders/HtmlDocsBuilder.py

+from collections import OrderedDict
+from importlib import reload
+
+from buildbot.plugins import steps, util
+from buildbot.steps.shell import ShellCommand
+from buildbot.steps.shell import WarningCountingShellCommand
+
+from zorg.buildbot.process import factory
+reload(factory)
+
+llvm_docs = OrderedDict([
+  # Project   Build target  Build path    Local path       Remote path
+  ("lnt",     ("html",      "docs",       "_build/html/",  "lnt")),
+])
+
+# We build with make for now. Change later if needed.
+build_cmd = 'make'
+
+def getHtmlDocsBuildFactory(
+        depends_on_projects = None,
+        clean = False,
+        env = None,
+        **kwargs):
+
+    if depends_on_projects is None:
+        # All the projects by default.
+        _depends_on_projects=llvm_docs.keys()
+    else:
+        # Make a local copy of depends_on_projects, as we are going to modify
+        # that.
+        _depends_on_projects=depends_on_projects[:]
+
+    # Prepare environmental variables. Set here all env we want everywhere.
+    merged_env = {
+        'TERM' : 'dumb' # Be cautious and disable color output from all tools.
+    }
+    if env is not None:
+        # Overwrite pre-set items with the given ones, so user can set anything.
+        merged_env.update(env)
+
+    # HTML Sphinx documentation builds in tree, each in its own directory.
+    # For that, make sure the obj_dir is the same as llvm_srcdir.
+    src_dir = kwargs.pop('llvm_srcdir', '.')
+    f = factory.LLVMBuildFactory(
+            clean=clean,
+            depends_on_projects=_depends_on_projects,
+            llvm_srcdir=src_dir,
+            obj_dir=src_dir,
+            **kwargs) # Pass through all the extra arguments.
+
+    # Build the documentation
+    for project in llvm_docs:
+
+        # Checkout the source code and remove all the untracked files, so
+        # we would build a fresh new documentation.
+        f.addStep(
+            steps.Git(
+                name='Checkout the {} source code'.format(project),
+                repourl=f.repourl_prefix + "llvm-{}.git".format(project),
+                mode='full',
+                method='fresh',
+                progress=True,
+                workdir=util.WithProperties(project),
+                env=merged_env,
+                **kwargs))
+
+        target, build_path, local_path, remote_path = llvm_docs[project]
+
+        build_dir = util.WithProperties(
+                            "{}".format("/".join([
+                                project,
+                                build_path])))
+        f.addStep(
+            steps.WarningCountingShellCommand(
+                name="Build {} documentation".format(project),
+                command=[build_cmd, target],
+                haltOnFailure=True,
+                workdir=build_dir,
+                env=merged_env,
+                **kwargs))
+
+        # Publish just built documentation
+        f.addStep(
+            ShellCommand(
+                name="Publish {}".format(project),
+                description=[
+                    "Publish", "just", "built", "documentation", "for",
+                    "{}".format(project)
+                    ],
+                command=[
+                    'rsync',
+                    '-vrl',
+                    '--delete', '--force', '--delay-updates', '--delete-delay',
+                    '--ignore-times',
+                    '--checksum',
+                    '-p', '--chmod=Du=rwx,Dg=rwx,Do=rx,Fu=rw,Fg=rw,Fo=r',
+                    "{}".format(local_path),
+                    "lists.llvm.org:web/{}".format(remote_path),
+                    ],
+                workdir=build_dir,
+                env=merged_env,
+            )
+        )
+
+    return f

zorg/buildbot/builders/LibcxxAndAbiBuilder.py

 import buildbot.steps.shell
 import buildbot.process.properties as properties
 
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.commands.CmakeCommand import CmakeCommand
 from zorg.buildbot.process.factory import LLVMBuildFactory
 from zorg.buildbot.builders import UnifiedTreeBuilder
 
 def getLibcxxAndAbiBuilder(f=None, env=None,
                            cmake_extra_opts=None, lit_extra_opts=None,
                            lit_extra_args=None, check_libcxx_abilist=False,
                            check_libcxx_benchmarks=None,
                            depends_on_projects=None,
                            use_cache=None,
+                           build_standalone=False,
                            **kwargs):
 
     if env is None:
         env = {}
     if cmake_extra_opts is None:
         cmake_extra_opts = {}
     if lit_extra_opts is None:
         lit_extra_opts = {}
     if lit_extra_args is None:
         lit_extra_args = []
 
     if depends_on_projects is None:
         depends_on_projects = ['libcxx','libcxxabi','libunwind']
 
     src_root = 'llvm'
+
     build_path = 'build'
 
+    if build_standalone:
+        src_to_build_dir = 'runtimes'
+    else:
+       src_to_build_dir = None
+
     if f is None:
         f = UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps(
                 depends_on_projects=depends_on_projects,
                 llvm_srcdir=src_root,
+                src_to_build_dir=src_to_build_dir,
                 obj_dir=build_path,
                 **kwargs) # Pass through all the extra arguments.
 
     rel_src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, build_path)
 
     # Specify the max number of threads using properties so LIT doesn't use
     # all the threads on the system.
     litTestArgs = '-vv --show-unsupported --show-xfail --threads=%(jobs)s'
if 'LIBCXX_CXX_ABI' not in cmake_extra_opts:
         cmake_opts.append('-DLIBCXX_CXX_ABI=libcxxabi')
 
     # Nuke/remake build directory and run CMake
     f.addStep(buildbot.steps.shell.ShellCommand(
         name='rm.builddir', command=['rm', '-rf', build_path],
         workdir=".",
         haltOnFailure=False))
 
+    if build_standalone:
+       if f.enable_projects:
+           CmakeCommand.applyRequiredOptions(cmake_opts, [
+            ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),
+            ])
+       if f.enable_runtimes:
+           CmakeCommand.applyRequiredOptions(cmake_opts, [
+            ('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.enable_runtimes)),
+            ])
+    else:
        CmakeCommand.applyRequiredOptions(cmake_opts, [
         ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)),
         ])
 
     f.addStep(buildbot.steps.shell.ShellCommand(
         name='cmake', command=['cmake', rel_src_dir] + cmake_opts,
         haltOnFailure=True, workdir=build_path, env=env))
 
-    # Build libcxxabi
     jobs_flag = properties.WithProperties('-j%(jobs)s')
+    # Build libcxxabi
+    if 'libcxxabi' in depends_on_projects:
         f.addStep(buildbot.steps.shell.ShellCommand(
-              name='build.libcxxabi', command=['make', jobs_flag, 'cxxabi'],
-              haltOnFailure=True, workdir=build_path))
+            haltOnFailure=True, workdir=build_path, env=env))
 
     # Build libcxx
+    if 'libcxx' in depends_on_projects:
         f.addStep(buildbot.steps.shell.ShellCommand(
-              name='build.libcxx', command=['make', jobs_flag, 'cxx'],
-              haltOnFailure=True, workdir=build_path))
+            haltOnFailure=True, workdir=build_path, env=env))
 
     # Test libc++abi
+    if 'libcxxabi' in depends_on_projects:
         f.addStep(LitTestCommand(
             name            = 'test.libcxxabi',
             command         = ['make', jobs_flag, 'check-cxxabi'],
             description     = ['testing', 'libcxxabi'],
             descriptionDone = ['test', 'libcxxabi'],
-        workdir         = build_path))
+            workdir         = build_path,
+            env             = env))
 
     # Test libc++
+    if 'libcxx' in depends_on_projects:
         f.addStep(LitTestCommand(
             name            = 'test.libcxx',
             command         = ['make', jobs_flag, 'check-cxx'],
             description     = ['testing', 'libcxx'],
             descriptionDone = ['test', 'libcxx'],
-        workdir         = build_path))
+            workdir         = build_path,
+            env             = env))
 
     if check_libcxx_abilist:
         f.addStep(buildbot.steps.shell.ShellCommand(
         name            = 'test.libcxx.abilist',
         command         = ['make', 'check-cxx-abilist'],
         description     = ['testing', 'libcxx', 'abi'],
         descriptionDone = ['test', 'libcxx', 'abi'],
-        workdir         = build_path))
+        workdir         = build_path,
+        env             = env))
 
     if check_libcxx_benchmarks:
       # Build the libc++ benchmarks
       f.addStep(buildbot.steps.shell.ShellCommand(
           name='build.libcxx.benchmarks',
           command=['make', jobs_flag, 'cxx-benchmarks'],
-          haltOnFailure=True, workdir=build_path))
+          haltOnFailure=True, workdir=build_path, env=env))
 
       # Run the benchmarks
       f.addStep(LitTestCommand(
           name            = 'test.libcxx.benchmarks',
           command         = ['make', jobs_flag, 'check-cxx-benchmarks'],
           description     = ['testing', 'libcxx', 'benchmarks'],
           descriptionDone = ['test', 'libcxx', 'benchmarks'],
-          workdir         = build_path))
+          workdir         = build_path,
+          env             = env))
 
     return f

zorg/buildbot/builders/OpenMPBuilder.py

 from buildbot.steps.shell import ShellCommand
 from buildbot.process.properties import WithProperties
+from buildbot.plugins import steps
 
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.commands.CmakeCommand import CmakeCommand
 from zorg.buildbot.commands.NinjaCommand import NinjaCommand
 from zorg.buildbot.builders import UnifiedTreeBuilder
 from zorg.buildbot.process.factory import LLVMBuildFactory
 
 def getOpenMPCMakeBuildFactory(
         jobs                = '%(jobs)s',   # Number of concurrent jobs.
         clean               = True,         # "clean" step is requested if true
         env                 = None,         # Environmental variables for all steps.
         ompt                = False,        # Whether to enable the OpenMP Tools Interface.
         test                = True,         # Test the built libraries.
         depends_on_projects = None,
+        enable_runtimes     = None,
+        extraCmakeArgs      = [],
+        install             = False,
+        testsuite           = False,
+        testsuite_sollvevv  = False,
+        extraTestsuiteCmakeArgs = [],
         **kwargs):
 
     # Prepare environmental variables. Set here all env we want everywhere.
     merged_env = {
         'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences.
                  }
     # Overwrite pre-set items with the given ones, so user can set anything.
     if env is not None:
         merged_env.update(env)
 
     llvm_srcdir = 'llvm.src'
     llvm_builddir = 'llvm.build'
+    llvm_instdir = 'llvm.inst'
+    testsuite_srcdir = "test-suite.src"
+    testsuite_builddir = "test-suite.build"
+    sollvevv_srcdir = "sollvevv.src"
 
     cleanBuildRequested = lambda step: clean or step.build.getProperty("clean", default=step.build.getProperty("clean_obj"))
 
     if depends_on_projects is None:
         # Monorepo configuration requires llvm and clang to get cmake work.
         depends_on_projects = ['llvm', 'clang', 'openmp']
 
     f = UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps(
             depends_on_projects=depends_on_projects,
+            enable_runtimes=enable_runtimes,
             llvm_srcdir=llvm_srcdir,
             obj_dir=llvm_builddir,
             cleanBuildRequested=cleanBuildRequested,
             env=merged_env,
             **kwargs) # Pass through all the extra arguments.
 
     f.addStep(
         ShellCommand(
             name            = 'clean',
             command         = ['rm', '-rf', f.obj_dir],
             warnOnFailure   = True,
             description     = ['clean'],
             doStepIf        = cleanBuildRequested,
             workdir         = '.',
             env             = merged_env))
 
     # Configure LLVM and OpenMP (and Clang, if requested).
     cmake_args = ['-DCMAKE_BUILD_TYPE=Release', '-DLLVM_ENABLE_ASSERTIONS=ON']
     if ompt:
         cmake_args += ['-DLIBOMP_OMPT_SUPPORT=ON']
     if test:
         lit_args = '-vv --show-unsupported --show-xfail -j %s' % jobs
         cmake_args += [WithProperties('-DLLVM_LIT_ARGS=%s' % lit_args)]
+    if install:
+        cmake_args += [WithProperties('-DCMAKE_INSTALL_PREFIX=%(builddir)s/' + llvm_instdir)]
+    cmake_args += extraCmakeArgs
+
+    if f.enable_projects:
+        CmakeCommand.applyDefaultOptions(cmake_args, [
+            ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),
+            ])
 
-    CmakeCommand.applyRequiredOptions(cmake_args, [
-        ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)),
+    if f.enable_runtimes:
+        CmakeCommand.applyDefaultOptions(cmake_args, [
+            ('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.enable_runtimes)),
             ])
 
     # Add llvm-lit and clang (if built) to PATH
     merged_env.update({
         'PATH': WithProperties('%(builddir)s/' + llvm_builddir + '/bin:${PATH}')})
 
     src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, f.obj_dir)
 
     f.addStep(CmakeCommand(name='configure-openmp',
if test:
         ninja_test_args = ['ninja', WithProperties('-j %s' % jobs)]
         f.addStep(
             LitTestCommand(
                 name        = 'test-openmp',
                 command     = ninja_test_args + ['check-openmp'],
                 description = 'test openmp',
                 workdir     = f.obj_dir,
                 env         = merged_env,
+                haltOnFailure=False,
+                flunkOnFailure=True))
+
+    clangexe = "%(builddir)s/" + llvm_builddir + "/bin/clang"
+    clangxxexe = "%(builddir)s/" + llvm_builddir + "/bin/clang++"
+    litexe = "%(builddir)s/" + llvm_builddir + "/bin/llvm-lit"
+    libdir = "%(builddir)s/" + llvm_builddir + "/lib"
+    if install:
+        f.addStep(steps.RemoveDirectory(name="LLVM: Clean Install Dir",
+                               dir=llvm_instdir,
+                               haltOnFailure=False))
+
+        f.addStep(NinjaCommand(name="LLVM: Install",
+                               description="installing",
+                               descriptionDone="install",
+                               descriptionSuffix="LLVM",
+                               workdir=f.obj_dir,
+                               targets=['install'],
+                               env=merged_env,
                                haltOnFailure=True))
+        # If installing, use the installed version of clang.
+        clangexe = "%(builddir)s/" + llvm_instdir + "/bin/clang"
+        clangxxexe = "%(builddir)s/" + llvm_instdir + "/bin/clang++"
+        libdir = "%(builddir)s/" + llvm_instdir + "/lib"
+
+
+    if testsuite:
+        f.addStep(steps.RemoveDirectory(name="Test-Suite: Clean Source",
+                           dir=testsuite_srcdir,
+                           haltOnFailure=False,
+                           warnOnFailure=True,
+                           doStepIf=cleanBuildRequested))
+
+        f.addGetSourcecodeForProject(name="Test-Suite: Checkout",
+            description="fetching",
+            descriptionDone="fetch",
+            descriptionSuffix="Test-Suite",
+            project='test-suite',
+            src_dir=testsuite_srcdir,
+            alwaysUseLatest=True)
+
+        if testsuite_sollvevv:
+            f.addStep(steps.RemoveDirectory(name="SOLLVE V&V: Clean Source",
+                           dir=sollvevv_srcdir,
+                           haltOnFailure=False,
+                           warnOnFailure=True,
+                           doStepIf=cleanBuildRequested))
+
+            f.addStep(steps.Git(name="SOLLVE V&V: Checkout",
+                    description="fetching",
+                    descriptionDone="fetch",
+                    descriptionSuffix="SOLLVE V&V",
+                    repourl='https://github.com/SOLLVE/sollve_vv.git',
+                    workdir=sollvevv_srcdir,
+                    alwaysUseLatest=True))
+
+        f.addStep(steps.RemoveDirectory(name="Test-Suite: Clean Build",
+                           dir=testsuite_builddir,
+                           haltOnFailure=False,
+                           warnOnFailure=True))
+
+        testsuite_options = [
+            "-DCMAKE_BUILD_TYPE=Release",
+            "-DTEST_SUITE_LIT_FLAGS=-vv;-s;-j6;-o;report.json",
+            "-DTEST_SUITE_EXTRA_C_FLAGS=-gline-tables-only",
+            "-DTEST_SUITE_EXTRA_CXX_FLAGS=-gline-tables-only",
+            WithProperties("-DCMAKE_C_COMPILER=" + clangexe),
+            WithProperties("-DCMAKE_CXX_COMPILER=" + clangxxexe),
+            WithProperties("-DTEST_SUITE_LIT=" + litexe),
+        ]
+        if testsuite_sollvevv:
+            testsuite_options += [
+                "-DTEST_SUITE_SUBDIRS=External/sollve_vv",
+                "-DTEST_SUITE_SOLLVEVV_ROOT=../" + sollvevv_srcdir,
+            ]
+        testsuite_options += extraTestsuiteCmakeArgs
+        f.addStep(CmakeCommand(name="Test-Suite: Configure",
+                            description="configuring",
+                            descriptionDone="configure",
+                            descriptionSuffix="Test-Suite",
+                            generator='Ninja',
+                            path='../' + testsuite_srcdir,
+                            workdir=testsuite_builddir,
+                            options=testsuite_options,
+                            haltOnFailure=True,
+                            logfiles={
+                                'CMakeCache.txt'   : 'CMakeCache.txt',
+                            }))
+
+        f.addStep(NinjaCommand(name="Test-Suite: Compile",
+                            description="compiling",
+                            descriptionDone="compile",
+                            descriptionSuffix="Test-Suite",
+                            options=['-k0'],      # Continue building; programs that don't compile will fail with NOEXE.
+                            haltOnFailure=False,
+                            flunkOnFailure=False, # SOLLVE V&V contains tests that clang have not been implemented yet.
+                            warnOnFailure=True,
+                            workdir=testsuite_builddir))
+
+        merged_env.update({
+            'LD_LIBRARY_PATH': WithProperties(libdir + ':${LD_LIBRARY_PATH}')
+            })
+        f.addStep(LitTestCommand(name="Test-Suite: Run",
+                            description="running",
+                            descriptionDone="run",
+                            descriptionSuffix="Test-Suite",
+                            command=[WithProperties(litexe), '-vv', '-s',  '-j6', '-o','report.json', '.'],
+                            haltOnFailure=False,
+                            flunkOnFailure=False, # SOLLVE V&V contains tests that clang have not been implemented yet.
+                            warnOnFailure=True,
+                            workdir=testsuite_builddir,
+                            env=merged_env,
+                            logfiles={
+                                'test.log'   : 'test.log',
+                                'report.json': 'report.json'
+                            }))
 
     return f

zorg/buildbot/builders/SanitizerBuilder.py

 # TODO: Change WithProperties to Interpolate
 from buildbot.process.properties import WithProperties
 from zorg.buildbot.commands.AnnotatedCommand import AnnotatedCommand
 from zorg.buildbot.process.factory import LLVMBuildFactory
 
 def getSanitizerBuildFactory(
     clean=False,
     depends_on_projects=None,
+    extra_configure_args=None,
     env=None,
     timeout=1200):
 
     # Prepare environmental variables. Set here all env we want everywhere.
     merged_env = {
         "TERM" : "dumb", # Make sure Clang doesn't use color escape sequences.
                  }
     # Clobber bot if we need a clean build.
     if clean:
         merged_env["BUILDBOT_CLOBBER"] = "1"
     # Overwrite pre-set items with the given ones, so user can set anything.
     if env is not None:
         merged_env.update(env)
 
     if depends_on_projects is None:
         depends_on_projects = [
             "llvm",
             "clang",
             "compiler-rt",
             "libcxx",
             "libcxxabi",
             "libunwind",
             "lld"]
 
+    if extra_configure_args:
+        extra_configure_args = ['--'] + list(extra_configure_args)
+    else:
+        extra_configure_args = []
+
     # Explicitly use '/' as separator, because it works on *nix and Windows.
     sanitizer_script_dir = "sanitizer_buildbot/sanitizers"
     sanitizer_script = "../%s/zorg/buildbot/builders/sanitizers/%s" % (sanitizer_script_dir, "buildbot_selector.py")
 
     f = LLVMBuildFactory(
         clean=clean,
         depends_on_projects=depends_on_projects,
         llvm_srcdir=sanitizer_script_dir)
 
     # Get sanitizer buildbot scripts.
     f.addGetSourcecodeForProject(
         name='update-annotate-scripts',
         project='zorg',
         src_dir=sanitizer_script_dir,
         alwaysUseLatest=True)
 
     # Run annotated command for sanitizer.
+    command = ['python', sanitizer_script] + extra_configure_args
     f.addStep(AnnotatedCommand(name="annotate",
                                description="annotate",
                                timeout=timeout,
                                haltOnFailure=True,
-                               command="python " + sanitizer_script,
+                               command=command,
                                env=merged_env))
     return f

zorg/buildbot/builders/SphinxDocsBuilder.py

def getSphinxDocsBuildFactory(
     if libcxx_html:
         depends_on_projects.append('libcxx')
         depends_on_projects.append('libcxxabi')
     if libunwind_html:
         depends_on_projects.append('libunwind')
 
     f = UnifiedTreeBuilder.getCmakeBuildFactory(
             depends_on_projects=depends_on_projects,
+            enable_runtimes=[], # Docs don't support runtimes build yet.
             llvm_srcdir=llvm_srcdir,
             obj_dir=llvm_objdir,
             extra_configure_args=cmake_args,
             **kwargs) # Pass through all the extra arguments.
 
     if llvm_html:
         f.addStep(NinjaCommand(name="docs-llvm-html",
                                haltOnFailure=True,
def getLLVMDocsBuildFactory(
     docs = [
         llvm_docs[project] for project in llvm_docs.keys()
         if project in _depends_on_projects
     ]
 
     f = UnifiedTreeBuilder.getCmakeBuildFactory(
             clean=clean,
             depends_on_projects=_depends_on_projects,
+            enable_runtimes=[], # Docs don't support runtimes build yet.
             extra_configure_args=cmake_args,
             env=merged_env,
             **kwargs) # Pass through all the extra arguments.
 
     UnifiedTreeBuilder.addNinjaSteps(
         f,
         targets=[d[0] for d in docs],
         checks=[],

zorg/buildbot/builders/UnifiedTreeBuilder.py

 from buildbot.plugins import steps, util
 from buildbot.steps.shell import SetPropertyFromCommand
 from buildbot.process.properties import WithProperties
 
 from zorg.buildbot.commands.CmakeCommand import CmakeCommand
 from zorg.buildbot.commands.NinjaCommand import NinjaCommand
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 
 from zorg.buildbot.conditions.FileConditions import FileDoesNotExist
 from zorg.buildbot.process.factory import LLVMBuildFactory
 
 import zorg.buildbot.builders.Util as builders_util
 
 def getLLVMBuildFactoryAndPrepareForSourcecodeSteps(
            depends_on_projects = None,
+           enable_runtimes = None,
            llvm_srcdir = None,
            obj_dir = None,
            install_dir = None,
            cleanBuildRequested = None,
            env = None,
            **kwargs):
 
     def cleanBuildRequestedByProperty(step):
         return step.build.getProperty("clean")
 
     if cleanBuildRequested is None:
         # We want a clean checkout only if requested by the property.
         cleanBuildRequested = cleanBuildRequestedByProperty
 
     f = LLVMBuildFactory(
             depends_on_projects=depends_on_projects,
+            enable_runtimes=enable_runtimes,
             llvm_srcdir=llvm_srcdir,
             obj_dir=obj_dir,
             install_dir=install_dir,
             cleanBuildRequested=cleanBuildRequested,
             **kwargs) # Pass through all the extra arguments.
 
     # Remove the source code for a clean checkout if requested by property.
     # TODO: Some Windows workers do not handle RemoveDirectory command well.
     # So, consider running "rmdir /S /Q <dir>" if the build runs on Windows.
     f.addStep(steps.RemoveDirectory(name='clean-src-dir',
               dir=f.monorepo_dir,
               haltOnFailure=False,
               flunkOnFailure=False,
               doStepIf=cleanBuildRequestedByProperty,
               ))
 
     return f
 
 def getLLVMBuildFactoryAndSourcecodeSteps(
            depends_on_projects = None,
+           enable_runtimes = None,
            llvm_srcdir = None,
+           src_to_build_dir = None,
            obj_dir = None,
            install_dir = None,
            cleanBuildRequested = None,
            **kwargs):
 
     f = getLLVMBuildFactoryAndPrepareForSourcecodeSteps(
             depends_on_projects=depends_on_projects,
+            enable_runtimes=enable_runtimes,
             llvm_srcdir=llvm_srcdir,
+            src_to_build_dir=src_to_build_dir,
             obj_dir=obj_dir,
             install_dir=install_dir,
             cleanBuildRequested=cleanBuildRequested,
             **kwargs) # Pass through all the extra arguments.
 
     # Get the source code.
     f.addGetSourcecodeSteps(**kwargs)
 
def addCmakeSteps(
     # So, consider running "rmdir /S /Q <dir>" if the build runs on Windows.
     f.addStep(steps.RemoveDirectory(name='clean-%s-dir' % obj_dir,
               dir=obj_dir,
               haltOnFailure=False,
               flunkOnFailure=False,
               doStepIf=cleanBuildRequested,
               ))
 
+    if f.enable_projects:
         CmakeCommand.applyDefaultOptions(cmake_args, [
-        ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)),
+            ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),
+            ])
+
+    if f.enable_runtimes:
+        CmakeCommand.applyDefaultOptions(cmake_args, [
+            ('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.enable_runtimes)),
             ])
 
     if install_dir:
         install_dir_rel = LLVMBuildFactory.pathRelativeTo(
                               install_dir,
                               obj_dir)
         CmakeCommand.applyRequiredOptions(cmake_args, [
             ('-DCMAKE_INSTALL_PREFIX=', install_dir_rel),
             ])
if install_dir:
                                description=["Install", "just", "built", "components"],
                                env=env or {},
                                workdir=obj_dir,
                                **kwargs # Pass through all the extra arguments.
                                ))
 
 def getCmakeBuildFactory(
            depends_on_projects = None,
+           enable_runtimes = None,
            llvm_srcdir = None,
            obj_dir = None,
            install_dir = None,
            clean = False,
            extra_configure_args = None,
            env = None,
            **kwargs):
 
     f = getLLVMBuildFactoryAndSourcecodeSteps(
             depends_on_projects=depends_on_projects,
+            enable_runtimes=enable_runtimes,
             llvm_srcdir=llvm_srcdir,
             obj_dir=obj_dir,
             install_dir=install_dir,
             **kwargs) # Pass through all the extra arguments.
 
     cleanBuildRequested = lambda step: step.build.getProperty("clean", default=step.build.getProperty("clean_obj")) or clean
     addCmakeSteps(
         f,
         cleanBuildRequested=cleanBuildRequested,
         obj_dir=f.obj_dir,
         install_dir=f.install_dir,
         extra_configure_args=extra_configure_args,
         env=env,
         **kwargs)
 
     return f
 
 
 def getCmakeWithNinjaBuildFactory(
            depends_on_projects = None,
+           enable_runtimes = None,
            targets = None,
            llvm_srcdir = None,
            obj_dir = None,
            checks = None,
            install_dir = None,
            clean = False,
            extra_configure_args = None,
            env = None,
def getCmakeWithNinjaBuildFactory(
 
     # Some options are required for this build no matter what.
     CmakeCommand.applyRequiredOptions(cmake_args, [
         ('-G',                      'Ninja'),
         ])
 
     f = getCmakeBuildFactory(
             depends_on_projects=depends_on_projects,
+            enable_runtimes=enable_runtimes,
             llvm_srcdir=llvm_srcdir,
             obj_dir=obj_dir,
             install_dir=install_dir,
             clean=clean,
             extra_configure_args=cmake_args,
             env=merged_env,
             **kwargs) # Pass through all the extra arguments.
 
     addNinjaSteps(
            f,
            obj_dir=f.obj_dir,
            targets=targets,
            checks=checks,
            install_dir=f.install_dir,
            env=merged_env,
            **kwargs)
 
     return f
 
 def getCmakeWithNinjaWithMSVCBuildFactory(
            depends_on_projects = None,
+           enable_runtimes = None,
            targets = None,
            llvm_srcdir = None,
            obj_dir = None,
            checks = None,
            install_dir = None,
            clean = False,
            extra_configure_args = None,
            # VS tools environment variable if using MSVC. For example,
def getCmakeWithNinjaWithMSVCBuildFactory(
     else:
         cmake_args = list()
 
     if checks is None:
         checks = ['check-all']
 
     f = getLLVMBuildFactoryAndSourcecodeSteps(
             depends_on_projects=depends_on_projects,
+            enable_runtimes=enable_runtimes,
             llvm_srcdir=llvm_srcdir,
             obj_dir=obj_dir,
             install_dir=install_dir,
             **kwargs) # Pass through all the extra arguments.
 
     f.addStep(SetPropertyFromCommand(
         command=builders_util.getVisualStudioEnvironment(vs, target_arch),
         extract_fn=builders_util.extractVSEnvironment))
addNinjaSteps(
            install_dir=f.install_dir,
            env=env,
            **kwargs)
 
     return f
 
 def getCmakeWithNinjaMultistageBuildFactory(
            depends_on_projects = None,
+           enable_runtimes = None,
            llvm_srcdir = None,
+           src_to_build_dir = None,
            obj_dir = None,
            checks = None,
            install_dir = None,
            clean = False,
            extra_configure_args = None,
            env = None,
            stages=2,
            stage_names=None,
def getCmakeWithNinjaMultistageBuildFactory(
     stage_objdirs = list()
     stage_installdirs = list()
     for s in stage_names:
         stage_objdirs.append("%s/%s" % (obj_dir, s))
         stage_installdirs.append("%s/%s" % (install_dir, s))
 
     f = getLLVMBuildFactoryAndPrepareForSourcecodeSteps(
             depends_on_projects=depends_on_projects,
+            enable_runtimes=enable_runtimes,
             llvm_srcdir=llvm_srcdir,
+            src_to_build_dir=src_to_build_dir,
             obj_dir=obj_dir,
             install_dir=install_dir,
             env=merged_env,
             stage_objdirs=stage_objdirs,
             stage_installdirs=stage_installdirs,
             stage_names=stage_names,
             **kwargs) # Pass through all the extra arguments.
 

zorg/buildbot/builders/annotated/annotated_builder.py

 from __future__ import print_function
 
 import util
 
 import argparse
 import errno
 import os
 import re
 import shutil
 import subprocess
 import sys
 
 from os.path import join as pjoin
 
 VSWHERE_PATH = "C:/Program Files (x86)/Microsoft Visual Studio/Installer/vswhere.exe"
+BUILDTOOLS_VSDEVCMD = "C:/BuildTools/Common7/Tools/VsDevCmd.bat"
 
 def get_argument_parser(*args, **kwargs):
     ap = argparse.ArgumentParser(*args, **kwargs)
     ap.add_argument('--jobs', help='Number of concurrent jobs to run')
     return ap
 
 
 class AnnotatedBuilder:
class AnnotatedBuilder:
     def stage_cmake_args(
         self,
         cmake_args,
         extra_cmake_args,
         c_compiler,
         cxx_compiler,
         linker,
         previous_stage_bin):
+        archiver = 'llvm-ar'
+        if os.name == 'nt':
+            archiver = 'llvm-lib'
         return (
             cmake_args + [
                 '-DCMAKE_AR=%s' % (
-                    util.cmake_pjoin(previous_stage_bin, 'llvm-ar'),),
+                    util.cmake_pjoin(previous_stage_bin, archiver),),
                 '-DCMAKE_RANLIB=%s' % (
                     util.cmake_pjoin(previous_stage_bin, 'llvm-ranlib'),),
             ] + self.cmake_compiler_flags(
                 c_compiler, cxx_compiler, linker, previous_stage_bin) +
             extra_cmake_args)
 
     def set_environment(self, env=None, vs_tools=None, arch=None):
         self.report_build_step('set-environment')
def run_steps(
         except Exception as e:
             self.report_step_exception(e)
             return 1
 
         return 0
 
 
 def get_vcvars(vs_tools, arch):
-    """Get the VC tools environment using vswhere.exe from VS 2017
+    """Get the VC tools environment using vswhere.exe or buildtools docker
+
+    This is intended to work either when VS is in its standard installation
+    location, or when the docker instructions have been followed, and we can
+    find Visual C++ in C:/BuildTools.
+
+    Visual Studio provides a docker image with instructions here:
+    https://docs.microsoft.com/en-us/visualstudio/install/build-tools-container?view=vs-2019
 
-    This code is following the guidelines from strategy 1 in this blog post:
+    This vswhere code is following the guidelines from strategy 1 in this blog
+    post:
         https://blogs.msdn.microsoft.com/vcblog/2017/03/06/finding-the-visual-c-compiler-tools-in-visual-studio-2017/
 
     It doesn't work when VS is not installed at the default location.
     """
     if not arch:
         # First check the wow64 processor architecture, since python is probably
         # 32-bit, then fall back to PROCESSOR_ARCHITECTURE.
         arch = os.environ.get('PROCESSOR_ARCHITEW6432', '').lower()
         if not arch:
             arch = os.environ.get('PROCESSOR_ARCHITECTURE', '').lower()
     else:
         arch = arch.lower()
 
     # Use vswhere.exe if it exists.
     if os.path.exists(VSWHERE_PATH):
         cmd = [VSWHERE_PATH, "-latest", "-property", "installationPath"]
-        vs_path = subprocess.check_output(cmd).strip()
+        vs_path = subprocess.check_output(cmd).decode(sys.stdout.encoding)
+        vs_path = vs_path.strip()
         util.report("Running vswhere to find VS: " + repr(cmd))
         util.report("vswhere output: " + vs_path)
         if not os.path.isdir(vs_path):
             raise ValueError("VS install path does not exist: " + vs_path)
         vcvars_path = pjoin(vs_path, 'VC', 'Auxiliary', 'Build',
                             'vcvarsall.bat')
+    elif os.path.exists(BUILDTOOLS_VSDEVCMD):
+        vcvars_path = BUILDTOOLS_VSDEVCMD
     elif vs_tools is None:
         vs_tools = os.path.expandvars('%VS140COMNTOOLS%')
         vcvars_path = pjoin(vs_tools, '..', '..', 'VC', 'vcvarsall.bat')
 
     # Newer vcvarsall.bat scripts aren't quiet, so direct them to NUL, aka
     # Windows /dev/null.
     cmd = util.shquote_cmd([vcvars_path, arch]) + ' > NUL && set'
     util.report("Running vcvars: " + cmd)
-    output = subprocess.check_output(cmd, shell=True)
+    output = \
+        subprocess.check_output(cmd, shell=True).decode(sys.stdout.encoding)
     new_env = {}
     for line in output.splitlines():
         var, val = line.split('=', 1)
         new_env[var] = val
     return new_env
 
 
 def main(argv):
     ap = get_argument_parser()
     args = ap.parse_args(argv[1:])
     builder = AnnotatedBuilder()
     builder.run_steps(jobs=args.jobs)
 
 
 if __name__ == '__main__':
     sys.exit(main(sys.argv))

zorg/buildbot/builders/annotated/buildbot-helper.sh

+#!/bin/bash
+
+# Buildbot Helper functions
+
+# Stop if we've encountered an error.
+halt_on_failure() {
+  echo "@@@HALT_ON_FAILURE@@@"
+}
+
+build_step() {
+  echo  "@@@BUILD_STEP ${@}@@@"
+}

zorg/buildbot/builders/annotated/hip-build.sh

+#!/bin/bash
+
+# Enable Error tracing
+set -o errtrace
+
+# Print trace for all commands ran before execution
+set -x
+
+# Include the Buildbot helper functions
+HERE="$(dirname $0)"
+. ${HERE}/buildbot-helper.sh
+
+# Ensure all commands pass, and not dereferencing unset variables.
+set -eu
+halt_on_failure
+
+BUILDBOT_ROOT=/buildbot
+LLVM_ROOT="${BUILDBOT_ROOT}/llvm-project"
+REVISION=${BUILDBOT_REVISION:-origin/main}
+AMDGPU_ARCHS=${AMDGPU_ARCHS:="gfx900;gfx906;gfx908;gfx1030"}
+
+# Set-up llvm-project
+if [ ! -d "${LLVM_ROOT}" ]; then
+  build_step "Cloning llvm-project repo"
+  git clone --progress https://github.com/llvm/llvm-project.git ${LLVM_ROOT}
+fi
+
+build_step "Updating llvm-project repo"
+git -C "${LLVM_ROOT}" fetch origin
+git -C "${LLVM_ROOT}" reset --hard ${REVISION}
+
+# Set-up llvm-test-suite
+TESTSUITE_ROOT="${BUILDBOT_ROOT}/llvm-test-suite"
+if [ ! -d "${TESTSUITE_ROOT}" ]; then
+  build_step "Cloning llvm-test-suite repo"
+  git clone --progress https://github.com/llvm/llvm-test-suite.git ${TESTSUITE_ROOT}
+fi
+
+build_step "Updating llvm-test-suite repo"
+git -C "${TESTSUITE_ROOT}" fetch origin
+git -C "${TESTSUITE_ROOT}" reset --hard origin/main
+
+# Set-up variables
+BUILDBOT_SLAVENAME=$(whoami)
+BUILD_DIR="${BUILDBOT_ROOT}/${BUILDBOT_SLAVENAME}/${BUILDBOT_BUILDERNAME}"
+DESTDIR=${BUILD_DIR}/install
+EXTERNAL_DIR=/buildbot/Externals
+
+build_step "Setting up the buildbot"
+echo "BUILDBOT_ROOT=${BUILDBOT_ROOT}"
+echo "LLVM_ROOT=${LLVM_ROOT}"
+echo "BUILD_DIR=${BUILD_DIR}"
+echo "DESTDIR=${DESTDIR}"
+echo "EXTERNAL_DIR=${EXTERNAL_DIR}"
+
+# Start building LLVM, Clang, Lld, clang-tools-extra, compiler-rt
+build_step "Configure LLVM Build"
+LLVM_BUILD_DIR="${BUILD_DIR}/llvm"
+mkdir -p "${LLVM_BUILD_DIR}"
+cd "${LLVM_BUILD_DIR}"
+cmake -G Ninja \
+  -DCMAKE_BUILD_TYPE="Release" \
+  -DCMAKE_VERBOSE_MAKEFILE=1 \
+  -DLLVM_TARGETS_TO_BUILD="AMDGPU;X86" \
+  -DLLVM_ENABLE_PROJECTS="clang;lld;clang-tools-extra;compiler-rt" \
+  -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \
+  -DLIBCXX_ENABLE_SHARED=OFF \
+  -DLIBCXX_ENABLE_STATIC=ON \
+  -DLIBCXX_INSTALL_LIBRARY=OFF \
+  -DLIBCXX_INSTALL_HEADERS=OFF \
+  -DLIBCXXABI_ENABLE_SHARED=OFF \
+  -DLIBCXXABI_ENABLE_STATIC=ON \
+  -DLIBCXXABI_INSTALL_STATIC_LIBRARY=OFF \
+  -DCMAKE_INSTALL_PREFIX="${DESTDIR}" \
+  -DLLVM_ENABLE_ASSERTIONS=ON \
+  -DLLVM_ENABLE_Z3_SOLVER=OFF \
+  -DLLVM_ENABLE_ZLIB=ON \
+  -DLLVM_LIT_ARGS="-v -vv" \
+  ${LLVM_ROOT}/llvm
+
+build_step "Building LLVM"
+ninja runtimes
+ninja
+
+build_step "Install LLVM"
+rm -rf "${DESTDIR}"
+ninja install-runtimes
+ninja install
+
+# Start building llvm-test-suite's hip tests
+build_step "Configuring HIP test-suite"
+TEST_BUILD_DIR=${BUILD_DIR}/test-suite-build
+rm -rf ${TEST_BUILD_DIR}
+mkdir -p ${TEST_BUILD_DIR}
+cd ${TEST_BUILD_DIR}
+PATH="${LLVM_BUILD_DIR}/bin:$PATH" cmake -G Ninja \
+  -DTEST_SUITE_SUBDIRS=External \
+  -DTEST_SUITE_EXTERNALS_DIR=${EXTERNAL_DIR}/ \
+  -DTEST_SUITE_COLLECT_CODE_SIZE=OFF \
+  -DTEST_SUITE_COLLECT_COMPILE_TIME=OFF \
+  -DAMDGPU_ARCHS="${AMDGPU_ARCHS}" \
+  -DCMAKE_CXX_COMPILER="${LLVM_BUILD_DIR}/bin/clang++" \
+  -DCMAKE_C_COMPILER="${LLVM_BUILD_DIR}/bin/clang" \
+  -DCMAKE_VERBOSE_MAKEFILE=ON \
+  ${TESTSUITE_ROOT}
+
+build_step "Building HIP test-suite"
+ninja hip-tests-simple
+
+build_step "Testing HIP test-suite"
+ninja check-hip-simple
+
+exit 0
+

zorg/buildbot/builders/annotated/libc-linux.py

 #!/usr/bin/python
 
 import argparse
 import os
 import subprocess
 import sys
 import traceback
 import util
 from contextlib import contextmanager
 
 
+def is_fullbuild_builder(builder_name):
+    return ('fullbuild' in builder_name.split('-'))
+
+
 def main(argv):
     ap = argparse.ArgumentParser()
     ap.add_argument('--asan', action='store_true', default=False,
                     help='Build with address sanitizer enabled.')
     ap.add_argument('--debug', action='store_true', default=False,
                     help='Build in debug mode.')
     args, _ = ap.parse_known_args()
 
     source_dir = os.path.join('..', 'llvm-project')
+    fullbuild = is_fullbuild_builder(os.environ.get('BUILDBOT_BUILDERNAME'))
 
     with step('cmake', halt_on_fail=True):
         projects = ['llvm', 'libc', 'clang', 'clang-tools-extra']
 
-        cmake_args = ['-GNinja']
+        # On most systems the default generator is make and the default
+        # compilers are gcc and g++. We make it explicit here that we want
+        # clang and ninja which reduces one step of setting environment
+        # variables when setting up workers.
+        cmake_args = ['-GNinja',
+                      '-DCMAKE_C_COMPILER=clang',
+                      '-DCMAKE_CXX_COMPILER=clang++']
         if args.debug:
             cmake_args.append('-DCMAKE_BUILD_TYPE=Debug')
         else:
             cmake_args.append('-DCMAKE_BUILD_TYPE=Release')
 
         if args.asan:
             cmake_args.append('-DLLVM_USE_SANITIZER=Address')
 
+        if fullbuild and not args.asan:
+            projects.append('compiler-rt')
+
         cmake_args.append('-DLLVM_ENABLE_PROJECTS={}'.format(';'.join(projects)))
 
+        if fullbuild and not args.asan:
+            cmake_args.append('-DLLVM_LIBC_INCLUDE_SCUDO=ON')
+
+        if fullbuild:
+            cmake_args.extend(['-DLLVM_LIBC_FULL_BUILD=ON',
+                               '-DLLVM_LIBC_ENABLE_LINTING=ON'])
+
         run_command(['cmake', os.path.join(source_dir, 'llvm')] + cmake_args)
 
     with step('build llvmlibc', halt_on_fail=True):
         run_command(['ninja', 'llvmlibc'])
 
     with step('check-libc'):
         run_command(['ninja', 'check-libc'])
 
-    if not args.asan:
-        with step('Loader Tests'):
+    if fullbuild and not args.asan:
+        with step('libc-loader-tests'):
             run_command(['ninja', 'libc_loader_tests'])
-        with step('Integration Tests'):
+        with step('libc-integration-test'):
             run_command(['ninja', 'libc-integration-test'])
+        with step('libc-fuzzer'):
+            run_command(['ninja', 'libc-fuzzer'])
+        with step('libc-scudo-integration-test'):
+            run_command(['ninja', 'libc-scudo-integration-test'])
         with step('AOR Tests'):
             aor_dir = os.path.join(source_dir, 'libc', 'AOR_v20.02')
             # Remove the AOR build dir.
             util.clean_dir(os.path.join(aor_dir, 'build'))
             run_command(['make', 'check'], directory=aor_dir)
-
-    if not args.debug:
         with step('Benchmark Utils Tests'):
             run_command(['ninja', 'libc-benchmark-util-tests'])
 
 
 @contextmanager
 def step(step_name, halt_on_fail=False):
     util.report('@@@BUILD_STEP {}@@@'.format(step_name))
     if halt_on_fail:

zorg/buildbot/builders/annotated/libc-windows.py

-#!/usr/bin/python
+
+import annotated_builder
+import util
 
 import argparse
 import os
 import subprocess
 import sys
 import traceback
-import util
 from contextlib import contextmanager
 
-
 def main(argv):
     ap = argparse.ArgumentParser()
     ap.add_argument('--asan', action='store_true', default=False,
                     help='Build with address sanitizer enabled.')
     ap.add_argument('--debug', action='store_true', default=False,
                     help='Build in debug mode.')
     args, _ = ap.parse_known_args()
 
     source_dir = os.path.join('..', 'llvm-project')
+    vc_vars = annotated_builder.get_vcvars(vs_tools=None, arch='amd64')
+    for (var, val) in vc_vars.items():
+        os.environ[var] = val
 
     with step('cmake', halt_on_fail=True):
-        projects = ['llvm', 'libc', 'clang', 'clang-tools-extra']
-
-        cmake_args = ['-GNinja']
+        # On most systems the default generator is make and the default
+        # compilers are gcc and g++. We make it explicit here that we want
+        # clang and ninja which reduces one step of setting environment
+        # variables when setting up workers.
+        ## adjust compiler location definitions based on VM config
+        cmake_args = ['-GNinja',
+                      '-DCMAKE_C_COMPILER=clang-cl',
+                      '-DCMAKE_CXX_COMPILER=clang-cl']
         if args.debug:
             cmake_args.append('-DCMAKE_BUILD_TYPE=Debug')
         else:
             cmake_args.append('-DCMAKE_BUILD_TYPE=Release')
 
         if args.asan:
             cmake_args.append('-DLLVM_USE_SANITIZER=Address')
 
-        cmake_args.append('-DLLVM_ENABLE_PROJECTS={}'.format(';'.join(projects)))
+        cmake_args.append('-DLLVM_ENABLE_PROJECTS=libc')
+        cmake_args.append('-DLLVM_TARGETS_TO_BUILD=X86')
+        cmake_args.append('-DLLVM_FORCE_BUILD_RUNTIME=libc')
+        cmake_args.append('-DLLVM_NATIVE_ARCH=x86_64')
+        cmake_args.append('-DLLVM_HOST_TRIPLE=x86_64-window-x86-gnu')
+        cmake_args.append('-DLLVM_LIBC_MPFR_INSTALL_PATH=C:/src/install')
 
         run_command(['cmake', os.path.join(source_dir, 'llvm')] + cmake_args)
 
     with step('build llvmlibc', halt_on_fail=True):
         run_command(['ninja', 'llvmlibc'])
 
     with step('check-libc'):
         run_command(['ninja', 'check-libc'])
 
-    if not args.asan:
-        with step('Loader Tests'):
-            run_command(['ninja', 'libc_loader_tests'])
-        with step('Integration Tests'):
-            run_command(['ninja', 'libc-integration-test'])
-        with step('AOR Tests'):
-            aor_dir = os.path.join(source_dir, 'libc', 'AOR_v20.02')
-            # Remove the AOR build dir.
-            util.clean_dir(os.path.join(aor_dir, 'build'))
-            run_command(['make', 'check'], directory=aor_dir)
-
-    if not args.debug:
-        with step('Benchmark Utils Tests'):
-            run_command(['ninja', 'libc-benchmark-util-tests'])
-
 
 @contextmanager
 def step(step_name, halt_on_fail=False):
     util.report('@@@BUILD_STEP {}@@@'.format(step_name))
     if halt_on_fail:
         util.report('@@@HALT_ON_FAILURE@@@')
     try:
         yield

zorg/buildbot/builders/sanitizers/buildbot_android_functions.sh

 #!/usr/bin/env bash
 
-CLOBBER="android_ndk android-ndk-* platform-tools *.zip shards_* test_android_* tested_arch_*"
-STAGE2_CLOBBER="compiler_rt_build_android_* llvm_build_android_*"
-STAGE1_CLOBBER="llvm_build64 ${STAGE2_CLOBBER}"
-
 ANDROID_NDK_VERSION=21
 ANDROID_API=24
 NDK_DIR=android_ndk
 
 function download_android_tools {
   local VERSION=android-ndk-r${ANDROID_NDK_VERSION}d
   local FILE_NAME=${VERSION}-linux-x86_64.zip
   local NDK_URL=https://dl.google.com/android/repository/${FILE_NAME}
   if  [[ "$(cat ${NDK_DIR}/android_ndk_url)" != ${NDK_URL} ]] ; then
     echo @@@BUILD_STEP downloading Android NDK@@@
     [[ -d ${NDK_DIR} ]] && rm -rf ${NDK_DIR}
     [[ -d ${VERSION} ]] && rm -rf ${VERSION}
     [[ -f ${FILE_NAME} ]] && rm -f ${FILE_NAME}
-    wget ${NDK_URL}
+    wget -e dotbytes=10m ${NDK_URL}
     unzip ${FILE_NAME} > /dev/null
     mv ${VERSION} ${NDK_DIR}
     echo ${NDK_URL} > ${NDK_DIR}/android_ndk_url
   fi
 
   if  [[ ! -d platform-tools ]] ; then
     echo @@@BUILD_STEP downloading Android Platform Tools@@@
     local FILE_NAME=platform-tools-latest-linux.zip
     [[ -f ${FILE_NAME} ]] && rm -f ${FILE_NAME}
-    wget https://dl.google.com/android/repository/${FILE_NAME}
+    wget -e dotbytes=10m https://dl.google.com/android/repository/${FILE_NAME}
     unzip ${FILE_NAME} > /dev/null
   fi
   export PATH=$ROOT/platform-tools/:$PATH
 }
 
 function build_stage2_android() {
   # Build self-hosted tree with fresh Clang and -Werror.
   local CMAKE_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_WERROR=ON ${STAGE1_AS_COMPILER} -DCMAKE_C_FLAGS=-gmlt -DCMAKE_CXX_FLAGS=-gmlt"
-  CMAKE_OPTIONS="${CMAKE_OPTIONS} -DLLVM_ENABLE_PROJECTS='clang;lld' -DLLVM_USE_LINKER=lld"
+  CMAKE_OPTIONS="${CMAKE_OPTIONS} -DLLVM_ENABLE_PROJECTS='clang;lld' -DLLVM_USE_LINKER=lld -DCLANG_DEFAULT_RTLIB=libgcc"
 
   if ccache -s ; then
     CMAKE_OPTIONS="${CMAKE_OPTIONS} -DLLVM_CCACHE_BUILD=ON"
-    rm_dirs llvm_build64
   fi
 
   echo @@@BUILD_STEP bootstrap clang@@@
-  rm_dirs ${STAGE2_CLOBBER}
-
   mkdir -p llvm_build64
   if  [[ "$(cat llvm_build64/CMAKE_OPTIONS)" != "${CMAKE_OPTIONS}" ]] ; then
     (cd llvm_build64 && cmake ${CMAKE_OPTIONS} -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON $LLVM && \
        echo ${CMAKE_OPTIONS} > CMAKE_OPTIONS) || echo @@@STEP_FAILURE@@@
   fi
   ninja -C llvm_build64 || (echo @@@STEP_FAILURE@@@ && exit 2)
 }
 
 function configure_android { # ARCH triple
   local _arch=$1
   local _triple=$2
 
   ANDROID_TOOLCHAIN=$ROOT/$NDK_DIR/toolchains/llvm/prebuilt/linux-x86_64
   echo "Building for Android API level $ANDROID_API"
 
   local ANDROID_LIBRARY_OUTPUT_DIR=$(ls -d $ROOT/llvm_build64/lib/clang/* | tail -1)
   local ANDROID_EXEC_OUTPUT_DIR=$ROOT/llvm_build64/bin
-  local ANDROID_FLAGS="--target=${_triple}${ANDROID_API} --sysroot=$ANDROID_TOOLCHAIN/sysroot -gcc-toolchain $ANDROID_TOOLCHAIN  -B$ANDROID_TOOLCHAIN"
+  local ANDROID_FLAGS="--target=${_triple}${ANDROID_API} --sysroot=$ANDROID_TOOLCHAIN/sysroot --gcc-toolchain=$ANDROID_TOOLCHAIN  -B$ANDROID_TOOLCHAIN"
   local ANDROID_CXX_FLAGS="$ANDROID_FLAGS -stdlib=libc++ -I/$ANDROID_TOOLCHAIN/sysroot/usr/include/c++/v1"
 
   local CLANG_PATH=$ROOT/llvm_build64/bin/clang
   local CLANGXX_PATH=$ROOT/llvm_build64/bin/clang++
 
   # Always clobber android build tree.
   # It has a hidden dependency on clang (through CXX) which is not known to
   # the build system.
if ! ninja -C llvm_build_android_$_arch llvm-symbolizer ; then
     echo @@@STEP_FAILURE@@@
   fi
   if ! ninja -C compiler_rt_build_android_$_arch ; then
     BUILD_RT_ERR="${BUILD_RT_ERR}|${_arch}|"
     echo @@@STEP_FAILURE@@@
   fi
 }
 
-# If a multiarch device has x86 as the first arch, remove everything else from
-# the list. This captures cases like [x86,armeabi-v7a], where the arm part is
-# software emulation and incompatible with ASan.
+# If a multiarch device's first arch starts with x86, remove archs that don't
+# start with x86. Handle cases like:
+#  - x86,armeabi-v7a
+#  - x86_64,x86,arm64-v8a,armeabi-v7a,armeabi
+#  - x86_64,arm64-v8a,x86,armeabi-v7a,armeabi
+# The arm part is software emulation and incompatible with ASan.
 function patch_abilist { # IN OUT
   local _abilist=$1
   local _out=$2
-  if [[ "$_abilist" == "x86,"* ]]; then
-    _abilist="x86"
+  if [[ "$_abilist" == "x86"* ]]; then
+    _abilist=$(echo $_abilist | tr , '\n' | grep '^x86')
+    _abilist=$(echo $_abilist | tr ' ' ,)
   fi
   eval $_out="'$_abilist'"
 }
 
 function restart_adb_server {
   ADB=adb
   echo @@@BUILD_STEP restart adb server@@@
   $ADB kill-server
   sleep 2
   $ADB start-server
   sleep 2
 }
 
 function test_on_device {
   local _serial=$1
   shift
 
   ABILIST=$(${ADB} -s $_serial shell getprop ro.product.cpu.abilist)
-  patch_abilist $ABILIST ABILIST
+  patch_abilist "$ABILIST" ABILIST
   for _arg in "$@"; do
     local _arch=${_arg%:*}
     local _abi=${_arg#*:}
     if [[ $BUILD_RT_ERR == *"|${_arch}|"* ]]; then
       echo "@@@STEP_FAILURE@@ skipping tests on ${_arch}"
       continue
     fi
     if [[ $ABILIST == *"$_abi"* ]]; then
       echo "$_serial" >> tested_arch_$_arch
       BUILD_ID=$(${ADB} -s $_serial shell getprop ro.build.id | tr -d '\r')
       BUILD_FLAVOR=$(${ADB} -s $_serial shell getprop ro.build.flavor | tr -d '\r')
+      (
+        # Test only one arch at a time to avoid simultaneously writes into the
+        # same compiler-rt build dir.
+        flock -x $lock_fd
         test_arch_on_device "$_arch" "$_serial" "$BUILD_ID" "$BUILD_FLAVOR"
+      ) {lock_fd}>$ROOT/${_arch}.lock
     fi
   done
 }
 
 function tail_pids {
   for LOG_PID in $1; do
     PID=${LOG_PID#*,}
     LOG=${LOG_PID%,*}
     tail -n +1 -F $LOG --pid=$PID
   done
   wait
 }
 
 function test_android {
-  if [[ "${BUILDBOT_SLAVENAME:-}" != "" ]]; then
+  if [[ -v BUILDBOT_BUILDERNAME ]]; then
     restart_adb_server
   fi
 
   ADB=adb
   echo @@@BUILD_STEP run all tests@@@
   ANDROID_DEVICES=$(${ADB} devices | grep 'device$' | sort -r | awk '{print $1}')
 
   rm -rf test_android_log_*
   rm -rf tested_arch_*
   rm -rf shards_*
   LOGS=
   for SERIAL in $ANDROID_DEVICES; do
     LOG="$(mktemp test_android_log_XXXX)"
     (test_on_device "$SERIAL" $@ 2>&1 >"$LOG") &
     LOGS="$LOGS $LOG,$!"
   done
   tail_pids "$LOGS"
 
+  # Return to avoid exception if we already have error.
+  [[ $BUILD_RT_ERR == "" ]] || return
+
   for _arg in "$@"; do
     local _arch=${_arg%:*}
     if [[ ! -f tested_arch_$_arch ]]; then
       echo @@@BUILD_STEP unavailable device android/$_arch@@@
       echo @@@STEP_EXCEPTION@@@
     fi
   done
 }
function run_tests_sharded {
   local _env=$3
 
   local NUM_SHARDS=4
   local _log_prefix=$(mktemp shards_XXXX_)
   echo @@@BUILD_STEP run $_test_name tests [$DEVICE_DESCRIPTION]@@@
   LOGS=
   for ((SHARD=0; SHARD < $NUM_SHARDS; SHARD++)); do
     LOG=${_log_prefix}_$SHARD
-    local ENV="$_env GTEST_TOTAL_SHARDS=$NUM_SHARDS GTEST_SHARD_INDEX=$SHARD LD_LIBRARY_PATH=$DEVICE_ROOT"
+    # 'adb shell <command>' on Fugu is missing TMPDIR.
+    local ENV="$_env GTEST_TOTAL_SHARDS=$NUM_SHARDS GTEST_SHARD_INDEX=$SHARD LD_LIBRARY_PATH=$DEVICE_ROOT TMPDIR=$DEVICE_TMPDIR"
     ( (run_command_on_device "$ENV $DEVICE_ROOT/$_test" || echo @@@STEP_FAILURE@@@) 2>&1 >${_log_prefix}_$SHARD ) &
     LOGS="$LOGS $LOG,$!"
   done
   tail_pids "$LOGS" || true
 }
 
 function test_arch_on_device {
   local _arch=$1
   local _serial=$2
   local _build_id=$3
   local _build_flavor=$4
 
   export DEVICE_DESCRIPTION=$_arch/$_build_flavor/$_build_id
 
   local LIBCXX_SHARED=$ANDROID_TOOLCHAIN/sysroot/usr/lib/${_arch}-linux-androideabi/libc++_shared.so
   if [[ ! -f $LIBCXX_SHARED ]] ; then
     local LIBCXX_SHARED=$ANDROID_TOOLCHAIN/sysroot/usr/lib/${_arch}-linux-android/libc++_shared.so
   fi
   local SYMBOLIZER_BIN=$ROOT/llvm_build_android_$_arch/bin/llvm-symbolizer
   local RT_DIR=$($ROOT/llvm_build64/bin/clang -print-resource-dir)/lib/linux
   local COMPILER_RT_BUILD_DIR=$ROOT/compiler_rt_build_android_$_arch
   export ADB=adb
+  export DEVICE_TMPDIR=/data/local/tmp
   export DEVICE_ROOT=/data/local/tmp/Output
   export ANDROID_SERIAL=$_serial
   echo "Serial $_serial"
 
   echo @@@BUILD_STEP device setup [$DEVICE_DESCRIPTION]@@@
   $ADB wait-for-device
   $ADB devices
 

zorg/buildbot/builders/sanitizers/buildbot_bootstrap_asan.sh

 #!/usr/bin/env bash
 
 set -x
 set -e
 set -u
 
 HERE="$(cd $(dirname $0) && pwd)"
 . ${HERE}/buildbot_functions.sh
 
 ROOT=`pwd`
 PLATFORM=`uname`
 export PATH="/usr/local/bin:$PATH"
 
 LLVM=$ROOT/llvm
-CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF"
 
 clobber
 
 buildbot_update
 
 # Stage 1
 
 build_stage1_clang
 
 # Stage 2 / AddressSanitizer
 
 build_stage2_asan
 
 check_stage2_asan
 
 # Stage 3 / AddressSanitizer
 
 export ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1"
 build_stage3_asan
 
 check_stage3_asan
 
+cleanup $STAGE1_DIR

zorg/buildbot/builders/sanitizers/buildbot_bootstrap_msan.sh

 #!/usr/bin/env bash
 
 set -x
 set -e
 set -u
 
 HERE="$(cd $(dirname $0) && pwd)"
 . ${HERE}/buildbot_functions.sh
 
 ROOT=`pwd`
 PLATFORM=`uname`
 export PATH="/usr/local/bin:$PATH"
 
 LLVM=$ROOT/llvm
-CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF"
 
 clobber
 
 buildbot_update
 
 # Stage 1
 
 build_stage1_clang
 
 # Stage 2 / Memory Sanitizer
 
 build_stage2_msan
 
 check_stage2_msan
 
 # Stage 3 / MemorySanitizer
 
 build_stage3_msan
 
 check_stage3_msan
+
+cleanup $STAGE1_DIR

zorg/buildbot/builders/sanitizers/buildbot_bootstrap_ubsan.sh

 #!/usr/bin/env bash
 
 set -x
 set -e
 set -u
 
 HERE="$(cd $(dirname $0) && pwd)"
 . ${HERE}/buildbot_functions.sh
 
 ROOT=`pwd`
 PLATFORM=`uname`
 export PATH="/usr/local/bin:$PATH"
 
 LLVM=$ROOT/llvm
-CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF"
 
 clobber
 
 buildbot_update
 
 # Stage 1
 
 build_stage1_clang
 
 # Stage 2 / UndefinedBehaviorSanitizer
 
 build_stage2_ubsan
 
 check_stage2_ubsan
 
 # Stage 3 / UndefinedBehaviorSanitizer
 
 build_stage3_ubsan
 
 check_stage3_ubsan
+
+cleanup $STAGE1_DIR

zorg/buildbot/builders/sanitizers/buildbot_cmake.sh

 #!/usr/bin/env bash
 
 set -x
 set -e
 set -u
 
 HERE="$(dirname $0)"
 . ${HERE}/buildbot_functions.sh
 
 ROOT=`pwd`
 PLATFORM=`uname`
 ARCH=`uname -m`
 export PATH="/usr/local/bin:$PATH"
 export ANDROID_SDK_HOME=$ROOT/../../..
 
-# Always clobber bootstrap build trees.
-rm -rf compiler_rt_build llvm_build* symbolizer_build*
+CMAKE_ARGS=""
+for arg in "$@"
+do
+    case $arg in
+        --CMAKE_ARGS=*)
+        CMAKE_ARGS="${arg#*=}"
+    esac
+done
 
 USE_CCACHE=
 if ccache -s ; then
   USE_CCACHE="-DLLVM_CCACHE_BUILD=ON"
-  rm -rf clang_build
 fi
 
+# Always clobber bootstrap build trees.
+clobber
+
 SUPPORTS_32_BITS=${SUPPORTS_32_BITS:-1}
 MAKE_JOBS=${MAX_MAKE_JOBS:-$(nproc)}
 LLVM=$ROOT/llvm
 ZLIB=$ROOT/zlib
 
-CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS:-} -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_PARALLEL_LINK_JOBS=10"
+CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS:-} -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_PARALLEL_LINK_JOBS=10 -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF ${CMAKE_ARGS}"
 ENABLE_LIBCXX_FLAG=
 if [ "$PLATFORM" == "Darwin" ]; then
   CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS} -DPYTHON_EXECUTABLE=/usr/bin/python"
   ENABLE_LIBCXX_FLAG="-DLLVM_ENABLE_LIBCXX=ON"
 fi
 
 if [ -e /usr/include/plugin-api.h ]; then
   CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_BINUTILS_INCDIR=/usr/include"

 fi
 CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_PROJECTS='${PROJECTS}'"
 
 if [[ "$CHECK_CFI" == "1" ]]; then
   # We need this after https://reviews.llvm.org/D62050
   CMAKE_COMMON_OPTIONS="$CMAKE_COMMON_OPTIONS -DLLVM_BUILD_LLVM_DYLIB=ON"
 fi
 
-CLOBBER="zlib clang_build"
-clobber
 
 buildbot_update
 
 COMPILER_RT=$LLVM/../compiler-rt
 LIBCXX=$LLVM/../libcxx
 
-echo @@@BUILD_STEP lint@@@
-CHECK_LINT=${COMPILER_RT}/lib/sanitizer_common/scripts/check_lint.sh
-${CHECK_LINT} || echo @@@STEP_WARNINGS@@@
-
 # Use both gcc and just-built Clang/LLD as a host compiler/linker for sanitizer
 # tests. Assume that self-hosted build tree should compile with -Werror.
 echo @@@BUILD_STEP build fresh toolchain@@@
-if [ ! -d clang_build ]; then
-  mkdir clang_build
-fi
+mkdir -p clang_build
 (cd clang_build && cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ${CMAKE_COMMON_OPTIONS} ${USE_CCACHE} $LLVM ) || (rm -rf clang_build ; echo @@@STEP_FAILURE@@@)
 
 BOOTSTRAP_BUILD_TARGETS="clang"
 if [[ "$CHECK_LLD" != "0" ]]; then
   BOOTSTRAP_BUILD_TARGETS="$BOOTSTRAP_BUILD_TARGETS lld"
 fi
 (cd clang_build && make $BOOTSTRAP_BUILD_TARGETS -j$MAKE_JOBS) || (echo @@@STEP_FAILURE@@@ ; exit 1) || echo @@@STEP_FAILURE@@@
 
if [ "$PLATFORM" == "Linux" ]; then
   check_in_gcc $CHECK_HWASAN hwasan
   check_in_gcc $CHECK_CFI cfi-and-supported
   check_in_gcc $CHECK_DFSAN dfsan
   check_in_gcc $CHECK_LSAN lsan
   check_in_gcc $CHECK_MSAN msan
   check_in_gcc $CHECK_SCUDO scudo
   check_in_gcc $CHECK_SCUDO_STANDALONE scudo_standalone
   LDFLAGS=-no-pie check_in_gcc $CHECK_TSAN tsan
+  LDFLAGS=-no-pie check_in_gcc $CHECK_TSAN tsan-shared
   check_in_gcc $CHECK_UBSAN ubsan
   check_in_gcc $CHECK_UBSAN ubsan-minimal
 fi
 
 ### From now on we use just-built Clang as a host compiler ###
 CLANG_PATH=${ROOT}/clang_build/bin
 # Build self-hosted tree with fresh Clang and -Werror.
 CMAKE_CLANG_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_WERROR=ON -DCMAKE_C_COMPILER=${CLANG_PATH}/clang -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++ -DCMAKE_C_FLAGS=-gmlt -DCMAKE_CXX_FLAGS=-gmlt"
 BUILD_TYPE=Release
 
 echo @@@BUILD_STEP bootstrap clang@@@
-if [ ! -d llvm_build64 ]; then
-  mkdir llvm_build64
-fi
+mkdir -p llvm_build64
 (cd llvm_build64 && cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE \
     ${CMAKE_CLANG_OPTIONS} -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \
     ${ENABLE_LIBCXX_FLAG} $LLVM) || echo @@@STEP_FAILURE@@@
 
 # First, build only Clang.
 (cd llvm_build64 && make -j$MAKE_JOBS clang) || echo @@@STEP_FAILURE@@@
 
 # If needed, install the headers to clang_build/include.
if [ "$PLATFORM" == "Linux" ]; then
   check_64bit $CHECK_ASAN asan-dynamic
   check_64bit $CHECK_HWASAN hwasan
   check_64bit $CHECK_CFI cfi-and-supported
   check_64bit $CHECK_DFSAN dfsan
   check_64bit $CHECK_LSAN lsan
   check_64bit $CHECK_MSAN msan
   # check_64bit $CHECK_SCUDO scudo, No check-scudo target for this config
   check_64bit $CHECK_TSAN tsan
+  check_64bit $CHECK_TSAN tsan-shared
   check_64bit $CHECK_UBSAN ubsan
   check_64bit $CHECK_UBSAN ubsan-minimal
 fi
 
 FRESH_CLANG_PATH=${ROOT}/llvm_build64/bin
 
 echo @@@BUILD_STEP build standalone compiler-rt@@@
 if [ ! -d compiler_rt_build ]; then
if [ "$PLATFORM" == "Linux" -a $HAVE_NINJA == 1 ]; then
   check_ninja $CHECK_HWASAN hwasan
   check_ninja $CHECK_CFI cfi-and-supported
   check_ninja $CHECK_DFSAN dfsan
   check_ninja $CHECK_LSAN lsan
   check_ninja $CHECK_MSAN msan
   check_ninja $CHECK_SCUDO scudo
   check_ninja $CHECK_SCUDO_STANDALONE scudo_standalone
   check_ninja $CHECK_TSAN tsan
+  check_ninja $CHECK_TSAN tsan-shared
   check_ninja $CHECK_UBSAN ubsan
   check_ninja $CHECK_UBSAN ubsan-minimal
 
   if [ "$CHECK_SYMBOLIZER" == "1" ]; then
     build_symbolizer 32 llvm_build_ninja
     build_symbolizer 64 llvm_build_ninja
 
     check_ninja_with_symbolizer() {
if [ "$CHECK_SYMBOLIZER" == "1" ]; then
     check_ninja_with_symbolizer $CHECK_HWASAN hwasan
     check_ninja_with_symbolizer $CHECK_CFI cfi-and-supported
     check_ninja_with_symbolizer $CHECK_DFSAN dfsan
     check_ninja_with_symbolizer $CHECK_LSAN lsan
     check_ninja_with_symbolizer $CHECK_MSAN msan
     check_ninja_with_symbolizer $CHECK_SCUDO scudo
     check_ninja_with_symbolizer $CHECK_SCUDO_STANDALONE scudo_standalone
     check_ninja_with_symbolizer $CHECK_TSAN tsan
+    check_ninja_with_symbolizer $CHECK_TSAN tsan-shared
     check_ninja_with_symbolizer $CHECK_UBSAN ubsan
   fi
 fi
+
+cleanup

zorg/buildbot/builders/sanitizers/buildbot_fast.sh

 #!/usr/bin/env bash
 
 set -x
 set -e
 set -u
 
 HERE="$(cd $(dirname $0) && pwd)"
 . ${HERE}/buildbot_functions.sh
 
 ROOT=`pwd`
 PLATFORM=`uname`
 export PATH="/usr/local/bin:$PATH"
 
 LLVM=$ROOT/llvm
-CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF"
 
 clobber
 
 # Stage 1
 
 build_stage1_clang_at_revison
 
 if ccache -s ; then

 
 check_stage2_asan
 
 # Stage 2 / UndefinedBehaviorSanitizer
 
 build_stage2_ubsan
 
 check_stage2_ubsan
+
+cleanup

zorg/buildbot/builders/sanitizers/buildbot_functions.sh

(
   echo
   echo "How to reproduce locally: https://github.com/google/sanitizers/wiki/SanitizerBotReproduceBuild"
   echo
   exit 0
 )
 echo @@@BUILD_STEP Prepare@@@
 
 BUILDBOT_CLOBBER="${BUILDBOT_CLOBBER:-}"
-BUILDBOT_REVISION="${BUILDBOT_REVISION:-origin/master}"
+BUILDBOT_REVISION="${BUILDBOT_REVISION:-origin/main}"
 
 function rm_dirs {
   while ! rm -rf $@ ; do sleep 1; done
 }
 
-function stage1_clobber {
-  rm_dirs llvm_build2_* llvm_build_* libcxx_build_* ${STAGE1_CLOBBER:-}
+function cleanup() {
+  [[ -v BUILDBOT_BUILDERNAME ]] || return 0
+  echo @@@BUILD_STEP cleanup@@@
+  rm_dirs llvm_build2_* llvm_build_* libcxx_build_* compiler_rt_build* symbolizer_build* $@
+  if ccache -s >/dev/null ; then
+    rm_dirs llvm_build64 clang_build
+  fi
+  ls
 }
 
 function clobber {
-  if [ "$BUILDBOT_CLOBBER" != "" ]; then
+  if [[ "$BUILDBOT_CLOBBER" != "" ]]; then
     echo @@@BUILD_STEP clobber@@@
-    rm_dirs svn_checkout llvm llvm-project llvm_build0 ${CLOBBER:-}
-    stage1_clobber
-    ! test "$(ls -A .)" || echo @@@STEP_EXCEPTION@@@
+    if [[ ! -v BUILDBOT_BUILDERNAME ]]; then
+      echo "Clobbering is supported only on buildbot only!"
+      exit 1
+    fi
+    rm_dirs *
+  else
+    BUILDBOT_BUILDERNAME=1 cleanup $@
   fi
 }
 
 BUILDBOT_MONO_REPO_PATH=${BUILDBOT_MONO_REPO_PATH:-}
 
 function buildbot_update {
   echo @@@BUILD_STEP update $BUILDBOT_REVISION@@@
   if [[ -d "$BUILDBOT_MONO_REPO_PATH" ]]; then
     LLVM=$BUILDBOT_MONO_REPO_PATH/llvm
   else
     (
       local DEPTH=100
       [[ -d llvm-project ]] || (
         mkdir -p llvm-project
         cd llvm-project
         git init
         git remote add origin https://github.com/llvm/llvm-project.git
+        git config --local advice.detachedHead false
       )
       cd llvm-project
-      git fetch --depth $DEPTH origin master
+      git fetch --depth $DEPTH origin main
       git clean -fd
       local REV=${BUILDBOT_REVISION}
-      if [[  "$REV" != "origin/master" ]] ; then
+      if [[  "$REV" != "origin/main" ]] ; then
         # "git fetch --depth 1 origin $REV" does not work with 2.11 on bots
         while true ; do
           git checkout $REV && break
-          git rev-list --pretty --max-count=1 origin/master
-          git rev-list --pretty --max-parents=0 origin/master
+          git rev-list --pretty --max-count=1 origin/main
+          git rev-list --pretty --max-parents=0 origin/main
           echo "DEPTH=$DEPTH is too small"
           [[ "$DEPTH" -le "1000000" ]] || exit 1
           DEPTH=$(( $DEPTH * 10 ))
           git fetch --depth $DEPTH origin
         done
       fi
       git checkout $REV
       git status
       git rev-list --pretty --max-count=1 HEAD
     ) || { echo @@@STEP_EXCEPTION@@@ ; exit 1 ; }
     LLVM=$ROOT/llvm-project/llvm
   fi
 }
 
 function common_stage1_variables {
   STAGE1_DIR=llvm_build0
   stage1_clang_path=$ROOT/${STAGE1_DIR}/bin
   llvm_symbolizer_path=${stage1_clang_path}/llvm-symbolizer
   STAGE1_AS_COMPILER="-DCMAKE_C_COMPILER=${stage1_clang_path}/clang -DCMAKE_CXX_COMPILER=${stage1_clang_path}/clang++"
 }
 
 function build_stage1_clang_impl {
-  rm -rf ${STAGE1_DIR}
   mkdir -p ${STAGE1_DIR}
   local cmake_stage1_options="${CMAKE_COMMON_OPTIONS}"
   cmake_stage1_options="${cmake_stage1_options} -DLLVM_ENABLE_PROJECTS='clang;compiler-rt;lld'"
   if ccache -s ; then
     cmake_stage1_options="${cmake_stage1_options} -DLLVM_CCACHE_BUILD=ON"
   fi
   (cd ${STAGE1_DIR} && cmake ${cmake_stage1_options} $LLVM && ninja)
 }
 
 function build_stage1_clang {
   echo @@@BUILD_STEP stage1 build all@@@
-  export STAGE1_DIR=llvm_build0
   common_stage1_variables
   build_stage1_clang_impl
-
-  echo @@@BUILD_STEP Clobber stage1 users
-  stage1_clobber
 }
 
 function build_stage1_clang_at_revison {
   common_stage1_variables
 
   curl -s https://raw.githubusercontent.com/chromium/chromium/master/tools/clang/scripts/update.py \
     | python - --output-dir=${STAGE1_DIR}
 
function build_stage2 {
   export STAGE2_DIR=${build_dir}
 
   common_stage2_variables
 
   if [ "$sanitizer_name" == "msan" ]; then
     export MSAN_SYMBOLIZER_PATH="${llvm_symbolizer_path}"
     local llvm_use_sanitizer="Memory"
     local fsanitize_flag="-fsanitize=memory"
-    BUILDBOT_MSAN_ORIGINS=${BUILDBOT_MSAN_ORIGINS:-}
-    if [ "$BUILDBOT_MSAN_ORIGINS" != "" ]; then
-      llvm_use_sanitizer="MemoryWithOrigins"
-    fi
     local build_type="Release"
   elif [ "$sanitizer_name" == "asan" ]; then
     export ASAN_SYMBOLIZER_PATH="${llvm_symbolizer_path}"
     export ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1"
     local llvm_use_sanitizer="Address"
     local fsanitize_flag="-fsanitize=address"
     local build_type="Release"
   elif [ "$sanitizer_name" == "ubsan" ]; then
function build_stage2 {
   (cd ${libcxx_build_dir} && \
     cmake \
       ${cmake_stage2_common_options} \
       -DLLVM_ENABLE_PROJECTS='libcxx;libcxxabi' \
       -DCMAKE_BUILD_TYPE=${build_type} \
       -DLLVM_USE_SANITIZER=${llvm_use_sanitizer} \
       $LLVM && \
     ninja cxx cxxabi) || echo $step_result
-  local sanitizer_ldflags="-lc++abi -Wl,--rpath=${ROOT}/${libcxx_build_dir}/lib -L${ROOT}/${libcxx_build_dir}/lib"
+
+  libcxx_runtime_path=$(dirname $(find ${ROOT}/${libcxx_build_dir} -name libc++.so))
+  local sanitizer_ldflags="-lc++abi -Wl,--rpath=${libcxx_runtime_path} -L${libcxx_runtime_path}"
   local sanitizer_cflags="-nostdinc++ -isystem ${ROOT}/${libcxx_build_dir}/include -isystem ${ROOT}/${libcxx_build_dir}/include/c++/v1"
 
   echo @@@BUILD_STEP stage2/$sanitizer_name build@@@
 
   # See http://llvm.org/bugs/show_bug.cgi?id=19071, http://www.cmake.org/Bug/view.php?id=15264
   local cmake_bug_workaround_cflags="$sanitizer_ldflags $fsanitize_flag -w"
   sanitizer_cflags="$sanitizer_cflags $cmake_bug_workaround_cflags"
 

zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh

 PLATFORM=`uname`
 export PATH="/usr/local/bin:$PATH"
 
 MAKE_JOBS=${MAX_MAKE_JOBS:-$(nproc)}
 LLVM=$ROOT/llvm
 LIBFUZZER=$LLVM/lib/Fuzzer
 # No assertions. Need to clean up the existing assertion failures first.
 # Also, the Fuzzer does not provide reproducers on assertion failures yet.
-CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_PARALLEL_LINK_JOBS=8 -DLIBFUZZER_ENABLE_TESTS=ON"
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_LIT_ARGS=-v\;--time-tests"
 
-CLOBBER=fuzzer-test-suite
-STAGE1_CLOBBER="RUNDIR-* $LIBFUZZER"
-clobber
+clobber RUNDIR-*
 
 # Make sure asan intercepts SIGABRT so that the fuzzer can print the test cases
 # for assertion failures.
 # export ASAN_OPTIONS=handle_abort=1:strip_path_prefix=build/llvm/
 
 buildbot_update
 
 # Stage 1

 RunFuzzerTest libpng-1.2.56        || echo @@@STEP_FAILURE@@@
 RunFuzzerTest libssh-2017-1272     || echo @@@STEP_FAILURE@@@
 RunFuzzerTest re2-2014-12-09       || echo @@@STEP_FAILURE@@@
 RunFuzzerTest c-ares-CVE-2016-5180 || echo @@@STEP_FAILURE@@@
 RunFuzzerTest openssl-1.0.1f       || echo @@@STEP_FAILURE@@@
 RunFuzzerTest openssl-1.0.2d       || echo @@@STEP_FAILURE@@@
 RunFuzzerTest proj4-2017-08-14     || echo @@@STEP_FAILURE@@@
 #RunFuzzerTest woff2-2016-05-06     || echo @@@STEP_WARNINGS@@@  # Often can't find the bug in the given time.
+
+cleanup $STAGE1_DIR RUNDIR-*

zorg/buildbot/builders/sanitizers/buildbot_qemu.sh

+#!/usr/bin/env bash
+
+set -x
+set -e
+set -u
+
+HERE="$(cd $(dirname $0) && pwd)"
+. ${HERE}/buildbot_functions.sh
+
+ROOT=`pwd`
+PLATFORM=`uname`
+export PATH="/usr/local/bin:$PATH"
+
+LLVM=$ROOT/llvm
+CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=OFF"
+
+clobber
+
+buildbot_update
+
+build_stage1_clang
+
+set +x # Avoid echoing STEP_FAILURE because of the command trace.
+MISSING_QEMU_IMAGE_MESSAGE=$(cat << 'EOF'
+=====================================================================
+Looks like you're missing the QEMU system images for x86_64 LAM
+HWASan testing. These system images aren't automatically generated
+as part of the buildbot script because they require root (sorry!).
+
+If you have the system images already built, you can run
+buildbot_qemu.sh with QEMU_IMAGE_DIR=path/to/qemu/images. Otherwise,
+you can build the images (only necessary once-per-clobber) by:
+ 1. WARNING: Read the script at https://github.com/google/sanitizers/blob/master/hwaddress-sanitizer/create_qemu_image.sh
+    You're running this as ROOT, so make sure you're comfortable with
+    that.
+ 2. In your terminal, run:
+      $ sudo su -c "bash <(wget -qO- https://raw.githubusercontent.com/google/sanitizers/master/hwaddress-sanitizer/create_qemu_image.sh)" root && \
+        sudo chown $(whoami) debian.*
+
+You can also choose to skip the x86_64 HWASan LAM testing by supplying
+SKIP_HWASAN_LAM=true in your invocation of this script.
+=====================================================================
+@@@STEP_EXCEPTION@@@
+EOF
+)
+set -x
+
+SKIP_HWASAN_LAM=${SKIP_HWASAN_LAM:-}
+
+function setup_lam_qemu_image {
+  # Full system emulation is required for x86_64 testing with LAM, as some
+  # sanitizers aren't friendly with usermode emulation under x86_64 LAM.
+  echo @@@BUILD_STEP Check x86_64 LAM Prerequisites@@@
+
+  # Allow specifying the QEMU image dir, otherwise assume in the local dir.
+  QEMU_IMAGE_DIR=${QEMU_IMAGE_DIR:=${ROOT}}
+
+  # Ensure the buildbot start script created a QEMU image for us.
+  (
+    ls ${QEMU_IMAGE_DIR}/debian.img
+    ls ${QEMU_IMAGE_DIR}/debian.id_rsa
+  ) ||
+  (
+    # Make the "missing file" error clearer by not effectively echoing twice.
+    set +x
+    echo "$MISSING_QEMU_IMAGE_MESSAGE"
+    exit 1
+  )
+}
+
+([[ -z "$SKIP_HWASAN_LAM" ]] && setup_lam_qemu_image) || SKIP_HWASAN_LAM=1
+
+COMPILER_BIN_DIR=$(readlink -f ${STAGE1_DIR})/bin
+
+function git_clone_at_revision {
+  local src_dir_name="${1}"
+  local git_url="${2}"
+  local revision="${3}"
+  local build_dir="${4}"
+  (
+    cd ${ROOT}
+    [[ -d ${src_dir_name} ]] || git clone ${git_url} ${src_dir_name} || exit 1
+    cd ${ROOT}/${src_dir_name}
+    git remote set-url origin ${git_url}
+    git fetch origin
+
+    # Short circuit if we already have a build at the correct revision.
+    [[ "$(git rev-parse HEAD)" == "$(git rev-parse ${revision})" ]] && exit 0
+
+    # Delete the build dir to ensure a rebuild happens at the new revision.
+    rm -rf ${build_dir}
+
+    git reset --hard ${revision}
+    git submodule update --init --recursive
+  )
+}
+
+function build_qemu {
+  local qemu_name="${1}"
+  local qemu_url="${2}"
+  local qemu_revision="${3}"
+
+  local build_dir="${ROOT}/${qemu_name}_build"
+
+  echo "@@@BUILD_STEP build ${qemu_name}@@@"
+  (
+    git_clone_at_revision ${qemu_name} ${qemu_url} ${qemu_revision} \
+      ${build_dir} || exit 1
+
+    ${build_dir}/qemu-x86_64 --version &&
+      ${build_dir}/qemu-system-x86_64 --version &&
+      ${build_dir}/qemu-img --version &&
+      exit 0
+
+    rm -rf ${build_dir} &&
+    mkdir -p ${build_dir} &&
+    cd ${build_dir} &&
+    ${ROOT}/${qemu_name}/configure --enable-linux-user &&
+    ninja &&
+    ${build_dir}/qemu-x86_64 --version &&
+    ${build_dir}/qemu-system-x86_64 --version &&
+    ${build_dir}/qemu-img --version
+  ) || (
+    echo "@@@STEP_EXCEPTION@@@"
+    exit 2
+  )
+}
+
+function build_lam_linux {
+  local build_dir="${ROOT}/lam_linux_build"
+
+  echo "@@@BUILD_STEP build lam linux@@@"
+  (
+    git_clone_at_revision lam_linux https://github.com/morehouse/linux.git \
+      origin/lam ${build_dir} || exit 1
+
+    ls ${build_dir}/arch/x86_64/boot/bzImage && exit 0
+
+    rm -rf ${build_dir} &&
+    mkdir -p ${build_dir} &&
+    cd ${ROOT}/lam_linux &&
+    make O=${build_dir} LD=ld.bfd defconfig &&
+    make O=${build_dir} LD=ld.bfd -j $(nproc) &&
+    ls ${build_dir}/arch/x86_64/boot/bzImage
+  ) || (
+    echo "@@@STEP_EXCEPTION@@@"
+    exit 2
+  )
+}
+
+build_qemu qemu https://github.com/vitalybuka/qemu.git origin/sanitizer_bot
+[[ -z "$SKIP_HWASAN_LAM" ]] && (
+  build_qemu lam_qemu https://github.com/morehouse/qemu.git origin/lam
+  build_lam_linux
+)
+
+SCUDO_BUILDS=
+
+function configure_scudo_compiler_rt {
+  local arch=$1
+  local target="${arch}-linux-gnu${2:-}"
+
+  local name=""
+  if [[ "$DBG" == "ON" ]] ; then
+    name=debug_
+  fi
+  name+="${arch}"
+
+  local qemu_cmd=""
+  if [[ "${QEMU:-}" != "0" ]] ; then
+    name+="_qemu"
+    local qemu_arch=${arch}
+    [[ "${arch}" == "powerpc64le" ]] && qemu_arch="ppc64le"
+    qemu_cmd="$ROOT/qemu_build/qemu-${qemu_arch} -L /usr/${target}"
+    if [[ ! -z "${QEMU_CPU:-}" ]] ; then
+      qemu_cmd+=" -cpu ${QEMU_CPU}"
+      name+="_${QEMU_CPU}"
+    fi
+  fi
+
+  local linker_flags=
+  [[ "${arch}" =~ "mips*" ]] && linker_flags="-latomic -Wl,-z,notext -Wno-unused-command-line-argument"
+
+  local out_dir=llvm_build2_${name}
+  rm -rf ${out_dir}
+  mkdir -p ${out_dir}
+
+  SCUDO_BUILDS+=" ${name}"
+
+  (
+    cd ${out_dir}
+
+    (
+      cmake \
+        ${CMAKE_COMMON_OPTIONS} \
+        -DCOMPILER_RT_DEBUG=$DBG \
+        -DLLVM_CONFIG_PATH=${COMPILER_BIN_DIR}/llvm-config \
+        -DCMAKE_C_COMPILER=${COMPILER_BIN_DIR}/clang \
+        -DCMAKE_CXX_COMPILER=${COMPILER_BIN_DIR}/clang++ \
+        -DCOMPILER_RT_HAS_LLD=ON \
+        -DCOMPILER_RT_TEST_USE_LLD=ON \
+        -DCMAKE_INSTALL_PREFIX=$(${COMPILER_BIN_DIR}/clang -print-resource-dir) \
+        -DLLVM_LIT_ARGS="-v --time-tests" \
+        -DCOMPILER_RT_BUILD_BUILTINS=OFF \
+        -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
+        -DCMAKE_CROSSCOMPILING=True \
+        -DCOMPILER_RT_INCLUDE_TESTS=ON \
+        -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
+        -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
+        -DCMAKE_CXX_FLAGS=-fPIC \
+        -DCMAKE_C_FLAGS=-fPIC \
+        -DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=lld ${linker_flags}" \
+        -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld ${linker_flags}" \
+        -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${target} ${linker_flags}" \
+        -DCMAKE_C_COMPILER_TARGET=${target} \
+        -DCMAKE_CXX_COMPILER_TARGET=${target} \
+        -DCOMPILER_RT_EMULATOR="${qemu_cmd:-}" \
+        $LLVM/../compiler-rt
+     ) >& configure.log
+  ) &
+}
+
+readonly LLVM_SYMBOLIZER_DIR="${ROOT}/llvm_build2_x86_64_symbolizer"
+
+function configure_llvm_symbolizer {
+  rm -rf ${LLVM_SYMBOLIZER_DIR}
+  mkdir -p ${LLVM_SYMBOLIZER_DIR}
+
+  (
+    cd ${LLVM_SYMBOLIZER_DIR}
+
+    (
+      cmake \
+        ${CMAKE_COMMON_OPTIONS} \
+        -DCMAKE_C_COMPILER=${COMPILER_BIN_DIR}/clang \
+        -DCMAKE_CXX_COMPILER=${COMPILER_BIN_DIR}/clang++ \
+        -DLLVM_BUILD_RUNTIME=OFF \
+        -DLLVM_STATIC_LINK_CXX_STDLIB=ON \
+        $LLVM
+     ) >& configure.log
+  ) &
+}
+
+function configure_hwasan_lam {
+  git_clone_at_revision sanitizers https://github.com/google/sanitizers.git \
+    origin/master ""
+  local script="${ROOT}/sanitizers/hwaddress-sanitizer/run_in_qemu_with_lam.sh"
+  ls ${script}
+
+  local out_dir=llvm_build2_x86_64_lam_qemu
+  rm -rf ${out_dir}
+  mkdir -p ${out_dir}
+
+  (
+    cd ${out_dir}
+
+    (
+      cmake \
+        ${CMAKE_COMMON_OPTIONS} \
+        -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;lld;libcxx;libcxxabi" \
+        -DCMAKE_C_COMPILER=${COMPILER_BIN_DIR}/clang \
+        -DCMAKE_CXX_COMPILER=${COMPILER_BIN_DIR}/clang++ \
+        -DLLVM_ENABLE_LLD=ON \
+        -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
+        -DLLVM_LIT_ARGS="-v --time-tests" \
+        -DCOMPILER_RT_EMULATOR="env ROOT=${ROOT} QEMU_IMAGE_DIR=${QEMU_IMAGE_DIR} ${script}" \
+        $LLVM
+     ) >& configure.log
+  ) &
+}
+
+function run_scudo_tests {
+  local name="${1}"
+  local out_dir=llvm_build2_${name}
+
+  echo "@@@BUILD_STEP scudo $name@@@"
+
+  (
+    cd ${out_dir}
+
+    cat configure.log
+
+    # Copy into clang resource dir to make -fsanitize= work in lit tests.
+    ninja install-scudo_standalone
+
+    ninja check-scudo_standalone || exit 3
+  ) || echo "@@@STEP_FAILURE@@@"
+}
+
+function build_llvm_symbolizer {
+  echo "@@@BUILD_STEP llvm-symbolizer x86_64@@@"
+
+  (
+    cd ${LLVM_SYMBOLIZER_DIR}
+
+    cat configure.log
+
+    ninja llvm-symbolizer || exit 3
+  ) || echo "@@@STEP_FAILURE@@@"
+}
+
+function run_hwasan_lam_tests {
+  local name="x86_64_lam_qemu"
+  local out_dir=llvm_build2_${name}
+
+  echo "@@@BUILD_STEP hwasan ${name}@@@"
+
+  (
+    cd ${out_dir}
+
+    cat configure.log
+
+    # LLD must be built first since HWASan tests use -fuse-ld=lld and the
+    # buildbots don't have LLD preinstalled.
+    ninja lld || exit 3
+
+    ninja check-hwasan-lam || exit 3
+  ) || echo "@@@STEP_FAILURE@@@"
+}
+
+echo "@@@BUILD_STEP configure@@@"
+
+for DBG in OFF ON ; do
+  QEMU=0 configure_scudo_compiler_rt x86_64
+  configure_scudo_compiler_rt x86_64
+  configure_scudo_compiler_rt arm eabihf
+  configure_scudo_compiler_rt aarch64
+  QEMU_CPU="cortex-a72" configure_scudo_compiler_rt aarch64
+  #configure_scudo_compiler_rt mips
+  #configure_scudo_compiler_rt mipsel
+  configure_scudo_compiler_rt mips64 abi64
+  configure_scudo_compiler_rt mips64el abi64
+  configure_scudo_compiler_rt powerpc64le
+done
+configure_llvm_symbolizer
+[[ -z "$SKIP_HWASAN_LAM" ]] && configure_hwasan_lam
+
+wait
+
+for B in $SCUDO_BUILDS ; do
+  run_scudo_tests $B
+done
+
+[[ -z "$SKIP_HWASAN_LAM" ]] && (
+  # Symbolizer only required for HWASan LAM tests.
+  build_llvm_symbolizer
+  run_hwasan_lam_tests
+)
+
+cleanup $STAGE1_DIR

zorg/buildbot/builders/sanitizers/buildbot_selector.py

 #!/usr/bin/python
 
 import os
 import subprocess
 import sys
 
 THIS_DIR=os.path.dirname(sys.argv[0])
-
+extra_args = sys.argv[1:]
+BOT_DIR = '/b'
 
 def bash(path):
     return 'bash ' + os.path.join(THIS_DIR, path)
 
 def cmd_call(path):
     return 'call ' + os.path.join(THIS_DIR, path)
 
 BOT_ASSIGNMENT = {
     'sanitizer-ppc64le-linux': bash('buildbot_cmake.sh'),
     'sanitizer-ppc64be-linux': bash('buildbot_cmake.sh'),
     'sanitizer-x86_64-linux': bash('buildbot_cmake.sh'),
     'sanitizer-x86_64-linux-fast': bash('buildbot_fast.sh'),
     'sanitizer-x86_64-linux-autoconf': bash('buildbot_standard.sh'),
     'sanitizer-x86_64-linux-fuzzer': bash('buildbot_fuzzer.sh'),
     'sanitizer-x86_64-linux-android': bash('buildbot_android.sh'),
     'sanitizer-x86_64-linux-bootstrap': bash('buildbot_bootstrap_asan.sh'),
+    'sanitizer-x86_64-linux-bootstrap-asan': bash('buildbot_bootstrap_asan.sh'),
     'sanitizer-x86_64-linux-bootstrap-msan': bash('buildbot_bootstrap_msan.sh'),
     'sanitizer-x86_64-linux-bootstrap-ubsan': bash('buildbot_bootstrap_ubsan.sh'),
-    'sanitizer-x86_64-linux-gn': 'true',
+    'sanitizer-x86_64-linux-qemu': bash('buildbot_qemu.sh'),
 }
 
 BOT_ADDITIONAL_ENV = {
     'sanitizer-ppc64le-linux': { 'HAVE_NINJA': '1', 'CHECK_LIBCXX': '0', 'CHECK_LLD': '0' },
     'sanitizer-ppc64be-linux': { 'HAVE_NINJA': '1', 'CHECK_LIBCXX': '0', 'CHECK_LLD': '0' },
     'sanitizer-x86_64-linux': { 'HAVE_NINJA' : '1' },
-    'sanitizer-x86_64-linux-bootstrap': {},
     'sanitizer-x86_64-linux-fast': {},
     'sanitizer-x86_64-linux-autoconf': {},
     'sanitizer-x86_64-linux-fuzzer': {},
     'sanitizer-x86_64-linux-android': {},
+    'sanitizer-x86_64-linux-bootstrap': {},
+    'sanitizer-x86_64-linux-bootstrap-asan': {},
     'sanitizer-x86_64-linux-bootstrap-msan': {},
     'sanitizer-x86_64-linux-bootstrap-ubsan': {},
-    'sanitizer-x86_64-linux-gn': {},
+    'sanitizer-x86_64-linux-qemu': { 'QEMU_IMAGE_DIR': BOT_DIR + '/qemu_image' },
 }
 
 def Main():
   builder = os.environ.get('BUILDBOT_BUILDERNAME')
   print "builder name: %s" % (builder)
-  cmd = BOT_ASSIGNMENT.get(builder)
+  cmd = BOT_ASSIGNMENT.get(builder) + ' ' + ' '.join(extra_args)
   if not cmd:
     sys.stderr.write('ERROR - unset/invalid builder name\n')
     sys.exit(1)
 
   print "%s runs: %s\n" % (builder, cmd)
   sys.stdout.flush()
 
   bot_env = os.environ
+  bot_env['BOT_DIR'] = BOT_DIR
   add_env = BOT_ADDITIONAL_ENV.get(builder)
   for var in add_env:
     bot_env[var] = add_env[var]
   if 'TMPDIR' in bot_env:
     del bot_env['TMPDIR']
 
   retcode = subprocess.call(cmd, env=bot_env, shell=True)
   sys.exit(retcode)
 
 
 if __name__ == '__main__':
   Main()

zorg/buildbot/builders/sanitizers/buildbot_standard.sh

 #!/usr/bin/env bash
 
 set -x
 set -e
 set -u
 
 HERE="$(dirname $0)"
 . ${HERE}/buildbot_functions.sh
 
 TSAN_DEBUG_BUILD_DIR=tsan_debug_build
 TSAN_FULL_DEBUG_BUILD_DIR=tsan_full_debug_build
 TSAN_RELEASE_BUILD_DIR=tsan_release_build
 
-rm -rf $TSAN_DEBUG_BUILD_DIR $TSAN_FULL_DEBUG_BUILD_DIR $TSAN_RELEASE_BUILD_DIR
-clobber
+CLEANUP="$TSAN_DEBUG_BUILD_DIR $TSAN_FULL_DEBUG_BUILD_DIR $TSAN_RELEASE_BUILD_DIR"
+clobber $CLEANUP
 
 ROOT=`pwd`
 PLATFORM=`uname`
 MAKE_JOBS=${MAX_MAKE_JOBS:-$(nproc)}
 
 LLVM=${ROOT}/llvm
 CMAKE_COMMON_OPTIONS="-GNinja -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON -DLLVM_USE_LINKER=gold"
 

 echo @@@BUILD_STEP build release tsan with clang@@@
 build_tsan "${TSAN_RELEASE_BUILD_DIR}" "-DCOMPILER_RT_DEBUG=OFF" "$ROOT/$TSAN_DEBUG_BUILD_DIR/bin/clang" "$ROOT/$TSAN_DEBUG_BUILD_DIR/bin/clang++"
 
 echo @@@BUILD_STEP tsan analyze@@@
 BIN=$(mktemp -t tsan_exe.XXXXXXXX)
 echo "int main() {return 0;}" | $TSAN_RELEASE_BUILD_DIR/bin/clang -x c++ - -fsanitize=thread -O2 -o ${BIN}
 COMPILER_RT=$LLVM/../compiler-rt
 $COMPILER_RT/lib/tsan/check_analyze.sh ${BIN} || echo @@@STEP_FAILURE@@@
+
+cleanup $CLEANUP

zorg/buildbot/changes/llvmgitpoller.py

 # LLVM buildbot needs to watch multiple projects within a single repository.
 # TODO: Handle both author and committer for a commit to build a correct blame list later.
 import re
 
 from twisted.python import log
 from datetime import datetime
 
 from twisted.internet import defer
 from buildbot.util import bytes2unicode
 from buildbot.plugins import changes
 
 class LLVMPoller(changes.GitPoller):
     """
-    Poll LLVM repository for changes and submit them to the change master.
+    Poll LLVM repository for changes and submit them for builds scheduling.
     Following Multiple LLVM Projects.
 
     This source will poll a remote LLVM git _monorepo_ for changes and submit
-    them to the change master."""
+    them for builds scheduling."""
 
     _repourl = "https://github.com/llvm/llvm-project"
-    _branch = "master"
 
-    compare_attrs = ["repourl", "branch", "workdir",
+    compare_attrs = ["repourl", "branches", "workdir",
                      "pollInterval", "gitbin", "usetimestamps",
                      "category", "project",
                      "projects"]
 
+    def _check_branches(branch):
+        log.msg("LLVMPoller: _check_branches: branch={}", branch)
+
+        if branch == "refs/heads/main":
+            # Always listen for changes in the main branch.
+            return True
+        else:
+            # We are also interested in the release branches.
+            # Some builders will be building changes from there as well.
+            return re.search(r"refs\/heads\/release\/\d\d+.*", branch)
+
     def __init__(self,
-                 repourl=_repourl, branch=_branch,
+                 repourl=_repourl, branches=_check_branches,
                  **kwargs):
 
         self.cleanRe = re.compile(r"Require(?:s?)\s*.*\s*clean build", re.IGNORECASE + re.MULTILINE)
         self.cleanCfg = re.compile(r"(CMakeLists\.txt$|\.cmake$|\.cmake\.in$)")
 
-        # TODO: Add support for an optional list of projects.
-        # For now we always watch all the projects.
+        # Note: We always watch all the projects, then schedulers decide
+        # to build or not to build.
 
-        super().__init__(repourl=repourl, branch=branch, **kwargs)
+        super().__init__(repourl=repourl, branches=branches, **kwargs)
 
     def _transform_path(self, fileList):
         """
         Parses the given list of files, and returns a list of two-entry tuples
         (PROJECT, [FILES]) if PROJECT is watched one,
         or None otherwise.
 
         NOTE: we don't change result path, just extract a project name.
def _process_changes(self, newRev, branch):
             if existingRev != newRev:
                 revList = [newRev]
                 if existingRev is None:
                     # This branch was completely unknown, rebuild
                     log.msg('LLVMPoller: rebuilding {} for new branch "{}"'.format(
                         newRev, branch))
                 else:
                     # This branch is known, but it now points to a different
-                    # commit than last time we saw it, rebuild.
+                    # commit than the last time we saw it, rebuild.
                     log.msg('LLVMPoller: rebuilding {} for updated branch "{}"'.format(
                         newRev, branch))
 
         self.changeCount = len(revList)
         self.lastRev[branch] = newRev
 
         if self.changeCount:
             log.msg('LLVMPoller: processing {} changes: {} from "{}" branch "{}"'.format(
def _process_changes(self, newRev, branch):
                 projects += [where_project]
 
                 if self.cleanRe.search(comments) or \
                    any([m for f in where_project_files for m in [self.cleanCfg.search(f)] if m]):
                     log.msg("LLVMPoller: creating a change with the 'clean_obj' property for r%s" % rev)
                     properties['clean_obj'] = (True, "change")
 
             log.msg("LLVMPoller: creating a change rev=%s" % rev)
-            log.msg("  >>> revision=%s, timestamp=%s, author=%s, committer=%s, project=%s, files=%s, comments=\"%s\", properties=%s" % \
-                (bytes2unicode(rev, encoding=self.encoding), datetime.fromtimestamp(timestamp), author, committer,
+            log.msg("  >>> branch=%s, revision=%s, timestamp=%s, author=%s, committer=%s, project=%s, files=%s, comments=\"%s\", properties=%s" % \
+                (bytes2unicode(self._removeHeads(branch)),
+                bytes2unicode(rev, encoding=self.encoding), datetime.fromtimestamp(timestamp), author, committer,
                 projects, files, comments, properties))
 
             yield self.master.data.updates.addChange(
                        author=author,
                        committer=committer if committer != author else None,
                        revision=bytes2unicode(rev, encoding=self.encoding),
                        files=files,
                        comments=comments,
                        when_timestamp=timestamp,
                        branch=bytes2unicode(self._removeHeads(branch)),
                        category=self.category, ## TODO: Figure out if we could support tags here
                        project=",".join(projects) if projects else "llvm-project",
                        # Always promote an external github url of the LLVM project with the changes.
                        repository=self._repourl,
                        src='git', # Must be one of the buildbot.process.users.srcs
                        properties=properties)

zorg/buildbot/commands/AnnotatedCommand.py

def __init__(self, **kwargs):
             'BUILDBOT_BLAMELIST':       util.Interpolate('%(prop:blamelist:-[])s'),
             'BUILDBOT_BRANCH':          util.Interpolate('%(prop:branch:-None)s'),
             'BUILDBOT_BUILDERNAME':     util.Interpolate('%(prop:buildername:-None)s'),
             'BUILDBOT_BUILDNUMBER':     util.Interpolate('%(prop:buildnumber:-None)s'),
             'BUILDBOT_CLOBBER':         util.Interpolate('%(prop:clobber:+1)s'),
             'BUILDBOT_GOT_REVISION':    util.Interpolate('%(prop:got_revision:-None)s'),
             'BUILDBOT_REVISION':        util.Interpolate('%(prop:revision:-None)s'),
             'BUILDBOT_SCHEDULER':       util.Interpolate('%(prop:scheduler:-None)s'),
-            'BUILDBOT_SLAVENAME':       util.Interpolate('%(prop:slavename:-None)s'),
-            'BUILDBOT_MSAN_ORIGINS':    util.Interpolate('%(prop:msan_origins:-)s'),
         }
         # Apply the passed in environment on top.
         old_env = kwargs.get('env') or {}
         env.update(old_env)
         # Change passed in args (ok as a copy is made internally).
         kwargs['env'] = env
 
         kwargs = self.setupShellMixin(kwargs)

zorg/buildbot/process/factory.py

 # TODO: Change WithProperties to Interpolate
 # TODO: Consider setting codebase to llvm-project.
 from collections import OrderedDict
 
 from buildbot.process.factory import BuildFactory
 from buildbot.plugins import util, steps
 
+_all_runtimes = frozenset([
+    "compiler-rt",
+    "libc",
+    "libcxx",
+    "libcxxabi",
+    "libunwind",
+    "openmp",
+])
+
 class LLVMBuildFactory(BuildFactory):
     """
     TODO: Document
+
+    depends_on_projects is a list of LLVM projects a produced builder
+    depends on. If None, it gets discovered depending on other params.
+
+    enable_runtimes is a list of enabled runtimes. If None,
+    it gets discovered based on the depends_on_projects list.
     """
 
     def __init__(self, steps=None, depends_on_projects=None, **kwargs):
         # Cannot use "super" here as BuildFactory is an old style class.
         BuildFactory.__init__(self, steps)
 
+        # Handle the dependencies.
         if depends_on_projects is None:
-            self.depends_on_projects = frozenset(['llvm'])
+            # llvm project is always included.
+            self.depends_on_projects = set(['llvm'])
         else:
             self.depends_on_projects = frozenset(depends_on_projects)
 
+        enable_runtimes = kwargs.pop('enable_runtimes', None)
+        # If specified, we either got a givem list of
+        # enabled runtimes, or "auto", or "all".
+
+        if enable_runtimes is None:
+            # For the backward compatibility, we do not use
+            # enable_runtimes unless it is requested explicitly.
+            self.enable_runtimes = frozenset([])
+        elif enable_runtimes == "auto":
+            # Let's build the list of runtimes based on the given
+            # depends_on_projects list.
+            self.enable_runtimes = \
+                self.depends_on_projects.intersection(_all_runtimes)
+        else:
+            if  enable_runtimes == "all":
+                # Let's replace the "all" placeholder by
+                # the actual list of all runtimes.
+                self.enable_runtimes = frozenset(_all_runtimes)
+            else:
+                # Let's just use the given list, no need to discover.
+                self.enable_runtimes = frozenset(enable_runtimes)
+
+            # Update the list of dependencies.
+            if depends_on_projects is None:
+                self.depends_on_projects.update(self.enable_runtimes)
+
+        # Build the list of projects to enable.
+        self.enable_projects = \
+            self.depends_on_projects.difference(self.enable_runtimes)
+
         # Directories.
-        self.llvm_srcdir = kwargs.pop('llvm_srcdir', None)
+        self.monorepo_dir = kwargs.pop('llvm_srcdir', None)
+        self.src_to_build_dir = kwargs.pop('src_to_build_dir', None)
         self.obj_dir = kwargs.pop('obj_dir', None)
         self.install_dir = kwargs.pop('install_dir', None)
 
         # Preserve the rest of the given extra attributes if any, so we could
         # expand the factory later.
         for k,v in kwargs.items():
             setattr(self, k, v)
 
-        self.monorepo_dir = self.llvm_srcdir or "llvm-project"
+        self.monorepo_dir = self.monorepo_dir or "llvm-project"
+        self.src_to_build_dir = self.src_to_build_dir or 'llvm'
         self.llvm_srcdir = \
-                "%(monorepo_dir)s/llvm" % {'monorepo_dir' : self.monorepo_dir}
+                "{}/{}".format(self.monorepo_dir, self.src_to_build_dir)
         self.obj_dir = \
                 self.obj_dir or "build"
 
         # Repourl_prefix could be specified per builder. Otherwise we use github.
         self.repourl_prefix = kwargs.pop('repourl_prefix', 'https://github.com/llvm/')
 
 
     @staticmethod
def pathRelativeTo(path, basePath):
                 # Everything matches.
                 rel_path = '.'
 
             return rel_path
 
 
     def addGetSourcecodeSteps(self, **kwargs):
         # Checkout the monorepo.
+        # Documentation: http://docs.buildbot.net/current/manual/configuration/buildsteps.html#git
         self.addStep(steps.Git(
                 name='Checkout the source code',
                 repourl=self.repourl_prefix + "llvm-project.git",
                 progress=True,
                 workdir=util.WithProperties(self.monorepo_dir),
+                retryFetch=True,
+                clobberOnFailure=True,
                 **kwargs))
 
 
     # Checkout a given LLVM project to the given directory.
     # TODO: Handle clean property and self.clean attribute.
     def addGetSourcecodeForProject(self, project, name=None, src_dir=None, **kwargs):
         # project contains a repo name which is not a part of the monorepo.
         #  We do not enforce it here, though.
def addGetSourcecodeForProject(self, project, name=None, src_dir=None, **kwargs):
         # Ignore workdir if given. We check out to src_dir.
         kwargs.pop('workdir', None)
 
         self.addStep(steps.Git(
                 name=name,
                 repourl=_repourl,
                 progress=True,
                 workdir=util.WithProperties(src_dir),
+                retryFetch=True,
+                clobberOnFailure=True,                
                 **kwargs))

zorg/buildbot/util/InformativeMailNotifier.py

-# TODO: Add support for clang crash dumps.
-# TODO: Better handle unit/regression tests failures
-
-# TODO: For debug purposes. Remove this later.
-from twisted.python import log
+import traceback
 
 from buildbot.plugins import reporters
+from buildbot.process.results import CANCELLED
+from buildbot.process.results import EXCEPTION
+from buildbot.process.results import FAILURE
+from buildbot.process.results import RETRY
+from buildbot.process.results import SKIPPED
+from buildbot.process.results import SUCCESS
+from buildbot.process.results import WARNINGS
+from buildbot.process.results import statusToString
+
+def get_log_details(build):
+    text = ""
+    try:
+        for step in build['steps']:
+            results = step['results']
+            if results == SUCCESS or results == SKIPPED:
+                continue
+
+            text += "Step {} ({}) {}: {}\n".format(step['number'], step['name'], statusToString(results), step['state_string'])
+
+            logs = step['logs']
+            if logs:
+                log_index = -1
+                log_type = 0
+                for i, _log in enumerate(logs):
+                    if _log['name'].startswith("FAIL: "): # Use only first logchunk FAIL:
+                        log_type = 3
+                        log_index = i
+                    elif log_type < 2 and _log['name'].startswith("warnings "):
+                        log_type = 2
+                        log_index = i
+                    elif log_type < 1 and _log['type'] == "s": # stdio
+                        log_type = 1
+                        log_index = i
+
+                if log_index < 0:
+                    continue
+
+                log_text = logs[log_index]['content']['content']
+                if log_type == 1:
+                    # Parse stdio
+                    lines = log_text.splitlines()
+                    for line in lines[:]:
+                        if line.startswith("h"):
+                            lines.remove(line)
+                    for j, line in enumerate(lines):
+                        if line.startswith("o") or line.startswith("e"):
+                            lines[j] = line[1:]
+                    for j, line in enumerate(lines):
+                        if line.find("FAIL:") != -1 or line.find("FAILED") != -1:
+                            if j > 10:
+                                del lines[:j-10] # Start 10 lines before FAIL
+                                lines = ["..."] + lines
+                            del lines[50:] # Keep up to 50 lines around FAIL
+                            break
+                    if len(lines) > 50:
+                        del lines[:len(lines)-50] # Otherwise keep last 50 lines
+                        lines = ["..."] + lines
+
+                    log_text = "\n".join(lines)
+
+                elif logs[log_index]['num_lines'] > 50:
+                    # Keep first 50 lines
+                    lines = log_text.splitlines()
+                    del lines[50:]
+                    log_text = "\n".join(lines + ["..."])
+
+                text += log_text + "\n"
+
+    except Exception as err:
+        print("Exception in LLVMMessageFormatter.get_log_details(): {}\n{}".format(err, traceback.format_exc()))
+        # TODO: We should send something useful in this case.
+
+    return dict(details=text)
 
-def _get_logs_and_tracebacks_from_build(build):
-    # TODO: Implement interesting parts of the logs and tracebacks extraction.
-    return dict()
+# TODO: Add build reason if we have that valid and available
+#Build Reason: {{ build['properties'].get('reason', ["<unknown>"])[0] }}
 
 MAIL_TEMPLATE = """\
 The Buildbot has detected a {{ status_detected }} on builder {{ buildername }} while building {{ projects }}.
+
 Full details are available at:
     {{ build_url }}
-Buildbot URL: {{ buildbot_url }}
+
 Worker for this Build: {{ workername }}
-Build Reason: {{ build['properties'].get('reason', ["<unknown>"])[0] }}
-Blamelist: {{ ", ".join(blamelist) }}
+Blamelist:
+    {{ ",\n    ".join(blamelist) }}
+
 {{ summary }}
+
+{{ details }}
 Sincerely,
 LLVM Buildbot
 """
 
 class LLVMMessageFormatter(reporters.MessageFormatter):
     def buildAdditionalContext(self, master, ctx):
-        #log.msg(">>> LLVMMessageFormatter.buildAdditionalContext got ctx={}".format(ctx))
         ctx.update(self.ctx)
-
-        build = ctx["build"]
-        build_interesting_data = _get_logs_and_tracebacks_from_build(build)
-        #log.msg(">>> LLVMMessageFormatter.buildAdditionalContext build_interesting_data={}",format(build_interesting_data))
-        ctx["build"].update(build_interesting_data)
-
+        ctx.update(get_log_details(ctx["build"]))
 
 LLVMInformativeMailNotifier = LLVMMessageFormatter(
     template=MAIL_TEMPLATE,
     template_type="plain",
     wantLogs=True,
     wantProperties=True,
     wantSteps=True,
 )

zorg/jenkins/build.py

def lldb_cmake_builder():
     run_cmd(conf.lldbbuilddir(), cmake_cmd)
     footer()
 
     header("Build")
     run_cmd(conf.lldbbuilddir(), [NINJA, '-v'])
     footer()
 
     header("Run Tests")
-    run_cmd(conf.lldbbuilddir(), [NINJA, '-v', 'check-debuginfo'])
     run_cmd(conf.lldbbuilddir(), ['/usr/bin/env', 'TERM=vt100', NINJA, '-v',
-                                  'check-lldb'])
+                                  'check-debuginfo', 'check-lldb', '-k2'])
     footer()
 
 
 def static_analyzer_benchmarks_builder():
     """Run static analyzer benchmarks"""
     header("Static Analyzer Benchmarks")
 
     benchmark_script = conf.workspace + "/utils-analyzer/SATestBuild.py"

zorg/jenkins/common.groovy

private def post_build() {
     // TODO: Notify IRC.
 }
 
 def task_pipeline(label, body) {
     node(label) {
         try {
             stage('main') {
                 dir('config') {
-                    git url: 'https://github.com/llvm/llvm-zorg.git', branch: 'master', poll: false
+                    git url: 'https://github.com/llvm/llvm-zorg.git', branch: 'main', poll: false
                 }
                 body()
             }
         } catch(hudson.AbortException e) {
             // No need to print the exception if something fails inside a 'sh'
             // step.
             currentBuild.result = 'FAILURE'
         } catch (Exception e) {
             currentBuild.result = 'FAILURE'
             throw e
         } finally {
             stage('post') {
                 post_build()
             }
         }
     }
 }
 
 def benchmark_pipeline(label, body) {
     properties([
+        disableResume(),
         parameters([
             string(name: 'ARTIFACT'),
             string(name: 'GIT_DISTANCE'),
             string(name: 'GIT_SHA')
         ])
     ])
 
     currentBuild.displayName = basename(params.ARTIFACT)
     task_pipeline(label) {
         clone_llvm_project('llvm-project', params.GIT_SHA)
         body()
     }
 }
 
 def testsuite_pipeline(label, body) {
     benchmark_pipeline(label) {
         dir('lnt') {
-            git url: 'https://github.com/llvm/llvm-lnt.git', branch: 'master', poll: false
+            git url: 'https://github.com/llvm/llvm-lnt.git', branch: 'main', poll: false
         }
         dir('test-suite') {
-            git url: 'https://github.com/llvm/llvm-test-suite.git', branch: 'master', poll: false
+            git url: 'https://github.com/llvm/llvm-test-suite.git', branch: 'main', poll: false
         }
         body()
     }
 }
 
 return this

zorg/jenkins/inspect_log.py

 _JENKINS_POOL_PATTERN = r'\[.*]\s'
 _JENKINS_POOL_REGEX = re.compile('^'+_JENKINS_POOL_PATTERN)
 
 
 class _Matcher(object):
     '''Matching engine. Keeps data structures to match a single line against
     a fixed list of patterns.'''
     def __init__(self, patterns):
-        # Create a combined master regex combining all patterns.
+        # Create a combined main regex combining all patterns.
         combined = ''
         merge = ''
         for pattern in patterns:
             # Convert named groups into anonymous ones to avoid name clashes.
             p = pattern.pattern
             p = re.sub(r'\(\?P<[^>]+>', '(?:', p)
             combined += merge
             combined += '(?:%s)' % p

zorg/jenkins/jobs/jobs/clang-stage1-RA

+pipeline {
+    agent { label 'green-dragon-20' }
+
+    parameters {
+        string(name: 'GIT_REVISION', defaultValue: '*/main', description: 'Git revision to build')
+    }
+
+    stages {
+        stage('Checkout') {
+            steps {
+                dir('llvm-project') {
+                    checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'https://github.com/llvm/llvm-project.git']]])
+                }
+                dir('llvm-zorg') {
+                    checkout([$class: 'GitSCM', branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/llvm/llvm-zorg.git']]])
+                }
+            }
+        }
+        stage('Build') {
+            steps {
+                timeout(90) {
+                    sh '''
+                    set -u
+                    rm -rf build.properties
+
+                    cd llvm-project
+                    git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
+
+                    git_desc=$(git describe --match "first_commit")
+                    export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
+
+                    sha=$(echo ${git_desc} | cut -f 3 -d "-")
+                    export GIT_SHA=${sha:1}
+
+                    # Also save the LLVM_REV until LNT server is taught about GIT
+                    export LLVM_REV=$(git show -q | grep "llvm-svn:" | cut -f2 -d":" | tr -d " ")
+
+                    cd -
+
+                    echo "GIT_DISTANCE=$GIT_DISTANCE" > build.properties
+                    echo "GIT_SHA=$GIT_SHA" >> build.properties
+                    echo "ARTIFACT=$JOB_NAME/clang-d$GIT_DISTANCE-g$GIT_SHA-t$BUILD_ID-b$BUILD_NUMBER.tar.gz" >> build.properties
+
+                    export PATH=$PATH:/usr/bin:/usr/local/bin
+
+                    # Set a MacOS minimum deployment target
+                    export MACOSX_DEPLOYMENT_TARGET=10.9
+
+                    rm -rf clang-build clang-install *.tar.gz
+                    python llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \
+                      --assertions --cmake-type=RelWithDebInfo \
+                      --sccache --projects="clang;clang-tools-extra;compiler-rt;libcxx" \
+                      --cmake-flag='-DLIBCXX_ENABLE_SHARED=OFF'\
+                      --cmake-flag='-DLIBCXX_ENABLE_STATIC=OFF' \
+                      --cmake-flag='-DLIBCXX_INCLUDE_TESTS=OFF' \
+                      --cmake-flag='-DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY=OFF'
+                    '''
+                }
+            }
+        }
+        stage('Test') {
+            steps {
+                timeout(90) {
+                    sh '''
+                    set -u
+                    export PATH=$PATH:/usr/bin:/usr/local/bin
+                    python llvm-zorg/zorg/jenkins/monorepo_build.py cmake testlong
+                    '''
+                }
+                junit 'clang-build/**/testresults.xunit.xml'
+            }
+        }
+    }
+    post {
+        always {
+            scanForIssues tool: clang()
+        }
+        regression {
+            emailext subject: '$DEFAULT_SUBJECT',
+                    presendScript: '$DEFAULT_PRESEND_SCRIPT',
+                    postsendScript: '$DEFAULT_POSTSEND_SCRIPT',
+                    recipientProviders: [
+                            [$class: 'CulpritsRecipientProvider'],
+                            [$class: 'DevelopersRecipientProvider'],
+                            [$class: 'RequesterRecipientProvider'],
+                    ],
+                    replyTo: '$DEFAULT_REPLYTO',
+                    to: '$DEFAULT_RECIPIENTS',
+                    body:'$DEFAULT_CONTENT'
+        }
+        success {
+            build job: 'clang-stage2-Rthinlto_relay', wait: false
+            build job: 'clang-stage2-cmake-RgSan_relay', wait: false
+            //build job: 'phase2_modules_relay', wait: false
+            build job: 'relay-lnt-test-suite', wait: false
+            build job: 'relay-lnt-ctmark', wait: false
+            build job: 'relay-test-suite-verify-machineinstrs', wait: false
+        }
+        unstable {
+            build job: 'clang-stage2-Rthinlto_relay', wait: false
+            build job: 'clang-stage2-cmake-RgSan_relay', wait: false
+            //build job: 'phase2_modules_relay', wait: false
+            build job: 'relay-lnt-test-suite', wait: false
+            build job: 'relay-lnt-ctmark', wait: false
+            build job: 'relay-test-suite-verify-machineinstrs', wait: false
+        }
+    }
+}
+

zorg/jenkins/jobs/jobs/clang-stage1-cmake-RA-incremental

+pipeline {
+    agent { label 'green-dragon-07' }
+
+    parameters {
+        string(name: 'GIT_REVISION', defaultValue: '*/main', description: 'Git revision to build')
+    }
+
+    stages {
+        stage('Checkout') {
+            steps {
+                dir('llvm-project') {
+                    checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'https://github.com/llvm/llvm-project.git']]])
+                }
+                dir('llvm-zorg') {
+                    checkout([$class: 'GitSCM', branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/llvm/llvm-zorg.git']]])
+                }
+            }
+        }
+        stage('Build') {
+            steps {
+                timeout(90) {
+                    sh '''
+                    set -u
+                    rm -rf build.properties
+
+                    cd llvm-project
+                    git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
+
+                    git_desc=$(git describe --match "first_commit")
+
+                    export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
+
+                    sha=$(echo ${git_desc} | cut -f 3 -d "-")
+                    export GIT_SHA=${sha:1}
+
+                    # Also save the LLVM_REV until LNT server is taught about GIT
+                    export LLVM_REV=$(git show -q | grep "llvm-svn:" | cut -f2 -d":" | tr -d " ")
+
+                    cd -
+
+                    export PATH=$PATH:/usr/bin:/usr/local/bin
+
+                    # Set a MacOS minimum deployment target
+                    export MACOSX_DEPLOYMENT_TARGET=10.9
+
+                    python llvm-zorg/zorg/jenkins/monorepo_build.py cmake build --assertions --projects="clang"
+
+                    # Removing the local artifact
+                    rm -rf clang-*.tar.gz
+                    '''
+                }
+            }
+        }
+        stage('Test') {
+            steps {
+                timeout(90) {
+                    sh '''
+                    set -u
+                    export PATH=$PATH:/usr/bin:/usr/local/bin
+
+                    rm -rf clang-build/testresults.xunit.xml
+
+                    python llvm-zorg/zorg/jenkins/monorepo_build.py cmake testlong
+                    '''
+                }
+                junit 'clang-build/**/testresults.xunit.xml'
+            }
+        }
+    }
+    post {
+        always {
+            scanForIssues tool: clang()
+        }
+        /* Disabled email notification.
+        regression {
+            emailext subject: '$DEFAULT_SUBJECT',
+                    presendScript: '$DEFAULT_PRESEND_SCRIPT',
+                    postsendScript: '$DEFAULT_POSTSEND_SCRIPT',
+                    recipientProviders: [
+                            [$class: 'CulpritsRecipientProvider'],
+                            [$class: 'DevelopersRecipientProvider'],
+                            [$class: 'RequesterRecipientProvider'],
+                    ],
+                    replyTo: '$DEFAULT_REPLYTO',
+                    to: '$DEFAULT_RECIPIENTS',
+                    body:'$DEFAULT_CONTENT'
+        }
+        */
+    }
+}

zorg/jenkins/jobs/jobs/clang-stage2-Rthinlto

+pipeline {
+    agent { label 'green-dragon-22' }
+
+    options { disableResume() }
+
+    parameters {
+        string(name: 'GIT_SHA', defaultValue: '*/main', description: 'Git revision to build')
+        string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'description')
+    }
+
+    stages {
+        stage('Checkout') {
+            steps {
+                dir('llvm-project') {
+                    checkout([$class: 'GitSCM', branches: [[name: params.GIT_SHA]], userRemoteConfigs: [[url: 'https://github.com/llvm/llvm-project.git']]])
+                }
+                dir('llvm-zorg') {
+                    checkout([$class: 'GitSCM', branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/llvm/llvm-zorg.git']]])
+                }
+            }
+        }
+        stage('Build') {
+            steps {
+                timeout(600) {
+                    sh '''
+                    set -u
+                    rm -rf build.properties
+
+                    cd llvm-project
+                    git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
+
+                    git_desc=$(git describe --match "first_commit")
+                    export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
+
+                    sha=$(echo ${git_desc} | cut -f 3 -d "-")
+                    export GIT_SHA=${sha:1}
+
+                    cd -
+
+                    echo "ARTIFACT=$JOB_NAME/clang-d$GIT_DISTANCE-g$GIT_SHA-t$BUILD_ID-b$BUILD_NUMBER.tar.gz" > build.properties
+
+                    export PATH=$PATH:/usr/bin:/usr/local/bin
+
+                    rm -rf clang-build clang-install *.tar.gz
+
+                    python llvm-zorg/zorg/jenkins/monorepo_build.py fetch
+                    python llvm-zorg/zorg/jenkins/monorepo_build.py clang build \
+                      --thinlto --projects="clang;compiler-rt;libcxx" \
+                      --cmake-flag="-DLIBCXX_ENABLE_SHARED=OFF" \
+                      --cmake-flag="-DLIBCXX_ENABLE_STATIC=OFF" \
+                      --cmake-flag="-DLIBCXX_INCLUDE_TESTS=OFF" \
+                      --cmake-flag="-DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY=OFF" \
+                      --cmake-flag="-DCMAKE_DSYMUTIL=$WORKSPACE/host-compiler/bin/dsymutil" \
+                      --sccache
+                    '''
+                }
+            }
+        }
+
+        stage('Test') {
+            steps {
+                timeout(240) {
+                    sh '''
+                    set -u
+                    export PATH=$PATH:/usr/bin:/usr/local/bin
+
+                    rm -rf clang-build/testresults.xunit.xml
+
+                    python llvm-zorg/zorg/jenkins/monorepo_build.py clang test
+                    '''
+                }
+                junit 'clang-build/**/testresults.xunit.xml'
+            }
+        }
+    }
+    post {
+        always {
+            scanForIssues tool: clang()
+        }
+        regression {
+            emailext subject: '$DEFAULT_SUBJECT',
+                    presendScript: '$DEFAULT_PRESEND_SCRIPT',
+                    postsendScript: '$DEFAULT_POSTSEND_SCRIPT',
+                    recipientProviders: [
+                            [$class: 'CulpritsRecipientProvider'],
+                            [$class: 'DevelopersRecipientProvider'],
+                            [$class: 'RequesterRecipientProvider'],
+                    ],
+                    replyTo: '$DEFAULT_REPLYTO',
+                    to: '$DEFAULT_RECIPIENTS',
+                    body:'$DEFAULT_CONTENT'
+        }
+    }
+}

zorg/jenkins/jobs/jobs/lldb-cmake

 #!/usr/bin/env groovy
 pipeline {
     agent { label 'green-dragon-10' }
     parameters {
-        string(name: 'GIT_REVISION', defaultValue: '*/master', description: 'Git revision to build')
-        string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project')
+        string(name: 'GIT_REVISION', defaultValue: '*/main', description: 'Git revision to build')
+        string(name: 'ARTIFACT', defaultValue: 'clang-stage2-Rthinlto/latest', description: 'Compiler artifact to use for building the project')
         string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
         string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
     }
     stages {
         stage('Checkout') {
             steps {
                 timeout(30) {
                     dir('llvm-project') {
                         checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]])
                     }
                     dir('llvm-zorg') {
-                        checkout([$class: 'GitSCM', branches: [[name: '*/master']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
+                        checkout([$class: 'GitSCM', branches: [[name: '*/main']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
                     }
                 }
             }
         }
         stage('Fetch Host Compiler') {
         steps {
             timeout(10) {
                     sh '''
stages {
                     sha=$(echo ${git_desc} | cut -f 3 -d "-")
                     export GIT_SHA=${sha:1}
 
                     cd -
 
                     export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print $2}')
                     python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
                       --assertions \
-                      --projects="clang;libcxx;libcxxabi;compiler-rt;lld;lldb;debuginfo-tests"  \
+                      --projects="clang;libcxx;libcxxabi;compiler-rt;lld;lldb;cross-project-tests"  \
                       --compiler-flag="-Wdocumentation" \
                       --cmake-type=Release || (echo "Retrying. debuginfo-tests may have failed due to high load."; sleep 30; python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
                       --assertions \
-                      --projects="clang;libcxx;libcxxabi;compiler-rt;lld;lldb;debuginfo-tests"  \
+                      --projects="clang;libcxx;libcxxabi;compiler-rt;lld;lldb;cross-project-tests"  \
                       --compiler-flag="-Wdocumentation" \
                       --cmake-type=Release)
                       '''
                 }
             }
         }
         stage('Test') {
             steps {
                 timeout(90) {
                     sh '''
                     set -u
                     export PATH=$PATH:/usr/bin:/usr/local/bin
-                    export LLDB_CAPTURE_REPRODUCER=1
 
                     rm -rf test/results.xml
 
                     export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print $2}')
                     python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake testlong
                     '''
                 }
             }

zorg/jenkins/jobs/jobs/lldb-cmake-matrix

 #!/usr/bin/env groovy
 
 pipeline {
     agent { label 'green-dragon-23' }
     parameters {
-        string(name: 'GIT_REVISION', defaultValue: '*/master', description: 'Git revision to build')
-        string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project')
+        string(name: 'GIT_REVISION', defaultValue: '*/main', description: 'Git revision to build')
+        string(name: 'ARTIFACT', defaultValue: 'clang-stage2-Rthinlto/latest', description: 'Compiler artifact to use for building the project')
         string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
         string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
     }
     stages {
         stage('Checkout') {
             steps {
                 timeout(30) {
                     dir('llvm-project') {
                         checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]])
                     }
                     dir('llvm-zorg') {
-                        checkout([$class: 'GitSCM', branches: [[name: '*/master']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
+                        checkout([$class: 'GitSCM', branches: [[name: '*/main']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
                     }
                 }
             }
         }
         stage('Fetch') {
             steps {
                 timeout(10) {
                     sh '''
stages {
                     set +e
                     python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test
                     set -e
                     '''
                 }
                 junit 'test/results.xml'
             }
         }
+        stage('Test Reproducers') {
+            steps {
+                timeout(120) {
+                    sh '''
+                    export PATH=$PATH:/usr/bin:/usr/local/bin
+
+                    set +e
+                    python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake \
+                        --cmake-test-target check-lldb-reproducers
+                    set -e
+                    '''
+                }
+            }
+        }
     }
 }

zorg/jenkins/jobs/jobs/lldb-cmake-reproducers

-#!/usr/bin/env groovy
-pipeline {
-    agent { label 'green-dragon-23' }
-    parameters {
-        string(name: 'GIT_REVISION', defaultValue: '*/master', description: 'Git revision to build')
-        string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project')
-        string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
-        string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
-    }
-    stages {
-        stage('Checkout') {
-            steps {
-                timeout(30) {
-                    dir('llvm-project') {
-                        checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]])
-                    }
-                    dir('llvm-zorg') {
-                        checkout([$class: 'GitSCM', branches: [[name: '*/master']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
-                    }
-                }
-            }
-        }
-        stage('Fetch Host Compiler') {
-        steps {
-            timeout(10) {
-                    sh '''
-                    python llvm-zorg/zorg/jenkins/monorepo_build.py fetch
-                    '''
-                }
-            }
-        }
-        stage('Build') {
-            steps {
-                timeout(90) {
-                    sh '''
-                    set -u
-                    rm -rf build.properties
-
-                    cd llvm-project
-                    git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
-
-                    git_desc=$(git describe --match "first_commit")
-
-                    export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
-
-                    sha=$(echo ${git_desc} | cut -f 3 -d "-")
-                    export GIT_SHA=${sha:1}
-
-                    cd -
-
-                    python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
-                      --assertions \
-                      --projects="clang;libcxx;libcxxabi;compiler-rt;lld;lldb;debuginfo-tests"  \
-                      --compiler-flag="-Wdocumentation" \
-                      --cmake-type=Release
-                    '''
-                }
-            }
-        }
-        stage('Test Reproducers') {
-            steps {
-                timeout(120) {
-                    sh '''
-                    set -u
-                    export PATH=$PATH:/usr/bin:/usr/local/bin
-                    rm -rf test/results.xml
-                    python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake \
-                        --cmake-test-target check-lldb-reproducers
-                    '''
-                }
-            }
-        }
-    }
-    post {
-        always {
-            scanForIssues tool: clang()
-            junit 'test/results.xml'
-        }
-    }
-}

zorg/jenkins/jobs/jobs/lldb-cmake-sanitized

 #!/usr/bin/env groovy
 pipeline {
     agent { label 'green-dragon-23' }
     parameters {
-        string(name: 'GIT_REVISION', defaultValue: '*/master', description: 'Git revision to build')
-        string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project')
+        string(name: 'GIT_REVISION', defaultValue: '*/main', description: 'Git revision to build')
+        string(name: 'ARTIFACT', defaultValue: 'clang-stage2-Rthinlto/latest', description: 'Compiler artifact to use for building the project')
         string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
         string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
     }
     stages {
         stage('Checkout') {
             steps {
                 timeout(30) {
                     dir('llvm-project') {
                         checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]])
                     }
                     dir('llvm-zorg') {
-                        checkout([$class: 'GitSCM', branches: [[name: '*/master']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
+                        checkout([$class: 'GitSCM', branches: [[name: '*/main']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
                     }
                 }
             }
         }
         stage('Fetch Host Compiler') {
         steps {
             timeout(10) {
                     sh '''

zorg/jenkins/jobs/jobs/lldb-cmake-standalone

 #!/usr/bin/env groovy
 pipeline {
     agent {label 'green-dragon-23'}
     parameters {
-        string(name: 'GIT_REVISION', defaultValue: '*/master', description: 'Git revision to build')
-        string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project')
+        string(name: 'GIT_REVISION', defaultValue: '*/main', description: 'Git revision to build')
+        string(name: 'ARTIFACT', defaultValue: 'clang-stage2-Rthinlto/latest', description: 'Compiler artifact to use for building the project')
         string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
         string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
     }
     stages {
         stage('Checkout') {
             steps {
                 timeout(30) {
                     dir('llvm-project') {
                         checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]])
                     }
                     dir('llvm-zorg') {
-                        checkout([$class: 'GitSCM', branches: [[name: '*/master']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
+                        checkout([$class: 'GitSCM', branches: [[name: '*/main']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
                     }
                 }
             }
         }
         stage('Fetch') {
             steps {
               timeout(90) {
                   sh '''

zorg/jenkins/jobs/jobs/llvm-coverage

+#!/usr/bin/env groovy
+pipeline {
+    agent { label 'green-dragon-23' }
+
+    parameters {
+        string(name: 'GIT_REVISION', defaultValue: '*/main', description: 'Git revision to build')
+        string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project')
+        string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
+        string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
+    }
+
+    stages {
+        stage('Checkout') {
+            steps {
+                dir('llvm-project') {
+                    checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]])
+                }
+                dir('llvm-zorg') {
+                    checkout([$class: 'GitSCM', branches: [[name: '*/main']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
+                }
+            }
+        }
+        stage('Fetch') {
+            steps {
+                timeout(10) {
+                    sh '''
+                    python llvm-zorg/zorg/jenkins/monorepo_build.py fetch
+                    '''
+                }
+            }
+        }
+        stage('Build') {
+            steps {
+                timeout(300) {
+                    sh '''
+                    set -u
+                    rm -rf build.properties
+
+                    cd llvm-project
+                    git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
+
+                    git_desc=$(git describe --match "first_commit")
+
+                    export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
+
+                    sha=$(echo ${git_desc} | cut -f 3 -d "-")
+                    export GIT_SHA=${sha:1}
+
+                    cd -
+
+                    export PATH=$PATH:/usr/bin:/usr/local/bin
+
+                    python llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \
+                      --cmake-build-target FileCheck \
+                      --cmake-build-target count \
+                      --cmake-build-target not \
+                      --cmake-flag="-DLLVM_TARGETS_TO_BUILD=host" \
+                      --cmake-type=Release \
+                      --projects "" \
+                      --noinstall \
+                      --noupload
+
+                    python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
+                      --assertions \
+                      --projects="clang;libcxx;libcxxabi;lldb;cross-project-tests"  \
+                      --cmake-flag="-DPYTHON_LIBRARY=/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7.dylib" \
+                      --cmake-flag="-DPYTHON_INCLUDE_DIR=/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m" \
+                      --cmake-flag="-DPYTHON_LIBRARY_DEBUG=/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7.dylib" \
+                      --cmake-flag="-DPYTHON_EXECUTABLE=/usr/local/Cellar/python/3.7.0/bin/python3.7" \
+                      --cmake-flag="-DLLDB_USE_SYSTEM_DEBUGSERVER=ON" \
+                      --cmake-flag="-DLLVM_BUILD_INSTRUMENTED_COVERAGE=ON" \
+                      --cmake-flag="-DLLVM_PROFILE_MERGE_POOL_SIZE=1" \
+                      --cmake-flag="-DLLVM_ENABLE_MODULES=Off" \
+                      --cmake-type=Release \
+                      --noupload
+                    '''
+                }
+            }
+        }
+        stage('Test') {
+            steps {
+                timeout(300) {
+                    sh '''
+                    set -u
+                    export PATH=$PATH:/usr/bin:/usr/local/bin
+
+                    rm -rf test/results.xml
+
+                    BUILD_DIR=$(pwd)/lldb-build
+                    FAST_BUILD_DIR=$(pwd)/clang-build
+                    REPORT_DIR=$(pwd)/coverage-reports
+                    HOST=$(pwd)/host-compiler/bin
+                    LLVM_PROFDATA=$HOST/llvm-profdata
+                    LLVM_COV=$HOST/llvm-cov
+                    ARTIFACT_PREP_SCRIPT=$WORKSPACE/llvm-project/llvm/utils/prepare-code-coverage-artifact.py
+
+                    FAST_TOOLS=(FileCheck count not)
+                    for TOOL in ${FAST_TOOLS[@]}; do
+                      cp $FAST_BUILD_DIR/bin/$TOOL $BUILD_DIR/bin/$TOOL
+                    done
+
+                    # Clear out any stale profiles.
+                    rm -rf $BUILD_DIR/profiles
+
+                    # Run the tests.
+                    IGNORE_ERRORS_OVERRIDE=1 python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake testlong || echo "Some tests may have failed."
+
+                    cd $BUILD_DIR
+                    ninja -k 0 check-llvm check-clang || echo "Some tests may have failed."
+                    cd -
+
+                    COV_BINARIES=$(find $BUILD_DIR/bin $BUILD_DIR/lib -depth 1 -type f -exec file {} \\; | grep Mach-O | cut -d':' -f1 | grep -vE '/(FileCheck|count|not)$' | xargs)
+
+                    rm -rf $REPORT_DIR
+                    mkdir -p $REPORT_DIR
+                    python $ARTIFACT_PREP_SCRIPT $LLVM_PROFDATA $LLVM_COV $BUILD_DIR/profiles $REPORT_DIR $COV_BINARIES --unified-report --restrict $WORKSPACE/llvm-project
+
+                    scp -r $REPORT_DIR [e-mail]:/Library/WebServer/Documents/coverage
+                    ssh [e-mail] "chmod -R 777 /Library/WebServer/Documents/coverage"
+
+                    rm -rf lldb-build/profiles
+                    '''
+                }
+            }
+        }
+    }
+
+    post {
+        always {    
+            scanForIssues tool: clang()
+            junit 'test/results.xml'
+        }
+    }
+}

zorg/jenkins/jobs/jobs/lnt-ctmark-aarch64-O0-g

 #!/usr/bin/env groovy
 def common = evaluate readTrusted('zorg/jenkins/common.groovy')
 common.testsuite_pipeline(label: 'green-dragon-12') {
+    timeout(30) {
         sh """
 LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake"
 LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-O0-g.cmake"
 config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}"
         """
     }
+}

zorg/jenkins/jobs/jobs/lnt-ctmark-aarch64-O3-flto

 #!/usr/bin/env groovy
 def common = evaluate readTrusted('zorg/jenkins/common.groovy')
 common.testsuite_pipeline(label: 'green-dragon-12') {
+    timeout(30) {
         sh """
 LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake"
 LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-O3-flto.cmake"
 config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}"
         """
     }
+}

zorg/jenkins/jobs/jobs/lnt-ctmark-aarch64-Os

 #!/usr/bin/env groovy
 def common = evaluate readTrusted('zorg/jenkins/common.groovy')
 common.testsuite_pipeline(label: 'green-dragon-12') {
+    timeout(30) {
         sh """
 LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake"
 LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-Os.cmake"
 config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}"
         """
     }
+}

zorg/jenkins/jobs/jobs/lnt-ctmark-aarch64-Oz

 #!/usr/bin/env groovy
 def common = evaluate readTrusted('zorg/jenkins/common.groovy')
 common.testsuite_pipeline(label: 'green-dragon-12') {
+    timeout(30) {
         sh """
 LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake"
 LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-Oz.cmake"
 config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}"
         """
     }
+}

zorg/jenkins/jobs/util/make_pipeline.py

<scm class="hudson.plugins.git.GitSCM" plugin="git@3.6.1">
       <configVersion>2</configVersion>
       <userRemoteConfigs>
         <hudson.plugins.git.UserRemoteConfig>
           <url>{pipeline_git_url}</url>
         </hudson.plugins.git.UserRemoteConfig>
       </userRemoteConfigs>
       <branches>
         <hudson.plugins.git.BranchSpec>
-          <name>*/master</name>
+          <name>*/main</name>
         </hudson.plugins.git.BranchSpec>
       </branches>
       <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
       <submoduleCfg class="list"/>
       <extensions/>
     </scm>
     <scriptPath>{pipeline_git_path}</scriptPath>
     <lightweight>true</lightweight>

zorg/jenkins/monorepo_build.py

def liblto(self):
             assert os.path.exists(clang_liblto_path), "host-compiler present," \
                                                       " but has no liblto."
             return clang_liblto_path
         else:
             return False
 
     def branch(self):
         """Figure out the source branch name.
-     Not using GIT_BRANCH env var from Jenkins as that includes the
+           Not using GIT_BRANCH env var from Jenkins as that includes the
            remote name too.
         """
         if not os.environ.get('TESTING', False):
             try:
                 cmd = ['git', '-C', conf.srcdir(), 'symbolic-ref', '--short', 'HEAD']
                 out = run_collect_output(cmd, stderr=subprocess.STDOUT).strip()
             except subprocess.CalledProcessError:
                 cmd = ['git', '-C', conf.srcdir(), 'rev-parse', 'HEAD']
                 out = run_collect_output(cmd, stderr=subprocess.STDOUT).strip()
             return out
-        return 'master'
+        return 'main'
 
     def link_memory_usage(self):
         """Guesstimate the maximum link memory usage for this build.
-           We are only building master here so we will just use that value
+           We are only building main here so we will just use that value
         """
         #  Determinited experimentally.
-        usages = {'master': 3.5}
-        if self.branch() == 'master':
-            return usages['master']
+        usages = {'main': 3.5}
+        if self.branch() == 'main':
+            return usages['main']
         else:
             raise NotImplementedError(
                 "Unknown link memory usage." + self.branch())
 
 
 # Global storage for configuration object.
 conf = None  # type: Configuration
 
if target == 'all' or target == 'test' or target == 'testlong':
         targets = [
             'check-all'] if target == 'testlong' or target == 'all' else conf.cmake_test_targets
 
         if not targets:
             # testlong and all do check all, otherwise check and check-clang
             # unless the user asked for something else.
             targets = ['check', 'check-clang']
 
-        run_cmd(conf.builddir(), ninja_cmd + targets)
+        run_cmd(conf.builddir(), ninja_cmd + targets, err_okay=True)
         footer()
 
     if conf.sccache:
         run_ws([conf.sccache_path, "--stop-server"])
 
 
 def clang_builder(target):
     """Build to set of commands to compile and test apple-clang"""
if not conf.device and (target == "test" or target == "all"):
             obj_dir = os.path.join(conf.workspace, obj_dir)
 
         cmd = [NINJA, '-v', '-k', '0', 'check-all']
 
         if conf.assertions:
             cmd[-1] += ' --param use_gmalloc=1 ' \
                        '--param gmalloc_path=$(xcodebuild -find-library' \
                        ' libgmalloc.dylib)'
-        run_cmd(obj_dir, cmd, env={'MALLOC_LOG_FILE': '/dev/null'})
+        run_cmd(obj_dir, cmd, env={'MALLOC_LOG_FILE': '/dev/null'}, err_okay=True)
 
     if conf.sccache:
         run_ws([conf.sccache_path, "--stop-server"])
 
 
 def parse_settings_from_output(working_dir, cmd):
     old_dir = os.getcwd()
     try:
def lldb_cmake_builder(target, variant=None):
     # Construct dotest.py arguments.
     dotest_args=['--arch', 'x86_64', '--build-dir',
                  conf.lldbbuilddir()+'/lldb-test-build.noindex',
                  '-t', '--env', 'TERM=vt100']
     dotest_args.extend(conf.dotest_flags)
 
     # Construct lit arguments.
     lit_args = ['-v', '--time-tests', '--shuffle',
-    '--xunit-xml-output={}'.format(results_file), '-v']
+                '--xunit-xml-output={}'.format(results_file), '-v']
     if conf.max_parallel_tests:
         lit_args.extend(['-j', conf.max_parallel_tests])
     if variant == 'sanitized':
         lit_args.extend(['--timeout 1200'])
 
     # Construct CMake invocation.
     cmake_cmd = ["/usr/local/bin/cmake", '-G', 'Ninja',
                  conf.llvmsrcdir(),
def lldb_cmake_builder(target, variant=None):
 
     if variant == 'sanitized':
         cmake_cmd.append('-DLLVM_TARGETS_TO_BUILD=X86')
         cmake_cmd.append('-DLLVM_USE_SANITIZER=Address;Undefined')
         # There is no need to compile the lldb tests with an asanified compiler
         # if we have a host compiler available.
         if conf.CC():
             cmake_cmd.extend([
-                '-DLLDB_TEST_C_COMPILER=' + conf.CC(),
-                '-DLLDB_TEST_CXX_COMPILER=' + conf.CC() + "++"
+                '-DLLDB_TEST_COMPILER=' + conf.CC(),
             ])
 
     if conf.compiler_flags:
         cmake_cmd.extend([
             "-DCMAKE_C_FLAGS={}".format(' '.join(conf.compiler_flags)),
             "-DCMAKE_CXX_FLAGS={}".format(' '.join(conf.compiler_flags))
         ])
 
     if conf.lldbtestcompiler():
         cmake_cmd.extend([
-            '-DLLDB_TEST_C_COMPILER=' + conf.lldbtestcompiler(),
-            '-DLLDB_TEST_CXX_COMPILER=' + conf.lldbtestcompiler() + "++"
+            '-DLLDB_TEST_COMPILER=' + conf.lldbtestcompiler(),
         ])
 
     cmake_cmd.extend(conf.cmake_flags)
 
     if conf.CC():
         cmake_cmd.extend([
             '-DCMAKE_C_COMPILER=' + conf.CC(),
             '-DCMAKE_CXX_COMPILER=' + conf.CC() + "++"
if target == 'all' or target == 'build':
         run_cmd(conf.lldbbuilddir(), [NINJA, '-v'])
         footer()
 
     if target == 'all' or target == 'install':
         header("Install")
         run_cmd(conf.lldbbuilddir(), [NINJA, '-v', 'install'])
         footer()
 
-    if target == 'all' or target == 'testlong':
-        header("Run Debug Info Tests")
-        run_cmd(conf.lldbbuilddir(), [NINJA, '-v', 'check-debuginfo'])
-        footer()
-
     if target == 'all' or target == 'test' or target == 'testlong':
         header("Run Tests")
-        run_cmd(conf.lldbbuilddir(),
-                ['/usr/bin/env', 'TERM=vt100', NINJA, '-v', 'check-lldb'])
+        if variant == 'matrix' or variant == 'sanitized':
+            test_command = ['/usr/bin/env', 'TERM=vt100', NINJA,
+                            '-v', 'check-lldb']
+        else:
+            test_command = ['/usr/bin/env', 'TERM=vt100', NINJA,
+                            '-v', 'check-debuginfo', 'check-lldb', '-k2']
+        run_cmd(conf.lldbbuilddir(), test_command)
         footer()
 
     for test_target in conf.cmake_test_targets:
         header("Run Custom Test: {0}".format(test_target))
         run_cmd(conf.lldbbuilddir(), [NINJA, '-k', '0', '-v', test_target])
         footer()
 
 
def http_download(url, dest):
     """
     try:
         print("GETting", url, "to", dest, "...")
         f = urllib.request.urlopen(url)
         # Open our local file for writing
         with open(dest, "wb") as local_file:
             local_file.write(f.read())
 
-    except urllib.HTTPError as e:
+    except urllib.error.HTTPError as e:
         print("HTTP Error:", e.code, url)
         sys.exit(1)
 
-    except urllib.URLError as e:
+    except urllib.error.URLError as e:
         print("URL Error:", e.reason, url)
         sys.exit(1)
     print("done.")
 
 
 def create_builddirs():
     create_dirs([conf.builddir(), conf.installdir()]);