This is an archive of the discontinued LLVM Phabricator instance.

Differential D154406

Support Windows test
Needs ReviewPublic

Authored by yinyangsx on Jul 3 2023, 7:44 PM.

Details

Reviewers
gkistanova
aaron.ballman
bollu
jdoerfert
sscalpone
aartbik

Diff Detail

Event Timeline

yinyangsx created this revision.Jul 3 2023, 7:44 PM
Herald added a reviewer: gkistanova. · View Herald TranscriptJul 3 2023, 7:44 PM
Herald added a project: Restricted Project. · View Herald Transcript
yinyangsx requested review of this revision.Jul 3 2023, 7:44 PM
Harbormaster completed remote builds in B242922: Diff 536937.Jul 3 2023, 7:44 PM
yinyangsx mentioned this in D151980: Add lit test for Win.Jul 5 2023, 6:43 PM
aaron.ballman added a reviewer: aaron.ballman.Jul 6 2023, 6:31 AM
aaron.ballman added a subscriber: aaron.ballman.
aaron.ballman added inline comments.
buildbot/osuosl/master/config/builders.py
859

Something smells off, but this is not my area of expertise so I could be wrong. We're not checking out compiler-rt, but below we have:
"-DLLVM_ENABLE_RUNTIMES=compiler-rt", which seems like a conflict.

yinyangsx added inline comments.Jul 6 2023, 11:34 PM
buildbot/osuosl/master/config/builders.py
859

It does look strange, although the same configuration works well under linux builder. Let us modify one line and run the test? what do you think?

aaron.ballman added inline comments.Jul 7 2023, 4:38 AM
buildbot/osuosl/master/config/builders.py
859

I think you should try changing checkout_compiler_rt= to True and see if the full test suite still passes for you.

yinyangsx updated this revision to Diff 540863.Jul 16 2023, 10:39 PM

Support Windows test

Harbormaster completed remote builds in B245726: Diff 540863.Jul 16 2023, 10:39 PM
aaron.ballman added inline comments.Jul 17 2023, 10:51 AM
buildbot/osuosl/master/config/builders.py
860

I think this still needs to be set to True so that we check out the latest compiler-rt, since we're enabling the runtime for it below.

yinyangsx added inline comments.Jul 17 2023, 6:29 PM
buildbot/osuosl/master/config/builders.py
860

In our email thread we agree that “compiler-rt“ should be removed from the LLVM_ENABLE_PROJECTS option. So check_compiler_rt has to be False. Otherwise, "compiler-t" will be added to LLVM_ENABLE_PROJECTS option.

aaron.ballman added a subscriber: phosek.Jul 18 2023, 5:22 AM
aaron.ballman added inline comments.
buildbot/osuosl/master/config/builders.py
860

In our email thread we agree that “compiler-rt“ should be removed from the LLVM_ENABLE_PROJECTS option. So check_compiler_rt has to be False. Otherwise, "compiler-t" will be added to LLVM_ENABLE_PROJECTS option.

Huh, that seems surprising to me. My understanding is that compiler-rt should never be added to LLVM_ENABLE_PROJECTS, it should only be added to LLVM_ENABLE_RUNTIMES to enable it. And if we're enabling it, we should be checking out the latest version so it's in step with changes to Clang.

CC @phosek for additional insights on the CMake incantations.

gkistanova added a comment.Aug 8 2023, 10:28 PM

Hi Yin,

I'm confused. There are multiple reviews for the same patch or very similar patches. Some already have been reviewed. Could you mark abandoned those you do not need any more and have a single one for the review, please?

DoyleLi added a subscriber: DoyleLi.Aug 16 2023, 8:10 PM
DoyleLi updated this revision to Diff 551000.Aug 16 2023, 10:35 PM
  • Resolve issues for missing files lib/clang/18/lib/windows/liborc_rt-x86_64.a
Herald added a reviewer: bollu. · View Herald TranscriptAug 16 2023, 10:35 PM
Herald added a reviewer: jdoerfert. · View Herald Transcript
Herald added a reviewer: sscalpone. · View Herald Transcript
Herald added a reviewer: aartbik. · View Herald Transcript
Herald added subscribers: wangpc, bviyer, jplehr and 38 others. · View Herald Transcript
Harbormaster completed remote builds in B253122: Diff 551000.Aug 16 2023, 10:35 PM
DoyleLi updated this revision to Diff 551002.Aug 16 2023, 10:42 PM

Revert the head commit of D154406

Harbormaster completed remote builds in B253124: Diff 551002.Aug 16 2023, 10:42 PM
DoyleLi updated this revision to Diff 551005.Aug 16 2023, 11:05 PM

Revert to last good commit.

Harbormaster completed remote builds in B253127: Diff 551005.Aug 16 2023, 11:05 PM
DoyleLi added a comment.Aug 16 2023, 11:09 PM

I tried different ways to revert the patch to last good commit, 540863.
But not wure why it still bring up huge difference within the patch.
So this one should be closed and I will submit a new one later.
Very sorry for inconvenient due to my experience on arc related commands.

DoyleLi updated this revision to Diff 551010.Aug 16 2023, 11:23 PM

Try to rever to last good commit, 540863

Harbormaster completed remote builds in B253130: Diff 551010.Aug 16 2023, 11:23 PM
DoyleLi added a comment.Aug 16 2023, 11:29 PM

New patch submitted to:
https://reviews.llvm.org/D158146

DoyleLi mentioned this in D158146: Support Windows test on intel64 AVX512.Aug 16 2023, 11:36 PM

Revision Contents

PathSize
7 lines
buildbot/
google/
docker/
buildbot-clangd-ubuntu-clang/
42 lines
2 lines
18 lines
buildbot-cpp20/
70 lines
1 line
buildbot-cpp20/
buildbot-clangd-ubuntu-clang/
23 lines
buildbot-mlir-nvidia/
buildbot-sanitizer-windows/
21 lines
1 line
1 line
26 lines
51 lines
terraform/
10 lines
58 lines
59 lines
332 lines
mlir-nvidia/
109 lines
mlir-nvidia/
google/
docker/
buildbot-mlir-nvidia/
23 lines
osuosl/
master/
config/
2 lines
10 lines
2447 lines
276 lines
126 lines
257 lines
279 lines
17 lines
llvmbisect/
llvmlab/
28 lines
33 lines
tasks/
cmake/
caches/
util/
6 lines
tasktool/
4 lines
tasktool/
repos/
2 lines
utils/
2 lines
test/
jenkins/
18 lines
zorg/
buildbot/
builders/
4 lines
18 lines
128 lines
58 lines
37 lines
122 lines
5 lines
8 lines
105 lines
2 lines
186 lines
37 lines
29 lines
211 lines
108 lines
35 lines
9 lines
annotated/
26 lines
12 lines
114 lines
111 lines
libc-windows.py
libc-linux.py
42 lines
250 lines
194 lines
85 lines
sanitizers/
17 lines
87 lines
42 lines
14 lines
11 lines
34 lines
63 lines
11 lines
184 lines
36 lines
443 lines
35 lines
393 lines
66 lines
33 lines
12 lines
changes/
31 lines
commands/
40 lines
26 lines
process/
62 lines
util/
102 lines
110 lines
jenkins/
10 lines
3 lines
13 lines
2 lines
jobs/
jobs/
103 lines
119 lines
104 lines
103 lines
162 lines
173 lines
350 lines
47 lines
44 lines
155 lines
8 lines
8 lines
8 lines
8 lines
3 lines
2 lines
2 lines
2 lines
2 lines
2 lines
2 lines
util/
2 lines
142 lines
17 lines

Diff 551010

.pylintrc

  • This file was added.
# add root folder to Pylint PYTHONPATH, so builders can be checked with zorg.*
# packages
[MASTER]
init-hook='import sys; sys.path.append(".")'
[FORMAT]
max-line-length=120
No newline at end of file

buildbot/google/docker/buildbot-clangd-ubuntu-clang/Dockerfile

#===-- Dockerfile --------------------------------------------------------===// #===-- Dockerfile --------------------------------------------------------===//
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information. # See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
# #
#===----------------------------------------------------------------------===// #===----------------------------------------------------------------------===//
# Docker image used for the mlir-nvidia builder # Docker image used for the clangd builder
# #
# Environment variables configurable at runtime: # Environment variables configurable at runtime:
# BUILDBOT_PORT - server port to connect to # BUILDBOT_PORT - server port to connect to
#===----------------------------------------------------------------------===// #===----------------------------------------------------------------------===//
# Use an official Ubuntu image as the base. # Use an official Ubuntu image as the base.
FROM ubuntu:18.04 FROM ubuntu:18.04
# Install build tools. # Install build tools.
RUN apt-get update; \ RUN apt-get update && \
apt-get install -y software-properties-common apt-transport-https ca-certificates \ apt-get install -y software-properties-common apt-transport-https ca-certificates \
ninja-build git wget gnupg \ ninja-build git wget gnupg \
# Install ccache for local caching of builds. # Install ccache for local caching of builds.
ccache \ ccache \
# Install python for buildbot and LIT. # Install python for buildbot and LIT.
python3 python3-pip python3-psutil ; \ python3 python3-pip python3-psutil && \
# Clean apt cache to reduce image size. # Clean apt cache to reduce image size.
apt-get clean apt-get clean
# install latest LLVM release # install latest LLVM release
RUN bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)";\ RUN wget https://apt.llvm.org/llvm.sh && \
chmod +x llvm.sh && \
./llvm.sh 11 && \
rm -rf llvm.sh && \
# clean apt cache to reduce image size # clean apt cache to reduce image size
apt-get clean apt-get clean
# configure default versions of LLVM tools # configure default versions of LLVM tools
RUN update-alternatives --install /usr/bin/clang clang /usr/bin/clang-11 100 ;\ RUN update-alternatives --install /usr/bin/clang clang /usr/bin/clang-11 100 ;\
update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-11 100 ;\ update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-11 100 ;\
update-alternatives --install /usr/bin/lld lld /usr/bin/lld-11 100 update-alternatives --install /usr/bin/ld.lld ld.lld /usr/bin/lld-11 100
# Configure LLVM tools.
ENV CC=clang
ENV CXX=clang++
ENV LD=LLD
# Ubuntu ships with old cmake version, install the latest one # Ubuntu ships with old cmake version, install the latest one
# from https://apt.kitware.com/ # from https://apt.kitware.com/
RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | \ RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | \
gpg --dearmor - | \ gpg --dearmor - | \
tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null ;\ tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null && \
apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' ;\ apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' && \
apt-get update ;\ apt-get update && \
apt-get install -y cmake ;\ apt-get install -y cmake && \
# Clean apt cache to reduce image size. # Clean apt cache to reduce image size.
apt-get clean apt-get clean
# Install build bot (server was at 2.8.5-dev at time of writing). # Install build bot (server was at 2.8.5-dev at time of writing).
RUN pip3 install buildbot-worker==2.8.4 RUN pip3 install buildbot-worker==2.8.4
# Build and install gRPC for clangd remote index, based on instructions # Build and install gRPC for clangd remote index, based on instructions
# https://github.com/llvm/llvm-project/tree/master/clang-tools-extra/clangd/index/remote#building-from-sources # https://github.com/llvm/llvm-project/tree/master/clang-tools-extra/clangd/index/remote#building-from-sources
# The grpc version shipped with Ubuntu 18.04 crashes on TSan checks. # The grpc version shipped with Ubuntu 18.04 crashes on TSan checks.
ENV GRPC_INSTALL_PATH=/usr/local/lib/grpc ENV GRPC_INSTALL_PATH=/usr/local/lib/grpc
RUN cd /tmp ; \ RUN cd /tmp ; \
git clone -b v1.33.2 --depth 1 --recursive --shallow-submodules https://github.com/grpc/grpc ; \ git clone -b v1.36.3 --depth 1 --recursive --shallow-submodules https://github.com/grpc/grpc && \
cd grpc ; \ cd grpc && mkdir build && cd build && \
mkdir build; cd build ; \ cmake -G Ninja -DgRPC_INSTALL=ON -DCMAKE_INSTALL_PREFIX=${GRPC_INSTALL_PATH} -DCMAKE_BUILD_TYPE=Release .. && \
cmake -G Ninja -DgRPC_INSTALL=ON -DCMAKE_INSTALL_PREFIX=${GRPC_INSTALL_PATH} -DCMAKE_BUILD_TYPE=Release .. ; \ cmake --build . --target install && \
cmake --build . --target install ; \
# Clean up sources to reduce image size. # Clean up sources to reduce image size.
cd / ; rm -rf /tmp/grpc cd / && rm -rf /tmp/grpc
# Workaround permissions issues when writing to named volumes # Workaround permissions issues when writing to named volumes
# https://github.com/docker/compose/issues/3270#issuecomment-206214034 # https://github.com/docker/compose/issues/3270#issuecomment-206214034
RUN mkdir -p /vol/test /vol/ccache /vol/worker ; \ RUN mkdir -p /vol/test /vol/ccache /vol/worker ; \
chmod -R 777 /vol chmod -R 777 /vol
# Volume to mount secrets into the container. # Volume to mount secrets into the container.
VOLUME /vol/secrets VOLUME /vol/secrets
Show All 18 Lines
# Allow the server port of this agent to be configurable during deployment. # Allow the server port of this agent to be configurable during deployment.
# This way we can connect the same image to production and integration. # This way we can connect the same image to production and integration.
# Ports: # Ports:
# 9990 - production # 9990 - production
# 9994 - integration # 9994 - integration
ENV BUILDBOT_PORT="9994" ENV BUILDBOT_PORT="9994"
# Configure LLVM tools.
ENV CC=clang
ENV CXX=clang++
ENV LD=LLD
# Run startup script. # Run startup script.
CMD /home/buildbot/run.sh CMD /home/buildbot/run.sh

buildbot/google/docker/buildbot-clangd-ubuntu-clang/VERSION

3 7

buildbot/google/docker/buildbot-clangd-ubuntu-clang/run.sh

  • This file was copied to buildbot/google/docker/buildbot-cpp20/run.sh.
#!/bin/bash #!/bin/bash
#===-- run.sh -------------------------------------------------------------===// #===-- run.sh -------------------------------------------------------------===//
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information. # See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
# #
#===----------------------------------------------------------------------===// #===----------------------------------------------------------------------===//
# This script will start the buildbot worker # This script will start the buildbot worker
#
#===----------------------------------------------------------------------===// #===----------------------------------------------------------------------===//
set -eu set -eu
# Read the worker password from a mounted file. # Read the worker password from a mounted file.
WORKER_PASSWORD=$(cat /vol/secrets/token) WORKER_PASSWORD=$(cat /vol/secrets/token)
# Set up buildbot host and maintainer info. # Set up buildbot host and maintainer info.
mkdir -p "${WORKER_NAME}/info/" mkdir -p "${WORKER_NAME}/info/"
echo "Christian Kuehnel <kuhnel@google.com>" > "${WORKER_NAME}/info/admin" echo "Christian Kuehnel <kuhnel@google.com>" > "${WORKER_NAME}/info/admin"
# generate the host information of this worker # generate the host information of this worker
( (
uname -a ; \ uname -a ; \
cat /proc/cpuinfo | grep "model name" | head -n1 | cut -d " " -f 3- ;\ cat /proc/cpuinfo | grep "model name" | head -n1 | cut -d " " -f 3- ;\
echo "number of cores: $(nproc)" ;\ echo "number of cores: $(nproc)" ;\
cat /proc/meminfo | grep MemTo ;\ cat /proc/meminfo | grep MemTo ;\
lsb_release -d | cut -f 2- ; \ lsb_release -d | cut -f 2- ; \
clang --version | head -n1 ; \ clang --version | head -n1 ; \
ld.lld-11 --version ; \ ld.lld --version ; \
cmake --version | head -n1 ; \ cmake --version | head -n1 ; \
) > ${WORKER_NAME}/info/host ) > ${WORKER_NAME}/info/host
echo "Host information:" echo "Host information:"
cat ${WORKER_NAME}/info/host cat ${WORKER_NAME}/info/host
# create the folder structure # create the folder structure
echo "creating worker ${WORKER_NAME} at port ${BUILDBOT_PORT}..." echo "creating worker ${WORKER_NAME} at port ${BUILDBOT_PORT}..."
buildbot-worker create-worker --keepalive=200 "${WORKER_NAME}" \ buildbot-worker create-worker --keepalive=200 "${WORKER_NAME}" \
lab.llvm.org:${BUILDBOT_PORT} "${WORKER_NAME}" "${WORKER_PASSWORD}" lab.llvm.org:${BUILDBOT_PORT} "${WORKER_NAME}" "${WORKER_PASSWORD}"
# start the daemon, this command return immetiately # start the daemon, this command returns immediately
echo "starting worker..." echo "starting worker..."
buildbot-worker start "${WORKER_NAME}" set +e
if ! buildbot-worker start "${WORKER_NAME}"; then
echo ERROR: starting worker failed. contents of twistd.log:
cat "${WORKER_NAME}/twistd.log"
exit 1
fi
# To keep the container running and produce log outputs: dump the worker # To keep the container running and produce log outputs: dump the worker
# log to stdout # log to stdout
echo "Following worker log..." echo "Following worker log..."
tail -f ${WORKER_NAME}/twistd.log tail -f ${WORKER_NAME}/twistd.log

buildbot/google/docker/buildbot-cpp20/Dockerfile

  • This file was added.
#===-- Dockerfile --------------------------------------------------------===//
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#
#===----------------------------------------------------------------------===//
# Environment variables configurable at runtime:
# BUILDBOT_PORT - server port to connect to
#===----------------------------------------------------------------------===//
FROM debian:sid
# Install build tools.
RUN apt-get update && \
apt-get install -y \
software-properties-common apt-transport-https ca-certificates \
git wget gnupg \
cmake ninja-build \
ccache \
python3 python3-pip python3-psutil && \
# Install clang-15 to make the minimal C++20 support in the toolchain.
echo "deb http://apt.llvm.org/unstable/ llvm-toolchain-15 main" >> /etc/apt/sources.list.d/llvm-15.list && \
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key >> /etc/apt/trusted.gpg.d/llvm.asc && \
apt-get update && \
apt-get install -y clang-15 lld-15 && \
apt-get clean
# Install build bot (server was at 2.8.5-dev at time of writing).
RUN pip3 install buildbot-worker==2.8.4
# Workaround permissions issues when writing to named volumes
# https://github.com/docker/compose/issues/3270#issuecomment-206214034
RUN mkdir -p /vol/test /vol/ccache /vol/worker ; \
chmod -R 777 /vol
# Volume to mount secrets into the container.
VOLUME /vol/secrets
# Volume to store data for local, manual testing of the container.
VOLUME /vol/test
# Volume to store ccache.
VOLUME /vol/ccache
ENV CCACHE_DIR=/vol/ccache
# Volume for worker working directory.
VOLUME /vol/worker
# Create user account, some tests fail if run as root.
RUN useradd buildbot --create-home
WORKDIR /vol/worker
# Copy startup script.
COPY run.sh /home/buildbot/
RUN chmod a+rx /home/buildbot/run.sh
USER buildbot
ENV WORKER_NAME="clang-debian-cpp20"
# Allow the server port of this agent to be configurable during deployment.
# This way we can connect the same image to production and integration.
# Ports:
# 9990 - production
# 9994 - integration
ENV BUILDBOT_PORT="9994"
# Configure LLVM tools.
ENV CC=clang-15
ENV CXX=clang++-15
ENV LD=ld.lld-15
# Run startup script.
CMD /home/buildbot/run.sh

buildbot/google/docker/buildbot-cpp20/VERSION

  • This file was added.
4

buildbot/google/docker/buildbot-cpp20/run.sh

  • This file was copied from buildbot/google/docker/buildbot-clangd-ubuntu-clang/run.sh.
#!/bin/bash #!/bin/bash
#===-- run.sh -------------------------------------------------------------===// #===-- run.sh -------------------------------------------------------------===//
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information. # See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
# #
#===----------------------------------------------------------------------===// #===----------------------------------------------------------------------===//
# This script will start the buildbot worker # This script will start the buildbot worker
# #
#===----------------------------------------------------------------------===// #===----------------------------------------------------------------------===//
set -eu set -eu
# Read the worker password from a mounted file. # Read the worker password from a mounted file.
WORKER_PASSWORD=$(cat /vol/secrets/token) WORKER_PASSWORD=$(cat /vol/secrets/token)
# Set up buildbot host and maintainer info. # Set up buildbot host and maintainer info.
mkdir -p "${WORKER_NAME}/info/" mkdir -p "${WORKER_NAME}/info/"
echo "Christian Kuehnel <kuhnel@google.com>" > "${WORKER_NAME}/info/admin" echo "Ilya Biryukov <ibiryukov@google.com>" > "${WORKER_NAME}/info/admin"
# generate the host information of this worker # generate the host information of this worker
( (
uname -a ; \ uname -a ; \
cat /proc/cpuinfo | grep "model name" | head -n1 | cut -d " " -f 3- ;\ cat /proc/cpuinfo | grep "model name" | head -n1 | cut -d " " -f 3- ;\
echo "number of cores: $(nproc)" ;\ echo "number of cores: $(nproc)" ;\
cat /proc/meminfo | grep MemTo ;\ cat /proc/meminfo | grep MemTo ;\
lsb_release -d | cut -f 2- ; \ lsb_release -d | cut -f 2- ; \
clang --version | head -n1 ; \ $CC --version | head -n1 ; \
ld.lld-11 --version ; \ $LD --version ; \
cmake --version | head -n1 ; \ cmake --version | head -n1 ; \
) > ${WORKER_NAME}/info/host ) > ${WORKER_NAME}/info/host
echo "Host information:" echo "Host information:"
cat ${WORKER_NAME}/info/host cat ${WORKER_NAME}/info/host
# create the folder structure # create the folder structure
echo "creating worker ${WORKER_NAME} at port ${BUILDBOT_PORT}..." echo "creating worker ${WORKER_NAME} at port ${BUILDBOT_PORT}..."
buildbot-worker create-worker --keepalive=200 "${WORKER_NAME}" \ buildbot-worker create-worker --keepalive=200 "${WORKER_NAME}" \
lab.llvm.org:${BUILDBOT_PORT} "${WORKER_NAME}" "${WORKER_PASSWORD}" lab.llvm.org:${BUILDBOT_PORT} "${WORKER_NAME}" "${WORKER_PASSWORD}"
# start the daemon, this command return immetiately # start the daemon, this command returns immediately
echo "starting worker..." echo "starting worker..."
buildbot-worker start "${WORKER_NAME}" set +e
if ! buildbot-worker start "${WORKER_NAME}"; then
echo ERROR: starting worker failed. contents of twistd.log:
cat "${WORKER_NAME}/twistd.log"
exit 1
fi
# To keep the container running and produce log outputs: dump the worker # To keep the container running and produce log outputs: dump the worker
# log to stdout # log to stdout
echo "Following worker log..." echo "Following worker log..."
tail -f ${WORKER_NAME}/twistd.log tail -f ${WORKER_NAME}/twistd.log

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile

  • This file was deleted.
#===-- Dockerfile --------------------------------------------------------===//
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#
#===----------------------------------------------------------------------===//
# Docker image used for the mlir-nvidia builder
#
# Environment variables configurable at runtime:
# BUILDBOT_PORT - server port to connect to
#===----------------------------------------------------------------------===//
# Use the image from NVIDIA as base
FROM nvidia/cuda:10.2-base-ubuntu18.04
# install build tools
RUN apt-get update; \
apt-get install -y software-properties-common apt-transport-https ca-certificates \
clang-8 lld-8 ninja-build git wget gnupg ccache \
python3 python3-pip python3-psutil pybind11-dev \
# dumb-init recommended in
# https://hub.docker.com/r/buildbot/buildbot-worker/dockerfile
dumb-init;\
update-alternatives --install /usr/bin/clang clang /usr/bin/clang-8 100 ;\
update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-8 100 ;\
update-alternatives --install /usr/bin/lld lld /usr/bin/lld-8 100
# install cuda
# avoid popups for keyboard configurations
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y cuda
# Ubuntu ships with old cmake version, install the latest one
# from https://apt.kitware.com/
RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | \
gpg --dearmor - | \
tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null ;\
apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' ;\
apt-get update ;\
apt-get install -y cmake
# Get the Vulkan SDK from LunarG. The CUDA meta package already pulls in NVIDIA
# Vulkan ICD.
RUN wget -qO - http://packages.lunarg.com/lunarg-signing-key-pub.asc 2>/dev/null | \
apt-key add - ;\
wget -qO /etc/apt/sources.list.d/lunarg-vulkan-1.2.141-bionic.list \
http://packages.lunarg.com/vulkan/1.2.141/lunarg-vulkan-1.2.141-bionic.list;\
apt-get update ;\
apt-get install -y vulkan-sdk
# Install build bot (server was at 2.8.5-dev at time of writing).
RUN pip3 install buildbot-worker==2.8.4
# Workaround permissions issues when writing to named volumes
# https://github.com/docker/compose/issues/3270#issuecomment-206214034
RUN mkdir -p /vol/test /vol/ccache /vol/worker ; \
chmod -R 777 /vol
# Volume to mount secrets into the container.
VOLUME /vol/secrets
# Volume to store data for local, manual testing of the container.
VOLUME /vol/test
# Volume to store ccache.
VOLUME /vol/ccache
ENV CCACHE_DIR=/vol/ccache
# Volume for worker working directory.
VOLUME /vol/worker
# Create user account, some tests fail if run as root.
RUN useradd buildbot --create-home
WORKDIR /vol/worker
# copy startup script
COPY run.sh /home/buildbot/
RUN chmod a+rx /home/buildbot/run.sh
USER buildbot
ENV WORKER_NAME="mlir-nvidia"
# Allow the server port of this agent to be configurable during deployment.
# This way we can connect the same image to production and integration.
# Ports:
# 9990 - production
# 9994 - integration
ENV BUILDBOT_PORT="9994"
CMD /home/buildbot/run.sh

buildbot/google/docker/buildbot-mlir-nvidia/VERSION

  • This file was deleted.
15

buildbot/google/docker/buildbot-mlir-nvidia/run.sh

  • This file was moved to buildbot/mlir-nvidia/run.sh.

buildbot/google/docker/buildbot-sanitizer-windows/Dockerfile

  • This file was added.
# escape=`
FROM gcr.io/sanitizer-bots/windows-base-vscode2019:8
ENV WORKER_NAME=sanitizer-windows
# copy script to start the agent
COPY run.ps1 .
COPY fix_buildbot_tac_paths.py .
# Set location for sccache cache
ENV SCCACHE_DIR="C:\volumes\sccache"
# Set the maximum sccache local cache size
ENV SCCACHE_CACHE_SIZE="10G"
# Move buildbot work dir to a volume to avoid of disc issues
VOLUME C:\volumes\buildbot
# Configure 32bit tools ("x86") instead of 64bit ("amd64")
CMD ["powershell.exe", "-NoLogo", "-ExecutionPolicy", "Bypass", `
"c:\\buildbot\\run.ps1"]

buildbot/google/docker/buildbot-sanitizer-windows/VERSION

  • This file was added.
1

buildbot/google/docker/buildbot-sanitizer-windows/admin

  • This file was added.
Reid Kleckner <rnk@google.com>

buildbot/google/docker/buildbot-sanitizer-windows/fix_buildbot_tac_paths.py

  • This file was added.
#!/usr/bin/env python3
# Replace backward slashes in the paths in a "buildbot.tac" file
import argparse
import re
import shutil
def fix_slashes(buildbot_tac_path : str):
shutil.copy(buildbot_tac_path, buildbot_tac_path + ".bak")
result = []
with open(buildbot_tac_path) as buildbot_tac_file:
buildbot_tac = buildbot_tac_file.readlines()
for line in buildbot_tac:
if line.lstrip().startswith('basedir'):
line = line.replace(r'\\','/')
result.append(line)
with open(buildbot_tac_path, 'w') as buildbot_tac_file:
buildbot_tac_file.writelines(result)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('buildbot_tac_path', help="path of 'buildbot.tac' file")
args = parser.parse_args()
fix_slashes(args.buildbot_tac_path)

buildbot/google/docker/buildbot-sanitizer-windows/run.ps1

  • This file was added.
# Stop the script on the first error
$ErrorActionPreference = "Stop"
# Read password from file
$WORKER_PASSWORD=Get-Content "C:\volumes\secrets\token"
#Create short drive alias "Q:" for worker dir, to keep path lengths <260
SUBST Q: C:\volumes\buildbot
Get-PSDrive
# chdir to the volume storing the temporary data
# This is a mounted volume
Set-Location Q:\
# Write host and admin information
# delete old folder if it exists
# configure powershell output to use UTF-8, otherwiese buildbot can't read the file
if ( Test-Path -Path 'info\' -PathType Container ) {
Remove-Item -Recurse -Force info
}
mkdir "info\"
$HOST_FILE="info\host"
$PSDefaultParameterValues['Out-File:Encoding'] = 'utf8'
Write-Output "Windows version: $(cmd /c ver)" > $HOST_FILE
Write-Output "VCTools version: $(ls C:\BuildTools\VC\Tools\MSVC | Select -ExpandProperty Name)" >> $HOST_FILE
Write-Output "Cores : $((Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors)" >> $HOST_FILE
Write-Output "RAM : $([int][Math]::Round((Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory / 1073741824)) GB" >> $HOST_FILE
Write-Output "Reid Kleckner <rnk@google.com>" > "info\admin"
# create the worker
Write-Output "creating worker..."
buildslave create-slave --keepalive=200 Q: `
"lab.llvm.org:9994" "$env:WORKER_NAME" "$WORKER_PASSWORD"
# Note: Powershell does NOT exit on non-zero return codes, so we need to check manually.
if ($LASTEXITCODE -ne 0) { throw "Exit code of 'buildslave create' is $LASTEXITCODE" }
# Replace backward slashes with forward slashes on buildbot.tac
# Otherwise Cmake will go into infinite re-configuration loops
python C:\buildbot\fix_buildbot_tac_paths.py buildbot.tac
if ($LASTEXITCODE -ne 0) { throw "Exit code of 'fix_buildbot_tac_paths' is $LASTEXITCODE" }
# start the daemon, as this does not print and logs to std out, run it in the background
Write-Output "starting worker..."
cmd /c Start /B buildslave start Q:
# Wait a bit until the logfile exists
Start-Sleep -s 5
# To keep the container running and produce log outputs: dump the worker
# log to stdout, this is the windows equivalent of `tail -f`
Get-Content -Path "Q:\twistd.log" -Wait

buildbot/google/terraform/README.md

Show First 20 Lines • Show All 46 Lines • ▼ Show 20 Lines
# Secrets # Secrets
To keep secrets a secret, they MUST not be stored in version control. The right To keep secrets a secret, they MUST not be stored in version control. The right
place on kubernetes is a "secret". To create a kubernetes secret for the agent place on kubernetes is a "secret". To create a kubernetes secret for the agent
token: token:
```bash ```bash
kubectl create secret generic buildbot-token-mlir-nvidia --from-file=token=<file name> kubectl create secret generic buildbot-token-<BUILDER> --from-file=token=<file name>
``` ```
The file in `<file name>` then must contain the password of the buildbot worker The file in `<file name>` then must contain the password of the buildbot worker
in plain text. In the `Deployment` of a container, the secret is defined as a in plain text. In the `Deployment` of a container, the secret is defined as a
special type of volume and mounted in the specified path. During runtime the special type of volume and mounted in the specified path. During runtime the
secret can then be read from that file. secret can then be read from that file.
An example: An example:
The secret `buildbot-token-mlir-nvidia` is defined (as above) in Kubernetes. The secret `buildbot-token-<BUILDER>` is defined (as above) in Kubernetes.
In the [deployment](buildbot/google/terraform/main.tf) `mlir-nvidia` it is In the [deployment](buildbot/google/terraform/main.tf) `<BUILDER>` it is
used as a volume of type `secret` and then mounted at `/secrets`. During the used as a volume of type `secret` and then mounted at `/secrets`. During the
runtime of the docker container, the script runtime of the docker container, the script
[run.sh](../docker/buildbot-mlir-nvidia/run.sh) reads the secret from the file [run.sh](../docker/buildbot-<BUILDER>/run.sh) reads the secret from the file
`/secrets/token` and uses it to create the worker configuration. `/secrets/token` and uses it to create the worker configuration.
# Using GPUs on Google Cloud # Using GPUs on Google Cloud
Terraform does not support deployments on GCP using a GPU at the moment. Terraform does not support deployments on GCP using a GPU at the moment.
So we need to deploy such cases using plain Kubernetes configuration files. So we need to deploy such cases using plain Kubernetes configuration files.
See this [issue](https://github.com/terraform-providers/terraform-provider-kubernetes/issues/149) See this [issue](https://github.com/terraform-providers/terraform-provider-kubernetes/issues/149)
for more details. for more details.
The buildbot mlir-nvidia is configured in `deployment-mlir-nvidia.yaml` in this The buildbot <BUILDER> is configured in `deployment-<BUILER>.yaml` in this
folder. folder.
For all non-GPU cases add a `"kubernetes_deployment"` to `main.tf`. For all non-GPU cases add a `"kubernetes_deployment"` to `main.tf`.
The contents is identical to the the Kubernetes file, just the markup is The contents is identical to the the Kubernetes file, just the markup is
different. different.
Kubernetes files are also declarative, so you can re-deploy them when you made Kubernetes files are also declarative, so you can re-deploy them when you made
a change. They can be deployed with: a change. They can be deployed with:
```bash ```bash
kubectl apply -f myfile.yaml kubectl apply -f myfile.yaml
``` ```

buildbot/google/terraform/deployment-clang-debian-cpp20.yaml

  • This file was added.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: clang-debian-cpp20
name: clang-debian-cpp20
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: clang-debian-cpp20
strategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: clang-debian-cpp20
spec:
automountServiceAccountToken: false
containers:
- env:
- name: BUILDBOT_PORT
value: "9990" # 9990 is production, 9994 is staging.
image: gcr.io/sanitizer-bots/buildbot-cpp20:4
name: buildbot-clang-debian-cpp20
resources:
limits:
cpu: "15"
memory: 45G
requests:
cpu: "15"
memory: 45G
volumeMounts:
- mountPath: /vol/secrets
mountPropagation: None
name: buildbot-token
- mountPath: /vol/cccache
mountPropagation: None
name: ccache-vol
- mountPath: /vol/worker
mountPropagation: None
name: worker-vol
nodeSelector:
pool: linux-16-core-pool
restartPolicy: Always
volumes:
- name: buildbot-token
secret:
optional: false
secretName: password-clangd-ubuntu-clang
- emptyDir: {}
name: ccache-vol
- emptyDir: {}
name: worker-vol

buildbot/google/terraform/deployment-clangd-ubuntu-clang.yaml

  • This file was added.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: clangd-ubuntu-clang
name: clangd-ubuntu-clang
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: clangd-ubuntu-clang
strategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: clangd-ubuntu-clang
spec:
automountServiceAccountToken: false
containers:
- env:
- name: BUILDBOT_PORT
value: "9990" # 9990 is production, 9994 is staging.
image: gcr.io/sanitizer-bots/buildbot-clangd-ubuntu-clang:7
name: buildbot-clangd-ubuntu-clang
resources:
limits:
cpu: "15"
memory: 45G
requests:
cpu: "15"
memory: 45G
volumeMounts:
- mountPath: /vol/secrets
mountPropagation: None
name: buildbot-token
- mountPath: /vol/ccache
mountPropagation: None
name: ccache-vol
- mountPath: /vol/worker
mountPropagation: None
name: worker-vol
nodeSelector:
pool: linux-16-core-pool
restartPolicy: Always
volumes:
- name: buildbot-token
secret:
optional: false
secretName: password-clangd-ubuntu-clang
- emptyDir: {}
name: ccache-vol
- emptyDir: {}
name: worker-vol

buildbot/google/terraform/deployment-mlir-nvidia-production.yaml

  • This file was deleted.
---
# deployment for production at http://lab.llvm.org:8011/
# The deployment for staging is in a separate file!
apiVersion: apps/v1
kind: Deployment
metadata:
name: mlir-nvidia-production
spec:
# number of instances we want to run
replicas: 1
selector:
matchLabels:
app: mlir-nvidia-production
# define strategy for updating the images
strategy:
rollingUpdate:
# do not deploy more replicas, as the buildbot server
# can't handle multiple workers with the same credentials
maxSurge: 0
# Allow to have 0 replicas during updates.
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: mlir-nvidia-production
spec:
containers:
# the image and version we want to run
- image: gcr.io/sanitizer-bots/buildbot-mlir-nvidia:15
name: mlir-nvidia-production
env:
# connect to production environment, running at port 9990
- name: BUILDBOT_PORT
value: "9990"
# reserve "<number of cores>-1" for this image, kubernetes also
# needs <1 core for management tools
resources:
limits:
cpu: "15"
memory: 10Gi
# also request to use the GPU
# Note: this does not work in terraform at the moment
nvidia.com/gpu: "1"
requests:
cpu: "15"
memory: 10Gi
nvidia.com/gpu: "1"
volumeMounts:
# mount the secrets into a folder
- mountPath: /vol/secrets
mountPropagation: None
name: buildbot-token
# storage for ccache
- mountPath: /vol/ccache
mountPropagation: None
name: ccache-vol
# work directory of buildbot
- mountPath: /vol/buildbot
mountPropagation: None
name: buildbot-vol
# specify the nood pool on which to deploy
nodeSelector:
pool: nvidia-16core-pool
restartPolicy: Always
volumes:
# declare the secret as a volume so we can mount it
- name: buildbot-token
secret:
optional: false
secretName: password-mlir-nvidia
- name: ccache-vol
emptyDir: {}
- name: buildbot-vol
emptyDir: {}

buildbot/google/terraform/deployment-mlir-nvidia-staging.yaml

  • This file was deleted.
---
# deployment for staging, this connects the agent to the staging
# environment at http://lab.llvm.org:8014/
# the deployment for production is in a separate file
apiVersion: apps/v1
kind: Deployment
metadata:
name: mlir-nvidia-staging
spec:
# number of instances we want to run
replicas: 1
selector:
matchLabels:
app: mlir-nvidia-staging
# define strategy for updating the images
strategy:
rollingUpdate:
# do not deploy more replicas, as the buildbot server
# can't handle multiple workers with the same credentials
maxSurge: 0
# Allow to have 0 replicas during updates.
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: mlir-nvidia-staging
spec:
containers:
# the image and version we want to run
- image: gcr.io/sanitizer-bots/buildbot-mlir-nvidia:15
name: mlir-nvidia-staging
# set environment variables
env:
# connect to staging environment, running at port 9994 (instead of 9990)
- name: BUILDBOT_PORT
value: "9994"
# reserve "<number of cores>-1" for this image, kubernetes also
# needs <1 core for management tools
resources:
limits:
cpu: "15"
memory: 10Gi
# also request to use the GPU
# Note: this does not work in terraform at the moment
nvidia.com/gpu: "1"
requests:
cpu: "15"
memory: 10Gi
nvidia.com/gpu: "1"
volumeMounts:
# mount the secrets into a folder
- mountPath: /secrets
mountPropagation: None
name: buildbot-token
# specify the nood pool on which to deploy
nodeSelector:
pool: nvidia-16core-pool
restartPolicy: Always
volumes:
# declare the secret as a volume so we can mount it
- name: buildbot-token
secret:
optional: false
secretName: password-mlir-nvidia
# mount the secrets into a folder
- mountPath: /vol/secrets
mountPropagation: None
name: buildbot-token
# storage for ccache
- mountPath: /vol/ccache
mountPropagation: None
name: ccache-vol
# work directory of buildbot
- mountPath: /vol/buildbot
mountPropagation: None
name: buildbot-vol
# specify the nood pool on which to deploy
nodeSelector:
pool: nvidia-16core-pool
restartPolicy: Always
volumes:
# declare the secret as a volume so we can mount it
- name: buildbot-token
secret:
optional: false
secretName: password-mlir-nvidia
- name: ccache-vol
emptyDir: {}
- name: buildbot-vol
emptyDir: {}

buildbot/google/terraform/main.tf

Show First 20 Lines • Show All 44 Lines • ▼ Show 20 Lines node_config {
# We need at least 2 vCPU to run all kubernetes services # We need at least 2 vCPU to run all kubernetes services
machine_type = "e2-medium" machine_type = "e2-medium"
# use preemptible, as this saves costs # use preemptible, as this saves costs
preemptible = true preemptible = true
} }
} }
resource "null_resource" "update_cluster" {
# Add NVIDIA driver daemonset.
depends_on = [google_container_cluster.primary]
# Update kubectl context for the cluster and apply nvidia's daemonset.
provisioner "local-exec" {
command = <<EOT
gcloud container clusters get-credentials cudabot
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml
EOT
}
}
# Create machines for mlir-nvidia
# Note: The buildbot mlir-nividia is deployed using a kubernetes file. See
# the README.md for details on GPUs.
resource "google_container_node_pool" "nvidia_16core_pool_nodes" {
name = "nvidia-16core-pool"
# specify a zone here (e.g. "-a") to avoid a redundant deployment
location = var.gcp_config.zone_a
cluster = google_container_cluster.primary.name
# use autoscaling to only create a machine when there is a deployment
autoscaling {
min_node_count = 0
max_node_count = 2
}
node_config {
# use preemptible, as this saves costs
preemptible = true
# FIXME upgrade to "n1-custom-24-32768"
machine_type = "n1-highcpu-16"
disk_size_gb = 100
# FIXME: test if SSDs are actually faster than HDDs for our use case
disk_type = "pd-ssd"
guest_accelerator {
type = "nvidia-tesla-t4"
count= 1
}
# set the premissions required for the deployment later
oauth_scopes = [
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring",
"https://www.googleapis.com/auth/devstorage.read_only",
]
# add a label to all machines of this type, so we can select them
# during deployment
labels = {
pool = "nvidia-16core-pool"
}
}
}
resource "null_resource" "deployment-mlir-nvidia" {
# Add NVIDIA driver daemonset.
depends_on = [null_resource.update_cluster]
triggers = {
t4_contents = filemd5("${path.module}/deployment-mlir-nvidia-production.yaml")
}
# Add NVIDIA daemonset and deploy mlir-nvidia.
# Using this workaround as terraform does not support GPUs on Google Cloud.
# https://github.com/terraform-providers/terraform-provider-kubernetes/issues/149
provisioner "local-exec" {
command = "kubectl apply -f ${path.module}/deployment-mlir-nvidia-production.yaml"
}
}
# node pool for windows machines
resource "google_container_node_pool" "windows_32core_pool_nodes" {
name = "windows-32core-pool"
# specify a zone here (e.g. "-a") to avoid a redundant deployment
location = var.gcp_config.zone_a
cluster = google_container_cluster.primary.name
# use autoscaling to only create a machine when there is a deployment
autoscaling {
min_node_count = 0
max_node_count = 1
}
node_config {
# use preemptible, as this saves costs
preemptible = true
machine_type = "e2-highcpu-32"
# Windows deployments tend to require more disk space, so using 300GB here.
disk_size_gb = 300
# FIXME: test if SSDs are actually faster than HDDs for our use case
disk_type = "pd-ssd"
# Configure Windows image. As Windows is picky about the combination of
# host and container OS versions, this must be compatible with the version
# in your container. Recommondation: Use LTSC for long-term stability.
# For details see
# https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility
# https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster-windows#choose_your_windows_server_node_image
image_type = "WINDOWS_LTSC"
# set the premissions required for the deployment later
oauth_scopes = [
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring",
"https://www.googleapis.com/auth/devstorage.read_only",
]
# add a label to all machines of this type, so we can select them
# during deployment
labels = {
pool = "win-32core-pool"
}
}
}
# Deployment for the buildbot windows10_vs2019 running on Windows rather than
# Linux.
# Note: Deploying this takes significantly longer (~35 min) than on Linux
# as the images tend to be larger (~18GB) and IO performance is lower.
resource "kubernetes_deployment" "windows10_vs2019" {
metadata {
name = "windows10-vs2019"
labels = {
app = "windows10_vs2019"
}
}
spec {
# create one instance of this container
replicas = 1
selector {
match_labels = {
app = "windows10_vs2019"
}
}
strategy{
rolling_update{
# do not deploy more replicas, as the buildbot server
# can't handle multiple workers with the same credentials
max_surge = 0
# Allow to have 0 replicas during updates.
max_unavailable = 1
}
type = "RollingUpdate"
}
template {
metadata {
labels = {
app = "windows10_vs2019"
}
}
spec {
container {
image = "${var.gcp_config.gcr_prefix}/buildbot-windows10-vs2019:17"
name = "windows10-vs2019"
# reserve "<number of cores>-1" for this image, kubernetes also
# needs <1 core for management tools
resources {
limits {
cpu = "31"
memory = "20Gi"
}
requests {
cpu = "31"
memory = "20Gi"
}
}
# mount the secrets into a folder
volume_mount {
mount_path = "c:\\volumes\\secrets"
name = "buildbot-token"
}
volume_mount {
mount_path = "c:\\volumes\\sccache"
name = "sccache-vol"
}
volume_mount {
mount_path = "c:\\volumes\\buildbot"
name = "buildbot-vol"
}
}
# select which node pool to deploy to
node_selector = {
pool = "win-32core-pool"
}
# restart in case of any crashes
restart_policy = "Always"
# select the secret to be mounted
volume {
name = "buildbot-token"
secret {
optional = false
secret_name = "password-windows10-vs2019"
}
}
volume {
name = "sccache-vol"
empty_dir {}
}
volume {
name = "buildbot-vol"
empty_dir {}
}
# Windows nodes from the node pool are marked with the taint
# "node.kubernetes.io/os=windows". So we need to "tolerate" this to
# deploy to such nodes.
toleration {
effect = "NoSchedule"
key = "node.kubernetes.io/os"
operator = "Equal"
value = "windows"
}
}
}
}
}
resource "google_container_node_pool" "linux_16_core_pool" { resource "google_container_node_pool" "linux_16_core_pool" {
name = "linux-16-core-pool" name = "linux-16-core-pool"
# specify a zone here (e.g. "-a") to avoid a redundant deployment # specify a zone here (e.g. "-a") to avoid a redundant deployment
location = var.gcp_config.zone_a location = var.gcp_config.zone_a
cluster = google_container_cluster.primary.name cluster = google_container_cluster.primary.name
# use autoscaling to only create a machine when there is a deployment # use autoscaling to only create a machine when there is a deployment
autoscaling { autoscaling {
min_node_count = 0 min_node_count = 0
max_node_count = 2 max_node_count = 2
} }
node_config { node_config {
# use preemptible, as this saves costs # use preemptible, as this saves costs
preemptible = true preemptible = true
#custom machine type: 16 core, 32 GB as tsan needs more RAM #custom machine type: 16 core, 48 GB as tsan needs more RAM
machine_type = "n2d-custom-16-32768" machine_type = "n2d-custom-16-49152"
disk_size_gb = 100 disk_size_gb = 100
disk_type = "pd-ssd" disk_type = "pd-ssd"
# set the premissions required for the deployment later # set the premissions required for the deployment later
oauth_scopes = [ oauth_scopes = [
"https://www.googleapis.com/auth/logging.write", "https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring", "https://www.googleapis.com/auth/monitoring",
"https://www.googleapis.com/auth/devstorage.read_only", "https://www.googleapis.com/auth/devstorage.read_only",
] ]
# add a label to all machines of this type, so we can select them # add a label to all machines of this type, so we can select them
# during deployment # during deployment
labels = { labels = {
pool = "linux-16-core-pool" pool = "linux-16-core-pool"
} }
} }
} }
resource "kubernetes_deployment" "clangd-ubuntu-clang" {
metadata {
name = "clangd-ubuntu-clang"
labels = {
app = "clangd-ubuntu-clang"
}
}
spec {
# create one instance of this container
replicas = 1
selector {
match_labels = {
app = "clangd-ubuntu-clang"
}
}
strategy{
rolling_update{
# do not deploy more replicas, as the buildbot server
# can't handle multiple workers with the same credentials
max_surge = 0
# Allow to have 0 replicas during updates.
max_unavailable = 1
}
type = "RollingUpdate"
}
template {
metadata {
labels = {
app = "clangd-ubuntu-clang"
}
}
spec {
container {
image = "${var.gcp_config.gcr_prefix}/buildbot-clangd-ubuntu-clang:3"
name = "buildbot-clangd-ubuntu-clang"
# reserve "<number of cores>-1" for this image, kubernetes also
# needs <1 core for management tools
resources {
limits {
cpu = "15"
memory = "28G"
}
requests {
cpu = "15"
memory = "28G"
}
}
# mount the secrets into a folder
volume_mount {
mount_path = "/vol/secrets"
name = "buildbot-token"
}
volume_mount {
mount_path = "/vol/cccache"
name = "ccache-vol"
}
volume_mount {
mount_path = "/vol/worker"
name = "worker-vol"
}
env {
# connect to production environment, running at port 9990
# staging would be at 9994
name = "BUILDBOT_PORT"
value = "9990"
}
}
# select which node pool to deploy to
node_selector = {
pool = "linux-16-core-pool"
}
# restart in case of any crashes
restart_policy = "Always"
# select the secret to be mounted
volume {
name = "buildbot-token"
secret {
optional = false
secret_name = "password-clangd-ubuntu-clang"
}
}
volume {
name = "ccache-vol"
empty_dir {}
}
volume {
name = "worker-vol"
empty_dir {}
}
}
}
}
}

buildbot/mlir-nvidia/Dockerfile

  • This file was added.
#===-- Dockerfile --------------------------------------------------------===//
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#
#===----------------------------------------------------------------------===//
# Docker image used for the mlir-nvidia builder
#
# Environment variables configurable at runtime:
# BUILDBOT_PORT - server port to connect to
#===----------------------------------------------------------------------===//
# Use the image from NVIDIA
FROM nvidia/cuda:12.1.1-devel-ubuntu20.04
# install extra build tools
ENV TZ=Etc/UTC
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
apt-transport-https \
ca-certificates \
ccache \
clang-10 \
# dumb-init recommended in
# https://hub.docker.com/r/buildbot/buildbot-worker/dockerfile
dumb-init \
git \
gnupg \
libpython3-dev \
lld-10 \
ninja-build \
python3 \
python3-pip \
python3-psutil \
software-properties-common \
wget \
&& update-alternatives --install /usr/bin/clang clang /usr/bin/clang-10 100 \
&& update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-10 100 \
&& update-alternatives --install /usr/bin/ld.lld lld /usr/bin/lld-10 100 \
&& rm -rf /var/lib/apt/lists/*
# LTS releases often bundle obsolete pip versions that cannot access newest
# Linux binary wheels. This pinned version is not special: it was just current
# at the time this was added. Refer to compatibility table:
# https://github.com/pypa/manylinux
RUN python3 -m pip install --upgrade pip==23.1.2
# Refer to mlir/lib/Bindings/Python/requirements.txt. Listed explicitly here
# and version pinned for consistency as this is a bot.
RUN python3 -m pip install numpy==1.24.3 pybind11==2.10.4 PyYAML==5.3.1
# Ubuntu ships with old cmake version, install the latest one
# from https://apt.kitware.com/
RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null \
| gpg --dearmor - \
| \
tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null \
&& apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' \
&& apt-get update \
&& apt-get install -y --no-install-recommends cmake \
&& rm -rf /var/lib/apt/lists/*
# Get the Vulkan SDK from LunarG.
RUN wget -qO - http://packages.lunarg.com/lunarg-signing-key-pub.asc 2>/dev/null \
| apt-key add - \
&& wget -qO /etc/apt/sources.list.d/lunarg-vulkan-1.2.141-bionic.list \
https://packages.lunarg.com/vulkan/1.3.243/lunarg-vulkan-1.3.243-focal.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends vulkan-sdk \
&& rm -rf /var/lib/apt/lists/*
# Right now the container-optimized OS on Google Cloud does not have the Vulkan
# components in the installed driver, install them in the container. This creates
# a coupling with the actual driver running on the VM unfortunately.
RUN apt-get update \
&& apt-get install -y --no-install-recommends libnvidia-gl-470-server \
&& rm -rf /var/lib/apt/lists/*
# Install build bot (server was at 2.8.5-dev at time of writing).
RUN pip3 install buildbot-worker==2.8.4
# Create user account, some tests fail if run as root.
RUN useradd buildbot --create-home
WORKDIR /vol/worker
RUN mkdir -p /vol/worker \
&& chown buildbot /vol/worker
# Configure ccache (default is 5GB caching)
RUN mkdir /home/buildbot/.ccache/ \
&& echo "max_size = 30.0G" > /home/buildbot/.ccache/ccache.conf \
&& chown -R buildbot /home/buildbot/.ccache
# copy startup script
COPY run.sh /home/buildbot/
RUN chmod a+rx /home/buildbot/run.sh
USER buildbot
ENV WORKER_NAME="mlir-nvidia"
# Allow the server port of this agent to be configurable during deployment.
# This way we can connect the same image to production and integration.
# Ports:
# 9990 - production
# 9994 - integration
ENV BUILDBOT_PORT="9994"
CMD /home/buildbot/run.sh

buildbot/mlir-nvidia/run.sh

  • This file was moved from buildbot/google/docker/buildbot-mlir-nvidia/run.sh.
#!/bin/bash #!/bin/bash
#===-- run.sh -------------------------------------------------------------===// #===-- run.sh -------------------------------------------------------------===//
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information. # See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
# #
#===----------------------------------------------------------------------===// #===----------------------------------------------------------------------===//
# This script will start the buildbot worker # This script will start the buildbot worker
# #
#===----------------------------------------------------------------------===// #===----------------------------------------------------------------------===//
set -eu set -eu
# Read the worker password from a mounted file. # Read the worker password from a mounted file.
WORKER_PASSWORD=$(cat /vol/secrets/token) if [[ -z "${WORKER_PASSWORD}" ]]; then
echo "Missing WORKER_PASSWORD environment variable"
exit 1
fi
# Set up buildbot host and maintainer info. # Set up buildbot host and maintainer info.
mkdir -p "${WORKER_NAME}/info/" mkdir -p "${WORKER_NAME}/info/"
echo "Christian Kuehnel <kuhnel@google.com>" > "${WORKER_NAME}/info/admin" echo "Mehdi Amini <joker.eph@gmail.com>" > "${WORKER_NAME}/info/admin"
# generate the host information of this worker # generate the host information of this worker
( (
uname -a ; \ uname -a ; \
cat /proc/cpuinfo | grep "model name" | head -n1 | cut -d " " -f 3- ;\ cat /proc/cpuinfo | grep "model name" | head -n1 | cut -d " " -f 3- ;\
echo "number of cores: $(nproc)" ;\ echo "number of cores: $(nproc)" ;\
nvidia-smi -L | cut -d "(" -f 1 ;\ nvidia-smi -L | cut -d "(" -f 1 ;\
lsb_release -d | cut -f 2- ; \ lsb_release -d | cut -f 2- ; \
clang --version | head -n1 ; \ clang --version | head -n1 ; \
ld.lld-8 --version ; \ ld.lld --version ; \
cmake --version | head -n1 ; \ cmake --version | head -n1 ; \
vulkaninfo 2>/dev/null | grep "Vulkan Instance" ; \ vulkaninfo 2>/dev/null | grep "Vulkan Instance" ; \
vulkaninfo 2>/dev/null | grep "apiVersion" | cut -d= -f2 | awk '{printf "NVIDIA Vulkan ICD Version: " $2 "\n"}' vulkaninfo 2>/dev/null | grep "apiVersion" | cut -d= -f2 | awk '{printf "NVIDIA Vulkan ICD Version: " $2 "\n"}'
) > ${WORKER_NAME}/info/host ) > ${WORKER_NAME}/info/host
echo "Full nvidia-smi output:" echo "Full nvidia-smi output:"
nvidia-smi nvidia-smi
echo "Full vulkaninfo:" echo "Full vulkaninfo:"
vulkaninfo vulkaninfo
echo "Host information:" echo "Host information:"
cat ${WORKER_NAME}/info/host cat ${WORKER_NAME}/info/host
# It looks like GKE sometimes deploys the container before the NVIDIA drivers # It looks like GKE sometimes deploys the container before the NVIDIA drivers
# are loaded on the host. In this case the GPU is not available during the # are loaded on the host. In this case the GPU is not available during the
# entire lifecycle of the container. Not sure how to fix this properly. # entire lifecycle of the container. Not sure how to fix this properly.
RETURN_CODE=$(nvidia-smi > /dev/null ; echo $?) RETURN_CODE=$(nvidia-smi > /dev/null ; echo $?)
if [[ "$RETURN_CODE" != "0" ]] ; then if [[ "$RETURN_CODE" != "0" ]] ; then
echo "ERROR: Failed to access NVIDIA graphics card." echo "ERROR: Failed to access NVIDIA graphics card."
echo "Exiting in 30 secs..." echo "Exiting in 30 secs..."
sleep 30 sleep 30
exit 1 exit 1
fi fi
# create the folder structure # create the folder structure
echo "creating worker ${WORKER_NAME} at port ${BUILDBOT_PORT}..." echo "creating worker ${WORKER_NAME} at port ${BUILDBOT_PORT}..."
buildbot-worker create-worker --keepalive=200 "${WORKER_NAME}" \ buildbot-worker create-worker --keepalive=200 "${WORKER_NAME}" \
lab.llvm.org:${BUILDBOT_PORT} "${WORKER_NAME}" "${WORKER_PASSWORD}" lab.llvm.org:${BUILDBOT_PORT} "${WORKER_NAME}" "${WORKER_PASSWORD}"
# start the worker, based on # start the worker, based on
# https://hub.docker.com/r/buildbot/buildbot-worker/dockerfile # https://hub.docker.com/r/buildbot/buildbot-worker/dockerfile
echo "starting worker..." echo "starting worker..."
/usr/bin/dumb-init twistd --pidfile= --nodaemon -l - --python="${WORKER_NAME}/buildbot.tac" /usr/bin/dumb-init twistd --pidfile= --nodaemon -l - --python="${WORKER_NAME}/buildbot.tac"
No newline at end of file

buildbot/osuosl/master/config/__init__.py

# Load local options. # Load local options.
import os import os
from importlib import reload from importlib import reload
from configparser import ConfigParser from configparser import ConfigParser
options = ConfigParser( options = ConfigParser(
interpolation=None, interpolation=None,
empty_lines_in_values=False, empty_lines_in_values=False,
allow_no_value=True, allow_no_value=True,
) )
options.read(os.path.join(os.path.dirname(__file__), 'local.cfg')) options.read(os.path.join(os.path.dirname(__file__), 'local.cfg'))
import config.auth import config.auth
import config.builders import config.builders
import config.release_builders
import config.schedulers import config.schedulers
import config.workers import config.workers
import config.status import config.status
# Note: The following modules should be reloaded in # Note: The following modules should be reloaded in
# a particular order to follow the dependency chain. # a particular order to follow the dependency chain.
reload(config.auth) reload(config.auth)
reload(config.workers) reload(config.workers)
reload(config.schedulers) reload(config.schedulers)
reload(config.builders) reload(config.builders)
reload(release_builders)
reload(config.status) reload(config.status)

buildbot/osuosl/master/config/auth.py

from buildbot.plugins import util from buildbot.plugins import util
#from twisted.python import log #from twisted.python import log
from zorg.buildbot.util.workerowner import WorkerOwnerAuthz
from zorg.buildbot.util.workerowner import WorkerEndpointMatcher
from zorg.buildbot.util.workerowner import RolesFromWorkerOwner
import config import config
def getAuth(): def getAuth():
# For test local setup use NoAuth instead. # For test local setup use NoAuth instead.
auth = util.GitHubAuth( auth = util.GitHubAuth(
clientId=str(config.options.get('GitHub Auth', 'clientId')), clientId=str(config.options.get('GitHub Auth', 'clientId')),
clientSecret=str(config.options.get('GitHub Auth', 'clientSecret')), clientSecret=str(config.options.get('GitHub Auth', 'clientSecret')),
apiVersion=4, apiVersion=4,
getTeamsMembership=True, getTeamsMembership=True,
debug=False, debug=False,
) )
return auth return auth
def getAuthz(): def getAuthz():
authz = util.Authz( authz = WorkerOwnerAuthz(
allowRules=[ allowRules=[
# Admins can do anything. # Admins can do anything.
# defaultDeny=False: if user does not have the admin role, # defaultDeny=False: if user does not have the admin role,
# we continue parsing rules. # we continue parsing rules.
util.AnyEndpointMatcher(role="LLVM Lab team", defaultDeny=False), util.AnyEndpointMatcher(role="LLVM Lab team", defaultDeny=False),
WorkerEndpointMatcher(role="worker-owner", defaultDeny=False),
# Allow authors to stop, force or rebuild their own builds, # Allow authors to stop, force or rebuild their own builds,
util.StopBuildEndpointMatcher(role="owner", defaultDeny=False), util.StopBuildEndpointMatcher(role="owner", defaultDeny=False),
# Allow bot owners to stop, force or rebuild on their own bots, # Allow bot owners to stop, force or rebuild on their own bots,
util.StopBuildEndpointMatcher(role="worker-owner"), util.StopBuildEndpointMatcher(role="worker-owner"),
# allow devs to force or rebuild any build. # allow devs to force or rebuild any build.
util.RebuildBuildEndpointMatcher(role="owner", defaultDeny=False), util.RebuildBuildEndpointMatcher(role="owner", defaultDeny=False),
util.RebuildBuildEndpointMatcher(role="worker-owner", defaultDeny=False), util.RebuildBuildEndpointMatcher(role="worker-owner", defaultDeny=False),
util.RebuildBuildEndpointMatcher(role="LLVM Committers"), util.RebuildBuildEndpointMatcher(role="LLVM Committers"),
util.ForceBuildEndpointMatcher(role="owner", defaultDeny=False), util.ForceBuildEndpointMatcher(role="owner", defaultDeny=False),
util.ForceBuildEndpointMatcher(role="worker-owner", defaultDeny=False), util.ForceBuildEndpointMatcher(role="worker-owner", defaultDeny=False),
util.ForceBuildEndpointMatcher(role="LLVM Committers"), util.ForceBuildEndpointMatcher(role="LLVM Committers"),
# Future-proof control endpoints. No parsing rules beyond this. # Future-proof control endpoints. No parsing rules beyond this.
# Allows anonymous to look at build results. # Allows anonymous to look at build results.
util.AnyControlEndpointMatcher(role="LLVM Lab team"), util.AnyControlEndpointMatcher(role="LLVM Lab team"),
], ],
roleMatchers=[ roleMatchers=[
util.RolesFromGroups(groupPrefix="llvm/"), util.RolesFromGroups(groupPrefix="llvm/"),
util.RolesFromGroups(groupPrefix="llvm/"),
# role owner is granted when property owner matches the email of the user # role owner is granted when property owner matches the email of the user
util.RolesFromOwner(role="owner"), util.RolesFromOwner(role="owner"),
RolesFromWorkerOwner(role="worker-owner"),
], ],
) )
return authz return authz

buildbot/osuosl/master/config/builders.py

# TODO: Rename workers with "slave" as a part of the name. from importlib import reload
from buildbot.plugins import util
from buildbot.process.properties import WithProperties from buildbot.process.properties import WithProperties
from zorg.buildbot.builders import ClangBuilder from zorg.buildbot.builders import ClangBuilder
from zorg.buildbot.builders import FlangBuilder from zorg.buildbot.builders import FlangBuilder
from zorg.buildbot.builders import PollyBuilder from zorg.buildbot.builders import PollyBuilder
from zorg.buildbot.builders import LLDBBuilder from zorg.buildbot.builders import LLDBBuilder
from zorg.buildbot.builders import SanitizerBuilder from zorg.buildbot.builders import SanitizerBuilder
from zorg.buildbot.builders import OpenMPBuilder from zorg.buildbot.builders import OpenMPBuilder
from zorg.buildbot.builders import LibcxxAndAbiBuilder
from zorg.buildbot.builders import SphinxDocsBuilder from zorg.buildbot.builders import SphinxDocsBuilder
from zorg.buildbot.builders import ABITestsuitBuilder from zorg.buildbot.builders import ABITestsuitBuilder
from zorg.buildbot.builders import ClangLTOBuilder from zorg.buildbot.builders import ClangLTOBuilder
from zorg.buildbot.builders import UnifiedTreeBuilder from zorg.buildbot.builders import UnifiedTreeBuilder
from zorg.buildbot.builders import AOSPBuilder from zorg.buildbot.builders import AOSPBuilder
from zorg.buildbot.builders import AnnotatedBuilder from zorg.buildbot.builders import AnnotatedBuilder
from zorg.buildbot.builders import LLDPerformanceTestsuite from zorg.buildbot.builders import LLDPerformanceTestsuite
from zorg.buildbot.builders import FuchsiaBuilder from zorg.buildbot.builders import FuchsiaBuilder
from zorg.buildbot.builders import XToolchainBuilder from zorg.buildbot.builders import XToolchainBuilder
from zorg.buildbot.builders import TestSuiteBuilder
from zorg.buildbot.builders import BOLTBuilder
from buildbot.plugins import util from zorg.buildbot.builders import HtmlDocsBuilder
from zorg.buildbot.builders import DoxygenDocsBuilder
reload(ClangBuilder)
reload(FlangBuilder)
reload(PollyBuilder)
reload(LLDBBuilder)
reload(SanitizerBuilder)
reload(OpenMPBuilder)
reload(SphinxDocsBuilder)
reload(ABITestsuitBuilder)
reload(ClangLTOBuilder)
reload(UnifiedTreeBuilder)
reload(AOSPBuilder)
reload(AnnotatedBuilder)
reload(LLDPerformanceTestsuite)
reload(FuchsiaBuilder)
reload(XToolchainBuilder)
reload(TestSuiteBuilder)
reload(BOLTBuilder)
reload(HtmlDocsBuilder)
reload(DoxygenDocsBuilder)
all = [ all = [
# Clang fast builders. # Clang fast builders.
{'name' : "clang-x86_64-debian-fast", {'name' : "clang-x86_64-debian-fast",
'tags' : ["clang", "fast"], 'tags' : ["clang", "fast"],
'collapseRequests': False, 'collapseRequests': False,
'workernames':["gribozavr4"], 'workernames':["gribozavr4"],
'builddir':"clang-x86_64-debian-fast", 'builddir':"clang-x86_64-debian-fast",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
llvm_srcdir="llvm.src", llvm_srcdir="llvm.src",
obj_dir="llvm.obj", obj_dir="llvm.obj",
clean=True, clean=True,
depends_on_projects=['llvm','clang','clang-tools-extra','compiler-rt'], depends_on_projects=['llvm','clang','clang-tools-extra','compiler-rt'],
extra_configure_args=[ extra_configure_args=[
"-DLLVM_CCACHE_BUILD=ON",
"-DCOMPILER_RT_BUILD_BUILTINS:BOOL=OFF", "-DCOMPILER_RT_BUILD_BUILTINS:BOOL=OFF",
"-DCOMPILER_RT_BUILD_ORC:BOOL=OFF",
"-DCOMPILER_RT_BUILD_SANITIZERS:BOOL=OFF", "-DCOMPILER_RT_BUILD_SANITIZERS:BOOL=OFF",
"-DCOMPILER_RT_BUILD_XRAY:BOOL=OFF", "-DCOMPILER_RT_BUILD_XRAY:BOOL=OFF",
"-DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF", "-DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF",
"-DCOMPILER_RT_BUILD_LIBFUZZER:BOOL=OFF",
"-DCMAKE_C_FLAGS=-Wdocumentation -Wno-documentation-deprecated-sync", "-DCMAKE_C_FLAGS=-Wdocumentation -Wno-documentation-deprecated-sync",
"-DCMAKE_CXX_FLAGS=-std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync", "-DCMAKE_CXX_FLAGS=-std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync",
], ],
env={ env={
'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin', 'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes', 'CC': 'clang', 'CXX': 'clang++',
})},
{'name' : "clang-x86_64-debian-new-pass-manager-fast",
'tags' : ["clang", "fast"],
'workernames': ["gribozavr4"],
'builddir': "clang-x86_64-debian-new-pass-manager-fast",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
llvm_srcdir="llvm.src",
obj_dir="llvm.obj",
clean=True,
depends_on_projects=['llvm','clang','clang-tools-extra','compiler-rt'],
extra_configure_args=[
"-DCOMPILER_RT_BUILD_BUILTINS:BOOL=OFF",
"-DCOMPILER_RT_BUILD_SANITIZERS:BOOL=OFF",
"-DCOMPILER_RT_BUILD_XRAY:BOOL=OFF",
"-DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF",
"-DENABLE_EXPERIMENTAL_NEW_PASS_MANAGER=ON",
"-DCMAKE_C_FLAGS=-Wdocumentation -Wno-documentation-deprecated-sync",
"-DCMAKE_CXX_FLAGS=-std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync",
],
env={
'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes',
})}, })},
{'name' : "llvm-clang-x86_64-win-fast", {'name' : "llvm-clang-x86_64-win-fast",
'tags' : ["clang", "fast"], 'tags' : ["clang", "fast"],
'collapseRequests': False, 'collapseRequests': False,
'workernames' : ["as-builder-3"], 'workernames' : ["as-builder-3"],
'builddir': "llvm-clang-x86_64-win-fast", 'builddir': "llvm-clang-x86_64-win-fast",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
vs="autodetect", vs="autodetect",
depends_on_projects=['llvm', 'clang'], depends_on_projects=['llvm', 'clang'],
clean=True, clean=True,
checks=[ checks=[
"check-llvm-unit", "check-llvm-unit",
"check-clang-unit"], "check-clang-unit"],
extra_configure_args=[ extra_configure_args=[
"-DLLVM_ENABLE_WERROR=OFF", "-DLLVM_ENABLE_WERROR=OFF",
"-DLLVM_TARGETS_TO_BUILD=ARM", "-DLLVM_TARGETS_TO_BUILD=ARM",
"-DLLVM_DEFAULT_TARGET_TRIPLE=armv7-unknown-linux-eabihf", "-DLLVM_DEFAULT_TARGET_TRIPLE=armv7-unknown-linux-eabihf",
"-DLLVM_ENABLE_ASSERTIONS=OFF", "-DLLVM_ENABLE_ASSERTIONS=OFF",
"-DLLVM_OPTIMIZED_TABLEGEN=OFF", "-DLLVM_OPTIMIZED_TABLEGEN=OFF",
"-DLLVM_LIT_ARGS=-v --threads=32"])}, "-DLLVM_LIT_ARGS=-v --threads=32"])},
{'name': "llvm-clang-x86_64-sie-ubuntu-fast",
'tags' : ["clang", "llvm", "clang-tools-extra", "lld", "cross-project-tests"],
'collapseRequests': False,
'workernames': ["sie-linux-worker"],
'builddir': "llvm-clang-x86_64-sie-ubuntu-fast",
'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=['llvm','clang','clang-tools-extra','lld','cross-project-tests'],
extra_configure_args=[
"-DCMAKE_C_COMPILER=gcc",
"-DCMAKE_CXX_COMPILER=g++",
"-DCMAKE_BUILD_TYPE=Release",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_CLANGD=OFF",
"-DLLVM_BUILD_RUNTIME=OFF",
"-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_INCLUDE_EXAMPLES=OFF",
"-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-scei-ps4",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_LIT_ARGS=--verbose -j100",
"-DLLVM_TARGETS_TO_BUILD=X86",
"-DLLVM_USE_LINKER=gold"])},
# Expensive checks builders. # Expensive checks builders.
{'name' : "llvm-clang-x86_64-expensive-checks-ubuntu", {'name' : "llvm-clang-x86_64-expensive-checks-ubuntu",
'tags' : ["llvm", "expensive-checks"], 'tags' : ["llvm", "expensive-checks"],
'workernames' : ["as-builder-4"], 'workernames' : ["as-builder-4"],
'builddir': "llvm-clang-x86_64-expensive-checks-ubuntu", 'builddir': "llvm-clang-x86_64-expensive-checks-ubuntu",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=["llvm"], depends_on_projects=["llvm", "lld"],
clean=True, clean=True,
extra_configure_args=[ extra_configure_args=[
"-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON", "-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
"-DLLVM_ENABLE_WERROR=OFF", "-DLLVM_ENABLE_WERROR=OFF",
"-DLLVM_USE_SPLIT_DWARF=ON",
"-DLLVM_USE_LINKER=gold",
"-DCMAKE_BUILD_TYPE=Debug", "-DCMAKE_BUILD_TYPE=Debug",
"-DCMAKE_CXX_FLAGS=-U_GLIBCXX_DEBUG", "-DCMAKE_CXX_FLAGS=-U_GLIBCXX_DEBUG -Wno-misleading-indentation",
"-DLLVM_LIT_ARGS=-vv -j32"])}, "-DLLVM_LIT_ARGS=-vv -j32"])},
{'name' : "llvm-clang-x86_64-expensive-checks-win", {'name' : "llvm-clang-x86_64-expensive-checks-win",
'tags' : ["llvm", "clang", "expensive-checks"], 'tags' : ["llvm", "expensive-checks"],
'workernames' : ["as-worker-93"], 'workernames' : ["as-worker-93"],
'builddir': "llvm-clang-x86_64-expensive-checks-win", 'builddir': "llvm-clang-x86_64-expensive-checks-win",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
vs="autodetect", vs="autodetect",
depends_on_projects=["llvm", "lld"],
clean=True, clean=True,
extra_configure_args=[ extra_configure_args=[
"-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON", "-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
"-DLLVM_ENABLE_WERROR=OFF", "-DLLVM_ENABLE_WERROR=OFF",
"-DCMAKE_BUILD_TYPE=Debug", "-DCMAKE_BUILD_TYPE=Debug"])},
"-DLLVM_LIT_ARGS=-vv"])},
{'name' : "llvm-clang-x86_64-expensive-checks-debian", {'name' : "llvm-clang-x86_64-expensive-checks-debian",
'tags' : ["llvm", "expensive-checks"], 'tags' : ["llvm", "expensive-checks"],
'collapseRequests' : False, 'collapseRequests' : False,
'workernames' : ["gribozavr4"], 'workernames' : ["gribozavr4"],
'builddir': "llvm-clang-x86_64-expensive-checks-debian", 'builddir': "llvm-clang-x86_64-expensive-checks-debian",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=["llvm"], depends_on_projects=["llvm", "lld"],
clean=True, clean=True,
extra_configure_args=[ extra_configure_args=[
"-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON", "-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
"-DLLVM_ENABLE_WERROR=OFF", "-DLLVM_ENABLE_WERROR=OFF",
"-DCMAKE_BUILD_TYPE=Release", "-DCMAKE_BUILD_TYPE=Release",
"-DCMAKE_CXX_FLAGS=-U_GLIBCXX_DEBUG", "-DCMAKE_CXX_FLAGS=-U_GLIBCXX_DEBUG",
"-DLLVM_LIT_ARGS=-v -vv -j96"], "-DLLVM_LIT_ARGS=-v -vv -j96"],
env={ env={
'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin', 'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes', 'CC': 'clang', 'CXX': 'clang++',
})}, })},
# Cross builders. # Cross builders.
{'name' : "llvm-clang-win-x-armv7l", {'name' : "llvm-clang-win-x-armv7l",
'tags' : ["clang", "llvm", "compiler-rt", "cross"," armv7l"], 'tags' : ["clang", "llvm", "compiler-rt", "cross", "armv7"],
'workernames' : ["as-builder-1"], 'workernames' : ["as-builder-1"],
'builddir': "x-armv7l", 'builddir': "x-armv7l",
'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory( 'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory(
vs="autodetect", vs="autodetect",
clean=True, clean=True,
checks=[ checks=[
"check-llvm", "check-llvm",
"check-clang", "check-clang",
"check-lld", "check-lld",
"check-compiler-rt" "check-compiler-rt-armv7-unknown-linux-gnueabihf"
], ],
checks_on_target = [ checks_on_target = [
("libunwind", ("libunwind",
["python", "bin/llvm-lit.py", ["python", "bin/llvm-lit.py",
"-v", "-vv", "--threads=32", "-v", "-vv", "--threads=32",
"runtimes/runtimes-bins/libunwind/test"]), "runtimes/runtimes-armv7-unknown-linux-gnueabihf-bins/libunwind/test"]),
("libc++abi", ("libc++abi",
["python", "bin/llvm-lit.py", ["python", "bin/llvm-lit.py",
"-v", "-vv", "--threads=32", "-v", "-vv", "--threads=32",
"runtimes/runtimes-bins/libcxxabi/test"]), "runtimes/runtimes-armv7-unknown-linux-gnueabihf-bins/libcxxabi/test"]),
("libc++", ("libc++",
['python', 'bin/llvm-lit.py', ['python', 'bin/llvm-lit.py',
'-v', '-vv', '--threads=32', '-v', '-vv', '--threads=32',
'runtimes/runtimes-bins/libcxx/test', 'runtimes/runtimes-armv7-unknown-linux-gnueabihf-bins/libcxx/test',
]) ])
], ],
extra_configure_args=[ extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=ARM",
"-DTOOLCHAIN_TARGET_TRIPLE=armv7-unknown-linux-gnueabihf",
"-DDEFAULT_SYSROOT=C:/buildbot/.arm-ubuntu", "-DDEFAULT_SYSROOT=C:/buildbot/.arm-ubuntu",
"-DZLIB_ROOT=C:/buildbot/.zlib-win32",
"-DLLVM_LIT_ARGS=-v -vv --threads=32", "-DLLVM_LIT_ARGS=-v -vv --threads=32",
WithProperties("%(remote_test_host:+-DREMOTE_TEST_HOST=)s%(remote_test_host:-)s"), WithProperties("%(remote_test_host:+-DREMOTE_TEST_HOST=)s%(remote_test_host:-)s"),
WithProperties("%(remote_test_user:+-DREMOTE_TEST_USER=)s%(remote_test_user:-)s"), WithProperties("%(remote_test_user:+-DREMOTE_TEST_USER=)s%(remote_test_user:-)s"),
], ],
cmake_cache="../llvm-project/clang/cmake/caches/CrossWinToARMLinux.cmake")}, cmake_cache="../llvm-project/clang/cmake/caches/CrossWinToARMLinux.cmake")},
{'name' : "llvm-clang-win-x-aarch64", {'name' : "llvm-clang-win-x-aarch64",
'tags' : ["clang", "llvm", "compiler-rt", "cross"," aarch64"], 'tags' : ["clang", "llvm", "compiler-rt", "cross", "aarch64"],
'workernames' : ["as-builder-2"], 'workernames' : ["as-builder-2"],
'builddir': "x-aarch64", 'builddir': "x-aarch64",
'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory( 'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory(
vs="autodetect", vs="autodetect",
clean=True, clean=True,
checks=[ checks=[
"check-llvm", "check-llvm",
"check-clang", "check-clang",
"check-lld", "check-lld",
"check-compiler-rt" "check-compiler-rt-aarch64-unknown-linux-gnu"
], ],
checks_on_target = [ checks_on_target = [
("libunwind", ("libunwind",
["python", "bin/llvm-lit.py", ["python", "bin/llvm-lit.py",
"-v", "-vv", "--threads=32", "-v", "-vv", "--threads=32",
"runtimes/runtimes-bins/libunwind/test"]), "runtimes/runtimes-aarch64-unknown-linux-gnu-bins/libunwind/test"]),
("libc++abi", ("libc++abi",
["python", "bin/llvm-lit.py", ["python", "bin/llvm-lit.py",
"-v", "-vv", "--threads=32", "-v", "-vv", "--threads=32",
"runtimes/runtimes-bins/libcxxabi/test"]), "runtimes/runtimes-aarch64-unknown-linux-gnu-bins/libcxxabi/test"]),
("libc++", ("libc++",
['python', 'bin/llvm-lit.py', ['python', 'bin/llvm-lit.py',
'-v', '-vv', '--threads=32', '-v', '-vv', '--threads=32',
'runtimes/runtimes-bins/libcxx/test', 'runtimes/runtimes-aarch64-unknown-linux-gnu-bins/libcxx/test',
]) ])
], ],
extra_configure_args=[ extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=AArch64", "-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DCMAKE_C_COMPILER_TARGET=aarch64-linux-gnu", "-DTOOLCHAIN_TARGET_TRIPLE=aarch64-unknown-linux-gnu",
"-DDEFAULT_SYSROOT=C:/buildbot/.aarch64-ubuntu", "-DDEFAULT_SYSROOT=C:/buildbot/.aarch64-ubuntu",
"-DZLIB_ROOT=C:/buildbot/.zlib-win32",
"-DLLVM_LIT_ARGS=-v -vv --threads=32", "-DLLVM_LIT_ARGS=-v -vv --threads=32",
WithProperties("%(remote_test_host:+-DREMOTE_TEST_HOST=)s%(remote_test_host:-)s"), WithProperties("%(remote_test_host:+-DREMOTE_TEST_HOST=)s%(remote_test_host:-)s"),
WithProperties("%(remote_test_user:+-DREMOTE_TEST_USER=)s%(remote_test_user:-)s"), WithProperties("%(remote_test_user:+-DREMOTE_TEST_USER=)s%(remote_test_user:-)s"),
], ],
cmake_cache="../llvm-project/clang/cmake/caches/CrossWinToARMLinux.cmake")}, cmake_cache="../llvm-project/clang/cmake/caches/CrossWinToARMLinux.cmake")},
# Clang builders. # Clang builders.
{'name': "clang-arm64-windows-msvc", {'name': "clang-arm64-windows-msvc",
'tags' : ["clang"], 'tags' : ["llvm", "clang", "lld", "flang"],
'workernames' : ["linaro-armv8-windows-msvc-01", "linaro-armv8-windows-msvc-02"], 'workernames' : ["linaro-armv8-windows-msvc-04"],
'builddir': "clang-arm64-windows-msvc", 'builddir': "clang-arm64-windows-msvc",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
vs="manual", vs="manual",
test=False, # Disable testing until MCJIT failures are fixed clean=False,
checkout_flang=True,
checkout_lld=True,
checkout_compiler_rt=False,
extra_cmake_args=[ extra_cmake_args=[
"-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-windows-msvc", "-DCMAKE_TRY_COMPILE_CONFIGURATION=Release",
"-DLLVM_HOST_TRIPLE=aarch64-windows-msvc", "-DLLVM_TARGETS_TO_BUILD='AArch64'",
"-DLLVM_TARGET_ARCH=AArch64", "-DCMAKE_C_COMPILER_LAUNCHER=sccache",
"-DCOMPILER_RT_BUILD_SANITIZERS=OFF", "-DCMAKE_CXX_COMPILER_LAUNCHER=sccache"])},
"-DCOMPILER_RT_BUILD_XRAY=OFF"])},
## ARMv8 check-all
{'name' : "clang-armv8-quick",
'tags' : ["clang"],
'workernames':["linaro-clang-armv8-quick"],
'builddir':"clang-armv8-quick",
'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False,
checkout_compiler_rt=False,
checkout_lld=False,
extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM'"])},
# Cortex-A15 LNT test-suite in Benchmark mode # Cortex-A15 LNT test-suite in Benchmark mode
{'name' : "clang-native-arm-lnt-perf", {'name' : "clang-native-arm-lnt-perf",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames':["linaro-tk1-02"], 'workernames':["linaro-tk1-02"],
'builddir':"clang-native-arm-lnt-perf", 'builddir':"clang-native-arm-lnt-perf",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_compiler_rt=False, checkout_compiler_rt=False,
checkout_lld=False, checkout_lld=False,
test=False, checks=[],
useTwoStage=False,
runTestSuite=True, runTestSuite=True,
testsuite_flags=[ testsuite_flags=[
'--cppflags', '-O3 -mcpu=cortex-a15 -mthumb', '--cppflags', '-O3 -mcpu=cortex-a15 -mthumb',
'--threads=1', '--build-threads=4', '--threads=1', '--build-threads=4',
'--use-perf=all', '--use-perf=all',
'--benchmarking-only', '--exec-multisample=3', '--benchmarking-only', '--exec-multisample=3',
'--exclude-stat-from-submission=compile'], '--exclude-stat-from-submission=compile'],
extra_cmake_args=[ extra_cmake_args=[
"-DLLVM_TARGETS_TO_BUILD='ARM'", "-DLLVM_TARGETS_TO_BUILD='ARM'",
"-DLLVM_PARALLEL_LINK_JOBS=2"], "-DLLVM_PARALLEL_LINK_JOBS=2"],
submitURL='http://lnt.llvm.org/submitRun', submitURL='http://lnt.llvm.org/submitRun',
testerName='LNT-Thumb2v7-A15-O3')}, testerName='LNT-Thumb2v7-A15-O3')},
# ARMv7 LNT test-suite in test-only mode # ARMv7 LNT test-suite in test-only mode
{'name' : "clang-cmake-armv7-lnt", {'name' : "clang-armv7-lnt",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["linaro-armv7-lnt"], 'workernames' : ["linaro-clang-armv7-lnt"],
'builddir': "clang-cmake-armv7-lnt", 'builddir': "clang-armv7-lnt",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_compiler_rt=False, checkout_compiler_rt=False,
checkout_lld=False, checkout_lld=False,
test=False, checks=[],
useTwoStage=False,
runTestSuite=True, runTestSuite=True,
testsuite_flags=[ testsuite_flags=[
'--cppflags', '-mcpu=cortex-a15 -marm', '--cppflags', '-mcpu=cortex-a15 -marm',
'--threads=32', '--build-threads=32'], '--threads=32', '--build-threads=32'])},
extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM'"])},
## ARMv7 check-all self-host NEON with CMake builder ## ARMv7 check-all 2-stage
{'name' : "clang-cmake-armv7-selfhost-neon", {'name' : "clang-armv7-2stage",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames': ["linaro-armv7-selfhost"], 'workernames': ["linaro-clang-armv7-2stage"],
'builddir':"clang-cmake-armv7-selfhost-neon", 'builddir':"clang-armv7-2stage",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_compiler_rt=False, checkout_compiler_rt=False,
checkout_lld=False, checkout_lld=False,
useTwoStage=True, useTwoStage=True,
testStage1=False, testStage1=False,
extra_cmake_args=[ extra_cmake_args=[
"-DCMAKE_C_FLAGS='-mcpu=cortex-a15 -marm'", "-DCMAKE_C_FLAGS='-mcpu=cortex-a15 -marm'",
"-DCMAKE_CXX_FLAGS='-mcpu=cortex-a15 -marm'", "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a15 -marm'"])},
"-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
## ARMv7 check-all with CMake builder ## ARMv7 run test-suite with GlobalISel enabled
{'name' : "clang-cmake-armv7-quick", {'name' : "clang-armv7-global-isel",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames':["linaro-armv7-quick"], 'workernames':["linaro-clang-armv7-global-isel"],
'builddir':"clang-cmake-armv7-quick", 'builddir':"clang-armv7-global-isel",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_compiler_rt=False, checkout_compiler_rt=False,
checkout_lld=False, checkout_lld=False,
extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
## ARMv7 Clang + LLVM run test-suite with GlobalISel enabled
{'name' : "clang-cmake-armv7-global-isel",
'tags' : ["clang"],
'workernames':["linaro-armv7-global-isel"],
'builddir':"clang-cmake-armv7-global-isel",
'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False,
checkout_compiler_rt=False,
checkout_lld=False,
test=True,
useTwoStage=False,
runTestSuite=True, runTestSuite=True,
testsuite_flags=[ testsuite_flags=[
'--cppflags', '-mcpu=cortex-a15 -marm -O0 -mllvm -global-isel -mllvm -global-isel-abort=0', '--cppflags', '-mcpu=cortex-a15 -marm -O0 -mllvm -global-isel -mllvm -global-isel-abort=0',
'--threads=32', '--build-threads=32'], '--threads=32', '--build-threads=32'])},
extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
## ARMv7 check-all self-host with CMake builder ## ARMv7 VFPv3 check-all 2-stage
{'name' : "clang-cmake-armv7-selfhost", {'name' : "clang-armv7-vfpv3-2stage",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["linaro-armv7-selfhost"], 'workernames' : ["linaro-clang-armv7-vfpv3-2stage"],
'builddir': "clang-cmake-armv7-selfhost", 'builddir': "clang-armv7-vfpv3-2stage",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_compiler_rt=False, checkout_compiler_rt=False,
checkout_lld=False, checkout_lld=False,
useTwoStage=True, useTwoStage=True,
testStage1=False, testStage1=False,
extra_cmake_args=[ extra_cmake_args=[
"-DCMAKE_C_FLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -marm'", "-DCMAKE_C_FLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -marm'",
"-DCMAKE_CXX_FLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -marm'", "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -marm'"])},
"-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
## AArch64 Clang+LLVM check-all + test-suite ## AArch64 check-all
{'name' : "clang-cmake-aarch64-quick", {'name' : "clang-aarch64-quick",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["linaro-aarch64-quick"], 'workernames' : ["linaro-clang-aarch64-quick"],
'builddir': "clang-cmake-aarch64-quick", 'builddir': "clang-aarch64-quick",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_compiler_rt=False, checkout_compiler_rt=False,
checkout_lld=False, checkout_lld=False,
test=True, extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='AArch64'"])},
useTwoStage=False,
runTestSuite=True,
testsuite_flags=[
'--cppflags', '-mcpu=cortex-a57',
'--threads=32', '--build-threads=32'],
extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
## AArch64 Self-hosting Clang+LLVM check-all + LLD + test-suite ## AArch64 check-all + LLD + test-suite 2-stage
{'name' : "clang-cmake-aarch64-lld", {'name' : "clang-aarch64-lld-2stage",
'tags' : ["lld"], 'tags' : ["lld"],
'workernames' : ["linaro-aarch64-lld"], 'workernames' : ["linaro-clang-aarch64-lld-2stage"],
'builddir':"clang-cmake-aarch64-lld", 'builddir':"clang-aarch64-lld-2stage",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_compiler_rt=True,
checkout_lld=True,
test=True,
useTwoStage=True, useTwoStage=True,
runTestSuite=True, runTestSuite=True,
testsuite_flags=[ testsuite_flags=[
'--cppflags', '-mcpu=cortex-a57 -fuse-ld=lld', '--cppflags', '-mcpu=cortex-a57 -fuse-ld=lld',
'--threads=32', '--build-threads=32'], '--threads=32', '--build-threads=32'],
extra_cmake_args=[ extra_cmake_args=[
"-DCMAKE_C_FLAGS='-mcpu=cortex-a57'", "-DCMAKE_C_FLAGS='-mcpu=cortex-a57'",
"-DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'", "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'",
"-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'", "-DLLVM_ENABLE_LLD=True",
"-DLLVM_ENABLE_LLD=True"])}, "-DLLVM_LIT_ARGS='-v'"])},
## AArch64 Clang+LLVM run test-suite at -O0 (GlobalISel is now default). ## AArch64 run test-suite at -O0 (GlobalISel is now default).
{'name' : "clang-cmake-aarch64-global-isel", {'name' : "clang-aarch64-global-isel",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["linaro-aarch64-global-isel"], 'workernames' : ["linaro-clang-aarch64-global-isel"],
'builddir': "clang-cmake-aarch64-global-isel", 'builddir': "clang-aarch64-global-isel",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_compiler_rt=False, checkout_compiler_rt=False,
checkout_lld=False, checkout_lld=False,
test=True,
useTwoStage=False,
runTestSuite=True, runTestSuite=True,
testsuite_flags=[ testsuite_flags=[
'--cppflags', '-O0', '--cppflags', '-O0',
'--threads=32', '--build-threads=32'], '--threads=32', '--build-threads=32'])},
extra_cmake_args=["-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])},
## ARMv7 without neon; check-all 2-stage full compiler-rt + testsuite ## AArch32 Self-hosting Clang+LLVM check-all + LLD + test-suite
{'name' : "clang-cmake-armv7-full", # Sanitizers build disabled due to PR38690
{'name' : "clang-armv8-lld-2stage",
'tags' : ["lld"],
'workernames' : ["linaro-clang-armv8-lld-2stage"],
'builddir': "clang-armv8-lld-2stage",
'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False,
useTwoStage=True,
runTestSuite=True,
testsuite_flags=[
'--cppflags', '-mcpu=cortex-a57 -fuse-ld=lld',
'--threads=32', '--build-threads=32'],
extra_cmake_args=[
"-DCMAKE_C_FLAGS='-mcpu=cortex-a57'",
"-DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'",
"-DCOMPILER_RT_BUILD_SANITIZERS=OFF",
"-DLLVM_ENABLE_LLD=True",
# lld tests cause us to hit thread limits
"-DLLVM_ENABLE_THREADS=OFF"])},
# AArch64 check-all + flang + compiler-rt + test-suite + SVE/SME
# mlir-integration-tests 2-stage
{'name' : "clang-aarch64-full-2stage",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["linaro-tk1-06", "linaro-tk1-07", "linaro-tk1-08", "linaro-tk1-09"], 'workernames' : ["linaro-clang-aarch64-full-2stage"],
'builddir': "clang-cmake-armv7-full", 'builddir': "clang-aarch64-full-2stage",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_flang=True,
checkout_lld=False, checkout_lld=False,
checkout_compiler_rt=True,
testStage1=False,
useTwoStage=True, useTwoStage=True,
testStage1=False,
runTestSuite=True, runTestSuite=True,
env={
'NO_STOP_MESSAGE':'1', # For Fortran test-suite
},
testsuite_flags=[ testsuite_flags=[
'--cppflags', '-mcpu=cortex-a15 -mfpu=vfpv3 -marm', '--cppflags', '-mcpu=cortex-a57',
'--threads=4', '--build-threads=4'], '--threads=32', '--build-threads=32'],
extra_cmake_args=[ extra_cmake_args=[
"-DCOMPILER_RT_TEST_COMPILER_CFLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -marm'", "-DCMAKE_C_FLAGS='-mcpu=cortex-a57'",
"-DLLVM_TARGETS_TO_BUILD='ARM'", "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'",
"-DLLVM_PARALLEL_LINK_JOBS=2"])}, "-DLLVM_LIT_ARGS='-v'",
"-DMLIR_INCLUDE_INTEGRATION_TESTS=True",
"-DMLIR_RUN_ARM_SVE_TESTS=True",
"-DMLIR_RUN_ARM_SME_TESTS=True",
"-DARM_EMULATOR_EXECUTABLE=qemu-aarch64"])},
# AArch64 Clang+LLVM+RT+LLD check-all + flang + test-suite +
# mlir-integration-tests w/SVE-Vector-Length-Agnostic Note that in this and
# other clang-aarch64-sve-* builders we set -mllvm
# -treat-scalable-fixed-error-as-warning=false to make compiler fail on
# non-critical SVE codegen issues. This helps us notice and fix SVE
# problems sooner rather than later.
{'name' : "clang-aarch64-sve-vla",
'tags' : ["clang"],
'workernames' : ["linaro-g3-01", "linaro-g3-02", "linaro-g3-03", "linaro-g3-04"],
'builddir': "clang-aarch64-sve-vla",
'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False,
checkout_flang=True,
runTestSuite=True,
env={
'NO_STOP_MESSAGE':'1', # For Fortran test-suite
},
testsuite_flags=[
'--cppflags', '-mcpu=neoverse-512tvb -mllvm -scalable-vectorization=preferred -mllvm -treat-scalable-fixed-error-as-warning=false -O3',
'--threads=32', '--build-threads=32'],
extra_cmake_args=[
"-DCMAKE_C_FLAGS='-mcpu=neoverse-512tvb'",
"-DCMAKE_CXX_FLAGS='-mcpu=neoverse-512tvb'",
"-DLLVM_ENABLE_LLD=True",
"-DMLIR_INCLUDE_INTEGRATION_TESTS=True",
"-DMLIR_RUN_ARM_SVE_TESTS=True",
"-DLLVM_LIT_ARGS='-v'"])},
## ARMv7 Thumb2 with neon; check-all 2-stage full compiler-rt + testsuite # AArch64 Clang+LLVM+RT+LLD check-all + flang + test-suite 2-stage w/SVE-Vector-Length-Agnostic
{'name' : "clang-cmake-thumbv7-full-sh", {'name' : "clang-aarch64-sve-vla-2stage",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["linaro-tk1-01", "linaro-tk1-03", "linaro-tk1-04", "linaro-tk1-05"], 'workernames' : ["linaro-g3-01", "linaro-g3-02", "linaro-g3-03", "linaro-g3-04"],
'builddir': "clang-cmake-thumbv7-full-sh", 'builddir': "clang-aarch64-sve-vla-2stage",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_lld=False, checkout_flang=True,
checkout_compiler_rt=True,
testStage1=False,
useTwoStage=True, useTwoStage=True,
testStage1=False,
runTestSuite=True, runTestSuite=True,
env={
'NO_STOP_MESSAGE':'1', # For Fortran test-suite
},
testsuite_flags=[ testsuite_flags=[
'--cppflags', '-mcpu=cortex-a15 -mthumb', '--cppflags', '-mcpu=neoverse-512tvb -mllvm -scalable-vectorization=preferred -mllvm -treat-scalable-fixed-error-as-warning=false -O3',
'--threads=4', '--build-threads=4'], '--threads=32', '--build-threads=32'],
extra_cmake_args=[ extra_cmake_args=[
"-DCMAKE_C_FLAGS='-mcpu=cortex-a15 -mthumb'", "-DCMAKE_C_FLAGS='-mcpu=neoverse-512tvb -mllvm -scalable-vectorization=preferred -mllvm -treat-scalable-fixed-error-as-warning=false'",
"-DCMAKE_CXX_FLAGS='-mcpu=cortex-a15 -mthumb'", "-DCMAKE_CXX_FLAGS='-mcpu=neoverse-512tvb -mllvm -scalable-vectorization=preferred -mllvm -treat-scalable-fixed-error-as-warning=false'",
"-DCOMPILER_RT_TEST_COMPILER_CFLAGS='-mcpu=cortex-a15 -mthumb'", "-DLLVM_ENABLE_LLD=True",
"-DLLVM_TARGETS_TO_BUILD='ARM'", "-DLLVM_LIT_ARGS='-v'"])},
"-DLLVM_PARALLEL_LINK_JOBS=2"])},
## AArch32 Self-hosting Clang+LLVM check-all + LLD + test-suite # AArch64 Clang+LLVM+RT+LLD check-all + flang + test-suite w/SVE-Vector-Length-Specific
# Sanitizers build disabled due to PR38690 {'name' : "clang-aarch64-sve-vls",
{'name' : "clang-cmake-armv8-lld", 'tags' : ["clang"],
'tags' : ["lld"], 'workernames' : ["linaro-g3-01", "linaro-g3-02", "linaro-g3-03", "linaro-g3-04"],
'workernames' : ["linaro-armv8-lld"], 'builddir': "clang-aarch64-sve-vls",
'builddir': "clang-cmake-armv8-lld",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_compiler_rt=True, checkout_flang=True,
checkout_lld=True,
test=True,
useTwoStage=True,
runTestSuite=True, runTestSuite=True,
env={
'NO_STOP_MESSAGE':'1', # For Fortran test-suite
},
testsuite_flags=[ testsuite_flags=[
'--cppflags', '-mcpu=cortex-a57 -fuse-ld=lld', '--cppflags', '-mcpu=neoverse-512tvb -msve-vector-bits=256 -mllvm -treat-scalable-fixed-error-as-warning=false -O3',
'--threads=32', '--build-threads=32'], '--threads=32', '--build-threads=32'],
extra_cmake_args=[ extra_cmake_args=[
"-DCMAKE_C_FLAGS='-mcpu=cortex-a57'", "-DCMAKE_C_FLAGS='-mcpu=neoverse-512tvb'",
"-DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'", "-DCMAKE_CXX_FLAGS='-mcpu=neoverse-512tvb'",
"-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'", "-DLLVM_ENABLE_LLD=True",
"-DCOMPILER_RT_BUILD_SANITIZERS=OFF", "-DMLIR_INCLUDE_INTEGRATION_TESTS=True",
"-DLLVM_ENABLE_LLD=True"])}, "-DMLIR_RUN_ARM_SVE_TESTS=True",
"-DLLVM_LIT_ARGS='-v'"])},
# AArch64 Clang+LLVM+RT check-all + flang + test-suite + self-hosting # AArch64 Clang+LLVM+RT+LLD check-all + flang + test-suite 2-stage w/SVE-Vector-Length-Specific
{'name' : "clang-cmake-aarch64-full", {'name' : "clang-aarch64-sve-vls-2stage",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["linaro-aarch64-full"], 'workernames' : ["linaro-g3-01", "linaro-g3-02", "linaro-g3-03", "linaro-g3-04"],
'builddir': "clang-cmake-aarch64-full", 'builddir': "clang-aarch64-sve-vls-2stage",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_compiler_rt=True,
checkout_flang=True, checkout_flang=True,
checkout_lld=False,
test=True,
useTwoStage=True, useTwoStage=True,
testStage1=False, testStage1=False,
runTestSuite=True, runTestSuite=True,
env={
'NO_STOP_MESSAGE':'1', # For Fortran test-suite
},
testsuite_flags=[ testsuite_flags=[
'--cppflags', '-mcpu=cortex-a57', '--cppflags', '-mcpu=neoverse-512tvb -msve-vector-bits=256 -mllvm -treat-scalable-fixed-error-as-warning=false -O3',
'--threads=32', '--build-threads=32'], '--threads=32', '--build-threads=32'],
extra_cmake_args=[ extra_cmake_args=[
"-DCMAKE_C_FLAGS='-mcpu=cortex-a57'", "-DCMAKE_C_FLAGS='-mcpu=neoverse-512tvb -msve-vector-bits=256 -mllvm -treat-scalable-fixed-error-as-warning=false'",
"-DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'", "-DCMAKE_CXX_FLAGS='-mcpu=neoverse-512tvb -msve-vector-bits=256 -mllvm -treat-scalable-fixed-error-as-warning=false'",
"-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'"])}, "-DLLVM_ENABLE_LLD=True",
"-DLLVM_LIT_ARGS='-v'"])},
{'name' : "clang-arm64-windows-msvc-2stage", {'name' : "clang-arm64-windows-msvc-2stage",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["linaro-armv8-windows-msvc-01", "linaro-armv8-windows-msvc-02"], 'workernames' : ["linaro-armv8-windows-msvc-01", "linaro-armv8-windows-msvc-02", "linaro-armv8-windows-msvc-03"],
'builddir': "clang-arm64-windows-msvc-2stage", 'builddir': "clang-arm64-windows-msvc-2stage",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
vs="manual", vs="manual",
test=False, # Disable testing until MCJIT failures are fixed
useTwoStage=True, useTwoStage=True,
testStage1=False, checkout_flang=True,
extra_cmake_args=[ extra_cmake_args=[
"-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-windows-msvc", "-DCMAKE_TRY_COMPILE_CONFIGURATION=Release",
"-DLLVM_HOST_TRIPLE=aarch64-windows-msvc", "-DCMAKE_C_COMPILER_LAUNCHER=sccache",
"-DLLVM_TARGET_ARCH=AArch64", "-DCMAKE_CXX_COMPILER_LAUNCHER=sccache",
# FIXME: compiler-rt\lib\sanitizer_common\sanitizer_unwind_win.cpp assumes WIN64 is x86_64,
# so, before that's fixed, disable everything that triggers its build.
"-DCOMPILER_RT_BUILD_SANITIZERS=OFF", "-DCOMPILER_RT_BUILD_SANITIZERS=OFF",
"-DCOMPILER_RT_BUILD_XRAY=OFF"])}, "-DCOMPILER_RT_BUILD_PROFILE=OFF"])},
{'name' : 'clang-x64-windows-msvc', {'name' : 'clang-x64-windows-msvc',
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ['windows-gcebot2'], 'workernames' : ['windows-gcebot2'],
'builddir': 'clang-x64-windows-msvc', 'builddir': 'clang-x64-windows-msvc',
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory( 'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="clang-windows.py", script="clang-windows.py",
depends_on_projects=['llvm', 'clang', 'lld', 'debuginfo-tests'])}, depends_on_projects=['llvm', 'clang', 'lld', 'debuginfo-tests'])},
{'name' : "clang-ppc64be-linux-lnt", {'name' : "clang-m68k-linux",
'tags' : ["clang", "ppc"], 'tags' : ["clang"],
'workernames' : ["ppc64be-clang-lnt-test"], 'workernames' : ["debian-akiko-m68k"],
'builddir': "clang-ppc64be-lnt", 'builddir': "clang-m68k-linux",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_lld=False, checkout_lld=False,
useTwoStage=False, useTwoStage=False,
runTestSuite=True, enable_runtimes=None,
stage1_config='Release', stage1_config='Release',
nt_flags=['--threads=16', '--build-threads=16'], extra_cmake_args=[
extra_cmake_args=["-DLLVM_ENABLE_ASSERTIONS=ON"])}, "-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=M68k"])},
{'name' : "clang-ppc64be-linux-multistage", {'name' : "clang-m68k-linux-cross",
'tags' : ["clang", "ppc"], 'tags' : ["clang"],
'workernames' : ["ppc64be-clang-multistage-test"], 'workernames' : ["suse-gary-m68k-cross"],
'builddir': "clang-ppc64be-multistage", 'builddir': "clang-m68k-linux-cross",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_lld=False, checkout_lld=False,
useTwoStage=True, checkout_compiler_rt=False,
useTwoStage=False,
stage1_config='Release', stage1_config='Release',
stage2_config='Release', extra_cmake_args=[
extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON'])}, "-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_TARGETS_TO_BUILD=X86",
"-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=M68k"])},
{'name' : "clang-ppc64le-linux-lnt", {'name' : "clang-mips64el-linux",
'tags' : ["clang", "ppc", "ppc64le"], 'tags' : ["clang"],
'workernames' : ["ppc64le-clang-lnt-test"], 'workernames' : ["debian-tritium-mips64el"],
'builddir': "clang-ppc64le-lnt", 'builddir': "clang-mips64el-linux",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_lld=False, checkout_lld=False,
enable_runtimes=None,
useTwoStage=False, useTwoStage=False,
runTestSuite=True,
stage1_config='Release', stage1_config='Release',
nt_flags=['--threads=16', '--build-threads=16'], extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON',
extra_cmake_args=["-DLLVM_ENABLE_ASSERTIONS=ON"])}, '-DLLVM_PARALLEL_LINK_JOBS=4',
'-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON',
'-DCMAKE_C_COMPILER_TARGET="mips64el-unknown-linux-gnu"',
'-DLLVM_TARGETS_TO_BUILD=Mips'])},
{'name' : "clang-ppc64le-linux-multistage", {'name' : "clang-ppc64be-linux-test-suite",
'tags' : ["clang", "ppc", "ppc64le"], 'tags' : ["clang", "ppc"],
'workernames' : ["ppc64le-clang-multistage-test"], 'workernames' : ["ppc64be-clang-test-suite"],
'builddir': "clang-ppc64le-multistage", 'builddir': "clang-ppc64be-test-suite",
'factory' : TestSuiteBuilder.getTestSuiteBuildFactory(
depends_on_projects=["llvm", "clang", "clang-tools-extra",
"compiler-rt"],
checks=['check-all', 'check-runtimes'],
extra_configure_args=[
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_LIT_ARGS=-v",
"-DLLVM_CCACHE_BUILD=ON"])},
{'name' : "clang-ppc64be-linux-multistage",
'tags' : ["clang", "ppc"],
'workernames' : ["ppc64be-clang-multistage-test"],
'builddir': "clang-ppc64be-multistage",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checks=['check-all', 'check-runtimes'],
checkout_lld=False, checkout_lld=False,
useTwoStage=True, useTwoStage=True,
stage1_config='Release', stage1_config='Release',
stage2_config='Release', stage2_config='Release',
extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON', '-DBUILD_SHARED_LIBS=ON'])}, extra_cmake_args=[
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_CCACHE_BUILD=ON"])},
{'name' : "clang-ppc64be-linux", {'name' : "clang-ppc64le-linux-test-suite",
'tags' : ["clang", "ppc"], 'tags' : ["clang", "ppc", "ppc64le"],
'workernames' : ["ppc64be-clang-test"], 'workernames' : ["ppc64le-clang-test-suite"],
'builddir': "clang-ppc64be", 'builddir': "clang-ppc64le-test-suite",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : TestSuiteBuilder.getTestSuiteBuildFactory(
clean=False, depends_on_projects=["llvm", "clang", "clang-tools-extra",
checkout_lld=False, "compiler-rt"],
useTwoStage=False, checks=['check-all', 'check-runtimes'],
stage1_config='Release', extra_configure_args=[
extra_cmake_args=["-DLLVM_ENABLE_ASSERTIONS=ON"])}, "-DLLVM_ENABLE_ASSERTIONS=ON",
"-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_LIT_ARGS=-v",
"-DLLVM_CCACHE_BUILD=ON"])},
{'name' : "clang-ppc64le-linux", {'name' : "clang-ppc64le-linux-multistage",
'tags' : ["clang", "ppc", "ppc64le"], 'tags' : ["clang", "ppc", "ppc64le"],
'workernames' : ["ppc64le-clang-test"], 'workernames' : ["ppc64le-clang-multistage-test"],
'builddir': "clang-ppc64le", 'builddir': "clang-ppc64le-multistage",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checks=['check-all', 'check-runtimes'],
checkout_lld=False, checkout_lld=False,
useTwoStage=False, useTwoStage=True,
stage1_config='Release', stage1_config='Release',
extra_cmake_args=["-DLLVM_ENABLE_ASSERTIONS=ON"])}, stage2_config='Release',
extra_cmake_args=[
'-DLLVM_ENABLE_ASSERTIONS=ON',
'-DBUILD_SHARED_LIBS=ON',
'-DLLVM_CCACHE_BUILD=ON'])},
{'name' : "clang-ppc64le-rhel", {'name' : "clang-ppc64le-rhel",
'tags' : ["clang", "ppc", "ppc64le"], 'tags' : ["clang", "ppc", "ppc64le"],
'workernames' : ["ppc64le-clang-rhel-test"], 'workernames' : ["ppc64le-clang-rhel-test"],
'builddir': "clang-ppc64le-rhel", 'builddir': "clang-ppc64le-rhel",
'factory' : ClangBuilder.getClangCMakeBuildFactory(clean=False, 'factory' : TestSuiteBuilder.getTestSuiteBuildFactory(
checkout_clang_tools_extra=True, depends_on_projects=["llvm", "clang", "clang-tools-extra",
checkout_compiler_rt=False, "lld", "compiler-rt"],
checkout_lld=True, checks=['check-all', 'check-runtimes'],
checkout_libcxx=False, extra_configure_args=[
useTwoStage=False, "-DLLVM_ENABLE_ASSERTIONS=On",
runTestSuite=True, "-DCMAKE_C_COMPILER=clang",
stage1_config='Release',
nt_flags=['--threads=16', '--build-threads=16'],
extra_cmake_args=[
'-DLLVM_ENABLE_PROJECTS=clang;llvm;clang-tools-extra;lld',
'-DLLVM_ENABLE_RUNTIMES=compiler-rt',
"-DLLVM_ENABLE_ASSERTIONS=On", "-DCMAKE_C_COMPILER=clang",
"-DCMAKE_CXX_COMPILER=clang++", "-DCMAKE_CXX_COMPILER=clang++",
"-DCLANG_DEFAULT_LINKER=lld", "-DCLANG_DEFAULT_LINKER=lld",
"-DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN:PATH=/opt/rh/devtoolset-7/root/usr", "-DLLVM_TOOL_GOLD_BUILD=0",
"-DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN:PATH=/opt/rh/devtoolset-7/root/usr", "-DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN:PATH=/usr",
"-DLLVM_BINUTILS_INCDIR=/usr/include", "-DBUILD_SHARED_LIBS=ON", "-DLLVM_ENABLE_WERROR=ON", "-DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN:PATH=/usr",
'-DLLVM_LIT_ARGS=-vj 20'])}, "-DLLVM_BINUTILS_INCDIR=/usr/include",
"-DBUILD_SHARED_LIBS=ON", "-DLLVM_ENABLE_WERROR=ON",
"-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_LIT_ARGS=-vj 20"])},
{'name' : "clang-ppc64-aix",
'tags' : ["clang", "aix", "ppc"],
'workernames' : ["aix-ppc64"],
'builddir': "clang-ppc64-aix",
'factory' : TestSuiteBuilder.getTestSuiteBuildFactory(
depends_on_projects=["llvm", "clang", "compiler-rt"],
clean=False,
extra_configure_args=[
"-DLLVM_ENABLE_ASSERTIONS=On",
"-DCMAKE_C_COMPILER=/usr/local/clang-15.0.0/bin/clang",
"-DCMAKE_CXX_COMPILER=/usr/local/clang-15.0.0/bin/clang++",
"-DPython3_EXECUTABLE:FILEPATH=/opt/freeware/bin/python3_64",
"-DLLVM_ENABLE_ZLIB=OFF", "-DLLVM_APPEND_VC_REV=OFF",
"-DLLVM_PARALLEL_LINK_JOBS=2",
"-DLLVM_ENABLE_WERROR=ON"]),
'env' : {'OBJECT_MODE': '64'}},
{'name' : "clang-s390x-linux", {'name' : "clang-s390x-linux",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["systemz-1"], 'workernames' : ["systemz-1"],
'builddir': "clang-s390x-linux", 'builddir': "clang-s390x-linux",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
jobs=4, jobs=4,
clean=False, clean=False,
checkout_lld=False, checkout_lld=False,
useTwoStage=False, useTwoStage=False,
stage1_config='Release', stage1_config='Release',
extra_cmake_args=[ extra_cmake_args=[
"-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON", "-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_LIT_ARGS=-v -j4 --param run_long_tests=true"])}, "-DLLVM_LIT_ARGS=-v -j4 --param run_long_tests=true"])},
{'name' : "clang-s390x-linux-multistage", {'name' : "clang-s390x-linux-multistage",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["systemz-1"], 'workernames' : ["systemz-1"],
'builddir': "clang-s390x-linux-multistage", 'builddir': "clang-s390x-linux-multistage",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
jobs=4, jobs=4,
clean=False, clean=False,
checkout_lld=False, checkout_lld=False,
useTwoStage=True, useTwoStage=True,
stage1_config='Release', stage1_config='Release',
stage2_config='Release', stage2_config='Release',
extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON'])}, extra_cmake_args=[
"-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON"])},
{'name' : "clang-s390x-linux-lnt", {'name' : "clang-s390x-linux-lnt",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["systemz-1"], 'workernames' : ["systemz-1"],
'builddir': "clang-s390x-linux-lnt", 'builddir': "clang-s390x-linux-lnt",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
jobs=4, jobs=4,
clean=False, clean=False,
checkout_lld=False, checkout_lld=False,
useTwoStage=False, useTwoStage=False,
runTestSuite=True, runTestSuite=True,
stage1_config='Release', stage1_config='Release',
testsuite_flags=['--threads=4', '--build-threads=4'], testsuite_flags=['--threads=4', '--build-threads=4'],
extra_cmake_args=["-DLLVM_ENABLE_ASSERTIONS=ON"])}, extra_cmake_args=[
"-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON"])},
{'name' : "clang-sparc64-linux-multistage", {'name' : "clang-sparc64-linux-multistage",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["debian-stadler-sparc64"], 'workernames' : ["debian-stadler-sparc64"],
'builddir': "clang-sparc64-linux-multistage", 'builddir': "clang-sparc64-linux-multistage",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_lld=False, checkout_lld=False,
enable_runtimes=None,
useTwoStage=True, useTwoStage=True,
stage1_config='Release', stage1_config='Release',
stage2_config='Release', stage2_config='Release',
extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON', extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON',
'-DLLVM_PARALLEL_LINK_JOBS=4', '-DLLVM_PARALLEL_LINK_JOBS=4',
'-DLLVM_TARGETS_TO_BUILD=Sparc'])}, '-DLLVM_TARGETS_TO_BUILD=Sparc'])},
## LoongArch64 Clang+LLVM build check-all + test-suite
{'name' : 'clang-loongarch64-linux',
'tags' : ['clang'],
'workernames' : ['loongson-loongarch64-clfs-clang-01'],
'builddir': 'clang-loongarch64-linux',
'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False,
runTestSuite=True,
checkout_clang_tools_extra=False,
checkout_compiler_rt=False,
checkout_lld=False,
testsuite_flags=['--threads=32', '--build-threads=32'],
extra_cmake_args=['-DLLVM_TARGETS_TO_BUILD=',
'-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=LoongArch',
'-DLLVM_ENABLE_PROJECTS=clang'])},
{'name' : "clang-hexagon-elf", {'name' : "clang-hexagon-elf",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["hexagon-build-02", "hexagon-build-03"], 'workernames' : ["hexagon-build-02", "hexagon-build-03"],
'builddir': "clang-hexagon-elf", 'builddir': "clang-hexagon-elf",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
jobs=16, jobs=16,
checkout_clang_tools_extra=False, checkout_clang_tools_extra=False,
checkout_compiler_rt=False, checkout_compiler_rt=False,
checkout_lld=False, checkout_lld=False,
env={'LD_LIBRARY_PATH': '/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/lib', 'PATH': ['/local/cmake-3.17.0/bin', '${PATH}']}, env={'LD_LIBRARY_PATH': '/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/lib'},
extra_cmake_args=[ extra_cmake_args=[
"-DCMAKE_BUILD_TYPE:STRING=Release", "-DCMAKE_BUILD_TYPE:STRING=Release",
"-DLLVM_TARGETS_TO_BUILD:STRING=Hexagon", "-DLLVM_TARGETS_TO_BUILD:STRING=Hexagon",
"-DTARGET_TRIPLE:STRING=hexagon-unknown-elf", "-DTARGET_TRIPLE:STRING=hexagon-unknown-elf",
"-DLLVM_DEFAULT_TARGET_TRIPLE:STRING=hexagon-unknown-elf", "-DLLVM_DEFAULT_TARGET_TRIPLE:STRING=hexagon-unknown-elf",
"-DLLVM_TARGET_ARCH:STRING=hexagon-unknown-elf", "-DLLVM_TARGET_ARCH:STRING=hexagon-unknown-elf",
"-DLLVM_BUILD_RUNTIME:BOOL=OFF", "-DLLVM_BUILD_RUNTIME:BOOL=OFF",
"-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON", "-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON",
"-DLLVM_ENABLE_PIC:BOOL=ON", "-DLLVM_ENABLE_PIC:BOOL=ON",
"-DLLVM_ENABLE_ASSERTIONS:BOOL=ON", "-DLLVM_ENABLE_ASSERTIONS:BOOL=ON",
"-DLLVM_INCLUDE_TOOLS:BOOL=ON", "-DLLVM_INCLUDE_TOOLS:BOOL=ON",
"-DLLVM_LIT_ARGS:STRING=-v", "-DLLVM_LIT_ARGS:STRING=-v",
"-DLLVM_ENABLE_LIBCXX:BOOL=ON", "-DLLVM_ENABLE_LIBCXX:BOOL=ON",
"-DWITH_POLLY:BOOL=OFF", "-DWITH_POLLY:BOOL=OFF",
"-DLINK_POLLY_INTO_TOOLS:BOOL=OFF", "-DLINK_POLLY_INTO_TOOLS:BOOL=OFF",
"-DPOLLY_BUILD_SHARED_LIB:BOOL=OFF", "-DPOLLY_BUILD_SHARED_LIB:BOOL=OFF",
"-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang", "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang",
"-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])}, "-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang++"])},
## X86_64 AVX2 Clang+LLVM check-all + test-suite ## X86_64 AVX512 Clang+LLVM check-all + test-suite
{'name' : "clang-cmake-x86_64-avx2-linux", {'name' : "clang-cmake-x86_64-avx512-linux",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["avx2-intel64"], 'workernames' : ["avx512-intel64"],
'builddir': "clang-cmake-x86_64-avx2-linux", 'builddir': "clang-cmake-x86_64-avx512-linux",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, clean=False,
checkout_clang_tools_extra=False, checkout_clang_tools_extra=False,
checkout_compiler_rt=False, checkout_compiler_rt=False,
checkout_lld=False, checkout_lld=False,
test=True,
useTwoStage=False, useTwoStage=False,
runTestSuite=True, runTestSuite=True,
nt_flags=['--cflag', '-march=broadwell', '--threads=80', '--build-threads=80'], testsuite_flags=['--cflag', '-march=cascadelake', '--threads=32', '--build-threads=32'],
env={'PATH':'/usr/bin/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}, env={'PATH':'/usr/bin/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'},
extra_cmake_args=[ extra_cmake_args=[
"-DLLVM_ENABLE_ASSERTIONS=ON", "-DLLVM_ENABLE_ASSERTIONS=ON",
"-DCMAKE_C_FLAGS='-march=broadwell'", "-DCMAKE_C_FLAGS='-march=cascadelake'",
"-DCMAKE_CXX_FLAGS='-march=broadwell'", "-DCMAKE_CXX_FLAGS='-march=cascadelake'",
"-DLLVM_ENABLE_RUNTIMES=compiler-rt",
"-DCOMPILER_RT_BUILD_SANITIZERS=OFF",
"-DLLVM_TARGETS_TO_BUILD='X86'"])}, "-DLLVM_TARGETS_TO_BUILD='X86'"])},
## X86_64 AVX2 LNT test-suite in Benchmark mode ## Windows X86_64 AVX512 Clang+LLVM check-all + test-suite
{'name' : "clang-cmake-x86_64-avx2-linux-perf", {'name' : "clang-cmake-x86_64-avx512-win",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["avx2-intel64"], 'workernames' : ["avx512-intel64-win"],
'builddir': "clang-cmake-x86_64-avx2-linux-perf", 'builddir': "clang-cmake-x86_64-avx512-win",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
vs="autodetect",
vs_target_arch='x64',
stage1_config='Debug',
clean=False, clean=False,
checkout_clang_tools_extra=False, checkout_clang_tools_extra=True,
aaron.ballmanUnsubmitted
Not Done
ReplyInline Actions

Something smells off, but this is not my area of expertise so I could be wrong. We're not checking out compiler-rt, but below we have:
"-DLLVM_ENABLE_RUNTIMES=compiler-rt", which seems like a conflict.

aaron.ballman: Something smells off, but this is not my area of expertise so I could be wrong. We're not…
yinyangsxAuthorUnsubmitted
Done
ReplyInline Actions

It does look strange, although the same configuration works well under linux builder. Let us modify one line and run the test? what do you think?

yinyangsx: It does look strange, although the same configuration works well under linux builder. Let us…
aaron.ballmanUnsubmitted
Not Done
ReplyInline Actions

I think you should try changing checkout_compiler_rt= to True and see if the full test suite still passes for you.

aaron.ballman: I think you should try changing `checkout_compiler_rt=` to `True` and see if the full test…
checkout_compiler_rt=False, checkout_compiler_rt=False,
aaron.ballmanUnsubmitted
Not Done
ReplyInline Actions

I think this still needs to be set to True so that we check out the latest compiler-rt, since we're enabling the runtime for it below.

aaron.ballman: I think this still needs to be set to `True` so that we check out the latest compiler-rt, since…
yinyangsxAuthorUnsubmitted
Done
ReplyInline Actions

In our email thread we agree that “compiler-rt“ should be removed from the LLVM_ENABLE_PROJECTS option. So check_compiler_rt has to be False. Otherwise, "compiler-t" will be added to LLVM_ENABLE_PROJECTS option.

yinyangsx: In our email thread we agree that “compiler-rt“ should be removed from the LLVM_ENABLE_PROJECTS…
aaron.ballmanUnsubmitted
Not Done
ReplyInline Actions

In our email thread we agree that “compiler-rt“ should be removed from the LLVM_ENABLE_PROJECTS option. So check_compiler_rt has to be False. Otherwise, "compiler-t" will be added to LLVM_ENABLE_PROJECTS option.

Huh, that seems surprising to me. My understanding is that compiler-rt should never be added to LLVM_ENABLE_PROJECTS, it should only be added to LLVM_ENABLE_RUNTIMES to enable it. And if we're enabling it, we should be checking out the latest version so it's in step with changes to Clang.

CC @phosek for additional insights on the CMake incantations.

aaron.ballman: > In our email thread we agree that “compiler-rt“ should be removed from the…
checkout_lld=False, checkout_lld=False,
test=False,
useTwoStage=False, useTwoStage=False,
runTestSuite=True, runTestSuite=True,
nt_flags=['--cflag', '-march=broadwell', '--threads=1', '--build-threads=80', '--use-perf', testsuite_flags=['--cflag', '-march=cascadelake', '--threads=32', '--build-threads=32'],
'--benchmarking-only', '--multisample=4', '--exclude-stat-from-submission=compile'],
env={'PATH':'/usr/bin/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'},
extra_cmake_args=[ extra_cmake_args=[
"-DCMAKE_C_FLAGS='-march=broadwell'", "-DCMAKE_C_FLAGS='-march=cascadelake'",
"-DCMAKE_CXX_FLAGS='-march=broadwell'", "-DCMAKE_CXX_FLAGS='-march=cascadelake'",
"-DLLVM_TARGETS_TO_BUILD='X86'"], "-DLLVM_ENABLE_RUNTIMES=compiler-rt",
submitURL='http://lnt.llvm.org/submitRun', "-DCOMPILER_RT_BUILD_SANITIZERS=OFF",
testerName='LNT-Broadwell-AVX2-O3')}, "-DLLVM_TARGETS_TO_BUILD=X86"])},
## X86_64 Clang+LLVM Run test-suite targeting AVX512 on SDE (Emulator) {'name' : "clang-xcore-ubuntu-20-x64",
{'name' : "clang-cmake-x86_64-sde-avx512-linux",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["sde-avx512-intel64"], 'workernames' : ["xcore-ubuntu20-x64"],
'builddir': "clang-cmake-x86_64-sde-avx512-linux", 'builddir': "clang-xcore-ubuntu-20-x64",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False, jobs=4,
checkout_clang_tools_extra=False, checkout_clang_tools_extra=False,
checkout_compiler_rt=False, checkout_compiler_rt=False,
checkout_lld=False, checkout_lld=False,
test=True, testStage1=True,
useTwoStage=False, useTwoStage=False,
runTestSuite=True, stage1_config='Release',
nt_flags=['--cflag', '-march=skylake-avx512', '--threads=80',
'--build-threads=80', '--make-param', "RUNUNDER=sde64 -skx --", '--make-param', 'USER_MODE_EMULATION=1',
'--make-param', 'RUNTIMELIMIT=1200'],
env={'PATH':'/home/ssglocal/tools/sde/latest:/usr/bin/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'},
extra_cmake_args=[ extra_cmake_args=[
"-DLLVM_TARGETS_TO_BUILD:STRING=XCore",
"-DLLVM_DEFAULT_TARGET_TRIPLE:STRING=xcore-unknown-unknown-elf",
"-DLLVM_ENABLE_THREADS:BOOL=OFF"])},
{'name' : "llvm-clang-x86_64-sie-win",
'tags' : ["llvm", "clang", "clang-tools-extra", "lld", "cross-project-tests"],
'workernames' : ["sie-win-worker"],
'builddir': "llvm-clang-x86_64-sie-win",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
vs="autodetect",
target_arch='x64',
depends_on_projects=['llvm','clang','clang-tools-extra','lld','cross-project-tests'],
clean=True,
extra_configure_args=[
"-DCMAKE_BUILD_TYPE=Release",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_CLANGD=OFF",
"-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-sie-ps5",
"-DLLVM_INCLUDE_EXAMPLES=OFF",
"-DLLVM_TARGETS_TO_BUILD=X86",
"-DLLVM_VERSION_SUFFIX=",
"-DLLVM_BUILD_RUNTIME=OFF",
"-DLLVM_ENABLE_ASSERTIONS=ON", "-DLLVM_ENABLE_ASSERTIONS=ON",
"-DCMAKE_C_FLAGS='-march=broadwell'", "-DLLVM_LIT_ARGS=--verbose"])},
"-DCMAKE_CXX_FLAGS='-march=broadwell'",
"-DLLVM_TARGETS_TO_BUILD='X86'"])},
## Armv7 build cache {'name': "cross-project-tests-sie-ubuntu",
{'name' : "clang-armv7-linux-build-cache", 'tags' : ["clang", "llvm", "lldb", "cross-project-tests"],
'tags' : ["clang"], 'workernames': ["doug-worker-1a"],
'workernames' : ["packet-linux-armv7-slave-1"], 'builddir': "cross-project-tests-sie-ubuntu",
'builddir': "clang-armv7-linux-build-cache", 'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
'factory' : ClangBuilder.getClangCMakeGCSBuildFactory( depends_on_projects=['llvm','clang','lldb','cross-project-tests'],
stage1_config='Release', checks = ['check-cross-project'],
clean=True, extra_configure_args=[
checkout_compiler_rt=False, "-DCMAKE_C_COMPILER=gcc",
test=False, "-DCMAKE_CXX_COMPILER=g++",
useTwoStage=False, "-DCMAKE_BUILD_TYPE=Release",
runTestSuite=False, "-DCLANG_ENABLE_ARCMT=OFF",
checkout_lld=True, "-DLLDB_ENABLE_PYTHON=TRUE",
checkout_libcxx=True, "-DLLVM_INCLUDE_EXAMPLES=OFF",
checkout_clang_tools_extra=False, "-DLLVM_ENABLE_ASSERTIONS=ON",
use_pixz_compression=False, "-DLLVM_LIT_ARGS=--verbose",
xz_compression_factor=0, "-DLLVM_PARALLEL_LINK_JOBS=8",
stage1_upload_directory='clang-armv7-linux', "-DLLVM_TARGETS_TO_BUILD=X86",
extra_cmake_args=[ "-DLLVM_USE_LINKER=gold"])},
"-DLLVM_TARGETS_TO_BUILD='ARM;AArch64;X86'",
"-DCMAKE_C_FLAGS='-mthumb'",
"-DCMAKE_CXX_FLAGS='-mthumb'",
],
env={'BUCKET': 'llvm-build-artifacts'})},
## AArch64 build cache
{'name' : "clang-aarch64-linux-build-cache",
'tags' : ["clang"],
'workernames' : ["packet-linux-aarch64-slave-1"],
'builddir': "clang-aarch64-linux-build-cache",
'factory' : ClangBuilder.getClangCMakeGCSBuildFactory(
stage1_config='Release',
clean=True,
checkout_compiler_rt=False,
test=False,
useTwoStage=False,
runTestSuite=False,
checkout_lld=True,
checkout_libcxx=True,
checkout_clang_tools_extra=False,
stage1_upload_directory='clang-aarch64-linux',
use_pixz_compression=True,
extra_cmake_args=[
"-DLLVM_TARGETS_TO_BUILD='ARM;AArch64;X86'",
#"-DCMAKE_C_FLAGS=''",
#"-DCMAKE_CXX_FLAGS=''",
],
env={'BUCKET': 'llvm-build-artifacts'})},
{'name' : "llvm-avr-linux", {'name': "cross-project-tests-sie-ubuntu-dwarf5",
'tags' : ["clang"], 'tags' : ["clang", "llvm", "lldb", "cross-project-tests"],
'workernames' : ["avr-build-01"], 'workernames': ["doug-worker-1b"],
'builddir': "llvm-avr-linux", 'builddir': "cross-project-tests-sie-ubuntu-dwarf5",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
jobs=8, depends_on_projects=['llvm','clang','lldb','cross-project-tests'],
clean=False, checks = ['check-cross-project'],
checkout_lld=False, extra_configure_args=[
extra_cmake_args=[ "-DCMAKE_C_COMPILER=gcc",
'-DLLVM_ENABLE_ASSERTIONS=ON', "-DCMAKE_CXX_COMPILER=g++",
# We need to compile the X86 backend due to a few generic CodeGen tests. "-DCMAKE_BUILD_TYPE=Release",
'-DLLVM_TARGETS_TO_BUILD=X86', "-DCLANG_ENABLE_ARCMT=OFF",
'-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=AVR', "-DLLDB_ENABLE_PYTHON=TRUE",
'-DBUILD_SHARED_LIBS=ON'])}, "-DLLVM_INCLUDE_EXAMPLES=OFF",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_LIT_ARGS=--verbose",
"-DLLVM_PARALLEL_LINK_JOBS=8",
"-DLLVM_TARGETS_TO_BUILD=X86",
"-DLLVM_USE_LINKER=gold"])},
{'name' : "clang-x64-ninja-win7", {'name': "llvm-clang-x86_64-gcc-ubuntu",
'tags' : ["clang"], 'tags' : ["llvm", "clang", "clang-tools-extra", "compiler-rt", "lld", "cross-project-tests"],
'workernames' : ["windows7-buildbot"], 'workernames': ["doug-worker-2a"],
'builddir': "clang-x64-ninja-win7", 'builddir': "llvm-clang-x86_64-gcc-ubuntu",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=False, depends_on_projects=['llvm','clang','clang-tools-extra','compiler-rt','lld','cross-project-tests'],
checkout_lld=False, extra_configure_args=[
vs="autodetect", "-DCMAKE_C_COMPILER=gcc",
vs_target_arch='x64', "-DCMAKE_CXX_COMPILER=g++",
testStage1=True, "-DCMAKE_BUILD_TYPE=Release",
useTwoStage=True, "-DCLANG_ENABLE_CLANGD=OFF",
stage1_config='Release', "-DLLVM_BUILD_RUNTIME=ON",
stage2_config='Release', "-DLLVM_BUILD_TESTS=ON",
extra_cmake_args=[ "-DLLVM_ENABLE_ASSERTIONS=ON",
'-DLLVM_ENABLE_ASSERTIONS=ON', "-DLLVM_INCLUDE_EXAMPLES=OFF",
'-DLLVM_TARGETS_TO_BUILD=X86'])}, "-DLLVM_LIT_ARGS=--verbose -j48",
"-DLLVM_PARALLEL_LINK_JOBS=16",
"-DLLVM_USE_LINKER=gold"])},
# Polly builders. # Polly builders.
{'name' : "polly-arm-linux", {'name' : "polly-arm-linux",
'tags' : ["polly"], 'tags' : ["polly"],
'workernames' : ["hexagon-build-02", "hexagon-build-03"], 'workernames' : ["hexagon-build-02", "hexagon-build-03"],
'builddir': "polly-arm-linux", 'builddir': "polly-arm-linux",
'factory' : PollyBuilder.getPollyBuildFactory( 'factory' : PollyBuilder.getPollyBuildFactory(
clean=True, clean=True,
install=True, install=True,
make='ninja', make='ninja',
jobs=16, jobs=16,
env={'LD_LIBRARY_PATH': '/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/lib', 'PATH': ['/local/cmake-3.17.0/bin', '${PATH}']}, env={'LD_LIBRARY_PATH': '/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/lib'},
extraCmakeArgs=[ extraCmakeArgs=[
"-G", "Ninja", "-G", "Ninja",
"-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'", "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'",
"-DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-gnueabi", "-DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-gnueabi",
"-DLLVM_TARGET_ARCH=arm-linux-gnueabi", "-DLLVM_TARGET_ARCH=arm-linux-gnueabi",
"-DLLVM_ENABLE_ASSERTIONS=True", "-DLLVM_ENABLE_ASSERTIONS=True",
"-DLLVM_ENABLE_LIBCXX:BOOL=ON", "-DLLVM_ENABLE_LIBCXX:BOOL=ON",
"-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang", "-DPOLLY_ENABLE_GPGPU_CODEGEN=OFF",
"-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])}, "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang",
"-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang++"])},
{'name' : "polly-x86_64-linux", {'name' : "polly-x86_64-linux",
'tags' : ["polly"], 'tags' : ["polly"],
'workernames' : ["polly-x86_64-fdcserver", "polly-x86_64-gce1"], 'workernames' : ["polly-x86_64-gce1"],
'builddir': "polly-x86_64-linux", 'builddir': "polly-x86_64-linux",
'factory' : PollyBuilder.getPollyBuildFactory( 'factory' : PollyBuilder.getPollyBuildFactory(
clean=False, clean=False,
install=False, install=False,
make='ninja', make='ninja',
extraCmakeArgs=[ extraCmakeArgs=[
"-G", "Ninja", "-G", "Ninja",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_ASSERTIONS=True", "-DLLVM_ENABLE_ASSERTIONS=True",
"-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'", "-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'",
"-DCLANG_ENABLE_ARCMT=OFF", "-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_STATIC_ANALYZER=OFF", "-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
"-DCLANG_ENABLE_OBJC_REWRITER=OFF",
"-DLLVM_ENABLE_LLD=ON",
"-DPOLLY_ENABLE_GPGPU_CODEGEN=ON"
])},
{'name' : "polly-x86_64-linux-plugin",
'tags' : ["polly"],
'workernames' : ["polly-x86_64-gce1"],
'builddir': "polly-x86_64-linux-plugin",
'factory' : PollyBuilder.getPollyBuildFactory(
clean=False,
install=False,
make='ninja',
extraCmakeArgs=[
"-G", "Ninja",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_ASSERTIONS=True",
"-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
"-DCLANG_ENABLE_OBJC_REWRITER=OFF",
"-DLLVM_ENABLE_LLD=ON",
"-DLLVM_POLLY_LINK_INTO_TOOLS=OFF",
"-DPOLLY_ENABLE_GPGPU_CODEGEN=OFF" # Not all required symbols available in opt executable
])},
{'name' : "polly-x86_64-linux-noassert",
'tags' : ["polly"],
'workernames' : ["polly-x86_64-gce1"],
'builddir': "polly-x86_64-linux-noassert",
'factory' : PollyBuilder.getPollyBuildFactory(
clean=False,
install=False,
make='ninja',
extraCmakeArgs=[
"-G", "Ninja",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_ASSERTIONS=False",
"-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
"-DCLANG_ENABLE_OBJC_REWRITER=OFF",
"-DLLVM_ENABLE_LLD=ON",
"-DPOLLY_ENABLE_GPGPU_CODEGEN=ON"
])},
{'name' : "polly-x86_64-linux-shared",
'tags' : ["polly"],
'workernames' : ["polly-x86_64-gce2"],
'builddir': "polly-x86_64-linux-shared",
'factory' : PollyBuilder.getPollyBuildFactory(
clean=False,
install=False,
make='ninja',
extraCmakeArgs=[
"-G", "Ninja",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_ASSERTIONS=True",
"-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
"-DCLANG_ENABLE_OBJC_REWRITER=OFF",
"-DLLVM_ENABLE_LLD=ON",
"-DBUILD_SHARED_LIBS=ON",
"-DPOLLY_ENABLE_GPGPU_CODEGEN=ON"
])},
{'name' : "polly-x86_64-linux-shared-plugin",
'tags' : ["polly"],
'workernames' : ["polly-x86_64-gce2"],
'builddir': "polly-x86_64-linux-shared-plugin",
'factory' : PollyBuilder.getPollyBuildFactory(
clean=False,
install=False,
make='ninja',
extraCmakeArgs=[
"-G", "Ninja",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_ASSERTIONS=True",
"-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
"-DCLANG_ENABLE_OBJC_REWRITER=OFF",
"-DLLVM_ENABLE_LLD=ON",
"-DBUILD_SHARED_LIBS=ON",
"-DLLVM_POLLY_LINK_INTO_TOOLS=OFF",
"-DPOLLY_ENABLE_GPGPU_CODEGEN=ON"
])},
{'name' : "polly-x86_64-linux-shlib",
'tags' : ["polly"],
'workernames' : ["polly-x86_64-gce2"],
'builddir': "polly-x86_64-linux-shlib",
'factory' : PollyBuilder.getPollyBuildFactory(
clean=False,
install=False,
make='ninja',
extraCmakeArgs=[
"-G", "Ninja",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_ASSERTIONS=True",
"-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
"-DCLANG_ENABLE_OBJC_REWRITER=OFF",
"-DLLVM_ENABLE_LLD=ON",
"-DLLVM_BUILD_LLVM_DYLIB=ON",
"-DLLVM_LINK_LLVM_DYLIB=ON",
"-DPOLLY_ENABLE_GPGPU_CODEGEN=ON"
])},
{'name' : "polly-x86_64-linux-shlib-plugin",
'tags' : ["polly"],
'workernames' : ["polly-x86_64-gce2"],
'builddir': "polly-x86_64-linux-shlib-plugin",
'factory' : PollyBuilder.getPollyBuildFactory(
clean=False,
install=False,
make='ninja',
extraCmakeArgs=[
"-G", "Ninja",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_ASSERTIONS=True",
"-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
"-DCLANG_ENABLE_OBJC_REWRITER=OFF",
"-DLLVM_ENABLE_LLD=ON",
"-DLLVM_BUILD_LLVM_DYLIB=ON",
"-DLLVM_LINK_LLVM_DYLIB=ON",
"-DLLVM_POLLY_LINK_INTO_TOOLS=OFF",
"-DPOLLY_ENABLE_GPGPU_CODEGEN=ON"
])}, ])},
{'name' : "polly-x86_64-linux-test-suite", {'name' : "polly-x86_64-linux-test-suite",
'tags' : ["polly"], 'tags' : ["polly"],
'workernames' : ["polly-x86_64-fdcserver", "polly-x86_64-gce2"], 'workernames' : ["polly-x86_64-fdcserver", "minipc-1050ti-linux"],
'builddir': "polly-x86_64-linux-test-suite", 'builddir': "polly-x86_64-linux-test-suite",
'factory' : PollyBuilder.getPollyBuildFactory( 'factory' : PollyBuilder.getPollyBuildFactory(
clean=False, clean=False,
install=False, install=False,
make='ninja', make='ninja',
extraCmakeArgs=[ extraCmakeArgs=[
"-G", "Ninja", "-G", "Ninja",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_ASSERTIONS=True", "-DLLVM_ENABLE_ASSERTIONS=True",
"-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'", "-DLLVM_TARGETS_TO_BUILD='X86;NVPTX'",
"-DCLANG_ENABLE_ARCMT=OFF", "-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_STATIC_ANALYZER=OFF", "-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
"-DCLANG_ENABLE_OBJC_REWRITER=OFF"
], ],
testsuite=True, testsuite=True,
extraTestsuiteCmakeArgs=["-G", "Ninja"] extraTestsuiteCmakeArgs=[
"-G", "Ninja",
"-DTEST_SUITE_COLLECT_COMPILE_TIME=OFF",
"-DTEST_SUITE_COLLECT_STATS=OFF",
"-DTEST_SUITE_COLLECT_CODE_SIZE=OFF",
WithProperties("-DTEST_SUITE_EXTERNALS_DIR=%(builddir)s/../../test-suite-externals"),
]
)}, )},
# AOSP builders. # AOSP builders.
{'name' : "aosp-O3-polly-before-vectorizer-unprofitable", {'name' : "aosp-O3-polly-before-vectorizer-unprofitable",
'tags' : ["polly", "aosp"], 'tags' : ["polly", "aosp"],
'workernames' : ["hexagon-build-03"], 'workernames' : ["hexagon-build-03"],
'builddir': "aosp", 'builddir': "aosp",
'factory' : AOSPBuilder.getAOSPBuildFactory( 'factory' : AOSPBuilder.getAOSPBuildFactory(
device="angler", device="angler",
extra_cmake_args=[ extra_cmake_args=[
"-G", "Ninja", "-G", "Ninja",
"-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'", "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64'",
"-DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-androideabi", "-DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-androideabi",
"-DLLVM_TARGET_ARCH=arm-linux-androideabi", "-DLLVM_TARGET_ARCH=arm-linux-androideabi",
"-DLLVM_ENABLE_ASSERTIONS=True", "-DLLVM_ENABLE_ASSERTIONS=True",
"-DLLVM_ENABLE_LIBCXX:BOOL=ON", "-DLLVM_ENABLE_LIBCXX:BOOL=ON",
"-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang", "-DPOLLY_ENABLE_GPGPU_CODEGEN=OFF",
"-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"], "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang",
"-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang++"],
timeout=240, timeout=240,
target_clang=None, target_clang=None,
target_flags="-Wno-error -O3 -mllvm -polly -mllvm -polly-position=before-vectorizer -mllvm -polly-process-unprofitable -fcommon", target_flags="-Wno-error -O3 -mllvm -polly -mllvm -polly-position=before-vectorizer -mllvm -polly-process-unprofitable -fcommon",
jobs=8, jobs=32,
extra_make_args=None, extra_make_args=None,
env={'LD_LIBRARY_PATH': '/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/lib', 'PATH': ['/local/cmake-3.17.0/bin', '${PATH}']}, env={'LD_LIBRARY_PATH': '/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/lib'},
clean=False, clean=False,
sync=False, sync=False,
patch=None)}, patch=None)},
# Reverse iteration builders. # Reverse iteration builders.
{'name' : "reverse-iteration", {'name' : "reverse-iteration",
'tags' : ["rev_iter"], 'tags' : ["rev_iter"],
'workernames' : ["hexagon-build-02", "hexagon-build-03"], 'workernames' : ["hexagon-build-02", "hexagon-build-03"],
'builddir': "reverse-iteration", 'builddir': "reverse-iteration",
'factory' : PollyBuilder.getPollyBuildFactory( 'factory' : PollyBuilder.getPollyBuildFactory(
clean=True, clean=True,
make='ninja', make='ninja',
jobs=16, jobs=16,
checkAll=True, checkAll=True,
env={'LD_LIBRARY_PATH': '/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/lib', 'PATH': ['/local/cmake-3.17.0/bin', '${PATH}']}, env={'LD_LIBRARY_PATH': '/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/lib'},
extraCmakeArgs=[ extraCmakeArgs=[
"-G", "Ninja", "-G", "Ninja",
"-DLLVM_REVERSE_ITERATION:BOOL=ON", "-DLLVM_REVERSE_ITERATION:BOOL=ON",
"-DLLVM_ENABLE_ASSERTIONS=True", "-DLLVM_ENABLE_ASSERTIONS=True",
"-DLLVM_ENABLE_LIBCXX:BOOL=ON", "-DLLVM_ENABLE_LIBCXX:BOOL=ON",
"-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang", "-DPOLLY_ENABLE_GPGPU_CODEGEN=ON",
"-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin/clang++"])}, "-DCMAKE_C_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang",
"-DCMAKE_CXX_COMPILER:FILEPATH=/local/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang++"])},
# LLDB builders. # LLDB builders.
{'name' : "lldb-x86_64-debian", {'name' : "lldb-x86_64-debian",
'tags' : ["lldb"], 'tags' : ["lldb"],
'workernames' : ["lldb-x86_64-debian"], 'workernames' : ["lldb-x86_64-debian"],
'builddir': "lldb-x86_64-debian", 'builddir': "lldb-x86_64-debian",
'factory' : LLDBBuilder.getLLDBCMakeBuildFactory( 'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
test=True, test=True,
extra_cmake_args=[ extra_cmake_args=[
'-DLLVM_ENABLE_ASSERTIONS=True', '-DLLVM_ENABLE_ASSERTIONS=True',
'-DLLVM_USE_LINKER=gold', '-DLLVM_USE_LINKER=gold',
'-DLLDB_ENABLE_PYTHON=True', '-DLLDB_ENABLE_PYTHON=True',
'-DLLDB_TEST_USER_ARGS=-t',
'-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DPYTHON_EXECUTABLE=/usr/bin/python3',
'-DCMAKE_C_COMPILER=clang', '-DCMAKE_C_COMPILER=clang',
'-DCMAKE_CXX_COMPILER=clang++'])}, '-DCMAKE_CXX_COMPILER=clang++'])},
{'name' : "lldb-aarch64-ubuntu", {'name' : "lldb-aarch64-ubuntu",
'tags' : ["lldb"], 'tags' : ["lldb"],
'workernames' : ["linaro-aarch64-lldb"], 'workernames' : ["linaro-lldb-aarch64-ubuntu"],
'builddir': "lldb-cmake-aarch64", 'builddir': "lldb-aarch64-ubuntu",
'factory' : LLDBBuilder.getLLDBCMakeBuildFactory( 'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
test=True, test=True,
clean=True,
extra_cmake_args=[ extra_cmake_args=[
'-DLLVM_ENABLE_ASSERTIONS=True', '-DLLVM_ENABLE_ASSERTIONS=True',
'-DLLVM_LIT_ARGS=-svj 8', '-DLLVM_LIT_ARGS=-vj 4',
'-DLLVM_USE_LINKER=gold'])}, '-DLLVM_USE_LINKER=gold'])},
{'name' : "lldb-arm-ubuntu", {'name' : "lldb-arm-ubuntu",
'tags' : ["lldb"], 'tags' : ["lldb"],
'workernames' : ["linaro-arm-lldb"], 'workernames' : ["linaro-lldb-arm-ubuntu"],
'builddir': "lldb-cmake-arm", 'builddir': "lldb-arm-ubuntu",
'factory' : LLDBBuilder.getLLDBCMakeBuildFactory( 'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
test=True, test=True,
clean=True,
extra_cmake_args=[ extra_cmake_args=[
'-DLLVM_ENABLE_ASSERTIONS=True', '-DLLVM_ENABLE_ASSERTIONS=True',
'-DLLVM_LIT_ARGS=-svj 8', '-DLLVM_LIT_ARGS=-vj 4',
'-DLLVM_USE_LINKER=gold'])}, '-DLLVM_USE_LINKER=gold'])},
{'name' : "lldb-x64-windows-ninja", {'name' : "lldb-aarch64-windows",
'tags' : ["lldb"], 'tags' : ["lldb"],
'workernames' : ["win-py3-buildbot"], 'workernames' : ["linaro-armv8-windows-msvc-05"],
'builddir': "lldb-x64-windows-ninja", 'builddir': "lldb-aarch64-windows",
'factory' : LLDBBuilder.getLLDBCMakeBuildFactory( 'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
clean=True, clean=True,
python_source_dir=r'"C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64"',
target_arch='x64',
vs="autodetect",
test=True, test=True,
extra_cmake_args=[ extra_cmake_args=[
'-DLLVM_ENABLE_ASSERTIONS=OFF', "-DCMAKE_C_COMPILER_LAUNCHER=sccache",
'-DLLVM_ENABLE_ZLIB=FALSE', "-DCMAKE_CXX_COMPILER_LAUNCHER=sccache",
'-DLLDB_ENABLE_PYTHON=TRUE'])}, '-DLLVM_LIT_ARGS=-v',
'-DLLDB_TEST_USER_ARGS=--skip-category=watchpoint'])},
# LLD builders. # LLD builders.
{'name' : "lld-x86_64-darwin",
'tags' : ["lld"],
'workernames' : ["as-worker-3"],
'builddir': "lld-x86_64-darwin",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
depends_on_projects=['llvm', 'lld'],
extra_configure_args=[
'-DLLVM_ENABLE_WERROR=OFF'])},
{'name' : "lld-x86_64-win", {'name' : "lld-x86_64-win",
'tags' : ["lld"], 'tags' : ["lld"],
'workernames' : ["as-worker-93"], 'workernames' : ["as-worker-93"],
'builddir': "lld-x86_64-win", 'builddir': "lld-x86_64-win",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
depends_on_projects=['llvm', 'lld'], depends_on_projects=['llvm', 'lld'],
vs="autodetect", vs="autodetect",
extra_configure_args = [ extra_configure_args = [
'-DLLVM_ENABLE_WERROR=OFF'])}, '-DLLVM_ENABLE_WERROR=OFF'])},
{'name' : "lld-x86_64-freebsd",
'tags' : ["lld"],
'workernames' : ["as-worker-4"],
'builddir': "lld-x86_64-freebsd",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=['llvm', 'lld'],
extra_configure_args=[
'-DCMAKE_EXE_LINKER_FLAGS=-lcxxrt',
'-DLLVM_ENABLE_WERROR=OFF'],
env={'CXXFLAGS' : "-std=c++11 -stdlib=libc++"})},
{'name' : "ppc64le-lld-multistage-test", {'name' : "ppc64le-lld-multistage-test",
'tags' : ["lld", "ppc", "ppc64le"], 'tags' : ["lld", "ppc", "ppc64le"],
'workernames' : ["ppc64le-lld-multistage-test"], 'workernames' : ["ppc64le-lld-multistage-test"],
'builddir': "ppc64le-lld-multistage-test", 'builddir': "ppc64le-lld-multistage-test",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaMultistageBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaMultistageBuildFactory(
extra_configure_args=[ extra_configure_args=[
'-DLLVM_ENABLE_ASSERTIONS=ON', '-DLLVM_ENABLE_ASSERTIONS=ON',
'-DLLVM_LIT_ARGS=-svj 256'], '-DLLVM_LIT_ARGS=-svj 256',
'-DLLVM_CCACHE_BUILD=ON'],
depends_on_projects=['llvm', 'clang', 'lld'])}, depends_on_projects=['llvm', 'clang', 'lld'])},
{'name' : "lld-x86_64-ubuntu-fast", {'name' : "lld-x86_64-ubuntu-fast",
'tags' : ["lld"], 'tags' : ["lld"],
'collapseRequests': False, 'collapseRequests': False,
'workernames' : ["as-builder-4"], 'workernames' : ["as-builder-4"],
'builddir' : "lld-x86_64-ubuntu-fast", 'builddir' : "lld-x86_64-ubuntu-fast",
'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
Show All 9 Lines# LLD builders.
'factory' : LLDPerformanceTestsuite.getFactory(targets=["bin/lld"])}, 'factory' : LLDPerformanceTestsuite.getFactory(targets=["bin/lld"])},
# LTO and ThinLTO builders. # LTO and ThinLTO builders.
{'name' : "clang-with-thin-lto-ubuntu", {'name' : "clang-with-thin-lto-ubuntu",
'tags' : ["clang","lld","LTO"], 'tags' : ["clang","lld","LTO"],
'workernames' : ["as-worker-92"], 'workernames' : ["as-worker-92"],
'builddir': "clang-with-thin-lto-ubuntu", 'builddir': "clang-with-thin-lto-ubuntu",
'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(jobs=72, lto='thin')}, 'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(
jobs=72,
lto='thin',
)},
{'name' : "clang-with-thin-lto-wpd-ubuntu", {'name' : "clang-with-thin-lto-wpd-ubuntu",
'tags' : ["clang","lld","LTO"], 'tags' : ["clang","lld","LTO"],
'workernames' : ["thinlto-x86-64-bot1"], 'workernames' : ["thinlto-x86-64-bot1", "thinlto-x86-64-bot2"],
'builddir': "clang-with-thin-lto-wpd-ubuntu", 'builddir': "clang-with-thin-lto-wpd-ubuntu",
'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory( 'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(
jobs=72, jobs=72,
lto='thin', lto='thin',
extra_configure_args=[ extra_configure_args=[
'-DLLVM_CCACHE_BUILD=ON', '-DLLVM_CCACHE_BUILD=ON',
'-DCMAKE_CXX_FLAGS="-O3 -Xclang -fwhole-program-vtables -fno-split-lto-unit"', ],
'-DCMAKE_C_FLAGS="-O3 -Xclang -fwhole-program-vtables -fno-split-lto-unit"', extra_configure_args_lto_stage=[
'-DCMAKE_EXE_LINKER_FLAGS="-Wl,--lto-whole-program-visibility"'])}, '-DCMAKE_CXX_FLAGS=-O3 -Xclang -fwhole-program-vtables -fno-split-lto-unit',
'-DCMAKE_C_FLAGS=-O3 -Xclang -fwhole-program-vtables -fno-split-lto-unit',
'-DCMAKE_EXE_LINKER_FLAGS=-Wl,--lto-whole-program-visibility -fuse-ld=lld'])},
{'name' : "clang-with-lto-ubuntu", {'name' : "clang-with-lto-ubuntu",
'tags' : ["clang","lld","LTO"], 'tags' : ["clang","lld","LTO"],
'workernames' : ["as-worker-91"], 'workernames' : ["as-worker-91"],
'builddir': "clang-with-lto-ubuntu", 'builddir': "clang-with-lto-ubuntu",
'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(jobs=72)}, 'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(
jobs=72,
extra_configure_args_lto_stage=[
'-DLLVM_PARALLEL_LINK_JOBS=14',
])},
]
# Common builders options for MLIR.
mlir_ubuntu_workers = [f"mlir-ubuntu-worker{i}" for i in range(5)]
mlir_default_cmake_options = [
'-DLLVM_CCACHE_BUILD=ON',
'-DLLVM_ENABLE_PROJECTS=mlir',
'-DLLVM_TARGETS_TO_BUILD=host;NVPTX;AMDGPU',
'-DLLVM_BUILD_EXAMPLES=ON',
'-DMLIR_INCLUDE_INTEGRATION_TESTS=ON',
'-DMLIR_ENABLE_BINDINGS_PYTHON=ON',
]
all += [
{'name' : "mlir-ubuntu-asan-ubsan-clang-12.0",
'tags' : ["mlir"],
'workernames' : mlir_ubuntu_workers,
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
llvm_srcdir="llvm.src",
obj_dir="llvm.obj",
clean=True,
targets = ['check-mlir-build-only'],
checks = ['check-mlir'],
depends_on_projects=['llvm','mlir'],
extra_configure_args= mlir_default_cmake_options + [
'-DLLVM_ENABLE_LLD=ON',
'-DBUILD_SHARED_LIBS=ON',
'-DLLVM_USE_SANITIZER=Address;Undefined',
'-DCMAKE_C_COMPILER=/compilers/clang-12.0/bin/clang',
'-DCMAKE_CXX_COMPILER=/compilers/clang-12.0/bin/clang++',
])},
# Builders for MLIR. {'name' : "mlir-ubuntu-gcc5.5-shared-libs",
'tags' : ["mlir"],
'workernames' : mlir_ubuntu_workers,
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
llvm_srcdir="llvm.src",
obj_dir="llvm.obj",
clean=True,
targets = ['check-mlir-build-only'],
checks = ['check-mlir'],
depends_on_projects=['llvm','mlir'],
extra_configure_args= mlir_default_cmake_options + [
'-DBUILD_SHARED_LIBS=ON',
'-DCMAKE_C_COMPILER=/usr/bin/gcc-5',
'-DCMAKE_CXX_COMPILER=/usr/bin/g++-5',
])},
{'name' : "mlir-ubuntu-clang-5-link-llvm-dylib",
'tags' : ["mlir"],
'workernames' : mlir_ubuntu_workers,
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
llvm_srcdir="llvm.src",
obj_dir="llvm.obj",
clean=True,
targets = ['check-mlir-build-only'],
checks = ['check-mlir'],
depends_on_projects=['llvm','mlir'],
extra_configure_args= mlir_default_cmake_options + [
'-DLLVM_ENABLE_LLD=OFF', # lld-5 is subject to http://llvm.org/pr49915
'-DLLVM_LINK_LLVM_DYLIB=ON',
'-DCMAKE_C_COMPILER=/compilers/clang-5.0/bin/clang',
'-DCMAKE_CXX_COMPILER=/compilers/clang-5.0/bin/clang++',
])},
{'name' : "mlir-ubuntu-gcc11-release-intel-sde",
'tags' : ["mlir"],
'workernames' : mlir_ubuntu_workers,
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
llvm_srcdir="llvm.src",
obj_dir="llvm.obj",
clean=True,
targets = ['check-mlir-build-only'],
checks = ['check-mlir'],
depends_on_projects=['llvm','mlir'],
extra_configure_args= mlir_default_cmake_options + [
'-DLLVM_ENABLE_LLD=ON',
'-DLLVM_ENABLE_ASSERTIONS=OFF',
'-DCMAKE_C_COMPILER=/usr/bin/gcc-11',
'-DCMAKE_CXX_COMPILER=/usr/bin/g++-11',
'-DMLIR_RUN_X86VECTOR_TESTS=ON',
'-DMLIR_RUN_AMX_TESTS=ON',
'-DINTEL_SDE_EXECUTABLE=/intel_sde/sde64'
])},
{'name' : "mlir-nvidia", {'name' : "mlir-nvidia",
'tags' : ["mlir"], 'tags' : ["mlir"],
'workernames' : ["mlir-nvidia"], 'workernames' : ["mlir-nvidia"],
'builddir': "mlir-nvidia", 'builddir': "mlir-nvidia",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
llvm_srcdir="llvm.src", llvm_srcdir="llvm.src",
obj_dir="llvm.obj", obj_dir="llvm.obj",
clean=True, clean=True,
targets = ['check-mlir-build-only'],
checks = ['check-mlir'],
depends_on_projects=['llvm','mlir'], depends_on_projects=['llvm','mlir'],
extra_configure_args=[ extra_configure_args=[
'-DLLVM_BUILD_EXAMPLES=ON', '-DLLVM_BUILD_EXAMPLES=ON',
'-DLLVM_TARGETS_TO_BUILD=host;NVPTX', '-DLLVM_TARGETS_TO_BUILD=host;NVPTX',
'-DLLVM_ENABLE_PROJECTS=mlir', '-DLLVM_ENABLE_PROJECTS=mlir',
'-DMLIR_CUDA_RUNNER_ENABLED=1', '-DMLIR_ENABLE_CUDA_RUNNER=1',
'-DMLIR_INCLUDE_INTEGRATION_TESTS=ON',
'-DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc', '-DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc',
'-DMLIR_VULKAN_RUNNER_ENABLED=1', '-DMLIR_ENABLE_VULKAN_RUNNER=1',
'-DBUILD_SHARED_LIBS=ON', '-DBUILD_SHARED_LIBS=ON',
'-DLLVM_CCACHE_BUILD=ON', '-DLLVM_CCACHE_BUILD=ON',
'-DMLIR_ENABLE_BINDINGS_PYTHON=ON',
'-DMLIR_RUN_CUDA_TENSOR_CORE_TESTS=ON',
], ],
env={ env={
'CC':'clang', 'CC':'clang',
'CXX': 'clang++', 'CXX': 'clang++',
'LD': 'lld', 'LD': 'lld',
})}, })},
{'name' : "mlir-windows", {'name' : "mlir-windows",
'tags' : ["mlir"], 'tags' : ["mlir"],
'workernames' : ["win-mlir-buildbot"], 'workernames' : ["win-mlir-buildbot"],
'builddir': "mlir-x64-windows-ninja", 'builddir': "mlir-x64-windows-ninja",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
clean=True, clean=True,
targets = ['check-mlir-build-only'],
checks = ['check-mlir'],
depends_on_projects=['llvm','mlir'], depends_on_projects=['llvm','mlir'],
vs="autodetect", vs="autodetect",
checks=['check-mlir'],
extra_configure_args=[ extra_configure_args=[
"-DLLVM_BUILD_EXAMPLES=ON", "-DLLVM_BUILD_EXAMPLES=ON",
"-DLLVM_ENABLE_PROJECTS=mlir", "-DLLVM_ENABLE_PROJECTS=mlir",
"-DMLIR_ENABLE_BINDINGS_PYTHON=ON",
"-DLLVM_ENABLE_WERROR=ON",
"-DLLVM_TARGETS_TO_BUILD='host;NVPTX;AMDGPU'", "-DLLVM_TARGETS_TO_BUILD='host;NVPTX;AMDGPU'",
])}, ])},
{'name' : 'ppc64le-mlir-rhel-clang', {'name' : 'ppc64le-mlir-rhel-clang',
'tags' : ["mlir", "ppc", "ppc64le"], 'tags' : ["mlir", "ppc", "ppc64le"],
'collapseRequests' : False, 'collapseRequests' : False,
'workernames' : ['ppc64le-flang-mlir-rhel-test'], 'workernames' : ['ppc64le-mlir-rhel-test'],
'builddir': 'ppc64le-mlir-rhel-clang-build', 'builddir': 'ppc64le-mlir-rhel-clang-build',
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True, clean=True,
depends_on_projects=['llvm', 'mlir'], depends_on_projects=['llvm', 'mlir'],
targets = ['check-mlir-build-only'],
checks=['check-mlir'], checks = ['check-mlir'],
extra_configure_args=[ extra_configure_args=[
'-DLLVM_TARGETS_TO_BUILD=PowerPC', '-DLLVM_TARGETS_TO_BUILD=PowerPC',
'-DLLVM_INSTALL_UTILS=ON', '-DLLVM_INSTALL_UTILS=ON',
'-DCMAKE_CXX_STANDARD=17', '-DCMAKE_CXX_STANDARD=17',
'-DLLVM_ENABLE_PROJECTS=mlir', '-DLLVM_ENABLE_PROJECTS=mlir',
'-DLLVM_LIT_ARGS=-vj 256', '-DLLVM_LIT_ARGS=-vj 256',
'-DLLVM_CCACHE_BUILD=ON',
], ],
env={ env={
'CC': 'clang', 'CC': 'clang',
'CXX': 'clang++', 'CXX': 'clang++',
'LD': 'lld', 'LD': 'lld',
'LD_LIBRARY_PATH': '/usr/lib64',
})}, })},
{'name' : 'mlir-s390x-linux',
'tags' : ["mlir", "s390x"],
'workernames' : ["systemz-1"],
'builddir': 'mlir-s390x-linux',
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
depends_on_projects=['llvm', 'mlir'],
checks=['check-mlir'],
extra_configure_args=[
"-DLLVM_CCACHE_BUILD=ON",
'-DLLVM_TARGETS_TO_BUILD=SystemZ',
'-DLLVM_ENABLE_PROJECTS=mlir',
'-DLLVM_LIT_ARGS=-vj 4',
])},
{'name' : "mlir-s390x-linux-werror",
'tags' : ["mlir", "s390x"],
'workernames' : ["onnx-mlir-nowarn-linux-s390x"],
'builddir': "onnx-mlir-nowarn-linux-s390x",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
checks = ['check-mlir'],
targets = ['check-mlir-build-only'],
depends_on_projects=['llvm','mlir'],
extra_configure_args=[
"-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_ENABLE_PROJECTS=mlir",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_ENABLE_RTTI=ON",
"-DLLVM_ENABLE_WERROR=ON",
"-DLLVM_TARGETS_TO_BUILD=host",
])},
# Sanitizer builders. # Sanitizer builders.
#
# bootstrap-asan, bootstrap-msan, and sanitizer-x86_64-linux-fast have steps
# with large memory usage, so assign them to different workers.
{'name' : "sanitizer-x86_64-linux", {'name' : "sanitizer-x86_64-linux",
'tags' : ["sanitizer"], 'tags' : ["sanitizer"],
'workernames' : ["sanitizer-buildbot1", "sanitizer-buildbot2"], 'workernames' : [
"sanitizer-buildbot1",
"sanitizer-buildbot2",
],
'builddir': "sanitizer-x86_64-linux", 'builddir': "sanitizer-x86_64-linux",
'factory' : SanitizerBuilder.getSanitizerBuildFactory()}, 'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
{'name' : "sanitizer-x86_64-linux-fast", {'name' : "sanitizer-x86_64-linux-fast",
'tags' : ["sanitizer"], 'tags' : ["sanitizer"],
'workernames' : ["sanitizer-buildbot1", "sanitizer-buildbot2"], 'workernames' : [
"sanitizer-buildbot3",
"sanitizer-buildbot4",
],
'builddir': "sanitizer-x86_64-linux-fast", 'builddir': "sanitizer-x86_64-linux-fast",
'factory' : SanitizerBuilder.getSanitizerBuildFactory()}, 'factory' : SanitizerBuilder.getSanitizerBuildFactory(
extra_depends_on_projects=["mlir", "clang-tools-extra"]
)},
{'name' : "sanitizer-x86_64-linux-bootstrap", {'name' : "sanitizer-x86_64-linux-bootstrap-asan",
'tags' : ["sanitizer"], 'tags' : ["sanitizer"],
'workernames' : ["sanitizer-buildbot3", "sanitizer-buildbot4"], 'workernames' : [
'builddir': "sanitizer-x86_64-linux-bootstrap", "sanitizer-buildbot1",
'factory' : SanitizerBuilder.getSanitizerBuildFactory()}, "sanitizer-buildbot2",
],
'builddir': "sanitizer-x86_64-linux-bootstrap-asan",
'factory' : SanitizerBuilder.getSanitizerBuildFactory(
extra_depends_on_projects=["mlir", "clang-tools-extra"]
)},
{'name' : "sanitizer-x86_64-linux-bootstrap-msan",
'tags' : ["sanitizer"],
'workernames' : [
"sanitizer-buildbot5",
"sanitizer-buildbot6",
],
'builddir': "sanitizer-x86_64-linux-bootstrap-msan",
'factory' : SanitizerBuilder.getSanitizerBuildFactory(
extra_depends_on_projects=["mlir", "clang-tools-extra"]
)},
{'name' : "sanitizer-x86_64-linux-bootstrap-ubsan", {'name' : "sanitizer-x86_64-linux-bootstrap-ubsan",
'tags' : ["sanitizer"], 'tags' : ["sanitizer"],
'workernames' : ["sanitizer-buildbot3", "sanitizer-buildbot4"], 'workernames' : [
"sanitizer-buildbot3",
"sanitizer-buildbot4",
],
'builddir': "sanitizer-x86_64-linux-bootstrap-ubsan", 'builddir': "sanitizer-x86_64-linux-bootstrap-ubsan",
'factory' : SanitizerBuilder.getSanitizerBuildFactory()}, 'factory' : SanitizerBuilder.getSanitizerBuildFactory(
extra_depends_on_projects=["mlir", "clang-tools-extra"]
)},
{'name' : "sanitizer-x86_64-linux-bootstrap-msan", {'name' : "sanitizer-x86_64-linux-autoconf",
'tags' : ["sanitizer"], 'tags' : ["sanitizer"],
'workernames' : ["sanitizer-buildbot7", "sanitizer-buildbot8"], 'workernames' : [
'builddir': "sanitizer-x86_64-linux-bootstrap-msan", "sanitizer-buildbot5",
"sanitizer-buildbot6",
],
'builddir': "sanitizer-x86_64-linux-autoconf",
'factory' : SanitizerBuilder.getSanitizerBuildFactory()}, 'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
{'name' : "sanitizer-x86_64-linux-autoconf", {'name' : "sanitizer-x86_64-linux-qemu",
'tags' : ["sanitizer"], 'tags' : ["sanitizer"],
'workernames' : ["sanitizer-buildbot7", "sanitizer-buildbot8"], 'workernames' : [
'builddir': "sanitizer-x86_64-linux-autoconf", "sanitizer-buildbot3",
"sanitizer-buildbot4",
],
'builddir': "sanitizer-x86_64-linux-qemu",
'factory' : SanitizerBuilder.getSanitizerBuildFactory()}, 'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
{'name' : "sanitizer-x86_64-linux-fuzzer", {'name' : "sanitizer-x86_64-linux-fuzzer",
'tags' : ["sanitizer"], 'tags' : ["sanitizer"],
'workernames' : ["sanitizer-buildbot7", "sanitizer-buildbot8"], 'workernames' : [
"sanitizer-buildbot1",
"sanitizer-buildbot2",
],
'builddir': "sanitizer-x86_64-linux-fuzzer", 'builddir': "sanitizer-x86_64-linux-fuzzer",
'factory' : SanitizerBuilder.getSanitizerBuildFactory()}, 'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
{'name' : "sanitizer-x86_64-linux-android", {'name' : "sanitizer-x86_64-linux-android",
'tags' : ["sanitizer"], 'tags' : ["sanitizer"],
'workernames' : ["sanitizer-buildbot6"], 'workernames' : [
"sanitizer-buildbot-android",
],
'builddir': "sanitizer-x86_64-linux-android", 'builddir': "sanitizer-x86_64-linux-android",
'factory' : SanitizerBuilder.getSanitizerBuildFactory()}, 'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
{'name' : "sanitizer-aarch64-linux-bootstrap-asan",
'tags' : ["sanitizer"],
'workernames' : [
"sanitizer-buildbot7",
"sanitizer-buildbot9",
"sanitizer-buildbot11",
],
'builddir': "sanitizer-aarch64-linux-bootstrap-asan",
'factory' : SanitizerBuilder.getSanitizerBuildFactory(
extra_depends_on_projects=["mlir", "clang-tools-extra"]
)},
{'name' : "sanitizer-aarch64-linux-bootstrap-hwasan",
'tags' : ["sanitizer"],
'workernames' : [
"sanitizer-buildbot7",
"sanitizer-buildbot9",
"sanitizer-buildbot11",
],
'builddir': "sanitizer-aarch64-linux-bootstrap-hwasan",
'factory' : SanitizerBuilder.getSanitizerBuildFactory(
extra_depends_on_projects=["mlir", "clang-tools-extra"]
)},
{'name' : "sanitizer-aarch64-linux-bootstrap-msan",
'tags' : ["sanitizer"],
'workernames' : [
"sanitizer-buildbot8",
"sanitizer-buildbot10",
"sanitizer-buildbot12",
],
'builddir': "sanitizer-aarch64-linux-bootstrap-msan",
'factory' : SanitizerBuilder.getSanitizerBuildFactory(
extra_depends_on_projects=["mlir", "clang-tools-extra"]
)},
{'name' : "sanitizer-aarch64-linux-bootstrap-ubsan",
'tags' : ["sanitizer"],
'workernames' : [
"sanitizer-buildbot8",
"sanitizer-buildbot10",
"sanitizer-buildbot12",
],
'builddir': "sanitizer-aarch64-linux-bootstrap-ubsan",
'factory' : SanitizerBuilder.getSanitizerBuildFactory(
extra_depends_on_projects=["mlir", "clang-tools-extra"]
)},
{'name' : "sanitizer-aarch64-linux-fuzzer",
'tags' : ["sanitizer"],
'workernames' : [
"sanitizer-buildbot8",
"sanitizer-buildbot10",
"sanitizer-buildbot12",
],
'builddir': "sanitizer-aarch64-linux-fuzzer",
'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
{'name' : "sanitizer-ppc64be-linux", {'name' : "sanitizer-ppc64be-linux",
'tags' : ["sanitizer", "ppc"], 'tags' : ["sanitizer", "ppc"],
'workernames' : ["ppc64be-sanitizer"], 'workernames' : ["ppc64be-sanitizer"],
'builddir': "sanitizer-ppc64be", 'builddir': "sanitizer-ppc64be",
'factory' : SanitizerBuilder.getSanitizerBuildFactory(timeout=1800)}, 'factory' : SanitizerBuilder.getSanitizerBuildFactory(timeout=1800)},
{'name' : "sanitizer-ppc64le-linux", {'name' : "sanitizer-ppc64le-linux",
'tags' : ["sanitizer", "ppc", "ppc64le"], 'tags' : ["sanitizer", "ppc", "ppc64le"],
'workernames' : ["ppc64le-sanitizer"], 'workernames' : ["ppc64le-sanitizer"],
'builddir': "sanitizer-ppc64le", 'builddir': "sanitizer-ppc64le",
'factory' : SanitizerBuilder.getSanitizerBuildFactory(timeout=1800)}, 'factory' : SanitizerBuilder.getSanitizerBuildFactory(timeout=1800)},
{'name' : "sanitizer-windows", {'name' : "sanitizer-windows",
'tags' : ["sanitizer"], 'tags' : ["sanitizer"],
'workernames' : ["sanitizer-windows"], 'workernames' : ["sanitizer-windows"],
'builddir': "sanitizer-windows", 'builddir': "sanitizer-windows",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory( 'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="sanitizer-windows.py", script="sanitizer-windows.py",
depends_on_projects=["llvm", "clang", "lld", "compiler-rt"])}, depends_on_projects=["llvm", "clang", "lld", "compiler-rt"])},
# OpenMP builders.
{'name' : "openmp-gcc-x86_64-linux-debian", {'name' : "openmp-gcc-x86_64-linux-debian",
'tags' : ["openmp"], 'tags' : ["openmp"],
'workernames' : ["gribozavr4"], 'workernames' : ["gribozavr4"],
'builddir': "openmp-gcc-x86_64-linux-debian", 'builddir': "openmp-gcc-x86_64-linux-debian",
'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory( 'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
extraCmakeArgs=[
'-DLLVM_CCACHE_BUILD=ON',
],
env={ env={
'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin', 'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes', 'CC': 'clang', 'CXX': 'clang++',
})}, })},
{'name' : "openmp-clang-x86_64-linux-debian", {'name' : "openmp-clang-x86_64-linux-debian",
'tags' : ["openmp"], 'tags' : ["openmp"],
'workernames' : ["gribozavr4"], 'workernames' : ["gribozavr4"],
'builddir': "openmp-clang-x86_64-linux-debian", 'builddir': "openmp-clang-x86_64-linux-debian",
'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory( 'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
extraCmakeArgs=[
'-DLLVM_CCACHE_BUILD=ON',
],
env={ env={
'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin', 'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes', 'CC': 'clang', 'CXX': 'clang++',
})}, })},
# Libc++ builders. {'name' : "openmp-offload-cuda-project",
'tags' : ["openmp"],
'workernames' : ["minipc-1050ti-linux"],
'builddir': "openmp-offload-cuda-project",
'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
clean=False,
enable_runtimes=[],
extraCmakeArgs=[
"-DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda",
"-DLIBOMPTARGET_BUILD_NVPTX_BCLIB=ON",
"-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_OBJC_REWRITER=OFF",
"-DLLVM_TARGETS_TO_BUILD=X86;NVPTX",
"-DLLVM_ENABLE_LLD=ON",
'-DLLVM_PARALLEL_LINK_JOBS=2',
],
install=True,
testsuite=True,
testsuite_sollvevv=True,
extraTestsuiteCmakeArgs=[
"-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
"-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
],
)},
{'name': 'libcxx-libcxxabi-x86_64-linux-debian', {'name' : "openmp-offload-cuda-runtime",
'tags' : ["libcxx"], 'tags' : ["openmp"],
'workernames': ['gribozavr4'], 'workernames' : ["minipc-1050ti-linux"],
'builddir': 'libcxx-libcxxabi-x86_64-linux-debian', 'builddir': "openmp-offload-cuda-runtime",
'factory': LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder( 'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
env={'CC': 'clang', 'CXX': 'clang++'}, clean=True,
lit_extra_args=['--shuffle'], enable_runtimes=['openmp'],
check_libcxx_abilist=True)}, extraCmakeArgs=[
"-DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda",
"-DLIBOMPTARGET_BUILD_NVPTX_BCLIB=ON",
"-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_OBJC_REWRITER=OFF",
"-DLLVM_TARGETS_TO_BUILD=X86;NVPTX",
"-DLLVM_ENABLE_LLD=ON",
'-DLLVM_PARALLEL_LINK_JOBS=2',
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
],
install=True,
testsuite=True,
testsuite_sollvevv=True,
extraTestsuiteCmakeArgs=[
"-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
"-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDFLAGS=-fopenmp-targets=nvptx64-nvidia-cuda;--cuda-path=/opt/cuda",
],
)},
{'name' : 'libcxx-libcxxabi-x86_64-linux-debian-noexceptions', # OpenMP AMDGPU Builders
'tags' : ["libcxx"], {'name' : "openmp-offload-amdgpu-runtime",
'workernames' : ['gribozavr4'], 'tags' : ["openmp"],
'builddir': 'libcxx-libcxxabi-x86_64-linux-debian-noexceptions', 'workernames' : ["omp-vega20-0"],
'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder( 'builddir': "openmp-offload-amdgpu-runtime",
env={'CC': 'clang', 'CXX': 'clang++'}, 'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
use_cache='Generic-noexceptions.cmake', clean=True,
lit_extra_args=['--shuffle'])}, enable_runtimes=['openmp'],
depends_on_projects=['llvm','clang','lld','openmp'],
extraCmakeArgs=[
"-DCMAKE_BUILD_TYPE=Release",
"-DCLANG_DEFAULT_LINKER=lld",
"-DLLVM_TARGETS_TO_BUILD=X86;AMDGPU",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_ENABLE_RUNTIMES=openmp",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
],
install=True,
testsuite=False,
testsuite_sollvevv=False,
extraTestsuiteCmakeArgs=[
"-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp-targets=amdgcn-amd-amdhsa;-Xopenmp-target=amdgcn-amd-amdhsa",
"-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDLAGS=-fopenmp-targets=amdgcn-amd-amdhsa;-Xopenmp-target=amdgcn-amd-amdhsa",
],
)},
{'name' : 'libcxx-libcxxabi-libunwind-x86_64-linux-debian', {'name' : "openmp-offload-libc-amdgpu-runtime",
'tags' : ["libcxx"], 'tags' : ["openmp"],
'workernames' : ['gribozavr4'], 'workernames' : ["omp-vega20-1"],
'builddir': 'libcxx-libcxxabi-libunwind-x86_64-linux-debian', 'builddir': "openmp-offload-libc-amdgpu-runtime",
'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder( 'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
env={'CC': 'clang', 'CXX': 'clang++'}, clean=True,
cmake_extra_opts={'LIBCXXABI_USE_LLVM_UNWINDER': 'ON'}, depends_on_projects=['llvm', 'clang', 'compiler-rt', 'libc', 'lld', 'openmp'],
lit_extra_args=['--shuffle'])}, extraCmakeArgs=[
"-DCMAKE_BUILD_TYPE=Release",
"-DCLANG_DEFAULT_LINKER=lld",
"-DLLVM_TARGETS_TO_BUILD=X86;AMDGPU",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLIBOMPTARGET_FOUND_AMDGPU_GPU=ON",
"-DLIBOMP_ARCHER_SUPPORT=OFF",
"-DLIBC_GPU_BUILD=ON",
"-DLIBC_GPU_ARCHITECTURES=gfx906",
"-DLIBC_GPU_TEST_ARCHITECTURE=gfx906",
"-DLIBC_GPU_TEST_JOBS=1",
],
install=True,
testsuite=False,
testsuite_sollvevv=False,
extraTestsuiteCmakeArgs=[
"-DTEST_SUITE_SOLLVEVV_OFFLOADING_CFLAGS=-fopenmp;-fopenmp-targets=amdgcn-amd-amdhsa;-Xopenmp-target=amdgcn-amd-amdhsa;-march=gfx906",
"-DTEST_SUITE_SOLLVEVV_OFFLOADING_LDLAGS=-fopenmp;-fopenmp-targets=amdgcn-amd-amdhsa;-Xopenmp-target=amdgcn-amd-amdhsa;-march=gfx906",
],
add_lit_checks=["check-clang", "check-llvm", "check-lld", "check-libc"]
)},
{'name' : 'libcxx-libcxxabi-singlethreaded-x86_64-linux-debian',
'tags' : ["libcxx"], # Whole-toolchain builders.
'workernames' : ['gribozavr4'],
'builddir': 'libcxx-libcxxabi-singlethreaded-x86_64-linux-debian',
'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
env={'CC': 'clang', 'CXX': 'clang++'},
use_cache='Generic-singlethreaded.cmake')},
# ARMv7 LibC++ and LibC++abi tests (require Clang+RT)
{'name' : 'libcxx-libcxxabi-libunwind-armv7-linux',
'tags' : ["libcxx"],
'workernames': ['linaro-tk1-02'],
'builddir': 'libcxx-libcxxabi-libunwind-armv7-linux',
'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
cmake_extra_opts={
'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
'CMAKE_C_FLAGS': '-mcpu=cortex-a15 -marm',
'CMAKE_CXX_FLAGS': '-mcpu=cortex-a15 -marm',
'LLVM_PARALLEL_LINK_JOBS': '2'})},
# ARMv8 LibC++ and LibC++abi tests (require Clang+RT)
{'name' : 'libcxx-libcxxabi-libunwind-armv8-linux',
'tags' : ["libcxx"],
'workernames' : ['linaro-armv8-libcxx'],
'builddir': 'libcxx-libcxxabi-libunwind-armv8-linux',
'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
cmake_extra_opts={
'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
'CMAKE_C_FLAGS': '-mcpu=cortex-a57 -marm',
'CMAKE_CXX_FLAGS': '-mcpu=cortex-a57 -marm'})},
# ARMv7 LibC++ and LibC++abi tests w/o EH (require Clang+RT)
{'name' : 'libcxx-libcxxabi-libunwind-armv7-linux-noexceptions',
'tags' : ["libcxx"],
'workernames' : ['linaro-tk1-02'],
'builddir': 'libcxx-libcxxabi-libunwind-armv7-linux-noexceptions',
'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
cmake_extra_opts={
'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
'LIBCXX_ENABLE_EXCEPTIONS': 'OFF',
'LIBCXXABI_ENABLE_EXCEPTIONS': 'OFF',
'CMAKE_C_FLAGS': '-mcpu=cortex-a15 -mthumb',
'CMAKE_CXX_FLAGS': '-mcpu=cortex-a15 -mthumb',
'LLVM_PARALLEL_LINK_JOBS': '2'})},
# ARMv8 LibC++ and LibC++abi tests w/o EH (require Clang+RT)
{'name' : 'libcxx-libcxxabi-libunwind-armv8-linux-noexceptions',
'tags' : ["libcxx"],
'workernames' : ['linaro-armv8-libcxx'],
'builddir': 'libcxx-libcxxabi-libunwind-armv8-linux-noexceptions',
'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
cmake_extra_opts={
'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
'LIBCXX_ENABLE_EXCEPTIONS': 'OFF',
'LIBCXXABI_ENABLE_EXCEPTIONS': 'OFF',
'CMAKE_C_FLAGS': '-mcpu=cortex-a57 -mthumb',
'CMAKE_CXX_FLAGS': '-mcpu=cortex-a57 -mthumb'})},
# AArch64 LibC++ and LibC++abi tests (require Clang+RT)
{'name' : 'libcxx-libcxxabi-libunwind-aarch64-linux',
'tags' : ["libcxx"],
'workernames' : ['linaro-aarch64-libcxx'],
'builddir': 'libcxx-libcxxabi-libunwind-aarch64-linux',
'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
cmake_extra_opts={
'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
'CMAKE_C_FLAGS': '-mcpu=cortex-a57',
'CMAKE_CXX_FLAGS': '-mcpu=cortex-a57'})},
{'name' : 'libcxx-libcxxabi-libunwind-aarch64-linux-noexceptions',
'tags' : ["libcxx"],
'workernames' : ['linaro-aarch64-libcxx'],
'builddir': 'libcxx-libcxxabi-libunwind-aarch64-linux-noexceptions',
'factory' : LibcxxAndAbiBuilder.getLibcxxAndAbiBuilder(
cmake_extra_opts={
'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
'LIBCXX_ENABLE_EXCEPTIONS': 'OFF',
'LIBCXXABI_ENABLE_EXCEPTIONS': 'OFF',
'CMAKE_C_FLAGS': '-mcpu=cortex-a57',
'CMAKE_CXX_FLAGS': '-mcpu=cortex-a57'})},
{'name': "fuchsia-x86_64-linux", {'name': "fuchsia-x86_64-linux",
'tags' : ["toolchain"], 'tags' : ["toolchain"],
'workernames' :["fuchsia-debian-64-us-central1-a-1", "fuchsia-debian-64-us-central1-b-1"], 'workernames' :["fuchsia-debian-64-us-central1-a-1", "fuchsia-debian-64-us-central1-b-1"],
'builddir': "fuchsia-x86_64-linux", 'builddir': "fuchsia-x86_64-linux",
'factory': FuchsiaBuilder.getFuchsiaToolchainBuildFactory()}, 'factory': FuchsiaBuilder.getFuchsiaToolchainBuildFactory()},
# libc Builders. # libc Builders.
{'name' : 'libc-x86_64-windows-dbg',
'tags' : ["libc"],
'workernames' : ['libc-x86_64-windows'],
'builddir': 'libc-x86_64-windows',
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="libc-windows.py",
depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
extra_args=['--debug'])},
{'name' : 'libc-arm32-debian-dbg',
'tags' : ["libc"],
'workernames' : ['libc-arm32-debian'],
'builddir': 'libc-arm32-debian-dbg',
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="libc-linux.py",
depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
extra_args=['--debug'])},
{'name' : 'libc-aarch64-ubuntu-dbg',
'tags' : ["libc"],
'workernames' : ['libc-aarch64-ubuntu'],
'builddir': 'libc-aarch64-ubuntu',
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="libc-linux.py",
depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
extra_args=['--debug'])},
{'name' : "libc-aarch64-ubuntu-fullbuild-dbg",
'tags' : ["libc"],
'workernames' : ["libc-aarch64-ubuntu"],
'builddir': "libc-aarch64-ubuntu-fullbuild-dbg",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="libc-linux.py",
depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
extra_args=['--debug'])},
{'name' : 'libc-x86_64-debian', {'name' : 'libc-x86_64-debian',
'tags' : ["libc"], 'tags' : ["libc"],
'workernames' : ['libc-x86_64-debian'], 'workernames' : ['libc-x86_64-debian'],
'builddir': 'libc-x86_64-debian', 'builddir': 'libc-x86_64-debian',
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory( 'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="libc-linux.py", script="libc-linux.py",
depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'])}, depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'])},
Show All 10 Lines# libc Builders.
'tags' : ["libc"], 'tags' : ["libc"],
'workernames' : ["libc-x86_64-debian"], 'workernames' : ["libc-x86_64-debian"],
'builddir': "libc-x86_64-debian-dbg-asan", 'builddir': "libc-x86_64-debian-dbg-asan",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory( 'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="libc-linux.py", script="libc-linux.py",
depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'], depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
extra_args=['--debug', '--asan'])}, extra_args=['--debug', '--asan'])},
# Flang builders. {'name' : "libc-x86_64-debian-dbg-runtimes-build",
'tags' : ["libc"],
'workernames' : ["libc-x86_64-debian"],
'builddir': "libc-x86_64-debian-dbg-runtimes-build",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="libc-linux.py",
depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
extra_args=['--debug'])},
{'name' : "flang-aarch64-ubuntu", {'name' : "libc-x86_64-debian-fullbuild-dbg",
'tags' : ["flang"], 'tags' : ["libc"],
'workernames' : ["flang-aarch64-ubuntu-build"], 'workernames' : ["libc-x86_64-debian-fullbuild"],
'builddir': "flang-aarch64-ubuntu-build", 'builddir': "libc-x86_64-debian-fullbuild-dbg",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
clean=True, script="libc-linux.py",
depends_on_projects=['llvm','mlir','clang','flang'], depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
extra_configure_args=[ extra_args=['--debug'])},
"-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DCMAKE_C_COMPILER=/usr/bin/gcc-9", {'name' : "libc-x86_64-debian-gcc-fullbuild-dbg",
"-DCMAKE_CXX_COMPILER=/usr/bin/g++-9", 'tags' : ["libc"],
"-DLLVM_INSTALL_UTILS=ON", 'workernames' : ["libc-x86_64-debian-fullbuild"],
"-DFLANG_BUILD_NEW_DRIVER=ON", 'builddir': "libc-x86_64-debian-gcc-fullbuild-dbg",
"-DCMAKE_CXX_STANDARD=17", 'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
])}, script="libc-linux.py",
depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
extra_args=['--debug'])},
{'name' : "libc-x86_64-debian-fullbuild-dbg-asan",
'tags' : ["libc"],
'workernames' : ["libc-x86_64-debian-fullbuild"],
'builddir': "libc-x86_64-debian-fullbuild-dbg-asan",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="libc-linux.py",
depends_on_projects=['llvm', 'libc', 'clang', 'clang-tools-extra'],
extra_args=['--debug', '--asan'])},
{'name' : "flang-aarch64-ubuntu-dylib", {'name' : "libc-x86_64-debian-dbg-lint",
'tags' : ["libc"],
'workernames' : ["libc-lint-worker"],
'builddir': "libc-x86_64-debian-dbg-lint",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="libc-linux.py",
depends_on_projects=['llvm', 'libc'],
extra_args=['--debug'])},
{'name' : 'libc-riscv64-debian-dbg',
'tags' : ["libc"],
'workernames' : ['libc-riscv64-debian'],
'builddir': 'libc-riscv64-debian-dbg',
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="libc-linux.py",
depends_on_projects=['llvm', 'libc'],
extra_args=['--debug'])},
{'name' : "libc-riscv64-debian-fullbuild-dbg",
'tags' : ["libc"],
'workernames' : ["libc-riscv64-debian"],
'builddir': "libc-riscv64-debian-fullbuild-dbg",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="libc-linux.py",
depends_on_projects=['llvm', 'libc'],
extra_args=['--debug'])},
# Flang builders.
{'name' : "flang-aarch64-dylib",
'tags' : ["flang"], 'tags' : ["flang"],
'workernames' : ["linaro-aarch64-flang-dylib"], 'workernames' : ["linaro-flang-aarch64-dylib"],
'builddir': "flang-aarch64-ubuntu-dylib", 'builddir': "flang-aarch64-dylib",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True, clean=True,
checks=['check-flang'],
depends_on_projects=['llvm','mlir','clang','flang'], depends_on_projects=['llvm','mlir','clang','flang'],
extra_configure_args=[ extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=AArch64", "-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DFLANG_BUILD_NEW_DRIVER=ON",
"-DLLVM_BUILD_LLVM_DYLIB=ON", "-DLLVM_BUILD_LLVM_DYLIB=ON",
"-DLLVM_LINK_LLVM_DYLIB=ON", "-DLLVM_LINK_LLVM_DYLIB=ON",
"-DCMAKE_CXX_STANDARD=17", "-DCMAKE_CXX_STANDARD=17",
])}, ])},
{'name' : "flang-aarch64-ubuntu-sharedlibs", {'name' : "flang-aarch64-sharedlibs",
'tags' : ["flang"], 'tags' : ["flang"],
'workernames' : ["linaro-aarch64-flang-sharedlibs"], 'workernames' : ["linaro-flang-aarch64-sharedlibs"],
'builddir': "flang-aarch64-ubuntu-sharedlibs", 'builddir': "flang-aarch64-sharedlibs",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True, clean=True,
checks=['check-flang'],
depends_on_projects=['llvm','mlir','clang','flang'], depends_on_projects=['llvm','mlir','clang','flang'],
extra_configure_args=[ extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=AArch64", "-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DFLANG_BUILD_NEW_DRIVER=ON",
"-DBUILD_SHARED_LIBS=ON", "-DBUILD_SHARED_LIBS=ON",
"-DLLVM_BUILD_EXAMPLES=ON", "-DLLVM_BUILD_EXAMPLES=ON",
"-DCMAKE_CXX_STANDARD=17", "-DCMAKE_CXX_STANDARD=17",
])}, ])},
{'name' : "flang-aarch64-ubuntu-out-of-tree", {'name' : "flang-aarch64-out-of-tree",
'tags' : ["flang"], 'tags' : ["flang"],
'workernames' : ["linaro-aarch64-flang-oot"], 'workernames' : ["linaro-flang-aarch64-out-of-tree"],
'builddir': "flang-aarch64-out-of-tree", 'builddir': "flang-aarch64-out-of-tree",
'factory' : FlangBuilder.getFlangOutOfTreeBuildFactory( 'factory' : FlangBuilder.getFlangOutOfTreeBuildFactory(
checks=['check-flang'],
llvm_extra_configure_args=[ llvm_extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=AArch64", "-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DCMAKE_CXX_STANDARD=17", "-DCMAKE_CXX_STANDARD=17",
"-DLLVM_ENABLE_WERROR=OFF", "-DLLVM_ENABLE_WERROR=OFF",
"-DLLVM_ENABLE_ASSERTIONS=ON", "-DLLVM_ENABLE_ASSERTIONS=ON",
"-DCMAKE_BUILD_TYPE=Release", "-DCMAKE_BUILD_TYPE=Release",
], ],
flang_extra_configure_args=[ flang_extra_configure_args=[
"-DFLANG_ENABLE_WERROR=ON", "-DFLANG_ENABLE_WERROR=ON",
"-DCMAKE_BUILD_TYPE=Release", "-DCMAKE_BUILD_TYPE=Release",
])}, ])},
{'name' : "flang-x86_64-linux", {'name' : "flang-aarch64-debug",
'tags' : ["flang"], 'tags' : ["flang"],
'workernames' : ["nersc-flang"], 'workernames' : ["linaro-flang-aarch64-debug"],
'builddir': "flang-x86_64-linux", 'builddir': "flang-aarch64-debug",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
checks=['check-flang'],
depends_on_projects=['llvm','mlir','clang','flang'], depends_on_projects=['llvm','mlir','clang','flang'],
extra_configure_args=[ extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=X86", "-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DCMAKE_BUILD_TYPE=Debug",
"-DCMAKE_CXX_STANDARD=17",
"-DLLVM_USE_LINKER=lld",
])},
{'name' : "flang-aarch64-latest-clang",
'tags' : ['flang'],
'workernames' : ["linaro-flang-aarch64-latest-clang"],
'builddir': "flang-aarch64-latest-clang",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
checks=['check-flang'],
depends_on_projects=['llvm','mlir','clang','flang'],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DLLVM_INSTALL_UTILS=ON", "-DLLVM_INSTALL_UTILS=ON",
"-DCMAKE_CXX_STANDARD=17", "-DCMAKE_CXX_STANDARD=17",
"-DLLVM_ENABLE_WERROR=OFF",
"-DFLANG_ENABLE_WERROR=ON",
"-DBUILD_SHARED_LIBS=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_ENABLE_LIBCXX=On",
"-DCMAKE_BUILD_TYPE=Release",
])},
{'name' : "flang-aarch64-release",
'tags' : ["flang"],
'workernames' : ["linaro-flang-aarch64-release"],
'builddir': "flang-aarch64-release",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
checks=['check-flang'],
depends_on_projects=['llvm','mlir','clang','flang'],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_ENABLE_ASSERTIONS=OFF",
"-DCMAKE_CXX_STANDARD=17",
])},
{'name' : "flang-aarch64-rel-assert",
'tags' : ["flang"],
'workernames' : ["linaro-flang-aarch64-rel-assert"],
'builddir': "flang-aarch64-rel-assert",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
checks=['check-flang'],
depends_on_projects=['llvm','mlir','clang','flang'],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_BUILD_EXAMPLES=ON",
"-DCMAKE_BUILD_TYPE=Release",
"-DCMAKE_CXX_STANDARD=17",
])},
{'name' : "flang-aarch64-latest-gcc",
'tags' : ['flang'],
'workernames' : ["linaro-flang-aarch64-latest-gcc"],
'builddir': "flang-aarch64-latest-gcc",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
checks=['check-flang'],
depends_on_projects=['llvm','mlir','clang','flang'],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DLLVM_INSTALL_UTILS=ON",
"-DCMAKE_CXX_STANDARD=17",
"-DLLVM_ENABLE_WERROR=OFF",
"-DFLANG_ENABLE_WERROR=ON",
"-DBUILD_SHARED_LIBS=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DCMAKE_BUILD_TYPE=Release",
])}, ])},
{'name' : "flang-x86_64-knl-linux", {'name' : "flang-x86_64-knl-linux",
'tags' : ["flang"], 'tags' : ["flang"],
'workernames' : ["alcf-theta-flang"], 'workernames' : ["alcf-theta-flang"],
'builddir': "flang-x86_64-knl-linux", 'builddir': "flang-x86_64-knl-linux",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=['llvm','mlir','clang','flang'], depends_on_projects=['llvm','mlir','clang','flang'],
extra_configure_args=[ extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=X86", "-DLLVM_TARGETS_TO_BUILD=X86",
"-DCMAKE_C_COMPILER=gcc", "-DCMAKE_C_COMPILER=gcc",
"-DCMAKE_CXX_COMPILER=g++", "-DCMAKE_CXX_COMPILER=g++",
"-DLLVM_INSTALL_UTILS=ON", "-DLLVM_INSTALL_UTILS=ON",
"-DFLANG_BUILD_NEW_DRIVER=ON",
"-DCMAKE_CXX_STANDARD=17", "-DCMAKE_CXX_STANDARD=17",
])}, ])},
{'name' : 'ppc64le-flang-rhel-clang', {'name' : 'ppc64le-flang-rhel-clang',
'tags' : ["flang", "ppc", "ppc64le"], 'tags' : ["flang", "ppc", "ppc64le"],
'collapseRequests' : False, 'collapseRequests' : False,
'workernames' : ['ppc64le-flang-mlir-rhel-test'], 'workernames' : ['ppc64le-flang-rhel-test'],
'builddir': 'ppc64le-flang-rhel-clang-build', 'builddir': 'ppc64le-flang-rhel-clang-build',
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True, clean=True,
depends_on_projects=['llvm', 'mlir', 'clang', 'flang'], depends_on_projects=['llvm', 'mlir', 'clang', 'flang'],
checks=['check-flang'], checks=['check-flang'],
extra_configure_args=[ extra_configure_args=[
'-DLLVM_TARGETS_TO_BUILD=PowerPC', '-DLLVM_TARGETS_TO_BUILD=PowerPC',
'-DLLVM_INSTALL_UTILS=ON', '-DLLVM_INSTALL_UTILS=ON',
'-DCMAKE_CXX_STANDARD=17', '-DCMAKE_CXX_STANDARD=17',
'-DLLVM_ENABLE_PROJECTS=flang', '-DLLVM_LIT_ARGS=-vj 256',
'-DLLVM_LIT_ARGS=-vj 256' '-DLLVM_CCACHE_BUILD=ON'
], ],
env={ env={
'CC': 'clang', 'CC': 'clang',
'CXX': 'clang++', 'CXX': 'clang++',
'LD': 'lld' 'LD': 'lld'
})}, })},
# Builders responsible building Sphinix documentation. {'name' : "flang-x86_64-windows",
'tags' : ["flang"],
'workernames' : ["minipc-ryzen-win"],
'builddir': "flang-x86_64-windows",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=['llvm','mlir','clang','flang'],
checks=['check-flang'],
install_dir="flang.install",
extra_configure_args=[
"-DCLANG_ENABLE_STATIC_ANALYZER=OFF",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_OBJC_REWRITER=OFF",
"-DLLVM_TARGETS_TO_BUILD=X86",
"-DLLVM_INSTALL_UTILS=ON",
"-DCMAKE_C_COMPILER=cl",
"-DCMAKE_CXX_COMPILER=cl",
"-DCMAKE_CXX_STANDARD=17",
'-DLLVM_PARALLEL_COMPILE_JOBS=4',
])},
# Builders responsible building Sphinx documentation.
{'name' : "llvm-sphinx-docs", {'name' : "llvm-sphinx-docs",
'tags' : ["llvm", "doc"], 'tags' : ["llvm", "doc"],
'workernames' : ["gribozavr3"], 'workernames' : ["gribozavr3"],
'builddir': "llvm-sphinx-docs", 'builddir': "llvm-sphinx-docs",
'factory': SphinxDocsBuilder.getSphinxDocsBuildFactory(llvm_html=True, llvm_man=True)}, 'factory': SphinxDocsBuilder.getSphinxDocsBuildFactory(llvm_html=True, llvm_man=True)},
{'name' : "clang-sphinx-docs", {'name' : "clang-sphinx-docs",
'tags' : ["clang", "doc"], 'tags' : ["clang", "doc"],
'workernames' : ["gribozavr3"], 'workernames' : ["gribozavr3"],
'builddir': "clang-sphinx-docs", 'builddir': "clang-sphinx-docs",
'factory' : SphinxDocsBuilder.getSphinxDocsBuildFactory(clang_html=True)}, 'factory' : SphinxDocsBuilder.getSphinxDocsBuildFactory(clang_html=True, clang_man=True)},
{'name' : "clang-tools-sphinx-docs", {'name' : "clang-tools-sphinx-docs",
'tags' : ["clang-tools", "doc"], 'tags' : ["clang-tools", "doc"],
'workernames':["gribozavr3"], 'workernames':["gribozavr3"],
'builddir':"clang-tools-sphinx-docs", 'builddir':"clang-tools-sphinx-docs",
'factory': SphinxDocsBuilder.getSphinxDocsBuildFactory(clang_tools_html=True)}, 'factory': SphinxDocsBuilder.getSphinxDocsBuildFactory(clang_tools_html=True)},
{'name' : "lld-sphinx-docs", {'name' : "lld-sphinx-docs",
'tags' : ["lld", "doc"], 'tags' : ["lld", "doc"],
'workernames' : ["gribozavr3"], 'workernames' : ["gribozavr3"],
'builddir': "lld-sphinx-docs", 'builddir': "lld-sphinx-docs",
'factory' : SphinxDocsBuilder.getSphinxDocsBuildFactory(lld_html=True)}, 'factory' : SphinxDocsBuilder.getSphinxDocsBuildFactory(lld_html=True)},
{'name' : "lldb-sphinx-docs",
'tags' : ["lldb", "doc"],
'workernames' : ["gribozavr3"],
'builddir': "lldb-sphinx-docs",
'factory' : SphinxDocsBuilder.getSphinxDocsBuildFactory(lldb_html=True)},
{'name':"libunwind-sphinx-docs", {'name':"libunwind-sphinx-docs",
'tags' : ["libunwind", "doc"], 'tags' : ["libunwind", "doc"],
'workernames':["gribozavr3"], 'workernames':["gribozavr3"],
'builddir':"libunwind-sphinx-docs", 'builddir':"libunwind-sphinx-docs",
'factory': SphinxDocsBuilder.getSphinxDocsBuildFactory(libunwind_html=True)}, 'factory': SphinxDocsBuilder.getSphinxRuntimesDocsBuildFactory(libunwind_html=True)},
# Sphinx doc Publisher # Sphinx doc Publisher
{'name' : "publish-sphinx-docs", {'name' : "publish-sphinx-docs",
'tags' : ["doc"], 'tags' : ["doc"],
'workernames' : ["as-worker-4"], 'workernames' : ["as-worker-4"],
'builddir': "publish-sphinx-docs", 'builddir': "publish-sphinx-docs",
'factory' : SphinxDocsBuilder.getLLVMDocsBuildFactory(clean=True)}, 'factory' : SphinxDocsBuilder.getLLVMDocsBuildFactory(clean=True)},
{'name' : "publish-runtimes-sphinx-docs",
'tags' : ["doc"],
'workernames' : ["as-worker-4"],
'builddir': "publish-runtimes-sphinx-docs",
'factory' : SphinxDocsBuilder.getLLVMRuntimesDocsBuildFactory(clean=True)},
{'name' : "publish-lnt-sphinx-docs",
'tags' : ["doc"],
'workernames' : ["as-worker-4"],
'builddir': "publish-lnt-sphinx-docs",
'factory' : HtmlDocsBuilder.getHtmlDocsBuildFactory()},
{'name' : "publish-doxygen-docs",
'tags' : ["doc"],
'workernames' : ["as-worker-4"], #FIXME: Temporarily disabled failing doxygen build - as-builder-8.
'builddir': "publish-doxygen-docs",
'factory' : DoxygenDocsBuilder.getLLVMDocsBuildFactory(
# Doxygen builds the final result for really
# long time without any output.
# We have to have a long timeout here.
timeout=172800)},
{'name' : "polly-sphinx-docs",
'tags' : ["llvm", "doc"],
'workernames' : ["polly-x86_64-gce1"],
'builddir': "polly-sphinx-docs",
'factory': SphinxDocsBuilder.getSphinxDocsBuildFactory(polly_html=True)},
# CUDA builders. # CUDA builders.
{'name' : "clang-cuda-k80", {'name' : "clang-cuda-k80",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["cuda-k80-0"], 'workernames' : ["cuda-k80-0"],
'builddir': "clang-cuda-k80", 'builddir': "clang-cuda-k80",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory( 'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="/buildbot/cuda-build", script="/buildbot/cuda-build",
Show All 9 Lines# CUDA builders.
{'name' : "clang-cuda-t4", {'name' : "clang-cuda-t4",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["cuda-t4-0"], 'workernames' : ["cuda-t4-0"],
'builddir': "clang-cuda-t4", 'builddir': "clang-cuda-t4",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory( 'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="/buildbot/cuda-build", script="/buildbot/cuda-build",
checkout_llvm_sources=False)}, checkout_llvm_sources=False)},
# HIP builders.
{'name' : "clang-hip-vega20",
'tags' : ["clang"],
'workernames' : ["hip-vega20-0"],
'builddir': "clang-hip-vega20",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="hip-build.sh",
checkout_llvm_sources=False,
script_interpreter=None)},
# VE builders.
{'name' : "clang-ve-ninja", {'name' : "clang-ve-ninja",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames':["nec-arrproto41"], 'workernames':["hpce-ve-main"],
'builddir':"clang-ve-ninja", 'builddir':"clang-ve-ninja",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
clean=True, script="ve-linux.py",
depends_on_projects=['llvm','clang','openmp'], depends_on_projects=['llvm', 'clang', 'compiler-rt', 'libcxx'])},
extra_configure_args=[ {'name' : "clang-ve-staging",
"-DLLVM_TARGETS_TO_BUILD=X86", 'tags' : ["clang"],
"-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=VE", 'workernames':["hpce-ve-staging"],
])}, 'builddir':"clang-ve-staging",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
# Latest stable fedora running on Red Hat internal OpenShift cluster (PSI). script="ve-linux.py",
depends_on_projects=['llvm', 'clang', 'compiler-rt', 'libcxx'])},
{'name' : 'x86_64-fedora-clang', # Build the LLVM dylib .so with all backends and link tools to it
'tags' : ['mlir'], {'name' : 'llvm-x86_64-debian-dylib',
'tags' : ['llvm'],
'collapseRequests': False, 'collapseRequests': False,
'workernames': ['fedora-llvm-x86_64'], 'workernames': ['gribozavr4'],
'builddir': 'x86_64-fedora-clang', 'builddir': 'llvm-x86_64-debian-dylib',
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True, clean=True,
depends_on_projects=['llvm', 'clang', 'clang-tools-extra', 'compiler-rt', 'lld', 'mlir'], depends_on_projects=['llvm'],
checks=['check-all'], checks=['check-all'],
extra_configure_args=[ extra_configure_args=[
'-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_BUILD_TYPE=Release',
'-DCMAKE_C_COMPILER=/usr/bin/gcc',
'-DCMAKE_CXX_COMPILER=/usr/bin/g++',
'-DLLVM_ENABLE_ASSERTIONS=On', '-DLLVM_ENABLE_ASSERTIONS=On',
'-DLLVM_BUILD_EXAMPLES=On', '-DLLVM_BUILD_EXAMPLES=Off',
"-DLLVM_LIT_ARGS=-v --xunit-xml-output test-results.xml", "-DLLVM_LIT_ARGS=-v --xunit-xml-output test-results.xml",
'-DLLVM_CCACHE_BUILD=On', '-DLLVM_TARGETS_TO_BUILD=all',
'-DLLVM_CCACHE_DIR=/ccache',
'-DLLVM_CCACHE_MAXSIZE=20G',
'-DLLVM_TARGETS_TO_BUILD=X86',
'-DCMAKE_EXPORT_COMPILE_COMMANDS=1', '-DCMAKE_EXPORT_COMPILE_COMMANDS=1',
'-DLLVM_BUILD_LLVM_DYLIB=On', '-DLLVM_BUILD_LLVM_DYLIB=On',
'-DLLVM_LINK_LLVM_DYLIB=On', '-DLLVM_LINK_LLVM_DYLIB=On',
'-DCLANG_LINK_CLANG_DYLIB=On',
'-DBUILD_SHARED_LIBS=Off', '-DBUILD_SHARED_LIBS=Off',
'-DLLVM_ENABLE_LLD=ON', '-DLLVM_ENABLE_LLD=Off',
])}, '-DLLVM_ENABLE_BINDINGS=Off',
'-DLLVM_CCACHE_BUILD=ON',
],
env={
'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
'CC': 'clang', 'CXX': 'clang++',
})},
{'name' : "clang-solaris11-amd64", {'name' : "clang-solaris11-amd64",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["solaris11-amd64"], 'workernames' : ["solaris11-amd64"],
'builddir': "clang-solaris11-amd64", 'builddir': "clang-solaris11-amd64",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
jobs=8, jobs=8,
clean=False, clean=False,
timeout=1800,
checkout_lld=False, checkout_lld=False,
enable_runtimes=None,
extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON', extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON',
'-DLLVM_TARGETS_TO_BUILD=X86', '-DLLVM_TARGETS_TO_BUILD=X86',
'-DLLVM_HOST_TRIPLE=amd64-pc-solaris2.11', '-DLLVM_HOST_TRIPLE=amd64-pc-solaris2.11',
'-DLLVM_PARALLEL_LINK_JOBS=4'])}, '-DLLVM_PARALLEL_LINK_JOBS=4'])},
{'name' : "clang-solaris11-sparcv9", {'name' : "clang-solaris11-sparcv9",
'tags' : ["clang"], 'tags' : ["clang"],
'workernames' : ["solaris11-sparcv9"], 'workernames' : ["solaris11-sparcv9"],
'builddir': "clang-solaris11-sparcv9", 'builddir': "clang-solaris11-sparcv9",
'factory' : ClangBuilder.getClangCMakeBuildFactory( 'factory' : ClangBuilder.getClangCMakeBuildFactory(
jobs=8, jobs=8,
clean=False, clean=False,
timeout=1800,
checkout_lld=False, checkout_lld=False,
enable_runtimes=None,
extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON', extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON',
'-DLLVM_TARGETS_TO_BUILD=Sparc', '-DLLVM_TARGETS_TO_BUILD=Sparc',
'-DLLVM_HOST_TRIPLE=sparcv9-sun-solaris2.11', '-DLLVM_HOST_TRIPLE=sparcv9-sun-solaris2.11',
'-DLLVM_PARALLEL_LINK_JOBS=4'])}, '-DLLVM_PARALLEL_LINK_JOBS=4'])},
{'name' : "clang-x86-ninja-win10",
'tags' : ["clang"],
'workernames' : ["windows10-vs2019"],
'builddir': "clang-x86-ninja-win10",
'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=True,
vs="autodetect",
vs_target_arch='x86',
testStage1=True,
useTwoStage=True,
stage1_config='Release',
stage2_config='Release',
# reduce scope of builds to get stable results
checkout_clang_tools_extra=False,
checkout_compiler_rt=False,
checkout_flang=False,
checkout_libcxx=False,
extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON',
'-DLLVM_TARGETS_TO_BUILD=X86',
'-DCMAKE_C_COMPILER_LAUNCHER=sccache',
'-DCMAKE_CXX_COMPILER_LAUNCHER=sccache',
'-DLLVM_ENABLE_ZLIB=OFF',
'-DLLVM_LIT_TOOLS_DIR=C:\\Program Files\\GnuWin32\\usr\\bin',
])},
# Flang builders.
{'name' : "flang-aarch64-ubuntu-clang",
'tags' : ['flang'],
'workernames' : ["flang-aarch64-ubuntu-clang-build"],
'builddir': "flang-aarch64-ubuntu-clang",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
depends_on_projects=['llvm','mlir','clang','flang'],
checks=['check-flang'],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DCMAKE_C_COMPILER=/usr/bin/clang-10",
"-DCMAKE_CXX_COMPILER=/usr/bin/clang++-10",
"-DLLVM_INSTALL_UTILS=ON",
"-DCMAKE_CXX_STANDARD=17",
"-DLLVM_ENABLE_WERROR=OFF",
"-DFLANG_ENABLE_WERROR=ON",
"-DBUILD_SHARED_LIBS=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_ENABLE_LIBCXX=On",
"-DFLANG_BUILD_NEW_DRIVER=ON",
"-DCMAKE_BUILD_TYPE=Release",
])},
{'name' : "flang-aarch64-ubuntu-gcc10",
'tags' : ['flang'],
'workernames' : ["flang-aarch64-ubuntu-gcc10-build"],
'builddir': "flang-aarch64-ubuntu-gcc10",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
depends_on_projects=['llvm','mlir','clang','flang'],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DCMAKE_C_COMPILER=/usr/bin/gcc-10",
"-DCMAKE_CXX_COMPILER=/usr/bin/g++-10",
"-DLLVM_INSTALL_UTILS=ON",
"-DCMAKE_CXX_STANDARD=17",
"-DLLVM_ENABLE_WERROR=OFF",
"-DFLANG_ENABLE_WERROR=ON",
"-DBUILD_SHARED_LIBS=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DFLANG_BUILD_NEW_DRIVER=ON",
"-DCMAKE_BUILD_TYPE=Release",
])},
# LLDB builders.
{'name' : "lldb-x86_64-fedora",
'tags' : ["lldb"],
'workernames' : ["lldb-x86_64-fedora"],
'builddir': "lldb-x86_64-fedora",
'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
clean=True,
test=True,
extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=True',
'-DLLVM_USE_LINKER=gold',
'-DLLVM_LIT_ARGS=-v'])},
# Builders for ML-driven compiler optimizations. # Builders for ML-driven compiler optimizations.
# Development mode build bot: tensorflow C APIs are present, and # Development mode build bot: tensorflow C APIs are present, and
# we can dynamically load models, and produce training logs. # we can dynamically load models, and produce training logs.
{'name' : "ml-opt-dev-x86-64", {'name' : "ml-opt-dev-x86-64",
'tags' : ['ml_opt'], 'tags' : ['ml_opt'],
'collapseRequests': False, 'collapseRequests': False,
'workernames' : ["ml-opt-dev-x86-64-b1"], 'workernames' : ["ml-opt-dev-x86-64-b1", "ml-opt-dev-x86-64-b2"],
'builddir': "ml-opt-dev-x86-64-b1", 'builddir': "ml-opt-dev-x86-64-b1",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True, clean=True,
depends_on_projects=['llvm'], depends_on_projects=['llvm'],
extra_configure_args=[ extra_configure_args=[
"-DCMAKE_BUILD_TYPE=Release", "-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_CCACHE_BUILD=ON", "-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON", "-DLLVM_ENABLE_ASSERTIONS=ON",
"-DTENSORFLOW_API_PATH=/tmp/tensorflow", "-DTENSORFLOW_C_LIB_PATH=/tmp/tensorflow",
"-C", "/tmp/tflitebuild/tflite.cmake",
"-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON" "-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON"
])}, ])},
# Both tensorflow C library, and the pip package, are present. # Both tensorflow C library, and the pip package, are present.
{'name' : "ml-opt-devrel-x86-64", {'name' : "ml-opt-devrel-x86-64",
'tags' : ["ml_opt"], 'tags' : ["ml_opt"],
'collapseRequests': False, 'collapseRequests': False,
'workernames' : ["ml-opt-devrel-x86-64-b1"], 'workernames' : ["ml-opt-devrel-x86-64-b1", "ml-opt-devrel-x86-64-b2"],
'builddir': "ml-opt-devrel-x86-64-b1", 'builddir': "ml-opt-devrel-x86-64-b1",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True, clean=True,
depends_on_projects=['llvm'], depends_on_projects=['llvm'],
extra_configure_args= [ extra_configure_args= [
"-DCMAKE_BUILD_TYPE=Release", "-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_CCACHE_BUILD=ON", "-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON", "-DLLVM_ENABLE_ASSERTIONS=ON",
"-DTENSORFLOW_API_PATH=/tmp/tensorflow", "-DTENSORFLOW_C_LIB_PATH=/tmp/tensorflow",
"-C", "/tmp/tflitebuild/tflite.cmake",
"-DTENSORFLOW_AOT_PATH=/var/lib/buildbot/.local/lib/python3.7/site-packages/tensorflow", "-DTENSORFLOW_AOT_PATH=/var/lib/buildbot/.local/lib/python3.7/site-packages/tensorflow",
"-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON" "-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON"
])}, ])},
# Release mode build bot: the model is pre-built and linked in the # Release mode build bot: the model is pre-built and linked in the
# compiler. Only the tensorflow pip package is needed, and out of it, # compiler. Only the tensorflow pip package is needed, and out of it,
# only saved_model_cli (the model compiler) and the thin C++ wrappers # only saved_model_cli (the model compiler) and the thin C++ wrappers
# in xla_aot_runtime_src (and include files) # in xla_aot_runtime_src (and include files)
{'name' : "ml-opt-rel-x86-64", {'name' : "ml-opt-rel-x86-64",
'tags' : ["ml_opt"], 'tags' : ["ml_opt"],
'collapseRequests': False, 'collapseRequests': False,
'workernames' : ["ml-opt-rel-x86-64-b1"], 'workernames' : ["ml-opt-rel-x86-64-b1", "ml-opt-rel-x86-64-b2"],
'builddir': "ml-opt-rel-x86-64-b1", 'builddir': "ml-opt-rel-x86-64-b1",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True, clean=True,
depends_on_projects=['llvm'], depends_on_projects=['llvm'],
extra_configure_args= [ extra_configure_args= [
"-DCMAKE_BUILD_TYPE=Release", "-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_CCACHE_BUILD=ON", "-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON", "-DLLVM_ENABLE_ASSERTIONS=ON",
"-DTENSORFLOW_AOT_PATH=/var/lib/buildbot/.local/lib/python3.7/site-packages/tensorflow" "-DTENSORFLOW_AOT_PATH=/var/lib/buildbot/.local/lib/python3.7/site-packages/tensorflow"
])}, ])},
# build clangd with remote-index enabled and check with TSan # build clangd with remote-index enabled and check with TSan
{'name': "clangd-ubuntu-tsan", {'name': "clangd-ubuntu-tsan",
'tags': ["clangd"], 'tags': ["clangd"],
'workernames': ["clangd-ubuntu-clang"], 'workernames': ["clangd-ubuntu-clang"],
'builddir': "clangd-ubuntu-tsan", 'builddir': "clangd-ubuntu-tsan",
'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory( 'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True, clean=True,
depends_on_projects=["llvm", "clang", "clang-tools-extra"], depends_on_projects=["llvm", "clang", "clang-tools-extra"],
checks=["check-clangd"], checks=["check-clangd"],
targets=["clangd", "clangd-index-server", "clangd-indexer"], targets=["clangd", "clangd-index-server", "clangd-indexer"],
extra_configure_args=[ extra_configure_args=[
"-DLLVM_CCACHE_BUILD=ON", "-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_USE_SANITIZER=Thread", "-DLLVM_USE_SANITIZER=Thread",
"-DCMAKE_BUILD_TYPE=Release", "-DCMAKE_BUILD_TYPE=Release",
"-DCLANGD_ENABLE_REMOTE=ON", "-DCLANGD_ENABLE_REMOTE=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON", "-DLLVM_ENABLE_ASSERTIONS=ON",
"-DGRPC_INSTALL_PATH=/usr/local/lib/grpc" "-DGRPC_INSTALL_PATH=/usr/local/lib/grpc",
"-DLLVM_OPTIMIZED_TABLEGEN=ON"
])},
# Build in C++20 configuration.
{'name': "clang-debian-cpp20",
'tags': ["clang", "c++20"],
'workernames': ["clang-debian-cpp20"],
'builddir': "clang-debian-cpp20",
'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
depends_on_projects=["llvm", "clang", "clang-tools-extra"],
extra_configure_args=[
"-DCMAKE_CXX_STANDARD=20",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_ENABLE_ASSERTIONS=ON",
# FIXME: Re-enable after cleaning up LLVM.
# https://github.com/llvm/llvm-project/issues/60101
"-DCMAKE_CXX_FLAGS=-Wno-deprecated-enum-enum-conversion -Wno-deprecated-declarations -Wno-deprecated-anon-enum-enum-conversion -Wno-ambiguous-reversed-operator",
])},
# Target ARC from Synopsys
{'name': "arc-builder",
'tags': ["clang", "lld"],
'workernames' : ["arc-worker"],
'builddir': "arc-folder",
'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=["llvm", "clang", "lld"],
extra_configure_args=[
"-DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON",
"-DLLVM_ENABLE_ASSERTIONS:BOOL=ON",
"-DLLVM_TOOL_CLANG_TOOLS_EXTRA_BUILD=0",
"-DLLVM_ENABLE_LIBPFM=OFF",
"-DLLVM_TARGETS_TO_BUILD=X86",
"-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=ARC",
])}, ])},
# BOLT builders managed by Meta
{'name' : 'bolt-x86_64-ubuntu-nfc',
'tags' : ["bolt"],
'collapseRequests': False,
'workernames' : ['bolt-worker'],
'builddir': "bolt-x86_64-ubuntu-nfc",
'factory' : BOLTBuilder.getBOLTCmakeBuildFactory(
bolttests=True,
depends_on_projects=['bolt', 'llvm'],
extra_configure_args=[
"-DLLVM_APPEND_VC_REV=OFF",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_PROJECTS=clang;lld;bolt",
"-DLLVM_TARGETS_TO_BUILD=X86;AArch64",
],
is_nfc=True,
)},
{'name': "bolt-x86_64-ubuntu-clang",
'tags': ["bolt"],
'workernames':["bolt-worker"],
'builddir': "bolt-x86_64-ubuntu-clang",
'factory' : BOLTBuilder.getBOLTCmakeBuildFactory(
bolttests=False,
clean=True,
depends_on_projects=['bolt', 'clang', 'lld', 'llvm'],
caches=[
'clang/cmake/caches/BOLT.cmake',
'clang/cmake/caches/BOLT-PGO.cmake',
],
targets=['clang-bolt'],
checks=['stage2-clang-bolt'],
extra_configure_args=[
"-DCMAKE_C_COMPILER=gcc",
"-DCMAKE_CXX_COMPILER=g++",
"-DLLVM_APPEND_VC_REV=OFF",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_LLD=ON",
"-DBOOTSTRAP_LLVM_ENABLE_LLD=ON",
"-DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_LLD=ON",
"-DPGO_INSTRUMENT_LTO=Thin",
],
)},
{'name': "bolt-x86_64-ubuntu-dylib",
'tags': ["bolt"],
'workernames':["bolt-worker"],
'builddir': "bolt-x86_64-ubuntu-dylib",
'factory' : BOLTBuilder.getBOLTCmakeBuildFactory(
bolttests=False,
depends_on_projects=['bolt', 'llvm'],
extra_configure_args=[
"-DLLVM_APPEND_VC_REV=OFF",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_PROJECTS=bolt",
"-DLLVM_TARGETS_TO_BUILD=X86;AArch64",
"-DLLVM_LINK_LLVM_DYLIB=ON",
"-DLLVM_ENABLE_LLD=ON",
"-DBOLT_CLANG_EXE=/usr/bin/clang",
"-DBOLT_LLD_EXE=/usr/bin/ld.lld",
],
)},
{'name': "bolt-x86_64-ubuntu-shared",
'tags': ["bolt"],
'workernames':["bolt-worker"],
'builddir': "bolt-x86_64-ubuntu-shared",
'factory' : BOLTBuilder.getBOLTCmakeBuildFactory(
bolttests=False,
depends_on_projects=['bolt', 'llvm'],
extra_configure_args=[
"-DLLVM_APPEND_VC_REV=OFF",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_PROJECTS=bolt",
"-DLLVM_TARGETS_TO_BUILD=X86;AArch64",
"-DBUILD_SHARED_LIBS=ON",
"-DLLVM_ENABLE_LLD=ON",
"-DBOLT_CLANG_EXE=/usr/bin/clang",
"-DBOLT_LLD_EXE=/usr/bin/ld.lld",
],
)},
{'name': "bolt-aarch64-ubuntu-clang-shared",
'tags': ["bolt"],
'workernames':["bolt-worker-aarch64"],
'builddir': "bolt-aarch64-ubuntu-clang-shared",
'factory' : BOLTBuilder.getBOLTCmakeBuildFactory(
bolttests=True,
depends_on_projects=['bolt', 'llvm'],
extra_configure_args=[
"-DCMAKE_C_COMPILER=clang",
"-DCMAKE_CXX_COMPILER=clang++",
"-DLLVM_APPEND_VC_REV=OFF",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_ENABLE_PROJECTS=bolt",
"-DLLVM_TARGETS_TO_BUILD=X86;AArch64",
"-DBUILD_SHARED_LIBS=ON",
"-DLLVM_USE_LINKER=mold",
"-DBOLT_CLANG_EXE=/usr/bin/clang",
"-DBOLT_LLD_EXE=/usr/bin/ld.lld",
],
)},
# AMD ROCm support.
{'name' : 'mlir-rocm-mi200',
'tags' : ["mlir"],
'workernames' : ['mi200-buildbot'],
'builddir': 'mlir-rocm-mi200',
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
depends_on_projects=['llvm', 'mlir'],
targets = ['check-mlir-build-only'],
checks = ['check-mlir'],
extra_configure_args= mlir_default_cmake_options + [
'-DLLVM_CCACHE_BUILD=ON',
'-DLLVM_ENABLE_ASSERTIONS=ON',
'-DLLVM_ENABLE_LLD=ON',
'-DMLIR_ENABLE_ROCM_RUNNER=ON',
'-DMLIR_ENABLE_ROCM_CONVERSIONS=ON',
'-DMLIR_INCLUDE_INTEGRATION_TESTS=ON',
],
env={
'CC': 'clang',
'CXX': 'clang++',
'LD': 'lld',
})},
# Standalone builder
{'name' : "standalone-build-x86_64",
'tags' : ["clang"],
'workernames':["standalone-build-x86_64"],
'builddir':"standalone-build-x86_64",
'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
script="standalone-build.sh",
checkout_llvm_sources=False,
script_interpreter=None)},
## CSKY check-all + test-suite in soft-float
{'name' : "clang-csky-soft",
'tags' : ["clang"],
'workernames' : ["thead-clang-csky"],
'builddir':"clang-csky-softfp",
'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False,
checkout_clang_tools_extra=False,
checkout_compiler_rt=False,
checkout_lld=False,
testStage1=True,
useTwoStage=False,
stage1_config='Release',
runTestSuite=True,
testsuite_flags=[
'--cflags', '-mcpu=c860 -latomic -DSMALL_PROBLEM_SIZE',
'--cppflags', '-mcpu=c860 -latomic -DSMALL_PROBLEM_SIZE',
'--run-under=/mnt/qemu/bin/qemu-cskyv2 -cpu c860 -csky-extend denormal=on -L /mnt/gcc-csky/csky-linux-gnuabiv2/libc/ck860 -E LD_LIBRARY_PATH=/mnt/gcc-csky/csky-linux-gnuabiv2/lib/ck860',
'--cmake-define=SMALL_PROBLEM_SIZE=On',
'--cmake-define=TEST_SUITE_USER_MODE_EMULATION=True',
'--threads=32', '--build-threads=32'],
extra_cmake_args=[
"-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD='CSKY'",
"-DLLVM_DEFAULT_TARGET_TRIPLE='csky-unknown-linux'",
"-DGCC_INSTALL_PREFIX=/mnt/gcc-csky/"])},
## CSKY check-all + test-suite in hard-float
{'name' : "clang-csky-hardfp",
'tags' : ["clang"],
'workernames' : ["thead-clang-csky"],
'builddir':"clang-csky-hardfp",
'factory' : ClangBuilder.getClangCMakeBuildFactory(
clean=False,
checkout_clang_tools_extra=False,
checkout_compiler_rt=False,
checkout_lld=False,
testStage1=True,
useTwoStage=False,
stage1_config='Release',
runTestSuite=True,
testsuite_flags=[
'--cflags', '-mcpu=c860 -latomic -mhard-float -DSMALL_PROBLEM_SIZE',
'--cppflags', '-mcpu=c860 -latomic -mhard-float -DSMALL_PROBLEM_SIZE',
'--run-under=/mnt/qemu/bin/qemu-cskyv2 -cpu c860 -csky-extend denormal=on -L /mnt/gcc-csky/csky-linux-gnuabiv2/libc/ck860/hard-fp -E LD_LIBRARY_PATH=/mnt/gcc-csky/csky-linux-gnuabiv2/lib/ck860/hard-fp',
'--cmake-define=SMALL_PROBLEM_SIZE=On',
'--cmake-define=TEST_SUITE_USER_MODE_EMULATION=True',
'--threads=32', '--build-threads=32'],
extra_cmake_args=[
"-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD='CSKY'",
"-DLLVM_DEFAULT_TARGET_TRIPLE='csky-unknown-linux'",
"-DGCC_INSTALL_PREFIX=/mnt/gcc-csky/"])},
# NVPTX builders
{'name' : "llvm-nvptx-nvidia-ubuntu",
'tags' : ["llvm", "nvptx"],
'collapseRequests': False,
'workernames' : ["as-builder-7"],
'builddir': "llvm-nvptx-nvidia-ubuntu",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=["llvm"],
clean=True,
checks=["check-llvm"],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=X86;NVPTX",
"-DLLVM_DEFAULT_TARGET_TRIPLE=nvptx-nvidia-cuda",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_LIT_ARGS=-vv --threads=32",
"-DLLVM_USE_LINKER=gold",
"-DBUILD_SHARED_LIBS=ON",
"-DLLVM_OPTIMIZED_TABLEGEN=ON"])},
{'name' : "llvm-nvptx64-nvidia-ubuntu",
'tags' : ["llvm", "nvptx"],
'collapseRequests': False,
'workernames' : ["as-builder-7"],
'builddir': "llvm-nvptx64-nvidia-ubuntu",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=["llvm"],
clean=True,
checks=["check-llvm"],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=X86;NVPTX",
"-DLLVM_DEFAULT_TARGET_TRIPLE=nvptx64-nvidia-cuda",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_LIT_ARGS=-vv --threads=32",
"-DLLVM_USE_LINKER=gold",
"-DBUILD_SHARED_LIBS=ON",
"-DLLVM_OPTIMIZED_TABLEGEN=ON"])},
{'name' : "llvm-nvptx-nvidia-win",
'tags' : ["llvm", "nvptx"],
'workernames' : ["as-builder-8"],
'builddir': "llvm-nvptx-nvidia-win",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
vs="autodetect",
depends_on_projects=["llvm"],
clean=True,
checks=["check-llvm"],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=X86;NVPTX",
"-DLLVM_DEFAULT_TARGET_TRIPLE=nvptx-nvidia-cuda",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_LIT_ARGS=-vv --threads=32",
"-DLLVM_OPTIMIZED_TABLEGEN=ON"])},
{'name' : "llvm-nvptx64-nvidia-win",
'tags' : ["llvm", "nvptx"],
'workernames' : ["as-builder-8"],
'builddir': "llvm-nvptx64-nvidia-win",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
vs="autodetect",
depends_on_projects=["llvm"],
clean=True,
checks=["check-llvm"],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=X86;NVPTX",
"-DLLVM_DEFAULT_TARGET_TRIPLE=nvptx64-nvidia-cuda",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_LIT_ARGS=-vv --threads=32",
"-DLLVM_OPTIMIZED_TABLEGEN=ON"])},
## RISC-V RV64GC check-all running under qemu-user.
{'name' : "clang-rv64gc-qemu-user-single-stage",
'tags' : ["llvm", "clang"],
'workernames' : ["rv64gc-qemu-user"],
'builddir': "clang-rv64gc",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=["llvm", "clang", "clang-tools-extra", "lld"],
checks=['check-all'],
extra_configure_args=[
"-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
"-DLLVM_TARGETS_TO_BUILD=all"],
env={
'CC':'clang',
'CXX': 'clang++',
})},
# GoogleCloud Buildguards (investigation/stat)
{'name' : "llvm-clang-buildguard5-ubuntu",
'tags' : ["llvm", "clang", "lld", "lldb", "clang-tools-extra"],
'collapseRequests' : False,
'workernames' : ["gc-builder-5"],
'builddir': "buildguard5",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=["llvm", "clang", "clang-tools-extra", "lld", "lldb", "compiler-rt", "libunwind", "libcxxabi", "libcxx"],
enable_runtimes="auto", # get runtimes from depends_on_projects.
checks=['check-all'],
clean=True,
extra_configure_args=[
"-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_ENABLE_WERROR=OFF",
"-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_USE_SPLIT_DWARF=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON",
"-DCLANG_DEFAULT_LINKER=lld",
"-DLLVM_USE_LINKER=gold",
"-DLLVM_OPTIMIZED_TABLEGEN=ON",
"-DCMAKE_CXX_FLAGS=-D__OPTIMIZE__ -Wno-misleading-indentation",
"-DBUILD_SHARED_LIBS=ON",
"-DLLVM_LIT_ARGS=-v -vv"],
env={
'CCACHE_DIR' : WithProperties("%(builddir)s/ccache-db"),
})},
{'name' : "llvm-clang-buildguard6-win",
'tags' : ["llvm", "clang", "lld", "lldb", "clang-tools-extra"],
'collapseRequests' : False,
'workernames' : ["gc-builder-6-win", "gc-builder-7-win"],
'builddir': "buildguard6",
'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory(
vs="autodetect",
checks=['check-all'],
clean=True,
extra_configure_args=[
"-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_ENABLE_WERROR=OFF",
"-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_USE_SPLIT_DWARF=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON",
"-DCLANG_DEFAULT_LINKER=lld",
"-DLLVM_OPTIMIZED_TABLEGEN=ON",
"-DCMAKE_CXX_FLAGS=-D__OPTIMIZE__",
"-DLLVM_LIT_ARGS=-v -vv"],
env={
'CCACHE_DIR' : WithProperties("%(builddir)s/ccache-db"),
})},
] ]

buildbot/osuosl/master/config/release_builders.py

  • This file was added.
from importlib import reload
from buildbot.process.properties import WithProperties
from zorg.buildbot.builders import ClangBuilder
from zorg.buildbot.builders import FlangBuilder
from zorg.buildbot.builders import PollyBuilder
from zorg.buildbot.builders import LLDBBuilder
from zorg.buildbot.builders import SanitizerBuilder
from zorg.buildbot.builders import OpenMPBuilder
from zorg.buildbot.builders import SphinxDocsBuilder
from zorg.buildbot.builders import ABITestsuitBuilder
from zorg.buildbot.builders import ClangLTOBuilder
from zorg.buildbot.builders import UnifiedTreeBuilder
from zorg.buildbot.builders import AOSPBuilder
from zorg.buildbot.builders import AnnotatedBuilder
from zorg.buildbot.builders import LLDPerformanceTestsuite
from zorg.buildbot.builders import FuchsiaBuilder
from zorg.buildbot.builders import XToolchainBuilder
# Release builders.
all = [
# Clang builders.
{'name' : "llvm-clang-x86_64-win-release",
'tags' : ["clang"],
'workernames' : ["as-builder-3"],
'builddir': "llvm-clang-x86_64-win-rel",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
vs="autodetect",
depends_on_projects=['llvm', 'clang'],
clean=True,
checks=[
"check-llvm-unit",
"check-clang-unit"],
extra_configure_args=[
"-DLLVM_ENABLE_WERROR=OFF",
"-DLLVM_TARGETS_TO_BUILD=ARM",
"-DLLVM_DEFAULT_TARGET_TRIPLE=armv7-unknown-linux-eabihf",
"-DLLVM_ENABLE_ASSERTIONS=OFF",
"-DLLVM_OPTIMIZED_TABLEGEN=OFF",
"-DLLVM_LIT_ARGS=-v --threads=32"])},
# Expensive checks builders.
{'name' : "llvm-clang-x86_64-expensive-checks-ubuntu-release",
'tags' : ["llvm", "expensive-checks"],
'workernames' : ["as-builder-4"],
'builddir': "llvm-clang-x86_64-expensive-checks-ubuntu-rel",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=["llvm", "lld"],
clean=True,
extra_configure_args=[
"-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
"-DLLVM_ENABLE_WERROR=OFF",
"-DLLVM_USE_SPLIT_DWARF=ON",
"-DLLVM_USE_LINKER=gold",
"-DCMAKE_BUILD_TYPE=Debug",
"-DCMAKE_CXX_FLAGS=-U_GLIBCXX_DEBUG -Wno-misleading-indentation",
"-DLLVM_LIT_ARGS=-vv -j32"])},
{'name' : "llvm-clang-x86_64-expensive-checks-win-release",
'tags' : ["llvm", "expensive-checks"],
'workernames' : ["as-worker-93"],
'builddir': "llvm-clang-x86_64-expensive-checks-win-rel",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
vs="autodetect",
depends_on_projects=["llvm", "lld"],
clean=True,
extra_configure_args=[
"-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
"-DLLVM_ENABLE_WERROR=OFF",
"-DCMAKE_BUILD_TYPE=Debug"])},
{'name' : "llvm-clang-x86_64-expensive-checks-debian-release",
'tags' : ["llvm", "expensive-checks"],
'workernames' : ["gribozavr4"],
'builddir': "llvm-clang-x86_64-expensive-checks-debian-rel",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=["llvm", "lld"],
clean=True,
extra_configure_args=[
"-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON",
"-DLLVM_ENABLE_WERROR=OFF",
"-DCMAKE_BUILD_TYPE=Release",
"-DCMAKE_CXX_FLAGS=-U_GLIBCXX_DEBUG",
"-DLLVM_LIT_ARGS=-v -vv -j96"],
env={
'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
'CC': 'clang', 'CXX': 'clang++',
})},
# Cross builders.
{'name' : "llvm-clang-win-x-armv7l-release",
'tags' : ["clang", "llvm", "compiler-rt", "cross", "armv7"],
'workernames' : ["as-builder-5"],
'builddir': "x-armv7l-rel",
'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory(
vs="autodetect",
clean=True,
checks=[
"check-llvm",
"check-clang",
"check-lld",
"check-compiler-rt-armv7-unknown-linux-gnueabihf"
],
checks_on_target = [
("libunwind",
["python", "bin/llvm-lit.py",
"-v", "-vv", "--threads=32",
"runtimes/runtimes-armv7-unknown-linux-gnueabihf-bins/libunwind/test"]),
("libc++abi",
["python", "bin/llvm-lit.py",
"-v", "-vv", "--threads=32",
"runtimes/runtimes-armv7-unknown-linux-gnueabihf-bins/libcxxabi/test"]),
("libc++",
['python', 'bin/llvm-lit.py',
'-v', '-vv', '--threads=32',
'runtimes/runtimes-armv7-unknown-linux-gnueabihf-bins/libcxx/test',
])
],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=ARM",
"-DTOOLCHAIN_TARGET_TRIPLE=armv7-unknown-linux-gnueabihf",
"-DDEFAULT_SYSROOT=C:/buildbot/.arm-ubuntu",
"-DZLIB_ROOT=C:/buildbot/.zlib-win32",
"-DLLVM_LIT_ARGS=-v -vv --threads=32",
WithProperties("%(remote_test_host:+-DREMOTE_TEST_HOST=)s%(remote_test_host:-)s"),
WithProperties("%(remote_test_user:+-DREMOTE_TEST_USER=)s%(remote_test_user:-)s"),
],
cmake_cache="../llvm-project/clang/cmake/caches/CrossWinToARMLinux.cmake")},
{'name' : "llvm-clang-win-x-aarch64-release",
'tags' : ["clang", "llvm", "compiler-rt", "cross", "aarch64"],
'workernames' : ["as-builder-6"],
'builddir': "x-aarch64-rel",
'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory(
vs="autodetect",
clean=True,
checks=[
"check-llvm",
"check-clang",
"check-lld",
"check-compiler-rt-aarch64-unknown-linux-gnu"
],
checks_on_target = [
("libunwind",
["python", "bin/llvm-lit.py",
"-v", "-vv", "--threads=32",
"runtimes/runtimes-aarch64-unknown-linux-gnu-bins/libunwind/test"]),
("libc++abi",
["python", "bin/llvm-lit.py",
"-v", "-vv", "--threads=32",
"runtimes/runtimes-aarch64-unknown-linux-gnu-bins/libcxxabi/test"]),
("libc++",
['python', 'bin/llvm-lit.py',
'-v', '-vv', '--threads=32',
'runtimes/runtimes-aarch64-unknown-linux-gnu-bins/libcxx/test',
])
],
extra_configure_args=[
"-DLLVM_TARGETS_TO_BUILD=AArch64",
"-DTOOLCHAIN_TARGET_TRIPLE=aarch64-unknown-linux-gnu",
"-DDEFAULT_SYSROOT=C:/buildbot/.aarch64-ubuntu",
"-DZLIB_ROOT=C:/buildbot/.zlib-win32",
"-DLLVM_LIT_ARGS=-v -vv --threads=32",
WithProperties("%(remote_test_host:+-DREMOTE_TEST_HOST=)s%(remote_test_host:-)s"),
WithProperties("%(remote_test_user:+-DREMOTE_TEST_USER=)s%(remote_test_user:-)s"),
],
cmake_cache="../llvm-project/clang/cmake/caches/CrossWinToARMLinux.cmake")},
# LLD builders.
{'name' : "lld-x86_64-win-release",
'tags' : ["lld"],
'workernames' : ["as-worker-93"],
'builddir': "lld-x86_64-win-rel",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
depends_on_projects=['llvm', 'lld'],
vs="autodetect",
extra_configure_args = [
'-DLLVM_ENABLE_WERROR=OFF'])},
{'name' : "lld-x86_64-ubuntu-release",
'tags' : ["lld"],
'workernames' : ["as-builder-4"],
'builddir' : "lld-x86_64-ubuntu-rel",
'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
clean=True,
extra_configure_args=[
'-DLLVM_ENABLE_WERROR=OFF'],
depends_on_projects=['llvm', 'lld'])},
# LTO and ThinLTO builders.
{'name' : "clang-with-thin-lto-ubuntu-release",
'tags' : ["clang", "lld", "LTO"],
'workernames' : ["as-worker-92"],
'builddir': "clang-with-thin-lto-ubuntu-rel",
'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(jobs=72, lto='thin')},
{'name' : "clang-with-lto-ubuntu-release",
'tags' : ["clang", "lld", "LTO"],
'workernames' : ["as-worker-91"],
'builddir': "clang-with-lto-ubuntu-rel",
'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(
jobs=72,
extra_configure_args_lto_stage=[
'-DLLVM_PARALLEL_LINK_JOBS=14',
])},
# OpenMP builders.
{'name' : "openmp-clang-x86_64-linux-debian-release",
'tags' : ["openmp"],
'workernames' : ["gribozavr4"],
'builddir': "openmp-clang-x86_64-linux-debian-rel",
'factory' : OpenMPBuilder.getOpenMPCMakeBuildFactory(
extraCmakeArgs=[
'-DLLVM_CCACHE_BUILD=ON',
],
env={
'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
'CC': 'clang', 'CXX': 'clang++',
})},
# Sony builders.
{'name' : "llvm-clang-x86_64-sie-win-release",
'tags' : ["llvm", "clang", "clang-tools-extra", "lld", "cross-project-tests"],
'workernames' : ["sie-win-worker"],
'builddir': "x86_64-sie-win-rel",
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(
vs="autodetect",
target_arch='x64',
depends_on_projects=['llvm','clang','clang-tools-extra','lld','cross-project-tests'],
clean=True,
extra_configure_args=[
"-DCMAKE_BUILD_TYPE=Release",
"-DCLANG_ENABLE_ARCMT=OFF",
"-DCLANG_ENABLE_CLANGD=OFF",
"-DLLVM_CCACHE_BUILD=ON",
"-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-sie-ps5",
"-DLLVM_INCLUDE_EXAMPLES=OFF",
"-DLLVM_TARGETS_TO_BUILD=X86",
"-DLLVM_VERSION_SUFFIX=",
"-DLLVM_BUILD_RUNTIME=OFF",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_LIT_ARGS=--verbose"])},
{'name': "llvm-clang-x86_64-gcc-ubuntu-release",
'tags' : ["llvm", "clang", "clang-tools-extra", "compiler-rt", "lld", "cross-project-tests"],
'workernames': ["doug-worker-2a"],
'builddir': "x86_64-gcc-rel",
'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
depends_on_projects=['llvm','clang','clang-tools-extra','compiler-rt','lld','cross-project-tests'],
extra_configure_args=[
"-DCMAKE_C_COMPILER=gcc",
"-DCMAKE_CXX_COMPILER=g++",
"-DCMAKE_BUILD_TYPE=Release",
"-DCLANG_ENABLE_CLANGD=OFF",
"-DLLVM_BUILD_RUNTIME=ON",
"-DLLVM_BUILD_TESTS=ON",
"-DLLVM_ENABLE_ASSERTIONS=ON",
"-DLLVM_INCLUDE_EXAMPLES=OFF",
"-DLLVM_LIT_ARGS=--verbose -j48",
"-DLLVM_PARALLEL_LINK_JOBS=16",
"-DLLVM_USE_LINKER=gold"])},
]

buildbot/osuosl/master/config/schedulers.py

Show All 13 Lines if cp:
changed_projects = frozenset(cp.split(',')) changed_projects = frozenset(cp.split(','))
if changed_projects.intersection(projects_of_interest): if changed_projects.intersection(projects_of_interest):
return True return True
return False return False
# Since we have many parametric builders, we dynamically build the minimum set # Since we have many parametric builders, we dynamically build the minimum set
# of schedulers, which covers all actually used combinations of dependencies. # of schedulers, which covers all actually used combinations of dependencies.
def getSingleBranchSchedulers( def getMainBranchSchedulers(
builders, builders,
explicitly_set_schedulers = None, explicitly_set_schedulers = None,
**kwargs): **kwargs):
""" """
I'm taking over all of not yet assigned builders with the I'm taking over all of not yet assigned builders with the
declared source code dependencies, and automatically generate declared source code dependencies, and automatically generate
a minimum set of SingleBranchSchedulers to handle all the declared a minimum set of SingleBranchSchedulers to handle all the declared
source code dependency combinations. source code dependency combinations.
""" """
builders_with_explicit_schedulers = set() builders_with_explicit_schedulers = set()
if explicitly_set_schedulers: if explicitly_set_schedulers:
# TODO: Make a list of builder names with already set schedulers. # TODO: Make a list of builder names with already set schedulers.
# builders_with_explicit_schedulers.add(builder) # builders_with_explicit_schedulers.add(builder)
pass pass
# For the builders created with LLVMBuildFactory or similar, # For the builders created with LLVMBuildFactory or similar,
# we always use automatic schedulers, # we always use automatic schedulers,
# unless schedulers already explicitly set. # unless schedulers already explicitly set.
builders_with_automatic_schedulers = [ builders_with_automatic_schedulers = [
builder for builder in builders builder for builder in builders
if builder.name not in builders_with_explicit_schedulers if builder.name not in builders_with_explicit_schedulers
if getattr(builder.factory, 'depends_on_projects', None) if getattr(builder.factory, 'depends_on_projects', None)
if 'release' not in getattr(builder, 'tags', [])
] ]
filter_branch = 'master' filter_branch = 'main'
treeStableTimer = kwargs.get('treeStableTimer', None) treeStableTimer = kwargs.get('treeStableTimer', None)
automatic_schedulers = [] automatic_schedulers = []
# Do we have any to take care of? # Do we have any to take care of?
if builders_with_automatic_schedulers: if builders_with_automatic_schedulers:
# Let's reconsile first to get a unique set of dependencies. # Let's reconsile first to get a unique set of dependencies.
# We need a set of unique sets of dependent projects. # We need a set of unique sets of dependent projects.
set_of_dependencies = set([ set_of_dependencies = set([
frozenset(getattr(b.factory, 'depends_on_projects')) frozenset(getattr(b.factory, 'depends_on_projects'))
for b in builders_with_automatic_schedulers for b in builders_with_automatic_schedulers
]) ])
for projects in set_of_dependencies: for projects in set_of_dependencies:
sch_builders = [ main_builders = [
b.name b.name
for b in builders_with_automatic_schedulers for b in builders_with_automatic_schedulers
if frozenset(getattr(b.factory, 'depends_on_projects')) == projects if frozenset(getattr(b.factory, 'depends_on_projects')) == projects
] ]
automatic_scheduler_name = filter_branch + ":" + ",".join(sorted(projects)) automatic_scheduler_name = filter_branch + ":" + ",".join(sorted(projects))
automatic_schedulers.append( automatic_schedulers.append(
schedulers.SingleBranchScheduler( schedulers.SingleBranchScheduler(
name=automatic_scheduler_name, name=automatic_scheduler_name,
treeStableTimer=kwargs.get('treeStableTimer', None), treeStableTimer=treeStableTimer,
reason="Merge to github %s branch" % filter_branch, reason="Merge to github {} branch".format(filter_branch),
builderNames=sch_builders, builderNames=main_builders,
change_filter=util.ChangeFilter( change_filter=util.ChangeFilter(
project_fn= \ project_fn= \
lambda c, projects_of_interest=frozenset(projects): lambda c, projects_of_interest=frozenset(projects):
isProjectOfInterest(c, projects_of_interest), isProjectOfInterest(c, projects_of_interest),
branch=filter_branch) branch=filter_branch)
) )
) )
log.msg( log.msg(
"Generated SingleBranchScheduler: { name='%s'" % automatic_scheduler_name, "Generated SingleBranchScheduler: {{ name='{}'".format(automatic_scheduler_name),
", builderNames=", sch_builders, ", builderNames=", main_builders,
", change_filter=", projects, " (branch: %s)" % filter_branch, ", change_filter=", projects, " (branch: {})".format(filter_branch),
", treeStableTimer=%s" % treeStableTimer, ", treeStableTimer={}".format(treeStableTimer),
"}") "}")
return automatic_schedulers return automatic_schedulers
def getReleaseBranchSchedulers(
builders,
explicitly_set_schedulers = None,
**kwargs):
"""
I'm taking over all of not yet assigned builders with the
declared source code dependencies and the 'release' tag,
and automatically generate a minimum set of SingleBranchSchedulers
to handle all the declared source code dependency combinations.
"""
builders_with_explicit_schedulers = set()
if explicitly_set_schedulers:
# TODO: Make a list of builder names with already set schedulers.
# builders_with_explicit_schedulers.add(builder)
pass
# For the builders created with LLVMBuildFactory or similar,
# we always use automatic schedulers,
# unless schedulers already explicitly set.
builders_with_automatic_schedulers = [
builder for builder in builders
if builder.name not in builders_with_explicit_schedulers
if getattr(builder.factory, 'depends_on_projects', None)
if 'release' in getattr(builder, 'tags', [])
]
treeStableTimer = kwargs.get('treeStableTimer', None)
automatic_schedulers = []
# Do we have any to take care of?
if builders_with_automatic_schedulers:
# Let's reconsile first to get a unique set of dependencies.
# We need a set of unique sets of dependent projects.
set_of_dependencies = set([
frozenset(getattr(b.factory, 'depends_on_projects'))
for b in builders_with_automatic_schedulers
])
for projects in set_of_dependencies:
release_builders = [
b.name
for b in builders_with_automatic_schedulers
if frozenset(getattr(b.factory, 'depends_on_projects')) == projects
]
automatic_scheduler_name = "release:" + ",".join(sorted(projects))
automatic_schedulers.append(
schedulers.SingleBranchScheduler(
name=automatic_scheduler_name,
treeStableTimer=treeStableTimer,
reason="Merge to github release branch",
builderNames=release_builders,
change_filter=util.ChangeFilter(
project_fn= \
lambda c, projects_of_interest=frozenset(projects):
isProjectOfInterest(c, projects_of_interest),
branch_fn= \
lambda branch: branch.startswith('release/'))
)
)
log.msg(
"Generated release SingleBranchScheduler: {{ name='{}'".format(automatic_scheduler_name),
", builderNames=", release_builders,
", change_filter=", projects, " (branch: {release/*})",
", treeStableTimer={}".format(treeStableTimer),
"}")
return automatic_schedulers
def getForceSchedulers(builders):
# TODO: Move these settings to the configuration file. # TODO: Move these settings to the configuration file.
_repourl = "https://github.com/llvm/llvm-project" _repourl = "https://github.com/llvm/llvm-project"
_branch = "master" _branch = "main"
def getForceSchedulers(builders):
# Walk over all builders and collect their names. # Walk over all builders and collect their names.
scheduler_builders = [ scheduler_builders = [
builder.name for builder in builders builder.name for builder in builders
] ]
# Create the force schedulers. # Create the force schedulers.
return [ schedulers.ForceScheduler( return [ schedulers.ForceScheduler(
name = "force-build-scheduler", name = "force-build-scheduler",
Show All 9 Lines return [ schedulers.ForceScheduler(
"Narrow down blamelist", "Narrow down blamelist",
], ],
default = "Build a particular revision" default = "Build a particular revision"
), ),
builderNames = scheduler_builders, builderNames = scheduler_builders,
codebases = [ codebases = [
util.CodebaseParameter( util.CodebaseParameter(
codebase = "", codebase = "",
branch = util.FixedParameter( branch = util.StringParameter(
name = "branch", name = "branch",
label = "branch:",
size = 64,
default = _branch default = _branch
), ),
revision = util.StringParameter( revision = util.StringParameter(
name = "revision", name = "revision",
label = "revision:", label = "revision:",
size = 45, size = 45,
default = '' default = ''
), ),
Show All 19 Lines return [ schedulers.ForceScheduler(
default = False default = False
) )
] ]
) )
] ]
# TODO: Abstract this kind of scheduler better. # TODO: Abstract this kind of scheduler better.
def getLntSchedulers(): def getLntSchedulers():
project = "lnt" _project = "lnt"
lnt_builders = [ _branch = 'master'
_repourl = "https://github.com/llvm/llvm-lnt"
_lnt_builders = [
"publish-lnt-sphinx-docs", "publish-lnt-sphinx-docs",
] ]
return [ return [
schedulers.SingleBranchScheduler( schedulers.Nightly(
name="lnt-scheduler", name="lnt-scheduler",
treeStableTimer=kwargs.get('treeStableTimer', None), builderNames=_lnt_builders,
reason="Merge to LNT github {} branch".format(filter_branch), reason='Periodic LNT build',
builderNames=lnt_builders, hour=1),
change_filter=util.ChangeFilter(
project_fn=project,
branch=filter_branch)),
schedulers.ForceScheduler( schedulers.ForceScheduler(
name = "force-build-scheduler", name = "lnt-force-build-scheduler",
label = "Force Build", label = "Force Build",
buttonName = "Force Build", buttonName = "Force Build",
reason = util.ChoiceStringParameter( reason = util.ChoiceStringParameter(
name = "reason", name = "reason",
label = "reason:", label = "reason:",
required = True, required = True,
choices = [ choices = [
"Build a particular revision", "Build a particular revision",
"Force clean build", "Force clean build",
"Narrow down blamelist", "Narrow down blamelist",
], ],
default = "Build a particular revision" default = "Build a particular revision"
), ),
builderNames = lnt_builders, builderNames = _lnt_builders,
codebases = [ codebases = [
util.CodebaseParameter( util.CodebaseParameter(
codebase = "", codebase = "",
branch = util.FixedParameter( branch = util.FixedParameter(
name = "branch", name = "branch",
default = _branch default = _branch
), ),
revision = util.StringParameter( revision = util.StringParameter(
name = "revision", name = "revision",
label = "revision:", label = "revision:",
size = 45, size = 45,
default = '' default = ''
), ),
repository = util.FixedParameter( repository = util.FixedParameter(
name = "repository", name = "repository",
default = _repourl default = _repourl
), ),
project = util.FixedParameter( project = util.FixedParameter(
name = "project", name = "project",
default = project default = _project
) )
) )
], ],
properties = [ properties = [
util.BooleanParameter( util.BooleanParameter(
name = "clean", name = "clean",
label = "Clean source code and build directory", label = "Clean source code and build directory",
default = False default = False
Show All 9 Lines

buildbot/osuosl/master/config/status.py

from buildbot.process.properties import Interpolate from buildbot.process.properties import Interpolate
from buildbot.plugins import reporters from buildbot.plugins import reporters
import config import config
from zorg.buildbot.util.InformativeMailNotifier import LLVMInformativeMailNotifier from zorg.buildbot.util.InformativeMailNotifier import LLVMInformativeMailNotifier
# Should be a single e-mail address # Should be a single e-mail address
status_email = str(config.options.get('Master Options', 'status_email')).split(',') status_email = str(config.options.get('Master Options', 'status_email')).split(',')
all = [ all = [
# Note: reporters.GitHubStatusPush requires txrequests package to allow # Report github status for all the release builders,
# interaction with GitHub REST API. # i.e. those with the "release" tag.
reporters.GitHubStatusPush( reporters.GitHubStatusPush(
str(config.options.get('GitHub Status', 'token')), str(config.options.get('GitHub Status', 'token')),
context = Interpolate("%(prop:buildername)s"), context = Interpolate("%(prop:buildername)s"),
verbose = True, # TODO: Turn off the verbosity once this is working reliably. verbose = True, # TODO: Turn off the verbosity once this is working reliably.
builders = [ builders = [
"llvm-clang-x86_64-expensive-checks-ubuntu", "llvm-clang-x86_64-expensive-checks-ubuntu",
"llvm-clang-x86_64-win-fast", "llvm-clang-x86_64-win-fast",
"clang-x86_64-debian-fast", "clang-x86_64-debian-fast",
"llvm-clang-x86_64-expensive-checks-debian", "llvm-clang-x86_64-expensive-checks-debian",
]), "llvm-clang-x86_64-sie-ubuntu-fast",
] + [
b.get('name') for b in config.release_builders.all
if 'release' in b.get('tags', [])
]
),
reporters.IRC( reporters.IRC(
useColors = False, useColors = False,
host = str(config.options.get('IRC', 'host')), host = str(config.options.get('IRC', 'host')),
nick = str(config.options.get('IRC', 'nick')), nick = str(config.options.get('IRC', 'nick')),
channels = str(config.options.get('IRC', 'channels')).split(','), channels = str(config.options.get('IRC', 'channels')).split(','),
#authz=... # TODO: Consider allowing "harmful" operations to authorizes users. #authz=... # TODO: Consider allowing "harmful" operations to authorizes users.
useRevisions = False, # FIXME: There is a bug in the buildbot useRevisions = False, # FIXME: There is a bug in the buildbot
showBlameList = True, showBlameList = True,
notify_events = str(config.options.get('IRC', 'notify_events')).split(','), notify_events = str(config.options.get('IRC', 'notify_events')).split(','),
), ),
reporters.MailNotifier( reporters.MailNotifier(
mode = ('problem',), mode = ('problem',),
fromaddr = "llvm.buildmaster@lab.llvm.org", # TODO: Change this to buildmaster@lab.llvm.org. fromaddr = "llvm.buildmaster@lab.llvm.org", # TODO: Change this to buildmaster@lab.llvm.org.
extraRecipients = status_email, extraRecipients = status_email,
extraHeaders = {"Reply-To": status_email[0]}, # The first from the list. extraHeaders = {"Reply-To": status_email[0]}, # The first from the list.
lookup = "lab.llvm.org", lookup = "lab.llvm.org",
messageFormatter = LLVMInformativeMailNotifier, messageFormatter = LLVMInformativeMailNotifier,
# TODO: For debug purposes only. Remove later. # TODO: For debug purposes only. Remove later.
dumpMailsToLog = True, dumpMailsToLog = True,
builders = [
b.get('name') for b in config.builders.all
if 'silent' not in b.get('tags', [])
]
), ),
# In addition to that the following notifiers are defined for special
# cases.
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["leandro.nunes@arm.com"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["clang-aarch64-linux-build-cache", "clang-armv7-linux-build-cache"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["gribozavr@gmail.com"], extraRecipients = ["gribozavr@gmail.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["clang-x86_64-debian-fast"]), builders = ["clang-x86_64-debian-fast"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["mstester.llvm@gmail.com"], extraRecipients = ["tbaeder@redhat.com", "tstellar@redhat.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["clang-x64-ninja-win7"]), builders = ["llvm-x86_64-debian-dylib"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["llvm.buildmaster@quicinc.com"], extraRecipients = ["llvm.buildmaster@quicinc.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["clang-hexagon-elf"]), builders = ["clang-hexagon-elf"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["Ulrich.Weigand@de.ibm.com"], extraRecipients = ["Ulrich.Weigand@de.ibm.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["clang-s390x-linux", "clang-s390x-linux-multistage", builders = ["clang-s390x-linux", "clang-s390x-linux-multistage",
"clang-s390x-linux-lnt"]), "clang-s390x-linux-lnt", "mlir-s390x-linux"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["sunil_srivastava@playstation.sony.com", extraRecipients = ["sunil_srivastava@playstation.sony.com",
"warren_ristow@playstation.sony.com"], "warren_ristow@playstation.sony.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["clang-x86_64-linux-abi-test"]), builders = ["clang-x86_64-linux-abi-test"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["me@dylanmckay.io"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["llvm-avr-linux"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["gkistanova@gmail.com"], extraRecipients = ["gkistanova@gmail.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["lld-x86_64-win", builders = ["lld-x86_64-win",
"lld-x86_64-freebsd", "lld-x86_64-darwin", "lld-x86_64-freebsd",
"clang-x86_64-linux-abi-test", "clang-x86_64-linux-abi-test",
"clang-with-lto-ubuntu", "clang-with-thin-lto-ubuntu", "clang-with-lto-ubuntu", "clang-with-thin-lto-ubuntu",
"llvm-clang-x86_64-expensive-checks-win", "llvm-clang-x86_64-expensive-checks-win",
"llvm-clang-x86_64-expensive-checks-ubuntu"]), "llvm-clang-x86_64-expensive-checks-ubuntu"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["efriedma@codeaurora.org", "huihuiz@codeaurora.org"], extraRecipients = ["efriedma@codeaurora.org", "huihuiz@codeaurora.org"],
Show All 20 Lines reporters.MailNotifier(
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["asb@lowrisc.org"], extraRecipients = ["asb@lowrisc.org"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["llvm-riscv-linux"]), builders = ["llvm-riscv-linux"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["stilis@microsoft.com", "jonas@devlieghere.com", extraRecipients = ["aifxbuildbotdri@microsoft.com", "namcvica@microsoft.com"],
"diprou@microsoft.com", "makudrya@microsoft.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["lldb-x64-windows-ninja"]), builders = ["mlir-windows"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["stilis@microsoft.com", "namcvica@microsoft.com"], extraRecipients = ["gongsu@us.ibm.com", "alexe@us.ibm.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["mlir-windows"]), builders = ["mlir-s390x-linux-werror"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["phosek@google.com"], extraRecipients = ["phosek@google.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["fuchsia-x86_64-linux"]), builders = ["fuchsia-x86_64-linux"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["jan.kratochvil@redhat.com"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["lldb-x86_64-fedora"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["kkleine@redhat.com"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["fedora-llvm-x86_64", "x86_64-fedora-clang"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["labath@google.com"], extraRecipients = ["labath@google.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["lldb-x86_64-debian"]), builders = ["lldb-x86_64-debian"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["omair.javaid@linaro.org"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["lldb-arm-ubuntu","lldb-aarch64-ubuntu"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["vvereschaka@accesssoftek.com"], extraRecipients = ["vvereschaka@accesssoftek.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["llvm-clang-x86_64-win-fast","lld-x86_64-ubuntu-fast", builders = ["llvm-clang-x86_64-win-fast","lld-x86_64-ubuntu-fast",
"llvm-clang-x86_64-expensive-checks-ubuntu", "llvm-clang-x86_64-expensive-checks-ubuntu",
"llvm-clang-win-x-armv7l", "llvm-clang-win-x-aarch64"]), "llvm-clang-win-x-armv7l", "llvm-clang-win-x-aarch64",
"llvm-nvptx-nvidia-ubuntu", "llvm-nvptx64-nvidia-ubuntu",
"llvm-nvptx-nvidia-win", "llvm-nvptx64-nvidia-win"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["llvm.buildbot@emea.nec.com"], extraRecipients = ["llvm.buildbot@emea.nec.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["clang-ve-ninja"]), builders = ["clang-ve-ninja"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["sivachandra@google.com", "paulatoth@google.com"], extraRecipients = ["lntue@google.com", "michaelrj@google.com",
"sivachandra@google.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["libc-x86_64-debian", "libc-x86_64_debian-dbg", builders = ["libc-x86_64-debian", "libc-x86_64_debian-dbg",
"libc-x86_64-debian-dbg-asan"]), "libc-x86_64-debian-dbg-runtimes-build",
"libc-x86_64-debian-dbg-asan", "libc-aarch64-ubuntu-dbg",
"libc-x86_64-windows-dbg", "libc-arm32-debian-dbg",
"libc-aarch64-ubuntu-fullbuild-dbg",
"libc-x86_64-debian-fullbuild-dbg",
"libc-x86_64-debian-gcc-fullbuild-dbg",
"libc-x86_64-debian-fullbuild-dbg-asan",
"libc-riscv64-debian-dbg",
"libc-riscv64-debian-fullbuild-dbg",
"libc-x86_64-debian-dbg-lint"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = ["aaron@aaronballman.com"], extraRecipients = ["aaron@aaronballman.com"],
subject = "Sphinx build %(builder)s Failure", subject = "Sphinx build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = ["publish-sphinx-docs"]), builders = ["publish-sphinx-docs"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr="llvm.buildmaster@lab.llvm.org", fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients=[ extraRecipients=[
"mlcompileropt-buildbot@google.com"], "mlcompileropt-buildbot@google.com"],
subject = "ML Compiler Opt Failure: %(builder)s", subject = "ML Compiler Opt Failure: %(builder)s",
mode = "failing", mode = "failing",
builders = [ builders = [
"ml-opt-dev-x86-64", "ml-opt-rel-x86-64", "ml-opt-devrel-x86-64"]), "ml-opt-dev-x86-64", "ml-opt-rel-x86-64", "ml-opt-devrel-x86-64"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org", fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = [ extraRecipients=[
"caroline.concatto@arm.com", "flang_llvm_buildbots@arm.com"], "tejohnson@google.com"],
subject = "ThinLTO WPD Failure: %(builder)s",
mode = "failing",
builders = ["clang-with-thin-lto-wpd-ubuntu"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["douglas.yung@sony.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = [ builders = ["llvm-clang-x86_64-sie-ubuntu-fast",
"flang-aarch64-ubuntu", "flang-aarch64-ubuntu-clang", "llvm-clang-x86_64-sie-win",
"flang-aarch64-ubuntu-gcc10"]), "llvm-clang-x86_64-sie-win-release",
"llvm-clang-x86_64-gcc-ubuntu",
"llvm-clang-x86_64-gcc-ubuntu-release",
"cross-project-tests-sie-ubuntu",
"cross-project-tests-sie-ubuntu-dwarf5"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["tom.weaver@sony.com"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["llvm-clang-x86_64-sie-ubuntu-fast",
"llvm-clang-x86_64-sie-win",
"llvm-clang-x86_64-gcc-ubuntu"]),
reporters.MailNotifier( reporters.MailNotifier(
fromaddr="llvm.buildmaster@lab.llvm.org", fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients=[ extraRecipients=[
"tejohnson@google.com"], "joker.eph@gmail.com"],
subject = "ThinLTO WPD Failure: %(builder)s", subject = "MLIR Build Failure: %(builder)s",
mode = "failing", mode = "failing",
builders = ["thinlto-x86-64-bot1"]), builders = ["mlir-nvidia"]),
reporters.MailNotifier(
fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients=[
"mlir-bugs-external+buildbot@googlegroups.com"],
subject = "MLIR Build Failure: %(builder)s",
mode = "failing",
builders = ["mlir-nvidia",
"mlir-windows",
"ppc64le-mlir-rhel-clang"]),
reporters.MailNotifier(
fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients=["dl.gcr.lightning.buildbot@amd.com"],
subject = "Build Failure: %(builder)s",
mode = "failing",
builders = ["clang-hip-vega20"]),
reporters.MailNotifier(
fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients=["llvm_arc_buildbot@synopsys.com", "heli@synopsys.com"],
subject = "Build Failure: %(builder)s",
mode = "failing",
builders = ["arc-builder"]),
reporters.MailNotifier(
fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients=["dl.gcr.lightning.buildbot@amd.com"],
subject = "Build Failure: %(builder)s",
mode = "failing",
builders = ["openmp-offload-amdgpu-runtime"]),
reporters.MailNotifier(
fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients=["dl.mlse.buildbot@amd.com"],
subject = "Build Failure: %(builder)s",
mode = "failing",
builders = ["mlir-rocm-mi200"]),
reporters.MailNotifier(
fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients=["flangbuilder@meinersbur.de"],
subject = "Build Failure (flang): %(builder)s",
mode = "failing",
builders = ["flang-x86_64-windows"]),
reporters.MailNotifier(
fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients=["offloadbuilder@meinersbur.de"],
subject = "Build Failure (offload): %(builder)s",
mode = "failing",
builders = ["openmp-offload-cuda-project","openmp-offload-cuda-runtime"]),
reporters.MailNotifier(
fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients=["pollybuilder@meinersbur.de"],
subject = "Build Failure (polly): %(builder)s",
mode = "failing",
builders = [
"polly-x86_64-linux",
"polly-x86_64-linux-noassert",
"polly-x86_64-linux-plugin",
"polly-x86_64-linux-shared",
"polly-x86_64-linux-shared-plugin",
"polly-x86_64-linux-shlib",
"polly-x86_64-linux-shlib-plugin",
"polly-sphinx-docs",
]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["orlando.hyams@sony.com"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["cross-project-tests-sie-ubuntu",
"llvm-clang-x86_64-sie-win"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["kkleine@redhat.com"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["standalone-build-x86_64"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["llvm-bolt@meta.com"],
subject = "BOLT NFC checks mismatch",
mode = ("warnings",),
builders = ["bolt-x86_64-ubuntu-nfc"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["luweining@loongson.cn", "chenli@loongson.cn"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["clang-loongarch64-linux"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["kadircet@google.com", "sammccall@google.com"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["clangd-ubuntu-tsan"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["kadircet@google.com", "ibiryukov@google.com"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["clang-debian-cpp20"]),
reporters.MailNotifier(
fromaddr = "llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients = ["llvm.buildbot.notification@intel.com"],
subject = "Build %(builder)s Failure",
mode = "failing",
builders = ["clang-cmake-x86_64-avx512-linux"]),
] ]
from twisted.python import log
# Returns a list of Status Targets. The results of each build will be # Returns a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot.plugins reporters has a variety # pushed to these targets. buildbot.plugins reporters has a variety
# to choose from, including email senders, and IRC bots. # to choose from, including email senders, and IRC bots.
def getReporters(): def getReporters():
if config.options.getboolean('Master Options', 'is_production'): if config.options.getboolean('Master Options', 'is_production'):
log.msg(">>> getReporters: Production mode. All reporters are registered.")
return all return all
else: else:
# Staging buildbot does not report issues. # Staging buildbot does not report issues.
log.msg(">>> getReporters: Staging buildbot does not report issues.")
return [] return []

buildbot/osuosl/master/config/workers.py

Show All 10 Lines
def get_all(): def get_all():
return [ return [
# FreeBSD # FreeBSD
create_worker("as-worker-4", properties={'jobs' : 24}, max_builds=2), create_worker("as-worker-4", properties={'jobs' : 24}, max_builds=2),
# Linux Ubuntu # Linux Ubuntu
create_worker("as-worker-5", properties={'jobs' : 16}), create_worker("as-worker-5", properties={'jobs' : 16}),
# Mac Pro 2.7 GHz 12-Core Intel Xeon E5, Maverick 10.9.2
create_worker("as-worker-3", properties={'jobs' : 8}, max_builds=4),
# ARMv7/ARMv8 Linaro workers # ARMv7/ARMv8 Linaro workers
create_worker("linaro-tk1-01", properties={'jobs' : 4}, max_builds=1),
create_worker("linaro-tk1-02", properties={'jobs' : 4}, max_builds=1), create_worker("linaro-tk1-02", properties={'jobs' : 4}, max_builds=1),
create_worker("linaro-tk1-03", properties={'jobs' : 4}, max_builds=1), create_worker("linaro-clang-armv7-lnt", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-tk1-04", properties={'jobs' : 4}, max_builds=1), create_worker("linaro-clang-armv7-2stage", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-tk1-05", properties={'jobs' : 4}, max_builds=1), create_worker("linaro-clang-armv7-global-isel", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-tk1-06", properties={'jobs' : 4}, max_builds=1), create_worker("linaro-clang-armv7-vfpv3-2stage", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-tk1-07", properties={'jobs' : 4}, max_builds=1), create_worker("linaro-clang-armv8-lld-2stage", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-tk1-08", properties={'jobs' : 4}, max_builds=1), create_worker("linaro-clang-armv8-quick", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-tk1-09", properties={'jobs' : 4}, max_builds=1), create_worker("linaro-lldb-arm-ubuntu", properties={'jobs' : 16}, max_builds=1),
create_worker("linaro-armv7-lnt", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-armv7-selfhost", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-armv7-quick", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-armv7-global-isel", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-armv8-lld", properties={'jobs' : 32}, max_builds=1),
# Libcxx testsuite has tests with timing assumptions. Run single-threaded to make
# sure we have plenty CPU cycle to satisfy timing assumptions.
create_worker("linaro-armv8-libcxx", properties={'jobs' : 1}, max_builds=1),
create_worker("linaro-arm-lldb", properties={'jobs' : 16}, max_builds=1),
# AArch64 Linaro workers # AArch64 Linaro workers
create_worker("linaro-aarch64-quick", properties={'jobs' : 32}, max_builds=1), create_worker("linaro-clang-aarch64-quick", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-aarch64-full", properties={'jobs' : 32}, max_builds=1), create_worker("linaro-clang-aarch64-lld-2stage", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-aarch64-global-isel", properties={'jobs' : 32}, max_builds=1), create_worker("linaro-clang-aarch64-global-isel", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-aarch64-lld", properties={'jobs' : 32}, max_builds=1), create_worker("linaro-clang-aarch64-full-2stage", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-aarch64-flang-oot", properties={'jobs' : 32}, max_builds=1), create_worker("linaro-lldb-aarch64-ubuntu", properties={'jobs': 16}, max_builds=1),
create_worker("linaro-aarch64-flang-dylib", properties={'jobs' : 32}, max_builds=1), create_worker("linaro-flang-aarch64-dylib", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-aarch64-flang-sharedlibs", properties={'jobs' : 32}, max_builds=1), create_worker("linaro-flang-aarch64-sharedlibs", properties={'jobs' : 32}, max_builds=1),
# Libcxx testsuite has tests with timing assumptions. Run single-threaded to make create_worker("linaro-flang-aarch64-out-of-tree", properties={'jobs' : 32}, max_builds=1),
# sure we have plenty CPU cycle to satisfy timing assumptions. create_worker("linaro-flang-aarch64-debug", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-aarch64-libcxx", properties={'jobs' : 1}, max_builds=1), create_worker("linaro-flang-aarch64-latest-clang", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-aarch64-lldb", properties={'jobs': 16}, max_builds=1), create_worker("linaro-flang-aarch64-release", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-flang-aarch64-rel-assert", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-flang-aarch64-latest-gcc", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-g3-01", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-g3-02", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-g3-03", properties={'jobs' : 32}, max_builds=1),
create_worker("linaro-g3-04", properties={'jobs' : 32}, max_builds=1),
# AArch64 # AArch64 Windows Microsoft Surface X Pro
create_worker("linaro-armv8-windows-msvc-01", properties={'jobs' : 8}, max_builds=1), create_worker("linaro-armv8-windows-msvc-01", properties={'jobs' : 8}, max_builds=1),
create_worker("linaro-armv8-windows-msvc-02", properties={'jobs' : 8}, max_builds=1), create_worker("linaro-armv8-windows-msvc-02", properties={'jobs' : 8}, max_builds=1),
create_worker("linaro-armv8-windows-msvc-03", properties={'jobs' : 8}, max_builds=1),
# ARMv7 build cache workers create_worker("linaro-armv8-windows-msvc-04", properties={'jobs' : 8}, max_builds=1),
create_worker("packet-linux-armv7-slave-1", properties={'jobs' : 64}, max_builds=1), create_worker("linaro-armv8-windows-msvc-05", properties={'jobs' : 8}, max_builds=1),
# AArch64 build cache worker
create_worker("packet-linux-aarch64-slave-1", properties={'jobs' : 64}, max_builds=1),
# Windows Server 2016 Intel Xeon(R) Quad 2.30 GHz, 56GB of RAM
create_worker("win-py3-buildbot", properties={'jobs' : 64}, max_builds=1),
# Windows Server 2016 Intel(R) Xeon(R) CPU @ 2.60GHz, 16 Core(s), 128 GB of RAM # Windows Server 2016 Intel(R) Xeon(R) CPU @ 2.60GHz, 16 Core(s), 128 GB of RAM
create_worker("win-mlir-buildbot", properties={'jobs' : 64}, max_builds=1), create_worker("win-mlir-buildbot", properties={'jobs' : 64}, max_builds=1),
# Linux s390x Ubuntu Focal, IBM z13 (5GHz), 64GB of RAM
create_worker("onnx-mlir-nowarn-linux-s390x", properties={'jobs' : 4}, max_builds=1),
# LoongArch64 Loongson-3C5000L-LL 2GHz, 128GB of RAM
# CLFS repo: https://github.com/sunhaiyong1978/CLFS-for-LoongArch
create_worker("loongson-loongarch64-clfs-clang-01", properties={'jobs': 32}, max_builds=1),
# MIPS Loongson-3A R4 (Loongson-3A4000) 64-bit little endian (mips64el)
create_worker("debian-tritium-mips64el", properties={'jobs': 1}, max_builds=1),
# Motorola 68k 32-bit big endian (m68k)
create_worker("debian-akiko-m68k", properties={'jobs': 1}, max_builds=1),
create_worker("suse-gary-m68k-cross", properties={'jobs': 4}, max_builds=1),
# POWER7 PowerPC big endian (powerpc64) # POWER7 PowerPC big endian (powerpc64)
create_worker("ppc64be-clang-test", properties={'jobs': 16}, max_builds=1), create_worker("ppc64be-clang-test-suite", max_builds=1),
create_worker("ppc64be-clang-lnt-test", properties={'jobs': 16, 'vcs_protocol': 'https'}, max_builds=1),
create_worker("ppc64be-clang-multistage-test", properties={'jobs': 16}, max_builds=1), create_worker("ppc64be-clang-multistage-test", properties={'jobs': 16}, max_builds=1),
create_worker("ppc64be-sanitizer", properties={'jobs': 16}, max_builds=1), create_worker("ppc64be-sanitizer", properties={'jobs': 16}, max_builds=1),
# POWER 8 PowerPC little endian (powerpc64le) # POWER 8 PowerPC little endian (powerpc64le)
create_worker("ppc64le-clang-test", properties={'jobs': 4}, max_builds=1), create_worker("ppc64le-clang-test-suite", max_builds=1),
create_worker("ppc64le-clang-lnt-test", properties={'jobs': 8}, max_builds=1),
create_worker("ppc64le-clang-multistage-test", properties={'jobs': 8}, max_builds=1), create_worker("ppc64le-clang-multistage-test", properties={'jobs': 8}, max_builds=1),
create_worker("ppc64le-sanitizer", properties={'jobs': 4}, max_builds=1), create_worker("ppc64le-sanitizer", properties={'jobs': 4}, max_builds=1),
create_worker("ppc64le-lld-multistage-test", max_builds=1), create_worker("ppc64le-lld-multistage-test", max_builds=1),
create_worker("ppc64le-clang-rhel-test", properties={'jobs': 4}, max_builds=1), create_worker("ppc64le-clang-rhel-test", properties={'jobs': 192}, max_builds=1),
create_worker("ppc64le-flang-mlir-rhel-test", max_builds=1), create_worker("ppc64le-flang-rhel-test", max_builds=1),
create_worker("ppc64le-mlir-rhel-test", max_builds=1),
# SPARC 64-bit big endian (sparc64) # SPARC 64-bit big endian (sparc64)
create_worker("debian-stadler-sparc64", properties={'jobs': 4}, max_builds=1), create_worker("debian-stadler-sparc64", properties={'jobs': 4}, max_builds=1),
# Ubuntu x86-64, Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz # Ubuntu x86-64, Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
create_worker("hexagon-build-02", properties={'jobs': 12, 'loadaverage': 32}, create_worker("hexagon-build-02", properties={'jobs': 12, 'loadaverage': 32},
max_builds=1), max_builds=1),
# Ubuntu x86-64, Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz # Ubuntu x86-64, Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
create_worker("hexagon-build-03", properties={'jobs': 16, 'loadaverage':32}, create_worker("hexagon-build-03", properties={'jobs': 16, 'loadaverage':32},
max_builds=1), max_builds=1),
# Ubuntu x86-64
create_worker("avr-build-01", properties={'jobs': 10}, max_builds=1),
# Debian Jessie x86-64 GCE instance. # Debian Jessie x86-64 GCE instance.
create_worker("gribozavr3", properties={'jobs': 1}, max_builds=2), create_worker("gribozavr3", properties={'jobs': 1}, max_builds=2),
# Debian x86-64 # Debian x86-64
create_worker("gribozavr4", properties={'jobs': 96}, max_builds=2), create_worker("gribozavr4", properties={'jobs': 96}, max_builds=2),
create_worker("gribozavr5", properties={'jobs': 96}, max_builds=2, create_worker("gribozavr5", properties={'jobs': 96}, max_builds=2,
missing_timeout=300, # notify after 5 minutes of missing missing_timeout=300, # notify after 5 minutes of missing
notify_on_missing=[ notify_on_missing=[
"gribozavr@gmail.com", "gribozavr@gmail.com",
"gkistanova@gmail.com"]), "gkistanova@gmail.com"]),
# Debian 7.7 x86_64 GCE instance # Ubuntu x86_64 6-core z440 workstation
create_worker("sanitizer-buildbot-android", properties={'jobs': 6}, max_builds=1),
# x86_64 GCE instances
create_worker("sanitizer-buildbot1", properties={'jobs': 64}, max_builds=3), create_worker("sanitizer-buildbot1", properties={'jobs': 64}, max_builds=3),
# Debian 7.7 x86_64 GCE instance
create_worker("sanitizer-buildbot2", properties={'jobs': 64}, max_builds=3), create_worker("sanitizer-buildbot2", properties={'jobs': 64}, max_builds=3),
# Debian 7.7 x86_64 GCE instance create_worker("sanitizer-buildbot3", properties={'jobs': 64}, max_builds=3),
create_worker("sanitizer-buildbot3", properties={'jobs': 64}, max_builds=2), create_worker("sanitizer-buildbot4", properties={'jobs': 64}, max_builds=3),
# Debian 7.7 x86_64 GCE instance create_worker("sanitizer-buildbot5", properties={'jobs': 64}, max_builds=3),
create_worker("sanitizer-buildbot4", properties={'jobs': 64}, max_builds=2), create_worker("sanitizer-buildbot6", properties={'jobs': 64}, max_builds=3),
# Ubuntu 14.04 x86_64 6-core z440 workstation # aarch64 GCE instances
create_worker("sanitizer-buildbot6", properties={'jobs': 6}, max_builds=1), create_worker("sanitizer-buildbot7", properties={'jobs': 48}, max_builds=2),
# Debian 7.7 x86_64 GCE instance create_worker("sanitizer-buildbot8", properties={'jobs': 48}, max_builds=3),
create_worker("sanitizer-buildbot7", properties={'jobs': 64}, max_builds=3), create_worker("sanitizer-buildbot9", properties={'jobs': 48}, max_builds=2),
# Debian 7.7 x86_64 GCE instance create_worker("sanitizer-buildbot10", properties={'jobs': 48}, max_builds=3),
create_worker("sanitizer-buildbot8", properties={'jobs': 64}, max_builds=3), create_worker("sanitizer-buildbot11", properties={'jobs': 48}, max_builds=2),
create_worker("sanitizer-buildbot12", properties={'jobs': 48}, max_builds=3),
# POWER 8 PowerPC AIX 7.2
create_worker("aix-ppc64", properties={'jobs': 10}, max_builds=1),
# IBM z13 (s390x), Ubuntu 16.04.2 # IBM z13 (s390x), Ubuntu 16.04.2
create_worker("systemz-1", properties={'jobs': 4, 'vcs_protocol': 'https'}, max_builds=4), create_worker("systemz-1", properties={'jobs': 4, 'vcs_protocol': 'https'}, max_builds=4),
# Windows Server 2012 x86_64 16-core GCE instance # Windows Server 2012 x86_64 16-core GCE instance
create_worker("sanitizer-windows", properties={'jobs': 16}, max_builds=1), create_worker("sanitizer-windows", properties={'jobs': 16}, max_builds=1),
# Windows Server 2012 x86_64 32-core GCE instance # Windows Server 2012 x86_64 32-core GCE instance
create_worker("windows-gcebot2", properties={'jobs': 32}, max_builds=1), create_worker("windows-gcebot2", properties={'jobs': 32}, max_builds=1),
# Ubuntu 14.04 x86_64-scei-ps4, 2 x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz # Ubuntu 14.04 x86_64-scei-ps4, 2 x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
create_worker("as-worker-91"), create_worker("as-worker-91", max_builds=1),
create_worker("as-worker-92"), create_worker("as-worker-92", max_builds=1),
# Windows 10 Pro x86_64-scei-ps4, 2 x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz # Windows 10 Pro x86_64-scei-ps4, 2 x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
create_worker("as-worker-93", properties={'jobs': 36}, max_builds=1), create_worker("as-worker-93", properties={'jobs': 36}, max_builds=1),
# WIP migration of the CUDA buildbot to GCE. # WIP migration of the CUDA buildbot to GCE.
create_worker("cuda-k80-0", max_builds=1), create_worker("cuda-k80-0", max_builds=1),
create_worker("cuda-p4-0", max_builds=1), create_worker("cuda-p4-0", max_builds=1),
create_worker("cuda-t4-0", max_builds=1), create_worker("cuda-t4-0", max_builds=1),
# X86_64 AVX2, Ubuntu 16.04.2, Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz # HIP on Ubuntu 18.04.5, Intel(R) Xeon(R) Gold 5218 @ 2.30GHz, Vega20 GPU
create_worker("avx2-intel64", properties={'jobs': 80}, max_builds=1), create_worker("hip-vega20-0", max_builds=1),
# X86_64 with SDE, Ubuntu 16.04.2, Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz # X86_64 AVX512, Ubuntu 22.04.1 LTS, Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
create_worker("sde-avx512-intel64", properties={'jobs': 80}, max_builds=1), create_worker("avx512-intel64", properties={'jobs': 80}, max_builds=1),
# X86_64 AVX512, Windows Server 2022 Standard, Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
create_worker("avx512-intel64-win", properties={'jobs': 80}, max_builds=1),
# Debian 9.0 x86_64 64-core GCE instances # Debian 9.0 x86_64 64-core GCE instances
create_worker("fuchsia-debian-64-us-central1-a-1", properties={'jobs': 64}, max_builds=1), create_worker("fuchsia-debian-64-us-central1-a-1", properties={'jobs': 64}, max_builds=1),
create_worker("fuchsia-debian-64-us-central1-b-1", properties={'jobs': 64}, max_builds=1), create_worker("fuchsia-debian-64-us-central1-b-1", properties={'jobs': 64}, max_builds=1),
# Fedora latest stable x86_64, Intel i5-2500, 4 cores, 12GB RAM
create_worker("lldb-x86_64-fedora", properties={'jobs': 4}, max_builds=1),
# Fedora latest stable, arch=x86_64, running on RedHat internal OpenShift PSI cluster
create_worker("fedora-llvm-x86_64", properties={'jobs': 64}, max_builds=1),
# Debian x86_64 Buster Xeon(R) Gold 6154 CPU @ 3.00GHz, 192GB RAM # Debian x86_64 Buster Xeon(R) Gold 6154 CPU @ 3.00GHz, 192GB RAM
create_worker("lldb-x86_64-debian", properties={'jobs': 72}, max_builds=1), create_worker("lldb-x86_64-debian", properties={'jobs': 72}, max_builds=1),
# Windows x86_64 32 CPUs, 125 GB RAM
create_worker("libc-x86_64-windows", properties={'jobs': 32}, max_builds=2),
# Debian arm32 single core, 512 MB RAM backed by 32 GB swap memory
create_worker("libc-arm32-debian", properties={'jobs': 1}, max_builds=1),
# Ubuntu aarch64 128 CPUs, 125 GB RAM
create_worker("libc-aarch64-ubuntu", properties={'jobs': 32}, max_builds=2),
# Debian x86_64 Intel Broadwell 32 CPUs, 120 GB RAM # Debian x86_64 Intel Broadwell 32 CPUs, 120 GB RAM
create_worker("libc-x86_64-debian", properties={'jobs': 32}, max_builds=2), create_worker("libc-x86_64-debian", properties={'jobs': 32}, max_builds=2),
# Debian x86_64 Intel Skylake 32 CPUs, 128 GB RAM
create_worker("libc-x86_64-debian-fullbuild", properties={'jobs': 32}, max_builds=2),
# Debian x86_64 Intel Cascade Lake 8 CPU, 32 GB RAM
create_worker("libc-lint-worker", properties={'jobs': 16}, max_builds=2),
# Debian riscv64 4 CPU, 4 GB RAM
create_worker("libc-riscv64-debian", properties={'jobs': 4}, max_builds=1),
# Windows Server on Xeon Gold 6130 (2x2.1GHz), 128Gb of RAM # Windows Server on Xeon Gold 6130 (2x2.1GHz), 128Gb of RAM
create_worker("as-builder-1", properties={ create_worker("as-builder-1", properties={
'remote_test_host': 'jetson6.lab.llvm.org', 'remote_test_host': 'jetson6.lab.llvm.org',
'remote_test_user': 'ubuntu' 'remote_test_user': 'ubuntu'
}, },
max_builds=1), max_builds=1),
# Windows Server on Xeon Gold 6130 (2x2.1GHz), 128Gb of RAM # Windows Server on Xeon Gold 6130 (2x2.1GHz), 128Gb of RAM
create_worker("as-builder-2", properties={ create_worker("as-builder-2", properties={
'remote_test_host': 'jetson9.lab.llvm.org', 'remote_test_host': 'jetson9.lab.llvm.org',
'remote_test_user': 'ubuntu' 'remote_test_user': 'ubuntu'
}, },
max_builds=1), max_builds=1),
# Windows dellfx2-sled3 # Windows dellfx2-sled3
create_worker("as-builder-3", max_builds=1), create_worker("as-builder-3", max_builds=1),
# Ubuntu 18.04.2 LTS x86_64 Intel(R) Xeon(R) Gold CPU @ 2.1GHz, 128GB RAM # Ubuntu 18.04.2 LTS x86_64 Intel(R) Xeon(R) Gold CPU @ 2.1GHz, 128GB RAM
create_worker("as-builder-4", properties={'jobs': 64}, max_builds=1), create_worker("as-builder-4", properties={'jobs': 64}, max_builds=1),
# Windows Server on Xeon Gold 6230 (2x2.1GHz), 256Gb of RAM
create_worker("as-builder-5", properties={ # arm
'remote_test_host': 'jetson4.lab.llvm.org',
'remote_test_user': 'ubuntu'
},
max_builds=1),
create_worker("as-builder-6", properties={ # aarch64
'remote_test_host': 'jetson8.lab.llvm.org',
'remote_test_user': 'ubuntu'
},
max_builds=1),
# NVPTX builders.
# Ubuntu 22.04 LTS x86_64 Intel Xeon 6330 CPU 2Sx64Cx128LP @ 2.0GHz, 256GB RAM
create_worker("as-builder-7", properties={'jobs': 64}, max_builds=1),
# Windows Server on Intel Xeon 6330 CPU 2Sx64Cx128LP @ 2.0GHz, 256GB RAM
create_worker("as-builder-8", max_builds=1),
# Solaris 11 # Solaris 11
create_worker("solaris11-amd64", properties={'jobs' : 8}, max_builds=1), create_worker("solaris11-amd64", properties={'jobs' : 8}, max_builds=1),
create_worker("solaris11-sparcv9", properties={'jobs' : 8}, max_builds=1), create_worker("solaris11-sparcv9", properties={'jobs' : 8}, max_builds=1),
# Windows 7 Intel(R) Core(TM) CPU i7-4790K (4.00GHz), 16GB of RAM # CentOS 7.8.1127 on Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz, 96GB RAM
create_worker("windows7-buildbot", properties={'jobs': 2}, max_builds=1), create_worker("hpce-aurora2", properties={'jobs': 8}, max_builds=1),
create_worker("hpce-ve-main", properties={'jobs': 8}, max_builds=1),
# Windows 10, Visual Studio 2019, Google Cloud 16 cores create_worker("hpce-ve-staging", properties={'jobs': 8}, max_builds=1),
create_worker("windows10-vs2019", properties={'jobs': 32}, max_builds=1),
# CentOS 7.5.1804 on Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz, 96GB RAM
create_worker("nec-arrproto41", properties={'jobs': 12}, max_builds=1),
# Ubuntu 18.04 amd64 on Google Cloud, 16 core, Nvidia Tesla T4 # Ubuntu 18.04 amd64 on Google Cloud, 16 core, Nvidia Tesla T4
create_worker("mlir-nvidia", properties={'jobs': 16}, max_builds=1), create_worker("mlir-nvidia", properties={'jobs': 16}, max_builds=1),
# Ubuntu 18.04 pool of VMs on GCP
create_worker("mlir-ubuntu-worker0", max_builds=1),
create_worker("mlir-ubuntu-worker1", max_builds=1),
create_worker("mlir-ubuntu-worker2", max_builds=1),
create_worker("mlir-ubuntu-worker3", max_builds=1),
create_worker("mlir-ubuntu-worker4", max_builds=1),
# Ubuntu 18.04 on Google Cloud, for machine configuration check # Ubuntu 18.04 on Google Cloud, for machine configuration check
# buildbot/google/terraform/main.tf # buildbot/google/terraform/main.tf
create_worker("clangd-ubuntu-clang", max_builds=1), create_worker("clangd-ubuntu-clang", max_builds=1),
# Debian bullseye, build in C++20 configuration, for worker
# configuration check buildbot/google/terraform/main.tf
create_worker("clang-debian-cpp20", max_builds=1),
# Ubuntu 18.04.LTS x86_64, GCE instance # Ubuntu 18.04.LTS x86_64, GCE instance
create_worker("polly-x86_64-gce1", properties={'jobs': 2}, max_builds=1), create_worker("polly-x86_64-gce1", properties={'jobs': 2}, max_builds=1),
create_worker("polly-x86_64-gce2", properties={'jobs': 2}, max_builds=1), create_worker("polly-x86_64-gce2", properties={'jobs': 2}, max_builds=1),
# Ubuntu 18.04.LTS x86_64, Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 32 GiB RAM # Ubuntu 18.04.LTS x86_64, Intel(R) Xeon(R) CPU X3460 @ 2.80GHz, 32 GiB RAM
create_worker("polly-x86_64-fdcserver", properties={'jobs': 8, 'loadaverage': 8}, max_builds=1), create_worker("polly-x86_64-fdcserver", properties={'jobs': 8, 'loadaverage': 8}, max_builds=1),
create_worker("flang-aarch64-ubuntu-build"), # Windows 10, AMD Ryzen 5 PRO 4650G, 16 GiB RAM
create_worker("flang-aarch64-ubuntu-clang-build", properties={'jobs': 14}), create_worker("minipc-ryzen-win", properties={'jobs': 12}, max_builds=1),
create_worker("flang-aarch64-ubuntu-gcc10-build", properties={'jobs': 14}),
create_worker("nersc-flang"), # Ubuntu 20.04.LTS x86_64, Intel(R) Core(TM) i5-9400F CPU @ 2.90Ghz, 16 GiB RAM, NVIDIA GeForce GTX 1050 Ti (Pascal, sm_61, 4GiB)
create_worker("minipc-1050ti-linux", properties={'jobs': 6}, max_builds=1),
create_worker("alcf-theta-flang", properties={'jobs': 12}, max_builds=1), create_worker("alcf-theta-flang", properties={'jobs': 12}, max_builds=1),
# ML-Driven Compiler Optimizations build workers (Ubuntu x86_64) # ML-Driven Compiler Optimizations build workers (Ubuntu x86_64)
create_worker("ml-opt-dev-x86-64-b1", create_worker("ml-opt-dev-x86-64-b1",
properties={'jobs': 64}, max_builds=1), properties={'jobs': 64}, max_builds=1,
notify_on_missing='mlcompileropt-buildbot@google.com'),
create_worker("ml-opt-rel-x86-64-b1", create_worker("ml-opt-rel-x86-64-b1",
properties={'jobs': 64}, max_builds=1), properties={'jobs': 64}, max_builds=1,
notify_on_missing='mlcompileropt-buildbot@google.com'),
create_worker("ml-opt-devrel-x86-64-b1", create_worker("ml-opt-devrel-x86-64-b1",
properties={'jobs': 64}, max_builds=1), properties={'jobs': 64}, max_builds=1,
notify_on_missing='mlcompileropt-buildbot@google.com'),
create_worker("ml-opt-dev-x86-64-b2",
properties={'jobs': 64}, max_builds=1,
notify_on_missing='mlcompileropt-buildbot@google.com'),
create_worker("ml-opt-rel-x86-64-b2",
properties={'jobs': 64}, max_builds=1,
notify_on_missing='mlcompileropt-buildbot@google.com'),
create_worker("ml-opt-devrel-x86-64-b2",
properties={'jobs': 64}, max_builds=1,
notify_on_missing='mlcompileropt-buildbot@google.com'),
# Ubuntu x86_64 # Ubuntu x86_64
create_worker("thinlto-x86-64-bot1", properties={'jobs': 64}, max_builds=1), create_worker("thinlto-x86-64-bot1", properties={'jobs': 64}, max_builds=1),
create_worker("thinlto-x86-64-bot2", properties={'jobs': 64}, max_builds=1),
# Ubuntu 20.04 on AWS, x86_64 PS4 target
create_worker("sie-linux-worker", properties={'jobs': 40}, max_builds=1),
# 2012 Mac Mini host, 16GB memory:
# - Ubuntu 18.04 in docker container
create_worker("doug-worker-1a", properties={'jobs': 10}, max_builds=1),
# - Ubuntu 22.04 in docker container
create_worker("doug-worker-1b", properties={'jobs': 10}, max_builds=1),
# Ubuntu 18.04 in docker container on Ryzen 4800U
create_worker("doug-worker-2a", properties={'jobs': 22}, max_builds=1),
# Windows Server 2019 on AWS, x86_64 PS4 target
create_worker("sie-win-worker", properties={'jobs': 64}, max_builds=1),
# XCore target, Ubuntu 20.04 x64 host
create_worker("xcore-ubuntu20-x64", properties={'jobs': 4}, max_builds=1),
# ARC Worker, CentOS 7.9 x86_64 Intel Xeon Platinum 8000 @ 3.6GHz, 32GB RAM
create_worker("arc-worker", properties={'jobs': 16}, max_builds=1),
# OpenMP on AMDGPU, Ubuntu 18.04.5, Intel(R) Xeon(R) Gold 5218 @ 2.30GHz with 64GB Memory, 1 Vega20 GPU with 16GB Memory
create_worker("omp-vega20-0", properties={'jobs': 32}, max_builds=1),
create_worker("omp-vega20-1", properties={'jobs': 32}, max_builds=1),
# AMD ROCm support, Ubuntu 18.04.6, AMD Ryzen @ 1.5 GHz, MI200 GPU
create_worker("mi200-buildbot", max_builds=1),
# BOLT workers
create_worker("bolt-worker", max_builds=1),
create_worker("bolt-worker-aarch64", max_builds=1),
# Fedora worker
create_worker("standalone-build-x86_64", max_builds=1),
# CSKY T-HEAD workers
create_worker("thead-clang-csky", properties={'jobs' : 32}, max_builds=1),
# RISC-V workers
create_worker("rv64gc-qemu-user", properties={'jobs' : 32}, max_builds=1),
# Google Cloud workers.
create_worker("gc-builder-5", properties={'jobs' : 112}, max_builds=1),
create_worker("gc-builder-6-win", properties={'jobs' : 112}, max_builds=1),
create_worker("gc-builder-7-win", properties={'jobs' : 112}, max_builds=1),
] ]

buildbot/osuosl/master/master.cfg

Show First 20 Lines • Show All 59 Lines • ▼ Show 20 Lines
####### BUILDERS ####### BUILDERS
from buildbot.plugins import util from buildbot.plugins import util
c['builders'] = builders = [ c['builders'] = builders = [
util.BuilderConfig(**b) for b in config.builders.all util.BuilderConfig(**b) for b in config.builders.all
] ]
for rb in config.release_builders.all:
# Make sure a release byuilder has the "release" tag.
tags = rb.get('tags', [])
if 'release' not in tags:
rb['tags'] = tags + ['release']
builders.append(util.BuilderConfig(**rb))
####### SCHEDULERS ####### SCHEDULERS
c['schedulers'] = config.schedulers.getSingleBranchSchedulers( c['schedulers'] = config.schedulers.getMainBranchSchedulers(
c['builders']) builders)
c['schedulers'].extend(config.schedulers.getLntSchedulers())
c['schedulers'].extend(config.schedulers.getReleaseBranchSchedulers(
builders,
treeStableTimer=5*60))
c['schedulers'].extend(config.schedulers.getForceSchedulers( c['schedulers'].extend(config.schedulers.getForceSchedulers(
c['builders'])) builders))
####### BUILDBOT SERVICES ####### BUILDBOT SERVICES
# 'services' is a list of BuildbotService items like reporter targets. The # 'services' is a list of BuildbotService items like reporter targets. The
# status of each build will be pushed to these targets. buildbot/reporters/*.py # status of each build will be pushed to these targets. buildbot/reporters/*.py
# has a variety to choose from, like IRC bots. # has a variety to choose from, like IRC bots.
# Configure MailNotifier, IRC, buildbot.reporters.github.GitHubStatusPush, buildbot.reporters.github.GitHubCommentPush # Configure MailNotifier, IRC, buildbot.reporters.github.GitHubStatusPush, buildbot.reporters.github.GitHubCommentPush
Show All 40 Lines

llvmbisect/llvmlab/llvmlab.py

Show All 40 LinesBUILD_NAME_REGEX = re.compile(
r"-([A-Z][A-Za-z]+)(\.(.*))?") r"-([A-Z][A-Za-z]+)(\.(.*))?")
class Build(object): class Build(object):
@staticmethod @staticmethod
def frombasename(str, url=None): def frombasename(str, url=None):
str = os.path.basename(str) str = os.path.basename(str)
revision = timestamp = build = None revision = revision_prefix = sha = timestamp = build = None
# Check if this is a BNI style build. # Check if this is a BNI style build.
m = BUILD_NAME_REGEX.match(str) m = BUILD_NAME_REGEX.match(str)
if m: if m:
name, _, major_str, _, minor_str, _, micro_str, build, \ name, _, major_str, _, minor_str, _, micro_str, build, \
_, suffix = m.groups() _, suffix = m.groups()
revision = [int(major_str)] revision = [int(major_str)]
if minor_str: if minor_str:
revision.append(int(minor_str)) revision.append(int(minor_str))
if micro_str: if micro_str:
revision.append(int(micro_str)) revision.append(int(micro_str))
return Build(name, tuple(revision), None, build, suffix) return Build(name, None, tuple(revision), None, None, build, suffix)
if '.' in str: if '.' in str:
str, suffix = str.split('.', 1) str, suffix = str.split('.', 1)
else: else:
suffix = None suffix = None
m = re.match(r'(.*)-b([0-9]+)', str) m = re.match(r'(.*)-b([0-9]+)', str)
if m: if m:
str, build = m.groups() str, build = m.groups()
build = int(build) build = int(build)
m = re.match(r'(.*)-t([0-9-]{8,10}_[0-9-]{6,8})', str) m = re.match(r'(.*)-t([0-9-]{8,10}_[0-9-]{6,8})', str)
if m: if m:
str, timestamp = m.groups() str, timestamp = m.groups()
m = re.match(r'(.*)-d([0-9]+)-(.*)', str)
if m:
str, revision, sha = m.groups()
revision = int(revision)
revision_prefix = 'd'
m = re.match(r'(.*)-r([0-9]+)', str) m = re.match(r'(.*)-r([0-9]+)', str)
if m: if m:
str, revision = m.groups() str, revision = m.groups()
revision = int(revision) revision = int(revision)
revision_prefix = 'r'
return Build(str, revision, timestamp, build, suffix, url) return Build(str, revision_prefix, revision, sha, timestamp,
build, suffix, url)
@staticmethod @staticmethod
def fromdata(data): def fromdata(data):
return Build(data['name'], data['revision'], data['timestamp'], return Build(data['name'], data['revision'], data['timestamp'],
data['build'], data['suffix']) data['build'], data['suffix'])
def todata(self): def todata(self):
return {'name': self.name, return {'name': self.name,
'revision': self.revision, 'revision': self.revision,
'sha': self.sha,
'timestamp': self.timestamp, 'timestamp': self.timestamp,
'build': self.build, 'build': self.build,
'suffix': self.suffix} 'suffix': self.suffix}
def __init__(self, name, revision, timestamp, build, suffix, url): def __init__(self, name, revision_prefix, revision, sha, timestamp,
build, suffix, url):
self.name = name self.name = name
self.revision_prefix = revision_prefix
self.revision = revision self.revision = revision
self.sha = sha
self.timestamp = timestamp self.timestamp = timestamp
self.build = build self.build = build
self.suffix = suffix self.suffix = suffix
self.url = url self.url = url
def tobasename(self, include_suffix=True): def tobasename(self, include_suffix=True):
basename = self.name basename = self.name
if self.revision is not None: if self.revision is not None:
if isinstance(self.revision, (tuple, list)): if isinstance(self.revision, (tuple, list)):
basename += '-' + '.'.join(str(r) for r in self.revision) basename += '-' + '.'.join(str(r) for r in self.revision)
else: else:
assert isinstance(self.revision, int) assert isinstance(self.revision, int)
basename += '-r%d' % self.revision basename += '-%s%d' % (self.revision_prefix, self.revision)
if self.sha is not None:
basename += '-%s' % self.sha
if self.timestamp is not None: if self.timestamp is not None:
basename += '-t%s' % self.timestamp basename += '-t%s' % self.timestamp
if self.build is not None: if self.build is not None:
if isinstance(self.build, str): if isinstance(self.build, str):
basename += '-' + self.build basename += '-' + self.build
else: else:
basename += '-b%d' % self.build basename += '-b%d' % self.build
if include_suffix and self.suffix is not None: if include_suffix and self.suffix is not None:
basename += '.%s' % self.suffix basename += '.%s' % self.suffix
return basename return basename
def __repr__(self): def __repr__(self):
return "%s%r" % (self.__class__.__name__, return "%s%r" % (self.__class__.__name__,
(self.name, self.revision, self.timestamp, self.build, (self.name, self.revision, self.sha, self.timestamp,
self.suffix)) self.build, self.suffix))
def __cmp__(self, other): def __cmp__(self, other):
return cmp((self.revision, self.timestamp, return cmp((self.revision, self.timestamp,
self.build, self.suffix, self.name), self.build, self.suffix, self.name),
((other.revision, other.timestamp, ((other.revision, other.timestamp,
other.build, other.suffix, other.name))) other.build, other.suffix, other.name)))
▲ Show 20 Lines • Show All 138 Lines • Show Last 20 Lines

requirements.txt

attrs==20.2.0 attrs==20.3.0
six==1.15.0 six==1.15.0
Automat==20.2.0 Automat==20.2.0
constantly==15.1.0 constantly==15.1.0
idna==2.10 idna==2.10
hyperlink==20.0.1 hyperlink==21.0.0
incremental==17.5.0 incremental==21.3.0
Jinja2==2.11.2 Jinja2==2.11.3
PyHamcrest==2.0.2
Tempita==0.5.2 Tempita==0.5.2
decorator==4.4.2 decorator==4.4.2
pbr==5.5.1 pbr==5.5.1
sqlparse==0.4.1 sqlparse==0.4.2
SQLAlchemy==1.3.20 SQLAlchemy==1.3.23
sqlalchemy-migrate==0.13.0 sqlalchemy-migrate==0.13.0
certifi==2020.6.20 certifi==2022.12.7
chardet==3.0.4 chardet==4.0.0
urllib3==1.25.11 urllib3==1.26.8
requests==2.24.0 requests==2.25.1
zope.interface==5.1.2 zope.interface==5.2.0
pycparser==2.20 pycparser==2.20
pyOpenSSL==19.1.0 pyOpenSSL==20.0.1
pyasn1==0.4.8 pyasn1==0.4.8
pyasn1-modules==0.2.8 pyasn1-modules==0.2.8
service-identity==18.1.0 service-identity==18.1.0
cryptography==3.2.1 cryptography==3.4.5
cffi==1.14.3 cffi==1.14.5
Twisted==20.3.0 Twisted==22.4.0
treq==20.9.0 treq==22.1.0

tasks/cmake/caches/util/xcode_sdk.cmake

Show First 20 Lines • Show All 45 Lines • ▼ Show 20 LinesDYLD_LIBRARY_PATH=\"${COMPILER_DIR}/../lib:$DYLD_LIBRARY_PATH\" ${SDK_TOOL_BIN} \"$@\"
set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY CACHE STRING "") set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY CACHE STRING "")
# Some benchmarks check for the function re_comp by using # Some benchmarks check for the function re_comp by using
# check_function_exists. This function is not available in any Xcode SDK, and # check_function_exists. This function is not available in any Xcode SDK, and
# the check returns YES with custom built compilers when we cross-compile, # the check returns YES with custom built compilers when we cross-compile,
# because of "CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY" explained just # because of "CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY" explained just
# above. We work around that by forcing it to NO. # above. We work around that by forcing it to NO.
set(HAVE_RE_COMP CACHE BOOL NO) set(HAVE_RE_COMP CACHE BOOL NO)
# Likewise, some benchmarks look for librt, which does not exist on darwin
# systems, but the check returns YES because we don't actually link, given
# the CMAKE_TRY_COMPILE_TARGET_TYPE setting above. We work around that by
# forcing it to NO.
set(HAVE_LIB_RT CACHE BOOL NO)
else() else()
# Search and use compiler coming with the SDK. # Search and use compiler coming with the SDK.
# Note that we do not search CMAKE_CXX_COMPILER here. cmake will try # Note that we do not search CMAKE_CXX_COMPILER here. cmake will try
# `${CMAKE_C_COMPILER}++` and use that if available. # `${CMAKE_C_COMPILER}++` and use that if available.
xcrun_find_update_cache(CMAKE_AR ar) xcrun_find_update_cache(CMAKE_AR ar)
xcrun_find_update_cache(CMAKE_NM nm) xcrun_find_update_cache(CMAKE_NM nm)
xcrun_find_update_cache(CMAKE_C_COMPILER clang) xcrun_find_update_cache(CMAKE_C_COMPILER clang)
xcrun_find_update_cache(CMAKE_RANLIB ranlib) xcrun_find_update_cache(CMAKE_RANLIB ranlib)
Show All 22 Lines

tasks/tasktool/README.md

Show First 20 Lines • Show All 86 Lines • ▼ Show 20 Lines
This is usually done by querying the repository definitions in the `repos` This is usually done by querying the repository definitions in the `repos`
directory (and `repos.try` in some modes). For example for an artifact directory (and `repos.try` in some modes). For example for an artifact
parameter called `llvm` we could have a `repos/llvm.json` configuration file parameter called `llvm` we could have a `repos/llvm.json` configuration file
to query the latest revision of a git repository: to query the latest revision of a git repository:
```json ```json
{ {
"url": "https://github.com/my-project.git", "url": "https://github.com/my-project.git",
"type": "git", "type": "git",
"default_rev": "refs/heads/master" "default_rev": "refs/heads/main"
} }
``` ```
To query the latest artifact from a directory in the A2 artifact server: To query the latest artifact from a directory in the A2 artifact server:
```json ```json
{ {
"url": "http://lab.llvm.org:8080/artifacts/clang-stage1-configure-RA", "url": "http://lab.llvm.org:8080/artifacts/clang-stage1-configure-RA",
"type": "artifact_server" "type": "artifact_server"
▲ Show 20 Lines • Show All 57 Lines • ▼ Show 20 Lines
VARIABLE="$(build get arg --optional PARAMETER_NAME)" VARIABLE="$(build get arg --optional PARAMETER_NAME)"
``` ```
Other default values can be supplied with the usual shell tricks: Other default values can be supplied with the usual shell tricks:
```bash ```bash
: ${VARIABLE:='DefaultValue'} : ${VARIABLE:='DefaultValue'}
``` ```
- Abstract artifacts like `refs/heads/master` in a git repository or - Abstract artifacts like `refs/heads/main` in a git repository or
`clang-stage2-cmake` on an A2 server are resolved to a concrete git revision `clang-stage2-cmake` on an A2 server are resolved to a concrete git revision
or a specific URL when the task is started. or a specific URL when the task is started.
- A task with a set of artifact and parameter inputs is called a build. - A task with a set of artifact and parameter inputs is called a build.
- `task try mytask.sh` resolves artifact inputs and executes a local build. - `task try mytask.sh` resolves artifact inputs and executes a local build.
- `task submit mytask.sh` resolves artifact inputs and submits a build to - `task submit mytask.sh` resolves artifact inputs and submits a build to
a queue on a jenkins server. a queue on a jenkins server.
- `task jenkinsrun mytask.sh` may be used to run the task as part of a - `task jenkinsrun mytask.sh` may be used to run the task as part of a
recurring jenking project. recurring jenking project.
### Config Artifact ### ### Config Artifact ###
The task files are part of a version control repository. When running a task The task files are part of a version control repository. When running a task
this repository is always checked out to a subdirectory named `config`. This this repository is always checked out to a subdirectory named `config`. This
allows task files to bundle and reference additional files in the repository allows task files to bundle and reference additional files in the repository
such as helper scripts shared between multiple tasks. such as helper scripts shared between multiple tasks.

tasks/tasktool/tasktool/repos/git.py

Show All 19 Lines if config.get('url') is None:
raise Exception("No 'url' specified") raise Exception("No 'url' specified")
def resolve_latest(config): def resolve_latest(config):
assert(config['type'] == 'git') assert(config['type'] == 'git')
rev = config.get('rev') rev = config.get('rev')
default_rev = config.pop('default_rev', None) default_rev = config.pop('default_rev', None)
if default_rev is None: if default_rev is None:
default_rev = "refs/heads/master" default_rev = "refs/heads/main"
if rev is not None: if rev is not None:
return return
url = config['url'] url = config['url']
revs = _git_ls_remote(url, default_rev) revs = _git_ls_remote(url, default_rev)
# Look for an exact match # Look for an exact match
if default_rev is not None: if default_rev is not None:
rev = revs.get(default_rev, None) rev = revs.get(default_rev, None)
# Otherwise hope that we only have a single match. # Otherwise hope that we only have a single match.
Show All 25 Lines

tasks/utils/venv.sh

echo "@@@ LNT VirtualEnv @@@" echo "@@@ LNT VirtualEnv @@@"
/usr/local/bin/virtualenv venv /usr/local/bin/virtualenv --python=python3 venv
set +u set +u
. venv/bin/activate . venv/bin/activate
set -u set -u
echo "@@@@@@@" echo "@@@@@@@"

test/jenkins/test_monorepo_build.py

# Testing for the Jenkins monorepo_build.py script # Testing for the Jenkins monorepo_build.py script
# #
# RUN: export TESTING=1 # RUN: export TESTING=1
# RUN: export JOB_NAME="FOO" # RUN: export JOB_NAME="FOO"
# RUN: export BUILD_NUMBER=321 # RUN: export BUILD_NUMBER=321
# RUN: export BRANCH=master # RUN: export BRANCH=main
# Tell monorepo_build.py to just print commands instead of running. # Tell monorepo_build.py to just print commands instead of running.
# RUN: mkdir -p %t.SANDBOX/host-compiler/lib %t.SANDBOX/host-compiler/bin %t.SANDBOX/llvm-project/llvm %t.SANDBOX/llvm-project/clang %t.SANDBOX/llvm-project/libcxx %t.SANDBOX/llvm-project/compiler-rt %t.SANDBOX/llvm-project/debuginfo-tests %t.SANDBOX/llvm-project/clang-tools-extra %t.SANDBOX/llvm-project/lldb # RUN: mkdir -p %t.SANDBOX/host-compiler/lib %t.SANDBOX/host-compiler/bin %t.SANDBOX/llvm-project/llvm %t.SANDBOX/llvm-project/clang %t.SANDBOX/llvm-project/compiler-rt %t.SANDBOX/llvm-project/debuginfo-tests %t.SANDBOX/llvm-project/clang-tools-extra %t.SANDBOX/llvm-project/lldb
# RUN: touch %t.SANDBOX/host-compiler/bin/clang # RUN: touch %t.SANDBOX/host-compiler/bin/clang
# RUN: python %{src_root}/zorg/jenkins/monorepo_build.py clang all > %t.log # RUN: python %{src_root}/zorg/jenkins/monorepo_build.py clang all > %t.log
# RUN: FileCheck --check-prefix CHECK-SIMPLE < %t.log %s # RUN: FileCheck --check-prefix CHECK-SIMPLE < %t.log %s
# CHECK-SIMPLE: @@@ Build Directory @@@ # CHECK-SIMPLE: @@@ Build Directory @@@
# CHECK-SIMPLE: cd # CHECK-SIMPLE: cd
# CHECK-SIMPLE: 'mkdir' '-p' # CHECK-SIMPLE: 'mkdir' '-p'
# CHECK-SIMPLE: @@@@@@ # CHECK-SIMPLE: @@@@@@
# CHECK-SIMPLE: @@@ Build Clang @@@ # CHECK-SIMPLE: @@@ Build Clang @@@
# CHECK-SIMPLE: cd # CHECK-SIMPLE: cd
# CHECK-SIMPLE: 'mkdir' './Build' './Root' # CHECK-SIMPLE: 'mkdir' './Build' './Root'
# CHECK-SIMPLE: cd # CHECK-SIMPLE: cd
# CHECK-SIMPLE: '/usr/local/bin/cmake' '-G' 'Ninja' '-C' # CHECK-SIMPLE: '/usr/local/bin/cmake' '-G' 'Ninja' '-C'
# CHECK-SIMPLE: '-DLLVM_ENABLE_ASSERTIONS:BOOL=FALSE' # CHECK-SIMPLE: '-DLLVM_ENABLE_ASSERTIONS:BOOL=FALSE'
# CHECK-SIMPLE: '-DCMAKE_BUILD_TYPE=RelWithDebInfo' # CHECK-SIMPLE: '-DCMAKE_BUILD_TYPE=RelWithDebInfo'
# CHECK-SIMPLE: '-DLLVM_ENABLE_PROJECTS=clang;clang-tools-extra;compiler-rt;libcxx' # CHECK-SIMPLE: '-DLLVM_ENABLE_PROJECTS=clang;clang-tools-extra;compiler-rt'
# CHECK-SIMPLE: '-DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja' # CHECK-SIMPLE: '-DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja'
# CHECK-SIMPLE: '-DLLVM_VERSION_PATCH=99' # CHECK-SIMPLE: '-DLLVM_VERSION_PATCH=99'
# CHECK-SIMPLE: '-DLLVM_VERSION_SUFFIX=""' # CHECK-SIMPLE: '-DLLVM_VERSION_SUFFIX=""'
# CHECK-SIMPLE: '-DLLVM_BUILD_EXTERNAL_COMPILER_RT=On' # CHECK-SIMPLE: '-DLLVM_BUILD_EXTERNAL_COMPILER_RT=On'
# CHECK-SIMPLE: '-DCLANG_COMPILER_RT_CMAKE_ARGS # CHECK-SIMPLE: '-DCLANG_COMPILER_RT_CMAKE_ARGS
# CHECK-SIMPLE: Apple.cmake' # CHECK-SIMPLE: Apple.cmake'
# CHECK-SIMPLE: '-DCOMPILER_RT_BUILD_SANITIZERS=On' # CHECK-SIMPLE: '-DCOMPILER_RT_BUILD_SANITIZERS=On'
# CHECK-SIMPLE: '-DCMAKE_INSTALL_PREFIX # CHECK-SIMPLE: '-DCMAKE_INSTALL_PREFIX
▲ Show 20 Lines • Show All 109 Lines • ▼ Show 20 Lines
# Test to check if timeout flag is actually being set. # Test to check if timeout flag is actually being set.
# RUN: python %{src_root}/zorg/jenkins/monorepo_build.py cmake all --timeout=900 > %t-timeout.log # RUN: python %{src_root}/zorg/jenkins/monorepo_build.py cmake all --timeout=900 > %t-timeout.log
# RUN: FileCheck --check-prefix CHECK-TIMEOUT < %t-timeout.log %s # RUN: FileCheck --check-prefix CHECK-TIMEOUT < %t-timeout.log %s
# CHECK-TIMEOUT: --timeout=900 # CHECK-TIMEOUT: --timeout=900
# Test to check if default timeout is being set to 600. # Test to check if default timeout is being set to 600.
# RUN: python %{src_root}/zorg/jenkins/monorepo_build.py cmake all > %t-timeout-default.log # RUN: python %{src_root}/zorg/jenkins/monorepo_build.py cmake all > %t-timeout-default.log
# RUN: FileCheck --check-prefix CHECK-TIMEOUT-DEFAULT < %t-timeout-default.log %s # RUN: FileCheck --check-prefix CHECK-TIMEOUT-DEFAULT < %t-timeout-default.log %s
# CHECK-TIMEOUT-DEFAULT: --timeout=600 # CHECK-TIMEOUT-DEFAULT: --timeout=600
No newline at end of file
# RUN: python %{src_root}/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
# RUN: --lldb-test-compiler="MY_LLDB_TEST_COMPILER" \
# RUN: --cmake-flag=-DLLVM_TARGETS_TO_BUILD=X86 > %t-lldb-configure.log
# RUN: FileCheck --check-prefix CHECK-LLDB-CONFIG < %t-lldb-configure.log %s
# CHECK-LLDB-CONFIG: -DLLDB_TEST_COMPILER=MY_LLDB_TEST_COMPILER
# CHECK-LLDB-CONFIG: -DLLVM_TARGETS_TO_BUILD=X86
# there may be other "target to build", but the one above should be the last:
# CHECK-LLDB-CONFIG-NOT: -DLLVM_TARGETS_TO_BUILD

zorg/buildbot/builders/AOSPBuilder.py

Show All 19 Lines
def getAOSPBuildCommand( def getAOSPBuildCommand(
device, # Target device for AOSP build device, # Target device for AOSP build
timeout=None, # Maximum CPU time in seconds, umlimited if 'None' timeout=None, # Maximum CPU time in seconds, umlimited if 'None'
target_clang=None, # Path to the Clang used for AOSP target build target_clang=None, # Path to the Clang used for AOSP target build
# Set target_clang to None to use AOSP's default Clang # Set target_clang to None to use AOSP's default Clang
target_flags=None, # Extra C/CXX flags for AOSP target build target_flags=None, # Extra C/CXX flags for AOSP target build
jobs=None, # Number of concurrent jobs jobs=None, # Number of concurrent jobs
extra_make_args=None): # Extra args for the make command extra_make_args=None): # Extra args for the make command
command = "source build/envsetup.sh" command = "/bin/bash -c \"source build/envsetup.sh"
command += " && lunch aosp_%s-userdebug" % device command += " && lunch aosp_%s-userdebug" % device
command += " && export LC_ALL=C"
command += " && make -k" command += " && make -k"
if timeout: if timeout:
command += " TIMEOUT=%s" % timeout command += " TIMEOUT=%s" % timeout
if target_clang: if target_clang:
command += " TARGET_CLANG=%s" % target_clang command += " TARGET_CLANG=%s" % target_clang
if target_flags: if target_flags:
command += " TARGET_FLAGS='%s'" % target_flags command += " TARGET_FLAGS='%s'" % target_flags
if extra_make_args: if extra_make_args:
command += " " + extra_make_args command += " " + extra_make_args
if jobs: if jobs:
command += " -j" + str(jobs) command += " -j" + str(jobs)
command += "\""
return command return command
def getAOSPBuildFactory( def getAOSPBuildFactory(
device, # Target device for AOSP build device, # Target device for AOSP build
extra_cmake_args=[], # Extra args for the LLVM cmake command extra_cmake_args=[], # Extra args for the LLVM cmake command
# This flag is ignored if build_clang is False # This flag is ignored if build_clang is False
timeout=None, # Maximum CPU time in seconds, umlimited if 'None' timeout=None, # Maximum CPU time in seconds, umlimited if 'None'
▲ Show 20 Lines • Show All 72 Lines • Show Last 20 Lines

zorg/buildbot/builders/AnnotatedBuilder.py

from buildbot.process.properties import WithProperties from buildbot.process.properties import WithProperties
from buildbot.steps.shell import SetProperty from buildbot.steps.shell import SetProperty
from zorg.buildbot.commands.AnnotatedCommand import AnnotatedCommand from zorg.buildbot.commands.AnnotatedCommand import AnnotatedCommand
from zorg.buildbot.process.factory import LLVMBuildFactory from zorg.buildbot.process.factory import LLVMBuildFactory
def getAnnotatedBuildFactory( def getAnnotatedBuildFactory(
script, script,
clean=False, clean=False,
depends_on_projects=None, depends_on_projects=None,
env=None, env=None,
extra_args=None, extra_args=None,
timeout=1200, timeout=1200,
checkout_llvm_sources=True): checkout_llvm_sources=True,
script_interpreter="python",
warnOnWarnings=False):
""" """
Returns a new build factory that uses AnnotatedCommand, which Returns a new build factory that uses AnnotatedCommand, which
allows the build to be run by version-controlled scripts that do allows the build to be run by version-controlled scripts that do
not require a buildmaster restart to update. not require a buildmaster restart to update.
script: script under "builders/annotated" to be run by python script: script under "builders/annotated" to be run by python
clean: set to true for a clean build of llvm clean: set to true for a clean build of llvm
depends_on_projects: which subprojects to enable depends_on_projects: which subprojects to enable
llvm must be first in the list llvm must be first in the list
(default: ["llvm", "clang", "compiler-rt", "libcxx", (default: ["llvm", "clang", "compiler-rt", "libcxx",
"libcxxabi", "libunwind", "lld"]) "libcxxabi", "libunwind", "lld"])
env: environment overrides (map; default is no overrides) env: environment overrides (map; default is no overrides)
extra_args: extra arguments to pass to the script (default: []) extra_args: extra arguments to pass to the script (default: [])
timeout: specifies the builder's timeout in seconds (default: 1200) timeout: specifies the builder's timeout in seconds (default: 1200)
script_interpreter: specifies the interpreter to run scripts (default: "python")
""" """
if depends_on_projects is None: if depends_on_projects is None:
depends_on_projects = [ depends_on_projects = [
"llvm", "llvm",
"clang", "clang",
"compiler-rt", "compiler-rt",
"libcxx", "libcxx",
"libcxxabi", "libcxxabi",
"libunwind", "libunwind",
"lld"] "lld"]
if extra_args is None: if extra_args is None:
extra_args = [] # We used to add --jobs to all script invocations. Perserve this
# for cases when the user did not specify extra_args, but allow
# overriding it if the user did specify extra_args.
extra_args = ["--jobs=%(jobs:-)s"]
f = LLVMBuildFactory( f = LLVMBuildFactory(
clean=clean, clean=clean,
depends_on_projects=depends_on_projects) depends_on_projects=depends_on_projects)
if clean: if clean:
f.addStep(SetProperty(property='clean', command='echo 1')) f.addStep(SetProperty(property='clean', command='echo 1'))
Show All 29 Lines if checkout_llvm_sources:
f.addGetSourcecodeSteps() f.addGetSourcecodeSteps()
extra_args_with_props = [WithProperties(arg) for arg in extra_args] extra_args_with_props = [WithProperties(arg) for arg in extra_args]
# Explicitly use '/' as separator, because it works on *nix and Windows. # Explicitly use '/' as separator, because it works on *nix and Windows.
if script.startswith('/'): if script.startswith('/'):
command = [script] command = [script]
else: else:
script_path = "../llvm-zorg/zorg/buildbot/builders/annotated/%s" % (script) script_path = "../llvm-zorg/zorg/buildbot/builders/annotated/%s" % (script)
command = ["python", script_path, WithProperties("--jobs=%(jobs:-)s")] # Handle scripts with script_interpreter, otherwise execute the script directly.
if script_interpreter:
command = [script_interpreter, script_path]
else:
command = [script_path]
command += extra_args_with_props command += extra_args_with_props
f.addStep(AnnotatedCommand(name="annotate", f.addStep(AnnotatedCommand(name="annotate",
description="annotate", description="annotate",
timeout=timeout, timeout=timeout,
haltOnFailure=True, haltOnFailure=True,
warnOnWarnings=warnOnWarnings,
command=command, command=command,
env=merged_env)) env=merged_env))
return f return f

zorg/buildbot/builders/BOLTBuilder.py

  • This file was added.
from buildbot.plugins import steps
from buildbot.steps.shell import ShellCommand
from zorg.buildbot.builders.UnifiedTreeBuilder import getLLVMBuildFactoryAndSourcecodeSteps, addCmakeSteps, addNinjaSteps
from zorg.buildbot.commands.LitTestCommand import LitTestCommand
from zorg.buildbot.commands.CmakeCommand import CmakeCommand
from zorg.buildbot.conditions.FileConditions import FileExists
from zorg.buildbot.process.factory import LLVMBuildFactory
def getBOLTCmakeBuildFactory(
clean = False,
bolttests = False,
is_nfc = False,
targets = None,
checks = None,
caches = None,
extra_configure_args = None,
env = None,
depends_on_projects = None,
**kwargs):
if env is None:
env = dict()
bolttests_dir = "bolt-tests"
cleanBuildRequested = lambda step: clean or step.build.getProperty("clean", default=step.build.getProperty("clean_obj"))
cleanBuildRequestedByProperty = lambda step: step.build.getProperty("clean")
if not targets:
targets = ['bolt']
if not checks:
checks = ['check-bolt']
f = getLLVMBuildFactoryAndSourcecodeSteps(
depends_on_projects=depends_on_projects,
**kwargs) # Pass through all the extra arguments.
if bolttests:
checks += ['check-large-bolt']
extra_configure_args += [
'-DLLVM_EXTERNAL_PROJECTS=bolttests',
'-DLLVM_EXTERNAL_BOLTTESTS_SOURCE_DIR=' + LLVMBuildFactory.pathRelativeTo(bolttests_dir, f.monorepo_dir),
]
# Clean checkout of bolt-tests if cleanBuildRequested
f.addSteps([
steps.RemoveDirectory(name="BOLT tests: clean",
dir=bolttests_dir,
haltOnFailure=True,
warnOnFailure=True,
doStepIf=cleanBuildRequestedByProperty),
steps.Git(name="BOLT tests: checkout",
description="fetching",
descriptionDone="fetch",
descriptionSuffix="BOLT Tests",
repourl='https://github.com/rafaelauler/bolt-tests.git',
workdir=bolttests_dir,
alwaysUseLatest=True),
])
# Some options are required for this build no matter what.
CmakeCommand.applyRequiredOptions(extra_configure_args, [
('-G', 'Ninja'),
])
if caches:
for cache in caches:
extra_configure_args += [f"-C../{f.monorepo_dir}/{cache}"]
addCmakeSteps(
f,
cleanBuildRequested=cleanBuildRequested,
extra_configure_args=extra_configure_args,
obj_dir=None,
env=env,
**kwargs)
addNinjaSteps(
f,
targets=targets,
checks=checks,
env=env,
**kwargs)
if is_nfc:
f.addSteps([
ShellCommand(
name='nfc-check-setup',
command=[f"../{f.monorepo_dir}/bolt/utils/nfc-check-setup.py"],
description=('Setup NFC testing'),
warnOnFailure=True,
haltOnFailure=False,
flunkOnFailure=False,
env=env),
ShellCommand(
name='check-bolt-different',
command='rm -f .llvm-bolt.diff; cmp -s bin/llvm-bolt.old bin/llvm-bolt.new || touch .llvm-bolt.diff',
description=('Check if llvm-bolt binaries are different and '
'skip the following nfc-check steps'),
haltOnFailure=False,
env=env),
LitTestCommand(
name='nfc-check-bolt',
command=['bin/llvm-lit', '-sv', '-j4',
# bolt-info will always mismatch in NFC mode
'--xfail=bolt-info.test',
'tools/bolt/test'],
description=["running", "NFC", "check-bolt"],
descriptionDone=["NFC", "check-bolt", "completed"],
warnOnFailure=True,
haltOnFailure=False,
flunkOnFailure=False,
doStepIf=FileExists('build/.llvm-bolt.diff'),
env=env),
LitTestCommand(
name='nfc-check-large-bolt',
command=['bin/llvm-lit', '-sv', '-j2',
'tools/bolttests'],
description=["running", "NFC", "check-large-bolt"],
descriptionDone=["NFC", "check-large-bolt", "completed"],
warnOnFailure=True,
haltOnFailure=False,
flunkOnFailure=False,
doStepIf=FileExists('build/.llvm-bolt.diff'),
env=env),
])
return f

zorg/buildbot/builders/ClangBuilder.py

import copy import copy
from datetime import datetime from datetime import datetime
from buildbot.plugins import util
from buildbot.process.properties import WithProperties, Property from buildbot.process.properties import WithProperties, Property
from buildbot.steps.shell import ShellCommand, SetProperty from buildbot.steps.shell import ShellCommand, SetProperty
from buildbot.steps.shell import WarningCountingShellCommand from buildbot.steps.shell import WarningCountingShellCommand
import zorg.buildbot.builders.Util as builders_util import zorg.buildbot.builders.Util as builders_util
from zorg.buildbot.commands.LitTestCommand import LitTestCommand from zorg.buildbot.commands.LitTestCommand import LitTestCommand
from zorg.buildbot.conditions.FileConditions import FileDoesNotExist from zorg.buildbot.conditions.FileConditions import FileDoesNotExist
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Lines f.addStep(ShellCommand(
command=['gsutil', 'cp', '../install.tar.xz', gcs_url], command=['gsutil', 'cp', '../install.tar.xz', gcs_url],
description=('uploading ' + package_name + description=('uploading ' + package_name +
'to storage bucket ...'), 'to storage bucket ...'),
workdir=install_prefix, workdir=install_prefix,
env=env)) env=env))
def getClangCMakeGCSBuildFactory( def getClangCMakeGCSBuildFactory(
clean=True, clean=True,
test=True, checks=None,
cmake='cmake', cmake='cmake',
jobs=None, jobs=None,
# VS tools environment variable if using MSVC. For example, # VS tools environment variable if using MSVC. For example,
# %VS120COMNTOOLS% selects the 2013 toolchain. # %VS120COMNTOOLS% selects the 2013 toolchain.
vs=None, vs=None,
vs_target_arch='x86', vs_target_arch='x86',
Show All 26 Linesdef getClangCMakeGCSBuildFactory(
# Use a lower compression level to generate the build-cache package faster. # Use a lower compression level to generate the build-cache package faster.
# defuault is 6 according to xz documentation # defuault is 6 according to xz documentation
xz_compression_factor=6, xz_compression_factor=6,
use_pixz_compression=False, use_pixz_compression=False,
# Triggers # Triggers
trigger_after_stage1=None): trigger_after_stage1=None):
return _getClangCMakeBuildFactory( return _getClangCMakeBuildFactory(
clean=clean, test=test, cmake=cmake, jobs=jobs, vs=vs, clean=clean, checks=checks, cmake=cmake, jobs=jobs, vs=vs,
vs_target_arch=vs_target_arch, useTwoStage=useTwoStage, vs_target_arch=vs_target_arch, useTwoStage=useTwoStage,
testStage1=testStage1, stage1_config=stage1_config, testStage1=testStage1, stage1_config=stage1_config,
stage2_config=stage2_config, runTestSuite=runTestSuite, stage2_config=stage2_config, runTestSuite=runTestSuite,
nt_flags=nt_flags, testsuite_flags=testsuite_flags, nt_flags=nt_flags, testsuite_flags=testsuite_flags,
submitURL=submitURL, testerName=testerName, submitURL=submitURL, testerName=testerName,
env=env, extra_cmake_args=extra_cmake_args, env=env, extra_cmake_args=extra_cmake_args,
checkout_clang_tools_extra=checkout_clang_tools_extra, checkout_clang_tools_extra=checkout_clang_tools_extra,
checkout_compiler_rt=checkout_compiler_rt, checkout_compiler_rt=checkout_compiler_rt,
checkout_lld=checkout_lld, checkout_lld=checkout_lld,
checkout_libcxx=checkout_libcxx, checkout_libcxx=checkout_libcxx,
stage1_upload_directory=stage1_upload_directory, stage1_upload_directory=stage1_upload_directory,
xz_compression_factor=xz_compression_factor, xz_compression_factor=xz_compression_factor,
use_pixz_compression=use_pixz_compression, use_pixz_compression=use_pixz_compression,
trigger_after_stage1=trigger_after_stage1) trigger_after_stage1=trigger_after_stage1)
def getClangCMakeBuildFactory( def getClangCMakeBuildFactory(
clean=True, clean=True,
test=True, checks=None,
cmake='cmake', cmake='cmake',
jobs=None, jobs=None,
timeout=None, # TODO: Implement support for timeout
# VS tools environment variable if using MSVC. For example, # VS tools environment variable if using MSVC. For example,
# %VS120COMNTOOLS% selects the 2013 toolchain. # %VS120COMNTOOLS% selects the 2013 toolchain.
vs=None, vs=None,
vs_target_arch='x86', vs_target_arch='x86',
# Multi-stage compilation # Multi-stage compilation
useTwoStage=False, useTwoStage=False,
Show All 13 Linesdef getClangCMakeBuildFactory(
extra_cmake_args=None, extra_cmake_args=None,
# Extra repositories # Extra repositories
checkout_clang_tools_extra=True, checkout_clang_tools_extra=True,
checkout_compiler_rt=True, checkout_compiler_rt=True,
checkout_lld=True, checkout_lld=True,
checkout_libcxx=False, checkout_libcxx=False,
checkout_flang=False, checkout_flang=False,
checkout_test_suite=False): checkout_test_suite=False,
enable_runtimes="auto"):
return _getClangCMakeBuildFactory( return _getClangCMakeBuildFactory(
clean=clean, test=test, cmake=cmake, jobs=jobs, vs=vs, clean=clean, checks=checks, cmake=cmake, jobs=jobs, vs=vs,
vs_target_arch=vs_target_arch, useTwoStage=useTwoStage, vs_target_arch=vs_target_arch, useTwoStage=useTwoStage,
testStage1=testStage1, stage1_config=stage1_config, testStage1=testStage1, stage1_config=stage1_config,
stage2_config=stage2_config, runTestSuite=runTestSuite, stage2_config=stage2_config, runTestSuite=runTestSuite,
nt_flags=nt_flags, testsuite_flags=testsuite_flags, nt_flags=nt_flags, testsuite_flags=testsuite_flags,
submitURL=submitURL, testerName=testerName, submitURL=submitURL, testerName=testerName,
env=env, extra_cmake_args=extra_cmake_args, env=env, extra_cmake_args=extra_cmake_args,
checkout_clang_tools_extra=checkout_clang_tools_extra, checkout_clang_tools_extra=checkout_clang_tools_extra,
checkout_lld=checkout_lld, checkout_lld=checkout_lld,
checkout_compiler_rt=checkout_compiler_rt, checkout_compiler_rt=checkout_compiler_rt,
checkout_libcxx=checkout_libcxx, checkout_libcxx=checkout_libcxx,
checkout_flang=checkout_flang, checkout_flang=checkout_flang,
checkout_test_suite=checkout_test_suite) checkout_test_suite=checkout_test_suite,
enable_runtimes=enable_runtimes)
def _getClangCMakeBuildFactory( def _getClangCMakeBuildFactory(
clean=True, clean=True,
test=True, checks=None,
cmake='cmake', cmake='cmake',
jobs=None, jobs=None,
# VS tools environment variable if using MSVC. For example, # VS tools environment variable if using MSVC. For example,
# %VS120COMNTOOLS% selects the 2013 toolchain. # %VS120COMNTOOLS% selects the 2013 toolchain.
vs=None, vs=None,
vs_target_arch='x86', vs_target_arch='x86',
Show All 17 Linesdef _getClangCMakeBuildFactory(
# Extra repositories # Extra repositories
checkout_clang_tools_extra=True, checkout_clang_tools_extra=True,
checkout_compiler_rt=True, checkout_compiler_rt=True,
checkout_lld=True, checkout_lld=True,
checkout_libcxx=False, checkout_libcxx=False,
checkout_test_suite=False, checkout_test_suite=False,
checkout_flang=False, checkout_flang=False,
enable_runtimes="auto",
# Upload artifacts to Google Cloud Storage (for the llvmbisect tool) # Upload artifacts to Google Cloud Storage (for the llvmbisect tool)
stage1_upload_directory=None, stage1_upload_directory=None,
# Use a lower compression level to generate the build-cache package faster # Use a lower compression level to generate the build-cache package faster
# default is 6 according to documentation # default is 6 according to documentation
xz_compression_factor=6, xz_compression_factor=6,
use_pixz_compression=False, use_pixz_compression=False,
# Triggers # Triggers
trigger_after_stage1=None): trigger_after_stage1=None):
############# PREPARING ############# PREPARING
if checks is None:
checks = ['check-all']
if nt_flags is None: if nt_flags is None:
nt_flags = [] nt_flags = []
if testsuite_flags is None: if testsuite_flags is None:
testsuite_flags = [] testsuite_flags = []
if env is None: if env is None:
env = {} env = {}
if extra_cmake_args is None: if extra_cmake_args is None:
extra_cmake_args = [] extra_cmake_args = []
Show All 19 Lines if checkout_libcxx:
depends_on_projects.append('libcxxabi') depends_on_projects.append('libcxxabi')
depends_on_projects.append('libunwind') depends_on_projects.append('libunwind')
if checkout_flang: if checkout_flang:
depends_on_projects.append('flang') depends_on_projects.append('flang')
depends_on_projects.append('mlir') depends_on_projects.append('mlir')
f = LLVMBuildFactory( f = LLVMBuildFactory(
depends_on_projects=depends_on_projects, depends_on_projects=depends_on_projects,
llvm_srcdir='llvm') llvm_srcdir='llvm',
enable_runtimes=enable_runtimes)
# Checkout the latest code for LNT # Checkout the latest code for LNT
# and the test-suite separately. Le's do this first, # and the test-suite separately. Le's do this first,
# so we wouldn't poison got_revision property. # so we wouldn't poison got_revision property.
if runTestSuite or checkout_test_suite: if runTestSuite or checkout_test_suite:
f.addGetSourcecodeForProject( f.addGetSourcecodeForProject(
project='lnt', project='lnt',
src_dir='test/lnt', src_dir='test/lnt',
Show All 11 Linesdef _getClangCMakeBuildFactory(
lit_args = "'-v" lit_args = "'-v"
if jobs is not None: if jobs is not None:
jobs_cmd = ["-j"+str(jobs)] jobs_cmd = ["-j"+str(jobs)]
lit_args += " -j"+str(jobs)+"'" lit_args += " -j"+str(jobs)+"'"
else: else:
lit_args += "'" lit_args += "'"
ninja_cmd = ['ninja'] + jobs_cmd ninja_cmd = ['ninja'] + jobs_cmd
ninja_install_cmd = ['ninja', 'install'] + jobs_cmd ninja_install_cmd = ['ninja', 'install'] + jobs_cmd
ninja_check_cmd = ['ninja', 'check-all'] + jobs_cmd ninja_check_cmd = ['ninja'] + checks + jobs_cmd
# Global configurations # Global configurations
stage1_build = 'stage1' stage1_build = 'stage1'
stage1_install = 'stage1.install' stage1_install = 'stage1.install'
stage2_build = 'stage2' stage2_build = 'stage2'
stage2_install = 'stage2.install' stage2_install = 'stage2.install'
# Set up VS environment, if appropriate. # Set up VS environment, if appropriate.
Show All 13 Lines f.addStep(ShellCommand(name='clean stage 1',
flunkOnFailure=False, flunkOnFailure=False,
description='cleaning stage 1', description='cleaning stage 1',
descriptionDone='clean', descriptionDone='clean',
workdir='.', workdir='.',
doStepIf=clean_build_requested)) doStepIf=clean_build_requested))
############# STAGE 1 ############# STAGE 1
if f.enable_projects:
CmakeCommand.applyRequiredOptions(extra_cmake_args, [ CmakeCommand.applyRequiredOptions(extra_cmake_args, [
('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)), ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),
]) ])
if f.enable_runtimes:
CmakeCommand.applyRequiredOptions(extra_cmake_args, [
('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.enable_runtimes)),
])
rel_src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, stage1_build) rel_src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, stage1_build)
f.addStep(ShellCommand(name='cmake stage 1', f.addStep(ShellCommand(name='cmake stage 1',
command=[cmake, "-G", "Ninja", rel_src_dir, command=[cmake, "-G", "Ninja", rel_src_dir,
"-DCMAKE_BUILD_TYPE="+stage1_config, "-DCMAKE_BUILD_TYPE="+stage1_config,
"-DLLVM_ENABLE_ASSERTIONS=True", "-DLLVM_ENABLE_ASSERTIONS=True",
"-DLLVM_LIT_ARGS="+lit_args, "-DLLVM_LIT_ARGS="+lit_args,
"-DCMAKE_INSTALL_PREFIX=../"+stage1_install] "-DCMAKE_INSTALL_PREFIX=../"+stage1_install]
+ extra_cmake_args, + extra_cmake_args,
haltOnFailure=True, haltOnFailure=True,
description='cmake stage 1', description='cmake stage 1',
workdir=stage1_build, workdir=stage1_build,
doStepIf=FileDoesNotExist("build.ninja"), doStepIf=FileDoesNotExist("build.ninja"),
env=env)) env=env))
f.addStep(WarningCountingShellCommand(name='build stage 1', f.addStep(WarningCountingShellCommand(name='build stage 1',
command=ninja_cmd, command=ninja_cmd,
haltOnFailure=True, haltOnFailure=True,
description='ninja all', description='ninja all',
workdir=stage1_build, workdir=stage1_build,
env=env)) env=env))
if test and testStage1: if checks and testStage1:
haltOnStage1Check = not useTwoStage and not runTestSuite haltOnStage1Check = not useTwoStage and not runTestSuite
f.addStep(LitTestCommand(name='ninja check 1', f.addStep(LitTestCommand(name='ninja check 1',
command=ninja_check_cmd, command=ninja_check_cmd,
haltOnFailure=haltOnStage1Check, haltOnFailure=haltOnStage1Check,
description=["checking stage 1"], description=["checking stage 1"],
descriptionDone=["stage 1 checked"], descriptionDone=["stage 1 checked"],
workdir=stage1_build, workdir=stage1_build,
env=env)) env=env))
Show All 19 Lines if stage1_upload_directory:
use_pixz_compression=use_pixz_compression, use_pixz_compression=use_pixz_compression,
xz_compression_factor=xz_compression_factor) xz_compression_factor=xz_compression_factor)
# Compute the cmake define flag to set the C and C++ compiler to clang. Use # Compute the cmake define flag to set the C and C++ compiler to clang. Use
# clang-cl if we used MSVC for stage1. # clang-cl if we used MSVC for stage1.
if not vs: if not vs:
cc = 'clang' cc = 'clang'
cxx = 'clang++' cxx = 'clang++'
fc = 'flang-new'
else: else:
cc = 'clang-cl.exe' cc = 'clang-cl.exe'
cxx = 'clang-cl.exe' cxx = 'clang-cl.exe'
fc = 'flang-new.exe'
############# STAGE 2 ############# STAGE 2
if useTwoStage: if useTwoStage:
# We always cleanly build the stage 2. If the compiler has been # We always cleanly build the stage 2. If the compiler has been
# changed on the stage 1, we cannot trust any of the intermediate file # changed on the stage 1, we cannot trust any of the intermediate file
# from the old compiler. And if the stage 1 compiler is the same, we # from the old compiler. And if the stage 1 compiler is the same, we
# should not build in the first place. # should not build in the first place.
Show All 34 Lines if useTwoStage:
f.addStep(WarningCountingShellCommand(name='build stage 2', f.addStep(WarningCountingShellCommand(name='build stage 2',
command=ninja_cmd, command=ninja_cmd,
haltOnFailure=True, haltOnFailure=True,
description='ninja all', description='ninja all',
workdir=stage2_build, workdir=stage2_build,
env=env)) env=env))
if test: if checks:
f.addStep(LitTestCommand(name='ninja check 2', f.addStep(LitTestCommand(name='ninja check 2',
command=ninja_check_cmd, command=ninja_check_cmd,
haltOnFailure=not runTestSuite, haltOnFailure=not runTestSuite,
description=["checking stage 2"], description=["checking stage 2"],
descriptionDone=["stage 2 checked"], descriptionDone=["stage 2 checked"],
workdir=stage2_build, workdir=stage2_build,
env=env)) env=env))
▲ Show 20 Lines • Show All 44 Lines • ▼ Show 20 Lines if runTestSuite:
lit = WithProperties('%(builddir)s/'+stage1_build+'/bin/llvm-lit') lit = WithProperties('%(builddir)s/'+stage1_build+'/bin/llvm-lit')
test_suite_cmd = [python, lnt, 'runtest', 'test-suite', test_suite_cmd = [python, lnt, 'runtest', 'test-suite',
'--no-timestamp', '--no-timestamp',
'--sandbox', sandbox, '--sandbox', sandbox,
'--test-suite', test_suite_dir, '--test-suite', test_suite_dir,
'--cc', cc, '--cc', cc,
'--cxx', cxx, '--cxx', cxx,
'--use-lit', lit] '--use-lit', lit]
# Enable fortran if flang is checked out
if checkout_flang:
fortran_flags = [
'--cmake-define=TEST_SUITE_FORTRAN:STRING=ON',
'--cmake-define=CMAKE_Fortran_FLAGS:STRING=' +
'-lpgmath',
util.Interpolate(
'--cmake-define=CMAKE_Fortran_COMPILER=' +
'%(prop:builddir)s/'+compiler_path+'/bin/'+fc)]
test_suite_cmd.extend(fortran_flags)
# Append any option provided by the user # Append any option provided by the user
test_suite_cmd.extend(testsuite_flags) test_suite_cmd.extend(testsuite_flags)
# Only submit if a URL has been specified # Only submit if a URL has been specified
if submitURL is not None: if submitURL is not None:
if not isinstance(submitURL, list): if not isinstance(submitURL, list):
submitURL = [submitURL] submitURL = [submitURL]
for url in submitURL: for url in submitURL:
Show All 9 Lines if runTestSuite:
# Steps to prepare, build and run LNT # Steps to prepare, build and run LNT
f.addStep(ShellCommand(name='clean sandbox', f.addStep(ShellCommand(name='clean sandbox',
command=['rm', '-rf', 'sandbox'], command=['rm', '-rf', 'sandbox'],
haltOnFailure=True, haltOnFailure=True,
description='removing sandbox directory', description='removing sandbox directory',
workdir='test', workdir='test',
env=env)) env=env))
f.addStep(ShellCommand(name='recreate sandbox', f.addStep(ShellCommand(name='recreate sandbox',
command=['virtualenv', 'sandbox'], command=['virtualenv', '--python=python3', 'sandbox'],
haltOnFailure=True, haltOnFailure=True,
description='recreating sandbox', description='recreating sandbox',
workdir='test', workdir='test',
env=env)) env=env))
f.addStep(ShellCommand(name='setup lit', f.addStep(ShellCommand(name='setup lit',
command=[python, lnt_setup, 'develop'], command=[python, lnt_setup, 'develop'],
haltOnFailure=True, haltOnFailure=True,
description='setting up LNT in sandbox', description='setting up LNT in sandbox',
Show All 15 Lines

zorg/buildbot/builders/ClangLTOBuilder.py

Show First 20 Lines • Show All 59 Lines • ▼ Show 20 Lines CmakeCommand.applyDefaultOptions(cmake_args, [
]) ])
# Some options are required for this stage no matter what. # Some options are required for this stage no matter what.
CmakeCommand.applyRequiredOptions(cmake_args, [ CmakeCommand.applyRequiredOptions(cmake_args, [
('-G', 'Ninja'), ('-G', 'Ninja'),
('-DCMAKE_INSTALL_PREFIX=', install_dir), ('-DCMAKE_INSTALL_PREFIX=', install_dir),
]) ])
if f.enable_projects:
CmakeCommand.applyRequiredOptions(cmake_args, [ CmakeCommand.applyRequiredOptions(cmake_args, [
('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)), ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),
])
if f.enable_runtimes:
CmakeCommand.applyRequiredOptions(cmake_args, [
('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.enable_runtimes)),
]) ])
# Note: On this stage we do not care of warnings, as we build with # Note: On this stage we do not care of warnings, as we build with
# a system toolchain and cannot control the environment. # a system toolchain and cannot control the environment.
# Warnings are likely, and we ignore them. # Warnings are likely, and we ignore them.
# Create configuration files with cmake # Create configuration files with cmake
f.addStep(CmakeCommand(name="cmake-configure-stage%s" % stage_num, f.addStep(CmakeCommand(name="cmake-configure-stage%s" % stage_num,
description=["stage%s cmake configure" % stage_num], description=["stage%s cmake configure" % stage_num],
▲ Show 20 Lines • Show All 97 Lines • ▼ Show 20 Lines cmake_args.append(
WithProperties( WithProperties(
"-DCMAKE_CXX_COMPILER=%(builddir)s/" + staged_install + "/bin/clang++" "-DCMAKE_CXX_COMPILER=%(builddir)s/" + staged_install + "/bin/clang++"
)) ))
cmake_args.append( cmake_args.append(
WithProperties( WithProperties(
"-DCMAKE_C_COMPILER=%(builddir)s/" + staged_install + "/bin/clang" "-DCMAKE_C_COMPILER=%(builddir)s/" + staged_install + "/bin/clang"
)) ))
if f.enable_projects:
CmakeCommand.applyRequiredOptions(cmake_args, [
('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),
])
if f.enable_runtimes:
CmakeCommand.applyRequiredOptions(cmake_args, [ CmakeCommand.applyRequiredOptions(cmake_args, [
('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)), ('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.enable_runtimes)),
]) ])
# Create configuration files with cmake # Create configuration files with cmake
f.addStep(CmakeCommand(name="cmake-configure-stage%s" % stage_num, f.addStep(CmakeCommand(name="cmake-configure-stage%s" % stage_num,
description=["stage%s cmake configure" % stage_num], description=["stage%s cmake configure" % stage_num],
haltOnFailure=True, haltOnFailure=True,
options=cmake_args, options=cmake_args,
path=src_dir, path=src_dir,
env=env, env=env,
Show All 33 Lines f.addStep(NinjaCommand(name="install-stage%s-compiler"% stage_num,
)) ))
def getClangWithLTOBuildFactory( def getClangWithLTOBuildFactory(
depends_on_projects = None, depends_on_projects = None,
clean = False, clean = False,
jobs = None, jobs = None,
extra_configure_args = None, extra_configure_args = None,
extra_configure_args_lto_stage = None,
compare_last_2_stages = True, compare_last_2_stages = True,
lto = None, # The string gets passed to -flto flag as is. Like -flto=thin. lto = None, # The string gets passed to -flto flag as is. Like -flto=thin.
env = None, env = None,
**kwargs): **kwargs):
# Set defaults # Set defaults
if depends_on_projects: if depends_on_projects:
depends_on_projects = list(depends_on_projects) depends_on_projects = list(depends_on_projects)
else: else:
# By default we link with LLD. # By default we link with LLD.
depends_on_projects = ['llvm', 'clang', 'lld'] depends_on_projects = ['llvm', 'clang', 'lld']
if lto is None: if lto is None:
lto = 'ON' lto = 'ON'
if extra_configure_args is None: if extra_configure_args is None:
extra_configure_args = [] extra_configure_args = []
else: else:
extra_configure_args = list(extra_configure_args) extra_configure_args = list(extra_configure_args)
if extra_configure_args_lto_stage is None:
extra_configure_args_lto_stage = []
else:
extra_configure_args_lto_stage = list(extra_configure_args_lto_stage)
# Make sure CMAKE_INSTALL_PREFIX and -G are not specified # Make sure CMAKE_INSTALL_PREFIX and -G are not specified
# in the extra_configure_args. We set them internally as needed. # in the extra_configure_args. We set them internally as needed.
# TODO: assert extra_configure_args. # TODO: assert extra_configure_args.
install_prefix_specified = any(a.startswith('-DCMAKE_INSTALL_PREFIX=') for a in extra_configure_args) install_prefix_specified = (any(a.startswith('-DCMAKE_INSTALL_PREFIX=') for a in extra_configure_args) or
any(a.startswith('-DCMAKE_INSTALL_PREFIX=') for a in extra_configure_args_lto_stage))
assert not install_prefix_specified, "Please do not explicitly specify the install prefix for multi-stage build." assert not install_prefix_specified, "Please do not explicitly specify the install prefix for multi-stage build."
# Prepare environmental variables. Set here all env we want everywhere. # Prepare environmental variables. Set here all env we want everywhere.
merged_env = { merged_env = {
'TERM' : 'dumb' # Be cautious and disable color output from all tools. 'TERM' : 'dumb' # Be cautious and disable color output from all tools.
} }
if env is not None: if env is not None:
# Overwrite pre-set items with the given ones, so user can set anything. # Overwrite pre-set items with the given ones, so user can set anything.
Show All 35 Lines _addSteps4StagedCompiler(f,
jobs=jobs, jobs=jobs,
extra_configure_args=extra_configure_args, extra_configure_args=extra_configure_args,
env=merged_env) env=merged_env)
# Build all the remaining stages with exactly the same configuration. # Build all the remaining stages with exactly the same configuration.
CmakeCommand.applyRequiredOptions(extra_configure_args, [ CmakeCommand.applyRequiredOptions(extra_configure_args, [
('-DLLVM_ENABLE_LTO=', lto), ('-DLLVM_ENABLE_LTO=', lto),
# NOTE: Diagnostic messages could contain __FILE__, thus final executable would depend on build path.
# We do not want that since we expect executables built from the same source code in different
# build directories be exactly the same.
('-DLLVM_ENABLE_ASSERTIONS=', 'OFF'),
]) ])
# If we build LLD, we would link with LLD. # If we build LLD, we would link with LLD.
# Otherwise we link with the system linker. # Otherwise we link with the system linker.
if 'lld' in depends_on_projects: if 'lld' in depends_on_projects:
CmakeCommand.applyRequiredOptions(extra_configure_args, [ CmakeCommand.applyRequiredOptions(extra_configure_args, [
('-DLLVM_ENABLE_LLD=', 'ON'), ('-DLLVM_ENABLE_LLD=', 'ON'),
]) ])
# The rest are test stages, which depend on the staged compiler we are ultimately after. # The rest are test stages, which depend on the staged compiler we are ultimately after.
s = f.staged_compiler_idx + 1 s = f.staged_compiler_idx + 1
staged_install = f.stage_installdirs[f.staged_compiler_idx] staged_install = f.stage_installdirs[f.staged_compiler_idx]
for i in range(s, len(f.stage_objdirs[s:]) + s): for i in range(s, len(f.stage_objdirs[s:]) + s):
configure_args = extra_configure_args[:] configure_args = extra_configure_args[:] + extra_configure_args_lto_stage[:]
configure_args.append( configure_args.append(
WithProperties( WithProperties(
"-DCMAKE_AR=%(builddir)s/" + staged_install + "/bin/llvm-ar" "-DCMAKE_AR=%(builddir)s/" + staged_install + "/bin/llvm-ar"
)) ))
configure_args.append( configure_args.append(
WithProperties( WithProperties(
"-DCMAKE_RANLIB=%(builddir)s/" + staged_install + "/bin/llvm-ranlib" "-DCMAKE_RANLIB=%(builddir)s/" + staged_install + "/bin/llvm-ranlib"
▲ Show 20 Lines • Show All 67 Lines • Show Last 20 Lines

zorg/buildbot/builders/DoxygenDocsBuilder.py

  • This file was added.
from collections import OrderedDict
from importlib import reload
from buildbot.plugins import steps, util
from buildbot.steps.shell import ShellCommand
from zorg.buildbot.builders import UnifiedTreeBuilder
from zorg.buildbot.commands import CmakeCommand
from zorg.buildbot.process import factory
reload(factory)
reload(UnifiedTreeBuilder)
reload(CmakeCommand)
llvm_docs = OrderedDict([
# Project Build target Local path Remote path
("llvm", ("doxygen", "docs/doxygen/html/", "llvm")),
("clang", (None, "tools/clang/docs/doxygen/html/", "cfe")),
("clang-tools-extra", (None, "tools/clang/tools/extra/docs/doxygen/html/", "cfe-extra")),
("flang", (None, "tools/flang/docs/doxygen/html/", "flang")),
("polly", (None, "tools/polly/docs/doxygen/html/", "polly")),
("openmp", (None, "projects/openmp/docs/doxygen/html/", "openmp")),
("lldb", ("lldb-cpp-doc", "tools/lldb/docs/cpp_reference/", "lldb/cpp_reference")),
# NOTE: 5/9/2020 lldb-python-doc fails to build. Disabled till be fixed.
#(None, ("lldb-python-doc", "tools/lldb/docs/python_reference/", "lldb")),
])
def getLLVMDocsBuildFactory(
clean = True,
depends_on_projects = None,
extra_configure_args = None,
timeout=10800,
env = None,
**kwargs):
if depends_on_projects is None:
# All the projects from llvm_docs, and remove all duplicates.
_depends_on_projects=list(set(
[project for project in llvm_docs if project]))
else:
# Make a local copy of depends_on_projects, as we are going to modify
# that.
_depends_on_projects=depends_on_projects[:]
# Some projects are interdependent for the purpose of documentation.
# Enforce the dependencies.
# TODO: Check later the dependencies for doxygen docs and enforce them
# here if needed.
# Make a local copy of the configure args, as we are going to modify that.
if extra_configure_args:
cmake_args = extra_configure_args[:]
else:
cmake_args = list()
# Prepare environmental variables. Set here all env we want everywhere.
merged_env = {
'TERM' : 'dumb' # Be cautious and disable color output from all tools.
}
if env is not None:
# Overwrite pre-set items with the given ones, so user can set anything.
merged_env.update(env)
CmakeCommand.CmakeCommand.applyDefaultOptions(cmake_args, [
("-G", "Ninja"),
("-DLLVM_ENABLE_DOXYGEN=", "ON"),
("-DLLVM_BUILD_DOCS=", "ON"),
("-DCLANG_TOOLS_EXTRA_INCLUDE_DOCS=", "ON"),
("-DLLVM_ENABLE_ASSERTIONS=", "OFF"),
("-DCMAKE_BUILD_TYPE=", "Release"),
])
f = UnifiedTreeBuilder.getCmakeBuildFactory(
clean=clean,
depends_on_projects=_depends_on_projects,
enable_runtimes=[], # Docs don't support runtimes build yet.
extra_configure_args=cmake_args,
env=merged_env,
**kwargs) # Pass through all the extra arguments.
# Build the documentation for all the projects.
for project in llvm_docs:
target = llvm_docs[project][0]
# Build only those with specifies targets.
if target:
UnifiedTreeBuilder.addNinjaSteps(
f,
# Doxygen builds the final result for really
# long time without any output.
# We have to have a long timeout at this step.
timeout=timeout,
targets=[target],
checks=[],
env=merged_env,
**kwargs)
# Publish just built documentation
for project in llvm_docs:
target, local_path, remote_path = llvm_docs[project]
f.addStep(
ShellCommand(
name="Publish {}".format(project or target),
description=[
"Publish", "just", "built", "documentation", "for",
"{}".format(project or target)
],
command=[
'rsync',
'-vrl',
'--delete', '--force', '--delay-updates', '--delete-delay',
'--ignore-times',
'--checksum',
'-p', '--chmod=Du=rwx,Dg=rwx,Do=rx,Fu=rw,Fg=rw,Fo=r',
"{}".format(local_path),
"lists.llvm.org:web/doxygen/{}".format(remote_path),
],
env=merged_env,
)
)
return f

zorg/buildbot/builders/FlangBuilder.py

from zorg.buildbot.commands.CmakeCommand import CmakeCommand from zorg.buildbot.commands.CmakeCommand import CmakeCommand
from zorg.buildbot.process.factory import LLVMBuildFactory from zorg.buildbot.process.factory import LLVMBuildFactory
from zorg.buildbot.builders.UnifiedTreeBuilder import addNinjaSteps, getCmakeWithNinjaBuildFactory from zorg.buildbot.builders.UnifiedTreeBuilder import addNinjaSteps, getCmakeWithNinjaBuildFactory
def getFlangOutOfTreeBuildFactory( def getFlangOutOfTreeBuildFactory(
checks = None, checks = None,
clean = False, clean = False,
llvm_extra_configure_args = None, llvm_extra_configure_args = None,
flang_extra_configure_args = None, flang_extra_configure_args = None,
env = None, env = None,
**kwargs): **kwargs):
if env is None: if env is None:
env = dict() env = dict()
f = getCmakeWithNinjaBuildFactory( f = getCmakeWithNinjaBuildFactory(
depends_on_projects=['llvm','mlir'], depends_on_projects=['llvm','clang','mlir'],
obj_dir="build_llvm", obj_dir="build_llvm",
checks=[], checks=[],
clean=clean, clean=clean,
extra_configure_args=llvm_extra_configure_args, extra_configure_args=llvm_extra_configure_args,
env=env, env=env,
**kwargs) **kwargs)
if checks is None: if checks is None:
Show All 11 Lines CmakeCommand.applyRequiredOptions(flang_cmake_args, [
]) ])
flang_obj_dir = "build_flang" flang_obj_dir = "build_flang"
flang_src_dir = "{}/flang".format(f.monorepo_dir) flang_src_dir = "{}/flang".format(f.monorepo_dir)
# Add LLVM_DIR and MLIR_DIR to the CMake invocation. # Add LLVM_DIR and MLIR_DIR to the CMake invocation.
llvm_dir = "{}/lib/cmake/llvm".format(f.obj_dir) llvm_dir = "{}/lib/cmake/llvm".format(f.obj_dir)
mlir_dir = "{}/lib/cmake/mlir".format(f.obj_dir) mlir_dir = "{}/lib/cmake/mlir".format(f.obj_dir)
clang_dir = "{}/lib/cmake/clang".format(f.obj_dir)
CmakeCommand.applyRequiredOptions(flang_cmake_args, [ CmakeCommand.applyRequiredOptions(flang_cmake_args, [
# We actually need the paths to be relative to the source directory, # We actually need the paths to be relative to the source directory,
# otherwise find_package can't locate the config files. # otherwise find_package can't locate the config files.
('-DLLVM_DIR:PATH=', ('-DLLVM_DIR:PATH=',
LLVMBuildFactory.pathRelativeTo(llvm_dir, flang_src_dir)), LLVMBuildFactory.pathRelativeTo(llvm_dir, flang_src_dir)),
('-DMLIR_DIR:PATH=', ('-DMLIR_DIR:PATH=',
LLVMBuildFactory.pathRelativeTo(mlir_dir, flang_src_dir)), LLVMBuildFactory.pathRelativeTo(mlir_dir, flang_src_dir)),
('-DCLANG_DIR:PATH=',
LLVMBuildFactory.pathRelativeTo(clang_dir, flang_src_dir)),
]) ])
# We can't use addCmakeSteps as that would use the path in f.llvm_srcdir. # We can't use addCmakeSteps as that would use the path in f.llvm_srcdir.
f.addStep(CmakeCommand(name="cmake-configure-flang", f.addStep(CmakeCommand(name="cmake-configure-flang",
haltOnFailure=True, haltOnFailure=True,
description=["CMake", "configure", "flang"], description=["CMake", "configure", "flang"],
options=flang_cmake_args, options=flang_cmake_args,
path=LLVMBuildFactory.pathRelativeTo(flang_src_dir, path=LLVMBuildFactory.pathRelativeTo(flang_src_dir,
Show All 14 Lines

zorg/buildbot/builders/FuchsiaBuilder.py

Show All 22 Lines if env is not None:
merged_env.update(env) merged_env.update(env)
src_dir = "llvm.src" src_dir = "llvm.src"
obj_dir = "llvm.obj" obj_dir = "llvm.obj"
install_dir = "llvm.install" install_dir = "llvm.install"
f = UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps( f = UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps(
depends_on_projects=[ depends_on_projects=[
"llvm", "bolt",
"clang", "clang",
"clang-tools-extra", "clang-tools-extra",
"compiler-rt", "compiler-rt",
"libcxx", "libcxx",
"libcxxabi", "libcxxabi",
"libunwind", "libunwind",
"lld" "lld",
"llvm",
"polly",
], ],
llvm_srcdir=src_dir, llvm_srcdir=src_dir,
obj_dir=obj_dir, obj_dir=obj_dir,
install_dir=install_dir, install_dir=install_dir,
**kwargs) # Pass through all the extra arguments. **kwargs) # Pass through all the extra arguments.
# Get Fuchsia SDK. # Get Fuchsia SDK.
sdk_dir = "fuchsia.sdk" sdk_dir = "fuchsia.sdk"
Show All 40 Linesdef getFuchsiaToolchainBuildFactory(
if extra_configure_args is None: if extra_configure_args is None:
cmake_options = [] cmake_options = []
else: else:
cmake_options = extra_configure_args[:] cmake_options = extra_configure_args[:]
# Some options are required for this stage no matter what. # Some options are required for this stage no matter what.
CmakeCommand.applyRequiredOptions(cmake_options, [ CmakeCommand.applyRequiredOptions(cmake_options, [
("-G", "Ninja"), ("-G", "Ninja"),
("-DLLVM_ENABLE_PROJECTS=", "clang;clang-tools-extra;lld"),
("-DLLVM_ENABLE_RUNTIMES=", "compiler-rt;libcxx;libcxxabi;libunwind"),
]) ])
# Set proper defaults. # Set proper defaults.
CmakeCommand.applyDefaultOptions(cmake_options, [ CmakeCommand.applyDefaultOptions(cmake_options, [
("-DBOOTSTRAP_LLVM_ENABLE_LTO=", "OFF"), ("-DBOOTSTRAP_LLVM_ENABLE_LTO=", "OFF"),
("-DLLVM_ENABLE_LTO=", "OFF"), ("-DLLVM_ENABLE_LTO=", "OFF"),
]) ])
▲ Show 20 Lines • Show All 48 Lines • Show Last 20 Lines

zorg/buildbot/builders/HtmlDocsBuilder.py

  • This file was added.
from collections import OrderedDict
from importlib import reload
from buildbot.plugins import steps, util
from buildbot.steps.shell import ShellCommand
from buildbot.steps.shell import WarningCountingShellCommand
from zorg.buildbot.process import factory
reload(factory)
llvm_docs = OrderedDict([
# Project Build target Build path Local path Remote path
("lnt", ("html", "docs", "_build/html/", "lnt")),
])
# We build with make for now. Change later if needed.
build_cmd = 'make'
def getHtmlDocsBuildFactory(
depends_on_projects = None,
clean = False,
env = None,
**kwargs):
if depends_on_projects is None:
# All the projects by default.
_depends_on_projects=llvm_docs.keys()
else:
# Make a local copy of depends_on_projects, as we are going to modify
# that.
_depends_on_projects=depends_on_projects[:]
# Prepare environmental variables. Set here all env we want everywhere.
merged_env = {
'TERM' : 'dumb' # Be cautious and disable color output from all tools.
}
if env is not None:
# Overwrite pre-set items with the given ones, so user can set anything.
merged_env.update(env)
# HTML Sphinx documentation builds in tree, each in its own directory.
# For that, make sure the obj_dir is the same as llvm_srcdir.
src_dir = kwargs.pop('llvm_srcdir', '.')
f = factory.LLVMBuildFactory(
clean=clean,
depends_on_projects=_depends_on_projects,
llvm_srcdir=src_dir,
obj_dir=src_dir,
**kwargs) # Pass through all the extra arguments.
# Build the documentation
for project in llvm_docs:
# Checkout the source code and remove all the untracked files, so
# we would build a fresh new documentation.
f.addStep(
steps.Git(
name='Checkout the {} source code'.format(project),
repourl=f.repourl_prefix + "llvm-{}.git".format(project),
mode='full',
method='fresh',
progress=True,
workdir=util.WithProperties(project),
env=merged_env,
**kwargs))
target, build_path, local_path, remote_path = llvm_docs[project]
build_dir = util.WithProperties(
"{}".format("/".join([
project,
build_path])))
f.addStep(
steps.WarningCountingShellCommand(
name="Build {} documentation".format(project),
command=[build_cmd, target],
haltOnFailure=True,
workdir=build_dir,
env=merged_env,
**kwargs))
# Publish just built documentation
f.addStep(
ShellCommand(
name="Publish {}".format(project),
description=[
"Publish", "just", "built", "documentation", "for",
"{}".format(project)
],
command=[
'rsync',
'-vrl',
'--delete', '--force', '--delay-updates', '--delete-delay',
'--ignore-times',
'--checksum',
'-p', '--chmod=Du=rwx,Dg=rwx,Do=rx,Fu=rw,Fg=rw,Fo=r',
"{}".format(local_path),
"lists.llvm.org:web/{}".format(remote_path),
],
workdir=build_dir,
env=merged_env,
)
)
return f

zorg/buildbot/builders/LLDBBuilder.py

Show First 20 Lines • Show All 42 Lines • ▼ Show 20 Lines if vs and vs != 'manual':
extract_fn=extractVSEnvironment)) extract_fn=extractVSEnvironment))
env = Property('vs_env') env = Property('vs_env')
f.addGetSourcecodeSteps() f.addGetSourcecodeSteps()
build_cmd=['ninja'] build_cmd=['ninja']
install_cmd = ['ninja','install'] install_cmd = ['ninja','install']
test_cmd = ['ninja','check-lldb'] test_cmd = ['ninja','check-lldb']
lit_args = '-v'
if jobs: if jobs:
build_cmd.append(WithProperties("-j%s" % jobs)) build_cmd.append(WithProperties("-j%s" % jobs))
install_cmd.append(WithProperties("-j%s" % jobs)) install_cmd.append(WithProperties("-j%s" % jobs))
test_cmd.append(WithProperties("-j%s" % jobs)) test_cmd.append(WithProperties("-j%s" % jobs))
############# CLEANING ############# CLEANING
cleanBuildRequested = lambda step: clean or step.build.getProperty("clean", default=step.build.getProperty("clean_obj")) cleanBuildRequested = lambda step: clean or step.build.getProperty("clean", default=step.build.getProperty("clean_obj"))
f.addStep(steps.RemoveDirectory(name='clean '+build_dir, f.addStep(steps.RemoveDirectory(name='clean '+build_dir,
dir=build_dir, dir=build_dir,
haltOnFailure=False, haltOnFailure=False,
flunkOnFailure=False, flunkOnFailure=False,
doStepIf=cleanBuildRequested doStepIf=cleanBuildRequested
)) ))
rel_src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, f.obj_dir) rel_src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, f.obj_dir)
cmake_options = [ cmake_options = [
"-G", "Ninja", "-G", "Ninja",
"-DCMAKE_BUILD_TYPE=" + config, "-DCMAKE_BUILD_TYPE=" + config,
"-DLLVM_LIT_ARGS='%s'" % lit_args,
"-DCMAKE_INSTALL_PREFIX=../install", "-DCMAKE_INSTALL_PREFIX=../install",
"-DLLVM_ENABLE_PROJECTS=%s" % ";".join(f.depends_on_projects), "-DLLVM_ENABLE_PROJECTS=%s" % ";".join(f.depends_on_projects),
] ]
if python_source_dir: if python_source_dir:
cmake_options.append("-DPYTHON_HOME=" + python_source_dir) cmake_options.append("-DPYTHON_HOME=" + python_source_dir)
if extra_cmake_args: if extra_cmake_args:
cmake_options += extra_cmake_args cmake_options += extra_cmake_args
Show All 35 Lines

zorg/buildbot/builders/LibcxxAndAbiBuilder.py

  • This file was deleted.
import buildbot.steps.shell
import buildbot.process.properties as properties
from zorg.buildbot.commands.LitTestCommand import LitTestCommand
from zorg.buildbot.commands.CmakeCommand import CmakeCommand
from zorg.buildbot.process.factory import LLVMBuildFactory
from zorg.buildbot.builders import UnifiedTreeBuilder
def getLibcxxAndAbiBuilder(f=None, env=None,
cmake_extra_opts=None, lit_extra_opts=None,
lit_extra_args=None, check_libcxx_abilist=False,
check_libcxx_benchmarks=None,
depends_on_projects=None,
use_cache=None,
**kwargs):
if env is None:
env = {}
if cmake_extra_opts is None:
cmake_extra_opts = {}
if lit_extra_opts is None:
lit_extra_opts = {}
if lit_extra_args is None:
lit_extra_args = []
if depends_on_projects is None:
depends_on_projects = ['libcxx','libcxxabi','libunwind']
src_root = 'llvm'
build_path = 'build'
if f is None:
f = UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps(
depends_on_projects=depends_on_projects,
llvm_srcdir=src_root,
obj_dir=build_path,
**kwargs) # Pass through all the extra arguments.
rel_src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, build_path)
# Specify the max number of threads using properties so LIT doesn't use
# all the threads on the system.
litTestArgs = '-vv --show-unsupported --show-xfail --threads=%(jobs)s'
if lit_extra_args:
litTestArgs += ' ' + ' '.join(lit_extra_args)
for key in lit_extra_opts:
litTestArgs += (' --param=' + key + '=' + lit_extra_opts[key])
cmake_opts = [properties.WithProperties('-DLLVM_LIT_ARGS='+litTestArgs)]
for key in cmake_extra_opts:
cmake_opts.append('-D' + key + '=' + cmake_extra_opts[key])
if use_cache:
libcxx_cache_dir = '%s/../libcxx/cmake/caches' % rel_src_dir
cache = '%s/%s' % (libcxx_cache_dir, use_cache)
cmake_opts.append('-C' + cache)
# FIXME: The libc++ abilist's are generated in release mode with debug
# symbols Other configurations may contain additional non-inlined symbols.
if check_libcxx_abilist and not 'CMAKE_BUILD_TYPE' in cmake_extra_opts:
cmake_opts.append('-DCMAKE_BUILD_TYPE=RELWITHDEBINFO')
# Force libc++ to use the in-tree libc++abi unless otherwise specified.
if 'LIBCXX_CXX_ABI' not in cmake_extra_opts:
cmake_opts.append('-DLIBCXX_CXX_ABI=libcxxabi')
# Nuke/remake build directory and run CMake
f.addStep(buildbot.steps.shell.ShellCommand(
name='rm.builddir', command=['rm', '-rf', build_path],
workdir=".",
haltOnFailure=False))
CmakeCommand.applyRequiredOptions(cmake_opts, [
('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)),
])
f.addStep(buildbot.steps.shell.ShellCommand(
name='cmake', command=['cmake', rel_src_dir] + cmake_opts,
haltOnFailure=True, workdir=build_path, env=env))
# Build libcxxabi
jobs_flag = properties.WithProperties('-j%(jobs)s')
f.addStep(buildbot.steps.shell.ShellCommand(
name='build.libcxxabi', command=['make', jobs_flag, 'cxxabi'],
haltOnFailure=True, workdir=build_path))
# Build libcxx
f.addStep(buildbot.steps.shell.ShellCommand(
name='build.libcxx', command=['make', jobs_flag, 'cxx'],
haltOnFailure=True, workdir=build_path))
# Test libc++abi
f.addStep(LitTestCommand(
name = 'test.libcxxabi',
command = ['make', jobs_flag, 'check-cxxabi'],
description = ['testing', 'libcxxabi'],
descriptionDone = ['test', 'libcxxabi'],
workdir = build_path))
# Test libc++
f.addStep(LitTestCommand(
name = 'test.libcxx',
command = ['make', jobs_flag, 'check-cxx'],
description = ['testing', 'libcxx'],
descriptionDone = ['test', 'libcxx'],
workdir = build_path))
if check_libcxx_abilist:
f.addStep(buildbot.steps.shell.ShellCommand(
name = 'test.libcxx.abilist',
command = ['make', 'check-cxx-abilist'],
description = ['testing', 'libcxx', 'abi'],
descriptionDone = ['test', 'libcxx', 'abi'],
workdir = build_path))
if check_libcxx_benchmarks:
# Build the libc++ benchmarks
f.addStep(buildbot.steps.shell.ShellCommand(
name='build.libcxx.benchmarks',
command=['make', jobs_flag, 'cxx-benchmarks'],
haltOnFailure=True, workdir=build_path))
# Run the benchmarks
f.addStep(LitTestCommand(
name = 'test.libcxx.benchmarks',
command = ['make', jobs_flag, 'check-cxx-benchmarks'],
description = ['testing', 'libcxx', 'benchmarks'],
descriptionDone = ['test', 'libcxx', 'benchmarks'],
workdir = build_path))
return f

zorg/buildbot/builders/OpenMPBuilder.py

from buildbot.steps.shell import ShellCommand from buildbot.steps.shell import ShellCommand
from buildbot.process.properties import WithProperties from buildbot.process.properties import WithProperties
from buildbot.plugins import steps
from zorg.buildbot.commands.LitTestCommand import LitTestCommand from zorg.buildbot.commands.LitTestCommand import LitTestCommand
from zorg.buildbot.commands.CmakeCommand import CmakeCommand from zorg.buildbot.commands.CmakeCommand import CmakeCommand
from zorg.buildbot.commands.NinjaCommand import NinjaCommand from zorg.buildbot.commands.NinjaCommand import NinjaCommand
from zorg.buildbot.builders import UnifiedTreeBuilder from zorg.buildbot.builders import UnifiedTreeBuilder
from zorg.buildbot.process.factory import LLVMBuildFactory from zorg.buildbot.process.factory import LLVMBuildFactory
def getOpenMPCMakeBuildFactory( def getOpenMPCMakeBuildFactory(
jobs = '%(jobs)s', # Number of concurrent jobs. jobs = '%(jobs)s', # Number of concurrent jobs.
clean = True, # "clean" step is requested if true clean = True, # "clean" step is requested if true
env = None, # Environmental variables for all steps. env = None, # Environmental variables for all steps.
ompt = False, # Whether to enable the OpenMP Tools Interface.
test = True, # Test the built libraries. test = True, # Test the built libraries.
depends_on_projects = None, depends_on_projects = None,
enable_runtimes = "auto",
extraCmakeArgs = None,
install = False,
testsuite = False,
testsuite_sollvevv = False,
extraTestsuiteCmakeArgs = None,
add_lit_checks = None,
**kwargs): **kwargs):
if extraCmakeArgs is None:
extraCmakeArgs = []
if extraTestsuiteCmakeArgs is None:
extraTestsuiteCmakeArgs = []
# Prepare environmental variables. Set here all env we want everywhere. # Prepare environmental variables. Set here all env we want everywhere.
merged_env = { merged_env = {
'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences. 'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences.
} }
# Overwrite pre-set items with the given ones, so user can set anything. # Overwrite pre-set items with the given ones, so user can set anything.
if env is not None: if env is not None:
merged_env.update(env) merged_env.update(env)
llvm_srcdir = 'llvm.src' llvm_srcdir = 'llvm.src'
llvm_builddir = 'llvm.build' llvm_builddir = 'llvm.build'
llvm_instdir = 'llvm.inst'
cleanBuildRequested = lambda step: clean or step.build.getProperty("clean", default=step.build.getProperty("clean_obj")) testsuite_srcdir = "test-suite.src"
testsuite_builddir = "test-suite.build"
sollvevv_srcdir = "sollvevv.src"
# If true, clean everything, including source dirs
def cleanBuildRequested(step):
return step.build.getProperty("clean")
# If true, clean build products; implied if cleanBuildRequested is true
def cleanObjRequested(step):
return cleanBuildRequested(step) or clean or step.build.getProperty("clean_obj")
if depends_on_projects is None: if depends_on_projects is None:
# Monorepo configuration requires llvm and clang to get cmake work. # Monorepo configuration requires llvm and clang to get cmake work.
depends_on_projects = ['llvm', 'clang', 'openmp'] depends_on_projects = ['llvm', 'clang', 'openmp']
f = UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps( f = UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps(
depends_on_projects=depends_on_projects, depends_on_projects=depends_on_projects,
enable_runtimes=enable_runtimes,
llvm_srcdir=llvm_srcdir, llvm_srcdir=llvm_srcdir,
obj_dir=llvm_builddir, obj_dir=llvm_builddir,
cleanBuildRequested=cleanBuildRequested, cleanBuildRequested=cleanBuildRequested,
env=merged_env, env=merged_env,
**kwargs) # Pass through all the extra arguments. **kwargs) # Pass through all the extra arguments.
f.addStep( f.addStep(steps.RemoveDirectory(name='clean',
ShellCommand( dir=f.obj_dir,
name = 'clean',
command = ['rm', '-rf', f.obj_dir],
warnOnFailure = True, warnOnFailure=True,
description = ['clean'], doStepIf=cleanObjRequested))
doStepIf = cleanBuildRequested,
workdir = '.',
env = merged_env))
# Configure LLVM and OpenMP (and Clang, if requested). # Configure LLVM and OpenMP (and Clang, if requested).
cmake_args = ['-DCMAKE_BUILD_TYPE=Release', '-DLLVM_ENABLE_ASSERTIONS=ON'] cmake_args = ['-DCMAKE_BUILD_TYPE=Release', '-DLLVM_ENABLE_ASSERTIONS=ON']
if ompt:
cmake_args += ['-DLIBOMP_OMPT_SUPPORT=ON']
if test: if test:
lit_args = '-vv --show-unsupported --show-xfail -j %s' % jobs lit_args = '-vv --show-unsupported --show-xfail -j %s' % jobs
cmake_args += [WithProperties('-DLLVM_LIT_ARGS=%s' % lit_args)] cmake_args += [WithProperties('-DLLVM_LIT_ARGS=%s' % lit_args)]
if install:
cmake_args += [WithProperties('-DCMAKE_INSTALL_PREFIX=%(builddir)s/' + llvm_instdir)]
cmake_args += extraCmakeArgs
if f.enable_projects:
CmakeCommand.applyDefaultOptions(cmake_args, [
('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),
])
CmakeCommand.applyRequiredOptions(cmake_args, [ if f.enable_runtimes:
('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)), CmakeCommand.applyDefaultOptions(cmake_args, [
('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.enable_runtimes)),
]) ])
# Add llvm-lit and clang (if built) to PATH # Add llvm-lit and clang (if built) to PATH
merged_env.update({ merged_env.update({
'PATH': WithProperties('%(builddir)s/' + llvm_builddir + '/bin:${PATH}')}) 'PATH': WithProperties('%(builddir)s/' + llvm_builddir + '/bin:${PATH}')})
src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, f.obj_dir) src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, f.obj_dir)
f.addStep(CmakeCommand(name='configure-openmp', f.addStep(CmakeCommand(name='configure-openmp',
Show All 25 Lines if test:
ninja_test_args = ['ninja', WithProperties('-j %s' % jobs)] ninja_test_args = ['ninja', WithProperties('-j %s' % jobs)]
f.addStep( f.addStep(
LitTestCommand( LitTestCommand(
name = 'test-openmp', name = 'test-openmp',
command = ninja_test_args + ['check-openmp'], command = ninja_test_args + ['check-openmp'],
description = 'test openmp', description = 'test openmp',
workdir = f.obj_dir, workdir = f.obj_dir,
env = merged_env, env = merged_env,
haltOnFailure=False,
flunkOnFailure=True))
# When requested run additional lit tests
if add_lit_checks != None:
for add_check in add_lit_checks:
ninja_test_args = ['ninja', WithProperties('-j %s' % jobs)]
f.addStep(LitTestCommand(
name = 'Add check ' + add_check,
command = [ninja_test_args, add_check],
description = ["Additional check in OpenMP for", add_check,],
env = merged_env,
workdir = f.obj_dir,
haltOnFailure = False,
flunkOnFailure=True))
clangexe = "%(builddir)s/" + llvm_builddir + "/bin/clang"
clangxxexe = "%(builddir)s/" + llvm_builddir + "/bin/clang++"
litexe = "%(builddir)s/" + llvm_builddir + "/bin/llvm-lit"
libdir = "%(builddir)s/" + llvm_builddir + "/lib"
if install:
f.addStep(steps.RemoveDirectory(name="LLVM: Clean Install Dir",
dir=llvm_instdir,
haltOnFailure=False))
f.addStep(NinjaCommand(name="LLVM: Install",
description="installing",
descriptionDone="install",
descriptionSuffix="LLVM",
workdir=f.obj_dir,
targets=['install'],
env=merged_env,
haltOnFailure=True)) haltOnFailure=True))
# If installing, use the installed version of clang.
clangexe = "%(builddir)s/" + llvm_instdir + "/bin/clang"
clangxxexe = "%(builddir)s/" + llvm_instdir + "/bin/clang++"
libdir = "%(builddir)s/" + llvm_instdir + "/lib"
if testsuite:
f.addStep(steps.RemoveDirectory(name="Test-Suite: Clean Source",
dir=testsuite_srcdir,
haltOnFailure=False,
warnOnFailure=True,
doStepIf=cleanBuildRequested))
f.addGetSourcecodeForProject(name="Test-Suite: Checkout",
description="fetching",
descriptionDone="fetch",
descriptionSuffix="Test-Suite",
project='test-suite',
src_dir=testsuite_srcdir,
alwaysUseLatest=True)
if testsuite_sollvevv:
f.addStep(steps.RemoveDirectory(name="SOLLVE V&V: Clean Source",
dir=sollvevv_srcdir,
haltOnFailure=False,
warnOnFailure=True,
doStepIf=cleanBuildRequested))
f.addStep(steps.Git(name="SOLLVE V&V: Checkout",
description="fetching",
descriptionDone="fetch",
descriptionSuffix="SOLLVE V&V",
repourl='https://github.com/SOLLVE/sollve_vv.git',
workdir=sollvevv_srcdir,
alwaysUseLatest=True))
f.addStep(steps.RemoveDirectory(name="Test-Suite: Clean Build",
dir=testsuite_builddir,
haltOnFailure=False,
warnOnFailure=True))
testsuite_options = [
"-DCMAKE_BUILD_TYPE=Release",
"-DTEST_SUITE_LIT_FLAGS=-vv;-s;-j6;-o;report.json",
"-DTEST_SUITE_EXTRA_C_FLAGS=-gline-tables-only",
"-DTEST_SUITE_EXTRA_CXX_FLAGS=-gline-tables-only",
WithProperties("-DCMAKE_C_COMPILER=" + clangexe),
WithProperties("-DCMAKE_CXX_COMPILER=" + clangxxexe),
WithProperties("-DTEST_SUITE_LIT=" + litexe),
]
if testsuite_sollvevv:
testsuite_options += [
"-DTEST_SUITE_SUBDIRS=External/sollve_vv",
"-DTEST_SUITE_SOLLVEVV_ROOT=../" + sollvevv_srcdir,
]
testsuite_options += extraTestsuiteCmakeArgs
f.addStep(CmakeCommand(name="Test-Suite: Configure",
description="configuring",
descriptionDone="configure",
descriptionSuffix="Test-Suite",
generator='Ninja',
path='../' + testsuite_srcdir,
workdir=testsuite_builddir,
options=testsuite_options,
haltOnFailure=True,
logfiles={
'CMakeCache.txt' : 'CMakeCache.txt',
}))
f.addStep(NinjaCommand(name="Test-Suite: Compile",
description="compiling",
descriptionDone="compile",
descriptionSuffix="Test-Suite",
options=['-k0'], # Continue building; programs that don't compile will fail with NOEXE.
haltOnFailure=False,
flunkOnFailure=False, # SOLLVE V&V contains tests that clang have not been implemented yet.
warnOnFailure=True,
workdir=testsuite_builddir))
merged_env.update({
'LD_LIBRARY_PATH': WithProperties(libdir + ':${LD_LIBRARY_PATH}')
})
f.addStep(LitTestCommand(name="Test-Suite: Run",
description="running",
descriptionDone="run",
descriptionSuffix="Test-Suite",
command=[WithProperties(litexe), '-vv', '-s', '-j6', '-o','report.json', '.'],
haltOnFailure=False,
flunkOnFailure=False, # SOLLVE V&V contains tests that clang have not been implemented yet.
warnOnFailure=True,
workdir=testsuite_builddir,
env=merged_env,
logfiles={
'test.log' : 'test.log',
'report.json': 'report.json'
}))
return f return f

zorg/buildbot/builders/PollyBuilder.py

Show All 38 Linesdef getPollyBuildFactory(
# Prepare environmental variables. Set here all env we want everywhere. # Prepare environmental variables. Set here all env we want everywhere.
merged_env = { merged_env = {
'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences. 'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences.
} }
if env: if env:
merged_env.update(env) # Overwrite pre-set items with the given ones, so user can set anything. merged_env.update(env) # Overwrite pre-set items with the given ones, so user can set anything.
depends_on_projects = ['llvm','clang','polly'] depends_on_projects = ['llvm','clang','polly']
if testsuite:
# XRay tests in test-suite require compiler-rt
depends_on_projects += ['compiler-rt']
cleanBuildRequestedByProperty = lambda step: step.build.getProperty("clean", False) # If true, clean everything, including source dirs
cleanBuildRequested = lambda step: clean or step.build.getProperty("clean", default=step.build.getProperty("clean_obj")) def cleanBuildRequested(step):
return step.build.getProperty("clean")
# If true, clean build products; implied if cleanBuildRequested is true
def cleanObjRequested(step):
return cleanBuildRequested(step) or clean or step.build.getProperty("clean_obj")
f = LLVMBuildFactory( f = LLVMBuildFactory(
depends_on_projects=depends_on_projects, depends_on_projects=depends_on_projects,
llvm_srcdir=llvm_srcdir, llvm_srcdir=llvm_srcdir,
obj_dir=llvm_objdir, obj_dir=llvm_objdir,
install_dir=llvm_instdir, install_dir=llvm_instdir,
cleanBuildRequested=cleanBuildRequested, cleanBuildRequested=cleanBuildRequested,
**kwargs) # Pass through all the extra arguments. **kwargs) # Pass through all the extra arguments.
f.addStep(steps.RemoveDirectory(name='clean-src-dir', f.addStep(steps.RemoveDirectory(name='clean-src-dir',
dir=f.monorepo_dir, dir=f.monorepo_dir,
warnOnFailure=True, warnOnFailure=True,
doStepIf=cleanBuildRequestedByProperty)) doStepIf=cleanBuildRequested))
# Get the source code. # Get the source code.
f.addGetSourcecodeSteps(**kwargs) f.addGetSourcecodeSteps(**kwargs)
# Clean build dir # Clean build dir
f.addStep(steps.RemoveDirectory(name='clean-build-dir', f.addStep(steps.RemoveDirectory(name='clean-build-dir',
dir=llvm_objdir, dir=llvm_objdir,
warnOnFailure=True, warnOnFailure=True,
doStepIf=cleanBuildRequested)) doStepIf=cleanObjRequested))
# Create configuration files with cmake # Create configuration files with cmake
cmakeCommand = ["cmake", "../%s/llvm" % llvm_srcdir, cmakeCommand = ["cmake", "../%s/llvm" % llvm_srcdir,
"-DCMAKE_COLOR_MAKEFILE=OFF", "-DCMAKE_COLOR_MAKEFILE=OFF",
"-DPOLLY_TEST_DISABLE_BAR=ON", "-DPOLLY_TEST_DISABLE_BAR=ON",
"-DPOLLY_ENABLE_GPGPU_CODEGEN=ON",
"-DCMAKE_BUILD_TYPE=Release", "-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_POLLY_LINK_INTO_TOOLS=ON", "-DLLVM_POLLY_LINK_INTO_TOOLS=ON",
"-DLLVM_ENABLE_PROJECTS=%s" % ";".join(f.depends_on_projects), "-DLLVM_ENABLE_PROJECTS=%s" % ";".join(f.depends_on_projects),
] + cmake_install + extraCmakeArgs ] + cmake_install + extraCmakeArgs
f.addStep(ShellCommand(name="cmake-configure", f.addStep(ShellCommand(name="cmake-configure",
command=cmakeCommand, command=cmakeCommand,
haltOnFailure=False, haltOnFailure=False,
description=["cmake configure"], description=["cmake configure"],
workdir=llvm_objdir, workdir=llvm_objdir,
env=merged_env)) env=merged_env,
logfiles={
'CMakeCache.txt' : llvm_objdir + '/CMakeCache.txt',
}))
# Build # Build
f.addStep(WarningCountingShellCommand(name="build", f.addStep(WarningCountingShellCommand(name="build",
command=build_cmd, command=build_cmd,
haltOnFailure=True, haltOnFailure=True,
description=["build"], description=["build"],
workdir=llvm_objdir, workdir=llvm_objdir,
env=merged_env)) env=merged_env))
clangexe = "%(builddir)s/" + llvm_objdir + "/bin/clang" clangexe = "%(builddir)s/" + llvm_objdir + "/bin/clang"
clangxxexe = "%(builddir)s/" + llvm_objdir + "/bin/clang++" clangxxexe = "%(builddir)s/" + llvm_objdir + "/bin/clang++"
litexe = "%(builddir)s/" + llvm_objdir + "/bin/llvm-lit" litexe = "%(builddir)s/" + llvm_objdir + "/bin/llvm-lit"
sizeexe = "%(builddir)s/" + llvm_objdir + "/bin/llvm-size" sizeexe = "%(builddir)s/" + llvm_objdir + "/bin/llvm-size"
# Clean install dir # Clean install dir
if install: if install:
f.addStep(steps.RemoveDirectory(name='clean-install-dir', f.addStep(steps.RemoveDirectory(name='clean-install-dir',
dir=llvm_instdir, dir=llvm_instdir,
haltOnFailure=False, haltOnFailure=False,
doStepIf=cleanBuildRequested)) doStepIf=cleanObjRequested))
f.addStep(ShellCommand(name="install", f.addStep(ShellCommand(name="install",
command=install_cmd, command=install_cmd,
haltOnFailure=True, haltOnFailure=True,
description=["install"], description=["install"],
workdir=llvm_objdir, workdir=llvm_objdir,
env=merged_env)) env=merged_env))
Show All 18 Lines else:
workdir=llvm_objdir, workdir=llvm_objdir,
env=merged_env)) env=merged_env))
if testsuite: if testsuite:
f.addStep(steps.RemoveDirectory(name='test-suite_clean-src-dir', f.addStep(steps.RemoveDirectory(name='test-suite_clean-src-dir',
dir=testsuite_srcdir, dir=testsuite_srcdir,
haltOnFailure=False, haltOnFailure=False,
warnOnFailure=True, warnOnFailure=True,
doStepIf=cleanBuildRequestedByProperty)) doStepIf=cleanBuildRequested))
f.addGetSourcecodeForProject( f.addGetSourcecodeForProject(
project='test-suite', project='test-suite',
src_dir=testsuite_srcdir, src_dir=testsuite_srcdir,
alwaysUseLatest=True) alwaysUseLatest=True)
f.addStep(steps.RemoveDirectory(name='test-suite_clean-build-dir', f.addStep(steps.RemoveDirectory(name='test-suite_clean-build-dir',
dir=testsuite_builddir, dir=testsuite_builddir,
haltOnFailure=False, haltOnFailure=False,
warnOnFailure=True)) warnOnFailure=True))
# -Wno-unused-command-line-argument is needed because linking will not uses the "-mllvm -polly" argument. # -Wno-unused-command-line-argument is needed because linking will not use the "-mllvm -polly" argument.
f.addStep(ShellCommand(name='test-suite_cmake-configure', f.addStep(ShellCommand(name='test-suite_cmake-configure',
description=["Test-Suite: cmake"], description=["Test-Suite: cmake"],
command=["cmake", '-B', testsuite_builddir, '-S', testsuite_srcdir, command=["cmake", '-B', testsuite_builddir, '-S', testsuite_srcdir,
"-DCMAKE_BUILD_TYPE=Release", "-DCMAKE_BUILD_TYPE=Release",
"-DTEST_SUITE_COLLECT_STATS=ON",
"-DTEST_SUITE_EXTRA_C_FLAGS=-Wno-unused-command-line-argument -mllvm -polly", "-DTEST_SUITE_EXTRA_C_FLAGS=-Wno-unused-command-line-argument -mllvm -polly",
"-DTEST_SUITE_EXTRA_CXX_FLAGS=-mllvm -polly", "-DTEST_SUITE_EXTRA_CXX_FLAGS=-Wno-unused-command-line-argument -mllvm -polly",
"-DTEST_SUITE_LIT_FLAGS=-vv;-o;report.json", "-DTEST_SUITE_LIT_FLAGS=-vv;-o;report.json",
WithProperties("-DCMAKE_C_COMPILER=" + clangexe), WithProperties("-DCMAKE_C_COMPILER=" + clangexe),
WithProperties("-DCMAKE_CXX_COMPILER=" + clangxxexe), WithProperties("-DCMAKE_CXX_COMPILER=" + clangxxexe),
WithProperties("-DTEST_SUITE_LLVM_SIZE=" + sizeexe), WithProperties("-DTEST_SUITE_LLVM_SIZE=" + sizeexe),
WithProperties("-DTEST_SUITE_LIT=" + litexe), WithProperties("-DTEST_SUITE_LIT=" + litexe),
] + extraTestsuiteCmakeArgs, ] + extraTestsuiteCmakeArgs,
haltOnFailure=True, haltOnFailure=True,
workdir='.', workdir='.',
env=merged_env)) env=merged_env,
logfiles={
'CMakeCache.txt' : testsuite_builddir + '/CMakeCache.txt',
}))
f.addStep(WarningCountingShellCommand(name='test-suite_build', f.addStep(WarningCountingShellCommand(name='test-suite_build',
description=["Test-Suite: build"], description=["Test-Suite: build"],
# Continue building; programs that don't compile will fail with NOEXE. # Continue building; programs that don't compile will fail with NOEXE.
command=[make, 'all', '-k0'] + jobs_cmd, command=[make, 'all', '-k0'] + jobs_cmd,
haltOnFailure=False, haltOnFailure=False,
flunkOnFailure=True, flunkOnFailure=True,
workdir=testsuite_builddir, workdir=testsuite_builddir,
env=merged_env)) env=merged_env))
f.addStep(LitTestCommand(name='test-suite_run', f.addStep(LitTestCommand(name='test-suite_run',
description=['Test-Suite: run'], description=['Test-Suite: run'],
command=[WithProperties(litexe), '-vv', '-o', 'report.json', '.'], command=[WithProperties(litexe), '-vv', '-s', '-o', 'report.json', '.'],
haltOnFailure=True, haltOnFailure=True,
workdir=testsuite_builddir, workdir=testsuite_builddir,
logfiles={ logfiles={
'test.log' : 'test.log', 'test.log' : 'test.log',
'report.json': 'report.json' 'report.json': 'report.json'
}, },
env=merged_env)) env=merged_env))
return f return f

zorg/buildbot/builders/SanitizerBuilder.py

# TODO: Change WithProperties to Interpolate # TODO: Change WithProperties to Interpolate
from buildbot.process.properties import WithProperties from buildbot.process.properties import WithProperties
from zorg.buildbot.commands.AnnotatedCommand import AnnotatedCommand from zorg.buildbot.commands.AnnotatedCommand import AnnotatedCommand
from zorg.buildbot.process.factory import LLVMBuildFactory from zorg.buildbot.process.factory import LLVMBuildFactory
def getSanitizerBuildFactory( def getSanitizerBuildFactory(
clean=False, clean=False,
depends_on_projects=None, extra_depends_on_projects=[],
extra_configure_args=None,
env=None, env=None,
timeout=1200): timeout=1200):
# Prepare environmental variables. Set here all env we want everywhere. # Prepare environmental variables. Set here all env we want everywhere.
merged_env = { merged_env = {
"TERM" : "dumb", # Make sure Clang doesn't use color escape sequences. "TERM" : "dumb", # Make sure Clang doesn't use color escape sequences.
} }
# Clobber bot if we need a clean build. # Clobber bot if we need a clean build.
if clean: if clean:
merged_env["BUILDBOT_CLOBBER"] = "1" merged_env["BUILDBOT_CLOBBER"] = "1"
# Overwrite pre-set items with the given ones, so user can set anything. # Overwrite pre-set items with the given ones, so user can set anything.
if env is not None: if env is not None:
merged_env.update(env) merged_env.update(env)
if depends_on_projects is None:
depends_on_projects = [ depends_on_projects = [
"llvm", "llvm",
"clang", "clang",
"compiler-rt", "compiler-rt",
"libcxx", "libcxx",
"libcxxabi", "libcxxabi",
"libunwind", "libunwind",
"lld"] "lld",
] + extra_depends_on_projects
if extra_configure_args:
extra_configure_args = ['--'] + list(extra_configure_args)
else:
extra_configure_args = []
# Explicitly use '/' as separator, because it works on *nix and Windows. # Explicitly use '/' as separator, because it works on *nix and Windows.
sanitizer_script_dir = "sanitizer_buildbot/sanitizers" sanitizer_script_dir = "sanitizer_buildbot/sanitizers"
sanitizer_script = "../%s/zorg/buildbot/builders/sanitizers/%s" % (sanitizer_script_dir, "buildbot_selector.py") sanitizer_script = "../%s/zorg/buildbot/builders/sanitizers/%s" % (sanitizer_script_dir, "buildbot_selector.py")
f = LLVMBuildFactory( f = LLVMBuildFactory(
clean=clean, clean=clean,
depends_on_projects=depends_on_projects, depends_on_projects=depends_on_projects,
llvm_srcdir=sanitizer_script_dir) llvm_srcdir=sanitizer_script_dir)
# Get sanitizer buildbot scripts. # Get sanitizer buildbot scripts.
f.addGetSourcecodeForProject( f.addGetSourcecodeForProject(
name='update-annotate-scripts', name='update-annotate-scripts',
project='zorg', project='zorg',
src_dir=sanitizer_script_dir, src_dir=sanitizer_script_dir,
alwaysUseLatest=True) alwaysUseLatest=True)
# Run annotated command for sanitizer. # Run annotated command for sanitizer.
command = ['python', sanitizer_script] + extra_configure_args
f.addStep(AnnotatedCommand(name="annotate", f.addStep(AnnotatedCommand(name="annotate",
description="annotate", description="annotate",
timeout=timeout, timeout=timeout,
haltOnFailure=True, haltOnFailure=True,
command="python " + sanitizer_script, command=command,
env=merged_env)) env=merged_env))
return f return f

zorg/buildbot/builders/SphinxDocsBuilder.py

from collections import OrderedDict from collections import OrderedDict
from buildbot.steps.shell import ShellCommand from buildbot.steps.shell import ShellCommand
from zorg.buildbot.builders import UnifiedTreeBuilder from zorg.buildbot.builders import UnifiedTreeBuilder
from zorg.buildbot.commands.CmakeCommand import CmakeCommand from zorg.buildbot.commands.CmakeCommand import CmakeCommand
from zorg.buildbot.commands.NinjaCommand import NinjaCommand from zorg.buildbot.commands.NinjaCommand import NinjaCommand
llvm_docs = OrderedDict([ llvm_docs = OrderedDict([
# Project Build target Local path Remote path # Project Build target Local path Remote path
("llvm", ("docs-llvm-html", "docs/html/", "llvm")), ("llvm", ("docs-llvm-html", "docs/html/", "llvm")),
("clang", ("docs-clang-html", "tools/clang/docs/html/", "cfe")), ("clang", ("docs-clang-html", "tools/clang/docs/html/", "cfe")),
("clang-tools-extra", ("docs-clang-tools-html", "tools/clang/tools/extra/docs/html/", "clang-tools-extra")), ("clang-tools-extra", ("docs-clang-tools-html", "tools/clang/tools/extra/docs/html/", "clang-tools-extra")),
("libcxx", ("docs-libcxx-html", "projects/libcxx/docs/html/", "libcxx")), ("libc", ("docs-libc-html", "libc/docs/html/", "libc")),
("libunwind", ("docs-libunwind-html", "projects/libunwind/docs/html/", "libunwind")), ("libcxx", ("docs-libcxx-html", "libcxx/docs/html/", "libcxx")),
("libunwind", ("docs-libunwind-html", "libunwind/docs/html/", "libunwind")),
("lld", ("docs-lld-html", "tools/lld/docs/html/", "lld")), ("lld", ("docs-lld-html", "tools/lld/docs/html/", "lld")),
("lldb", ("docs-lldb-html", "tools/lldb/docs/html/", "lldb")), ("lldb", ("docs-lldb-html", "tools/lldb/docs/html/", "lldb")),
('flang', ("docs-flang-html", "tools/flang/docs/html/", "flang")), ('flang', ("docs-flang-html", "tools/flang/docs/html/", "flang")),
("openmp", ("docs-openmp-html", "projects/openmp/docs/html/", "openmp")), ("openmp", ("docs-openmp-html", "projects/openmp/docs/html/", "openmp")),
("polly", ("docs-polly-html", "tools/polly/docs/html/", "polly")), ("polly", ("docs-polly-html", "tools/polly/docs/html/", "polly")),
]) ])
def getSphinxDocsBuildFactory( def getSphinxDocsBuildFactory(
llvm_html = False, # Build LLVM HTML documentation llvm_html = False, # Build LLVM HTML documentation
llvm_man = False, # Build LLVM man pages llvm_man = False, # Build LLVM man pages
clang_html = False, # Build Clang HTML documentation clang_html = False, # Build Clang HTML documentation
clang_man = False, # Build Clang man pages
clang_tools_html = False, # Build Clang Extra Tools HTML documentation clang_tools_html = False, # Build Clang Extra Tools HTML documentation
lld_html = False, # Build LLD HTML documentation lld_html = False, # Build LLD HTML documentation
libcxx_html = False, # Build Libc++ HTML documentation
libunwind_html = False, # Build libunwind HTML documentation
lldb_html = False, # Build LLDB HTML documentation lldb_html = False, # Build LLDB HTML documentation
polly_html = False, # Build Polly HTML documentation
extra_configure_args = None, extra_configure_args = None,
**kwargs): **kwargs):
if extra_configure_args: if extra_configure_args:
cmake_args = extra_configure_args[:] cmake_args = extra_configure_args[:]
else: else:
cmake_args = list() cmake_args = list()
# Set proper defaults for the config flags. # Set proper defaults for the config flags.
CmakeCommand.applyDefaultOptions(cmake_args, [ CmakeCommand.applyDefaultOptions(cmake_args, [
('-G', 'Ninja'), ('-G', 'Ninja'),
('-DLLVM_ENABLE_SPHINX=', 'ON'), ('-DLLVM_ENABLE_SPHINX=', 'ON'),
('-DSPHINX_OUTPUT_HTML=', 'ON'), ('-DSPHINX_OUTPUT_HTML=', 'ON'),
('-DSPHINX_OUTPUT_MAN=', 'ON'), ('-DSPHINX_OUTPUT_MAN=', 'ON'),
('-DLLDB_INCLUDE_TESTS=', 'OFF'), ('-DLLDB_INCLUDE_TESTS=', 'OFF'),
('-DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=', 'ON'), ('-DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=', 'ON'),
('-DLLVM_ENABLE_ASSERTIONS=', 'OFF'), ('-DLLVM_ENABLE_ASSERTIONS=', 'OFF'),
]) ])
llvm_srcdir = 'llvm/src' llvm_srcdir = 'llvm/src'
llvm_objdir = 'llvm/build' llvm_objdir = 'llvm/build'
depends_on_projects = ['llvm'] depends_on_projects = ['llvm']
if clang_html or clang_tools_html or lldb_html: if clang_html or clang_man or clang_tools_html or lldb_html:
depends_on_projects.append('clang') depends_on_projects.append('clang')
if clang_tools_html: if clang_tools_html:
depends_on_projects.append('clang-tools-extra') depends_on_projects.append('clang-tools-extra')
if lld_html: if lld_html:
depends_on_projects.append('lld') depends_on_projects.append('lld')
if lldb_html: if lldb_html:
depends_on_projects.append('lldb') depends_on_projects.append('lldb')
if libcxx_html: if polly_html:
depends_on_projects.append('libcxx') depends_on_projects.append('polly')
depends_on_projects.append('libcxxabi')
if libunwind_html:
depends_on_projects.append('libunwind')
f = UnifiedTreeBuilder.getCmakeBuildFactory( f = UnifiedTreeBuilder.getCmakeBuildFactory(
depends_on_projects=depends_on_projects, depends_on_projects=depends_on_projects,
llvm_srcdir=llvm_srcdir, llvm_srcdir=llvm_srcdir,
obj_dir=llvm_objdir, obj_dir=llvm_objdir,
extra_configure_args=cmake_args, extra_configure_args=cmake_args,
**kwargs) # Pass through all the extra arguments. **kwargs) # Pass through all the extra arguments.
Show All 16 Linesdef getSphinxDocsBuildFactory(
if clang_html: if clang_html:
f.addStep(NinjaCommand(name="docs-clang-html", f.addStep(NinjaCommand(name="docs-clang-html",
haltOnFailure=True, haltOnFailure=True,
description=["Build Clang Sphinx HTML documentation"], description=["Build Clang Sphinx HTML documentation"],
workdir=llvm_objdir, workdir=llvm_objdir,
targets=['docs-clang-html'] targets=['docs-clang-html']
)) ))
if clang_man:
f.addStep(NinjaCommand(name="docs-clang-man",
haltOnFailure=True,
description=["Build Clang Sphinx man pages"],
workdir=llvm_objdir,
targets=['docs-clang-man']
))
if clang_tools_html: if clang_tools_html:
f.addStep(NinjaCommand(name="docs-clang-tools-html", f.addStep(NinjaCommand(name="docs-clang-tools-html",
haltOnFailure=True, haltOnFailure=True,
description=["Build Clang Extra Tools Sphinx HTML documentation"], description=["Build Clang Extra Tools Sphinx HTML documentation"],
workdir=llvm_objdir, workdir=llvm_objdir,
targets=['docs-clang-tools-html'] targets=['docs-clang-tools-html']
)) ))
if lld_html: if lld_html:
f.addStep(NinjaCommand(name="docs-lld-html", f.addStep(NinjaCommand(name="docs-lld-html",
haltOnFailure=True, haltOnFailure=True,
description=["Build LLD Sphinx HTML documentation"], description=["Build LLD Sphinx HTML documentation"],
workdir=llvm_objdir, workdir=llvm_objdir,
targets=['docs-lld-html'] targets=['docs-lld-html']
)) ))
if lldb_html: if lldb_html:
f.addStep(NinjaCommand(name="docs-lldb-html", f.addStep(NinjaCommand(name="docs-lldb-html",
haltOnFailure=True, haltOnFailure=True,
description=["Build LLDB Sphinx HTML documentation"], description=["Build LLDB Sphinx HTML documentation"],
workdir=llvm_objdir, workdir=llvm_objdir,
targets=['docs-lldb-html'] targets=['docs-lldb-html']
)) ))
if polly_html:
f.addStep(NinjaCommand(name="docs-polly-html",
haltOnFailure=True,
description=["Build Polly Sphinx HTML documentation"],
workdir=llvm_objdir,
targets=['docs-polly-html']
))
return f
def getSphinxRuntimesDocsBuildFactory(
libcxx_html = False, # Build Libc++ HTML documentation
libunwind_html = False, # Build libunwind HTML documentation
libc_html = False, # Build Libc HTML documentation
extra_configure_args = None,
**kwargs):
if extra_configure_args:
cmake_args = extra_configure_args[:]
else:
cmake_args = list()
# Set proper defaults for the config flags.
CmakeCommand.applyDefaultOptions(cmake_args, [
('-G', 'Ninja'),
('-DLLVM_ENABLE_SPHINX=', 'ON'),
('-DSPHINX_OUTPUT_HTML=', 'ON'),
('-DSPHINX_OUTPUT_MAN=', 'ON'),
('-DLLDB_INCLUDE_TESTS=', 'OFF'),
('-DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=', 'ON'),
('-DLLVM_ENABLE_ASSERTIONS=', 'OFF'),
])
depends_on_runtimes = []
if libcxx_html:
depends_on_runtimes.append('libcxx')
depends_on_runtimes.append('libcxxabi')
if libunwind_html:
depends_on_runtimes.append('libunwind')
if libc_html:
depends_on_runtimes.append('libc')
f = UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps(
depends_on_projects=depends_on_runtimes,
enable_runtimes=depends_on_runtimes,
src_to_build_dir='runtimes',
**kwargs) # Pass through all the extra arguments.
UnifiedTreeBuilder.addCmakeSteps(
f,
cleanBuildRequested=f.cleanBuildRequested,
obj_dir=f.obj_dir,
install_dir=f.install_dir,
extra_configure_args=cmake_args,
**kwargs)
if libcxx_html: if libcxx_html:
f.addStep(NinjaCommand(name="docs-libcxx-html", f.addStep(NinjaCommand(name="docs-libcxx-html",
haltOnFailure=True, haltOnFailure=True,
description=["Build Libc++ Sphinx HTML documentation"], description=["Build Libc++ Sphinx HTML documentation"],
workdir=llvm_objdir, workdir=f.obj_dir,
targets=['docs-libcxx-html'] targets=['docs-libcxx-html']
)) ))
if libunwind_html: if libunwind_html:
f.addStep(NinjaCommand(name="docs-libunwind-html", f.addStep(NinjaCommand(name="docs-libunwind-html",
haltOnFailure=True, haltOnFailure=True,
description=["Build libunwind Sphinx HTML documentation"], description=["Build libunwind Sphinx HTML documentation"],
workdir=llvm_objdir, workdir=f.obj_dir,
targets=['docs-libunwind-html'] targets=['docs-libunwind-html']
)) ))
if libc_html:
f.addStep(NinjaCommand(name="docs-libc-html",
haltOnFailure=True,
description=["Build libc Sphinx HTML documentation"],
workdir=f.obj_dir,
targets=['docs-libc-html']
))
return f return f
def getLLVMDocsBuildFactory( def getLLVMDocsBuildFactory(
clean = False, clean = False,
depends_on_projects = None, depends_on_projects = None,
extra_configure_args = None, extra_configure_args = None,
env = None, env = None,
**kwargs): **kwargs):
if depends_on_projects is None: if depends_on_projects is None:
# All the projects by default. # All the projects by default.
_depends_on_projects=[ _depends_on_projects=[
"llvm", "llvm",
"clang", "clang",
"clang-tools-extra", "clang-tools-extra",
"libcxx",
"libcxxabi",
"libunwind",
"lld", "lld",
"lldb", "lldb",
"flang", "flang",
"openmp", "openmp",
"polly", "polly",
] ]
else: else:
# Make a local copy of depends_on_projects, as we are going to modify # Make a local copy of depends_on_projects, as we are going to modify
# that. # that.
_depends_on_projects=depends_on_projects[:] _depends_on_projects=depends_on_projects[:]
# Some projects are interdependent for the purpose of documentation. # Some projects are interdependent for the purpose of documentation.
# Enforce the dependencies. # Enforce the dependencies.
if ("clang-tools-extra" in _depends_on_projects or \ if ("clang-tools-extra" in _depends_on_projects or \
"lldb" in _depends_on_projects "lldb" in _depends_on_projects
) and "clang" not in _depends_on_projects: ) and "clang" not in _depends_on_projects:
_depends_on_projects.append("clang") _depends_on_projects.append("clang")
if "libcxx" in _depends_on_projects and \
"libcxxabi" not in _depends_on_projects:
_depends_on_projects.append("libcxxabi")
if "libcxxabi" in _depends_on_projects and \
"libcxx" not in _depends_on_projects:
_depends_on_projects.append("libcxx")
# Make a local copy of the configure args, as we are going to modify that. # Make a local copy of the configure args, as we are going to modify that.
if extra_configure_args: if extra_configure_args:
cmake_args = extra_configure_args[:] cmake_args = extra_configure_args[:]
else: else:
cmake_args = list() cmake_args = list()
# Prepare environmental variables. Set here all env we want everywhere. # Prepare environmental variables. Set here all env we want everywhere.
▲ Show 20 Lines • Show All 53 Lines • ▼ Show 20 Lines for target, local_path, remote_path in docs:
"{}".format(local_path), "{}".format(local_path),
"lists.llvm.org:web/{}".format(remote_path), "lists.llvm.org:web/{}".format(remote_path),
], ],
env=merged_env, env=merged_env,
) )
) )
return f return f
def getLLVMRuntimesDocsBuildFactory(
clean = False,
depends_on_runtimes = None,
extra_configure_args = None,
env = None,
**kwargs):
if depends_on_runtimes is None:
# All the projects by default.
_depends_on_runtimes=[
"libcxx",
"libcxxabi",
"libunwind",
"libc",
]
else:
# Make a local copy of depends_on_runtimes, as we are going to modify
# that.
_depends_on_runtimes=depends_on_runtimes[:]
# Some runtimes are interdependent for the purpose of documentation.
# Enforce the dependencies.
if "libcxx" in _depends_on_runtimes and \
"libcxxabi" not in _depends_on_runtimes:
_depends_on_runtimes.append("libcxxabi")
if "libcxxabi" in _depends_on_runtimes and \
"libcxx" not in _depends_on_runtimes:
_depends_on_runtimes.append("libcxx")
# Make a local copy of the configure args, as we are going to modify that.
if extra_configure_args:
cmake_args = extra_configure_args[:]
else:
cmake_args = list()
# Prepare environmental variables. Set here all env we want everywhere.
merged_env = {
'TERM' : 'dumb' # Be cautious and disable color output from all tools.
}
if env is not None:
# Overwrite pre-set items with the given ones, so user can set anything.
merged_env.update(env)
CmakeCommand.applyDefaultOptions(cmake_args, [
("-G", "Ninja"),
("-DLLVM_ENABLE_SPHINX=", "ON"),
("-DSPHINX_OUTPUT_HTML=", "ON"),
("-DSPHINX_OUTPUT_MAN=", "OFF"),
("-DSPHINX_WARNINGS_AS_ERRORS=", "OFF"),
("-DLLVM_ENABLE_ASSERTIONS=", "OFF"),
("-DCMAKE_BUILD_TYPE=", "Release"),
])
# Build docs for each of the runtimes this builder depends on
docs = [
llvm_docs[project] for project in llvm_docs.keys()
if project in _depends_on_runtimes
]
cleanBuildRequested = lambda step: step.build.getProperty("clean", default=step.build.getProperty("clean_obj")) or clean
f = UnifiedTreeBuilder.getLLVMBuildFactoryAndSourcecodeSteps(
depends_on_projects=_depends_on_runtimes,
enable_runtimes=_depends_on_runtimes,
src_to_build_dir='runtimes',
cleanBuildRequested=cleanBuildRequested,
**kwargs) # Pass through all the extra arguments.
UnifiedTreeBuilder.addCmakeSteps(
f,
cleanBuildRequested=cleanBuildRequested,
obj_dir=f.obj_dir,
install_dir=f.install_dir,
extra_configure_args=cmake_args,
env=merged_env,
**kwargs)
UnifiedTreeBuilder.addNinjaSteps(
f,
targets=[d[0] for d in docs],
checks=[],
env=merged_env,
**kwargs)
# Publish just built documentation
for target, local_path, remote_path in docs:
f.addStep(
ShellCommand(
name="Publish {}".format(target),
description=[
"Publish", "just", "built", "documentation", "for",
"{}".format(target),
],
command=[
'rsync',
'-vrl',
'--delete', '--force', '--delay-updates', '--delete-delay',
'--ignore-times',
'--checksum',
'-p', '--chmod=Du=rwx,Dg=rwx,Do=rx,Fu=rw,Fg=rw,Fo=r',
"{}".format(local_path),
"lists.llvm.org:web/{}".format(remote_path),
],
env=merged_env,
)
)
return f

zorg/buildbot/builders/TestSuiteBuilder.py

  • This file was added.
from zorg.buildbot.builders.UnifiedTreeBuilder import getCmakeWithNinjaBuildFactory
from buildbot.plugins import util
from buildbot.steps.shell import ShellCommand
from zorg.buildbot.commands.CmakeCommand import CmakeCommand
from zorg.buildbot.commands.NinjaCommand import NinjaCommand
from zorg.buildbot.commands.LitTestCommand import LitTestCommand
# This builder is uses UnifiedTreeBuilders and adds running
# llvm-test-suite with cmake and ninja step.
def addTestSuiteStep(
f,
compiler_dir = None,
env = None,
lit_args = [],
**kwargs):
cc = util.Interpolate('-DCMAKE_C_COMPILER=' + '%(prop:builddir)s/'+compiler_dir+'/bin/clang')
cxx = util.Interpolate('-DCMAKE_CXX_COMPILER=' + '%(prop:builddir)s/'+compiler_dir+'/bin/clang++')
lit = util.Interpolate('%(prop:builddir)s/' + compiler_dir + '/bin/llvm-lit')
test_suite_base_dir = util.Interpolate('%(prop:builddir)s/' + 'test')
test_suite_src_dir = util.Interpolate('%(prop:builddir)s/' + 'test/test-suite')
test_suite_workdir = util.Interpolate('%(prop:builddir)s/' + 'test/build-test-suite')
cmake_lit_arg = util.Interpolate('-DTEST_SUITE_LIT:FILEPATH=' + '%(prop:builddir)s/' + compiler_dir + '/bin/llvm-lit')
# used for cmake building test-suite step
options = [cc, cxx, cmake_lit_arg]
# always clobber the build directory to test each new compiler
f.addStep(ShellCommand(name='Clean Test Suite Build dir',
command=['rm', '-rf', test_suite_workdir],
haltOnFailure=True,
description='Removing the Test Suite build directory',
workdir=test_suite_base_dir,
env=env))
f.addGetSourcecodeForProject(
project='test-suite',
src_dir=test_suite_src_dir,
alwaysUseLatest=True)
f.addStep(CmakeCommand(name='cmake Test Suite',
haltOnFailure=True,
description='Running cmake on Test Suite dir',
workdir=test_suite_workdir,
options=options,
path=test_suite_src_dir,
generator='Ninja'))
f.addStep(NinjaCommand(name='ninja Test Suite',
description='Running Ninja on Test Suite dir',
haltOnFailure=True,
workdir=test_suite_workdir))
f.addStep(LitTestCommand(name='Run Test Suite with lit',
haltOnFailure=True,
description='Running test suite tests',
workdir=test_suite_workdir,
command=[lit] + lit_args + ['.'],
env=env,
**kwargs))
return f
def getTestSuiteBuildFactory(
depends_on_projects = None,
enable_runtimes = "auto",
targets = None,
llvm_srcdir = None,
obj_dir = None,
checks = None,
install_dir = None,
clean = False,
extra_configure_args = None,
env = None,
**kwargs):
# handle the -DCMAKE args for lit
lit_args = list()
if any("DLLVM_LIT_ARGS" in arg for arg in extra_configure_args):
arg = [arg for arg in extra_configure_args
if "DLLVM_LIT_ARGS" in arg][0]
lit_args = arg.split("=")[1]
lit_args = lit_args.split(" ")
f = getCmakeWithNinjaBuildFactory(
depends_on_projects = depends_on_projects,
enable_runtimes = enable_runtimes,
targets = targets,
llvm_srcdir = llvm_srcdir,
obj_dir = obj_dir,
checks = checks,
install_dir = install_dir,
clean = clean,
extra_configure_args = extra_configure_args,
env = env,
**kwargs)
addTestSuiteStep(f,
compiler_dir=f.obj_dir,
env=env,
lit_args=lit_args,
**kwargs)
return f

zorg/buildbot/builders/UnifiedTreeBuilder.py

from buildbot.plugins import steps, util from buildbot.plugins import steps, util
from buildbot.steps.shell import SetPropertyFromCommand from buildbot.steps.shell import SetPropertyFromCommand
from buildbot.process.properties import WithProperties from buildbot.process.properties import WithProperties
from zorg.buildbot.commands.CmakeCommand import CmakeCommand from zorg.buildbot.commands.CmakeCommand import CmakeCommand
from zorg.buildbot.commands.NinjaCommand import NinjaCommand from zorg.buildbot.commands.NinjaCommand import NinjaCommand
from zorg.buildbot.commands.LitTestCommand import LitTestCommand from zorg.buildbot.commands.LitTestCommand import LitTestCommand
from zorg.buildbot.conditions.FileConditions import FileDoesNotExist from zorg.buildbot.conditions.FileConditions import FileDoesNotExist
from zorg.buildbot.process.factory import LLVMBuildFactory from zorg.buildbot.process.factory import LLVMBuildFactory
import zorg.buildbot.builders.Util as builders_util import zorg.buildbot.builders.Util as builders_util
def getLLVMBuildFactoryAndPrepareForSourcecodeSteps( def getLLVMBuildFactoryAndPrepareForSourcecodeSteps(
depends_on_projects = None, depends_on_projects = None,
enable_runtimes = "auto",
llvm_srcdir = None, llvm_srcdir = None,
obj_dir = None, obj_dir = None,
install_dir = None, install_dir = None,
cleanBuildRequested = None, cleanBuildRequested = None,
env = None, env = None,
**kwargs): **kwargs):
def cleanBuildRequestedByProperty(step): def cleanBuildRequestedByProperty(step):
return step.build.getProperty("clean") return step.build.getProperty("clean")
if cleanBuildRequested is None: if cleanBuildRequested is None:
# We want a clean checkout only if requested by the property. # We want a clean checkout only if requested by the property.
cleanBuildRequested = cleanBuildRequestedByProperty cleanBuildRequested = cleanBuildRequestedByProperty
f = LLVMBuildFactory( f = LLVMBuildFactory(
depends_on_projects=depends_on_projects, depends_on_projects=depends_on_projects,
enable_runtimes=enable_runtimes,
llvm_srcdir=llvm_srcdir, llvm_srcdir=llvm_srcdir,
obj_dir=obj_dir, obj_dir=obj_dir,
install_dir=install_dir, install_dir=install_dir,
cleanBuildRequested=cleanBuildRequested, cleanBuildRequested=cleanBuildRequested,
**kwargs) # Pass through all the extra arguments. **kwargs) # Pass through all the extra arguments.
# Remove the source code for a clean checkout if requested by property. # Remove the source code for a clean checkout if requested by property.
# TODO: Some Windows workers do not handle RemoveDirectory command well. # TODO: Some Windows workers do not handle RemoveDirectory command well.
# So, consider running "rmdir /S /Q <dir>" if the build runs on Windows. # So, consider running "rmdir /S /Q <dir>" if the build runs on Windows.
f.addStep(steps.RemoveDirectory(name='clean-src-dir', f.addStep(steps.RemoveDirectory(name='clean-src-dir',
dir=f.monorepo_dir, dir=f.monorepo_dir,
haltOnFailure=False, haltOnFailure=False,
flunkOnFailure=False, flunkOnFailure=False,
doStepIf=cleanBuildRequestedByProperty, doStepIf=cleanBuildRequestedByProperty,
)) ))
return f return f
def getLLVMBuildFactoryAndSourcecodeSteps( def getLLVMBuildFactoryAndSourcecodeSteps(
depends_on_projects = None, depends_on_projects = None,
enable_runtimes = "auto",
llvm_srcdir = None, llvm_srcdir = None,
src_to_build_dir = None,
obj_dir = None, obj_dir = None,
install_dir = None, install_dir = None,
cleanBuildRequested = None, cleanBuildRequested = None,
**kwargs): **kwargs):
f = getLLVMBuildFactoryAndPrepareForSourcecodeSteps( f = getLLVMBuildFactoryAndPrepareForSourcecodeSteps(
depends_on_projects=depends_on_projects, depends_on_projects=depends_on_projects,
enable_runtimes=enable_runtimes,
llvm_srcdir=llvm_srcdir, llvm_srcdir=llvm_srcdir,
src_to_build_dir=src_to_build_dir,
obj_dir=obj_dir, obj_dir=obj_dir,
install_dir=install_dir, install_dir=install_dir,
cleanBuildRequested=cleanBuildRequested, cleanBuildRequested=cleanBuildRequested,
**kwargs) # Pass through all the extra arguments. **kwargs) # Pass through all the extra arguments.
# Get the source code. # Get the source code.
f.addGetSourcecodeSteps(**kwargs) f.addGetSourcecodeSteps(**kwargs)
Show All 25 Linesdef addCmakeSteps(
# So, consider running "rmdir /S /Q <dir>" if the build runs on Windows. # So, consider running "rmdir /S /Q <dir>" if the build runs on Windows.
f.addStep(steps.RemoveDirectory(name='clean-%s-dir' % obj_dir, f.addStep(steps.RemoveDirectory(name='clean-%s-dir' % obj_dir,
dir=obj_dir, dir=obj_dir,
haltOnFailure=False, haltOnFailure=False,
flunkOnFailure=False, flunkOnFailure=False,
doStepIf=cleanBuildRequested, doStepIf=cleanBuildRequested,
)) ))
if f.enable_projects:
CmakeCommand.applyDefaultOptions(cmake_args, [ CmakeCommand.applyDefaultOptions(cmake_args, [
('-DLLVM_ENABLE_PROJECTS=', ";".join(f.depends_on_projects)), ('-DLLVM_ENABLE_PROJECTS=', ";".join(f.enable_projects)),
])
if f.enable_runtimes:
CmakeCommand.applyDefaultOptions(cmake_args, [
('-DLLVM_ENABLE_RUNTIMES=', ";".join(f.enable_runtimes)),
]) ])
if install_dir: if install_dir:
install_dir_rel = LLVMBuildFactory.pathRelativeTo( install_dir_rel = LLVMBuildFactory.pathRelativeTo(
install_dir, install_dir,
obj_dir) obj_dir)
CmakeCommand.applyRequiredOptions(cmake_args, [ CmakeCommand.applyRequiredOptions(cmake_args, [
('-DCMAKE_INSTALL_PREFIX=', install_dir_rel), ('-DCMAKE_INSTALL_PREFIX=', install_dir_rel),
]) ])
▲ Show 20 Lines • Show All 88 Lines • ▼ Show 20 Linesdef addNinjaSteps(
# Note: At this point env could be None, a dictionary, or a Property object. # Note: At this point env could be None, a dictionary, or a Property object.
if isinstance(env, dict): if isinstance(env, dict):
check_env = env.copy() if env else dict() check_env = env.copy() if env else dict()
check_env['NINJA_STATUS'] = check_env.get('NINJA_STATUS', "%e [%u/%r/%f] ") check_env['NINJA_STATUS'] = check_env.get('NINJA_STATUS', "%e [%u/%r/%f] ")
else: else:
check_env = env or {} check_env = env or {}
for check in checks: for check in checks:
f.addStep(LitTestCommand(name="test-%s%s" % (step_name, check), f.addStep(LitTestCommand(name="test-%s-%s" % (step_name, check),
command=['ninja', check], command=['ninja', check],
description=[ description=[
"Test", "just", "built", "components", "for", "Test", "just", "built", "components", "for",
check, check,
], ],
env=check_env, env=check_env,
workdir=obj_dir, workdir=obj_dir,
**kwargs # Pass through all the extra arguments. **kwargs # Pass through all the extra arguments.
)) ))
# Install just built components # Install just built components
if install_dir: if install_dir:
# TODO: Run this step only if none of the prevous failed. # TODO: Run this step only if none of the prevous failed.
f.addStep(NinjaCommand(name="install-%sall" % step_name, f.addStep(NinjaCommand(name="install-%sall" % step_name,
targets=["install"], targets=["install"],
description=["Install", "just", "built", "components"], description=["Install", "just", "built", "components"],
env=env or {}, env=env or {},
workdir=obj_dir, workdir=obj_dir,
**kwargs # Pass through all the extra arguments. **kwargs # Pass through all the extra arguments.
)) ))
def getCmakeBuildFactory( def getCmakeBuildFactory(
depends_on_projects = None, depends_on_projects = None,
enable_runtimes = "auto",
llvm_srcdir = None, llvm_srcdir = None,
obj_dir = None, obj_dir = None,
install_dir = None, install_dir = None,
clean = False, clean = False,
extra_configure_args = None, extra_configure_args = None,
env = None, env = None,
**kwargs): **kwargs):
f = getLLVMBuildFactoryAndSourcecodeSteps( f = getLLVMBuildFactoryAndSourcecodeSteps(
depends_on_projects=depends_on_projects, depends_on_projects=depends_on_projects,
enable_runtimes=enable_runtimes,
llvm_srcdir=llvm_srcdir, llvm_srcdir=llvm_srcdir,
obj_dir=obj_dir, obj_dir=obj_dir,
install_dir=install_dir, install_dir=install_dir,
**kwargs) # Pass through all the extra arguments. **kwargs) # Pass through all the extra arguments.
cleanBuildRequested = lambda step: step.build.getProperty("clean", default=step.build.getProperty("clean_obj")) or clean cleanBuildRequested = lambda step: step.build.getProperty("clean", default=step.build.getProperty("clean_obj")) or clean
addCmakeSteps( addCmakeSteps(
f, f,
cleanBuildRequested=cleanBuildRequested, cleanBuildRequested=cleanBuildRequested,
obj_dir=f.obj_dir, obj_dir=f.obj_dir,
install_dir=f.install_dir, install_dir=f.install_dir,
extra_configure_args=extra_configure_args, extra_configure_args=extra_configure_args,
env=env, env=env,
**kwargs) **kwargs)
return f return f
def getCmakeWithNinjaBuildFactory( def getCmakeWithNinjaBuildFactory(
depends_on_projects = None, depends_on_projects = None,
enable_runtimes = "auto",
targets = None, targets = None,
llvm_srcdir = None, llvm_srcdir = None,
obj_dir = None, obj_dir = None,
checks = None, checks = None,
install_dir = None, install_dir = None,
clean = False, clean = False,
extra_configure_args = None, extra_configure_args = None,
env = None, env = None,
Show All 18 Linesdef getCmakeWithNinjaBuildFactory(
# Some options are required for this build no matter what. # Some options are required for this build no matter what.
CmakeCommand.applyRequiredOptions(cmake_args, [ CmakeCommand.applyRequiredOptions(cmake_args, [
('-G', 'Ninja'), ('-G', 'Ninja'),
]) ])
f = getCmakeBuildFactory( f = getCmakeBuildFactory(
depends_on_projects=depends_on_projects, depends_on_projects=depends_on_projects,
enable_runtimes=enable_runtimes,
llvm_srcdir=llvm_srcdir, llvm_srcdir=llvm_srcdir,
obj_dir=obj_dir, obj_dir=obj_dir,
install_dir=install_dir, install_dir=install_dir,
clean=clean, clean=clean,
extra_configure_args=cmake_args, extra_configure_args=cmake_args,
env=merged_env, env=merged_env,
**kwargs) # Pass through all the extra arguments. **kwargs) # Pass through all the extra arguments.
addNinjaSteps( addNinjaSteps(
f, f,
obj_dir=f.obj_dir, obj_dir=f.obj_dir,
targets=targets, targets=targets,
checks=checks, checks=checks,
install_dir=f.install_dir, install_dir=f.install_dir,
env=merged_env, env=merged_env,
**kwargs) **kwargs)
return f return f
def getCmakeWithNinjaWithMSVCBuildFactory( def getCmakeWithNinjaWithMSVCBuildFactory(
depends_on_projects = None, depends_on_projects = None,
enable_runtimes = "auto",
targets = None, targets = None,
llvm_srcdir = None, llvm_srcdir = None,
obj_dir = None, obj_dir = None,
checks = None, checks = None,
install_dir = None, install_dir = None,
clean = False, clean = False,
extra_configure_args = None, extra_configure_args = None,
# VS tools environment variable if using MSVC. For example, # VS tools environment variable if using MSVC. For example,
# %VS140COMNTOOLS% selects the 2015 toolchain. # %VS140COMNTOOLS% selects the 2015 toolchain.
vs=None, vs=None,
target_arch=None, target_arch=None,
env = None, env = None,
**kwargs): **kwargs):
assert not env, "Can't have custom builder env vars with MSVC build"
# Make a local copy of the configure args, as we are going to modify that. # Make a local copy of the configure args, as we are going to modify that.
if extra_configure_args: if extra_configure_args:
cmake_args = extra_configure_args[:] cmake_args = extra_configure_args[:]
else: else:
cmake_args = list() cmake_args = list()
if checks is None: if checks is None:
checks = ['check-all'] checks = ['check-all']
f = getLLVMBuildFactoryAndSourcecodeSteps( f = getLLVMBuildFactoryAndSourcecodeSteps(
depends_on_projects=depends_on_projects, depends_on_projects=depends_on_projects,
enable_runtimes=enable_runtimes,
llvm_srcdir=llvm_srcdir, llvm_srcdir=llvm_srcdir,
obj_dir=obj_dir, obj_dir=obj_dir,
install_dir=install_dir, install_dir=install_dir,
**kwargs) # Pass through all the extra arguments. **kwargs) # Pass through all the extra arguments.
f.addStep(SetPropertyFromCommand( f.addStep(SetPropertyFromCommand(
command=builders_util.getVisualStudioEnvironment(vs, target_arch), command=builders_util.getVisualStudioEnvironment(vs, target_arch),
extract_fn=builders_util.extractVSEnvironment)) extract_fn=builders_util.extractVSEnvironment,
env=env))
env = util.Property('vs_env') env = util.Property('vs_env')
cleanBuildRequested = lambda step: step.build.getProperty("clean", default=step.build.getProperty("clean_obj")) or clean cleanBuildRequested = lambda step: step.build.getProperty("clean", default=step.build.getProperty("clean_obj")) or clean
addCmakeSteps( addCmakeSteps(
f, f,
generator='Ninja', generator='Ninja',
cleanBuildRequested=cleanBuildRequested, cleanBuildRequested=cleanBuildRequested,
Show All 11 Lines addNinjaSteps(
install_dir=f.install_dir, install_dir=f.install_dir,
env=env, env=env,
**kwargs) **kwargs)
return f return f
def getCmakeWithNinjaMultistageBuildFactory( def getCmakeWithNinjaMultistageBuildFactory(
depends_on_projects = None, depends_on_projects = None,
enable_runtimes = "auto",
llvm_srcdir = None, llvm_srcdir = None,
src_to_build_dir = None,
obj_dir = None, obj_dir = None,
checks = None, checks = None,
install_dir = None, install_dir = None,
clean = False, clean = False,
extra_configure_args = None, extra_configure_args = None,
env = None, env = None,
stages=2, stages=2,
stage_names=None, stage_names=None,
Show All 32 Linesdef getCmakeWithNinjaMultistageBuildFactory(
stage_objdirs = list() stage_objdirs = list()
stage_installdirs = list() stage_installdirs = list()
for s in stage_names: for s in stage_names:
stage_objdirs.append("%s/%s" % (obj_dir, s)) stage_objdirs.append("%s/%s" % (obj_dir, s))
stage_installdirs.append("%s/%s" % (install_dir, s)) stage_installdirs.append("%s/%s" % (install_dir, s))
f = getLLVMBuildFactoryAndPrepareForSourcecodeSteps( f = getLLVMBuildFactoryAndPrepareForSourcecodeSteps(
depends_on_projects=depends_on_projects, depends_on_projects=depends_on_projects,
enable_runtimes=enable_runtimes,
llvm_srcdir=llvm_srcdir, llvm_srcdir=llvm_srcdir,
src_to_build_dir=src_to_build_dir,
obj_dir=obj_dir, obj_dir=obj_dir,
install_dir=install_dir, install_dir=install_dir,
env=merged_env, env=merged_env,
stage_objdirs=stage_objdirs, stage_objdirs=stage_objdirs,
stage_installdirs=stage_installdirs, stage_installdirs=stage_installdirs,
stage_names=stage_names, stage_names=stage_names,
**kwargs) # Pass through all the extra arguments. **kwargs) # Pass through all the extra arguments.
▲ Show 20 Lines • Show All 104 Lines • Show Last 20 Lines

zorg/buildbot/builders/XToolchainBuilder.py

Show All 34 Linesdef getCmakeWithMSVCBuildFactory(
if install_dir is None: if install_dir is None:
install_dir = 'install' install_dir = 'install'
# Prepare environmental variables. Set here all env we want for all steps. # Prepare environmental variables. Set here all env we want for all steps.
merged_env = { merged_env = {
'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences. 'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences.
} }
if env is not None: if env is not None:
assert not vs, "Cannot have custom builder env vars with VS setup."
# Overwrite pre-set items with the given ones, so user can set anything. # Overwrite pre-set items with the given ones, so user can set anything.
merged_env.update(env) merged_env.update(env)
# Make a local copy of the configure args, as we are going to modify that. # Make a local copy of the configure args, as we are going to modify that.
if extra_configure_args: if extra_configure_args:
cmake_args = extra_configure_args[:] cmake_args = extra_configure_args[:]
else: else:
cmake_args = list() cmake_args = list()
▲ Show 20 Lines • Show All 60 Lines • ▼ Show 20 Lines f.addStep(ShellCommand(name='clean-%s-dir' % f.obj_dir,
workdir='.', workdir='.',
env=merged_env, env=merged_env,
doStepIf=clean_build_requested)) doStepIf=clean_build_requested))
if vs: if vs:
# Configure MSVC environment if requested. # Configure MSVC environment if requested.
f.addStep(SetPropertyFromCommand( f.addStep(SetPropertyFromCommand(
command=builders_util.getVisualStudioEnvironment(vs, None), command=builders_util.getVisualStudioEnvironment(vs, None),
extract_fn=builders_util.extractVSEnvironment)) extract_fn=builders_util.extractVSEnvironment,
env=merged_env))
merged_env = Property('vs_env') merged_env = Property('vs_env')
# Since this is a build of a cross toolchain, we build only the host-side
# tools first by the host system compiler. Libraries will be cross-compiled.
cmake_args.append(InterpolateToPosixPath(
'-DLLVM_AR=%(builddir)s/' + f.obj_dir + '/bin/llvm-ar.exe')),
CmakeCommand.applyDefaultOptions(cmake_args, [ CmakeCommand.applyDefaultOptions(cmake_args, [
('-G', 'Ninja'), ('-G', 'Ninja'),
('-DLLVM_ENABLE_PROJECTS=', 'llvm;clang;clang-tools-extra;lld'), ('-DLLVM_ENABLE_PROJECTS=', 'llvm;clang;clang-tools-extra;lld'),
('-DCMAKE_BUILD_TYPE=', 'Release'), ('-DCMAKE_BUILD_TYPE=', 'Release'),
('-DCMAKE_CXX_FLAGS=', '-D__OPTIMIZE__'), ('-DCMAKE_CXX_FLAGS=', '-D__OPTIMIZE__'),
('-DLLVM_ENABLE_ASSERTIONS=', 'ON'), ('-DLLVM_ENABLE_ASSERTIONS=', 'ON'),
('-DLLVM_LIT_ARGS=', '-v -vv'), ('-DLLVM_LIT_ARGS=', '-v -vv'),
]) ])
▲ Show 20 Lines • Show All 90 Lines • Show Last 20 Lines

zorg/buildbot/builders/annotated/annotated_builder.py

from __future__ import print_function from __future__ import print_function
import util import util
import argparse import argparse
import errno import errno
import os import os
import re import re
import shutil import shutil
import subprocess import subprocess
import sys import sys
from os.path import join as pjoin from os.path import join as pjoin
VSWHERE_PATH = "C:/Program Files (x86)/Microsoft Visual Studio/Installer/vswhere.exe" VSWHERE_PATH = "C:/Program Files (x86)/Microsoft Visual Studio/Installer/vswhere.exe"
BUILDTOOLS_VSDEVCMD = "C:/BuildTools/Common7/Tools/VsDevCmd.bat"
def get_argument_parser(*args, **kwargs): def get_argument_parser(*args, **kwargs):
ap = argparse.ArgumentParser(*args, **kwargs) ap = argparse.ArgumentParser(*args, **kwargs)
ap.add_argument('--jobs', help='Number of concurrent jobs to run') ap.add_argument('--jobs', help='Number of concurrent jobs to run')
return ap return ap
class AnnotatedBuilder: class AnnotatedBuilder:
▲ Show 20 Lines • Show All 182 Lines • ▼ Show 20 Linesclass AnnotatedBuilder:
def stage_cmake_args( def stage_cmake_args(
self, self,
cmake_args, cmake_args,
extra_cmake_args, extra_cmake_args,
c_compiler, c_compiler,
cxx_compiler, cxx_compiler,
linker, linker,
previous_stage_bin): previous_stage_bin):
archiver = 'llvm-ar'
if os.name == 'nt':
archiver = 'llvm-lib'
return ( return (
cmake_args + [ cmake_args + [
'-DCMAKE_AR=%s' % ( '-DCMAKE_AR=%s' % (
util.cmake_pjoin(previous_stage_bin, 'llvm-ar'),), util.cmake_pjoin(previous_stage_bin, archiver),),
'-DCMAKE_RANLIB=%s' % ( '-DCMAKE_RANLIB=%s' % (
util.cmake_pjoin(previous_stage_bin, 'llvm-ranlib'),), util.cmake_pjoin(previous_stage_bin, 'llvm-ranlib'),),
] + self.cmake_compiler_flags( ] + self.cmake_compiler_flags(
c_compiler, cxx_compiler, linker, previous_stage_bin) + c_compiler, cxx_compiler, linker, previous_stage_bin) +
extra_cmake_args) extra_cmake_args)
def set_environment(self, env=None, vs_tools=None, arch=None): def set_environment(self, env=None, vs_tools=None, arch=None):
self.report_build_step('set-environment') self.report_build_step('set-environment')
▲ Show 20 Lines • Show All 92 Lines • ▼ Show 20 Lines def run_steps(
except Exception as e: except Exception as e:
self.report_step_exception(e) self.report_step_exception(e)
return 1 return 1
return 0 return 0
def get_vcvars(vs_tools, arch): def get_vcvars(vs_tools, arch):
"""Get the VC tools environment using vswhere.exe from VS 2017 """Get the VC tools environment using vswhere.exe or buildtools docker
This is intended to work either when VS is in its standard installation
location, or when the docker instructions have been followed, and we can
find Visual C++ in C:/BuildTools.
Visual Studio provides a docker image with instructions here:
https://docs.microsoft.com/en-us/visualstudio/install/build-tools-container?view=vs-2019
This code is following the guidelines from strategy 1 in this blog post: This vswhere code is following the guidelines from strategy 1 in this blog
post:
https://blogs.msdn.microsoft.com/vcblog/2017/03/06/finding-the-visual-c-compiler-tools-in-visual-studio-2017/ https://blogs.msdn.microsoft.com/vcblog/2017/03/06/finding-the-visual-c-compiler-tools-in-visual-studio-2017/
It doesn't work when VS is not installed at the default location. It doesn't work when VS is not installed at the default location.
""" """
if not arch: if not arch:
# First check the wow64 processor architecture, since python is probably # First check the wow64 processor architecture, since python is probably
# 32-bit, then fall back to PROCESSOR_ARCHITECTURE. # 32-bit, then fall back to PROCESSOR_ARCHITECTURE.
arch = os.environ.get('PROCESSOR_ARCHITEW6432', '').lower() arch = os.environ.get('PROCESSOR_ARCHITEW6432', '').lower()
if not arch: if not arch:
arch = os.environ.get('PROCESSOR_ARCHITECTURE', '').lower() arch = os.environ.get('PROCESSOR_ARCHITECTURE', '').lower()
else: else:
arch = arch.lower() arch = arch.lower()
# Use vswhere.exe if it exists. # Use vswhere.exe if it exists.
if os.path.exists(VSWHERE_PATH): if os.path.exists(VSWHERE_PATH):
cmd = [VSWHERE_PATH, "-latest", "-property", "installationPath"] cmd = [VSWHERE_PATH, "-latest", "-property", "installationPath"]
vs_path = subprocess.check_output(cmd).strip() vs_path = subprocess.check_output(cmd).decode(sys.stdout.encoding)
vs_path = vs_path.strip()
util.report("Running vswhere to find VS: " + repr(cmd)) util.report("Running vswhere to find VS: " + repr(cmd))
util.report("vswhere output: " + vs_path) util.report("vswhere output: " + vs_path)
if not os.path.isdir(vs_path): if not os.path.isdir(vs_path):
raise ValueError("VS install path does not exist: " + vs_path) raise ValueError("VS install path does not exist: " + vs_path)
vcvars_path = pjoin(vs_path, 'VC', 'Auxiliary', 'Build', vcvars_path = pjoin(vs_path, 'VC', 'Auxiliary', 'Build',
'vcvarsall.bat') 'vcvarsall.bat')
elif os.path.exists(BUILDTOOLS_VSDEVCMD):
vcvars_path = BUILDTOOLS_VSDEVCMD
elif vs_tools is None: elif vs_tools is None:
vs_tools = os.path.expandvars('%VS140COMNTOOLS%') vs_tools = os.path.expandvars('%VS140COMNTOOLS%')
vcvars_path = pjoin(vs_tools, '..', '..', 'VC', 'vcvarsall.bat') vcvars_path = pjoin(vs_tools, '..', '..', 'VC', 'vcvarsall.bat')
# Newer vcvarsall.bat scripts aren't quiet, so direct them to NUL, aka # Newer vcvarsall.bat scripts aren't quiet, so direct them to NUL, aka
# Windows /dev/null. # Windows /dev/null.
cmd = util.shquote_cmd([vcvars_path, arch]) + ' > NUL && set' cmd = util.shquote_cmd([vcvars_path, arch]) + ' > NUL && set'
util.report("Running vcvars: " + cmd) util.report("Running vcvars: " + cmd)
output = subprocess.check_output(cmd, shell=True) output = \
subprocess.check_output(cmd, shell=True).decode(sys.stdout.encoding)
new_env = {} new_env = {}
for line in output.splitlines(): for line in output.splitlines():
var, val = line.split('=', 1) var, val = line.split('=', 1)
new_env[var] = val new_env[var] = val
return new_env return new_env
def main(argv): def main(argv):
ap = get_argument_parser() ap = get_argument_parser()
args = ap.parse_args(argv[1:]) args = ap.parse_args(argv[1:])
builder = AnnotatedBuilder() builder = AnnotatedBuilder()
builder.run_steps(jobs=args.jobs) builder.run_steps(jobs=args.jobs)
if __name__ == '__main__': if __name__ == '__main__':
sys.exit(main(sys.argv)) sys.exit(main(sys.argv))

zorg/buildbot/builders/annotated/buildbot-helper.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/bin/bash
# Buildbot Helper functions
# Stop if we've encountered an error.
halt_on_failure() {
echo "@@@HALT_ON_FAILURE@@@"
}
build_step() {
echo "@@@BUILD_STEP ${@}@@@"
}

zorg/buildbot/builders/annotated/hip-build.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/bin/bash
# Enable Error tracing
set -o errtrace
# Print trace for all commands ran before execution
set -x
# Include the Buildbot helper functions
HERE="$(dirname $0)"
. ${HERE}/buildbot-helper.sh
# Ensure all commands pass, and not dereferencing unset variables.
set -eu
halt_on_failure
BUILDBOT_ROOT=/buildbot
LLVM_ROOT="${BUILDBOT_ROOT}/llvm-project"
REVISION="${BUILDBOT_REVISION:-origin/main}"
AMDGPU_ARCHS=${AMDGPU_ARCHS:="gfx900;gfx906;gfx908;gfx1030"}
# Set-up llvm-project
if [ ! -d "${LLVM_ROOT}" ]; then
build_step "Cloning llvm-project repo"
git clone --progress https://github.com/llvm/llvm-project.git ${LLVM_ROOT}
fi
build_step "Updating llvm-project repo"
git -C "${LLVM_ROOT}" fetch origin
git -C "${LLVM_ROOT}" reset --hard "${REVISION}"
# Set-up llvm-test-suite
TESTSUITE_ROOT="${BUILDBOT_ROOT}/llvm-test-suite"
if [ ! -d "${TESTSUITE_ROOT}" ]; then
build_step "Cloning llvm-test-suite repo"
git clone --progress https://github.com/llvm/llvm-test-suite.git ${TESTSUITE_ROOT}
fi
build_step "Updating llvm-test-suite repo"
git -C "${TESTSUITE_ROOT}" fetch origin
git -C "${TESTSUITE_ROOT}" reset --hard origin/main
# Set-up variables
BUILDBOT_SLAVENAME=$(whoami)
BUILD_DIR="${BUILDBOT_ROOT}/${BUILDBOT_SLAVENAME}/${BUILDBOT_BUILDERNAME}"
DESTDIR=${BUILD_DIR}/install
EXTERNAL_DIR=/buildbot/Externals
build_step "Setting up the buildbot"
echo "BUILDBOT_ROOT=${BUILDBOT_ROOT}"
echo "LLVM_ROOT=${LLVM_ROOT}"
echo "BUILD_DIR=${BUILD_DIR}"
echo "DESTDIR=${DESTDIR}"
echo "EXTERNAL_DIR=${EXTERNAL_DIR}"
# Start building LLVM, Clang, Lld, clang-tools-extra, compiler-rt
build_step "Configure LLVM Build"
LLVM_BUILD_DIR="${BUILD_DIR}/llvm"
mkdir -p "${LLVM_BUILD_DIR}"
cd "${LLVM_BUILD_DIR}"
cmake -G Ninja \
-DCMAKE_BUILD_TYPE="Release" \
-DCMAKE_VERBOSE_MAKEFILE=1 \
-DLLVM_TARGETS_TO_BUILD="AMDGPU;X86" \
-DLLVM_ENABLE_PROJECTS="clang;lld;clang-tools-extra;compiler-rt" \
-DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \
-DLIBCXX_ENABLE_SHARED=OFF \
-DLIBCXX_ENABLE_STATIC=ON \
-DLIBCXX_INSTALL_LIBRARY=OFF \
-DLIBCXX_INSTALL_HEADERS=OFF \
-DLIBCXXABI_ENABLE_SHARED=OFF \
-DLIBCXXABI_ENABLE_STATIC=ON \
-DLIBCXXABI_INSTALL_STATIC_LIBRARY=OFF \
-DCMAKE_INSTALL_PREFIX="${DESTDIR}" \
-DLLVM_ENABLE_ASSERTIONS=ON \
-DLLVM_ENABLE_Z3_SOLVER=OFF \
-DLLVM_ENABLE_ZLIB=ON \
-DLLVM_LIT_ARGS="-v -vv" \
${LLVM_ROOT}/llvm
build_step "Building LLVM"
ninja runtimes
ninja
build_step "Install LLVM"
rm -rf "${DESTDIR}"
ninja install-runtimes
ninja install
# Start building llvm-test-suite's hip tests
build_step "Configuring HIP test-suite"
TEST_BUILD_DIR=${BUILD_DIR}/test-suite-build
rm -rf ${TEST_BUILD_DIR}
mkdir -p ${TEST_BUILD_DIR}
cd ${TEST_BUILD_DIR}
PATH="${LLVM_BUILD_DIR}/bin:$PATH" cmake -G Ninja \
-DTEST_SUITE_SUBDIRS=External \
-DTEST_SUITE_EXTERNALS_DIR=${EXTERNAL_DIR}/ \
-DTEST_SUITE_COLLECT_CODE_SIZE=OFF \
-DTEST_SUITE_COLLECT_COMPILE_TIME=OFF \
-DAMDGPU_ARCHS="${AMDGPU_ARCHS}" \
-DCMAKE_CXX_COMPILER="${LLVM_BUILD_DIR}/bin/clang++" \
-DCMAKE_C_COMPILER="${LLVM_BUILD_DIR}/bin/clang" \
-DCMAKE_VERBOSE_MAKEFILE=ON \
${TESTSUITE_ROOT}
build_step "Building HIP test-suite"
ninja hip-tests-simple
build_step "Testing HIP test-suite"
ninja check-hip-simple
exit 0

zorg/buildbot/builders/annotated/libc-linux.py

  • This file was copied to zorg/buildbot/builders/annotated/libc-windows.py.
#!/usr/bin/python #!/usr/bin/python
import argparse import argparse
import os import os
import subprocess import subprocess
import sys import sys
import traceback import traceback
import util import util
from contextlib import contextmanager from contextlib import contextmanager
def is_fullbuild_builder(builder_name):
return ('fullbuild' in builder_name.split('-'))
def is_runtimes_builder(builder_name):
return ('runtimes' in builder_name.split('-'))
def is_gcc_builder(builder_name):
return ('gcc' in builder_name.split('-'))
def is_lint_builder(builder_name):
return ('lint' in builder_name.split('-'))
def is_riscv_builder(builder_name):
return 'riscv' in builder_name
def is_x86_64_builder(builder_name):
return 'x86_64' in builder_name
def main(argv): def main(argv):
ap = argparse.ArgumentParser() ap = argparse.ArgumentParser()
ap.add_argument('--asan', action='store_true', default=False, ap.add_argument('--asan', action='store_true', default=False,
help='Build with address sanitizer enabled.') help='Build with address sanitizer enabled.')
ap.add_argument('--debug', action='store_true', default=False, ap.add_argument('--debug', action='store_true', default=False,
help='Build in debug mode.') help='Build in debug mode.')
args, _ = ap.parse_known_args() args, _ = ap.parse_known_args()
source_dir = os.path.join('..', 'llvm-project') source_dir = os.path.join('..', 'llvm-project')
builder_name = os.environ.get('BUILDBOT_BUILDERNAME')
fullbuild = is_fullbuild_builder(builder_name)
runtimes_build = is_runtimes_builder(builder_name)
gcc_build = is_gcc_builder(builder_name)
lint_build = is_lint_builder(builder_name)
riscv_build = is_riscv_builder(builder_name)
x86_64_build = is_x86_64_builder(builder_name)
if gcc_build:
cc = 'gcc'
cxx = 'g++'
else:
if lint_build:
full_build = True
cc = '/home/libc-lint-tools/bin/clang'
cxx = '/home/libc-lint-tools/bin/clang++'
clang_tidy = '/home/libc-lint-tools/bin/clang-tidy'
else:
cc = 'clang'
cxx = 'clang++'
with step('cmake', halt_on_fail=True): with step('cmake', halt_on_fail=True):
projects = ['llvm', 'libc', 'clang', 'clang-tools-extra'] # On most systems the default generator is make, and the default
# compilers are gcc and g++. We make the compiler and the generator
# explicit here, which reduces one step of setting environment
# variables when setting up workers.
cmake_args = ['-GNinja',
'-DCMAKE_C_COMPILER=%s' % cc,
'-DCMAKE_CXX_COMPILER=%s' % cxx]
if lint_build:
cmake_args.append('-DLLVM_LIBC_CLANG_TIDY=%s' % clang_tidy)
if runtimes_build:
projects = ['llvm', 'clang']
cmake_args.append('-DLLVM_ENABLE_RUNTIMES=libc')
else:
projects = ['llvm', 'libc']
cmake_args = ['-GNinja']
if args.debug: if args.debug:
cmake_args.append('-DCMAKE_BUILD_TYPE=Debug') cmake_args.append('-DCMAKE_BUILD_TYPE=Debug')
else: else:
cmake_args.append('-DCMAKE_BUILD_TYPE=Release') cmake_args.append('-DCMAKE_BUILD_TYPE=Release')
if args.asan: if args.asan:
cmake_args.append('-DLLVM_USE_SANITIZER=Address') cmake_args.append('-DLLVM_USE_SANITIZER=Address')
if fullbuild and not args.asan and not lint_build and not riscv_build:
projects.extend(['clang', 'compiler-rt'])
cmake_args.append('-DLLVM_ENABLE_PROJECTS={}'.format(';'.join(projects))) cmake_args.append('-DLLVM_ENABLE_PROJECTS={}'.format(';'.join(projects)))
if fullbuild and not args.asan and not lint_build and not riscv_build:
cmake_args.append('-DLLVM_LIBC_INCLUDE_SCUDO=ON')
cmake_args.append('-DLIBC_INCLUDE_BENCHMARKS=ON')
if fullbuild:
cmake_args.extend(['-DLLVM_LIBC_FULL_BUILD=ON']),
run_command(['cmake', os.path.join(source_dir, 'llvm')] + cmake_args) run_command(['cmake', os.path.join(source_dir, 'llvm')] + cmake_args)
with step('build llvmlibc', halt_on_fail=True): if lint_build:
run_command(['ninja', 'llvmlibc']) with step('lint libc'):
run_command(['ninja', 'libc-lint'])
return
with step('build libc'):
run_command(['ninja', 'libc'])
if fullbuild:
with step('build libc-startup'):
run_command(['ninja', 'libc-startup'])
if runtimes_build:
with step('check-libc'): with step('check-libc'):
run_command(['ninja', 'check-libc']) run_command(['ninja', 'check-libc'])
else:
with step('libc-unit-tests'):
run_command(['ninja', 'libc-unit-tests'])
if not args.asan: if fullbuild and not args.asan:
with step('Loader Tests'): with step('libc-integration-tests'):
run_command(['ninja', 'libc_loader_tests']) run_command(['ninja', 'libc-integration-tests'])
with step('Integration Tests'): with step('libc-api-test'):
run_command(['ninja', 'libc-integration-test']) run_command(['ninja', 'libc-api-test'])
if gcc_build or ('riscv' in builder_name):
# The rest of the targets are either not yet gcc-clean or
# not yet availabe on riscv.
return
with step('libc-scudo-integration-test'):
run_command(['ninja', 'libc-scudo-integration-test'])
with step('AOR Tests'): with step('AOR Tests'):
aor_dir = os.path.join(source_dir, 'libc', 'AOR_v20.02') aor_dir = os.path.join(source_dir, 'libc', 'AOR_v20.02')
# Remove the AOR build dir. # Remove the AOR build dir.
util.clean_dir(os.path.join(aor_dir, 'build')) util.clean_dir(os.path.join(aor_dir, 'build'))
run_command(['make', 'check'], directory=aor_dir) run_command(['make', 'check'], directory=aor_dir)
if not args.debug:
with step('Benchmark Utils Tests'): with step('Benchmark Utils Tests'):
run_command(['ninja', 'libc-benchmark-util-tests']) run_command(['ninja', 'libc-benchmark-util-tests'])
if not (fullbuild or runtimes_build) and x86_64_build:
with step('libc-fuzzer'):
run_command(['ninja', 'libc-fuzzer'])
@contextmanager @contextmanager
def step(step_name, halt_on_fail=False): def step(step_name, halt_on_fail=False):
util.report('@@@BUILD_STEP {}@@@'.format(step_name)) util.report('@@@BUILD_STEP {}@@@'.format(step_name))
if halt_on_fail: if halt_on_fail:
util.report('@@@HALT_ON_FAILURE@@@') util.report('@@@HALT_ON_FAILURE@@@')
try: try:
yield yield
Show All 20 Lines

zorg/buildbot/builders/annotated/libc-windows.py

  • This file was copied from zorg/buildbot/builders/annotated/libc-linux.py.
#!/usr/bin/python
import annotated_builder
import util
import argparse import argparse
import os import os
import subprocess import subprocess
import sys import sys
import traceback import traceback
import util
from contextlib import contextmanager from contextlib import contextmanager
def main(argv): def main(argv):
ap = argparse.ArgumentParser() ap = argparse.ArgumentParser()
ap.add_argument('--asan', action='store_true', default=False, ap.add_argument('--asan', action='store_true', default=False,
help='Build with address sanitizer enabled.') help='Build with address sanitizer enabled.')
ap.add_argument('--debug', action='store_true', default=False, ap.add_argument('--debug', action='store_true', default=False,
help='Build in debug mode.') help='Build in debug mode.')
args, _ = ap.parse_known_args() args, _ = ap.parse_known_args()
source_dir = os.path.join('..', 'llvm-project') source_dir = os.path.join('..', 'llvm-project')
vc_vars = annotated_builder.get_vcvars(vs_tools=None, arch='amd64')
for (var, val) in vc_vars.items():
os.environ[var] = val
with step('cmake', halt_on_fail=True): with step('cmake', halt_on_fail=True):
projects = ['llvm', 'libc', 'clang', 'clang-tools-extra'] # On most systems the default generator is make and the default
# compilers are gcc and g++. We make it explicit here that we want
cmake_args = ['-GNinja'] # clang and ninja which reduces one step of setting environment
# variables when setting up workers.
## adjust compiler location definitions based on VM config
cmake_args = ['-GNinja',
'-DCMAKE_C_COMPILER=clang-cl',
'-DCMAKE_CXX_COMPILER=clang-cl']
if args.debug: if args.debug:
cmake_args.append('-DCMAKE_BUILD_TYPE=Debug') cmake_args.append('-DCMAKE_BUILD_TYPE=Debug')
else: else:
cmake_args.append('-DCMAKE_BUILD_TYPE=Release') cmake_args.append('-DCMAKE_BUILD_TYPE=Release')
if args.asan: if args.asan:
cmake_args.append('-DLLVM_USE_SANITIZER=Address') cmake_args.append('-DLLVM_USE_SANITIZER=Address')
cmake_args.append('-DLLVM_ENABLE_PROJECTS={}'.format(';'.join(projects))) cmake_args.append('-DLLVM_ENABLE_PROJECTS=libc')
cmake_args.append('-DLLVM_TARGETS_TO_BUILD=X86')
cmake_args.append('-DLLVM_FORCE_BUILD_RUNTIME=libc')
cmake_args.append('-DLLVM_NATIVE_ARCH=x86_64')
cmake_args.append('-DLLVM_HOST_TRIPLE=x86_64-window-x86-gnu')
cmake_args.append('-DLLVM_LIBC_MPFR_INSTALL_PATH=C:/src/install')
run_command(['cmake', os.path.join(source_dir, 'llvm')] + cmake_args) run_command(['cmake', os.path.join(source_dir, 'llvm')] + cmake_args)
with step('build llvmlibc', halt_on_fail=True): with step('build llvmlibc', halt_on_fail=True):
run_command(['ninja', 'llvmlibc']) run_command(['ninja', 'llvmlibc'])
with step('check-libc'): with step('check-libc'):
run_command(['ninja', 'check-libc']) run_command(['ninja', 'check-libc'])
if not args.asan:
with step('Loader Tests'):
run_command(['ninja', 'libc_loader_tests'])
with step('Integration Tests'):
run_command(['ninja', 'libc-integration-test'])
with step('AOR Tests'):
aor_dir = os.path.join(source_dir, 'libc', 'AOR_v20.02')
# Remove the AOR build dir.
util.clean_dir(os.path.join(aor_dir, 'build'))
run_command(['make', 'check'], directory=aor_dir)
if not args.debug:
with step('Benchmark Utils Tests'):
run_command(['ninja', 'libc-benchmark-util-tests'])
@contextmanager @contextmanager
def step(step_name, halt_on_fail=False): def step(step_name, halt_on_fail=False):
util.report('@@@BUILD_STEP {}@@@'.format(step_name)) util.report('@@@BUILD_STEP {}@@@'.format(step_name))
if halt_on_fail: if halt_on_fail:
util.report('@@@HALT_ON_FAILURE@@@') util.report('@@@HALT_ON_FAILURE@@@')
try: try:
yield yield
Show All 20 Lines

zorg/buildbot/builders/annotated/standalone-build.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/bin/bash
# This script builds LLVM and Clang in standalone mode that means it first
# builds LLVM and installs it into a specific directory. That directory is then
# used when building Clang which depends on it.
# Enable Error tracing
set -o errtrace
# Print trace for all commands ran before execution
set -x
# Include the Buildbot helper functions
HERE="$(realpath $(dirname $0))"
. ${HERE}/buildbot-helper.sh
# Ensure all commands pass, and not dereferencing unset variables.
set -eu
halt_on_failure
BUILDBOT_ROOT=${BUILDBOT_ROOT:-${HERE}}
REVISION="${BUILDBOT_REVISION:-origin/main}"
LLVM_ROOT="${BUILDBOT_ROOT}/llvm-project"
BUILD_TYPE=Release
INSTALL_ROOT_DIR=${BUILDBOT_ROOT}/install
BUILD_ROOT_DIR=${BUILDBOT_ROOT}/build
# Enable CMake tracing during configure per component
CMAKE_TRACE_CONFIGURE_LLVM=${CMAKE_TRACE_CONFIGURE_LLVM:-}
CMAKE_TRACE_CONFIGURE_CLANG=${CMAKE_TRACE_CONFIGURE_CLANG:-}
CMAKE_TRACE_CONFIGURE_LLD=${CMAKE_TRACE_CONFIGURE_LLD:-}
# Turn CMake's verbose mode for build, check and install steps per component
CMAKE_VERBOSE_LLVM=${CMAKE_VERBOSE_LLVM:-}
CMAKE_VERBOSE_CLANG=${CMAKE_VERBOSE_CLANG:-}
CMAKE_VERBOSE_LLD=${CMAKE_VERBOSE_LLD:-}
# Enable CMake verbose-mode during building
CMAKE_VERBOSE_BUILD_LLVM=${CMAKE_VERBOSE_BUILD_LLVM:-${CMAKE_VERBOSE_LLVM}}
CMAKE_VERBOSE_BUILD_CLANG=${CMAKE_VERBOSE_BUILD_CLANG:-${CMAKE_VERBOSE_CLANG}}
CMAKE_VERBOSE_BUILD_LLD=${CMAKE_VERBOSE_BUILD_LLD:-${CMAKE_VERBOSE_LLD}}
# Enable CMake verbose-mode during checking
CMAKE_VERBOSE_CHECK_LLVM=${CMAKE_VERBOSE_CHECK_LLVM:-${CMAKE_VERBOSE_LLVM}}
CMAKE_VERBOSE_CHECK_CLANG=${CMAKE_VERBOSE_CHECK_CLANG:-${CMAKE_VERBOSE_CLANG}}
CMAKE_VERBOSE_CHECK_LLD=${CMAKE_VERBOSE_CHECK_LLD:-${CMAKE_VERBOSE_LLD}}
# Enable CMake verbose-mode during installation
CMAKE_VERBOSE_INSTALL_LLVM=${CMAKE_VERBOSE_INSTALL_LLVM:-${CMAKE_VERBOSE_LLVM}}
CMAKE_VERBOSE_INSTALL_CLANG=${CMAKE_VERBOSE_INSTALL_CLANG:-${CMAKE_VERBOSE_CLANG}}
CMAKE_VERBOSE_INSTALL_LLD=${CMAKE_VERBOSE_INSTALL_LLD:-${CMAKE_VERBOSE_LLD}}
install_dir() {
echo ${INSTALL_ROOT_DIR}/$1
}
build_dir() {
echo ${BUILD_ROOT_DIR}/$1
}
setup_llvm_project() {
build_step "Setup llvm-project"
if [ ! -d "${LLVM_ROOT}" ]; then
build_step "Cloning llvm-project repo"
git clone --progress https://github.com/llvm/llvm-project.git ${LLVM_ROOT}
fi
build_step "Updating llvm-project repo"
git -C "${LLVM_ROOT}" fetch origin
git -C "${LLVM_ROOT}" reset --hard "${REVISION}"
git -C "${LLVM_ROOT}" sparse-checkout init --cone
}
pre_build_cleanup() {
build_step "Pre-build cleanup"
rm -rf ${INSTALL_ROOT_DIR}
rm -rf ${BUILD_ROOT_DIR}
}
build_python_lit() {
local PYTHON_LIT_BUILD_DIR=$(build_dir python-lit)
local PYTHON_LIT_INSTALL_DIR=$(install_dir python-lit)
build_step "Sparse checkout out python-lit"
git -C "${LLVM_ROOT}" sparse-checkout set llvm/utils/lit
build_step "Building python-lit"
cd "${LLVM_ROOT}"/llvm/utils/lit
python3 setup.py build '--executable=/usr/bin/python3 -s'
# TODO(kkleine): Fix failing test: FAIL: lit :: shtest-not.py (43 of 50)
#build_step "Testing python-lit"
#python3 lit.py tests
build_step "Installing python-lit"
rm -rf ${PYTHON_LIT_INSTALL_DIR}
python3 setup.py install -O1 --skip-build --root ${PYTHON_LIT_INSTALL_DIR}
build_step "Removing python-lit build dir"
rm -rf ${PYTHON_LIT_BUILD_DIR}
cd -
}
# We don't install python to the system so we need help python to find it
discover_python_lit() {
build_step "Help discover python-lit module"
local PYTHON_LIT_INSTALL_DIR=$(install_dir python-lit)
local pyver=$(echo 'import sys; print("{}.{}".format(sys.version_info.major, sys.version_info.minor))' | python3)
export PYTHONPATH="${PYTHONPATH:-}:${PYTHON_LIT_INSTALL_DIR}/usr/local/lib/python${pyver}/site-packages"
}
build_llvm() {
local PYTHON_LIT_INSTALL_DIR=$(install_dir python-lit)
local LLVM_BUILD_DIR=$(build_dir llvm)
local LLVM_INSTALL_DIR=$(install_dir llvm)
build_step "Sparse checkout out llvm"
git -C "${LLVM_ROOT}" sparse-checkout set llvm cmake third-party
build_step "Configuring llvm"
cmake ${CMAKE_TRACE_CONFIGURE_LLVM} \
-S ${LLVM_ROOT}/llvm \
-B ${LLVM_BUILD_DIR} \
-G Ninja \
-DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
-DLLVM_BUILD_LLVM_DYLIB=ON \
-DLLVM_LINK_LLVM_DYLIB=ON \
-DLLVM_INCLUDE_BENCHMARKS=OFF \
-DLLVM_INSTALL_UTILS=ON \
-DCMAKE_INSTALL_PREFIX=${LLVM_INSTALL_DIR} \
-DLLVM_EXTERNAL_LIT=${PYTHON_LIT_INSTALL_DIR}/usr/local/bin/lit \
-DLLVM_INCLUDE_UTILS:BOOL=ON \
-DLLVM_INSTALL_UTILS:BOOL=ON \
-DLLVM_UTILS_INSTALL_DIR:PATH=${LLVM_INSTALL_DIR}/bin \
-DLLVM_CCACHE_BUILD=ON \
-DLLVM_INSTALL_GTEST=ON
build_step "Building llvm"
cmake --build ${LLVM_BUILD_DIR} ${CMAKE_VERBOSE_BUILD_LLVM}
build_step "Testing llvm"
LD_LIBRARY_PATH="${LLVM_INSTALL_DIR}/lib64" cmake --build ${LLVM_BUILD_DIR} --target check-all ${CMAKE_VERBOSE_CHECK_LLVM}
build_step "Installing llvm"
rm -rf ${LLVM_INSTALL_DIR}
cmake --install ${LLVM_BUILD_DIR} ${CMAKE_VERBOSE_INSTALL_LLVM}
# This is meant to extinguish any dependency on files being taken
# from the llvm build dir when building clang.
build_step "Removing llvm build dir"
rm -rf "${LLVM_BUILD_DIR}"
}
build_clang() {
local PYTHON_LIT_INSTALL_DIR=$(install_dir python-lit)
local LLVM_INSTALL_DIR=$(install_dir llvm)
local CLANG_BUILD_DIR=$(build_dir clang)
local CLANG_INSTALL_DIR=$(install_dir clang)
build_step "Sparse checkout out clang"
git -C "${LLVM_ROOT}" sparse-checkout set clang cmake
build_step "Configuring clang"
cmake ${CMAKE_TRACE_CONFIGURE_CLANG} \
-S ${LLVM_ROOT}/clang \
-B ${CLANG_BUILD_DIR} \
-G Ninja \
-DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
-DCLANG_LINK_CLANG_DYLIB=ON \
-DCLANG_INCLUDE_TESTS=ON \
-DLLVM_EXTERNAL_LIT=${PYTHON_LIT_INSTALL_DIR}/usr/local/bin/lit \
-DCMAKE_INSTALL_PREFIX=${CLANG_INSTALL_DIR} \
-DLLVM_ROOT=${LLVM_INSTALL_DIR} \
-DLLVM_ENABLE_ASSERTIONS=ON \
-DLLVM_TABLEGEN_EXE=${LLVM_INSTALL_DIR}/bin/llvm-tblgen \
-DLLVM_CCACHE_BUILD=ON
build_step "Building clang"
LD_LIBRARY_PATH="${LLVM_INSTALL_DIR}/lib64" cmake --build ${CLANG_BUILD_DIR} ${CMAKE_VERBOSE_BUILD_CLANG}
# TODO(kwk): Add Clang check
build_step "Installing clang"
rm -rf ${CLANG_INSTALL_DIR}
cmake --install ${CLANG_BUILD_DIR} ${CMAKE_VERBOSE_INSTALL_CLANG}
build_step "Removing clang build dir"
rm -rf ${CLANG_BUILD_DIR}
}
build_lld() {
local LLVM_INSTALL_DIR=$(install_dir llvm)
local PYTHON_LIT_INSTALL_DIR=$(install_dir python-lit)
local LLD_BUILD_DIR=$(build_dir lld)
local LLD_INSTALL_DIR=$(install_dir lld)
build_step "Sparse checkout out lld"
git -C "${LLVM_ROOT}" sparse-checkout set lld cmake libunwind
# We don't want to checkout the llvm source tree but sadly there are paths
# like ${LLVM_MAIN_SRC_DIR}/../libunwind/include in lld source code. They
# resolve to /<SOMEPATH>/../llvm/../libunwind/include which makes absolutely
# no sense when the llvm dir doesn't exist. Let's fix this by just providing
# the empty llvm dir so that paths are resolved without errors.
# TODO: I don't know how to fix this easily
rm -rf "${LLVM_ROOT}"/llvm && mkdir "${LLVM_ROOT}"/llvm
rm -rf ${LLD_INSTALL_DIR}
rm -rf ${LLD_BUILD_DIR}
build_step "Configuring lld"
cmake ${CMAKE_TRACE_CONFIGURE_LLD} \
-S ${LLVM_ROOT}/lld \
-B ${LLD_BUILD_DIR} \
-G Ninja \
-DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
-DLLVM_LINK_LLVM_DYLIB=ON \
-DCMAKE_INSTALL_PREFIX=${LLD_INSTALL_DIR} \
-DLLVM_ROOT=${LLVM_INSTALL_DIR} \
-DLLVM_CCACHE_BUILD=ON \
-DLLVM_INCLUDE_TESTS=ON \
-DLLVM_EXTERNAL_LIT=${PYTHON_LIT_INSTALL_DIR}/usr/local/bin/lit
build_step "Building lld"
LD_LIBRARY_PATH="${LLVM_INSTALL_DIR}/lib64" cmake --build ${LLD_BUILD_DIR} ${CMAKE_VERBOSE_BUILD_LLD}
build_step "Testing lld"
LD_LIBRARY_PATH="${LLVM_INSTALL_DIR}/lib64" cmake --build ${LLD_BUILD_DIR} --target check-lld ${CMAKE_VERBOSE_CHECK_LLD}
build_step "Installing lld"
rm -rf ${LLD_INSTALL_DIR}
cmake --install ${LLD_BUILD_DIR} ${CMAKE_VERBOSE_INSTALL_LLD}
build_step "Removing lld build dir"
rm -rf ${LLD_BUILD_DIR}
}
setup_llvm_project
pre_build_cleanup
build_python_lit
discover_python_lit
build_llvm
build_clang
build_lld
exit 0

zorg/buildbot/builders/annotated/ve-linux-steps.make

  • This file was added.
##### Interface Variables & Targets #####
BUILDROOT?=$(error "BUILDROOT has to be the path to the worker's build/ directory.")
# Renders one target per line in make order. Each target will made with a
# build step with the 'get-steps' annotated builder (ve-linux.py).
get-steps:
@echo "prepare"
@echo "build-llvm"
@echo "check-llvm"
@echo "install-llvm"
@echo "build-crt-ve"
@echo "check-crt-ve"
@echo "install-crt-ve"
@echo "build-runtimes-ve"
@echo "install-runtimes-ve"
# @echo "check-runtimes-ve"
##### Tools #####
CMAKE?=cmake
NINJA?=ninja
TOOL_CONFIG_CACHE?=${HOME}/tools-config.cmake
##### Derived Configuration #####
# Path
MONOREPO=${BUILDROOT}/../llvm-project
# Build foders
LLVM_BUILD=${BUILDROOT}/build_llvm
CRT_BUILD_VE=${BUILDROOT}/build_crt_ve
RUNTIMES_BUILD_VE=${BUILDROOT}/build_runtimes_ve
# 'Install' into the LLVM build tree.
LLVM_PREFIX=${BUILDROOT}/install
# Install prefix structure
BUILT_CLANG=${LLVM_PREFIX}/bin/clang
BUILT_CLANGXX=${LLVM_PREFIX}/bin/clang++
X86_TARGET=x86_64-unknown-linux-gnu
VE_TARGET=ve-unknown-linux-gnu
### LLVM
LLVM_BUILD_TYPE=RelWithDebInfo
### Compiler-RT
CRT_BUILD_TYPE=Release
CRT_OPTFLAGS=-O2
CRT_TEST_OPTFLAGS=-O2
## Runtimes
RUNTIMES_BUILD_TYPE=Release
RUNTIMES_OPTFLAGS=-O2
##### Build Steps #####
# Standalone build has been prohibited. However, runtime build is not
# possible for VE because crt-ve is needed to be compiled by just compiled
# llvm. Such bootstrap build CMakefile is not merged yet. Check
# https://reviews.llvm.org/D89492 for details.
#
# As a result, we compile llvm for ve using following three steps.
# 1. Build llvm for X86 and VE with only X86 runtimes.
# 1.1 check-llvm
# 2. Build llvm for X86 and VE with x86 and VE compiler-rt runtimes.
# 2.1 check-compiler-rt for VE
# 3. Build x86 and VE all possible runtimes using 2.
### Vanilla LLVM stage ###
build-llvm:
touch "${TOOL_CONFIG_CACHE}"
mkdir -p "${LLVM_BUILD}"
cd "${LLVM_BUILD}" && ${CMAKE} "${MONOREPO}/llvm" -G Ninja \
-C "${TOOL_CONFIG_CACHE}" \
-DCMAKE_BUILD_TYPE="${LLVM_BUILD_TYPE}" \
-DCMAKE_INSTALL_PREFIX="${LLVM_PREFIX}" \
-DCLANG_LINK_CLANG_DYLIB=Off \
-DLLVM_BUILD_LLVM_DYLIB=Off \
-DLLVM_LINK_LLVM_DYLIB=Off \
-DLLVM_TARGETS_TO_BUILD="X86;VE" \
-DLLVM_ENABLE_PROJECTS="clang" \
-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=On
cd "${LLVM_BUILD}" && ${NINJA}
check-llvm:
cd "${LLVM_BUILD}" && ${NINJA} check-all
install-llvm:
cd "${LLVM_BUILD}" && ${NINJA} install
### Compiler-RT ###
build-crt-ve:
mkdir -p "${CRT_BUILD_VE}"
cd "${CRT_BUILD_VE}" && ${CMAKE} "${MONOREPO}/llvm" -G Ninja \
-C "${TOOL_CONFIG_CACHE}" \
-DCMAKE_BUILD_TYPE="${CRT_BUILD_TYPE}" \
-DCMAKE_INSTALL_PREFIX="${LLVM_PREFIX}" \
-DCMAKE_C_COMPILER="${BUILT_CLANG}" \
-DCMAKE_CXX_COMPILER="${BUILT_CLANGXX}" \
-DCMAKE_CXX_FLAGS_RELEASE="${CRT_OPTFLAGS}" \
-DCMAKE_C_FLAGS_RELEASE="${CRT_OPTFLAGS}" \
-DCLANG_LINK_CLANG_DYLIB=Off \
-DLLVM_BUILD_LLVM_DYLIB=Off \
-DLLVM_LINK_LLVM_DYLIB=Off \
-DLLVM_TARGETS_TO_BUILD="X86;VE" \
-DLLVM_ENABLE_PROJECTS="clang" \
-DLLVM_ENABLE_RUNTIMES="compiler-rt" \
-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=On \
-DLLVM_INSTALL_UTILS=On \
-DLLVM_RUNTIME_TARGETS="${X86_TARGET};${VE_TARGET}" \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_BUILTINS=On \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_CRT=On \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_SANITIZERS=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_XRAY=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_LIBFUZZER=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_PROFILE=On \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_MEMPROF=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_ORC=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_GWP_ASAN=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_USE_BUILTINS_LIBRARY=On \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_TEST_COMPILER_CFLAGS="-target ${VE_TARGET} ${CRT_TEST_OPTFLAGS}"
cd "${CRT_BUILD_VE}" && ${NINJA}
check-crt-ve:
cd "${CRT_BUILD_VE}" && ${NINJA} check-compiler-rt-ve-unknown-linux-gnu
install-crt-ve:
cd "${CRT_BUILD_VE}" && ${NINJA} install
### Runtimes ###
# It is not possible to compile clang nor compiler-rt with other runtimes
# at once for VE. Because the crtbegin bootstrap mechanism is merged yet.
# See https://reviews.llvm.org/D89492 for details. Without this patch,
# running regression tests for runtimes is also not possible for VE.
#
# Once this patch is merged, it will be possible to compile runtimes
# with following defines using single build step.
# -DLLVM_ENABLE_PROJECTS="clang" \
# -DLLVM_ENABLE_RUNTIMES="compiler-rt;libunwind" \
# -DRUNTIMES_${VE_TARGET}_COMPILER_RT_BOOTSTRAP=On \
build-runtimes-ve:
mkdir -p "${RUNTIMES_BUILD_VE}"
cd "${RUNTIMES_BUILD_VE}" && ${CMAKE} "${MONOREPO}/llvm" -G Ninja \
-C "${TOOL_CONFIG_CACHE}" \
-DCMAKE_BUILD_TYPE="${RUNTIMES_BUILD_TYPE}" \
-DCMAKE_INSTALL_PREFIX="${LLVM_PREFIX}" \
-DCMAKE_C_COMPILER="${BUILT_CLANG}" \
-DCMAKE_CXX_COMPILER="${BUILT_CLANGXX}" \
-DCMAKE_CXX_FLAGS_RELEASE="${RUNTIMES_OPTFLAGS}" \
-DCMAKE_C_FLAGS_RELEASE="${RUNTIMES_OPTFLAGS}" \
-DCLANG_LINK_CLANG_DYLIB=Off \
-DLLVM_BUILD_LLVM_DYLIB=Off \
-DLLVM_LINK_LLVM_DYLIB=Off \
-DLLVM_TARGETS_TO_BUILD="X86;VE" \
-DLLVM_ENABLE_PROJECTS="" \
-DLLVM_ENABLE_RUNTIMES="libunwind" \
-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=On \
-DLLVM_RUNTIME_TARGETS="${X86_TARGET};${VE_TARGET}" \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_BUILTINS=On \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_CRT=On \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_SANITIZERS=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_XRAY=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_LIBFUZZER=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_PROFILE=On \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_MEMPROF=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_ORC=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_BUILD_GWP_ASAN=Off \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_USE_BUILTINS_LIBRARY=On \
-DRUNTIMES_${VE_TARGET}_LIBCXXABI_USE_LLVM_UNWINDER=On \
-DRUNTIMES_${VE_TARGET}_LIBCXXABI_USE_COMPILER_RT=On \
-DRUNTIMES_${VE_TARGET}_LIBCXX_USE_COMPILER_RT=On \
-DRUNTIMES_${VE_TARGET}_CMAKE_C_COMPILER="${BUILT_CLANG}" \
-DRUNTIMES_${VE_TARGET}_COMPILER_RT_TEST_COMPILER_CFLAGS="-target ${VE_TARGET} ${RUNTIMES_TEST_OPTFLAGS}" \
-DRUNTIMES_${VE_TARGET}_LIBCXXABI_TEST_COMPILER_CFLAGS="-target ${VE_TARGET} ${RUNTIMES_TEST_OPTFLAGS}" \
-DRUNTIMES_${VE_TARGET}_LIBCXX_TEST_COMPILER_CFLAGS="-target ${VE_TARGET} ${RUNTIMES_TEST_OPTFLAGS}" \
-DRUNTIMES_${VE_TARGET}_LIBUNWIND_TEST_COMPILER_CFLAGS="-target ${VE_TARGET} ${RUNTIMES_TEST_OPTFLAGS}"
cd "${RUNTIMES_BUILD_VE}" && ${NINJA}
check-runtimes-ve:
cd "${RUNTIMES_BUILD_VE}" && ${NINJA} check-runtimes-ve-unknown-linux-gnu
install-runtimes-ve:
cd "${RUNTIMES_BUILD_VE}" && ${NINJA} install
# Clearout the temporary install prefix.
prepare:
# Build everything from scratch - TODO: incrementalize later.
rm -rf "${LLVM_PREFIX}"
rm -rf "${LLVM_BUILD}"
rm -rf "${CRT_BUILD_VE}"
rm -rf "${RUNTIMES_BUILD_VE}"

zorg/buildbot/builders/annotated/ve-linux.py

  • This file was added.
#!/usr/bin/env python3
import argparse
import os
import subprocess
import sys
import traceback
import util
from contextlib import contextmanager
def main(argv):
# Avoid a buildmaster update by hard-coding this for now.
makefile = 've-linux-steps.make'
# TODO: Make Makefile configurable and update buildmaster.
# ap = argparse.ArgumentParser()
# ap.add_argument('makefile', help='The Makefile to use. (relative to annoted/ folder of llvm-zorg).')
# args, _ = ap.parse_known_args()
# makefile = args.makefile
worker_dir = os.path.abspath(os.path.join('..'))
annotated_dir = os.path.join(worker_dir, 'llvm-zorg', 'zorg', 'buildbot', 'builders', 'annotated')
makefile_path = os.path.join(annotated_dir, makefile)
# Query step list from makefile.
build_targets=[]
with step('prepare', halt_on_fail=True):
build_targets = get_steps(makefile_path)
make_vars = {
'BUILDROOT' : os.path.join(worker_dir, 'build')
}
for target in build_targets:
halt_on_fail = not ("check-" in target)
with step(target, halt_on_fail=halt_on_fail):
make_cmd = build_make_cmd(makefile_path, target, make_vars)
run_command(make_cmd, cwd='.')
@contextmanager
def step(step_name, halt_on_fail=False):
util.report('@@@BUILD_STEP {}@@@'.format(step_name))
if halt_on_fail:
util.report('@@@HALT_ON_FAILURE@@@')
try:
yield
except Exception as e:
if isinstance(e, subprocess.CalledProcessError):
util.report(
'{} exited with return code {}.'.format(e.cmd, e.returncode)
)
util.report('The build step threw an exception...')
traceback.print_exc()
util.report('@@@STEP_FAILURE@@@')
finally:
sys.stdout.flush()
def get_steps(makefile):
try:
make_cmd = build_make_cmd(makefile, 'get-steps')
raw_steps = capture_cmd_stdout(make_cmd)
return raw_steps.decode('utf-8').split('\n')[:-1]
except:
return []
def build_make_cmd(makefile, target, make_vars={}):
make_cmd = ['make', '-f', makefile]
if not target is None:
make_cmd.append(target)
for k,v in make_vars.items():
make_cmd += ["{}={}".format(k, v)]
return make_cmd
def capture_cmd_stdout(cmd, **kwargs):
return subprocess.run(cmd, shell=False, check=True, stdout=subprocess.PIPE, **kwargs).stdout
def run_command(cmd, **kwargs):
util.report_run_cmd(cmd, **kwargs)
if __name__ == '__main__':
sys.path.append(os.path.dirname(__file__))
sys.exit(main(sys.argv))

zorg/buildbot/builders/sanitizers/buildbot_android.sh

#!/usr/bin/env bash #!/usr/bin/env bash
set -x set -x
set -e set -e
set -u set -u
HERE="$(cd $(dirname $0) && pwd)" HERE="$(cd $(dirname $0) && pwd)"
. ${HERE}/buildbot_functions.sh . ${HERE}/buildbot_functions.sh
. ${HERE}/buildbot_android_functions.sh . ${HERE}/buildbot_android_functions.sh
ROOT=`pwd`
PLATFORM=`uname`
LOCAL_IPS=`hostname -I` LOCAL_IPS=`hostname -I`
export PATH="/usr/local/bin:$PATH"
LLVM=$ROOT/llvm
CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS:-}"
CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS:-} -GNinja -DCMAKE_BUILD_TYPE=Release"
if [ -e /usr/include/plugin-api.h ]; then
CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS}"
fi
clobber clobber
download_android_tools download_android_tools
# Stage 1 # Stage 1
build_stage1_clang_at_revison build_stage1_clang_at_revison
### From now on we use just-built Clang as a host compiler ### ### From now on we use just-built Clang as a host compiler ###
buildbot_update buildbot_update
CMAKE_COMMON_OPTIONS="$CMAKE_COMMON_OPTIONS -DLLVM_ENABLE_ASSERTIONS=ON" CMAKE_COMMON_OPTIONS+=" -DLLVM_ENABLE_ASSERTIONS=ON"
build_stage2_android build_stage2_android
echo @@@BUILD_STEP run cmake@@@ echo @@@BUILD_STEP run cmake@@@
configure_android aarch64 aarch64-linux-android configure_android aarch64 aarch64-linux-android
# Testing armv7 instead of plain arm to work around # Testing armv7 instead of plain arm to work around
# https://code.google.com/p/android/issues/detail?id=68779 # https://code.google.com/p/android/issues/detail?id=68779
configure_android arm armv7-linux-androideabi configure_android arm armv7-linux-androideabi
configure_android i686 i686-linux-android configure_android i686 i686-linux-android
build_android aarch64 build_android aarch64
build_android arm build_android arm
build_android i686 build_android i686
# Arm hardware is temporarily offline # i686:x86 hardware is offline.
test_android i686:x86 arm:armeabi-v7a aarch64:arm64-v8a test_android arm:armeabi-v7a aarch64:arm64-v8a

zorg/buildbot/builders/sanitizers/buildbot_android_functions.sh

#!/usr/bin/env bash #!/usr/bin/env bash
CLOBBER="android_ndk android-ndk-* platform-tools *.zip shards_* test_android_* tested_arch_*"
STAGE2_CLOBBER="compiler_rt_build_android_* llvm_build_android_*"
STAGE1_CLOBBER="llvm_build64 ${STAGE2_CLOBBER}"
ANDROID_NDK_VERSION=21 ANDROID_NDK_VERSION=21
ANDROID_API=24 ANDROID_API=24
NDK_DIR=android_ndk NDK_DIR=android_ndk
function download_android_tools { function download_android_tools {
local VERSION=android-ndk-r${ANDROID_NDK_VERSION}d local VERSION=android-ndk-r${ANDROID_NDK_VERSION}d
local FILE_NAME=${VERSION}-linux-x86_64.zip local FILE_NAME=${VERSION}-linux-x86_64.zip
local NDK_URL=https://dl.google.com/android/repository/${FILE_NAME} local NDK_URL=https://dl.google.com/android/repository/${FILE_NAME}
if [[ "$(cat ${NDK_DIR}/android_ndk_url)" != ${NDK_URL} ]] ; then if [[ "$(cat ${NDK_DIR}/android_ndk_url)" != ${NDK_URL} ]] ; then
echo @@@BUILD_STEP downloading Android NDK@@@ echo @@@BUILD_STEP downloading Android NDK@@@
[[ -d ${NDK_DIR} ]] && rm -rf ${NDK_DIR} [[ -d ${NDK_DIR} ]] && rm -rf ${NDK_DIR}
[[ -d ${VERSION} ]] && rm -rf ${VERSION} [[ -d ${VERSION} ]] && rm -rf ${VERSION}
[[ -f ${FILE_NAME} ]] && rm -f ${FILE_NAME} [[ -f ${FILE_NAME} ]] && rm -f ${FILE_NAME}
wget ${NDK_URL} wget -e dotbytes=10m ${NDK_URL}
unzip ${FILE_NAME} > /dev/null unzip ${FILE_NAME} > /dev/null
mv ${VERSION} ${NDK_DIR} mv ${VERSION} ${NDK_DIR}
echo ${NDK_URL} > ${NDK_DIR}/android_ndk_url echo ${NDK_URL} > ${NDK_DIR}/android_ndk_url
fi fi
if [[ ! -d platform-tools ]] ; then if [[ ! -d platform-tools ]] ; then
echo @@@BUILD_STEP downloading Android Platform Tools@@@ echo @@@BUILD_STEP downloading Android Platform Tools@@@
local FILE_NAME=platform-tools-latest-linux.zip local FILE_NAME=platform-tools-latest-linux.zip
[[ -f ${FILE_NAME} ]] && rm -f ${FILE_NAME} [[ -f ${FILE_NAME} ]] && rm -f ${FILE_NAME}
wget https://dl.google.com/android/repository/${FILE_NAME} wget -e dotbytes=10m https://dl.google.com/android/repository/${FILE_NAME}
unzip ${FILE_NAME} > /dev/null unzip ${FILE_NAME} > /dev/null
fi fi
export PATH=$ROOT/platform-tools/:$PATH export PATH=$ROOT/platform-tools/:$PATH
} }
function build_stage2_android() { function build_stage2_android() {
# Build self-hosted tree with fresh Clang and -Werror. # Build self-hosted tree with fresh Clang and -Werror.
local CMAKE_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_WERROR=ON ${STAGE1_AS_COMPILER} -DCMAKE_C_FLAGS=-gmlt -DCMAKE_CXX_FLAGS=-gmlt" local CMAKE_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_WERROR=ON ${STAGE1_AS_COMPILER} -DCMAKE_C_FLAGS=-gmlt -DCMAKE_CXX_FLAGS=-gmlt"
CMAKE_OPTIONS="${CMAKE_OPTIONS} -DLLVM_ENABLE_PROJECTS='clang;lld' -DLLVM_USE_LINKER=lld" CMAKE_OPTIONS="${CMAKE_OPTIONS} -DLLVM_ENABLE_PROJECTS='clang;lld' -DCLANG_DEFAULT_RTLIB=libgcc"
if ccache -s ; then
CMAKE_OPTIONS="${CMAKE_OPTIONS} -DLLVM_CCACHE_BUILD=ON"
rm_dirs llvm_build64
fi
echo @@@BUILD_STEP bootstrap clang@@@ echo @@@BUILD_STEP bootstrap clang@@@
rm_dirs ${STAGE2_CLOBBER}
mkdir -p llvm_build64 mkdir -p llvm_build64
if [[ "$(cat llvm_build64/CMAKE_OPTIONS)" != "${CMAKE_OPTIONS}" ]] ; then if [[ "$(cat llvm_build64/CMAKE_OPTIONS)" != "${CMAKE_OPTIONS}" ]] ; then
(cd llvm_build64 && cmake ${CMAKE_OPTIONS} -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON $LLVM && \ (cd llvm_build64 && cmake ${CMAKE_OPTIONS} -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON $LLVM && \
echo ${CMAKE_OPTIONS} > CMAKE_OPTIONS) || echo @@@STEP_FAILURE@@@ echo ${CMAKE_OPTIONS} > CMAKE_OPTIONS) || build_failure
fi fi
ninja -C llvm_build64 || (echo @@@STEP_FAILURE@@@ && exit 2) ninja -C llvm_build64 || (build_failure && exit 2)
} }
function configure_android { # ARCH triple function configure_android { # ARCH triple
local _arch=$1 local _arch=$1
local _triple=$2 local _triple=$2
ANDROID_TOOLCHAIN=$ROOT/$NDK_DIR/toolchains/llvm/prebuilt/linux-x86_64 ANDROID_TOOLCHAIN=$ROOT/$NDK_DIR/toolchains/llvm/prebuilt/linux-x86_64
echo "Building for Android API level $ANDROID_API" echo "Building for Android API level $ANDROID_API"
local ANDROID_LIBRARY_OUTPUT_DIR=$(ls -d $ROOT/llvm_build64/lib/clang/* | tail -1) local ANDROID_LIBRARY_OUTPUT_DIR=$(ls -d $ROOT/llvm_build64/lib/clang/* | tail -1)
local ANDROID_EXEC_OUTPUT_DIR=$ROOT/llvm_build64/bin local ANDROID_EXEC_OUTPUT_DIR=$ROOT/llvm_build64/bin
local ANDROID_FLAGS="--target=${_triple}${ANDROID_API} --sysroot=$ANDROID_TOOLCHAIN/sysroot -gcc-toolchain $ANDROID_TOOLCHAIN -B$ANDROID_TOOLCHAIN" local ANDROID_FLAGS="--target=${_triple}${ANDROID_API} --sysroot=$ANDROID_TOOLCHAIN/sysroot --gcc-toolchain=$ANDROID_TOOLCHAIN -B$ANDROID_TOOLCHAIN"
local ANDROID_CXX_FLAGS="$ANDROID_FLAGS -stdlib=libc++ -I/$ANDROID_TOOLCHAIN/sysroot/usr/include/c++/v1" local ANDROID_CXX_FLAGS="$ANDROID_FLAGS -stdlib=libc++ -I/$ANDROID_TOOLCHAIN/sysroot/usr/include/c++/v1"
local CLANG_PATH=$ROOT/llvm_build64/bin/clang local CLANG_PATH=$ROOT/llvm_build64/bin/clang
local CLANGXX_PATH=$ROOT/llvm_build64/bin/clang++ local CLANGXX_PATH=$ROOT/llvm_build64/bin/clang++
# Always clobber android build tree. # Always clobber android build tree.
# It has a hidden dependency on clang (through CXX) which is not known to # It has a hidden dependency on clang (through CXX) which is not known to
# the build system. # the build system.
rm_dirs compiler_rt_build_android_$_arch llvm_build_android_$_arch rm_dirs compiler_rt_build_android_$_arch llvm_build_android_$_arch
mkdir -p compiler_rt_build_android_$_arch mkdir -p compiler_rt_build_android_$_arch
mkdir -p llvm_build_android_$_arch mkdir -p llvm_build_android_$_arch
(cd llvm_build_android_$_arch && cmake \ (cd llvm_build_android_$_arch && cmake \
${CMAKE_COMMON_OPTIONS} \
-DLLVM_ENABLE_WERROR=OFF \ -DLLVM_ENABLE_WERROR=OFF \
-DCMAKE_C_COMPILER=$CLANG_PATH \ -DCMAKE_C_COMPILER=$CLANG_PATH \
-DCMAKE_CXX_COMPILER=$CLANGXX_PATH \ -DCMAKE_CXX_COMPILER=$CLANGXX_PATH \
-DCMAKE_ASM_FLAGS="$ANDROID_FLAGS" \ -DCMAKE_ASM_FLAGS="$ANDROID_FLAGS" \
-DCMAKE_C_FLAGS="$ANDROID_FLAGS" \ -DCMAKE_C_FLAGS="$ANDROID_FLAGS" \
-DCMAKE_CXX_FLAGS="$ANDROID_CXX_FLAGS" \ -DCMAKE_CXX_FLAGS="$ANDROID_CXX_FLAGS" \
-DLLVM_BINUTILS_INCDIR=/usr/include \ -DLLVM_BINUTILS_INCDIR=/usr/include \
-DCMAKE_EXE_LINKER_FLAGS="-pie" \ -DCMAKE_EXE_LINKER_FLAGS="-pie" \
-DCMAKE_SKIP_RPATH=ON \ -DCMAKE_SKIP_RPATH=ON \
-DLLVM_BUILD_RUNTIME=OFF \ -DLLVM_BUILD_RUNTIME=OFF \
-DLLVM_TABLEGEN=$ROOT/llvm_build64/bin/llvm-tblgen \ -DLLVM_TABLEGEN=$ROOT/llvm_build64/bin/llvm-tblgen \
${CMAKE_COMMON_OPTIONS} \ $LLVM || build_failure) &
$LLVM || echo @@@STEP_FAILURE@@@) &
local COMPILER_RT_OPTIONS="$(readlink -f $LLVM/../compiler-rt)" local COMPILER_RT_OPTIONS="$(readlink -f $LLVM/../compiler-rt)"
(cd compiler_rt_build_android_$_arch && cmake \ (cd compiler_rt_build_android_$_arch && cmake \
${CMAKE_COMMON_OPTIONS} \
-DCMAKE_C_COMPILER=$CLANG_PATH \ -DCMAKE_C_COMPILER=$CLANG_PATH \
-DCMAKE_CXX_COMPILER=$CLANGXX_PATH \ -DCMAKE_CXX_COMPILER=$CLANGXX_PATH \
-DLLVM_CONFIG_PATH=$ROOT/llvm_build64/bin/llvm-config \ -DLLVM_CMAKE_DIR=$ROOT/llvm_build64 \
-DCOMPILER_RT_BUILD_BUILTINS=OFF \ -DCOMPILER_RT_BUILD_BUILTINS=OFF \
-DCOMPILER_RT_INCLUDE_TESTS=ON \ -DCOMPILER_RT_INCLUDE_TESTS=ON \
-DCOMPILER_RT_ENABLE_WERROR=ON \ -DCOMPILER_RT_ENABLE_WERROR=ON \
-DCMAKE_ASM_FLAGS="$ANDROID_FLAGS" \ -DCMAKE_ASM_FLAGS="$ANDROID_FLAGS" \
-DCMAKE_C_FLAGS="$ANDROID_FLAGS" \ -DCMAKE_C_FLAGS="$ANDROID_FLAGS" \
-DCMAKE_CXX_FLAGS="$ANDROID_CXX_FLAGS" \ -DCMAKE_CXX_FLAGS="$ANDROID_CXX_FLAGS" \
-DSANITIZER_CXX_ABI="libcxxabi" \ -DSANITIZER_CXX_ABI="libcxxabi" \
-DCOMPILER_RT_TEST_COMPILER_CFLAGS="$ANDROID_FLAGS" \ -DCOMPILER_RT_TEST_COMPILER_CFLAGS="$ANDROID_FLAGS" \
-DCOMPILER_RT_TEST_TARGET_TRIPLE=$_triple \ -DCOMPILER_RT_DEFAULT_TARGET_TRIPLE=$_triple \
-DCOMPILER_RT_OUTPUT_DIR="$ANDROID_LIBRARY_OUTPUT_DIR" \ -DCOMPILER_RT_OUTPUT_DIR="$ANDROID_LIBRARY_OUTPUT_DIR" \
-DCOMPILER_RT_EXEC_OUTPUT_DIR="$ANDROID_EXEC_OUTPUT_DIR" \ -DCOMPILER_RT_EXEC_OUTPUT_DIR="$ANDROID_EXEC_OUTPUT_DIR" \
-DLLVM_LIT_ARGS="-vv --show-unsupported --show-xfail" \ -DLLVM_LIT_ARGS="-vv --show-unsupported --show-xfail" \
${CMAKE_COMMON_OPTIONS} \ ${COMPILER_RT_OPTIONS} || build_failure) &
${COMPILER_RT_OPTIONS} || echo @@@STEP_FAILURE@@@) &
} }
BUILD_RT_ERR="" BUILD_RT_ERR=""
function build_android { function build_android {
local _arch=$1 local _arch=$1
wait wait
echo @@@BUILD_STEP build android/$_arch@@@ echo @@@BUILD_STEP build android/$_arch@@@
if ! ninja -C llvm_build_android_$_arch llvm-symbolizer ; then if ! ninja -C llvm_build_android_$_arch llvm-symbolizer ; then
BUILD_RT_ERR="${BUILD_RT_ERR}|${_arch}|" BUILD_RT_ERR="${BUILD_RT_ERR}|${_arch}|"
echo @@@STEP_FAILURE@@@ build_failure
fi fi
if ! ninja -C compiler_rt_build_android_$_arch ; then if ! ninja -C compiler_rt_build_android_$_arch ; then
BUILD_RT_ERR="${BUILD_RT_ERR}|${_arch}|" BUILD_RT_ERR="${BUILD_RT_ERR}|${_arch}|"
echo @@@STEP_FAILURE@@@ build_failure
fi fi
} }
# If a multiarch device has x86 as the first arch, remove everything else from # If a multiarch device's first arch starts with x86, remove archs that don't
# the list. This captures cases like [x86,armeabi-v7a], where the arm part is # start with x86. Handle cases like:
# software emulation and incompatible with ASan. # - x86,armeabi-v7a
# - x86_64,x86,arm64-v8a,armeabi-v7a,armeabi
# - x86_64,arm64-v8a,x86,armeabi-v7a,armeabi
# The arm part is software emulation and incompatible with ASan.
function patch_abilist { # IN OUT function patch_abilist { # IN OUT
local _abilist=$1 local _abilist=$1
local _out=$2 local _out=$2
if [[ "$_abilist" == "x86,"* ]]; then if [[ "$_abilist" == "x86"* ]]; then
_abilist="x86" _abilist=$(echo $_abilist | tr , '\n' | grep '^x86')
_abilist=$(echo $_abilist | tr ' ' ,)
fi fi
eval $_out="'$_abilist'" eval $_out="'$_abilist'"
} }
function restart_adb_server { function restart_adb_server {
ADB=adb ADB=adb
echo @@@BUILD_STEP restart adb server@@@ echo @@@BUILD_STEP restart adb server@@@
$ADB kill-server $ADB kill-server
sleep 2 sleep 2
$ADB start-server $ADB start-server
sleep 2 sleep 2
} }
function test_on_device { function test_on_device {
local _serial=$1 local _serial=$1
shift shift
ABILIST=$(${ADB} -s $_serial shell getprop ro.product.cpu.abilist) ABILIST=$(${ADB} -s $_serial shell getprop ro.product.cpu.abilist)
patch_abilist $ABILIST ABILIST patch_abilist "$ABILIST" ABILIST
for _arg in "$@"; do for _arg in "$@"; do
local _arch=${_arg%:*} local _arch=${_arg%:*}
local _abi=${_arg#*:} local _abi=${_arg#*:}
if [[ $BUILD_RT_ERR == *"|${_arch}|"* ]]; then if [[ $BUILD_RT_ERR == *"|${_arch}|"* ]]; then
echo "@@@STEP_FAILURE@@ skipping tests on ${_arch}" echo "skipping tests on ${_arch}"
build_failure
continue continue
fi fi
if [[ $ABILIST == *"$_abi"* ]]; then if [[ $ABILIST == *"$_abi"* ]]; then
echo "$_serial" >> tested_arch_$_arch echo "$_serial" >> tested_arch_$_arch
BUILD_ID=$(${ADB} -s $_serial shell getprop ro.build.id | tr -d '\r') BUILD_ID=$(${ADB} -s $_serial shell getprop ro.build.id | tr -d '\r')
BUILD_FLAVOR=$(${ADB} -s $_serial shell getprop ro.build.flavor | tr -d '\r') BUILD_FLAVOR=$(${ADB} -s $_serial shell getprop ro.build.flavor | tr -d '\r')
(
# Test only one arch at a time to avoid simultaneously writes into the
# same compiler-rt build dir.
flock -x $lock_fd
test_arch_on_device "$_arch" "$_serial" "$BUILD_ID" "$BUILD_FLAVOR" test_arch_on_device "$_arch" "$_serial" "$BUILD_ID" "$BUILD_FLAVOR"
) {lock_fd}>$ROOT/${_arch}.lock
fi fi
done done
} }
function tail_pids { function tail_pids {
for LOG_PID in $1; do for LOG_PID in $1; do
PID=${LOG_PID#*,} PID=${LOG_PID#*,}
LOG=${LOG_PID%,*} LOG=${LOG_PID%,*}
tail -n +1 -F $LOG --pid=$PID tail -n +1 -F $LOG --pid=$PID
done done
wait wait
} }
function test_android { function test_android {
if [[ "${BUILDBOT_SLAVENAME:-}" != "" ]]; then if [[ -v BUILDBOT_BUILDERNAME ]]; then
restart_adb_server restart_adb_server
fi fi
ADB=adb ADB=adb
echo @@@BUILD_STEP run all tests@@@ echo @@@BUILD_STEP run all tests@@@
ANDROID_DEVICES=$(${ADB} devices | grep 'device$' | sort -r | awk '{print $1}') ANDROID_DEVICES=$(${ADB} devices | grep 'device$' | sort -r | awk '{print $1}')
rm -rf test_android_log_* rm -rf test_android_log_*
rm -rf tested_arch_* rm -rf tested_arch_*
rm -rf shards_* rm -rf shards_*
LOGS=
for SERIAL in $ANDROID_DEVICES; do for SERIAL in $ANDROID_DEVICES; do
LOG="$(mktemp test_android_log_XXXX)" test_on_device "$SERIAL" $@
(test_on_device "$SERIAL" $@ 2>&1 >"$LOG") &
LOGS="$LOGS $LOG,$!"
done done
tail_pids "$LOGS"
# Return to avoid exception if we already have error.
[[ $BUILD_RT_ERR == "" ]] || return
for _arg in "$@"; do for _arg in "$@"; do
local _arch=${_arg%:*} local _arch=${_arg%:*}
if [[ ! -f tested_arch_$_arch ]]; then if [[ ! -f tested_arch_$_arch ]]; then
echo @@@BUILD_STEP unavailable device android/$_arch@@@ echo @@@BUILD_STEP unavailable device android/$_arch@@@
echo @@@STEP_EXCEPTION@@@ echo @@@STEP_EXCEPTION@@@
fi fi
done done
Show All 12 Linesfunction run_tests_sharded {
local _env=$3 local _env=$3
local NUM_SHARDS=4 local NUM_SHARDS=4
local _log_prefix=$(mktemp shards_XXXX_) local _log_prefix=$(mktemp shards_XXXX_)
echo @@@BUILD_STEP run $_test_name tests [$DEVICE_DESCRIPTION]@@@ echo @@@BUILD_STEP run $_test_name tests [$DEVICE_DESCRIPTION]@@@
LOGS= LOGS=
for ((SHARD=0; SHARD < $NUM_SHARDS; SHARD++)); do for ((SHARD=0; SHARD < $NUM_SHARDS; SHARD++)); do
LOG=${_log_prefix}_$SHARD LOG=${_log_prefix}_$SHARD
local ENV="$_env GTEST_TOTAL_SHARDS=$NUM_SHARDS GTEST_SHARD_INDEX=$SHARD LD_LIBRARY_PATH=$DEVICE_ROOT" # 'adb shell <command>' on Fugu is missing TMPDIR.
( (run_command_on_device "$ENV $DEVICE_ROOT/$_test" || echo @@@STEP_FAILURE@@@) 2>&1 >${_log_prefix}_$SHARD ) & local ENV="$_env GTEST_TOTAL_SHARDS=$NUM_SHARDS GTEST_SHARD_INDEX=$SHARD LD_LIBRARY_PATH=$DEVICE_ROOT TMPDIR=$DEVICE_TMPDIR"
( (run_command_on_device "$ENV $DEVICE_ROOT/$_test" || build_failure) 2>&1 >${_log_prefix}_$SHARD ) &
LOGS="$LOGS $LOG,$!" LOGS="$LOGS $LOG,$!"
done done
tail_pids "$LOGS" || true tail_pids "$LOGS" || true
} }
function test_arch_on_device { function test_arch_on_device {
local _arch=$1 local _arch=$1
local _serial=$2 local _serial=$2
local _build_id=$3 local _build_id=$3
local _build_flavor=$4 local _build_flavor=$4
export DEVICE_DESCRIPTION=$_arch/$_build_flavor/$_build_id export DEVICE_DESCRIPTION=$_arch/$_build_flavor/$_build_id
local LIBCXX_SHARED=$ANDROID_TOOLCHAIN/sysroot/usr/lib/${_arch}-linux-androideabi/libc++_shared.so local LIBCXX_SHARED=$ANDROID_TOOLCHAIN/sysroot/usr/lib/${_arch}-linux-androideabi/libc++_shared.so
if [[ ! -f $LIBCXX_SHARED ]] ; then if [[ ! -f $LIBCXX_SHARED ]] ; then
local LIBCXX_SHARED=$ANDROID_TOOLCHAIN/sysroot/usr/lib/${_arch}-linux-android/libc++_shared.so local LIBCXX_SHARED=$ANDROID_TOOLCHAIN/sysroot/usr/lib/${_arch}-linux-android/libc++_shared.so
fi fi
local SYMBOLIZER_BIN=$ROOT/llvm_build_android_$_arch/bin/llvm-symbolizer local SYMBOLIZER_BIN=$ROOT/llvm_build_android_$_arch/bin/llvm-symbolizer
local RT_DIR=$($ROOT/llvm_build64/bin/clang -print-resource-dir)/lib/linux local RT_DIR=$($ROOT/llvm_build64/bin/clang -print-resource-dir)/lib/linux
local COMPILER_RT_BUILD_DIR=$ROOT/compiler_rt_build_android_$_arch local COMPILER_RT_BUILD_DIR=$ROOT/compiler_rt_build_android_$_arch
export ADB=adb export ADB=adb
export DEVICE_TMPDIR=/data/local/tmp
export DEVICE_ROOT=/data/local/tmp/Output export DEVICE_ROOT=/data/local/tmp/Output
export ANDROID_SERIAL=$_serial export ANDROID_SERIAL=$_serial
echo "Serial $_serial" echo "Serial $_serial"
echo @@@BUILD_STEP device setup [$DEVICE_DESCRIPTION]@@@ echo @@@BUILD_STEP device setup [$DEVICE_DESCRIPTION]@@@
$ADB wait-for-device $ADB wait-for-device
$ADB devices $ADB devices
$ADB shell getprop ro.build.version.release
# Kill leftover symbolizers. TODO: figure out what's going on. # Kill leftover symbolizers. TODO: figure out what's going on.
$ADB shell pkill llvm-symbolizer || true $ADB shell pkill llvm-symbolizer || true
$ADB shell rm -rf $DEVICE_ROOT $ADB shell rm -rf $DEVICE_ROOT
$ADB shell mkdir $DEVICE_ROOT $ADB shell mkdir $DEVICE_ROOT
FILES="$SYMBOLIZER_BIN FILES="$SYMBOLIZER_BIN
$RT_DIR/libclang_rt.*-android.so $RT_DIR/libclang_rt.*-android.so
$LIBCXX_SHARED $LIBCXX_SHARED
$COMPILER_RT_BUILD_DIR/lib/sanitizer_common/tests/SanitizerTest $COMPILER_RT_BUILD_DIR/lib/sanitizer_common/tests/SanitizerTest
$COMPILER_RT_BUILD_DIR/lib/asan/tests/AsanTest $COMPILER_RT_BUILD_DIR/lib/asan/tests/AsanTest
$COMPILER_RT_BUILD_DIR/lib/asan/tests/AsanNoinstTest" $COMPILER_RT_BUILD_DIR/lib/asan/tests/AsanNoinstTest"
for F in $FILES ; do for F in $FILES ; do
( $ADB push $F $DEVICE_ROOT/ >/dev/null || echo @@@STEP_FAILURE@@@ )& ( $ADB push $F $DEVICE_ROOT/ >/dev/null || build_failure )&
done done
wait wait
echo @@@BUILD_STEP run lit tests [$DEVICE_DESCRIPTION]@@@ echo @@@BUILD_STEP run lit tests [$DEVICE_DESCRIPTION]@@@
(cd $COMPILER_RT_BUILD_DIR && ninja check-all) || echo @@@STEP_FAILURE@@@ (cd $COMPILER_RT_BUILD_DIR && ninja check-all) || build_failure
run_tests_sharded sanitizer_common SanitizerTest "" run_tests_sharded sanitizer_common SanitizerTest ""
run_tests_sharded asan AsanNoinstTest "" run_tests_sharded asan AsanNoinstTest ""
run_tests_sharded "instrumented asan" AsanTest "ASAN_OPTIONS=start_deactivated=1" run_tests_sharded "instrumented asan" AsanTest "ASAN_OPTIONS=start_deactivated=1"
} }

zorg/buildbot/builders/sanitizers/buildbot_bisect_run.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/usr/bin/env bash
# This script uses BUILDBOT_REVISION to read bad and good patch for bisect.
# The format is: BUILDBOT_REVISION=good:bad
set -x
set -e
set -u
HERE="$(cd $(dirname $0) && pwd)"
. ${HERE}/buildbot_functions.sh
BUILDBOT_REVISION=origin/main buildbot_update
echo @@@BUILD_STEP bisecting ${BUILDBOT_REVISION}@@@
# Try to get them out from the bisect string in BUILDBOT_REVISION first.
GOOD="${BUILDBOT_REVISION/:*/}"
BAD="${BUILDBOT_REVISION/*:/}"
# If not provided through BUILDBOT_REVISION, use some defaults.
if [[ "$BAD" == "" ]]; then
BAD="origin/main"
fi
if [[ "$GOOD" == "" ]]; then
GOOD="origin/main~100"
fi
cd "${LLVM}/.."
(
export BUILDBOT_BISECT_MODE=1
if git bisect start $BAD $GOOD; then
git bisect run bash -c "cd $ROOT && $*"
fi
echo @@@BUILD_STEP bisect result@@@
git bisect log
) || true
git bisect reset
build_exception # Bisect is neither FAIL nor PASS.

zorg/buildbot/builders/sanitizers/buildbot_bisect_test.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/usr/bin/env bash
set -x
set -e
set -u
HERE="$(cd $(dirname $0) && pwd)"
. ${HERE}/buildbot_functions.sh
buildbot_update
if git -c $LLVM merge-base HEAD "${1}" | grep "${1}"; then
build_failure
fi

zorg/buildbot/builders/sanitizers/buildbot_bootstrap_asan.sh

#!/usr/bin/env bash #!/usr/bin/env bash
set -x set -x
set -e set -e
set -u set -u
HERE="$(cd $(dirname $0) && pwd)" HERE="$(cd $(dirname $0) && pwd)"
. ${HERE}/buildbot_functions.sh . ${HERE}/buildbot_functions.sh
ROOT=`pwd` CMAKE_COMMON_OPTIONS+=" -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF"
PLATFORM=`uname`
export PATH="/usr/local/bin:$PATH"
LLVM=$ROOT/llvm
CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
clobber clobber
buildbot_update buildbot_update
# Stage 1 # Stage 1
build_stage1_clang build_stage1_clang
check_stage1_asan
# Stage 2 / AddressSanitizer # Stage 2 / AddressSanitizer
build_stage2_asan build_stage2_asan
check_stage2_asan check_stage2_asan
# Stage 3 / AddressSanitizer # Stage 3 / AddressSanitizer
export ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1" export ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1"
build_stage3_asan build_stage3_asan
check_stage3_asan check_stage3_asan
cleanup

zorg/buildbot/builders/sanitizers/buildbot_bootstrap_hwasan.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/usr/bin/env bash
set -x
set -e
set -u
HERE="$(cd $(dirname $0) && pwd)"
. ${HERE}/buildbot_functions.sh
CMAKE_COMMON_OPTIONS+=" -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF"
clobber
buildbot_update
# Stage 1
build_stage1_clang
check_stage1_hwasan
# Stage 2 / HWAddressSanitizer
build_stage2_hwasan
check_stage2_hwasan
# Stage 3 / HWAddressSanitizer
build_stage3_hwasan
check_stage3_hwasan
cleanup

zorg/buildbot/builders/sanitizers/buildbot_bootstrap_msan.sh

#!/usr/bin/env bash #!/usr/bin/env bash
set -x set -x
set -e set -e
set -u set -u
HERE="$(cd $(dirname $0) && pwd)" HERE="$(cd $(dirname $0) && pwd)"
. ${HERE}/buildbot_functions.sh . ${HERE}/buildbot_functions.sh
ROOT=`pwd` CMAKE_COMMON_OPTIONS+=" -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF"
PLATFORM=`uname`
export PATH="/usr/local/bin:$PATH"
LLVM=$ROOT/llvm
CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
clobber clobber
buildbot_update buildbot_update
# Stage 1 # Stage 1
build_stage1_clang build_stage1_clang
check_stage1_msan
for I in 1 2 ; do
# Stage 2 / Memory Sanitizer # Stage 2 / Memory Sanitizer
build_stage2_msan build_stage2_msan
check_stage2_msan (
check_stage2_msan |& tee stage2_msan.log
exit ${PIPESTATUS[0]}
)
if grep "WARNING: MemorySanitizer" stage2_msan.log ; then
# Stage 2 / MemoryWithOriginsSanitizer
(
build_stage2_msan_track_origins
check_stage2_msan_track_origins
)
fi
# Stage 3 / MemorySanitizer # Stage 3 / MemorySanitizer
(
{
build_stage3_msan build_stage3_msan
check_stage3_msan check_stage3_msan
} |& tee stage3_msan.log
exit ${PIPESTATUS[0]}
)
if grep "WARNING: MemorySanitizer" stage3_msan.log ; then
# Stage 3 / MemoryWithOriginsSanitizer
(
build_stage2_msan_track_origins
build_stage3_msan_track_origins
check_stage3_msan_track_origins
)
fi
# FIXME: stage3/msan check_1 crashes.
break
cleanup
# Repeat with strict settings.
CMAKE_COMMON_OPTIONS+=" -DCMAKE_C_FLAGS=-fno-inline"
CMAKE_COMMON_OPTIONS+=" -DCMAKE_CXX_FLAGS=-fno-inline"
CMAKE_COMMON_OPTIONS+=" -DCMAKE_C_FLAGS_RELEASE=-Oz"
CMAKE_COMMON_OPTIONS+=" -DCMAKE_CXX_FLAGS_RELEASE=-Oz"
done
cleanup
No newline at end of file

zorg/buildbot/builders/sanitizers/buildbot_bootstrap_ubsan.sh

#!/usr/bin/env bash #!/usr/bin/env bash
set -x set -x
set -e set -e
set -u set -u
HERE="$(cd $(dirname $0) && pwd)" HERE="$(cd $(dirname $0) && pwd)"
. ${HERE}/buildbot_functions.sh . ${HERE}/buildbot_functions.sh
ROOT=`pwd` CMAKE_COMMON_OPTIONS+=" -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF"
PLATFORM=`uname`
export PATH="/usr/local/bin:$PATH"
LLVM=$ROOT/llvm
CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
clobber clobber
buildbot_update buildbot_update
# Stage 1 # Stage 1
build_stage1_clang build_stage1_clang
check_stage1_ubsan
# Stage 2 / UndefinedBehaviorSanitizer # Stage 2 / UndefinedBehaviorSanitizer
build_stage2_ubsan build_stage2_ubsan
check_stage2_ubsan check_stage2_ubsan
# Stage 3 / UndefinedBehaviorSanitizer # Stage 3 / UndefinedBehaviorSanitizer
build_stage3_ubsan build_stage3_ubsan
check_stage3_ubsan check_stage3_ubsan
cleanup

zorg/buildbot/builders/sanitizers/buildbot_cmake.sh

#!/usr/bin/env bash #!/usr/bin/env bash
set -x set -x
set -e set -e
set -u set -u
HERE="$(dirname $0)" HERE="$(dirname $0)"
. ${HERE}/buildbot_functions.sh . ${HERE}/buildbot_functions.sh
ROOT=`pwd`
PLATFORM=`uname` PLATFORM=`uname`
ARCH=`uname -m` ARCH=`uname -m`
export PATH="/usr/local/bin:$PATH" ninja --version || export PATH="/usr/local/bin:$PATH"
export ANDROID_SDK_HOME=$ROOT/../../.. export ANDROID_SDK_HOME=$ROOT/../../..
# Always clobber bootstrap build trees. CMAKE_ARGS=""
rm -rf compiler_rt_build llvm_build* symbolizer_build* for arg in "$@"
do
case $arg in
--CMAKE_ARGS=*)
CMAKE_ARGS="${arg#*=}"
esac
done
USE_CCACHE= # Always clobber bootstrap build trees.
if ccache -s ; then clobber
USE_CCACHE="-DLLVM_CCACHE_BUILD=ON"
rm -rf clang_build
fi
SUPPORTS_32_BITS=${SUPPORTS_32_BITS:-1} SUPPORTS_32_BITS=${SUPPORTS_32_BITS:-1}
MAKE_JOBS=${MAX_MAKE_JOBS:-$(nproc)}
LLVM=$ROOT/llvm
ZLIB=$ROOT/zlib ZLIB=$ROOT/zlib
CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS:-} -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_PARALLEL_LINK_JOBS=10" CMAKE_COMMON_OPTIONS+=" -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_PARALLEL_LINK_JOBS=10 -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF ${CMAKE_ARGS}"
ENABLE_LIBCXX_FLAG= ENABLE_LIBCXX_FLAG=
if [ "$PLATFORM" == "Darwin" ]; then if [ "$PLATFORM" == "Darwin" ]; then
CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS} -DPYTHON_EXECUTABLE=/usr/bin/python" CMAKE_COMMON_OPTIONS+=" -DPYTHON_EXECUTABLE=/usr/bin/python"
ENABLE_LIBCXX_FLAG="-DLLVM_ENABLE_LIBCXX=ON" ENABLE_LIBCXX_FLAG="-DLLVM_ENABLE_LIBCXX=ON"
fi fi
if [ -e /usr/include/plugin-api.h ]; then if [ -e /usr/include/plugin-api.h ]; then
CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_BINUTILS_INCDIR=/usr/include" CMAKE_COMMON_OPTIONS+=" -DLLVM_BINUTILS_INCDIR=/usr/include"
fi fi
CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_PROJECTS='clang;compiler-rt'" CMAKE_COMMON_OPTIONS+=" -DLLVM_ENABLE_PROJECTS='clang;compiler-rt'"
CHECK_LIBCXX=${CHECK_LIBCXX:-1} CHECK_LIBCXX=${CHECK_LIBCXX:-1}
CHECK_SYMBOLIZER=${CHECK_SYMBOLIZER:-$CHECK_LIBCXX} CHECK_SYMBOLIZER=${CHECK_SYMBOLIZER:-$CHECK_LIBCXX}
CHECK_LLD=${CHECK_LLD:-1} CHECK_LLD=${CHECK_LLD:-1}
CHECK_ASAN=0 CHECK_ASAN=0
CHECK_HWASAN=0 CHECK_HWASAN=0
CHECK_UBSAN=0 CHECK_UBSAN=0
CHECK_MSAN=0 CHECK_MSAN=0
CHECK_TSAN=0 CHECK_TSAN=0
CHECK_LSAN=0 CHECK_LSAN=0
CHECK_DFSAN=0 CHECK_DFSAN=0
CHECK_SCUDO=0
CHECK_SCUDO_STANDALONE=0 CHECK_SCUDO_STANDALONE=0
CHECK_CFI=0 CHECK_CFI=0
case "$ARCH" in case "$ARCH" in
x86_64) x86_64)
CHECK_ASAN=1 CHECK_ASAN=1
CHECK_HWASAN=1 CHECK_HWASAN=1
CHECK_UBSAN=1 CHECK_UBSAN=1
CHECK_MSAN=1 CHECK_MSAN=1
CHECK_TSAN=1 CHECK_TSAN=1
CHECK_LSAN=1 CHECK_LSAN=1
CHECK_DFSAN=1 CHECK_DFSAN=1
CHECK_SCUDO=1
CHECK_SCUDO_STANDALONE=1 CHECK_SCUDO_STANDALONE=1
CHECK_CFI=1 CHECK_CFI=1
;; ;;
aarch64) aarch64)
CHECK_ASAN=1 CHECK_ASAN=1
CHECK_UBSAN=1 CHECK_UBSAN=1
CHECK_MSAN=1 CHECK_MSAN=1
CHECK_TSAN=1 CHECK_TSAN=1
CHECK_LSAN=1 CHECK_LSAN=1
CHECK_DFSAN=1 CHECK_DFSAN=1
CHECK_SCUDO=1
;; ;;
mips64*) mips64*)
CHECK_ASAN=1 CHECK_ASAN=1
CHECK_UBSAN=1 CHECK_UBSAN=1
CHECK_MSAN=1 CHECK_MSAN=1
CHECK_TSAN=1 CHECK_TSAN=1
CHECK_LSAN=1 CHECK_LSAN=1
CHECK_DFSAN=1 CHECK_DFSAN=1
;; ;;
ppc64*) ppc64*)
CHECK_ASAN=1 CHECK_ASAN=1
CHECK_UBSAN=1 CHECK_UBSAN=1
CHECK_MSAN=1 CHECK_MSAN=1
CHECK_TSAN=1 CHECK_TSAN=1
CHECK_SCUDO=1 CMAKE_COMMON_OPTIONS+=" -DLLVM_TARGETS_TO_BUILD=PowerPC"
if [[ "$ARCH" == "ppc64le" ]]; then
CMAKE_COMMON_OPTIONS+=" -DLLVM_LIT_ARGS=-vj256"
else
CMAKE_COMMON_OPTIONS+=" -DLLVM_LIT_ARGS=-vj80"
fi
;; ;;
i*86) i*86)
CHECK_UBSAN=1 CHECK_UBSAN=1
CHECK_ASAN=1 CHECK_ASAN=1
;; ;;
mips*) mips*)
CHECK_UBSAN=1 CHECK_UBSAN=1
CHECK_ASAN=1 CHECK_ASAN=1
;; ;;
arm*) arm*)
CHECK_UBSAN=1 CHECK_UBSAN=1
CHECK_ASAN=1 CHECK_ASAN=1
;; ;;
s390x) s390x)
CHECK_UBSAN=1 CHECK_UBSAN=1
;; ;;
esac esac
PROJECTS="clang;compiler-rt;libcxx;libcxxabi" PROJECTS="clang;compiler-rt"
if [[ "$CHECK_LLD" != "0" ]]; then if [[ "$CHECK_LLD" != "0" ]]; then
PROJECTS="${PROJECTS};lld" PROJECTS+=";lld"
fi fi
CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_PROJECTS='${PROJECTS}'" CMAKE_COMMON_OPTIONS+=" -DLLVM_ENABLE_PROJECTS='${PROJECTS}' -DLLVM_ENABLE_RUNTIMES=libcxx;libcxxabi"
if [[ "$CHECK_CFI" == "1" ]]; then if [[ "$CHECK_CFI" == "1" ]]; then
# We need this after https://reviews.llvm.org/D62050 # We need this after https://reviews.llvm.org/D62050
CMAKE_COMMON_OPTIONS="$CMAKE_COMMON_OPTIONS -DLLVM_BUILD_LLVM_DYLIB=ON" CMAKE_COMMON_OPTIONS+=" -DLLVM_BUILD_LLVM_DYLIB=ON"
fi fi
CLOBBER="zlib clang_build"
clobber
buildbot_update buildbot_update
COMPILER_RT=$LLVM/../compiler-rt COMPILER_RT=$LLVM/../compiler-rt
LIBCXX=$LLVM/../libcxx LIBCXX=$LLVM/../libcxx
echo @@@BUILD_STEP lint@@@
CHECK_LINT=${COMPILER_RT}/lib/sanitizer_common/scripts/check_lint.sh
${CHECK_LINT} || echo @@@STEP_WARNINGS@@@
# Use both gcc and just-built Clang/LLD as a host compiler/linker for sanitizer # Use both gcc and just-built Clang/LLD as a host compiler/linker for sanitizer
# tests. Assume that self-hosted build tree should compile with -Werror. # tests. Assume that self-hosted build tree should compile with -Werror.
echo @@@BUILD_STEP build fresh toolchain@@@ echo @@@BUILD_STEP build fresh toolchain@@@
if [ ! -d clang_build ]; then mkdir -p clang_build
mkdir clang_build (cd clang_build && cmake ${CMAKE_COMMON_OPTIONS} $LLVM ) || (rm -rf clang_build ; build_failure)
fi
(cd clang_build && cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ${CMAKE_COMMON_OPTIONS} ${USE_CCACHE} $LLVM ) || (rm -rf clang_build ; echo @@@STEP_FAILURE@@@)
BOOTSTRAP_BUILD_TARGETS="clang" BOOTSTRAP_BUILD_TARGETS="clang"
if [[ "$CHECK_LLD" != "0" ]]; then if [[ "$CHECK_LLD" != "0" ]]; then
BOOTSTRAP_BUILD_TARGETS="$BOOTSTRAP_BUILD_TARGETS lld" BOOTSTRAP_BUILD_TARGETS="$BOOTSTRAP_BUILD_TARGETS lld"
fi fi
(cd clang_build && make $BOOTSTRAP_BUILD_TARGETS -j$MAKE_JOBS) || (echo @@@STEP_FAILURE@@@ ; exit 1) || echo @@@STEP_FAILURE@@@ (cd clang_build && ninja $BOOTSTRAP_BUILD_TARGETS) || build_failure
# If we're building with libcxx, install the headers to clang_build/include. # If we're building with libcxx, install the headers to clang_build/include.
if [ ! -z ${ENABLE_LIBCXX_FLAG} ]; then if [ ! -z ${ENABLE_LIBCXX_FLAG} ]; then
(cd clang_build && make -C ${LIBCXX} installheaders \ (cd clang_build && ninja -C ${LIBCXX} installheaders \
HEADER_DIR=${PWD}/include) || echo @@@STEP_FAILURE@@@ HEADER_DIR=${PWD}/include) || build_failure
fi fi
# Do a sanity check on Linux: build and test sanitizers using gcc as a host # Check on Linux: build and test sanitizers using gcc as a host
# compiler. # compiler.
if [ "$PLATFORM" == "Linux" ]; then if [ "$PLATFORM" == "Linux" ]; then
check_in_gcc() { check_in_gcc() {
CONDITION=$1 CONDITION=$1
SANITIZER=$2 SANITIZER=$2
if [ "$CONDITION" == "1" ]; then if [ "$CONDITION" == "1" ]; then
echo @@@BUILD_STEP check-$SANITIZER in gcc build@@@ echo @@@BUILD_STEP check-$SANITIZER in gcc build@@@
(cd clang_build && make -j$MAKE_JOBS check-$SANITIZER) || echo @@@STEP_FAILURE@@@ (cd clang_build && ninja check-$SANITIZER) || build_failure
fi fi
} }
check_in_gcc 1 sanitizer check_in_gcc 1 sanitizer
check_in_gcc $CHECK_ASAN asan check_in_gcc $CHECK_ASAN asan
check_in_gcc $CHECK_HWASAN hwasan check_in_gcc $CHECK_HWASAN hwasan
check_in_gcc $CHECK_CFI cfi-and-supported check_in_gcc $CHECK_CFI cfi-and-supported
check_in_gcc $CHECK_DFSAN dfsan check_in_gcc $CHECK_DFSAN dfsan
check_in_gcc $CHECK_LSAN lsan check_in_gcc $CHECK_LSAN lsan
check_in_gcc $CHECK_MSAN msan check_in_gcc $CHECK_MSAN msan
check_in_gcc $CHECK_SCUDO scudo
check_in_gcc $CHECK_SCUDO_STANDALONE scudo_standalone check_in_gcc $CHECK_SCUDO_STANDALONE scudo_standalone
LDFLAGS=-no-pie check_in_gcc $CHECK_TSAN tsan LDFLAGS=-no-pie check_in_gcc $CHECK_TSAN tsan
check_in_gcc $CHECK_UBSAN ubsan check_in_gcc $CHECK_UBSAN ubsan
check_in_gcc $CHECK_UBSAN ubsan-minimal check_in_gcc $CHECK_UBSAN ubsan-minimal
fi fi
### From now on we use just-built Clang as a host compiler ### ### From now on we use just-built Clang as a host compiler ###
CLANG_PATH=${ROOT}/clang_build/bin CLANG_PATH=${ROOT}/clang_build/bin
# Build self-hosted tree with fresh Clang and -Werror. # Build self-hosted tree with fresh Clang and -Werror.
CMAKE_CLANG_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_WERROR=ON -DCMAKE_C_COMPILER=${CLANG_PATH}/clang -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++ -DCMAKE_C_FLAGS=-gmlt -DCMAKE_CXX_FLAGS=-gmlt" CMAKE_CLANG_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_WERROR=ON -DCMAKE_C_COMPILER=${CLANG_PATH}/clang -DCMAKE_CXX_COMPILER=${CLANG_PATH}/clang++ -DCMAKE_C_FLAGS=-gmlt -DCMAKE_CXX_FLAGS=-gmlt"
BUILD_TYPE=Release
echo @@@BUILD_STEP bootstrap clang@@@ echo @@@BUILD_STEP bootstrap clang@@@
if [ ! -d llvm_build64 ]; then mkdir -p llvm_build64
mkdir llvm_build64 (cd llvm_build64 && cmake ${CMAKE_CLANG_OPTIONS} \
fi -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \
(cd llvm_build64 && cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE \ ${ENABLE_LIBCXX_FLAG} \
${CMAKE_CLANG_OPTIONS} -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \ $LLVM) || build_failure
${ENABLE_LIBCXX_FLAG} $LLVM) || echo @@@STEP_FAILURE@@@
# First, build only Clang. # First, build only Clang.
(cd llvm_build64 && make -j$MAKE_JOBS clang) || echo @@@STEP_FAILURE@@@ (cd llvm_build64 && ninja clang) || build_failure
# If needed, install the headers to clang_build/include. # If needed, install the headers to clang_build/include.
if [ ! -z ${ENABLE_LIBCXX_FLAG} ]; then if [ ! -z ${ENABLE_LIBCXX_FLAG} ]; then
(cd llvm_build64 && make -C ${LIBCXX} installheaders \ (cd llvm_build64 && ninja -C ${LIBCXX} installheaders \
HEADER_DIR=${PWD}/include) || echo @@@STEP_FAILURE@@@ HEADER_DIR=${PWD}/include) || build_failure
fi fi
# Now build everything else. # Now build everything else.
(cd llvm_build64 && make -j$MAKE_JOBS) || echo @@@STEP_FAILURE@@@ (cd llvm_build64 && ninja) || build_failure
# Symbolizer dependencies. # Symbolizer dependencies.
(cd llvm_build64 && make -j$MAKE_JOBS llvm-ar llvm-link llvm-tblgen opt) || echo @@@STEP_FAILURE@@@ (cd llvm_build64 && ninja llvm-ar llvm-link llvm-tblgen opt) || build_failure
if [ "$CHECK_CFI" == "1" ]; then if [ "$CHECK_CFI" == "1" ]; then
# FIXME: Make these true dependencies of check-cfi-and-supported when # FIXME: Make these true dependencies of check-cfi-and-supported when
# compiler-rt is configured as an external project. # compiler-rt is configured as an external project.
(cd llvm_build64 && make -j$MAKE_JOBS LLVMgold opt sanstats) || echo @@@STEP_FAILURE@@@ (cd llvm_build64 && ninja LLVMgold opt sanstats) || build_failure
fi fi
check_64bit() { check_64bit() {
CONDITION=$1 CONDITION=$1
SANITIZER=$2 SANITIZER=$2
if [ "$CONDITION" == "1" ]; then if [ "$CONDITION" == "1" ]; then
echo @@@BUILD_STEP 64-bit check-$SANITIZER@@@ echo @@@BUILD_STEP 64-bit check-$SANITIZER@@@
(cd llvm_build64 && make -j$MAKE_JOBS check-$SANITIZER) || echo @@@STEP_FAILURE@@@ (cd llvm_build64 && ninja check-$SANITIZER) || build_failure
fi fi
} }
check_64bit 1 sanitizer check_64bit 1 sanitizer
check_64bit $CHECK_ASAN asan check_64bit $CHECK_ASAN asan
if [ "$PLATFORM" == "Linux" ]; then if [ "$PLATFORM" == "Linux" ]; then
check_64bit $CHECK_ASAN asan-dynamic check_64bit $CHECK_ASAN asan-dynamic
check_64bit $CHECK_HWASAN hwasan check_64bit $CHECK_HWASAN hwasan
check_64bit $CHECK_CFI cfi-and-supported check_64bit $CHECK_CFI cfi-and-supported
check_64bit $CHECK_DFSAN dfsan check_64bit $CHECK_DFSAN dfsan
check_64bit $CHECK_LSAN lsan check_64bit $CHECK_LSAN lsan
check_64bit $CHECK_MSAN msan check_64bit $CHECK_MSAN msan
# check_64bit $CHECK_SCUDO scudo, No check-scudo target for this config
check_64bit $CHECK_TSAN tsan check_64bit $CHECK_TSAN tsan
check_64bit $CHECK_UBSAN ubsan check_64bit $CHECK_UBSAN ubsan
check_64bit $CHECK_UBSAN ubsan-minimal check_64bit $CHECK_UBSAN ubsan-minimal
fi fi
FRESH_CLANG_PATH=${ROOT}/llvm_build64/bin FRESH_CLANG_PATH=${ROOT}/llvm_build64/bin
echo @@@BUILD_STEP build standalone compiler-rt@@@ echo @@@BUILD_STEP build standalone compiler-rt@@@
if [ ! -d compiler_rt_build ]; then if [ ! -d compiler_rt_build ]; then
mkdir compiler_rt_build mkdir compiler_rt_build
fi fi
(cd compiler_rt_build && cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE \ (cd compiler_rt_build && cmake -GNinja \
-DCMAKE_C_COMPILER=${FRESH_CLANG_PATH}/clang \ -DCMAKE_C_COMPILER=${FRESH_CLANG_PATH}/clang \
-DCMAKE_CXX_COMPILER=${FRESH_CLANG_PATH}/clang++ \ -DCMAKE_CXX_COMPILER=${FRESH_CLANG_PATH}/clang++ \
-DCOMPILER_RT_INCLUDE_TESTS=ON \ -DCOMPILER_RT_INCLUDE_TESTS=ON \
-DCOMPILER_RT_ENABLE_WERROR=ON \ -DCOMPILER_RT_ENABLE_WERROR=ON \
-DLLVM_CONFIG_PATH=${FRESH_CLANG_PATH}/llvm-config \ -DLLVM_CMAKE_DIR=${ROOT}/llvm_build64 \
$COMPILER_RT) || echo @@@STEP_FAILURE@@@ $COMPILER_RT) || build_failure
(cd compiler_rt_build && make -j$MAKE_JOBS) || echo @@@STEP_FAILURE@@@ (cd compiler_rt_build && ninja) || build_failure
echo @@@BUILD_STEP test standalone compiler-rt@@@ echo @@@BUILD_STEP test standalone compiler-rt@@@
(cd compiler_rt_build && make -j$MAKE_JOBS check-all) || echo @@@STEP_FAILURE@@@ (cd compiler_rt_build && ninja check-all) || build_failure
build_symbolizer() {
echo @@@BUILD_STEP build $1-bit symbolizer for $2@@@
if [ ! -d symbolizer_build$1 ]; then
mkdir symbolizer_build$1
fi
(cd symbolizer_build$1 && ZLIB_SRC=$ZLIB FLAGS=-m$1 \
CLANG=${FRESH_CLANG_PATH}/clang \
bash -eux $COMPILER_RT/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh \
$(dirname $(find ../$2/ -name libclang_rt.*.a | head -n1)) || echo @@@STEP_FAILURE@@@)
}
if [ "$CHECK_SYMBOLIZER" == "1" ]; then
echo @@@BUILD_STEP update zlib@@@
(cd $ZLIB && git pull --rebase) || \
git clone https://github.com/madler/zlib.git $ZLIB || echo @@@STEP_FAILURE@@@
build_symbolizer 32 compiler_rt_build
build_symbolizer 64 compiler_rt_build
echo @@@BUILD_STEP test standalone compiler-rt with symbolizer@@@
(cd compiler_rt_build && make -j$MAKE_JOBS check-all) || echo @@@STEP_FAILURE@@@
fi
HAVE_NINJA=${HAVE_NINJA:-1} HAVE_NINJA=${HAVE_NINJA:-1}
if [ "$PLATFORM" == "Linux" -a $HAVE_NINJA == 1 ]; then if [ "$PLATFORM" == "Linux" -a $HAVE_NINJA == 1 ]; then
echo @@@BUILD_STEP build with ninja@@@ echo @@@BUILD_STEP build with ninja@@@
if [ ! -d llvm_build_ninja ]; then if [ ! -d llvm_build_ninja ]; then
mkdir llvm_build_ninja mkdir llvm_build_ninja
fi fi
CMAKE_NINJA_OPTIONS="${CMAKE_CLANG_OPTIONS} -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -G Ninja" CMAKE_NINJA_OPTIONS="${CMAKE_CLANG_OPTIONS} -GNinja"
(cd llvm_build_ninja && cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE \ (cd llvm_build_ninja && cmake \
${CMAKE_NINJA_OPTIONS} $LLVM) || echo @@@STEP_FAILURE@@@ ${CMAKE_NINJA_OPTIONS} $LLVM) || build_failure
ln -sf llvm_build_ninja/compile_commands.json $LLVM ln -sf llvm_build_ninja/compile_commands.json $LLVM
(cd llvm_build_ninja && ninja) || echo @@@STEP_FAILURE@@@ (cd llvm_build_ninja && ninja) || build_failure
check_ninja() { check_ninja() {
CONDITION=$1 CONDITION=$1
SANITIZER=$2 SANITIZER=$2
if [ "$CONDITION" == "1" ]; then if [ "$CONDITION" == "1" ]; then
echo @@@BUILD_STEP ninja check-$SANITIZER@@@ echo @@@BUILD_STEP ninja check-$SANITIZER@@@
(cd llvm_build_ninja && ninja check-$SANITIZER) || echo @@@STEP_FAILURE@@@ (cd llvm_build_ninja && ninja check-$SANITIZER) || build_failure
fi fi
} }
check_ninja 1 sanitizer check_ninja 1 sanitizer
check_ninja $CHECK_ASAN asan check_ninja $CHECK_ASAN asan
check_ninja $CHECK_HWASAN hwasan check_ninja $CHECK_HWASAN hwasan
check_ninja $CHECK_CFI cfi-and-supported check_ninja $CHECK_CFI cfi-and-supported
check_ninja $CHECK_DFSAN dfsan check_ninja $CHECK_DFSAN dfsan
check_ninja $CHECK_LSAN lsan check_ninja $CHECK_LSAN lsan
check_ninja $CHECK_MSAN msan check_ninja $CHECK_MSAN msan
check_ninja $CHECK_SCUDO scudo
check_ninja $CHECK_SCUDO_STANDALONE scudo_standalone check_ninja $CHECK_SCUDO_STANDALONE scudo_standalone
check_ninja $CHECK_TSAN tsan check_ninja $CHECK_TSAN tsan
check_ninja $CHECK_UBSAN ubsan check_ninja $CHECK_UBSAN ubsan
check_ninja $CHECK_UBSAN ubsan-minimal check_ninja $CHECK_UBSAN ubsan-minimal
if [ "$CHECK_SYMBOLIZER" == "1" ]; then if [ "$CHECK_SYMBOLIZER" == "1" ]; then
build_symbolizer() {
echo @@@BUILD_STEP build $1-bit symbolizer for $2@@@
if [ ! -d symbolizer_build$1 ]; then
mkdir symbolizer_build$1
fi
(cd symbolizer_build$1 && ZLIB_SRC=$ZLIB FLAGS=-m$1 \
CLANG=${FRESH_CLANG_PATH}/clang \
bash -eux $COMPILER_RT/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh \
$(dirname $(find ../$2/ -name libclang_rt.*.a | head -n1)) || build_failure)
}
echo @@@BUILD_STEP update zlib@@@
(cd $ZLIB && git pull --rebase) || \
git clone https://github.com/madler/zlib.git $ZLIB || build_failure
build_symbolizer 32 llvm_build_ninja build_symbolizer 32 llvm_build_ninja
build_symbolizer 64 llvm_build_ninja build_symbolizer 64 llvm_build_ninja
check_ninja_with_symbolizer() { check_ninja_with_symbolizer() {
CONDITION=$1 CONDITION=$1
SANITIZER=$2 SANITIZER=$2
# Disabled, tests are not working yet. if [ "$CONDITION" == "1" ]; then
if [ "$CONDITION" == "-1" ]; then echo @@@BUILD_STEP ninja check-$SANITIZER with internal symbolizer@@@
echo @@@BUILD_STEP ninja check-$SANITIZER with symbolizer@@@ (cd llvm_build_ninja && ninja check-$SANITIZER) || build_failure
(cd llvm_build_ninja && ninja check-$SANITIZER) || echo @@@STEP_FAILURE@@@
fi fi
} }
check_ninja_with_symbolizer 1 sanitizer # TODO: Replace LIT_FILTER_OUT with lit features.
check_ninja_with_symbolizer $CHECK_ASAN asan LIT_FILTER_OUT=":: (max_allocation_size.cpp|Linux/soft_rss_limit_mb_test.cpp)$" check_ninja_with_symbolizer 1 sanitizer
check_ninja_with_symbolizer $CHECK_HWASAN hwasan LIT_FILTER_OUT=":: TestCases/Linux/check_memcpy.c$" check_ninja_with_symbolizer $CHECK_ASAN asan
check_ninja_with_symbolizer $CHECK_CFI cfi-and-supported # check_ninja_with_symbolizer $CHECK_HWASAN hwasan
# check_ninja_with_symbolizer $CHECK_CFI cfi-and-supported
check_ninja_with_symbolizer $CHECK_DFSAN dfsan check_ninja_with_symbolizer $CHECK_DFSAN dfsan
check_ninja_with_symbolizer $CHECK_LSAN lsan LIT_FILTER_OUT=":: TestCases/(realloc_too_big.c|recoverable_leak_check.cpp|suppressions_file.cpp)$" check_ninja_with_symbolizer $CHECK_LSAN lsan
check_ninja_with_symbolizer $CHECK_MSAN msan LIT_FILTER_OUT=":: Linux/check_memcpy.c$" check_ninja_with_symbolizer $CHECK_MSAN msan
check_ninja_with_symbolizer $CHECK_SCUDO scudo
check_ninja_with_symbolizer $CHECK_SCUDO_STANDALONE scudo_standalone check_ninja_with_symbolizer $CHECK_SCUDO_STANDALONE scudo_standalone
check_ninja_with_symbolizer $CHECK_TSAN tsan LIT_FILTER_OUT=":: Linux/check_memcpy.c$" check_ninja_with_symbolizer $CHECK_TSAN tsan
check_ninja_with_symbolizer $CHECK_UBSAN ubsan LIT_FILTER_OUT=":: TestCases/TypeCheck/(vptr-virtual-base.cpp|vptr.cpp)$" check_ninja_with_symbolizer $CHECK_UBSAN ubsan
fi fi
fi fi
cleanup

zorg/buildbot/builders/sanitizers/buildbot_fast.sh

#!/usr/bin/env bash #!/usr/bin/env bash
set -x set -x
set -e set -e
set -u set -u
HERE="$(cd $(dirname $0) && pwd)" HERE="$(cd $(dirname $0) && pwd)"
. ${HERE}/buildbot_functions.sh . ${HERE}/buildbot_functions.sh
ROOT=`pwd`
PLATFORM=`uname`
export PATH="/usr/local/bin:$PATH"
LLVM=$ROOT/llvm
CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON"
clobber clobber
# Stage 1 # Stage 1
build_stage1_clang_at_revison build_stage1_clang_at_revison
if ccache -s ; then CMAKE_COMMON_OPTIONS+=" -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF"
CMAKE_COMMON_OPTIONS="${CMAKE_COMMON_OPTIONS} -DLLVM_CCACHE_BUILD=ON"
fi
buildbot_update
# Stage 2 / Memory Sanitizer # Some of them are slow.
STAGE2_SKIP_TEST_CXX=1
build_stage2_msan buildbot_update
check_stage2_msan # Stage 2 / Address and Undefined Sanitizer
# Stage 2 / AddressSanitizer build_stage2_asan_ubsan
build_stage2_asan check_stage2_asan_ubsan
check_stage2_asan # Stage 2 / Memory Sanitizer
# Stage 2 / UndefinedBehaviorSanitizer CMAKE_COMMON_OPTIONS+=" -DCMAKE_C_FLAGS=-fno-inline"
CMAKE_COMMON_OPTIONS+=" -DCMAKE_CXX_FLAGS=-fno-inline"
CMAKE_COMMON_OPTIONS+=" -DCMAKE_C_FLAGS_RELEASE=-Oz"
CMAKE_COMMON_OPTIONS+=" -DCMAKE_CXX_FLAGS_RELEASE=-Oz"
build_stage2_msan
build_stage2_ubsan check_stage2_msan
check_stage2_ubsan cleanup

zorg/buildbot/builders/sanitizers/buildbot_functions.sh

#!/usr/bin/env bash #!/usr/bin/env bash
ROOT=`pwd`
LLVM=$ROOT/llvm
BUILDBOT_CLOBBER="${BUILDBOT_CLOBBER:-}"
BUILDBOT_REVISION="${BUILDBOT_REVISION:-origin/main}"
HOST_CLANG_REVISION=llvmorg-$(curl https://api.github.com/repos/llvm/llvm-project/releases/latest -s | jq .name -r | cut -f2 -d' ')
CMAKE_COMMON_OPTIONS+=" -DLLVM_APPEND_VC_REV=OFF -GNinja -DCMAKE_BUILD_TYPE=Release"
export LC_ALL=C
if ccache -s ; then
CMAKE_COMMON_OPTIONS+=" -DLLVM_CCACHE_BUILD=ON"
fi
if lld --version ; then
CMAKE_COMMON_OPTIONS+=" -DLLVM_USE_LINKER=lld"
fi
function include_config() {
local P=.
while true ; do
local F=${P}/sanitizer_buildbot_config
if [[ -f ${F} ]] ; then
. ${F}
break
fi
[[ "${P}" -ef '/' ]] && break
P="${P}/.."
done
}
include_config
echo @@@BUILD_STEP Info@@@ echo @@@BUILD_STEP Info@@@
( (
set +e set +e
env env | sort
echo echo
uptime uptime
echo echo
ulimit -n 100000 ulimit -n 1000000
ulimit -a ulimit -a
echo echo
df -h df -h
echo echo
ccache -s ccache -ps
echo
echo "How to reproduce locally: https://github.com/google/sanitizers/wiki/SanitizerBotReproduceBuild"
echo
exit 0 exit 0
) )
echo @@@BUILD_STEP Prepare@@@ echo @@@BUILD_STEP Prepare@@@
BUILDBOT_CLOBBER="${BUILDBOT_CLOBBER:-}" export LIT_OPTS="--time-tests"
BUILDBOT_REVISION="${BUILDBOT_REVISION:-origin/master}"
function rm_dirs { function rm_dirs {
while ! rm -rf $@ ; do sleep 1; done while ! rm -rf $@ ; do sleep 1; done
} }
function stage1_clobber { function cleanup() {
rm_dirs llvm_build2_* llvm_build_* libcxx_build_* ${STAGE1_CLOBBER:-} [[ -v BUILDBOT_BUILDERNAME ]] || return 0
echo @@@BUILD_STEP cleanup@@@
rm_dirs llvm_build2_* llvm_build_* libcxx_build_* compiler_rt_build* symbolizer_build* $@
if ccache -s >/dev/null ; then
rm_dirs llvm_build64 clang_build
fi
ls
} }
function clobber { function clobber {
if [ "$BUILDBOT_CLOBBER" != "" ]; then if [[ "$BUILDBOT_CLOBBER" != "" ]]; then
echo @@@BUILD_STEP clobber@@@ echo @@@BUILD_STEP clobber@@@
rm_dirs svn_checkout llvm llvm-project llvm_build0 ${CLOBBER:-} if [[ ! -v BUILDBOT_BUILDERNAME ]]; then
stage1_clobber echo "Clobbering is supported only on buildbot only!"
! test "$(ls -A .)" || echo @@@STEP_EXCEPTION@@@ exit 1
fi
rm_dirs *
else
BUILDBOT_BUILDERNAME=1 cleanup $@
fi fi
} }
BUILDBOT_MONO_REPO_PATH=${BUILDBOT_MONO_REPO_PATH:-} BUILDBOT_MONO_REPO_PATH=${BUILDBOT_MONO_REPO_PATH:-}
function buildbot_update { function buildbot_update {
echo @@@BUILD_STEP update $BUILDBOT_REVISION@@@ if [[ "${BUILDBOT_BISECT_MODE:-}" == "1" ]]; then
echo "@@@BUILD_STEP bisect status@@@"
(
cd llvm-project
git bisect visualize --oneline
git bisect log
git status
)
LLVM=$ROOT/llvm-project/llvm
return 0
fi
echo "@@@BUILD_STEP update $BUILDBOT_REVISION@@@"
if [[ -d "$BUILDBOT_MONO_REPO_PATH" ]]; then if [[ -d "$BUILDBOT_MONO_REPO_PATH" ]]; then
LLVM=$BUILDBOT_MONO_REPO_PATH/llvm LLVM=$BUILDBOT_MONO_REPO_PATH/llvm
else else
( (
local DEPTH=100 local DEPTH=100
[[ -d llvm-project ]] || ( [[ -d llvm-project ]] || (
mkdir -p llvm-project mkdir -p llvm-project
cd llvm-project cd llvm-project
git init git init
git remote add origin https://github.com/llvm/llvm-project.git git remote add origin https://github.com/llvm/llvm-project.git
git config --local advice.detachedHead false
) )
cd llvm-project cd llvm-project
git fetch --depth $DEPTH origin master git fetch origin
git clean -fd git clean -fd
local REV=${BUILDBOT_REVISION} git reset --hard
if [[ "$REV" != "origin/master" ]] ; then git checkout -f "${BUILDBOT_REVISION}"
# "git fetch --depth 1 origin $REV" does not work with 2.11 on bots
while true ; do
git checkout $REV && break
git rev-list --pretty --max-count=1 origin/master
git rev-list --pretty --max-parents=0 origin/master
echo "DEPTH=$DEPTH is too small"
[[ "$DEPTH" -le "1000000" ]] || exit 1
DEPTH=$(( $DEPTH * 10 ))
git fetch --depth $DEPTH origin
done
fi
git checkout $REV
git status git status
git rev-list --pretty --max-count=1 HEAD git rev-list --pretty --max-count=1 HEAD
) || { echo @@@STEP_EXCEPTION@@@ ; exit 1 ; } ) || { build_exception ; exit 1 ; }
LLVM=$ROOT/llvm-project/llvm LLVM=$ROOT/llvm-project/llvm
fi fi
} }
function common_stage1_variables { function common_stage1_variables {
STAGE1_DIR=llvm_build0 STAGE1_DIR=llvm_build0
stage1_clang_path=$ROOT/${STAGE1_DIR}/bin stage1_clang_path=$ROOT/${STAGE1_DIR}/bin
llvm_symbolizer_path=${stage1_clang_path}/llvm-symbolizer llvm_symbolizer_path=${stage1_clang_path}/llvm-symbolizer
export LLVM_SYMBOLIZER_PATH="${llvm_symbolizer_path}"
STAGE1_AS_COMPILER="-DCMAKE_C_COMPILER=${stage1_clang_path}/clang -DCMAKE_CXX_COMPILER=${stage1_clang_path}/clang++" STAGE1_AS_COMPILER="-DCMAKE_C_COMPILER=${stage1_clang_path}/clang -DCMAKE_CXX_COMPILER=${stage1_clang_path}/clang++"
} }
function build_stage1_clang_impl { function build_stage1_clang_impl {
rm -rf ${STAGE1_DIR} [[ -f "${STAGE1_DIR}/delete_next_time" ]] && rm -rf "${STAGE1_DIR}"
mkdir -p ${STAGE1_DIR} mkdir -p ${STAGE1_DIR}
local cmake_stage1_options="${CMAKE_COMMON_OPTIONS}" local cmake_stage1_options="${CMAKE_COMMON_OPTIONS} -DLLVM_ENABLE_PROJECTS='clang;compiler-rt;lld'"
cmake_stage1_options="${cmake_stage1_options} -DLLVM_ENABLE_PROJECTS='clang;compiler-rt;lld'" if clang -v ; then
if ccache -s ; then cmake_stage1_options+=" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++"
cmake_stage1_options="${cmake_stage1_options} -DLLVM_CCACHE_BUILD=ON"
fi fi
(cd ${STAGE1_DIR} && cmake ${cmake_stage1_options} $LLVM && ninja) (cd ${STAGE1_DIR} && cmake ${cmake_stage1_options} $LLVM && ninja) || {
touch "${STAGE1_DIR}/delete_next_time"
return 1
}
md5sum ${STAGE1_DIR}/bin/clang || true
} }
function build_stage1_clang { function build_stage1_clang {
echo @@@BUILD_STEP stage1 build all@@@ echo @@@BUILD_STEP stage1 build all@@@
export STAGE1_DIR=llvm_build0
common_stage1_variables common_stage1_variables
build_stage1_clang_impl build_stage1_clang_impl
echo @@@BUILD_STEP Clobber stage1 users
stage1_clobber
} }
function build_stage1_clang_at_revison { function download_clang_from_chromium {
common_stage1_variables common_stage1_variables
curl -s https://raw.githubusercontent.com/chromium/chromium/master/tools/clang/scripts/update.py \ curl -s https://raw.githubusercontent.com/chromium/chromium/main/tools/clang/scripts/update.py \
| python - --output-dir=${STAGE1_DIR} | python3 - --output-dir=${STAGE1_DIR}
echo @@@BUILD_STEP using pre-built stage1 clang at $(cat ${STAGE1_DIR}/cr_build_revision)@@@ echo @@@BUILD_STEP using pre-built stage1 clang at $(cat ${STAGE1_DIR}/cr_build_revision)@@@
} }
function build_clang_at_release_tag {
common_stage1_variables
if [ -r ${STAGE1_DIR}/host_clang_revision ] && \
[ "$(cat ${STAGE1_DIR}/host_clang_revision)" == $HOST_CLANG_REVISION ]
then
echo "@@@BUILD_STEP using pre-built stage1 clang at r${HOST_CLANG_REVISION}@@@"
else
BUILDBOT_MONO_REPO_PATH= BUILDBOT_REVISION="${HOST_CLANG_REVISION}" buildbot_update
rm -rf ${STAGE1_DIR}
echo @@@BUILD_STEP build stage1 clang at $HOST_CLANG_REVISION@@@
build_stage1_clang_impl && \
( echo $HOST_CLANG_REVISION > ${STAGE1_DIR}/host_clang_revision )
fi
}
function build_stage1_clang_at_revison {
build_clang_at_release_tag
}
function common_stage2_variables { function common_stage2_variables {
cmake_stage2_common_options="\ cmake_stage2_common_options="${CMAKE_COMMON_OPTIONS} ${STAGE1_AS_COMPILER}"
${CMAKE_COMMON_OPTIONS} ${STAGE1_AS_COMPILER} -DLLVM_USE_LINKER=lld"
} }
function build_stage2 { function build_stage2 {
local sanitizer_name=$1 local sanitizer_name=$1
local step_result=$2 echo @@@BUILD_STEP stage2/$sanitizer_name build libcxx@@@
local libcxx_build_dir=libcxx_build_${sanitizer_name} local libcxx_build_dir=libcxx_build_${sanitizer_name}
local build_dir=llvm_build_${sanitizer_name} local build_dir=llvm_build_${sanitizer_name}
export STAGE2_DIR=${build_dir} export STAGE2_DIR=${build_dir}
local cmake_libcxx_cflags=
common_stage2_variables common_stage2_variables
local fno_sanitize_flag=
if [ "$sanitizer_name" == "msan" ]; then if [ "$sanitizer_name" == "msan" ]; then
export MSAN_SYMBOLIZER_PATH="${llvm_symbolizer_path}" export MSAN_SYMBOLIZER_PATH="${llvm_symbolizer_path}"
local llvm_use_sanitizer="Memory" llvm_use_sanitizer="Memory"
local fsanitize_flag="-fsanitize=memory" fsanitize_flag="-fsanitize=memory"
BUILDBOT_MSAN_ORIGINS=${BUILDBOT_MSAN_ORIGINS:-} elif [ "$sanitizer_name" == "msan_track_origins" ]; then
if [ "$BUILDBOT_MSAN_ORIGINS" != "" ]; then export MSAN_SYMBOLIZER_PATH="${llvm_symbolizer_path}"
llvm_use_sanitizer="MemoryWithOrigins" llvm_use_sanitizer="MemoryWithOrigins"
fi fsanitize_flag="-fsanitize=memory -fsanitize-memory-track-origins"
local build_type="Release"
elif [ "$sanitizer_name" == "asan" ]; then elif [ "$sanitizer_name" == "asan" ]; then
export ASAN_SYMBOLIZER_PATH="${llvm_symbolizer_path}" export ASAN_SYMBOLIZER_PATH="${llvm_symbolizer_path}"
export ASAN_OPTIONS="check_initialization_order=true:detect_stack_use_after_return=1:detect_leaks=1" export ASAN_OPTIONS="check_initialization_order=true"
local llvm_use_sanitizer="Address" llvm_use_sanitizer="Address"
local fsanitize_flag="-fsanitize=address" fsanitize_flag="-fsanitize=address"
local build_type="Release" # FIXME: False ODR violations in libcxx tests.
# https://github.com/google/sanitizers/issues/1017
cmake_libcxx_cflags="-mllvm -asan-use-private-alias=1"
elif [ "$sanitizer_name" == "hwasan" ]; then
export HWASAN_SYMBOLIZER_PATH="${llvm_symbolizer_path}"
export HWASAN_OPTIONS="abort_on_error=1"
llvm_use_sanitizer="HWAddress"
fsanitize_flag="-fsanitize=hwaddress -mllvm -hwasan-use-after-scope=1"
# FIXME: Support globals with DSO https://github.com/llvm/llvm-project/issues/57206
cmake_stage2_common_options+=" -DLLVM_ENABLE_PLUGINS=OFF"
elif [ "$sanitizer_name" == "ubsan" ]; then elif [ "$sanitizer_name" == "ubsan" ]; then
export UBSAN_OPTIONS="external_symbolizer_path=${llvm_symbolizer_path}:print_stacktrace=1" export UBSAN_OPTIONS="external_symbolizer_path=${llvm_symbolizer_path}:print_stacktrace=1"
local llvm_use_sanitizer="Undefined" llvm_use_sanitizer="Undefined"
local fsanitize_flag="-fsanitize=undefined" fsanitize_flag="-fsanitize=undefined"
local build_type="Release" # FIXME: After switching to LLVM_ENABLE_RUNTIMES, vptr has infitine
# recursion.
fno_sanitize_flag+=" -fno-sanitize=vptr"
elif [ "$sanitizer_name" == "asan_ubsan" ]; then
export ASAN_SYMBOLIZER_PATH="${llvm_symbolizer_path}"
export ASAN_OPTIONS="check_initialization_order=true"
llvm_use_sanitizer="Address;Undefined"
fsanitize_flag="-fsanitize=address,undefined"
# FIXME: After switching to LLVM_ENABLE_RUNTIMES, vptr has infitine
# recursion.
fno_sanitize_flag+=" -fno-sanitize=vptr"
else else
echo "Unknown sanitizer!" echo "Unknown sanitizer!"
exit 1 exit 1
fi fi
# Don't use libc++/libc++abi in UBSan builds (due to known bugs).
echo @@@BUILD_STEP stage2/$sanitizer_name build libcxx@@@
rm -rf ${libcxx_build_dir}
mkdir -p ${libcxx_build_dir} mkdir -p ${libcxx_build_dir}
(cd ${libcxx_build_dir} && \ (cd ${libcxx_build_dir} && \
cmake \ cmake \
${cmake_stage2_common_options} \ ${cmake_stage2_common_options} \
-DLLVM_ENABLE_PROJECTS='libcxx;libcxxabi' \ -DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi' \
-DCMAKE_BUILD_TYPE=${build_type} \
-DLLVM_USE_SANITIZER=${llvm_use_sanitizer} \ -DLLVM_USE_SANITIZER=${llvm_use_sanitizer} \
$LLVM && \ -DCMAKE_C_FLAGS="${fsanitize_flag} ${cmake_libcxx_cflags} ${fno_sanitize_flag}" \
ninja cxx cxxabi) || echo $step_result -DCMAKE_CXX_FLAGS="${fsanitize_flag} ${cmake_libcxx_cflags} ${fno_sanitize_flag}" \
local sanitizer_ldflags="-lc++abi -Wl,--rpath=${ROOT}/${libcxx_build_dir}/lib -L${ROOT}/${libcxx_build_dir}/lib" $LLVM/../runtimes && \
local sanitizer_cflags="-nostdinc++ -isystem ${ROOT}/${libcxx_build_dir}/include -isystem ${ROOT}/${libcxx_build_dir}/include/c++/v1" ninja cxx cxxabi) || build_failure
local libcxx_runtime_path=$(dirname $(find ${ROOT}/${libcxx_build_dir} -name libc++.so))
local sanitizer_ldflags="-Wl,--rpath=${libcxx_runtime_path} -L${libcxx_runtime_path}"
local sanitizer_cflags="-nostdinc++ -isystem ${ROOT}/${libcxx_build_dir}/include -isystem ${ROOT}/${libcxx_build_dir}/include/c++/v1 $fsanitize_flag"
echo @@@BUILD_STEP stage2/$sanitizer_name build@@@ echo @@@BUILD_STEP stage2/$sanitizer_name build@@@
# See http://llvm.org/bugs/show_bug.cgi?id=19071, http://www.cmake.org/Bug/view.php?id=15264 # See http://llvm.org/bugs/show_bug.cgi?id=19071, http://www.cmake.org/Bug/view.php?id=15264
local cmake_bug_workaround_cflags="$sanitizer_ldflags $fsanitize_flag -w" sanitizer_cflags+=" $sanitizer_ldflags -w"
sanitizer_cflags="$sanitizer_cflags $cmake_bug_workaround_cflags"
rm -rf ${build_dir}
mkdir -p ${build_dir} mkdir -p ${build_dir}
local cmake_stage2_clang_options="-DLLVM_ENABLE_PROJECTS='clang;lld;clang-tools-extra'" local cmake_stage2_clang_options="-DLLVM_ENABLE_PROJECTS='clang;lld;clang-tools-extra;mlir'"
if [[ "$(arch)" == "aarch64" ]] ; then
# FIXME: clangd tests fail.
cmake_stage2_clang_options="-DLLVM_ENABLE_PROJECTS='clang;lld;mlir'"
fi
(cd ${build_dir} && \ (cd ${build_dir} && \
cmake \ cmake \
${cmake_stage2_common_options} \ ${cmake_stage2_common_options} \
${cmake_stage2_clang_options} \ ${cmake_stage2_clang_options} \
-DCMAKE_BUILD_TYPE=${build_type} \
-DLLVM_USE_SANITIZER=${llvm_use_sanitizer} \ -DLLVM_USE_SANITIZER=${llvm_use_sanitizer} \
-DLLVM_ENABLE_LIBCXX=ON \ -DLLVM_ENABLE_LIBCXX=ON \
-DCMAKE_C_FLAGS="${sanitizer_cflags}" \ -DCMAKE_C_FLAGS="${sanitizer_cflags}" \
-DCMAKE_CXX_FLAGS="${sanitizer_cflags}" \ -DCMAKE_CXX_FLAGS="${sanitizer_cflags}" \
-DCMAKE_EXE_LINKER_FLAGS="${sanitizer_ldflags}" \ -DCMAKE_EXE_LINKER_FLAGS="${sanitizer_ldflags}" \
$LLVM && \ $LLVM && \
ninja) || echo $step_result time ninja) || build_failure
md5sum ${build_dir}/bin/clang || true
} }
function build_stage2_msan { function build_stage2_msan {
build_stage2 msan @@@STEP_FAILURE@@@ build_stage2 msan
}
function build_stage2_msan_track_origins {
build_stage2 msan_track_origins
} }
function build_stage2_asan { function build_stage2_asan {
build_stage2 asan @@@STEP_FAILURE@@@ build_stage2 asan
}
function build_stage2_hwasan {
build_stage2 hwasan
} }
function build_stage2_ubsan { function build_stage2_ubsan {
build_stage2 ubsan @@@STEP_FAILURE@@@ build_stage2 ubsan
}
function build_stage2_asan_ubsan {
build_stage2 asan_ubsan
}
function check_stage1 {
local sanitizer_name=$1
echo @@@BUILD_STEP stage1/$sanitizer_name check-sanitizer@@@
ninja -C ${STAGE1_DIR} check-sanitizer || build_failure
# Uses by asan and hwasan.
echo @@@BUILD_STEP stage1/$sanitizer_name check-lsan@@@
ninja -C ${STAGE1_DIR} check-lsan || build_failure
echo @@@BUILD_STEP stage1/$sanitizer_name check-${sanitizer_name}@@@
ninja -C ${STAGE1_DIR} check-${sanitizer_name} || build_failure
}
function check_stage1_msan {
check_stage1 msan
}
function check_stage1_asan {
check_stage1 asan
}
function check_stage1_hwasan {
check_stage1 hwasan
}
function check_stage1_ubsan {
check_stage1 ubsan
}
function check_stage1_asan_ubsan {
check_stage1 asan_ubsan
} }
function check_stage2 { function check_stage2 {
local sanitizer_name=$1 local sanitizer_name=$1
local step_result=$2
local build_dir=${STAGE2_DIR} if [[ "${STAGE2_SKIP_TEST_CXX:-}" != "1" ]] ; then
(
# Very slow, run in background.
LIT_OPTS+=" --timeout=1500"
(
# Very slow.
export LIT_FILTER_OUT="modules_include.sh.cpp"
LIT_FILTER_OUT+="|std/algorithms/alg.modifying.operations/alg.transform/ranges.transform.pass.cpp"
LIT_FILTER_OUT+="|std/containers/sequences/deque/deque.modifiers/insert_iter_iter.pass.cpp"
LIT_FILTER_OUT+="|std/numerics/rand/rand.dist/rand.dist.bern/rand.dist.bern.negbin/eval.pass.cpp"
LIT_FILTER_OUT+="|std/numerics/rand/rand.dist/rand.dist.samp/rand.dist.samp.discrete/eval.pass.cpp"
LIT_FILTER_OUT+="|std/utilities/charconv/charconv.msvc/test.pass.cpp"
LIT_FILTER_OUT+="|std/utilities/format/format.functions/format_to_n.locale.pass.cpp"
LIT_FILTER_OUT+="|std/utilities/format/format.functions/format_to_n.pass.cpp"
LIT_FILTER_OUT+="|std/utilities/format/format.functions/format_to.locale.pass.cpp"
LIT_FILTER_OUT+="|std/utilities/format/format.functions/format_to.pass.cpp"
LIT_FILTER_OUT+="|std/utilities/format/format.functions/format.locale.pass.cpp"
LIT_FILTER_OUT+="|std/utilities/format/format.functions/format.pass.cpp"
LIT_FILTER_OUT+="|std/utilities/format/format.functions/formatted_size.locale.pass.cpp"
LIT_FILTER_OUT+="|std/utilities/format/format.functions/formatted_size.pass.cpp"
LIT_FILTER_OUT+="|std/utilities/format/format.functions/vformat"
LIT_FILTER_OUT+="|std/utilities/variant/variant.visit/visit_return_type.pass.cpp"
LIT_FILTER_OUT+="|std/utilities/variant/variant.visit/visit.pass.cpp"
if [[ "$(arch)" == "aarch64" && "$sanitizer_name" == "asan" ]] ; then
# TODO: Investigate one leak and two slowest tests.
LIT_FILTER_OUT+="|test_vector2.pass.cpp|catch_multi_level_pointer.pass.cpp"
LIT_FILTER_OUT+="|guard_threaded_test.pass.cpp"
fi
if [[ "$(arch)" == "aarch64" && "$sanitizer_name" == "msan" ]] ; then
# TODO: Investigate one slow tests.
LIT_FILTER_OUT+="|catch_multi_level_pointer.pass.cpp"
LIT_FILTER_OUT+="|guard_threaded_test.pass.cpp"
LIT_FILTER_OUT+="|test_demangle.pass.cpp"
fi
if [[ "$(arch)" == "aarch64" && "$sanitizer_name" == "hwasan" ]] ; then
# TODO: Investigate one slow tests.
LIT_FILTER_OUT+="|catch_multi_level_pointer.pass.cpp"
LIT_FILTER_OUT+="|guard_threaded_test.pass.cpp"
LIT_FILTER_OUT+="|test_demangle.pass.cpp"
LIT_FILTER_OUT+="|test_vector2.pass.cpp"
LIT_FILTER_OUT+="|forced_unwind2.pass.cpp"
fi
ninja -C libcxx_build_${sanitizer_name} check-cxx check-cxxabi
) || build_failure
) &>check_cxx.log &
fi
echo @@@BUILD_STEP stage2/$sanitizer_name check@@@ echo @@@BUILD_STEP stage2/$sanitizer_name check@@@
ninja -C ${build_dir} check-all || echo $step_result ninja -C ${STAGE2_DIR} check-all || build_failure
if [[ "${STAGE2_SKIP_TEST_CXX:-}" != "1" ]] ; then
echo @@@BUILD_STEP stage2/$sanitizer_name check-cxx@@@
wait
sleep 5
cat check_cxx.log
fi
} }
function check_stage2_msan { function check_stage2_msan {
check_stage2 msan @@@STEP_FAILURE@@@ check_stage2 msan
}
function check_stage2_msan_track_origins {
check_stage2 msan_track_origins
} }
function check_stage2_asan { function check_stage2_asan {
check_stage2 asan @@@STEP_FAILURE@@@ check_stage2 asan
}
function check_stage2_hwasan {
check_stage2 hwasan
} }
function check_stage2_ubsan { function check_stage2_ubsan {
check_stage2 ubsan @@@STEP_FAILURE@@@ check_stage2 ubsan
}
function check_stage2_asan_ubsan {
check_stage2 asan_ubsan
} }
function build_stage3 { function build_stage3 {
local sanitizer_name=$1 local sanitizer_name=$1
local step_result=$2 echo @@@BUILD_STEP build stage3/$sanitizer_name build@@@
local build_dir=llvm_build2_${sanitizer_name} local build_dir=llvm_build2_${sanitizer_name}
local clang_path=$ROOT/${STAGE2_DIR}/bin local clang_path=$ROOT/${STAGE2_DIR}/bin
echo @@@BUILD_STEP build stage3/$sanitizer_name build@@@ local sanitizer_cflags=
rm -rf ${build_dir}
mkdir -p ${build_dir} mkdir -p ${build_dir}
local stage3_projects='clang;lld;clang-tools-extra'
if [[ "$(arch)" == "aarch64" ]] ; then
# FIXME: clangd tests fail.
stage3_projects='clang;lld'
fi
(cd ${build_dir} && \ (cd ${build_dir} && \
cmake \ cmake \
${CMAKE_COMMON_OPTIONS} \ ${CMAKE_COMMON_OPTIONS} \
-DLLVM_ENABLE_PROJECTS='clang;lld,clang-tools-extra' \ -DLLVM_ENABLE_PROJECTS="${stage3_projects}" \
-DCMAKE_C_COMPILER=${clang_path}/clang \ -DCMAKE_C_COMPILER=${clang_path}/clang \
-DCMAKE_CXX_COMPILER=${clang_path}/clang++ \ -DCMAKE_CXX_COMPILER=${clang_path}/clang++ \
-DLLVM_USE_LINKER=lld \ -DCMAKE_CXX_FLAGS="${sanitizer_cflags}" \
-DLLVM_CCACHE_BUILD=OFF \
$LLVM && \ $LLVM && \
ninja clang) || echo $step_result time ninja) || build_failure
md5sum ${build_dir}/bin/clang* || true
} }
function build_stage3_msan { function build_stage3_msan {
build_stage3 msan @@@STEP_FAILURE@@@ build_stage3 msan
}
function build_stage3_msan_track_origins {
build_stage3 msan_track_origins
} }
function build_stage3_asan { function build_stage3_asan {
build_stage3 asan @@@STEP_FAILURE@@@ build_stage3 asan
}
function build_stage3_hwasan {
build_stage3 hwasan
} }
function build_stage3_ubsan { function build_stage3_ubsan {
build_stage3 ubsan @@@STEP_FAILURE@@@ build_stage3 ubsan
} }
function check_stage3 { function check_stage3 {
local sanitizer_name=$1 local sanitizer_name=$1
local step_result=$2 echo @@@BUILD_STEP stage3/$sanitizer_name check@@@
local build_dir=llvm_build2_${sanitizer_name} local build_dir=llvm_build2_${sanitizer_name}
echo @@@BUILD_STEP stage3/$sanitizer_name check@@@ (cd ${build_dir} && env && ninja check-all) || build_failure
(cd ${build_dir} && ninja check-all) || echo $step_result
} }
function check_stage3_msan { function check_stage3_msan {
check_stage3 msan @@@STEP_FAILURE@@@ check_stage3 msan
}
function check_stage3_msan_track_origins {
check_stage3 msan_track_origins
} }
function check_stage3_asan { function check_stage3_asan {
check_stage3 asan @@@STEP_FAILURE@@@ check_stage3 asan
}
function check_stage3_hwasan {
check_stage3 hwasan
} }
function check_stage3_ubsan { function check_stage3_ubsan {
check_stage3 ubsan @@@STEP_FAILURE@@@ check_stage3 ubsan
}
function build_failure() {
# In bisect mode exit early.
echo
echo "How to reproduce locally: https://github.com/google/sanitizers/wiki/SanitizerBotReproduceBuild"
echo
sleep 5
echo "@@@STEP_FAILURE@@@"
if [[ "${BUILDBOT_BISECT_MODE:-}" == "1" ]] ; then
exit 1
fi
}
function build_exception() {
echo
echo "How to reproduce locally: https://github.com/google/sanitizers/wiki/SanitizerBotReproduceBuild"
echo
sleep 5
echo "@@@STEP_EXCEPTION@@@"
} }

zorg/buildbot/builders/sanitizers/buildbot_fuzzer.sh

#!/usr/bin/env bash #!/usr/bin/env bash
set -x set -x
set -e set -e
set -u set -u
HERE="$(cd $(dirname $0) && pwd)" HERE="$(cd $(dirname $0) && pwd)"
. ${HERE}/buildbot_functions.sh . ${HERE}/buildbot_functions.sh
ROOT=`pwd`
PLATFORM=`uname`
export PATH="/usr/local/bin:$PATH"
MAKE_JOBS=${MAX_MAKE_JOBS:-$(nproc)} MAKE_JOBS=${MAX_MAKE_JOBS:-$(nproc)}
LLVM=$ROOT/llvm
LIBFUZZER=$LLVM/lib/Fuzzer LIBFUZZER=$LLVM/lib/Fuzzer
# No assertions. Need to clean up the existing assertion failures first. # No assertions. Need to clean up the existing assertion failures first.
# Also, the Fuzzer does not provide reproducers on assertion failures yet. # Also, the Fuzzer does not provide reproducers on assertion failures yet.
CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_PARALLEL_LINK_JOBS=8 -DLIBFUZZER_ENABLE_TESTS=ON" CMAKE_COMMON_OPTIONS+=" -DLLVM_ENABLE_ASSERTIONS=OFF"
CLOBBER=fuzzer-test-suite clobber RUNDIR-*
STAGE1_CLOBBER="RUNDIR-* $LIBFUZZER"
clobber
# Make sure asan intercepts SIGABRT so that the fuzzer can print the test cases # Make sure asan intercepts SIGABRT so that the fuzzer can print the test cases
# for assertion failures. # for assertion failures.
# export ASAN_OPTIONS=handle_abort=1:strip_path_prefix=build/llvm/ # export ASAN_OPTIONS=handle_abort=1:strip_path_prefix=build/llvm/
buildbot_update buildbot_update
# Stage 1 # Stage 1
build_stage1_clang build_stage1_clang
# export ASAN_SYMBOLIZER_PATH="${llvm_symbolizer_path}" # export ASAN_SYMBOLIZER_PATH="${llvm_symbolizer_path}"
export PATH="$(readlink -f ${STAGE1_DIR}/bin):$PATH" export PATH="$(readlink -f ${STAGE1_DIR}/bin):$PATH"
echo @@@BUILD_STEP check-fuzzer@@@ echo @@@BUILD_STEP check-fuzzer@@@
(cd ${STAGE1_DIR} && ninja check-fuzzer) || echo @@@STEP_FAILURE@@@ (cd ${STAGE1_DIR} && ninja check-fuzzer) || build_failure
echo @@@BUILD_STEP get fuzzer-test-suite @@@ echo @@@BUILD_STEP get fuzzer-test-suite @@@
[ ! -e fuzzer-test-suite ] && git clone https://github.com/google/fuzzer-test-suite.git [ ! -e fuzzer-test-suite ] && git clone https://github.com/google/fuzzer-test-suite.git
(cd fuzzer-test-suite && git pull) (cd fuzzer-test-suite && git pull)
RunFuzzerTest() { RunFuzzerTest() {
echo @@@BUILD_STEP test "$1" fuzzer@@@ echo @@@BUILD_STEP test "$1" fuzzer@@@
ln -sf $LIBFUZZER . ln -sf $LIBFUZZER .
export FUZZING_ENGINE=fsanitize_fuzzer export FUZZING_ENGINE=fsanitize_fuzzer
`pwd`/fuzzer-test-suite/build-and-test.sh "$1" `pwd`/fuzzer-test-suite/build-and-test.sh "$1"
echo echo
} }
ulimit -t 3600 ulimit -t 3600
# FIXME: Some aarch64 tests are extremly slow, maybe related to
export JOBS=${MAKE_JOBS} export JOBS=${MAKE_JOBS}
RunFuzzerTest libxml2-v2.9.2 || echo @@@STEP_FAILURE@@@ RunFuzzerTest libxml2-v2.9.2 || build_failure
RunFuzzerTest libpng-1.2.56 || echo @@@STEP_FAILURE@@@ [[ "$(arch)" == "aarch64" ]] || RunFuzzerTest libpng-1.2.56 || build_failure
RunFuzzerTest libssh-2017-1272 || echo @@@STEP_FAILURE@@@ RunFuzzerTest libssh-2017-1272 || build_failure
RunFuzzerTest re2-2014-12-09 || echo @@@STEP_FAILURE@@@ RunFuzzerTest re2-2014-12-09 || build_failure
RunFuzzerTest c-ares-CVE-2016-5180 || echo @@@STEP_FAILURE@@@ RunFuzzerTest c-ares-CVE-2016-5180 || build_failure
RunFuzzerTest openssl-1.0.1f || echo @@@STEP_FAILURE@@@ RunFuzzerTest openssl-1.0.1f || build_failure
RunFuzzerTest openssl-1.0.2d || echo @@@STEP_FAILURE@@@ [[ "$(arch)" == "aarch64" ]] || RunFuzzerTest openssl-1.0.2d || build_failure
RunFuzzerTest proj4-2017-08-14 || echo @@@STEP_FAILURE@@@ [[ "$(arch)" == "aarch64" ]] || RunFuzzerTest proj4-2017-08-14 || build_failure
#RunFuzzerTest woff2-2016-05-06 || echo @@@STEP_WARNINGS@@@ # Often can't find the bug in the given time. #RunFuzzerTest woff2-2016-05-06 || build_failure # Often can't find the bug in the given time.
cleanup $STAGE1_DIR RUNDIR-*

zorg/buildbot/builders/sanitizers/buildbot_qemu.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/usr/bin/env bash
set -x
set -e
set -u
HERE="$(cd $(dirname $0) && pwd)"
. ${HERE}/buildbot_functions.sh
clobber
build_stage1_clang_at_revison
CMAKE_COMMON_OPTIONS+=" -DLLVM_ENABLE_ASSERTIONS=ON"
buildbot_update
readonly STAGE2_DIR=llvm_build2_host
(
echo @@@BUILD_STEP build host clang@@@
COMPILER_BIN_DIR="$(readlink -f ${STAGE1_DIR})/bin"
rm -rf ${STAGE2_DIR}
mkdir -p ${STAGE2_DIR}
cd ${STAGE2_DIR}
cmake \
${CMAKE_COMMON_OPTIONS} \
-DLLVM_ENABLE_PROJECTS="clang;compiler-rt;lld" \
-DCMAKE_C_COMPILER=${COMPILER_BIN_DIR}/clang \
-DCMAKE_CXX_COMPILER=${COMPILER_BIN_DIR}/clang++ \
-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
$LLVM && ninja
) || build_failure
set +x # Avoid echoing STEP_FAILURE because of the command trace.
MISSING_QEMU_IMAGE_MESSAGE=$(cat << 'EOF'
=====================================================================
Looks like you're missing the QEMU system images for x86_64 LAM
HWASan testing. These system images aren't automatically generated
as part of the buildbot script because they require root (sorry!).
If you have the system images already built, you can run
buildbot_qemu.sh with QEMU_IMAGE_DIR=path/to/qemu/images. Otherwise,
you can build the images (only necessary once-per-clobber) by:
1. WARNING: Read the script at https://github.com/google/sanitizers/blob/master/hwaddress-sanitizer/create_qemu_image.sh
You're running this as ROOT, so make sure you're comfortable with
that.
2. In your terminal, run:
$ sudo su -c "bash <(wget -qO- https://raw.githubusercontent.com/google/sanitizers/master/hwaddress-sanitizer/create_qemu_image.sh)" root && \
sudo chown $(whoami) debian.*
You can also choose to skip the x86_64 HWASan LAM testing by supplying
SKIP_HWASAN_LAM=true in your invocation of this script.
=====================================================================
EOF
)
set -x
SKIP_HWASAN_LAM=${SKIP_HWASAN_LAM:-}
function setup_lam_qemu_image {
# Full system emulation is required for x86_64 testing with LAM, as some
# sanitizers aren't friendly with usermode emulation under x86_64 LAM.
echo @@@BUILD_STEP Check x86_64 LAM Prerequisites@@@
# Allow specifying the QEMU image dir, otherwise assume in the local dir.
QEMU_IMAGE_DIR=${QEMU_IMAGE_DIR:=${ROOT}}
# Ensure the buildbot start script created a QEMU image for us.
{
QEMU_IMAGE="${QEMU_IMAGE_DIR}/debian.img"
QEMU_SSH_KEY="${QEMU_IMAGE_DIR}/debian.id_rsa"
ls ${QEMU_IMAGE}
ls ${QEMU_SSH_KEY}
} || {
# Make the "missing file" error clearer by not effectively echoing twice.
set +x
echo "$MISSING_QEMU_IMAGE_MESSAGE"
build_exception
exit 2
}
}
[[ -z "$SKIP_HWASAN_LAM" ]] && setup_lam_qemu_image || SKIP_HWASAN_LAM=1
function git_clone_at_revision {
local src_dir_name="${1}"
local git_url="${2}"
local revision="${3}"
local build_dir="${4}"
(
cd ${ROOT}
[[ -d ${src_dir_name} ]] || git clone ${git_url} ${src_dir_name} || exit 1
cd ${ROOT}/${src_dir_name}
git remote set-url origin ${git_url}
git fetch origin
# Short circuit if we already have a build at the correct revision.
[[ "$(git rev-parse HEAD)" == "$(git rev-parse ${revision})" ]] && exit 0
# Delete the build dir to ensure a rebuild happens at the new revision.
rm -rf ${build_dir}
git reset --hard ${revision}
git submodule update --init --recursive
)
}
function build_qemu {
local qemu_name="${1}"
local qemu_url="${2}"
local qemu_revision="${3}"
local build_dir="${ROOT}/${qemu_name}_build"
echo "@@@BUILD_STEP build ${qemu_name}@@@"
(
git_clone_at_revision ${qemu_name} ${qemu_url} ${qemu_revision} \
${build_dir} || exit 1
${build_dir}/qemu-x86_64 --version &&
${build_dir}/qemu-system-x86_64 --version &&
${build_dir}/qemu-img --version &&
exit 0
rm -rf ${build_dir} &&
mkdir -p ${build_dir} &&
cd ${build_dir} &&
${ROOT}/${qemu_name}/configure --enable-linux-user &&
ninja &&
${build_dir}/qemu-x86_64 --version &&
${build_dir}/qemu-system-x86_64 --version &&
${build_dir}/qemu-img --version
) || (
build_exception
exit 2
)
}
function build_lam_linux {
echo "@@@BUILD_STEP build lam linux@@@"
local build_dir="${ROOT}/lam_linux_build"
LAM_KERNEL="${build_dir}/arch/x86_64/boot/bzImage"
(
git_clone_at_revision lam_linux https://github.com/vitalybuka/linux.git \
origin/lam ${build_dir} || exit 1
ls "${LAM_KERNEL}" && exit 0
rm -rf ${build_dir} &&
mkdir -p ${build_dir} &&
cd ${ROOT}/lam_linux &&
make O=${build_dir} LD=ld.bfd defconfig &&
make O=${build_dir} LD=ld.bfd -j $(nproc) &&
ls "${LAM_KERNEL}"
) || (
build_exception
exit 2
)
}
build_qemu qemu https://github.com/vitalybuka/qemu.git origin/sanitizer_bot
[[ -z "$SKIP_HWASAN_LAM" ]] && build_lam_linux
SCUDO_BUILDS=
function configure_scudo_compiler_rt {
local arch=$1
local target="${arch}-linux-gnu${2:-}"
local name=""
if [[ "$DBG" == "ON" ]] ; then
name=debug_
fi
name+="${arch}"
local qemu_cmd=""
if [[ "${QEMU:-}" != "0" ]] ; then
name+="_qemu"
local qemu_arch=${arch}
[[ "${arch}" == "powerpc64le" ]] && qemu_arch="ppc64le"
qemu_cmd="$ROOT/qemu_build/qemu-${qemu_arch} -L /usr/${target}"
if [[ ! -z "${QEMU_CPU:-}" ]] ; then
qemu_cmd+=" -cpu ${QEMU_CPU}"
name+="_${QEMU_CPU}"
fi
fi
local linker_flags=
[[ "${arch}" =~ "mips*" ]] && linker_flags="-latomic -Wl,-z,notext -Wno-unused-command-line-argument"
local out_dir=llvm_build2_${name}
rm -rf ${out_dir}
mkdir -p ${out_dir}
SCUDO_BUILDS+=" ${name}"
(
# We need a fresh base compiler to test standalone build, as this config
# does not build own clang.
local COMPILER_BIN_DIR="$(readlink -f ${STAGE2_DIR})/bin"
cd ${out_dir}
(
cmake \
${CMAKE_COMMON_OPTIONS} \
-DCOMPILER_RT_DEBUG=$DBG \
-DLLVM_CMAKE_DIR="${COMPILER_BIN_DIR}/.." \
-DCMAKE_C_COMPILER=${COMPILER_BIN_DIR}/clang \
-DCMAKE_CXX_COMPILER=${COMPILER_BIN_DIR}/clang++ \
-DCOMPILER_RT_HAS_LLD=ON \
-DCOMPILER_RT_TEST_USE_LLD=ON \
-DCMAKE_INSTALL_PREFIX=$(${COMPILER_BIN_DIR}/clang -print-resource-dir) \
-DCOMPILER_RT_BUILD_BUILTINS=OFF \
-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
-DCMAKE_CROSSCOMPILING=True \
-DCOMPILER_RT_INCLUDE_TESTS=ON \
-DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
-DCMAKE_CXX_FLAGS=-fPIC \
-DCMAKE_C_FLAGS=-fPIC \
-DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=lld ${linker_flags}" \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld ${linker_flags}" \
-DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${target} ${linker_flags}" \
-DCMAKE_C_COMPILER_TARGET=${target} \
-DCMAKE_CXX_COMPILER_TARGET=${target} \
-DCOMPILER_RT_EMULATOR="${qemu_cmd:-}" \
$LLVM/../compiler-rt
) >& configure.log
) &
}
function configure_hwasan_lam {
local out_dir=llvm_build2_x86_64_lam_qemu
rm -rf ${out_dir}
mkdir -p ${out_dir}
(
# We don't need fresh base compiler, as this config will build own clang.
local COMPILER_BIN_DIR="$(readlink -f ${STAGE1_DIR})/bin"
cd ${out_dir}
(
# STAGE1_DIR is build once, so we can use CCACHE.
cmake \
${CMAKE_COMMON_OPTIONS} \
-DLLVM_ENABLE_PROJECTS="clang;compiler-rt;lld" \
-DCMAKE_C_COMPILER=${COMPILER_BIN_DIR}/clang \
-DCMAKE_CXX_COMPILER=${COMPILER_BIN_DIR}/clang++ \
-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
-DCOMPILER_RT_EMULATOR="env SSH_CONTROL_SOCKET=${SSH_CONTROL_SOCKET} ${HERE}/ssh_run.sh" \
$LLVM
) >& configure.log
) &
}
function run_scudo_tests {
local name="${1}"
local out_dir=llvm_build2_${name}
echo "@@@BUILD_STEP scudo $name@@@"
(
cd ${out_dir}
cat configure.log
# Copy into clang resource dir to make -fsanitize= work in lit tests.
ninja install-scudo_standalone
ninja check-scudo_standalone || exit 3
) || build_failure
}
QEMU_PID=""
readonly QEMU_TMPDIR="$ROOT/qemu_tmp"
readonly SSH_CONTROL_SOCKET="${QEMU_TMPDIR}/ssh-control-socket"
function kill_qemu {
if kill "${QEMU_PID}"; then
echo "Waiting for QEMU to shutdown..." >&2
{
timeout -k 5s 5s wait "${QEMU_PID}" && return
kill -9 "${QEMU_PID}" && wait "${QEMU_PID}" || true
} &>/dev/null
fi
}
function boot_qemu {
trap kill_qemu EXIT
# Path to a qemu-system-x86_64 binary built with LAM support.
local QEMU="${ROOT}/qemu_build/qemu-system-x86_64"
# Path to a qemu-img binary.
local QEMU_IMG="${ROOT}/qemu_build/qemu-img"
echo "Booting QEMU..." >&2
# Try up to 10 random port numbers until one succeeds.
for i in {0..10}; do
rm -rf ${QEMU_TMPDIR}
mkdir -p ${QEMU_TMPDIR}
# Create a delta image to boot from.
local DELTA_IMAGE="${QEMU_TMPDIR}/delta.qcow2"
"${QEMU_IMG}" create -F raw -b "${QEMU_IMAGE}" -f qcow2 "${DELTA_IMAGE}"
local SSH_PORT="$(shuf -i 1000-65535 -n 1)"
"${QEMU}" -hda "${DELTA_IMAGE}" -nographic \
-net "user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:${SSH_PORT}-:22" \
-net "nic,model=e1000" -machine "type=q35,accel=tcg" \
-smp $(($(nproc) / 2)) -cpu "qemu64,+la57,+lam" -kernel "${LAM_KERNEL}" \
-append "root=/dev/sda net.ifnames=0 console=ttyS0" -m "16G" &
QEMU_PID=$!
# If QEMU is running, the port number worked.
sleep 1
ps -p "${QEMU_PID}" &>/dev/null || continue
echo "Waiting for QEMU ssh daemon..." >&2
for i in {0..10}; do
echo "SSH into VM, try ${i}" >&2
sleep 15
# Set up persistent SSH connection for faster command execution inside QEMU.
ssh -p "${SSH_PORT}" -o "StrictHostKeyChecking=no" \
-o "UserKnownHostsFile=/dev/null" -o "ControlPersist=30m" \
-M -S "${SSH_CONTROL_SOCKET}" -i "${QEMU_SSH_KEY}" root@localhost "uname -a" 1>&2 &&
return
done
kill_qemu
done
# Fail fast if QEMU is not running.
ps -p "${QEMU_PID}" &>/dev/null
}
function run_hwasan_lam_tests {
local name="x86_64_lam_qemu"
local out_dir=llvm_build2_${name}
echo "@@@BUILD_STEP configure hwasan ${name}@@@"
(
cd ${out_dir}
cat configure.log
# Build most stuff before starting VM.
echo "@@@BUILD_STEP build tools ${name}@@@"
ninja clang lld llvm-symbolizer || exit 3
echo "@@@BUILD_STEP start LAM QEMU@@@"
boot_qemu || build_exception
ssh -S "${SSH_CONTROL_SOCKET}" root@localhost \
"mkdir -p /b && mount -t nfs 10.0.2.10:/b /b"
echo
echo "@@@BUILD_STEP test hwasan ${name}@@@"
# Fixme: figure out why is the test failing with detect_leaks tunred on.
export LIT_FILTER_OUT="print-memory-usage.c"
ninja check-hwasan-lam || exit 3
) || build_failure
}
echo "@@@BUILD_STEP configure@@@"
for DBG in OFF ON ; do
QEMU=0 configure_scudo_compiler_rt x86_64
configure_scudo_compiler_rt x86_64
# FIXME: investigate and reenable configure_scudo_compiler_rt arm eabihf
configure_scudo_compiler_rt aarch64
QEMU_CPU="cortex-a72" configure_scudo_compiler_rt aarch64
#configure_scudo_compiler_rt mips
#configure_scudo_compiler_rt mipsel
configure_scudo_compiler_rt mips64 abi64
configure_scudo_compiler_rt mips64el abi64
configure_scudo_compiler_rt powerpc64le
done
[[ -z "$SKIP_HWASAN_LAM" ]] && configure_hwasan_lam
wait
for B in $SCUDO_BUILDS ; do
run_scudo_tests $B
done
[[ -z "$SKIP_HWASAN_LAM" ]] && (
run_hwasan_lam_tests
)
cleanup $STAGE2_DIR

zorg/buildbot/builders/sanitizers/buildbot_selector.py

#!/usr/bin/python #!/usr/bin/python
import os import os
import subprocess import subprocess
import sys import sys
THIS_DIR=os.path.dirname(sys.argv[0]) THIS_DIR=os.path.dirname(sys.argv[0])
extra_args = sys.argv[1:]
BOT_DIR = '/b'
def in_script_dir(path):
def bash(path): return os.path.join(THIS_DIR, path)
return 'bash ' + os.path.join(THIS_DIR, path)
def cmd_call(path):
return 'call ' + os.path.join(THIS_DIR, path)
BOT_ASSIGNMENT = { BOT_ASSIGNMENT = {
'sanitizer-ppc64le-linux': bash('buildbot_cmake.sh'), 'sanitizer-ppc64le-linux': 'buildbot_cmake.sh',
'sanitizer-ppc64be-linux': bash('buildbot_cmake.sh'), 'sanitizer-ppc64be-linux': 'buildbot_cmake.sh',
'sanitizer-x86_64-linux': bash('buildbot_cmake.sh'), 'sanitizer-x86_64-linux': 'buildbot_cmake.sh',
'sanitizer-x86_64-linux-fast': bash('buildbot_fast.sh'), 'sanitizer-x86_64-linux-fast': 'buildbot_fast.sh',
'sanitizer-x86_64-linux-autoconf': bash('buildbot_standard.sh'), 'sanitizer-x86_64-linux-autoconf': 'buildbot_standard.sh',
'sanitizer-x86_64-linux-fuzzer': bash('buildbot_fuzzer.sh'), 'sanitizer-x86_64-linux-fuzzer': 'buildbot_fuzzer.sh',
'sanitizer-x86_64-linux-android': bash('buildbot_android.sh'), 'sanitizer-x86_64-linux-android': 'buildbot_android.sh',
'sanitizer-x86_64-linux-bootstrap': bash('buildbot_bootstrap_asan.sh'), 'sanitizer-x86_64-linux-bootstrap-asan': 'buildbot_bootstrap_asan.sh',
'sanitizer-x86_64-linux-bootstrap-msan': bash('buildbot_bootstrap_msan.sh'), 'sanitizer-x86_64-linux-bootstrap-msan': 'buildbot_bootstrap_msan.sh',
'sanitizer-x86_64-linux-bootstrap-ubsan': bash('buildbot_bootstrap_ubsan.sh'), 'sanitizer-x86_64-linux-bootstrap-ubsan': 'buildbot_bootstrap_ubsan.sh',
'sanitizer-x86_64-linux-gn': 'true', 'sanitizer-x86_64-linux-qemu': 'buildbot_qemu.sh',
'sanitizer-aarch64-linux-fuzzer': 'buildbot_fuzzer.sh',
'sanitizer-aarch64-linux-bootstrap-asan': 'buildbot_bootstrap_asan.sh',
'sanitizer-aarch64-linux-bootstrap-hwasan': 'buildbot_bootstrap_hwasan.sh',
'sanitizer-aarch64-linux-bootstrap-msan': 'buildbot_bootstrap_msan.sh',
'sanitizer-aarch64-linux-bootstrap-ubsan': 'buildbot_bootstrap_ubsan.sh',
} }
BOT_ADDITIONAL_ENV = { BOT_ADDITIONAL_ENV = {
'sanitizer-ppc64le-linux': { 'HAVE_NINJA': '1', 'CHECK_LIBCXX': '0', 'CHECK_LLD': '0' }, 'sanitizer-ppc64le-linux': { 'CHECK_LIBCXX': '0' },
'sanitizer-ppc64be-linux': { 'HAVE_NINJA': '1', 'CHECK_LIBCXX': '0', 'CHECK_LLD': '0' }, 'sanitizer-ppc64be-linux': { 'CHECK_LIBCXX': '0' },
'sanitizer-x86_64-linux': { 'HAVE_NINJA' : '1' }, 'sanitizer-x86_64-linux': {},
'sanitizer-x86_64-linux-bootstrap': {},
'sanitizer-x86_64-linux-fast': {}, 'sanitizer-x86_64-linux-fast': {},
'sanitizer-x86_64-linux-autoconf': {}, 'sanitizer-x86_64-linux-autoconf': {},
'sanitizer-x86_64-linux-fuzzer': {}, 'sanitizer-x86_64-linux-fuzzer': {},
'sanitizer-x86_64-linux-android': {}, 'sanitizer-x86_64-linux-android': {},
'sanitizer-x86_64-linux-bootstrap-asan': {},
'sanitizer-x86_64-linux-bootstrap-msan': {}, 'sanitizer-x86_64-linux-bootstrap-msan': {},
'sanitizer-x86_64-linux-bootstrap-ubsan': {}, 'sanitizer-x86_64-linux-bootstrap-ubsan': {},
'sanitizer-x86_64-linux-gn': {}, 'sanitizer-x86_64-linux-qemu': { 'QEMU_IMAGE_DIR': BOT_DIR + '/qemu_image' },
'sanitizer-aarch64-linux-fuzzer': {},
'sanitizer-aarch64-linux-bootstrap-asan': {},
'sanitizer-aarch64-linux-bootstrap-hwasan': {},
'sanitizer-aarch64-linux-bootstrap-msan': {},
'sanitizer-aarch64-linux-bootstrap-ubsan': {},
} }
def Main(): def Main():
builder = os.environ.get('BUILDBOT_BUILDERNAME') builder = os.environ.get('BUILDBOT_BUILDERNAME')
print "builder name: %s" % (builder) revision = os.environ.get('BUILDBOT_REVISION')
cmd = BOT_ASSIGNMENT.get(builder) print("builder name: %s" % (builder))
cmd = [in_script_dir(BOT_ASSIGNMENT.get(builder))] + extra_args
if not cmd: if not cmd:
sys.stderr.write('ERROR - unset/invalid builder name\n') sys.stderr.write('ERROR - unset/invalid builder name\n')
sys.exit(1) sys.exit(1)
print "%s runs: %s\n" % (builder, cmd) print("%s runs: %s\n" % (builder, ' '.join(cmd)))
sys.stdout.flush() sys.stdout.flush()
bot_env = os.environ bot_env = os.environ
bot_env['BOT_DIR'] = BOT_DIR
add_env = BOT_ADDITIONAL_ENV.get(builder) add_env = BOT_ADDITIONAL_ENV.get(builder)
for var in add_env: for var in add_env:
bot_env[var] = add_env[var] bot_env[var] = add_env[var]
if 'TMPDIR' in bot_env: if 'TMPDIR' in bot_env:
del bot_env['TMPDIR'] del bot_env['TMPDIR']
retcode = subprocess.call(cmd, env=bot_env, shell=True) if ':' in revision:
sys.exit(retcode) cmd = [in_script_dir('buildbot_bisect_run.sh')] + cmd
sys.exit(subprocess.call(cmd, env=bot_env))
if __name__ == '__main__': if __name__ == '__main__':
Main() Main()

zorg/buildbot/builders/sanitizers/buildbot_standard.sh

#!/usr/bin/env bash #!/usr/bin/env bash
set -x set -x
set -e set -e
set -u set -u
HERE="$(dirname $0)" HERE="$(dirname $0)"
. ${HERE}/buildbot_functions.sh . ${HERE}/buildbot_functions.sh
TSAN_DEBUG_BUILD_DIR=tsan_debug_build TSAN_DEBUG_BUILD_DIR=tsan_debug_build
TSAN_FULL_DEBUG_BUILD_DIR=tsan_full_debug_build TSAN_FULL_DEBUG_BUILD_DIR=tsan_full_debug_build
TSAN_RELEASE_BUILD_DIR=tsan_release_build TSAN_RELEASE_BUILD_DIR=tsan_release_build
rm -rf $TSAN_DEBUG_BUILD_DIR $TSAN_FULL_DEBUG_BUILD_DIR $TSAN_RELEASE_BUILD_DIR CLEANUP="$TSAN_DEBUG_BUILD_DIR $TSAN_FULL_DEBUG_BUILD_DIR $TSAN_RELEASE_BUILD_DIR"
clobber clobber $CLEANUP
ROOT=`pwd` CMAKE_COMMON_OPTIONS+=" -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON"
PLATFORM=`uname`
MAKE_JOBS=${MAX_MAKE_JOBS:-$(nproc)}
LLVM=${ROOT}/llvm
CMAKE_COMMON_OPTIONS="-GNinja -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON -DLLVM_USE_LINKER=gold"
function build_tsan { function build_tsan {
local build_dir=$1 local build_dir=$1
local extra_cmake_args="$2 -DLLVM_ENABLE_PROJECTS='clang;compiler-rt'" local extra_cmake_args="$2 -DLLVM_ENABLE_PROJECTS='clang;compiler-rt'"
local targets="clang llvm-symbolizer llvm-config FileCheck not" local targets="clang llvm-symbolizer llvm-config FileCheck not"
if [ ! -d $build_dir ]; then if [ ! -d $build_dir ]; then
mkdir $build_dir mkdir $build_dir
fi fi
(cd $build_dir && CC="$3" CXX="$4" cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \ (cd $build_dir && CC="$3" CXX="$4" cmake \
${CMAKE_COMMON_OPTIONS} ${extra_cmake_args} \ ${CMAKE_COMMON_OPTIONS} ${extra_cmake_args} \
${LLVM}) ${LLVM})
(cd $build_dir && ninja ${targets}) || echo @@@STEP_FAILURE@@@ (cd $build_dir && ninja ${targets}) || build_failure
(cd $build_dir && ninja compiler-rt-clear) || echo @@@STEP_FAILURE@@@ (cd $build_dir && ninja tsan) || build_failure
(cd $build_dir && ninja tsan) || echo @@@STEP_FAILURE@@@
} }
buildbot_update buildbot_update
USE_CCACHE=
if ccache -s ; then
USE_CCACHE="-DLLVM_CCACHE_BUILD=ON"
fi
echo @@@BUILD_STEP build fresh clang + debug compiler-rt@@@ echo @@@BUILD_STEP build fresh clang + debug compiler-rt@@@
build_tsan "${TSAN_DEBUG_BUILD_DIR}" "-DCOMPILER_RT_DEBUG=ON $USE_CCACHE" gcc g++ build_tsan "${TSAN_DEBUG_BUILD_DIR}" "-DCOMPILER_RT_DEBUG=ON" gcc g++
echo @@@BUILD_STEP test tsan in debug compiler-rt build@@@ echo @@@BUILD_STEP test tsan in debug compiler-rt build@@@
(cd $TSAN_DEBUG_BUILD_DIR && ninja check-tsan) || echo @@@STEP_FAILURE@@@ (cd $TSAN_DEBUG_BUILD_DIR && ninja check-tsan) || build_failure
echo @@@BUILD_STEP build tsan with stats and debug output@@@ echo @@@BUILD_STEP build tsan with stats and debug output@@@
build_tsan "${TSAN_FULL_DEBUG_BUILD_DIR}" "-DCOMPILER_RT_DEBUG=ON -DCOMPILER_RT_TSAN_DEBUG_OUTPUT=ON -DLLVM_INCLUDE_TESTS=OFF $USE_CCACHE" gcc g++ build_tsan "${TSAN_FULL_DEBUG_BUILD_DIR}" "-DCOMPILER_RT_DEBUG=ON -DCOMPILER_RT_TSAN_DEBUG_OUTPUT=ON -DLLVM_INCLUDE_TESTS=OFF" gcc g++
echo @@@BUILD_STEP build release tsan with clang@@@ echo @@@BUILD_STEP build release tsan with clang@@@
build_tsan "${TSAN_RELEASE_BUILD_DIR}" "-DCOMPILER_RT_DEBUG=OFF" "$ROOT/$TSAN_DEBUG_BUILD_DIR/bin/clang" "$ROOT/$TSAN_DEBUG_BUILD_DIR/bin/clang++" build_tsan "${TSAN_RELEASE_BUILD_DIR}" "-DCOMPILER_RT_DEBUG=OFF" "$ROOT/$TSAN_DEBUG_BUILD_DIR/bin/clang" "$ROOT/$TSAN_DEBUG_BUILD_DIR/bin/clang++"
echo @@@BUILD_STEP tsan analyze@@@ echo @@@BUILD_STEP tsan analyze@@@
BIN=$(mktemp -t tsan_exe.XXXXXXXX) BIN=$(mktemp -t tsan_exe.XXXXXXXX)
echo "int main() {return 0;}" | $TSAN_RELEASE_BUILD_DIR/bin/clang -x c++ - -fsanitize=thread -O2 -o ${BIN} echo "int main() {return 0;}" | $TSAN_RELEASE_BUILD_DIR/bin/clang -x c++ - -fsanitize=thread -O2 -o ${BIN}
COMPILER_RT=$LLVM/../compiler-rt COMPILER_RT=$LLVM/../compiler-rt
$COMPILER_RT/lib/tsan/check_analyze.sh ${BIN} || echo @@@STEP_FAILURE@@@ $COMPILER_RT/lib/tsan/check_analyze.sh ${BIN} || build_failure
cleanup $CLEANUP

zorg/buildbot/builders/sanitizers/ssh_run.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/bin/bash -eu
#
# Runs the given command over ssh using SSH_CONTROL_SOCKET.
#
function run_in_qemu {
ssh -S "${SSH_CONTROL_SOCKET}" root@localhost "${@}"
}
# Run binary in QEMU.
ENV="HWASAN_OPTIONS=\"${HWASAN_OPTIONS:-}\" PATH=\"${PATH:-}\""
run_in_qemu "cd $(pwd) && ${ENV} $@"

zorg/buildbot/changes/llvmgitpoller.py

# LLVM buildbot needs to watch multiple projects within a single repository. # LLVM buildbot needs to watch multiple projects within a single repository.
# TODO: Handle both author and committer for a commit to build a correct blame list later. # TODO: Handle both author and committer for a commit to build a correct blame list later.
import re import re
from twisted.python import log from twisted.python import log
from datetime import datetime from datetime import datetime
from twisted.internet import defer from twisted.internet import defer
from buildbot.util import bytes2unicode from buildbot.util import bytes2unicode
from buildbot.plugins import changes from buildbot.plugins import changes
class LLVMPoller(changes.GitPoller): class LLVMPoller(changes.GitPoller):
""" """
Poll LLVM repository for changes and submit them to the change master. Poll LLVM repository for changes and submit them for builds scheduling.
Following Multiple LLVM Projects. Following Multiple LLVM Projects.
This source will poll a remote LLVM git _monorepo_ for changes and submit This source will poll a remote LLVM git _monorepo_ for changes and submit
them to the change master.""" them for builds scheduling."""
_repourl = "https://github.com/llvm/llvm-project" _repourl = "https://github.com/llvm/llvm-project"
_branch = "master"
compare_attrs = ["repourl", "branch", "workdir", compare_attrs = ["repourl", "branches", "workdir",
"pollInterval", "gitbin", "usetimestamps", "pollInterval", "gitbin", "usetimestamps",
"category", "project", "category", "project",
"projects"] "projects"]
def _check_branches(branch):
if branch == "refs/heads/main":
# Always listen for changes in the main branch.
return True
else:
# We are also interested in the release branches.
# Some builders will be building changes from there as well.
return re.search(r"refs\/heads\/release\/\d\d+.*", branch)
def __init__(self, def __init__(self,
repourl=_repourl, branch=_branch, repourl=_repourl, branches=_check_branches,
**kwargs): **kwargs):
self.cleanRe = re.compile(r"Require(?:s?)\s*.*\s*clean build", re.IGNORECASE + re.MULTILINE) self.cleanRe = re.compile(r"Require(?:s?)\s*.*\s*clean build", re.IGNORECASE + re.MULTILINE)
self.cleanCfg = re.compile(r"(CMakeLists\.txt$|\.cmake$|\.cmake\.in$)") self.cleanCfg = re.compile(r"(CMakeLists\.txt$|\.cmake$|\.cmake\.in$)")
# TODO: Add support for an optional list of projects. # Note: We always watch all the projects, then schedulers decide
# For now we always watch all the projects. # to build or not to build.
super().__init__(repourl=repourl, branch=branch, **kwargs) super().__init__(repourl=repourl, branches=branches, **kwargs)
def _transform_path(self, fileList): def _transform_path(self, fileList):
""" """
Parses the given list of files, and returns a list of two-entry tuples Parses the given list of files, and returns a list of two-entry tuples
(PROJECT, [FILES]) if PROJECT is watched one, (PROJECT, [FILES]) if PROJECT is watched one,
or None otherwise. or None otherwise.
NOTE: we don't change result path, just extract a project name. NOTE: we don't change result path, just extract a project name.
▲ Show 20 Lines • Show All 59 Lines • ▼ Show 20 Lines def _process_changes(self, newRev, branch):
if existingRev != newRev: if existingRev != newRev:
revList = [newRev] revList = [newRev]
if existingRev is None: if existingRev is None:
# This branch was completely unknown, rebuild # This branch was completely unknown, rebuild
log.msg('LLVMPoller: rebuilding {} for new branch "{}"'.format( log.msg('LLVMPoller: rebuilding {} for new branch "{}"'.format(
newRev, branch)) newRev, branch))
else: else:
# This branch is known, but it now points to a different # This branch is known, but it now points to a different
# commit than last time we saw it, rebuild. # commit than the last time we saw it, rebuild.
log.msg('LLVMPoller: rebuilding {} for updated branch "{}"'.format( log.msg('LLVMPoller: rebuilding {} for updated branch "{}"'.format(
newRev, branch)) newRev, branch))
self.changeCount = len(revList) self.changeCount = len(revList)
self.lastRev[branch] = newRev self.lastRev[branch] = newRev
if self.changeCount: if self.changeCount:
log.msg('LLVMPoller: processing {} changes: {} from "{}" branch "{}"'.format( log.msg('LLVMPoller: processing {} changes: {} from "{}" branch "{}"'.format(
Show All 35 Lines def _process_changes(self, newRev, branch):
projects += [where_project] projects += [where_project]
if self.cleanRe.search(comments) or \ if self.cleanRe.search(comments) or \
any([m for f in where_project_files for m in [self.cleanCfg.search(f)] if m]): any([m for f in where_project_files for m in [self.cleanCfg.search(f)] if m]):
log.msg("LLVMPoller: creating a change with the 'clean_obj' property for r%s" % rev) log.msg("LLVMPoller: creating a change with the 'clean_obj' property for r%s" % rev)
properties['clean_obj'] = (True, "change") properties['clean_obj'] = (True, "change")
log.msg("LLVMPoller: creating a change rev=%s" % rev) log.msg("LLVMPoller: creating a change rev=%s" % rev)
log.msg(" >>> revision=%s, timestamp=%s, author=%s, committer=%s, project=%s, files=%s, comments=\"%s\", properties=%s" % \ log.msg(" >>> branch=%s, revision=%s, timestamp=%s, author=%s, committer=%s, project=%s, files=%s, comments=\"%s\", properties=%s" % \
(bytes2unicode(rev, encoding=self.encoding), datetime.fromtimestamp(timestamp), author, committer, (bytes2unicode(self._removeHeads(branch)),
bytes2unicode(rev, encoding=self.encoding), datetime.fromtimestamp(timestamp), author, committer,
projects, files, comments, properties)) projects, files, comments, properties))
yield self.master.data.updates.addChange( yield self.master.data.updates.addChange(
author=author, author=author,
committer=committer if committer != author else None, committer=committer if committer != author else None,
revision=bytes2unicode(rev, encoding=self.encoding), revision=bytes2unicode(rev, encoding=self.encoding),
files=files, files=files,
comments=comments, comments=comments,
when_timestamp=timestamp, when_timestamp=timestamp,
branch=bytes2unicode(self._removeHeads(branch)), branch=bytes2unicode(self._removeHeads(branch)),
category=self.category, ## TODO: Figure out if we could support tags here category=self.category, ## TODO: Figure out if we could support tags here
project=",".join(projects) if projects else "llvm-project", project=",".join(projects) if projects else "llvm-project",
# Always promote an external github url of the LLVM project with the changes. # Always promote an external github url of the LLVM project with the changes.
repository=self._repourl, repository=self._repourl,
src='git', # Must be one of the buildbot.process.users.srcs src='git', # Must be one of the buildbot.process.users.srcs
properties=properties) properties=properties)

zorg/buildbot/commands/AnnotatedCommand.py

Show First 20 Lines • Show All 91 Lines • ▼ Show 20 Lines def __init__(self, parent_step, *args, **kwargs):
self._loglines = [] self._loglines = []
self._logLock = defer.DeferredLock() self._logLock = defer.DeferredLock()
self._request_finish = False self._request_finish = False
@defer.inlineCallbacks @defer.inlineCallbacks
def addStep(self): def addStep(self):
debuglog("AnnotatedBuildStep: adding step '%s': buildid=%s" % ( debuglog("AnnotatedBuildStep: adding step '{}': buildid={}".format(
self.name, self.build.buildid)) self.name, self.build.buildid))
yield buildstep.BuildStep.addStep(self) yield buildstep.BuildStep.addStep(self)
# Put ourselves into a list of processed steps of the build object. # Put ourselves into a list of processed steps of the build object.
self.build.executedSteps.append(self) self.build.executedSteps.append(self)
# Init result to SUCCESS by default. # Init result to SUCCESS by default.
self.results = results.SUCCESS self.results = results.SUCCESS
self.description = [self.name] self.description = [self.name]
self.stdio_log = yield self.addLog('stdio') self.stdio_log = yield self.addLog('stdio')
debuglog("AnnotatedBuildStep: added step '%s': stepid=%s, buildid=%s" % ( debuglog("AnnotatedBuildStep: added step '{}': stepid={}, buildid={}".format(
self.name, self.stepid, self.build.buildid)) self.name, self.stepid, self.build.buildid))
@defer.inlineCallbacks @defer.inlineCallbacks
def finishStep(self): def finishStep(self):
hidden = False hidden = False
debuglog("AnnotatedBuildStep: finish step '%s': stepid=%s, buildid=%s, results=%s" % ( debuglog("AnnotatedBuildStep: finish step '{}': stepid={}, buildid={}, results={}".format(
self.name, self.stepid, self.build.buildid, self.results)) self.name, self.stepid, self.build.buildid, self.results))
yield self.master.data.updates.finishStep(self.stepid, self.results, yield self.master.data.updates.finishStep(self.stepid, self.results,
hidden) hidden)
# finish unfinished logs # finish unfinished logs
yield self.finishUnfinishedLogs() yield self.finishUnfinishedLogs()
@defer.inlineCallbacks @defer.inlineCallbacks
def startStep(self, remote, done=False): def startStep(self, remote, done=False):
debuglog("AnnotatedBuildStep::startStep() starting '%s': " debuglog("AnnotatedBuildStep::startStep() starting '{}': "
"buildid=%s, done=%s" % ( "buildid={}, done={}".format(
self.name, self.build.buildid, done)) self.name, self.build.buildid, done))
try: try:
yield self.addStep() yield self.addStep()
try: try:
self.realUpdateSummary() self.realUpdateSummary()
# The "main" step has finished already. # The "main" step has finished already.
Show All 24 Lines def startStep(self, remote, done=False):
# update the summary one last time, make sure that completes, # update the summary one last time, make sure that completes,
# and then don't update it any more. # and then don't update it any more.
self.realUpdateSummary() self.realUpdateSummary()
yield self.realUpdateSummary.stop() yield self.realUpdateSummary.stop()
# Update step status in the database. # Update step status in the database.
yield self.finishStep() yield self.finishStep()
debuglog("AnnotatedBuildStep::startStep() completed '%s': " debuglog("AnnotatedBuildStep::startStep() completed '{}': "
"stepid=%s, buildid=%s, results=%s" % ( "stepid={}, buildid={}, results={}".format(
self.name, self.stepid, self.build.buildid, self.results)) self.name, self.stepid, self.build.buildid, self.results))
def requestFinish(self, status=None): def requestFinish(self, status=None):
debuglog("AnnotatedBuildStep::requestFinish(%r): '%s': " debuglog("AnnotatedBuildStep::requestFinish({}): '{}': "
"stepid=%s, buildid=%s, results=%r" % ( "stepid={}, buildid={}, results={}".format(
status, self.name, self.stepid, self.build.buildid, self.results)) status, self.name, self.stepid, self.build.buildid, self.results))
# Update the current step status with the worst result. # Update the current step status with the worst result.
if status is not None: if status is not None:
self.updateStatus(status) self.updateStatus(status)
self._request_finish = True self._request_finish = True
def setStepText(self, text=None): def setStepText(self, text=None):
if text is not None: if text is not None:
Show All 24 Linesclass AnnotatedBuildStep(buildstep.BuildStep):
@deferredLocked("_logLock") @deferredLocked("_logLock")
def _addLogs(self, text): def _addLogs(self, text):
if text is not None: if text is not None:
self._loglines.append(text) self._loglines.append(text)
return defer.succeed(None) return defer.succeed(None)
@defer.inlineCallbacks @defer.inlineCallbacks
def run(self): def run(self):
debuglog("AnnotatedBuildStep::run() starting '%s' step: " debuglog("AnnotatedBuildStep::run() starting '{}' step: "
"stepid=%s, buildid=%s" % ( "stepid={}, buildid={}".format(
self.name, self.stepid, self.build.buildid)) self.name, self.stepid, self.build.buildid))
# Save previously collected log lines. # Save previously collected log lines.
yield self._flushLogs() yield self._flushLogs()
while not self._request_finish: while not self._request_finish:
# Sleep for .1 second. Let arrive more logs. # Sleep for .1 second. Let arrive more logs.
yield asyncSleep(.1) yield asyncSleep(.1)
if self._loglines: if self._loglines:
yield self._flushLogs() yield self._flushLogs()
# and the last one time. # and the last one time.
if self._loglines: if self._loglines:
yield self._flushLogs() yield self._flushLogs()
debuglog("AnnotatedBuildStep::run() exiting '%s' step: " debuglog("AnnotatedBuildStep::run() exiting '{}' step: "
"stepid=%s, buildid=%s, results=%s" % ( "stepid={}, buildid={}, results={}".format(
self.name, self.stepid, self.build.buildid, self.results)) self.name, self.stepid, self.build.buildid, self.results))
return self.results return self.results
@defer.inlineCallbacks @defer.inlineCallbacks
def scheduleStdout(self, text): def scheduleStdout(self, text):
if text is None: if text is None:
return return
▲ Show 20 Lines • Show All 74 Lines • ▼ Show 20 Lines def __init__(self, **kwargs):
'BUILDBOT_BLAMELIST': util.Interpolate('%(prop:blamelist:-[])s'), 'BUILDBOT_BLAMELIST': util.Interpolate('%(prop:blamelist:-[])s'),
'BUILDBOT_BRANCH': util.Interpolate('%(prop:branch:-None)s'), 'BUILDBOT_BRANCH': util.Interpolate('%(prop:branch:-None)s'),
'BUILDBOT_BUILDERNAME': util.Interpolate('%(prop:buildername:-None)s'), 'BUILDBOT_BUILDERNAME': util.Interpolate('%(prop:buildername:-None)s'),
'BUILDBOT_BUILDNUMBER': util.Interpolate('%(prop:buildnumber:-None)s'), 'BUILDBOT_BUILDNUMBER': util.Interpolate('%(prop:buildnumber:-None)s'),
'BUILDBOT_CLOBBER': util.Interpolate('%(prop:clobber:+1)s'), 'BUILDBOT_CLOBBER': util.Interpolate('%(prop:clobber:+1)s'),
'BUILDBOT_GOT_REVISION': util.Interpolate('%(prop:got_revision:-None)s'), 'BUILDBOT_GOT_REVISION': util.Interpolate('%(prop:got_revision:-None)s'),
'BUILDBOT_REVISION': util.Interpolate('%(prop:revision:-None)s'), 'BUILDBOT_REVISION': util.Interpolate('%(prop:revision:-None)s'),
'BUILDBOT_SCHEDULER': util.Interpolate('%(prop:scheduler:-None)s'), 'BUILDBOT_SCHEDULER': util.Interpolate('%(prop:scheduler:-None)s'),
'BUILDBOT_SLAVENAME': util.Interpolate('%(prop:slavename:-None)s'),
'BUILDBOT_MSAN_ORIGINS': util.Interpolate('%(prop:msan_origins:-)s'),
} }
# Apply the passed in environment on top. # Apply the passed in environment on top.
old_env = kwargs.get('env') or {} old_env = kwargs.get('env') or {}
env.update(old_env) env.update(old_env)
# Change passed in args (ok as a copy is made internally). # Change passed in args (ok as a copy is made internally).
kwargs['env'] = env kwargs['env'] = env
kwargs = self.setupShellMixin(kwargs) kwargs = self.setupShellMixin(kwargs)
Show All 11 Lines def __init__(self, **kwargs):
# Use this lock to "run" the annotated steps. # Use this lock to "run" the annotated steps.
self.initLock = defer.DeferredLock() self.initLock = defer.DeferredLock()
self._annotated_finished = False self._annotated_finished = False
def _getLastAnnotatedStep(self): def _getLastAnnotatedStep(self):
return self.annotated_steps[-1] if self.annotated_steps else None return self.annotated_steps[-1] if self.annotated_steps else None
def _updateLastAnnotatedStepStatus(self, status): def _updateLastAnnotatedStepStatus(self, status):
debuglog(">>> AnnotatedCommand::_updateLastAnnotatedStepStatus(%r)" % status) debuglog(">>> AnnotatedCommand::_updateLastAnnotatedStepStatus({})".format(status))
# Alway update the common annotate command status. # Alway update the common annotate command status.
self.annotate_status = results.worst_status(self.annotate_status, status) self.annotate_status = results.worst_status(self.annotate_status, status)
s = self._getLastAnnotatedStep() s = self._getLastAnnotatedStep()
if s is not None: if s is not None:
s.updateStatus(status) s.updateStatus(status)
if self.halt_on_failure and status in [results.FAILURE, results.EXCEPTION]: if self.halt_on_failure and status in [results.FAILURE, results.EXCEPTION]:
if self.cmd: if self.cmd:
self.cmd.interrupt("Failed annotated step") self.cmd.interrupt("Failed annotated step")
def _updateLastAnnotatedStepText(self, text = None): def _updateLastAnnotatedStepText(self, text = None):
"""Updating a step text. None to clean up.""" """Updating a step text. None to clean up."""
debuglog(">>> AnnotatedCommand::_updateLastAnnotatedStepText('%s')" % (text if text else '')) debuglog(">>> AnnotatedCommand::_updateLastAnnotatedStepText('{}')".format(
text if text else ''))
s = self._getLastAnnotatedStep() s = self._getLastAnnotatedStep()
if s is not None: if s is not None:
s.setStepText(text) s.setStepText(text)
def _updateLastAnnotatedStepSummaryInfo(self, text = None): def _updateLastAnnotatedStepSummaryInfo(self, text = None):
"""Updating a step summary info. None to clean up.""" """Updating a step summary info. None to clean up."""
debuglog(">>> AnnotatedCommand::_updateLastAnnotatedStepSummaryInfo('%s')" % (text if text else '')) debuglog(">>> AnnotatedCommand::_updateLastAnnotatedStepSummaryInfo('{}')".format(
text if text else ''))
s = self._getLastAnnotatedStep() s = self._getLastAnnotatedStep()
if s is not None: if s is not None:
s.setStepSummary(text) s.setStepSummary(text)
def _fixupActiveAnnotatedStep(self): def _fixupActiveAnnotatedStep(self):
debuglog(">>> AnnotatedCommand::_fixupActiveAnnotatedStep()") debuglog(">>> AnnotatedCommand::_fixupActiveAnnotatedStep()")
# Fixup the active step # Fixup the active step
s = self._getLastAnnotatedStep() s = self._getLastAnnotatedStep()
if s is not None: if s is not None:
# Finish the active step with its current status. # Finish the active step with its current status.
s.requestFinish() s.requestFinish()
def _scheduleNewAnnotatedStep(self, name, logline): def _scheduleNewAnnotatedStep(self, name, logline):
debuglog(">>> AnnotatedCommand::_scheduleNewAnnotatedStep('%s')" % name) debuglog(">>> AnnotatedCommand::_scheduleNewAnnotatedStep('{}')".format(name))
self._fixupActiveAnnotatedStep() self._fixupActiveAnnotatedStep()
s = AnnotatedBuildStep(parent_step=self, name=name) s = AnnotatedBuildStep(parent_step=self, name=name)
# Doing this last so that @@@BUILD_STEP... occurs in the log of the new # Doing this last so that @@@BUILD_STEP... occurs in the log of the new
# step. # step.
s.scheduleStdout(logline) s.scheduleStdout(logline)
# Make it ready to consume the logs and status updates. # Make it ready to consume the logs and status updates.
▲ Show 20 Lines • Show All 112 Lines • ▼ Show 20 Lines def processAnnotatedCommand(self, line):
# We will need to save it in some cases. # We will need to save it in some cases.
args['logline'] = ln args['logline'] = ln
return ancmd, args return ancmd, args
return None, None return None, None
ancmd, args = parse_annotate_cmd(line) ancmd, args = parse_annotate_cmd(line)
if ancmd: if ancmd:
debuglog(">>> AnnotatedCommand::processAnnotatedCommand(): %s, %r" % (ancmd, args)) debuglog(">>> AnnotatedCommand::processAnnotatedCommand(): {}, {}".format(ancmd, args))
try: try:
if ancmd == STEP_LINK: if ancmd == STEP_LINK:
s = self._getLastAnnotatedStep() s = self._getLastAnnotatedStep()
if s is not None: if s is not None:
s.addURL(args['link_label'], args['link_url']) s.addURL(args['link_label'], args['link_url'])
else: else:
logging.msg("Warning: missed link for annotated step. " logging.msg("Warning: missed link for annotated step. "
Show All 33 Lines

zorg/buildbot/commands/LitTestCommand.py

Show All 13 Linesclass LitLogObserver(LogLineObserver):
# Regular expressions for verbose log start and stop markers. Verbose log # Regular expressions for verbose log start and stop markers. Verbose log
# output always immediately follow a test. # output always immediately follow a test.
kTestVerboseLogStartRE = re.compile(r"""\*{4,80} TEST '(.*)' .*""") kTestVerboseLogStartRE = re.compile(r"""\*{4,80} TEST '(.*)' .*""")
kTestVerboseLogStopRE = re.compile(r"""\*{10,80}""") kTestVerboseLogStopRE = re.compile(r"""\*{10,80}""")
# These are the codes for which we will include the log output in the buildbot # These are the codes for which we will include the log output in the buildbot
# step results. # step results.
failingCodes = set(['FAIL', 'XPASS', 'KPASS', 'UNRESOLVED', 'TIMEOUT', 'ERROR']) failingCodes = set(['FAIL', 'XPASS', 'KPASS', 'UNRESOLVED', 'TIMEOUT', 'ERROR', 'NOEXE'])
# Regular expressions for start of summary marker. # Regular expressions for start of summary marker.
kStartSummaryRE = re.compile(r'^Failing Tests \(\d*\)$') kStartSummaryRE = re.compile(r'^Failing Tests \(\d*\)$')
def __init__(self, maxLogs=None, parseSummaryOnly=False): def __init__(self, maxLogs=None, parseSummaryOnly=False):
super().__init__() super().__init__()
self.resultCounts = {} self.resultCounts = {}
self.maxLogs = maxLogs self.maxLogs = maxLogs
self.numLogs = 0 self.numLogs = 0
▲ Show 20 Lines • Show All 106 Lines • ▼ Show 20 Lines if self.parserStarted is True:
if m: if m:
# Remember the last test result and update the result counts. # Remember the last test result and update the result counts.
self.lastTestResult = (code, name) = m.groups() self.lastTestResult = (code, name) = m.groups()
self.resultCounts[code] = self.resultCounts.get(code, 0) + 1 self.resultCounts[code] = self.resultCounts.get(code, 0) + 1
return return
class LitTestCommand(Test): class LitTestCommand(Test):
resultNames = {'FAIL':'unexpected failures', resultNames = {'FAIL':'unexpected failures',
'XPASS':'unexpected passes',
'PASS':'expected passes', 'PASS':'expected passes',
'XFAIL':'expected failures', 'XFAIL':'expected failures',
'XPASS':'unexpected passes',
'KFAIL':'known failures', 'KFAIL':'known failures',
'KPASS':'unknown passes', 'KPASS':'unknown passes',
'UNRESOLVED':'unresolved testcases', 'UNRESOLVED':'unresolved testcases',
'UNTESTED':'untested testcases', 'UNTESTED':'untested testcases',
'REGRESSED':'runtime performance regression', 'REGRESSED':'runtime performance regression',
'IMPROVED':'runtime performance improvement', 'IMPROVED':'runtime performance improvement',
'UNSUPPORTED':'unsupported tests', 'UNSUPPORTED':'unsupported tests',
'TIMEOUT':'timeout waiting for results'} 'SKIPPED':'skipped tests',
'TIMEOUT':'timeout waiting for results',
'NOEXE':'test executable is missing'}
def __init__(self, ignore=[], flaky=[], max_logs=20, parseSummaryOnly=False, def __init__(self, ignore=[], flaky=[], max_logs=20, parseSummaryOnly=False,
*args, **kwargs): *args, **kwargs):
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
self.maxLogs = int(max_logs) self.maxLogs = int(max_logs)
self.logObserver = LitLogObserver(self.maxLogs, parseSummaryOnly) self.logObserver = LitLogObserver(self.maxLogs, parseSummaryOnly)
self.addFactoryArguments(max_logs=max_logs) self.addFactoryArguments(max_logs=max_logs)
self.addFactoryArguments(parseSummaryOnly=parseSummaryOnly) self.addFactoryArguments(parseSummaryOnly=parseSummaryOnly)
self.addLogObserver('stdio', self.logObserver) self.addLogObserver('stdio', self.logObserver)
def evaluateCommand(self, cmd): def evaluateCommand(self, cmd):
# Always report failure if the command itself failed. # Always report failure if the command itself failed.
if cmd.rc != 0: if cmd.rc != 0:
return FAILURE return FAILURE
# Otherwise, report failure if there were failures in the log. # Otherwise, report failure if there were failures in the log.
if self.logObserver.hadFailure(): if self.logObserver.hadFailure():
return FAILURE return FAILURE
return SUCCESS return SUCCESS
def describe(self, done=False): def describe(self, done=False):
description = Test.describe(self, done) or list() description = Test.describe(self, done) or list()
for name, count in self.logObserver.resultCounts.items():
if name in self.resultNames: # We use resultNames here so that the printed order is always the same.
description.append('{0} {1}'.format(count, self.resultNames[name])) # resultCounts is a dictionary that goes by insertion order instead.
else: for resultName, resultDescription in self.resultNames.items():
count = self.logObserver.resultCounts.get(resultName, 0)
if count:
print(">>>> LitLogObserver.describe: description={}".format(description))
description.append('{0} {1}'.format(count, resultDescription))
unknown = set(self.logObserver.resultCounts.keys()) - set(self.resultNames.keys())
for name in unknown:
description.append('Unexpected test result output ' + name) description.append('Unexpected test result output ' + name)
return description return description

zorg/buildbot/process/factory.py

# TODO: Change WithProperties to Interpolate # TODO: Change WithProperties to Interpolate
# TODO: Consider setting codebase to llvm-project. # TODO: Consider setting codebase to llvm-project.
from collections import OrderedDict from collections import OrderedDict
from buildbot.process.factory import BuildFactory from buildbot.process.factory import BuildFactory
from buildbot.plugins import util, steps from buildbot.plugins import util, steps
_all_runtimes = frozenset([
"compiler-rt",
"libc",
"libcxx",
"libcxxabi",
"libunwind",
"openmp",
])
class LLVMBuildFactory(BuildFactory): class LLVMBuildFactory(BuildFactory):
""" """
TODO: Document TODO: Document
depends_on_projects is a list of LLVM projects a produced builder
depends on. If None, it gets discovered depending on other params.
enable_runtimes is a list of enabled runtimes. If None,
it gets discovered based on the depends_on_projects list.
""" """
def __init__(self, steps=None, depends_on_projects=None, **kwargs): def __init__(self, steps=None, depends_on_projects=None, **kwargs):
# Cannot use "super" here as BuildFactory is an old style class. # Cannot use "super" here as BuildFactory is an old style class.
BuildFactory.__init__(self, steps) BuildFactory.__init__(self, steps)
# Handle the dependencies.
if depends_on_projects is None: if depends_on_projects is None:
self.depends_on_projects = frozenset(['llvm']) # llvm project is always included.
self.depends_on_projects = set(['llvm'])
else: else:
self.depends_on_projects = frozenset(depends_on_projects) self.depends_on_projects = frozenset(depends_on_projects)
enable_runtimes = kwargs.pop('enable_runtimes', None)
# If specified, we either got a givem list of
# enabled runtimes, or "auto", or "all".
if enable_runtimes is None:
# For the backward compatibility, we do not use
# enable_runtimes unless it is requested explicitly.
self.enable_runtimes = frozenset([])
elif enable_runtimes == "auto":
# Let's build the list of runtimes based on the given
# depends_on_projects list.
self.enable_runtimes = \
self.depends_on_projects.intersection(_all_runtimes)
else:
if enable_runtimes == "all":
# Let's replace the "all" placeholder by
# the actual list of all runtimes.
self.enable_runtimes = frozenset(_all_runtimes)
else:
# Let's just use the given list, no need to discover.
self.enable_runtimes = frozenset(enable_runtimes)
# Update the list of dependencies.
if depends_on_projects is None:
self.depends_on_projects.update(self.enable_runtimes)
# Build the list of projects to enable.
self.enable_projects = \
self.depends_on_projects.difference(self.enable_runtimes)
# Directories. # Directories.
self.llvm_srcdir = kwargs.pop('llvm_srcdir', None) self.monorepo_dir = kwargs.pop('llvm_srcdir', None)
self.src_to_build_dir = kwargs.pop('src_to_build_dir', None)
self.obj_dir = kwargs.pop('obj_dir', None) self.obj_dir = kwargs.pop('obj_dir', None)
self.install_dir = kwargs.pop('install_dir', None) self.install_dir = kwargs.pop('install_dir', None)
# Preserve the rest of the given extra attributes if any, so we could # Preserve the rest of the given extra attributes if any, so we could
# expand the factory later. # expand the factory later.
for k,v in kwargs.items(): for k,v in kwargs.items():
setattr(self, k, v) setattr(self, k, v)
self.monorepo_dir = self.llvm_srcdir or "llvm-project" self.monorepo_dir = self.monorepo_dir or "llvm-project"
self.src_to_build_dir = self.src_to_build_dir or 'llvm'
self.llvm_srcdir = \ self.llvm_srcdir = \
"%(monorepo_dir)s/llvm" % {'monorepo_dir' : self.monorepo_dir} "{}/{}".format(self.monorepo_dir, self.src_to_build_dir)
self.obj_dir = \ self.obj_dir = \
self.obj_dir or "build" self.obj_dir or "build"
# Repourl_prefix could be specified per builder. Otherwise we use github. # Repourl_prefix could be specified per builder. Otherwise we use github.
self.repourl_prefix = kwargs.pop('repourl_prefix', 'https://github.com/llvm/') self.repourl_prefix = kwargs.pop('repourl_prefix', 'https://github.com/llvm/')
@staticmethod @staticmethod
Show All 23 Lines def pathRelativeTo(path, basePath):
# Everything matches. # Everything matches.
rel_path = '.' rel_path = '.'
return rel_path return rel_path
def addGetSourcecodeSteps(self, **kwargs): def addGetSourcecodeSteps(self, **kwargs):
# Checkout the monorepo. # Checkout the monorepo.
# Documentation: http://docs.buildbot.net/current/manual/configuration/buildsteps.html#git
self.addStep(steps.Git( self.addStep(steps.Git(
name='Checkout the source code', name='Checkout the source code',
repourl=self.repourl_prefix + "llvm-project.git", repourl=self.repourl_prefix + "llvm-project.git",
progress=True, progress=True,
workdir=util.WithProperties(self.monorepo_dir), workdir=util.WithProperties(self.monorepo_dir),
retryFetch=True,
clobberOnFailure=True,
**kwargs)) **kwargs))
# Checkout a given LLVM project to the given directory. # Checkout a given LLVM project to the given directory.
# TODO: Handle clean property and self.clean attribute. # TODO: Handle clean property and self.clean attribute.
def addGetSourcecodeForProject(self, project, name=None, src_dir=None, **kwargs): def addGetSourcecodeForProject(self, project, name=None, src_dir=None, **kwargs):
# project contains a repo name which is not a part of the monorepo. # project contains a repo name which is not a part of the monorepo.
# We do not enforce it here, though. # We do not enforce it here, though.
Show All 12 Lines def addGetSourcecodeForProject(self, project, name=None, src_dir=None, **kwargs):
# Ignore workdir if given. We check out to src_dir. # Ignore workdir if given. We check out to src_dir.
kwargs.pop('workdir', None) kwargs.pop('workdir', None)
self.addStep(steps.Git( self.addStep(steps.Git(
name=name, name=name,
repourl=_repourl, repourl=_repourl,
progress=True, progress=True,
workdir=util.WithProperties(src_dir), workdir=util.WithProperties(src_dir),
retryFetch=True,
clobberOnFailure=True,
**kwargs)) **kwargs))

zorg/buildbot/util/InformativeMailNotifier.py

# TODO: Add support for clang crash dumps. import traceback
# TODO: Better handle unit/regression tests failures
# TODO: For debug purposes. Remove this later.
from twisted.python import log
from buildbot.plugins import reporters from buildbot.plugins import reporters
from buildbot.process.results import CANCELLED
from buildbot.process.results import EXCEPTION
from buildbot.process.results import FAILURE
from buildbot.process.results import RETRY
from buildbot.process.results import SKIPPED
from buildbot.process.results import SUCCESS
from buildbot.process.results import WARNINGS
from buildbot.process.results import statusToString
def get_log_details(build):
text = ""
try:
for step in build['steps']:
results = step['results']
if results == SUCCESS or results == SKIPPED:
continue
text += "Step {} ({}) {}: {}\n".format(step['number'], step['name'], statusToString(results), step['state_string'])
logs = step['logs']
if logs:
log_index = -1
log_type = 0
for i, _log in enumerate(logs):
if _log['name'].startswith("FAIL: "): # Use only first logchunk FAIL:
log_type = 3
log_index = i
elif log_type < 2 and _log['name'].startswith("warnings "):
log_type = 2
log_index = i
elif log_type < 1 and _log['type'] == "s": # stdio
log_type = 1
log_index = i
if log_index < 0:
continue
log_text = logs[log_index]['content']['content']
if log_type == 1:
# Parse stdio
lines = log_text.splitlines()
for line in lines[:]:
if line.startswith("h"):
lines.remove(line)
for j, line in enumerate(lines):
if line.startswith("o") or line.startswith("e"):
lines[j] = line[1:]
for j, line in enumerate(lines):
if line.find("FAIL:") != -1 or line.find("FAILED") != -1:
if j > 10:
del lines[:j-10] # Start 10 lines before FAIL
lines = ["..."] + lines
del lines[50:] # Keep up to 50 lines around FAIL
break
if len(lines) > 50:
del lines[:len(lines)-50] # Otherwise keep last 50 lines
lines = ["..."] + lines
log_text = "\n".join(lines)
elif logs[log_index]['num_lines'] > 50:
# Keep first 50 lines
lines = log_text.splitlines()
del lines[50:]
log_text = "\n".join(lines + ["..."])
text += log_text + "\n"
except Exception as err:
print("Exception in LLVMMessageFormatter.get_log_details(): {}\n{}".format(err, traceback.format_exc()))
# TODO: We should send something useful in this case.
return dict(details=text)
def _get_logs_and_tracebacks_from_build(build): # TODO: Add build reason if we have that valid and available
# TODO: Implement interesting parts of the logs and tracebacks extraction. #Build Reason: {{ build['properties'].get('reason', ["<unknown>"])[0] }}
return dict()
MAIL_TEMPLATE = """\ MAIL_TEMPLATE = """\
The Buildbot has detected a {{ status_detected }} on builder {{ buildername }} while building {{ projects }}. The Buildbot has detected a {{ status_detected }} on builder {{ buildername }} while building {{ projects }}.
Full details are available at: Full details are available at:
{{ build_url }} {{ build_url }}
Buildbot URL: {{ buildbot_url }}
Worker for this Build: {{ workername }} Worker for this Build: {{ workername }}
Build Reason: {{ build['properties'].get('reason', ["<unknown>"])[0] }} Blamelist:
Blamelist: {{ ", ".join(blamelist) }} {{ ",\n ".join(blamelist) }}
{{ summary }} {{ summary }}
{{ details }}
Sincerely, Sincerely,
LLVM Buildbot LLVM Buildbot
""" """
class LLVMMessageFormatter(reporters.MessageFormatter): class LLVMMessageFormatter(reporters.MessageFormatter):
def buildAdditionalContext(self, master, ctx): def buildAdditionalContext(self, master, ctx):
#log.msg(">>> LLVMMessageFormatter.buildAdditionalContext got ctx={}".format(ctx))
ctx.update(self.ctx) ctx.update(self.ctx)
ctx.update(get_log_details(ctx["build"]))
build = ctx["build"]
build_interesting_data = _get_logs_and_tracebacks_from_build(build)
#log.msg(">>> LLVMMessageFormatter.buildAdditionalContext build_interesting_data={}",format(build_interesting_data))
ctx["build"].update(build_interesting_data)
LLVMInformativeMailNotifier = LLVMMessageFormatter( LLVMInformativeMailNotifier = LLVMMessageFormatter(
template=MAIL_TEMPLATE, template=MAIL_TEMPLATE,
template_type="plain", template_type="plain",
wantLogs=True, wantLogs=True,
wantProperties=True, wantProperties=True,
wantSteps=True, wantSteps=True,
) )

zorg/buildbot/util/workerowner.py

  • This file was added.
import re
from twisted.internet import defer
from buildbot.util import unicode2bytes
from buildbot.www.authz.endpointmatchers import Match
from buildbot.www.authz.endpointmatchers import EndpointMatcherBase
from buildbot.www.authz.roles import RolesFromBase
from buildbot.www.authz.authz import Forbidden
from buildbot.www.authz.authz import Authz
class WorkerOwnerMatch(Match):
def __init__(self, master, workers, **kwargs):
super().__init__(master, **kwargs)
self.workerOwners = []
for worker in workers:
if 'workerinfo' in worker:
workerinfo = worker['workerinfo']
if 'admin' in workerinfo:
email = re.search('.*<(.*)>.*', workerinfo['admin'])
if email:
self.workerOwners.append(email.group(1))
def getWorkerOwners(self):
return self.workerOwners
class WorkerEndpointMatcher(EndpointMatcherBase):
def __init__(self, **kwargs):
super().__init__(**kwargs)
@defer.inlineCallbacks
def match_WorkerEndpoint(self, epobject, epdict, options):
worker = yield epobject.get({}, epdict)
return WorkerOwnerMatch(self.master, [worker])
@defer.inlineCallbacks
def match_BuildRequestEndpoint_cancel(self, epobject, epdict, options):
buildrequest = yield epobject.get({}, epdict)
if 'builderid' in buildrequest:
workers = yield self.master.db.workers.getWorkers(builderid=buildrequest['builderid'])
if workers:
return WorkerOwnerMatch(self.master, workers)
return None
class RolesFromWorkerOwner(RolesFromBase):
def __init__(self, role):
super().__init__()
self.role = role
def getRolesFromUser(self, userDetails, workerOwners):
if 'email' in userDetails:
if workerOwners and userDetails['email'] in workerOwners:
return [self.role]
return []
class WorkerOwnerAuthz(Authz):
def __init__(self, roleMatchers=None, **kwargs):
super().__init__(roleMatchers=roleMatchers, **kwargs)
self.workerOwnerRoleMatchers = [
r for r in roleMatchers if isinstance(r, RolesFromWorkerOwner)]
self.roleMatchers = [ # Remove workerOwnerRoleMatchers from roleMatchers
r for r in self.roleMatchers if r not in self.workerOwnerRoleMatchers]
def setMaster(self, master):
super().setMaster(master)
for r in self.workerOwnerRoleMatchers:
r.setAuthz(self)
@defer.inlineCallbacks
def assertUserAllowed(self, ep, action, options, userDetails):
IsWorkerOwnerMatch = False
roles = self.getRolesFromUser(userDetails)
for rule in self.allowRules:
match = yield rule.match(ep, action, options)
if match is not None:
# only try to get owner if there are owner Matchers
if self.ownerRoleMatchers:
owner = yield match.getOwner()
if owner:
for r in self.ownerRoleMatchers:
roles.update(set(r.getRolesFromUser(userDetails, owner)))
if self.workerOwnerRoleMatchers and isinstance(match, WorkerOwnerMatch):
IsWorkerOwnerMatch = True
workerOwners = match.getWorkerOwners()
if workerOwners:
for r in self.workerOwnerRoleMatchers:
roles.update(set(r.getRolesFromUser(userDetails, workerOwners)))
for role in roles:
if self.match(role, rule.role): # fnmatch or re.match
return None
if not rule.defaultDeny:
continue # check next suitable rule if not denied
# f" You need to have role '{rule.role}'."
error_msg = unicode2bytes(" You must be the worker owner." if IsWorkerOwnerMatch else
" You must be an author of the commit triggered the build.")
raise Forbidden(error_msg)
return None

zorg/jenkins/build.py

Show All 9 Lines
import argparse import argparse
import shutil import shutil
import math import math
import re import re
import xml.etree.ElementTree as ET import xml.etree.ElementTree as ET
from contextlib import contextmanager from contextlib import contextmanager
from urllib2 import urlopen, URLError, HTTPError from urllib2 import urlopen, URLError, HTTPError
SERVER = "labmaster2.lab.llvm.org" SERVER = "green-dragon-21.local"
NINJA = "/usr/local/bin/ninja" NINJA = "/usr/local/bin/ninja"
# Add dependency checker to the Python path. # Add dependency checker to the Python path.
# For relative reference to the dependency file. # For relative reference to the dependency file.
here = os.path.dirname(os.path.abspath(__file__)) here = os.path.dirname(os.path.abspath(__file__))
sys.path.append(os.path.abspath(here + "/../../dep/")) sys.path.append(os.path.abspath(here + "/../../dep/"))
import dep # noqa import dep # noqa
Show All 39 Lines def __init__(self, args):
self._lldb_src_dir = os.environ.get('LLDB_SRC_DIR', 'lldb') self._lldb_src_dir = os.environ.get('LLDB_SRC_DIR', 'lldb')
self._build_dir = os.environ.get('BUILD_DIR', 'clang-build') self._build_dir = os.environ.get('BUILD_DIR', 'clang-build')
self._lldb_build_dir = os.environ.get('LLDB_BUILD_DIR', 'lldb-build') self._lldb_build_dir = os.environ.get('LLDB_BUILD_DIR', 'lldb-build')
self._install_dir = os.environ.get('INSTALL_DIR', 'clang-install') self._install_dir = os.environ.get('INSTALL_DIR', 'clang-install')
self.j_level = os.environ.get('J_LEVEL', None) self.j_level = os.environ.get('J_LEVEL', None)
self.max_parallel_tests = os.environ.get('MAX_PARALLEL_TESTS', None) self.max_parallel_tests = os.environ.get('MAX_PARALLEL_TESTS', None)
self.max_parallel_links = os.environ.get('MAX_PARALLEL_LINKS', None) self.max_parallel_links = os.environ.get('MAX_PARALLEL_LINKS', None)
self.host_compiler_url = os.environ.get('HOST_URL', self.host_compiler_url = os.environ.get('HOST_URL',
'http://labmaster2.local/artifacts/') 'http://green-dragon-21.local/artifacts/')
self.artifact_url = os.environ.get('ARTIFACT', 'NONE') self.artifact_url = os.environ.get('ARTIFACT', 'NONE')
self.job_name = os.environ.get('JOB_NAME', 'NONE') self.job_name = os.environ.get('JOB_NAME', 'NONE')
self.build_id = os.environ.get('BUILD_ID', 'NONE') self.build_id = os.environ.get('BUILD_ID', 'NONE')
self.build_number = os.environ.get('BUILD_NUMBER', 'NONE') self.build_number = os.environ.get('BUILD_NUMBER', 'NONE')
self.svn_rev = os.environ.get('LLVM_REV', 'NONE') self.svn_rev = os.environ.get('LLVM_REV', 'NONE')
self.nobootstrap = True self.nobootstrap = True
self.device = None self.device = None
self._svn_url_cache = None self._svn_url_cache = None
▲ Show 20 Lines • Show All 307 Lines • ▼ Show 20 Lines if target == "all" or target == "build":
'-DCLANG_REPOSITORY_STRING={}'.format( '-DCLANG_REPOSITORY_STRING={}'.format(
conf.branch()), conf.branch()),
'-DCLANG_APPEND_VC_REV=On', '-DCLANG_APPEND_VC_REV=On',
'-DSVN_REVISION={}'.format(conf.svn_rev), '-DSVN_REVISION={}'.format(conf.svn_rev),
'-DLLVM_BUILD_TESTS=On', '-DLLVM_BUILD_TESTS=On',
'-DLLVM_INCLUDE_TESTS=On', '-DLLVM_INCLUDE_TESTS=On',
'-DCLANG_INCLUDE_TESTS=On', '-DCLANG_INCLUDE_TESTS=On',
'-DLLVM_INCLUDE_UTILS=On', '-DLLVM_INCLUDE_UTILS=On',
'-DLIBCXX_INSTALL_HEADERS=On',
'-DLIBCXX_OVERRIDE_DARWIN_INSTALL=On',
'-DLIBCXX_INSTALL_LIBRARY=Off',
'-DCMAKE_MACOSX_RPATH=On', '-DCMAKE_MACOSX_RPATH=On',
] ]
if dyld_path: if dyld_path:
cmake_command += ['-DDYLD_LIBRARY_PATH=' + dyld_path] cmake_command += ['-DDYLD_LIBRARY_PATH=' + dyld_path]
if libtool_path: if libtool_path:
cmake_command += ['-DCMAKE_LIBTOOL=' + libtool_path] cmake_command += ['-DCMAKE_LIBTOOL=' + libtool_path]
▲ Show 20 Lines • Show All 201 Lines • ▼ Show 20 Linesdef lldb_cmake_builder():
run_cmd(conf.lldbbuilddir(), cmake_cmd) run_cmd(conf.lldbbuilddir(), cmake_cmd)
footer() footer()
header("Build") header("Build")
run_cmd(conf.lldbbuilddir(), [NINJA, '-v']) run_cmd(conf.lldbbuilddir(), [NINJA, '-v'])
footer() footer()
header("Run Tests") header("Run Tests")
run_cmd(conf.lldbbuilddir(), [NINJA, '-v', 'check-debuginfo'])
run_cmd(conf.lldbbuilddir(), ['/usr/bin/env', 'TERM=vt100', NINJA, '-v', run_cmd(conf.lldbbuilddir(), ['/usr/bin/env', 'TERM=vt100', NINJA, '-v',
'check-lldb']) 'check-debuginfo', 'check-lldb', '-k2'])
footer() footer()
def static_analyzer_benchmarks_builder(): def static_analyzer_benchmarks_builder():
"""Run static analyzer benchmarks""" """Run static analyzer benchmarks"""
header("Static Analyzer Benchmarks") header("Static Analyzer Benchmarks")
benchmark_script = conf.workspace + "/utils-analyzer/SATestBuild.py" benchmark_script = conf.workspace + "/utils-analyzer/SATestBuild.py"
▲ Show 20 Lines • Show All 513 Lines • Show Last 20 Lines

zorg/jenkins/clang_build_dependencies.dep

os_version == 10.15.5 # All nodes should be on 10.15.5 os_version == 10.15.5 # All nodes should be on 10.15.5
brew cmake >= 3.17.2 # All nodes have 3.17.2 brew cmake >= 3.17.2 # All nodes have 3.17.2
brew ninja >= 1.8.2 # Stage2 verbose output brew ninja >= 1.8.2 # Stage2 verbose output
pip psutil >= 5.4.5 # Required for timeouts in the GTest Suite. pip psutil >= 5.4.5 # Required for timeouts in the GTest Suite.
pip six >= 1.11.0 # Required for urllib xcode >= 11.5 # Node configs are not currently all on the same Xcode.
xcode == 11.5

zorg/jenkins/common.groovy

Show All 24 Lines
// //
// FIXME: Make this do a --bare or --mirror clone instead of a full clone. This // FIXME: Make this do a --bare or --mirror clone instead of a full clone. This
// is very space-intensive. // is very space-intensive.
private def clone_llvm_project(name, sha) { private def clone_llvm_project(name, sha) {
dir("pseudo-checkout-${name}") { dir("pseudo-checkout-${name}") {
checkout poll: false, changelog: true, scm: [ checkout poll: false, changelog: true, scm: [
$class: 'GitSCM', $class: 'GitSCM',
branches: [[name: sha ]], branches: [[name: sha ]],
userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']] extensions: [[
$class: 'CloneOption',
reference: '/Users/Shared/llvm-project.git'
]],
userRemoteConfigs: [[url: 'https://github.com/llvm/llvm-project.git']]
] ]
} }
} }
private def post_build() { private def post_build() {
// Analyze build log. // Analyze build log.
def base_url = 'http://labmaster2:8080/green' def base_url = 'http://labmaster2:8080/green'
def build_url = currentBuild.getRawBuild().getUrl() def build_url = currentBuild.getRawBuild().getUrl()
Show All 34 Linesprivate def post_build() {
// TODO: Notify IRC. // TODO: Notify IRC.
} }
def task_pipeline(label, body) { def task_pipeline(label, body) {
node(label) { node(label) {
try { try {
stage('main') { stage('main') {
dir('config') { dir('config') {
git url: 'https://github.com/llvm/llvm-zorg.git', branch: 'master', poll: false git url: 'https://github.com/llvm/llvm-zorg.git', branch: 'main', poll: false
} }
body() body()
} }
} catch(hudson.AbortException e) { } catch(hudson.AbortException e) {
// No need to print the exception if something fails inside a 'sh' // No need to print the exception if something fails inside a 'sh'
// step. // step.
currentBuild.result = 'FAILURE' currentBuild.result = 'FAILURE'
} catch (Exception e) { } catch (Exception e) {
currentBuild.result = 'FAILURE' currentBuild.result = 'FAILURE'
throw e throw e
} finally { } finally {
stage('post') { stage('post') {
post_build() post_build()
} }
} }
} }
} }
def benchmark_pipeline(label, body) { def benchmark_pipeline(label, body) {
properties([ properties([
disableResume(),
parameters([ parameters([
string(name: 'ARTIFACT'), string(name: 'ARTIFACT'),
string(name: 'GIT_DISTANCE'), string(name: 'GIT_DISTANCE'),
string(name: 'GIT_SHA') string(name: 'GIT_SHA')
]) ])
]) ])
currentBuild.displayName = basename(params.ARTIFACT) currentBuild.displayName = basename(params.ARTIFACT)
task_pipeline(label) { task_pipeline(label) {
clone_llvm_project('llvm-project', params.GIT_SHA) clone_llvm_project('llvm-project', params.GIT_SHA)
body() body()
} }
} }
def testsuite_pipeline(label, body) { def testsuite_pipeline(label, body) {
benchmark_pipeline(label) { benchmark_pipeline(label) {
dir('lnt') { dir('lnt') {
git url: 'https://github.com/llvm/llvm-lnt.git', branch: 'master', poll: false git url: 'https://github.com/llvm/llvm-lnt.git', branch: 'main', poll: false
} }
dir('test-suite') { dir('test-suite') {
git url: 'https://github.com/llvm/llvm-test-suite.git', branch: 'master', poll: false git url: 'https://github.com/llvm/llvm-test-suite.git', branch: 'main', poll: false
} }
body() body()
} }
} }
return this return this

zorg/jenkins/inspect_log.py

Show All 20 Lines
_JENKINS_POOL_PATTERN = r'\[.*]\s' _JENKINS_POOL_PATTERN = r'\[.*]\s'
_JENKINS_POOL_REGEX = re.compile('^'+_JENKINS_POOL_PATTERN) _JENKINS_POOL_REGEX = re.compile('^'+_JENKINS_POOL_PATTERN)
class _Matcher(object): class _Matcher(object):
'''Matching engine. Keeps data structures to match a single line against '''Matching engine. Keeps data structures to match a single line against
a fixed list of patterns.''' a fixed list of patterns.'''
def __init__(self, patterns): def __init__(self, patterns):
# Create a combined master regex combining all patterns. # Create a combined main regex combining all patterns.
combined = '' combined = ''
merge = '' merge = ''
for pattern in patterns: for pattern in patterns:
# Convert named groups into anonymous ones to avoid name clashes. # Convert named groups into anonymous ones to avoid name clashes.
p = pattern.pattern p = pattern.pattern
p = re.sub(r'\(\?P<[^>]+>', '(?:', p) p = re.sub(r'\(\?P<[^>]+>', '(?:', p)
combined += merge combined += merge
combined += '(?:%s)' % p combined += '(?:%s)' % p
▲ Show 20 Lines • Show All 116 Lines • Show Last 20 Lines

zorg/jenkins/jobs/jobs/clang-san-iossim

  • This file was added.
pipeline {
agent { label 'green-dragon-08' }
parameters {
string(name: 'GIT_REVISION', defaultValue: '*/main', description: 'Git revision to build')
}
stages {
stage('Checkout') {
steps {
dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'https://github.com/llvm/llvm-project.git']]])
}
dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/llvm/llvm-zorg.git']]])
}
}
}
stage('Build') {
steps {
timeout(90) {
sh '''
set -u
cd llvm-project
git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1}
cd -
export PATH=$PATH:/usr/bin:/usr/local/bin
# Set a MacOS minimum deployment target
export MACOSX_DEPLOYMENT_TARGET=10.14
rm -rf clang-install *.tar.gz
python llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \
--assertions --cmake-type=RelWithDebInfo \
--projects="clang;clang-tools-extra;compiler-rt"
'''
}
}
}
stage('Test') {
steps {
timeout(90) {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
export SANITIZER_IOSSIM_TEST_DEVICE_IDENTIFIER='iPhone 12'
COMPILER_RT_TEST_DIR="${WORKSPACE}/clang-build/tools/clang/runtime/compiler-rt-bins/test"
cd $COMPILER_RT_TEST_DIR/asan && python3 ${WORKSPACE}/clang-build/./bin/llvm-lit \
--xunit-xml-output=testresults-asan-IOSSimX86_64Config.xunit.xml -v -vv --timeout=600 \
$COMPILER_RT_TEST_DIR/asan/IOSSimX86_64Config/
cd $COMPILER_RT_TEST_DIR/tsan && python3 ${WORKSPACE}/clang-build/./bin/llvm-lit \
--xunit-xml-output=testresults-tsan-IOSSimX86_64Config.xunit.xml -v -vv --timeout=600 \
$COMPILER_RT_TEST_DIR/tsan/IOSSimX86_64Config/
cd $COMPILER_RT_TEST_DIR/ubsan && python3 ${WORKSPACE}/clang-build/./bin/llvm-lit \
--xunit-xml-output=testresults-ubsan-AddressSanitizer-iossim-x86_64.xunit.xml -v -vv --timeout=600 \
$COMPILER_RT_TEST_DIR/ubsan/AddressSanitizer-iossim-x86_64/
cd $COMPILER_RT_TEST_DIR/ubsan && python3 ${WORKSPACE}/clang-build/./bin/llvm-lit \
--xunit-xml-output=testresults-ubsan-Standalone-iossim-x86_64.xunit.xml -v -vv --timeout=600 \
$COMPILER_RT_TEST_DIR/ubsan/Standalone-iossim-x86_64/
cd $COMPILER_RT_TEST_DIR/ubsan && python3 ${WORKSPACE}/clang-build/./bin/llvm-lit \
--xunit-xml-output=testresults-ubsan-ThreadSanitizer-iossim-x86_64.xunit.xml -v -vv --timeout=600 \
$COMPILER_RT_TEST_DIR/ubsan/ThreadSanitizer-iossim-x86_64/
'''
}
junit 'clang-build/**/testresults-*.xunit.xml'
}
}
}
post {
success {
scanForIssues tool: clang()
}
regression {
emailext subject: '$DEFAULT_SUBJECT',
presendScript: '$DEFAULT_PRESEND_SCRIPT',
postsendScript: '$DEFAULT_POSTSEND_SCRIPT',
recipientProviders: [
[$class: 'CulpritsRecipientProvider'],
[$class: 'DevelopersRecipientProvider'],
[$class: 'RequesterRecipientProvider'],
],
replyTo: '$DEFAULT_REPLYTO',
to: '$DEFAULT_RECIPIENTS',
body:'$DEFAULT_CONTENT'
}
}
}

zorg/jenkins/jobs/jobs/clang-stage1-RA

  • This file was added.
pipeline {
agent { label 'green-dragon-02' }
parameters {
string(name: 'GIT_REVISION', defaultValue: '*/main', description: 'Git revision to build')
}
stages {
stage('Checkout') {
steps {
dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [
[name: params.GIT_REVISION]
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-project.git']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
]])
}
dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [
[name: '*/main']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-zorg.git']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-zorg.git']
]])
}
}
}
stage('Build') {
steps {
timeout(90) {
sh '''
set -u
rm -rf build.properties
cd llvm-project
git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1}
# Also save the LLVM_REV until LNT server is taught about GIT
export LLVM_REV=$(git show -q | grep "llvm-svn:" | cut -f2 -d":" | tr -d " ")
cd -
echo "GIT_DISTANCE=$GIT_DISTANCE" > build.properties
echo "GIT_SHA=$GIT_SHA" >> build.properties
echo "ARTIFACT=$JOB_NAME/clang-d$GIT_DISTANCE-g$GIT_SHA-t$BUILD_ID-b$BUILD_NUMBER.tar.gz" >> build.properties
export PATH=$PATH:/usr/bin:/usr/local/bin
# Set a MacOS minimum deployment target
export MACOSX_DEPLOYMENT_TARGET=10.14
rm -rf clang-build clang-install *.tar.gz
python llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \
--assertions --cmake-type=RelWithDebInfo \
--projects="clang;clang-tools-extra;compiler-rt"
'''
}
}
}
stage('Test') {
steps {
timeout(90) {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
python llvm-zorg/zorg/jenkins/monorepo_build.py cmake testlong
'''
}
junit 'clang-build/**/testresults.xunit.xml'
}
}
}
post {
always {
scanForIssues tool: clang()
}
regression {
emailext subject: '$DEFAULT_SUBJECT',
presendScript: '$DEFAULT_PRESEND_SCRIPT',
postsendScript: '$DEFAULT_POSTSEND_SCRIPT',
recipientProviders: [
[$class: 'CulpritsRecipientProvider'],
[$class: 'DevelopersRecipientProvider'],
[$class: 'RequesterRecipientProvider'],
],
replyTo: '$DEFAULT_REPLYTO',
to: '$DEFAULT_RECIPIENTS',
body:'$DEFAULT_CONTENT'
}
success {
build job: 'clang-stage2-Rthinlto_relay', wait: false
build job: 'clang-stage2-cmake-RgSan_relay', wait: false
//build job: 'phase2_modules_relay', wait: false
build job: 'relay-lnt-test-suite', wait: false
build job: 'relay-lnt-ctmark', wait: false
build job: 'relay-test-suite-verify-machineinstrs', wait: false
}
unstable {
build job: 'clang-stage2-Rthinlto_relay', wait: false
build job: 'clang-stage2-cmake-RgSan_relay', wait: false
//build job: 'phase2_modules_relay', wait: false
build job: 'relay-lnt-test-suite', wait: false
build job: 'relay-lnt-ctmark', wait: false
build job: 'relay-test-suite-verify-machineinstrs', wait: false
}
}
}

zorg/jenkins/jobs/jobs/clang-stage1-cmake-RA-incremental

  • This file was added.
pipeline {
agent { label 'green-dragon-04' }
parameters {
string(name: 'GIT_REVISION', defaultValue: '*/main', description: 'Git revision to build')
}
stages {
stage('Checkout') {
steps {
dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [
[name: params.GIT_REVISION]
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-project.git']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
]])
}
dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [
[name: '*/main']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-zorg.git']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-zorg.git']
]])
}
}
}
stage('Build') {
steps {
timeout(120) {
sh '''
set -u
rm -rf build.properties
cd llvm-project
git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1}
# Also save the LLVM_REV until LNT server is taught about GIT
export LLVM_REV=$(git show -q | grep "llvm-svn:" | cut -f2 -d":" | tr -d " ")
cd -
export PATH=$PATH:/usr/bin:/usr/local/bin
# Set a MacOS minimum deployment target
export MACOSX_DEPLOYMENT_TARGET=10.14
python llvm-zorg/zorg/jenkins/monorepo_build.py cmake build --assertions --projects="clang"
# Removing the local artifact
rm -rf clang-*.tar.gz
'''
}
}
}
stage('Test') {
steps {
timeout(120) {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
rm -rf clang-build/testresults.xunit.xml
python llvm-zorg/zorg/jenkins/monorepo_build.py cmake testlong
'''
}
junit 'clang-build/**/testresults.xunit.xml'
}
}
}
post {
always {
scanForIssues tool: clang()
}
/* Disabled email notification.
regression {
emailext subject: '$DEFAULT_SUBJECT',
presendScript: '$DEFAULT_PRESEND_SCRIPT',
postsendScript: '$DEFAULT_POSTSEND_SCRIPT',
recipientProviders: [
[$class: 'CulpritsRecipientProvider'],
[$class: 'DevelopersRecipientProvider'],
[$class: 'RequesterRecipientProvider'],
],
replyTo: '$DEFAULT_REPLYTO',
to: '$DEFAULT_RECIPIENTS',
body:'$DEFAULT_CONTENT'
}
*/
}
}

zorg/jenkins/jobs/jobs/clang-stage2-Rthinlto

  • This file was added.
pipeline {
agent { label 'green-dragon-22' }
options { disableResume() }
parameters {
string(name: 'GIT_SHA', defaultValue: '*/main', description: 'Git revision to build')
string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'description')
}
stages {
stage('Checkout') {
steps {
dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [
[name: params.GIT_SHA]
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-project.git']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
]])
}
dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [
[name: '*/main']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-zorg.git']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-zorg.git']
]])
}
}
}
stage('Build') {
steps {
timeout(600) {
sh '''
set -u
rm -rf build.properties
cd llvm-project
git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1}
cd -
echo "ARTIFACT=$JOB_NAME/clang-d$GIT_DISTANCE-g$GIT_SHA-t$BUILD_ID-b$BUILD_NUMBER.tar.gz" > build.properties
export PATH=$PATH:/usr/bin:/usr/local/bin
rm -rf clang-build clang-install *.tar.gz
python llvm-zorg/zorg/jenkins/monorepo_build.py fetch
python llvm-zorg/zorg/jenkins/monorepo_build.py clang build \
--thinlto --projects="clang;compiler-rt" \
--cmake-flag="-DCMAKE_DSYMUTIL=$WORKSPACE/host-compiler/bin/dsymutil"
'''
}
}
}
stage('Test') {
steps {
timeout(240) {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
rm -rf clang-build/testresults.xunit.xml
python llvm-zorg/zorg/jenkins/monorepo_build.py clang test
'''
}
junit 'clang-build/**/testresults.xunit.xml'
}
}
}
post {
always {
scanForIssues tool: clang()
}
regression {
emailext subject: '$DEFAULT_SUBJECT',
presendScript: '$DEFAULT_PRESEND_SCRIPT',
postsendScript: '$DEFAULT_POSTSEND_SCRIPT',
recipientProviders: [
[$class: 'CulpritsRecipientProvider'],
[$class: 'DevelopersRecipientProvider'],
[$class: 'RequesterRecipientProvider'],
],
replyTo: '$DEFAULT_REPLYTO',
to: '$DEFAULT_RECIPIENTS',
body:'$DEFAULT_CONTENT'
}
}
}

zorg/jenkins/jobs/jobs/lldb-cmake

  • This file was deleted.
#!/usr/bin/env groovy
pipeline {
agent { label 'green-dragon-10' }
parameters {
string(name: 'GIT_REVISION', defaultValue: '*/master', description: 'Git revision to build')
string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project')
string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
}
stages {
stage('Checkout') {
steps {
timeout(30) {
dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]])
}
dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [[name: '*/master']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
}
}
}
}
stage('Fetch Host Compiler') {
steps {
timeout(10) {
sh '''
python llvm-zorg/zorg/jenkins/monorepo_build.py fetch
'''
}
}
}
stage('Build') {
steps {
timeout(120) {
sh '''
set -u
rm -rf build.properties
cd llvm-project
git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1}
cd -
export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print $2}')
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
--assertions \
--projects="clang;libcxx;libcxxabi;compiler-rt;lld;lldb;debuginfo-tests" \
--compiler-flag="-Wdocumentation" \
--cmake-type=Release || (echo "Retrying. debuginfo-tests may have failed due to high load."; sleep 30; python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
--assertions \
--projects="clang;libcxx;libcxxabi;compiler-rt;lld;lldb;debuginfo-tests" \
--compiler-flag="-Wdocumentation" \
--cmake-type=Release)
'''
}
}
}
stage('Test') {
steps {
timeout(90) {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
export LLDB_CAPTURE_REPRODUCER=1
rm -rf test/results.xml
export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print $2}')
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake testlong
'''
}
}
}
}
post {
always {
scanForIssues tool: clang()
junit 'test/results.xml'
}
changed {
emailext subject: '$DEFAULT_SUBJECT',
presendScript: '$DEFAULT_PRESEND_SCRIPT',
postsendScript: '$DEFAULT_POSTSEND_SCRIPT',
recipientProviders: [
[$class: 'CulpritsRecipientProvider'],
[$class: 'DevelopersRecipientProvider'],
[$class: 'RequesterRecipientProvider'],
],
replyTo: '$DEFAULT_REPLYTO',
to: '$DEFAULT_RECIPIENTS',
body:'$DEFAULT_CONTENT'
}
}
}

zorg/jenkins/jobs/jobs/lldb-cmake-as

  • This file was added.
#!/usr/bin/env groovy
pipeline {
agent { label 'green-dragon-13' }
parameters {
string(name: 'GIT_SHA', defaultValue: '*/main', description: 'Git revision to build')
string(name: 'ARTIFACT', defaultValue: 'clang-stage2-Rthinlto/latest', description: 'Compiler artifact to use for building the project')
string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
}
stages {
stage('Checkout') {
steps {
timeout(30) {
dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [
[name: params.GIT_SHA]
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-project.git',
noTags: true, timeout: 30]
]])
}
dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [
[name: '*/main']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-zorg.git']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-zorg.git']
]])
}
}
}
}
stage('Build') {
steps {
timeout(120) {
sh '''
set -u
rm -rf build.properties
cd llvm-project
git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1}
cd -
if test -f /opt/codesign-workaround.sh; then sh /opt/codesign-workaround.sh init lldb-build/bin/debugserver; fi
export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print $2}')
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
--assertions \
--projects="clang;lld;lldb;cross-project-tests" \
--runtimes="libcxx;libcxxabi;compiler-rt" \
--compiler-flag="-Wdocumentation" \
--cmake-type=Release || (echo "Retrying. debuginfo-tests may have failed due to high load."; sleep 30; python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
--assertions \
--projects="clang;lld;lldb;cross-project-tests" \
--runtimes="libcxx;libcxxabi;compiler-rt" \
--compiler-flag="-Wdocumentation" \
--cmake-type=Release)
if test -f /opt/codesign-workaround.sh; then sh /opt/codesign-workaround.sh sign lldb-build/bin/debugserver; fi
'''
}
}
}
stage('Clean Test Results') {
steps {
timeout(10) {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
rm -rf test/results.xml
'''
}
}
}
stage('Test (Debug Info)') {
steps {
timeout(30) {
// We want to continue with the rest of the pipeline even
// if the "Debug Info" tests fail, so catch the error but
// label this stage and the whole pipeline as FAILURE.
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print $2}')
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-debuginfo test
'''
}
}
}
}
stage('Test (LLDB)') {
steps {
timeout(90) {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print $2}')
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake test
'''
}
}
}
stage('Upload artifacts (LLDB)') {
steps {
timeout(90) {
sh '''
set -u
cd llvm-project
git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1}
cd -
python3 llvm-zorg/zorg/jenkins/monorepo_build.py properties
'''
}
}
}
}
post {
always {
scanForIssues tool: clang()
junit 'test/results.xml'
}
changed {
emailext subject: '$DEFAULT_SUBJECT',
presendScript: '$DEFAULT_PRESEND_SCRIPT',
postsendScript: '$DEFAULT_POSTSEND_SCRIPT',
recipientProviders: [
[$class: 'CulpritsRecipientProvider'],
[$class: 'DevelopersRecipientProvider'],
[$class: 'RequesterRecipientProvider'],
],
replyTo: '$DEFAULT_REPLYTO',
to: '$DEFAULT_RECIPIENTS',
body:'$DEFAULT_CONTENT'
}
success {
build job: 'relay-lldb', wait: false
}
}
}

zorg/jenkins/jobs/jobs/lldb-cmake-intel

  • This file was added.
#!/usr/bin/env groovy
pipeline {
agent { label 'green-dragon-03' }
parameters {
string(name: 'GIT_SHA', defaultValue: '*/main', description: 'Git revision to build')
string(name: 'ARTIFACT', defaultValue: 'clang-stage2-Rthinlto/latest', description: 'Compiler artifact to use for building the project')
string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
}
stages {
stage('Checkout') {
steps {
timeout(30) {
dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [
[name: params.GIT_SHA]
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-project.git',
noTags: true, timeout: 30]
]])
}
dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [
[name: '*/main']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-zorg.git']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-zorg.git']
]])
}
}
}
}
stage('Fetch Host Compiler') {
steps {
timeout(10) {
sh '''
echo "Skipping geting the just-built compiler to work around an issue with libcxx."
echo python3 llvm-zorg/zorg/jenkins/monorepo_build.py fetch
rm -rf host-compiler
'''
}
}
}
stage('Build') {
steps {
timeout(120) {
sh '''
set -u
rm -rf build.properties
cd llvm-project
git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1}
cd -
if test -f /opt/codesign-workaround.sh; then sh /opt/codesign-workaround.sh init lldb-build/bin/debugserver; fi
export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print $2}')
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
--assertions \
--projects="clang;lld;lldb;cross-project-tests" \
--runtimes="libcxx;libcxxabi;compiler-rt" \
--compiler-flag="-Wdocumentation" \
--cmake-type=Release || (echo "Retrying. debuginfo-tests may have failed due to high load."; sleep 30; python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
--assertions \
--projects="clang;lld;lldb;cross-project-tests" \
--runtimes="libcxx;libcxxabi;compiler-rt" \
--compiler-flag="-Wdocumentation" \
--cmake-type=Release)
if test -f /opt/codesign-workaround.sh; then sh /opt/codesign-workaround.sh sign lldb-build/bin/debugserver; fi
'''
}
}
}
stage('Clean Test Results') {
steps {
timeout(10) {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
rm -rf test/results.xml
'''
}
}
}
stage('Test (Debug Info)') {
steps {
timeout(30) {
// We want to continue with the rest of the pipeline even
// if the "Debug Info" tests fail, so catch the error but
// label this stage and the whole pipeline as FAILURE.
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print $2}')
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-debuginfo test
'''
}
}
}
}
stage('Test (LLDB)') {
steps {
timeout(90) {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print $2}')
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake test
'''
}
}
}
stage('Upload artifacts (LLDB)') {
steps {
timeout(90) {
sh '''
set -u
cd llvm-project
git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1}
cd -
python3 llvm-zorg/zorg/jenkins/monorepo_build.py properties
'''
}
}
}
}
post {
always {
scanForIssues tool: clang()
junit 'test/results.xml'
}
changed {
emailext subject: '$DEFAULT_SUBJECT',
presendScript: '$DEFAULT_PRESEND_SCRIPT',
postsendScript: '$DEFAULT_POSTSEND_SCRIPT',
recipientProviders: [
[$class: 'CulpritsRecipientProvider'],
[$class: 'DevelopersRecipientProvider'],
[$class: 'RequesterRecipientProvider'],
],
replyTo: '$DEFAULT_REPLYTO',
to: '$DEFAULT_RECIPIENTS',
body:'$DEFAULT_CONTENT'
}
success {
build job: 'relay-lldb', wait: false
}
}
}

zorg/jenkins/jobs/jobs/lldb-cmake-matrix

#!/usr/bin/env groovy #!/usr/bin/env groovy
pipeline { pipeline {
agent { label 'green-dragon-23' } agent { label 'green-dragon-09' }
parameters { parameters {
string(name: 'GIT_REVISION', defaultValue: '*/master', description: 'Git revision to build') string(name: 'GIT_SHA', defaultValue: '*/main', description: 'Git revision to build')
string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project') string(name: 'ARTIFACT', defaultValue: 'clang-stage2-Rthinlto/latest', description: 'Compiler artifact to use for building the project')
string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...') string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building') string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
} }
environment {
MATRIX_COMMON_BUILD_PARAMETERS = '--cmake-flag=-DLLVM_TARGETS_TO_BUILD=X86'
}
stages { stages {
stage('Checkout') { stage('Checkout') {
steps { steps {
timeout(30) { timeout(30) {
dir('llvm-project') { dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]]) checkout([$class: 'GitSCM', branches: [
[name: params.GIT_SHA]
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-project.git',
noTags: true, timeout: 30]
]])
} }
dir('llvm-zorg') { dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [[name: '*/master']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]]) checkout([$class: 'GitSCM', branches: [
[name: '*/main']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-zorg.git']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-zorg.git']
]])
} }
} }
} }
} }
stage('Fetch') { stage('Fetch') {
steps { steps {
timeout(10) { timeout(10) {
sh ''' sh '''
python llvm-zorg/zorg/jenkins/monorepo_build.py fetch # The relay job sets ARTIFACT to the last good LLDB build.
# Here we use it to download the last good Clang.
export ARTIFACT=clang-stage2-Rthinlto/latest
python3 llvm-zorg/zorg/jenkins/monorepo_build.py fetch
echo "Skipping geting the just-built compiler to work around an issue with libcxx."
rm -rf host-compiler
''' '''
} }
} }
} }
stage('Build LLDB') { stage('Build LLDB') {
steps { steps {
timeout(90) { timeout(90) {
sh ''' sh '''
set -u set -u
rm -rf build.properties rm -rf build.properties
cd llvm-project cd llvm-project
git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
git_desc=$(git describe --match "first_commit") git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-") export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-") sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1} export GIT_SHA=${sha:1}
cd - cd -
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
if test -f /opt/codesign-workaround.sh; then sh /opt/codesign-workaround.sh init lldb-build/bin/debugserver; fi
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \ python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
$MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \ --assertions \
--projects="clang;libcxx;libcxxabi;lldb" \ --projects="clang;lldb" \
--runtimes="libcxx;libcxxabi" \
--cmake-type=Release \ --cmake-type=Release \
--dotest-flag="--skip-category gmodules" \ --dotest-flag="--skip-category gmodules" \
--dotest-flag="--arch=x86_64" \ --dotest-flag="--arch=x86_64"
--cmake-flag="-DLLVM_TARGETS_TO_BUILD=X86"
if test -f /opt/codesign-workaround.sh; then sh /opt/codesign-workaround.sh sign lldb-build/bin/debugserver; fi
''' '''
} }
} }
} }
stage('Test DWARF2') { stage('Test DWARF2') {
steps { steps {
timeout(60) { timeout(60) {
sh ''' sh '''
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \ python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
$MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \ --assertions \
--projects="clang;libcxx;libcxxabi;lldb" \ --lldb-test-compiler="$WORKSPACE/lldb-build/bin/clang" \
--projects="clang;lldb" \
--runtimes="libcxx;libcxxabi" \
--cmake-type=Release \ --cmake-type=Release \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="gmodules" \ --dotest-flag="gmodules" \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="watchpoint" \ --dotest-flag="watchpoint" \
--dotest-flag="--skip-category" \
--dotest-flag="llgs" \
--dotest-flag="--skip-category" \
--dotest-flag="debugserver" \
--dotest-flag="--arch=x86_64" \ --dotest-flag="--arch=x86_64" \
--dotest-flag="--dwarf-version=2" --dotest-flag="--dwarf-version=2"
set +e set +e
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test # Give the system some time to recover.
sleep 120
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test
set -e set -e
''' '''
} }
junit 'test/results.xml' junit 'test/results.xml'
} }
} }
stage('Test DWARF4') { stage('Test DWARF4') {
steps { steps {
timeout(60) { timeout(60) {
sh ''' sh '''
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \ python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
$MATRIX_COMMON_BUILD_PARAMETERS \
--lldb-test-compiler="$WORKSPACE/lldb-build/bin/clang" \
--assertions \ --assertions \
--projects="clang;libcxx;libcxxabi;lldb" \ --projects="clang;lldb" \
--runtimes="libcxx;libcxxabi" \
--cmake-type=Release \ --cmake-type=Release \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="gmodules" \ --dotest-flag="gmodules" \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="watchpoint" \ --dotest-flag="watchpoint" \
--dotest-flag="--skip-category" \
--dotest-flag="llgs" \
--dotest-flag="--skip-category" \
--dotest-flag="debugserver" \
--dotest-flag="--arch=x86_64" \ --dotest-flag="--arch=x86_64" \
--dotest-flag="--dwarf-version=4" --dotest-flag="--dwarf-version=4"
set +e set +e
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test # Give the system some time to recover.
sleep 120
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test
set -e set -e
''' '''
} }
junit 'test/results.xml' junit 'test/results.xml'
} }
} }
stage('Test DWARF5') { stage('Test DWARF5') {
steps { steps {
timeout(60) { timeout(60) {
sh ''' sh '''
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \ python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
$MATRIX_COMMON_BUILD_PARAMETERS \
--lldb-test-compiler="$WORKSPACE/lldb-build/bin/clang" \
--assertions \ --assertions \
--projects="clang;libcxx;libcxxabi;lldb" \ --projects="clang;lldb" \
--runtimes="libcxx;libcxxabi" \
--cmake-type=Release \ --cmake-type=Release \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="gmodules" \ --dotest-flag="gmodules" \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="watchpoint" \ --dotest-flag="watchpoint" \
--dotest-flag="--skip-category" \
--dotest-flag="llgs" \
--dotest-flag="--skip-category" \
--dotest-flag="debugserver" \
--dotest-flag="--arch=x86_64" \ --dotest-flag="--arch=x86_64" \
--dotest-flag="--dwarf-version=5" --dotest-flag="--dwarf-version=5"
set +e set +e
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test # Give the system some time to recover.
sleep 120
echo "****************************************************************"
echo "** DWARF 5 tests are temporarily being skipped because the many test failures result in extremely large log output."
echo "****************************************************************"
echo python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test
set -e set -e
''' '''
} }
junit 'test/results.xml'
} }
} }
stage('Build Clang 5.0.2') { stage('Build Clang 7.0.1') {
steps { steps {
timeout(30) { timeout(30) {
dir('clang_502') { dir('clang_701') {
checkout([$class: 'GitSCM', branches: [[name: "llvmorg-5.0.2"]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]]) checkout([$class: 'GitSCM', branches: [
[name: "llvmorg-7.0.1"]
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
], extensions: [
[$class: 'CloneOption', timeout: 30,
reference: '/Users/Shared/llvm-project.git']
]])
} }
} }
timeout(90) { timeout(90) {
sh ''' sh '''
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
export SRC_DIR='clang_502' export SRC_DIR='clang_701'
export BUILD_DIR='clang_502_build' export BUILD_DIR='clang_701_build'
python llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \ python3 llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \
$MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \ --assertions \
--noupload \ --noupload \
--noinstall \ --noinstall \
--projects="clang;libcxx;libcxxabi" --projects="clang;libcxx;libcxxabi"
''' '''
} }
} }
} }
stage('Test Clang 5.0.2') { stage('Test Clang 7.0.1') {
steps { steps {
timeout(60) { timeout(60) {
sh ''' sh '''
build_dir=$WORKSPACE/clang_701_build
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
export LLDB_TEST_COMPILER="$WORKSPACE/clang_502_build/bin/clang" python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \ $MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \ --assertions \
--projects="clang;libcxx;libcxxabi;lldb" \ --lldb-test-compiler="$build_dir/bin/clang" \
--projects="clang;lldb" \
--runtimes="libcxx;libcxxabi" \
--cmake-type=Release \ --cmake-type=Release \
--dotest-flag="--libcxx-include-dir=$build_dir/include/c++/v1/" \
--dotest-flag="--libcxx-library-dir=$build_dir/lib/" \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="gmodules" \ --dotest-flag="gmodules" \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="watchpoint" \ --dotest-flag="watchpoint" \
--dotest-flag="--skip-category" \
--dotest-flag="llgs" \
--dotest-flag="--skip-category" \
--dotest-flag="debugserver" \
--dotest-flag="--arch=x86_64" --dotest-flag="--arch=x86_64"
set +e set +e
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test # Give the system some time to recover.
sleep 120
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test
set -e set -e
''' '''
} }
junit 'test/results.xml' junit 'test/results.xml'
} }
} }
stage('Build Clang 7.0.1') { stage('Build Clang 9.0.0') {
steps { steps {
timeout(30) { timeout(30) {
dir('clang_701') { dir('clang_900') {
checkout([$class: 'GitSCM', branches: [[name: "llvmorg-7.0.1"]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]]) checkout([$class: 'GitSCM', branches: [
[name: "llvmorg-9.0.0"]
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
], extensions: [
[$class: 'CloneOption', timeout: 30,
reference: '/Users/Shared/llvm-project.git']
]])
} }
} }
timeout(90) { timeout(90) {
sh ''' sh '''
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
export SRC_DIR='clang_701' export SRC_DIR='clang_900'
export BUILD_DIR='clang_701_build' export BUILD_DIR='clang_900_build'
python llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \ python3 llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \
$MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \ --assertions \
--noupload \ --noupload \
--noinstall \ --noinstall \
--projects="clang;libcxx;libcxxabi" --projects="clang;libcxx;libcxxabi"
''' '''
} }
} }
} }
stage('Test Clang 7.0.1') { stage('Test Clang 9.0.0') {
steps { steps {
timeout(60) { timeout(60) {
sh ''' sh '''
build_dir=$WORKSPACE/clang_900_build
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
export LLDB_TEST_COMPILER="$WORKSPACE/clang_701_build/bin/clang" python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \ $MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \ --assertions \
--projects="clang;libcxx;libcxxabi;lldb" \ --lldb-test-compiler="$build_dir/bin/clang" \
--projects="clang;lldb" \
--runtimes="libcxx;libcxxabi" \
--cmake-type=Release \ --cmake-type=Release \
--dotest-flag="--libcxx-include-dir=$build_dir/include/c++/v1/" \
--dotest-flag="--libcxx-library-dir=$build_dir/lib/" \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="gmodules" \ --dotest-flag="gmodules" \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="watchpoint" \ --dotest-flag="watchpoint" \
--dotest-flag="--skip-category" \
--dotest-flag="llgs" \
--dotest-flag="--skip-category" \
--dotest-flag="debugserver" \
--dotest-flag="--arch=x86_64" --dotest-flag="--arch=x86_64"
set +e set +e
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test # Give the system some time to recover.
sleep 120
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test
set -e set -e
''' '''
} }
junit 'test/results.xml' junit 'test/results.xml'
} }
} }
stage('Build Clang 9.0.0') { stage('Build Clang 11.0.1') {
steps { steps {
timeout(30) { timeout(30) {
dir('clang_900') { dir('clang_1101') {
checkout([$class: 'GitSCM', branches: [[name: "llvmorg-9.0.0"]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]]) checkout([$class: 'GitSCM', branches: [
[name: "llvmorg-11.0.1"]
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
], extensions: [
[$class: 'CloneOption', timeout: 30,
reference: '/Users/Shared/llvm-project.git']
]])
} }
} }
timeout(90) { timeout(90) {
sh ''' sh '''
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
export SRC_DIR='clang_900' export SRC_DIR='clang_1101'
export BUILD_DIR='clang_900_build' export BUILD_DIR='clang_1101_build'
python llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \ python3 llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \
$MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \ --assertions \
--noupload \ --noupload \
--noinstall \ --noinstall \
--projects="clang;libcxx;libcxxabi" --projects="clang;libcxx;libcxxabi"
''' '''
} }
} }
} }
stage('Test Clang 9.0.0') { stage('Test Clang 11.0.1') {
steps {
timeout(60) {
sh '''
build_dir=$WORKSPACE/clang_1101_build
export PATH=$PATH:/usr/bin:/usr/local/bin
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
$MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \
--lldb-test-compiler="$build_dir/bin/clang" \
--projects="clang;lldb" \
--runtimes="libcxx;libcxxabi" \
--cmake-type=Release \
--dotest-flag="--libcxx-include-dir=$build_dir/include/c++/v1/" \
--dotest-flag="--libcxx-library-dir=$build_dir/lib/" \
--dotest-flag="--skip-category" \
--dotest-flag="gmodules" \
--dotest-flag="--skip-category" \
--dotest-flag="watchpoint" \
--dotest-flag="--skip-category" \
--dotest-flag="llgs" \
--dotest-flag="--skip-category" \
--dotest-flag="debugserver" \
--dotest-flag="--arch=x86_64"
set +e
# Give the system some time to recover.
sleep 120
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test
set -e
'''
}
junit 'test/results.xml'
}
}
stage('Build Clang 13.0.0') {
steps {
timeout(30) {
dir('clang_1300') {
checkout([$class: 'GitSCM', branches: [
[name: "llvmorg-13.0.0"]
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
], extensions: [
[$class: 'CloneOption', timeout: 30,
reference: '/Users/Shared/llvm-project.git']
]])
}
}
timeout(90) {
sh '''
export PATH=$PATH:/usr/bin:/usr/local/bin
export SRC_DIR='clang_1300'
export BUILD_DIR='clang_1300_build'
python3 llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \
$MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \
--noupload \
--noinstall \
--projects="clang;libcxx;libcxxabi"
'''
}
}
}
stage('Test Clang 13.0.0') {
steps {
timeout(60) {
sh '''
build_dir=$WORKSPACE/clang_1300_build
export PATH=$PATH:/usr/bin:/usr/local/bin
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
$MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \
--lldb-test-compiler="$build_dir/bin/clang" \
--projects="clang;lldb" \
--runtimes="libcxx;libcxxabi" \
--cmake-type=Release \
--dotest-flag="--libcxx-include-dir=$build_dir/include/c++/v1/" \
--dotest-flag="--libcxx-library-dir=$build_dir/lib/" \
--dotest-flag="--skip-category" \
--dotest-flag="gmodules" \
--dotest-flag="--skip-category" \
--dotest-flag="watchpoint" \
--dotest-flag="--skip-category" \
--dotest-flag="llgs" \
--dotest-flag="--skip-category" \
--dotest-flag="debugserver" \
--dotest-flag="--arch=x86_64"
set +e
# Give the system some time to recover.
sleep 120
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test
set -e
'''
}
junit 'test/results.xml'
}
}
stage('Build Clang 15.0.1') {
steps {
timeout(30) {
dir('clang_1501') {
checkout([$class: 'GitSCM', branches: [
[name: "llvmorg-15.0.1"]
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
], extensions: [
[$class: 'CloneOption', timeout: 30,
reference: '/Users/Shared/llvm-project.git']
]])
}
}
timeout(90) {
sh '''
export PATH=$PATH:/usr/bin:/usr/local/bin
export SRC_DIR='clang_1501'
export BUILD_DIR='clang_1501_build'
python3 llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \
$MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \
--noupload \
--noinstall \
--projects="clang;libcxx;libcxxabi"
'''
}
}
}
stage('Test Clang 15.0.1') {
steps { steps {
timeout(60) { timeout(60) {
sh ''' sh '''
build_dir=$WORKSPACE/clang_1501_build
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
export LLDB_TEST_COMPILER="$WORKSPACE/clang_900_build/bin/clang" python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \ $MATRIX_COMMON_BUILD_PARAMETERS \
--assertions \ --assertions \
--projects="clang;libcxx;libcxxabi;lldb" \ --lldb-test-compiler="$build_dir/bin/clang" \
--projects="clang;lldb" \
--runtimes="libcxx;libcxxabi" \
--cmake-type=Release \ --cmake-type=Release \
--dotest-flag="--libcxx-include-dir=$build_dir/include/c++/v1/" \
--dotest-flag="--libcxx-library-dir=$build_dir/lib/" \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="gmodules" \ --dotest-flag="gmodules" \
--dotest-flag="--skip-category" \ --dotest-flag="--skip-category" \
--dotest-flag="watchpoint" \ --dotest-flag="watchpoint" \
--dotest-flag="--skip-category" \
--dotest-flag="llgs" \
--dotest-flag="--skip-category" \
--dotest-flag="debugserver" \
--dotest-flag="--arch=x86_64" --dotest-flag="--arch=x86_64"
set +e set +e
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test # Give the system some time to recover.
sleep 120
python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix test
set -e set -e
''' '''
} }
junit 'test/results.xml' junit 'test/results.xml'
} }
} }
} }
} }

zorg/jenkins/jobs/jobs/lldb-cmake-reproducers

  • This file was deleted.
#!/usr/bin/env groovy
pipeline {
agent { label 'green-dragon-23' }
parameters {
string(name: 'GIT_REVISION', defaultValue: '*/master', description: 'Git revision to build')
string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project')
string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
}
stages {
stage('Checkout') {
steps {
timeout(30) {
dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]])
}
dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [[name: '*/master']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]])
}
}
}
}
stage('Fetch Host Compiler') {
steps {
timeout(10) {
sh '''
python llvm-zorg/zorg/jenkins/monorepo_build.py fetch
'''
}
}
}
stage('Build') {
steps {
timeout(90) {
sh '''
set -u
rm -rf build.properties
cd llvm-project
git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1}
cd -
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
--assertions \
--projects="clang;libcxx;libcxxabi;compiler-rt;lld;lldb;debuginfo-tests" \
--compiler-flag="-Wdocumentation" \
--cmake-type=Release
'''
}
}
}
stage('Test Reproducers') {
steps {
timeout(120) {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
rm -rf test/results.xml
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake \
--cmake-test-target check-lldb-reproducers
'''
}
}
}
}
post {
always {
scanForIssues tool: clang()
junit 'test/results.xml'
}
}
}

zorg/jenkins/jobs/jobs/lldb-cmake-sanitized

#!/usr/bin/env groovy #!/usr/bin/env groovy
pipeline { pipeline {
agent { label 'green-dragon-23' } agent { label 'green-dragon-09' }
parameters { parameters {
string(name: 'GIT_REVISION', defaultValue: '*/master', description: 'Git revision to build') string(name: 'GIT_SHA', defaultValue: '*/main', description: 'Git revision to build')
string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project') string(name: 'ARTIFACT', defaultValue: 'clang-stage2-Rthinlto/latest', description: 'Compiler artifact to use for building the project')
string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...') string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building') string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
} }
stages { stages {
stage('Checkout') { stage('Checkout') {
steps { steps {
timeout(30) { timeout(30) {
dir('llvm-project') { dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]]) checkout([$class: 'GitSCM', branches: [
[name: params.GIT_SHA]
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-project.git',
noTags: true, timeout: 30]
]])
} }
dir('llvm-zorg') { dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [[name: '*/master']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]]) checkout([$class: 'GitSCM', branches: [
[name: '*/main']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-zorg.git']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-zorg.git']
]])
} }
} }
} }
} }
stage('Fetch Host Compiler') { stage('Fetch Host Compiler') {
steps { steps {
timeout(10) { timeout(10) {
sh ''' sh '''
python llvm-zorg/zorg/jenkins/monorepo_build.py fetch # The relay job sets ARTIFACT to the last good LLDB build.
# Here we use it to download the last good Clang.
export ARTIFACT=clang-stage2-Rthinlto/latest
python3 llvm-zorg/zorg/jenkins/monorepo_build.py fetch
echo "Skipping geting the just-built compiler to work around an issue with libcxx."
rm -rf host-compiler
''' '''
} }
} }
} }
stage('Build') { stage('Build') {
steps { steps {
timeout(120) { timeout(240) {
sh ''' sh '''
set -u set -u
rm -rf build.properties rm -rf build.properties
cd llvm-project cd llvm-project
git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
git_desc=$(git describe --match "first_commit") git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-") export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-") sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1} export GIT_SHA=${sha:1}
cd - cd -
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
if test -f /opt/codesign-workaround.sh; then sh /opt/codesign-workaround.sh init lldb-build/bin/debugserver; fi
# Running too many asanified threads is too stressful for the kernel and we get >90% system time. # Running too many asanified threads is too stressful for the kernel and we get >90% system time.
export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print int(($2+1)/2)}') export MAX_PARALLEL_TESTS=$(sysctl hw.physicalcpu |awk '{print int(($2+1)/2)}')
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-sanitized build \ python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-sanitized build \
--assertions \ --assertions \
--projects="clang;libcxx;libcxxabi;lld;lldb" \ --projects="clang;lld;lldb" \
--runtimes="libcxx;libcxxabi" \
--cmake-type=Release --cmake-type=Release
if test -f /opt/codesign-workaround.sh; then sh /opt/codesign-workaround.sh sign lldb-build/bin/debugserver; fi
''' '''
} }
} }
} }
stage('Test') { stage('Test') {
steps { steps {
timeout(120) { timeout(240) {
sh ''' sh '''
set -u set -u
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
rm -rf test/results.xml rm -rf test/results.xml
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-sanitized test python3 llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-sanitized test
''' '''
} }
} }
} }
} }
post { post {
always { always {
scanForIssues tool: clang() scanForIssues tool: clang()
junit 'test/results.xml' junit 'test/results.xml'
} }
} }
} }

zorg/jenkins/jobs/jobs/lldb-cmake-standalone

#!/usr/bin/env groovy #!/usr/bin/env groovy
pipeline { pipeline {
agent {label 'green-dragon-23'} agent {label 'green-dragon-23'}
parameters { parameters {
string(name: 'GIT_REVISION', defaultValue: '*/master', description: 'Git revision to build') string(name: 'GIT_SHA', defaultValue: '*/main', description: 'Git revision to build')
string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project') string(name: 'ARTIFACT', defaultValue: 'clang-stage2-Rthinlto/latest', description: 'Compiler artifact to use for building the project')
string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...') string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building') string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
} }
stages { stages {
stage('Checkout') { stage('Checkout') {
steps { steps {
timeout(30) { timeout(30) {
dir('llvm-project') { dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [[name: params.GIT_REVISION]], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-project.git']]]) checkout([$class: 'GitSCM', branches: [
[name: params.GIT_SHA]
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-project.git',
noTags: true, timeout: 30]
]])
} }
dir('llvm-zorg') { dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [[name: '*/master']], userRemoteConfigs: [[url: 'http://labmaster3.local/git/llvm-zorg.git']]]) checkout([$class: 'GitSCM', branches: [
[name: '*/main']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-zorg.git']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-zorg.git']
]])
} }
} }
} }
} }
stage('Fetch') { stage('Fetch') {
steps { steps {
timeout(90) { timeout(90) {
sh ''' sh '''
python llvm-zorg/zorg/jenkins/monorepo_build.py fetch # The relay job sets ARTIFACT to the last good LLDB build.
# Here we use it to download the last good Clang.
export ARTIFACT=clang-stage2-Rthinlto/latest
python3 llvm-zorg/zorg/jenkins/monorepo_build.py fetch
echo "Skipping geting the just-built compiler to work around an issue with libcxx."
rm -rf host-compiler
''' '''
} }
} }
} }
stage('Clang Build') { stage('Clang Build') {
steps { steps {
timeout(90) { timeout(90) {
sh ''' sh '''
Show All 9 Lines stages {
sha=$(echo ${git_desc} | cut -f 3 -d "-") sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1} export GIT_SHA=${sha:1}
cd - cd -
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
python llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \ python3 llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \
--assertions \ --assertions \
--noupload \ --noupload \
--projects="clang;libcxx;libcxxabi" \ --projects="clang" \
--runtimes="libcxx;libcxxabi" \
--cmake-flag="-DLLVM_INSTALL_UTILS=On" \ --cmake-flag="-DLLVM_INSTALL_UTILS=On" \
--cmake-flag="-DLLVM_INSTALL_TOOLCHAIN_ONLY=Off" \ --cmake-flag="-DLLVM_INSTALL_TOOLCHAIN_ONLY=Off" \
--cmake-flag="-DLLVM_TARGETS_TO_BUILD=X86" --cmake-flag="-DLLVM_TARGETS_TO_BUILD=X86"
''' '''
} }
} }
} }
stage('Ninja Standalone Build (build tree)') { stage('Ninja Standalone Build (build tree)') {
Show All 13 Lines stages {
sha=$(echo ${git_desc} | cut -f 3 -d "-") sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1} export GIT_SHA=${sha:1}
cd - cd -
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
export LLDB_STANDALONE_TYPE="build-tree" export LLDB_STANDALONE_TYPE="build-tree"
python $WORKSPACE/llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-standalone build \ python3 $WORKSPACE/llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-standalone build \
--assertions \ --assertions \
--cmake-type=Release --cmake-type=Release
''' '''
} }
} }
} }
stage('Ninja Standalone Build (install tree)') { stage('Ninja Standalone Build (install tree)') {
steps { steps {
Show All 12 Lines stages {
sha=$(echo ${git_desc} | cut -f 3 -d "-") sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1} export GIT_SHA=${sha:1}
cd - cd -
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
export LLDB_STANDALONE_TYPE="install-tree" export LLDB_STANDALONE_TYPE="install-tree"
python $WORKSPACE/llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-standalone build \ python3 $WORKSPACE/llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-standalone build \
--assertions \ --assertions \
--cmake-type=Release --cmake-type=Release
''' '''
} }
} }
} }
stage('Xcode Standalone Build (build tree)') { stage('Xcode Standalone Build (build tree)') {
steps { steps {
Show All 11 Lines stages {
sha=$(echo ${git_desc} | cut -f 3 -d "-") sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1} export GIT_SHA=${sha:1}
cd - cd -
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
python $WORKSPACE/llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-xcode build \ python3 $WORKSPACE/llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-xcode build \
--assertions \ --assertions \
--cmake-type=Release --cmake-type=Release
''' '''
} }
} }
} }
stage('Xcode Standalone Test (build tree)') { stage('Xcode Standalone Test (build tree)') {
steps { steps {
timeout(30) { timeout(30) {
sh ''' sh '''
set -u set -u
export PATH=$PATH:/usr/bin:/usr/local/bin export PATH=$PATH:/usr/bin:/usr/local/bin
python $WORKSPACE/llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-xcode test \ echo "TEMPORARILY DISABLED (runs with too many jobs in parallel)" python3 $WORKSPACE/llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-xcode test \
--assertions \ --assertions \
--cmake-type=Release --cmake-type=Release
''' '''
} }
} }
} }
} }
post { post {
Show All 15 Lines

zorg/jenkins/jobs/jobs/llvm-coverage

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/usr/bin/env groovy
pipeline {
agent { label 'green-dragon-23' }
parameters {
string(name: 'GIT_REVISION', defaultValue: '*/main', description: 'Git revision to build')
string(name: 'ARTIFACT', defaultValue: 'clang-stage1-RA/latest', description: 'Compiler artifact to use for building the project')
string(name: 'BUILD_TYPE', defaultValue: 'Release', description: 'Default CMake build type; one of: Release, Debug, ...')
string(name: 'CLEAN', defaultValue: "false", description: 'Whether or not to clean the build directory before building')
}
stages {
stage('Checkout') {
steps {
dir('llvm-project') {
checkout([$class: 'GitSCM', branches: [
[name: params.GIT_REVISION]
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-project.git']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-project.git']
]])
}
dir('llvm-zorg') {
checkout([$class: 'GitSCM', branches: [
[name: '*/main']
], extensions: [
[$class: 'CloneOption',
reference: '/Users/Shared/llvm-zorg.git']
], userRemoteConfigs: [
[url: 'https://github.com/llvm/llvm-zorg.git']
]])
}
}
}
stage('Fetch') {
steps {
timeout(10) {
sh '''
python llvm-zorg/zorg/jenkins/monorepo_build.py fetch
'''
}
}
}
stage('Build') {
steps {
timeout(180) {
sh '''
set -u
rm -rf build.properties
cd llvm-project
git tag -a -m "First Commit" first_commit 97724f18c79c7cc81ced24239eb5e883bf1398ef || true
git_desc=$(git describe --match "first_commit")
export GIT_DISTANCE=$(echo ${git_desc} | cut -f 2 -d "-")
sha=$(echo ${git_desc} | cut -f 3 -d "-")
export GIT_SHA=${sha:1}
cd -
export PATH=$PATH:/usr/bin:/usr/local/bin
python llvm-zorg/zorg/jenkins/monorepo_build.py cmake build \
--cmake-build-target FileCheck \
--cmake-build-target count \
--cmake-build-target not \
--cmake-flag="-DLLVM_TARGETS_TO_BUILD=host" \
--cmake-type=Release \
--projects "" \
--noinstall \
--noupload
python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake build \
--assertions \
--projects="clang;lldb;cross-project-tests" \
--runtimes="libcxx;libcxxabi" \
--cmake-flag="-DPYTHON_LIBRARY=/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7.dylib" \
--cmake-flag="-DPYTHON_INCLUDE_DIR=/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m" \
--cmake-flag="-DPYTHON_LIBRARY_DEBUG=/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7.dylib" \
--cmake-flag="-DPYTHON_EXECUTABLE=/usr/local/Cellar/python/3.7.0/bin/python3.7" \
--cmake-flag="-DLLDB_USE_SYSTEM_DEBUGSERVER=ON" \
--cmake-flag="-DLLVM_BUILD_INSTRUMENTED_COVERAGE=ON" \
--cmake-flag="-DLLVM_PROFILE_MERGE_POOL_SIZE=1" \
--cmake-flag="-DLLVM_ENABLE_MODULES=Off" \
--cmake-flag="-DLLVM_TARGETS_TO_BUILD=X86;ARM;AArch64;AMDGPU" \
--cmake-type=Release \
--noupload
'''
}
}
}
stage('Test') {
steps {
timeout(480) {
sh '''
set -u
export PATH=$PATH:/usr/bin:/usr/local/bin
rm -rf test/results.xml
BUILD_DIR=$(pwd)/lldb-build
FAST_BUILD_DIR=$(pwd)/clang-build
REPORT_DIR=$(pwd)/coverage-reports
HOST=$(pwd)/host-compiler/bin
LLVM_PROFDATA=$HOST/llvm-profdata
LLVM_COV=$HOST/llvm-cov
ARTIFACT_PREP_SCRIPT=$WORKSPACE/llvm-project/llvm/utils/prepare-code-coverage-artifact.py
FAST_TOOLS=(FileCheck count not)
for TOOL in ${FAST_TOOLS[@]}; do
cp $FAST_BUILD_DIR/bin/$TOOL $BUILD_DIR/bin/$TOOL
done
# Clear out any stale profiles.
rm -rf $BUILD_DIR/profiles
# Run the tests.
IGNORE_ERRORS_OVERRIDE=1 python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake testlong || echo "Some tests may have failed."
cd $BUILD_DIR
ninja -k 0 check-llvm check-clang || echo "Some tests may have failed."
cd -
# Temporarily exclude debugserver from the reports.
# It is a universal binary and the coverage report script
# doesn't yet know how to handle it.
rm -rf $BUILD_DIR/bin/debugserver
COV_BINARIES=$(find $BUILD_DIR/bin $BUILD_DIR/lib -depth 1 -type f -exec file {} \\; | grep Mach-O | cut -d':' -f1 | grep -vE '/(FileCheck|count|not)$' | xargs)
rm -rf $REPORT_DIR
mkdir -p $REPORT_DIR
python $ARTIFACT_PREP_SCRIPT $LLVM_PROFDATA $LLVM_COV $BUILD_DIR/profiles $REPORT_DIR $COV_BINARIES --unified-report --restrict $WORKSPACE/llvm-project
scp -r $REPORT_DIR buildslave@labmaster2.local:/Library/WebServer/Documents/coverage
ssh buildslave@labmaster2.local "chmod -R 777 /Library/WebServer/Documents/coverage"
rm -rf lldb-build/profiles
'''
}
}
}
}
post {
always {
scanForIssues tool: clang()
junit 'test/results.xml'
}
}
}

zorg/jenkins/jobs/jobs/lnt-ctmark-aarch64-O0-g

#!/usr/bin/env groovy #!/usr/bin/env groovy
def common = evaluate readTrusted('zorg/jenkins/common.groovy') def common = evaluate readTrusted('zorg/jenkins/common.groovy')
common.testsuite_pipeline(label: 'green-dragon-12') { common.testsuite_pipeline(label: 'green-dragon-11') {
timeout(30) {
sh """ sh """
LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake" LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake"
LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-O0-g.cmake" LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-O0-g.cmake"
config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}" config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}"
""" """
} }
}

zorg/jenkins/jobs/jobs/lnt-ctmark-aarch64-O3-flto

#!/usr/bin/env groovy #!/usr/bin/env groovy
def common = evaluate readTrusted('zorg/jenkins/common.groovy') def common = evaluate readTrusted('zorg/jenkins/common.groovy')
common.testsuite_pipeline(label: 'green-dragon-12') { common.testsuite_pipeline(label: 'green-dragon-11') {
timeout(30) {
sh """ sh """
LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake" LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake"
LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-O3-flto.cmake" LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-O3-flto.cmake"
config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}" config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}"
""" """
} }
}

zorg/jenkins/jobs/jobs/lnt-ctmark-aarch64-Os

#!/usr/bin/env groovy #!/usr/bin/env groovy
def common = evaluate readTrusted('zorg/jenkins/common.groovy') def common = evaluate readTrusted('zorg/jenkins/common.groovy')
common.testsuite_pipeline(label: 'green-dragon-12') { common.testsuite_pipeline(label: 'green-dragon-11') {
timeout(30) {
sh """ sh """
LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake" LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake"
LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-Os.cmake" LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-Os.cmake"
config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}" config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}"
""" """
} }
}

zorg/jenkins/jobs/jobs/lnt-ctmark-aarch64-Oz

#!/usr/bin/env groovy #!/usr/bin/env groovy
def common = evaluate readTrusted('zorg/jenkins/common.groovy') def common = evaluate readTrusted('zorg/jenkins/common.groovy')
common.testsuite_pipeline(label: 'green-dragon-12') { common.testsuite_pipeline(label: 'green-dragon-11') {
timeout(30) {
sh """ sh """
LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake" LNT_FLAGS+=" -C config/tasks/cmake/caches/target-arm64-iphoneos.cmake"
LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-Oz.cmake" LNT_FLAGS+=" -C config/tasks/cmake/caches/opt-Oz.cmake"
config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}" config/tasks/task jenkinsrun config/tasks/lnt-ctmark.sh -a compiler="${params.ARTIFACT}" -D LNT_FLAGS="\${LNT_FLAGS}"
""" """
} }
}

zorg/jenkins/jobs/jobs/relay-lldb

  • This file was added.
#!/usr/bin/env groovy
def relay = evaluate readTrusted('zorg/jenkins/relay.groovy')
relay.lldb_pipeline "lldb-cmake-(.+)"

zorg/jenkins/jobs/jobs/test-suite-verify-machineinstrs-aarch64-O0-g

#!/usr/bin/env groovy #!/usr/bin/env groovy
def common = evaluate readTrusted('zorg/jenkins/common.groovy') def common = evaluate readTrusted('zorg/jenkins/common.groovy')
common.testsuite_pipeline(label: 'green-dragon-12') { common.testsuite_pipeline(label: 'green-dragon-11') {
sh """ sh """
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-arm64-iphoneos.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-arm64-iphoneos.cmake"
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O0-g.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O0-g.cmake"
config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}" config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}"
""" """
} }

zorg/jenkins/jobs/jobs/test-suite-verify-machineinstrs-aarch64-O3

#!/usr/bin/env groovy #!/usr/bin/env groovy
def common = evaluate readTrusted('zorg/jenkins/common.groovy') def common = evaluate readTrusted('zorg/jenkins/common.groovy')
common.testsuite_pipeline(label: 'green-dragon-12') { common.testsuite_pipeline(label: 'green-dragon-11') {
sh """ sh """
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-arm64-iphoneos.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-arm64-iphoneos.cmake"
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O3.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O3.cmake"
config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}" config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}"
""" """
} }

zorg/jenkins/jobs/jobs/test-suite-verify-machineinstrs-aarch64-globalisel-O0-g

#!/usr/bin/env groovy #!/usr/bin/env groovy
def common = evaluate readTrusted('zorg/jenkins/common.groovy') def common = evaluate readTrusted('zorg/jenkins/common.groovy')
common.testsuite_pipeline(label: 'green-dragon-12') { common.testsuite_pipeline(label: 'green-dragon-11') {
sh """ sh """
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-arm64-iphoneos.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-arm64-iphoneos.cmake"
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O0-g.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O0-g.cmake"
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/global-isel.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/global-isel.cmake"
config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}" config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}"
""" """
} }

zorg/jenkins/jobs/jobs/test-suite-verify-machineinstrs-x86_64-O0-g

#!/usr/bin/env groovy #!/usr/bin/env groovy
def common = evaluate readTrusted('zorg/jenkins/common.groovy') def common = evaluate readTrusted('zorg/jenkins/common.groovy')
common.testsuite_pipeline(label: 'green-dragon-12') { common.testsuite_pipeline(label: 'green-dragon-11') {
sh """ sh """
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-x86_64-macos.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-x86_64-macos.cmake"
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O0-g.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O0-g.cmake"
config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}" config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}"
""" """
} }

zorg/jenkins/jobs/jobs/test-suite-verify-machineinstrs-x86_64-O3

#!/usr/bin/env groovy #!/usr/bin/env groovy
def common = evaluate readTrusted('zorg/jenkins/common.groovy') def common = evaluate readTrusted('zorg/jenkins/common.groovy')
common.testsuite_pipeline(label: 'green-dragon-12') { common.testsuite_pipeline(label: 'green-dragon-11') {
sh """ sh """
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-x86_64-macos.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-x86_64-macos.cmake"
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O3.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O3.cmake"
config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}" config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}"
""" """
} }

zorg/jenkins/jobs/jobs/test-suite-verify-machineinstrs-x86_64h-O3

#!/usr/bin/env groovy #!/usr/bin/env groovy
def common = evaluate readTrusted('zorg/jenkins/common.groovy') def common = evaluate readTrusted('zorg/jenkins/common.groovy')
common.testsuite_pipeline(label: 'green-dragon-12') { common.testsuite_pipeline(label: 'green-dragon-11') {
sh """ sh """
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-x86_64h-macos.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/target-x86_64h-macos.cmake"
CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O3.cmake" CMAKE_FLAGS+=" -C ../config/tasks/cmake/caches/opt-O3.cmake"
config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}" config/tasks/task jenkinsrun config/tasks/test-suite-verify-machineinstrs.sh -a compiler="${params.ARTIFACT}" -D CMAKE_FLAGS="\${CMAKE_FLAGS}"
""" """
} }

zorg/jenkins/jobs/util/make_pipeline.py

Show All 26 Lines <scm class="hudson.plugins.git.GitSCM" plugin="git@3.6.1">
<configVersion>2</configVersion> <configVersion>2</configVersion>
<userRemoteConfigs> <userRemoteConfigs>
<hudson.plugins.git.UserRemoteConfig> <hudson.plugins.git.UserRemoteConfig>
<url>{pipeline_git_url}</url> <url>{pipeline_git_url}</url>
</hudson.plugins.git.UserRemoteConfig> </hudson.plugins.git.UserRemoteConfig>
</userRemoteConfigs> </userRemoteConfigs>
<branches> <branches>
<hudson.plugins.git.BranchSpec> <hudson.plugins.git.BranchSpec>
<name>*/master</name> <name>*/main</name>
</hudson.plugins.git.BranchSpec> </hudson.plugins.git.BranchSpec>
</branches> </branches>
<doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations> <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
<submoduleCfg class="list"/> <submoduleCfg class="list"/>
<extensions/> <extensions/>
</scm> </scm>
<scriptPath>{pipeline_git_path}</scriptPath> <scriptPath>{pipeline_git_path}</scriptPath>
<lightweight>true</lightweight> <lightweight>true</lightweight>
Show All 29 Lines

zorg/jenkins/monorepo_build.py

"""Build and test clangs.""" """Build and test clangs."""
from __future__ import print_function from __future__ import print_function
import sys import sys
import logging import logging
import os import os
import subprocess import subprocess
import datetime import datetime
import time import time
import argparse import argparse
import shutil import shutil
import math import math
import re import re
import requests
import xml.etree.ElementTree as ET import xml.etree.ElementTree as ET
from contextlib import contextmanager from contextlib import contextmanager
from six.moves import urllib
SERVER = "labmaster2.lab.llvm.org" SERVER = "green-dragon-21.local"
NINJA = "/usr/local/bin/ninja" NINJA = "/usr/local/bin/ninja"
MODULE_CACHE_REGEX = re.compile(r'-fmodules-cache-path=([^\"]+)') MODULE_CACHE_REGEX = re.compile(r'-fmodules-cache-path=([^\"]+)')
# Add dependency checker to the Python path. # Add dependency checker to the Python path.
# For relative reference to the dependency file. # For relative reference to the dependency file.
here = os.path.dirname(os.path.abspath(__file__)) here = os.path.dirname(os.path.abspath(__file__))
Show All 12 Linesdef next_section(name):
"""Jenkins is setup to parse @@@ xyz @@@ as a new section of the buildlog """Jenkins is setup to parse @@@ xyz @@@ as a new section of the buildlog
with title xyz. The section ends with @@@@@@ """ with title xyz. The section ends with @@@@@@ """
footer() footer()
header(name) header(name)
def header(name): def header(name):
print("@@@", name, "@@@") print("@@@", name, "@@@")
print("Starting at: " + time.strftime("%FT%T"))
def footer(): def footer():
print("Completed at: " + time.strftime("%FT%T")) print("Completed at: " + time.strftime("%FT%T"))
print("@@@@@@") print("@@@@@@")
def quote_sh_string(string): def quote_sh_string(string):
▲ Show 20 Lines • Show All 52 Lines • ▼ Show 20 Lines def __init__(self, args):
self._llvm_src_dir = os.environ.get('LLVM_SRC_DIR', 'llvm') self._llvm_src_dir = os.environ.get('LLVM_SRC_DIR', 'llvm')
self._lldb_src_dir = os.environ.get('LLDB_SRC_DIR', 'lldb') self._lldb_src_dir = os.environ.get('LLDB_SRC_DIR', 'lldb')
self._build_dir = os.environ.get('BUILD_DIR', 'clang-build') self._build_dir = os.environ.get('BUILD_DIR', 'clang-build')
self._lldb_build_dir = os.environ.get('LLDB_BUILD_DIR', 'lldb-build') self._lldb_build_dir = os.environ.get('LLDB_BUILD_DIR', 'lldb-build')
self._lldb_standalone_build_dir = os.environ.get('LLDB_STANDALONE_BUILD_DIR', 'lldb-standalone-build') self._lldb_standalone_build_dir = os.environ.get('LLDB_STANDALONE_BUILD_DIR', 'lldb-standalone-build')
self._lldb_standalone_type = os.environ.get('LLDB_STANDALONE_TYPE', 'build-tree') self._lldb_standalone_type = os.environ.get('LLDB_STANDALONE_TYPE', 'build-tree')
self._lldb_xcode_build_dir = os.environ.get('LLDB_XCODE_BUILD_DIR', 'lldb-xcode-build') self._lldb_xcode_build_dir = os.environ.get('LLDB_XCODE_BUILD_DIR', 'lldb-xcode-build')
self._lldb_install_dir = os.environ.get('LLDB_INSTALL_DIR', 'lldb-install') self._lldb_install_dir = os.environ.get('LLDB_INSTALL_DIR', 'lldb-install')
self._lldb_test_compiler = os.environ.get('LLDB_TEST_COMPILER', '')
self._install_dir = os.environ.get('INSTALL_DIR', 'clang-install') self._install_dir = os.environ.get('INSTALL_DIR', 'clang-install')
self.j_level = os.environ.get('J_LEVEL', None) self.j_level = os.environ.get('J_LEVEL', None)
self.max_parallel_tests = os.environ.get('MAX_PARALLEL_TESTS', None) self.max_parallel_tests = os.environ.get('MAX_PARALLEL_TESTS', None)
self.max_parallel_links = os.environ.get('MAX_PARALLEL_LINKS', None) self.max_parallel_links = os.environ.get('MAX_PARALLEL_LINKS', None)
self.host_compiler_url = os.environ.get('HOST_URL', self.host_compiler_url = os.environ.get('HOST_URL',
'http://labmaster2.local/artifacts/') 'http://green-dragon-21.local/artifacts/')
self.artifact_url = os.environ.get('ARTIFACT', 'NONE') self.artifact_url = os.environ.get('ARTIFACT', 'NONE')
self.job_name = os.environ.get('JOB_NAME', 'NONE') self.job_name = os.environ.get('JOB_NAME', 'NONE')
self.build_id = os.environ.get('BUILD_ID', 'NONE') self.build_id = os.environ.get('BUILD_ID', 'NONE')
self.build_number = os.environ.get('BUILD_NUMBER', 'NONE') self.build_number = os.environ.get('BUILD_NUMBER', 'NONE')
self.svn_rev = os.environ.get('LLVM_REV', 'NONE') self.svn_rev = os.environ.get('LLVM_REV', 'NONE')
self.git_sha = os.environ.get('GIT_SHA', 'NONE') self.git_sha = os.environ.get('GIT_SHA', 'NONE')
self.git_distance = os.environ.get('GIT_DISTANCE', 'NONE') self.git_distance = os.environ.get('GIT_DISTANCE', 'NONE')
self.nobootstrap = True self.nobootstrap = True
self.device = None self.device = None
self.node_name = os.environ.get('NODE_NAME', None) self.node_name = os.environ.get('NODE_NAME', None)
self.lldb_test_archs = os.environ.get('LLDB_TEST_ARCHS', 'x86_64').split()
# Import all of the command line arguments into the config object # Import all of the command line arguments into the config object
self.__dict__.update(vars(args)) self.__dict__.update(vars(args))
def builddir(self): def builddir(self):
"""The build output directory for this compile.""" """The build output directory for this compile."""
return os.path.join(self.workspace, self._build_dir) return os.path.join(self.workspace, self._build_dir)
Show All 20 Linesclass Configuration(object):
def lldbsrcdir(self): def lldbsrcdir(self):
"""The derived source directory for this lldb build.""" """The derived source directory for this lldb build."""
return os.path.join(self.workspace, self._src_dir, self._lldb_src_dir) return os.path.join(self.workspace, self._src_dir, self._lldb_src_dir)
def lldbinstalldir(self): def lldbinstalldir(self):
"""The install directory for the lldb build.""" """The install directory for the lldb build."""
return os.path.join(self.workspace, self._lldb_install_dir) return os.path.join(self.workspace, self._lldb_install_dir)
def lldbtestcompiler(self):
"""The compiler used to build LLDB tests."""
return self._lldb_test_compiler
def installdir(self): def installdir(self):
"""The install directory for the compile.""" """The install directory for the compile."""
return os.path.join(self.workspace, self._install_dir) return os.path.join(self.workspace, self._install_dir)
def lldbstandalonetype(self): def lldbstandalonetype(self):
"""The type of standalone build: against the build or install tree.""" """The type of standalone build: against the build or install tree."""
return self._lldb_standalone_type; return self._lldb_standalone_type;
Show All 16 Lines def liblto(self):
assert os.path.exists(clang_liblto_path), "host-compiler present," \ assert os.path.exists(clang_liblto_path), "host-compiler present," \
" but has no liblto." " but has no liblto."
return clang_liblto_path return clang_liblto_path
else: else:
return False return False
def branch(self): def branch(self):
"""Figure out the source branch name. """Figure out the source branch name.
Not using GIT_BRANCH env var from Jenkins as that includes the Not using GIT_BRANCH env var from Jenkins as that includes the
remote name too. remote name too.
""" """
if not os.environ.get('TESTING', False): if not os.environ.get('TESTING', False):
try: try:
cmd = ['git', '-C', conf.srcdir(), 'symbolic-ref', '--short', 'HEAD'] cmd = ['git', '-C', conf.srcdir(), 'symbolic-ref', '--short', 'HEAD']
out = run_collect_output(cmd, stderr=subprocess.STDOUT).strip() out = run_collect_output(cmd, stderr=subprocess.STDOUT).strip()
except subprocess.CalledProcessError: except subprocess.CalledProcessError:
cmd = ['git', '-C', conf.srcdir(), 'rev-parse', 'HEAD'] cmd = ['git', '-C', conf.srcdir(), 'rev-parse', 'HEAD']
out = run_collect_output(cmd, stderr=subprocess.STDOUT).strip() out = run_collect_output(cmd, stderr=subprocess.STDOUT).strip()
return out return out
return 'master' return 'main'
def link_memory_usage(self): def link_memory_usage(self):
"""Guesstimate the maximum link memory usage for this build. """Guesstimate the maximum link memory usage for this build.
We are only building master here so we will just use that value We are only building main here so we will just use that value
""" """
# Determinited experimentally. # Determinited experimentally.
usages = {'master': 3.5} usages = {'main': 3.5}
if self.branch() == 'master': if self.branch() == 'main':
return usages['master'] return usages['main']
else: else:
raise NotImplementedError( raise NotImplementedError(
"Unknown link memory usage." + self.branch()) "Unknown link memory usage." + self.branch())
# Global storage for configuration object. # Global storage for configuration object.
conf = None # type: Configuration conf = None # type: Configuration
def cmake_builder(target): def cmake_builder(target):
if not os.getenv("TESTING"): if not os.getenv("TESTING"):
dep.parse_dependencies([here + "/clang_build_dependencies.dep"]) dep.parse_dependencies([here + "/clang_build_dependencies.dep"])
env = [] env = []
dyld_path = "" dyld_path = ""
if conf.lto and conf.liblto(): if conf.lto and conf.liblto():
dyld_path = conf.liblto() dyld_path = conf.liblto()
env.extend(["env", "DYLD_LIBRARY_PATH=" + dyld_path]) env.extend(["env", "DYLD_LIBRARY_PATH=" + dyld_path])
cmake_cmd = env + ["/usr/local/bin/cmake", "-G", "Ninja", cmake_cmd = env + ["/usr/local/bin/cmake", "-G", "Ninja",
'-DCMAKE_MAKE_PROGRAM=' + NINJA, '-DCMAKE_MAKE_PROGRAM=' + NINJA,
"-DCMAKE_INSTALL_PREFIX=" + conf.installdir(), "-DCMAKE_INSTALL_PREFIX=" + conf.installdir(),
"-DLLVM_ENABLE_PROJECTS=" + conf.llvm_enable_projects, "-DLLVM_ENABLE_PROJECTS=" + conf.llvm_enable_projects,
conf.llvmsrcdir()] conf.llvmsrcdir()]
if conf.llvm_enable_runtimes:
cmake_cmd += ["-DLLVM_ENABLE_RUNTIMES=" + conf.llvm_enable_runtimes]
compiler_flags = conf.compiler_flags compiler_flags = conf.compiler_flags
max_parallel_links = conf.max_parallel_links max_parallel_links = conf.max_parallel_links
if conf.lto: if conf.lto:
if conf.thinlto: if conf.thinlto:
cmake_cmd += ["-DLLVM_PARALLEL_LINK_JOBS=1"] cmake_cmd += ["-DLLVM_PARALLEL_LINK_JOBS=1"]
else: else:
cmake_cmd += ["-DLLVM_PARALLEL_LINK_JOBS=" + str(max_link_jobs())] cmake_cmd += ["-DLLVM_PARALLEL_LINK_JOBS=" + str(max_link_jobs())]
▲ Show 20 Lines • Show All 87 Lines • ▼ Show 20 Lines if target == 'all' or target == 'test' or target == 'testlong':
targets = [ targets = [
'check-all'] if target == 'testlong' or target == 'all' else conf.cmake_test_targets 'check-all'] if target == 'testlong' or target == 'all' else conf.cmake_test_targets
if not targets: if not targets:
# testlong and all do check all, otherwise check and check-clang # testlong and all do check all, otherwise check and check-clang
# unless the user asked for something else. # unless the user asked for something else.
targets = ['check', 'check-clang'] targets = ['check', 'check-clang']
run_cmd(conf.builddir(), ninja_cmd + targets) run_cmd(conf.builddir(), ninja_cmd + targets, err_okay=True)
footer() footer()
if conf.sccache: if conf.sccache:
run_ws([conf.sccache_path, "--stop-server"]) run_ws([conf.sccache_path, "--stop-server"])
def clang_builder(target): def clang_builder(target):
"""Build to set of commands to compile and test apple-clang""" """Build to set of commands to compile and test apple-clang"""
▲ Show 20 Lines • Show All 61 Lines • ▼ Show 20 Lines if target == "all" or target == "build":
install_prefix), install_prefix),
'-DCLANG_REPOSITORY_STRING={}'.format( '-DCLANG_REPOSITORY_STRING={}'.format(
conf.branch()), conf.branch()),
'-DCLANG_APPEND_VC_REV=On', '-DCLANG_APPEND_VC_REV=On',
'-DLLVM_BUILD_TESTS=On', '-DLLVM_BUILD_TESTS=On',
'-DLLVM_INCLUDE_TESTS=On', '-DLLVM_INCLUDE_TESTS=On',
'-DCLANG_INCLUDE_TESTS=On', '-DCLANG_INCLUDE_TESTS=On',
'-DLLVM_INCLUDE_UTILS=On', '-DLLVM_INCLUDE_UTILS=On',
'-DLIBCXX_INSTALL_HEADERS=On',
'-DLIBCXX_OVERRIDE_DARWIN_INSTALL=On',
'-DLIBCXX_INSTALL_LIBRARY=Off',
'-DCMAKE_MACOSX_RPATH=On', '-DCMAKE_MACOSX_RPATH=On',
] ]
if conf.llvm_enable_runtimes:
cmake_command.append('-DLLVM_ENABLE_RUNTIMES={}'.format(conf.llvm_enable_runtimes))
if conf.sccache: if conf.sccache:
cmake_command.append('-DCMAKE_C_COMPILER_LAUNCHER=' + conf.sccache_path) cmake_command.append('-DCMAKE_C_COMPILER_LAUNCHER=' + conf.sccache_path)
cmake_command.append('-DCMAKE_CXX_COMPILER_LAUNCHER=' + conf.sccache_path) cmake_command.append('-DCMAKE_CXX_COMPILER_LAUNCHER=' + conf.sccache_path)
if dyld_path: if dyld_path:
cmake_command += ['-DDYLD_LIBRARY_PATH=' + dyld_path] cmake_command += ['-DDYLD_LIBRARY_PATH=' + dyld_path]
if libtool_path: if libtool_path:
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Lines if not conf.device and (target == "test" or target == "all"):
obj_dir = os.path.join(conf.workspace, obj_dir) obj_dir = os.path.join(conf.workspace, obj_dir)
cmd = [NINJA, '-v', '-k', '0', 'check-all'] cmd = [NINJA, '-v', '-k', '0', 'check-all']
if conf.assertions: if conf.assertions:
cmd[-1] += ' --param use_gmalloc=1 ' \ cmd[-1] += ' --param use_gmalloc=1 ' \
'--param gmalloc_path=$(xcodebuild -find-library' \ '--param gmalloc_path=$(xcodebuild -find-library' \
' libgmalloc.dylib)' ' libgmalloc.dylib)'
run_cmd(obj_dir, cmd, env={'MALLOC_LOG_FILE': '/dev/null'}) run_cmd(obj_dir, cmd, env={'MALLOC_LOG_FILE': '/dev/null'}, err_okay=True)
if conf.sccache: if conf.sccache:
run_ws([conf.sccache_path, "--stop-server"]) run_ws([conf.sccache_path, "--stop-server"])
def parse_settings_from_output(working_dir, cmd): def parse_settings_from_output(working_dir, cmd):
old_dir = os.getcwd() old_dir = os.getcwd()
try: try:
Show All 17 Linesdef lldb_cmake_builder(target, variant=None):
test_dir = os.path.join(conf.workspace, 'test') test_dir = os.path.join(conf.workspace, 'test')
results_file = os.path.join(test_dir, 'results.xml') results_file = os.path.join(test_dir, 'results.xml')
create_dirs([conf.lldbbuilddir(), test_dir]) create_dirs([conf.lldbbuilddir(), test_dir])
cmake_build_type = conf.cmake_build_type if conf.cmake_build_type else 'RelWithDebInfo' cmake_build_type = conf.cmake_build_type if conf.cmake_build_type else 'RelWithDebInfo'
# Construct dotest.py arguments. # Construct dotest.py arguments.
dotest_args=['--arch', 'x86_64', '--build-dir', dotest_args=['--build-dir', conf.lldbbuilddir()+'/lldb-test-build.noindex',
conf.lldbbuilddir()+'/lldb-test-build.noindex',
'-t', '--env', 'TERM=vt100'] '-t', '--env', 'TERM=vt100']
dotest_args.extend(conf.dotest_flags) dotest_args.extend(conf.dotest_flags)
# Construct lit arguments. # Construct lit arguments.
lit_args = ['-v', '--time-tests', '--shuffle', lit_args = ['-v', '--time-tests', '--shuffle',
'--xunit-xml-output={}'.format(results_file), '-v'] '--xunit-xml-output={}'.format(results_file), '-v']
if conf.max_parallel_tests: if conf.max_parallel_tests:
lit_args.extend(['-j', conf.max_parallel_tests]) lit_args.extend(['-j', conf.max_parallel_tests])
if variant == 'sanitized': if variant == 'sanitized':
lit_args.extend(['--timeout 1200']) lit_args.extend(['--timeout 1200'])
# Construct CMake invocation. # Construct CMake invocation.
cmake_cmd = ["/usr/local/bin/cmake", '-G', 'Ninja', cmake_cmd = ["/usr/local/bin/cmake", '-G', 'Ninja',
conf.llvmsrcdir(), conf.llvmsrcdir(),
'-DCMAKE_BUILD_TYPE={}'.format(cmake_build_type), '-DCMAKE_BUILD_TYPE={}'.format(cmake_build_type),
'-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON',
'-DLLVM_TARGETS_TO_BUILD=X86;ARM;AArch64',
'-DCMAKE_INSTALL_PREFIX={}'.format(conf.lldbinstalldir()), '-DCMAKE_INSTALL_PREFIX={}'.format(conf.lldbinstalldir()),
'-DCMAKE_MAKE_PROGRAM={}'.format(NINJA), '-DCMAKE_MAKE_PROGRAM={}'.format(NINJA),
'-DLLDB_TEST_USER_ARGS='+';'.join(dotest_args), '-DLLDB_TEST_USER_ARGS='+';'.join(dotest_args),
'-DLLDB_ENFORCE_STRICT_TEST_REQUIREMENTS=On',
'-DLLDB_ENABLE_PYTHON=On', '-DLLDB_ENABLE_PYTHON=On',
'-DLLVM_ENABLE_ASSERTIONS:BOOL={}'.format("TRUE" if conf.assertions else "FALSE"), '-DLLVM_ENABLE_ASSERTIONS:BOOL={}'.format("TRUE" if conf.assertions else "FALSE"),
'-DLLVM_ENABLE_MODULES=On', '-DLLVM_ENABLE_MODULES=On',
'-DLLVM_ENABLE_PROJECTS={}'.format(conf.llvm_enable_projects), '-DLLVM_ENABLE_PROJECTS={}'.format(conf.llvm_enable_projects),
'-DLLVM_LIT_ARGS={}'.format(' '.join(lit_args)), '-DLLVM_LIT_ARGS={}'.format(' '.join(lit_args)),
'-DLLVM_VERSION_PATCH=99'] '-DLLVM_VERSION_PATCH=99']
if conf.llvm_enable_runtimes:
cmake_cmd.append('-DLLVM_ENABLE_RUNTIMES={}'.format(conf.llvm_enable_runtimes))
if conf.lua:
cmake_cmd.append('-DLLDB_ENABLE_LUA=On')
if variant == 'sanitized': if variant == 'sanitized':
cmake_cmd.append('-DLLVM_TARGETS_TO_BUILD=X86') cmake_cmd.append('-DLLVM_TARGETS_TO_BUILD=X86')
cmake_cmd.append('-DLLVM_USE_SANITIZER=Address;Undefined') cmake_cmd.append('-DLLVM_USE_SANITIZER=Address;Undefined')
# There is no need to compile the lldb tests with an asanified compiler # There is no need to compile the lldb tests with an asanified compiler
# if we have a host compiler available. # if we have a host compiler available.
if conf.CC(): if conf.CC():
cmake_cmd.extend([ cmake_cmd.extend([
'-DLLDB_TEST_C_COMPILER=' + conf.CC(), '-DLLDB_TEST_COMPILER=' + conf.CC(),
'-DLLDB_TEST_CXX_COMPILER=' + conf.CC() + "++"
]) ])
if conf.compiler_flags: if conf.compiler_flags:
cmake_cmd.extend([ cmake_cmd.extend([
"-DCMAKE_C_FLAGS={}".format(' '.join(conf.compiler_flags)), "-DCMAKE_C_FLAGS={}".format(' '.join(conf.compiler_flags)),
"-DCMAKE_CXX_FLAGS={}".format(' '.join(conf.compiler_flags)) "-DCMAKE_CXX_FLAGS={}".format(' '.join(conf.compiler_flags))
]) ])
if conf.lldbtestcompiler(): if conf.lldb_test_compiler is not None:
cmake_cmd.extend([ cmake_cmd.extend([
'-DLLDB_TEST_C_COMPILER=' + conf.lldbtestcompiler(), '-DLLDB_TEST_COMPILER=' + conf.lldb_test_compiler,
'-DLLDB_TEST_CXX_COMPILER=' + conf.lldbtestcompiler() + "++"
]) ])
cmake_cmd.extend(conf.cmake_flags) cmake_cmd.extend(conf.cmake_flags)
if conf.CC(): if conf.CC():
cmake_cmd.extend([ cmake_cmd.extend([
'-DCMAKE_C_COMPILER=' + conf.CC(), '-DCMAKE_C_COMPILER=' + conf.CC(),
'-DCMAKE_CXX_COMPILER=' + conf.CC() + "++" '-DCMAKE_CXX_COMPILER=' + conf.CC() + "++"
Show All 13 Lines if target == 'all' or target == 'build':
run_cmd(conf.lldbbuilddir(), [NINJA, '-v']) run_cmd(conf.lldbbuilddir(), [NINJA, '-v'])
footer() footer()
if target == 'all' or target == 'install': if target == 'all' or target == 'install':
header("Install") header("Install")
run_cmd(conf.lldbbuilddir(), [NINJA, '-v', 'install']) run_cmd(conf.lldbbuilddir(), [NINJA, '-v', 'install'])
footer() footer()
if target == 'all' or target == 'testlong':
header("Run Debug Info Tests")
run_cmd(conf.lldbbuilddir(), [NINJA, '-v', 'check-debuginfo'])
footer()
if target == 'all' or target == 'test' or target == 'testlong': if target == 'all' or target == 'test' or target == 'testlong':
header("Run Tests") header("Run Tests")
run_cmd(conf.lldbbuilddir(), if variant == 'debuginfo':
['/usr/bin/env', 'TERM=vt100', NINJA, '-v', 'check-lldb']) test_command = ['/usr/bin/env', 'TERM=vt100', NINJA,
'-v', 'check-debuginfo']
else:
test_command = ['/usr/bin/env', 'TERM=vt100', NINJA,
'-v', 'check-lldb']
run_cmd(conf.lldbbuilddir(), test_command)
footer() footer()
for test_target in conf.cmake_test_targets: for test_target in conf.cmake_test_targets:
header("Run Custom Test: {0}".format(test_target)) header("Run Custom Test: {0}".format(test_target))
run_cmd(conf.lldbbuilddir(), [NINJA, '-k', '0', '-v', test_target]) run_cmd(conf.lldbbuilddir(), [NINJA, '-k', '0', '-v', test_target])
footer() footer()
Show All 26 Linesdef lldb_cmake_standalone_builder(target):
cmake_cmd = ['/usr/local/bin/cmake', '-G', 'Ninja', cmake_cmd = ['/usr/local/bin/cmake', '-G', 'Ninja',
conf.lldbsrcdir(), conf.lldbsrcdir(),
'-DLLVM_TARGETS_TO_BUILD=X86', '-DLLVM_TARGETS_TO_BUILD=X86',
'-DCMAKE_BUILD_TYPE={}'.format(cmake_build_type), '-DCMAKE_BUILD_TYPE={}'.format(cmake_build_type),
'-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON',
'-DCMAKE_MAKE_PROGRAM={}'.format(NINJA), '-DCMAKE_MAKE_PROGRAM={}'.format(NINJA),
'-DLLDB_TEST_USER_ARGS='+';'.join(dotest_args), '-DLLDB_TEST_USER_ARGS='+';'.join(dotest_args),
'-DLLDB_ENFORCE_STRICT_TEST_REQUIREMENTS=On',
'-DLLVM_ENABLE_ASSERTIONS:BOOL={}'.format( '-DLLVM_ENABLE_ASSERTIONS:BOOL={}'.format(
"TRUE" if conf.assertions else "FALSE"), "TRUE" if conf.assertions else "FALSE"),
'-DLLVM_ENABLE_MODULES=Off', '-DLLVM_ENABLE_MODULES=Off',
'-DLLVM_DIR={}'.format(llvm_dir), '-DLLVM_DIR={}'.format(llvm_dir),
'-DClang_DIR={}'.format(clang_dir), '-DClang_DIR={}'.format(clang_dir),
'-DLLVM_LIT_ARGS=--xunit-xml-output={} -v'.format( '-DLLVM_LIT_ARGS=--xunit-xml-output={} -v'.format(
results_file), results_file),
'-DLLVM_VERSION_PATCH=99'] '-DLLVM_VERSION_PATCH=99']
Show All 29 Linesdef lldb_cmake_xcode_builder(target):
xcode_cache = os.path.join(conf.lldbsrcdir(), 'cmake', 'caches', 'Apple-lldb-Xcode.cmake') xcode_cache = os.path.join(conf.lldbsrcdir(), 'cmake', 'caches', 'Apple-lldb-Xcode.cmake')
cmake_cmd = ['/usr/local/bin/cmake', cmake_cmd = ['/usr/local/bin/cmake',
'-C', xcode_cache, '-C', xcode_cache,
'-G', 'Xcode', '-G', 'Xcode',
'-L', conf.lldbsrcdir(), '-L', conf.lldbsrcdir(),
'-DLLVM_ENABLE_ASSERTIONS:BOOL={}'.format("TRUE" if conf.assertions else "FALSE"), '-DLLVM_ENABLE_ASSERTIONS:BOOL={}'.format("TRUE" if conf.assertions else "FALSE"),
'-DLLVM_ENABLE_MODULES=Off', '-DLLVM_ENABLE_MODULES=Off',
'-DLLDB_ENFORCE_STRICT_TEST_REQUIREMENTS=On',
'-DLLVM_DIR={}'.format(llvm_dir), '-DLLVM_DIR={}'.format(llvm_dir),
'-DClang_DIR={}'.format(clang_dir), '-DClang_DIR={}'.format(clang_dir),
'-DLLVM_VERSION_PATCH=99'] '-DLLVM_VERSION_PATCH=99']
cmake_cmd.extend(conf.cmake_flags) cmake_cmd.extend(conf.cmake_flags)
build_cmd = ['/usr/local/bin/cmake', build_cmd = ['/usr/local/bin/cmake',
'--build', '.', '--build', '.',
'--config', cmake_build_type] '--config', cmake_build_type]
▲ Show 20 Lines • Show All 50 Lines • ▼ Show 20 Linesdef static_analyzer_benchmarks_builder():
footer() footer()
def http_download(url, dest): def http_download(url, dest):
"""Safely download url to dest. """Safely download url to dest.
Print error and exit if download fails. Print error and exit if download fails.
""" """
chunk_size = 100 * 1024 * 1024
try: try:
print("GETting", url, "to", dest, "...") print("GETting", url, "to", dest, "...")
f = urllib.request.urlopen(url) session = requests.Session()
r = session.get(url)
r.raise_for_status()
# Open our local file for writing # Open our local file for writing
with open(dest, "wb") as local_file: with open(dest, "wb") as local_file:
local_file.write(f.read()) for chunk in r.iter_content(chunk_size=chunk_size):
local_file.write(chunk)
except urllib.HTTPError as e: except requests.exceptions.HTTPError as e:
print("HTTP Error:", e.code, url) print("HTTP Error:", e.response.status_code, url)
sys.exit(1) sys.exit(1)
except urllib.URLError as e:
print("URL Error:", e.reason, url)
sys.exit(1)
print("done.") print("done.")
def create_builddirs(): def create_builddirs():
create_dirs([conf.builddir(), conf.installdir()]); create_dirs([conf.builddir(), conf.installdir()]);
def fetch_compiler(): def fetch_compiler():
▲ Show 20 Lines • Show All 63 Lines • ▼ Show 20 Linesdef build_upload_artifact():
lng_cmd = ["ssh", "buildslave@" + SERVER, lng_cmd = ["ssh", "buildslave@" + SERVER,
"ln", "-fs", "/Library/WebServer/Documents/artifacts/" + "ln", "-fs", "/Library/WebServer/Documents/artifacts/" +
conf.job_name + "/" + artifact_name, conf.job_name + "/" + artifact_name,
"/Library/WebServer/Documents/artifacts/" + "/Library/WebServer/Documents/artifacts/" +
conf.job_name + "/g" + conf.git_sha] conf.job_name + "/g" + conf.git_sha]
run_cmd(conf.workspace, lng_cmd) run_cmd(conf.workspace, lng_cmd)
def build_upload_properties():
"""Create artifact for this build, and upload to server."""
if conf.noupload:
print('Not uploading artificats')
return
header("Uploading Artifact")
prop_file = "last_good_build.properties"
with open(prop_file, 'w') as prop_fd:
prop_fd.write("LLVM_REV={}\n".format(conf.svn_rev))
prop_fd.write("GIT_DISTANCE={}\n".format(conf.git_distance))
prop_fd.write("GIT_SHA={}\n".format(conf.git_sha))
mkdir_cmd = ["ssh", "buildslave@" + SERVER, "mkdir", "-p", "/Library/WebServer/Documents/artifacts/" + conf.job_name]
run_cmd(conf.workspace, mkdir_cmd)
upload_cmd = ["scp", prop_file,
"buildslave@" + SERVER + ":/Library/WebServer/Documents/artifacts/" +
conf.job_name + "/"]
run_cmd(conf.workspace, upload_cmd)
def run_cmd(working_dir, cmd, env=None, sudo=False, err_okay=False): def run_cmd(working_dir, cmd, env=None, sudo=False, err_okay=False):
"""Run a command in a working directory, and make sure it returns zero.""" """Run a command in a working directory, and make sure it returns zero."""
assert type(cmd) == list, "Not a list: {}".format(type(cmd)) assert type(cmd) == list, "Not a list: {}".format(type(cmd))
old_cwd = os.getcwd() old_cwd = os.getcwd()
if env: if env:
envs = [] envs = []
for key, value in env.items(): for key, value in env.items():
Show All 23 Linesdef run_cmd(working_dir, cmd, env=None, sudo=False, err_okay=False):
logging.info("Command took {} seconds".format( logging.info("Command took {} seconds".format(
(end_time - start_time).seconds)) (end_time - start_time).seconds))
return return_code return return_code
KNOWN_TARGETS = ['all', 'configure', 'build', 'test', 'testlong', 'install'] KNOWN_TARGETS = ['all', 'configure', 'build', 'test', 'testlong', 'install']
KNOWN_BUILDS = [ KNOWN_BUILDS = [
'clang', 'cmake', 'lldb-cmake', 'lldb-cmake-standalone', 'clang',
'lldb-cmake-xcode', 'lldb-cmake-sanitized', 'lldb-cmake-matrix', 'fetch', 'cmake',
'artifact', 'static-analyzer-benchmarks' 'lldb-cmake',
'lldb-cmake-standalone',
'lldb-cmake-debuginfo',
'lldb-cmake-xcode',
'lldb-cmake-sanitized',
'lldb-cmake-matrix',
'fetch',
'artifact',
'properties',
'static-analyzer-benchmarks',
] ]
def query_sdk_path(sdk_name): def query_sdk_path(sdk_name):
"""Get the path to the sdk named using xcrun. """Get the path to the sdk named using xcrun.
When $TESTING define, just give a dummy back. We do this because xcrun When $TESTING define, just give a dummy back. We do this because xcrun
could fail if the sdk you want is not installed, and that is silly for could fail if the sdk you want is not installed, and that is silly for
▲ Show 20 Lines • Show All 83 Lines • ▼ Show 20 Lines parser.add_argument("build_target",
nargs='?', nargs='?',
help="The targets to call (build, check, etc).", help="The targets to call (build, check, etc).",
choices=KNOWN_TARGETS) choices=KNOWN_TARGETS)
parser.add_argument('--assertions', dest='assertions', action='store_true') parser.add_argument('--assertions', dest='assertions', action='store_true')
parser.add_argument('--lto', dest='lto', action='store_true') parser.add_argument('--lto', dest='lto', action='store_true')
parser.add_argument('--thinlto', dest='thinlto', action='store_true') parser.add_argument('--thinlto', dest='thinlto', action='store_true')
parser.add_argument('--debug', dest='debug', action='store_true') parser.add_argument('--debug', dest='debug', action='store_true')
parser.add_argument('--lua', dest='lua', action='store_true')
parser.add_argument('--cmake-type', dest='cmake_build_type', parser.add_argument('--cmake-type', dest='cmake_build_type',
help="Override cmake type Release, Debug, " help="Override cmake type Release, Debug, "
"RelWithDebInfo and MinSizeRel") "RelWithDebInfo and MinSizeRel")
parser.add_argument('--cmake-flag', dest='cmake_flags', parser.add_argument('--cmake-flag', dest='cmake_flags',
action='append', default=[], action='append', default=[],
help='Set an arbitrary cmake flag') help='Set an arbitrary cmake flag')
parser.add_argument('--dotest-flag', dest='dotest_flags', parser.add_argument('--dotest-flag', dest='dotest_flags',
action='append', default=[], action='append', default=[],
Show All 12 Lines parser.add_argument('--noinstall', dest='noinstall', action='store_true',
help="Disable the install stage, build only.") help="Disable the install stage, build only.")
parser.add_argument('--sccache', dest='sccache', parser.add_argument('--sccache', dest='sccache',
action='store_true', default=False, action='store_true', default=False,
help="If the builds supports it, use sccache.") help="If the builds supports it, use sccache.")
parser.add_argument('--globalisel', dest='globalisel', parser.add_argument('--globalisel', dest='globalisel',
action='store_true', help="Turn on the experimental" action='store_true', help="Turn on the experimental"
" GlobalISel CMake flag.") " GlobalISel CMake flag.")
parser.add_argument('--projects', dest='llvm_enable_projects', parser.add_argument('--projects', dest='llvm_enable_projects',
default="clang;clang-tools-extra;compiler-rt;libcxx", default="clang;clang-tools-extra;compiler-rt",
help="Semicolon seperated list of projects to build.") help="Semicolon seperated list of projects to enable.")
parser.add_argument('--runtimes', dest='llvm_enable_runtimes',
default="",
help="Semicolon seperated list of runtimes to enable.")
parser.add_argument('--timeout', dest='timeout', type=int, default='600', parser.add_argument('--timeout', dest='timeout', type=int, default='600',
help='Individual test timeout in seconds.') help='Individual test timeout in seconds.')
parser.add_argument('--lldb-test-compiler',
help='The compiler used to build LLDB tests.')
args = parser.parse_args() args = parser.parse_args()
if args.thinlto: if args.thinlto:
args.lto = True args.lto = True
return args return args
def main(): def main():
"""Run a build based on command line args and ENV.""" """Run a build based on command line args and ENV."""
global conf global conf
args = parse_args() args = parse_args()
conf = Configuration(args) conf = Configuration(args)
create_builddirs() create_builddirs()
try: try:
if args.build_type == 'clang': if args.build_type == 'clang':
clang_builder(args.build_target) clang_builder(args.build_target)
elif args.build_type == 'lldb-cmake': elif args.build_type == 'lldb-cmake':
lldb_cmake_builder(args.build_target) lldb_cmake_builder(args.build_target)
elif args.build_type == 'lldb-cmake-debuginfo':
lldb_cmake_builder(args.build_target, 'debuginfo')
elif args.build_type == 'lldb-cmake-sanitized': elif args.build_type == 'lldb-cmake-sanitized':
lldb_cmake_builder(args.build_target, 'sanitized') lldb_cmake_builder(args.build_target, 'sanitized')
elif args.build_type == 'lldb-cmake-matrix': elif args.build_type == 'lldb-cmake-matrix':
lldb_cmake_builder(args.build_target, 'matrix') lldb_cmake_builder(args.build_target, 'matrix')
elif args.build_type == 'lldb-cmake-standalone': elif args.build_type == 'lldb-cmake-standalone':
lldb_cmake_standalone_builder(args.build_target) lldb_cmake_standalone_builder(args.build_target)
elif args.build_type == 'lldb-cmake-xcode': elif args.build_type == 'lldb-cmake-xcode':
lldb_cmake_xcode_builder(args.build_target) lldb_cmake_xcode_builder(args.build_target)
elif args.build_type == 'cmake': elif args.build_type == 'cmake':
cmake_builder(args.build_target) cmake_builder(args.build_target)
elif args.build_type == 'fetch': elif args.build_type == 'fetch':
fetch_compiler() fetch_compiler()
elif args.build_type == 'artifact': elif args.build_type == 'artifact':
build_upload_artifact() build_upload_artifact()
elif args.build_type == 'properties':
build_upload_properties()
elif args.build_type == 'static-analyzer-benchmarks': elif args.build_type == 'static-analyzer-benchmarks':
static_analyzer_benchmarks_builder() static_analyzer_benchmarks_builder()
except subprocess.CalledProcessError as exct: except subprocess.CalledProcessError as exct:
print('Command failed: {}'.format(exct)) print('Command failed: {}'.format(exct))
print('Command:', exct.cmd) print('Command:', exct.cmd)
sys.exit(1) sys.exit(1)
if __name__ == '__main__': if __name__ == '__main__':
main() main()

zorg/jenkins/relay.groovy

Show All 22 Linesprivate def relay_steps(job_pattern, artifact_url, last_good_properties_url) {
// compiler artifact that should be used for this run and which llvm // compiler artifact that should be used for this run and which llvm
// revision it is based on. // revision it is based on.
propfile = basename(last_good_properties_url) propfile = basename(last_good_properties_url)
sh """ sh """
rm -f ${propfile} rm -f ${propfile}
curl -fksSO "${last_good_properties_url}" curl -fksSO "${last_good_properties_url}"
""" """
def props = readProperties file: propfile def props = readProperties file: propfile
def artifact = "http://labmaster2.local/artifacts/${props.ARTIFACT}" def artifact = "http://green-dragon-21.local/artifacts/${props.ARTIFACT}"
currentBuild.setDisplayName("${props.GIT_DISTANCE}-${props.GIT_SHA}") currentBuild.setDisplayName("${props.GIT_DISTANCE}-${props.GIT_SHA}")
// Trigger all jobs with names matching the `job_pattern` regex. // Trigger all jobs with names matching the `job_pattern` regex.
def joblist = get_matching_jobs(job_pattern) def joblist = get_matching_jobs(job_pattern)
def parallel_builds = [:] def parallel_builds = [:]
for (j in joblist) { for (j in joblist) {
def shortname = j[0] def shortname = j[0]
def jobname = j[1] def jobname = j[1]
Show All 11 Lines for (j in joblist) {
] ]
build job: jobname, parameters: job_params build job: jobname, parameters: job_params
} }
} }
parallel parallel_builds parallel parallel_builds
} }
def pipeline(job_pattern, def pipeline(job_pattern,
artifact_url='http://labmaster2.local/artifacts/', artifact_url='http://green-dragon-21.local/artifacts/',
last_good_properties_url='http://labmaster2.local/artifacts/clang-stage1-RA/last_good_build.properties') { last_good_properties_url='http://green-dragon-21.local/artifacts/clang-stage1-RA/last_good_build.properties') {
node('master') { node('master') {
stage('main') { stage('main') {
relay_steps job_pattern, artifact_url, last_good_properties_url relay_steps job_pattern, artifact_url, last_good_properties_url
} }
} }
} }
def lldb_pipeline(job_pattern,
artifact_url='http://green-dragon-21.local/artifacts/',
last_good_properties_url='http://green-dragon-21.local/artifacts/lldb-cmake/last_good_build.properties') {
node('master') {
stage('main') {
relay_steps job_pattern, artifact_url, last_good_properties_url
}
}
}
return this return this
zorg/jenkins/jobs/jobs/lldb-cmake-matrix