Revamp LLVM WoA buildbots

[omjavaid]

Summary

This patch makes following changes to LLVM Windows on Arm buildbots:

1. Improves clang-arm64-windows-msvc builder. It will run in similar configuration to linux clang quick builder with ninja check enabled on a dedicated relatively fast worker machine.
2. Enables ninja check-all for both stage1 and stage2 on clang-arm64-windows-msvc-2stage builder.
3. Adds an extra worker for clang-arm64-windows-msvc-2stage builder. Making 3 workers in total testing two stage builds.
4. Removes redundant CMAKE flags from various all WoA builders.
5. Enables object file caching using sccache
6. Update worker for lldb-aarch64-windows builder. It will be dedicated Surface X pro machines.

[DavidSpickett]

buildbot/osuosl/master/config/builders.py

243

Can we make the claim this is quick when flang is involved? I'm seeing anything from 10 minutes to 30 minutes on the existing flang-aarch64-release bot. Though if it's quick relative to the other WoA bot I could accept that.

I suppose we painted ourselves into a corner naming our single architecture backend bots "quick" :)

244–245

Do you need this in the new bot too?

Ditto all the extra_cmake_args here. Are we at the point now that we don't need them?

[omjavaid]

buildbot/osuosl/master/config/builders.py

243

A test run for above builder is running here: http://139.178.84.207:49161/#/builders/18

We may not be able to call it quick meeting the standards of Linux server grade hardware. However compared to other arm64-windows-msvc bots this should build faster. Build times range from 35 mins upto 2 hours. The normal clang-arm64-windows-msvc bots takes 4 hours straight.

Re flang we adds about 10-15% more files to be built and about 10% more unit/regression tests to be run. I assume we can live with that because for now we dont have hardware to put separate flang bots in place for arm64-windows.

Re naming I guess we call it quick assuming a clean build takes more time and quick bots dont do a clean build so that may be the source of this naming convention.

[omjavaid]

buildbot/osuosl/master/config/builders.py

244–245

Near future plan is to make this clang-arm64-windows-msvc bot redundant and keep the quick and 2stage ones.

Yes I am removing the extra flags after testing. We no longer need most of them.

[DavidSpickett]

If we're going to remove the existing single stage bot later anyway, we can always rename this one to the same thing if it's too slow to be "quick" by most standards.

The linked build doesn't load for me, but in any case, what tests are we running? check-flag we know can work, are we up to being able to run check-all? (which I think is what this will result in)

Nice milestone if so.

[gkistanova]

Thanks for the patch, Muhammad.

If you plan to replace the clang-arm64-windows-msvc builder, then maybe just do that? You can move it to the staging, and change the build config to match what you have for the "quick" builder, make sure it is green, and get the new builder back to production? There is a quite time around the New Year, so it might be a good point in time for that.

buildbot/osuosl/master/config/builders.py

239

The builddir must be unique. The easiest would be just skipping this arg and let the default machinery do its magic. Otherwise, make it unique, please. Currently it is the same as in the clang-arm64-windows-msvc builder.

[omjavaid]

Add more changes updating all LLVM Windows on Arm builders.

[omjavaid]

buildbot/osuosl/master/config/builders.py

239

Fixed in latest rev.

[omjavaid]

Ping! @DavidSpickett @gkistanova

[gkistanova]

Thanks for making the changes!

The patch looks good with a nit pick.

Since you replaced clang-arm64-windows-msvc, could you keep the same builder name, please?
The new builder is faster than the old one, which is great. Doesn’t make it a quick one, though. Keeping the same name seems reasonable.

[omjavaid]

In D140384#4035219, @gkistanova wrote:

Thanks for making the changes!

The patch looks good with a nit pick.

Since you replaced clang-arm64-windows-msvc, could you keep the same builder name, please?
The new builder is faster than the old one, which is great. Doesn’t make it a quick one, though. Keeping the same name seems reasonable.

Ack I will update the bot name. But Is there a criteria based on speed or configuration to call a bot Quick. This bot that I am adding is doing about 15 builds per day compared to 25 builds per day of LLVM AArch64 Linux quick bot. Most builds run for an Average build time of about 25 - 30 minutes on this new windows bot however some builds do take about 2 hours. The Linux bot with similar configuration take about 10 - 15 minutes and they dont have those 1 - 2 builds per day taking couple of hours.

[gkistanova]

The meaning is changing over time as we are getting more and more fast builders. For now, I'd call a builder quick if it is able to build each commit and stay current with the build queue in general (something like 3-5 minutes response time or better).

[DavidSpickett]

This looks fine but I agree, let's keep the clang-arm64-windows-msvc name.

[omjavaid]

Rename clang-arm64-windows-msvc-quick back to clang-arm64-windows-msvc.

[DavidSpickett]

LGTM

[gkistanova]

Thanks, Muhammad!
Looks good.

buildbot/osuosl/master/config/builders.py

'factory' : XToolchainBuilder.getCmakeWithMSVCBuildFactory(
                         WithProperties("%(remote_test_host:+-DREMOTE_TEST_HOST=)s%(remote_test_host:-)s"),
                         WithProperties("%(remote_test_user:+-DREMOTE_TEST_USER=)s%(remote_test_user:-)s"),
                     ],
                     cmake_cache="../llvm-project/clang/cmake/caches/CrossWinToARMLinux.cmake")},
 
 # Clang builders.
 
     {'name': "clang-arm64-windows-msvc",
-    'tags' : ["clang"],
-    'workernames' : ["linaro-armv8-windows-msvc-01", "linaro-armv8-windows-msvc-02"],
+    'tags' : ["llvm", "clang", "lld", "flang"],
+    'workernames' : ["linaro-armv8-windows-msvc-04"],
     'builddir': "clang-arm64-windows-msvc",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     vs="manual",
+                    clean=False,
                     checkout_flang=True,
-                    # Only enabling checks we expect to work.
-                    checks=['check-flang'],
+                    checkout_lld=True,
+                    checkout_compiler_rt=False,
                     extra_cmake_args=[
                         "-DCMAKE_TRY_COMPILE_CONFIGURATION=Release",
-                        "-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-pc-windows-msvc",
-                        "-DLLVM_HOST_TRIPLE=aarch64-pc-windows-msvc",
-                        "-DLLVM_TARGET_ARCH=AArch64",
-                        "-DCMAKE_C_FLAGS=-fms-compatibility-version=19.27",
-                        "-DCMAKE_CXX_FLAGS=-fms-compatibility-version=19.27",
-                        # The BUILTINS environment variable is expected to already exist
-                        # on the worker when it is launched.
-                        "-DCMAKE_EXE_LINKER_FLAGS=%BUILTINS%",
-                        "-DCMAKE_SHARED_LINKER_FLAGS=%BUILTINS%",
-                        "-DCMAKE_STATIC_LINKER_FLAGS=%BUILTINS%",
-                        "-DCMAKE_MODULE_LINKER_FLAGS=%BUILTINS%",
-                        # FIXME: compiler-rt\lib\sanitizer_common\sanitizer_unwind_win.cpp assumes WIN64 is x86_64,
-                        #        so, before that's fixed, disable everything that triggers its build.
-                        "-DCOMPILER_RT_BUILD_SANITIZERS=OFF",
-                        "-DCOMPILER_RT_BUILD_MEMPROF=OFF",
-                        "-DCOMPILER_RT_BUILD_XRAY=OFF"])},
+                        "-DLLVM_TARGETS_TO_BUILD='AArch64'",
+                        "-DCMAKE_C_COMPILER_LAUNCHER=sccache",
+                        "-DCMAKE_CXX_COMPILER_LAUNCHER=sccache"])},
 
     ## ARMv8 check-all
     {'name' : "clang-armv8-quick",
     'tags'  : ["clang"],
     'workernames':["linaro-clang-armv8-quick"],
     'builddir':"clang-armv8-quick",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     clean=False,
'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     extra_cmake_args=[
                         "-DCMAKE_C_FLAGS='-mcpu=a64fx -msve-vector-bits=512 -mllvm -treat-scalable-fixed-error-as-warning=false'",
                         "-DCMAKE_CXX_FLAGS='-mcpu=a64fx -msve-vector-bits=512 -mllvm -treat-scalable-fixed-error-as-warning=false'",
                         "-DLLVM_ENABLE_LLD=True",
                         "-DLLVM_LIT_ARGS='-v -j12'"])},
 
     {'name' : "clang-arm64-windows-msvc-2stage",
     'tags'  : ["clang"],
-    'workernames' : ["linaro-armv8-windows-msvc-01", "linaro-armv8-windows-msvc-02"],
+    'workernames' : ["linaro-armv8-windows-msvc-01", "linaro-armv8-windows-msvc-02", "linaro-armv8-windows-msvc-03"],
     'builddir': "clang-arm64-windows-msvc-2stage",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     vs="manual",
-                    checks=[], # Disable testing until MCJIT failures are fixed
                     useTwoStage=True,
-                    testStage1=False,
+                    checkout_flang=True,
                     extra_cmake_args=[
                         "-DCMAKE_TRY_COMPILE_CONFIGURATION=Release",
-                        "-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-pc-windows-msvc",
-                        "-DLLVM_HOST_TRIPLE=aarch64-pc-windows-msvc",
-                        "-DLLVM_TARGET_ARCH=AArch64",
-                        "-DCMAKE_C_FLAGS=-fms-compatibility-version=19.27",
-                        "-DCMAKE_CXX_FLAGS=-fms-compatibility-version=19.27",
+                        "-DCMAKE_C_COMPILER_LAUNCHER=sccache",
+                        "-DCMAKE_CXX_COMPILER_LAUNCHER=sccache",                         
                         # FIXME: compiler-rt\lib\sanitizer_common\sanitizer_unwind_win.cpp assumes WIN64 is x86_64,
                         #        so, before that's fixed, disable everything that triggers its build.
                         "-DCOMPILER_RT_BUILD_SANITIZERS=OFF",
-                        "-DCOMPILER_RT_BUILD_MEMPROF=OFF",
-                        "-DCOMPILER_RT_BUILD_XRAY=OFF"])},
+                        "-DCOMPILER_RT_BUILD_PROFILE=OFF"])},
 
     {'name' : 'clang-x64-windows-msvc',
     'tags'  : ["clang"],
     'workernames' : ['windows-gcebot2'],
     'builddir': 'clang-x64-windows-msvc',
     'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
                     script="clang-windows.py",
                     depends_on_projects=['llvm', 'clang', 'lld', 'debuginfo-tests'])},
'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
                         '-DLLDB_ENABLE_PYTHON=TRUE',
                         '-DLLDB_TEST_USER_ARGS=--skip-category=watchpoint',
                         '-DLLVM_ENABLE_ASSERTIONS=OFF',
                         '-DLLVM_ENABLE_ZLIB=FALSE',
                         '-DLLVM_LIT_ARGS=-vj 8'])},
 
     {'name' : "lldb-aarch64-windows",
     'tags'  : ["lldb"],
-    'workernames' : ["linaro-lldb-aarch64-windows"],
+    'workernames' : ["linaro-armv8-windows-msvc-05"],
     'builddir': "lldb-aarch64-windows",
     'factory' : LLDBBuilder.getLLDBCMakeBuildFactory(
                     clean=True,
                     test=True,
                     extra_cmake_args=[
-                        '-DCLANG_DEFAULT_LINKER=lld',
-                        '-DLLDB_ENABLE_PYTHON=TRUE',
-                        '-DLLDB_TEST_USER_ARGS=--skip-category=watchpoint',
-                        '-DLLVM_NATIVE_ARCH=AArch64',
-                        '-DLLVM_HOST_TRIPLE=aarch64-windows-msvc',
-                        '-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-windows-msvc',
-                        '-DLLVM_FORCE_USE_OLD_TOOLCHAIN=True',
-                        '-DLLVM_ENABLE_ASSERTIONS=OFF',
-                        '-DLLVM_ENABLE_ZLIB=FALSE',
-                        '-DLLVM_LIT_ARGS=-vj 4'])},
+                        "-DCMAKE_C_COMPILER_LAUNCHER=sccache",
+                        "-DCMAKE_CXX_COMPILER_LAUNCHER=sccache",
+                        '-DLLDB_TEST_USER_ARGS=--skip-category=watchpoint'])},
 
 # LLD builders.
 
     {'name' : "lld-x86_64-win",
     'tags'  : ["lld"],
     'workernames' : ["as-worker-93"],
     'builddir': "lld-x86_64-win",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaWithMSVCBuildFactory(

buildbot/osuosl/master/config/workers.py

return [
         create_worker("linaro-flang-aarch64-latest-clang", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-flang-aarch64-release", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-flang-aarch64-rel-assert", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-flang-aarch64-latest-gcc", properties={'jobs' : 32}, max_builds=1),
 
         # AArch64 Windows Microsoft Surface X Pro
         create_worker("linaro-armv8-windows-msvc-01", properties={'jobs' : 8}, max_builds=1),
         create_worker("linaro-armv8-windows-msvc-02", properties={'jobs' : 8}, max_builds=1),
-        create_worker("linaro-lldb-aarch64-windows", properties={'jobs' : 8}, max_builds=1),
+        create_worker("linaro-armv8-windows-msvc-03", properties={'jobs' : 8}, max_builds=1),
+        create_worker("linaro-armv8-windows-msvc-04", properties={'jobs' : 8}, max_builds=1),
+        create_worker("linaro-armv8-windows-msvc-05", properties={'jobs' : 8}, max_builds=1),
 
         # Windows Server 2016 Intel Xeon(R) Quad 2.30 GHz, 56GB of RAM
         create_worker("win-py3-buildbot", properties={'jobs' : 64}, max_builds=1),
 
         # Windows Server 2016 Intel(R) Xeon(R) CPU @ 2.60GHz, 16 Core(s), 128 GB of RAM
         create_worker("win-mlir-buildbot", properties={'jobs' : 64}, max_builds=1),
 
         # LoongArch64 Loongson-3C5000L-LL 2GHz, 128GB of RAM