[buildbot] Add a LoongArch buildbot on CLFS

[leecheechen]

Summary

Add a new builder and a new worker for LoongArch.
The worker runs build, check-all and test-suite on Loongson-3C5000L-LL
machine.

[leecheechen]

Modify the names of builder and worker from loongarch to loongarch64

[SixWeining]

Move @wangleiat and me to subscribers as we have no authority to review buildbot changes.

[SixWeining]

Add some guys who help reviewing LoongArch changes to the subscriber list. I think they may be interested in this change. Thanks.

[leecheechen]

Update Summary.

[xen0n]

Thanks for the builder contribution, and congratulations! I'm very pleased to be a part of this bring-up effort. Graduating from experimental status Soon™️ :-)

buildbot/osuosl/master/config/builders.py

814

It seems single quotes are used for string literals elsewhere. Consistency would be nice.

825

I currently build AArch64 and RISCV too, as they are prominent arches too. Maybe adding Mips would be nice as well, AFAIK Loongson claimed to "maintain the previous MIPS models for 10 years" a few years ago.

buildbot/osuosl/master/config/workers.py

67

mention CLFS and give a link to the CLFS repo, given it's not a widely-known distro spin?

[leecheechen]

Update Summaryy

[SixWeining]

buildbot/osuosl/master/config/builders.py

825

https://llvm.org/docs/HowToAddABuilder.html#best-practices-for-configuring-a-fast-builder says:

Restrict what you build and test
Think hard about why you’re setting up a bot, and restrict your build configuration as much as you can. Basic functionality is probably already covered by other bots, and you don’t need to duplicate that testing. You only need to be building and testing the unique parts of the configuration. (e.g. For a multi-stage clang builder, you probably don’t need to be enabling every target or building all the various utilities.)

I think most targets including X86/AArch64/RISCV/Mips are already coverred by other bots. To make the bot faster, we should turn on as few as targets here.

[xen0n]

buildbot/osuosl/master/config/builders.py

825

https://llvm.org/docs/HowToAddABuilder.html#best-practices-for-configuring-a-fast-builder says:

Restrict what you build and test
Think hard about why you’re setting up a bot, and restrict your build configuration as much as you can. Basic functionality is probably already covered by other bots, and you don’t need to duplicate that testing. You only need to be building and testing the unique parts of the configuration. (e.g. For a multi-stage clang builder, you probably don’t need to be enabling every target or building all the various utilities.)

I think most targets including X86/AArch64/RISCV/Mips are already coverred by other bots. To make the bot faster, we should turn on as few as targets here.

Hmm I must be so coffee-deprived while writing that comment that I forgot this. No problem only building x86 and ourselves then.

[leecheechen]

Use single quotes for string literals.
Add a link to CLFS repo.

[xen0n]

buildbot/osuosl/master/config/builders.py

814

Should now be "clfs-clang-loongarch64-test"?

buildbot/osuosl/master/config/status.py

330

Ditto.

[leecheechen]

buildbot/osuosl/master/config/builders.py

814

The builder may run build on different operating systems. For this reason, I didn't add the system info in its name.

buildbot/osuosl/master/config/status.py

330

The same as mentioned above.

[xen0n]

buildbot/osuosl/master/config/builders.py

814

Okay so the job name and builder name is way too similar then. I'd suggest picking a name for the builder instance, such as loongson-loongarch64-clfs-clang-01, while keeping the job name as-is. See how e.g. Linaro builders all have names like linaro-ARCH-OS-CC-NUMBER. (The architecture name is considered by many to be too loooong, so you may choose to use a shorter one like loongson-loong64-xxx instead, the important part is to clearly convey the information that this is a LoongArch builder contributed by Loongson running CLFS & Clang.)

[leecheechen]

buildbot/osuosl/master/config/builders.py

814

Thanks. I will change the worker name to "loongson-loongarch64-clfs-clang-01" according to your suggestion. The builder name will be changed to "clang-loongarch64-linux", thus it is not too similar to the worker name, and also keeps consistency with other builders.

[leecheechen]

Modify the names of builder and worker.

[leecheechen]

Use single quotes for string literal.

[leecheechen]

Enable check-all.

[leecheechen]

Update summary.

[SixWeining]

Add @rengolin to reviewer list.

[leecheechen]

hi, @gkistanova , I sent the buildbot-worker-access-name and buildbot-worker-access-password to your email ([e-mail]) on November 25, 2022. And updated the buildbot-worker-access-name to “loongson-loongarch64-clfs-clang-01” on November 28, 2022. I would like to confirm whether you have received these two emails. Thanks!

[rengolin]

buildbot/osuosl/master/config/builders.py

825

On the Arm32 bots, we really only build Arm, not even x86. Of all targets, x86 is perhaps the one with the most coverage, so you really don't need to build it on other targets.

827

Avoid adding temporal state on the commit. Whenever you fix those issues you won't come back here and update (or at least, shouldn't). These files are rarely changed even when the bots themselves change.

The only reason you'll have to change this again is when LoongArch moves out of experimental, or if you want to build more stuff on the same build.

[SixWeining]

buildbot/osuosl/master/config/builders.py

827

@rengolin Do you mean we could remove line 826 to line 834 and make /home/loongson/gcc/gcc-install/bin/g{cc,++} as the default C/C++ compiler on this bot?

[rengolin]

buildbot/osuosl/master/config/builders.py

827

No, I mean just remove the comments because what compiler you use and why is a matter for the bot maintainer only.

If you need a special compiler, go for it, don't need to explain how you got there. :)

You should create issues (either llvm or your local bugtracker) to fix those problems, and before we move out of experimental, you need to be able to build it with any GCC/clang that we officially support (because other bots will start building your backend and running your tests), but until then, take your time, fix your bugs, etc.

Changes to this file are rare and require additional screening from other bot owners, Galina, as well as a build master restart (by Galina), so you shouldn't add comments about internal infrastructure.

[SixWeining]

buildbot/osuosl/master/config/builders.py

825

So we can define:
-DLLVM_TARGETS_TO_BUILD=“”
Right?

827

Sorry, I’m a little confused.

The default gcc13 in’/usr/bin’ on the bot has the bug mentioned here https://github.com/llvm/llvm-project/issues/59227. But if it was fixed in future, we may choose to use it but not ‘/home/loongson/gcc/gcc-install/bin/gcc'. This would cause a change to this file again. So why not remove these 2 lines this time and make the make the custom gcc as default?

'-DCMAKE_C_COMPILER=/home/loongson/gcc/gcc-install/bin/gcc',
'-DCMAKE_CXX_COMPILER=/home/loongson/gcc/gcc-install/bin/g++',

And I think the bug of gcc13 is not LoongArch specific because we met the same issue on X86. What’s more, we find the specific gcc commit causing introducing this bug. This commit seems is not arch related.

PS: The reason why no bots report such issue is that no bots use latest gcc13 and run check-clang in release build.

[rengolin]

buildbot/osuosl/master/config/builders.py

825

If that works, sure.

827

Whatever works on your system. I normally just install clang from packages and use that to build, but any compiler you have that works should be fine.

If GCC 13 doesn't work then this is a bug, as the current requirements state GCC >= 7. But that's not for this patch. Here you just build however you can.

[xen0n]

buildbot/osuosl/master/config/builders.py

825

So we can define:
-DLLVM_TARGETS_TO_BUILD=“”
Right?

I just confirmed this works both for cross and native builds. Will save so much time each run...

827

IMO, I don't think hard-coding the absolute path is a good idea either, due to the involved process hinted by @rengolin; as the builder is a CLFS installation which is already hugely non-standard by itself, I'd say just swap out the gcc and g++ commands and symlink the "fixed" version into place before the situation is resolved. (CLFS for LoongArch is AFAIK basically maintained by one person, with dubious automation and reproducibility, if at all. I remember the CLFS for LoongArch maintainer said he has a huge script automating everything, but it's never shared on the project repo at least not yet.)

And, as the nonstandard toolchain is a problem in need of fixing, I'd suggest following the advice of making a public issue, so people don't forget to revisit later.

[leecheechen]

Revised to @xen0n @rengolin comments. Thank you for all the reviews.

[rengolin]

LGTM, thanks! Please check with @gkistanova before merging.

[leecheechen]

@gkistanova ping

[leecheechen]

Enable test-suite

[leecheechen]

Update summary.

[SixWeining]

Hi @gkistanova, this change has been accepted and could you please take a look? We hope the bot for LoongArch could be setted up as soon as possiple so that we can request to move LoongArch out of experimental before LLVM16 release branch is created. Thanks.

[rengolin]

Galina may be on extended holiday.

When maintaining bots at Linaro, we had a local Buildmaster where we could test our changes and new bots. If you have one, you could expose that for now, so that you can share build failures in case of breakages from others, until Galina comes back.

[SixWeining]

In D138672#3980784, @rengolin wrote:

Galina may be on extended holiday.

When maintaining bots at Linaro, we had a local Buildmaster where we could test our changes and new bots. If you have one, you could expose that for now, so that you can share build failures in case of breakages from others, until Galina comes back.

Thanks Renato.

If self-maintained buildbot can meet the requirement for a back-end to be promoted to official, we would have a try.

[rengolin]

In D138672#3980995, @SixWeining wrote:

If self-maintained buildbot can meet the requirement for a back-end to be promoted to official, we would have a try.

It's a bit of a grey area, but the key requirement is:

"The build target check-all must pass with the new target built, and where applicable, the test-suite must also pass without errors, in at least one configuration (publicly demonstrated, for example, via buildbots)."

It doesn't specify where.

The tests must be "demonstrated on a public buildbot", which can be tied to your own Buildmaster, or in Github Actions, Jenkins, Buildkite, DevOps, etc.

The main reason to get this bot in the upstream Buildmaster is that users get warned on breakages and provides a single entry point to CI, so that people can find your builders.

We've been trying to spread the cost of CI for many years but it has proven a hard problem for compiler developers to solve. :)

[leecheechen]

Build and run test-suite in parallel.

[gkistanova]

LGTM.

Thanks everyone for helping with the review!

[leecheechen]

rebase

buildbot/osuosl/master/config/builders.py

'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     checkout_lld=False,
                     useTwoStage=True,
                     stage1_config='Release',
                     stage2_config='Release',
                     extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON',
                                       '-DLLVM_PARALLEL_LINK_JOBS=4',
                                       '-DLLVM_TARGETS_TO_BUILD=Sparc'])},
 
+    ## LoongArch64 Clang+LLVM build check-all
+    {'name' : 'clang-loongarch64-linux',
+    'tags'  : ['clang'],
+    'workernames' : ['loongson-loongarch64-clfs-clang-01'],
+    'builddir': 'clang-loongarch64-linux',
+    'factory' : ClangBuilder.getClangCMakeBuildFactory(
+                    clean=False,
+                    runTestSuite=False,
+                    checkout_clang_tools_extra=False,
+                    checkout_compiler_rt=False,
+                    checkout_lld=False,
+                    extra_cmake_args=['-DLLVM_TARGETS_TO_BUILD=',
+                                      '-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=LoongArch',
+                                      '-DLLVM_ENABLE_PROJECTS=clang'])},
+
     {'name' : "clang-hexagon-elf",
     'tags'  : ["clang"],
     'workernames' : ["hexagon-build-02", "hexagon-build-03"],
     'builddir': "clang-hexagon-elf",
     'factory' : ClangBuilder.getClangCMakeBuildFactory(
                     jobs=16,
                     checkout_clang_tools_extra=False,
                     checkout_compiler_rt=False,

buildbot/osuosl/master/config/status.py

reporters.MailNotifier(
                     "llvm-clang-x86_64-sie-win"]),
     reporters.MailNotifier(
         fromaddr = "[e-mail]",
         sendToInterestedUsers = False,
         extraRecipients = ["[e-mail]"],
         subject = "Build %(builder)s Failure",
         mode = "failing",
         builders = ["standalone-build-x86_64"]),
-
+    reporters.MailNotifier(
+        fromaddr = "[e-mail]",
+        sendToInterestedUsers = False,
+        extraRecipients = ["[e-mail]", "[e-mail]"],
+        subject = "Build %(builder)s Failure",
+        mode = "failing",
+        builders = ["clang-loongarch64-linux"]),
 ]
 
 
 from twisted.python import log
 
 # Returns a list of Status Targets. The results of each build will be
 # pushed to these targets. buildbot.plugins reporters has a variety
 # to choose from, including email senders, and IRC bots.
 def getReporters():
     if config.options.getboolean('Master Options', 'is_production'):
         log.msg(">>> getReporters: Production mode. All reporters are registered.")
         return all
     else:
         # Staging buildbot does not report issues.
         log.msg(">>> getReporters: Staging buildbot does not report issues.")
         return []

buildbot/osuosl/master/config/workers.py

return [
         create_worker("linaro-lldb-aarch64-windows", properties={'jobs' : 8}, max_builds=1),
 
         # Windows Server 2016 Intel Xeon(R) Quad 2.30 GHz, 56GB of RAM
         create_worker("win-py3-buildbot", properties={'jobs' : 64}, max_builds=1),
 
         # Windows Server 2016 Intel(R) Xeon(R) CPU @ 2.60GHz, 16 Core(s), 128 GB of RAM
         create_worker("win-mlir-buildbot", properties={'jobs' : 64}, max_builds=1),
 
+        # LoongArch64 Loongson-3C5000L-LL 2GHz, 128GB of RAM
+        # CLFS repo: https://github.com/sunhaiyong1978/CLFS-for-LoongArch
+        create_worker("loongson-loongarch64-clfs-clang-01", properties={'jobs': 32}, max_builds=1),
+
         # MIPS Loongson-3A R4 (Loongson-3A4000) 64-bit little endian (mips64el)
         create_worker("debian-tritium-mips64el", properties={'jobs': 1}, max_builds=1),
 
         # Motorola 68k 32-bit big endian (m68k)
         create_worker("debian-akiko-m68k", properties={'jobs': 1}, max_builds=1),
         create_worker("suse-gary-m68k-cross", properties={'jobs': 4}, max_builds=1),
 
         # POWER7 PowerPC big endian (powerpc64)