Builder for LLVM Embedded Toolchain for Arm

[michaelplatings]

Summary

Hi Galina,

My team at Arm are interested in adding a buildbot for LLVM Embedded Toolchain for Arm

The LLVM Embedded Toolchain for Arm project is essentially LLVM + picolibc, built for ARM and AArch64 targets. In the short to medium term we'd like to set up staging builds and see what kind of failures it alerts us to. We anticipate that it will identify changes in LLVM that break llvm-test-suite on Arm targets. Once we get to a state where the buildbot reliably catches such cases, we'd like to promote the buildbot from staging to main.

My main question at this stage: is the overall approach in the change below acceptable to you? Most of the tasks are performed by a single call to the project's build.py. If you're OK with the general approach I'll continue with it and add changes to buildbot/osuosl/master/config

Many thanks

[DavidSpickett]

@gkistanova some background here is the project (https://github.com/ARM-software/LLVM-embedded-toolchain-for-Arm) is being done by Arm but is public facing and intended to be shared with others. The long term plan is for Linaro to host this bot (silent or otherwise).

The one issue I thought you/the community might have is storing a config for a build that is llvm plus something else, not pure llvm. Most existing bots are only using llvm repos.

One comparable builder might be fuchsia-x86_64-linux which dowloads the Fuchsia sdk as part of the build, so it's not unheard of. What do you think?

zorg/buildbot/builders/LLVMEmbeddedToolchainForArmBuilder.py

28

What's the story here, is there a reason to mirror (if that's not just a copy paste error)?

I see https://github.com/picolibc/picolibc is that not the right place?

[michaelplatings]

zorg/buildbot/builders/LLVMEmbeddedToolchainForArmBuilder.py

28

You're right that's a copy-paste error.

[michaelplatings]

@gkistanova I'd really love to get your thoughts on this please

[gkistanova]

Thanks for the background, Michael.

I’m still trying to wrap my mind around this. Sorry for being slow.

You are going to build a downstream toolchain along with the 3rd party libc by a custom script. What do you expect people do if this builder became red on some LLVM commit?

If you want these build failures to be actionable, all envvars and build steps should be visible in logs along with the corresponding printouts. Expecting all LLVM developers being familiar with your toolchain and its build script does not sound too reasonable, does it? You may consider using annotated builder instead if changing that build script is an option. You also have to make this builder fast in this case. What is the expected response time?

Making this builder actionable for the LLVM community effectively means we would need to support those external dependencies up to some degree. At least far enough to figure out a potential problem not being there + a clear escalation path if any bug fix is required there. Could you point me to the RFC where these were discussed and agreed, please?

If this is something not directly actionable by the LLVM community, i.e. on a build failure you will do the troubleshooting and debugging, fixing the external dependencies if an issue is there, or providing a bug report with a reproducer in case of LLVM issue, then it looks more like a part of the ARM embedded toolchain CI rather than a part of the LLVM public buildbot.

I might still be missing something here, though.

[michaelplatings]

Thanks Galina, that's great feedback and I really appreciate it.

You are going to build a downstream toolchain along with the 3rd party libc by a custom script.

Yes, but as far as possible the toolchain will be vanilla LLVM. We know that divergence from upstream is very costly so we only patch LLVM when we must. (The patch is currently small)
The ultimate dream is that LLVM itself could work as a full toolchain, and LLVM Embedded Toolchain for Arm becomes obsolete, but there's a way to go before that could be possible.

What do you expect people do if this builder became red on some LLVM commit?

Initially, nothing. We'll only set up staging builds. We don't yet know what kind of feedback these builds will produce.
We hope that the builder will identify genuine bugs in LLVM. If so then we'll raise the question of whether the builder can be promoted to main.
However we may find that it mostly picks up issues with the C library integration. In that case the builder may stay on staging indefinitely. I understand from this page that would be OK ("builders can run indefinitely on the staging buildmaster").

If you want these build failures to be actionable, all envvars and build steps should be visible in logs along with the corresponding printouts.

Thanks, I'll make sure that requirement is met, regardless of whether we're on staging or main.

Expecting all LLVM developers being familiar with your toolchain and its build script does not sound too reasonable, does it? You may consider using annotated builder instead if changing that build script is an option.

I'm inclined to agree with you. Ideally the build would just be a matter of running cmake with the right arguments. However we need to at least patch the LLVM sources, which is immediately a divergence from the usual LLVM workflow. So I think that some knowledge of the specifics of LLVM Embedded Toolchain for Arm is unfortunately unavoidable to build it, and for the same reason I don't think annotated_builder will work. However I think we could simplify the build script a lot. I see FuchsiaBuilder has custom steps, and we could do something similar if the steps to set up the build were sufficiently simplified. Would something like FuchsiaBuilder be acceptable?

What is the expected response time?

I hope to keep it under 15 minutes on average.

Could you point me to the RFC where these were discussed and agreed, please?

There's no such RFC. Would you agree that the time to raise an RFC would be if/when we have evidence from the staging builds that the builder would be useful to the wider community?

[michaelplatings]

@gkistanova I've updated the builder to use CMake directly. Is this something you'd be more comfortable with putting in staging?

[michaelplatings]

@gkistanova ping

[michaelplatings]

ping

[michaelplatings]

Raised an RFC: https://discourse.llvm.org/t/rfc-testing-of-newlib-picolibc/66509

[gkistanova]

Hi Michael.

Thanks for raising the RFC!

Assuming it will go well, the only missing piece is to build the LLVM mainline instead of https://github.com/ARM-software/LLVM-embedded-toolchain-for-Arm.git.

[michaelplatings]

Hi Galina, this change does build LLVM mainline (see line 42). LLVM-embedded-toolchain-for-Arm.git is not a fork of LLVM, it's just a build script really. I've updated the RFC to try to clarify it.
Thanks

[arichardson]

zorg/buildbot/builders/LLVMEmbeddedToolchainForArmBuilder.py

47

I think it might make sense to pin this to a known good specific commit so that picolibc changes can't cause a libc++ builder failure?

[michaelplatings]

zorg/buildbot/builders/LLVMEmbeddedToolchainForArmBuilder.py

47

Agreed. That's certainly a change that would have to be made prior to moving the builder out of staging. My intention is that the commit ID would be stored in the LLVM-embedded-toolchain-for-Arm repo and read from there. Some automation would need to be set up to automatically update the commit ID periodically. Until then, and while the builder is only in staging, I'd like to check out the latest commit.

[gkistanova]

In D131077#3921152, @michaelplatings wrote:

Hi Galina, this change does build LLVM mainline (see line 42).

Sorry for not being more clear.

Besides of the main builder purpose, it should also do the following

1. Subscribe for commits in LLVM projects it depends on (this is done by the depends_on_projects arg of the LLVMBuildFactory),
2. Build a particular commit the buildbot requested to build, not the top of the trunk always,
3. Respect the clean and clean_obj params to force the source code and build artifacts and temp files cleaning for a particular build. This is Ok to always remove previous build files, but making a clean checkout every time is a significant time waste.

This should be either coded or reused by using an existing build factory as a starting point for yours.

You can take FuchsiaBuilder.py as a good example. It uses UnifiedTreeBuilder for dialing with commits and LLVM source code, then adds extra build steps specific to the Fuchsia toolchain.

You may also want to look at XToolchainBuilder.py, which builds and tests a cross-toolchain based on cmake cache file. Maybe it already does almost everything you are after?

Unless I'm missing something, the extra patches, you apply to llvm-project and to picolibc, are temporary, and the issues they address will be fixed in the mainline projects sooner rather than later. This will simplify the matter for sure. Like those steps of applying the patches will go away.

zorg/buildbot/builders/LLVMEmbeddedToolchainForArmBuilder.py

103

Can this be just a part of the build environment? Something you would install once along with all other prerequisites, instead of installing for each build?

zorg/buildbot/builders/LLVMEmbeddedToolchainForArmBuilder.py

+from buildbot.plugins import steps
+from buildbot.process.factory import BuildFactory
+from buildbot.steps.shell import ShellCommand
+
+from zorg.buildbot.commands.CmakeCommand import CmakeCommand
+from zorg.buildbot.commands.NinjaCommand import NinjaCommand
+
+
+def getLLVMEmbeddedToolchainForArmBuildFactory(
+    test=True,
+    env=None,  # Environmental variables for all steps.
+    extra_configure_args=None,
+    **kwargs
+):
+    # Prepare environmental variables. Set here all env we want everywhere.
+    merged_env = {
+        "TERM": "dumb",  # Make sure Clang doesn't use color escape sequences.
+    }
+    if env is not None:
+        # Overwrite pre-set items with the given ones, so user can set anything.
+        merged_env.update(env)
+
+    f = BuildFactory()
+
+    # Check out & patch repos.
+
+    f.addStep(
+        steps.Git(
+            name="Check out LLVM-embedded-toolchain-for-Arm",
+            repourl="https://github.com/ARM-software/LLVM-embedded-toolchain-for-Arm.git",
+            progress=True,
+            workdir="LLVM-embedded-toolchain-for-Arm",
+            retryFetch=True,
+            clobberOnFailure=True,
+            method="fresh",
+        )
+    )
+
+    for repo, repourl, patch_file in [
+        (
+            "llvm-project",
+            "https://github.com/llvm/llvm-project.git",
+            "llvm-HEAD.patch",
+        ),
+        (
+            "picolibc",
+            "https://github.com/picolibc/picolibc.git",
+            "picolibc-HEAD.patch",
+        ),
+    ]:
+        f.addStep(
+            steps.Git(
+                name="Check out %s" % repo,
+                repourl=repourl,
+                progress=True,
+                workdir=repo,
+                retryFetch=True,
+                clobberOnFailure=True,
+                method="fresh",
+            )
+        )
+        f.addStep(
+            ShellCommand(
+                name="Reset %s" % repo,
+                command=["git", "reset", "--hard"],
+                workdir=repo,
+                haltOnFailure=True,
+                env=merged_env,
+            )
+        )
+        f.addStep(
+            ShellCommand(
+                name="Patch %s" % repo,
+                command=[
+                    "git",
+                    "apply",
+                    "../LLVM-embedded-toolchain-for-Arm/patches/%s"
+                    % patch_file,
+                ],
+                workdir=repo,
+                haltOnFailure=True,
+                env=merged_env,
+            )
+        )
+
+    # Clean build directory.
+    f.addStep(steps.RemoveDirectory(dir="build"))
+    f.addStep(steps.MakeDirectory(dir="build"))
+
+    # Install Meson.
+    f.addStep(
+        ShellCommand(
+            name="create-python-venv",
+            command=["python3", "-m", "venv", "venv"],
+            haltOnFailure=True,
+            env=merged_env,
+        )
+    )
+
+    f.addStep(
+        ShellCommand(
+            name="install-meson",
+            command=["./venv/bin/pip", "install", "meson==0.61.5"],
+            haltOnFailure=True,
+            env=merged_env,
+        )
+    )
+
+    # Configure.
+    cmake_options = (
+        [] if extra_configure_args is None else extra_configure_args[:]
+    )
+
+    CmakeCommand.applyRequiredOptions(
+        cmake_options,
+        [
+            ("-G", "Ninja"),
+            ("-DFETCHCONTENT_SOURCE_DIR_LLVMPROJECT=", "../llvm-project"),
+            ("-DFETCHCONTENT_SOURCE_DIR_PICOLIBC=", "../picolibc"),
+            (
+                "-DMESON_EXECUTABLE=",
+                util.Interpolate("%(prop:builddir)s/venv/bin/meson"),
+            ),
+        ],
+    )
+
+    f.addStep(
+        CmakeCommand(
+            name="Configure",
+            options=cmake_options,
+            path="../LLVM-embedded-toolchain-for-Arm",
+            haltOnFailure=True,
+            description=["configure"],
+            workdir="build",
+            env=merged_env,
+            doStepIf=FileDoesNotExist("CMakeCache.txt"),
+        )
+    )
+
+    # Build.
+    f.addStep(
+        NinjaCommand(
+            name="Build",
+            targets=["llvm-toolchain"],
+            haltOnFailure=True,
+            description=["build"],
+            workdir="build",
+            env=merged_env,
+        )
+    )
+
+    # Test.
+    f.addStep(
+        NinjaCommand(
+            name="Test",
+            targets=["check-llvm-toolchain"],
+            haltOnFailure=True,
+            description=["check"],
+            workdir="build",
+            env=merged_env,
+            doStepIf=test,
+        )
+    )
+
+    return f