Add BOLTBuilder: BOLT custom builder

[Amir]

Summary

The builder is based on UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory, but adds one extra checkout step
between getLLVMBuildFactoryAndSourcecodeSteps and addCmakeSteps/addNinjaSteps.
Builder checks out binary tests from another repo, currently at https://github.com/rafaelauler/bolt-tests.
The plan is to transfer the repository over to llvm organization in the future.

The configuration is tested with a temporary build master, which points to the current BOLT repo instead of the monorepo.

This diff is a part of BOLT upstreaming:
https://lists.llvm.org/pipermail/llvm-dev/2021-December/154290.html

[gkistanova]

Thanks, Amir!

The patch looks good with one thing:

zorg/buildbot/builders/BOLTBuilder.py

39

This would remove the source code every time when only obj files removal is requested. Not sure if you want this.

Most likely you want to remove the source code only when clean build property is evaluated as true. Please see the discussion of the similar issue in https://reviews.llvm.org/D107193.

[Amir]

Addressed review suggestions (cleanBuildRequestedByProperty)

[gkistanova]

zorg/buildbot/builders/BOLTBuilder.py

49

With doStepIf=cleanBuildRequestedByProperty here your builder would not receive any new commits to the test-suite, unless someone manually requires a clean build. Are you sure this is what you want?

Running git unconditionally here should work fine. It would pull only new commits if any, unless the source code got deleted or something went really bad with the checkout. Should serve the need, I think.

I did a quick check and see some relatively recent commits to the BOLT test suite.

[Amir]

Always checkout bolt-tests repo

[Amir]

zorg/buildbot/builders/BOLTBuilder.py

49

Thanks for pointing this out! Yes, it makes sense to pull new commits unconditionally.

[gkistanova]

LGTM

[Amir]

@gkistanova, thank you for the review! Provided there are no objections from other reviewers, can you please land this diff for me? I don't have commit access to LLVM repos yet.

[gkistanova]

Sure, I commit this for you.

zorg/buildbot/builders/BOLTBuilder.py

+from buildbot.plugins import steps
+from zorg.buildbot.commands.CmakeCommand import CmakeCommand
+from zorg.buildbot.builders.UnifiedTreeBuilder import getLLVMBuildFactoryAndSourcecodeSteps, addCmakeSteps, addNinjaSteps
+from zorg.buildbot.process.factory import LLVMBuildFactory
+
+def getBOLTCmakeBuildFactory(
+           clean = False,
+           bolttests = False,
+           extra_configure_args = None,
+           env = None,
+           **kwargs):
+
+    if env is None:
+        env = dict()
+
+    bolttests_dir = "bolt-tests"
+
+    cleanBuildRequested = lambda step: clean or step.build.getProperty("clean", default=step.build.getProperty("clean_obj"))
+
+    checks = ['check-bolt']
+    extra_steps = []
+
+    f = getLLVMBuildFactoryAndSourcecodeSteps(
+            depends_on_projects=['bolt'],
+            **kwargs) # Pass through all the extra arguments.
+
+    if bolttests:
+        checks += ['check-large-bolt']
+        extra_configure_args += [
+            '-DLLVM_EXTERNAL_PROJECTS=bolttests',
+            '-DLLVM_EXTERNAL_BOLTTESTS_SOURCE_DIR=' + LLVMBuildFactory.pathRelativeTo(bolttests_dir, f.monorepo_dir),
+            ]
+        # Clean checkout of bolt-tests if cleanBuildRequested
+        f.addSteps([
+            steps.RemoveDirectory(name="BOLT tests: clean",
+                dir=bolttests_dir,
+                haltOnFailure=True,
+                warnOnFailure=True,
+                doStepIf=cleanBuildRequested),
+
+            steps.Git(name="BOLT tests: checkout",
+                description="fetching",
+                descriptionDone="fetch",
+                descriptionSuffix="BOLT Tests",
+                repourl='https://github.com/rafaelauler/bolt-tests.git',
+                workdir=bolttests_dir,
+                doStepIf=cleanBuildRequested,
+                alwaysUseLatest=True),
+            ])
+
+    # Some options are required for this build no matter what.
+    CmakeCommand.applyRequiredOptions(extra_configure_args, [
+        ('-G',                      'Ninja'),
+        ])
+
+    addCmakeSteps(
+        f,
+        cleanBuildRequested=cleanBuildRequested,
+        extra_configure_args=extra_configure_args,
+        obj_dir=None,
+        env=env,
+        **kwargs)
+
+    addNinjaSteps(
+        f,
+        targets = ['llvm-bolt'],
+        checks=checks,
+        env=env,
+        **kwargs)
+
+    return f