[libc builder] Use AnnotatedBuilder instead of LLVMBuilder.

[sivachandra]

This part of the fix you wanted me to do in https://reviews.llvm.org/D69655

zorg/buildbot/builders/AnnotatedBuilder.py

46 ↗

(On Diff #227502)

I am not sure if this is correct, but the comment seems to indicate it has to under if clean.

[gkistanova]

Hello Siva,

How AnnotatedBuilder handles clean arg doesn't seem related to adding libc builders. May I suggest removing this change from your patch and make a separate review item if you want to change that behavior, please?

You do not need LibcBuilder factory at all. Just use that AnnotatedBuilder.getAnnotatedBuildFactory in the builders.py for your build configurations. The same is for the z`org/buildbot/builders/libc/libc_vars.py` file, which is not used, unless I'm missing something in your patch.

To add your libc builders you just need to change buildbot/osuosl/master/config/builders.py using nnotatedBuilder.getAnnotatedBuildFactory there, buildbot/osuosl/master/config/slaves.py to add your computer, and buildbot/osuosl/master/config/status.py for notifier. Very close to what you had for these files in https://reviews.llvm.org/D69655.

I think I got the idea of what you are trying to do and can propose a patch if you would like me to. Unless you want to do that by yourself, of course.

[sivachandra]

Use the module libc_vars.

[sivachandra]

In D69727#1733518, @gkistanova wrote:

Hello Siva,

How AnnotatedBuilder handles clean arg doesn't seem related to adding libc builders. May I suggest removing this change from your patch and make a separate review item if you want to change that behavior, please?

I have removed that change from this patch now.

You do not need LibcBuilder factory at all. Just use that AnnotatedBuilder.getAnnotatedBuildFactory in the builders.py for your build configurations. The same is for the z`org/buildbot/builders/libc/libc_vars.py` file, which is not used, unless I'm missing something in your patch.

That was my mistake to not have used libc_vars. Since I want the ability to pass different params to the annotation script, I created this additional module for the names of the env vars that will be set. I created a new builder function for libc also for the same reason (which is to be able to parametrize the builder.)

I think I got the idea of what you are trying to do and can propose a patch if you would like me to. Unless you want to do that by yourself, of course.

If this latest update does not suit you, feel free to land a change for me.

zorg/buildbot/builders/LibcBuilder.py

-from zorg.buildbot.commands.AnnotatedCommand import AnnotatedCommand
-from zorg.buildbot.process.factory import LLVMBuildFactory
+from zorg.buildbot.builders import AnnotatedBuilder
+from zorg.buildbot.builders.libc import libc_vars
 
 LIBC_BUILDER_DIR = "libc_builder"
 LLVM_ZORG = "%s/llvm-zorg" % LIBC_BUILDER_DIR
 ANNOTATED_STEP_RUNNER = (
     "%s/zorg/buildbot/builders/libc/annotated_step_runner.py" % LLVM_ZORG)
 
 
 def getBuildFactory(clean=False, asan=False, timeout=2400):
-    f = LLVMBuildFactory(clean=clean, is_legacy_mode=False,
-                         depends_on_projects=["llvm", "libc"])
-
-    # Get llvm-zorg
-    f.addGetSourcecodeForProject(
-        name='checkout-zorg',
-        project='zorg',
-        src_dir=LIBC_BUILDER_DIR,
-        alwaysUseLatest=True)
-
-    additional_env = {}
-    if clean:
-        additional_env["BUILDBOT_CLOBBER"] = "1"
-
-    annotated_step_cmd = [ANNOTATED_STEP_RUNNER]
-    if clean:
-        annotated_step_cmd.append("--clean")
+    libc_env = {}
     if asan:
-        annotated_step_cmd.append("--asan")
+        libc_env[libc_vars.LIBC_ASAN_BUILD] = 1
 
-    f.addStep(AnnotatedCommand(name='run_annotated_steps',
-                               description=["Run annotated steps"],
-                               command=annotated_step_command,
-                               haltOnFailure=True,
-                               timeout=timeout,
-                               env=additional_env))
-    return f 
+    return AnnotatedBuilder.getAnnotatedBuildFactory(
+        script="libc_annotated_step_runner.py",
+        clean=clean,
+        depends_on_projects=["llvm", "libc"],
+        env=libc_env)

zorg/buildbot/builders/annotated/libc_annotated_step_runner.py

 #! /usr/bin/python
 
 import argparse
 import os
 
 BUILD_DIR = "build"
 MONO_REPO_DIR = "llvm-project"
 
 
 def get_options():
     parser = argparse.ArgumentParser()
     parser.add_argument("--asan", action="store_true",
                         help="Build with address sanitizer enabled.")
     parser.add_argument("--clean", action="store_true",
                         help="Do a clobber/clean build.")
     return parser.parse_args()
 
 
 def run_cmd(cmd, pwd):
     print("Running command: %s" % cmd)
     print("Currently all commands are passthrough.")
     return 0
 
 
 def run_step(step_name, cmd, pwd=None):
     print("@@@BUILD_STEP %s@@@" % step_name)
     retcode = run_cmd(cmd, pwd)
     if retcode == 0:
         print("@@@BUILD_STEP %s@@@" % step_name)
     else:
         print("@@@STEP_FAILURE@@@")
 
 
 def create_build_dir(opts):
     if not os.path.exists(BUILD_DIR):
       run_step("create_build_dir", ["mkdir", "build"])
 
 
 def clean_build(opts):
     should_clobber = (os.environ.get("BUILDBOT_CLOBBER", "0") == "1")
     if opts.clean or should_clobber:
         run_step("clean_build", ["rm", "-rf", "*"], BUILD_DIR)
 
 
 def update_monorepo(opts):
     if not os.path.exists(MONO_REPO_DIR):
         run_step("clone_llvm_monorepo",
                  ["git", "clone", "https://github.com/llvm/llvm-project.git"])
     run_step("update_llvm_monorepo", ["git", "pull", "-r"], MONO_REPO_DIR)
 
     revision = os.environ.get("BUILDBOT_REVISION")
     run_step("checkout_revision", ["git", "checkout", revision], MONO_REPO_DIR)
 
 
 def cmake_configure(opts):
     cmd = ["cmake", "-G", "Ninja", "../llvm_project/llvm",
            "-DLLVM_ENABLE_PROJECTS=libc"]
     if opts.asan:
         cmd.append("-DLLVM_ENABLE_SANITIZER=Address")
     run_step("cmake_configure", cmd, BUILD_DIR)
 
 
 def build_llvmlibc(opts):
     cmd = ["ninja", "llvmlibc"]
     run_step("build_llvmlibc", cmd, BUILD_DIR)
 
 
 def run_unittests(opts):
     cmd = ["ninja", "llvm_libc_unittests"]
     run_step("unittests", cmd, BUILD_DIR)
 
 
 STEPS = [
     create_build_dir,
     clean_build,
     update_monorepo,
     cmake_configure,
     build_llvmlibc,
     run_unittests,
 ]
 
 
 def main():
     opts = get_options()
     for step in STEPS:
         step(opts)
 
 
 if __name__ == "__main__":
     main()

zorg/buildbot/builders/libc/libc_vars.py

+LIBC_ASAN_BUILD = "LIBC_ASAN_BUILD"