Add ziglang builder

[infrandomness]

Fix unwanted indentation modifications and use ZigBuilder

[infrandomness]

• Correct typos

[tstellar]

How long do these tests take to run? Could you add this to the list of release builders too?

[infrandomness]

In D100674#2752636, @tstellar wrote:

How long do these tests take to run?

Generally about an hour on my server

Could you add this to the list of release builders too?

I am not quite sure how to do that;

Also I had questions on about if I needed to rebuild llvm, clang and lld-devel to test; is it possible to use what was built beforehand by previous tests on the same commit ?

buildbot/osuosl/master/config/builders.py

41–56

Those indentation changes are an oopsie, I am reverting them back

[gkistanova]

Hello Brice,

Thanks for the patch.

Please see my comments inline.

buildbot/osuosl/master/config/builders.py

28

This is an unrelated cosmetic change. Please remove.

zorg/buildbot/builders/ZigBuilder.py

8

Is there anything missing in zorg.buildbot.commands.CmakeCommand for your usage?

9

This should be

from zorg.buildbot.process.factory import LLVMBuildFactory

15

This is more for historical reasons and for backward compatibility. It is better to use jobs property instead, which is defined for your worker, as the number of jobs is more related to a particular worker hardware than to the builder itself.

81

It seems zorg.buildbot.commands.NinjaCommand should work for you here and in all other places where you run ninja. It will handle the jobs and loadaverage properties properly, and specifies a specific output format.

94

Is this a correct description? The command suggests that you run zig, not ninja.

[tstellar]

In D100674#2753238, @infrandomness wrote:

In D100674#2752636, @tstellar wrote:

How long do these tests take to run?

Generally about an hour on my server

Could you add this to the list of release builders too?

I am not quite sure how to do that;

There are examples in buildbot/osuosl/master/config/release_builders.py

Also I had questions on about if I needed to rebuild llvm, clang and lld-devel to test; is it possible to use what was built beforehand by previous tests on the same commit ?

I'm not sure I understand the question. You should have to rebuild llvm, clang, and lld for every commit.

[gkistanova]

Also I had questions on about if I needed to rebuild llvm, clang and lld-devel to test; is it possible to use what was built beforehand by previous tests on the same commit ?

If I get the question right, you will build incrementally unless you remove the build directory content for each build, which specified clean=True, does.
Remove that and you will reuse the results of the previous build with all pros and cons.

[infrandomness]

In D100674#2756183, @gkistanova wrote:

Also I had questions on about if I needed to rebuild llvm, clang and lld-devel to test; is it possible to use what was built beforehand by previous tests on the same commit ?

If I get the question right, you will build incrementally unless you remove the build directory content for each build, which specified clean=True, does.
Remove that and you will reuse the results of the previous build with all pros and cons.

Oh, right, will apply new changes

[infrandomness]

In D100674#2755840, @tstellar wrote:

In D100674#2753238, @infrandomness wrote:

In D100674#2752636, @tstellar wrote:

How long do these tests take to run?

Generally about an hour on my server

Could you add this to the list of release builders too?

I am not quite sure how to do that;

There are examples in buildbot/osuosl/master/config/release_builders.py

Will do

[gkistanova]

If this is fine with you, let's finish this patch first, stage the builder to make sure it is reliably green, then you could move it to the production build bot and add to the release_builders.py.

[infrandomness]

• Remove unrelated cosmetics
• Fix imports
• Remove jobs
• Replace the factories

[infrandomness]

Fail

[infrandomness]

I give up

buildbot/osuosl/master/config/builders.py

 from zorg.buildbot.builders import ABITestsuitBuilder
 from zorg.buildbot.builders import ClangLTOBuilder
 from zorg.buildbot.builders import UnifiedTreeBuilder
 from zorg.buildbot.builders import AOSPBuilder
 from zorg.buildbot.builders import AnnotatedBuilder
 from zorg.buildbot.builders import LLDPerformanceTestsuite
 from zorg.buildbot.builders import FuchsiaBuilder
 from zorg.buildbot.builders import XToolchainBuilder
+from zorg.buildbot.builders import ZigBuilder
 
 from zorg.buildbot.builders import HtmlDocsBuilder
 from zorg.buildbot.builders import DoxygenDocsBuilder
 
+
 reload(HtmlDocsBuilder)
 reload(DoxygenDocsBuilder)
 
 
 all = [
 
 # Clang fast builders.
 
     {'name' : "clang-x86_64-debian-fast",
     'tags'  : ["clang", "fast"],
     'collapseRequests': False,
     'workernames':["gribozavr4"],
     'builddir':"clang-x86_64-debian-fast",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     llvm_srcdir="llvm.src",
                     obj_dir="llvm.obj",
                     clean=True,
                     depends_on_projects=['llvm','clang','clang-tools-extra','compiler-rt'],
                     extra_configure_args=[
                         "-DCOMPILER_RT_BUILD_BUILTINS:BOOL=OFF",
                         "-DCOMPILER_RT_BUILD_SANITIZERS:BOOL=OFF",
                         "-DCOMPILER_RT_BUILD_XRAY:BOOL=OFF",
                         "-DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF",
                         "-DCMAKE_C_FLAGS=-Wdocumentation -Wno-documentation-deprecated-sync",
                         "-DCMAKE_CXX_FLAGS=-std=c++11 -Wdocumentation -Wno-documentation-deprecated-sync",
                     ],
                     env={
                         'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin',
                         'CC': 'ccache clang', 'CXX': 'ccache clang++', 'CCACHE_CPP2': 'yes',
                     })},
 
     {'name' : "clang-x86_64-debian-new-pass-manager-fast",
     'tags'  : ["clang", "fast"],
     'workernames': ["gribozavr4"],
     'builddir': "clang-x86_64-debian-new-pass-manager-fast",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     llvm_srcdir="llvm.src",
                     obj_dir="llvm.obj",
'factory': UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
              "-DLLVM_CCACHE_BUILD=ON",
              "-DLLVM_USE_SANITIZER=Thread",
              "-DCMAKE_BUILD_TYPE=Release",
              "-DCLANGD_ENABLE_REMOTE=ON",
              "-DLLVM_ENABLE_ASSERTIONS=ON",
              "-DGRPC_INSTALL_PATH=/usr/local/lib/grpc"
          ])},
 
-
-]
+    {'name': "ziglang-carbon",
+     'tags': ["ziglang"],
+     'workernames': ["ziglang-carbon-linux-x86_64"],
+     'builddir': "ziglang-carbon-root",
+     'factory': ZigBuilder.getZiglangToolchainBuildFactory(
+         clean=True,
+         test=True,
+         env={
+             'CC': 'clang', 'CXX': 'clang++', 'LD': 'lld',
+         }
+     )}
+]
 No newline at end of file

buildbot/osuosl/master/config/workers.py

return [
         # Ubuntu x86_64
         create_worker("thinlto-x86-64-bot1", properties={'jobs': 64}, max_builds=1),
 
         # Ubuntu 20.04 on AWS, x86_64 PS4 target
         create_worker("sie-linux-builder", properties={'jobs': 40}, max_builds=1),
 
         # XCore target, Ubuntu 20.04 x64 host
         create_worker("xcore-ubuntu20-x64", properties={'jobs': 4}, max_builds=1),
+
+        # Ziglang worker
+        create_worker("ziglang-carbon", properties={'jobs': 24}, max_builds=1),
         ]

zorg/buildbot/builders/ZigBuilder.py

+import json
+import urllib.request
+
+from buildbot.plugins import steps
+from buildbot.steps.shell import ShellCommand, WarningCountingShellCommand
+from buildbot.process.properties import WithProperties, Property
+from buildbot.commands.CmakeCommand import CmakeCommand
+from buildbot.process.factory import LLVMBuildFactory
+
+
+def getZiglangToolchainBuildFactory(
+        clean=False,
+        env=None,
+        jobs="%(jobs)s",
+        extra_configure_args=None,
+        extra_cmake_args=None,
+        config='Release',
+        install=False,
+        testTimeout=2400,
+        test=False):
+    build_env = {
+        "TERM": "dumb",
+    }
+
+    if env is not None:
+        build_env.update(env)
+
+    llvm_src_dir = "llvm"
+    llvm_build_dir = "build"
+
+    f = LLVMBuildFactory(
+        depends_on_projects=[
+            "llvm",
+            "clang",
+            "lld"
+        ],
+        obj_dir=llvm_build_dir)
+
+    f.addGetSourcecodeSteps()
+
+    build_cmd = ["ninja"]
+    install_cmd = ["ninja", "install"]
+    test_cmd = ["./build/zig", "build", "test"]
+
+    if jobs:
+        build_cmd.append(WithProperties("-j%s" % jobs))
+        install_cmd.append(WithProperties("-j%s" % jobs))
+        test_cmd.append(WithProperties("-j%s" % jobs))
+
+    cleanBuildRequested = lambda step: clean or step.build.getProperty("clean",
+                                                                       default=step.build.getProperty("clean_obj"))
+
+    f.addStep(steps.RemoveDirectory(
+        name="clean "+llvm_build_dir,
+        dir=llvm_build_dir,
+        haltOnFailure=False,
+        doStepIf=cleanBuildRequested
+    ))
+
+    zig_rel_src_dir = LLVMBuildFactory.pathRelativeTo(f.llvm_srcdir, f.obj_dir)
+    cmake_options = [
+        "-G", "Ninja",
+        "-DCMAKE_BUILD_TYPE=" + config,
+        "-DCMAKE_INSTALL_PREFIX=../install",
+        "-DLLVM_ENABLE_PROJECTS=%s" % ";".join(f.depends_on_projects),
+    ]
+
+    if extra_configure_args:
+        cmake_options += extra_cmake_args
+
+    f.addStep(CmakeCommand(
+        name="cmake-configure",
+        description=["cmake configure"],
+        haltOnFailure=True,
+        path=zig_rel_src_dir,
+        env=env,
+        workdir=llvm_build_dir
+    ))
+
+    f.addStep(WarningCountingShellCommand(
+        name='build',
+        command=build_cmd,
+        haltOnFailure=True,
+        description='ninja build',
+        workdir=llvm_build_dir,
+        env=env
+    ))
+
+    f.addStep(ShellCommand(
+        name='test',
+        command=test_cmd,
+        timeout=testTimeout,
+        description='ninja test',
+        workdir=llvm_build_dir,
+        doStepIf=bool(test),
+        env=env))
+
+    f.addStep(ShellCommand(
+        name="install",
+        command=install_cmd,
+        description="ninja install",
+        workdir=llvm_build_dir,
+        doStepif=bool(install),
+        env=env
+    ))
+
+    zig_src_dir = ["zig"],
+    zig_build_dir = ["build"],
+    zig_test_cmd = [zig_build_dir, "/zig", "build", "test"]
+
+    json_text = urllib.request.urlopen("https://ziglang.org/download/index.json").read().decode()
+    src_url = json.load(json_text)['master']['src']['tarball']
+
+    f.addStep(ShellCommand(
+        name="fetch-src",
+        command=["curl", "-Sf", "-o", "zig.tar.xz", src_url],
+        description=["download", "zig source code"],
+        workdir=zig_src_dir,
+        haltOnFailure=True
+    ))
+
+    f.addStep(ShellCommand(
+        name="extract-src",
+        command=["tar", " xvf", "zig.tar.xz"],
+        description=["extract", "zig source code"],
+        workdir=zig_src_dir
+    ))
+
+    f.addStep(CmakeCommand(
+        name="cmake-configure",
+        description=["cmake configure"],
+        haltOnFailure=True,
+        env=env,
+        options=cmake_options.pop(),
+        workdir=zig_build_dir
+    ))
+
+    f.assStep(ShellCommand(
+        name="test-zig",
+        command=zig_test_cmd,
+        timeOut=testTimeout,
+        workdir=zig_src_dir,
+        description=""
+    ))
 No newline at end of file