[zorg] netbsd: Update project list, rename and move to toolchain category

[mgorny]

Summary

The netbsd builder builds more LLVM projects than the previous
LLVM+Clang+LLDB assumption. Update the project dependencies to ensure
correct blame. Also rename it and move to toolchain group to avoid
people wrongly assuming their change couldn't affect bot with 'lldb'
in its name.

[krytarowski]

I would include compiler-rt in one go as we will enable its builds sooner than later.

[Hahnfeld]

Would it make sense to rename the builder? lldb-amd64-ninja-netbsd8 is just not true based on that list. IIRC there are other unified bots, that test a variety of projects?

[mgorny]

Included compiler-rt

[mgorny]

In D58505#1405719, @Hahnfeld wrote:

Would it make sense to rename the builder? lldb-amd64-ninja-netbsd8 is just not true based on that list. IIRC there are other unified bots, that test a variety of projects?

Apparently Fuchsia has one, and puts it in 'toolchain' category. Maybe we should just name it 'amd64-ninja-netbsd8' or 'netbsd8-amd64-ninja'. Are you maybe aware if rename is something that can be done lightly in zorg, or does it require some extra work?

[krytarowski]

In D58505#1405719, @Hahnfeld wrote:

Would it make sense to rename the builder? lldb-amd64-ninja-netbsd8 is just not true based on that list. IIRC there are other unified bots, that test a variety of projects?

Maybe yes.. but we will keep receiving questions month ago what happened to the older bot. I still keep getting prompts about lldb-amd64-ninja-netbsd7.. the last time like few days ago.

I recommend to keep the name for legacy reasons.

[Hahnfeld]

In D58505#1405760, @mgorny wrote:

In D58505#1405719, @Hahnfeld wrote:

Would it make sense to rename the builder? lldb-amd64-ninja-netbsd8 is just not true based on that list. IIRC there are other unified bots, that test a variety of projects?

Apparently Fuchsia has one, and puts it in 'toolchain' category. Maybe we should just name it 'amd64-ninja-netbsd8' or 'netbsd8-amd64-ninja'. Are you maybe aware if rename is something that can be done lightly in zorg, or does it require some extra work?

When I last added a builder to an existing slave, it was as easy as putting up a patch, committing it to zorg and waiting for the buildbot server to be restarted. Maybe just wait for @gkistanova to chime in...

[krytarowski]

In D58505#1405760, @mgorny wrote:

In D58505#1405719, @Hahnfeld wrote:

Would it make sense to rename the builder? lldb-amd64-ninja-netbsd8 is just not true based on that list. IIRC there are other unified bots, that test a variety of projects?

Apparently Fuchsia has one, and puts it in 'toolchain' category. Maybe we should just name it 'amd64-ninja-netbsd8' or 'netbsd8-amd64-ninja'. Are you maybe aware if rename is something that can be done lightly in zorg, or does it require some extra work?

If we are going to rename it, we probably shall go for netbsd-amd64-tnf.

ninja` is just unimportant detail today it used to be interesting in the time of autoconf vs ninja option (and one worked better than the other), later we might switch to something else and just follow upstream without making non-default options.

netbsd8 soon won't be true once we will upgrade to the -9 branch (or 9.99.x).

tnf specifies hosting by The NetBSD Foundation and a magic shortcut to distinguish it from other potential bots hosted by someone else.

[Hahnfeld]

In D58505#1405772, @krytarowski wrote:

In D58505#1405760, @mgorny wrote:

In D58505#1405719, @Hahnfeld wrote:

Would it make sense to rename the builder? lldb-amd64-ninja-netbsd8 is just not true based on that list. IIRC there are other unified bots, that test a variety of projects?

Apparently Fuchsia has one, and puts it in 'toolchain' category. Maybe we should just name it 'amd64-ninja-netbsd8' or 'netbsd8-amd64-ninja'. Are you maybe aware if rename is something that can be done lightly in zorg, or does it require some extra work?

If we are going to rename it, we probably shall go for netbsd-amd64-tnf.

ninja` is just unimportant detail today it used to be interesting in the time of autoconf vs ninja option (and one worked better than the other), later we might switch to something else and just follow upstream without making non-default options.

netbsd8 soon won't be true once we will upgrade to the -9 branch (or 9.99.x).

tnf specifies hosting by The NetBSD Foundation and a magic shortcut to distinguish it from other potential bots hosted by someone else.

+1 for dropping ninja and the OS version, but I don't think it's common to have a sponsor in the builder name, it's usually somewhere in the slave.

[labath]

+1 for changing the names. With "lldb" in the name, I wouldn't be surprised if people just ignored emails from it because "their patch cannot possibly have anything to do with lldb". Or, OTOH, what has happened to me, I opened the page, expecting to see a relevant failure, only that the failure was due to some libc++ flakes.

[gkistanova]

I'd prefer seeing getLLDBScriptCommandsFactory changed to use LLVMBuildFactory with the properly defined dependencies, rather than another unique scheduler for one or very few builders.
If you would use LLVMBuildFactory , build master will take care of the correct scheduling the changes for you.
You may take a look at zorg/buildbot/builders/FuchsiaBuilder.py as an example of how this could be done easy.

Thanks

Galina

[mgorny]

I'm planning to switch into proper factory once our bot is stable. However, at this moment I really find the cheap script factory convenient for testing changes to the build process, and I expect to be making a big overhaul before we can call it stable.

[krytarowski]

We need to use our current approach with factory... we are actively working on LLDB test targets and we will be attaching compiler-rt next.

[gkistanova]

I made the LLDBBuilder.getLLDBScriptCommandsFactory change for you.

Now you can pass depends_on_projects to the LLDBBuilder.getLLDBScriptCommandsFactory when you instantiate lldb-amd64-ninja-netbsd8 in builders.py.

Something like this should do:

{'name': "lldb-amd64-ninja-netbsd8",
 'slavenames': ["lldb-amd64-ninja-netbsd8"],
 'builddir': "netbsd8",
 'category' : 'lldb',
 'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
                depends_on_projects=[
                        "cfe",
                        "clang-tools-extra",
                        "compiler-rt",
                        "libcxx",
                        "libcxxabi",
                        "libunwind",
                        "lld",
                        "lldb",
                        "llvm",
                        "openmp"
                        "polly",
                    ],
                downloadBinary=False,
                runTest=True)},

[mgorny]

Thank you. I've also taken the opportunity to rename the bot and move it into toolchain group, as suggested by others.

One thing I'm wondering is that there seems not to be a scheduler for that group. Are the builds triggered somehow implicitly, or maybe there are no builds triggered at the moment?

[gkistanova]

Thanks, Michał, for making this change.

Moving the bot to the toolchain group is fine. The group does not have a designated scheduler, so, buildmaster would automatically create an appropriate scheduler based on the specified list of dependencies.
In you case a build would be triggered by any change in any of the projects listed in the depends_on_projects list you have specified.

This is what you are after, right?

Thanks

Galina

[mgorny]

Thank you! Merged now.

zorg/trunk/buildbot/osuosl/master/config/builders.py

return [
          'slavenames': ["lldb-build1-ubuntu-1404"],
          'builddir': "buildWorkingDir",
          'category' : 'lldb',
          'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
                     downloadBinary=False,
                     buildAndroid=False,
                     runTest=True,
                     extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=ON'])},
-        {'name': "lldb-amd64-ninja-netbsd8",
-         'slavenames': ["lldb-amd64-ninja-netbsd8"],
-         'builddir': "netbsd8",
-         'category' : 'lldb',
-         'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
-                    downloadBinary=False,
-                    runTest=True)},
         {'name': "lldb-amd64-ninja-freebsd11",
          'slavenames': ["lldb-amd64-ninja-freebsd11"],
          'builddir': "scratch",
          'category' : 'lldb',
          'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
                     downloadBinary=False,
                     runTest=False)},
         {'name': "lldb-x86_64-fedora",

 # Toolchain builders.
 def _get_toolchain_builders():
       return [
           {'name': "fuchsia-x86_64-linux",
            'slavenames' :["fuchsia-debian-64-us-central1-a-1", "fuchsia-debian-64-us-central1-b-1"],
            'builddir': "fuchsia-x86_64-linux",
            'factory': FuchsiaBuilder.getToolchainBuildFactory(),
            'category': 'fuchsia'},
+          {'name': "netbsd-amd64",
+           'slavenames': ["netbsd-amd64"],
+           'builddir': "netbsd8",
+           'category' : 'netbsd',
+           'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
+                      downloadBinary=False,
+                      runTest=True,
+                      depends_on_projects=[
+                          "cfe",
+                          "clang-tools-extra",
+                          "compiler-rt",
+                          "libcxx",
+                          "libcxxabi",
+                          "libunwind",
+                          "lld",
+                          "lldb",
+                          "llvm",
+                          "openmp",
+                          "polly",
+                      ])},
         ]
 
 # Experimental and stopped builders
 def _get_on_demand_builders():
     return [
         ]
 
 def _get_experimental_scheduled_builders():

zorg/trunk/buildbot/osuosl/master/config/slaves.py

return [
         create_slave("ps4-buildslave1a", properties={'jobs': 64}, max_builds=2),
 
         # Windows 10 Pro x86_64-scei-ps4, 2 x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
         create_slave("ps4-buildslave2", properties={'jobs': 36}, max_builds=1),
 
         # Ubuntu 16.04.3 LTS x86_64-scei-ps4, 2 x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
         create_slave("ps4-buildslave4"),
 
-        # NetBSD 8.x amd64
-        create_slave("lldb-amd64-ninja-netbsd8", properties={'jobs': 8}, max_builds=1),
+        # NetBSD amd64
+        create_slave("netbsd-amd64", properties={'jobs': 8}, max_builds=1),
         # FreeBSD 11.0-CURRENT amd64
         create_slave("lldb-amd64-ninja-freebsd11", properties={'jobs': 3}, max_builds=1),
 
         # FreeBSD 11.0-CURRENT amd64, Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz
         create_slave("freebsd01", properties={'jobs': 3}, max_builds=1),
 
         # Ubuntu 16.04 x86_64, 2 x Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 64GB of RAM
         create_slave("cuda-build-test-01", properties={'jobs': 72}, max_builds=1),

zorg/trunk/buildbot/osuosl/master/config/status.py

return [
             addLogs=False,
             num_lines = 15),
         InformativeMailNotifier(
             fromaddr = "[e-mail]",
             sendToInterestedUsers= False,
             extraRecipients = ["[e-mail]", "[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",
-            builders = ["lldb-amd64-ninja-netbsd7",
-                        "lldb-amd64-ninja-netbsd8"],
+            builders = ["netbsd-amd64"],
             addLogs=False,
             num_lines = 15),
         InformativeMailNotifier(
             fromaddr = "[e-mail]",
             sendToInterestedUsers = False,
             extraRecipients = ["[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",