make AnnotatedBuilder use LLVMBuildFactory and depends_on_projects

[inglorion]

Summary

This makes AnnotatedBuilder derive from LLVMBuildFactory
instead of the standard buildbot factory. This means it will have
the same properties and methods as the other builders - in particular,
depends_on_projects, which is used for scheduling.

[rnk]

Do you think we should use LLVMBuildFactory.addSVNSteps to drive the checkout process, or is it better to keep that as part of the annotator script? Keeping it in the script lets us add more repos without doing buildmaster restarts, which is nice, but if we do that, then the scheduling logic gets out of sync.

[inglorion]

What do you mean by "the scheduling logic gets out of sync"?

[rnk]

In D34065#778428, @inglorion wrote:

What do you mean by "the scheduling logic gets out of sync"?

Suppose ThinLTO develops a dependency on libprofile in compiler-rt. If we update the annotator script to check out compiler-rt without updating the depends_on_projects list on the buildmaster, then builds will not be scheduled after compiler-rt changes. By using depends_on_projects, we're moving the knowledge of what source to check out back into the build master config. If adding a new subproject dependency requires a buildmaster restart, maybe we should go back to using the master side SVN checkout steps.

[inglorion]

I see. Yes, it's unfortunate that this essentially repeats the list of projects in multiple places. On the other hand, I like having as much as possible in the part that can be modified without requiring a buildmaster reload, which would speak for leaving the checkout code in the builder. This is also what the sanitizer builders do. They trigger on changes to any of llvm, cfe, compiler-rt, libcxx, libcxxabi, libunwind, and lld, which we could also do here. It's basically triggering on everything, which is safe. Theoretically, it might increase resource usage a bit, but the ThinLTO builder as-is is basically always building, so the number of builds triggered and the amount of work the worker does should be the same. Would you like me to make that change?

[rnk]

In D34065#779693, @inglorion wrote:

I see. Yes, it's unfortunate that this essentially repeats the list of projects in multiple places. On the other hand, I like having as much as possible in the part that can be modified without requiring a buildmaster reload, which would speak for leaving the checkout code in the builder. This is also what the sanitizer builders do. They trigger on changes to any of llvm, cfe, compiler-rt, libcxx, libcxxabi, libunwind, and lld, which we could also do here. It's basically triggering on everything, which is safe. Theoretically, it might increase resource usage a bit, but the ThinLTO builder as-is is basically always building, so the number of builds triggered and the amount of work the worker does should be the same. Would you like me to make that change?

Not really. Let's leave it as is, I guess.

buildbot/osuosl/master/config/builders.py

return [
          'builddir' : "clang-with-thin-lto-ubuntu",
          'factory': ClangLTOBuilder.getClangWithLTOBuildFactory(jobs=16, lto='thin'),
          'category'   : 'lld'},
 
         {'name': "clang-with-thin-lto-windows",
          'slavenames' :["windows-lld-thinlto-1"],
          'builddir': "clang-with-thin-lto-windows",
          'factory': AnnotatedBuilder.getAnnotatedBuildFactory(
-             script="clang-with-thin-lto-windows.py"),
+             script="clang-with-thin-lto-windows.py",
+             depends_on_projects=['llvm', 'clang', 'lld']),
          'category'   : 'lld'},
 
         {'name' : "clang-lld-x86_64-2stage",
          'slavenames' : ["am1i-slv1", "am1i-slv3"],
          'builddir' : "clang-lld-x86_64-2stage",
          'factory': UnifiedTreeBuilder.getCmakeWithNinjaMultistageBuildFactory(
                                   depends_on_projects=['llvm', 'clang', 'lld']),
          'category'   : 'lld'},

zorg/buildbot/builders/AnnotatedBuilder.py

 import os
 
 import buildbot
-import buildbot.process.factory
 from buildbot.process.properties import WithProperties
 from buildbot.steps.shell import SetProperty, ShellCommand
 from buildbot.steps.source import SVN
 from zorg.buildbot.commands.AnnotatedCommand import AnnotatedCommand
+from zorg.buildbot.process.factory import LLVMBuildFactory
 
 
 def getAnnotatedBuildFactory(
     script,
     clean=False,
+    depends_on_projects=None,
     env=None,
     timeout=1200):
     """
     Returns a new build factory that uses AnnotatedCommand, which
     allows the build to be run by version-controlled scripts that do
     not require a buildmaster restart to update.
     """
 
-    f = buildbot.process.factory.BuildFactory()
+    f = LLVMBuildFactory(
+        depends_on_projects=depends_on_projects,
+        llvm_srcdir='llvm.src')
 
     if clean:
       f.addStep(SetProperty(property='clean', command='echo 1'))
 
     # We normally use the clean property to indicate that we want a
     # clean build, but AnnotatedCommand uses the clobber property
     # instead. Therefore, set clobber if clean is set to a truthy
     # value.  This will cause AnnotatedCommand to set