[zorg] Support Windows/Linux Clang CMake builds with unified builder.

[rengolin]

Summary

This adds the basic ability to build Clang with cmake on both Windows (MSVC), and Linux. The default options work as the current getClangCMakeBuilder providing Linux builds.

CMake generators can be selected. Ninja, MSVC, and Unix Makefiles generators have been tested.

Windows and Linux builds have also been tested. Example builders for Hexagon show options for both Windows and Linux.

This might be an interim step. By adding a dictionary of llvm projects and SVN/Git parameters for fetching an LLVM based source tree, and general this becomes a generic LLVM CMake builder with the user passing a list of projects desired in the build.

[rfoos]

zorg/buildbot/builders/ClangBuilder.py

28

This should be a unified getLLVMSource using a dictionary for SVN arguments/tree locations. The use passing a list of projects for update.

59

This particular one is used by an lldb experimental builder and should be removed.

508

This is original behavior but allows the caller to pass an entire step 2 definition.

The only exception is the stage1/stage2 framework for the built compiler(s).

563

This section needs to support the jobs property.

[rengolin]

Overall, looks good. I think we're getting there. I'll wait until you finish your refactoring based on your comments and look at my comments.

To actually put this in production, I'd be a lot more confident if we could make it run on a staging environment before changing it. So, whenever you're happy with it, I'll apply the patch on my local buildmaster and set it to run with my boards. Other interested parties can do the same, and when everyone is happy, we push and apply.

cheers,
--renato

zorg/buildbot/builders/ClangBuilder.py

508

Maybe better as a dictionary? Like:

build { stage : 1, { cmd = "ninja", install = cmd + " install", check = cmd + " check-all" }
          stage : 2, { cmd = "ninja", install = cmd + " install", check = cmd + " check-all" } }

or whatever Python would do for something like that. :)

539

getClangSource already checks out clang-tools-extra

595

I'm not sure why you set the property here, since this is normally a config option, not a run-time one.

But this could be potentially good if we can detect failure on previous runs and automatically clean after every failure, to make sure it's not an infrastructure failure. Not for this iteration, though.

[rfoos]

I completely agree with several other testers.

I have a Windows/Linux staging environment set up on my servers, but
this will catch all of the things I might miss.

--rick

[gkistanova]

Thanks for working on this, Rick!

I like the changes you made and how they are done.
Please also see my comments in-line.

Thanks

Galina

zorg/buildbot/builders/ClangBuilder.py

41

There is checkout_clang_tools_extra getClangCMakeBuildFactory param which you are ignoring here. If you want to check out always, then you don't need the checkout_clang_tools param, and vice versa.

Though, if you up to introducing a list of components to check out, this issue will be automatically solved by that change.

490

Maybe rename 'Visual' to 'VisualStudio' or 'VC'?

508

Ordered list would do better, I think.

[ 
{ name="stage 1", source=[<list of (components and directories tuples)>], cmd = "ninja", install = cmd + " install", check = cmd + " check-all" },
{ name="stage 2", cmd = "ninja", install = cmd + " install", check = cmd + " check-all" },
...
]

As a next re-factoring pass we shall standardize the build parameters names to have the dictionary as much common for all the builders as practical.

551

No need in "is not None" check here.

if slave_envCmd:

would give you exactly what you want. We do not want to run an empty command.

562

Do you want to have 0 as a valid value? If you do not, then checking it as

if jobs:

would do.

I'm sure you have figured this by now that you are loosing the "job" property here. Could you reconcile the param with the property instead, please?

565

The same is here. It would be great to have a way to use the "loadaverage" property similar to the "job" one. Not a show stopper at this point, though.

595

Renato: This way someone could define clean=True property in the builder Web UI and rebuild a particular incremental build cleanly. Very nice feature to have.

How about renaming "doCleanIf" to something more descriptive? Like "cleanBuildRequested" or similar?

659

I think we should always do a clean build on the stage 2. It does not make much sense to use intermediate files built by a different compiler (the compiler on the stage 1 has been changed, right?).

What do you think?

[rfoos]

zorg/buildbot/builders/ClangBuilder.py

41

This keeps the old behavior. There are other special cases in the builders that might make a single project list from the caller.

f = getLLVMSource(f,'llvm',['svn-llvm','svn-clang'])

# Extra stuff that will be built/tested
if checkout_clang_tools_extra:
    f = getLLVMSource(f,'llvm',['svn-clang-tools-extra'])
if checkout_compiler_rt:
    f = getLLVMSource(f,'llvm',['svn-compiler-rt'])

490

if key in cmakeGenerator:
'Visual' is a search key for strings like "Microsoft Visual Studio 12.0". Otherwise I'd be glad to change it.

I've moved cmakeProjectFile to a parameter. I think it's time to drop that code. It's a bit confusing to second guess the caller. The default will be 'build.ninja'.

508

I'll try the ordered list. It's a balance between easy for the caller, or too confusing to use.

539

Fixed.

551

Fixed.

565

All the buildslaves define jobs, but only one defines loadaverage. To make it easier to change, I have to do something if the loadaverage property is not defined. Not sure I have the right answer in this code.

595

Yes, the Web UI to force a build was what I was thinking of. The property would only apply to a single build.

Fixed. I like cleanBuildRequested much better.

659

The default should be as you said.

Unless we soft code what step 2 does, yes it makes sense that the user can't override this.

[rfoos]

zorg/buildbot/builders/ClangBuilder.py

508

Please Review:
Here's the list I'm implementing per comments.
Since it is a list of dictionaries, the caller will have to specify everything.
Selections, like useTwoStage, or testStage1 are specified by leaving or adding to the build list.

The key names in the dictionary describe actions for one or more cmake build, install, check steps.

def getCMakeBuildFactory(
        build = [
        {'name': "stage 1",
        'source': ['svn-llvm','svn-clang'],
        'source_dir': 'llvm',
        'config': "Release",
        'slave_envCmd': None,
        'cmakeGenerator': 'Ninja',
        'cmakeProjectfile': 'build.ninja',
        'extra_cmake_args': [],
        'build_cmd': ['ninja'],
        'build_dir': 'stage1',
        'install_cmd': ['ninja', 'install'],
        'install_dir': None,
        'check_cmd': ['ninja', 'check-all'],
        'check_dir': None,
        },
        {'name': "stage 2",
        'config': "Release",
        'slave_envCmd': None,
        'cmakeGenerator': 'Ninja',
        'cmakeProjectfile': 'build.ninja',
        'extra_cmake_args': [],
        'build_cmd': ['ninja'],
        'build_dir': None,
        'install_cmd': ['ninja', 'install'],
        'install_dir': None,
        'check_cmd': ['ninja', 'check-all'],
        'check_dir': None,
        },
        ]
        ):
return

[rfoos]

SVN/Git checkout using list of projects [svn-llvm, svn-clang]

Build stages driven by caller defined list of dictionaries
[{stage1},{stage2},...]

Allow user to update individual values in stage dictionaries update=[{sources=[svn-llvn,svn-clang,svn-compiler-rt]}].

The caller does not need to specify the entire set of stage dictionaries to change a few things for thier needs.

Default stages set to do a self hosted linux clang build.

Moved code into new function name. Tested on Windows/Linux, MSVC, Makefiles, Ninja.

[gkistanova]

Thanks, Rick!

I like how the ordered array of stages is coming together.
Skipping optional parameters (those with the default values) would help to keep the definition short and clear and still flexible.

Please also see my comments in-line.

Thanks

Galina

zorg/buildbot/builders/ClangBuilder.py

68

May I suggest using scmSvn and scmGit functions references directly in the arrays in llvmSourceTree?
You do not need another mapping between a string and a function, really.

And maybe better names? How about SVN and Git (no need to import SVN and Git from buildbot.steps.source in this case, just use fully qualified names in the local SVN and Git implementation)? :)

The source definition would look like

'svn-llvm': [SVN,"%s",'http://llvm.org/svn/llvm-project/llvm/','trunk','update'],

523

How about using just one parameter 'usePreviousStageResults' (or some better name) instead of useTwoStage and useBuiltClang?

Besides, since we support multiple stages, useTwoStage sounds a bit misleading.

This is a cosmetic issue, not a show stopper.

540

Maybe reduce the list of parameters for those that match exactly the defaults you are applying when you use the dictionary below? Like skipping 'extra_cmake_args' and such.

This is cosmetic, though.

541

I feel like I'm missing a use case for the update parameter. Could you elaborate, please?

563

You do not really use haveSource, do you?
What was the original idea?

652

It looks like this function is not used.

[rfoos]

zorg/buildbot/builders/ClangBuilder.py

68

Excellent idea. I need to get my head around it a little better, but I'll do it.

523

Using one previous stage results is much better.

Eliminating defaults from the argument list is a great idea. I left the argument list as a working example as there are quite a few things that need to be entered/remembered.

541

Rather than tackling re-defining the entire list, I found MSVC followed by a clang built compiler easier to do with an update to the original list.

Probably will need something to document all the things you can do with this.

ClangBuilder.getCMakeBuildFactory(
                update=[
                {
                'cmakeGenerator': "Visual Studio 12",
                'cmakeProjectfile': "ALL_BUILD.vcxproj",
                # MSBuild command with user arguments.
                'build_cmd': ["MSBuild","/p:Configuration=RelWithDebInfo",
                "/maxcpucount", "ALL_BUILD.vcxproj"],
                'install_cmd': ["MSBuild","/p:Configuration=RelWithDebInfo",
                "/maxcpucount", "INSTALL.vcxproj"],
                'check_cmd': ["MSBuild","/p:Configuration=RelWithDebInfo",
                "/maxcpucount", "check-clang.vcxproj"],
                'check_dir': 'stage1/tools/clang/test',
                'slave_envCmd': r""""%VS120COMNTOOLS%\..\..\VC\vcvarsall.bat" %PROCESSOR_ARCHITECTURE% & set""",
                'extra_cmake_args': [
                 '-DLLVM_BUILD_RUNTIME:BOOL=OFF',
                 '-DLIBCLANG_BUILD_STATIC=ON',
                 '-DLLVM_ENABLE_PIC:BOOL=ON',
                 '-DLLVM_MINIMAL_INSTALL:BOOL=ON',
                 '-DLLVM_ENABLE_ASSERTIONS:BOOL=ON',
                 '-DLLVM_ENABLE_STATS:BOOL=ON',
                 '-DLLVM_LIT_ARGS:STRING=--verbose --no-progress-bar --param build_config=Win32',
                 '-DCMAKE_BUILD_TYPE=RelWithDebInfo',
                 '-DLLVM_APPEND_VC_REV:BOOL=ON',
                 "-DLLVM_TARGETS_TO_BUILD='Hexagon'"
                "-DTARGET_TRIPLE:STRING=hexagon-unknown-elf",
                "-DLLVM_DEFAULT_TARGET_TRIPLE:STRING=hexagon-unknown-elf",
                "-DLLVM_TARGET_ARCH:STRING=hexagon-unknown-elf"
                 ]})

I haven't figured out how to delete a key with update. I may need to go back around and check for both keys and values so that everything can be updated.

Using defaults more intelligently will this by reducing the number of entries.

563

If you don't specify source and try to build something, what should I do and how should I do it.

I haven't figured that out yet. Bombing out of a checkconfig would be the best way, but I don't know of a way to do it and pass back why something failed yet.

652

Yes, I used it in the routine below (which will be replaced by this routine). I'll remove it.

[gkistanova]

zorg/buildbot/builders/ClangBuilder.py

68

Something like this:

def SVN(f, llvmsrcdir, project=''):
    (workdir, baseURL, branch, mode) = llvmSourceTree[project]
    f = f.addStep(
        buildbot.steps.source.SVN(
            name=project,
            mode=mode,
            baseURL=baseURL,
            defaultBranch=branch,
            workdir=workdir % llvmsrcdir))
    return f

...

llvmSourceTree = {
    'svn-llvm': [SVN,"%s",'http://llvm.org/svn/llvm-project/llvm/','trunk','update'],
    ...

And since we use the conflicting names, the import of buildbot.steps.source shouldn't make them available by their short names.

541

I still don't think I see clearly how update will be used with the stages array. I mean, let's say you have 2 stages, stage 1 and stage 2 as described by the build array.

Are you thinking of generating a stage 2 definition by applying an update item for the stage 2 to the build item for the stage 1?

Or you want to combine a final definition for the stage 2 by applying the stage 2 build item with stage 2 update item?

If this the second, why not making the stage 2 build definition correctly in the first place? One can provide a generator instead of an array and describe steps one by one using some more complex logic.

Anyway, it i not like you have to convince me to get this patch committed. I guess I'll see it later from the real usage. Just wanted to make sure it still makes sense after the transition to the stages description array.

563

Frankly, I would let it be as described in the stages array, without trying to sanitize. I.e. if there is no source code specified at all, the first clean build would fail, providing enough information for troubleshooting.

If we want to support a case when every stage could have its own source, then we should work out the default case, when the next stage uses the source from the previous one unless otherwise specified explicitly. The most usual setup when only the first stage specify the source and all sequential stages share it would be supported naturally.

The one important thing here is to make sure every time the source gets checked out, it would checkout the same revision a particular build is for. Meaning that only buildbot SVN/Git/whatever steps are used and not some custom shell commands.

Anyway, if you feel strongly toward sanitizing, thrown exceptions will be exposed at the checkconfig.

[rfoos]

I have the updates done, and now testing.

Properties needed some re-work, and job control is added.

zorg/buildbot/builders/ClangBuilder.py

68

Yes, and it works. Thanks.

def getLLVMSource(f,llvmsrcdir='.',project_list=[]):
  for project in project_list:
      if llvmSourceTree[project]:
          llvmSourceTree[project][0](f,llvmsrcdir,project)
return f

541

The default builder will do a self-hosted ninja build. With defaults removed it will look something like this.

def getCMakeBuildFactory(
        build=[
        {'name': "stage 1",
        'build_dir': 'stage1.build',
        'install_dir': 'stage1.install',
        'test': 'ignoreFail',
        },
        {'name': "stage 2",
        'useBuiltClang': True,
        'built_dir': 'stage1.install',
        'build_dir': 'stage2.build',
        'install_dir': 'stage2.install',
        'test': 'ignoreFail',
        },
        ],
        update=[] # Update each stage [{},...]
        ):

Once the caller touches build, all of this goes away and all of the build stages need to be specified.

Update allows a few variables to be modified without breaking the self-hosted build. It's a convenience that could be removed once we specify a bunch of builders.

Update is completely optional.

ATM I think it's useful, but I could change my mind when everyone tries to use this.

563

I tried several forms of assert. I give up.

I like your way better.

[rfoos]

Incorporated all comments.
llvmGetSource direct calls from source tree dictionary.
Added user environment variables.
Fixed ignore fail
Added install for use Previous built compiler.
Windows escaping fixes to built compiler path.

[gkistanova]

zorg/buildbot/builders/ClangBuilder.py

513

It looks like you have changed your mind and have a separate flag for the loadaverage. Could you update the description to address this, please?

540

You do not really need an empty array here, do you? None would do.

611

'stage1.install' is not always a previous stage.
We either

1. require the previous stage to do the install and use that as the path (in this case usePreviousStageResults is just a flag), or

2. always require a valid path in usePreviousStageResults value without providing any default value.

I'm in favor of # 1, but either would do.

636

This looks inconsistent with the usage of -l option in build_jobs, etc. as described in the doc strings.
Did you change your mind in favor of having a separate 'build_loadaverage' parameter and forgot to change the description above?

660

Setting all 3 flags (haltOnFailure, warnOnFailure, and flunkOnFailure) to the same value doesn't look right.

663

It doesn't seem the usePreviousStageResults for this stage should require installation step.
I can see if you would check the usePreviousStageResults of the next stage for this purpose, though.

[rfoos]

Thanks for looking at it!

zorg/buildbot/builders/ClangBuilder.py

513

Not exactly. My intent was that jobs and loadaverage are not arguments, but rather completely specified by the caller.

The default is specifying them as properties defined by the buildslave executing the command.

The second reason for xxx_jobs is the MSBuild case where only jobs can be specified in /maxcpucount:%(jobs)s.

Since most of the buildslaves have jobs define, perhaps the best default is '-j$(jobs)s -l%(jobs)'.

540

yes, the comment says what I want None is better.

611

Allowing the caller to define stages introduced extra information that two consecutive stages needed to be aware of.

If I

1. make the default install step True;
2. create an iterator for stages;
3. set the default install stage to stage<i>.install (as it is now)

The usePreviousStageResults with no value would know to look for stage<i-1>.install.

This would support #1, and would be much like the current behavior w/o requiring communication between stages.

A value in usePreviousStageResults would be a filepath to an install directory, an option supporting #2.

It does too many installs, but it avoids having stages look ahead in the stage loop.

Sound good?

636

Oops, yes this is a bug. Thanks!

660

By default, all are true.
If I need to ignore failures, halt and flunk must be false.

Doing that I had a builder take the warn on failure, and instead of green I had yellow jobs in the summary.

Yes, it seems like overkill.

maybe I can say only flunk on failure, and not specify the others and the build step will do the right thing. I'll try that and see.

663

Yes, a bug left over from the previous code that declared install the first step of stage 2.

[gkistanova]

Hello Rick,
Just checking how things are going with this patch.
I might have a day or two to make a pass on this, if this is fine with you and you are not about to publish a new version of the change, of course.

[rfoos]

Yes, I've been tied up with several things.

If I can't get something up later this afternoon, please go for it, this patch is really needed.

[rengolin]

This revision is not several months stale and will have to be rebased and heavily modified.

I'll close this one and we can start a new one when either of us have time to look at this.

zorg/buildbot/builders/ClangBuilder.py

 import buildbot.process.factory
 import os
 
-from buildbot.process.properties import WithProperties
+from copy import deepcopy
+from buildbot.process.properties import WithProperties, Property
 from buildbot.steps.shell import Configure, ShellCommand, SetProperty
 from buildbot.steps.shell import WarningCountingShellCommand
-from buildbot.steps.source import SVN
 from buildbot.steps.transfer import FileDownload
+from buildbot.steps.slave import RemoveDirectory
 
 import zorg.buildbot.util.artifacts as artifacts
 import zorg.buildbot.builders.Util as builders_util

 import zorg.buildbot.commands.BatchFileDownload as batch_file_download
 import zorg.buildbot.commands.LitTestCommand as lit_test_command
 
+def slave_env_glob2list(rc, stdout, stderr):
+    '''Extract function for SetPropertyCommand. Loads Slave Environment
+    into a dictionary, and returns slave_env property for ShellCommands.'''
+    if not rc:
+        slave_env_dict = dict(l.strip().split('=',1) 
+            for l in stdout.split('\n') if len(l.split('=',1))==2)
+    return {'slave_env': slave_env_dict}
+
+def SVN(f,llvmsrcdir,project=''):
+    (scm, workdir, baseURL, branch, mode) = llvmSourceTree[project]
+    f = f.addStep(buildbot.steps.source.SVN(name=project,
+            mode=mode,
+            baseURL=baseURL,
+            defaultBranch=branch,
+            workdir=workdir % llvmsrcdir))
+    return f
+
+def Git(f,llvmsrcdir,project=''):
+    (scm, workdir, baseURL, branch, mode) = llvmSourceTree[project]
+    f = f.addStep(buildbot.steps.source.Git(name=project,
+            repourl=baseURL,
+            branchranch=branch,
+            alwaysUseLatest=True,
+            haltOnFailure=True,
+            workdir=workdir % llvmsrcdir))
+    return f
+
+# SCM, workdir, baseURL, branch, mode
+llvmSourceTree = {
+    'svn-llvm': [SVN,"%s",'http://llvm.org/svn/llvm-project/llvm/','trunk','update'],
+    'svn-clang': [SVN,"%s/tools/clang",'http://llvm.org/svn/llvm-project/cfe/','trunk','update'],
+    'svn-clang-tools-extra': [SVN,"%s/tools/clang/tools/extra",'http://llvm.org/svn/llvm-project/clang-tools-extra/','trunk','update'],
+    'svn-lldb': [SVN,"%s/tools/lldb",'http://llvm.org/svn/llvm-project/lldb/','trunk','update'],
+    'svn-polly': [SVN,"%s/tools/polly",'http://llvm.org/svn/llvm-project/polly/','trunk','update'],
+    'svn-libcxx': [SVN,"%s/projects/libcxx",'http://llvm.org/svn/llvm-project/libcxx/','trunk','update'],
+    'svn-libcxx-pull': [SVN,"%s/projects/libcxx",'http://llvm.org/svn/llvm-project/libcxx/','trunk','pull'],
+    'svn-libcxxabi': [SVN,"%s/projects/libcxxabi",'http://llvm.org/svn/llvm-project/libcxxabi/','trunk','update'],
+    'svn-compiler-rt': [SVN,"%s/projects/compiler-rt",'http://llvm.org/svn/llvm-project/compiler-rt/','trunk','update'],
+    'svn-dragonegg': [SVN,"dragonegg.src",'http://llvm.org/svn/llvm-project/dragonegg/','trunk','update'],
+    'svn-test-suite': [SVN,"test-suite.src",'http://llvm.org/svn/llvm-project/test-suite/','trunk','update'],
+    'svn-libiomp5': [SVN,"openmp.src",'http://llvm.org/svn/llvm-project/openmp/','trunk','update'],
+    'svn-clang-modern-gdb-tests': [SVN,"clang-tests/src",'http://llvm.org/svn/llvm-project/clang-tests-external/trunk/gdb/7.5/','','update'],
+    'git-imath': [Git,"imath.src",'git://codeaurora.org/quic/iml/imath/','isl','update'],
+    'git-isl': [Git,"isl.src",'http://repo.or.cz/r/isl.git/','master','update'],
+}
+# TODO 'svn-lit': [SVN,"lit.src",'http://llvm.org/svn/llvm-project/llvm/trunk/utils/lit','trunk','incremental'],
+# Git mirror not needed 'git-llvm': [Git,"%s",'http://llvm.org/git/llvm','master','update'],
+
+def getLLVMSource(f,llvmsrcdir='.',project_list=[]):
+    for project in project_list:
+        if llvmSourceTree[project]:
+            llvmSourceTree[project][0](f,llvmsrcdir,project)
+    return f
+
 def getClangBuildFactory(
             triple=None,
             clean=True,

                                haltOnFailure=True,
                                workdir='.'))
     else:
-        f.addStep(SVN(name='svn-llvm',
+        f.addStep(buildbot.steps.source.SVN(name='svn-llvm',
                       mode='update',
                       baseURL='http://llvm.org/svn/llvm-project/llvm/',
                       defaultBranch='trunk',
                       workdir=llvm_srcdir))
-        f.addStep(SVN(name='svn-clang',
+        f.addStep(buildbot.steps.source.SVN(name='svn-clang',
                       mode='update',
                       baseURL='http://llvm.org/svn/llvm-project/cfe/',
                       defaultBranch='trunk',
                       workdir='%s/tools/clang' % llvm_srcdir))
-        f.addStep(SVN(name='svn-clang-tools-extra',
+        f.addStep(buildbot.steps.source.SVN(name='svn-clang-tools-extra',
                       mode='update',
                       baseURL='http://llvm.org/svn/llvm-project/clang-tools-extra/',
                       defaultBranch='trunk',
                       workdir='%s/tools/clang/tools/extra' % llvm_srcdir))
         if checkout_compiler_rt:
-            f.addStep(SVN(name='svn-compiler-rt',
+            f.addStep(buildbot.steps.source.SVN(name='svn-compiler-rt',
                           mode='update',
                           baseURL='http://llvm.org/svn/llvm-project/compiler-rt/',
                           defaultBranch='trunk',

 
     return f
 
-# CMake Linux builds
+""" CMake Build Factory
+name: 'stage', Name: step name.
+source: [] List of projects from llvmSourceTree.
+'source_dir': 'llvm',
+'config': 'Release' Debug, Release, RelWithDebug.
+'slave_envCmd': r'', Execute command on slave, get slave env.
+env: {}, Update slave environment variables. Superceeded by slave_envCmd.
+'cmakeGenerator': 'Ninja', CMake Generator
+'cmakeProjectfile': 'build.ninja',Project file after CMake is run.
+'extra_cmake_args': [],Extra arguments to CMake.
+'build_cmd': ['ninja'], Build source code command.
+build_jobs: '-j%(jobs)s -l%(loadaverage)', job control if supported.
+'build_dir': 'stage1.build',
+'install_cmd': ['ninja', 'install'],
+'install_dir': 'stage1.install',
+install_cmd_jobs: '-j%(jobs)s -l%(loadaverage)', job control if supported.
+'check_cmd': ['ninja', 'check-all'],
+'check_dir': '', sub-directory from build_dir to start tests.
+check_cmd_jobs: '-j%(jobs)s -l%(loadaverage)', job control if supported.
+'useTwoStage': True, Use compiler from previous stage.
+'test': '', Execute test step (True) and Ignore Failures (ignoreFail)
+clean: False, clean build requested.
+"""
+def getCMakeBuildFactory(
+            build=[
+            {'name': "stage 1",
+            'build_dir': 'stage1.build',
+            'install_dir': 'stage1.install',
+            'install': True,
+            'test': 'ignoreFail',
+            },
+            {'name': "stage 2",
+            'usePreviousStageResults': 'stage1.install',
+            'build_dir': 'stage2.build',
+            'install_dir': 'stage2.install',
+            'test': 'ignoreFail',
+            },
+            ],
+            update=[] # Update each stage [{},...]
+            ):
+
+    # Apply updates to each stage. [{},{}]
+    if update:
+        newBuild = deepcopy(build)
+        for u,b in zip(update,newBuild):
+            b.update(u)
+    else:
+        newBuild = build
+
+    f = buildbot.process.factory.BuildFactory()
+    
+    # Must be initialized before multiple stage settings begin.
+    env = Property('env')
+    
+    # Drive thru build stages.
+    for stage in newBuild:
+        # Set the stage name
+        stageName = stage.get('name','stage')
+
+        # Check for source first.
+        stage_source = stage.get('source_dir','llvm')
+        if 'source' in stage:
+            f = getLLVMSource(f,stage_source,stage['source'])
+
+        # Check for user specified environment updates.
+        # Note: This is superceeded by slave_envCmd within a stage.
+        if 'env' in stage:
+            f = zorg.buildbot.util.phasedbuilderutils.setProperty(
+                    f,new_property='env', new_value=stage['env'])
+            env=Property('env')
+        
+        # Set and return slave environment to be used in all commands.
+        if 'slave_envCmd' in stage:
+            f.addStep(SetProperty(
+                name='get slave env',
+                command=stage['slave_envCmd'],
+                extract_fn=slave_env_glob2list))
+            env=Property('slave_env')
+
+        # Global configurations
+        stage_build = stage.get('build_dir','stage.build')
+        stage_install = stage.get('install_dir','stage.install')
+        if 'check_dir' in stage:
+            stage_check = stage_build+"/"+str(stage['check_dir'])
+        else:
+            stage_check = stage_build
+
+        ############# CLEANING
+        cleanBuildRequested = lambda step: step.build.getProperty("clean") or stage.get('clean',False)
+        f.addStep(RemoveDirectory(name=stageName+': clean',
+                    description='cleaning %s' % stage_build,
+                    dir=stage_build,
+                    warnOnFailure=True,
+                    doStepIf=cleanBuildRequested
+                    ))
+        cmakeProjectfile = stage.get('cmakeProjectfile','build.ninja')
+        f.addStep(SetProperty(name=stageName+': check cmake project files',
+                    workdir=stage_build,
+                    command=["sh", "-c",
+                           "test -e %s && echo OK || echo Missing" % cmakeProjectfile],
+                    flunkOnFailure=False,
+                    property="exists_cmake_projectfile"))
+
+        ############# STAGE
+        cmakeGenerator = stage.get('cmakeGenerator','Ninja')
+        stage_config = stage.get('config','Release')
+        lit_args = stage.get('lit_args',"-v")
+        usePreviousClang = []
+        if 'usePreviousStageResults' in stage:
+            built_clang_dir = stage.get('usePreviousStageResults','stage1.install')
+            # CMake escapes these variables. Windows \ is fatal.
+            usePreviousClang=[
+            "-DCMAKE_C_COMPILER=../"+built_clang_dir+"/bin/clang",
+            "-DCMAKE_CXX_COMPILER=../"+built_clang_dir+"/bin/clang++"
+            ]
+        extra_cmake_args = stage.get('extra_cmake_args',[])
+        f.addStep(ShellCommand(name=stageName+': cmake',
+                    command=["cmake", "-G", cmakeGenerator, "../"+stage_source,
+                            "-DCMAKE_BUILD_TYPE="+stage_config,
+                            "-DLLVM_ENABLE_ASSERTIONS=True",
+                            "-DLLVM_LIT_ARGS="+lit_args,
+                            "-DCMAKE_INSTALL_PREFIX=../"+stage_install]
+                            + usePreviousClang
+                            + extra_cmake_args,
+                    haltOnFailure=True,
+                    description='cmake %s' % stage_config,
+                    workdir=stage_build,
+                    doStepIf=lambda step: step.build.getProperty("exists_cmake_projectfile") != "OK",
+                    env=env))
+
+        build_cmd = [stage.get('build_cmd',['ninja'])]
+        if 'build_jobs' in stage:
+            build_cmd.append(WithProperties(stage['build_jobs']))
+        if 'build_loadaverage' in stage:
+            build_cmd.append(WithProperties(stage['build_loadaverage']))
+        f.addStep(WarningCountingShellCommand(name=stageName+': build',
+                    command=build_cmd,
+                    haltOnFailure=True,
+                    description='building %s' % stage_build,
+                    descriptionDone=["stage built"],
+                    workdir=stage_build,
+                    env=env))
+
+        if 'test' in stage:
+            check_cmd = stage.get('check_cmd',['ninja', 'check-all'])
+            if 'check_jobs' in stage:
+                check_cmd.append(WithProperties(stage['check_jobs']))
+            if 'check_loadaverage' in stage:
+                check_cmd.append(WithProperties(stage['check_loadaverage']))
+            ignoreTestFail = bool(stage['test'] != 'ignoreFail')
+            f.addStep(lit_test_command.LitTestCommand(name=stageName+': check',
+                        command=check_cmd,
+                        description="checking %s" % stage_check,
+                        descriptionDone="stage checked",
+                        workdir=stage_check,
+                        haltOnFailure=ignoreTestFail,
+                        warnOnFailure=ignoreTestFail,
+                        flunkOnFailure=ignoreTestFail,
+                        env=env))
+
+        ############# STAGE 2
+        if 'usePreviousStageResults' or 'install' in stage:
+            install_cmd = stage.get('install_cmd',['ninja', 'install'])
+            if 'install_jobs' in stage:
+                install_cmd.append(WithProperties(stage['install_jobs']))
+            if 'install_loadaverage' in stage:
+                install_cmd.append(WithProperties(stage['install_loadaverage']))
+            ignoreInstallFail = True
+            if 'install' in stage:
+                ignoreInstallFail = bool(stage['install'] != 'ignoreFail')
+            f.addStep(ShellCommand(name=stageName+': install',
+                        command=install_cmd,
+                        description='installing %s' % stage_install,
+                        descriptionDone=["stage installed"],
+                        workdir=stage_build,
+                        haltOnFailure=ignoreInstallFail,
+                        warnOnFailure=ignoreInstallFail,
+                        flunkOnFailure=ignoreInstallFail,
+                        env=env))
+    return f
+
+# CMake Linux/Windows builds
 def getClangCMakeBuildFactory(
             clean=True,
             test=True,

     f = buildbot.process.factory.BuildFactory()
 
     # We *must* checkout at least Clang+LLVM
-    f.addStep(SVN(name='svn-llvm',
+    f.addStep(buildbot.steps.source.SVN(name='svn-llvm',
                   mode='update', baseURL='http://llvm.org/svn/llvm-project/llvm/',
                   defaultBranch='trunk',
                   workdir='llvm'))
-    f.addStep(SVN(name='svn-clang',
+    f.addStep(buildbot.steps.source.SVN(name='svn-clang',
                   mode='update', baseURL='http://llvm.org/svn/llvm-project/cfe/',
                   defaultBranch='trunk',
                   workdir='llvm/tools/clang'))

 
     # Extra stuff that will be built/tested
     if checkout_clang_tools_extra:
-        f.addStep(SVN(name='svn-clang-tools-extra',
+        f.addStep(buildbot.steps.source.SVN(name='svn-clang-tools-extra',
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/clang-tools-extra/',
                       defaultBranch='trunk',
                       workdir='llvm/tools/clang/tools/extra'))
     if checkout_compiler_rt:
-        f.addStep(SVN(name='svn-compiler-rt',
+        f.addStep(buildbot.steps.source.SVN(name='svn-compiler-rt',
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/compiler-rt/',
                       defaultBranch='trunk',
                       workdir='llvm/projects/compiler-rt'))

     f = buildbot.process.factory.BuildFactory()
 
     if update:
-        f.addStep(SVN(name='svn-llvm',
+        f.addStep(buildbot.steps.source.SVN(name='svn-llvm',
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/llvm/',
                       defaultBranch='trunk',
                       workdir='llvm'))
-        f.addStep(SVN(name='svn-clang',
+        f.addStep(buildbot.steps.source.SVN(name='svn-clang',
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/cfe/',
                       defaultBranch='trunk',
                       workdir='llvm/tools/clang'))
-        f.addStep(SVN(name='svn-clang-tools-extra',
+        f.addStep(buildbot.steps.source.SVN(name='svn-clang-tools-extra',
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/clang-tools-extra/',
                       defaultBranch='trunk',
                       workdir='llvm/tools/clang/tools/extra'))

     f = buildbot.process.factory.BuildFactory()
 
     if update:
-        f.addStep(SVN(name='svn-llvm',
+        f.addStep(buildbot.steps.source.SVN(name='svn-llvm',
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/llvm/',
                       defaultBranch='trunk',
                       workdir='llvm'))
-        f.addStep(SVN(name='svn-clang',
+        f.addStep(buildbot.steps.source.SVN(name='svn-clang',
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/cfe/',
                       defaultBranch='trunk',
                       workdir='llvm/tools/clang'))
-        f.addStep(SVN(name='svn-clang-tools-extra',
+        f.addStep(buildbot.steps.source.SVN(name='svn-clang-tools-extra',
                       mode='update', baseURL='http://llvm.org/svn/llvm-project/clang-tools-extra/',
                       defaultBranch='trunk',
                       workdir='llvm/tools/clang/tools/extra'))

             'CC_UNDER_TEST=%s/bin/clang' % install_prefix),
                  WithProperties(
             'CXX_UNDER_TEST=%s/bin/clang++' % install_prefix)]
-    f.addStep(SVN(name='svn-clang-gcc-tests', mode='update',
+    f.addStep(buildbot.steps.source.SVN(name='svn-clang-gcc-tests', mode='update',
                   baseURL='http://llvm.org/svn/llvm-project/clang-tests/',
                   defaultBranch='trunk', workdir='clang-tests'))
     gcc_dg_ignores = ignores.get('gcc-4_2-testsuite', {})

             'CC_UNDER_TEST=%s/bin/clang' % install_prefix),
                  WithProperties(
             'CXX_UNDER_TEST=%s/bin/clang++' % install_prefix)]
-    f.addStep(SVN(name='svn-clang-gdb-tests', mode='update',
+    f.addStep(buildbot.steps.source.SVN(name='svn-clang-gdb-tests', mode='update',
                   baseURL='http://llvm.org/svn/llvm-project/clang-tests/',
                   defaultBranch='trunk', workdir='clang-tests'))
     f.addStep(commands.SuppressionDejaGNUCommand.SuppressionDejaGNUCommand(

                                 'CXX_FOR_TARGET=\'{0}/bin/clang++\' '
                                 'CFLAGS_FOR_TARGET=\'-w -fno-limit-debug-info\''
                                 .format(install_prefix))]
-    f.addStep(SVN(name='svn-clang-modern-gdb-tests', mode='update',
+    f.addStep(buildbot.steps.source.SVN(name='svn-clang-modern-gdb-tests', mode='update',
                   svnurl='http://llvm.org/svn/llvm-project/clang-tests-external/trunk/gdb/7.5',
                   workdir='clang-tests/src'))
     f.addStep(Configure(command='../src/configure',