Teach zorg to support cmake and modules for Clang builders

[rsmith]

Summary

Adds modules= and cmake= keyword arguments to getClangBuildFactory. Right now, the modules build requires a two-stage cmake-based bootstrap. To use this you'll need a modularized libc and libc++ in a place where the build will find them.

My modules buildbot is configured using this builder:

{'name' : "clang-x86_64-linux-selfhost-rel",
 'slavenames' : ["localhost"],
 'builddir' : "clang-x86_64-linux-selfhost-rel",
 'factory' : ClangBuilder.getClangBuildFactory(triple='x86_64-pc-linux-gnu',
                                      useTwoStage=True,
                                      modules=True,
                                      clean=False,
                                      stage1_config='Release+Asserts',
                                      stage2_config='Release+Asserts',
                                      extra_configure_args=[
                                          '-DCMAKE_C_COMPILER=/var/lib/buildbot/bin/clang',
                                          '-DCMAKE_CXX_COMPILER=/var/lib/buildbot/bin/clang++',
                                          '-DCMAKE_CXX_FLAGS=-stdlib=libc++',
                                      ],
                                      cmake='/var/lib/buildbot/bin/cmake')},

... and seems to work.

[rsmith]

Ping

[dblaikie]

zorg/buildbot/builders/ClangBuilder.py

46

This flag seems unused apart from the assertion below. Should it instead conditionalize the use of modules down around line 355 as well as being used in this assertion?

251

Trivial & can be committed separately if you like, of course.

331

Making this unconditional is an independent bug fix, I assume? (stage2 wouldn't benefit from any caching, so it should always be a clean build)

[rsmith]

Conditionalize adding of -fmodules etc on the modules flag, and don't change existing two-stage buildbots' behavior with respect to cleaning stage 2 before rebuilding it.

[rsmith]

zorg/buildbot/builders/ClangBuilder.py

46

Yes, it should =) Done, but I've not tested the cmake build /without/ modules.

251

Committed separately.

331

I've changed this so we only unconditionally clean stage2 for a cmake build. I don't want to change the behavior of existing buildbots, even though it doesn't seem very sensible to do an incremental build of stage2.

[dblaikie]

Looks good. One optional comment

(wow, phab markup does fun things with leading #... )

zorg/buildbot/builders/ClangBuilder.py

332

Maybe rephrase this (it leaves fairly ambiguous as to why we want to clean for the configure build here - the answer being "there's no known reason") maybe:

FIXME: verify that there's no need to skip cleaning for the configure stage 2 build (it doesn't make much sense) and remove this condition

?

[rsmith]

Thanks! I switched this over to enabling modules via the new -DLLVM_ENABLE_MODULES=1 flag and committed as r218557.

zorg/buildbot/builders/ClangBuilder.py

332

Done.

zorg/buildbot/builders/ClangBuilder.py

def getClangBuildFactory(
             use_pty_in_tests=False,
             trunk_revision=None,
             force_checkout=False,
             extra_clean_step=None,
             checkout_compiler_rt=False,
             run_gdb=False,
             run_modern_gdb=False,
             run_gcc=False,
-            merge_functions=False):
+            merge_functions=False,
+            cmake=None,
+            modules=False):
+    assert not modules or (cmake and useTwoStage), \
+           "modules build requires 2 stage cmake build for now"
+
     # Prepare environmental variables. Set here all env we want everywhere.
     merged_env = {
         'TERM' : 'dumb' # Make sure Clang doesn't use color escape sequences.
                  }
     if env is not None:
         # Overwrite pre-set items with the given ones, so user can set anything.
         merged_env.update(env)
 
def getClangBuildFactory(
     if clean and llvm_srcdir != llvm_1_objdir:
         f.addStep(ShellCommand(name="rm-llvm.obj.stage1",
                                command=["rm", "-rf", llvm_1_objdir],
                                haltOnFailure=True,
                                description=["rm build dir", "llvm"],
                                workdir=".",
                                env=merged_env))
 
+    if not clean:
+        if cmake:
+            expected_makefile = 'Makefile'
+        else:
+            expected_makefile = 'Makefile.config'
+        f.addStep(SetProperty(name="Makefile_isready",
+                              workdir=llvm_1_objdir,
+                              command=["sh", "-c",
+                                       "test -e %s && echo OK || echo Missing" % expected_makefile],
+                              flunkOnFailure=False,
+                          property="exists_Makefile"))
+
+    if not cmake:
         # Force without llvm-gcc so we don't run afoul of Frontend test failures.
         base_configure_args = [WithProperties("%%(builddir)s/%s/configure" % llvm_srcdir),
                                '--disable-bindings']
         base_configure_args += extra_configure_args
         if triple:
             base_configure_args += ['--build=%s' % triple,
                                     '--host=%s' % triple]
         args = base_configure_args + [WithProperties("--prefix=%%(builddir)s/%s" % llvm_1_installdir)]
         args += builders_util.getConfigArgs(stage1_config)
-    if not clean:
-        f.addStep(SetProperty(name="Makefile_isready",
-                              workdir=llvm_1_objdir,
-                              command=["sh", "-c",
-                                       "test -e Makefile.config && echo OK || echo Missing"],
-                              flunkOnFailure=False,
-                          property="exists_Makefile"))
+
         f.addStep(Configure(command=args,
                             workdir=llvm_1_objdir,
                             description=['configuring',stage1_config],
                             descriptionDone=['configure',stage1_config],
                             env=merged_env,
                             doStepIf=lambda step: step.build.getProperty("exists_Makefile") != "OK"))
+    else:
+        cmake_triple_arg = []
+        if triple:
+            cmake_triple_arg = ['-DLLVM_HOST_TRIPLE=%s' % triple]
+        f.addStep(ShellCommand(name='cmake',
+                               command=[cmake,
+                                        '-DLLVM_BUILD_TESTS=ON',
+                                        '-DCMAKE_BUILD_TYPE=%s' % stage1_config] +
+                                       cmake_triple_arg +
+                                       extra_configure_args +
+                                       ["../" + llvm_srcdir],
+                               description='cmake stage1',
+                               workdir=llvm_1_objdir,
+                               env=merged_env,
+                               doStepIf=lambda step: step.build.getProperty("exists_Makefile") != "OK"))
 
     # Make clean if using in-dir builds.
     if clean and llvm_srcdir == llvm_1_objdir:
         f.addStep(WarningCountingShellCommand(name="clean-llvm",
                                               command=[make, "clean"],
                                               haltOnFailure=True,
                                               description="cleaning llvm",
                                               descriptionDone="clean llvm",
f.addStep(WarningCountingShellCommand(name="compile",
                                           env=merged_env))
 
     if examples:
         f.addStep(WarningCountingShellCommand(name="compile.examples",
                                               command=['nice', '-n', '10',
                                                        make, WithProperties("-j%s" % jobs),
                                                        "BUILD_EXAMPLES=1"],
                                               haltOnFailure=True,
                                               description=["compiling", stage1_config, "examples"],
                                               descriptionDone=["compile", stage1_config, "examples"],
                                               workdir=llvm_1_objdir,
                                               env=merged_env))
 
     clangTestArgs = '-v -j %s' % jobs
     if valgrind:
         clangTestArgs += ' --vg'
         if valgrindLeakCheck:
if test:
                                    flunkOnFailure=not run_gdb,
                                    description=["checking"],
                                    descriptionDone=["checked"],
                                    workdir=llvm_1_objdir,
                                    usePTY=use_pty_in_tests,
                                    env=merged_env))
 
     # Install llvm and clang.
-    if llvm_1_installdir:
+    if llvm_1_installdir and not cmake: # FIXME: install for cmake build
         f.addStep(ShellCommand(name="rm-install.clang.stage1",
                                command=["rm", "-rf", llvm_1_installdir],
                                haltOnFailure=True,
                                description=["rm install dir", "clang"],
                                workdir=".",
                                env=merged_env))
         f.addStep(WarningCountingShellCommand(name="install.clang.stage1",
                                               command = ['nice', '-n', '10',
if not useTwoStage:
                                    warnOnFailure=True,
                                    flunkOnFailure=False,
                                    haltOnFailure=False,
                                    env=merged_env))
 
         return f
 
     # Clean up llvm (stage 2).
-    if clean:
+    #
+    # It does not make sense to skip this for a bootstrap builder,
+    # and we don't support doing so for cmake builds.
+    if clean or cmake:
         f.addStep(ShellCommand(name="rm-llvm.obj.stage2",
                                command=["rm", "-rf", llvm_2_objdir],
                                haltOnFailure=True,
                                description=["rm build dir", "llvm", "(stage 2)"],
                                workdir=".",
                                env=merged_env))
 
     # Configure llvm (stage 2).
+    if not cmake:
         args = base_configure_args + [WithProperties("--prefix=%(builddir)s/" + llvm_2_installdir)]
         args += builders_util.getConfigArgs(stage2_config)
         local_env = dict(merged_env)
         local_env.update({
             'CC'  : WithProperties("%%(builddir)s/%s/bin/clang"   % llvm_1_installdir),
             'CXX' : WithProperties("%%(builddir)s/%s/bin/clang++" % llvm_1_installdir)})
 
         f.addStep(Configure(name="configure.llvm.stage2",
                             command=args,
                             haltOnFailure=True,
                             workdir=llvm_2_objdir,
                             description=["configure", "llvm", "(stage 2)",
                                          stage2_config],
                             env=local_env))
+    else:
+        c_flags = ''
+        cxx_flags = ''
+        if modules:
+            c_flags += '-fmodules -fmodule-map-file=/usr/include/module.modulemap -fmodules-cache-path=%%(builddir)s/%s/module-cache' % llvm_2_objdir
+            # Ignore *_EXPORTS macros; they generate additional pointless module variants.
+            ignored_exports_macros = [
+                'BugpointPasses', 'LLVMHello', 'bugpoint', 'llvm_stress',
+                 'opt', 'llc', 'JITTests', 'lli', 'LTO']
+            c_flags += ''.join(' -fmodules-ignore-macro=%s_EXPORTS' % ignored_exports_macro
+                               for ignored_exports_macro in ignored_exports_macros)
+            cxx_flags += '-fcxx-modules -stdlib=libc++'
+
+        f.addStep(ShellCommand(name='cmake',
+                               command=[cmake,
+                                        '-DLLVM_BUILD_TESTS=ON',
+                                        WithProperties('-DCMAKE_C_COMPILER=%%(builddir)s/%s/bin/clang' % llvm_1_objdir), # FIXME use installdir
+                                        WithProperties('-DCMAKE_CXX_COMPILER=%%(builddir)s/%s/bin/clang++' % llvm_1_objdir),
+                                        '-DCMAKE_BUILD_TYPE=%s' % stage2_config,
+                                        WithProperties('-DCMAKE_C_FLAGS=%s' % c_flags),
+                                        WithProperties('-DCMAKE_CXX_FLAGS=%s %s' % (c_flags, cxx_flags)),
+                                        "../" + llvm_srcdir],
+                               description='cmake stage2',
+                               workdir=llvm_2_objdir,
+                               env=merged_env))
 
     # Build llvm (stage 2).
     f.addStep(WarningCountingShellCommand(name="compile.llvm.stage2",
                                           command=['nice', '-n', '10',
                                                    make, WithProperties("-j%s" % jobs)],
                                           haltOnFailure=True,
                                           description=["compiling", "(stage 2)",
                                                        stage2_config],
                                           descriptionDone=["compile", "(stage 2)",
                                                            stage2_config],
                                           workdir=llvm_2_objdir,
                                           env=merged_env))
 
     if test:
         f.addStep(lit_test_command.LitTestCommand(name='check-all',
                                    command=[make, "check-all", "VERBOSE=1",
                                             WithProperties("LIT_ARGS=%s" % clangTestArgs),
                                             WithProperties("EXTRA_TESTDIRS=%s" % extraTestDirs)],
                                    description=["checking"],
                                    descriptionDone=["checked"],
                                    workdir=llvm_2_objdir,
                                    usePTY=use_pty_in_tests,
                                    env=merged_env))
 
+    if llvm_2_installdir and not cmake: # FIXME: install for cmake build
         # Install clang (stage 2).
         f.addStep(ShellCommand(name="rm-install.clang.stage2",
                                command=["rm", "-rf", llvm_2_installdir],
                                haltOnFailure=True,
                                description=["rm install dir", "clang"],
                                workdir=".",
                                env=merged_env))
         f.addStep(WarningCountingShellCommand(name="install.clang.stage2",
                                               command = ['nice', '-n', '10',
                                                          make, 'install-clang'],
                                               haltOnFailure=True,
                                               description=["install", "clang",
                                                            "(stage 2)"],
                                               workdir=llvm_2_objdir,
                                               env=merged_env))
 
     if package_dst:
         name = WithProperties(
             "%(builddir)s/" + llvm_2_objdir +
             "/clang-r%(got_revision)s-b%(buildnumber)s.tgz")
         f.addStep(ShellCommand(name='pkg.tar',
                                description="tar root",
                                command=["tar", "zcvf", name, "./"],