Add support of multiple test compilers and architecture for LLDB cmake builder

[chying]

Summary

-build lldb test with clang/totclang/gcc4.8.2 as compilers and x86_64/i386 as architecture

[sivachandra]

buildbot/osuosl/master/config/builders.py

722

Can we have two archs on the same build slave??

zorg/buildbot/builders/LLDBBuilder.py

246

This should be initialized with the default system architecture.

248

You should probably check for presence of clang, gcc or cc in that order and default a test compiler to that.

303

You should probably have a step which cleans up the source tree and the build tree after each test step. Side effects from a previous test run could potentially affect a consequent test run.

[sivachandra]

+r vharron just so that he is aware that multiple compilers get tested on a single build slave. He probably already knows, but just in case!

[chying]

buildbot/osuosl/master/config/builders.py

722

Yes, it's one of the test branches to run i386 build on x86_64 machine.
The test result of i386 build running on x86_64 vs. i386 build running on 32 bit machine might be different, so next step would be to run i386 build on 32 bit machine.

zorg/buildbot/builders/LLDBBuilder.py

246

The thought was that if no test arch or compilers is specified, then lldb-test will be skipped.

248

Same as above. If no test_compilers were specified, lldb-test will be skipped

303

Since build/test is running in a separate build folder, and I checked that there's no files added or changed to source tree after running test.
But as you suggested, I will create separate output folder for each test so there's no possible interference.

[sivachandra]

buildbot/osuosl/master/config/builders.py

722

OK. Since getLLDBUbuntuCMakeBuildFactory is a generic utility, you should check whether the architectures listed are compatible in getLLDBUbuntuCMakeBuildFactory. For example, using x86_64 lldb to debug x86_64 binaries on i386 is fine. But, using i386 LLDB is probably not OK for debugging x86_64 binaries. You could for now keep something very simple like and stick to x86_64 and i386 like this:

if platform.machine() == 'x86_64':
    compatible_archs = ['x86_64', 'i386']
    assert(all([x in compatible_archs for x in test_archs]))

zorg/buildbot/builders/LLDBBuilder.py

236

Plural test_archs is probably more indicative of the fact that a sequence is expected.

246

In which case, this should be noted in a comment near the function header. Also, instead of making test_arch={}, the intent is more obvious if you have something like this after you add the compile step:

if not test_arch:
    return

Note: Even if you end up keeping code as is, initializing to an empty list [] is more meaningful than an empty dict {}.

248

Same as my comment with test_arch.

303

OK about the source tree. How about the build tree? Does running tests create any side effects in the build tree apart from lldb-test-traces?

[chying]

Clean svn source tree after running lldb-test
Put test trace logs to separate folder for each compiler/arch

[chying]

buildbot/osuosl/master/config/builders.py

722

This code will run on buildbot master machine, so it will return the machine architecture of master host. And assert will also fail on master side if any unsupported arch is specified.
The only way to run code on slave side is by factory.addStep(...).
We need to create some script on the fly and invoke script on slave machine to achieve this.
I suggest to keep the code as is for now, and come back to this when we have more time.

zorg/buildbot/builders/LLDBBuilder.py

246

yes, i changed to empty list in the next revision.

303

I take back the statement that "there's no files added or changed to source tree after running test".
Actually there're *.pyc and some output files left in the source tree.
I did not found these cause they're ignored by default.
Added step to clean these files in the next revision.

And yes, lldb-test-traces is the only side effect happened in build tree.

[sivachandra]

Please address comments before committing.

buildbot/osuosl/master/config/builders.py

722

OK. Sorry, I missed the fact that this runs on the master. If you do not have a simple solution for now, put a TODO about this just so that readers are aware of the problem.

zorg/buildbot/builders/LLDBBuilder.py

232

s/bulids/builds

233

s/sequnce/sequence
s/doset/dosep

234

Just so that it doesn't seem I missed, you should ideally put pydoc style comments. However, it does not seem to be the style in this file. Hence, I leave it up to you.

234

I think a better comment would be:

If test_archs or test_compilers are None or empty, test steps will not be added to the build factory.

248

Please modify lines 256 to 259 to this:

if (not test_archs) or (not test_compilers):
    return f

and move these two lines to after line 305.

[chying]

Address several nits based on feedback.

Add comments for TODO task - check listed archs are compatible with host
Correct typos in comment message
Return function early if no test arch or compiler specified

buildbot/osuosl/master/config/builders.py

return [
                         '--enable-optimized',
                         '--enable-assertions'],
                     env={'SHELL':"/bin/bash"})},
         {'name': "lldb-x86_64-ubuntu-14.04-cmake",
          'slavenames': ["lldb-build1-ubuntu-1404"],
          'builddir': "buildWorkingDir",
          'factory': LLDBBuilder.getLLDBUbuntuCMakeBuildFactory(
                     build_compiler="clang",
-                    test_compiler="clang",
                     build_type="Debug",
+                    test_arch=['x86_64', 'i386'],
+                    test_compilers=['clang', 'totclang', 'gcc4.8.2'],
                     env={'SHELL':"/bin/bash"})},
         {'name': "lldb-x86_64-darwin-13.4",
          'slavenames': ["lldb-x86_64-darwin-13.4"],
          'builddir': "buildDir",
          'factory': LLDBBuilder.getLLDBxcodebuildFactory()},
        ]
 
 # Offline.

zorg/buildbot/builders/LLDBBuilder.py

f.addStep(LitTestCommand(name="test lldb",
                                       make],
                              description="test lldb",
                              env=env,
                              workdir='%s/tools/lldb/test' % llvm_objdir))
 
     return f
 
 #Cmake bulids on Ubuntu
 #Build command sequnce - cmake, ninja, ninja check-lldb
 def getLLDBUbuntuCMakeBuildFactory(
             build_compiler,
-            test_compiler,
             build_type,
+            test_arch=None,
+            test_compilers=None,
             jobs='%(jobs)s',
             env=None,
             *args,
             **kwargs):
 
     if env is None:
         env={}
+    if test_arch is None:
+        test_arch={}
+    if test_compilers is None:
+        test_compilers={}
 
     llvm_srcdir = "llvm"
     llvm_builddir = "build"
 
     f = buildbot.process.factory.BuildFactory()
 
     # Determine the build directory.
     f.addStep(SetProperty(name="get_builddir",
                           command=["pwd"],
                           property="builddir",
                           description="set build dir",
                           workdir="."))
     # Get source code
     f = getLLDBSource(f,llvm_srcdir)
 
     # Construct cmake
     cmake_args = ["cmake", "-GNinja"]
     if build_compiler == "clang":
         cmake_args.append("-DCMAKE_C_COMPILER=clang")
         cmake_args.append("-DCMAKE_CXX_COMPILER=clang++")
     elif build_compiler == "gcc":
         cmake_args.append("-DCMAKE_C_COMPILER=gcc")
         cmake_args.append("-DCMAKE_CXX_COMPILER=g++")
 
-    if test_compiler == "clang":
-        cmake_args.append("-DLLDB_TEST_COMPILER=clang")
-    elif test_compiler == "gcc":
-        cmake_args.append("-DLLDB_TEST_COMPILER=gcc")
-
     cmake_args.append(WithProperties("-DCMAKE_BUILD_TYPE=%s" % build_type))
     cmake_args.append(WithProperties("../%s" % llvm_srcdir))
 
     # Clean Build Folder
     f.addStep(ShellCommand(name="clean",
                            command="rm -rf *",
                            description="clear build folder",
                            env=env,
                            workdir='%s' % llvm_builddir))
     # Configure
     f.addStep(Configure(name='configure/cmake',
                         command=cmake_args,
                         env=env,
                         workdir=llvm_builddir))
     # Compile
     f.addStep(WarningCountingShellCommand(name="compile/ninja",
                                           command=['nice', '-n', '10',
                                                    'ninja', WithProperties("-j%s" % jobs)],
                                           env=env,
                                           haltOnFailure=True,
                                           workdir=llvm_builddir))
-    # Test
-    f.addStep(LitTestCommand(name="test lldb",
-                             command=['nice', '-n', '10',
-                                      'ninja',
-                                      'check-lldb'],
+
+    # Run Test with list of compilers and archs
+    bindir='%(builddir)s/' + llvm_builddir + '/bin'
+
+    for compiler in test_compilers:
+        # find full path for top of tree clang
+        if compiler=='totclang':
+            compiler=bindir+'/clang'
+        for arch in test_arch:
+            f.addStep(LitTestCommand(name="test lldb (%s-%s)" % (compiler, arch),
+                                     command=['../%s/tools/lldb/test/dosep.py' % llvm_srcdir,
+                                              '--options',
+                                              WithProperties(''.join(['-q ',
+                                                                      '--arch=%s ' % arch,
+                                                                      '--executable ' + bindir + '/lldb ',
+                                                                      '-C ' + compiler + ' ',
+                                                                      '-s lldb-test-traces ',
+                                                                      '-u CXXFLAGS ',
+                                                                      '-u CFLAGS']))],
                                      description="test lldb",
                                      parseSummaryOnly=True,
                                      env=env,
                                      workdir='%s' % llvm_builddir))
     return f
 
 def getLLDBxcodebuildFactory(use_cc=None):
     f = buildbot.process.factory.BuildFactory()
     f.addStep(SetProperty(name='get_builddir',
                           command=['pwd'],
                           property='builddir',
                           description='set build dir',