Add support for lldb remote test

[chying]

Summary

• Write function to add lldb remote test steps
• Modify lldb cmake factory to accept optional remote test related arguments
• Add remote test on x86&clang-3.5 to cmake builder

[chaoren]

Please see comments below:

buildbot/osuosl/master/config/builders.py

724

Does this have to be hardcoded? You should probably use a hostname instead if it must be hardcoded.

zorg/buildbot/builders/LLDBBuilder.py

266

screen -d -m

Why not ssh -f?

--listen *:5432

Is this remote machine accessible from anywhere besides the build machine? If you know the hostname of the build machine, you should probably use that instead of *. Also, must the port be hardcoded?

[sivachandra]

zorg/buildbot/builders/LLDBBuilder.py

295

Lines 272-295 and lines 385-406 should probably live in a common function.

376

The remote test steps should probably be added after the local test steps? That is, after line 406?

[chying]

buildbot/osuosl/master/config/builders.py

724

The remote test instance was created with static ip, so this should be fine. It was not easy to debug with hostname, because gce instance cannot be reached by hostname from workstation(maybe there is some way that i don't know about).
But hostname could be used from one gce to another.
I will find if hostname works.

zorg/buildbot/builders/LLDBBuilder.py

266

"ssh -f" doesn't detach the process from current shell, the process is still running in the background when using "ssh -f". Since buildbot will wait for current process exit to continue next step, so use "screen -d -m" to return immediately.

Yes, it's accessible from any machine for authorized users.
Same as above, I will find if hostname works.
Port could be specified by builder.

[chying]

zorg/buildbot/builders/LLDBBuilder.py

295

Yes, I will re-factor the code.

376

I put it here, to avoid making changes to 378-379
But I will re-factor the code as suggested above.

[chying]

Refactor code to have a common getLLDBTestSteps function
Use hostname instead of ip address for remote host
Remote host only listen to connection from builder

[chying]

Remove local workaround

[chying]

Remove hard-coded values of remote target information, get these from cfg file

[sivachandra]

Mostly LGTM.
You should probably add vharron to the reviews, or at least add him as a subscriber so that he in the know of the changes you are making, and also of the changes that I am suggesting.

zorg/buildbot/builders/LLDBBuilder.py

310

You should put the remote config in a JSON file. That will eliminate parsing. Something like remote_cfg.json. You can generate this JSON from Python interpreter (and need not hand create it).

313

For my knowledge: Do the extracted dictionary keys become properties automagically?

319

Nit: Fix the indentation of the function params here to align with the opening '('.

[chying]

zorg/buildbot/builders/LLDBBuilder.py

310

That is probably a good idea. I have to learn more about JSON.
This will require Json parser installed, like 'jq', is that right?

313

Yes, dictionary keys returned from extract_fn will be added to property list. This code has been verified.

319

This should be ok based on PEP8.
https://www.python.org/dev/peps/pep-0008/

[chying]

Put configuration of remote target to a Json file

[sivachandra]

LGTM to avoid another round trip. But please address comments before submitting.

zorg/buildbot/builders/LLDBBuilder.py

14

Nit: This is standard library module. Should be above 'import os' at line #1.

311

The JSON file itself should have keys 'remote_host', 'remote_port', and 'remote_dir'. You just do this:

return json.loads(stdout)

319

Nit (repeating from last time): Fix the indentation of the params to align to the opening '('.

buildbot/osuosl/master/config/builders.py

return [
         {'name': "lldb-x86_64-ubuntu-14.04-cmake",
          'slavenames': ["lldb-build1-ubuntu-1404"],
          'builddir': "buildWorkingDir",
          'factory': LLDBBuilder.getLLDBUbuntuCMakeBuildFactory(
                     build_compiler="clang",
                     build_type="Debug",
                     test_archs=['x86_64', 'i386'],
                     test_compilers=['clang', 'totclang', 'gcc4.8.2'],
+                    remote_ip="173.255.116.181",
+                    remote_dir="/lldb-buildbot/remotetest",
+                    remote_test_archs=['x86_64'],
+                    remote_test_compilers=['clang'],
                     env={'SHELL':"/bin/bash"})},
         {'name': "lldb-x86_64-darwin-13.4",
          'slavenames': ["lldb-x86_64-darwin-13.4"],
          'builddir': "buildDir",
          'factory': LLDBBuilder.getLLDBxcodebuildFactory()},
        ]
 
 # Offline.

zorg/buildbot/builders/LLDBBuilder.py

 import os
 
 import buildbot
 import buildbot.process.factory
 from buildbot.steps.source import SVN
 from buildbot.steps.shell import Configure, SetProperty
 from buildbot.steps.shell import ShellCommand, WarningCountingShellCommand
 from buildbot.steps.slave import RemoveDirectory
 from buildbot.process.properties import WithProperties, Property
 import zorg.buildbot.commands.BatchFileDownload as batch_file_download
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.builders.Util import getVisualStudioEnvironment
 from zorg.buildbot.builders.Util import extractSlaveEnvironment
 
 # We *must* checkout at least Clang, LLVM, and LLDB.  Once we add a step to run
 # tests (e.g. ninja check-lldb), we will also need to add a step for LLD, since
 # MSVC LD.EXE cannot link executables with DWARF debug info.
 def getLLDBSource(f,llvmTopDir='llvm'):
     f.addStep(SVN(name='svn-llvm',
                   mode='update', baseURL='http://llvm.org/svn/llvm-project/llvm/',
                   defaultBranch='trunk',
                   workdir=llvmTopDir))
def getLLDBBuildFactory(
     f.addStep(LitTestCommand(name="test lldb",
                              command=['nice', '-n', '10',
                                       make],
                              description="test lldb",
                              env=env,
                              workdir='%s/tools/lldb/test' % llvm_objdir))
 
     return f
+# Add steps to run lldb test on remote target
+def addLLDBRemoteTest(f,
+                      bindir,
+                      remote_ip,
+                      remote_dir,
+                      remote_test_archs,
+                      remote_test_compilers,
+                      env):
+
+    if None in (remote_ip, remote_dir, remote_test_archs, remote_test_compilers):
+        return f
+    llvm_srcdir = "llvm"
+    llvm_builddir = "build"
+    # rsync
+    f.addStep(ShellCommand(name="rsync lldb-server",
+                           command="rsync -hav bin/lldb-server* %s:%s" % (remote_ip, remote_dir),
+                           description="rsync lldb-server",
+                           env=env,
+                           workdir='%s' % llvm_builddir))
+    f.addStep(ShellCommand(name="rsync python2.7",
+                           command="rsync -havL lib/python2.7 %s:%s" % (remote_ip, remote_dir),
+                           description="rsync python2.7",
+                           env=env,
+                           workdir='%s' % llvm_builddir))
+    # launch lldb-server
+    f.addStep(ShellCommand(name="launch lldb-server",
+                           command="ssh %s screen -d -m %s/lldb-server platform --listen *:5432 --server" % (remote_ip, remote_dir),
+                           description="launch lldb-server on remote host",
+                           env=env,
+                           workdir='%s' % llvm_builddir))
+
+    # Run Test with list of compilers and archs
+    for compiler in remote_test_compilers:
+        # find full path for top of tree clang
+        if compiler=='totclang':
+            compilerPath=bindir+'/clang'
+        else:
+            compilerPath = compiler
+        for arch in remote_test_archs:
+            DOTEST_OPTS=''.join(['--executable ' + bindir + '/lldb ',
+                                 '-A %s ' % arch,
+                                 '-C %s ' % compilerPath,
+                                 '-s lldb-test-traces-%s-%s ' % (compiler, arch),
+                                 '-u CXXFLAGS ',
+                                 '-u CFLAGS ',
+                                 '--platform-name remote-linux ',
+                                 '--platform-url connect://%s:5432 ' % remote_ip,
+                                 '--platform-working-dir %s' % remote_dir])
+            f.addStep(LitTestCommand(name="test lldb Remote (%s-%s)" % (compiler, arch),
+                                     command=['../%s/tools/lldb/test/dosep.py' % llvm_srcdir,
+                                              '--options',
+                                              WithProperties(DOTEST_OPTS)],
+                                     description="test lldb",
+                                     parseSummaryOnly=True,
+                                     workdir='%s' % llvm_builddir))
+            f=cleanSVNSourceTree(f, '%s/tools/lldb' % llvm_srcdir)
+    # terminate lldb-server on remote host
+    f.addStep(ShellCommand(name="terminate lldb-server",
+                           command="ssh %s pkill lldb-server" % remote_ip,
+                           description="terminate lldb-server",
+                           env=env,
+                           workdir='%s' % llvm_builddir))
+    return f
 
 # Cmake bulid on Ubuntu
 # Build command sequence - cmake, ninja, ./dosep
 # Note: If test_archs or test_compilers is not specified, lldb-test will not be added to build factory
 def getLLDBUbuntuCMakeBuildFactory(
             build_compiler,
             build_type,
             test_archs=None,
             test_compilers=None,
+            remote_ip=None,
+            remote_dir=None,
+            remote_test_archs=None,
+            remote_test_compilers=None,
             jobs='%(jobs)s',
             env=None,
             *args,
             **kwargs):
 
     if env is None:
         env={}
 
     llvm_srcdir = "llvm"
     llvm_builddir = "build"
+    bindir='%(builddir)s/' + llvm_builddir + '/bin'
 
     f = buildbot.process.factory.BuildFactory()
 
     # Determine the build directory.
     f.addStep(SetProperty(name="get_builddir",
                           command=["pwd"],
                           property="builddir",
                           description="set build dir",
f.addStep(Configure(name='configure/cmake',
                         workdir=llvm_builddir))
     # Compile
     f.addStep(WarningCountingShellCommand(name="compile/ninja",
                                           command=['nice', '-n', '10',
                                                    'ninja', WithProperties("-j%s" % jobs)],
                                           env=env,
                                           haltOnFailure=True,
                                           workdir=llvm_builddir))
+    # Remote test
+    f = addLLDBRemoteTest(f,
+                          bindir,
+                          remote_ip,
+                          remote_dir,
+                          remote_test_archs,
+                          remote_test_compilers,
+                          env)
     # Skip lldb-test if no test compiler or arch is specified
     if not test_archs or not test_compilers:
         return f
 
     # TODO: it will be good to check that architectures listed in test_archs are compatible with host architecture
     # For now, the caller of this function should make sure that each target architecture is supported by builder machine
 
     # Run Test with list of compilers and archs
-    bindir='%(builddir)s/' + llvm_builddir + '/bin'
-
     for compiler in test_compilers:
         # find full path for top of tree clang
         if compiler=='totclang':
             compilerPath=bindir+'/clang'
         else:
             compilerPath = compiler
         for arch in test_archs:
             f.addStep(LitTestCommand(name="test lldb (%s-%s)" % (compiler, arch),