Add support for lldb remote test

[chying]

Summary

• Write function to add lldb remote test steps
• Modify lldb cmake factory to accept optional remote test related arguments
• Add remote test on x86&clang-3.5 to cmake builder

[chaoren]

Please see comments below:

buildbot/osuosl/master/config/builders.py

724 ↗

(On Diff #22915)

Does this have to be hardcoded? You should probably use a hostname instead if it must be hardcoded.

zorg/buildbot/builders/LLDBBuilder.py

266 ↗

(On Diff #22915)

screen -d -m

Why not ssh -f?

--listen *:5432

Is this remote machine accessible from anywhere besides the build machine? If you know the hostname of the build machine, you should probably use that instead of *. Also, must the port be hardcoded?

[sivachandra]

zorg/buildbot/builders/LLDBBuilder.py

295 ↗

(On Diff #22915)

Lines 272-295 and lines 385-406 should probably live in a common function.

376 ↗

(On Diff #22915)

The remote test steps should probably be added after the local test steps? That is, after line 406?

[chying]

buildbot/osuosl/master/config/builders.py

724 ↗

(On Diff #22915)

The remote test instance was created with static ip, so this should be fine. It was not easy to debug with hostname, because gce instance cannot be reached by hostname from workstation(maybe there is some way that i don't know about).
But hostname could be used from one gce to another.
I will find if hostname works.

zorg/buildbot/builders/LLDBBuilder.py

266 ↗

(On Diff #22915)

"ssh -f" doesn't detach the process from current shell, the process is still running in the background when using "ssh -f". Since buildbot will wait for current process exit to continue next step, so use "screen -d -m" to return immediately.

Yes, it's accessible from any machine for authorized users.
Same as above, I will find if hostname works.
Port could be specified by builder.

[chying]

zorg/buildbot/builders/LLDBBuilder.py

295 ↗

(On Diff #22915)

Yes, I will re-factor the code.

376 ↗

(On Diff #22915)

I put it here, to avoid making changes to 378-379
But I will re-factor the code as suggested above.

[chying]

Refactor code to have a common getLLDBTestSteps function
Use hostname instead of ip address for remote host
Remote host only listen to connection from builder

[chying]

Remove local workaround

[chying]

Remove hard-coded values of remote target information, get these from cfg file

[sivachandra]

Mostly LGTM.
You should probably add vharron to the reviews, or at least add him as a subscriber so that he in the know of the changes you are making, and also of the changes that I am suggesting.

zorg/buildbot/builders/LLDBBuilder.py

310 ↗

(On Diff #23031)

You should put the remote config in a JSON file. That will eliminate parsing. Something like remote_cfg.json. You can generate this JSON from Python interpreter (and need not hand create it).

313 ↗

(On Diff #23031)

For my knowledge: Do the extracted dictionary keys become properties automagically?

367 ↗

(On Diff #23031)

Nit: Fix the indentation of the function params here to align with the opening '('.

[chying]

zorg/buildbot/builders/LLDBBuilder.py

310 ↗

(On Diff #23031)

That is probably a good idea. I have to learn more about JSON.
This will require Json parser installed, like 'jq', is that right?

313 ↗

(On Diff #23031)

Yes, dictionary keys returned from extract_fn will be added to property list. This code has been verified.

367 ↗

(On Diff #23031)

This should be ok based on PEP8.
https://www.python.org/dev/peps/pep-0008/

[chying]

Put configuration of remote target to a Json file

[sivachandra]

LGTM to avoid another round trip. But please address comments before submitting.

zorg/buildbot/builders/LLDBBuilder.py

14 ↗

(On Diff #23099)

Nit: This is standard library module. Should be above 'import os' at line #1.

312 ↗

(On Diff #23099)

The JSON file itself should have keys 'remote_host', 'remote_port', and 'remote_dir'. You just do this:

return json.loads(stdout)

369 ↗

(On Diff #23099)

Nit (repeating from last time): Fix the indentation of the params to align to the opening '('.

zorg/trunk/buildbot/osuosl/master/config/builders.py

return [
         {'name': "lldb-x86_64-ubuntu-14.04-cmake",
          'slavenames': ["lldb-build1-ubuntu-1404"],
          'builddir': "buildWorkingDir",
          'factory': LLDBBuilder.getLLDBUbuntuCMakeBuildFactory(
                     build_compiler="clang",
                     build_type="Debug",
                     test_archs=['x86_64', 'i386'],
                     test_compilers=['clang', 'totclang', 'gcc4.8.2'],
+                    remote_test_archs=['x86_64'],
+                    remote_test_compilers=['clang'],
                     env={'SHELL':"/bin/bash"})},
         {'name': "lldb-x86_64-darwin-13.4",
          'slavenames': ["lldb-x86_64-darwin-13.4"],
          'builddir': "buildDir",
          'factory': LLDBBuilder.getLLDBxcodebuildFactory()},
        ]
 
 # Offline.

zorg/trunk/zorg/buildbot/builders/LLDBBuilder.py

 import os
 
 import buildbot
 import buildbot.process.factory
 from buildbot.steps.source import SVN
 from buildbot.steps.shell import Configure, SetProperty
 from buildbot.steps.shell import ShellCommand, WarningCountingShellCommand
 from buildbot.steps.slave import RemoveDirectory
 from buildbot.process.properties import WithProperties, Property
 import zorg.buildbot.commands.BatchFileDownload as batch_file_download
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 from zorg.buildbot.builders.Util import getVisualStudioEnvironment
 from zorg.buildbot.builders.Util import extractSlaveEnvironment
+import json
 
 # We *must* checkout at least Clang, LLVM, and LLDB.  Once we add a step to run
 # tests (e.g. ninja check-lldb), we will also need to add a step for LLD, since
 # MSVC LD.EXE cannot link executables with DWARF debug info.
 def getLLDBSource(f,llvmTopDir='llvm'):
     f.addStep(SVN(name='svn-llvm',
                   mode='update', baseURL='http://llvm.org/svn/llvm-project/llvm/',
                   defaultBranch='trunk',
def getLLDBBuildFactory(
     f.addStep(LitTestCommand(name="test lldb",
                              command=['nice', '-n', '10',
                                       make],
                              description="test lldb",
                              env=env,
                              workdir='%s/tools/lldb/test' % llvm_objdir))
 
     return f
+# Add test steps from list of compilers and archs
+def getLLDBTestSteps(f,
+                     isRemoteTest,
+                     bindir,
+                     test_archs,
+                     test_compilers,
+                     remote_host=None,
+                     remote_port=None,
+                     remote_dir=None,
+                     env=None):
+    # Skip test steps if no test compiler or arch is specified
+    if None in (test_archs, test_compilers):
+        return f
+    llvm_srcdir = "llvm"
+    llvm_builddir = "build"
+    for compiler in test_compilers:
+        # find full path for top of tree clang
+        if compiler=='totclang':
+            compilerPath=bindir+'/clang'
+        else:
+            compilerPath = compiler
+        for arch in test_archs:
+            DOTEST_OPTS=''.join(['--executable ' + bindir + '/lldb ',
+                                 '-A %s ' % arch,
+                                 '-C %s ' % compilerPath,
+                                 '-s lldb-test-traces-%s-%s ' % (compiler, arch),
+                                 '-u CXXFLAGS ',
+                                 '-u CFLAGS'])
+            testname = "local"
+            if(isRemoteTest):
+                DOTEST_OPTS+=''.join([' --platform-name remote-linux ',
+                                      '--platform-url connect://%(remote_host)s:%(remote_port)s ',
+                                      '--platform-working-dir %(remote_dir)s'])
+                testname = "remote"
+            f.addStep(LitTestCommand(name="test lldb %s (%s-%s)" % (testname, compiler, arch),
+                                     command=['../%s/tools/lldb/test/dosep.py' % llvm_srcdir,
+                                              '--options',
+                                              WithProperties(DOTEST_OPTS)],
+                                     description="test lldb",
+                                     parseSummaryOnly=True,
+                                     workdir='%s' % llvm_builddir))
+            f=cleanSVNSourceTree(f, '%s/tools/lldb' % llvm_srcdir)
+    return f
+
+# Add steps to run lldb test on remote target
+def getLLDBRemoteTestSteps(f,
+                           bindir,
+                           remote_test_archs,
+                           remote_test_compilers,
+                           env):
+
+    if None in (remote_test_archs, remote_test_compilers):
+        return f
+    llvm_srcdir = "llvm"
+    llvm_builddir = "build"
+    # get hostname
+    slave_hostname=None
+    f.addStep(SetProperty(name="get hostname",
+                          command=["hostname"],
+                          property="slave_hostname",
+                          description="set slave hostname",
+                          workdir="."))
+    # get configuration of remote target
+    # config file should be placed under builddir on builder machine
+    # file name: remote_cfg.json
+    # content: json format with keys "remote_host", "remote_port", and "remote_dir"
+    # example: {"remote_host":"remotehostname","remote_port":"1234","remote_dir":"/path/to/dir"}
+    def getRemoteCfg(rc, stdout, stderr):
+       return json.loads(stdout)
+    f.addStep(SetProperty(name="get remote target",
+                          command="cat remote_cfg.json",
+                          extract_fn=getRemoteCfg,
+                          description="get remote target",
+                          workdir="."))
+    # rsync
+    f.addStep(ShellCommand(name="rsync lldb-server",
+                           command=WithProperties("rsync -hav bin/lldb-server* %(remote_host)s:%(remote_dir)s"),
+                           description="rsync lldb-server",
+                           haltOnFailure=True,
+                           env=env,
+                           workdir='%s' % llvm_builddir))
+    f.addStep(ShellCommand(name="rsync python2.7",
+                           command=WithProperties("rsync -havL lib/python2.7 %(remote_host)s:%(remote_dir)s"),
+                           description="rsync python2.7",
+                           haltOnFailure=True,
+                           env=env,
+                           workdir='%s' % llvm_builddir))
+    # launch lldb-server
+    f.addStep(ShellCommand(name="launch lldb-server",
+                           command=WithProperties("ssh %(remote_host)s screen -d -m %(remote_dir)s/lldb-server platform --listen %(slave_hostname)s:%(remote_port)s --server"),
+                           description="launch lldb-server on remote host",
+                           env=env,
+                           haltOnFailure=True,
+                           workdir='%s' % llvm_builddir))
+    # test steps
+    f = getLLDBTestSteps(f,
+                         True,
+                         bindir,
+                         remote_test_archs,
+                         remote_test_compilers,
+                         '%(remote_host)s',
+                         '%(remote_port)s',
+                         '%(remote_dir)s',
+                         env)
+    # terminate lldb-server on remote host
+    f.addStep(ShellCommand(name="terminate lldb-server remote",
+                           command=WithProperties("ssh %(remote_host)s pkill lldb-server"),
+                           description="terminate lldb-server",
+                           env=env,
+                           workdir='%s' % llvm_builddir))
+    return f
 
 # Cmake bulid on Ubuntu
 # Build command sequence - cmake, ninja, ./dosep
 # Note: If test_archs or test_compilers is not specified, lldb-test will not be added to build factory
-def getLLDBUbuntuCMakeBuildFactory(
-            build_compiler,
+def getLLDBUbuntuCMakeBuildFactory(build_compiler,
                                    build_type,
                                    test_archs=None,
                                    test_compilers=None,
+                                   remote_test_archs=None,
+                                   remote_test_compilers=None,
                                    jobs='%(jobs)s',
                                    env=None,
                                    *args,
                                    **kwargs):
 
     if env is None:
         env={}
 
     llvm_srcdir = "llvm"
     llvm_builddir = "build"
+    bindir='%(builddir)s/' + llvm_builddir + '/bin'
 
     f = buildbot.process.factory.BuildFactory()
 
     # Determine the build directory.
     f.addStep(SetProperty(name="get_builddir",
                           command=["pwd"],
                           property="builddir",
                           description="set build dir",
f.addStep(Configure(name='configure/cmake',
                         workdir=llvm_builddir))
     # Compile
     f.addStep(WarningCountingShellCommand(name="compile/ninja",
                                           command=['nice', '-n', '10',
                                                    'ninja', WithProperties("-j%s" % jobs)],
                                           env=env,
                                           haltOnFailure=True,
                                           workdir=llvm_builddir))
-    # Skip lldb-test if no test compiler or arch is specified
-    if not test_archs or not test_compilers:
-        return f
 
     # TODO: it will be good to check that architectures listed in test_archs are compatible with host architecture
     # For now, the caller of this function should make sure that each target architecture is supported by builder machine
 
-    # Run Test with list of compilers and archs
-    bindir='%(builddir)s/' + llvm_builddir + '/bin'
-
-    for compiler in test_compilers:
-        # find full path for top of tree clang
-        if compiler=='totclang':
-            compilerPath=bindir+'/clang'
-        else:
-            compilerPath = compiler
-        for arch in test_archs:
-            f.addStep(LitTestCommand(name="test lldb (%s-%s)" % (compiler, arch),
-                                     command=['../%s/tools/lldb/test/dosep.py' % llvm_srcdir,
-                                              '--options',
-                                              WithProperties(''.join(['-q ',
-                                                                      '--arch=%s ' % arch,
-                                                                      '--executable ' + bindir + '/lldb ',
-                                                                      '-C ' + compilerPath + ' ',
-                                                                      '-s lldb-test-traces-%s-%s ' % (compiler, arch),
-                                                                      '-u CXXFLAGS ',
-                                                                      '-u CFLAGS']))],
-                                     description="test lldb",
-                                     parseSummaryOnly=True,
-                                     env=env,
-                                     workdir='%s' % llvm_builddir))
-            f=cleanSVNSourceTree(f, '%s/tools/lldb' % llvm_srcdir)
+    # Add local test steps
+    f = getLLDBTestSteps(f,
+                         False,
+                         bindir,
+                         test_archs,
+                         test_compilers)
+    # Remote test
+    f = getLLDBRemoteTestSteps(f,
+                               bindir,
+                               remote_test_archs,
+                               remote_test_compilers,
+                               env)
     return f
 
 def getLLDBxcodebuildFactory(use_cc=None):
     f = buildbot.process.factory.BuildFactory()
     f.addStep(SetProperty(name='get_builddir',
                           command=['pwd'],
                           property='builddir',
                           description='set build dir',