Archive test traces to outer directory named by build number after lldb-test

[chying]

Summary

The test traces will be stored temporarily on slave machine
There will be a sheduled task running on slave machine to zip and upload traces to google storage

[sivachandra]

zorg/buildbot/builders/LLDBBuilder.py

306 ↗

(On Diff #24849)

The summary note says that a separate task would zip and upload traces to google storage. Why cant it be done inline right here as part of this step?

[chying]

zorg/buildbot/builders/LLDBBuilder.py

306 ↗

(On Diff #24849)

There were two concerns behind this, one is that certain builder might not have access to google storage, or would upload the traces to different locations. Another is that in case of network latency, doing this step inline would slow down the builder, zip&upload step normally only take 6 seconds though.

[sivachandra]

zorg/buildbot/builders/LLDBBuilder.py

306 ↗

(On Diff #24849)

I don't understand the first of the concerns. For the second one, chromium builders all archive to google storage. To my knowledge, the archive step was never treated as a time bottle neck.

[chying]

zorg/buildbot/builders/LLDBBuilder.py

306 ↗

(On Diff #24849)

For the first point, I was saying that adding this step inline will require any builders calling this function have google storage access, i.e. builders have no google storage access won't be able to call this factory function.
I am fine with adding this step inline, all of the current lldb builders calling this function have google storage access.

[sivachandra]

zorg/buildbot/builders/LLDBBuilder.py

306 ↗

(On Diff #24849)

Unless there is a compelling reason, I do not see why this step should not be inline. If not inline, you are also burdening yourselves with the maintenance of another process.

[chying]

zip and upload test traces to gs by shell command

[sivachandra]

lgtm

zorg/trunk/zorg/buildbot/builders/LLDBBuilder.py

def getLLDBUbuntuCMakeBuildFactory(build_compiler,
     # Remote test steps
     if remote_configs is not None:
         for config in remote_configs:
             f = getLLDBRemoteTestSteps(f,
                                        bindir,
                                        build_type,
                                        config,
                                        env)
+    # archive test traces
+    f = archiveLLDBTestTraces(f)
+    return f
 
+# zip and upload test traces to google storage
+def archiveLLDBTestTraces(f):
+    llvm_builddir = 'build'
+    f.addStep(ShellCommand(name="compress test traces",
+                           command=WithProperties("zip -r build-%(buildnumber)s lldb-test-traces-*"),
+                           description="zip",
+                           haltOnFailure=False,
+                           workdir='%s' % llvm_builddir))
+    f.addStep(ShellCommand(name="upload test traces",
+                           command=['gsutil',
+                                    'mv',
+                                    WithProperties('build-%(buildnumber)s.zip'),
+                                    WithProperties('gs://lldb_test_traces/%(buildername)s/')],
+                           description="upload to Google Storage",
+                           haltOnFailure=False,
+                           workdir='%s' % llvm_builddir))
     return f
 
 # for cmake and compile
 def getLLDBCmakeAndCompileSteps(f,
                                 build_compiler,
                                 build_type,
                                 ninja_target,
                                 bindir,