Disable for new Linux OS runs: upload test traces

[jankratochvil]

Summary

Currently my Linux OS run always fails the last step such as in:
http://lab.llvm.org:8014/builders/lldb-x86_64-fedora-28-cmake/builds/251

upload test traces './uploadTestTrace.sh 251 ...' failed

Pavel Labath even suggested this step is only internal for Google and it should be removed.

[jankratochvil]

ping

[jankratochvil]

ping#2

[rengolin]

This makes sense to me, but I'm not well versed in LLDB testing. Adding Pavel and Omair to approve it.

[omjavaid]

This was much needed. LGTM

[labath]

This is fine for enabling the usage of the buildbot factory for bots which do not have the same setup as the (now-mostly-unmaintained "linux cmake" buildbot). However, I would still recommend to create a new factory with a setup more similar to the other llvm bots, as this factory is very unlike any other existing llvm bots. Plus it uses a deprecated flow for running tests -- it uses dotest.py as a driver while tests are now normally run through lit.

[jankratochvil]

I have followed the @labath advice and ported the Windows lit run to Linux. I have been running it with local buildbot master but please double-check what you can.
I have merged the patch also with a rename of my buildslave which is the currently only one using that getLLDBCMakeBuildFactory.
Be aware existing slaves (lldb-x86-windows-msvc2015 and lldb-x64-windows-ninja) have been ported to the new getLLDBCMakeBuildFactory, hopefuly they run fine.
Also please rename the password for lldb-x86_64-fedora-28-cmake to be now for lldb-x86_64-fedora as it better matches what it is.
The new slave is running already:

2019-02-06 20:47:50+0100 [-] Connecting to lab.llvm.org:9990
2019-02-06 20:47:55+0100 [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective
2019-02-06 20:47:55+0100 [Broker,client] unauthorized login; check slave name and password

Thanks.

[labath]

Thank you for taking the time to do this. I may soon start using this factory myself. :)

The changes seem pretty straightforward to me, but it would be great if @stella.stamenova could look at this too.

[stella.stamenova]

This looks good. After it's checked in, it won't take effect until the master buildbot is updated though. You'd need @gkistanova to update it. She should also sign off on this.

[zturner]

I just uploaded a patch to delete a bunch of abandoned / broken LLDB bots, its' here. https://reviews.llvm.org/D57911

It looks like I assumed the Fedora one was abandoned, but from this patch, it looks like perhaps it's actually not? If that is the case, I'll update my patch to not remove that one.

[jankratochvil]

In D54386#1389280, @zturner wrote:

https://reviews.llvm.org/D57911
It looks like I assumed the Fedora one was abandoned, but from this patch, it looks like perhaps it's actually not? If that is the case, I'll update my patch to not remove that one.

Fedora one has been being prepared as a silent one to be deployed as a real one - by this patch. I also plan to deploy a CentOS/RHEL one (or two) after the Fedora one runs fine. Please do not remove it (but then I rename it anyway so I can also update this patch to add it back under a new name, I do not mind).

[gkistanova]

Thanks for making this more clean, Jan!
Since your bot is on the silent master yet, I can push the change there as soon as you commit.

zorg/trunk/buildbot/osuosl/master/config/builders.py

def _get_rev_iter_builders():
     ]
 
 # LLDB builders.
 def _get_lldb_builders():
     return [
         {'name': "lldb-x86-windows-msvc2015",
          'slavenames': ["zturner-win2008"],
          'builddir': "lldb-windows-x86",
-         'factory': LLDBBuilder.getLLDBWindowsCMakeBuildFactory(test=False)},
+         'factory': LLDBBuilder.getLLDBCMakeBuildFactory(
+                    test=False,
+                    python_source_dir=r'C:/Python35',
+                    vs=r"""%VS140COMNTOOLS%""")},
         # Disable the builder till we fix the cmake configuration
         #{'name': "lldb-x86-win7-msvc",
         #'slavenames': ["hexagon-build-01"],
         #'builddir': "builddir/lldb-win7-msvc",
-        #'factory': LLDBBuilder.getLLDBWindowsCMakeBuildFactory(config='Debug')},
+        #'factory': LLDBBuilder.getLLDBCMakeBuildFactory(
+        #           config='Debug',
+        #           python_source_dir=r'C:/Python35',
+        #           vs=r"""%VS140COMNTOOLS%""")},
         {'name': "lldb-x86_64-ubuntu-14.04-buildserver",
          'slavenames': ["lldb-linux-android-buildserver"],
          'builddir': "lldb-android-buildserver",
          'category' : 'lldb',
          'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
                     downloadBinary=False,
                     buildAndroid=True,
                     runTest=False,
return [
                     runTest=True)},
         {'name': "lldb-amd64-ninja-freebsd11",
          'slavenames': ["lldb-amd64-ninja-freebsd11"],
          'builddir': "scratch",
          'category' : 'lldb',
          'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
                     downloadBinary=False,
                     runTest=False)},
-        {'name': "lldb-x86_64-fedora-28-cmake",
-         'slavenames': ["lldb-x86_64-fedora-28-cmake"],
-         'builddir': "lldb-x86_64-fedora-28-cmake",
+        {'name': "lldb-x86_64-fedora",
+         'slavenames': ["lldb-x86_64-fedora"],
+         'builddir': "lldb-x86_64-fedora",
          'category' : 'lldb',
-         'factory': LLDBBuilder.getLLDBScriptCommandsFactory(
-                    downloadBinary=False,
-                    runTest=True,
-                    extra_cmake_args=["-DLLVM_ENABLE_ASSERTIONS=True",
-                                      "-DLLVM_USE_LINKER=gold"])},
+         'factory': LLDBBuilder.getLLDBCMakeBuildFactory(
+                    clean=True,
+                    test=True,
+                    extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=True',
+                                      '-DLLVM_USE_LINKER=gold'])},
         {'name': "lldb-x64-windows-ninja",
          'slavenames': ["win-py3-buildbot"],
          'builddir': "lldb-x64-windows-ninja",
          'category' : 'lldb',
-         'factory': LLDBBuilder.getLLDBWindowsCMakeBuildFactory(
+         'factory': LLDBBuilder.getLLDBCMakeBuildFactory(
                     clean=True,
                     python_source_dir=r'"C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64"',
                     target_arch='x64',
                     vs="autodetect",
                     test=True,
                     extra_cmake_args=['-DLLVM_ENABLE_ASSERTIONS=OFF'])}
        ]
 

zorg/trunk/buildbot/osuosl/master/config/slaves.py

return [
 
         # Defunct.
         # Debian 86_64, 2 x 6-core Opteron 2.6 GHz
         #create_slave("osu8", properties={'jobs' : 6}, max_builds=2),
 
         # OpenBSD amd64
         create_slave("openbsd-amd64", properties={'jobs': 2}, max_builds=1),
 
-        # test only: Fedora 28 x86_64, Intel Xeon E5-2630v3, 32 HT cores, 64GB RAM
-        create_slave("lldb-x86_64-fedora-28-cmake", properties={'jobs': 16}, max_builds=1),
+        # test only: Fedora latest stable x86_64, Intel i5-2500, 4 cores, 12GB RAM
+        create_slave("lldb-x86_64-fedora", properties={'jobs': 4}, max_builds=1),
 
         # Ubuntu 14.04 x86_64, Intel(R) Xeon(R) CPU @ 2.30GHz
         #create_slave("llgo-builder", properties={'jobs': 2}, max_builds=1),
 
         # LLVM Lab slaves
         #create_slave("hpproliant1", properties={'jobs': 6}, max_builds=1),
 
         # Windows 6.1.7601 x86-64, Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz

zorg/trunk/buildbot/osuosl/master/config/status.py

return [
         InformativeMailNotifier(
             fromaddr = "[e-mail]",
             sendToInterestedUsers= False,
             extraRecipients = ["[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",
             builders = ["fuchsia-x86_64-linux"],
             addLogs=False),
+        InformativeMailNotifier(
+            fromaddr = "[e-mail]",
+            sendToInterestedUsers = True,
+            extraRecipients = ["[e-mail]"],
+            subject="Build %(builder)s Failure",
+            mode = "failing",
+            builders = ["lldb-x86_64-fedora"],
+            addLogs=False),
         ]

zorg/trunk/zorg/buildbot/builders/LLDBBuilder.py

 # List all untracked files, and remove one by one
 def cleanSVNSourceTree(f, srcdir='llvm'):
     f.addStep(ShellCommand(name='clean svn source %s' % srcdir,
                            command="svn status --no-ignore | grep '^[I?]' | cut -c 9- | while IFS= read -r f; do echo \"$f\"; rm -rf \"$f\"; done",
                            description="clean SVN source tree",
                            workdir='%s' % srcdir))
     return f
 
-# CMake Windows builds
-def getLLDBWindowsCMakeBuildFactory(
+# CMake builds
+def getLLDBCMakeBuildFactory(
             clean=False,
             cmake='cmake',
             jobs="%(jobs)s",
 
             # Source directory containing a built python
-            python_source_dir=r'C:/Python35',
+            python_source_dir=None,
 
             # Default values for VS devenv and build configuration
-            vs=r"""%VS140COMNTOOLS%""",
+            vs=None,
             config='Release',
             target_arch='x86',
 
             extra_cmake_args=None,
             test=False,
             install=False):
 
     ############# PREPARING
     f = buildbot.process.factory.BuildFactory()
 
     # Determine Slave Environment and Set MSVC environment.
+    if vs:
         f.addStep(SetProperty(
             command=getVisualStudioEnvironment(vs, target_arch),
             extract_fn=extractSlaveEnvironment))
 
     f = getLLDBSource(f,'llvm')
 
     build_cmd=['ninja']
     install_cmd = ['ninja','install']
     test_cmd = ['ninja','check-lldb']
 
     if jobs:
f.addStep(RemoveDirectory(name='clean '+build_dir,
                 haltOnFailure=False,
                 flunkOnFailure=False,
                 doStepIf=cleanBuildRequested
                 ))
 
     cmake_cmd = [
         "cmake", "-G", "Ninja", "../llvm",
         "-DCMAKE_BUILD_TYPE=" + config,
-        "-DPYTHON_HOME=" + python_source_dir,
         "-DCMAKE_INSTALL_PREFIX=../install"
         ]
+    if python_source_dir:
+        cmake_cmd.append("-DPYTHON_HOME=" + python_source_dir)
     if extra_cmake_args:
         cmake_cmd += extra_cmake_args
     # Note: ShellCommand does not pass the params with special symbols right.
     # The " ".join is a workaround for this bug.
     f.addStep(ShellCommand(name="cmake-configure",
                            description=["cmake configure"],
                            command=WithProperties(" ".join(cmake_cmd)),
                            haltOnFailure=True,