Create an experimental Windows LLDB Builder

[zturner]

Summary

This creates a Windows buildbot for LLDB, with the following properties:

1. Only builds x86.
2. Only compiles, does not run tests.
3. Uses MSVC 2013 as the compiler, with the CMake / ninja generator.

I wanted to test this by setting up a local zorg buildmaster and connecting my slave to it, but I was unable to get it working properly. I'm going on vacation next week, so I thought I would put this up in the meantime and at least see if I'm going about this all wrong, or if this code looks like a reasonable starting point.

[timurrrr]

+Evgeniy who was adding other bots lately.

buildbot/osuosl/master/config/builders.py

617 ↗

(On Diff #17526)

please specify the VS version where these env values came from

630 ↗

(On Diff #17526)

Are you sure the framework envs are required? I don't remember setting them on our other bots

643 ↗

(On Diff #17526)

Ditto NETFX dirs

zorg/buildbot/builders/LLDBBuilder.py

29 ↗

(On Diff #17526)

Evgeniy might have a different opinion, e.g. suggest to use annotator instead?

[eugenis]

zorg/buildbot/builders/LLDBBuilder.py

29 ↗

(On Diff #17526)

Whatever works for you.

[timurrrr]

What would you use yourself if you wanted to create a brand new bot?

[zturner]

Removed manual setting of environment and do it automatically using vcvarsall. Also cleaned up the interface to the lldb cmake windows build factory generator by sinking some more logic into the factory generator method.

Still having trouble setting up a zorg buildmaster locally, but I will keep trying.

[zturner]

Post-holiday bump. I assume people are back to their regular schedules by now, so Galina could you please take a look?

[gkistanova]

Hello Zachary,

Thanks for working on this!

It is almost there. Please see my comments inline above.
Feel free to commit with the fixed ## 2 and 4.

Don't worry about having a local master. I'll stage the changes and will let you know if anything would go wrong. I feel much more comfortable with the current patch.

buildbot/osuosl/master/config/slaves.py

174 ↗

(On Diff #17674)

The 'jobs' property is not used by the factory. See the note # 4 below.

zorg/buildbot/builders/LLDBBuilder.py

28 ↗

(On Diff #17674)

1. Having all the env vars from a slave could be too much and noisy in the logs. How about white list only what you really need? This is not a show stopper, though.

39 ↗

(On Diff #17674)

2. Could you use raw strings here and everywhere else for Windows paths with dir separators, please? Like r'C:\src\python\'.

45 ↗

(On Diff #17674)

3. Maybe add an optional env param here and support optional env property propagated from a slave and builder definitions to merge with the envs you are requesting from the slave?

In some cases it might be needed to define something in addition.

Not a how stopper and could wait till someone would really need the feature. Just be nice to have it.

72 ↗

(On Diff #17674)

4. Here you are missing the 'jobs' property defined with the slave. You need to reconcile the 'jobs' param with the 'jobs' property before building the cmd string.

[zturner]

Thanks for the comments. I will work on a fix for the issues you pointed out.

zorg/buildbot/builders/LLDBBuilder.py

28 ↗

(On Diff #17674)

I thought about that, but it's difficult to know exactly which ones we need. There are some that are obvious, like include paths, library paths, etc, but some that are not so obvious. In theory we just need to be able find cl.exe, link.exe, and include / library paths, but the use of many of the other variables is kind of mysterious and not well documented by Microsoft, so it's not clear if we will trigger their access somehow.

In other words, I can come up with a whitelist but I would not be 100% confident that it would be accurate.

39 ↗

(On Diff #17674)

Unfortunately it seems like a raw string cannot end with a single backslash. This might be a bug in python, but I tested it out. r'C:\src\python\' is an error, and r'C:\src\python\\' actually puts two backslashes at the end.

I can probably change it to not include any backslashes at the end though, and then just make sure that the case of 0 backslashes and 1 backslash are both correctly handled in the function.

[gkistanova]

zorg/buildbot/builders/LLDBBuilder.py

28 ↗

(On Diff #17674)

Agree. This could be time consuming. Adding a FIXME: with the description of the issue would do for now.

[zturner]

Fixed suggested issues. I didn't fix #3 yet because I may need that feature when I add support for running the test suite on the build bot, which I plan to implement soon. To keep the changelist smaller and easier to diagnose errors, I will implement it then (if it ends up being needed).

I'm also not sure I did the property resolution correctly. I tried to go from some example about how other builders do it, but again with no way to test it in a real master/slave environment I'm not sure it's correct.

[rfoos]

zorg/buildbot/builders/LLDBBuilder.py

14 ↗

(On Diff #17674)

I think these are intended to define a 32 or 64 bit host executable for X86 or ARM.

They not to define an LLVM cross compiler target arch (i.e. AARCH64, Hexagon...)

To match the build host, vcvarsall %PROCESSOR_ARCHITECTURE%

21 ↗

(On Diff #17674)

Why not use buildbot's SetProperty? It has a facility for returning stdout. This seems like more work to support.

28 ↗

(On Diff #17674)

I'm not sure the env is so much longer using this method. The existing slave environment on Windows is quite long, and many of the variables are needed for other purposes.

Once you have the slave environment into a dictionary you can define merge and strip functions.

39 ↗

(On Diff #17674)

I second the motion for 0 trailing backslashes. I've not had a case where this has been a problem to date.

x=r"""c:\asdf\\"""
print x

c:\asdf\\

I think it's a bug too, and that I would have no luck in getting it accepted by the python community :)

Since you have to quote all the paths in windows, this appears to work

x=r""""c:\asdf\ fdas \""""
print x

"c:\asdf\ fdas \"

But I think something else that escapes this string will mess it up later.

45 ↗

(On Diff #17674)

I'm suggesting a generic way to invoke VCVARSALL.BAT that works on any buildslave.

This is for a 64 bit MSVC build, using a 64 bit version of Python.
slavenvCmd=r"""PATH=C:\Python2764;C:\Python2764\Scripts;%PATH% & "%VS120COMNTOOLS%\..\..\VC\vcvarsall.bat" amd64 & set""",

(As part of this, I have to admit that I don't have a generic method for identifying a 64 bit python needed for 64 bit builds of LLDB)

If you have multiple MSVC installations, these variables provide a reliable index to vcvarsall.bat. The locations are installation dependant, and the values can be different for each buildslave.

VS100COMNTOOLS=c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools
\
VS110COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools
\
VS120COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools
\
ATM, MSVC 2013 is all that works. As it was in the past, MSVC 2014/2015 will become an option soon.

side note, VSINSTALLDIR isn't defined until after vcvarsall is run and a specific version selected.

72 ↗

(On Diff #17674)

I'm having the same bug, but can we add loadaverage?

I have fast servers, and they overload themselves, keep going and plow through builds with 140+ loadaverages (32 being 100% utilization)

[zturner]

zorg/buildbot/builders/LLDBBuilder.py

14 ↗

(On Diff #17674)

For the above 2 cases, it's talking about both the host AND the target. In other words:

"vcvarsall x86" will use a 32-bit toolchain that produces 32-bit target executables
"vcvarsall amd64" will use a 64-bit toolchain that produces 64-bit target executables.

There are actually more values you can specify for the first argument to vcvarsall for cross compilation, but I'm not supporting those here.

I provided this argument because this way we can (later) provide slaves for both x86 and x64 on the same physical builder.

21 ↗

(On Diff #17674)

I guess because I don't know how it works. Isn't this the standard pythonic way to do this though?

45 ↗

(On Diff #17674)

I don't want to put Python in the path because LLDB requires linking against a manually compiled python, and I don't want anything to get confused. Also if we ever make a builder with a new version of MSVC (say 2015), you would have to change this. It seems like the way I've done it is sufficient, you just supply it a path to any version of MSVC, where it defaults to MSVC 2013, and it just works.

[zturner]

zorg/buildbot/builders/LLDBBuilder.py

45 ↗

(On Diff #17674)

Read this for information about why LLDB requires a custom built version of Python

http://lldb.llvm.org/build.html#BuildingLldbOnWindows

One reason I'm fond of the solution I've got here is because both the python source dir and the MSVC dir are configurable, so it's future proof against MSVC versions and should easily handle the case of specifying a 64-bit python. Just build 32 and 64-bit pythons from different source trees, and pass a different value for the python source dir.

[rfoos]

zorg/buildbot/builders/LLDBBuilder.py

21 ↗

(On Diff #17674)

buildbotic vs. pythonic :)
I'm OK with it as long as the subprocess is running on the buildslave. I had thought that code like that was running on the buildmaster.

45 ↗

(On Diff #17674)

Currently I'm building with a standard python release, and running with a pre-built MSVC2013 python.

We were doing fairly well without it, but when LLDB is a debug version, the python also has to be a debug version and none of the released pythons are built for debug.

changing the slave_envCmd from the builder is all you have to do to switch things as well, but the slave_envCmd also works for Linux r"""printenv""" which was one of my goals in a Windows/Linux build factory.

[zturner]

zorg/buildbot/builders/LLDBBuilder.py

45 ↗

(On Diff #17674)

Not sure what you mean about building vs. running. LLDB embeds python, and then links against itself via an extension module, so whatever you build with is also what you run with. The extension module must be compiled with the same version of MSVC as the version of Python that LLDB links against.

AFAIK there is nothing special about Debug / Release here. I used to think there was, but then I found out that the same issue is present for Release. So you always need to compile python, regardless of whether it's a debug or release build of LLDB.

[rfoos]

zorg/buildbot/builders/LLDBBuilder.py

45 ↗

(On Diff #17674)

You are right. Somebody else stashed an MSVC2013 python and I missed it. We are linking against a 32 or 64 bit version of MSVC2013 python.

So we don't need to compile python on each build, but once for each build config matching the lldb build config.

[gkistanova]

zorg/buildbot/builders/LLDBBuilder.py

72 ↗

(On Diff #17674)

Yes, at the end we need both, the job and the loadaverage.

[zturner]

Galina, Are you ok with the patch as is, now that I've addressed 1, 2, and 4?

[gkistanova]

Looks good to me. Please feel free to commit.

The "job" property wouldn't work, but this is fine for now. I'll follow up.

zorg/trunk/buildbot/osuosl/master/config/builders.py

return [
                                                     extra_configure_args=['--enable-cxx11', '--enable-optimized', '--enable-assertions'],
                                                     env={'PATH':'/home/llvmbb/bin/clang-latest/bin:/home/llvmbb/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games'})},
         {'name': "lldb-x86_64-freebsd",
          'slavenames': ["as-bldslv5"],
          'builddir': "lldb-x86_64-freebsd",
          'factory': LLDBBuilder.getLLDBBuildFactory(triple=None, # use default
                                                     make='gmake',
                                                     extra_configure_args=['--enable-cxx11', '--enable-optimized', '--enable-assertions'])},
+        {'name': "lldb-x86-windows-msvc",
+         'slavenames': ["zturner-win2008"],
+         'builddir': "lldb-windows-x86",
+         'factory': LLDBBuilder.getLLDBWindowsCMakeBuildFactory(config='Debug')},
        ]
 
 # Offline.
 {'name': "lldb-x86_64-linux",
  'slavenames': ["gcc20"],
  'builddir': "lldb-x86_64",
  'factory': LLDBBuilder.getLLDBBuildFactory(triple="x86_64-pc-linux-gnu",
                                             env={'CXXFLAGS' : '-std=c++0x'})},

zorg/trunk/buildbot/osuosl/master/config/slaves.py

32},
         create_slave("systemz-1", properties={'jobs': 4}, max_builds=1),
 
         # Ubuntu 14.04 x86_64, Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz
         create_slave("ericwf-buildslave", properties={'jobs': 4}, max_builds=2),
         # Ubuntu 14.10 x86_64, Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
         create_slave('ericwf-buildslave2', properties={'jobs': 4}, max_builds=2),
         # OS X 10.10 x86_64, Intel Core 2 Duo @ 2.40GHz
         create_slave("ericwf-osx-slave", properties={'jobs': 2}, max_builds=1),
+
+        # Windows Server 2008 R2, Quad 2.6GHz Intel Xeon(R) 4GB RAM
+        create_slave("zturner-win2008", properties={'jobs': 4}, max_builds=1),
         # Defunct.
 
 #        # GCC Compile Farm Slaves, see http://gcc.gnu.org/wiki/CompileFarm
 
 #        # gcc10  2TB   2x12x1.5 GHz AMD Opteron Magny-Cours / 64 GB RAM / Supermicro AS-1022G-BTF / Debian x86-64
 #        create_slave("gcc10", properties={'jobs' : 12}, max_builds=1),
 #        # gcc11  580G  2x 2x2.0 GHz AMD Opteron 2212 / 4GB RAM / Dell SC1345 / Debian x86-64
 #        create_slave("gcc11", properties={'jobs' : 2}, max_builds=1),

zorg/trunk/zorg/buildbot/builders/LLDBBuilder.py

 import os
+import subprocess
 
 import buildbot
 import buildbot.process.factory
 from buildbot.steps.source import SVN
 from buildbot.steps.shell import Configure, SetProperty
 from buildbot.steps.shell import ShellCommand, WarningCountingShellCommand
 from buildbot.process.properties import WithProperties
+import zorg.buildbot.commands.BatchFileDownload as batch_file_download
 from zorg.buildbot.commands.LitTestCommand import LitTestCommand
 
+def generateVisualStudioEnvironment(vs_install_dir, target_arch):
+    arch_arg = {'x86': 'x86', 'x64': 'amd64', 'amd64': 'amd64'}.get(target_arch, None)
+    if arch_arg is None:
+        return None
+    
+    vcvars_command = "\"" + os.path.join(vs_install_dir, 'VC', 'vcvarsall.bat') + "\""
+    vcvars_command = "%s %s && set" % (vcvars_command, arch_arg)
+    process = subprocess.Popen(vcvars_command, shell = True, stdout=subprocess.PIPE, stderr = None)
+    (stdout, _) = process.communicate()
+    vars = stdout.splitlines()
+    env = {}
+    # At some point it may be worth trying to see if we can create an trimmed down whitelist of only
+    # those environment variables which are necessary in order for the parts of the toolchain that
+    # we need, such as the compiler and linker.
+    for var in vars:
+        keyval_pair = var.split('=')
+        key = keyval_pair[0]
+        value = keyval_pair[1]
+        env[key] = value
+
+    return env
+
+# CMake Windows builds
+def getLLDBWindowsCMakeBuildFactory(
+            clean=True,
+            cmake='cmake',
+            jobs="%(jobs)s",
+
+            # Source directory containing a built python
+            python_source_dir=r'C:\src\python',
+
+            # Default values for VS version and build configuration
+            vs_install_dir=r'C:\Program Files (x86)\Microsoft Visual Studio 12.0',
+            config='Release',
+            target_arch='x86',
+
+            extra_cmake_args=[]):
+
+    ############# PREPARING
+    f = buildbot.process.factory.BuildFactory()
+
+    env = generateVisualStudioEnvironment(vs_install_dir, target_arch)
+
+    # We *must* checkout at least Clang, LLVM, and LLDB.  Once we add a step to run
+    # tests (e.g. ninja check-lldb), we will also need to add a step for LLD, since
+    # MSVC LD.EXE cannot link executables with DWARF debug info.
+    f.addStep(SVN(name='svn-llvm',
+                  mode='update', baseURL='http://llvm.org/svn/llvm-project/llvm/',
+                  defaultBranch='trunk',
+                  workdir='llvm'))
+    f.addStep(SVN(name='svn-clang',
+                  mode='update', baseURL='http://llvm.org/svn/llvm-project/cfe/',
+                  defaultBranch='trunk',
+                  workdir='llvm/tools/clang'))
+    f.addStep(SVN(name='svn-lldb',
+                  mode='update', baseURL='http://llvm.org/svn/llvm-project/lldb/',
+                  defaultBranch='trunk',
+                  workdir='llvm/tools/lldb'))
+
+    ninja_cmd=['ninja', WithProperties("-j%s" % jobs)]
+
+    # Global configurations
+    build_dir='build'
+
+    ############# CLEANING
+    if clean:
+        f.addStep(ShellCommand(name='clean',
+                               command=['rmdir', '/S/Q', build_dir],
+                               warnOnFailure=True,
+                               description='Cleaning',
+                               descriptionDone='clean',
+                               workdir='.',
+                               env=env))
+
+    if config.lower() == 'release':
+        python_lib = 'python27.lib'
+        python_exe = 'python.exe'
+    elif config.lower() == 'debug':
+        python_lib = 'python27_d.lib'
+        python_exe = 'python_d.exe'
+
+    python_lib = os.path.join(python_source_dir, 'PCbuild', python_lib)
+    python_exe = os.path.join(python_source_dir, 'PCbuild', python_exe)
+    python_include = os.path.join(python_source_dir, 'Include')
+
+    # Use batch files instead of ShellCommand directly, Windows quoting is
+    # borked. FIXME: See buildbot ticket #595 and buildbot ticket #377.
+    f.addStep(batch_file_download.BatchFileDownload(name='cmakegen',
+                                command=[cmake, "-G", "Ninja", "../llvm",
+                                         "-DCMAKE_BUILD_TYPE="+config,
+                                         # Need to use our custom built version of python
+                                         '-DPYTHON_LIBRARY=' + python_lib,
+                                         '-DPYTHON_INCLUDE_DIR=' + python_include,
+                                         '-DPYTHON_EXECUTABLE=' + python_exe]
+                                         + extra_cmake_args,
+                                workdir=build_dir))
+
+    f.addStep(ShellCommand(name='cmake',
+                           command=['cmakegen.bat'],
+                           haltOnFailure=True,
+                           description='cmake gen',
+                           workdir=build_dir,
+                           env=env))
+
+    f.addStep(WarningCountingShellCommand(name='build',
+                                          command=ninja_cmd,
+                                          haltOnFailure=True,
+                                          description='ninja build',
+                                          workdir=build_dir,
+                                          env=env))
+
+    return f
+
 def getLLDBBuildFactory(
             triple,
             outOfDir=False,
             useTwoStage=False,
             make='make',
             jobs='%(jobs)s',
             extra_configure_args=[],
             env={},