Page MenuHomePhabricator

[zorg] Port LLDB cmake build factory to git
ClosedPublic

Authored by labath on Oct 23 2019, 6:45 AM.

Details

Summary

The changes appear to be straight-forward, but I don't know how to test them
without committing to zorg.

Event Timeline

labath created this revision.Oct 23 2019, 6:45 AM

FYI my local buildbot master errors out this way:

2019-10-23 16:10:12+0200 [-] error while parsing config file
2019-10-23 16:10:12+0200 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/home/buildbot/.local/lib/python2.7/site-packages/buildbot-latest-py2.7.egg/buildbot/master.py", line 197, in loadTheConfigFile
	    d = self.loadConfig(f)
	  File "/home/buildbot/.local/lib/python2.7/site-packages/buildbot-latest-py2.7.egg/buildbot/master.py", line 579, in loadConfig
	    d.addCallback(do_load)
	  File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 322, in addCallback
	    callbackKeywords=kw)
	  File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 311, in addCallbacks
	    self._runCallbacks()
	--- <exception caught here> ---
	  File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/home/buildbot/.local/lib/python2.7/site-packages/buildbot-latest-py2.7.egg/buildbot/master.py", line 226, in do_load
	    exec f in localDict
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/master.cfg", line 83, in <module>
	    c['builders'] = builders = list(config.builders.get_builders())
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/config/builders.py", line 1444, in get_builders
	    for b in _get_lldb_builders():
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/config/builders.py", line 817, in _get_lldb_builders
	    '-DLLVM_LIT_ARGS="-v"'])},
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/zorg/buildbot/builders/LLDBBuilder.py", line 77, in getLLDBCMakeBuildFactory
	    f.addGetSourcecodeSteps()
	exceptions.AttributeError: BuildFactory instance has no attribute 'addGetSourcecodeSteps'
	
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/master.cfg", line 83, in <module>
	    c['builders'] = builders = list(config.builders.get_builders())
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/config/builders.py", line 1444, in get_builders
	    for b in _get_lldb_builders():
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/config/builders.py", line 817, in _get_lldb_builders
	    '-DLLVM_LIT_ARGS="-v"'])},
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/zorg/buildbot/builders/LLDBBuilder.py", line 77, in getLLDBCMakeBuildFactory
	    f.addGetSourcecodeSteps()
	exceptions.AttributeError: BuildFactory instance has no attribute 'addGetSourcecodeSteps'
labath updated this revision to Diff 226154.Oct 23 2019, 9:35 AM

Create the appropriate build factory

Thanks for trying it out. The last update should fix that error. Could I trouble you to give it another spin?

2019-10-23 20:34:53+0200 [-] error while parsing config file
2019-10-23 20:34:53+0200 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/home/buildbot/.local/lib/python2.7/site-packages/buildbot-latest-py2.7.egg/buildbot/master.py", line 197, in loadTheConfigFile
	    d = self.loadConfig(f)
	  File "/home/buildbot/.local/lib/python2.7/site-packages/buildbot-latest-py2.7.egg/buildbot/master.py", line 579, in loadConfig
	    d.addCallback(do_load)
	  File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 322, in addCallback
	    callbackKeywords=kw)
	  File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 311, in addCallbacks
	    self._runCallbacks()
	--- <exception caught here> ---
	  File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/home/buildbot/.local/lib/python2.7/site-packages/buildbot-latest-py2.7.egg/buildbot/master.py", line 226, in do_load
	    exec f in localDict
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/master.cfg", line 83, in <module>
	    c['builders'] = builders = list(config.builders.get_builders())
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/config/builders.py", line 1444, in get_builders
	    for b in _get_lldb_builders():
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/config/builders.py", line 817, in _get_lldb_builders
	    '-DLLVM_LIT_ARGS="-v"'])},
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/zorg/buildbot/builders/LLDBBuilder.py", line 110, in getLLDBCMakeBuildFactory
	    "cmake", "-G", "Ninja", "../" + self.monorepo_dir,
	exceptions.NameError: global name 'self' is not defined

I could pack some patched buildbot-0.8.5 and patched zorg-git running on Fedora 30 (and its Python version) so that you can run it yourself. But I am also fine to just testing it.

sivachandra added inline comments.
zorg/buildbot/builders/LLDBBuilder.py
110

Should this be os.path.join(os.pardir, f.monorepo_dir, "llvm") ?

After the @sivachandra 's fix I get:

2019-10-23 21:28:24+0200 [-] LLVMGitPoller: using workdir '/quad/home/buildbot/zorg-git/buildbot/osuosl/master/gitpoller-workdir'
2019-10-23 21:28:24+0200 [-] LLVMGitPoller: initializing working dir from http://git.lab.llvm.org/llvm/llvm-project.git
2019-10-23 21:28:24+0200 [-] configuration update complete
2019-10-23 21:28:25+0200 [-] while initializing LLVMGitPoller repository
	Traceback (most recent call last):
	  File "/usr/lib64/python2.7/site-packages/twisted/internet/_baseprocess.py", line 64, in maybeCallProcessEnded
	    proto.processEnded(Failure(reason))
	  File "/usr/lib64/python2.7/site-packages/twisted/internet/utils.py", line 163, in processEnded
	    self.deferred.callback((out, err, code))
	  File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 460, in callback
	    self._startRunCallbacks(result)
	  File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
	    self._runCallbacks()
	--- <exception caught here> ---
	  File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/zorg/buildbot/changes/llvmgitpoller.py", line 419, in _convert_nonzero_to_failure
	    raise EnvironmentError('command failed with exit code %d: %s' % (code, stderr))
	exceptions.EnvironmentError: command failed with exit code 128: fatal: unable to access 'http://git.lab.llvm.org/llvm/llvm-project.git/': Could not resolve host: git.lab.llvm.org
zorg/buildbot/builders/LLDBBuilder.py
110

I was curious there is no longer trailing "llvm" as I am using cmake ../llvm-monorepo/llvm/. Yes, it does compile with the new expression there:

"cmake", "-G", "Ninja", os.path.join(os.pardir, f.monorepo_dir, "llvm"),
sivachandra added inline comments.Oct 23 2019, 1:39 PM
zorg/buildbot/builders/LLDBBuilder.py
71

I wonder why it is getting git.lab.llvm.org! Can you try with an additional arg to the LLVMBuildFactory constructor:

repourl_prefix="http://github.com/llvm/"
sivachandra added inline comments.Oct 23 2019, 1:42 PM
zorg/buildbot/builders/LLDBBuilder.py
71

May be https instead of http.

labath marked 2 inline comments as done.Oct 23 2019, 2:22 PM
labath added inline comments.
zorg/buildbot/builders/LLDBBuilder.py
71

Are you sure that's needed? I don't see anyone else setting that. @jankratochvil, could this be something specific to your setup?

110

Technically, using os.path.join, is not correct here, because this is a path on the buildbot, not the path on the host which runs the master. In practice that doesn't matter because the master runs on a posix system. The existing code is pretty inconsistent about the usage, but the ../ seems to be a bit more common.

labath updated this revision to Diff 226183.Oct 23 2019, 2:23 PM

s/self.monorepo_dir/f.monorepo_dir

Thanks. This may come in useful, though I hope I don't have to go back in here for a long time. :)

sivachandra added inline comments.Oct 23 2019, 3:08 PM
zorg/buildbot/builders/LLDBBuilder.py
71

If I am reading the code right, it should not be needed. I suggested that to see if @jankratochvil can make progress, which validates the rest of this patch.

110

Right. The main point I was trying to convey was to add the path to the llvm directory.

I am only interested in seeing what you eventually land as I want to use the same for the LLVM libc builders.

labath updated this revision to Diff 226200.Oct 23 2019, 10:26 PM

add '/llvm' ...

jankratochvil added inline comments.Oct 24 2019, 3:44 AM
zorg/buildbot/builders/LLDBBuilder.py
71

It is still at: https://github.com/llvm/llvm-zorg/blob/master/buildbot/osuosl/master/master.cfg#L59
When I delete local /home/buildbot/zorg-git/buildbot/osuosl/master/gitpoller-workdir it will get rebuilt and it errors again:

2019-10-24 12:34:33+0200 [-] adding 3 new changesources, removing 0
2019-10-24 12:34:33+0200 [-] LLVMGitPoller: using workdir '/quad/home/buildbot/zorg-git/buildbot/osuosl/master/gitpoller-workdir'
2019-10-24 12:34:33+0200 [-] LLVMGitPoller: initializing working dir from http://git.lab.llvm.org/llvm/llvm-project.git
2019-10-24 12:34:33+0200 [-] configuration update complete
2019-10-24 12:34:33+0200 [-] while initializing LLVMGitPoller repository
        Traceback (most recent call last):
          File "/usr/lib64/python2.7/site-packages/twisted/internet/_baseprocess.py", line 64, in maybeCallProcessEnded
            proto.processEnded(Failure(reason))
          File "/usr/lib64/python2.7/site-packages/twisted/internet/utils.py", line 163, in processEnded
            self.deferred.callback((out, err, code))
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 460, in callback
            self._startRunCallbacks(result)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/quad/home/buildbot/zorg-git/buildbot/osuosl/master/zorg/buildbot/changes/llvmgitpoller.py", line 419, in _convert_nonzero_to_failure
            raise EnvironmentError('command failed with exit code %d: %s' % (code, stderr))
        exceptions.EnvironmentError: command failed with exit code 128: fatal: unable to access 'http://git.lab.llvm.org/llvm/llvm-project.git/': Could not resolve host: git.lab.llvm.org
sivachandra added inline comments.Oct 24 2019, 8:43 AM
zorg/buildbot/builders/LLDBBuilder.py
71

Ah, sorry! I missed that you were showing the master stack trace.

I wonder if its a repo which only the master can see. Look at this: http://lab.llvm.org:8011/builders/clang-ppc64be-linux-lnt/builds/32069

It seems to indicate that the master is tracking the git.lab.llvm repo, but "Checkout the source code" step is fetching from github.

labath marked 5 inline comments as done.Oct 24 2019, 8:50 AM
labath added inline comments.
zorg/buildbot/builders/LLDBBuilder.py
71

I'm afraid I am starting to get a bit lost here. Have we reached some kind of a conclusion? Should we maybe just commit it and see what happens? The bots are broken now anyway...

110

Got it. Thanks.

jankratochvil accepted this revision.Oct 24 2019, 9:29 AM
jankratochvil added inline comments.
zorg/buildbot/builders/LLDBBuilder.py
71

I agree your current patch should be an improvement of the current situation. I have no idea what is that LLVMGitPoller but it seems to be already broken and it is probably not required for the buildbots IIUC.

This revision is now accepted and ready to land.Oct 24 2019, 9:29 AM
labath marked 7 inline comments as done.Oct 24 2019, 2:21 PM

Thanks for your help Jan. Unfortunately, it looks like I don't know how to commit this to zorg. Does anyone know what's the process for that? @gkistanova, could you advise?

labath closed this revision.Oct 25 2019, 9:42 AM

Managed to commit this today (b37c659). We'll see what happens after a master restart.