This is an archive of the discontinued LLVM Phabricator instance.

convert SBDebugger::***FileHandle() wrappers to native files.
ClosedPublic

Authored by lawrence_danna on Oct 10 2019, 7:46 PM.

Details

Summary

This patch converts the swig wrappers for SetInputFileHandle() and friends
to emulate the old behavior using SetInputFile().

This will clear the way for deleting the FILE* typemaps altogether.

Event Timeline

lawrence_danna created this revision.Oct 10 2019, 7:46 PM
Herald added a project: Restricted Project. · View Herald TranscriptOct 10 2019, 7:46 PM
labath added inline comments.Oct 11 2019, 1:12 AM
lldb/packages/Python/lldbsuite/test/python_api/file_handle/TestFileHandle.py
827–835

I think these will fail with swig<=3. See my r374322. I personally don't think you need these checks for _every_ api taking accepting a file object, but if you do want to have them, you'll need to account for the differences in swig versions somehow.

fix for swig 3

lawrence_danna marked an inline comment as done.Oct 11 2019, 11:00 AM
labath accepted this revision.Oct 14 2019, 12:37 AM
This revision is now accepted and ready to land.Oct 14 2019, 12:37 AM
This revision was automatically updated to reflect the committed changes.
mgorny added a subscriber: mgorny.Oct 21 2019, 1:06 PM

I'm afraid this causes a few test failures on NetBSD, e.g.: http://lab.llvm.org:8011/builders/netbsd-amd64/builds/22255/steps/run%20unit%20tests/logs/FAIL%3A%20lldb-api%3A%3ATestDefaultConstructorForAPIObjects.py

Could you help me debugging it? I haven't been working on the Python API that deep, so I'd use any help I can get. For a start, a way to make that error message point to a specific operation would be helpful.

I'm afraid this causes a few test failures on NetBSD, e.g.: http://lab.llvm.org:8011/builders/netbsd-amd64/builds/22255/steps/run%20unit%20tests/logs/FAIL%3A%20lldb-api%3A%3ATestDefaultConstructorForAPIObjects.py

Could you help me debugging it? I haven't been working on the Python API that deep, so I'd use any help I can get. For a start, a way to make that error message point to a specific operation would be helpful.

sure, um do you have a netbsd machine handy?

sure, um do you have a netbsd machine handy?

If you mean one I could you access to, then I'm afraid not.

sure, um do you have a netbsd machine handy?

If you mean one I could you access to, then I'm afraid not.

ok, i've got a VM, how do I install git on this thing?

I spoke too soon, I don't have a *usable* netbsd VM.

@mgorny, Do you have a machine you can debug on yourself? Do those errors happen if you run the same test on an interactive terminal? It could be the stdin stream is not set to a valid file descriptor for the CI run. Can you step through the SWIG wrappers and tell me where it errors out?

@mgorny

can you at least point me at the setup scripts buildbot us using for netbsd? What dependencies are installed? How is cmake invoked? I'm getting a ton on unrelated problems just trying to build it.

I spoke too soon, I don't have a *usable* netbsd VM.

@mgorny, Do you have a machine you can debug on yourself?

Yes.

Do those errors happen if you run the same test on an interactive terminal?

Yes, if by that you mean lit run on console. I haven't tried running it outside lit (yet).

Can you step through the SWIG wrappers and tell me where it errors out?

I'm planning to do this later today.

can you at least point me at the setup scripts buildbot us using for netbsd? What dependencies are installed? How is cmake invoked? I'm getting a ton on unrelated problems just trying to build it.

https://github.com/mgorny/netbsd-llvm-build/tree/master/buildbotScripts/bashShell/svntotbuild

We're using setEnv, checkoutSource, cmake, buildLocal, in this order. I suppose only cmake matters to you, the stuff in build is a hack to reduce memory usage. Also relevant is that we're running 8.x kernel there (netbsd-8 branch if you're using git).

We install stuff via pkgsrc. Once you do the bootstrap, /usr/pkgsrc/devel/cmake, /usr/pkgsrc/devel/git and /usr/pkgsrc/devel/swig3 is what you need, optionally /usr/pkgsrc/devel/ninja-build. I don't recall needing anything else.

@mgorny

I can't get anything to work. I've tried running a local VM with virtualbox but it's networking driver crashes my kernel. I've tried a local VM with VMware but it won't boot netbsd. I've tried AWS but they only have netbsd 7, which is too old. I've tried google cloud, but their image creator script only works for netbsd 9, and python won't build because x11 isn't installed. I've tried installing the pkgsrc binaries from netbsd 8 onto netbsd 9, but that doesn't work either. I'm completely at a loss. I can't figure out how to make a netbsd VM that can actually build LLDB.

Do you have a machine image on AWS or google cloud or even a VMDK or something that I could use?

That's dedication. I wasn't able to get it to run on qemu-kvm either. Regular qemu works bit the performance is terrible. I ended installing it as a second system on my laptop.

I'll try to figure it out myself. If you could suggest roughly what syscalls are involved, that might be helpful. Otherwise, I'll figure out how to run it outside of lit and/or add some cheap debugging hacks.

krytarowski added a subscriber: krytarowski.EditedOct 22 2019, 3:35 AM

@mgorny

I can't get anything to work. I've tried running a local VM with virtualbox but it's networking driver crashes my kernel. I've tried a local VM with VMware but it won't boot netbsd. I've tried AWS but they only have netbsd 7, which is too old. I've tried google cloud, but their image creator script only works for netbsd 9, and python won't build because x11 isn't installed. I've tried installing the pkgsrc binaries from netbsd 8 onto netbsd 9, but that doesn't work either. I'm completely at a loss. I can't figure out how to make a netbsd VM that can actually build LLDB.

Do you have a machine image on AWS or google cloud or even a VMDK or something that I could use?

For the python part, you need to either install all basesystem sets (including x11) or set X11_TYPE=modular in /etc/mk.conf.

All my NetBSD setups are either native NetBSD or inside qemu or qemu+haxm (both hosted on NetBSD). I cannot test virtualbox or vmware as of now as I don't have non-NetBSD computers.

For the git part you will need to install mozilla-rootcerts and follow post-install instructions, as otherwise https:// won't work nicely.

lawrence_danna added a comment.EditedOct 22 2019, 10:06 AM

@mgorny

I can't get anything to work. I've tried running a local VM with virtualbox but it's networking driver crashes my kernel. I've tried a local VM with VMware but it won't boot netbsd. I've tried AWS but they only have netbsd 7, which is too old. I've tried google cloud, but their image creator script only works for netbsd 9, and python won't build because x11 isn't installed. I've tried installing the pkgsrc binaries from netbsd 8 onto netbsd 9, but that doesn't work either. I'm completely at a loss. I can't figure out how to make a netbsd VM that can actually build LLDB.

Do you have a machine image on AWS or google cloud or even a VMDK or something that I could use?

For the python part, you need to either install all basesystem sets (including x11) or set X11_TYPE=modular in /etc/mk.conf.

I tried X11_TYPE, it does nothing. I'm a bit confused there because I did not have a file called /etc/mk.conf.

Is there a way to install those missing basesystem sets once the machine is up? Or do I have to go back and reinstall from scratch?

PS: oh, of course I have to set the environment variable MAKECONF=/etc/mk.conf

now I'm stuck on this trying to install cmake.

pkg_add: no pkg found for 'libunistring>=0.9.4', sorry.
pkg_add: Can't install dependency libunistring>=0.9.4
pkg_add: 1 package addition failed
*** Error code 1
krytarowski added a comment.EditedOct 22 2019, 10:47 AM

now I'm stuck on this trying to install cmake.

pkg_add: no pkg found for 'libunistring>=0.9.4', sorry.
pkg_add: Can't install dependency libunistring>=0.9.4
pkg_add: 1 package addition failed
*** Error code 1

I recommend to just use pkgsrc from source code. As of today packages are imperfect and always there is a set of things that are not prebuilt.

If there are mixed packages for binary packages vs source ones, it's reasonable to rm -rf /usr/pkg /var/db/pkg before starting.

Steps:

  1. Download pkgsrc from tarball.

http://cdn.netbsd.org/pub/pkgsrc/pkgsrc-2019Q3/pkgsrc-2019Q3.tar.bz2

  1. Unpack, e.g. into /usr/pkgsrc
  2. cd /usr/pkgsrc/devel/git-base && make install
  3. cd /usr/pkgsrc/security/mozilla-rootcerts && make install

And follow the MESSAGE commands to finish the installation.

  1. Follow this process for all the dependencies (if I remember correctly: cmake, swig3, ninja-build).

@mgorny

I can't get anything to work. I've tried running a local VM with virtualbox but it's networking driver crashes my kernel. I've tried a local VM with VMware but it won't boot netbsd. I've tried AWS but they only have netbsd 7, which is too old. I've tried google cloud, but their image creator script only works for netbsd 9, and python won't build because x11 isn't installed. I've tried installing the pkgsrc binaries from netbsd 8 onto netbsd 9, but that doesn't work either. I'm completely at a loss. I can't figure out how to make a netbsd VM that can actually build LLDB.

Do you have a machine image on AWS or google cloud or even a VMDK or something that I could use?

For the python part, you need to either install all basesystem sets (including x11) or set X11_TYPE=modular in /etc/mk.conf.

I tried X11_TYPE, it does nothing. I'm a bit confused there because I did not have a file called /etc/mk.conf.

Is there a way to install those missing basesystem sets once the machine is up? Or do I have to go back and reinstall from scratch?

The typical approach is to fetch set file and untarball it into /, all other approaches are at most wrappers to this. The list of installed sets is stored in /etc/mtree/.

Please remember to use tar options to preserve file properties.

http://pub.nethence.com/bsd/wo.sysinst notes: tar xzphfe $set.tgz -C /.

Another option is to disable x11 option either for all packages or for python. In my opinion it's easier to just set the modular in the X11_TYPE variable.

PS: oh, of course I have to set the environment variable MAKECONF=/etc/mk.conf

/etc/mk.conf is the default file for MAKECONF, no need to set this variable.

In the meantime, I found the root issue. It's in PythonDataObjects.cpp, PythonFile::ConvertToFile(bool borrowed). The function unconditionally calls flush() on the fd. However, it's invalid to call flush() on stdin or — more generally — on file open only for reading. I'm going to submit a patch in a few minutes.

In the meantime, I found the root issue. It's in PythonDataObjects.cpp, PythonFile::ConvertToFile(bool borrowed). The function unconditionally calls flush() on the fd. However, it's invalid to call flush() on stdin or — more generally — on file open only for reading. I'm going to submit a patch in a few minutes.

🎉