convert SBDebugger::***FileHandle() wrappers to native files.

[lawrence_danna]

Summary

This patch converts the swig wrappers for SetInputFileHandle() and friends
to emulate the old behavior using SetInputFile().

This will clear the way for deleting the FILE* typemaps altogether.

[labath]

lldb/packages/Python/lldbsuite/test/python_api/file_handle/TestFileHandle.py

842–850

I think these will fail with swig<=3. See my r374322. I personally don't think you need these checks for _every_ api taking accepting a file object, but if you do want to have them, you'll need to account for the differences in swig versions somehow.

[lawrence_danna]

fix for swig 3

[lawrence_danna]

rebased.

[mgorny]

I'm afraid this causes a few test failures on NetBSD, e.g.: http://lab.llvm.org:8011/builders/netbsd-amd64/builds/22255/steps/run%20unit%20tests/logs/FAIL%3A%20lldb-api%3A%3ATestDefaultConstructorForAPIObjects.py

Could you help me debugging it? I haven't been working on the Python API that deep, so I'd use any help I can get. For a start, a way to make that error message point to a specific operation would be helpful.

[lawrence_danna]

In D68856#1717207, @mgorny wrote:

I'm afraid this causes a few test failures on NetBSD, e.g.: http://lab.llvm.org:8011/builders/netbsd-amd64/builds/22255/steps/run%20unit%20tests/logs/FAIL%3A%20lldb-api%3A%3ATestDefaultConstructorForAPIObjects.py

Could you help me debugging it? I haven't been working on the Python API that deep, so I'd use any help I can get. For a start, a way to make that error message point to a specific operation would be helpful.

sure, um do you have a netbsd machine handy?

[mgorny]

In D68856#1717247, @lawrence_danna wrote:

sure, um do you have a netbsd machine handy?

If you mean one I could you access to, then I'm afraid not.

[lawrence_danna]

In D68856#1717257, @mgorny wrote:

In D68856#1717247, @lawrence_danna wrote:

sure, um do you have a netbsd machine handy?

If you mean one I could you access to, then I'm afraid not.

ok, i've got a VM, how do I install git on this thing?

[lawrence_danna]

I spoke too soon, I don't have a *usable* netbsd VM.

@mgorny, Do you have a machine you can debug on yourself? Do those errors happen if you run the same test on an interactive terminal? It could be the stdin stream is not set to a valid file descriptor for the CI run. Can you step through the SWIG wrappers and tell me where it errors out?

[lawrence_danna]

@mgorny

can you at least point me at the setup scripts buildbot us using for netbsd? What dependencies are installed? How is cmake invoked? I'm getting a ton on unrelated problems just trying to build it.

[mgorny]

In D68856#1717332, @lawrence_danna wrote:

I spoke too soon, I don't have a *usable* netbsd VM.

@mgorny, Do you have a machine you can debug on yourself?

Yes.

Do those errors happen if you run the same test on an interactive terminal?

Yes, if by that you mean lit run on console. I haven't tried running it outside lit (yet).

Can you step through the SWIG wrappers and tell me where it errors out?

I'm planning to do this later today.

In D68856#1717567, @lawrence_danna wrote:

can you at least point me at the setup scripts buildbot us using for netbsd? What dependencies are installed? How is cmake invoked? I'm getting a ton on unrelated problems just trying to build it.

https://github.com/mgorny/netbsd-llvm-build/tree/master/buildbotScripts/bashShell/svntotbuild

We're using setEnv, checkoutSource, cmake, buildLocal, in this order. I suppose only cmake matters to you, the stuff in build is a hack to reduce memory usage. Also relevant is that we're running 8.x kernel there (netbsd-8 branch if you're using git).

We install stuff via pkgsrc. Once you do the bootstrap, /usr/pkgsrc/devel/cmake, /usr/pkgsrc/devel/git and /usr/pkgsrc/devel/swig3 is what you need, optionally /usr/pkgsrc/devel/ninja-build. I don't recall needing anything else.

[lawrence_danna]

@mgorny

I can't get anything to work. I've tried running a local VM with virtualbox but it's networking driver crashes my kernel. I've tried a local VM with VMware but it won't boot netbsd. I've tried AWS but they only have netbsd 7, which is too old. I've tried google cloud, but their image creator script only works for netbsd 9, and python won't build because x11 isn't installed. I've tried installing the pkgsrc binaries from netbsd 8 onto netbsd 9, but that doesn't work either. I'm completely at a loss. I can't figure out how to make a netbsd VM that can actually build LLDB.

Do you have a machine image on AWS or google cloud or even a VMDK or something that I could use?

[mgorny]

That's dedication. I wasn't able to get it to run on qemu-kvm either. Regular qemu works bit the performance is terrible. I ended installing it as a second system on my laptop.

I'll try to figure it out myself. If you could suggest roughly what syscalls are involved, that might be helpful. Otherwise, I'll figure out how to run it outside of lit and/or add some cheap debugging hacks.

[krytarowski]

In D68856#1717683, @lawrence_danna wrote:

@mgorny

I can't get anything to work. I've tried running a local VM with virtualbox but it's networking driver crashes my kernel. I've tried a local VM with VMware but it won't boot netbsd. I've tried AWS but they only have netbsd 7, which is too old. I've tried google cloud, but their image creator script only works for netbsd 9, and python won't build because x11 isn't installed. I've tried installing the pkgsrc binaries from netbsd 8 onto netbsd 9, but that doesn't work either. I'm completely at a loss. I can't figure out how to make a netbsd VM that can actually build LLDB.

Do you have a machine image on AWS or google cloud or even a VMDK or something that I could use?

For the python part, you need to either install all basesystem sets (including x11) or set X11_TYPE=modular in /etc/mk.conf.

All my NetBSD setups are either native NetBSD or inside qemu or qemu+haxm (both hosted on NetBSD). I cannot test virtualbox or vmware as of now as I don't have non-NetBSD computers.

[krytarowski]

For the git part you will need to install mozilla-rootcerts and follow post-install instructions, as otherwise https:// won't work nicely.

[lawrence_danna]

In D68856#1717767, @krytarowski wrote:

In D68856#1717683, @lawrence_danna wrote:

@mgorny

I can't get anything to work. I've tried running a local VM with virtualbox but it's networking driver crashes my kernel. I've tried a local VM with VMware but it won't boot netbsd. I've tried AWS but they only have netbsd 7, which is too old. I've tried google cloud, but their image creator script only works for netbsd 9, and python won't build because x11 isn't installed. I've tried installing the pkgsrc binaries from netbsd 8 onto netbsd 9, but that doesn't work either. I'm completely at a loss. I can't figure out how to make a netbsd VM that can actually build LLDB.

Do you have a machine image on AWS or google cloud or even a VMDK or something that I could use?

For the python part, you need to either install all basesystem sets (including x11) or set X11_TYPE=modular in /etc/mk.conf.

I tried X11_TYPE, it does nothing. I'm a bit confused there because I did not have a file called /etc/mk.conf.

Is there a way to install those missing basesystem sets once the machine is up? Or do I have to go back and reinstall from scratch?

PS: oh, of course I have to set the environment variable MAKECONF=/etc/mk.conf

[lawrence_danna]

now I'm stuck on this trying to install cmake.

pkg_add: no pkg found for 'libunistring>=0.9.4', sorry.
pkg_add: Can't install dependency libunistring>=0.9.4
pkg_add: 1 package addition failed
*** Error code 1

[krytarowski]

In D68856#1718096, @lawrence_danna wrote:

now I'm stuck on this trying to install cmake.

pkg_add: no pkg found for 'libunistring>=0.9.4', sorry.
pkg_add: Can't install dependency libunistring>=0.9.4
pkg_add: 1 package addition failed
*** Error code 1

I recommend to just use pkgsrc from source code. As of today packages are imperfect and always there is a set of things that are not prebuilt.

If there are mixed packages for binary packages vs source ones, it's reasonable to rm -rf /usr/pkg /var/db/pkg before starting.

Steps:

1. Download pkgsrc from tarball.

http://cdn.netbsd.org/pub/pkgsrc/pkgsrc-2019Q3/pkgsrc-2019Q3.tar.bz2

2. Unpack, e.g. into /usr/pkgsrc
3. cd /usr/pkgsrc/devel/git-base && make install
4. cd /usr/pkgsrc/security/mozilla-rootcerts && make install

And follow the MESSAGE commands to finish the installation.

5. Follow this process for all the dependencies (if I remember correctly: cmake, swig3, ninja-build).

In D68856#1718075, @lawrence_danna wrote:

In D68856#1717767, @krytarowski wrote:

In D68856#1717683, @lawrence_danna wrote:

@mgorny

I can't get anything to work. I've tried running a local VM with virtualbox but it's networking driver crashes my kernel. I've tried a local VM with VMware but it won't boot netbsd. I've tried AWS but they only have netbsd 7, which is too old. I've tried google cloud, but their image creator script only works for netbsd 9, and python won't build because x11 isn't installed. I've tried installing the pkgsrc binaries from netbsd 8 onto netbsd 9, but that doesn't work either. I'm completely at a loss. I can't figure out how to make a netbsd VM that can actually build LLDB.

Do you have a machine image on AWS or google cloud or even a VMDK or something that I could use?

For the python part, you need to either install all basesystem sets (including x11) or set X11_TYPE=modular in /etc/mk.conf.

I tried X11_TYPE, it does nothing. I'm a bit confused there because I did not have a file called /etc/mk.conf.

Is there a way to install those missing basesystem sets once the machine is up? Or do I have to go back and reinstall from scratch?

The typical approach is to fetch set file and untarball it into /, all other approaches are at most wrappers to this. The list of installed sets is stored in /etc/mtree/.

Please remember to use tar options to preserve file properties.

http://pub.nethence.com/bsd/wo.sysinst notes: tar xzphfe $set.tgz -C /.

Another option is to disable x11 option either for all packages or for python. In my opinion it's easier to just set the modular in the X11_TYPE variable.

PS: oh, of course I have to set the environment variable MAKECONF=/etc/mk.conf

/etc/mk.conf is the default file for MAKECONF, no need to set this variable.

[mgorny]

In the meantime, I found the root issue. It's in PythonDataObjects.cpp, PythonFile::ConvertToFile(bool borrowed). The function unconditionally calls flush() on the fd. However, it's invalid to call flush() on stdin or — more generally — on file open only for reading. I'm going to submit a patch in a few minutes.

[lawrence_danna]

In D68856#1718240, @mgorny wrote:

In the meantime, I found the root issue. It's in PythonDataObjects.cpp, PythonFile::ConvertToFile(bool borrowed). The function unconditionally calls flush() on the fd. However, it's invalid to call flush() on stdin or — more generally — on file open only for reading. I'm going to submit a patch in a few minutes.

🎉

lldb/packages/Python/lldbsuite/test/python_api/file_handle/TestFileHandle.py

def handleCmd(self, cmd, check=True, collect_result=True):
         self.debugger.GetOutputFile().Flush()
         self.debugger.GetErrorFile().Flush()
         if collect_result and check:
             self.assertTrue(ret.Succeeded())
         return ret.GetOutput()
 
 
     @add_test_categories(['pyapi'])
-    @skipIfWindows # FIXME pre-existing bug, should be fixed
-                   # when we delete the FILE* typemaps.
     def test_legacy_file_out_script(self):
         with open(self.out_filename, 'w') as f:
             self.debugger.SetOutputFileHandle(f, False)
             # scripts print to output even if you capture the results
             # I'm not sure I love that behavior, but that's the way
             # it's been for a long time.  That's why this test works
             # even with collect_result=True.
             self.handleCmd('script 1+1')
             self.debugger.GetOutputFileHandle().write('FOO\n')
         lldb.SBDebugger.Destroy(self.debugger)
         with open(self.out_filename, 'r') as f:
             self.assertEqual(readStrippedLines(f), ['2', 'FOO'])
 
 
     @add_test_categories(['pyapi'])
     def test_legacy_file_out(self):
         with open(self.out_filename, 'w') as f:
             self.debugger.SetOutputFileHandle(f, False)
             self.handleCmd('p/x 3735928559', collect_result=False, check=False)
         lldb.SBDebugger.Destroy(self.debugger)
         with open(self.out_filename, 'r') as f:
             self.assertIn('deadbeef', f.read())
 
     @add_test_categories(['pyapi'])
-    @skipIfWindows # FIXME pre-existing bug, should be fixed
-                   # when we delete the FILE* typemaps.
     def test_legacy_file_err_with_get(self):
         with open(self.out_filename, 'w') as f:
             self.debugger.SetErrorFileHandle(f, False)
             self.handleCmd('lolwut', check=False, collect_result=False)
             f2 = self.debugger.GetErrorFileHandle()
             f2.write('FOOBAR\n')
             f2.flush()
         lldb.SBDebugger.Destroy(self.debugger)
def test_legacy_file_error(self):
             self.handleCmd('lolwut', check=False, collect_result=False)
         with open(self.out_filename, 'r') as f:
             errors = f.read()
             self.assertTrue(re.search(r'error:.*lolwut', errors))
 
     @add_test_categories(['pyapi'])
     def test_sbfile_type_errors(self):
         sbf = lldb.SBFile()
-        self.assertRaises(TypeError, sbf.Write, None)
-        self.assertRaises(TypeError, sbf.Read, None)
-        self.assertRaises(TypeError, sbf.Read, b'this bytes is not mutable')
-        self.assertRaises(TypeError, sbf.Write, u"ham sandwich")
-        self.assertRaises(TypeError, sbf.Read, u"ham sandwich")
+        self.assertRaises(Exception, sbf.Write, None)
+        self.assertRaises(Exception, sbf.Read, None)
+        self.assertRaises(Exception, sbf.Read, b'this bytes is not mutable')
+        self.assertRaises(Exception, sbf.Write, u"ham sandwich")
+        self.assertRaises(Exception, sbf.Read, u"ham sandwich")
 
 
     @add_test_categories(['pyapi'])
     def test_sbfile_write_fileno(self):
         with open(self.out_filename, 'w') as f:
             sbf = lldb.SBFile(f.fileno(), "w", False)
             self.assertTrue(sbf.IsValid())
             e, n = sbf.Write(b'FOO\nBAR')
def test_identity(self):
             sbf.Close()
             self.assertTrue(f.closed)
 
         with open(self.out_filename, 'r') as f:
             self.assertEqual("foobar", f.read().strip())
 
 
     @add_test_categories(['pyapi'])
     def test_back_and_forth(self):
         with open(self.out_filename, 'w') as f:
             # at each step here we're borrowing the file, so we have to keep
             # them all alive until the end.
             sbf = lldb.SBFile.Create(f, borrow=True)
             def i(sbf):
                 for i in range(10):
                     f = sbf.GetFile()
                     yield f
                     sbf = lldb.SBFile.Create(f, borrow=True)
                     yield sbf
                     sbf.Write(str(i).encode('ascii') + b"\n")
             files = list(i(sbf))
         with open(self.out_filename, 'r') as f:
             self.assertEqual(list(range(10)), list(map(int, f.read().strip().split())))
 
+
+    @add_test_categories(['pyapi'])
+    def test_set_filehandle_none(self):
+        self.assertRaises(Exception, self.debugger.SetOutputFile, None)
+        self.assertRaises(Exception, self.debugger.SetOutputFile, "ham sandwich")
+        self.assertRaises(Exception, self.debugger.SetOutputFileHandle, "ham sandwich")
+        self.assertRaises(Exception, self.debugger.SetInputFile, None)
+        self.assertRaises(Exception, self.debugger.SetInputFile, "ham sandwich")
+        self.assertRaises(Exception, self.debugger.SetInputFileHandle, "ham sandwich")
+        self.assertRaises(Exception, self.debugger.SetErrorFile, None)
+        self.assertRaises(Exception, self.debugger.SetErrorFile, "ham sandwich")
+        self.assertRaises(Exception, self.debugger.SetErrorFileHandle, "ham sandwich")
+
+        with open(self.out_filename, 'w') as f:
+            status = self.debugger.SetOutputFile(f)
+            self.assertTrue(status.Success())
+            status = self.debugger.SetErrorFile(f)
+            self.assertTrue(status.Success())
+            self.debugger.SetOutputFileHandle(None, False)
+            self.debugger.SetErrorFileHandle(None, False)
+            sbf = self.debugger.GetOutputFile()
+            if sys.version_info.major >= 3:
+                # python 2 lacks PyFile_FromFd, so GetFile() will
+                # have to duplicate the file descriptor and make a FILE*
+                # in order to convert a NativeFile it back to a python
+                # file.
+                self.assertEqual(sbf.GetFile().fileno(), 1)
+            sbf = self.debugger.GetErrorFile()
+            if sys.version_info.major >= 3:
+                self.assertEqual(sbf.GetFile().fileno(), 2)
+        with open(self.out_filename, 'r') as f:
+            status = self.debugger.SetInputFile(f)
+            self.assertTrue(status.Success())
+            self.debugger.SetInputFileHandle(None, False)
+            sbf = self.debugger.GetInputFile()
+            if sys.version_info.major >= 3:
+                self.assertEqual(sbf.GetFile().fileno(), 0)

lldb/scripts/interface/SBDebugger.i

public:
     SetAsync (bool b);
 
     bool
     GetAsync ();
 
     void
     SkipLLDBInitFiles (bool b);
 
-    %feature("autodoc", "DEPRECATED, use SetInputFile");
-    void
-    SetInputFileHandle (FILE *f, bool transfer_ownership);
+    %pythoncode %{
+        def SetOutputFileHandle(self, file, transfer_ownership):
+            "DEPRECATED, use SetOutputFile"
+            if file is None:
+                import sys
+                file = sys.stdout
+            self.SetOutputFile(SBFile.Create(file, borrow=True))
 
-    %feature("autodoc", "DEPRECATED, use SetOutputFile");
-    void
-    SetOutputFileHandle (FILE *f, bool transfer_ownership);
+        def SetInputFileHandle(self, file, transfer_ownership):
+            "DEPRECATED, use SetInputFile"
+            if file is None:
+                import sys
+                file = sys.stdin
+            self.SetInputFile(SBFile.Create(file, borrow=True))
 
-    %feature("autodoc", "DEPRECATED, use SetErrorFile");
-    void
-    SetErrorFileHandle (FILE *f, bool transfer_ownership);
+        def SetErrorFileHandle(self, file, transfer_ownership):
+            "DEPRECATED, use SetErrorFile"
+            if file is None:
+                import sys
+                file = sys.stderr
+            self.SetErrorFile(SBFile.Create(file, borrow=True))
+    %}
 
-    %feature("autodoc", "DEPRECATED, use GetInputFile");
-    FILE *
-    GetInputFileHandle ();
 
-    %feature("autodoc", "DEPRECATED, use GetOutputFile");
-    FILE *
-    GetOutputFileHandle ();
+    %extend {
 
-    %feature("autodoc", "DEPRECATED, use GetErrorFile");
-    FILE *
-    GetErrorFileHandle ();
+        lldb::FileSP GetInputFileHandle() {
+            return self->GetInputFile().GetFile();
+        }
+
+        lldb::FileSP GetOutputFileHandle() {
+            return self->GetOutputFile().GetFile();
+        }
+
+        lldb::FileSP GetErrorFileHandle() {
+            return self->GetErrorFile().GetFile();
+        }
+    }
 
     SBError
     SetInputFile (SBFile file);
 
     SBError
     SetOutputFile (SBFile file);
 
     SBError