This is an archive of the discontinued LLVM Phabricator instance.

[mlir][python] Add pyi stub files to enable auto completion.
ClosedPublic

Authored by stellaraccident on Nov 28 2021, 2:16 PM.

Details

Summary

There is no completely automated facility for generating stubs that are both accurate and comprehensive for native modules. After some experimentation, I found that MyPy's stubgen does the best at generating correct stubs with a few caveats that are relatively easy to fix:

  • Some types resolve to cross module symbols incorrectly.
  • staticmethod and classmethod signatures seem to always be completely generic and need to be manually provided.
  • It does not generate an all which, from testing, causes namespace pollution to be visible to IDE code completion.

As a first step, I did the following:

  • Ran stubgen for _mlir.ir, _mlir.passmanager, and _mlirExecutionEngine.
  • Manually looked for all instances where unnamed arguments were being emitted (i.e. as 'arg0', etc) and updated the C++ side to include names (and re-ran stubgen to get a good initial state).
  • Made/noted a few structural changes to each pyi file to make it minimally functional.
  • Added the pyi files to the CMake rules so they are installed and visible.

To test, I added a .env file to the root of the project with PYTHONPATH=... set as per instructions. Then reload the developer window (in VsCode) and verify that completion works for various changes to test cases.

There are still a number of overly generic signatures, but I want to check in this low-touch baseline before iterating on more ambiguous changes. This is already a big improvement.

Diff Detail

Event Timeline

stellaraccident requested review of this revision.Nov 28 2021, 2:16 PM

Nice! It's a shame that static methods are not handled, can we hack something in the generator?

More generally, what's the plan for maintaining these? It feels almost easier to just require manual changes every time a binding method is added/modified, same as we would do for C++ .h/.cpp.

mlir/python/mlir/_mlir_libs/_mlir/__init__.pyi
8–9

Still arg0 here.

mlir/python/mlir/_mlir_libs/_mlir/ir.pyi
7

Do you remember which signatures? Could be nice to put a comment next to them to simplify future updates.

Nice! It's a shame that static methods are not handled, can we hack something in the generator?

I wasn't able to figure out out. Pybind11-stubgen with a bunch of hacks was doing marginally better but had so many correctness issues that I switched to the vanilla mypy version and decided to just do manual updates in a follow-up. There aren't that many.

More generally, what's the plan for maintaining these? It feels almost easier to just require manual changes every time a binding method is added/modified, same as we would do for C++ .h/.cpp.

Yeah, that's what I was thinking: maybe do a couple of iterations on the generated version and then it is manual. Updating a few signatures as making changes isn't very hard. If making a big API change, you can always rerun the generator and then copy paste what you need and revert.

ftynse accepted this revision.Nov 29 2021, 1:18 PM
This revision is now accepted and ready to land.Nov 29 2021, 1:18 PM
stellaraccident marked 2 inline comments as done.Nov 29 2021, 8:00 PM
stellaraccident added inline comments.
mlir/python/mlir/_mlir_libs/_mlir/ir.pyi
7

I re-ran it and looked at diffs - it was pretty nit picky changes that I was having trouble summarizing. I think this is a one way trap door and if someone runs it, looking at graphical diff is the best way to spot things.

This revision was automatically updated to reflect the committed changes.
stellaraccident marked an inline comment as done.