Page MenuHomePhabricator

[mlir][python] Reorganize MLIR python into namespace packages.
ClosedPublic

Authored by stellaraccident on Mar 5 2021, 6:13 PM.

Details

Summary
  • Only leaf packages are non-namespace packages. This allows most of the top levels to be split into different directories or deployment packages. In the previous state, the presence of __init__.py files at each level meant that the entire tree could only ever exist in one physical directory on the path.
  • This changes the API usage slightly: import mlir will no longer do a deep import of mlir.ir, etc. This may necessitate some client code changes.
  • Dialect gen code was restructured so that the user is responsible for providing the my_dialect.py file, which then must import its peer _my_dialect_ops_gen. This gives complete control of the dialect namespace to the user instead of to tablegen code, allowing further dialect-specific python APIs.
  • Correspondingly, the previous extension modules _my_dialect.py are now _my_dialect_ops_ext.py.
  • Now that the linalg namespace is open, moved the linalg_opdsl tool into it.
  • This may require some corresponding downstream adjustments to npcomp, circt, et al:
    • Probably some shallow imports need to be converted to deep imports (i.e. not import mlir brings in the world).
    • Each tablegen generated dialect now needs an explicit foo.py which does a from ._foo_ops_gen import *. This is similar to the way that generated code operates in the C++ world.
    • If providing dialect op extensions, those need to be moved from _foo.py -> _foo_ops_ext.py.

Diff Detail

Event Timeline

stellaraccident created this revision.Mar 5 2021, 6:13 PM
stellaraccident requested review of this revision.Mar 5 2021, 6:13 PM

Remove cmake cache errantly committed.

Not sure that I am qualified to accept this revision, but I do approve in general. The points in the summary make sense and sound good to me.

We haven't started using the dialect binding generators in CIRCT, but I will be looking at that this week. If this is the direction things are going, I will make sure to keep CIRCT on top of the latest here.

mehdi_amini accepted this revision.Mar 8 2021, 8:54 PM

Sweet!

mlir/lib/Bindings/Python/mlir/_cext_loader.py
66

Nice! I was actually wondering why we were doing this when I added my modules there, but punted on digging.

This revision is now accepted and ready to land.Mar 8 2021, 8:54 PM
mlir/lib/Bindings/Python/mlir/_cext_loader.py
66

Yeah, just cruft. When all of this started, there was only the "ir" package! I had forgotten that we had this in there like this.