Page MenuHomePhabricator

Move MLIR python sources to mlir/python.
ClosedPublic

Authored by stellaraccident on Apr 28 2021, 3:12 PM.

Details

Summary
  • NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends).
  • Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup.

Diff Detail

Event Timeline

stellaraccident requested review of this revision.Apr 28 2021, 3:12 PM
stellaraccident edited reviewers, added: ftynse, mehdi_amini; removed: herhut.Apr 29 2021, 2:16 PM
stellaraccident retitled this revision from DRAFT: Move MLIR python sources to mlir/python. to Move MLIR python sources to mlir/python..Apr 29 2021, 2:17 PM
stellaraccident edited the summary of this revision. (Show Details)
stellaraccident edited the summary of this revision. (Show Details)

Rebase.

Seems mechanical, LGTM on the principle.

Seems mechanical, LGTM on the principle.

Yes, mostly mechanical with CMake fixups that became apparent when things were separated.

Tests seem to fail for me locally

: 'RUN: at line 1';   /usr/bin/python3.9 /home/zinenko/llvm-project/mlir/test/python/ir/value.py | /home/zinenko/llvm-project/build2/bin/FileCheck /home/zinenko/llvm-project/mlir/test/python/ir/value.py
--
Exit Code: 2

Command Output (stderr):
--
Traceback (most recent call last):
  File "/home/zinenko/llvm-project/build2/python/mlir/__init__.py", line 23, in <module>
    import _mlir_libs
ModuleNotFoundError: No module named '_mlir_libs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zinenko/llvm-project/build2/python/mlir/_cext_loader.py", line 12, in <module>
    import _mlir_libs
ModuleNotFoundError: No module named '_mlir_libs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zinenko/llvm-project/mlir/test/python/ir/value.py", line 4, in <module>
    from mlir.ir import *
  File "/home/zinenko/llvm-project/build2/python/mlir/__init__.py", line 28, in <module>
    from ._dlloader import preload_dependency as _preload_dependency
  File "/home/zinenko/llvm-project/build2/python/mlir/_dlloader.py", line 6, in <module>
    import platform
  File "/usr/lib/python3.9/platform.py", line 117, in <module>
    import re
  File "/usr/lib/python3.9/re.py", line 124, in <module>
    import enum
  File "/usr/lib/python3.9/enum.py", line 2, in <module>
    from types import MappingProxyType, DynamicClassAttribute
  File "/home/zinenko/llvm-project/mlir/test/python/ir/types.py", line 4, in <module>
    from mlir.ir import *
  File "/home/zinenko/llvm-project/build2/python/mlir/ir.py", line 6, in <module>
    from ._cext_loader import _reexport_cext
  File "/home/zinenko/llvm-project/build2/python/mlir/_cext_loader.py", line 17, in <module>
    from ._dlloader import preload_dependency as _preload_dependency
ImportError: cannot import name 'preload_dependency' from partially initialized module 'mlir._dlloader' (most likely due to a circular import) (/home/zinenko/llvm-project/build2/python/mlir/_dlloader.py)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /home/zinenko/llvm-project/build2/bin/FileCheck /home/zinenko/llvm-project/mlir/test/python/ir/value.py

--

********************
********************
Failed Tests (15):
  MLIR :: python/ir/affine_expr.py
  MLIR :: python/ir/affine_map.py
  MLIR :: python/ir/array_attributes.py
  MLIR :: python/ir/attributes.py
  MLIR :: python/ir/context_lifecycle.py
  MLIR :: python/ir/context_managers.py
  MLIR :: python/ir/debug.py
  MLIR :: python/ir/dialects.py
  MLIR :: python/ir/insertion_point.py
  MLIR :: python/ir/integer_set.py
  MLIR :: python/ir/location.py
  MLIR :: python/ir/module.py
  MLIR :: python/ir/operation.py
  MLIR :: python/ir/types.py
  MLIR :: python/ir/value.py

Tests seem to fail for me locally

: 'RUN: at line 1';   /usr/bin/python3.9 /home/zinenko/llvm-project/mlir/test/python/ir/value.py | /home/zinenko/llvm-project/build2/bin/FileCheck /home/zinenko/llvm-project/mlir/test/python/ir/value.py
--
Exit Code: 2

Command Output (stderr):
--
Traceback (most recent call last):
  File "/home/zinenko/llvm-project/build2/python/mlir/__init__.py", line 23, in <module>
    import _mlir_libs
ModuleNotFoundError: No module named '_mlir_libs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zinenko/llvm-project/build2/python/mlir/_cext_loader.py", line 12, in <module>
    import _mlir_libs
ModuleNotFoundError: No module named '_mlir_libs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zinenko/llvm-project/mlir/test/python/ir/value.py", line 4, in <module>
    from mlir.ir import *
  File "/home/zinenko/llvm-project/build2/python/mlir/__init__.py", line 28, in <module>
    from ._dlloader import preload_dependency as _preload_dependency
  File "/home/zinenko/llvm-project/build2/python/mlir/_dlloader.py", line 6, in <module>
    import platform
  File "/usr/lib/python3.9/platform.py", line 117, in <module>
    import re
  File "/usr/lib/python3.9/re.py", line 124, in <module>
    import enum
  File "/usr/lib/python3.9/enum.py", line 2, in <module>
    from types import MappingProxyType, DynamicClassAttribute
  File "/home/zinenko/llvm-project/mlir/test/python/ir/types.py", line 4, in <module>
    from mlir.ir import *
  File "/home/zinenko/llvm-project/build2/python/mlir/ir.py", line 6, in <module>
    from ._cext_loader import _reexport_cext
  File "/home/zinenko/llvm-project/build2/python/mlir/_cext_loader.py", line 17, in <module>
    from ._dlloader import preload_dependency as _preload_dependency
ImportError: cannot import name 'preload_dependency' from partially initialized module 'mlir._dlloader' (most likely due to a circular import) (/home/zinenko/llvm-project/build2/python/mlir/_dlloader.py)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /home/zinenko/llvm-project/build2/bin/FileCheck /home/zinenko/llvm-project/mlir/test/python/ir/value.py

--

********************
********************
Failed Tests (15):
  MLIR :: python/ir/affine_expr.py
  MLIR :: python/ir/affine_map.py
  MLIR :: python/ir/array_attributes.py
  MLIR :: python/ir/attributes.py
  MLIR :: python/ir/context_lifecycle.py
  MLIR :: python/ir/context_managers.py
  MLIR :: python/ir/debug.py
  MLIR :: python/ir/dialects.py
  MLIR :: python/ir/insertion_point.py
  MLIR :: python/ir/integer_set.py
  MLIR :: python/ir/location.py
  MLIR :: python/ir/module.py
  MLIR :: python/ir/operation.py
  MLIR :: python/ir/types.py
  MLIR :: python/ir/value.py

Tried to ping you on discord. Included here as well:

Regarding https://reviews.llvm.org/D101493, were you starting from a pre-existing build directory? (I had blown mine away and ninja check-mlir from scratch works)
Just verified again from a fresh build and tests pass.
Notably: this file should not exist (it was deleted quite some time ago): /home/zinenko/llvm-project/build2/python/mlir/__init__.py

Tried to ping you on discord. Included here as well:

Sorry, I don't follow discord. Any synchronous discussion media basically shuts me off because of nine-hour time zone difference with most of the people. By the time I see the message, the discussion has usually ended or diverged and any reaction to previous messages would be largely off topic.

Regarding https://reviews.llvm.org/D101493, were you starting from a pre-existing build directory?

Yes, I was starting from a pre-existing build directory, but there seems to be another problem even with a clean directory.

FAIL: MLIR :: python/ir/value.py (766 of 954)
******************** TEST 'MLIR :: python/ir/value.py' FAILED ********************
Script:
--
: 'RUN: at line 1';   /usr/bin/python3.9 /home/zinenko/llvm-project/mlir/test/python/ir/value.py | /home/zinenko/llvm-project/build/bin/FileCheck /home/zinenko/llvm-project/mlir/test/python/ir/value.py
--
Exit Code: 1

Command Output (stderr):
--
Traceback (most recent call last):
  File "/home/zinenko/llvm-project/build/python/mlir/_cext_loader.py", line 12, in <module>
    import _mlir_libs
ModuleNotFoundError: No module named '_mlir_libs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zinenko/llvm-project/mlir/test/python/ir/value.py", line 4, in <module>
    from mlir.ir import *
  File "/home/zinenko/llvm-project/build/python/mlir/ir.py", line 6, in <module>
    from ._cext_loader import _reexport_cext
  File "/home/zinenko/llvm-project/build/python/mlir/_cext_loader.py", line 17, in <module>
    from ._dlloader import preload_dependency as _preload_dependency
  File "/home/zinenko/llvm-project/build/python/mlir/_dlloader.py", line 6, in <module>
    import platform
  File "/usr/lib/python3.9/platform.py", line 117, in <module>
    import re
  File "/usr/lib/python3.9/re.py", line 124, in <module>
    import enum
  File "/usr/lib/python3.9/enum.py", line 2, in <module>
    from types import MappingProxyType, DynamicClassAttribute
  File "/home/zinenko/llvm-project/mlir/test/python/ir/types.py", line 25, in <module>
    run(testParsePrint)
  File "/home/zinenko/llvm-project/mlir/test/python/ir/types.py", line 8, in run
    f()
  File "/home/zinenko/llvm-project/mlir/test/python/ir/types.py", line 15, in testParsePrint
    ctx = Context()
NameError: name 'Context' is not defined

This particular problem seems to come from the name clash between the types.py test and the types.py distributed with Python. It's strange that you don't hit the problem.

bondhugula requested changes to this revision.Apr 30 2021, 11:27 PM
bondhugula added a subscriber: bondhugula.

I think you are missing an update to docs/Bindings/Python.md? (The paths there have to change.)

This revision now requires changes to proceed.Apr 30 2021, 11:27 PM

Address comments and rebase.

I think you are missing an update to docs/Bindings/Python.md? (The paths there have to change.)

Updated.

This particular problem seems to come from the name clash between the types.py test and the types.py distributed with Python. It's strange that you don't hit the problem.

Strange. In lieu of debugging, I just renamed that to builtin_types.py.

Alex: Made a change that might appease your machine. Let me know.

ftynse accepted this revision.May 3 2021, 10:44 AM

I tried that myself before with renaming (back) to ir_types and it worked, so it's okay.

bondhugula resigned from this revision.May 3 2021, 10:46 AM

Thanks for updating the docs!

This revision is now accepted and ready to land.May 3 2021, 10:46 AM
This revision was landed with ongoing or failed builds.May 3 2021, 11:37 AM
This revision was automatically updated to reflect the committed changes.