[mlir] Move PyConcreteType to header. NFC.

[jdd]

Summary

This allows out-of-tree users to derive PyConcreteType to bind custom
types.

The Type version of https://reviews.llvm.org/D101063/new/

[stellaraccident]

Works for me, but I will make the PSA that by expanding the public API in this way, we are heading in to really deep waters when it comes to linkage issues and bugs. We can run this forward a bit and see, but know that actually shipping a shared mlir python package in combination with other projects has a lot of sharp edges that will need resolution at some point.

[mikeurbach]

PSA noted. When I mentioned this on discourse, it seemed like this and the previous change for attributes were expansions we could get some mileage out of without totally going off the deep end. Are there immediate linkage concerns with deriving PyConcreteType out of tree, or just the general concerns we have been discussing?

[jdd]

Thanks @stellaraccident. So I assume you'd be opposed to making this a real public API by moving some of the python headers into 'mlir/include'? I was going to do that next since as of now, it's quite awkward to #include in out-of-tree code.

[stellaraccident]

In D101496#2724291, @mikeurbach wrote:

PSA noted. When I mentioned this on discourse, it seemed like this and the previous change for attributes were expansions we could get some mileage out of without totally going off the deep end. Are there immediate linkage concerns with deriving PyConcreteType out of tree, or just the general concerns we have been discussing?

Generic concerns from me:

• This does not scale beyond a single out of tree project controlling the entire Python environment by providing the mlir native libraries and its project libraries (i.e. circt). If another MLIR project wanted to co-exist, it would need to be built against the same installed MLIR libraries.
• The compatibility rules for pybind11 to interop at the C++ level between multiple shared-library extensions are arcane and hard to satisfy for things not intimately built together (i.e. basically the tuple of (compiler, standard library, compile options, etc) all have to match and are checked specifically. Penalty for violation is that the types at the Python level are actually forked (the binding is actually stored in a dict in the Python builtin module with a key hard-coded to the above parameters). This usually causes chaos to ensue and is not completely obvious.
• We still have the lingering TypeID linkage issue (I've been hunting it down for 6 months and haven't run it to ground completely yet). This will work one day and not the next, based on past experience. This bug unfortunately can happen when you obey all of the other rules. I have to assume we'll track it down eventually...
• Other projects who have gone down this path have eventually had to grow a platform specific loader stub which can adapt things at runtime and work around issues that come from distributing binaries.
• Those other projects have typically also had to embrace a deployment strategy that lets them pip install headers/dev libraries and rework end projects to build against that (which provides a bit of a source of truth to make sure everything is at the same revision/config options/etc). LLVM's source instability will hurt here since we have very limited and unpredictable compatibility windows between projects.

An answer that neatly side-steps this for projects like circt and npcomp, which merely want to use MLIR for their own ends, and have limited benefit from joining on a single shared installation, is to move the MLIR Python binding in a direction so that it can be *privately embedded* in such a project (i.e. as an circt.mlir private module vs a shared dependency on mlir). I've tried to keep the source tree in a rough shape to make such an evolution practical at some point, but it would be definite work.

[stellaraccident]

In D101496#2724341, @jdd wrote:

Thanks @stellaraccident. So I assume you'd be opposed to making this a real public API by moving some of the python headers into 'mlir/include'? I was going to do that next since as of now, it's quite awkward to #include in out-of-tree code.

I'm not opposed, but I am advising caution: the distance between doing this and real, unsolvable problems, is really short.

[ftynse]

@stellaraccident do you mind replicating your comments in https://llvm.discourse.group/t/python-bindings-for-out-of-tree-projects/3339 so we don't spread the discussion?

mlir/lib/Bindings/Python/IRModule.h

public:
   /// may be a pre-existing object. Ownership of the underlying MlirType
   /// is taken by calling this function.
   static PyType createFromCapsule(pybind11::object capsule);
 
 private:
   MlirType type;
 };
 
+/// CRTP base classes for Python types that subclass Type and should be
+/// castable from it (i.e. via something like IntegerType(t)).
+/// By default, type class hierarchies are one level deep (i.e. a
+/// concrete type class extends PyType); however, intermediate python-visible
+/// base classes can be modeled by specifying a BaseTy.
+template <typename DerivedTy, typename BaseTy = PyType>
+class PyConcreteType : public BaseTy {
+public:
+  // Derived classes must define statics for:
+  //   IsAFunctionTy isaFunction
+  //   const char *pyClassName
+  using ClassTy = pybind11::class_<DerivedTy, BaseTy>;
+  using IsAFunctionTy = bool (*)(MlirType);
+
+  PyConcreteType() = default;
+  PyConcreteType(PyMlirContextRef contextRef, MlirType t)
+      : BaseTy(std::move(contextRef), t) {}
+  PyConcreteType(PyType &orig)
+      : PyConcreteType(orig.getContext(), castFrom(orig)) {}
+
+  static MlirType castFrom(PyType &orig) {
+    if (!DerivedTy::isaFunction(orig)) {
+      auto origRepr = pybind11::repr(pybind11::cast(orig)).cast<std::string>();
+      throw SetPyError(PyExc_ValueError, llvm::Twine("Cannot cast type to ") +
+                                             DerivedTy::pyClassName +
+                                             " (from " + origRepr + ")");
+    }
+    return orig;
+  }
+
+  static void bind(pybind11::module &m) {
+    auto cls = ClassTy(m, DerivedTy::pyClassName);
+    cls.def(pybind11::init<PyType &>(), pybind11::keep_alive<0, 1>());
+    cls.def_static("isinstance", [](PyType &otherType) -> bool {
+      return DerivedTy::isaFunction(otherType);
+    });
+    DerivedTy::bindDerived(cls);
+  }
+
+  /// Implemented by derived classes to add methods to the Python subclass.
+  static void bindDerived(ClassTy &m) {}
+};
+
 /// Wrapper around the generic MlirValue.
 /// Values are managed completely by the operation that resulted in their
 /// definition. For op result value, this is the operation that defines the
 /// value. For block argument values, this is the operation that contains the
 /// block to which the value is an argument (blocks cannot be detached in Python
 /// bindings so such operation always exists).
 class PyValue {
 public:

mlir/lib/Bindings/Python/IRTypes.cpp

 namespace {
 
 /// Checks whether the given type is an integer or float type.
 static int mlirTypeIsAIntegerOrFloat(MlirType type) {
   return mlirTypeIsAInteger(type) || mlirTypeIsABF16(type) ||
          mlirTypeIsAF16(type) || mlirTypeIsAF32(type) || mlirTypeIsAF64(type);
 }
 
-/// CRTP base classes for Python types that subclass Type and should be
-/// castable from it (i.e. via something like IntegerType(t)).
-/// By default, type class hierarchies are one level deep (i.e. a
-/// concrete type class extends PyType); however, intermediate python-visible
-/// base classes can be modeled by specifying a BaseTy.
-template <typename DerivedTy, typename BaseTy = PyType>
-class PyConcreteType : public BaseTy {
-public:
-  // Derived classes must define statics for:
-  //   IsAFunctionTy isaFunction
-  //   const char *pyClassName
-  using ClassTy = py::class_<DerivedTy, BaseTy>;
-  using IsAFunctionTy = bool (*)(MlirType);
-
-  PyConcreteType() = default;
-  PyConcreteType(PyMlirContextRef contextRef, MlirType t)
-      : BaseTy(std::move(contextRef), t) {}
-  PyConcreteType(PyType &orig)
-      : PyConcreteType(orig.getContext(), castFrom(orig)) {}
-
-  static MlirType castFrom(PyType &orig) {
-    if (!DerivedTy::isaFunction(orig)) {
-      auto origRepr = py::repr(py::cast(orig)).cast<std::string>();
-      throw SetPyError(PyExc_ValueError, Twine("Cannot cast type to ") +
-                                             DerivedTy::pyClassName +
-                                             " (from " + origRepr + ")");
-    }
-    return orig;
-  }
-
-  static void bind(py::module &m) {
-    auto cls = ClassTy(m, DerivedTy::pyClassName);
-    cls.def(py::init<PyType &>(), py::keep_alive<0, 1>());
-    cls.def_static("isinstance", [](PyType &otherType) -> bool {
-      return DerivedTy::isaFunction(otherType);
-    });
-    DerivedTy::bindDerived(cls);
-  }
-
-  /// Implemented by derived classes to add methods to the Python subclass.
-  static void bindDerived(ClassTy &m) {}
-};
-
 class PyIntegerType : public PyConcreteType<PyIntegerType> {
 public:
   static constexpr IsAFunctionTy isaFunction = mlirTypeIsAInteger;
   static constexpr const char *pyClassName = "IntegerType";
   using PyConcreteType::PyConcreteType;
 
   static void bindDerived(ClassTy &c) {
     c.def_static(