Page MenuHomePhabricator

Remove libMLIRPublicAPI DSO.
ClosedPublic

Authored by stellaraccident on Jul 20 2021, 8:22 AM.

Details

Summary

libMLIRPublicAPI.so came into existence early when the Python and C-API were being co-developed because the Python extensions need a single DSO which exports the C-API to link against. It really should never have been exported as a mondo library in the first place, which has caused no end of problems in different linking modes, etc (i.e. the CAPI tests depended on it).

This patch does a mechanical move that:

  • Makes the C-API tests link directly to their respective libraries.
  • Creates a libMLIRPythonCAPI as part of the Python bindings which assemble to exact DSO that they need.

This has the effect that the C-API is no longer monolithic and can be subset and used piecemeal in a modular fashion, which is necessary for downstreams to only pay for what they use. There are additional, more fundamental changes planned for how the Python API is assembled which should make it more out of tree friendly, but this minimal first step is necessary to break the fragile dependency between the C-API and Python API.

Downstream actions required:

  • If using the C-API and linking against MLIRPublicAPI, you must instead link against its constituent components. As a reference, the Python API dependencies are in lib/Bindings/Python/CMakeLists.txt and approximate the full set of dependencies available.
  • If you have a Python API project that was previously linking against MLIRPublicAPI (i.e. to add its own C-API DSO), you will want to s/MLIRPublicAPI/MLIRPythonCAPI/ and all should be as it was. There are larger changes coming in this area but this part is incremental.

Diff Detail

Event Timeline

stellaraccident requested review of this revision.Jul 20 2021, 8:22 AM

Fix Windows build.

I'm not sure I understand the problem you're trying to fix here: having a single DSO exporting all of the C API seems valuable to me.

And I didn't quite get the problem with C unit-test linking to it? Is it because the C unit-test would embed another copy of libSupport from libMLIRPublicApi.so ?

It may very well be that libMLIRPublicApi.so should only link libMLIR.so and no other static archive than the C API bindings. Clients of libMLIRPublicApi.so that require any C++ code should get it from libMLIR.so as well.

The complexity is always when you want more fine grain control over the amount of code to include: people building specific compilers using MLIR are likely to at some point customize their own build and not use libMLIRPublicApi.so / libMLIR.so, but as a starting point and as a point of inter-op between projects it seems like a good setup?

mlir/cmake/modules/AddMLIR.cmake
185–186

I'm a bit puzzled bit this method now: what is the public_c_api_library remaining? Aren't you removing it?

What is the intended use the MLIR_PUBLIC_C_API_LIBS property now?

mlir/lib/Bindings/Python/CMakeLists.txt
52

Carrying back my comment from the Mingw "fix" last week: shouldn't ${_DEPS} be just libMLIR.so when it is used?

I'm not sure I understand the problem you're trying to fix here: having a single DSO exporting all of the C API seems valuable to me.

And I didn't quite get the problem with C unit-test linking to it? Is it because the C unit-test would embed another copy of libSupport from libMLIRPublicApi.so ?

There has been a running firefight in downstreams (i.e. circt is carrying a local patch that effectively neuters this) with respect to this. Having this kind of bottleneck where we force dynamic linking just because you want to use the C-API was a bad design and doesn't cover how we actually want to use it (i.e. statically link). It turns out that this whole setup is super unstable across platforms, and it turns out that everyone who is using it is feeling pain. Better to remove it and layer this like everything else instead of having it be its own weird thing.

It may very well be that libMLIRPublicApi.so should only link libMLIR.so and no other static archive than the C API bindings. Clients of libMLIRPublicApi.so that require any C++ code should get it from libMLIR.so as well.

I don't think we can say that. That may be a way that people want to use it, but it is very much not want we want to be doing for Python, which is the reason I created this DSO to begin with.

The complexity is always when you want more fine grain control over the amount of code to include: people building specific compilers using MLIR are likely to at some point customize their own build and not use libMLIRPublicApi.so / libMLIR.so, but as a starting point and as a point of inter-op between projects it seems like a good setup?

It hasn't worked out that way. I would be ok having a *normal* library named MLIRPublicAPI that folks can use as a convenience (ie. a static lib for static builds and a shared lib for shared builds) but we shouldn't be introducing a hard dependency on a DSO in this way.

This all might be easier to chat about f2f. I'll ping you.

mlir/cmake/modules/AddMLIR.cmake
185–186

Right - forgot to remove the property. I think this still has some value with respect to the visibility support. I'm planning to generalize that a bit in a followup, which may obviate the need for this function -- just not there yet.

mlir/lib/Bindings/Python/CMakeLists.txt
52

Carrying back my comment from the Mingw "fix" last week: shouldn't ${_DEPS} be just libMLIR.so when it is used?

Not for the Python API. We don't (and will not) use libMLIR for this use case. In the common deployment case, this is a statically linked DSO, per best practices for Python deployment.

mehdi_amini accepted this revision.Jul 20 2021, 11:07 AM
This revision is now accepted and ready to land.Jul 20 2021, 11:07 AM

Fix windows build better.

Remove global property of CAPI libs.

Thanks for the review and conversation, Mehdi. Summarizing:

  • We need to be switching to a mode where the Python APIs compose for building Python based systems, and those need to be based on static linkage in the way starting to be set down here.
  • The MLIR C-API tests depending on fine-grained libs helps keep things tidy and, since they are written in C, there is already an implicit barrier keeping them depending purely on the C-API (which is the layering we want).
  • Nothing in this patch is opposed to having a single "Public C-API" library entry point for default use in the future. It is just that with the above two, the only in-tree (and known required) uses of libMLIRPublicAPI.so are now gone, and it is better to remove things that lack a reason for existing. It can be added back later if useful, and at that point, it should just be a normal library (not a forced shared-library).

Fix windows visibility (verified locally).

Fix python API on windows (some pre-existing test failures but generally works).

This revision was automatically updated to reflect the committed changes.